(Author: Roland Rashleigh-Berry
Date: 04 Apr 2011)
Introduction
If you skipped reading about the "production scripts" then that is not
a problem, but you should be aware of three of them, which are "sasb",
"sc" and "lis2ps". You should always use "sasb" to
run your sas program and this should always be done in batch. You can then
use "sc" to scan your log for errors if any were detected by "sasb".
You should always use "lis2ps" to print your output if your output
was created with the reporting system.
To help you understand the purposes of the utility scripts, they will
be split into sections. That way you will get an idea of why
they were written and hopefully may give you an idea of when a script might
be useful in a new setting.
(When you use your browser to view the scripts, you will probably
see a first line like this "# !/bin/bash #". If you view the same script
as "Source", from the "View" pull-down menu, you will see two lines instead.
The first is "# !/bin/bash" and the second is "#<pre><b>". I have
inserted this second line so that your browser can correctly display the
script from that point on. The "<pre>" tells the browser that what follows
is "pre-formatted" and the "<b>" tells it to display it as "bold". This
second line will not affect the operation of the script because any line
(except the first) that starts with a "#" is treated as a comment).
Batch development
Some of the scripts you use for developing your code in batch were introduced
on this page. You should read that page before
reading any further in this section. To briefly recap, you can set up a
Unix variable named "prog" (that you export) and you set this to the name
of the program you are working on (no extension). This makes a number of
scripts available for you to use that emulates an interactive sas session.
You have "pgm" to edit your code, "log" to look at the log, "lst" to look
at the ".lst" file and "lis" to look at the ".lis" or ".lis*" output files.
These scripts can be viewed below.
pgm log lst lis
There are a few scripts very much like these for when you are doing
validation programming. Your validation program name should start with
a "v" and the rest of the name should match the program you are validating.
If you stick with this naming convention then you can use some script to
look at the "original" listing, program and titles member. These are below.
opgm olis otitles
SAS program and script headers
There are a number of scripts available to create a sas program header
or a script header but you should only use these scripts if the headers
they create comply with your site standards. There are usually no site
standards for script headers but there is likely to be one for sas program
headers and sas macro headers. For the scripts there is "shdr" and "sohdr".
The first is for simple scripts and the second is for scripts that use
options. The second one will ask you a lot of questions about what options
your script uses and what default values apply so you should not use it
unless you know what you are doing. These are below.
shdr sohdr
For sas programs and sas macros, your site will already have its own
standards so you should not use the following scripts unless the headers
they create comply with site standards. Even if this is not the case,
these scripts could be used as the basis for creating new scripts which
comply with site standards. The first script, "ohdr", is for an "old" header
that is a box style with characters on the right to form a right boundary.
The scripts that follow it are for headers that do not have a right boundary.
"hdr" is for an ordinary sas program header, "mhdr" is for a macro header
and "lmhdr" is for a local macro header (i.e. a study-level macro). These
scripts try to put in the drugname, protocol and increment by reading parts
of the current directory name. They will probably need amending to make
sure that this is done correctly.
ohdr hdr mhdr lmhdr
All these scripts try to be polite by putting in your name and not just
your Unix userid. It uses the following script for this. You can use it
on its own to match a userid to a person's name.
getname
File renaming
There is a utility called "grename" that you can use to rename your
files. It will only work on files that you own. Sometimes file naming conventions
will change while you are working on a study. You might have many programs
and titles members that need to be renamed, so you can use this utility
to do it. It would be wise to make a backup of all the files that may be
affected before you run it. This is below.
grename
Once you have renamed your files you will have a new problem. For titles
files, the program name inside must match the file name. To find out which
are wrong you can use the following script. Note the "-f" option that you
can use to fix what is wrong.
mybadtitlesprogs
If your sas program headers used the same convention as that created
by "hdr", then there are more scripts you can use to fix wrong information
in your program headers. These scripts are below. Again, note the "-f"
option (which should have no effect if your headers do not match that convention).
mybaddrug mybadprot mybadinc
String search and replace
Most Unix installations have a search and replace utility but you have
got to be careful with it. There was one written especially for Spectre
called "greplace" which is a lot safer in that it only acts on files that
you own and will not update files where the target string is not found.
You
might have "greplace" where you work but it might not be the same one so
be careful.
greplace
antigrep
You have hopefully heard of "grep" and used it to search your files for
a character string or regular expression. But sometimes you need to know
which of your files do not contain a string. Suppose you were working
on a study, you might have copied programs for another study and used the
code as the basis of your new program. But did you update the information
in the header? Is the new protocol in there? You can find out with "antigrep".
antigrep
"fsv" scripts
Since the emphasis in Spectre is to develop programs in batch, you might
think you have been robbed of the ease of accessing and browsing datasets
from your interactive sas session. Luckily, this is not the case. Everything
you need has been covered by a script. That and more. The "fsv" scripts
are for using "fsview" to browse sas datasets and there are a few of these.
They have to be run from the same directory in which the datasets are stored.
"fsv" will browse the dataset with user formats applied, "fsvraw" will
browse a dataset with no user formats applied, "fsvdc" will show both unformatted
values along with their formatted values. "fsvacct" will allow you to browse
a dataset with fields from the acct dataset merged in that you specify.
fsv fsvraw fsvdc fsvacct
"contents" scripts
You might have the need to look at the contents of a sas dataset while
you are using batch. There are two scripts for this, an ordinary version
and an "l" ("long") version. When using these scripts, you might want to
pipe the output to "grep" to select only what you are interested in seeing.
contents contentsl
summary
This is for running a "proc summary" on a dataset in the current directory
and displaying the output in the terminal window. I use this a lot for
identifying observations with strange or incorrect values which I then
copy and paste into an email. You can apply a "where clause" to the dataset
and to the output.
summary
Format viewing
Sometimes you need to know what sas formats exist and what these formats
contain. Sometimes these will be central formats and drug formats. Sometimes
they will be for formats held in a local directory. There
is no need to start an interactive sas session as there are scripts to
do this. The first two might need amending to make sure they can find the
central and drug formats. The last two, for local formats, should not have
a problem.
showfmts showfmt showlfmts showlfmt
Printing other paginated output
You should print out all reports created using the reporting system with
"lis2ps" but if you need to print other paginated reports, which
are nothing to do with the reporting system, then you can use the utilities
"a4ps" or "usps" (for A4 paper or US Letter paper). These
work like "lis2ps" does but do not look in the titles dataset. To send
to a printer you use the "-p" option and follow it by the printer name,
just like you do with "lis2ps". Please bear in mind that this is for
paginated reports and that does not include sas programs or logs.
In most cases it will be able to produce a report but not always. You can
adjust margins using these scripts.
a4ps usps
Ad-hoc report line size and page size
If you are working completely outside the reporting system, creating ad-hoc
reports, you might want the layout to be similar to that coming out of
Spectre but you will need to know what linesize and pagesize values to
use. There are two scripts for this. You can specify the margins and it
will give you the linesize and pagesize values for a few different font
sizes.
a4lsps uslsps
orphans
"Orphans" are files that look as though they have been created by the reporting
system but there is no entry in the titles dataset for them. Perhaps some
programs have been renamed, leaving behind old .lis, .chk and .lst files.
You can list these out for everybody or list out just your own using these
two scripts.
orphans myorphans
rescue
Sometimes you might accidentally delete one of your sas programs (I deleted
a whole library of them once). Rather than wait for them to be restored
by the Unix administrator (which is the best option if you are sure there
are up-to-date backups) then you might like to use "rescue" instead. You
run this on your program log. If you use it, you will have to carefully
check the code it creates. It will give any restored program the suffix
"_rescue".
rescue
timestamp
You can create a timestamp that gets written to your home directory using
the "timestamp" script. After you have done this, you can get a
list of all the .lis, programs and titles files you have created since
then to remind you of what you have been working on.
timestamp mynewprogs mynewtitles mynewlis
myfiles
If you want a list of your own files you can use the "myfiles" script but
note that it uses the owner as seen by the "ls -l" command. You can use
this to give you a list of files owned by other users as well. This is
a very useful script that you will use often if you copy your files from
one directory to another.
myfiles
You have to know a little about Unix commands to get the best out of
this utility. Suppose you wanted to copy all your titles files from one
directory to another, you could first see the list with this command:
myfiles *.titles
...and then you could copy all these to another directory using a command
like this:
cp -p $(myfiles *.titles) /a/new/directory
wt
"wt" is short for "which titles file". If you know a table or appendix
reference number (that is the number that follows "Table" or "Appendix")
then you can find out the titles member and hence the program name using
the "wt" script. I use it a lot and that is why I made it
only two characters long. It saves typing.
wt
whosgot
Sometimes you will be trying to create a new sas dataset but you will get
an error message saying that someone else has a "lock" on it. To find out
the userid of this person then use the "whosgot" script. You can use the
script "getname" to match the userid to the person's name.
whosgot
makemyrun
There is an equivalent to the production "makerun" script that you can
use. That is "makemyrun". It creates shell scripts with your userid as
part of the name so you can run your own programs.
makemyrun
Cygwin users
I have Cygwin on my PC at home. Most of the scripts on this web site should
work with Cygwin but not the ones that call SAS, assuming you have
SAS installed on Windows. The scripts that call SAS rely on the use of
the option "-stdio" which is not available with Windows SAS. Also Windows
SAS will complain about Unix-style file names. So if you are trying to
install this reporting system on Cygwin and get it working then you are
going to be disappointed. But if you are determined to get some of the
scripts that call SAS working, then I have converted a few to use Cygwin
that you can maybe use as a guide. One more thing - if you have SAS Learning
Edition installed then you will not be able to use "proc printto" as this
very simple procedure was not included in the package. One of the scripts
I converted to run on Cygwin was "summary" (below) but although I converted
it it still will not work due to its reliance on "proc printto".
crindex_win contents_win contentsl_win summary_win
An "I wrote a script today" story
I mentioned somewhere else that it is often quicker to write a script to
help you with a repetitive task than it is to do it by hand. That's if
you know how to write scripts, of course. I am a self-taught script writer
but I think if I had been sent on courses then I would have learned five
times faster and my script writing skills would be better than they are
now. But I manage and can write scripts that work in a fairly short time.
I spent weeks writing scripts trying to get bookmarked PDFs to work so
I have had plenty of practice. So today I wrote a script and I will tell
you the story of why and how. Are you sitting comfortably? Right, then
I'll begin:
I am working on a study using Spectre, so of course I have hundreds
of programs producing output. Somebody points out that all tables and listings
should have a solid footnote at the bottom of the page, whether there is
footnote information or not, and some of my outputs lack that solid footnote.
Looking through these hundreds of outputs and fixing them will take all
day. So I know I have a script called "pages" that I can use to
extract the first page of output. The solid footnote line might not be
near the bottom of the page but will be in the last ten non-blank lines.
I know I can drop blank lines using "grep -v". I know I can pick
out the last ten lines using "tail -10". I know there is a version
of "grep" that can be used in "quiet" mode that will just tell me whether
it found what I was looking for or not from the return code. I can't remember
where it is, but "man grep" will tell me. I can't remember how to
loop through a list of input files but I can do a search on "for file
in" in the script library and get the correct syntax. Once I know these
things, I decide to call the new utility "nofootline" and the "shdr"
script will write a nice script header for me to start with. So I do this
and after fifteen minutes of messing about, I have my script and it seems
to work OK. Not only does it save me time but it is applicable to other
studies as well so other people can usefully use it. I have saved time
for myself and potentially saved time for other people in the future. This
script can now become a QC tool. This is "Unix" for you. Once you
reach this stage of being able to "think scripts" and "write scripts" you
will not want to work on a different platform. Here is the script I wrote.
nofootline
Yet another "I wrote a script today" story
Five days after I wrote the above script, an announcement went round for
another study that all occurences of "study medication" in any mixture
of case needed to be changed to "study drug" in all the output. So what
was needed was to search the ".lis" files for this literal string (in any
mixture of case) and find out the owners of the corresponding ".titles"
members (since the owner of the ".lis" file might be the person who ran
the program suite and not the programmer who wrote the code). It took me
less than fifteen minutes to write the script and you can see the script
below. But this brings me to an important point and that is - Spectre is
all about high efficiency with programmers maybe handling hundreds of programs
at a time and so Spectre needs the Unix/Linux platform to achieve high
efficiency. There is no point in trying to port Spectre to another
platform, though it is possible, since high efficiency would not be possible
on a non Unix/Linux platform.
lis2owner
Conclusion
You have been introduced to a number of utility scripts on this page. There
are others not shown here. To see the full list you can always use the
command "showme scripts" and then you can browse the headers of
any of these scripts by using "showme xyz" (where "xyz" is the script name).
If you know these scripts exist then they should be able to speed your
work. If "showme scripts" is not set up to work then you can see what it
would produce below.
@index.txt
Use the "Back" button of your browser
to return to the previous page.