(Author: Roland
Rashleigh-Berry
Date: 02 Nov 2006)
Introduction
If you skipped reading about the "production scripts" then that is not
a problem, but you should be aware of three of them, which are "sasb",
"sc" and "lis2ps". You should always use "sasb" to
run your sas program and this should always be done in batch. You can
then
use "sc" to scan your log for errors if any were detected by
"sasb".
You should always use "lis2ps" to print your output if your
output
was created with the reporting system.
To help you understand the purposes of the utility scripts, they
will
be split into sections. That way you will get an idea of why
they were written and hopefully may give you an idea of when a script
might
be useful in a new setting.
(When you use your browser to view the scripts, you will probably
see a first line like this "# !/bin/bash #". If you view the same
script
as "Source", from the "View" pull-down menu, you will see two lines
instead.
The first is "# !/bin/bash" and the second is "#<pre><b>".
I have
inserted this second line so that your browser can correctly display
the
script from that point on. The "<pre>" tells the browser that
what follows
is "pre-formatted" and the "<b>" tells it to display it as
"bold". This
second line will not affect the operation of the script because any
line
(except the first) that starts with a "#" is treated as a comment).
Batch development
Some of the scripts you use for developing your code in batch were
introduced
on this page. You should read that page
before
reading any further in this section. To briefly recap, you can set up a
Unix variable named "prog" (that you export) and you set this to the
name
of the program you are working on (no extension). This makes a number
of
scripts available for you to use that emulates an interactive sas
session.
You have "pgm" to edit your code, "log" to look at the log, "lst" to
look
at the ".lst" file and "lis" to look at the ".lis" or ".lis*" output
files.
These scripts can be viewed below.
pgm log lst lis
There are a few scripts very much like these for when you are doing
validation programming. Your validation program name should start with
a "v" and the rest of the name should match the program you are
validating.
If you stick with this naming convention then you can use some script
to
look at the "original" listing, program and titles member. These are
below.
opgm olis otitles
SAS program and script headers
There are a number of scripts available to create a sas program header
or a script header but you should only use these scripts if the headers
they create comply with your site standards. There are usually no site
standards for script headers but there is likely to be one for sas
program
headers and sas macro headers. For the scripts there is "shdr" and
"sohdr".
The first is for simple scripts and the second is for scripts that use
options. The second one will ask you a lot of questions about what
options
your script uses and what default values apply so you should not use it
unless you know what you are doing. These are below.
shdr sohdr
For sas programs and sas macros, your site will already have its own
standards so you should not use the following scripts unless the
headers
they create comply with site standards. Even if this is not the
case,
these scripts could be used as the basis for creating new scripts which
comply with site standards. The first script, "ohdr", is for an "old"
header
that is a box style with characters on the right to form a right
boundary.
The scripts that follow it are for headers that do not have a right
boundary.
"hdr" is for an ordinary sas program header, "mhdr" is for a macro
header
and "lmhdr" is for a local macro header (i.e. a study-level macro).
These
scripts try to put in the drugname, protocol and increment by reading
parts
of the current directory name. They will probably need amending to make
sure that this is done correctly.
ohdr hdr mhdr lmhdr
All these scripts try to be polite by putting in your name and not
just
your Unix userid. It uses the following script for this. You can use it
on its own to match a userid to a person's name.
getname
File renaming
There is a utility called "grename" that you can use to rename
your
files. It will only work on files that you own. Sometimes file naming
conventions
will change while you are working on a study. You might have many
programs
and titles members that need to be renamed, so you can use this utility
to do it. It would be wise to make a backup of all the files that may
be
affected before you run it. This is below.
grename
Once you have renamed your files you will have a new problem. For
titles
files, the program name inside must match the file name. To find out
which
are wrong you can use the following script. Note the "-f" option that
you
can use to fix what is wrong.
mybadtitlesprogs
If your sas program headers used the same convention as that created
by "hdr", then there are more scripts you can use to fix wrong
information
in your program headers. These scripts are below. Again, note the "-f"
option (which should have no effect if your headers do not match that
convention).
mybaddrug mybadprot mybadinc
String search and replace
Most Unix installations have a search and replace utility but you have
got to be careful with it. There was one written especially for Spectre
called "greplace" which is a lot safer in that it only acts on files
that
you own and will not update files where the target string is not found.
You
might have "greplace" where you work but it might not be the same one
so
be careful.
greplace
antigrep
You have hopefully heard of "grep" and used it to search your files for
a character string or regular expression. But sometimes you need to
know
which of your files do not contain a string. Suppose you were
working
on a study, you might have copied programs for another study and used
the
code as the basis of your new program. But did you update the
information
in the header? Is the new protocol in there? You can find out with
"antigrep".
antigrep
"fsv" scripts
Since the emphasis in Spectre is to develop programs in batch, you
might
think you have been robbed of the ease of accessing and browsing
datasets
from your interactive sas session. Luckily, this is not the case.
Everything
you need has been covered by a script. That and more. The "fsv" scripts
are for using "fsview" to browse sas datasets and there are a few of
these.
They have to be run from the same directory in which the datasets are
stored.
"fsv" will browse the dataset with user formats applied, "fsvraw" will
browse a dataset with no user formats applied, "fsvdc" will show both
unformatted
values along with their formatted values. "fsvacct" will allow you to
browse
a dataset with fields from the acct dataset merged in that you specify.
fsv fsvraw fsvdc fsvacct
"contents" scripts
You might have the need to look at the contents of a sas dataset while
you are using batch. There are two scripts for this, an ordinary
version
and an "l" ("long") version. When using these scripts, you might want
to
pipe the output to "grep" to select only what you are interested in
seeing.
contents contentsl
summary
This is for running a "proc summary" on a dataset in the current
directory
and displaying the output in the terminal window. I use this a lot for
identifying observations with strange or incorrect values which I then
copy and paste into an email. You can apply a "where clause" to the
dataset
and to the output.
summary
Format viewing
Sometimes you need to know what sas formats exist and what these
formats
contain. Sometimes these will be central formats and drug formats.
Sometimes
they will be for formats held in a local directory.
There
is no need to start an interactive sas session as there are scripts to
do this. The first two might need amending to make sure they can find
the
central and drug formats. The last two, for local formats, should not
have
a problem.
showfmts showfmt showlfmts showlfmt
Printing other paginated output
You should print out all reports created using the reporting system
with
"lis2ps" but if you need to print other paginated reports, which
are nothing to do with the reporting system, then you can use the
utilities
"a4ps" or "usps" (for A4 paper or US Letter paper). These
work like "lis2ps" does but do not look in the titles dataset. To send
to a printer you use the "-p" option and follow it by the printer name,
just like you do with "lis2ps". Please bear in mind that this is for
paginated reports and that does not include sas programs or
logs.
In most cases it will be able to produce a report but not always. You
can
adjust margins using these scripts.
a4ps usps
Ad-hoc report line size and page size
If you are working completely outside the reporting system, creating
ad-hoc
reports, you might want the layout to be similar to that coming out of
Spectre but you will need to know what linesize and pagesize values to
use. There are two scripts for this. You can specify the margins and it
will give you the linesize and pagesize values for a few different font
sizes.
a4lsps uslsps
orphans
"Orphans" are files that look as though they have been created by the
reporting
system but there is no entry in the titles dataset for them. Perhaps
some
programs have been renamed, leaving behind old .lis, .chk and .lst
files.
You can list these out for everybody or list out just your own using
these
two scripts.
orphans myorphans
rescue
Sometimes you might accidentally delete one of your sas programs (I
deleted
a whole library of them once). Rather than wait for them to be restored
by the Unix administrator (which is the best option if you are sure
there
are up-to-date backups) then you might like to use "rescue" instead.
You
run this on your program log. If you use it, you will have to carefully
check the code it creates. It will give any restored program the suffix
"_rescue".
rescue
timestamp
You can create a timestamp that gets written to your home directory
using
the "timestamp" script. After you have done this, you can get a
list of all the .lis, programs and titles files you have created since
then to remind you of what you have been working on.
timestamp mynewprogs mynewtitles mynewlis
myfiles
If you want a list of your own files you can use the "myfiles" script
but
note that it uses the owner as seen by the "ls -l" command. You can use
this to give you a list of files owned by other users as well. This is
a very useful script that you will use often if you copy your files
from
one directory to another.
myfiles
You have to know a little about Unix commands to get the best out of
this utility. Suppose you wanted to copy all your titles files from one
directory to another, you could first see the list with this command:
myfiles *.titles
...and then you could copy all these to another directory using a
command
like this:
cp -p $(myfiles *.titles) /a/new/directory
wt
"wt" is short for "which titles file". If you know a table or appendix
reference number (that is the number that follows "Table" or
"Appendix")
then you can find out the titles member and hence the program name
using
the "wt" script. I use it a lot and that is why I made it
only two characters long. It saves typing.
wt
whosgot
Sometimes you will be trying to create a new sas dataset but you will
get
an error message saying that someone else has a "lock" on it. To find
out
the userid of this person then use the "whosgot" script. You can use
the
script "getname" to match the userid to the person's name.
whosgot
makemyrun
There is an equivalent to the production "makerun" script that you can
use. That is "makemyrun". It creates shell scripts with your userid as
part of the name so you can run your own programs.
makemyrun
Cygwin users
I have Cygwin on my PC at home. Most of the scripts on this web site
should
work with Cygwin but not the ones that call SAS, assuming you
have
SAS installed on Windows. The scripts that call SAS rely on the use of
the option "-stdio" which is not available with Windows SAS. Also
Windows
SAS will complain about Unix-style file names. So if you are trying to
install this reporting system on Cygwin and get it working then you are
going to be disappointed. But if you are determined to get some of the
scripts that call SAS working, then I have converted a few to use
Cygwin
that you can maybe use as a guide. One more thing - if you have SAS
Learning
Edition installed then you will not be able to use "proc printto" as
this
very simple procedure was not included in the package. One of the
scripts
I converted to run on Cygwin was "summary" (below) but although I
converted
it it still will not work due to its reliance on "proc printto".
crindex_cygwin contents_cygwin contentsl_cygwin summary_cygwin
An "I wrote a script today" story
I mentioned somewhere else that it is often quicker to write a script
to
help you with a repetitive task than it is to do it by hand. That's if
you know how to write scripts, of course. I am a self-taught script
writer
but I think if I had been sent on courses then I would have learned
five
times faster and my script writing skills would be better than they are
now. But I manage and can write scripts that work in a fairly short
time.
I spent weeks writing scripts trying to get bookmarked PDFs to work so
I have had plenty of practice. So today I wrote a script and I will
tell
you the story of why and how. Are you sitting comfortably? Right, then
I'll begin:
I am working on a study using Spectre, so of course I have hundreds
of programs producing output. Somebody points out that all tables and
listings
should have a solid footnote at the bottom of the page, whether there
is
footnote information or not, and some of my outputs lack that solid
footnote.
Looking through these hundreds of outputs and fixing them will take all
day. So I know I have a script called "pages" that I can use to
extract the first page of output. The solid footnote line might not be
near the bottom of the page but will be in the last ten non-blank
lines.
I know I can drop blank lines using "grep -v". I know I can pick
out the last ten lines using "tail -10". I know there is a
version
of "grep" that can be used in "quiet" mode that will just tell me
whether
it found what I was looking for or not from the return code. I can't
remember
where it is, but "man grep" will tell me. I can't remember how
to
loop through a list of input files but I can do a search on "for
file
in" in the script library and get the correct syntax. Once I know
these
things, I decide to call the new utility "nofootline" and the "shdr"
script will write a nice script header for me to start with. So I do
this
and after fifteen minutes of messing about, I have my script and it
seems
to work OK. Not only does it save me time but it is applicable to other
studies as well so other people can usefully use it. I have saved time
for myself and potentially saved time for other people in the future.
This
script can now become a QC tool. This is "Unix" for you. Once
you
reach this stage of being able to "think scripts" and "write scripts"
you
will not want to work on a different platform. Here is the script I
wrote.
nofootline
Yet another "I wrote a script today" story
Five days after I wrote the above script, an announcement went round
for
another study that all occurences of "study medication" in any mixture
of case needed to be changed to "study drug" in all the output. So what
was needed was to search the ".lis" files for this literal string (in
any
mixture of case) and find out the owners of the corresponding ".titles"
members (since the owner of the ".lis" file might be the person who ran
the program suite and not the programmer who wrote the code). It took
me
less than fifteen minutes to write the script and you can see the
script
below. But this brings me to an important point and that is - Spectre
is
all about high efficiency with programmers maybe handling hundreds of
programs
at a time and so Spectre needs the Unix/Linux platform to achieve
high
efficiency. There is no point in trying to port Spectre to another
platform, though it is possible, since high efficiency would not be
possible
on a non Unix/Linux platform.
lis2owner
Conclusion
You have been introduced to a number of utility scripts on this page.
There
are others not shown here. To see the full list you can always use the
command "showme scripts" and then you can browse the headers of
any of these scripts by using "showme xyz" (where "xyz" is the script
name).
If you know these scripts exist then they should be able to speed your
work. If "showme scripts" is not set up to work then you can see what
it
would produce below.
@index.txt
Use the "Back" button of your
browser
to return to the previous page.