(Author: Roland Rashleigh-Berry
Date: 24 Aug 2015)
Introduction
This page contain some resources for the validation of Spectre. Note the
word resources is used. What you will find on this page is
not
a recommendation for how to validate Spectre. Instead you will find
my thinking and some code that you may or may not wish to use as a basis
for your validation effort. You should follow your own procedures to do
the validation of Spectre. Hopefully you have Standard
Operating Procedures to cover this activity. If you don't then you
will have to create a validation plan and stick to it and provide full
documentation to prove that this plan was carried out to successful conclusion.
But again, I can not recommend how this should be done.
Spectre will only be able to deliver its full potential as a fast and
efficient reporting system when it has been validated. Compared to a situation
where no standard reporting macros exist, you should find that it greatly
speeds up the production of tables. What it does not speed up is
the validation of these tables. Validation of tables will become the new
bottleneck once you have Spectre installed and running, until you can validate
the system. But having a validated system will not save you all the
validation work, the work will merely shift into parameter checking, data
selection checking and perhaps code review. But this should be a lot faster
than double programming table output.
Validated vs. non-validated
What I will recommend is that you do validate the system,
in particular the two major reporting macros %unistats and
%npcttab.
I will give my reasoning here for your consideration.
If you can validate the %unistats
and %npcttab macros,
then assuming these macros will feature heavily in table production, you
would probably save QC time overall for a study than if you double programmed.
Maybe the study budget could be used for the validation in such a case.
You will maybe have different data cuts for a study. If so you might end
up having to rerun your validation programs for each data cut and having
to check the figures each time. But if the macros and data selection process
were already validated then no extra work should be required for the additional
data cuts if the data selection process had not changed.
Assuming the macros you had validated were stable and could be used for
other studies then a lot of validation effort would be saved for those
other studies.
Testing vs. no testing
Some of the components of Spectre are not on the critical path of producing
output for submission to regulatory agencies. This applies to every utility
script that is not flagged as being a "production" script in the header
and it applies to some macros as well. For example, %complibs
is used to compare libraries of SAS® datasets. It is not used in a
production run to produce reports. It is itself a "validation" macro. Just
as you don't validate your validation code you use to double program tables
then I would suggest you do not need to validate a macro such as this.
So my suggestion is to not bother to validate anything not flagged as being
"production" in the header. Of course, this is a suggestion only
and not a recommendation. For the scripts and macros marked "production",
then there are two approaches you can take to testing them, as will be
explained.
Black box testing vs. white box testing
If you test the functionality of a system without regards to the internals
of a system then this is known as "functionality testing" or "black
box testing". It is known as "black box" testing because you can't
or don't look inside of it. If you know about the internals of a system
and you test these internal components then this is known as "white
box testing", "glass box testing", "structural testing", "clear box
testing", "open box testing" plus a number of other terms. With "white
box" testing you can see inside of it, you know its structure,
and you test the things you can see. My suggestion is that
you differentiate between the values shown in reports
(in the data area) and the running, layout, collection, conversion to PDFs
and delivery of the reports themselves. For the values you see in
reports (in the data area - not page numbers, for example) my suggestion
is to use both "white box testing" and "black box testing".
For things not related to the values in reports to use "black
box testing" only. If you have used Spectre to report a study then
you will soon know if Spectre is failing to run some programs it should
have run or failed to incorporate all the reports it should have done in
the PDF. The script "titlesvsbkmarks"
is run against the collection of PDFs and it will identify any table/appendix
reference numbers that are missing from the PDF bookmarks. So just by using
Spectre to report a study and by people confirming that all the reports
are present in the PDFs then some "black box" testing has been done even
if you didn't realize it and you didn't document it. Perhaps checking this
and documenting your findings could be part of the "black box" testing
that would form part of your validation effort. But again, this is a suggestion
and not a recommendation.
Testing %unistats and %npcttab
My suggestionfor these two macros is to use both "white
box" and "black box" testing. "Black Box" testing could be done
along the lines as the demonstrations shown in the introductory pages for
%unistats
and %npcttab. "White box" testing
could be performed on these macros by testing the components of %unistats
and %npcttab to ensure
those components are working as they should, and then perhaps more detailed
testing could be done with those components working together. Of the two
macros, %unistats will take longer to validate as it has far more functionality.
If you decide to do "white box" testing of these two macros then you will
need to test the low level sub macros they use. This is discussed in the
next section.
Low level macro testing
This is a good point to explain how I go about writing macros such as %unistats
and %npcttab because I think this reasoning should reflect on the way they
are tested. When I think about writing a complex macro, I know that I will
not be able to write it as a single macro. Large single macros are very
difficult to write mainly because they are especially difficult to debug.
The messages you get about macro errors often don't tell you much about
where it is going wrong so debugging large complex macros can be nearly
impossible. So what I do, when I know I have to write a complex macro,
is to isolate sub tasks that could be usefully performed by macros, especially
if these macros could be used in other complex macros not yet written.
I then proceed to write these sub macros and ensure they are working and
reliable and then I use these as building blocks in the more complex macros
I am trying to write. This may not be the "correct" approach from a "project
life cycle" methodology point of view but it is the only way I know of
getting the job done. So what I have done over the course of many years
(since 1986), is to collect a set of macros that might come in useful when
writing more complex macros. You will find this collection on the "sasautos
extensions" page of my old web site here.
You will also see that most of these macros have simple test packs. These
test packs are what I have used to convince myself that the macros are
doing what they are supposed to do. So when it comes to testing the complex
macros in Spectre, I think it makes sense to follow the same approach that
I have taken and validate the sub macros first and the larger macros that
call them after this has been done.
What I have done for the low level macro testing for %unistats and %npcttab
is to copy the approach I have used on my "sasautos extensions" page and
to have test packs for all the low level macros that you can run for yourself,
copy and amend to create your own versions of the test code that you feel
test the macros sufficiently for validation purposes. Again, this is a
suggestion and not a recommendation. You must follow your own test plan
and not something I suggest. What you will find on the page you can link
to below are the low level macro test packs that I have used and the log
or list output, arranged in alphabetical order.
The low level macros referred to in the last section do not cover the "partner"
macros of %unistats and %npcttab. Each has a p-values macro named %unipvals
and %npctpvals so
that the task of calculating p-values, that might change in the future,
is kept separate from the more stable parts of the macro. The same applies
to the "partner" macro to %unistats which does the printing in simple cases,
named
%unicatrep.
The intention is that the stable parts can be validated on their own without
the "partner" macros. How you do this is up to you. Note that if %unistats
is used to produce a dataset instead of a report then there will likely
be data manipulation of the output dataset before the report is created,
so even if the macro is fully validated you will still have some validation
work left to do. You might like to consider validating the macros where
this technique is used using "black box" testing and not bothering to validate
the way %unistats creates output files in a similar way to the demonstrations
given for these two macros.
Conclusion
This page introduced you to some ideas about how to validate Spectre. In
the end, it is you who must decide how this is done, and I wish you every
success in your endeavours.
Use the "Back" button of your browser
to return to the previous page
SAS and all other SAS Institute Inc. product or service names are registered
trademarks or trademarks of SAS Institute Inc. in the USA and other countries.
® indicates USA registration.