%==========================================================================
\chapter{Testing Procedures}
\label{Testing Procedures}

The HYPRE testing structure is designed to allow test suites to be easily added,
modified and removed.  Tests are designed for each interface and the options it
allows, i.e. different solvers and preconditioners.  Scripts to run the test codes 
are written so that they can be run stand-alone for individual testing or by a 
controller script for regression testing.

A controller script is run nightly on local and remote platforms to test the 
library with all changes made during the day.  The script checks out the latest 
version of the code from the source code repository; configures and builds the
library on the specified platform, runs the driver codes, identifies the success
or failure of the tests and notifies the development team by email. 

%==========================================================================
\section{Test Directory Structure}
\label{Test Directory Structure}

The test directory is contained in the source code repository as the \file{test} 
subdirectory of the top-level directory, \file{linear\_solvers}.  It contains:
\begin{enumerate}
\item \file{Makefile}; rules to build test codes
\item driver codes (\file{.c, .cxx, .C, .f} files)
\item subdirectory \file{AUTOTEST}; contains scripts and files needed for automated tests 
\item TEST\_* subdirectories; contain scripts and files needed to run the driver codes 
\item \file{runtest.sh} test driver utility script
\end{enumerate}

%==========================================================================
\section{Editing Existing Tests}
\label{Editing Existing Tests}

A developer may want to edit existing tests if they have added new capabilities to any
part of the library.  In this case, the developer would edit the appropriate scripts,
input files and test codes to include the new information.  All of the changes are 
manually tested and debugged before being added to the source code repository. Changing
existing tests does NOT require changes to the automated testing scripts.

%==========================================================================
\section{Adding New Tests}
\label{Adding New Tests}

A developer may add new tests in response to reported errors, when a new capability
is added, significant code changes are made, or they determine that existing tests 
are no longer sufficient.  Adding new test cases may require changes to the 
automated testing scripts.

The following steps are done in the \file{linear\_solvers/test} and/or \file{TEST\_*} directories:
\begin{enumerate}
\item Create and test a driver program, e.g. \file{somedriver.c}
\item Create and test a Bourne shell script to run the driver, \file{somedriver.sh}
\item Modify the \file{Makefile} in the test subdirectory as follows:
  \begin{itemize}
  \item Add library information to the \code{LFLAGS} variable
  \item Add target information to the \code{Targets} section
  \item Add rules to the \code{Rules} section
  \end{itemize}
\item Test all changes; debug as needed
\item Commit all files, new and modified, to the source code repository
\end{enumerate}

%==========================================================================
\section{Manual Tests}
\label{Manual Tests}

The developer is responsible for testing their code before it is committed to the
source code repository as well as creating the necessary driver codes and scripts.
Each script is written so that it can be run in stand-alone mode or by another 
script.  These scripts are used regularly by the development team in their individual
testing phases.

%==========================================================================
\section{Automated Tests}
\label{Automated Tests}

The primary script for automated testing is named \file{autotest}.  It is a Bourne 
shell script that controls extracting the source code from the repository, building 
the library, running test suites on local and/or remote platforms, recording results 
and emailing them to the development team. 

It is run nightly by the CRON controller to test the changes made to HYPRE during 
the day.  CRON expects to find the script in \file{/usr/casc/hypre/i686-pc-linux-gnu/AUTOTEST}
directory on the CASC cluster. If \file{autotest} has been modified in the \file{linear\_solvers/test}
directory, it must be manually copied to \file{/usr/casc/hypre/i686-pc-linux-gnu/AUTOTEST} 
before the CRON job is initiated.  See the section below on how to modify CRON jobs.

The \file{/usr/casc/hypre/i686-pc-linux-gnu/AUTOTEST} directory also contains:
\begin{enumerate}
\item a \file{linear\_solvers} directory which is the last extracted from the repository
by \file{autotest}; latest documentation is in the \file{docs} subdirectory
\item a \file{log} directory which is an archive of autotest summary files
\item cron log files for each day of the week, i.e. \file{autotest.cron.Mon.log}
\end{enumerate}

%==========================================================================
\section{Changing the CRON Job}
\label{Changing the CRON Job}

Changes to the CRON job can be necessitated by changes to the \file{autotest} script or the
file that tells CRON when and where to run each test, which is named \file{cronfile} and
found in the \file{linear\_solvers/test/AUTOTEST} directory.

The \file{cronfile} contains the commands for CRON to execute.  It only needs to be modified
if the tests being done on any given night need to be changed, new platforms are added or 
existing platforms are removed.  Currently CRON is initiated on the CASC cluster.

To terminate and restart the CRON job:
\begin{enumerate}
\item crontab -r          (stops current run)
\item verify all editing is completed and files in appropriate locations
\item crontab <filename>  (name of file containing cron commands, i.e. \file{cronfile})
\end{enumerate}

To list current CRON jobs:
\begin{itemize}
\item crontab -l 
\end{itemize}

%==========================================================================
\section{The runtest.sh Script}
\label{The runtest.sh Script}

This script is the test driver utility script.  It is assumed that there are 
test directories \file{test/TEST\_{solver}}, where {solver} is the solver/interface
test name, i. e. ij, struct, sstruct, etc.  These directories contain:
\begin{itemize}
\item Individual test scripts named {test\_name}.jobs that provide the mpirun
execution syntax; {test\_name} is user defined to describe the test, i. e. solver, 
default, matrix, etc.
\item Test run output files named {test\_name}.out.{number}
\item Individual scripts to compare, usually using diff, output files from
corresponding {test\_name}.jobs scripts
\end{itemize}

Usage:
./runtest.sh [-d|-debug] [-n|-norun] [-h|-help] [-t|-trace] TEST\_solver/test\_name.sh

\begin{verbatim}
where:
    solver      name of test
    test_name   user-defined name of test script
   -debug       turn on debug messages
   -help        print help message
   -norun       do not execute codes; echo each command
   -trace       turn on debug and echo each command
\end{verbatim}

Example usage to run all tests in the TEST\_sstruct directory with debug option turned on:
\begin{verbatim}
./runtest.sh -d TEST_sstruct/*.sh
\end{verbatim}

%==========================================================================
\section{The autotest Script}
\label{The autotest Script}

This script is the high-level wrapper for automated testing.  It determines, based 
on its input options, which platforms and options are to be tested.  It controls
extracting the source code from the repository, copying, building and running the
code on either local or remote platforms.

Usage:
\begin{verbatim}
./autotest [-nocvs|-rev REV] [-d|-debug] [-h|-help] host [host ... host]

where:
   -debug       turn on debug messages
   -help        print help message
   -nocvs       do not checkout latest source code; use existing \file{linear\_solvers} directory
   -rev REV     check out version REV from the source code repository
   host         current valid host options are:
                   -gps        Compaq system
                   -gpsmp      Compaq system executing OpenMP
                   -ilx        Intel Linux
                   -insure     CASC Linux executing insure++
                   -mcr        Intel Linux, CHAOS
                   -pengra     Intel Linux, CHAOS
                   -tux149     CASC RedHat Enterprise Linux
                   -tuxgcc     CASC RedHat Enterprise Linux - latest GCC compilers
                   -tuxpgi     CASC RedHat Enterprise Linux - latest PGI compilers
                   -tuxabsoft  CASC RedHat Enterprise Linux - Absoft-Pro8 Fortran90
                   -tuxlahey   CASC RedHat Enterprise Linux - Lahey Fortran
                   -tuxintel   CASC RedHat Enterprise Linux - latest Intel compilers
                   -tuxkai     CASC RedHat Enterprise Linux - latest KAI KCC compilers
\end{verbatim}

Example usage to run version 1.9.0b in debug mode on local CASC and remote MCR:
\begin{verbatim}
./autotest -rev V1-9-0b -debug -tux149 -mcr
\end{verbatim}