Testharness documentation

Overview

The test harness provides a system for scripting regression tests for Java classes and their methods. It consists of two parts:

Unit tests are defined within test scripts. These are plain text files which contain a sequence of method test definitions interspersed with comment lines used to describe tests and their expected results. Blank lines may be added for improved readability. Each test definition is a line containing the name of the method to be executed, a list of parameter values and an optional comment at the end of the line, which is typically used to say whether this test is expected to succeed or fail and to show the expected return value, if any.

Any number of parameters may be supplied. If the constructor or method being tested requires more parameters than are given on the test line, a non-fatal error is reported.

This is a short example script:

----------------------------------------------------------------
# re01.txt - clean re_error() test.
# ======== - error with simple text description
#
reporterror re01                # OK
re_error "error text 1"         # "re01: error text 1" expected

#
# No errors expected
#
----------------------------------------------------------------

The first test instantiates runs the ReportError() constructor with the string "re01" as its only parameter. The second test runs the re_error() method with "error text 1" as its only parameter.

The output from running a test script is intended to be an easily readable file, with outputs from each test, including error reports, immediately following the line that defined the test.

Regression testing is easily managed by a shell script that can:

Test script processing

Every line in a test script is written to an output file via stdout, preceded by a line number. Comment lines may appear anywhere and have a hash, #, as the first character. Comments and blank lines are copied to the output but otherwise ignored.

Each non-comment line defines a method call. This takes the form:

method param1 param2 .... # comment documenting expected results

The method name must start at the first character in the line and parameters are separated by one or more whitespace characters. There may be zero or more parameters on each line. There is no limit on the number of parameters for each test. Quoted parameters may contain whitespace. Unquoted parameters must not contain whitespace.

If a parameter contains whitespace it must be 'quoted' by enclosing it in single or double quotes ( 'param text' or "param text" ). Quoted parameters must be terminated with the same single or double quote mark that they started with, but may contain the other type of quote, so "String val '%s'" and 'Char val "%c"' will both be parsed correctly, but "stringvalue' will include all following parameters until either another double quote or the end of the parameter list is found.

The parameter list may be followed by a comment, starting with a hash, which is a good way of documenting the expected results from a method call.

When the text has been parsed into method name plus a list of parameter strings, a parallel list of integer numeric values is set up. Each parameter is first tested to see if it is a boolean value. If its first character is T,t or 1 its numeric value is set to 1 (TRUE) and if its first character is F,f or 0 its numeric value to 0 (FALSE).

If neither comparison matches, the parameter is used as input to Integer.parseInt() and the result is used as its numeric value.

The callAdapter() method, which will be described later, is now called to execute the required method test. It is passed the method name and both string and integer versions of the parameter list, runs the test and returns the test results by calling one or more of the following methods, which each write an output line to stdout immediately after the script line that defined the test:

Function calledOutput
void setboolreply(int n); Returned: true|false
void setcharreply(char c); Returned: 'c'
void setdoublereply(double d); Returned: n.nnnnnn
void seterror(char *e); Error: error text
void setfloatreply(float f); Returned: n.nnnnnn
void setintreply(int r); Returned: nnnn
void setstringreply(char *s); Returned: "string value"

The number and type of results shown is determined by the way in which you customise callAdapter() - see below.

The number of calls to seterror() are counted and output when the test script has been completely processed.

Note:

Customisation

This process will normally only require changes to the Custom.java source file. Make a copy of Custom.java and custom.h before you start, so you still have the original files available building further test harneses

Custom.java provides the interface between the test harness methodality in RunTest.java and the C methods being tested. It contains two methods:

Tests are driven by code in RunTest.java, which uses command line arguments and options to set run conditions and specify which test scripts will be run. Runtest interprets each test script, using the callAdapter() method in this module to exercise the methods being tested, and writes the results of each call to stdout immediately after the script line that defined the test.

public void link(RunTest rt)

This is called by RunTest to pass the rt reference to Custom to enable calls to the output reporting methods in the RunTest instance. If a class being tested needs access to the current debug level or to call a tracing or error reporting class, such as org.gregorie.envron.ReportError

, this can be created locally and adding a statement such as:

debug = rt.getDebugLevel();

debug is an int declared in your Custom subclass which takes positive values: zero means diagnostic tracing is off and successively higher values should select progressively more diagnostic detail.

public void callAdapter(String fn, ArrayList p, ArrayList n)

This is a code-specific call adapter which must be modified to call the set of methods to be tested.

Entry and exit is reported if debug level is 2 or more. Custom.java is an example implementation that contains the code to run one example method, whose prototype is:

int example(String s int i) is a runnable private method whose only use is to allow the code in Custom to compile and run successfully, exercising all the output functions in RunTest. It should be removed as part of copying Custom to create a customised subclass.

The basic if..then..else chain of method calls should be preserved, but any or all if-actions can be converted into code blocks and/or may call additional privately defined methods.

The methods used to return results, report errors, etc, are defined in RunTest.java. They are setError(), setNote(), setResult() and setResultWithHex() methods.

The final item in the condition chain should not be modified. It is there to trap method names in test scripts that don't match any of the methods being tested.

When callAdapter() is called by code in RunTest.java:

debug (normally zero) can be set to 1 to show callAdapter() entry and exit conditions by adding a single -d option to the command line. -dd shows more detail.
fn A String that must contain the name of the class, method or pseudo-method being tested. Pseudo methods are any code that uses setResult(), setNote() etc together with calling methods of methods being tested. For instance, once the getters of a class have been tested, it may make sense to write a pseudo-method that calls all getters because this would simplify test scripts that look at corner cases, etc.
ArrayList p is an array containing the call parameters as strings. Unused parameters have a NULL pointer, so can cause crashes.
ArrayList n is an array containing the call parameters as int values with those that weren't numeric set to zero.

It is the customiser's responsibility to map the contents of the p and n arrays to the parameter list of each method being tested. Any additional code that may be needed to provide the correct mapping can be added as required, e.g. converting p.get(1) to a char or n.get(2) to a double.

On exit callAdapter() should return true if the overall call didn't encounter any fatal errors and false if an error was encountered that was serious enough to abandon the run part-way through the test script.

void showHelp()

The text in showHelp() is displayed if the test harness is run with the '-?' option. It should say that this is a customised test harness, describe what methods or modules are being tested and identify the (set of) test scripts it uses, e.g. that their names are all of the form data/mytest99.txt

Other methods

Custom also includes a number of private methods that may be useful when manipulating or converting parameters:

Compilation

This process will normally only require changes to the Custom.java source file. Make a copy of Custom.java and adapt ant's build XML file to compile your subclass.. Finger trouble aside, if you don't change any method prototypes or headers a new test harness should build and run pretty much straight away.

NOTE: compiling the test harness requires my environ library, which you can download from https://www.libelle-systems.com and it should be put in /usr/local/lib.