Index

Test development with T2C: a typical workflow

The T2C ("Template-to-C") framework is designed for development of the "normal"-quality tests that can be used with the TETware Lite test harness.

Test development with T2C usually consists of the following steps.

Workflow

1. Analysis of the documentation and interface grouping

First of all, before trying to write conformance tests for some interfaces one should examine the documentation of these interfaces to find out what is to be tested.

Then the interfaces to be tested should be split into some functional groups. Sometimes the grouping is already done in the documentation. For instance, "Glib Reference Manual" is divided in sections such as "Memory Allocation", "String Utility Functions", "Key-value file parser", etc. Interfaces described in each section form a single functional group in this case.

The functional groups of interfaces should not be very large. The recommended maximum size of a group is 10–15 interfaces.

Cyclic dependency Ideally, the functional groups should have no cyclic dependencies between each other.
Group "A" depends on group "B" if we need interfaces from "B" to test those from "A". If "A" depends on "B", "B" depends on "C" while "C" depends on "A" (or even "B" depends on "A") it is a cyclic dependency. Situations like this should be avoided.


For each group of interfaces a T2C file will be created and placed in a proper place in the special directory structure. The T2C file will contain the tests for these interfaces in a format described below. The appropriate directory structure for the test suite is also created at this stage.

The documentation for the interfaces is assumed to be in html format.

2. Requirement (assertion) markup

At this stage the so called elementary requirements (or assertions) are extracted from the documentation. Each requirement is given a unique ID and is linked to an excerpt from the documentation. The excerpt is marked up in a special way in KompoZer html editor enhanced with T2C ReqMarkup plugin.

By default this excerpt represents the requirement's text associated with the respective ID. The text can be reformulated if necessary to improve readability. This reformulated text does not replace the original one in the documentation, it affects only requirement's textual representation associated with a given ID.

The reformulated requirement's text (or the original text if the reformulated text is not specified) is output to the TET journal in case of an assertion failure.

The result of this step is the html file with the documentation enriched with the marked up requirements for the interfaces.

ReqTools javascript provided with T2C Framework could be helpful for viewing this documentation in a web browser. ReqTools shows IDs of requirements for each marked up fragment of the text as well as a table of all requirements in the document, etc. For the present, it has been tested for Mozilla Firefox, Opera and Internet Explorer. It may not work properly in Konqueror though.

3. Generating T2C file template

Once the requirements have been marked up, the ReqMarkup plugin generates a template for the test in the T2C format.

Overall structure of a T2C file is shown below:
T2C file structure

The sections of a T2C file are marked up with special tags: <GLOBAL> ... </GLOBAL>, <CODE> ... </CODE>, etc. The code in these sections is a plain C, although the test case code (<CODE> section) can also contain placeholders for the test purpose parameters. The sets of such parameters are specified in the <PURPOSE> sections. At the step 6 the T2C code generator will create a C function for each test purpose by substituting the actual parameter values instead of these placeholders. That is, the contents of the <CODE> section will be used as a template for the C code of the test.

Test code generation

For each interface a test case template is created with a call of REQ macro for each requirement. REQ is used to check the requirement and report failures. It receives a requirement ID, an expression to evaluate and some other data as its arguments. If the expression evaluates to 0, it means that the requirement (assertion) has failed. In this case the appropriate message is output to the TET journal along with the requirement text and the test result code is set to FAIL.

4. Populating the T2C file template

This is the most important phase when the tests in the T2C format are written.

If it is necessary to provide some global data for the tests, perhaps along with the code for its initialization and cleanup, it can be specified in the <GLOBAL>, <STARTUP> and <CLEANUP> sections of the T2C file, respectively.

The T2C file header specifying the name of the library and the functional group of interfaces under test should also be provided.

The T2C Editor Eclipse plugin can be helpful for visial editing of T2C files providing advanced navigation among the sections along with other useful features. However, T2C files can be edited with any text editor as well. The development of the T2C Editor plugin is now in progress.

It is not recommended to use TETware Lite API directly from the test case templates. Macros and API functions provided by the T2C support library are higher level means to perform common operations.

5. Creating requirement catalog

Based on the html with the requirements marked up (see step 2), a requirement catalog is created. This catalog is used by the tests in failure reporting. It contains the ID and the text (original or reformulated, if specified) for each requirement.

The catalog is created by ReqMarkup plugin in KompoZer html editor.

The currently used catalog file format is rather simple. The catalog file is an xml file that contains an ordinary header ("<?xml version="1.0"?>"), a root element ("requirements") and one or more "req" elements within the root element. Requirement id is specified in the "id"attribute of this tag, while its body is the requirement's text. (See the example below.)

<?xml version="1.0"?>
<requirements>
<req id="atk_streamable_content_get_mime_type.01">
Returns a gchar* representing the specified mime type.
</req>
<req id="app.atk_streamable_content_get_mime_type.02">
streamable: a GObject instance that implements AtkStreamableContentIface.
</req>
<req id="app.atk_streamable_content_get_mime_type.03">
i: a gint representing the position of the mime type starting from 0.
</req>
<req id="atk_streamable_content_get_n_mime_types.01">
Returns a gint which is the number of mime types supported by the object.
</req>
<req id="app.atk_streamable_content_get_n_mime_types.02">
streamable: a GObject instance that implements AtkStreamableContentIface.
</req>
<req id="atk_streamable_content_get_stream.01">
Returns a GIOChannel that contains the content of the specified mime type.
</req>
<req id="app.atk_streamable_content_get_stream.02">
streamable: a GObject instance that implements AtkStreamableContentIface.
</req>
<req id="app.atk_streamable_content_get_stream.03">
mime_type: a gchar* representing the mime type.
</req>
</requirements> 

Sometimes the requirements for a group of interfaces are specified in several html files rather than one. ReqMarkup can generate the complete requirement catalog from all these files in this case.

6. Generating C code and makefiles

Now that the tests in T2C format and the requirement catalog files are prepared, it is time to invoke the T2C code generator to finally create the C code of the tests as well as necessary makefiles for individual tests and the test suite as a whole.

For this to work properly, T2C files, requirement catalog files and other files should be placed in a proper directory structure (described here).

Once the directory structure is prepared and each file is where it should be, one can run the code generator.

The code generator also creates special targets in the generated makefiles that can be used to build a test without using the TET test harness (see the details here). The test executable built in this way does not depend on TET at all which can simplify debugging. There is no free lunch, of course. Many of the TET features are not available in this case such as advanced signal handling and error reporting, etc.

7. Building, executing and debugging the tests

The test suite is ready for building. One can now build it using the makefiles generated at the previous step, then run the tests or debug some of these outside the TETWare Lite test harness.

Index