Let's consider the following examples.
Assume we want to develop tests for the two groups of interfaces:
Test development for the first group is described in detail below. As for the second one, its only purpose is to demonstrate some important features that are not used in the 1st example (such as common include files, access to test data, etc.). These features are described later.
The source code of the examples can be found in the samples subdirectory.
For this case study you need the following.
After you have downloaded T2C Framework, please do the following.
All the 12 glib array interfaces that are present in the LSB form a single functional group.
Before we create a T2C file for it, consider the directory structure of the sample test suite. (The general test suite structure is described here.) The root of this directory structure is assumed to be specified in the T2C_SUITE_ROOT environment variable.The samples directory contains subdirs for each sample subsystem: sample-01-t2c for the 1st example and sample-02-t2c for the 2nd one. There are also tet_code and tetexec.cfg files in this directory that are required to execute the tests under TET.
There are several scripts here too:It is not mandatory to use these scripts but they can be helpful.
build_all.sh and clean_all.sh use the gen_tests and clean scripts found in the respective subsystem directories.
A directory for each subsystem (sample-01-t2c and sample-02-t2c) should contain the following files and subdirectories:
a configuration file (*.cfg) for the T2C code generator.
gen_tests - test generation script.
clean - clean up script.
src/ - sources of the tests (in T2C format) for the respective library. Each subdirectory contains tests for a respective group of interfaces.
include/ (if present) - a common directory containing header files for the tests (see an example).
reqs/ - requirement catalog.
testdata/ (if present) - data used for testing (along with the sources and makefiles if the data needs to be built, e.g. special modules, etc.). The subdirs of this directory have the same names as the t2c-files with tests, which data they contain. (For instance, samples/sample-02-t2c/testdata/AtkStreamableContent contains data for the tests from samples/sample-02-t2c/src/atk_samples/AtkStreamableContent.t2c.) The path to the test data can be obtained in the test code via T2C_GET_DATA_PATH()
macro.
scenarios/ (created by the T2C C code generator) - TET scenario files.
tests/ (created by the T2C C code generator) - generated C files with the tests, makefiles necessary to build them.
Let us markup elementary requirements for the behaviour of g_array interfaces in the documentation. We will use KompoZer enhanced with T2C ReqMarkup plugin.
Open samples/html/sample-01/glib-Arrays.html in KompoZer. Switch to Normal view if it is not already active.
For now you need not worry about how these requirements are going to be tested. Leave this for step 4 and just mark them up as described below.
Only those parts of the text that contain the requirements for the behaviour of the interfaces are marked up. That is why the following fragments should NOT be marked up in this case:
The results of the requirement markup for the "Glib Arrays" interface group are in this file: samples/html/sample-01/results/glib-Arrays.html
Let us begin with g_array_new.
Press Ctrl+R, Alt+R or select "Assign REQ" in the context menu, on the toolbar or in the "Insert" menu.
Specify requirement identifier. Each requirement we mark up should have a unique identifier. It will be used in the requirement catalogs, output to the TET journal in case of failure, etc.
Most of the time it has the following syntax:
<function>.NN, where NN is a number (an "index") containing at least 2 digits, e.g.:
g_array_new.01
The numbers should start from 01 although it is not mandatory.
More complex cases will be explained later as we encounter them.
So the requirement contained in the selected fragment now has an id: g_array_new.01.
The "Original text" field shows the text we have just selected. If the complete text of the requirement differs from it, we can specify the complete text manually in the "Requirement text" field. This can be useful, for example, when a requirement is formulated in several fragments of the documentation rather than in one contiguous block and the text needs to be "assembled" or when some part of the requirement comes from the very structure of the documentation or from description of other interfaces, etc. We use this functionality to make the requirement's text complete and clear if necessary. It is important that the requirement (its meaning) remains the same.
If we do not specify the requirement's text explicitly, the original text is used as the requirement's text.
The requirement's text is output to the TET journal in case of a failure.
Press "Add". Now the text block we selected represents a requirement with ID g_array_new.01.
The text blocks we have marked up are highlighted in yellow. To toggle the highlighting, press "Highlight REQ" on the toolbar.
There is another requirement for g_array_new in the description of 'zero_terminated' parameter: "TRUE if the array should have an extra element at the end which is set to 0.". The requirement can be reformulated this way:
"If 'zero_terminated' is TRUE, the array shall have have an extra element at the end which is filled with 0s."
Like g_array_new.01, select the appropriate text block in the documentation, press Ctrl+R, then edit the ID (you may leave the id suggested by ReqMarkup: g_array_new.02).
The "+" button increments by 1 the last index in the ID. For example, "foo.02" becomes "foo.03", "bar.05.03.01" becomes "bar.05.03.02", etc. The situation where there is more than one index in the requirements ID is described below concerning the requirements for g_array_free.
Remember that requirement id should be unique. Within a single html this is enforced by ReqMarkup.
The prefixes ("app", "ext", etc.) described below are not taken into account when ReqMarkup checks the IDs for collision (uniqueness violation). For example, the IDs "g_array_new.06" and "ext.g_array_new.06" should not exist in a document at the same time while "g_array_new.05" and "g_array_sized_new.05" do not conflict with each other.
The indices in the ID may not go in order, i.e. it is allowed that there are "foo.03" and "foo.05", but there is no "foo.04".
Press "Add" when the ID is specified.
We added the requirement but forgot to explicitly specify its text. We need to do this because the text we have selected differs from the complete text of this requirement. Now we should do the following:
Do not select any text, just press Ctrl+R. The "REQ Properties" dialog appears again.
Select g_array_new.02 in the list of existing IDs. The "Add" button turns into "Overwrite". Enter the text of the requirement.
Press "Overwrite".
Note that you cannot change requirement ID this way for the current ReqMarkup version. To change the ID without modifying the html source directly, you can do the following in KompoZer:
Position the cursor on the marked up text block of the requirement. The KompoZer's status bar will show something like this: "<span id="g_array_new.03" class="req">"
Right-click this region of the status bar and select "Advanced Properties" from the context menu.
Edit the value of the "id" attribute in the dialog, then press "OK".
The description of the 'clear_' parameter ("TRUE if GArray elements should be automatically cleared to 0 when they are allocated.") should be marked up in a similar way as above. The requirement can be reformulated as follows:
"If 'clear_' is TRUE, the GArray elements shall be automatically cleared to 0 when they are allocated."
g_array_new.03 will be the ID of this requirement.
If you have accidentally added a requirement, you can always remove it using the "REQ Properties" dialog:
You can also remove all requirements marked up in a document by pressing "Remove Markup" button on the toolbar. Beware: this operation cannot be undone.
There seems to be no requirements in the description of the 'element_size' parameter. Leave this description as it is.
There is a requirement in the description of the return value but we have marked up the same requirement before as "g_array_new.01". Still we should mark up this text fragment too but give it a special ID: "&g_array_new.01". The '&' symbol prepended to the ID means that the requirement from this text block has already been marked up somewhere in the same html with ID "g_array_new.01".
This new marked up text block in fact refers to some other requirement in the document (recall C++ references and how '&' is used to denote them - hence the similar notation for the markup).
The referenced requirement may be before or after its "reference" in the document, it does not matter. It should exist in the same document though. References to the requirements from the different documents are not allowed for the present time.
There can be unlimited number of references to a requirement. References to references are prohibited.
From the common sense and from what the documentation says about g_array_sized_new ("the size of the array is still 0."), we can assume that the size of the newly created array is 0. That is, the len
field of this GArray
contains a meaningful value and this value is 0. The documentation says nothing about this, its authors probably considered it obvious. Well, assume it is not so obvious for us.
If we suppose this requirement holds for g_array_new, we can insert new text right in the documentation (the html document) and mark it up. The ID is assigned as usual except it gets the "ext" prefix: ext.g_array_new.06.
"ext" means "extension" in this case. Such requirements with ext-prefixed IDs ("ext-requirements", for brevity) extend the documentation, add something to it that it missed before.
You might wonder if there is any difference between adding an ext-requirement and manually specifying requirement's text. The difference is crucial: when we specify requirement's text, we must not change the meaning of the requirement. So we must not add any meaning to the requirement it has not had before. More often than not, we just try to make the requirement's text clearer or "assemble" it from several parts. If we are rather going to add something new to the documentation, we should use an ext-requirement for that. In practice the line between these cases is sometimes blurred, especially if the requirement's text is somewhat ambiguous.
Checking ext-requirements can be turned on via the T2C configuration parameters. To enable checking of such requirements, you should specify -DCHECK_EXT_REQS in the COMPILER_FLAGS parameter in a generator config file.
We have finished marking up the requirements for g_array_new. Do not forget to save the html file we have been editing.
There is a set of tools in T2C framework that can be useful when the html file with the requirement markup is viewed in a web browser. These tools are provided by the ReqTools javascript ($T2C_ROOT/reqmarkup/reqtools/reqtools.js, the samples use a copy: samples/html/reqtools.js). If you open the html files provided for this case study in a text editor, you will notice the line
<script language="javascript" src="../../reqtools.js"></script>
at the beginning of the <body>. It is all about that.
Now if you open the html in a browser (currently works with Firefox and Opera, may not work in Konqueror though), you will see a "Show Requirements" link at the top. If you click this link, some other links will appear ("Table >>", etc.) which will be discussed later. Besides that the IDs will now be shown (in curly brackets) before the text of each requirement marked up in this html. This can help to determine if some fragment has been marked up wrong.
Click "Hide Requirements" to return to the original view of the document.
If you click on the link (">>") near the word "Table", a table of all requirements marked up in the document will open. The table shows the text of each requirement as well as the references to the requirements. You can click the requirement ID to go to the corresponding fragment of the documentation.
The requirements for g_array_sized_new interface can be marked up in the similar way as above. There are some things you should consider when performing the markup.
There are two requirements for g_array_sized_new in the first sentence of its description rather than one as for g_array_new. The first one is that the function creates a new array while the second one is that the created array has reserved_size elements preallocated.
That is, the principle "one sentence – one requirement" is not always true.
There are no requirements for g_array_sized_new in the sentence "This avoids frequent reallocation, if you are going to add many elements to the array.".
Now we can markup the requirements for g_array_append_vals, g_array_prepend_vals and so on up to g_array_remove_range inclusive. This should not be a problem.
Note that g_array_append_val, g_array_prepend_val and g_array_insert_val are not LSB-interfaces. So we neither markup nor check the requirements for those unlike g_array_append_vals, g_array_prepend_vals and g_array_insert_val.
When there are many requirements marked up in an html-document, it can be difficult to find the one you need in the list in the "REQ Properties" dialog. The possibility to specify which requirements to show in the list (i.e. to filter them) can prove handy in this case.
A filter is a regular expression and it can be set in the "Filter" the "REQ Properties" dialog. Only those IDs that match the filter will be shown in the list. The empty filter matches any ID.
Often a substring is specified as a filter that should be in the IDs we are interested in (for example, "&g_array_new"). In fact any regular expression can be used here provided its syntax is acceptable by JavaScript. (The syntax is described here for instance.)
Now let us markup the requirements for the interfaces g_array_sort and g_array_sort_with_data.
First, note that the description of g_array_sort_with_data refers to the description of g_array_sort: "Like g_array_sort(), but <…>". That is, the requirements for both g_array_sort and g_array_sort_with_data are specified here at the same time.
In the situations like this several IDs separated by semicolons are specified for the corresponding fragment of the documentation instead of one. For example,
"g_array_sort.01;g_array_sort_with_data.01"
By the way the text of this requirement should probably be reformulated to explicitly indicate that sorting is done in ascending order. This is almost obvious, of course, but still.
Second, consider the following fragment:
"which should be a qsort()-style comparison function (returns -1 for first arg is less than second arg, 0 for equal, 1 if first arg is greater than second arg)."
There is a requirement here too but not for the g_array_sort function but rather for any application that uses this function. The application may only pass such a comparison function to g_array_sort that meets this requirement. Otherwise nothing is guaranteed.
Requirements like this ("requirements for application", "app-requirements") are often found in the description of input parameters of a function, etc. We need to mark up these too. The rules for assigning IDs are the same as above except they get the "app" prefix, for example:
"app.g_array_sort_with_data.03"
"app.g_array_sort.02;app.g_array_sort_with_data.02"
"ext.app.my_func.63"
If an ID has more than one prefix, their order does not matter. I.e. "app.ext.my_func.63" and "ext.app.my_func.63" is the same ID.
There is another requirement in the description of g_array_sort_with_data in the following fragment:
"the comparison function receives a user data argument."
This fragment means that g_array_sort_with_data is implemented so that when the comparison function is called during sorting, its last argument contains the value of user_data. So it is a requirement indeed and needs to be marked up.
At this stage marking up the requirements for g_array_set_size should pose no problem.
When marking up the requirements for g_array_free, consider the last sentence:
"Returns : the element data if free_segment is FALSE, otherwise NULL"
Here several possibilities for the function to behave are described. The compound requirements like this are marked up in the following way.
A so called "parent" requirement is associated with a common part of the text of these requirements ("Returns:" in this case). Let us set "g_array_free.03" as its ID.
Then two "child" requirements are associated with the following text fragements: "the element data if free_segment is FALSE" and "otherwise NULL". These requirements get the IDs "g_array_free.03.01" and "g_array_free.03.02", respectively. That is, we simply append one more index to the ID of the parent requirement.
The child requirements can be formulated as follows:
g_array_free.03.01: "The function returns the element data if free_segment is FALSE."
g_array_free.03.02: "The function returns NULL if free_segment is TRUE."
The parent requirement ("g_array_free.03") is considered checked if and only if all its children have been checked (g_array_free.03.01 and g_array_free.03.02).
Child requirements may have child requirements of their own. The IDs are assigned the same way: for instance the children of "g_array_free.03.02" will be "g_array_free.03.02.01", "g_array_free.03.02.02", etc.
Before we finish marking up the requirements, let us take a closer look at the description of GArray structure at the beginning of the document. There are two public fields in this structure, data and len.
The description of these fields imposes some restrictions on the behaviour of g_array interfaces.
For example, consider the len field:
"guint len; the number of elements in the GArray."
This means that if a function changes the array's length, len should change accordingly. That is, say, if we use g_array_append_vals() to add 3 elements to an array, len should increase by 3 after that.
On the other hand, if a function does not change the number of elements in an array (like g_array_sort()), len should remain the same.
The implicit requirements like these usually affect all or the most of the interfaces in the particular group. So we should mark them up specifying the lists of ids like the following:
g_array_new.07; g_array_sized_new.07; g_array_set_size.07; ...
For the next such requirement we would use
g_array_new.08; g_array_sized_new.08; g_array_set_size.08; ...
Using long lists of ids that differ only in the indexes can be tedious and error-prone. There is a more convenient way to handle this: "define"-blocks.
To create a "define"-block, press "Defines" button on the toolbar. A dialog will appear displaying all "define"-blocks found in the document. Press "Add" button and enter name (@g_array_funcs) and value for a new "define"-block in the "Define Editor" dialog, then press "Add" button there.
In this case the value is
g_array_new; g_array_sized_new; g_array_set_size; g_array_free;
g_array_append_vals; g_array_insert_vals; g_array_prepend_vals;
g_array_remove_index; g_array_remove_index_fast; g_array_remove_range;
g_array_sort; g_array_sort_with_data
The "Defines" dialog will now show us the newly added "define"-block:
It is recommended (but not mandatory) for the names of define-blocks to begin with '@'. This makes it easier to distinguish such names from the ordinary function names, etc.
Now we can use "@g_array_funcs" instead of this list of function names in the requirement ids: the description of data field will have the id "@g_array_funcs.07", while the description of len - "@g_array_funcs.08".
In order to keep this example (relatively) simple, some of these requirements are not checked in the sample tests we provide.
The ReqTools script can also show the define blocks present in the html file. Click on ">>" to the right of the word "defines" at the beginning of the document to see them. (If there are no define blocks in the html, this link will not be shown.)
Markup of the requirements in the "Glib Arrays" documentation is now complete.
A T2C file contains a template of the C code for each test case. It is in fact much like "a document with holes" because the placeholders can be left in it to be filled later with the actual test purpose parameters. (Parameter substitution will be done automatically by the T2C C code generator.)
The ReqMarkup plugin can generate a test case template for each interface. To show the template, press "Show Template" button on the toolbar. The template will be shown in a separate window. If you want a template for a specific interface rather than for all interfaces at once, specify the name of the interface in "Filter by function name" field and press "Generate" button (any regular expresion can be used as a filter here).
If the window showing the template is open, its contents are not updated automatically when the corresponding html file changes. To update the template, you should press "Generate" button in this window.
Overall structure of a T2C file is shown here. Note that the T2C section tags (<GLOBAL>, <BLOCK>, etc.) should reside on separate lines in the file. Apart from the tags, these lines may contain only whitespace characters.
Insert a special header at the 1st line of the file (no newlines before are allowed):
#library libglib-2.0 #libsection Arrays
Here we specify the name of the library the interfaces under test belong to (#library) and the name of the interface group that is tested (#libsection).
A line beginning with '#' is a comment (and is not considered by the generator) unless '#' is followed by 'library', 'libsection' or a standard C preprocessor directive like 'define', 'undef', 'line', 'ifdef', etc.
Example:# This is a comment in a T2C file.
#include directives required by the tests should be specified in a <GLOBAL> section that is located right after the T2C file header. The comparison functions for sorting the arrays should also be defined here along with any other global data and functions the tests need. (See samples/sample-01-t2c/src/glib/glib_arrays.t2c).
<GLOBAL> #include <glib-2.0/glib.h> // GCompareFunc // A comparison function for array elements (necessary for sorting) gint array_cmp (gconstpointer a, gconstpointer b) { if (a && b) { if (*((int*)a) < *((int*)b)) { return -1; } if (*((int*)a) > *((int*)b)) { return 1; } } return 0; } // GCompareDataFunc // A comparison function for array elements that also receives // user data argument (necessary for sorting) gint array_cmp_with_data (gconstpointer a, gconstpointer b, gpointer data) { if (a && b && data) { if (*((int*)a) - *((int*)b) < *((int*)data)) { return -1; } if (*((int*)a) - *((int*)b) > *((int*)data)) { return 1; } } return 0; } </GLOBAL>
If it is necessary to perform initialization and/or cleanup of some global resources, the respective code should be placed in the <STARTUP> section. Similarly, to clean up these objects place the appropriate code in the <CLEANUP> section.
Example from samples/sample-02-t2c/src/atk_samples/AtkStreamableContent.t2c:
<GLOBAL> #include <atk/atk.h> #include <AtkStreamableContent/MyAtkStreamableContent.h> #include <useful_functions.h> AtkStreamableContent* obj = NULL; </GLOBAL> <STARTUP> g_type_init(); OBJECT_NEW(obj, MY_TYPE_ATK_STREAMABLE_CONTENT, "MyAtkStreamableContent"); </STARTUP> <CLEANUP> OBJECT_UNREF(obj); </CLEANUP>
As far as our 1st example ("Glib Arrays") is concerned, we do not need such initialization and cleanup. So it is ok to omit these two sections.
Do not use the <CLEANUP> section to release the resources allocated in the test cases rather than in <STARTUP>. This may cause resource leak. Place such cleanup code in the <FINALLY> subsection of a test case block. See samples/sample-01-t2c/src/glib/glib_arrays.t2c for an example that uses <FINALLY>.
Each test purpose is executed in a separate process. Code from <STARTUP> and <CLEANUP> sections is executed in a parent process of the test purpose processes and it is done only once.
Each test purpose has its own copy of the global data. So it is pointless to try transfering data between different test purposes via global variables. Ideally, the execution of a test purpose should not affect any other test purpose.
For each interface a test case template block has been generated for you by ReqMarkup in KompoZer (a <BLOCK> section).
Layout of the <BLOCK> section:<TARGETS> … </TARGETS> <DEFINE> … </DEFINE> // optional <CODE> … </CODE> <FINALLY> … </FINALLY> // optional <PURPOSE> … </PURPOSE> // zero or more
You should place these subsections in the same order as they are listed above.
The <TARGETS> subsection contains the list of interfaces being tested in this test case, one per line. For example, three interfaces are actually tested in one of the test cases in samples/sample-01-t2c/src/glib/glib_arrays.t2c, so the <TARGETS> subsection of the respective <BLOCK> section looks as follows:
<TARGETS> g_array_set_size g_array_new g_array_sized_new </TARGETS>
The <CODE> subsection contains the common code for each test purpose of this <BLOCK>. The placeholders in the code to be filled by the particular test purpose parameters can be specified here. For example,
int nResult = 2 + <%0%> * <%1%>;
During the generation of a C function for a test purpose the generator will replace <%0%> and <%1%> with the values from the appropriate <PURPOSE> section.
The <PURPOSE> subsection specifies parameters of a particular test purpose (one parameter per line). The first of these parameters will replace <%0%> in the <CODE> section, the next one will be for <%1%> and so on. Up to 256 parameters are allowed (<%0%> - <%255%>).
Example:<PURPOSE> 2 a+b </PURPOSE>
<PURPOSE> subsection is optional. If no parameters are necessary for a test purpose, you may omit this section or specify it but leave it empty, like this:
<PURPOSE> </PURPOSE>
Multiple <PURPOSE> sections are allowed in a <BLOCK>.
For each <PURPOSE> subsection a single C function (in fact, a test purpose) will be generated using the template code specified in the <CODE> section.
In the <DEFINE> section you can list the #define directives for the test purpose parameters. The code generator will place these directives at the beginning of the generated test purpose function. Corresponding #undef-directives will be inserted at the end of this function.
This feature can be used to replace <%…%> with more readable symbolic names which can be quite convenient.
<DEFINE> subsection is optional.
<DEFINE> #define QUANTITY <%0%> #define PRICE <%1%> </DEFINE>Now instead
int nResult = 2 + <%0%> * <%1%>;we can write the following in the <CODE> subsection:
int nResult = 2 + QUANTITY * PRICE;
The code from the <FINALLY> subsection is always executed in the test purpose regardless of whether the requirement checks in REQs pass or fail. You can use this subsection to release the resources local to the test purpose, e.g. free previously allocated memory, close files, etc. (To release global test case resources, use the <CLEANUP> section described above.) The <FINALLY> subsection is optional.
The complete example of a <BLOCK> section with all its subsections is shown below. Usage of REQ(), TRACE() and other macros is explained later.
Example:<BLOCK> <TARGETS> g_array_remove_index_fast </TARGETS> <DEFINE> #define INDEX <%0%> #define VALS <%1%> #define TYPE <%2%> </DEFINE> <CODE> GArray *ga = NULL; GArray *new_ga = NULL; int old_len; int last_el; TYPE vals[] = VALS; ga = g_array_new(FALSE, TRUE, sizeof(TYPE)); if (ga == NULL) { ABORT_TEST_PURPOSE("g_array_new() returned NULL."); } ga = g_array_append_vals(ga, vals, sizeof(vals) / sizeof(TYPE)); if (ga == NULL) { ABORT_TEST_PURPOSE("g_array_append_vals() returned NULL."); } old_len = ga->len; last_el = g_array_index(ga, TYPE, old_len - 1); new_ga = g_array_remove_index_fast(ga, INDEX); /* * the GArray. * * [The function returns a pointer to the modified GArray.] */ REQ("g_array_remove_index_fast.03", "g_array_remove_index_fast returned NULL", new_ga); REQ("g_array_remove_index_fast.03", "The returned GArray pointer does not match the original one.", new_ga == ga); TRACE("The length of the array is %d (should be %d).", ga->len, old_len - 1); /* * Removes the element at the given index from a GArray. */ REQ("g_array_remove_index_fast.01;g_array_remove_index_fast.08", "", ga->len = old_len - 1); if (INDEX < old_len) { /* * The last element in the array is used to fill in the space */ REQ("g_array_remove_index_fast.02", "The last element of the array did not fill in the space.", g_array_index(ga, TYPE, INDEX) == last_el); } </CODE> <FINALLY> if (ga) { g_array_free(ga, TRUE); } </FINALLY> <PURPOSE> 8 {19, 89, -1, 8, 7, 190, 9, 10, 28, 56} int </PURPOSE> <PURPOSE> 0 {19, 89, -1, 8, 7, 190, 9, 10, 28, 56} int </PURPOSE> <PURPOSE> 9 {19, 89, -1, 8, 7, 190, 9, 10, 28, 56} int </PURPOSE> </BLOCK>
The <CODE> subsection contains at least one REQ() call for each marked up requirement for the particular interface.
REQ(<list_of_IDs>, <comment>, <expression>);Example:
REQ("fake.01;foo.04", "Incorrect multiplication result", ARG0 * ARG1 == nCorrect);
If <expression> is nonzero, the execution of the test purpose goes on. A message is also output to the TET journal indicating that the requirements with IDs listed in the 1st REQ argument have been checked.
Otherwise, i.e. if the requirement is violated, another kind of message is output to the TET journal that contains the list of the requirement IDs, the text of these requirements and also the comment specified as the second REQ argument. The execution of the test purpose is aborted in this case and the result code is set to FAIL.
If execution of a test purpose has not been aborted due to a failed requirement check or some unexpected failure (for example, segmentation fault, glibc error, etc.), the test result code is set to PASS.
REQ(<list_of_IDs>, <comment>, <expression>);is equivalent to
TRACE0(<comment>); REQ(<list_of_IDs>, "", <expression>);(See the description of TRACE0 below.)
You can use TODO_REQ() macro as the <expression> parameter for the requirements, checks for which are yet to be written. In this case no record goes to the TET journal and the execution of the test goes on as if the check has passed.
Unlike this, REQs with TRUE (or 1) as the expression do add records to the TET journal as described above, despite such checks never fail. The REQs like these can be used to report that the requirement is covered in the test even if it is satisfied automatically due to the way the test is organized.
If the REQ fails, the subsequent REQs in this test purpose WILL NOT be checked. The code in the <FINALLY> subsection (if present) will be executed and then the test purpose will terminate.
Sometimes more than one requirement is actually checked in a single REQ. It can happen that we are unable to determine which requirement has failed when the <expression> evaluates to FALSE (i.e. 0). This is often the case for get_XXX() and set_XXX() functions: we can often check that set_XXX() has worked as needed only by calling get_XXX() and then comparing the value it has returned with the value we tried to set by set_XXX. If the values do not match we cannot really say whether it was get_XXX() or set_XXX() (or both) that went wrong.
We can specify the list of corresponding requirement IDs in the 1st parameter of REQ in situations like this.
Example:/* * If both key and group_name are NULL, then comment will be written * above the first group in the file. */ TRACE("set_comment() was called for \"%s\", get_comment() returned \"%s\".", COMMENT, ret_cmnt); REQ("g_key_file_set_comment.03;g_key_file_get_comment.03", "", is_comment_equal(COMMENT, ret_cmnt));
If the expression in this REQ is false the message output to the journal will say that at least one of the listed requirements has failed.
Do not confuse the list of IDs in a REQ in situations like this with the list of IDs we specified during the markup (see the markup of the documentation for g_array_sort()). These lists serve completely different purposes. In the latter case a REQ() call is generated by ReqMarkup for each requirement in the list, so that the ID list for each REQ() still contains one and only one ID.
If the failed requirement is a requirement for the application using the interface being tested (i.e. a requirement with an "app."-prefixed ID) the displayed message indicates that there can be a bug in the test case itself (perhaps a test case developer error).
You should use TRACE instead of printf (the syntax is the same):
TRACE("The length of the array is %d (should be %d).", ga->len, old_len - 1);instead of
printf("The length of the array is %d (should be %d).", ga->len, old_len - 1);TRACE0 should be used instead of calling printf() with the only argument:
TRACE0(str);rather than
printf(str);
This macro sets the result code to PASS (if no failure has occured before in this test) and ends the test purpose.
Code in the <FINALLY> section will be executed anyway.
One can encounter a situation when an error is found during the execution of the startup function, and it makes no sense to execute the test case after that. (For instance, the initialization may have failed for some global user-defined object.)
In this case you should call INIT_FAILED("…") in the <STARTUP> section providing appropriate description of the failure as its argument. This description will be written to the TET journal. No test purpose will be executed after that for the test cases from this t2c-file. All the test purposes will be marked as UNINITIATED.
Example:<STARTUP> g_type_init (); img = ATK_IMAGE (g_object_new (TEST_TYPE_SIMPLE_IMAGE, NULL)); if (!img) { INIT_FAILED("Unable to create a TestSimpleImage instance."); } </STARTUP>
Use this macro to abort test purpose execution if something wrong happens (e.g. memory allocation failed for some local data, etc.) The specified message will go to the TET journal, the test purpose result will be set to UNRESOLVED and (after the execution of the <FINALLY> section) the test purpose terminates.
Example:ga = g_array_new(FALSE, TRUE, SIZE); if (ga == NULL) { ABORT_TEST_PURPOSE("g_array_new() returned NULL."); }
Use this macro in the test purpose to abort execution if the feature to be checked is not supported by the system under test. The specified message goes to the journal and should describe the situation, e.g. "Dynamic loading of modules is not supported". The result of this test purpose will be set to UNSUPPORTED.
Code in the <FINALLY> section will be executed anyway.
All the macros described above are defined in <t2c.h>.
The t2c files for the examples described here can be found in samples/sample-01-t2c/src/glib/, samples/sample-02-t2c/src/atk_samples/. The tests in these files may not be perfect, of course, (and there are also some TODO_REQ()s there) but still they can be useful.
For each t2c-file there should be an xml-file with the same name in the "reqs" subdirectories of samples/sample-01-t2c/ and samples/sample-02-t2c/. Each of the files should contain a list of the requirements to be checked in the tests from the t2c file and is in fact a requirement catalog for a given interface group.
This catalog will be used by the tests in failure reporting. It contains the ID and the text for each requirement.
Let us create a requirement catalog for "Glib Arrays" group.
First of all, open the html file with marked up requirements in KompoZer if it is not already open. Then press "Catalogue of Reqs" button on the toolbar. A dialog will appear:
The html files from which the catalog will be created are listed in the "Source files" box. Currently opened file is already listed there by default, this is exactly what we need in this case. If we wanted to create a single requirement catalog from several html files, we would press "Add" button to add remaining files ("Remove" button could be used to remove unnecessary items from the list). This functionality is useful when the requirements for a group of interfaces are specified in several html files rather than one.
Press "Browse" and specify an xml file that will contain the requirement catalog, let it be samples/sample-01-t2c/reqs/glib_arrays.xml in this case. Press "Generate". Requirement catalog will be created and saved in the specified file.
If you open the newly created xml file with a text editor, you will see something like this:
<?xml version="1.0"?> <requirements> <req id="g_array_append_vals.01"> Adds len elements onto the end of the array. </req> <req id="g_array_append_vals.02"> The function returns a pointer to the modified GArray. </req> <req id="g_array_free.01"> Frees the memory allocated for the GArray. </req> <req id="g_array_free.02"> If free_segment is TRUE it frees the actual element data as well. </req> <req id="g_array_free.03.01"> The function returns the element data if free_segment is FALSE. </req> <req id="g_array_free.03.02"> The function returns NULL if free_segment is TRUE. </req> … </requirements>
The requirement's ID is specified in the "id" attribute of the "req" tag. The contents of this tag (the text between ">" and "</req>") are the requirement's text.
The requirement catalog for "Glib Arrays" is ready. Now you can create a requirement catalog for "AtkStreamableContent" interface group from our second example.
If the tests are organized in appropriate directory structure described above, you can just change current directory to samples/ and run build_all.sh from there to generate the C code and build the tests. Make sure that T2C Framework is already built and the T2C_ROOT environment variable is set properly (see the instructions here) before executing build_all.sh
build_all.sh first sets some necessary environment variables (T2C_SUITE_ROOT, PATH, PKG_CONFIG_PATH, etc.) then builds the T2C code generator. The generator is then used to create C code from T2C files and makefiles to build test executables from these C sources. Finally, these makefiles are executed along with the makefiles in the testdata_src subdirs (if present).
The command-line syntax of the T2C code generator is described here. You can look at the gen_tests scripts found in each subsuite directory to see which parameters are actually used for T2C to process the presented samples.
To run all the tests, execute the run_tests.sh script from the samples/ directory.
If you want to run the tests from a particular "subsuite" (sample-01-t2c or sample-02-t2c), specify the name of this suite as a parameter for run_tests.sh.
Example:./run_tests.sh sample-01-t2c
In fact, run_tests.sh performs some auxiliary operations and then runs the TET test case controller from the samples/ directory:
tcc -e .
The test results can be found in the TET journal (samples/results/0001e/journal).
A standalone version of a test can also be built from the generated C source and executed without TET environment. This can be useful, for example, to debug the test and the target system or to obtain more data about the behaviour of the tested interfaces in case of failure.
The instructions on how to build and execute a standalone version of the test are available here. A brief description is also provided in a comment block in the beginning of each generated C file.
Index