When developing LSB tests we define three testing quality grades:
- Shallow – simple tests with the only guaranteed purpose of ensuring the interface does not crash (in many cases it is additionally checked that the interface doesn’t return an error code) being called with some particular correct parameters and in the correct environment. This is close to “existence” or “smoke” or “sanity” tests (but beware - these terms are interpreted differently by different experts).
- Normal – this is the most reasonable level of testing achievable by tests written in plain C or C++. The tests check the main functionality and may check a few error scenarios. Most of the existing LSB tests are of this quality.
- Deep – this is the level when most of the specification assertions are tested in various conditions/states. This is usually used for most important and critical software.
Note: the borders between the testing grades are obscure, the scale is continuous actually.
To develop tests of corresponding grades, we are using different approaches:
- Shallow tests – we have been developing and maturing a new technology and tools (AZOV Framework) for automatic shallow tests generation based on some description of the interfaces, their parameters, interlinks and default values in the LSB database. The core idea here is to augment the database to contain enough information about the interfaces and their dependencies that would allow automatically building correct call chains representing typical scenarios of interface usage. The first version of the shallow testing framework is ready and is now being used for Qt3 tests development. The cost of developing shallow tests is mainly in populating the information in the database + debugging automatically generated tests.
- Normal tests – this is basically tests written manually in C or C++, though we do use some automation here as well. We have been inspired by the generator that we found in the existing LSB tests for GTK – gtkvts. Basically, it allows automatically running the same code with different parameters thus achieving better testing quality without duplicating the code. However, we found particular gtkvts implementation limited and we used only the idea while implemented our own test development technology and tools. These tools, T2C Framework, were successfully used in test development for GTK+ libraries, libstdcxx, Freetype, etc. In version 2.0, key T2C tools were redesigned and completely rewritten to provide more automation capabilities in various use cases. Still, backward compatibility with the previous versions of T2C is mostly maintained with respect to the format of input files and public API.
- Deep tests - this level of testing is hard to achieve by manual tests in C; so advanced testing technologies like ISP RAS UniTESK are necessary here. Our former project (OLVER) was based on this technology. Now we are adapting/improving it to become LSB tests for LSB Core part.
For the normal and deep tests, it is important to have linkage to the specification assertions’ text. This means when a test fails it should say which particular assertion in the standard’s text is violated as well as provide info about particular mismatch (like XX expected, YY returned). Normal and deep tests being developed in ISP RAS do have this feature and analyzing reports and debugging failures is much easier with this approach. Leveraged by visual execution environment and interactive HTML reports this gives unprecedented comfort for the users. Apart of better reporting, enabling assertion catalog also enables measuring test coverage in terms of the number of assertions checked by the tests, which gives advanced level of quality measurement.