The AZOV framework is designed for automated development of shallow-quality tests that check run-time presence of interfaces and absence of critical errors in simple use cases. AZOV helps to quickly create numerous API tests for large systems using structured syntactic information about target interfaces from the main LSB Database augmented by additional information assigned by test developers.
The AZOV test development framework consists of the following components:
- LSB Database Extension contains 11 new tables (linked with the existing ones) to keep additional information about initializing correct parameters and execution environment for target interfaces and about constraints on return values.
- Web UI for test developers to specify the additional information (i.e. visually work with data in the DB extension).
- Code Generator that automatically generates tests from the extended database. It can generate tests in both plain C (for standalone debugging of failures) and T2C (for reusing good test execution and reporting facilities) formats.
The basic idea is to provide the code generator with enough information to generate correct invocation of the target interface in a typical use case and check basic constraints on the return value. The issue is that, for many interfaces, correct invocation requires preparing correct environment and correct parameter values. And this in turn may require calling other interfaces. AZOV database extension keeps information about various initialization means for various environments and semantic parameter types (specialized types). Such information can be reused for multiple interfaces each time the same semantic types are used, which greatly speeds up the development. Correct values for simple types are created by generator completely automatically. When an interface has initialization means assigned for all of its parameters (and if necessary for initial environment), the generator can automatically compose a test for this interface.
- Test development for numerous interfaces requires relatively small effort per test and thus is cheap enough. The efficiency is best revealed for interface sets with the number of interfaces starting from hundreds.
- Fully automatic generation of tests based on information in the extended LSB Database.
- Existing information about interfaces is reused from the LSB Database.
- Additional information about interfaces is handled via handy visual user interface.
- T2C Framework features can be reused for debugging and executing shallow tests.
Currently, test suites for Qt3 (9792 interfaces tested), Qt4 (10803 interfaces tested) and libxml2 (1284 interfaces tested) libraries are being developed with the help of the AZOV technology. They will be included into the LSB 4.0 certification test suite.
The AZOV Code Generator has been implemented as a separate tool (API Sanity Checker) to generate tests directly from header files (i.e. without the need to upload data to the extended LSB database). The AZOV Web UI is not published to the public domain yet.