LSB is an interface standard and LSB certification is based on running automated tests that verify conformance of a Linux product (distribution or applications) to the requirements of LSB. There are two types of LSB certification:
- Certification of distributions - ensure that certified distributions provide all LSB interfaces and that the interfaces behave as specified.
- Certification of applications - ensure that certified applications use only LSB interfaces and that they do this in a correct way.
DTK Manager uses underlying test suites for actual testing. A test suite contains a number of test cases each of which checks one or more LSB interfaces. As LSB contains a great number of interfaces, we have to use different levels of test quality for different kinds of interfaces to optimize value/cost. We define three testing quality grades and use different tools/technologies to develop LSB tests of corresponding quality:
- Shallow tests – simple tests with the only guaranteed purpose of ensuring the interface does not crash (in many cases it is additionally checked that the interface doesn’t return an error code) being called with some particular correct parameters and in the correct environment. This is close to “existence” or “smoke” or “sanity” tests (but beware - these terms are interpreted differently by different experts). We use a new technology and tools (AZOV Framework) for automatic shallow tests generation based on some description of the interfaces, their parameters, interlinks and default values in the LSB database. The core idea here is to augment the database to contain enough information about the interfaces and their dependencies that would allow automatically building correct call chains representing typical scenarios of interface usage.
- Normal tests – this is the most reasonable level of testing quality. Normal tests check the main functionality and may check a few error scenarios. Most of the existing LSB tests are of this quality. Though normal tests are basically manual C/C++ tests that use high level testing API helpers and underlying harness, we do use some automation here as well. We have developed T2C Framework, the first versions of which were inspired by the ideas used in GTKVTS: automatically running the same code with different parameters. This helps achieve better testing quality without manually duplicating the source code.
- Deep tests – this is the level when most of the specification assertions are tested in various conditions/states. This level of testing is hard to achieve by manual tests in C; so advanced testing technologies like ISPRAS UniTESK are necessary here. Our former project (OLVER) was based on this technology. Now we are adapting/improving it to become LSB tests for LSB Core part.
Please use the links above to check the current development status of corresponding tests. Test coverage of the old legacy tests (LSB 3.1) as well as test coverage improved by the new tests (LSB 3.2 and later) can be viewed in LSB Navigator.