API Sanity Checker

From ISP_RAS
(Redirected from API Sanity Autotest)
Jump to: navigation, search

API Sanity Checker is an automatic generator of basic unit tests for shared C/C++ libraries. It is able to generate reasonable (in most, but unfortunately not all, cases) input data for parameters and compose simple ("sanity" or "shallow"-quality) test cases for every function in the API through the analysis of declarations in header files. The quality of generated tests allows to check absence of critical errors in simple use cases. The tool is able to build and execute generated tests and detect crashes (segfaults), aborts, all kinds of emitted signals, non-zero program return code and program hanging. It may be considered as a tool for out-of-the-box low-cost sanity checking (fuzzing) of the library API or as a test development framework for initial generation of templates for advanced tests. Also it supports universal T2C format of tests, random test generation mode, specialized data types and other useful features.

See also:


Contents

Downloads

Releases

All releases can be downloaded from this page or github.com.

Latest release: 1.98.6

Previous release: 1.12.11

Git

Read-only access to the latest development version:
   git clone git://github.com/lvc/api-sanity-checker.git  

License

This program is free software. You may use, redistribute and/or modify it under the terms of either the GNU GPL or LGPL.

Supported Platforms

GNU/Linux, FreeBSD, Mac OS X, MS Windows, Haiku/BeOS.

System Requirements

  • Linux, FreeBSD
    • ABI Compliance Checker (1.98.7 or newer)
    • G++ (3.0-4.7, recommended 4.5 or newer)
    • GNU Binutils (readelf, c++filt, objdump)
    • Perl 5 (5.8 or newer)
    • Ctags (5.8 or newer)
    • WARNING: if you are using ccache program (i.e. gcc points to /usr/lib/ccache/gcc) then it should be newer than 3.1.2 or disabled.
  • MS Windows
    • ABI Compliance Checker (1.98.7 or newer)
    • MinGW (3.0-4.7, recommended 4.5 or newer)
    • MS Visual C++ (dumpbin, undname, cl)
    • Active Perl 5 (5.8 or newer)
    • Ctags (5.8 or newer)
    • Add tool locations to the PATH environment variable
    • Run vsvars32.bat script (C:\Microsoft Visual Studio 9.0\Common7\Tools\)

Examples of Generated Tests

Library Version Number of Tests Problems Found
FreeType2 2.3.11 178 13
Glibc 2.13 1996 340
libX11 1.3.4 778 286

Detectable Problems

  • Crash (segfault, signal SEGV)
  • Abort (signal ABRT)
  • All emitted signals: FPE, BUS, ILL and others
  • Non-zero exit code
  • Program hanging
  • Requirement failure (if specified)

Installation

The tool is ready-to-use after extracting the archive. You can also use a Makefile to install the tool into the system:
 cd api-sanity-checker-x.y.z/ 

 sudo perl Makefile.pl -install --prefix=PREFIX [/usr, /usr/local, ...] 

This command will install an  api-sanity-checker  program in the  PREFIX/bin  system directory.

To verify that the tool is installed correctly and it works on your host run:
 cd tmp/ 

 api-sanity-checker -test 

Usage

For generating, building and running tests you should provide the XML descriptor for your library version. It is a simple XML-file that specifies version number, paths to header files and shared libraries and optionally some other information. An example of the descriptor is the following (0.3.4.xml):

<version>
    0.3.4
</version>

<headers>
    /usr/local/libssh/0.3.4/include/
</headers>

<libs>
    /usr/local/libssh/0.3.4/lib/
</libs>

Generating Tests

Command for generating a test suite:
   api-sanity-checker -lib NAME -d VER.xml -gen 

Viewing Tests

You can view generated tests using the index file:
   tests/NAME/VER/view_tests.html 
or manually in the directory:
   tests/NAME/VER/groups/ 

Building Tests

Command for building tests:
   api-sanity-checker -l NAME -d VER.xml -build 

Running Tests

Command for running tests:
   api-sanity-checker -l NAME -d VER.xml -run 

After a time will be generated test report:
   test_results/NAME/VER/test_results.html 

Improving Tests

To improve generated tests quality, you can provide the collection of specialized types for the library.

Command-Line Options

The tool has many useful options for manipulating with the test generation and execution processes. See the list of all options on this page.

Most useful options:

Examples

Generate, build and execute the tests for the libssh library:
   api-sanity-checker -l libssh -d 0.3.4.xml -gen -build -run 

Tests will be generated to:
   tests/libssh/0.3.4/groups/ 
   tests/libssh/0.3.4/view_tests.html 

The report will be generated to:
   test_results/libssh/0.3.4/test_results.html 

Generate tests in the Template2Code (T2C) format:
   api-sanity-checker -l libssh -d 0.3.4.xml -gen -t2c 

The package with T2C tests will be generated to:
   tests_t2c/libssh/0.3.4/t2c-libssh-tests-0.3.4.tar.gz 

Execute the tests using the Xvfb server:
   api-sanity-checker -l allegro -d 4.9.21.xml -run -xvfb 

Tutorial

See "Smoke test" article at glibc wiki.

Similar Tools

  1. Trinity - The Linux system call fuzzer,
  2. iknowthis - A system call fuzzer for UNIX-like systems,

How the Tool Works

The basic idea of the test data generation algorithm is to recursively initialize parameters of a function using the values returned (or returned through the out-parameter) by other functions for structured data types (class, struct, union) or by some simple values for intrinsic data types (int, float, enum, ...). The recursion step includes the heuristic selection of the appropriate function, that should be called to initialize complex parameters for other functions. If some parameter of a function cannot be initialized, then the algorithm tries to select other function.

Let's see the example test for FT_Activate_Size ( FT_Size size ) function from the FreeType2 library:

   #include <freetype/freetype.h>
   int main(int argc, char *argv[])
   {
       FT_Library alibrary = 0;
       FT_Init_FreeType(&alibrary);//initialize "alibrary"
       
       FT_Face face = 0;
       FT_New_Face(
           alibrary,
           "sample.ttf",
           0,
           &face);//initialize "face"
       
       FT_Size size = 0;
       FT_New_Size(face, &size);//initialize "size"
       
       FT_Activate_Size(size);//target call
       return 0;
   }

In this test case the parameter "size" of target function FT_Activate_Size is initialized through the call of FT_New_Size function using its 2nd out-parameter. The first parameter "face" of FT_New_Size function is recursively initialized by the use of FT_New_Face function's 4th out-parameter. And finally the first parameter "alibrary" of FT_New_Face is initialized by the call of FT_Init_FreeType function on the 3rd recursion step. Other parameters of FT_New_Face function are initialized by intrinsic values.

Bugs

Please send your bug reports, feature requests and questions to the the maintainer, post to issue tracker or mailing list.

Applications

LSB 4.0 certification test suites for Qt3 (9792 interfaces tested), Qt4 (10803 interfaces tested) and libxml2 (1284 interfaces tested) libraries were developed with the help of this tool, also known as AZOV Code Generator in the past (2007-2009).

Future Plans

  • Add CUnit format support. See 2011 GSoC LSB projects.
  • Extending support of Splint and ACSL annotations.
  • Making better distinction between pointer arguments (*x) and arrays (x[]).
  • More kinds of generated test data files (images, fonts, ...).
  • Option for generating bound values for parameters.

Maintainers

The tool was originally developed by the Russian Linux Verification Center at ISPRAS and since 1.12.9 version it's developed by the ROSA Laboratory in Russia. Andrey Ponomarenko is the leader of this project.

Credits

We would like to thank everyone who has contributed to the success of this project!

Articles

  1. wikipedia.org, “Sanity testing in software development”
  2. wikipedia.org, “Fuzz testing”
  3. R. S. Zybin , V. V. Kuliamin , A. V. Ponomarenko , V. V. Rubanov and E. S. Chernov, “Automation of broad sanity test generation”
Personal tools