Distribution Checker Nightly Run HOWTO
Distribution Checker may be used for scheduled (e.g. nightly) runs of LSB and Moblin certification tests on multiple test systems with collection of the results on a single server. This document describes how to organize this process for LSB Distribution Checker. In the case of Moblin Distribution Checker everything goes in the similar way.
A typical architecture of a test farm is presented on the following picture.
Test farms can contain one or more systems used for test execution (test stands) and a server for collecting test results. Collected results can be seen and analyzed through Web interface. For large test farms, it could be sensible to use a mirror of the LF server with test packages.
Typical workflow includes:
- Put necessary tests to the server (preferably local but LF main FTP can be used by default).
- Prepare target systems to be tested (e.g. take development sources snapshot, build and install them).
- Setup a Test Results Server for collecting test results (expose an ssh accessible writable directory).
- Run Distribution Checker via command line on all the test systems - it will take necessary tests from the server, run them and upload results to the Test Results Server.
- Analyze all the results collected at the Test Results Server via Web-interface.
The following sections describe:
- how to setup a local mirror server with test packages to be run,
- how to setup Test Stands with systems to be tested,
- how to setup a Test Results Server to store test results,
- how to browse test results collected.
By default Distribution Checker looks for test packages on the ftp.linux-foundation.org server in the '/pub/lsb' directory as specified in Manifest file. Required packages are listed in the Manifest (/var/opt/lsb/test/manager/manifest/Manifest). To make Distribution Checker use files from another place, Manifest should be slightly modified (see Modifying Manifest below).
It is smart to set up a local mirror with test packages to be run (especially if your test farm consists of many test stands). It will reduce your traffic and load on LF's server and will speed up the process.
We have prepared a little utility to maintain a mirror. It takes a list of files to be mirrored from the Manifest file, and allows you to download only those files that are required for running tests with the Distribution Checker. This utility can be found at /opt/lsb/test/manager/utils/server_scripts/mirror.pl. It is based on several modules from DTK Manager located in the 'utils' directory; that are Common.pm , Manifest.pm and Download.pm. Copy mirror.pl script to a directory on your mirror server and put these modules to the same directory.
Usage of mirror.pl is as follows:
- First, you should download the latest version of Manifest file.
- Then you should automatically modify this Manifest file as described in section Modifying Manifest below.
- Then run mirror.pl:
./mirror.pl -m ./Manifest -s 'LSB 4.0' -S cert -p rpm --mirror --dir /srv/ftp/pub/lsb --purge-old
This command line tells mirror.pl to download all files required for certification (-S cert) for standard LSB 4.0 (-s 'LSB 4.0') of a rpm-based system (-p rpm). A mirror will be created in the directory /srv/ftp/pub/lsb. Option --purge-old means that unneeded files in this directory will be removed (be careful).
This script supports parameters for using proxy. See ./mirror.pl --help for description of available options.
To make Distribution Checker take files from another host, Manifest file should be modified. There are two places in the file to be modified:
- At the very beginning of the file:
that points to the original Manifest file. Change this URL so it should point to your modified copy of Manifest. This copy should be available for downloading by users from your server.
- The section [SERVERS] (usually near the middle of the file). This section specifies main directory at the remote file server. By default there are two locations: 'http://ftp.linuxfoundation.org/' and the same for 'FTP' protocol. You should replace both these lines with an address of your mirror server. (Make sure to keep backslash '/' at the end of string.)
The following script may be used to automate these actions:
sed 's!^URL = .*!URL = http://yourhost.org/path/to/Manifest!' -i Manifest sed 's!^\s\+\(ht\|f\)tp://\S\+! \1tp://yourhost.org/!' -i Manifest
Replace "yourhost.org" and "/path/to" with name of your host and proper path in both lines of the script.
Then copy the modified Manifest to each client machine into directory /var/opt/lsb/test/manager/manifest. This is needed be done only once, to make the new URL be used on these machines for automatic updates.
Test Stands Setup
Nightly test run usually consists of two steps:
- Preparation of System Under Test.
- Test Execution.
Preparation of System Under Test
System Under Test preparation step is to update the target system to be tested with any changes necessary and make it a working Linux system. The typical examples are:
- For distribution developers: it is necessary to test nightly builds of a distribution under development. In this case the preparation step includes building the distribution from the most recent sources and its deployment to the test stand.
- For upstream component developers: it is necessary to nightly test the latest snapshot of an upstream component under development. In this case preparation step is to build the component from the latest sources and update the Linux at the test stand with this new component.
When a system under test is ready, test execution can be started. Distribution Checker takes care of automated test running and uploading results to a test results server.
Command line interface to the DTK Manager is provided by utils/dist-checker.pl script. Here is an example of a command line to be used for nightly test runs (single line):
/opt/lsb/test/manager/utils/dist-checker.pl all --standard LSB4.0 -S cert -v2 –D \ --comment 'Nightly run on '`uname -nm` \ --post-cmd '/home/user/your_upload_results_script.sh' # optionally
All the options are described below:
- all defines a set of tests. In this example all tests were chosen for running. You can select a custom set of tests; in order to do that, specify one or more test names separated by space. The list of available tests may be obtained using the --list option.
- --standard LSB4.0 means that system will be tested against standard LSB 4.0. See --list for the list supported standards.
- -S cert Use certification set of testsuites. Alternative values are 'beta' or 'snapshot' (see --list). If you're running tests from snapshots, you may want to specify --force-reinstall option for Distribution Checker to always reinstall test packages.
- -v2 parameter turns on verbose mode. There are three modes: -v0 is silent, -v1 is normal (default), -v2 is verbose and -v3 is for debug. Verbose mode affects only what is printed on the console and to <result_dir>/log file. In addition to it a full (-v3) debug log is always available as <result_dir>/verbose_log.
- -D enables Distribution Checker to download packages from the Internet.
- --post-cmd '/home/user/your_process_results_script.sh' parameter tells Distribution Checker to run the script after the report has been prepared. The test result tarball file name will be passed to the script as a parameter. The script may for example upload test results to a remote server or do anything you write in it.
- --comment 'Nightly run on '`uname -nm` attaches a short description to this test run.
You can read more about available command line options here.
Test Results Server Setup
You need to choose a directory at the test results server for storing test results collected from test stands. Make sure the directory chosen has enough room for storing the test results (Up to 90Mb per full test set result, or 12 Mb in tar.gz) and it is writable for uploading test data from test stands.
Web Server Setup
We have developed a simple interface for visual browsing of collected test results. You should setup a web server with PHP support on the tests results server machine (e.g. Apache) in order to use it. The interface is just auto_dtk_results.php page that can be found at the 'utils/server_scripts' directory of Distribution Checker installation (usually /opt/lsb/test/manager/utils/server_scripts/auto_dtk_results.php). Copy this page to an appropriate directory (www), and setup your web server so this page could be accessed via HTTP. The results directory should also be accessible (you could create a symlink to the results directory in your www directory).
Then edit parameters at the beginning of the auto_dtk_results.php file. That are $result_dir, $www_root and $www_this variables (explained in comments in the file). To enable journal comparison you should copy jdiff.pl utility from Distribution Checker (from the 'utils' directory) with necessary modules (Common.pm and Report.pm) to a directory on the results server. Then uncomment and modify $utils_dir parameter in the auto_dtk_results.php script so that it points to directory with jdiff.pl.
Browsing Test Results
An expert responsible for analyzing test reports can use browser to view the table of uploaded test reports. Use 'auto_dtk_results.php'.
The results page consists of a table each row of which describes a test run: the first column contains links to a detailed report, the second column contains information about the test run, and the last column contains your comment. To view any particular report, just click the corresponding link in the first column. If the report comparison feature is enabled then one can select two or more reports and press 'Compare Reports' button. Reports are being compared journal-to-journal. It may take some time to analyze large journals.
'Detailed Report' page contains information about every runned test suite. All the failed tests that have already been waived will be marked as such in the report. Report also contains links to test run logs – they could be helpful when discovering a source of failures.
Should you have any problems in implementing nightly test runs with Distribution Checker, please contact us via e-mail linux _at_ ispras.ru.