LSB DTK Manager Nightly Run HOWTO
LSB DTK Manager supports automation of scheduled (e.g. nightly) runs of the LSB certification tests on multiple test systems with collection of the results on a single server. This document describes how to organize this process.
A typical architecture of a test farm is presented on the following picture.
Test farms can contain one or more systems used for test execution (test stands) and a server for collecting test results. Collected results can be seen and analyzed through Web interface. For large test farms, it could be sensible to use a mirror of the LF FTP server with test packages.
Typical workflow includes:
- Place necessary tests to a FTP server (preferably local but LF main FTP can be used by default).
- Prepare target systems to be tested (e.g. take development sources snapshot, build and install them).
- Setup a Test Results Server for collecting test results (expose an ssh accessible writable directory).
- Run DTK Manager via command line on all the test systems - it will take necessary tests from the FTP, run them and upload results to the Test Results Server.
- Analyze all the results collected at the Test Results Server via Web-interface.
The following sections describe:
- how to setup a local FTP Mirror with test packages to be run,
- how to setup Test Stands with systems to be tested,
- how to setup a Test Results Server to store test results,
- how to browse test results collected.
FTP Mirror Setup
By default DTK Manager looks for test packages on the ftp.linux-foundation.org server in the '/pub/lsb' directory in the following subdirectories: 'test_suites', 'app-battery' and 'snapshots' (daily snapshot builds). DTK Manager can be told to look for packages in another directory on another server (see the Test Execution below).
It is smart to set up a local mirror with test packages to be run (especially if your test farm consists of many test stands). It will reduce your traffic and load on LF's server and will speed up the process.
You can maintain your FTP mirror as you like. We're using ftpcopy for this. Note, that ftpcopy can't work with a proxy. In this case you could try more advanced tools, e.g. lftp.
There is our script for ftpcopy:
ftpcopy -x *.deb -x */archive/* \ ftp://ftp.linux-foundation.org/pub/lsb/test_suites/ \ pub/lsb/test_suites -s --ignore-time –-progress -l 3 ftpcopy -x *.deb -x */archive/* \ ftp://ftp.linux-foundation.org/pub/lsb/app-battery/ \ pub/lsb/app-battery –s --ignore-time --progress –l 3
Here are one line for mirroring the 'test_suites' directory and one line for the 'app-battery' directory. Here are excluded all *.deb files because DTK Manager doesn't use them (it uses alien instead). Archive directories are also excluded. For other options, please look the program's man page.
Test Stands Setup
Nightly test run usually consists of two steps:
- Preparation of System Under Test.
- Test Execution.
Preparation of System Under Test
System Under Test preparation step is to update the target system to be tested with any changes necessary and make it a working Linux system. The typical examples are:
- For distribution developers: it is necessary to test nightly builds of a distribution under development. In this case the preparation step includes building the distribution from the most recent sources and its deployment to the test stand.
- For upstream component developers: it is necessary to nightly test the latest snapshot of an upstream component under development. In this case preparation step is to build the component from the latest sources and update the Linux at the test stand with this new component.
When a system under test is ready, test execution can be started. LSB DTK Manager takes care of automatic test execution and uploading results to a test results server.
The command line interface to the DTK Manager is provided by the autotest-ext/lsb-tef.pl perl script. Here is an example of command line that can be used for nightly test runs (one line):
# /opt/lsb/test/manager/autotest-ext/lsb-tef.pl all --lsb=3.1 -v2 –D --ftp '<your-ftp-mirror>/dirname' --upload '<user>@<host>:/remote/directory' --comment 'Nightly run on <host>' 2>&1 |tee /tmp/tests_output.8888
All the options are described below:
- all defines a set of tests. In this example all tests are selected to be run. You can select a custom set of tests. In order to do that, specify one or more test names separated by spaces. The names of available tests could be obtained using the --help option.
- --lsb=3.1 means that the distribution will be tested against LSB version 3.1. Available values here are '3.0', '3.1' and (in later versions of DTK Manager) 'snapshot'. If you're going to run tests from snapshot, please read the corresponding section below.
- -v2 parameter turns on verbose mode. There are three modes: -v0 is silent, -v1 is normal (default), and -v2 is verbose (debug). Verbose mode is helpful for exploring unexpected failures.
- -D enables DTK Manager to download packages from the Internet.
- --ftp '<your-ftp-mirror>/dirname' option changes FTP server from default (ftp.linux foundation.org) to another (mirror) server. '/dirname' is a base directory at the mirror. For example it is '/pub/lsb' for the LF server. Note: actual path to tests is constructed by concatenating the dirname with proper subfolder defined by --lsb= option.
- --upload '<user>@<host>:/remote/directory' says DTK Manager to upload the results via SSH to the specified directory. Actually, two programs are used for this: rsync and expect. Be sure to install them if you are using this option. Also, please, read the section Passing the SSH Password below.
- --comment 'Nightly run on <host>' attaches a short description which will be displayed in the results table.
- And the last part is 2>&1 |tee /tmp/tests_output.8888. The tee program is used here to save all the output to a log file. It is important to use this particular file name and the path because DTK Manager looks for this file to add it to the results tarball.
You can read about other available command line options here.
Passing SSH Password
For uploading files on a remote host via ssh, a valid password shall be provided. This could be done in several ways.
1. The most insecure way is to use the '--ssh-password' option. Example:
# ./lsb-tef.pl cmdchk --upload 'user@host:/dir' --ssh-password 'password'
In that situation any user can see the password running the command 'ps aux' .
2. Another way is to provide the password in the SSH_PASSWORD environment variable. Example:
# export SSH_PASSWORD=password ; ./lsb-tef.pl cmdchk --upload 'user@host:/dir'
In that case the password isn't displayed by the 'ps aux' command, and also isn't visible (for non-root users) in the /proc/<pid>/environ file.
3. One more way is to write the password into a file that is readable for root only, and then run DTK Manager as follows:
# cat PASSFILE | ./lsb-tef.pl cmdchk --upload 'user@host:/dir'
DTK Manager takes password from the standard input at the start.
4. The similar way is to specify the password in a profile file, and then use it as follows:
# ./lsb-tef.pl cmdchk -f '<profile filename>'
All the command line parameters can be passed to DTK Manager in a profile. Read the LSB DTK Manager Profile Syntax page for more information.
5. Another way of SSH authorization is to setup passwordless SSH based on authorized keys. In that case you should also provide any word as password because if no password specified, DTK Manager asks for it from the standard input at the start.
Choose the way of authorization you believe to be secure enough.
Running snapshot tests (development version of tests)
To run tests from snapshots, use the --lsb=snapshot key.
Also we recommend using the two following options:
- --force-dl makes DTK Manager download all packages before running tests. Also it will re-download cached packages. Without this option DTK Manager will download packages right before running each particular test and won't re-download cached files.
- --force-reinstall makes DTK Manager reinstall all tests. By default DTK Manager doesn't reinstall the package if the installed version matches. Such behavior isn't good when a package being built every night, but the filename (and version) keeps the same.
Test Results Server Setup
You need to choose a directory at the test results server for storing test results collected from test stands. Make sure the directory chosen has enough room for storing the test results (~40Mb per full test set result) and it is writable for the 'user' used for uploading test data from the test stands (see --upload option in the Command Line section).
Web Server Setup
We have developed simple interface for visual browsing of collected test results. You should setup a web server with PHP support on the tests results server machine (e.g. Apache) in order to use it. The interface is just auto_dtk_results.php page that can be found at the 'php_auto_results' directory of DTK Manager installation (usually /opt/lsb/test/manager/php_auto_results/auto_dtk_results.php). Copy this page to an appropriate directory (www), and setup your web server so this page could be accessed via HTTP. The results directory should also be accessible (you could create a symlink to the results directory in your www directory).
Then edit $result_dir and $www_root variables in the auto_dtk_results.php file. The first variable shall contain the file system path to results. The second variable shall contain the HTTP hyperlink to this directory. auto_dtk_results.php script reads the results directory and builds a table with hyperlinks to reports, short descriptions and comments.
Note: the current version of auto_dtk_results.php is quite simple – this will be upgraded later with filters/sorting and additional decorations.
Browsing Test Results
An expert responsible for analyzing test reports can use its browser to view the table of uploaded test reports. Use 'http://your_results_server/auto_dtk_results.php'.
The results page consists of a table each row of which describes a test run: the first column contains links to a detailed report, the second column contains statistics of test run, and the last column contains your comment. To view any particular report, just click the corresponding link in the first column.
The Detailed Report page contains information about every test suite executed. Any failed tests that have already been waived will be marked as such in the report. It also contains links to test run logs – they could be helpful when discovering sources of failures.
Should you have any problems in implementing nightly test runs with DTK Manager, please contact us via e-mail linux _at_ ispras.ru.