Problem Database 2

From ISP_RAS
Jump to: navigation, search

Problem_db2 file describes known issues and test suite deficiencies.

Some complex example of a problem record:

STANDARD LSB >= 3.2 | Moblin
ARCHITECTURE x86,x86-64
TESTSUITE core < 3.2 | runtime >= 3.2 & != 4.0.12
TEST -g /24_iterators/istreambuf_iterator/{2,32}.cc {2..10}
   | /24_iterators/istreambuf_iterator/52.cc {15,20..25,30}
  SEVERITY failed
   MARKER -r _\d+_iterators_istreambuf_iterator_\d+\.exe
   MARKER what(): float
   OR
   MARKER terminate called after throwing an instance of 'std::runtime_error'
TESTCASE -r /\d+_iterators/istreambuf_iterator/\d+\.cc
 TESTPOINT 3..15,10,20
  SEVERITY failed,unresolved
   MARKER ^
 TESTPOINT 33
  SEVERITY failed
   MARKER ^
COMMENT Probably a bug in test suite header files (works with normal header files).
COMMENT The same problems are PR 0065 and PR 0095
LINK http://bugs.linuxbase.org/show_bug.cgi?id=1234 | bug #1234
RESOLUTION Test suite deficiency
WAIVE fip
END

Each problem record consists of several lines and ends with the 'END' line.

Each line starts with a keyword, usually followed by some information.

Problem_db2 file consists of two kinds of lines: matchers and info lines. Mat—Āhers are used to define the problem case and then detect it when the error appears in a test journal. Info lines provide description of the problem shown in the report.

Keywords are case insensitive, the rest of the text in line is usually case-sensitive.

Matchers:

STANDARD - specify standard version against which the problem will be matched. It restricts problem matching, so specify it only if the problem shouldn't be processed when testing against another standard version. (See version spec syntax for complex cases.)

ARCHITECTURE - specify architectures (comma-separated, without spaces) if the problem is specific to some architectures. Don't specify ARCHITECTURE if the problem isn't architecture specific. Allowed architecture names are: x86, x86-64, IA64, PPC32, PPC64, S390, S390X.

TESTSUITE - specify test suite name. (See version spec syntax for complex cases.)
Note: For version bounding of an entry, the test suite name must be on both sides of the pipe symbol (core < 4.0.0 | core > 4.1.0 & < 4.1.2). There is currently a bug in dist-checker that prevents bounding from working correctly: bug 3227.

TESTCASE - specify problem test case name. By default test case name is a fixed line. You may pick it from a test journal or report. If the test case is prefixed with -g key, then it is treated as 'globbed' expression: enumerations in curly braces will be expanded multiplying the set of matching test cases. (See the example.) Nested curly braces aren't allowed. Another key: -r states for regular expression (Perl Compatible Regular Expressions). Use regular expressions carefully.

You may specify several testcases joining them with ' | '. Note that space after bar is required.

TESTPOINT - specify test case points. Default format for test points is range format. If the test journal doesn't exploit test points, put '0' here. You can also specify '*' which matches any test point number. TESTPOINT line hierarchically attaches to the most recent TESTCASE line (the example above demonstrates it). TESTPOINT line may be omitted -- this case is equivalent to 'TESTPOINT *'.

TEST - specify problem test case name and test point in one line. This is just a mix of TESTCASE and TESTPOINT specifications. However, there are differences: test case shouldn't contain spaces, both test case and test points are in the glob format (i.e. put test points in braces if you specify more than one item).

Don't mix TEST specifications with TESTCASE or TESTPOINT lines - this will lead to an unobvious behaviour.

SEVERITY - specify expected test case severity. Look in the report for it. Several severities may be specified comma-separated. SEVERITY line hierarchically attaches to the most recent TEST or TESTPOINT line. SEVERITY line may be omitted -- this is equivalent to 'SEVERITY *'.

MARKER - specify expected text issued by the test. Default format is fixed string which to be searched in the message. Regular expressions are also allowed using the '-r' prefix (Perl Compatible Regular Expressions format). MARKER lines hierarchically attach to the most recent SEVERITY line. Several MARKER lines form a block in which the lines are searched sequentially in the order they are specified. Several MARKER blocks may be joined by 'OR' line.

A special shortcut 'MARKER ^' exists to allow to reuse the whole previous MARKER declaration (more precisely, everything below the previous SEVERITY line).

Info Lines:

COMMENT - specify problem information. COMMENT lines are joined together keeping line breaks. It is ok to have a very long line after the COMMENT keyword -- it will be wrapped automatically. It is much better than splitting text into many short COMMENT lines. Start another COMMENT line only if you wish to enforce line break.

COMMENT format is plain text. You can insert some html markup by wrapping it into {noreplace}...{/noreplace} block.

LINK - append a hyperlink to the comment text. Format of the link line is as follows: http://someurl.org | Link title.

RESOLUTION - a short (two-three words) description of the problem. It appears in the last column of problems table in report.

WAIVE - waive the problem. You may also specify another verdict to be enforced for this test case (e.g. 'WAIVE warning' will change test result to 'warning').

END


See also:

Personal tools