Open Test todo

Author: Baptiste Lepilleur
Contact: blep@users.sourceforge.net

Contents

General status

Open Test is a fairly new concept and hence has a low level of maturity. Everything is to be done.

The TestRunner interface has been defined and implemented for CppUnit 2.

Everything related to test driver need to be defined and implemented. This involves:

Property tree structure

Suite and test declarations

TestDeclarator::beginSuite() and TestDeclarator::addTest().

  • time-out delay
  • description,
  • input/expected values (for functional testing based on input/output)

Test runner configuration

TestRunner::runTests().

  • is logging enabled
  • number of threads

Test configuration

TestPlanEntry::configuration().

Two categories of data are associated to a test at this stage of the testing:

  • test specific runner configuration
  • test descriptive data

Test specific runner configuration:

  • enable logging

Test descriptive data:

  • input/expected values override

Test execution result

TestRunTracker::addTestInfo().

Notes: this method probably needs to be modified since it can be called multiple times (for example, every time a line of log is added).

  • test status: success ?
  • test status type: success/failure/fault...
  • test result message
  • test result source code location
  • test result (input/) actual/expected
  • test result log

Simple text based TestDriver

Implementing a simple text based TestDriver is required to validate the interface of the TestRunner. It should use direct method call for communication with the TestRunner (which means no monitoring).

Features:

Status: partially implemented in opentest/texttestdriver.h.

Test monitoring

The test monitoring feature needs to be implemented. It requires that the TestDriver spawn a child process that run the TestRunner and communicate with it.

Communication between the TestDriver and the TestRunner should be stream based as it allows the usage of various implementation, such as shared memory, pipe, and network... Ideally, all implementations should provide the same interface, only requiring the TestDriver to bother about the implementation type at connection time.

A spike solution is being worked on to explore shared memory on Windows (Baptiste: a working implementation is in place but need a lot of clean-up).

GUI based TestDriver

QT will likely be the first GUI toolkit to be ported as it is multi-platform and is easier to implement than MFC or wxWidget.

TestDriver selection from command-line

It should be possible to select the TestDriver to use from the command-line. This implies that all test drivers provide a similar interface.

The user should be able to choose between text driver and GUI driver from the command-line (use plug-in for TestDriver ?).