[[TableOfContents]] 

= Objectives =

The CppUnit code base has grown into a monster and there are still many features that I would like to see added.

CppUnit2 is an attempt to propose a much cleaner implementation of the current features, and to follow good C++ practices.

The main objectives are:
 * use of smart-pointer for memory management
 * better support for custom test cases (repeat...) and parametrized test methods
 * flexible enough to allow sharing of a fixture across test cases (bad practice as it breaks test isolation, but may save a lot of time at execution if the setup is heavy).
 * better assertion support (equality testing, object conversion to string,...)
 * association of user data with test cases
 * thread support (passing assertion exceptions across threads, assertion...)
 * input-based test case support (read input data from external source, such as XML files...)
 * python binding

= Getting the sources =

CppUnit2 sources are finally available on CVS. You can obtain them as follows:
{{{
cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/cppunit login
cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/cppunit co cppunit2
}}}

= Features =

 * /Assertions
 * /DescriptiveData
 * /ConvertToString
 * /EqualityTesting
 * /ParametrizedTestCase

= Compiler Requirements =

Four major requirements:
 * C++ exception
 * namespace support
 * enough template support to be able to implement /GenericFunctor
 * C++ style cast (easy to remove)
 * explicit keyword (easy to remove)

Compiler features that are '''not''' required:
 * function template overloading
 * template member support
 * run-time type information (rtti)
 * template partial specialization

Also, CppUnit2 should compile correctly on MSVC6/7.x and gcc 2.95, as those are probably the two most widely used compilers.

= Compatibility with CppUnit 1.x =

 CppUnit2 is implemented in a different namespace (Cpp``UT). This allows a compatibility layer to be created, as well as having both frameworks cohabiting. 

 It should be possible to build a compatibility layer for test fixtures, assertions, auto registration, hence preserving most of the written test code. Most of the remaining code written against other would like need to be ported (test listener, test registry, test runner, test context...).

= Build system =

 Experience with the autotools has shown that it is far from being perfect (AIX still doesn't support shared libraries), and that very few people actually know how to maintain and fix such build scripts. Also, the autotools don't work on Windows (managing builds on Windows with a total of about 8 configurations per project would be too much work).

 [Scons http://scons.sf.net] seems to be a good candidate for multiplatform builds. It should also be fairly easy for a user to 'hack' into the build script to adapt it to their compiler when not supported.

= Memory management =

 In CppUnit 1, it was difficult to know when passing a parameter what its expected life was and who was responsible to delete it.

 CppUnit2 solves this by using smart-pointer for dynamically allocated objects, and the following conventions:
  * pass by smart-pointer when the ownership of the parameter is transfered/shared.
  * pass by const reference indicates a 'pass by value', the parameter will be copied if needed.
  * pass by non const reference indicates that the ownership will not be assumed. The life-cycle requirement depends on the semantic of the function. In most case, if it is a constructor, then the reference is expected to be valid as long as the object being constructed. 

 The use of smart-pointer has greatly simplified the overall implementation (especially the TestFactoryRegistry).

 CppUnit2 comes with its own smart-pointer, which implements a subset of the generic smart-pointer that has been adopted by the standard C++ committee (neither weak pointer, nor checked_deleter are supported).

 A standard conformant implementation ([http://www.boost.org/libs/smart_ptr/smart_ptr.htm boost:shared_ptr]) can be used if CPPUT_USE_BOOST_SHARED_PTR is defined.

= Parametrized test method =
 In the implementation of the test case, the callback for setUp(), run() and tearDown() are provided through generic functors. Those functors can be bound to a struct functor, a C function or a C++ method. Parameters can also be bound by the functors so that when the call is made, specific values are passed as parameters.

= Test fixture sharing for test case methods / shared resources =
 When writing heavy functional test cases which have a time-consuming setUp()/tearDown() (often database related), you often want the setUp()/tearDown() step to be executed only once to run all the test cases to decrease the time required to run the test.

 The obvious drawback of this is that tests are not isolated from each other, which means that a test may pass because a test before it was run, or a test may fail because a previous test failed.

 This can easily be implemented by dynamically controlling the behavior of Fixture``Factory (should the last created fixture be returned, or should a new one be returned).

 * Open issue: how should the user specify that a shared fixture policy needs to be used?
 * Possible solutions: 
  * make this a 'property' of the fixture creation and have the Fixture``Factory look up those properties
  * provide specific methods on the Fixture``Factory and wrap them into macros for helper macro integration

 Notes: currently trying to approach this from a different angle. Shared test fixtures with single setUp/tearDown are complex to implement with some of the new features of the framework (such as multiple execution of a test case or execution of a few test cases of a test fixture). The rough idea is to dissociate the setUp/tearDown from the test fixture (let's call them ''resource'') and have the test framework properly handle those resources.

 After some thinking, the shared test fixture will be taken out. It's basically trying to achieve what the resource concept would do without materializing it. The idea of the resource concept is to have test cases depend on some resources. A resource is a simple C++ object with a setUp/tearDown, which can be retrieved using a key (likely a string identifier). A resource is shared among multiple objects and is initialized only once per test run.

 What's interesting about resources is that you have explicit resource dependency tracking and that, for example, in a concurrent test runner, you could prevent two tests using the same resource from running concurrently. This puts more complexity on the test driver side, but it provides much more flexiblity and make things easier to understand.

= Assertions =
   The goals when writing the new assertions facility were to:
  * make it easier to write custom assertions (specifically, being able to write complex assertions by reusing simple ones, support for default values in arguments)
  * decouple customization of conversion to string and equality comparison in equality assertions
  * provide more flexibility in customization of conversion to string and equality comparison
  * allow string types to be decorated with "..." so whitespace can be seen
  * provide string-specific assertions
  * provide some support for string types other than std::string.

  Two alternative designs were implemented and experimented with before electing the second one the winner.

  Here is how an assertion is written with CppUnit2:
{{{
# define CPPUT_ASSERT            \
   CPPUT_BEGIN_ASSERTION_MACRO() \
   ::CppUT::checkTrue

void checkTrue( bool shouldBeTrue, const Message &message )
{
   if ( shouldBeTrue )
      return;

   Message newMessage( message );
   newMessage.insertAt( 0, translate( "Assertion failed: expression did not evaluate to true." ) );
   fail( newMessage );
}
}}}

 The CPPUT_ASSERT macro first captures the location of the assertion (file, line...) with CPPUT_BEGIN_ASSERTION_MACRO() and then forwards the call to the function checkTrue() which implements the assertion. The final call to fail() will actually throw the assertion exception, using the location that was previously captured.

 This idiom supports overloading, default parameter values and templatized assertion functions, and conversion of macro parameters to strings (for insertion in messages) is still possible.

''Note: currently investigating the possibility of capturing the call stack in addition to the file and source line.''

 Here is a list of the assertions provided. The names should speak for themselves:
 * Basic assertions
  * CPPUT_ASSERT
  * CPPUT_ASSERT_EXPR( expression )
  * CPPUT_ASSERT_FALSE
  * CPPUT_ASSERT_EXPR_FALSE( expression )
  * CPPUT_ASSERT_EQUAL
  * CPPUT_ASSERT_NOT_EQUAL
  * CPPUT_ASSERT_THROW( expression, Exception``Type )
  * CPPUT_ASSERT_NO_THROW( expression )
  * CPPUT_ASSERT_ASSERTION_FAIL( assertion )
  * CPPUT_ASSERT_ASSERTION_PASS( assertion )
 * Assertions working with string
  * CPPUT_ASSERTSTR_START
  * CPPUT_ASSERTSTR_END
  * CPPUT_ASSERTSTR_CONTAIN
  * CPPUT_ASSERTSTR_EQUAL
 * Enumeration assertions
  * CPPUT_ASSERT_SEQUENCE_EQUAL
  * CPPUT_ASSERT_SET_EQUAL
  * CPPUT_ASSERT_STL_SEQUENCE_EQUAL
  * CPPUT_ASSERT_STL_SET_EQUAL

 The conversion of objects to strings, and test of two objects for equality services have been separated from the equality assertion implementation. As they are orthogonal concepts, they can now be customized separately.

 The new assertions working with strings introduced another service: conversion of a string type to an std::string. This allows the testing framework to work with unknown string type, as long as the user provides the required conversion functions. Those assertions will always convert their string arguments to std::strings before proceeding with the assertion test. This allows testing of two ''const char *'' pointers for equality as strings instead of pointers.

 The conversion of objects to strings will recognize string types and automatically escape and quote them. No more chasing non-printable characters on equality failure messages.

 Customization is done completely within headers and can therefore be done on a per-translation unit/application basis.

''Note: some regular-expression-based assertions would be useful.''
 
= Test descriptive data =
   Functional testing and environments that require extensive documentation often wish to associate data to test cases, such as a description for example.

   It would be desirable to work with other test case management projects and define a standard and shared xml test output. Applications to view the test results could be developed and managed separately from the test library. It would also allow the consistent management of multi-language systems. A system that has perl, Java, and C++ could have language-specific test frameworks, but share common test-reporting tools via the common xml test output.

  Also, data attached to test cases could be used for auditing existing tests, managing dependencies, concurrency...

  Being able to associate data with the result of a test case, whether successful or not, would be very useful. You could attach data such as time to run, some structured output of a failure (list of expected values/list of actual values...)

= Threaded test cases =

 Having a test case that spans multiple threads to check for thread-safety or interaction is a fairly common task. 

 This puts the following requirements over the framework:
 * thread-safety
 * test failure propagation from a thread spawned by a test case to the test framework

 == Thread-safety ==

 All components used in multiple threads must be thread-safe. (This mainly concerns assertions.)

 The checkpoint mecanism used by the assertions used a static variable. It was made thread-safe by the use of thread local storage. To achieve this, a minimal portable thread api is embedded in CppUnit2.

 The smart-pointer reference counter needs to be thread-safe.

 Care must be taken when manipulating std::strings across thread as they often use non-thread-safe reference counting.

 Notes: WIN32 implementations for this are done. pthread & MacOS implementations need to be done.  (Mac OS X uses pthread for its threading model)

 == Test failure propagation ==

 Assertions that occur in threads spawned by a test case need to be transfered to the main test framework thread and associated with the test case.

 This is achieved by reusing the Test``Failure``Guard``Chain components which capture test case uncaught exceptions and assertions. These components are reused to protect spawned thread main loop and 'memorize' failures. Those failures can then be dispatched in the main test framework thread.

== Test plug-in ==

 There are two main uses for plug-ins in cppunit:
 * writing unit tests in plug-ins
 * customizing/extending cppunit at run-time

 Unit test plug-ins are already handled by linking against a dll build of cppunit 2. The static registry is shared among users of the dll. A simple runner needed to be written though.

 Customization/extension should provide more flexibility than in cppunit 1. A very interesting goal to achieve would be to allow graphic test runner to be a simple 'extension', though this might be complex to achieve.

= Input-based test cases =
  
= Python binding =

Expose execution of unit tests, and other high-level functionnalities in python. If possible provide integration with python unit testing module.

 * Ease the development of new test runners (test plug-in runner...) and increase their reusability
 * Integration with python environment for unit testing

= Integration with test-driving environments =

 Study possiblity to integrate the test framework with:
 * [Cruise``Control http://cruisecontrol.sourceforge.net/]
 * [Cruise``Control.NET http://ccnet.thoughtworks.com/]
 * [QMTest http://www.codesourcery.com/qm/qmtest]
 * ?

= Architecture =

The test hierarchy:

attachment:uml_test.gif

Main changes compared to CppUnit 1.x include the use of the visitor pattern, a reference to the parent test in each test, and the use of generic functors in test case implementation.

 == Visitor pattern to distinguish between test case and test suite ==
   It simplifies implementation, and gives much more flexibility to explore the test hierarchy...

 == Each test references its parent ==
   * Helps provide useful reports on failure (the full test suite path can be easily reported). 
   * Does not depend on execution order to track which test suite the test belongs to. 
   * Avoid the ugly hack in the test fixture to prepend the method name with the fixture class name to report the suite the test belongs to.

 == Generic Functor in test case implementation ==
   * Test setUp/tearDown and run can be implemented in any form (functor, C function, member function)
   * Parameters can be bound to the generic functor implementing setUp/tearDown and run, thereby allowing simple test parametrization.
`
= To do =
 * build system
 * thread api for pthread (need to implement Thread``Id``Type)
 * thread api for Mac OS
 * thread guard (failure propagator for spawned thread in test case)
 * test plug-in
 * test listener plug-in
 * plug-in loader & manager
 * progress listener
 * result outputter (text, compiler, xml)
 * test plan (find test by pattern...)
 * test runner (mfc, qt, command line based)
 * input-based test case (likely to have an impact on the core design)
 * shared test fixture with single setUp/tearDown (likely to have an impact on the core design)

= Progress =
 * 2004/06/16:
  * enhanced assertenum.h to support convertion to string customization through functor.
  * idea: enhance class Message to carry a bag of properties, allowing structured result. Find a way to get this into success too (could be used to store elapsed time...). At the current time in cppunit 1, we must use a test listener and can only store results with the xml outputter.
 * 2004/02/08:
  * incorporated the portable dll loading api of cppunit 1
  * added header content descriptions to README.
