14 Common Test Hooks

14.1 General

The Common Test Hook (CTH) framework allows extensions of the default behavior of Common Test using hooks before and after all test suite calls. CTHs allow advanced Common Test users to abstract out behavior that is common to multiple test suites without littering all test suites with library calls. This can be used for logging, starting, and monitoring external systems, building C files needed by the tests, and so on.

In brief, CTH allows you to do the following:

• Manipulate the runtime configuration before each suite configuration call.
• Manipulate the return of all suite configuration calls, and in extension, the result of the tests themselves.

The following sections describe how to use CTHs, when they are run, and how to manipulate the test results in a CTH.

14.2 Installing a CTH

A CTH can be installed in multiple ways in your test run. You can do it for all tests in a run, for specific test suites, and for specific groups within a test suite. If you want a CTH to be present in all test suites within your test run, there are three ways to accomplish that, as follows:

• Add -ct_hooks as an argument to ct_run. To add multiple CTHs using this method, append them to each other using the keyword and, that is, ct_run -ct_hooks cth1 [{debug,true}] and cth2 ....
• Add tag ct_hooks to your Test Specification.
• Add tag ct_hooks to your call to ct:run_test/1.

CTHs can also be added within a test suite. This is done by returning {ct_hooks,[CTH]} in the configuration list from suite/0, init_per_suite/1, or init_per_group/2.

In this case, CTH can either be only the module name of the CTH or a tuple with the module name and the initial arguments, and optionally the hook priority of the CTH. For example, one of the following:

• {ct_hooks,[my_cth_module]}
• {ct_hooks,[{my_cth_module,[{debug,true}]}]}
• {ct_hooks,[{my_cth_module,[{debug,true}],500}]}

Overriding CTHs

By default, each installation of a CTH causes a new instance of it to be activated. This can cause problems if you want to override CTHs in test specifications while still having them in the suite information function. The id/1 callback exists to address this problem. By returning the same id in both places, Common Test knows that this CTH is already installed and does not try to install it again.

CTH Execution Order

By default, each CTH installed is executed in the order that they are installed for init calls, and then reversed for end calls. This is not always desired, so Common Test allows the user to specify a priority for each hook. The priority can either be specified in the CTH function init/2 or when installing the hook. The priority specified at installation overrides the priority returned by the CTH.

14.3 CTH Scope

Once the CTH is installed into a certain test run it remains there until its scope is expired. The scope of a CTH depends on when it is installed, see the following table. Function init/2 is called at the beginning of the scope and function terminate/1 is called when the scope ends.

CTH Processes and Tables

CTHs are run with the same process scoping as normal test suites, that is, a different process executes the init_per_suite hooks then the init_per_group or per_testcase hooks. So if you want to spawn a process in the CTH, you cannot link with the CTH process, as it exits after the post hook ends. Also, if you for some reason need an ETS table with your CTH, you must spawn a process that handles it.

External Configuration Data and Logging

Configuration data values in the CTH can be read by calling ct:get_config/1,2,3 (as explained in section Requiring and Reading Configuration Data). The configuration variables in question must, as always, first have been required by a suite-, group-, or test case information function, or by function ct:require/1/2. The latter can also be used in CT hook functions.

The CT hook functions can call any logging function in the ct interface to print information to the log files, or to add comments in the suite overview page.

14.4 Manipulating Tests

Through CTHs the results of tests and configuration functions can be manipulated. The main purpose to do this with CTHs is to allow common patterns to be abstracted out from test suites and applied to multiple test suites without duplicating any code. All the callback functions for a CTH follow a common interface described hereafter.

Common Test always calls all available hook functions, even pre- and post hooks for configuration functions that are not implemented in the suite. For example, pre_init_per_suite(x_SUITE, ...) and post_init_per_suite(x_SUITE, ...) are called for test suite x_SUITE, even if it does not export init_per_suite/1. With this feature hooks can be used as configuration fallbacks, and all configuration functions can be replaced with hook functions.

Pre Hooks

In a CTH, the behavior can be hooked in before the following functions:

• init_per_suite
• init_per_group
• init_per_testcase
• end_per_testcase
• end_per_group
• end_per_suite

This is done in the CTH functions called pre_<name of function>. These functions take the arguments SuiteName, Name (group or test case name, if applicable), Config, and CTHState. The return value of the CTH function is always a combination of a result for the suite/group/test and an updated CTHState.

To let the test suite continue on executing, return the configuration list that you want the test to use as the result.

All pre hooks, except pre_end_per_testcase/4, can skip or fail the test by returning a tuple with skip or fail, and a reason as the result.

Example:

pre_init_per_suite(SuiteName, Config, CTHState) ->
  case db:connect() of
    {error,_Reason} ->
      {{fail, "Could not connect to DB"}, CTHState};
    {ok, Handle} ->
      {[{db_handle, Handle} | Config], CTHState#state{ handle = Handle }}
  end.

Post Hooks

In a CTH, behavior can be hooked in after the following functions:

• init_per_suite
• init_per_group
• init_per_testcase
• end_per_testcase
• end_per_group
• end_per_suite

This is done in the CTH functions called post_<name of function>. These functions take the arguments SuiteName, Name (group or test case name, if applicable), Config, Return, and CTHState. Config in this case is the same Config as the testcase is called with. Return is the value returned by the testcase. If the testcase fails by crashing, Return is {'EXIT',{{Error,Reason},Stacktrace}}.

The return value of the CTH function is always a combination of a result for the suite/group/test and an updated CTHState. If you do not want the callback to affect the outcome of the test, return the Return data as it is given to the CTH. You can also modify the test result. By returning the Config list with element tc_status removed, you can recover from a test failure. As in all the pre hooks, it is also possible to fail/skip the test case in the post hook.

Example:

post_end_per_testcase(_Suite, _TC, Config, {'EXIT',{_,_}}, CTHState) ->
  case db:check_consistency() of
    true ->
      %% DB is good, pass the test.
      {proplists:delete(tc_status, Config), CTHState};
    false ->
      %% DB is not good, mark as skipped instead of failing
      {{skip, "DB is inconsisten!"}, CTHState}
  end;
post_end_per_testcase(_Suite, _TC, Config, Return, CTHState) ->
  %% Do nothing if tc does not crash.
  {Return, CTHState}.

Skip and Fail Hooks

After any post hook has been executed for all installed CTHs, on_tc_fail or on_tc_skip is called if the testcase failed or was skipped, respectively. You cannot affect the outcome of the tests any further at this point.

14.5 Synchronizing External User Applications with Common Test

CTHs can be used to synchronize test runs with external user applications. The init function can, for example, start and/or communicate with an application that has the purpose of preparing the SUT for an upcoming test run, or initialize a database for saving test data to during the test run. The terminate function can similarly order such an application to reset the SUT after the test run, and/or tell the application to finish active sessions and terminate. Any system error- or progress reports generated during the init- or termination stage are saved in the Pre- and Post Test I/O Log. (This is also true for any printouts made with ct:log/2 and ct:pal/2).

To ensure that Common Test does not start executing tests, or closes its log files and shuts down, before the external application is ready for it, Common Test can be synchronized with the application. During startup and shutdown, Common Test can be suspended, simply by having a CTH evaluate a receive expression in the init- or terminate function. The macros ?CT_HOOK_INIT_PROCESS (the process executing the hook init function) and ?CT_HOOK_TERMINATE_PROCESS (the process executing the hook terminate function) each specifies the name of the correct Common Test process to send a message to. This is done to return from the receive. These macros are defined in ct.hrl.

14.6 Example CTH

The following CTH logs information about a test run into a format parseable by file:consult/1 (in Kernel):

 %%% Common Test Example Common Test Hook module.
 -module(example_cth).

 %% Callbacks
 -export([id/1]).
 -export([init/2]).

 -export([pre_init_per_suite/3]).
 -export([post_init_per_suite/4]).
 -export([pre_end_per_suite/3]).
 -export([post_end_per_suite/4]).

 -export([pre_init_per_group/4]).
 -export([post_init_per_group/5]).
 -export([pre_end_per_group/4]).
 -export([post_end_per_group/5]).

 -export([pre_init_per_testcase/4]).
 -export([post_init_per_testcase/5]).
 -export([pre_end_per_testcase/4]).
 -export([post_end_per_testcase/5]).

 -export([on_tc_fail/4]).
 -export([on_tc_skip/4]).

 -export([terminate/1]).

 -record(state, { file_handle, total, suite_total, ts, tcs, data }).

 %% Return a unique id for this CTH.
 id(Opts) ->
   proplists:get_value(filename, Opts, "/tmp/file.log").

 %% Always called before any other callback function. Use this to initiate
 %% any common state. 
 init(Id, Opts) ->
     {ok,D} = file:open(Id,[write]),
     {ok, #state{ file_handle = D, total = 0, data = [] }}.

 %% Called before init_per_suite is called.
 pre_init_per_suite(Suite,Config,State) ->
     {Config, State#state{ suite_total = 0, tcs = [] }}.

 %% Called after init_per_suite.
 post_init_per_suite(Suite,Config,Return,State) ->
     {Return, State}.

 %% Called before end_per_suite.
 pre_end_per_suite(Suite,Config,State) ->
     {Config, State}.

 %% Called after end_per_suite.
 post_end_per_suite(Suite,Config,Return,State) ->
     Data = {suites, Suite, State#state.suite_total, lists:reverse(State#state.tcs)},
     {Return, State#state{ data = [Data | State#state.data] ,
                           total = State#state.total + State#state.suite_total } }.

 %% Called before each init_per_group.
 pre_init_per_group(Suite,Group,Config,State) ->
     {Config, State}.

 %% Called after each init_per_group.
 post_init_per_group(Suite,Group,Config,Return,State) ->
     {Return, State}.

 %% Called before each end_per_group.
 pre_end_per_group(Suite,Group,Config,State) ->
     {Config, State}.

 %% Called after each end_per_group.
 post_end_per_group(Suite,Group,Config,Return,State) ->
     {Return, State}.

 %% Called before each init_per_testcase.
 pre_init_per_testcase(Suite,TC,Config,State) ->
     {Config, State#state{ ts = now(), total = State#state.suite_total + 1 } }.

 %% Called after each init_per_testcase (immediately before the test case).
 post_init_per_testcase(Suite,TC,Config,Return,State) ->
     {Return, State}

%% Called before each end_per_testcase (immediately after the test case).
 pre_end_per_testcase(Suite,TC,Config,State) ->
     {Config, State}.

 %% Called after each end_per_testcase.
 post_end_per_testcase(Suite,TC,Config,Return,State) ->
     TCInfo = {testcase, Suite, TC, Return, timer:now_diff(now(), State#state.ts)},
     {Return, State#state{ ts = undefined, tcs = [TCInfo | State#state.tcs] } }.

 %% Called after post_init_per_suite, post_end_per_suite, post_init_per_group,
 %% post_end_per_group and post_end_per_testcase if the suite, group or test case failed.
 on_tc_fail(Suite, TC, Reason, State) ->
     State.

 %% Called when a test case is skipped by either user action
 %% or due to an init function failing.  
 on_tc_skip(Suite, TC, Reason, State) ->
     State.

 %% Called when the scope of the CTH is done
 terminate(State) ->
     io:format(State#state.file_handle, "~p.~n",
                [{test_run, State#state.total, State#state.data}]),
     file:close(State#state.file_handle),
     ok.

14.7 Built-In CTHs

Common Test is delivered with some general-purpose CTHs that can be enabled by the user to provide generic testing functionality. Some of these CTHs are enabled by default when common_test is started to run. They can be disabled by setting enable_builtin_hooks to false on the command line or in the test specification. The following two CTHs are delivered with Common Test:

cth_log_redirect

Built-in

Captures all log events that would normally be printed by the default logger handler, and prints them to the current test case log. If an event cannot be associated with a test case, it is printed in the Common Test framework log. This happens for test cases running in parallel and events occuring in-between test cases. You can configure the level of SASL reports using the normal SASL mechanisms.

cth_surefire

Not built-in

Captures all test results and outputs them as surefire XML into a file. The created file is by default called junit_report.xml. The file name can be changed by setting option path for this hook, for example:

-ct_hooks cth_surefire [{path,"/tmp/report.xml"}]

If option url_base is set, an extra attribute named url is added to each testsuite and testcase XML element. The value is constructed from url_base and a relative path to the test suite or test case log, respectively, for example:

-ct_hooks cth_surefire [{url_base, "http://myserver.com/"}]

gives an URL attribute value similar to

"http://myserver.com/[email protected]_11.19.39/ x86_64-unknown-linux-gnu.my_test.logs/run.2012-12-12_11.19.39/suite.log.html"

Surefire XML can, for example, be used by Jenkins to display test results.