11 Using Common Test for Large-Scale Testing

11.1 General

Large-scale automated testing requires running multiple independent test sessions in parallel. This is accomplished by running some Common Test nodes on one or more hosts, testing different target systems. Configuring, starting, and controlling the test nodes independently can be a cumbersome operation. To aid this kind of automated large-scale testing, Common Test offers a master test node component, Common Test Master, which handles central configuration and control in a system of distributed Common Test nodes.

The Common Test Master server runs on one dedicated Erlang node and uses distributed Erlang to communicate with any number of Common Test test nodes, each hosting a regular Common Test server. Test specifications are used as input to specify what to test on which test nodes, using what configuration.

The Common Test Master server writes progress information to HTML log files similarly to the regular Common Test server. The logs contain test statistics and links to the log files written by each independent Common Test server.

The Common Test Master API is exported by module ct_master.

11.2 Use

Common Test Master requires all test nodes to be on the same network and share a common file system. Common Test Master cannot start test nodes automatically. The nodes must be started in advance for Common Test Master to be able to start test sessions on them.

Tests are started by calling ct_master:run(TestSpecs) or ct_master:run(TestSpecs, InclNodes, ExclNodes)

TestSpecs is either the name of a test specification file (string) or a list of test specifications. If it is a list, the specifications are handled (and the corresponding tests executed) in sequence. An element in a TestSpecs list can also be list of test specifications. The specifications in such a list are merged into one combined specification before test execution.

Example:

ct_master:run(["ts1","ts2",["ts3","ts4"]])

Here, the tests specified by "ts1" run first, then the tests specified by "ts2", and finally the tests specified by both "ts3" and "ts4".

The InclNodes argument to run/3 is a list of node names. Function run/3 runs the tests in TestSpecs just like run/1, but also takes any test in TestSpecs, which is not explicitly tagged with a particular node name, and execute it on the nodes listed in InclNodes. By using run/3 this way, any test specification can be used, with or without node information, in a large-scale test environment.

ExclNodes is a list of nodes to be excluded from the test. That is, tests that are specified in the test specification to run on a particular node are not performed if that node is listed in ExclNodes at runtime.

If Common Test Master fails initially to connect to any of the test nodes specified in a test specification or in the InclNodes list, the operator is prompted with the option to either start over again (after manually checking the status of the nodes in question), to run without the missing nodes, or to abort the operation.

When tests start, Common Test Master displays information to console about the involved nodes. Common Test Master also reports when tests finish, successfully or unsuccessfully. If connection is lost to a node, the test on that node is considered finished. Common Test Master does not attempt to re-establish contact with the failing node.

At any time, to get the current status of the test nodes, call function ct_master:progress().

To stop one or more tests, use function ct_master:abort() (to stop all) or ct_master:abort(Nodes).

For details about the Common Test Master API, see module ct_master.

11.3 Test Specifications

The test specifications used as input to Common Test Master are fully compatible with the specifications used as input to the regular Common Test server. The syntax is described in section Test Specifications in section Running Tests and Analyzing Results.

All test specification terms can have a NodeRefs element. This element specifies which node or nodes a configuration operation or a test is to be executed on. NodeRefs is defined as follows:

NodeRefs = all_nodes | [NodeRef] | NodeRef

NodeRef = NodeAlias | node() | master

A NodeAlias (atom()) is used in a test specification as a reference to a node name (so the node name only needs to be declared once, which also can be achieved using constants). The alias is declared with a node term as follows:

{node, NodeAlias, NodeName}

If NodeRefs has the value all_nodes, the operation or test is performed on all specified test nodes. (Declaring a term without a NodeRefs element has the same effect). If NodeRefs has the value master, the operation is only performed on the Common Test Master node (namely set the log directory or install an event handler).

Consider the example in section Test Specifications in section Running Tests and Analysing Results, now extended with node information and intended to be executed by Common Test Master:

{define, 'Top', "/home/test"}.
{define, 'T1', "'Top'/t1"}.
{define, 'T2', "'Top'/t2"}.
{define, 'T3', "'Top'/t3"}.
{define, 'CfgFile', "config.cfg"}.
{define, 'Node', ct_node}.

{node, node1, 'Node@host_x'}.
{node, node2, 'Node@host_y'}.

{logdir, master, "'Top'/master_logs"}.
{logdir, "'Top'/logs"}.

{config, node1, "'T1'/'CfgFile'"}.
{config, node2, "'T2'/'CfgFile'"}.
{config, "'T3'/'CfgFile'"}.

{suites, node1, 'T1', all}.
{skip_suites, node1, 'T1', [t1B_SUITE,t1D_SUITE], "Not implemented"}.
{skip_cases, node1, 'T1', t1A_SUITE, [test3,test4], "Irrelevant"}.
{skip_cases, node1, 'T1', t1C_SUITE, [test1], "Ignore"}.

{suites, node2, 'T2', [t2B_SUITE,t2C_SUITE]}.
{cases, node2, 'T2', t2A_SUITE, [test4,test1,test7]}.

{skip_suites, 'T3', all, "Not implemented"}.

This example specifies the same tests as the original example. But now if started with a call to ct_master:run(TestSpecName), test t1 is executed on node ct_node@host_x (node1), test t2 on ct_node@host_y (node2) and test t3 on both node1 and node2. Configuration file t1 is only read on node1 and configuration file t2 only on node2, while the configuration file t3 is read on both node1 and node2. Both test nodes write log files to the same directory. (However, the Common Test Master node uses a different log directory than the test nodes.)

If the test session is instead started with a call to ct_master:run(TestSpecName, [ct_node@host_z], [ct_node@host_x]), the result is that test t1 does not run on ct_node@host_x (or any other node) while test t3 runs on both ct_node@host_y and ct_node@host_z.

A nice feature is that a test specification that includes node information can still be used as input to the regular Common Test server (as described in section Test Specifications). The result is that any test specified to run on a node with the same name as the Common Test node in question (typically ct@somehost if started with the ct_run program), is performed. Tests without explicit node association are always performed too, of course.

11.4 Automatic Startup of Test Target Nodes

Initial actions can be started and performed automatically on test target nodes using test specification term init.

Two subterms are supported, node_start and eval.

Example:

{node, node1, node1@host1}.
{node, node2, node1@host2}.
{node, node3, node2@host2}.
{node, node4, node1@host3}.
{init, node1, [{node_start, [{callback_module, my_slave_callback}]}]}.
{init, [node2, node3], {node_start, [{username, "ct_user"}, {password, "ct_password"}]}}.
{init, node4, {eval, {module, function, []}}}.

This test specification declares that node1@host1 is to be started using the user callback function callback_module:my_slave_callback/0, and nodes node1@host2 and node2@host2 are to be started with the default callback module ct_slave. The specified username and password are used to log on to remote host host2. Also, function module:function/0 is evaluated on node1@host3, and the result of this call is printed to the log.

The default callback module ct_slave, has the following features:

• Starting Erlang target nodes on local or remote hosts (application SSH is used for communication).
• Ability to start an Erlang emulator with more flags (any flags supported by erl are supported).
• Supervision of a node being started using internal callback functions. Used to prevent hanging nodes. (Configurable.)
• Monitoring of the master node by the slaves. A slave node can be stopped if the master node terminates. (Configurable.)
• Execution of user functions after a slave node is started. Functions can be specified as a list of {Module, Function, Arguments} tuples.