Test_HOWTO.txt

Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements.  See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License.  You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

HOW TO WRITE A TEST
===================

See document qpid-interop-test-devel-overview.txt for an overview of
Tests and Shims, and the relationship between them.

0. Introduction
===============

The Test is the top-level entry point for a particular test objective
in which the clients are to be tested against each other. The overall
idea is to have the clients send and receive messages in such a way as
to test their interoperability. The shims are stand-alone executables
written in the client API / language which send or recieve the messages
so as to achieve the test objectives. The Test calls the shims to
perform the sends and receives, and evaluates the data it received in
order to pass or fail the test objective.

Each test has one or more Shims written to perform the specific
messaging task for that test. Each shim is a stand-alone executable
binary or script which can be called by the Test, and the parameters
sent to the shims and the data received from the shims is
specific to that Test. The practice of using JSON for sending
data to and from the shims has been adopted, as it is fairly widely
available, and can handle the necessary data structure.

1. Test name and location
=========================
Each Test is a single stand-alone Python program located in the
src/python/qpid_interop_test directory. The test is named
<test-name>_test.py

2. Common modules
=================
The following modules are available to Tests:

2.1 qit_broker_props.py
-----------------------
  Opens a connection to a broker and reads the connection properties in
  order to return a string containing the broker name. This is useful
  if you need to know the broker name for any reason. One of the common
  uses of this is to conditionally skip certain tests which fail based
  on the broker against which they are running.

2.2 qit_common.py
-----------------
  Code and classes common to most tests.

  This module includes class QitTestTypeMap which contains empty maps
  useful for storing test data vs data type for some tests. The class
  containing this map is subclasses and populated with test data for
  each test. This implies that the test data is saved as literals
  within the Test itself (which is current practice). However, using
  this technique is optional, and test data can be obtained by any
  useful means, so long as it is easy to access and maintain.

  NOTE: All tests except amqp_complex_types_test use this technique.

  NOTE: Test amqp_complex_types_test stores its test data in JSON
  files, then uses a code generator to emit test code for each client.

2.3 qit_errors.py
-----------------
  Defines error classes for use with tests. Currently this only
  defines a single error class (InteropTestError) which accepts
  a text error message, but more complex error classes may
  be added if needed.

2.4 qit_jms_types.py
--------------------
  Used by JMS tests, this module defines common QpidJms values and
  operations.

2.5 qit_shim.py
---------------
  This module defines the interface for calling, sending parameters
  to and receiving data from Shims.

  The Test will use separate worker threads for launching the sender
  and receiver shims (so that they are running simultaneously) so that
  the broker will have both connections open at the same time. Some
  "brokers" (like the Qpid Dispatch Router, which has no message storage
  ability) require that the receiver is launched before the sender in
  order that there is something to consume the message immediately
  they are received.

  Each shim object defines how the JSON parameters (and if necessary,
  other parameters - such as Java classpath) are sent to the shim for
  both the send and receive cases.

                                 +-- Sender
  Thread <-- ShimWorkerThread <--+
                                 +-- Receiver

  Every client API under test must have a shim object in shims.py. Note
  that this is shared by all Tests using this cleint API. The class
  structure is as follows:

          +-- ProtonPythonShim
          |
          +-- ProtonCppShim
          |
          +-- QpidJmsShim
  Shim <--+
          +-- <client_4>
          |
          +-- <client_5>
          |
          ...

  Each shim object contains an instance of Sender and Receiver when it is
  created, and which can be launched on its own thread. When the Sender
  and Receiver instance are run, they call the corresponding client Shim
  executable with the correct parameters for the test. When the Shim
  executable finishes, any data in stdout and stderr are piped back to
  the shim object for processing.

  HINT: There are some useful debug print statements in the shim Sender
  and Receiver run() methods. If enabled during shim development, these
  will print out the full parameters containing the JSON data for each
  shim as it is launched. In addition, other statements will print
  returned data from each shim. Once you have this, it is easy to run
  the shim independently from the test using these parameters. This is
  especially useful if you need to use a debugger on the shim.

2.5 qit_xunit_log.py
--------------------
  This module allows test results to be stored in xunit format.

3. Command-line options
=======================
The Tests should accept the following command-line options:
parameter         | action     | default          | metavar      | help
------------------+------------+------------------+--------------+------
--sender          | store      | localhost:5672   | IP-ADDR:PORT | Node to which test suite will send messages
--receiver        | store      | localhost:5672   | IP-ADDR:PORT | Node from which test suite will receive messages
--no-skip         | store_true |                  |              | Do not skip tests that are excluded by default for reasons of a known bug [1]
--broker-type     | store      |                  | BROKER_NAME  | Disable test of broker type (using connection properties) by specifying the broker name, or "None" [2]
--timeout         | store      |  varies per test | SEC          | Timeout for test in seconds (%d sec). If test is not complete in this time, it will be terminated
--include-shim    | append     |                  | SHIM-NAME    | Name of shim to include [3][4]
--exclude-shim    | append     |                  | SHIM-NAME    | Name of shim to exclude [3][5]
--xunit-log       | store_true |                  |              | Enable xUnit logging of test results
--xunit-log-dir   | store      | cwd()/xunit_logs | LOG-DIR-PATH | Default xUnit log directory where xUnit logs are written [xunit_logs dir in current directory (%s)]
--description     | store      |                  | DESCR        | Detailed description of test, used in xUnit logs
--broker-topology | store      |                  | DESCR        | Detailed description of broker topology used in test, used in xUnit logs

Notes:
[1] --no-skip: Only applies if you are using the broke name (through
    qit_broker_props.py) to exclude some tests bassed on the broker
    name.
[2] --broker-type: Should skip checking for the broker name (through
    qit_broker_props.py) and use the given name. "None" may have
    specific meaning to some tests for broker that don't return
    connection properties.
[3] --include-shim, --exclude-shim: These are mutually exclusive.
[4] --include-shim: This option generally clears the list of known
    shims and uses those supplied by this argument. This argument may
    be used multiple times.
[5] --exclude-shim: This option generally uses the list of known
    shims but deletes this argument from the list. This argument may
    be used multiple times.

Parameters for allowing for filtering of test cases should also be
provided so that by a combination of the above shim parameters and
the test data parameters, a test can be narrowed down to a single
test case. For example, in a type test, mutually exclusive parameters
which include or exclude individual types would be appropriate.

4. General Test layout
======================
Tests perform the following tasks:

4.1 Define test cases (and data)
--------------------------------
This is generally done by subclassing class QitTestTypeMap (see 2.2
above). However, any method to obtain a map containing test cases as
the index and the test data for that case as the value is appropriate.

If known test failures exist for certain brokers, then define a
BROKER_SKIP map in which the affected test cases are the index,
and each value consists of a sub-map with the broker name as the
index and a text reason for the skip as the value.

If a client has a known failure or if it does not support certain
types, then define a CLIENT_SKIP map in which the affected test cases
are the index, and each value consists of a sub-map with the client
name as the index and a text reason for the skip as the value.

NOTE: Every test skipped in this manner should reference a JIRA issue
for why that test is failing. This makes it easier to track and
re-enable the tests when the issue is resolved. It also encourages
reporting of issues.

4.2 Define your test class
--------------------------
Derive your test class from class QitTestCase. Make sure it has a run_test()
method, and which should receive as parameters the send and receive shim
objects being used for this test case. This method should perform the
following tasks:
* Create a unique queue name for this test;
* Create a Sender object by calling the shim create_sender() method;
* Create a Receiver object by calling the shim create_receiver() method;
* Run the Sender and Receiver by calling the start() method on each object.
  This will cause the Shims to be called on separate threads through the OS.
* Wait for the Sender and Receiver shims to finish executing by calling
  join().
* Process any returned errors by failing the test.
* If no errors are present, process the returned data and make a decision
  to pass or fail the test based on the received data values. Usually these
  are compared with the sent values to determine correctness.

4.3 Create your test cases
--------------------------
These can be created either statically or dynamically. How you choose to do it
depends on the type of test and how repetitive it is. If the tests are very
different and specific, then using the static method will work better

4.3.1 Static tests methods
--------------------------
Create one method per test labeled test<test_name> in the test class. See
https://docs.python.org/3.8/library/unittest.html for details. Each test should
accomplish a single objective using a single or limited set of similar
messages.

4.3.2 Dynamically created test methods
--------------------------------------
Were tests consist of the permutations and combinations of a number of test
shims and test values, it is possible to create the test methods dynamically
which saves a lot of error and repetition. An example may be found in
amqp_types-test, where the method which creates the test is
create_testcase_class(amqp_type, shim_product, timeout).

4.4 Add test options
--------------------
Use argparse.ArgumentParser to add test arguments. See section 3 above for
recommended options. Individual tests may also define additional test-specific
options as needed to control/modify that test.

4.5 Create a shim map
---------------------
In the main section of the test, create a shim map as follows:

{<shim_name>: <shim_instance>,
 ...
}

the product of which is used to run the shims against each other when
dynamically creating test cases.

5. Write the shims
==================
Each test needs to call one or more shims to perform the task of sending and
receiving messages. If only one shim exists, then it will test sending and
receiving against itself. See Shim_HOWTO for details on shim writing.

6. Test
=======
Get it working. 'Nuff said.

7. Add to the suite at Qpid Interop Test
========================================
If you have added a useful test, consider adding it to Qpid Interop Test.
Submit a JIRA at https://issues.apache.org/jira/browse/QPIDIT/.