AddingTestCases.txt

  1tdc - Adding test cases for tdc
  2
  3Author: Lucas Bates - lucasb@mojatatu.com
  4
  5ADDING TEST CASES
  6-----------------
  7
  8User-defined tests should be added by defining a separate JSON file.  This
  9will help prevent conflicts when updating the repository. Refer to
 10template.json for the required JSON format for test cases.
 11
 12Include the 'id' field, but do not assign a value. Running tdc with the -i
 13option will generate a unique ID for that test case.
 14
 15tdc will recursively search the 'tc-tests' subdirectory (or the
 16directories named with the -D option) for .json files.  Any test case
 17files you create in these directories will automatically be included.
 18If you wish to store your custom test cases elsewhere, be sure to run
 19tdc with the -f argument and the path to your file, or the -D argument
 20and the path to your directory(ies).
 21
 22Be aware of required escape characters in the JSON data - particularly
 23when defining the match pattern. Refer to the supplied json test files
 24for examples when in doubt.  The match pattern is written in json, and
 25will be used by python.  So the match pattern will be a python regular
 26expression, but should be written using json syntax.
 27
 28
 29TEST CASE STRUCTURE
 30-------------------
 31
 32Each test case has required data:
 33
 34id:           A unique alphanumeric value to identify a particular test case
 35name:         Descriptive name that explains the command under test
 36skip:         A completely optional key, if the corresponding value is "yes"
 37              then tdc will not execute the test case in question. However,
 38              this test case will still appear in the results output but
 39              marked as skipped. This key can be placed anywhere inside the
 40              test case at the top level.
 41dependsOn:    Same as 'skip', but the value is executed as a command. The test
 42              is skipped when the command returns non-zero.
 43category:     A list of single-word descriptions covering what the command
 44              under test is testing. Example: filter, actions, u32, gact, etc.
 45setup:        The list of commands required to ensure the command under test
 46              succeeds. For example: if testing a filter, the command to create
 47              the qdisc would appear here.
 48	      This list can be empty.
 49	      Each command can be a string to be executed, or a list consisting
 50	      of a string which is a command to be executed, followed by 1 or
 51	      more acceptable exit codes for this command.
 52	      If only a string is given for the command, then an exit code of 0
 53	      will be expected.
 54cmdUnderTest: The tc command being tested itself.
 55expExitCode:  The code returned by the command under test upon its termination.
 56              tdc will compare this value against the actual returned value.
 57verifyCmd:    The tc command to be run to verify successful execution.
 58              For example: if the command under test creates a gact action,
 59              verifyCmd should be "$TC actions show action gact"
 60matchPattern: A regular expression to be applied against the output of the
 61              verifyCmd to prove the command under test succeeded. This pattern
 62              should be as specific as possible so that a false positive is not
 63              matched.
 64matchCount:   How many times the regex in matchPattern should match. A value
 65              of 0 is acceptable.
 66teardown:     The list of commands to clean up after the test is completed.
 67              The environment should be returned to the same state as when
 68              this test was started: qdiscs deleted, actions flushed, etc.
 69	      This list can be empty.
 70	      Each command can be a string to be executed, or a list consisting
 71	      of a string which is a command to be executed, followed by 1 or
 72	      more acceptable exit codes for this command.
 73	      If only a string is given for the command, then an exit code of 0
 74	      will be expected.
 75
 76
 77SETUP/TEARDOWN ERRORS
 78---------------------
 79
 80If an error is detected during the setup/teardown process, execution of the
 81tests will immediately stop with an error message and the namespace in which
 82the tests are run will be destroyed. This is to prevent inaccurate results
 83in the test cases.  tdc will output a series of TAP results for the skipped
 84tests.
 85
 86Repeated failures of the setup/teardown may indicate a problem with the test
 87case, or possibly even a bug in one of the commands that are not being tested.
 88
 89It's possible to include acceptable exit codes with the setup/teardown command
 90so that it doesn't halt the script for an error that doesn't matter. Turn the
 91individual command into a list, with the command being first, followed by all
 92acceptable exit codes for the command.
 93
 94Example:
 95
 96A pair of setup commands.  The first can have exit code 0, 1 or 255, the
 97second must have exit code 0.
 98
 99        "setup": [
100            [
101                "$TC actions flush action gact",
102                0,
103                1,
104                255
105            ],
106            "$TC actions add action reclassify index 65536"
107        ],