Navigation :

Run Test

Overview

The run_test binary program allows users to execute one or more JSON-defined test scripts against a running N2SVCD TesterApp application. This program uses the ExecuteTest REST API method to begin executing each test run, and then uses the QueryTest REST API method to periodically check each test run until it completes.

This program may be used to either:

A single test run, which executes a single instance.
A single test run, which executes multiple instances (specify --total greater than one).
Multiple test runs, each of which run in sequence (specify more than one .json file).
Multiple test runs, running in parallel (specify more than one .json file, and specify --parallel greater than one).

The default installation location for this binary is /usr/share/n2svcd/bin/run_test.

Running run_test --help will show the program usage.

usage: run_test <json-file> [... <json-file>]
        --host <json-rpc-host>          Default = 'localhost'.
        --port <json-rpc-port>          Default = 9009.
        --quiet                         (suppress progress noise)
        --space                         (output a newline between tests)
        --echo                          (echo JSON RPC request content)
        --failure-only                  (when running multiple test scripts only print failed tests in summary)
        --parallel <number_of_tests>    The number of tests to run in parallel. Up to this number will be actioned at once.

        --dpc <point-code>              Default = 4114.
        --no-dpc                        (don't send a DPC)
        --dssn <subsystem-number>       Default = 106.
        --no-dssn                       (don't send DSSN)
        --dgt_digits <digits>           Default = ''.
        --dgt_noa <nature-of-address>   Default = ''.
        --dgt_np <numbering-plan>       Default = ''.
        --dgt_tt <translation-type>     Default = ''.
        --ssp_ac <ati3|camel1|camel2|camel3|camel4|cs1|etsi|ussd2|a.b...>
                                        Default = 'camel3'.

        --smpp <smpp-app-name>          Default = 'default'.
        --pi <pi-app-name>              Default = 'default'.
        --osd <osd-app-name>            Default = 'default'.
        --diameter <diameter-app-name>  Default = 'default'.
        --sigtran <sigtran-app-name>    Default = 'default'.
        --rest <rest-app-name>          Default = 'default'.

        --google-test-output <output_file_location> Output a Google test formatted XML file that can be used by invoking applications to display run statistics.

        --cps <cps>
        --backlog <backlog>
        --ramp <ramp-seconds>           (requires --cps and/or --backlog)
        --total <total>                 (requires --cps and/or --backlog)

        --trace_level <0|1|2|3>         (request n2svcd to return debug traces)
        --trace <0|1|2|3>               (request n2svcd to return debug traces)
        --debug                         (--trace_level = 1)
        --dump                          (--trace_level = 2)
        --spam                          (--trace_level = 3)

        --load-details                  (track test details under load)

        --globals                       (Print $g at the end of the run)

Usage

run_test can be used to run a single JSON test definition, in which case a single test summary will be output.

./run_test ../test/regression/set2-INAP/tester_app/gb001_a_party_hangup.json

[START ../test/regression/set2-INAP/tester_app/gb001_a_party_hangup.json]
[STARTED...]
[COMPLETED 1(1) of 1 instances in 0.208 seconds.  Aborted = 0, Failed = 0, Cancelled = 0, Rate = 4.815/sec]
>>> FIRST CALL CHECKS
... [S=01/O=01] [PASS] Received operation inap.ssp_from_scp.FurnishChargingInformation as expected.
... [S=01/O=01] [PASS] String 'FCI.[0].tag_hex' is present and has expected value '001c'.
... [S=01/O=01] [PASS] String 'FCI.[0].value_hex' is present and has expected value '0100ffffffffffffffffffffffffffffffffffff'.
... [S=01/O=01] [PASS] String 'FCI.[1].tag_hex' is present and has expected value '002c'.
... [S=01/O=01] [PASS] String 'FCI.[1].value_hex' is present and has expected value '0200000000000000001000000000000000000000'.
... [S=01/O=01] [PASS] String 'FCI.[2].tag_hex' is present and has expected value '004c'.
... [S=01/O=01] [PASS] String 'FCI.[2].value_hex' is present and has expected value '0000000000000017228700000049300000005042'.
... [S=01/O=01] [PASS] String 'FCI.[3].tag_hex' is present and has expected value '999c'.
... [S=01/O=01] [PASS] String 'FCI.[3].value_hex' is present and has expected value '0000000000000000000000000000000000000000'.
... [S=02/O=02] [PASS] Received operation inap.ssp_from_scp.ApplyCharging as expected.
... [S=02/O=02] [PASS] Integer 'R1.maxConversationTime' is present and has expected value 120.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.warningTimeBeforeExpiry' is present and has expected value 110.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.informationToSend.inbandInfo.messageID.elementaryMessageID' is present and has expected value 500.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.informationToSend.inbandInfo.numberOfRepetitions' is present and has expected value 1.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.informationToSend.inbandInfo.duration' is present and has expected value 0.
... [S=02/O=02] [PASS] Integer 'R1.treatmentForFirstWarning.informationToSend.inbandInfo.interval' is present and has expected value 0.
... [S=03/O=03] [PASS] Received operation inap.ssp_from_scp.RequestReportBCSMEvent as expected.
... [S=03/O=03] [PASS] Integer 'BCSM(5,2)monitorMode' is present and has expected value 0.
... [S=03/O=03] [PASS] Integer 'BCSM(6,2)monitorMode' is present and has expected value 0.
... [S=03/O=03] [PASS] Integer 'BCSM(6,2)dPSpecificCriteria.applicationTimer' is present and has expected value 300.
... [S=03/O=03] [PASS] Integer 'BCSM(10,1)monitorMode' is present and has expected value 1.
... [S=03/O=03] [PASS] Integer 'BCSM(4)monitorMode' is present and has expected value 0.
... [S=03/O=03] [PASS] Integer 'BCSM(9,2)monitorMode' is present and has expected value 0.
... [S=03/O=03] [PASS] Integer 'BCSM(9,1)monitorMode' is present and has expected value 1.
... [S=04/O=04] [PASS] Received operation inap.ssp_from_scp.Connect as expected.
... [S=04/O=04] [PASS] String 'destinationRoutingAddress_digits' is present and has expected value '049340221'.
... [S=04/O=04] [PASS] Integer 'destinationRoutingAddress_noa' is present and has expected value 3.
... [S=05/O=05] [PASS] This is a static pass string.
... [S=06/O=06] [PASS] Correctly received DRA digits 049340221.
>>> First Instance all tests returned PASS.

run_test may also be run using multiple test definitions. This will output a summary of each individual test (see above), as well as a final summary of all the tests when complete.

./run_test ../test/regression/set2-INAP/tester_app/*.json
...

SUMMARY OF 16 RUNS.
  IDX 1st-PASS/FAIL : #START #FINISH #ABORT #FAILED #CANCEL : JSON Filename
    0       29    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb001_a_party_hangup.json
    1       27    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb002_b_party_hangup.json
    2       32    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb003_timer_expires.json
    3       39    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb004_post_call_interaction.json
    4       27    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/gb100_simple_charged.json
    5        5    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap001_idp_ext_rc_logic.json
    6       15    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap002_ctr_pa_pacui.json
    7       11    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap003_sinap_fci.json
    8       22    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap004_etc_pa_pacui_double.json
    9       11    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap004_etc_pa_pacui.json
   10       11    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap005_etc_pa_pacui_error.json
   11       10    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap006_etc_pa_pacui_abort.json
   12       13    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap007_connect_bcsm_cir.json
   13        3    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap100_simple_connect.json
   14        3    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap101_vars_connect.json
   15       11    0        1       1      0       0       0   ../test/regression/set2-INAP/tester_app/inap200_intester_camel.json

Common Options

The following common options may be specified when executing run_test.

Option	Type	Description
`--host`	String	Specify a non standard JSON-RPC server listen host for `TesterApp`. (Default: `localhost`)
`--port`	String	Specify a non standard JSON-RPC server listen port for `TesterApp`. (Default: `9009`)
`--quiet`	Switch	Specifying `--quiet` will suppress the more verbose output logging of each assertion statement from the generated output.
`--echo`	Switch	Outputs the JSON result message from the JSON REST RPC after running each test.
`--trace_level`	Number	Sets the global trace level that N2SVCD will pass to all executing applications during the test.
`--trace`	Number	Increments the global trace level to match that of the provided`--trace` level. This is effectively an alias for `--trace_level`.
`--debug`	Switch	Sets the global trace level to `DEBUG`. This is the same as setting `trace_level = 1`.
`--dump`	Switch	Sets the global trace level to `DUMP`. This is the same as setting `trace_level = 2`.
`--spam`	Switch	Sets the global trace level to `SPAM`. This is the same as setting `trace_level = 3`.
`--globals`	Switch	Outputs any stored global variables that were generated during the test run.
`--space`	Switch	Inserts whitespace between test results when executing multiple tests.
`--failure-only`	Switch	When executing multiple tests only failed tests will be reported when generating the final execution summary.
`--parallel`	Number	When executing multiple tests `parallel` indicates the number of tests to have running at one time. Default = `1`
`--google-test-output`	String	If specified a Google test output formatted XML file will be generated to the provided path. Google test XML may be used by downstream services such as Teamcity in order to generate reporting for test executions.

Load Testing Options

The following load testing options may be specified when executing run_test. Using these options will automatically execute run_test in performance testing mode, which will disable some standard output.

Option	Type	Description
`--cps`	Number	Executes a concurrent number of calls equal to the `--cps` value. If no `--total` is provided the total number of calls will be ten times the value of `--cps`.
`--backlog`	Number	Instructs `run_test` to process tests as fast as possible, having a minimum of the `--backlog` specified value in progress at one time. If no `--total` flag is provided, the total number of calls will be ten times the value of `--backlog`.
`--ramp`	Number	Increments the concurrent number of calls by the specified value of `--ramp` over the duration of the performance test. This is intended to be used with `--total` and a combination of either `--cps` or `--backlog`, otherwise only one test will be performed.
`--total`	Number	Determines the number of total test iterations that `run_test` will perform. If specified, at least one of `--cps` or `--backlog` must also be specified.
`--load-details`	Switch	Allows the matching of inner JSON test operation content under load test conditions.

SCCP Protocol Options

The following SCCP protocol options may be specified when executing run_test.

Option	Type	Description
`--dpc`	Number	Specifies the Destination Point Code when sending outbound SCCP messages. (Default: `4114`)
`--no-dpc`	Switch	Disables the inclusion of the Destination Point Code when sending outbound SCCP messages. If specified, the value of `--dpc` will be ignored.
`--dssn`	Number	Specifies the Destination Sub-System Number when sending outbound SCCP messages. (Default: `106`)
`--no-dssn`	Switch	Disables the inclusion of the Destination Sub-System Number when sending outbound SCCP messages. If specified, the value of `--dssn` will be ignored.
`--dgt_digits`	String	Specifies the Destination Global Title Digits when sending outbound SCCP messages. If not specified, this message parameter will not be sent.
`--dgt_noa`	Number	Specifies the Destination Global Title Nature of Address Indicator when sending outbound SCCP messages. If not specified, this message parameter will not be sent.
`--dgt_np`	Number	Specifies the Destination Global Title Numbering Plan when sending outbound SCCP messages. If not specified, this message parameter will not be sent.
`--dgt_tt`	Number	Specifies the Destination Global Title Translation Type when sending outbound SCCP messages. If not specified, this message parameter will not be sent.
`--ssp_ac`	String	Specifies the Application Context to use when sending INAP encoded messages. (Default: `camel3`)

Application Options

The following application options may be specified when executing run_test.

Option	Type	Description
`--smpp`	String	Specifies the SMPP application to utilize when executing a test that has SMPP operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file. (Default: `default`)
`--pi`	String	Specifies the PI application to utilize when executing a test that has PI operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file. (Default: `default`)
`--osd`	String	Specifies the OSD application to utilize when executing a test that has OSD operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file. (Default: `default`)
`--diameter`	String	Specifies the Diameter application to utilize when executing a test that has Diameter operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file. (Default: `default`)
`--sigtran`	String	Specifies the SIGTRAN application to utilize when executing a test that has SIGTRAN operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file. (Default: `default`)
`--rest`	String	Specifies the REST application to utilize when executing a test that has REST operations. The value must be either a real or virtual named application defined in the N2SVCD configuration file. (Default: `default`)