v-and-v/testium
2
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 7 Wiki Activity

Project restart

Browse Source
This commit is contained in:
Foué
2025-12-29 10:46:05 +01:00
commit 59d19cb48c
388 changed files with 48020 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
doc/manual/sphinx/source/test_items/check_test_item.rst Normal file
View File

@@ -0,0 +1,12 @@
**check** test item
============================================================
The ``check`` test item returns the result of a python string evaluation:
.. code-block:: yaml
:caption: example of ``check`` test item usage
- check:
name: check test item example
values:
- '"tictactoe" in "$(my_global_string)"'

136
doc/manual/sphinx/source/test_items/console_test_item.rst Normal file
View File

@@ -0,0 +1,136 @@
.. _sec_console_test_item:
**console** test item
============================================================
The console test item is of the form:
.. code-block:: yaml
:caption: example of ``console`` test item usage
- console:
name: test name in GUI
console_name: console name in dict
steps:
- open:
protocol: telnet
telnet_host: $(target_ip)
telnet_port: $(target_port)
- writeln: reset
- read_until: {expected: U-Boot, timeout: 50}
- write: $(boot_vxworks_1)
- writeln: $(boot_vxworks_2)
- read_until:
expected: U-Boot
timeout: 15
- read_until:
expected: Something that will never occurs
timeout: 5
no_fail: True
mute: True
- close:
Attributes
-----------------------
Beside common test items attributes, console test item has specific attributes:
* ``console_name``: console instance name
* ``write_delay``: optional parameter giving the delay to wait in
milliseconds between each character sent.
* ``steps``: a sequence of actions to be applied to the console, as listed above.
The console test item steps accept the parameters and configurations defined in the next sections.
All the following actions support the ``name`` attribute. The ``name`` is concatenated with
the step type in the *testium* GUI, and recalled in the test log and reports.
``open`` action
-------------------------
The ``open`` action initializes the console with the attributes defined as described below.
The console instance is then added to the ``console_instances`` entry of the global
variables (cf :ref:`global variables<sec_global_variables>`).
Open step accepts the following attribute:
* ``protocol``: Setting of the console protocol, supported protocol are listed
in table below
* Other attributes are dependent of the protocol in used and are listed
in table below
.. table:: console protocols
:widths: 20, 30, 50
+---------------+------------------------+-------------------------------------------+
| **Protocol** | **protocol parameter** | **Description** |
+---------------+------------------------+-------------------------------------------+
|``telnet`` | ``telnet_host`` | hostname of the target. |
| +------------------------+-------------------------------------------+
| | ``telnet_port`` | port of the telnet server of the target. |
+---------------+------------------------+-------------------------------------------+
|``ssh`` | ``ssh_host`` | Hostname or IP address of the target. |
| +------------------------+-------------------------------------------+
| | ``ssh_user`` | port of the telnet server of the target. |
| +------------------------+-------------------------------------------+
| | ``ssh_pwd`` | Password (optional). |
+---------------+------------------------+-------------------------------------------+
|``serial`` | ``serial_port`` | Serial port to the target. |
| +------------------------+-------------------------------------------+
| | ``serial_baudrate`` | Baud rate of the serial connection. |
| +------------------------+-------------------------------------------+
| | ``buffered`` | Optinal boolean parameter. If ``False``, |
| | | it forces the |
| | | console to read directly the device. |
| | | Default: ``True``. |
+---------------+------------------------+-------------------------------------------+
|``rawtcp`` | ``tcp_host`` | hostname of the target. |
| +------------------------+-------------------------------------------+
| | ``tcp_port`` | port of the rawtcp server of the target. |
+---------------+------------------------+-------------------------------------------+
|``terminal`` | ``terminal_path`` | Path of the terminal console. |
+ +------------------------+-------------------------------------------+
| | ``shell`` | Shell to execute in the terminal |
| | | Default: /usr/bin/env bash |
+---------------+------------------------+-------------------------------------------+
* ``log``: is available only for Telnet and Serial console and is a path to a folder or a file, where the log will be stored.
``close`` action
---------------------------
The ``close`` action closes the console devices and removes its instance from
the ``console_instances`` list accessible in the global variables
(cf :ref:`global variables<sec_global_variables>`).
No parameters required for this action.
``write`` action
---------------------------
``write`` action takes as parameter the string to be written on the console.
``writeln`` action
-------------------------
writeln function is similar to the write function except that a '\n' (newline) character is sent at the end of the string to be written.
``read_until`` action
----------------------------
The ``read_until`` action is waiting for a string pattern from the console,
its parameter are listed below
* ``expected``: Character string to wait for
* ``timeout``: Timeout setting for the action (in seconds)
* ``no_fail``: Boolean value (``True`` or ``False``) leading to no error reported
if the expected input is not read
* ``mute``: Boolean value (``True`` or ``False``) does not log any readen data
The text read by the ``read_until`` action is stored in the global
variable named ``cn_<test_name>`` (See :ref:`global variables<sec_global_variables>`
for more detail on accessing global variables from test items and scripts).
In the example above, the global variable ``$(cn_test name in GUI)``
would be created at the end of the step. It would contain the resulting
data of the read.

BIN
doc/manual/sphinx/source/test_items/dialog_choices.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 56 KiB

109
doc/manual/sphinx/source/test_items/dialog_choices_test_item.rst Normal file
View File

@@ -0,0 +1,109 @@
.. _sec_dialog_choices_test_item:
**dialog_choices** test item
============================================================
This test item displays a dialog asking a question and waiting for
a selection to be done among defined list of items.
These selectable items can be passed as a tree.
The :numref:`Figure %s<choices-dialog>` displays an example of this item.
.. figure:: dialog_choices.png
:name: choices-dialog
:figwidth: 50 %
:width: 50 %
:align: center
choices dialog
The item parameters corresponding to :numref:`Figure %s<choices-dialog>`
is shown below.
.. code-block:: yaml
:caption: example of ``choices_dialog`` test item usage
- dialog_choices:
name: Choices
question: Select the items you want
icon: $(test_directory)/document.png
choices:
- name: choice 1
description: My first choice description
icon: $(test_directory)/document-save.png
choices:
- name: choice 1.1
description: My choice 1 first subchoice description
icon: $(test_directory)/Label.png
- name: choice 1.2
description: My choice 1 first subchoice description
- name: choice 2
description: My second choice description
icon: $(test_directory)/image.png
- name: choice 3
description: My third choice description
icon: $(test_directory)/image.png
choices:
- name: choice 3.1
description: My choice 3 first subchoice description
- name: choice 3.2
description: My choice 3 second subchoice description
icon: $(test_directory)/Label.png
Attributes
---------------
The supported attributes of the ``dialog_choices`` test item are:
* ``question``: Question to be displayed in the dialog box.
* ``choices``: List of the choicies presented to the user.
* ``icon``: Optional. Path of the icon used in the
selection tree, for all the items by default.
``Choices`` attribute content
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Each choice element is a dictionary which can have the following attributes
* ``name``: name of the choice to be done.
* ``description``: description of the choice to be done.
* ``icon``: Optional. Path of the icon displayed in the
selection tree in front of the corresponding choice.
* ``choices``: List of sub-choicies presented to the user (recursive).
Feature
------------------
The dialog references test item creates the ``cs_<name of test item>`` entry in the
global dictionary.
In the example above, the global variable name containing the
result of the test item would be ``cs_Choices``, and it would contain an
object of this form:
.. code-block::
:caption: example of result of the ``dialog_choices`` test item
[
{'name': 'choice 1',
'checked': True,
'choices': [ {'name': 'choice 1.1', 'checked': True},
{'name': 'choice 1.2', 'checked': False} ]
},
{ 'name': 'choice 2',
'checked': False},
{ 'name': 'choice 3',
'checked': True,
'choices': [ {'name': 'choice 3.1', 'checked': False},
{'name': 'choice 3.2', 'checked': True} ]
}
]
See :ref:`global variables<sec_global_variables>` for more detail
on how to access to global variables from test items and scripts.

27
doc/manual/sphinx/source/test_items/dialog_image_test_item.rst Normal file
View File

@@ -0,0 +1,27 @@
**dialog_image** test item
============================================================
This test item displays an image within a dialog box.
``dialog_image`` test item has the following description format
.. code-block:: yaml
:caption: example of ``dialog_image`` test item usage
- dialog_image:
name: dialog image test item
question: operator question
filename: imageToBeDisplayed.jpg
Attributes
----------------------
``dialog_image`` has the following specific attributes:
* ``question``: Question to be displayed in the dialog box
* ``filename``: File name of the image to be displayed in the dialog box.
Feature
----------------------
The test returns a ``FAIL`` if the answer is No and ``PASS`` if yes.

24
doc/manual/sphinx/source/test_items/dialog_message_test_item.rst Normal file
View File

@@ -0,0 +1,24 @@
**dialog_message** test item
============================================================
This test item displays a simple dialog asking a question and returning the entered value.
dialog_message test item has the following description format
.. code-block:: yaml
:caption: example of ``dialog_message`` test item usage
- dialog_message:
name: dialog value test item
question: operator question
Attributes
---------------------
``dialog_message`` has the following specific attribute:
* ``question``: Sentence to be displayed in the dialog box
Feature
---------------------
Just display the message.

25
doc/manual/sphinx/source/test_items/dialog_note_test_item.rst Normal file
View File

@@ -0,0 +1,25 @@
**dialog_note** test items
============================================================
This test item displays a simple dialog allowing to enter some text and printing the entered value in logs.
``dialog_note`` test item has the following description format
.. code-block:: yaml
:caption: example of ``dialog_note`` test item usage
- dialog_note:
name: dialog value test item
question: operator question
Attributes
-----------------
``dialog_note`` has the following specific attribute:
* ``question``: Question to be displayed in the dialog box
Feature
--------------------
Prints the entered text in the log.

26
doc/manual/sphinx/source/test_items/dialog_question_test_item.rst Normal file
View File

@@ -0,0 +1,26 @@
**dialog_question** test item
============================================================
This test item displays a simple dialog asking a question and returning
the entered value.
``dialog_question`` test item has the following description format
.. code-block:: yaml
:caption: example of ``dialog_question`` test item usage
- dialog_question:
name: dialog value test item
question: operator question
Attributes
--------------------
``dialog_question`` has the following specific attribute:
* ``question``: Question to be asked in the dialog box
Feature
----------------------
The test returns a ``FAIL`` if the answer is No and ``PASS`` if yes.

BIN
doc/manual/sphinx/source/test_items/dialog_reference.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 17 KiB

54
doc/manual/sphinx/source/test_items/dialog_reference_test_item.rst Normal file
View File

@@ -0,0 +1,54 @@
.. _sec_dialog_references_test_item:
**dialog_references** test item
============================================================
This test item displays a dialog asking a question and waiting for
references of the devices under test.
This test item has the following format:
.. code-block:: yaml
:caption: example of ``dialog_references`` test item usage
- dialog_references:
name: ask for a reference
question: Please give the reference of the product
reference:
- ref 1
- ref 2/rev
The example above displays the dialog box in :numref:`Figure %s<dialog-reference>`.
.. figure:: dialog_reference.png
:name: dialog-reference
:figwidth: 50 %
:width: 50 %
:align: center
dialog reference
Attributes
---------------
All the following attributes are mandatory.
* ``question``: Question to be displayed in the dialog box.
* ``reference``: For each of this parameter in the test correspond to a
row to fill in the dialog.
Every field for a reference can be pre-filled using separating
each filed with an '/' (cf :numref:`Figure %s<dialog-reference>`).
Feature
------------------
The dialog references test item creates the ``tested_items`` entry in the
global_dict global variable. This entry is a list of dictionaries of
this form:
.. code-block:: text
:caption: example of ``tested_items`` global variable result of ``dialog_reference``
test item
[{'reference': 'XXXXX', 'revision': 'YYYYY', 'serial': 'ZZZZZ'}, …]

29
doc/manual/sphinx/source/test_items/dialog_value_test_item.rst Normal file
View File

@@ -0,0 +1,29 @@
.. _sec_dialog_value_test_item:
**dialog_value** test items
============================================================
This test item displays a simple dialog asking a question and returning the entered value.
``dialog_value`` test item has the following description format
.. code-block:: yaml
:caption: example of ``dialog_value`` test item usage
- dialog_value:
name: dialog value test item
question: operator question
Attributes
-------------------
``dialog_value`` has the following specific attribute:
* ``question``: Question to be displayed in the dialog box
* ``default``: default value to place in the dialog form (optional)
Feature
----------------------
The returned value is added in the global variable entry with the key being the
``dialog_value`` test item name.

108
doc/manual/sphinx/source/test_items/func_test_item.rst Normal file
View File

@@ -0,0 +1,108 @@
.. _sec_func_item:
**py_func** test item
============================================================
The ``py_func`` test item is used to execute custom python scripts with the given
input parameters.
There are two modes for executing a ``py_func`` item. The class mode and the function mode.
class py_func item
-------------------------
This is the normal way of calling some custom python code.
A class must be defined and derived from ``FunctionItem`` from the ``libs.testium`` module.
From this class it is possible to define some custom reported values with the following API
* ``reportValue(key, value)``: This ``FunctionItem`` method is adding a value added to the report,
* ``reportedValues()``: This ``FunctionItem`` method is retrieving the current report values.
.. code-block:: python
:caption: ``py_func`` test item implementation example
:name: scriptFunctionItem.py
import py_func.tm as tm
class TestItemFunc(tm.FunctionItem)
def exec(param1, param2, param4, param4):
...
self.reportValue('my_reported_value', reported_value)
print(self.reportedValues())
return 10
The ``exec`` method of the ``FunctionItem`` derived class is executed while running the ``py_func`` test item.
.. code-block:: yaml
:caption: legacy ``py_func`` test item implementation
- py_func:
name: function test item
file: scriptTestFile.py
func_name: TestItemFunc
param:
- 123
- 0.123
- True
- $(global_dict_key)
expected_result: 10
**legacy py_func**
The legacy py_func test item is of the form:
.. code-block:: python
:caption: legacy ``py_func`` python function example
:name: scriptTestFile.py
def dummy_func(param1, param2, param4, param4):
...
return 10
There is no possibility to access the report features in that mode.
.. code-block:: yaml
:caption: corresponding ``py_func`` tum extract
- py_func:
name: function test item
file: scriptTestFile.py
func_name: funcToBeExecuted
param:
- 123
- 0.123
- True
- $(global_dict_key)
expected_result: 10
**Attributes**
Beside common test items attributes, py_func item has specific attribute, some of which being mandatory.
* ``file``: the script file name that contains the function to be executed.
Only python script format is supported.
* ``func_name``: The function name to be executed.
* ``param``: This is a list of parameters that are passed to the function
in the order they are presented in the script. These parameters are not
mandatory and are highly dependent of the function prototype.
.. code-block:: yaml
:caption: ``py_func`` test item example of usage
- py_func:
file: script_name.py
func_name: methodName
param:
- $(my_param)
The result of the function (after eventual post treatment) is stored in the global
variable named ``fn_<func_name>``
(See :ref:`global variables<sec_global_variables>` for more detail
on how to access to global variables from test items and scripts).
In the example above, the global variable ``$(fn_function test item)``
would be created at the end of the item execution. It would contain the resulting
value of the funcToBeExecuted python function.

17
doc/manual/sphinx/source/test_items/git_test_item.rst Normal file
View File

@@ -0,0 +1,17 @@
**git** test item
============================================================
Git test item allows
this item has the following description format
.. code-block:: yaml
:caption: ``git`` test item usage example
- git:
name: git test item
repo: [$(test_directory), "/path_to/another/repo"]
Attributes
-----------------
* ``repo``: a string or list of string path to the root of the git repositery(ies) to follow.

26
doc/manual/sphinx/source/test_items/group_test_item.rst Normal file
View File

@@ -0,0 +1,26 @@
**group** test item
============================================================
This element is of the following form:
.. code-block:: yaml
:caption: ``group`` test item usage example
- group:
name: Group Item
condition: "'$(OS)' == 'Linux'"
steps:
- unittest_file:
test_file: test_prod_rio6_8093.py
test_method:
...
- sleep:
timeout: 10
The ``group`` element is used to manage a sequence of item as a group.
Attributes
--------------------
* The ``steps`` list describes the sequence executed in the group.
It is a list of any of the testium test items,

222
doc/manual/sphinx/source/test_items/items_common_attributes.rst Normal file
View File

@@ -0,0 +1,222 @@
.. _sec_item_common:
Items common attributes
============================================================
All test items have common attributes independently of their types, which are
listed in next table, those are all optional parameters and their default value
if not provided is given in the table as well.
.. table:: test items common attributes
:widths: 25, 15, 60
+-----------------------+-------------------+-------------------------------------------------------+
| **Parameter name** | **Default value** | **Description** |
+-----------------------+-------------------+-------------------------------------------------------+
|``name`` | test item type | This is the test item name as displayed |
| | | in the test tree window of the |
| | | testium. |
| | | |
| | | This attribute is also supported by actions of |
| | | ``console``, ``jsonrpc`` or ``plot`` test items. |
| | | |
| | | Default value, if not provided, is the |
| | | test item type. |
+-----------------------+-------------------+-------------------------------------------------------+
|``stop_on_failure`` |``False`` | If ``stop_on_failure`` is set to |
| | | ``True``, the test sequence execution |
| | | stops on test tem failure and no |
| | | further test items are executed, |
| | | except those withexecute_on_stop |
| | | attribute set (see below) |
| | | |
| | | It depends on the test item to take it |
| | | into account or not. |
| | | For example it makes sense to use it |
| | | for ``unittest_file`` test type |
| | | because it can contain many sub-tests, |
| | | but not for sleep test type. |
| | | In cycles, it means that the child |
| | | sequence execution is stopped at first |
| | | failure. It also means that the |
| | | remaining loops are not executed. |
+-----------------------+-------------------+-------------------------------------------------------+
|``execute_on_stop`` |``False`` | When this attribute is set to True, the |
| | | test item is always run, even on test |
| | | failure of any test before. |
| | | This feature is useful, to end the |
| | | test |
| | | sequence properly on test failure |
| | | (switch off power supplies, climatic |
| | | chamber temperature set to ambient |
| | | temperature….) |
+-----------------------+-------------------+-------------------------------------------------------+
|``skipped`` |``False`` | The test item execution is to be |
| | | skipped during test sequence execution |
| | | if set to ``True``. It will be |
| | | displayed as failed in the report. |
+-----------------------+-------------------+-------------------------------------------------------+
| ``no_fail`` | ``False`` | The result of the test step is forced to PASS if this |
| | | attribute is set to ``true``. |
+-----------------------+-------------------+-------------------------------------------------------+
|``doc`` |``""`` | Documentation for the test item that |
| | | appears in the test doc field and the |
| | | contextual text window in the testium |
| | | GUI. |
+-----------------------+-------------------+-------------------------------------------------------+
|``Key`` | / | This attribute defines a key which |
| | | will be attached to the test result and |
| | | which will allow to be filtered during |
| | | the report generation. |
+-----------------------+-------------------+-------------------------------------------------------+
|``report`` | / | This attribute defines values (a dictionary) which |
| | | will be added in the ``data`` field of the report. |
+-----------------------+-------------------+-------------------------------------------------------+
| ``condition`` | / | The test item is not executed if its |
| | | ``condition`` attribute content is |
| | | evaluated as ``False``. |
| | | see :ref:`Conditional |
| | | execution<sec_conditional_execution>`. |
+-----------------------+-------------------+-------------------------------------------------------+
| ``process_result`` | / | Process an evaluation of the process_result |
| | | and store it in the result |
| | | see :ref:`Process result<sec_process_result>` |
| | | for details. |
+-----------------------+-------------------+-------------------------------------------------------+
| ``expected_result`` | / | Expected result value or string. |
| | | see :ref:`Expected result<sec_expected_result>` |
| | | for details. |
+-----------------------+-------------------+-------------------------------------------------------+
last test result
-----------------------------------------------
The global variable ``last_test_result`` is automatically set at the end of a test item execution.
If the corresponding test item does not return any acutal, the content of the ``last_test_result``
variable will be the test success (``PASS``, ``FAIL`` or ``SKIP``).
It the test item returns a value, the ``last_test_result`` variable will contain the returned value.
The main test items returning a value are:
* :ref:`console<sec_console_test_item>` test item,
* :ref:`jsonrpc<sec_jsonrpc_test_item>` test item,
* :ref:`dialog references<sec_dialog_references_test_item>` test item,
* :ref:`dialog value<sec_dialog_value_test_item>` test item.
Test timings
-----------------------------------------------
After the execution of a test step, the following global variables are set :
* ``ts_start_<item_name>``
* ``ts_end_<item_name>``
and
* duration: ``ts_duration_<item_name>``
See :ref:`global variables<sec_global_variables>` for more detail on how to access to global variables from
test items and scripts.
Skipped test items
-----------------------------------------------
A variable named ``skipped_test_item`` can be defined in the global variable entries
or in configuration file (see :ref:`config files<sec_configuration_files>`) as a list of item to be skipped.
.. _sec_conditional_execution:
Conditional execution
-----------------------------------------------
The ``condition`` attribute content is evaluated as a python string.
.. _sec_process_result:
Process result
-----------------------------------------------
The ``process_result`` attribute can be applied to all the test items. However, it's behavior is different
depending if the test item is returning a value or not.
The ``process_result`` attribute content is evaluated as a python line.
The special ``$(result)`` variable is replaced in the ``process_result`` attribute content with the test result value.
The process result is done before the ``expected_result``
If the result of the evaluation is a boolean, the test will be *PASSED* if ``True``, and *FAIL* otherwise.
.. _sec_expected_result:
Expected result
-----------------------------------------------
The ``expected_result`` attribute can be applied to all the test items. However, it's behavior is different
depending if the test item is returning a value or not.
The test items returning a value are:
* :ref:`dialog_references test item<sec_dialog_references_test_item>`
* :ref:`dialog_value test items<sec_dialog_value_test_item>`
* :ref:`py_func test item<sec_func_item>`
* :ref:`dialog_choices test item<sec_dialog_choices_test_item>`
* :ref:`json_rpc test item<sec_jsonrpc_test_item>`
For test items which don't return a value, the ``expected_result`` attribute content is
compared to ``PASS`` or ``FAIL``.
The ``expected_result`` attribute content is a simple comparison with ``$(result)``.
If the result and the expected_result is equal, the test will be *PASSED* if ``True``, and *FAIL* otherwise.
The special ``$(result)`` variable is replaced in the ``expected_result`` attribute content with the test result value.
Export attribute
-----------------------------------------------
.. code-block:: yaml
:caption: Example of ``export`` common attribute usage
- check:
name: Example of result specific to the step 001
values:
- $(last_test_result) == PASS
key:
- GID-1510554_step_1
report:
reported_list: <@ random.sample(range(0,20), k=10) @>
reported_float: <@ math.sqrt(float(1)) @>
reported_str: This is my reported sentence
.. _sec_item_default_folded:
Container items GUI default folding
============================================================
The container items are items which are the parent of other test items. For example loops and groups
are container test items.
In the GUI, if the user wants that a container test item is folded when he opens a test, the ``.``
character has to be place before the test item declaration.
See an example below:
.. code-block:: yaml
:caption: example of ``loop`` folded by default in the GUI
- .loop:
doc: An example loop
name: An example loop
...

178
doc/manual/sphinx/source/test_items/json-rpc_test_item.rst Normal file
View File

@@ -0,0 +1,178 @@
.. _sec_jsonrpc_test_item:
**jsonrpc** test item
============================================================
The `jsonrpc` test item is used to access jsonrpc servers, by sending queries and analysing the
answers.
It supports JSONRPC `v1.0 <https://www.jsonrpc.org/specification_v1>`_ or
`v2.0 <https://www.jsonrpc.org/specification>`_.
This test item can access the jsonrpc server by using an existing
:ref:`console<sec_console_test_item>` or directly using a UDP protocol.
Two low level *adapters* can be then chosen: ``udp`` or ``console``.
Example of ``jsonrpc`` test item with the console adapter:
.. code-block:: yaml
:caption: ``json_rpc`` test item usage example
- json_rpc:
name: JSONRPC console Query
doc: JSONRPC console Query not waiting (only send)
console:
name : jsonrpc_server
prompt: "@@>"
timeout: 1
version: "2.0"
steps:
- query:
method: echo
params:
a: Hello world
b: [0, 1, 2, 3]
id: 3095372
no_wait: True
- [...]
- json_rpc:
name: JSONRPC console Reception
doc: JSONRPC console reception of the previous request
console: {name : jsonrpc_server}
timeout: 1
steps:
- receive:
name: console reception
id: 3095372
timeout: 0.5
Attributes
-----------------------
the jsonrpc attributes are:
* ``timeout``: global communication timeout in seconds. It is a floating point number.
* ``version``: "1.0" or "2.0" (as a string) depending on the version of the JSONRPC
standard which is supported.
* ``mute``: a boolean giving the verbosity of the jsonrpc exchanges on the log output.
* An :ref:`Adapter<sec_jsonrpc_adapters>` is to be chosen between:
* Console,
* UDP,
* ``steps``: a sequence of actions as described in the sections below.
.. _sec_jsonrpc_adapters:
Steps
-----------------------
the jsonrpc steps can be of the following:
* ``open``: used by UDP to open the socket explicitely,
* ``close``: used by UDP adapter to close the socket explicitely,
* ``query``: performs a complete or partial JSONRPC call,
* ``receive``: used to receive the JSONRPC result of call previously
done by the ``query`` action.
If no ``expected_value`` attribute is defined for ``query`` or ``receive`` actions,
the success of the step will depend on the value returned by the JSONRPC frame.
Indeed, this protocol defines a mean to notify if the remote procedure has succeeded
or failed.
All the actions support the ``name`` attribute. The ``name`` is concatenated with
the action type in the *testium* GUI, and recalled in the test log and reports.
adapter attributes
^^^^^^^^^^^^^^^^^^^^^^
The adapters attributes are listed in the table below.
.. flat-table:: jsonrpc adapters
:header-rows: 2
:stub-columns: 1
:widths: 10 20 15 10 10 10
* - :rspan:`1` adapter
- :cspan:`2` attribute
- :cspan:`1` Description
* - attribute
- *type*
* - :rspan:`2` Console
- ``console``
- *dictionary*
- The console adapter configuration
* - ``console.name``
- *string*
- The name of the console which will be retrieved from
the :ref:`global variables<sec_global_variables>`. See also
the :ref:`console test item<sec_console_test_item>`.
* - ``console.prompt``
- *string*
- the eventual enclosing suffix of the jsonrpc frame.
* - :rspan:`4` UDP
- ``udp``
- *dictionary*
- The UDP adapter configuration
* - ``udp.server``
- *string*
- UDP server hostname or IP address.
* - ``udp.snd_port``
- *integer*
- UDP server listening port
* - ``udp.rcv_port``
- *integer*
- UDP answer reception port (on client side)
* - ``bufsize``
- *integer*
- the maximum expected size of the buffer received while waiting for
a jsonrpc frame.
``open`` action
-------------------------
The ``open`` jsonrpc action is only used with the
`UDP adapter<sec_jsonrpc_adapters>` but is mandatory before any ``query`` action.
No parameter is required.
``close`` action
---------------------------
The ``close`` jsonrpc action is only used with the
`UDP adapter<sec_jsonrpc_adapters>` but is mandatory after JSONRPC transfers are finished.
No parameter is required.
``query`` action
---------------------------
The ``query`` jsonrpc action has the following attributes:
* ``method``: JSONRPC method to be called,
* ``params``: JSONRPC param (must be conforming to the version defined above), by default it is an empty list.
* ``id``: JSONRPC id. If not defined or starts with ``rand``, it is chosen randomly.
Otherwise it must be an integer value,
* ``timeout``: reception timeout in seconds. It is a floating point number.
It is by default the jsonrpc timeout.
* ``no_wait``: Optional boolean. False by default. This attribute defines if
the reception is performed in this step (reception can be done appart, in the
``receive`` action described below),
``receive`` action
---------------------------
The ``receive`` jsonrpc action has the following attributes:
* ``id``: JSONRPC id as an integer value,
* ``timeout``: reception timeout in seconds. It is a floating point number,
It is by default the jsonrpc timeout.

25
doc/manual/sphinx/source/test_items/let_test_item.rst Normal file
View File

@@ -0,0 +1,25 @@
**let** test item
============================================================
This element is of the following form:
.. code-block:: yaml
:caption: ``let`` test item usage example
- let:
name: Let Item
values:
key1: value1
key2: value2
eval:
key3: $(variable)[$(loop_index)]
The ``let`` element is used to set values in the global directory.
Attributes
----------------
* The values list gives the {<key>, <value>} couples to set in the
global directory,
* The eval list gives the strings to evaluate prior to its storage into
the <key> of global directory.

95
doc/manual/sphinx/source/test_items/loop_test_item.rst Normal file
View File

@@ -0,0 +1,95 @@
.. _sec_loop_item:
**loop** test items
============================================================
This element is of the following form:
.. code-block:: yaml
:caption: ``loop`` test item usage example
- loop:
name: Cycle Temperature
iterator: 10
steps:
- unittest_file:
test_file: test_prod_rio6_8093.py
- py_func:
name: function test item
file: scriptTestFile.py
func_name: funcToBeExecuted
param:
- $(loop_param)
exit_condition:
file: script_name.py
func_name: methodName
The loop element executes repeatedly the ``steps`` sequence of items.
The configuration of the iteration process is done according to the iterator
cycle sub-item. As described later in this chapter the iterator is
configurable per cycle and allows to call a python function at each
cycle loop.
Attributes
----------------
Below are described loop test item specific attributes.
* ``Iterator``: giving the number of loop iteration (see dedicated chapter below).
* ``steps``: describes the sequence executed at each cycle; it is
a list of any of the testium test items.
* ``exit_condition``: allows to exit the loop. If False is returned, loop continues
else, it breaks. exit_condition attributes are:
* ``time``: the loop stops after the time (in minutes) is elapsed (optional)
* ``value``: the loop stops when the content of the value attribute is
evaluated as True (optional)
* ``file``: the loop the script file name that contains a function to be
executed on each loop. Only python script format is supported (optional
if another exit_condition attribute is defined)
* ``func_name``: the function to execute on each loop when the file attribute
is defined. The function referenced by the ``func_name`` attribute must
have two parameters: the current loop iterator value and the report,
even if they are not used. This attribute is mandatory if the file
attribute is defined.
* ``eval``: optional parameter allowing post treatment of the function result.
It is a python evaluable string in which the ``$(result)`` keywork
is replaced by the actual function call result (see exemple below).
.. code-block:: yaml
:caption: ``loop`` exit condition
- loop:
...
exit_condition:
file: script_name.py
func_name: methodName
eval: $(result) < 2
**Iterator**
The iterator attribute can be of the following types:
* An integer giving the cycle loop number,
* A list. The number of elements of the list gives the loop number, and
the list member are the consecutive loop parameters,
* Undefined. Then cycle loops until the exit condition is reached.
**Loop variables**
The following loop variables are automatically defined:
* ``$(loop_param)``: parameter of the loop. It contains the iterator value.
* ``$(loop_index)``: index of the loop, starting with 0 and incremented at each cycle.
* ``$(loop_index_inverse)``: inverse of index of the loop, starting from cycle length
-1 and decremented at each cycle.
* ``$(loop_count)``: loop total iteration number. If the number of loops is undefined
its value is the python ``inf``.
When these variables are found in a parameter, an attribute, etc, a
loop is searched recursively in the test hierarchy. And the variable value
is replaced by the corresponding loop value.
If more than one loop exists in the test item hierarchy, the lowest level
loop iterator is used.

180
doc/manual/sphinx/source/test_items/plot_test_item.rst Normal file
View File

@@ -0,0 +1,180 @@
**plot** test item
============================================================
This test item is used to display runtime values of tests variables or any evolving value in
a independent external window.
The plot window is defined using the ``plot`` test item:
.. code-block:: yaml
:caption: ``plot`` test item usage example
- plot:
name: test name in GUI
plot_name: plot identifier
steps:
- open:
- add:
...
Attributes
----------------------
In addition to common test items attributes, console test item has specific attributes:
* ``plot_name``: plot window instance name.
* ``steps``: a sequence of actions to be applied to the plot window. More than one action can be
executed in a ``plot`` item.
The plot test item can accept the actions described in further sections.
All the following actions support the ``name`` attribute. The ``name`` is concatenated with
the action type in the *testium* GUI, and recalled in the test log and reports.
``open`` action
----------------------
This action initializes and opens the plot window with the corresponding attributes as defined below.
This action accepts one optional ``path`` parameter defining a path where are stored the
plot lines values in csv format.
.. code-block:: yaml
:caption: ``plot`` ``open`` action
- plot:
name: Open the plot window
plot_name: plot identifier
steps:
- open:
name: open the plot
log_path: $(test_directory)/tmp
``close`` action
----------------------
The ``close`` action closes the plot window and removes its from the managed instances of *testium*.
This action does not have mandatory parameters. However, ``close`` optional action parameters are:
* ``wait_dialog_exit``: Boolean value. If set to True, the window
is kept opened until the user closes it manually.
* ``timeout``: Value expressed in seconds. It is active if the ``wait_dialog_exit`` is set to True.
If this parameter is defined, and if not closed manually, the dialog window is kept opened until
the timeout elapses.
.. code-block:: yaml
:caption: ``plot`` ``close`` action
- plot:
name: Closes the plot
plot_name: plot identifier
steps:
- close:
wait_dialog_exit: True
timeout: 600
.. note::
When the ``close`` action is entered, the ``periodic`` plots are stopped.
``add`` action
----------------------
The ``add`` action is used to add a single data to the ``plot`` window.
.. code-block:: yaml
:caption: ``plot`` ``add`` action
- plot:
name: Add to the plot
plot_name: plot identifier
steps:
- add:
name: add value 1 & 2
value1: $(loop_index)
value2: $(loop_index)+2
The parameter of the ``add`` action is a dictionnary of (*key*, *values*) pairs where the *key* is the plot line name and
*value* is the numeric value to add to the plot line.
The *value* content is evaluated as a python statement if not a number, but a string.
``periodic`` action
----------------------
This action allows to specify a python function to be called and which result is used to update the
plot.
``periodic`` plots are updated automatically and don't require further steps in a test sequence, once executed.
``periodic`` action parameters are:
* ``period``: period of the automatic value update.
* ``file``: python file containing the function to call.
* ``func_name``: the name of the python function to be periodicaly called.
* ``eval``: optional parameter allowing post treatment of the function result.
The ``eval`` parameter of the periodic action is a python evaluable string in which the $(result) keywork
is replaced by the actual function call result.
The result of the action must be a dictionnary of (*key*, *values*) pairs where the *key* is the plot line name and
*value* is the numeric value to add to the plot line.
.. code-block:: yaml
:caption: ``plot`` ``periodic`` action
- plot:
name: Add periodic to the plot
plot_name: plot identifier
steps:
- periodic:
period: 1
file: $(test_path)$(psep)plot.py
func_name: random_value
eval: '{"periodic": $(result)}'
``last_value`` action
----------------------
The ``last_value`` action returns the last values added to the plot (periodicaly or not) into
the global variables entries.
``last_value`` action parameters are:
* ``name``: Optional parameter giving the list of measures to be returned. If
it is not defined, all the measures are returned.
.. code-block:: yaml
:caption: ``plot`` ``last_value`` action
- plot:
name: Plot measure_1 value
plot_name: plot identifier
steps:
- last_value:
name: [measure_1]
The result of the action is stored in the global
variable named ``plv_<item_name>`` in the example above, it would be
``$(plv_Plot measure_1 value)``. See :ref:`global variables<sec_global_variables>` for more detail
on how to access to global variables from test items and scripts.
``export`` action
----------------------
The ``export`` action saves the plot window data in various formats to the filesystem.
.. code-block:: yaml
:caption: ``plot`` ``export`` action
- plot:
name: Plot export
plot_name: plot identifier
steps:
- export: $(my_custom_path)/plot_export.pdf
- export: $(my_custom_path)/plot_export.csv
At the time of writing of this documentation, ``.pdf`` and ``.csv`` files are supported.

51
doc/manual/sphinx/source/test_items/report_test_item.rst Normal file
View File

@@ -0,0 +1,51 @@
**report** test item
============================================================
This test item exports a report file.
To have this functionality activated, a ``report section`` must be defined at the root of the test file.
The root report section is described in :ref:`report<sec_reports>` section.
report test item has the following description format
.. code-block:: yaml
:caption: ``report`` test item usage example
- report:
name: Intermediate report
export:
- junit:
path: $(home)/reports/report-key-1.junit
pattern:
- Unittest%
key: report-key-1
- text:
file_name: report-key-1.txt
path: $(home)/reports
key:
- report-key-1Attributes
This item is useful to generate intermediate reports in any format other than ``sqlite``. Nevertheless,
if ``sqlite`` export is defined, It won't generate anything.
Attributes
---------------------
``report`` test item has the following specific attributes:
* ``export``: reports to be exported. It is a list of the reports exports to be executed.
The supported exports are:
* ``junit``
* ``json``
* ``html``
* text
The export sub-attributes (see example above) may contain the following attributes.
* ``path``: path of the report files directory,
* ``filename``: report file name,
* ``Pattern``: list of the patterns (applied on test names) used to select the
tests to exportinto the report,
* ``Key``: list of selected keys which are used to select the tests to export
into the report.

30
doc/manual/sphinx/source/test_items/run_test_item.rst Normal file
View File

@@ -0,0 +1,30 @@
**run** test item
============================================================
This test item executes a new instance of testium.
.. code-block:: yaml
:caption: ``run`` test item usage example
- run:
name: Execute TUM
tum_fime: example_cycle.tum
python_path: python3
testium_path: /home/francois/projets/testium-new-report/testium.pyw
log_file: $(home)/reports/test.log
report_file: $(home)/reports/test.rep
Attributes
---------------------
run test item has the following specific attributes:
* ``tum_fime``: mandatory the path of the file to execute, it can be relative to current execution folder,
* ``param_file`` (optional) the path of the parameter file to use, otherwise default parameter file is used.
* ``python_path`` (optional) the path of a specific python to run your scripts,
* ``testium_path`` (optional) the path of a specific testium to run your scripts,
* ``log_file`` (optional) the path of log file to register, if not provided a file is created with timestamp at the location of TUM file.
* ``report_file`` (optional), the path of report file to create
* ``start_time`` (optional), start time for the script execution, in HH:MM format.
* ``end_time`` (optional), end time for an execution within a time frame, in HH:MM format.
* ``wait_for_exec`` (optional). True or False, wait to be in the execution window defined by start_time and end_time to run the script.

18
doc/manual/sphinx/source/test_items/sleep_test_item.rst Normal file
View File

@@ -0,0 +1,18 @@
**sleep** test item
============================================================
sleep test item has the following description format
.. code-block:: yaml
:caption: ``sleep`` test item example usage
- sleep:
name: sleep test item
timeout: 10
dialog: True
Attributes
---------------
* ``timeout``: sleep duration in second or in relative date format like "2d 5h 31m 3s", which translate into 2 days, 5 hours, 31 minutes and 3 seconds.
* ``dialog``: If set to True, a window showing the remaining time to wait is displayed (optional parameter set to ``False`` by default)

81
doc/manual/sphinx/source/test_items/unittest_file_test_item.rst Normal file
View File

@@ -0,0 +1,81 @@
**unittest_file** test item
============================================================
unittest_file test item allows the execution of unittest test script which
is part of python standard libraries.
The tum file prototype is as followed:
.. code-block:: yaml
:caption: ``unittest_file`` test item usage example
- unittest_file:
name: unitTest test item
test_file: unitTestScript.py
test_method:
- test_1
- test_2
Attributes
------------------
Beside common test items attributes, unittest test item has specific attribute, some of which being mandatory.
* ``test_file``: it is the name (and eventually path) of the unittest file
to be processed.
* ``test_method``: it is an optional unittest_file test sub-item. If one or more
elements are present, the unittest python script file is parsed and only
the corresponding methods are included in the test tree. Otherwise, all
the test methods are included in the test tree.
Access to global variables entries
----------------------------------
``unittest`` file tests instances have access to the testium global variables
by using the :ref:`helper's library<sec_python_helper_library>`.
Report value from unittest
----------------------------------
Value can be added to the test report from unitTest test at runtime.
.. code-block:: python
:caption: example of ``unittest`` test item python function
from unittest import (TestCase)
class DummyTests(TestCase):
def test_01_report(self):
self.reported_values['key reported']= 'value_reported'
Console use example with unittest item
-----------------------------------------
Here is an example how to use the console module from python ``unittest``.
.. code-block:: python
:caption: example of a *testium* console usage from a ``unittest`` python function
from unittest import (TestCase)
import console
class DummyTests(TestCase):
@classmethod
def setUpClass(cls):
cls.consA0= console.TelnetConsole('cons name','192.168.98.123',7001)
cls.consA0.open()
cls.promptA0 = 'test-computer>'
def test_01_console(self):
self.consA0.write('config')
self.assertEqual(self.consA0.read_until(self.promptA0, 10), 0)
self.consA0.write('lsusb && echo "Done."\n')
status, read_data = self.consA0.read_until('Done.',
10, return_data=True)
self.assertEqual(status, 0)
if read_data.find('ID 04f2:b684 Chicony Electronics Co.')!=-1:
index=0
@classmethod
def tearDownClass(cls):
cls.consA0.close()