282 lines
15 KiB
ReStructuredText
282 lines
15 KiB
ReStructuredText
.. _sec_item_common:
|
|
|
|
Items common attributes
|
|
============================================================
|
|
|
|
All test items have common attributes independently of their types, which are
|
|
listed in next table, those are all optional parameters and their default value
|
|
if not provided is given in the table as well.
|
|
|
|
.. table:: test items common attributes
|
|
:widths: 25, 15, 60
|
|
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
| **Parameter name** | **Default value** | **Description** |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
|``name`` | test item type | This is the test item name as displayed |
|
|
| | | in the test tree window of the |
|
|
| | | testium. |
|
|
| | | |
|
|
| | | This attribute is also supported by actions of |
|
|
| | | ``console``, ``jsonrpc`` or ``plot`` test items. |
|
|
| | | |
|
|
| | | Default value, if not provided, is the |
|
|
| | | test item type. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
|``stop_on_failure`` |``False`` | If ``stop_on_failure`` is set to |
|
|
| | | ``True``, the test sequence execution |
|
|
| | | stops on test tem failure and no |
|
|
| | | further test items are executed, |
|
|
| | | except those withexecute_on_stop |
|
|
| | | attribute set (see below) |
|
|
| | | |
|
|
| | | It depends on the test item to take it |
|
|
| | | into account or not. |
|
|
| | | For example it makes sense to use it |
|
|
| | | for ``unittest`` test type |
|
|
| | | because it can contain many sub-tests, |
|
|
| | | but not for sleep test type. |
|
|
| | | In cycles, it means that the child |
|
|
| | | sequence execution is stopped at first |
|
|
| | | failure. It also means that the |
|
|
| | | remaining loops are not executed. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
|``execute_on_stop`` |``False`` | When this attribute is set to True, the |
|
|
| | | test item is always run, even on test |
|
|
| | | failure of any test before. |
|
|
| | | This feature is useful, to end the |
|
|
| | | test |
|
|
| | | sequence properly on test failure |
|
|
| | | (switch off power supplies, climatic |
|
|
| | | chamber temperature set to ambient |
|
|
| | | temperature….) |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
|``skipped`` |``False`` | The test item execution is to be |
|
|
| | | skipped during test sequence execution |
|
|
| | | if set to ``True``. It will be |
|
|
| | | displayed as failed in the report. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
| ``no_fail`` | ``False`` | The result of the test step is forced to PASS if this |
|
|
| | | attribute is set to ``true``. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
|``doc`` |``""`` | Documentation for the test item that |
|
|
| | | appears in the test doc field and the |
|
|
| | | contextual text window in the testium |
|
|
| | | GUI. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
|``Key`` | / | This attribute defines a key which |
|
|
| | | will be attached to the test result and |
|
|
| | | which will allow to be filtered during |
|
|
| | | the report generation. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
|``report`` | / | This attribute defines values (a dictionary) which |
|
|
| | | will be added in the ``data`` field of the report. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
| ``condition`` | / | The test item is executed if its |
|
|
| | | ``condition`` attribute content is a boolean |
|
|
| | | ``True``. |
|
|
| | | see :ref:`Conditional |
|
|
| | | execution<sec_conditional_execution>`. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
| ``process_result`` | / | Process an evaluation of the process_result |
|
|
| | | and store it in the result |
|
|
| | | see :ref:`Process result<sec_process_result>` |
|
|
| | | for details. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
| ``expected_result`` | / | Expected result value or string. |
|
|
| | | see :ref:`Expected result<sec_expected_result>` |
|
|
| | | for details. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
| ``store_result`` | / | Store the test result in a global variable. |
|
|
| | | see :ref:`Store result<sec_store_result>` |
|
|
| | | for details. |
|
|
+-----------------------+-------------------+-------------------------------------------------------+
|
|
|
|
|
|
last test result
|
|
-----------------------------------------------
|
|
|
|
The global variable ``last_step_result`` is automatically set at the end of a test item execution.
|
|
|
|
If the corresponding test item does not return any acutal, the content of the ``last_step_result``
|
|
variable will be the test success (``PASS``, ``FAIL`` or ``SKIP``).
|
|
|
|
It the test item returns a value, the ``last_step_result`` variable will contain the returned value.
|
|
|
|
The main test items returning a value are:
|
|
|
|
* :ref:`console<sec_console_test_item>` test item,
|
|
* :ref:`jsonrpc<sec_jsonrpc_test_item>` test item,
|
|
* :ref:`dialog references<sec_dialog_references_test_item>` test item,
|
|
* :ref:`dialog value<sec_dialog_value_test_item>` test item.
|
|
|
|
Test timings
|
|
-----------------------------------------------
|
|
|
|
After the execution of a test step, the following global variables are set :
|
|
|
|
* ``ts_start_<item_name>``
|
|
|
|
* ``ts_end_<item_name>``
|
|
|
|
and
|
|
|
|
* duration: ``ts_duration_<item_name>``
|
|
|
|
See :ref:`global variables<sec_global_variables>` for more detail on how to access to global variables from
|
|
test items and scripts.
|
|
|
|
Skipped test items
|
|
-----------------------------------------------
|
|
|
|
A variable named ``skipped_test_item`` can be defined in the global variable entries
|
|
or in configuration file (see :ref:`config files<sec_configuration_files>`) as a list of item to be skipped.
|
|
|
|
.. _sec_conditional_execution:
|
|
|
|
Conditional execution
|
|
-----------------------------------------------
|
|
|
|
The ``condition`` attribute content must be a boolean value (if not ``True``, the condition is considered
|
|
not met).
|
|
|
|
|
|
.. _sec_process_result:
|
|
|
|
Process result
|
|
-----------------------------------------------
|
|
|
|
The ``process_result`` attribute can be applied to all the test items. However, it's behavior is different
|
|
depending if the test item is returning a value or not.
|
|
|
|
The ``process_result`` attribute content is evaluated as a python line.
|
|
|
|
The special ``$(result)`` variable is replaced in the ``process_result`` attribute content with the test result value.
|
|
|
|
The process result is done before the ``expected_result``
|
|
|
|
If the result of the evaluation is a boolean, the test will be *PASSED* if ``True``, and *FAIL* otherwise.
|
|
|
|
.. _sec_expected_result:
|
|
|
|
Expected result
|
|
-----------------------------------------------
|
|
|
|
The ``expected_result`` attribute can be applied to all the test items. However, it's behavior is different
|
|
depending if the test item is returning a value or not.
|
|
|
|
The test items returning a value are:
|
|
|
|
* :ref:`dialog_references test item<sec_dialog_references_test_item>`
|
|
|
|
* :ref:`dialog_value test items<sec_dialog_value_test_item>`
|
|
|
|
* :ref:`py_func test item<sec_py_func_item>`
|
|
|
|
* :ref:`dialog_choices test item<sec_dialog_choices_test_item>`
|
|
|
|
* :ref:`json_rpc test item<sec_jsonrpc_test_item>`
|
|
|
|
For test items which don't return a value, the ``expected_result`` attribute content is
|
|
compared to ``PASS`` or ``FAIL``.
|
|
|
|
The ``expected_result`` attribute content is a simple comparison with ``$(result)``.
|
|
|
|
If the result and the expected_result is equal, the test will be *PASSED* if ``True``, and *FAIL* otherwise.
|
|
|
|
The special ``$(result)`` variable is replaced in the ``expected_result`` attribute content with the test result value.
|
|
|
|
|
|
.. _sec_store_result:
|
|
|
|
Store result
|
|
-----------------------------------------------
|
|
|
|
The ``store_result`` attribute stores the test result into a named global variable,
|
|
making it available to subsequent test items via ``$(variable_name)``.
|
|
|
|
If the test item returns a value (e.g. ``py_func``, ``json_rpc``), that value is stored.
|
|
If ``process_result`` is also specified, the stored value is the post-processed result.
|
|
|
|
If the test item produces no value (result is ``None``), the stored value is the
|
|
test status string: ``"PASS"`` or ``"FAIL"``, evaluated after ``expected_result``
|
|
but **before** ``no_fail``. This ensures the real outcome is captured even when
|
|
``no_fail: True`` would otherwise mask a failure.
|
|
|
|
.. code-block:: yaml
|
|
:caption: Store a function return value
|
|
|
|
- py_func:
|
|
name: Read sensor
|
|
func_name: read_temperature
|
|
store_result: temperature
|
|
|
|
- py_func:
|
|
name: Check temperature in range
|
|
func_name: check_range
|
|
param: [$(temperature), 20, 30]
|
|
|
|
.. code-block:: yaml
|
|
:caption: Store a post-processed value
|
|
|
|
- py_func:
|
|
name: Get firmware version string
|
|
func_name: get_version
|
|
process_result: "'$(result)'.split('.')[0]"
|
|
store_result: fw_major
|
|
|
|
.. code-block:: yaml
|
|
:caption: Store the pass/fail status of a test with no return value
|
|
|
|
- console:
|
|
name: Send command
|
|
console_name: device
|
|
steps:
|
|
- writeln: reboot
|
|
- read_until: {expected: "ready", timeout: 10}
|
|
store_result: reboot_status
|
|
|
|
- py_func:
|
|
name: Use reboot status
|
|
func_name: log_status
|
|
param: [$(reboot_status)]
|
|
|
|
|
|
Export attribute
|
|
-----------------------------------------------
|
|
|
|
.. code-block:: yaml
|
|
:caption: Example of ``export`` common attribute usage
|
|
|
|
- check:
|
|
name: Example of result specific to the step 001
|
|
values:
|
|
- $(last_step_result) == PASS
|
|
key:
|
|
- GID-1510554_step_1
|
|
report:
|
|
reported_list: <| random.sample(range(0,20), k=10) |>
|
|
reported_float: <| math.sqrt(float(1)) |>
|
|
reported_str: This is my reported sentence
|
|
|
|
|
|
.. _sec_item_default_folded:
|
|
|
|
Container items GUI default folding
|
|
============================================================
|
|
|
|
The container items are items which are the parent of other test items. For example loops and groups
|
|
are container test items.
|
|
|
|
In the GUI, if the user wants that a container test item is folded when he opens a test, the ``.``
|
|
character has to be place before the test item declaration.
|
|
|
|
See an example below:
|
|
|
|
.. code-block:: yaml
|
|
:caption: example of ``loop`` folded by default in the GUI
|
|
|
|
- .loop:
|
|
doc: An example loop
|
|
name: An example loop
|
|
... |