testium/doc/tutorial.md

Tutorial — testing a small Python utility

This walk-through builds, step by step, a testium campaign that exercises a small Python module. Each section adds one feature; you can follow along by editing a single .tum file and re-running it.

If you have not yet run testium, start with quick_start.md.

The code under test

Create calc.py next to your .tum file:

def add(a, b):
    return a + b

def divide(a, b):
    return a / b

Step 1 — a static check

The simplest item is check: it evaluates an expression and the test passes iff the expression is truthy. Create tutorial.tum:

main:
    name: calc.py campaign
    steps:
        - check:
            name: addition is correct
            values:
                - <| 2 + 3 == 5 |>

The <| ... |> markers turn the body into a Python expression evaluated at run time. Run it:

./run.sh -b -- tutorial.tum

Step 2 — call your code with py_func

check only sees Python literals; to exercise calc.py we need a py_func item. Replace the step:

        - py_func:
            name: add 2 and 3
            file: calc.py
            func_name: add
            param: [2, 3]
            expected_result: 5

expected_result makes the item PASS only when the function returns exactly that value.

The result is also stored in the global dict under pfn_<name> (here pfn_add 2 and 3).

Anywhere in a .tum, $(key) is replaced at runtime by the value stored in the global dict under key. A subsequent step can read the result back with $(pfn_<name>):

        - check:
            name: result was 5
            values:
                - <| $(pfn_add 2 and 3) == 5 |>

Step 3 — group several checks

Wrap the steps in a group to keep them visually together and let testium report a per-group status:

main:
    name: calc.py campaign
    steps:
        - group:
            name: add
            steps:
                - py_func:
                    name: 2 + 3
                    file: calc.py
                    func_name: add
                    param: [2, 3]
                    expected_result: 5
                - py_func:
                    name: -1 + 1
                    file: calc.py
                    func_name: add
                    param: [-1, 1]
                    expected_result: 0
        - group:
            name: divide
            steps:
                - py_func:
                    name: 6 / 2
                    file: calc.py
                    func_name: divide
                    param: [6, 2]
                    expected_result: 3.0

A group fails as soon as one of its steps fails (set stop_on_failure: false to keep going).

Step 4 — define a variable with let

Avoid hard-coding the same number twice with a variable:

        - let:
            name: define numerator
            values:
                - num: 6
        - py_func:
            name: divide num by 2
            file: calc.py
            func_name: divide
            param:
                - $(num)
                - 2
            expected_result: 3.0

$(num) expands to the global dict entry — when the stored value is a number it is substituted as a number, no need to wrap it in <| ... |>.

Step 5 — conditional execution

Skip a step when a condition is false:

        - py_func:
            name: divide by zero only on linux
            condition: <| "$(os)" == "Linux" |>
            file: calc.py
            func_name: divide
            param: [1, 0]

Items skipped this way report SKIP and do not affect the overall result.

Step 6 — generate a report

Add a report block at the root of the file:

main:
    name: calc.py campaign
    steps:
        # ... your steps here ...

report:
    enabled: true
    log_stored: true
    export:
        - junit:
            path: ./reports
            file_name: calc.xml
        - html:
            path: ./reports
            file_name: calc.html

The path directory must exist before the test runs — testium does not create it. Create it once:

mkdir -p reports

Re-run the test — ./reports/calc.xml (CI-friendly) and ./reports/calc.html (human-friendly) are produced. Set log_stored: true to include each item's captured stdout.

Step 7 — share state between calls

By default each py_func runs in its own short-lived subprocess. To keep state across calls, use context_id:

        - py_func:
            name: open
            file: calc.py
            func_name: open_resource
            context_id: my_ctx
        - py_func:
            name: use
            file: calc.py
            func_name: use_resource
            context_id: my_ctx

Both steps share the same persistent Python interpreter, so calc.py can store any object in module-level globals or in tm.setgd().

To share data without context_id, write it to the testium global dict via the JSON-RPC bridge:

import py_func.tm as tm

def producer():
    tm.setgd("computed", 42)

def consumer():
    return tm.gd("computed")

Where to go next

• doc/examples/ — one runnable .tum per feature (cycles, dialogs, console, plots, parallel, run-of-tum, …).
• doc/manual/testium_manual.pdf — full reference manual covering every test item, every attribute and the YAML syntax extensions.