v-and-v/testium
2
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 7 Wiki Activity

docs: refocus README on users, add quick_start + tutorial, fill CONTRIBUTING

Browse Source 
- README.md: pruned developer-oriented sections (Sphinx setup, Qt
  Creator workflow, VSCode debugging, release procedure, AppImage
  Wayland note) and replaced them with a user-facing layout: pre-built
  releases pointer, quick start, manual install, troubleshooting,
  licence.
- CONTRIBUTING.md: absorbed the developer content (debugging in VSCode,
  Qt GUI regen, Sphinx build, validation suite — batch + GUI variants,
  cross-distrib check, release procedure).
- doc/quick_start.md: 5-minute path from install to a passing test,
  in batch mode and in the GUI.
- doc/tutorial.md: guided walk-through against a small calc.py
  module — check, py_func, expected_result, $(...) expansion, group,
  let, condition, report (with the mkdir reminder), context_id.
- CLAUDE.md: subprocess API contract, bins.py, report-exporter
  plugin section, packaging matrix (wheel / PyInstaller / Flatpak /
  .deb work-in-progress), refreshed recent-fixes list. README/CLAUDE
  validation command no longer carries the spurious "-l" flag (which
  is GUI-only and a no-op in batch).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Foué
2026-05-05 09:18:59 +02:00
parent d3c5bd01e5
commit b9475c6e9b
5 changed files with 494 additions and 181 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
CLAUDE.md
View File

@@ -242,7 +242,7 @@ The `.deb` work-in-progress lives in `package/deb/`:
## Validation tests ## Validation tests
Located in `test/validation/`. Run with `-b` flag: Located in `test/validation/`. Run with `-b` flag:
``` ```
./run.sh -b -l mon_log.log -- test/validation/main.tum ./run.sh -b -- test/validation/main.tum
``` ```
Parallel item tests: `test/validation/items/parallel/test.tum` Parallel item tests: `test/validation/items/parallel/test.tum`

101
CONTRIBUTING.md
View File

@@ -45,7 +45,7 @@ For existing files, keep the header that is already there.
3. Commit with a clear message (one logical change per commit). 3. Commit with a clear message (one logical change per commit).
4. Make sure the validation suite still passes: 4. Make sure the validation suite still passes:
``` ```
./run.sh -b -l mon_log.log -- test/validation/main.tum ./run.sh -b -- test/validation/main.tum
``` ```
5. Open a pull request against `main`. 5. Open a pull request against `main`.
@@ -56,6 +56,105 @@ For existing files, keep the header that is already there.
- Add or update tests in `test/validation/` for new test items or behaviours - Add or update tests in `test/validation/` for new test items or behaviours
- Update `CLAUDE.md` and the Sphinx manual for user-visible changes - Update `CLAUDE.md` and the Sphinx manual for user-visible changes
## Development
### Debugging in VSCode
The recommended workflow:
1. Add a debug configuration to `.vscode/launch.json`:
```json
{
"configurations": [
{
"name": "Python : testium",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/src/testium",
"console": "integratedTerminal",
"args": ["-g"],
"justMyCode": true
}
]
}
```
2. Install `debugpy` in the venv: `python -m pip install debugpy`.
3. Open the *Run and Debug* tab and press play. testium starts; load and
run a `.tum` file. Set breakpoints where you want to investigate.
### Qt GUI modification
UI files (`*.ui`) are edited in **Qt Creator**. After editing, regenerate
the corresponding Python and resource files:
```sh
scripts/qt_generate.sh
```
Icons come from <https://github.com/free-icons/free-icons>.
### Sphinx documentation
```sh
pip install sphinx linuxdoc
doc/manual/sphinx/build_doc.sh
```
PDF generation requires `texlive`:
```sh
sudo apt install texlive-full
```
### Validation suite
Batch mode (CI-friendly, headless):
```sh
./run.sh -b -- test/validation/main.tum
```
GUI mode (loads the suite, click *Run* to execute and inspect the tree):
```sh
./run.sh test/validation/main.tum
```
GUI run-and-close (executes the suite, then closes):
```sh
./run.sh -r -- test/validation/main.tum
```
Subset run via the `items` define (works in any mode):
```sh
./run.sh -b -d "items=['parallel','common']" -- test/validation/main.tum
```
### Cross-distribution check
`package/deb/test_distro.sh` spins up a Docker/Podman container of the
target image, installs the expected system Python deps via apt (with
pip fallback for what is missing), installs the testium wheel and runs
the validation suite end-to-end. Currently green on `debian:bookworm`,
`debian:trixie`, `ubuntu:24.04`.
```sh
./package/deb/test_distro.sh debian:trixie
```
## Release procedure
1. Update `release_note.txt`.
2. Bump the version in `src/VERSION`.
3. Make sure the documentation is up to date — rebuild with
`doc/manual/sphinx/build_doc.sh` if needed.
4. Push and tag the commit with the new version.
5. Build the binary release: `package/pyinstaller/build.sh`.
6. Run the validation suite against each generated binary.
7. Confirm all validation results are green before publishing.
## Reporting security issues ## Reporting security issues
Please do **not** report security vulnerabilities through public GitHub Please do **not** report security vulnerabilities through public GitHub

283
README.md
View File

@@ -1,185 +1,110 @@
# Documentation # testium
[See here](doc/manual/testium_manual.pdf). testium is a YAML-driven test sequencer for hardware-in-the-loop and
integration testing. A test campaign is described in a `.tum` file as a tree
of items (checks, console interactions, Python/Lua functions, parallel blocks,
dialogs, …); testium executes the tree, captures results, and produces
reports in several formats.
# License ## Documentation
Copyright (c) 2025-2026 François Dausseur. * [Quick start](doc/quick_start.md) — install and run your first test in
five minutes.
* [Tutorial](doc/tutorial.md) — guided walk-through of the most common
test items with a runnable example.
* [User manual (PDF)](doc/manual/testium_manual.pdf) — full reference.
* [`doc/examples/`](doc/examples/) — runnable `.tum` snippets.
## Pre-built releases
Pre-built artifacts are published at
<https://git.beafrancois.fr/v-and-v/testium/releases>:
* **Python wheel** (`testium-<version>-py3-none-any.whl`) — install with
`pip install testium-*.whl`. Lighter than the binary; pulls Python
dependencies from PyPI on install.
* **Self-contained Linux binary** (`testium`, built with PyInstaller) —
runnable directly, no Python installation required on the host. Lua
support still needs a system `lua` interpreter and the `lua-socket` /
`lua-cjson` modules.
* **Flatpak** — *coming soon.*
## Quick start
From a checkout of the repository:
| OS | Command |
|----|---------|
| Linux | `./run.sh` |
| Windows (cmd) | `run.bat` |
| Windows (PowerShell) | `run.ps1` |
The wrapper creates a Python virtual environment on first run and starts
testium in GUI mode. Add `-b path/to/test.tum` to run a test in batch mode.
## Manual installation
If the wrapper script does not fit your environment, set up testium manually:
```sh
python3 -m venv .venv
source .venv/bin/activate
pip install -r src/requirements.txt
```
Required Python packages (see `src/requirements.txt`):
`pyside6`, `pyserial`, `pyyaml`, `pexpect`, `gitpython`, `jinja2`, `colorama`,
`matplotlib`, `junit-xml`, `lxml`.
For tests using `lua_func` items, install Lua (>= 5.1) plus the `socket` and
`cjson` modules. On Debian/Ubuntu:
```sh
sudo apt install lua5.4 lua-socket lua-cjson
```
Run testium:
```sh
python3 src/testium # GUI
python3 src/testium -b mytest.tum # batch
```
## Troubleshooting
### `wl_proxy_marshal_flags` symbol error
```
testium: symbol lookup error: ... undefined symbol: wl_proxy_marshal_flags
```
Force the X11 Qt backend:
```sh
export QT_QPA_PLATFORM=xcb
testium
```
### `xcb plugin missing`
```
qt.qpa.plugin: Could not load the Qt platform plugin "xcb"
```
Install the missing system libraries:
```sh
sudo apt install libxcb-cursor0 libicu-dev libxcb-cursor-dev
```
## License
Copyright © 2025-2026 François Dausseur.
testium is distributed under the **European Union Public Licence v. 1.2 testium is distributed under the **European Union Public Licence v. 1.2
(EUPL-1.2)** — see the [LICENSE](LICENSE) file for the full text. (EUPL-1.2)** — see [`LICENSE`](LICENSE) for the full text. SPDX:
`EUPL-1.2`.
SPDX identifier: `EUPL-1.2` Contributions are accepted under the same licence (inbound = outbound).
See [`CONTRIBUTING.md`](CONTRIBUTING.md) for development setup, debugging
Contributions are accepted under the same licence (inbound = outbound). See workflow, and the release procedure.
[CONTRIBUTING.md](CONTRIBUTING.md) for details.
# run testium
From the root path, on windows `cmd`:
run.bat
On windows powershell:
run.ps1
On linux:
./run.sh
The virtual environment is created if needed and *testium* is started.
# Manual setup
A python virtual environment should be created:
python3 -m venv <testium_venv>
## Requirements
In the virtual environment, the following modules must be installed:
* pyside6
* pyserial
* pyyaml
* pexpect
* gitpython
* jinja2
* colorama
* matplotlib
* junit-xml
* lxml
A `requirements.txt` file is also available in the git repository in the path `testium/src/`.
## run testium
from the testium path, execute
python3 -m src/testium
# Doc generation
## Install sphinx
pip install sphinx linuxdoc
## Generate the doc
Execute
doc/manual/sphinx/./build_doc.sh
This command works if texlive package has been installed on the system. It can be done by invoking the following command.
sudo apt install texlive-full
# QT GUI
## QT GUI modification
Open the ".ui" file with `qtcreator` and modify the gui. Then regenerate the python code.
On linux, a helper script has been created:
scripts/./qt_generate.sh
# Debugging
In order to debug testium or your python script executed within testium.
## In VSCODE
This is the prefered method :
1. Create a debug configuration like the following:
```
"configurations": [
{
"name": "Python : testium",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/src/testium",
"console": "integratedTerminal",
"args": ["-g"],
"justMyCode": true
},
]
```
2. Install debugpy module in python
python -m pip install debugpy
3. Then get to the "RUN AND DEBUG" tab and press the play button.
4. A testium window will pops up ; start execution of your tum.
5. Do not forget to put breakpoints where you want to investigate.
## Icons
Icons are coming from the following site: https://github.com/free-icons/free-icons.git
# testium Release
## Pre-requisite
A `python` virtual environment must have been set as described above.
### Install pyinstaller
Install `pyinstaller` package using pip.
## Generate the binary package
The procedure for a binary release is as follows:
1. update the `release_note.txt` file
2. modify the version in `src/VERSION` file
3. be sure that the documentation is up to date, and if not execute `doc/manual/sphinx/build_doc.sh` script
4. push modifications and create a tag with the new version on the git repository
5. generate an executable file by calling `package/pyinstaller/./build.sh`
6. run the complete validation test for each generated binary
7. check that all the validation results are OK
# Troubleshooting
## The testium exe crashes `wl_proxy_marshal_flags`
### Error message
/testium: symbol lookup error: /tmp/_MEIOhDCPF/libQt6WaylandClient.so.6: undefined symbol: wl_proxy_marshal_flags
### Solution
Set the appropriate environment variable
export QT_QPA_PLATFORM=xcb
testium
## xcb plugin missing
### Error message
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
### Solution
A package is missing
sudo apt install libxcb-cursor0
sudo apt-get install libicu-dev
sudo apt-get install libxcb-cursor-dev
## The testium appimage crashes when opening a file
This is usually because wayland is defined as the default X server.
To change it :
* Disable Wayland by uncommenting WaylandEnable=false in the `/etc/gdm3/daemon.conf`
* Add `QT_QPA_PLATFORM=xcb` in `/etc/environment`
* After a reboot, check that the environment variable value returns `x11`:
$ echo $XDG_SESSION_TYPE
x11

66
doc/quick_start.md Normal file
View File

@@ -0,0 +1,66 @@
# Quick start
Five minutes from zero to a passing test.
## Install
From a checkout of the repository:
```sh
./run.sh --version # Linux
run.bat # Windows cmd
```
The wrapper creates a Python virtual environment on first run and verifies
testium starts. If you prefer a manual install, see the README.
## Your first test
Create `hello.tum`:
```yaml
main:
name: hello world
steps:
- check:
name: 1 + 1 makes 2
values:
- <| 1 + 1 == 2 |>
```
Run it in batch mode:
```sh
./run.sh -b -- hello.tum
```
You should see something like:
```
-----> step "1 + 1 makes 2" started
Check passed
<----- step "1 + 1 makes 2" finished: PASS
Test run success.
```
Replace `==` with `!=` and re-run — the step now ends with **FAIL** and
the process exits with code 1.
## Open it in the GUI
```sh
./run.sh hello.tum
```
The test tree appears in the left panel; click *Run test* in the toolbar.
Each item turns green or red live as it executes. Use `F1` on a selected
item to open its detail panel.
## Where to go next
* [`doc/tutorial.md`](tutorial.md) — a guided walk-through of the most
common test items (`py_func`, `let`, `group`, `condition`, `report`).
* [`doc/examples/`](examples/) — runnable `.tum` snippets covering one
feature each.
* [`doc/manual/testium_manual.pdf`](manual/testium_manual.pdf) —
full reference manual.

223
doc/tutorial.md Normal file
View File

@@ -0,0 +1,223 @@
# Tutorial — testing a small Python utility
This walk-through builds, step by step, a testium campaign that exercises
a small Python module. Each section adds one feature; you can follow
along by editing a single `.tum` file and re-running it.
If you have not yet run testium, start with [`quick_start.md`](quick_start.md).
## The code under test
Create `calc.py` next to your `.tum` file:
```python
def add(a, b):
return a + b
def divide(a, b):
return a / b
```
## Step 1 — a static check
The simplest item is `check`: it evaluates an expression and the test
passes iff the expression is truthy. Create `tutorial.tum`:
```yaml
main:
name: calc.py campaign
steps:
- check:
name: addition is correct
values:
- <| 2 + 3 == 5 |>
```
The `<| ... |>` markers turn the body into a Python expression evaluated
at run time. Run it:
```sh
./run.sh -b -- tutorial.tum
```
## Step 2 — call your code with `py_func`
`check` only sees Python literals; to exercise `calc.py` we need a
`py_func` item. Replace the step:
```yaml
- py_func:
name: add 2 and 3
file: calc.py
func_name: add
param: [2, 3]
expected_result: 5
```
`expected_result` makes the item PASS only when the function returns
exactly that value.
The result is also stored in the global dict under `pfn_<name>`
(here `pfn_add 2 and 3`).
Anywhere in a `.tum`, `$(key)` is replaced at runtime by the value
stored in the global dict under `key`. A subsequent step can read the
result back with `$(pfn_<name>)`:
```yaml
- check:
name: result was 5
values:
- <| $(pfn_add 2 and 3) == 5 |>
```
## Step 3 — group several checks
Wrap the steps in a `group` to keep them visually together and let
testium report a per-group status:
```yaml
main:
name: calc.py campaign
steps:
- group:
name: add
steps:
- py_func:
name: 2 + 3
file: calc.py
func_name: add
param: [2, 3]
expected_result: 5
- py_func:
name: -1 + 1
file: calc.py
func_name: add
param: [-1, 1]
expected_result: 0
- group:
name: divide
steps:
- py_func:
name: 6 / 2
file: calc.py
func_name: divide
param: [6, 2]
expected_result: 3.0
```
A group fails as soon as one of its steps fails (set
`stop_on_failure: false` to keep going).
## Step 4 — define a variable with `let`
Avoid hard-coding the same number twice with a variable:
```yaml
- let:
name: define numerator
values:
- num: 6
- py_func:
name: divide num by 2
file: calc.py
func_name: divide
param:
- $(num)
- 2
expected_result: 3.0
```
`$(num)` expands to the global dict entry — when the stored value is a
number it is substituted as a number, no need to wrap it in `<| ... |>`.
## Step 5 — conditional execution
Skip a step when a condition is false:
```yaml
- py_func:
name: divide by zero only on linux
condition: <| "$(os)" == "Linux" |>
file: calc.py
func_name: divide
param: [1, 0]
```
Items skipped this way report `SKIP` and do not affect the overall
result.
## Step 6 — generate a report
Add a `report` block at the root of the file:
```yaml
main:
name: calc.py campaign
steps:
# ... your steps here ...
report:
enabled: true
log_stored: true
export:
- junit:
path: ./reports
file_name: calc.xml
- html:
path: ./reports
file_name: calc.html
```
The `path` directory must exist before the test runs — testium does not
create it. Create it once:
```sh
mkdir -p reports
```
Re-run the test — `./reports/calc.xml` (CI-friendly) and
`./reports/calc.html` (human-friendly) are produced. Set
`log_stored: true` to include each item's captured stdout.
## Step 7 — share state between calls
By default each `py_func` runs in its own short-lived subprocess.
To keep state across calls, use `context_id`:
```yaml
- py_func:
name: open
file: calc.py
func_name: open_resource
context_id: my_ctx
- py_func:
name: use
file: calc.py
func_name: use_resource
context_id: my_ctx
```
Both steps share the same persistent Python interpreter, so `calc.py`
can store any object in module-level globals or in `tm.setgd()`.
To share data without `context_id`, write it to the testium global dict
via the JSON-RPC bridge:
```python
import py_func.tm as tm
def producer():
tm.setgd("computed", 42)
def consumer():
return tm.gd("computed")
```
## Where to go next
* [`doc/examples/`](examples/) — one runnable `.tum` per feature
(cycles, dialogs, console, plots, parallel, run-of-tum, …).
* [`doc/manual/testium_manual.pdf`](manual/testium_manual.pdf) — full
reference manual covering every test item, every attribute and the
YAML syntax extensions.