4 Commits

Author SHA1 Message Date
358ade8c98 Inc version 2026-05-05 09:21:43 +02:00
46bdb44cfb Route py_func/lua_func subprocess stdio into the parent log
stdout/stderr of the subprocesses were going to DEVNULL — early-startup
errors (lua require failures, exceptions before stdio_redir kicks in)
were lost.

New helper proc_drain.drain_to_log spawns a daemon thread per pipe that
print()s each line through stdio_redir, so it reaches the log + live
output. Used by py_process and lua_process with [py_func]/[lua_func]
prefixes.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-05 09:20:53 +02:00
41519c97cb Fix testium --version reporting "unknown" when installed from a wheel
get_testium_version() used pkg_resources (deprecated, slow to import)
and a narrow catch on git.InvalidGitRepositoryError; any other git
exception fell through to the outer except and returned "unknown".

- Use importlib.metadata.version("testium") to read the wheel
  version that setuptools bakes from src/VERSION at build time. Works
  out of any source checkout — pip-installed copies report
  "<x.y> (wheel release)" instead of "unknown".
- Source-checkout path: tried first when prefs.git_supported, broadly
  catches Exception so a missing repo / detached worktree / etc. no
  longer hides the wheel-metadata fallback.
- PyInstaller path: graceful "unknown (binary release)" if the bundled
  VERSION file is unreadable, instead of an unhandled exception.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-05 09:19:22 +02:00
b9475c6e9b docs: refocus README on users, add quick_start + tutorial, fill CONTRIBUTING
- README.md: pruned developer-oriented sections (Sphinx setup, Qt
  Creator workflow, VSCode debugging, release procedure, AppImage
  Wayland note) and replaced them with a user-facing layout: pre-built
  releases pointer, quick start, manual install, troubleshooting,
  licence.
- CONTRIBUTING.md: absorbed the developer content (debugging in VSCode,
  Qt GUI regen, Sphinx build, validation suite — batch + GUI variants,
  cross-distrib check, release procedure).
- doc/quick_start.md: 5-minute path from install to a passing test,
  in batch mode and in the GUI.
- doc/tutorial.md: guided walk-through against a small calc.py
  module — check, py_func, expected_result, $(...) expansion, group,
  let, condition, report (with the mkdir reminder), context_id.
- CLAUDE.md: subprocess API contract, bins.py, report-exporter
  plugin section, packaging matrix (wheel / PyInstaller / Flatpak /
  .deb work-in-progress), refreshed recent-fixes list. README/CLAUDE
  validation command no longer carries the spurious "-l" flag (which
  is GUI-only and a no-op in batch).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-05 09:18:59 +02:00
10 changed files with 588 additions and 213 deletions

View File

@@ -242,7 +242,7 @@ The `.deb` work-in-progress lives in `package/deb/`:
## Validation tests
Located in `test/validation/`. Run with `-b` flag:
```
./run.sh -b -l mon_log.log -- test/validation/main.tum
./run.sh -b -- test/validation/main.tum
```
Parallel item tests: `test/validation/items/parallel/test.tum`

View File

@@ -45,7 +45,7 @@ For existing files, keep the header that is already there.
3. Commit with a clear message (one logical change per commit).
4. Make sure the validation suite still passes:
```
./run.sh -b -l mon_log.log -- test/validation/main.tum
./run.sh -b -- test/validation/main.tum
```
5. Open a pull request against `main`.
@@ -56,6 +56,105 @@ For existing files, keep the header that is already there.
- Add or update tests in `test/validation/` for new test items or behaviours
- Update `CLAUDE.md` and the Sphinx manual for user-visible changes
## Development
### Debugging in VSCode
The recommended workflow:
1. Add a debug configuration to `.vscode/launch.json`:
```json
{
"configurations": [
{
"name": "Python : testium",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/src/testium",
"console": "integratedTerminal",
"args": ["-g"],
"justMyCode": true
}
]
}
```
2. Install `debugpy` in the venv: `python -m pip install debugpy`.
3. Open the *Run and Debug* tab and press play. testium starts; load and
run a `.tum` file. Set breakpoints where you want to investigate.
### Qt GUI modification
UI files (`*.ui`) are edited in **Qt Creator**. After editing, regenerate
the corresponding Python and resource files:
```sh
scripts/qt_generate.sh
```
Icons come from <https://github.com/free-icons/free-icons>.
### Sphinx documentation
```sh
pip install sphinx linuxdoc
doc/manual/sphinx/build_doc.sh
```
PDF generation requires `texlive`:
```sh
sudo apt install texlive-full
```
### Validation suite
Batch mode (CI-friendly, headless):
```sh
./run.sh -b -- test/validation/main.tum
```
GUI mode (loads the suite, click *Run* to execute and inspect the tree):
```sh
./run.sh test/validation/main.tum
```
GUI run-and-close (executes the suite, then closes):
```sh
./run.sh -r -- test/validation/main.tum
```
Subset run via the `items` define (works in any mode):
```sh
./run.sh -b -d "items=['parallel','common']" -- test/validation/main.tum
```
### Cross-distribution check
`package/deb/test_distro.sh` spins up a Docker/Podman container of the
target image, installs the expected system Python deps via apt (with
pip fallback for what is missing), installs the testium wheel and runs
the validation suite end-to-end. Currently green on `debian:bookworm`,
`debian:trixie`, `ubuntu:24.04`.
```sh
./package/deb/test_distro.sh debian:trixie
```
## Release procedure
1. Update `release_note.txt`.
2. Bump the version in `src/VERSION`.
3. Make sure the documentation is up to date — rebuild with
`doc/manual/sphinx/build_doc.sh` if needed.
4. Push and tag the commit with the new version.
5. Build the binary release: `package/pyinstaller/build.sh`.
6. Run the validation suite against each generated binary.
7. Confirm all validation results are green before publishing.
## Reporting security issues
Please do **not** report security vulnerabilities through public GitHub

283
README.md
View File

@@ -1,185 +1,110 @@
# Documentation
# testium
[See here](doc/manual/testium_manual.pdf).
testium is a YAML-driven test sequencer for hardware-in-the-loop and
integration testing. A test campaign is described in a `.tum` file as a tree
of items (checks, console interactions, Python/Lua functions, parallel blocks,
dialogs, …); testium executes the tree, captures results, and produces
reports in several formats.
# License
## Documentation
Copyright (c) 2025-2026 François Dausseur.
* [Quick start](doc/quick_start.md) — install and run your first test in
five minutes.
* [Tutorial](doc/tutorial.md) — guided walk-through of the most common
test items with a runnable example.
* [User manual (PDF)](doc/manual/testium_manual.pdf) — full reference.
* [`doc/examples/`](doc/examples/) — runnable `.tum` snippets.
## Pre-built releases
Pre-built artifacts are published at
<https://git.beafrancois.fr/v-and-v/testium/releases>:
* **Python wheel** (`testium-<version>-py3-none-any.whl`) — install with
`pip install testium-*.whl`. Lighter than the binary; pulls Python
dependencies from PyPI on install.
* **Self-contained Linux binary** (`testium`, built with PyInstaller) —
runnable directly, no Python installation required on the host. Lua
support still needs a system `lua` interpreter and the `lua-socket` /
`lua-cjson` modules.
* **Flatpak** — *coming soon.*
## Quick start
From a checkout of the repository:
| OS | Command |
|----|---------|
| Linux | `./run.sh` |
| Windows (cmd) | `run.bat` |
| Windows (PowerShell) | `run.ps1` |
The wrapper creates a Python virtual environment on first run and starts
testium in GUI mode. Add `-b path/to/test.tum` to run a test in batch mode.
## Manual installation
If the wrapper script does not fit your environment, set up testium manually:
```sh
python3 -m venv .venv
source .venv/bin/activate
pip install -r src/requirements.txt
```
Required Python packages (see `src/requirements.txt`):
`pyside6`, `pyserial`, `pyyaml`, `pexpect`, `gitpython`, `jinja2`, `colorama`,
`matplotlib`, `junit-xml`, `lxml`.
For tests using `lua_func` items, install Lua (>= 5.1) plus the `socket` and
`cjson` modules. On Debian/Ubuntu:
```sh
sudo apt install lua5.4 lua-socket lua-cjson
```
Run testium:
```sh
python3 src/testium # GUI
python3 src/testium -b mytest.tum # batch
```
## Troubleshooting
### `wl_proxy_marshal_flags` symbol error
```
testium: symbol lookup error: ... undefined symbol: wl_proxy_marshal_flags
```
Force the X11 Qt backend:
```sh
export QT_QPA_PLATFORM=xcb
testium
```
### `xcb plugin missing`
```
qt.qpa.plugin: Could not load the Qt platform plugin "xcb"
```
Install the missing system libraries:
```sh
sudo apt install libxcb-cursor0 libicu-dev libxcb-cursor-dev
```
## License
Copyright © 2025-2026 François Dausseur.
testium is distributed under the **European Union Public Licence v. 1.2
(EUPL-1.2)** — see the [LICENSE](LICENSE) file for the full text.
(EUPL-1.2)** — see [`LICENSE`](LICENSE) for the full text. SPDX:
`EUPL-1.2`.
SPDX identifier: `EUPL-1.2`
Contributions are accepted under the same licence (inbound = outbound). See
[CONTRIBUTING.md](CONTRIBUTING.md) for details.
# run testium
From the root path, on windows `cmd`:
run.bat
On windows powershell:
run.ps1
On linux:
./run.sh
The virtual environment is created if needed and *testium* is started.
# Manual setup
A python virtual environment should be created:
python3 -m venv <testium_venv>
## Requirements
In the virtual environment, the following modules must be installed:
* pyside6
* pyserial
* pyyaml
* pexpect
* gitpython
* jinja2
* colorama
* matplotlib
* junit-xml
* lxml
A `requirements.txt` file is also available in the git repository in the path `testium/src/`.
## run testium
from the testium path, execute
python3 -m src/testium
# Doc generation
## Install sphinx
pip install sphinx linuxdoc
## Generate the doc
Execute
doc/manual/sphinx/./build_doc.sh
This command works if texlive package has been installed on the system. It can be done by invoking the following command.
sudo apt install texlive-full
# QT GUI
## QT GUI modification
Open the ".ui" file with `qtcreator` and modify the gui. Then regenerate the python code.
On linux, a helper script has been created:
scripts/./qt_generate.sh
# Debugging
In order to debug testium or your python script executed within testium.
## In VSCODE
This is the prefered method :
1. Create a debug configuration like the following:
```
"configurations": [
{
"name": "Python : testium",
"type": "python",
"request": "launch",
"program": "${workspaceFolder}/src/testium",
"console": "integratedTerminal",
"args": ["-g"],
"justMyCode": true
},
]
```
2. Install debugpy module in python
python -m pip install debugpy
3. Then get to the "RUN AND DEBUG" tab and press the play button.
4. A testium window will pops up ; start execution of your tum.
5. Do not forget to put breakpoints where you want to investigate.
## Icons
Icons are coming from the following site: https://github.com/free-icons/free-icons.git
# testium Release
## Pre-requisite
A `python` virtual environment must have been set as described above.
### Install pyinstaller
Install `pyinstaller` package using pip.
## Generate the binary package
The procedure for a binary release is as follows:
1. update the `release_note.txt` file
2. modify the version in `src/VERSION` file
3. be sure that the documentation is up to date, and if not execute `doc/manual/sphinx/build_doc.sh` script
4. push modifications and create a tag with the new version on the git repository
5. generate an executable file by calling `package/pyinstaller/./build.sh`
6. run the complete validation test for each generated binary
7. check that all the validation results are OK
# Troubleshooting
## The testium exe crashes `wl_proxy_marshal_flags`
### Error message
/testium: symbol lookup error: /tmp/_MEIOhDCPF/libQt6WaylandClient.so.6: undefined symbol: wl_proxy_marshal_flags
### Solution
Set the appropriate environment variable
export QT_QPA_PLATFORM=xcb
testium
## xcb plugin missing
### Error message
qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found.
### Solution
A package is missing
sudo apt install libxcb-cursor0
sudo apt-get install libicu-dev
sudo apt-get install libxcb-cursor-dev
## The testium appimage crashes when opening a file
This is usually because wayland is defined as the default X server.
To change it :
* Disable Wayland by uncommenting WaylandEnable=false in the `/etc/gdm3/daemon.conf`
* Add `QT_QPA_PLATFORM=xcb` in `/etc/environment`
* After a reboot, check that the environment variable value returns `x11`:
$ echo $XDG_SESSION_TYPE
x11
Contributions are accepted under the same licence (inbound = outbound).
See [`CONTRIBUTING.md`](CONTRIBUTING.md) for development setup, debugging
workflow, and the release procedure.

66
doc/quick_start.md Normal file
View File

@@ -0,0 +1,66 @@
# Quick start
Five minutes from zero to a passing test.
## Install
From a checkout of the repository:
```sh
./run.sh --version # Linux
run.bat # Windows cmd
```
The wrapper creates a Python virtual environment on first run and verifies
testium starts. If you prefer a manual install, see the README.
## Your first test
Create `hello.tum`:
```yaml
main:
name: hello world
steps:
- check:
name: 1 + 1 makes 2
values:
- <| 1 + 1 == 2 |>
```
Run it in batch mode:
```sh
./run.sh -b -- hello.tum
```
You should see something like:
```
-----> step "1 + 1 makes 2" started
Check passed
<----- step "1 + 1 makes 2" finished: PASS
Test run success.
```
Replace `==` with `!=` and re-run — the step now ends with **FAIL** and
the process exits with code 1.
## Open it in the GUI
```sh
./run.sh hello.tum
```
The test tree appears in the left panel; click *Run test* in the toolbar.
Each item turns green or red live as it executes. Use `F1` on a selected
item to open its detail panel.
## Where to go next
* [`doc/tutorial.md`](tutorial.md) — a guided walk-through of the most
common test items (`py_func`, `let`, `group`, `condition`, `report`).
* [`doc/examples/`](examples/) — runnable `.tum` snippets covering one
feature each.
* [`doc/manual/testium_manual.pdf`](manual/testium_manual.pdf) —
full reference manual.

223
doc/tutorial.md Normal file
View File

@@ -0,0 +1,223 @@
# Tutorial — testing a small Python utility
This walk-through builds, step by step, a testium campaign that exercises
a small Python module. Each section adds one feature; you can follow
along by editing a single `.tum` file and re-running it.
If you have not yet run testium, start with [`quick_start.md`](quick_start.md).
## The code under test
Create `calc.py` next to your `.tum` file:
```python
def add(a, b):
return a + b
def divide(a, b):
return a / b
```
## Step 1 — a static check
The simplest item is `check`: it evaluates an expression and the test
passes iff the expression is truthy. Create `tutorial.tum`:
```yaml
main:
name: calc.py campaign
steps:
- check:
name: addition is correct
values:
- <| 2 + 3 == 5 |>
```
The `<| ... |>` markers turn the body into a Python expression evaluated
at run time. Run it:
```sh
./run.sh -b -- tutorial.tum
```
## Step 2 — call your code with `py_func`
`check` only sees Python literals; to exercise `calc.py` we need a
`py_func` item. Replace the step:
```yaml
- py_func:
name: add 2 and 3
file: calc.py
func_name: add
param: [2, 3]
expected_result: 5
```
`expected_result` makes the item PASS only when the function returns
exactly that value.
The result is also stored in the global dict under `pfn_<name>`
(here `pfn_add 2 and 3`).
Anywhere in a `.tum`, `$(key)` is replaced at runtime by the value
stored in the global dict under `key`. A subsequent step can read the
result back with `$(pfn_<name>)`:
```yaml
- check:
name: result was 5
values:
- <| $(pfn_add 2 and 3) == 5 |>
```
## Step 3 — group several checks
Wrap the steps in a `group` to keep them visually together and let
testium report a per-group status:
```yaml
main:
name: calc.py campaign
steps:
- group:
name: add
steps:
- py_func:
name: 2 + 3
file: calc.py
func_name: add
param: [2, 3]
expected_result: 5
- py_func:
name: -1 + 1
file: calc.py
func_name: add
param: [-1, 1]
expected_result: 0
- group:
name: divide
steps:
- py_func:
name: 6 / 2
file: calc.py
func_name: divide
param: [6, 2]
expected_result: 3.0
```
A group fails as soon as one of its steps fails (set
`stop_on_failure: false` to keep going).
## Step 4 — define a variable with `let`
Avoid hard-coding the same number twice with a variable:
```yaml
- let:
name: define numerator
values:
- num: 6
- py_func:
name: divide num by 2
file: calc.py
func_name: divide
param:
- $(num)
- 2
expected_result: 3.0
```
`$(num)` expands to the global dict entry — when the stored value is a
number it is substituted as a number, no need to wrap it in `<| ... |>`.
## Step 5 — conditional execution
Skip a step when a condition is false:
```yaml
- py_func:
name: divide by zero only on linux
condition: <| "$(os)" == "Linux" |>
file: calc.py
func_name: divide
param: [1, 0]
```
Items skipped this way report `SKIP` and do not affect the overall
result.
## Step 6 — generate a report
Add a `report` block at the root of the file:
```yaml
main:
name: calc.py campaign
steps:
# ... your steps here ...
report:
enabled: true
log_stored: true
export:
- junit:
path: ./reports
file_name: calc.xml
- html:
path: ./reports
file_name: calc.html
```
The `path` directory must exist before the test runs — testium does not
create it. Create it once:
```sh
mkdir -p reports
```
Re-run the test — `./reports/calc.xml` (CI-friendly) and
`./reports/calc.html` (human-friendly) are produced. Set
`log_stored: true` to include each item's captured stdout.
## Step 7 — share state between calls
By default each `py_func` runs in its own short-lived subprocess.
To keep state across calls, use `context_id`:
```yaml
- py_func:
name: open
file: calc.py
func_name: open_resource
context_id: my_ctx
- py_func:
name: use
file: calc.py
func_name: use_resource
context_id: my_ctx
```
Both steps share the same persistent Python interpreter, so `calc.py`
can store any object in module-level globals or in `tm.setgd()`.
To share data without `context_id`, write it to the testium global dict
via the JSON-RPC bridge:
```python
import py_func.tm as tm
def producer():
tm.setgd("computed", 42)
def consumer():
return tm.gd("computed")
```
## Where to go next
* [`doc/examples/`](examples/) — one runnable `.tum` per feature
(cycles, dialogs, console, plots, parallel, run-of-tum, …).
* [`doc/manual/testium_manual.pdf`](manual/testium_manual.pdf) — full
reference manual covering every test item, every attribute and the
YAML syntax extensions.

View File

@@ -1 +1 @@
0.1
0.2

View File

@@ -8,6 +8,7 @@ from runtime.jrpc import JsonRpcClient
from interpreter.utils.paths import subproc_path
from runtime.tum_except import ETUMRuntimeError
from interpreter.utils import bins
from interpreter.utils.proc_drain import drain_to_log
class LuaProcessBase:
@@ -93,10 +94,14 @@ class LuaProcessBase:
self._process = subprocess.Popen(
params, env=env, cwd=func_proc_path,
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
restore_signals=False,
)
# Route subprocess stdout/stderr (lua require failures, syntax
# errors, anything written to fd 1/2 before the in-script
# remote_print is set up) into the parent's log.
drain_to_log(self._process, prefix="[lua_func] ")
self._rpc = JsonRpcClient(
"localhost", self._port, req_handler=self._req_handler

View File

@@ -0,0 +1,48 @@
"""Drain a subprocess stdout/stderr into testium's print pipeline.
Captured lines go through the parent's stdio_redir, so they reach the
test log AND the live output (terminal in batch mode, GUI text panel
in -r mode). This is essential for diagnosing early-startup errors
of py_func / lua_func subprocesses (missing modules, unhandled
exceptions before the in-process redirection kicks in, lua
``require`` failures, anything written to fd 1/2 directly).
"""
import threading
def _drain_pipe(pipe, prefix):
try:
for raw in iter(pipe.readline, b""):
line = raw.decode("utf-8", errors="replace").rstrip("\r\n")
if not line:
continue
if prefix:
print(f"{prefix}{line}")
else:
print(line)
finally:
try:
pipe.close()
except Exception:
pass
def drain_to_log(process, prefix=""):
"""Spawn daemon threads that read ``process.stdout`` and
``process.stderr`` line by line and print each line through the
parent's stdout (so it reaches the log + live output).
Each thread exits cleanly when the subprocess closes the
corresponding pipe (i.e. when it exits). Daemon flag ensures they
do not block testium exit.
"""
threads = []
for pipe in (process.stdout, process.stderr):
if pipe is None:
continue
t = threading.Thread(
target=_drain_pipe, args=(pipe, prefix), daemon=True,
)
t.start()
threads.append(t)
return threads

View File

@@ -7,6 +7,7 @@ import api.testium as tm
from runtime.tum_except import ETUMRuntimeError
from interpreter.utils.paths import testium_path, subproc_path
from interpreter.utils import bins
from interpreter.utils.proc_drain import drain_to_log
class PyProcessBase:
@@ -77,10 +78,15 @@ class PyProcessBase:
self._process = subprocess.Popen(
params, env=env, cwd=func_proc_path,
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
restore_signals=False,
)
# Route subprocess stdout/stderr (early-startup errors,
# unhandled exceptions, anything written to fd 1/2 before the
# in-process JSON-RPC stdio_redir kicks in) into the parent's
# log.
drain_to_log(self._process, prefix="[py_func] ")
self._rpc = JsonRpcClient(
"localhost", self._port, req_handler=self._req_handler

View File

@@ -31,39 +31,42 @@ def get_version(path :str)-> str:
return "Warning git not supported in your settings, version of {} unknown".format(path)
def get_testium_version():
# case where we're executing from an Appimage
# AppImage
if 'APPIMAGE' in os.environ:
ver = 'unknown'
if 'SEQUENCER_REV' in os.environ:
ver = os.getenv('SEQUENCER_REV')
return (ver + " (binary release)")
ver = os.getenv('SEQUENCER_REV', 'unknown')
return ver + " (binary release)"
# case where we're executing from pyinstaller exe
# PyInstaller frozen exe
if getattr(sys, 'frozen', False):
file_path = os.path.join(sys._MEIPASS, "VERSION")
with open(file_path, 'r') as file:
ver = file.read()
return (ver + " (binary release)")
try:
with open(file_path, 'r') as f:
ver = f.read().strip()
return ver + " (binary release)"
except OSError:
return "unknown (binary release)"
# Executed from sources
try:
if prefs.settings.git_supported:
# Source checkout: prefer git revision when available
if prefs.settings.git_supported:
try:
git = import_module("git")
path = tm.get_main_dir()
try:
return repo_rev(path)
except git.InvalidGitRepositoryError:
pkg_rec = import_module("pkg_resources")
try:
ret = pkg_rec.get_distribution("testium").version
_cached_versions.update({path: ret})
return str(ret) + " (wheel release)"
except:
return "Warning : testium not versioned"
else:
return "Warning git not supported in your settings, version of testium is unknown."
except:
return ("Unknown")
return repo_rev(tm.get_main_dir())
except Exception:
# Not a git repo (typical pip install): fall through.
pass
# Pip-installed wheel: use the package metadata baked from VERSION
try:
from importlib.metadata import version as _pkg_version
from importlib.metadata import PackageNotFoundError
try:
return _pkg_version("testium") + " (wheel release)"
except PackageNotFoundError:
pass
except ImportError:
pass
return "unknown"
def get_modifications(path : str)-> str: