From b9475c6e9b24c68093d63fd583c0683f27ecceed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Fran=C3=A7ois?= Date: Tue, 5 May 2026 09:18:59 +0200 Subject: [PATCH] docs: refocus README on users, add quick_start + tutorial, fill CONTRIBUTING MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - README.md: pruned developer-oriented sections (Sphinx setup, Qt Creator workflow, VSCode debugging, release procedure, AppImage Wayland note) and replaced them with a user-facing layout: pre-built releases pointer, quick start, manual install, troubleshooting, licence. - CONTRIBUTING.md: absorbed the developer content (debugging in VSCode, Qt GUI regen, Sphinx build, validation suite — batch + GUI variants, cross-distrib check, release procedure). - doc/quick_start.md: 5-minute path from install to a passing test, in batch mode and in the GUI. - doc/tutorial.md: guided walk-through against a small calc.py module — check, py_func, expected_result, $(...) expansion, group, let, condition, report (with the mkdir reminder), context_id. - CLAUDE.md: subprocess API contract, bins.py, report-exporter plugin section, packaging matrix (wheel / PyInstaller / Flatpak / .deb work-in-progress), refreshed recent-fixes list. README/CLAUDE validation command no longer carries the spurious "-l" flag (which is GUI-only and a no-op in batch). Co-Authored-By: Claude Opus 4.7 --- CLAUDE.md | 2 +- CONTRIBUTING.md | 101 +++++++++++++++- README.md | 283 +++++++++++++++++---------------------------- doc/quick_start.md | 66 +++++++++++ doc/tutorial.md | 223 +++++++++++++++++++++++++++++++++++ 5 files changed, 494 insertions(+), 181 deletions(-) create mode 100644 doc/quick_start.md create mode 100644 doc/tutorial.md diff --git a/CLAUDE.md b/CLAUDE.md index 8cbeb46..bc9f61a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -242,7 +242,7 @@ The `.deb` work-in-progress lives in `package/deb/`: ## Validation tests Located in `test/validation/`. Run with `-b` flag: ``` -./run.sh -b -l mon_log.log -- test/validation/main.tum +./run.sh -b -- test/validation/main.tum ``` Parallel item tests: `test/validation/items/parallel/test.tum` diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5f9138a..bd4f1c4 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -45,7 +45,7 @@ For existing files, keep the header that is already there. 3. Commit with a clear message (one logical change per commit). 4. Make sure the validation suite still passes: ``` - ./run.sh -b -l mon_log.log -- test/validation/main.tum + ./run.sh -b -- test/validation/main.tum ``` 5. Open a pull request against `main`. @@ -56,6 +56,105 @@ For existing files, keep the header that is already there. - Add or update tests in `test/validation/` for new test items or behaviours - Update `CLAUDE.md` and the Sphinx manual for user-visible changes +## Development + +### Debugging in VSCode + +The recommended workflow: + +1. Add a debug configuration to `.vscode/launch.json`: + ```json + { + "configurations": [ + { + "name": "Python : testium", + "type": "python", + "request": "launch", + "program": "${workspaceFolder}/src/testium", + "console": "integratedTerminal", + "args": ["-g"], + "justMyCode": true + } + ] + } + ``` +2. Install `debugpy` in the venv: `python -m pip install debugpy`. +3. Open the *Run and Debug* tab and press play. testium starts; load and + run a `.tum` file. Set breakpoints where you want to investigate. + +### Qt GUI modification + +UI files (`*.ui`) are edited in **Qt Creator**. After editing, regenerate +the corresponding Python and resource files: + +```sh +scripts/qt_generate.sh +``` + +Icons come from . + +### Sphinx documentation + +```sh +pip install sphinx linuxdoc +doc/manual/sphinx/build_doc.sh +``` + +PDF generation requires `texlive`: + +```sh +sudo apt install texlive-full +``` + +### Validation suite + +Batch mode (CI-friendly, headless): + +```sh +./run.sh -b -- test/validation/main.tum +``` + +GUI mode (loads the suite, click *Run* to execute and inspect the tree): + +```sh +./run.sh test/validation/main.tum +``` + +GUI run-and-close (executes the suite, then closes): + +```sh +./run.sh -r -- test/validation/main.tum +``` + +Subset run via the `items` define (works in any mode): + +```sh +./run.sh -b -d "items=['parallel','common']" -- test/validation/main.tum +``` + +### Cross-distribution check + +`package/deb/test_distro.sh` spins up a Docker/Podman container of the +target image, installs the expected system Python deps via apt (with +pip fallback for what is missing), installs the testium wheel and runs +the validation suite end-to-end. Currently green on `debian:bookworm`, +`debian:trixie`, `ubuntu:24.04`. + +```sh +./package/deb/test_distro.sh debian:trixie +``` + +## Release procedure + +1. Update `release_note.txt`. +2. Bump the version in `src/VERSION`. +3. Make sure the documentation is up to date — rebuild with + `doc/manual/sphinx/build_doc.sh` if needed. +4. Push and tag the commit with the new version. +5. Build the binary release: `package/pyinstaller/build.sh`. +6. Run the validation suite against each generated binary. +7. Confirm all validation results are green before publishing. + ## Reporting security issues Please do **not** report security vulnerabilities through public GitHub diff --git a/README.md b/README.md index d664f50..a526e3e 100644 --- a/README.md +++ b/README.md @@ -1,185 +1,110 @@ -# Documentation +# testium -[See here](doc/manual/testium_manual.pdf). +testium is a YAML-driven test sequencer for hardware-in-the-loop and +integration testing. A test campaign is described in a `.tum` file as a tree +of items (checks, console interactions, Python/Lua functions, parallel blocks, +dialogs, …); testium executes the tree, captures results, and produces +reports in several formats. -# License +## Documentation -Copyright (c) 2025-2026 François Dausseur. +* [Quick start](doc/quick_start.md) — install and run your first test in + five minutes. +* [Tutorial](doc/tutorial.md) — guided walk-through of the most common + test items with a runnable example. +* [User manual (PDF)](doc/manual/testium_manual.pdf) — full reference. +* [`doc/examples/`](doc/examples/) — runnable `.tum` snippets. + +## Pre-built releases + +Pre-built artifacts are published at +: + +* **Python wheel** (`testium--py3-none-any.whl`) — install with + `pip install testium-*.whl`. Lighter than the binary; pulls Python + dependencies from PyPI on install. +* **Self-contained Linux binary** (`testium`, built with PyInstaller) — + runnable directly, no Python installation required on the host. Lua + support still needs a system `lua` interpreter and the `lua-socket` / + `lua-cjson` modules. +* **Flatpak** — *coming soon.* + +## Quick start + +From a checkout of the repository: + +| OS | Command | +|----|---------| +| Linux | `./run.sh` | +| Windows (cmd) | `run.bat` | +| Windows (PowerShell) | `run.ps1` | + +The wrapper creates a Python virtual environment on first run and starts +testium in GUI mode. Add `-b path/to/test.tum` to run a test in batch mode. + +## Manual installation + +If the wrapper script does not fit your environment, set up testium manually: + +```sh +python3 -m venv .venv +source .venv/bin/activate +pip install -r src/requirements.txt +``` + +Required Python packages (see `src/requirements.txt`): +`pyside6`, `pyserial`, `pyyaml`, `pexpect`, `gitpython`, `jinja2`, `colorama`, +`matplotlib`, `junit-xml`, `lxml`. + +For tests using `lua_func` items, install Lua (>= 5.1) plus the `socket` and +`cjson` modules. On Debian/Ubuntu: + +```sh +sudo apt install lua5.4 lua-socket lua-cjson +``` + +Run testium: + +```sh +python3 src/testium # GUI +python3 src/testium -b mytest.tum # batch +``` + +## Troubleshooting + +### `wl_proxy_marshal_flags` symbol error + +``` +testium: symbol lookup error: ... undefined symbol: wl_proxy_marshal_flags +``` + +Force the X11 Qt backend: + +```sh +export QT_QPA_PLATFORM=xcb +testium +``` + +### `xcb plugin missing` + +``` +qt.qpa.plugin: Could not load the Qt platform plugin "xcb" +``` + +Install the missing system libraries: + +```sh +sudo apt install libxcb-cursor0 libicu-dev libxcb-cursor-dev +``` + +## License + +Copyright © 2025-2026 François Dausseur. testium is distributed under the **European Union Public Licence v. 1.2 -(EUPL-1.2)** — see the [LICENSE](LICENSE) file for the full text. +(EUPL-1.2)** — see [`LICENSE`](LICENSE) for the full text. SPDX: +`EUPL-1.2`. -SPDX identifier: `EUPL-1.2` - -Contributions are accepted under the same licence (inbound = outbound). See -[CONTRIBUTING.md](CONTRIBUTING.md) for details. - -# run testium - -From the root path, on windows `cmd`: - - run.bat - -On windows powershell: - - run.ps1 - -On linux: - - ./run.sh - -The virtual environment is created if needed and *testium* is started. - -# Manual setup - -A python virtual environment should be created: - - python3 -m venv - -## Requirements - -In the virtual environment, the following modules must be installed: - -* pyside6 -* pyserial -* pyyaml -* pexpect -* gitpython -* jinja2 -* colorama -* matplotlib -* junit-xml -* lxml - -A `requirements.txt` file is also available in the git repository in the path `testium/src/`. - - -## run testium - -from the testium path, execute - - python3 -m src/testium - -# Doc generation - -## Install sphinx - - pip install sphinx linuxdoc - -## Generate the doc - -Execute - - doc/manual/sphinx/./build_doc.sh - -This command works if texlive package has been installed on the system. It can be done by invoking the following command. - - sudo apt install texlive-full - -# QT GUI - -## QT GUI modification - -Open the ".ui" file with `qtcreator` and modify the gui. Then regenerate the python code. - -On linux, a helper script has been created: - scripts/./qt_generate.sh - -# Debugging - -In order to debug testium or your python script executed within testium. - -## In VSCODE - -This is the prefered method : - -1. Create a debug configuration like the following: - -``` - "configurations": [ - { - "name": "Python : testium", - "type": "python", - "request": "launch", - "program": "${workspaceFolder}/src/testium", - "console": "integratedTerminal", - "args": ["-g"], - "justMyCode": true - }, - ] -``` - -2. Install debugpy module in python - - python -m pip install debugpy -3. Then get to the "RUN AND DEBUG" tab and press the play button. -4. A testium window will pops up ; start execution of your tum. -5. Do not forget to put breakpoints where you want to investigate. - -## Icons - -Icons are coming from the following site: https://github.com/free-icons/free-icons.git - -# testium Release - -## Pre-requisite - -A `python` virtual environment must have been set as described above. - -### Install pyinstaller - -Install `pyinstaller` package using pip. - -## Generate the binary package - -The procedure for a binary release is as follows: - -1. update the `release_note.txt` file -2. modify the version in `src/VERSION` file -3. be sure that the documentation is up to date, and if not execute `doc/manual/sphinx/build_doc.sh` script -4. push modifications and create a tag with the new version on the git repository -5. generate an executable file by calling `package/pyinstaller/./build.sh` -6. run the complete validation test for each generated binary -7. check that all the validation results are OK - -# Troubleshooting - -## The testium exe crashes `wl_proxy_marshal_flags` - -### Error message - - /testium: symbol lookup error: /tmp/_MEIOhDCPF/libQt6WaylandClient.so.6: undefined symbol: wl_proxy_marshal_flags - -### Solution - -Set the appropriate environment variable - - export QT_QPA_PLATFORM=xcb - testium - -## xcb plugin missing - -### Error message - - qt.qpa.plugin: Could not load the Qt platform plugin "xcb" in "" even though it was found. - -### Solution - -A package is missing - - sudo apt install libxcb-cursor0 - sudo apt-get install libicu-dev - sudo apt-get install libxcb-cursor-dev - -## The testium appimage crashes when opening a file - -This is usually because wayland is defined as the default X server. - -To change it : - -* Disable Wayland by uncommenting WaylandEnable=false in the `/etc/gdm3/daemon.conf` -* Add `QT_QPA_PLATFORM=xcb` in `/etc/environment` -* After a reboot, check that the environment variable value returns `x11`: - - $ echo $XDG_SESSION_TYPE - x11 +Contributions are accepted under the same licence (inbound = outbound). +See [`CONTRIBUTING.md`](CONTRIBUTING.md) for development setup, debugging +workflow, and the release procedure. diff --git a/doc/quick_start.md b/doc/quick_start.md new file mode 100644 index 0000000..15d8cf3 --- /dev/null +++ b/doc/quick_start.md @@ -0,0 +1,66 @@ +# Quick start + +Five minutes from zero to a passing test. + +## Install + +From a checkout of the repository: + +```sh +./run.sh --version # Linux +run.bat # Windows cmd +``` + +The wrapper creates a Python virtual environment on first run and verifies +testium starts. If you prefer a manual install, see the README. + +## Your first test + +Create `hello.tum`: + +```yaml +main: + name: hello world + steps: + - check: + name: 1 + 1 makes 2 + values: + - <| 1 + 1 == 2 |> +``` + +Run it in batch mode: + +```sh +./run.sh -b -- hello.tum +``` + +You should see something like: + +``` +-----> step "1 + 1 makes 2" started +Check passed +<----- step "1 + 1 makes 2" finished: PASS +Test run success. +``` + +Replace `==` with `!=` and re-run — the step now ends with **FAIL** and +the process exits with code 1. + +## Open it in the GUI + +```sh +./run.sh hello.tum +``` + +The test tree appears in the left panel; click *Run test* in the toolbar. +Each item turns green or red live as it executes. Use `F1` on a selected +item to open its detail panel. + +## Where to go next + +* [`doc/tutorial.md`](tutorial.md) — a guided walk-through of the most + common test items (`py_func`, `let`, `group`, `condition`, `report`). +* [`doc/examples/`](examples/) — runnable `.tum` snippets covering one + feature each. +* [`doc/manual/testium_manual.pdf`](manual/testium_manual.pdf) — + full reference manual. diff --git a/doc/tutorial.md b/doc/tutorial.md new file mode 100644 index 0000000..5393c10 --- /dev/null +++ b/doc/tutorial.md @@ -0,0 +1,223 @@ +# Tutorial — testing a small Python utility + +This walk-through builds, step by step, a testium campaign that exercises +a small Python module. Each section adds one feature; you can follow +along by editing a single `.tum` file and re-running it. + +If you have not yet run testium, start with [`quick_start.md`](quick_start.md). + +## The code under test + +Create `calc.py` next to your `.tum` file: + +```python +def add(a, b): + return a + b + +def divide(a, b): + return a / b +``` + +## Step 1 — a static check + +The simplest item is `check`: it evaluates an expression and the test +passes iff the expression is truthy. Create `tutorial.tum`: + +```yaml +main: + name: calc.py campaign + steps: + - check: + name: addition is correct + values: + - <| 2 + 3 == 5 |> +``` + +The `<| ... |>` markers turn the body into a Python expression evaluated +at run time. Run it: + +```sh +./run.sh -b -- tutorial.tum +``` + +## Step 2 — call your code with `py_func` + +`check` only sees Python literals; to exercise `calc.py` we need a +`py_func` item. Replace the step: + +```yaml + - py_func: + name: add 2 and 3 + file: calc.py + func_name: add + param: [2, 3] + expected_result: 5 +``` + +`expected_result` makes the item PASS only when the function returns +exactly that value. + +The result is also stored in the global dict under `pfn_` +(here `pfn_add 2 and 3`). + +Anywhere in a `.tum`, `$(key)` is replaced at runtime by the value +stored in the global dict under `key`. A subsequent step can read the +result back with `$(pfn_)`: + +```yaml + - check: + name: result was 5 + values: + - <| $(pfn_add 2 and 3) == 5 |> +``` + +## Step 3 — group several checks + +Wrap the steps in a `group` to keep them visually together and let +testium report a per-group status: + +```yaml +main: + name: calc.py campaign + steps: + - group: + name: add + steps: + - py_func: + name: 2 + 3 + file: calc.py + func_name: add + param: [2, 3] + expected_result: 5 + - py_func: + name: -1 + 1 + file: calc.py + func_name: add + param: [-1, 1] + expected_result: 0 + - group: + name: divide + steps: + - py_func: + name: 6 / 2 + file: calc.py + func_name: divide + param: [6, 2] + expected_result: 3.0 +``` + +A group fails as soon as one of its steps fails (set +`stop_on_failure: false` to keep going). + +## Step 4 — define a variable with `let` + +Avoid hard-coding the same number twice with a variable: + +```yaml + - let: + name: define numerator + values: + - num: 6 + - py_func: + name: divide num by 2 + file: calc.py + func_name: divide + param: + - $(num) + - 2 + expected_result: 3.0 +``` + +`$(num)` expands to the global dict entry — when the stored value is a +number it is substituted as a number, no need to wrap it in `<| ... |>`. + +## Step 5 — conditional execution + +Skip a step when a condition is false: + +```yaml + - py_func: + name: divide by zero only on linux + condition: <| "$(os)" == "Linux" |> + file: calc.py + func_name: divide + param: [1, 0] +``` + +Items skipped this way report `SKIP` and do not affect the overall +result. + +## Step 6 — generate a report + +Add a `report` block at the root of the file: + +```yaml +main: + name: calc.py campaign + steps: + # ... your steps here ... + +report: + enabled: true + log_stored: true + export: + - junit: + path: ./reports + file_name: calc.xml + - html: + path: ./reports + file_name: calc.html +``` + +The `path` directory must exist before the test runs — testium does not +create it. Create it once: + +```sh +mkdir -p reports +``` + +Re-run the test — `./reports/calc.xml` (CI-friendly) and +`./reports/calc.html` (human-friendly) are produced. Set +`log_stored: true` to include each item's captured stdout. + +## Step 7 — share state between calls + +By default each `py_func` runs in its own short-lived subprocess. +To keep state across calls, use `context_id`: + +```yaml + - py_func: + name: open + file: calc.py + func_name: open_resource + context_id: my_ctx + - py_func: + name: use + file: calc.py + func_name: use_resource + context_id: my_ctx +``` + +Both steps share the same persistent Python interpreter, so `calc.py` +can store any object in module-level globals or in `tm.setgd()`. + +To share data without `context_id`, write it to the testium global dict +via the JSON-RPC bridge: + +```python +import py_func.tm as tm + +def producer(): + tm.setgd("computed", 42) + +def consumer(): + return tm.gd("computed") +``` + +## Where to go next + +* [`doc/examples/`](examples/) — one runnable `.tum` per feature + (cycles, dialogs, console, plots, parallel, run-of-tum, …). +* [`doc/manual/testium_manual.pdf`](manual/testium_manual.pdf) — full + reference manual covering every test item, every attribute and the + YAML syntax extensions.