Fix run item and batch mode robustness
- run item: rename tum_fime→tum, remove stdout=PIPE (deadlock with
spawn), support batch mode (-b), SUCCESS on any completed subprocess
regardless of sub-test result
- batch.py: fix control("loaded") deadlock via daemon thread + Event +
is_alive() polling; fix premature finish on gd_update messages;
propagate success flag from finished message; guard control("close")
- process.py: include success flag in send_finished message
- py_process/lua_process: add stdout/stderr=DEVNULL to Popen
- test_run.py: fix finished detection ("id" in m and m["id"] is None)
- testium_win.py: track run_exit_code, SIGABRT handler, clean exit
- __init__.py: sys.exit with batch success flag
- Add run item validation tests and CLAUDE.md documentation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
94
CLAUDE.md
Normal file
94
CLAUDE.md
Normal file
@@ -0,0 +1,94 @@
|
||||
# Testium — Claude Context
|
||||
|
||||
## What is testium
|
||||
|
||||
Testium is a test sequencer/runner written in Python. It executes YAML-based test scripts ("`.tum`" files) and supports three execution modes:
|
||||
|
||||
- **GUI mode** (default, no flag): PySide6 Qt application (`src/testium/main_win/`)
|
||||
- **Batch mode** (`-b` / `--batch-execution`): headless, non-interactive, runs tests and exits
|
||||
- **Terminal mode** (`-m` / `--terminal`): interactive REPL in the terminal (partially implemented / work-in-progress)
|
||||
|
||||
Run from repo root: `./run.sh` (Linux) or `run.bat` / `run.ps1` (Windows).
|
||||
Direct invocation: `python3 -m src/testium [-b|-m] <test_file.tum>`
|
||||
|
||||
## Architecture
|
||||
|
||||
### Entry point
|
||||
`src/testium/__init__.py` — parses CLI args, dispatches to the three modes.
|
||||
`multiprocessing.set_start_method('spawn')` is called early (required for Linux dialog subprocesses).
|
||||
|
||||
### Core execution
|
||||
- `src/testium/interpreter/process.py` — `TestProcess(multiprocessing.Process)`: runs the test in a child process. Stdout is redirected via a `StringQueue` → pipe → parent thread (`capture_stdout`) that writes to real stdout.
|
||||
- `src/testium/interpreter/batch.py` — `Batch`: parent-side orchestrator for `-b` mode. Creates the `msg_queue`, starts `TestProcess`, waits for the "finished" signal.
|
||||
- `src/testium/interpreter/test_set.py` — `TestSet`: builds and executes the tree of test items.
|
||||
- `src/testium/interpreter/test_items/test_item*.py` — one file per test item type (check, cycle, group, let, unittest, py_func, lua_func, console, git, dialogs, report, …).
|
||||
|
||||
### Communication channels (parent ↔ child process)
|
||||
- `msg_queue` (`multiprocessing.Queue`): carries status messages from child to parent.
|
||||
- Item status: `{"id": <non-None>, "name": ..., "status": "started"|"finished", ...}`
|
||||
- Global dict updates: `{"type": "gd_update"|"gd_delete", "key": ..., "value": ...}` — **no "id" key**
|
||||
- Process finished: `{"id": None, "name": "test_process", "status": "finished"}` — id key present but `None`
|
||||
- `tst_ctrl` (`TestSetController`): sends control commands (execute, stop, pause, close, …) from parent to child.
|
||||
- stdout pipe (`multiprocessing.Pipe`): streams test output from child back to parent's `capture_stdout` thread.
|
||||
|
||||
### Stdout pipeline (batch mode)
|
||||
```
|
||||
test item print()
|
||||
→ sys.stdout (StringQueue, in child)
|
||||
→ send_stdout thread (child) → pipe → capture_stdout thread (parent)
|
||||
→ print() → sys.stdout (TermLog wrapping real stdout, in parent)
|
||||
→ terminal
|
||||
```
|
||||
|
||||
### Global dictionary
|
||||
`src/testium/interpreter/utils/globdict.py` — shared state accessible from test scripts via `tm.gd()` / `tm.setgd()`. When `set_update_queue()` is active (during test execution), every `setgd`/`delgd` on a non-`_`-prefixed key pushes a message to `msg_queue`.
|
||||
|
||||
### Coloring (`-o` disables it)
|
||||
`src/testium/interpreter/utils/termlog.py` — `TermLog` wraps stdout with colorama-based line coloring (PASS=green, FAIL=red, WARN=yellow, …). Applied in parent process for batch/terminal modes. Auto-detects light/dark terminal background via (in order): `COLORFGBG` env var, OSC 11 query, default dark.
|
||||
|
||||
### Dialog items in batch mode
|
||||
All dialog items (`dialog_image`, `dialog_question`, `dialog_references`, `dialog_value`, `dialog_message`, `dialog_choices`, `dialog_note`) follow this rule in non-interactive text mode (`-b`):
|
||||
- `auto_result` defined in the `.tum` → result controlled by it (`ok`/`yes` → SUCCESS, `cancel`/`no` → FAIL)
|
||||
- `auto_result` absent → FAIL with `"Dialog not supported in batch mode"`
|
||||
- `sleep dialog: true` → exception: just sleeps normally, no GUI, no failure
|
||||
|
||||
`auto_result` (and `auto_value` for value/note dialogs) is intended for the validation test suite (`test/validation/`) only.
|
||||
|
||||
## Key files
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `src/testium/__init__.py` | CLI entry, mode dispatch |
|
||||
| `src/testium/interpreter/batch.py` | `-b` mode orchestrator |
|
||||
| `src/testium/interpreter/process.py` | Child test process |
|
||||
| `src/testium/interpreter/terminal.py` | `-m` mode (WIP) |
|
||||
| `src/testium/interpreter/test_set.py` | Test tree builder/executor |
|
||||
| `src/testium/interpreter/utils/globdict.py` | Global variable dict |
|
||||
| `src/testium/interpreter/utils/termlog.py` | Terminal color output |
|
||||
| `src/lib/stdout_redirect.py` | `StdioRedirect` singleton (`stdio_redir`) |
|
||||
| `src/lib/string_queue.py` | Thread-safe string buffer used for stdout redirection |
|
||||
| `src/testium/libs/testium.py` | Public API for test scripts (`tm.*`) |
|
||||
|
||||
## Known issues / WIP
|
||||
- `-m` (terminal mode) is not functional yet.
|
||||
- `text_no_pyside` branch: work in progress on text mode improvements.
|
||||
|
||||
### `run` item
|
||||
`src/testium/interpreter/test_items/test_item_run.py` — launches a `.tum` file in a new testium instance (`-b` in batch mode, `-r` in GUI mode). Result:
|
||||
- **SUCCESS** if the sub-instance launched and ran to completion (exit code is ignored)
|
||||
- **FAILURE** if the file is not found, `wait_for_exec` is set without `start_time`/`end_time`, the time window was not reached, or any other launch error
|
||||
|
||||
The sub-test's own pass/fail result is intentionally not propagated.
|
||||
|
||||
## Recent fixes (branch `text_no_pyside`)
|
||||
- `batch.py`: premature loop exit when `gd_update` messages (no `"id"` key) were mistaken for the "finished" signal — fix: `"id" in m and m["id"] is None`
|
||||
- `batch.py`: `control("loaded")` deadlock if `TestProcess` crashed before `cmd_th` started — fix: daemon thread + `threading.Event` + `is_alive()` polling
|
||||
- `termlog.py`: `COLOR_DEFAULT = Fore.WHITE` invisible on light terminals; added auto-detection + light palette. Also fixed `write()` residue accumulation bug (`s[pos:]` → `s[pos+1:]`).
|
||||
- Dialog items: `auto_result`/`auto_value` now used in non-interactive text mode; dialogs without `auto_result` FAIL immediately in batch mode.
|
||||
- `run` item: removed `stdout=PIPE` (caused deadlock with `multiprocessing` spawn); simplified result to SUCCESS on any completed subprocess.
|
||||
|
||||
## Validation tests
|
||||
Located in `test/validation/`. Run with `-b` flag against each `.tum` file.
|
||||
|
||||
## Dependencies
|
||||
See `src/requirements.txt`. Key ones: `pyside6`, `pyyaml`, `jinja2`, `colorama`, `gitpython`, `pexpect`, `matplotlib`.
|
||||
Reference in New Issue
Block a user