diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..4743b83 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,94 @@ +# Testium — Claude Context + +## What is testium + +Testium is a test sequencer/runner written in Python. It executes YAML-based test scripts ("`.tum`" files) and supports three execution modes: + +- **GUI mode** (default, no flag): PySide6 Qt application (`src/testium/main_win/`) +- **Batch mode** (`-b` / `--batch-execution`): headless, non-interactive, runs tests and exits +- **Terminal mode** (`-m` / `--terminal`): interactive REPL in the terminal (partially implemented / work-in-progress) + +Run from repo root: `./run.sh` (Linux) or `run.bat` / `run.ps1` (Windows). +Direct invocation: `python3 -m src/testium [-b|-m] ` + +## Architecture + +### Entry point +`src/testium/__init__.py` — parses CLI args, dispatches to the three modes. +`multiprocessing.set_start_method('spawn')` is called early (required for Linux dialog subprocesses). + +### Core execution +- `src/testium/interpreter/process.py` — `TestProcess(multiprocessing.Process)`: runs the test in a child process. Stdout is redirected via a `StringQueue` → pipe → parent thread (`capture_stdout`) that writes to real stdout. +- `src/testium/interpreter/batch.py` — `Batch`: parent-side orchestrator for `-b` mode. Creates the `msg_queue`, starts `TestProcess`, waits for the "finished" signal. +- `src/testium/interpreter/test_set.py` — `TestSet`: builds and executes the tree of test items. +- `src/testium/interpreter/test_items/test_item*.py` — one file per test item type (check, cycle, group, let, unittest, py_func, lua_func, console, git, dialogs, report, …). + +### Communication channels (parent ↔ child process) +- `msg_queue` (`multiprocessing.Queue`): carries status messages from child to parent. + - Item status: `{"id": , "name": ..., "status": "started"|"finished", ...}` + - Global dict updates: `{"type": "gd_update"|"gd_delete", "key": ..., "value": ...}` — **no "id" key** + - Process finished: `{"id": None, "name": "test_process", "status": "finished"}` — id key present but `None` +- `tst_ctrl` (`TestSetController`): sends control commands (execute, stop, pause, close, …) from parent to child. +- stdout pipe (`multiprocessing.Pipe`): streams test output from child back to parent's `capture_stdout` thread. + +### Stdout pipeline (batch mode) +``` +test item print() + → sys.stdout (StringQueue, in child) + → send_stdout thread (child) → pipe → capture_stdout thread (parent) + → print() → sys.stdout (TermLog wrapping real stdout, in parent) + → terminal +``` + +### Global dictionary +`src/testium/interpreter/utils/globdict.py` — shared state accessible from test scripts via `tm.gd()` / `tm.setgd()`. When `set_update_queue()` is active (during test execution), every `setgd`/`delgd` on a non-`_`-prefixed key pushes a message to `msg_queue`. + +### Coloring (`-o` disables it) +`src/testium/interpreter/utils/termlog.py` — `TermLog` wraps stdout with colorama-based line coloring (PASS=green, FAIL=red, WARN=yellow, …). Applied in parent process for batch/terminal modes. Auto-detects light/dark terminal background via (in order): `COLORFGBG` env var, OSC 11 query, default dark. + +### Dialog items in batch mode +All dialog items (`dialog_image`, `dialog_question`, `dialog_references`, `dialog_value`, `dialog_message`, `dialog_choices`, `dialog_note`) follow this rule in non-interactive text mode (`-b`): +- `auto_result` defined in the `.tum` → result controlled by it (`ok`/`yes` → SUCCESS, `cancel`/`no` → FAIL) +- `auto_result` absent → FAIL with `"Dialog not supported in batch mode"` +- `sleep dialog: true` → exception: just sleeps normally, no GUI, no failure + +`auto_result` (and `auto_value` for value/note dialogs) is intended for the validation test suite (`test/validation/`) only. + +## Key files + +| Path | Role | +|------|------| +| `src/testium/__init__.py` | CLI entry, mode dispatch | +| `src/testium/interpreter/batch.py` | `-b` mode orchestrator | +| `src/testium/interpreter/process.py` | Child test process | +| `src/testium/interpreter/terminal.py` | `-m` mode (WIP) | +| `src/testium/interpreter/test_set.py` | Test tree builder/executor | +| `src/testium/interpreter/utils/globdict.py` | Global variable dict | +| `src/testium/interpreter/utils/termlog.py` | Terminal color output | +| `src/lib/stdout_redirect.py` | `StdioRedirect` singleton (`stdio_redir`) | +| `src/lib/string_queue.py` | Thread-safe string buffer used for stdout redirection | +| `src/testium/libs/testium.py` | Public API for test scripts (`tm.*`) | + +## Known issues / WIP +- `-m` (terminal mode) is not functional yet. +- `text_no_pyside` branch: work in progress on text mode improvements. + +### `run` item +`src/testium/interpreter/test_items/test_item_run.py` — launches a `.tum` file in a new testium instance (`-b` in batch mode, `-r` in GUI mode). Result: +- **SUCCESS** if the sub-instance launched and ran to completion (exit code is ignored) +- **FAILURE** if the file is not found, `wait_for_exec` is set without `start_time`/`end_time`, the time window was not reached, or any other launch error + +The sub-test's own pass/fail result is intentionally not propagated. + +## Recent fixes (branch `text_no_pyside`) +- `batch.py`: premature loop exit when `gd_update` messages (no `"id"` key) were mistaken for the "finished" signal — fix: `"id" in m and m["id"] is None` +- `batch.py`: `control("loaded")` deadlock if `TestProcess` crashed before `cmd_th` started — fix: daemon thread + `threading.Event` + `is_alive()` polling +- `termlog.py`: `COLOR_DEFAULT = Fore.WHITE` invisible on light terminals; added auto-detection + light palette. Also fixed `write()` residue accumulation bug (`s[pos:]` → `s[pos+1:]`). +- Dialog items: `auto_result`/`auto_value` now used in non-interactive text mode; dialogs without `auto_result` FAIL immediately in batch mode. +- `run` item: removed `stdout=PIPE` (caused deadlock with `multiprocessing` spawn); simplified result to SUCCESS on any completed subprocess. + +## Validation tests +Located in `test/validation/`. Run with `-b` flag against each `.tum` file. + +## Dependencies +See `src/requirements.txt`. Key ones: `pyside6`, `pyyaml`, `jinja2`, `colorama`, `gitpython`, `pexpect`, `matplotlib`. diff --git a/doc/manual/sphinx/source/test_items/run_test_item.rst b/doc/manual/sphinx/source/test_items/run_test_item.rst index cf74bcb..7b71c09 100644 --- a/doc/manual/sphinx/source/test_items/run_test_item.rst +++ b/doc/manual/sphinx/source/test_items/run_test_item.rst @@ -8,9 +8,8 @@ This test item executes a new instance of testium. - run: name: Execute TUM - tum_fime: example_cycle.tum + tum: example_cycle.tum python_bin: python3 - testium_path: /home/francois/projets/testium-new-report/testium.pyw log_file: $(home)/reports/test.log report_file: $(home)/reports/test.rep @@ -19,7 +18,7 @@ Attributes run test item has the following specific attributes: -* ``tum_fime``: mandatory the path of the file to execute, it can be relative to current execution folder, +* ``tum``: mandatory the path of the file to execute, it can be relative to current execution folder, * ``param_file`` (optional) the path of the parameter file to use, otherwise default parameter file is used. * ``python_bin`` (optional) the path of a specific python to run your scripts, * ``testium_path`` (optional) the path of a specific testium to run your scripts, diff --git a/src/testium/__init__.py b/src/testium/__init__.py index 179f3e7..886b05f 100755 --- a/src/testium/__init__.py +++ b/src/testium/__init__.py @@ -125,6 +125,7 @@ def main(): from interpreter.batch import Batch b = Batch(tf, cf, defines, rf, args.report_type, pn, args.no_color, text_mode=True) + sys.exit(0 if b.success else 1) else: from main_win.testium_win import MainWin diff --git a/src/testium/interpreter/batch.py b/src/testium/interpreter/batch.py index 9fa7521..a44b63c 100644 --- a/src/testium/interpreter/batch.py +++ b/src/testium/interpreter/batch.py @@ -1,6 +1,7 @@ import os import sys import platform +import threading from time import sleep from signal import signal, SIGINT from queue import Empty @@ -8,7 +9,7 @@ from multiprocessing import Queue from interpreter.process import TestProcess from interpreter.utils.test_ctrl import TestSetController -from lib.tum_except import ETUMFileError +from lib.tum_except import ETUMFileError, ETUMRuntimeError from lib.stdout_redirect import stdio_redir @@ -52,6 +53,7 @@ class Batch: signal(SIGINT, self.sigint_handler) + self._success = False msg_queue = Queue() self.tst_ctrl = TestSetController() tst_proc = TestProcess( @@ -64,8 +66,17 @@ class Batch: ) tst_proc.start() - while not self.tst_ctrl.control("loaded"): - sleep(0.1) + # Wait for TestProcess to finish loading. + # Run the blocking control("loaded") in a daemon thread so we + # can watch for unexpected process death in the main thread. + _loaded_event = threading.Event() + def _wait_loaded(): + self.tst_ctrl.control("loaded") + _loaded_event.set() + threading.Thread(target=_wait_loaded, daemon=True).start() + while not _loaded_event.wait(timeout=0.1): + if not tst_proc.is_alive(): + raise ETUMRuntimeError("TestProcess terminated unexpectedly during load") self.tst_ctrl.control( "report", @@ -82,6 +93,7 @@ class Batch: m = msg_queue.get(timeout=0.2) if "id" in m and m["id"] is None: # id key present and None -> finished + self._success = m.get("success", False) break except Empty: if not tst_proc.is_alive(): @@ -89,7 +101,8 @@ class Batch: continue # Close the process and wait for termination - self.tst_ctrl.control("close") + if tst_proc.is_alive(): + self.tst_ctrl.control("close") tst_proc.join() except Exception as e: @@ -98,6 +111,10 @@ class Batch: finally: stdio_redir.restore() + @property + def success(self): + return self._success + def sigint_handler(self, signal_received, frame): try: self.tst_ctrl.control("stop", timeout=5) diff --git a/src/testium/interpreter/process.py b/src/testium/interpreter/process.py index 2fd93c6..ee470a2 100644 --- a/src/testium/interpreter/process.py +++ b/src/testium/interpreter/process.py @@ -283,7 +283,7 @@ Is the python exec path correct ?""" engine.stop() engine.join() # Sends signal to the GUI - self.send_finished() + self.send_finished(success=test_set.success()) globdict.set_update_queue(None) restore_gd(gdict) except Exception as e: @@ -339,8 +339,10 @@ Is the python exec path correct ?""" stdio_redir.restore() stdio_redir.stop() - def send_finished(self): + def send_finished(self, success=None): status = {"id": None, "name": "test_process", "status": "finished"} + if success is not None: + status["success"] = success self.__squeue.put(status) def execute(self): diff --git a/src/testium/interpreter/test_items/test_item_run.py b/src/testium/interpreter/test_items/test_item_run.py index 59a45ee..39f805a 100644 --- a/src/testium/interpreter/test_items/test_item_run.py +++ b/src/testium/interpreter/test_items/test_item_run.py @@ -31,7 +31,7 @@ class TestItemRun(TestItem): self._type = cst.TYPE_RUN self.is_container = False with item_load_context(self.cmd(), self.name(), self.seqFilename()): - self.tum_fime = self._prms.getParam('tum_fime', required=True) + self.tum_file = self._prms.getParam('tum', required=True) self.param_file = self._prms.getParam('param_file', default='') self.python_bin = self._prms.getParam('python_bin', default='') self.testium_path = self._prms.getParam('testium_path', default='') @@ -43,39 +43,43 @@ class TestItemRun(TestItem): @test_run def execute(self): - res = -1 try: - file_path = self._prms.expanse(self.tum_fime) + file_path = self._prms.expanse(self.tum_file) if not os.path.exists(file_path) and not os.path.isabs(file_path): - file_path = os.path.join(tm.gd('test_directory'), self.tum_fime) + file_path = os.path.join(tm.gd('test_directory'), file_path) if not os.path.isfile(file_path): raise ETUMRuntimeError( '"{}" file could not be found'.format(file_path)) - self.tum_fime = file_path + self.tum_file = file_path pf = self._prms.expanse(self.param_file) pp = self._prms.expanse(self.python_bin) sp = self._prms.expanse(self.testium_path) lp = self._prms.expanse(self.log_path) rp = self._prms.expanse(self.report_path) cmd = [] + if sp == '': + sp = sys.argv[0] if pp != '': cmd.append(pp) - if sp == '': - sp = os.path.join(tm.get_main_dir(), "testium.pyw") + elif not os.path.isfile(sp) or not os.access(sp, os.X_OK): + cmd.append(sys.executable) cmd.append(sp) - if lp == '': - lp = os.path.splitext(self.tum_fime)[0] + "_" + \ - datetime.utcnow().isoformat(timespec='seconds') + '.log' - cmd.append("-r") + if tm.text_mode(): + cmd.append("-b") + else: + cmd.append("-r") + if lp == '': + lp = os.path.splitext(self.tum_file)[0] + "_" + \ + datetime.utcnow().isoformat(timespec='seconds') + '.log' + cmd.append("-l") + cmd.append('"' + lp + '"') if pf != '': cmd.append("-c") cmd.append('"' + pf + '"') - cmd.append("-l") - cmd.append('"' + lp + '"') if rp != '': cmd.append("-p") cmd.append('"' + rp + '"') - cmd.append(self.tum_fime) + cmd.append(self.tum_file) for c in cmd: print(c, end = ' ') @@ -90,31 +94,23 @@ class TestItemRun(TestItem): raise ETUMRuntimeError( '"wait_for_exec" set but not start_time or end_time') + r = None if self.wait_for_exec: while not nowInBetween(self.start_time, self.end_time): sleep(60) - r = subprocess.run( - cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + r = subprocess.run(cmd) elif self.start_time is not None and self.end_time is not None: if nowInBetween(self.start_time, self.end_time): - r = subprocess.run( - cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + r = subprocess.run(cmd) elif self.start_time is not None: if self.start_time < datetime.now().time(): - r = subprocess.run( - cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + r = subprocess.run(cmd) else: - r = subprocess.run( - cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + r = subprocess.run(cmd) if isinstance(r, subprocess.CompletedProcess): - print((r.stdout).decode()) - print(r.stderr.decode()) - res = r.returncode - if res >= 0: self.result.set(TestValue.SUCCESS) else: - self.result.set(TestValue.FAILURE, - 'Test execution returned negative value.') + self.result.set(TestValue.FAILURE, 'Sub-test did not execute') except: traceback.print_exception(*sys.exc_info()) self.result.set(TestValue.FAILURE, 'Unrecoverable "run" item error') diff --git a/src/testium/interpreter/utils/lua_process.py b/src/testium/interpreter/utils/lua_process.py index ca19fe3..2b919e7 100644 --- a/src/testium/interpreter/utils/lua_process.py +++ b/src/testium/interpreter/utils/lua_process.py @@ -191,7 +191,10 @@ class LuaProcessBase: self._process = subprocess.Popen( params, env=env, cwd=func_proc_path, - stdin=subprocess.DEVNULL, restore_signals=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + restore_signals=False, ) self._rpc = JsonRpcClient( diff --git a/src/testium/interpreter/utils/py_process.py b/src/testium/interpreter/utils/py_process.py index 531dbc6..faeef01 100644 --- a/src/testium/interpreter/utils/py_process.py +++ b/src/testium/interpreter/utils/py_process.py @@ -160,7 +160,10 @@ class PyProcessBase: self._process = subprocess.Popen( params, env=env, cwd=func_proc_path, - stdin=subprocess.DEVNULL, restore_signals=False, + stdin=subprocess.DEVNULL, + stdout=subprocess.DEVNULL, + stderr=subprocess.DEVNULL, + restore_signals=False, ) self._rpc = JsonRpcClient( diff --git a/src/testium/main_win/test_run/test_run.py b/src/testium/main_win/test_run/test_run.py index fba9a59..d09c044 100644 --- a/src/testium/main_win/test_run/test_run.py +++ b/src/testium/main_win/test_run/test_run.py @@ -28,7 +28,7 @@ class ThreadTestStatus(QThread): self.gdUpdated.emit(m["key"], m["value"]) elif msg_type == "gd_delete": self.gdDeleted.emit(m["key"]) - elif m.get("id", None) is None: + elif "id" in m and m["id"] is None: self.testSetIsFinished.emit() else: self.statusToBeUpdated.emit(m) diff --git a/src/testium/main_win/test_runner.py b/src/testium/main_win/test_runner.py index 9fb9f0a..ad77e55 100644 --- a/src/testium/main_win/test_runner.py +++ b/src/testium/main_win/test_runner.py @@ -118,6 +118,7 @@ class TestRunner: self.logFileHandler = None w.textLog.appendPlainText("Test is finished") + w.run_exit_code = 0 if w.treeTests.getGlobalSuccess() else 1 if w.runandclose: w.on_actionExit_triggered() diff --git a/src/testium/main_win/testium_win.py b/src/testium/main_win/testium_win.py index 4088f2a..8672e29 100755 --- a/src/testium/main_win/testium_win.py +++ b/src/testium/main_win/testium_win.py @@ -1,18 +1,14 @@ import sys import os -import subprocess -import traceback import webbrowser -from time import sleep from multiprocessing import Queue -from queue import Empty from threading import Thread import shutil # Qt -from PySide6 import QtGui, QtWidgets +from PySide6 import QtGui from PySide6.QtGui import QAction, QShortcut, QIcon, QPixmap, QTextCursor, QDesktopServices, QTextCursor -from PySide6.QtCore import Slot, QUrl, Qt, QTimer, QDateTime +from PySide6.QtCore import Slot, QUrl, Qt, QTimer from PySide6.QtWidgets import ( QApplication, @@ -92,6 +88,7 @@ class MainWindow(QMainWindow, Ui_MainWindow): self.test_service = None self.threadTestStatus = None self._signals_connected = False + self.run_exit_code = -1 # -1 = test not yet completed self.timer = QTimer() self.timer.setSingleShot(False) @@ -359,14 +356,20 @@ class MainWindow(QMainWindow, Ui_MainWindow): self.treeTests.saveSizes() prefs.settings.sync() + def closeEvent(self, event): + self.on_exiting() + event.accept() + def on_exiting(self): - if self.runner.state == TestState.IDLE: - self.save_settings() - self.file_manager.clear_process() - self.threadTestStatus.stop() - self.threadOutput.stop() - self.threadOutput.wait() - self.threadTestStatus.wait() + try: + if self.runner.state == TestState.IDLE: + self.save_settings() + self.file_manager.clear_process() + finally: + self.threadTestStatus.stop() + self.threadOutput.stop() + self.threadOutput.wait() + self.threadTestStatus.wait() def show_checkboxes(self, hidden=None): if hidden: @@ -685,5 +688,17 @@ def MainWin( debug, ) + import signal + import os as _os + + def _sigabrt_handler(signum, frame): + # Qt crash: exit with the test result if known, -1 if test never completed + _os._exit(ui.run_exit_code) + + signal.signal(signal.SIGABRT, _sigabrt_handler) + ui.show() - sys.exit(app.exec_()) + app.exec_() + exit_code = ui.run_exit_code if ui.run_exit_code >= 0 else 0 + del ui + sys.exit(exit_code) diff --git a/test/validation/items/run/param.yaml b/test/validation/items/run/param.yaml new file mode 100644 index 0000000..0af0f7f --- /dev/null +++ b/test/validation/items/run/param.yaml @@ -0,0 +1 @@ +no_param: Null diff --git a/test/validation/items/run/sub_fail.tum b/test/validation/items/run/sub_fail.tum new file mode 100644 index 0000000..c86b417 --- /dev/null +++ b/test/validation/items/run/sub_fail.tum @@ -0,0 +1,7 @@ +main: + name: run sub-test (always fail) + steps: + - check: + name: fail + values: + - false diff --git a/test/validation/items/run/sub_pass.tum b/test/validation/items/run/sub_pass.tum new file mode 100644 index 0000000..072658f --- /dev/null +++ b/test/validation/items/run/sub_pass.tum @@ -0,0 +1,7 @@ +main: + name: run sub-test (always pass) + steps: + - check: + name: pass + values: + - true diff --git a/test/validation/items/run/test.tum b/test/validation/items/run/test.tum new file mode 100644 index 0000000..e54c1c1 --- /dev/null +++ b/test/validation/items/run/test.tum @@ -0,0 +1,25 @@ +# run item: launches a .tum file in a new testium instance. +# In batch mode the sub-instance runs with -b; in GUI mode with -r. +# The run item result is SUCCESS if the sub-instance launched successfully, +# regardless of its own test result. + +- run: + name: run PASS (valid file, passing sub-test) + key: $(test)_PASS + tum: $(test_path)$(psep)sub_pass.tum + +- run: + name: run PASS (valid file, failing sub-test) + key: $(test)_PASS + tum: $(test_path)$(psep)sub_fail.tum + +- run: + name: run FAIL (file not found) + key: $(test)_FAIL + tum: $(test_path)$(psep)non_existent.tum + +- run: + name: run FAIL (wait_for_exec without time window) + key: $(test)_FAIL + tum: $(test_path)$(psep)sub_pass.tum + wait_for_exec: true