Fix run item and batch mode robustness

- run item: rename tum_fime→tum, remove stdout=PIPE (deadlock with spawn), support batch mode (-b), SUCCESS on any completed subprocess regardless of sub-test result - batch.py: fix control("loaded") deadlock via daemon thread + Event + is_alive() polling; fix premature finish on gd_update messages; propagate success flag from finished message; guard control("close") - process.py: include success flag in send_finished message - py_process/lua_process: add stdout/stderr=DEVNULL to Popen - test_run.py: fix finished detection ("id" in m and m["id"] is None) - testium_win.py: track run_exit_code, SIGABRT handler, clean exit - __init__.py: sys.exit with batch success flag - Add run item validation tests and CLAUDE.md documentation Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-27 07:49:16 +02:00
parent a3e449cc7d
commit 60dbcf0252
15 changed files with 225 additions and 54 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -0,0 +1,94 @@
+# Testium — Claude Context
+
+## What is testium
+
+Testium is a test sequencer/runner written in Python. It executes YAML-based test scripts ("`.tum`" files) and supports three execution modes:
+
+- **GUI mode** (default, no flag): PySide6 Qt application (`src/testium/main_win/`)
+- **Batch mode** (`-b` / `--batch-execution`): headless, non-interactive, runs tests and exits
+- **Terminal mode** (`-m` / `--terminal`): interactive REPL in the terminal (partially implemented / work-in-progress)
+
+Run from repo root: `./run.sh` (Linux) or `run.bat` / `run.ps1` (Windows).  
+Direct invocation: `python3 -m src/testium [-b|-m] <test_file.tum>`
+
+## Architecture
+
+### Entry point
+`src/testium/__init__.py` — parses CLI args, dispatches to the three modes.  
+`multiprocessing.set_start_method('spawn')` is called early (required for Linux dialog subprocesses).
+
+### Core execution
+- `src/testium/interpreter/process.py` — `TestProcess(multiprocessing.Process)`: runs the test in a child process. Stdout is redirected via a `StringQueue` → pipe → parent thread (`capture_stdout`) that writes to real stdout.
+- `src/testium/interpreter/batch.py` — `Batch`: parent-side orchestrator for `-b` mode. Creates the `msg_queue`, starts `TestProcess`, waits for the "finished" signal.
+- `src/testium/interpreter/test_set.py` — `TestSet`: builds and executes the tree of test items.
+- `src/testium/interpreter/test_items/test_item*.py` — one file per test item type (check, cycle, group, let, unittest, py_func, lua_func, console, git, dialogs, report, …).
+
+### Communication channels (parent ↔ child process)
+- `msg_queue` (`multiprocessing.Queue`): carries status messages from child to parent.
+  - Item status: `{"id": <non-None>, "name": ..., "status": "started"|"finished", ...}`
+  - Global dict updates: `{"type": "gd_update"|"gd_delete", "key": ..., "value": ...}` — **no "id" key**
+  - Process finished: `{"id": None, "name": "test_process", "status": "finished"}` — id key present but `None`
+- `tst_ctrl` (`TestSetController`): sends control commands (execute, stop, pause, close, …) from parent to child.
+- stdout pipe (`multiprocessing.Pipe`): streams test output from child back to parent's `capture_stdout` thread.
+
+### Stdout pipeline (batch mode)
+```
+test item print()
+  → sys.stdout (StringQueue, in child)
+  → send_stdout thread (child) → pipe → capture_stdout thread (parent)
+  → print() → sys.stdout (TermLog wrapping real stdout, in parent)
+  → terminal
+```
+
+### Global dictionary
+`src/testium/interpreter/utils/globdict.py` — shared state accessible from test scripts via `tm.gd()` / `tm.setgd()`. When `set_update_queue()` is active (during test execution), every `setgd`/`delgd` on a non-`_`-prefixed key pushes a message to `msg_queue`.
+
+### Coloring (`-o` disables it)
+`src/testium/interpreter/utils/termlog.py` — `TermLog` wraps stdout with colorama-based line coloring (PASS=green, FAIL=red, WARN=yellow, …). Applied in parent process for batch/terminal modes. Auto-detects light/dark terminal background via (in order): `COLORFGBG` env var, OSC 11 query, default dark.
+
+### Dialog items in batch mode
+All dialog items (`dialog_image`, `dialog_question`, `dialog_references`, `dialog_value`, `dialog_message`, `dialog_choices`, `dialog_note`) follow this rule in non-interactive text mode (`-b`):
+- `auto_result` defined in the `.tum` → result controlled by it (`ok`/`yes` → SUCCESS, `cancel`/`no` → FAIL)
+- `auto_result` absent → FAIL with `"Dialog not supported in batch mode"`
+- `sleep dialog: true` → exception: just sleeps normally, no GUI, no failure
+
+`auto_result` (and `auto_value` for value/note dialogs) is intended for the validation test suite (`test/validation/`) only.
+
+## Key files
+
+| Path | Role |
+|------|------|
+| `src/testium/__init__.py` | CLI entry, mode dispatch |
+| `src/testium/interpreter/batch.py` | `-b` mode orchestrator |
+| `src/testium/interpreter/process.py` | Child test process |
+| `src/testium/interpreter/terminal.py` | `-m` mode (WIP) |
+| `src/testium/interpreter/test_set.py` | Test tree builder/executor |
+| `src/testium/interpreter/utils/globdict.py` | Global variable dict |
+| `src/testium/interpreter/utils/termlog.py` | Terminal color output |
+| `src/lib/stdout_redirect.py` | `StdioRedirect` singleton (`stdio_redir`) |
+| `src/lib/string_queue.py` | Thread-safe string buffer used for stdout redirection |
+| `src/testium/libs/testium.py` | Public API for test scripts (`tm.*`) |
+
+## Known issues / WIP
+- `-m` (terminal mode) is not functional yet.
+- `text_no_pyside` branch: work in progress on text mode improvements.
+
+### `run` item
+`src/testium/interpreter/test_items/test_item_run.py` — launches a `.tum` file in a new testium instance (`-b` in batch mode, `-r` in GUI mode). Result:
+- **SUCCESS** if the sub-instance launched and ran to completion (exit code is ignored)
+- **FAILURE** if the file is not found, `wait_for_exec` is set without `start_time`/`end_time`, the time window was not reached, or any other launch error
+
+The sub-test's own pass/fail result is intentionally not propagated.
+
+## Recent fixes (branch `text_no_pyside`)
+- `batch.py`: premature loop exit when `gd_update` messages (no `"id"` key) were mistaken for the "finished" signal — fix: `"id" in m and m["id"] is None`
+- `batch.py`: `control("loaded")` deadlock if `TestProcess` crashed before `cmd_th` started — fix: daemon thread + `threading.Event` + `is_alive()` polling
+- `termlog.py`: `COLOR_DEFAULT = Fore.WHITE` invisible on light terminals; added auto-detection + light palette. Also fixed `write()` residue accumulation bug (`s[pos:]` → `s[pos+1:]`).
+- Dialog items: `auto_result`/`auto_value` now used in non-interactive text mode; dialogs without `auto_result` FAIL immediately in batch mode.
+- `run` item: removed `stdout=PIPE` (caused deadlock with `multiprocessing` spawn); simplified result to SUCCESS on any completed subprocess.
+
+## Validation tests
+Located in `test/validation/`. Run with `-b` flag against each `.tum` file.
+
+## Dependencies
+See `src/requirements.txt`. Key ones: `pyside6`, `pyyaml`, `jinja2`, `colorama`, `gitpython`, `pexpect`, `matplotlib`.
--- a/doc/manual/sphinx/source/test_items/run_test_item.rst
+++ b/doc/manual/sphinx/source/test_items/run_test_item.rst
@@ -8,9 +8,8 @@ This test item executes a new instance of testium.

    - run:
        name: Execute TUM
-        tum_fime: example_cycle.tum
+        tum: example_cycle.tum
        python_bin: python3
-        testium_path: /home/francois/projets/testium-new-report/testium.pyw
        log_file: $(home)/reports/test.log
        report_file: $(home)/reports/test.rep

@@ -19,7 +18,7 @@ Attributes

 run test item has the following specific attributes:

-* ``tum_fime``: mandatory the path of the file to execute, it can be relative to current execution folder,
+* ``tum``: mandatory the path of the file to execute, it can be relative to current execution folder,
 * ``param_file`` (optional) the path of the parameter file to use, otherwise default parameter file is used.
 * ``python_bin`` (optional) the path of a specific python to run your scripts,
 * ``testium_path`` (optional) the path of a specific testium to run your scripts,
--- a/src/testium/init.py
+++ b/src/testium/init.py
@@ -125,6 +125,7 @@ def main():

        from interpreter.batch import Batch
        b = Batch(tf, cf, defines, rf, args.report_type, pn, args.no_color, text_mode=True)
+        sys.exit(0 if b.success else 1)

    else:
        from main_win.testium_win import MainWin
--- a/src/testium/interpreter/batch.py
+++ b/src/testium/interpreter/batch.py
@@ -1,6 +1,7 @@
 import os
 import sys
 import platform
+import threading
 from time import sleep
 from signal import signal, SIGINT
 from queue import Empty
@@ -8,7 +9,7 @@ from multiprocessing import Queue

 from interpreter.process import TestProcess
 from interpreter.utils.test_ctrl import TestSetController
-from lib.tum_except import ETUMFileError
+from lib.tum_except import ETUMFileError, ETUMRuntimeError
 from lib.stdout_redirect import stdio_redir


@@ -52,6 +53,7 @@ class Batch:

                signal(SIGINT, self.sigint_handler)

+                self._success = False
                msg_queue = Queue()
                self.tst_ctrl = TestSetController()
                tst_proc = TestProcess(
@@ -64,8 +66,17 @@ class Batch:
                )
                tst_proc.start()

-                while not self.tst_ctrl.control("loaded"):
-                    sleep(0.1)
+                # Wait for TestProcess to finish loading.
+                # Run the blocking control("loaded") in a daemon thread so we
+                # can watch for unexpected process death in the main thread.
+                _loaded_event = threading.Event()
+                def _wait_loaded():
+                    self.tst_ctrl.control("loaded")
+                    _loaded_event.set()
+                threading.Thread(target=_wait_loaded, daemon=True).start()
+                while not _loaded_event.wait(timeout=0.1):
+                    if not tst_proc.is_alive():
+                        raise ETUMRuntimeError("TestProcess terminated unexpectedly during load")

                self.tst_ctrl.control(
                    "report",
@@ -82,6 +93,7 @@ class Batch:
                        m = msg_queue.get(timeout=0.2)
                        if "id" in m and m["id"] is None:
                            # id key present and None -> finished
+                            self._success = m.get("success", False)
                            break
                    except Empty:
                        if not tst_proc.is_alive():
@@ -89,6 +101,7 @@ class Batch:
                        continue

                # Close the process and wait for termination
+                if tst_proc.is_alive():
                    self.tst_ctrl.control("close")
                tst_proc.join()

@@ -98,6 +111,10 @@ class Batch:
        finally:
            stdio_redir.restore()

+    @property
+    def success(self):
+        return self._success
+
    def sigint_handler(self, signal_received, frame):
        try:
            self.tst_ctrl.control("stop", timeout=5)
--- a/src/testium/interpreter/process.py
+++ b/src/testium/interpreter/process.py
@@ -283,7 +283,7 @@ Is the python exec path correct ?"""
                                    engine.stop()
                                    engine.join()
                                # Sends signal to the GUI
-                                self.send_finished()
+                                self.send_finished(success=test_set.success())
                                globdict.set_update_queue(None)
                                restore_gd(gdict)
                        except Exception as e:
@@ -339,8 +339,10 @@ Is the python exec path correct ?"""
        stdio_redir.restore()
        stdio_redir.stop()

-    def send_finished(self):
+    def send_finished(self, success=None):
        status = {"id": None, "name": "test_process", "status": "finished"}
+        if success is not None:
+            status["success"] = success
        self.__squeue.put(status)

    def execute(self):
--- a/src/testium/interpreter/test_items/test_item_run.py
+++ b/src/testium/interpreter/test_items/test_item_run.py
@@ -31,7 +31,7 @@ class TestItemRun(TestItem):
        self._type = cst.TYPE_RUN
        self.is_container = False
        with item_load_context(self.cmd(), self.name(), self.seqFilename()):
-            self.tum_fime = self._prms.getParam('tum_fime', required=True)
+            self.tum_file = self._prms.getParam('tum', required=True)
            self.param_file = self._prms.getParam('param_file', default='')
            self.python_bin = self._prms.getParam('python_bin', default='')
            self.testium_path = self._prms.getParam('testium_path', default='')
@@ -43,39 +43,43 @@ class TestItemRun(TestItem):

    @test_run
    def execute(self):
-        res = -1
        try:
-            file_path = self._prms.expanse(self.tum_fime)
+            file_path = self._prms.expanse(self.tum_file)
            if not os.path.exists(file_path) and not os.path.isabs(file_path):
-                file_path = os.path.join(tm.gd('test_directory'), self.tum_fime)
+                file_path = os.path.join(tm.gd('test_directory'), file_path)
            if not os.path.isfile(file_path):
                raise ETUMRuntimeError(
                    '"{}" file could not be found'.format(file_path))
-            self.tum_fime = file_path
+            self.tum_file = file_path
            pf = self._prms.expanse(self.param_file)
            pp = self._prms.expanse(self.python_bin)
            sp = self._prms.expanse(self.testium_path)
            lp = self._prms.expanse(self.log_path)
            rp = self._prms.expanse(self.report_path)
            cmd = []
+            if sp == '':
+                sp = sys.argv[0]
            if pp != '':
                cmd.append(pp)
-            if sp == '':
-                sp = os.path.join(tm.get_main_dir(), "testium.pyw")
+            elif not os.path.isfile(sp) or not os.access(sp, os.X_OK):
+                cmd.append(sys.executable)
            cmd.append(sp)
-            if lp == '':
-                lp = os.path.splitext(self.tum_fime)[0] + "_" + \
-                    datetime.utcnow().isoformat(timespec='seconds') + '.log'
+            if tm.text_mode():
+                cmd.append("-b")
+            else:
                cmd.append("-r")
+                if lp == '':
+                    lp = os.path.splitext(self.tum_file)[0] + "_" + \
+                        datetime.utcnow().isoformat(timespec='seconds') + '.log'
+                cmd.append("-l")
+                cmd.append('"' + lp + '"')
            if pf != '':
                cmd.append("-c")
                cmd.append('"' + pf + '"')
-            cmd.append("-l")
-            cmd.append('"' + lp + '"')
            if rp != '':
                cmd.append("-p")
                cmd.append('"' + rp + '"')
-            cmd.append(self.tum_fime)
+            cmd.append(self.tum_file)
            for c in cmd:
                print(c, end = ' ')

@@ -90,31 +94,23 @@ class TestItemRun(TestItem):
                raise ETUMRuntimeError(
                    '"wait_for_exec" set but not start_time or end_time')

+            r = None
            if self.wait_for_exec:
                while not nowInBetween(self.start_time, self.end_time):
                    sleep(60)
-                r = subprocess.run(
-                    cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                r = subprocess.run(cmd)
            elif self.start_time is not None and self.end_time is not None:
                if nowInBetween(self.start_time, self.end_time):
-                    r = subprocess.run(
-                        cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                    r = subprocess.run(cmd)
            elif self.start_time is not None:
                if self.start_time < datetime.now().time():
-                    r = subprocess.run(
-                        cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                    r = subprocess.run(cmd)
            else:
-                r = subprocess.run(
-                    cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+                r = subprocess.run(cmd)
            if isinstance(r, subprocess.CompletedProcess):
-                print((r.stdout).decode())
-                print(r.stderr.decode())
-                res = r.returncode
-            if res >= 0:
                self.result.set(TestValue.SUCCESS)
            else:
-                self.result.set(TestValue.FAILURE,
-                                'Test execution returned negative value.')
+                self.result.set(TestValue.FAILURE, 'Sub-test did not execute')
        except:
            traceback.print_exception(*sys.exc_info())
            self.result.set(TestValue.FAILURE, 'Unrecoverable "run" item error')
--- a/src/testium/interpreter/utils/lua_process.py
+++ b/src/testium/interpreter/utils/lua_process.py
@@ -191,7 +191,10 @@ class LuaProcessBase:

        self._process = subprocess.Popen(
            params, env=env, cwd=func_proc_path,
-            stdin=subprocess.DEVNULL, restore_signals=False,
+            stdin=subprocess.DEVNULL,
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            restore_signals=False,
        )

        self._rpc = JsonRpcClient(
--- a/src/testium/interpreter/utils/py_process.py
+++ b/src/testium/interpreter/utils/py_process.py
@@ -160,7 +160,10 @@ class PyProcessBase:

        self._process = subprocess.Popen(
            params, env=env, cwd=func_proc_path,
-            stdin=subprocess.DEVNULL, restore_signals=False,
+            stdin=subprocess.DEVNULL,
+            stdout=subprocess.DEVNULL,
+            stderr=subprocess.DEVNULL,
+            restore_signals=False,
        )

        self._rpc = JsonRpcClient(
--- a/src/testium/main_win/test_run/test_run.py
+++ b/src/testium/main_win/test_run/test_run.py
@@ -28,7 +28,7 @@ class ThreadTestStatus(QThread):
                        self.gdUpdated.emit(m["key"], m["value"])
                    elif msg_type == "gd_delete":
                        self.gdDeleted.emit(m["key"])
-                    elif m.get("id", None) is None:
+                    elif "id" in m and m["id"] is None:
                        self.testSetIsFinished.emit()
                    else:
                        self.statusToBeUpdated.emit(m)
--- a/src/testium/main_win/test_runner.py
+++ b/src/testium/main_win/test_runner.py
@@ -118,6 +118,7 @@ class TestRunner:
        self.logFileHandler = None

        w.textLog.appendPlainText("Test is finished")
+        w.run_exit_code = 0 if w.treeTests.getGlobalSuccess() else 1
        if w.runandclose:
            w.on_actionExit_triggered()

--- a/src/testium/main_win/testium_win.py
+++ b/src/testium/main_win/testium_win.py
@@ -1,18 +1,14 @@
 import sys
 import os
-import subprocess
-import traceback
 import webbrowser
-from time import sleep
 from multiprocessing import Queue
-from queue import Empty
 from threading import Thread
 import shutil

 # Qt
-from PySide6 import QtGui, QtWidgets
+from PySide6 import QtGui
 from PySide6.QtGui import QAction, QShortcut, QIcon, QPixmap, QTextCursor, QDesktopServices, QTextCursor
-from PySide6.QtCore import Slot, QUrl, Qt, QTimer, QDateTime
+from PySide6.QtCore import Slot, QUrl, Qt, QTimer

 from PySide6.QtWidgets import (
    QApplication,
@@ -92,6 +88,7 @@ class MainWindow(QMainWindow, Ui_MainWindow):
        self.test_service = None
        self.threadTestStatus = None
        self._signals_connected = False
+        self.run_exit_code = -1  # -1 = test not yet completed

        self.timer = QTimer()
        self.timer.setSingleShot(False)
@@ -359,10 +356,16 @@ class MainWindow(QMainWindow, Ui_MainWindow):
        self.treeTests.saveSizes()
        prefs.settings.sync()

+    def closeEvent(self, event):
+        self.on_exiting()
+        event.accept()
+
    def on_exiting(self):
+        try:
            if self.runner.state == TestState.IDLE:
                self.save_settings()
            self.file_manager.clear_process()
+        finally:
            self.threadTestStatus.stop()
            self.threadOutput.stop()
            self.threadOutput.wait()
@@ -685,5 +688,17 @@ def MainWin(
        debug,
    )

+    import signal
+    import os as _os
+
+    def _sigabrt_handler(signum, frame):
+        # Qt crash: exit with the test result if known, -1 if test never completed
+        _os._exit(ui.run_exit_code)
+
+    signal.signal(signal.SIGABRT, _sigabrt_handler)
+
    ui.show()
-    sys.exit(app.exec_())
+    app.exec_()
+    exit_code = ui.run_exit_code if ui.run_exit_code >= 0 else 0
+    del ui
+    sys.exit(exit_code)
--- a/test/validation/items/run/param.yaml
+++ b/test/validation/items/run/param.yaml
@@ -0,0 +1 @@
+no_param: Null
--- a/test/validation/items/run/sub_fail.tum
+++ b/test/validation/items/run/sub_fail.tum
@@ -0,0 +1,7 @@
+main:
+    name: run sub-test (always fail)
+    steps:
+        - check:
+            name: fail
+            values:
+                - false
--- a/test/validation/items/run/sub_pass.tum
+++ b/test/validation/items/run/sub_pass.tum
@@ -0,0 +1,7 @@
+main:
+    name: run sub-test (always pass)
+    steps:
+        - check:
+            name: pass
+            values:
+                - true
--- a/test/validation/items/run/test.tum
+++ b/test/validation/items/run/test.tum
@@ -0,0 +1,25 @@
+# run item: launches a .tum file in a new testium instance.
+# In batch mode the sub-instance runs with -b; in GUI mode with -r.
+# The run item result is SUCCESS if the sub-instance launched successfully,
+# regardless of its own test result.
+
+- run:
+    name: run PASS (valid file, passing sub-test)
+    key: $(test)_PASS
+    tum: $(test_path)$(psep)sub_pass.tum
+
+- run:
+    name: run PASS (valid file, failing sub-test)
+    key: $(test)_PASS
+    tum: $(test_path)$(psep)sub_fail.tum
+
+- run:
+    name: run FAIL (file not found)
+    key: $(test)_FAIL
+    tum: $(test_path)$(psep)non_existent.tum
+
+- run:
+    name: run FAIL (wait_for_exec without time window)
+    key: $(test)_FAIL
+    tum: $(test_path)$(psep)sub_pass.tum
+    wait_for_exec: true