A self-loading item that can't load its module/file (unittest test file
with a missing import, pytest not installed on the host, ...) no longer
aborts the whole test load. TestSet._load_item() wraps load(), warns at
load time and records item._load_error; @test_run turns it into a clean
run-time FAILURE. The rest of the campaign loads and runs.
Scoped to module-loading items (unittest; pytest once merged). Structural
action loading stays fail-fast.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Use the official pytest logo (devicon) for the pytest item instead of
reusing python.png — visually distinct from the py_func (Python logo)
item. Three 64x64 theme variants (color RGBA, black RGBA silhouette,
white LA), declared in the QRC and compiled into testium_core_win_rc.py;
_ITEM_CONFIG "pytest" now points to pytest.png.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New test_items/pytest_test_item.rst (params, host-subprocess execution,
pytest-must-be-installed note), wired into the toctree. Regenerated PDF.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
0.3 is reserved for the upcoming pytest item release. Regenerated the
manual PDF so its version stamp matches.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Variable substitution ($(...)) must use the runtime global dict, so it
must happen at run time (execute), never at load (__init__).
- console telnet_port: was never expanded — `telnet_port: $(port)` stayed
literal. Now expanded at run (processed=True in execute, like the other
host/port params).
- test_item base: stop_on_failure / execute_on_stop are now stored raw and
resolved at run time via properties (so a $(...) flag reflects the
runtime value, not the load-time one).
- cycle iterator and git repo: drop the redundant load-time expansion
(execute() already re-expands them).
- tested_references: fetch 'reference' raw, expand each value in execute().
Justified load-time exceptions kept: name, doc, skipped (static/GUI at
load) and unittest test_method (drives child loading at load).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Document that read_until's 'expected' accepts a list (match any) and the
new 'regex' flag, with examples and the bounded-window limitation note.
Regenerated manual PDF.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
read_until:
- 'expected' now accepts a list of values (succeeds on any match).
- new 'regex: true' flag: each pattern is a Python regex (re.search over a
bounded tail, Console.REGEX_WINDOW). Reports which pattern matched.
Serial console robustness & clarity:
- failed open() raises a clear ETUMRuntimeError ("Serial device '…' does not
exist." / permission hint) instead of a raw pyserial traceback.
- a console whose open failed is safely "not open" (init _thd=None +
isOpened guards in readchar/read_nowait/close) — no more cascading
AttributeError: '_thd' on subsequent read steps.
- action handlers: one-liner for expected (ETUMRuntimeError) errors, full
traceback kept for unexpected ones. All console errors use testium
exceptions (ETUMRuntimeError).
Flatpak: grant --device=all so serial adapters (/dev/ttyUSB*, /dev/ttyACM*)
are visible in the sandbox.
Validation: new read_until list/regex (match + no-match) cases in
items/console/test.tum.
Version: 0.3.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run a user pytest file as a testium item, surfacing each collected test
as a child with its own PASS/FAIL/SKIP, duration and failure message.
Mirrors the unittest item but runs pytest in a subprocess on the host
interpreter (bins.python_bin(), like py_func/lua_func) so it works across
every packaging channel. A stdlib-only pytest plugin streams collected
node-ids and per-test results over stdout via sentinels; the parent parses
them live. Params: test_file, test_method. stop_on_failure maps to -x;
disabled children are reported NORUN without running.
Wiring: TYPE_PYTEST / TYPE_PYTEST_STEP constants, test_init registration,
self-loading branch in test_set, GUI tree icon. Schema/LSP pick it up
automatically from the declarative PARAMS.
Validation: test/validation/items/pytest/ (validation venv now installs
pytest).
WIP: paused mid-feature (DESIGN.md documented; manual section pending).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- PyInstaller exe built windowed (console=False) with package/testium.ico
as the embedded icon (BMP entries for shell compatibility).
- Suppress stray subprocess console windows in the frozen Windows build via
paths.no_window_kwargs() (CREATE_NO_WINDOW); wheel/source unchanged.
Applied to py_process, lua_process, bins probes, sys_app_path_win.
- New per-user Inno Setup installer (package/innosetup/): no admin,
version-scoped AppId/dir so versions install side-by-side, one Start
Menu entry per version, .ico shipped for shortcut/uninstall icons.
- DESIGN.md + release_note.txt updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
bins.ensure() now stores the resolved interpreter path under
python_bin / lua_bin when the key is unset, so test scripts can use
$(python_bin) / $(lua_bin) in GUI mode (no -d override). Restores the
behaviour lost when bins.py centralised resolution. A user-provided
value is left untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The subprocess now binds port 0, prints the bound port on stdout after
listen(), and the parent connects only once it reads that port. Removes
the reserve/close/rebind race and SO_REUSEADDR, and the connect-before-
ready timing guess that failed intermittently on Windows. wait_ready()
no longer hangs when a connection attempt fails.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make the suite run cleanly on Windows.
Product code:
- __init__.py: force UTF-8 on stdout/stderr. The Windows console code
page (cp1252) cannot encode the box-drawing/accented characters the
runner prints, which crashed the parent capture_stdout thread. Only
the stream encoders are reconfigured; the locale default used to read
cp1252 config files is left untouched.
- report_export_junit/html: open the report file with encoding="utf-8"
(XML/HTML are UTF-8) instead of the platform default, matching the
txt/json exporters.
Validation:
- run.bat: source mode now sets up its own venv and runs testium from
src\ directly instead of delegating to the project run.bat (which
launches the GUI and drops its arguments). Installs the fake_exporter
entry-point plugin (report_plugin) and the [lsp] extra, and runs the
same lsp_check.py pre-flight as run.sh.
- jsonrpc/test.tum: launch the echo server via "$(python_bin)" instead
of "python3" (the Microsoft Store stub on Windows).
- post_execution.py: write the JUnit XML with encoding="utf-8".
- restore items/run/sub_pass.tum and sub_fail.tum, deleted by mistake in
d97d00c "removed test logs".
- term console via flatpak-spawn --host so host venvs resolve (bins.host_console_command)
- QSettings sync() before subprocess kill in choices/tested-refs dialogs
- console regression test: fails on the in-sandbox 0.2.1 console
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Manual (modes.rst) and README: install the extension from Open VSX in
VSCodium/Cursor/etc., and as a .vsix by hand in Microsoft VSCode; note
that testium must be on PATH or set via testium.serverPath.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set the app id via setDesktopFileName so the window stops inheriting the
launcher's class ("python3" under the AppImage), which is what GNOME was
keying the wrong icon off. On native Wayland the task-bar icon comes from
an installed desktop file matched to the app id (setWindowIcon is
ignored there), so on Linux drop an idempotent desktop entry + 256px icon
under ~/.local/share. Flatpak keeps its own id/desktop; Windows / macOS
use the window icon. No-op off Linux.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
load_test_recursively expanded nested lists and included 'sequence'
entries by splicing each into the step list and rebuilding the whole
list every time (O(n^2)). The list branch also rebuilt after an in-place
splice, duplicating entries when a nested list held more than one item.
Replace both with a single linear _flatten_actions pass. Build phase
~12% faster at 6k items; the real fix is the duplication (a nested
2-element list now yields a,b,c,d not a,b,c,c,d). Validation suite
identical (post-exec SUCCESS, same verdicts/tracebacks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Base the TUM loaders (and the param-file load) on yaml.CLoader when
PyYAML is built with libyaml, falling back to the pure-Python Loader
otherwise. Same ParserError/ScannerError, same custom !include
constructors. YAML parse time ~8x lower; validation suite identical
(same verdicts, same 8 expected-fail tracebacks, post-exec SUCCESS).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Shared jinja Environment + compiled-template cache keyed on (path, mtime,
size), and render to an in-memory StringIO instead of a temp file.
Behaviour unchanged (validation suite passes); template time -10..40x,
total load -20..30% on template-heavy trees.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generator + in-process harness timing the real loader's three stages and
template/YAML call counts, across tunable profiles. cases/ git-ignored;
see test/benchmark/README.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The wheel channel was installed without [lsp], so 'testium lsp' reported pygls
missing. Install testium[lsp] in the wheel venv — validating the wheel's
language-server-capable form, like the AppImage installs ...whl[lsp].
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
modes.rst: new 'Language server (editor support)' section — testium lsp /
testium schema, the testium_assist client, the [lsp] extra. tum_syntax.rst:
a note on parameter validation (unknown-key warning, missing-required error).
Builds clean (sphinx html, no RST errors).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 'main' merge (bd1cd03) brought a more detailed AppImage entry (with the
libfuse2 requirement); my earlier branch had added a simpler one, leaving two.
Keep main's, drop the duplicate, retain the language-server note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README: add the AppImage release, an 'Editor support' section (testium lsp /
schema, [lsp] extra, the testium_assist client), note the LSP ships in every
channel. DESIGN.md: document build_all parallelism, --ram tmpfs mode and the
Ctrl+C job-tree kill. release_note.txt: 0.2 entries for the language server and
the build_all parallel/--ram work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Previously the reaping loop waited on jobs in array order, so a finished build's
OK/FAILED line was delayed until the loop reached its PID (e.g. appimage done
but unreported while flatpak still ran). Use 'wait -n -p' to print each result
as soon as that build finishes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
flatpak-builder mounts its state dir via rofiles-fuse; FUSE mounts fail on
/dev/shm ('fusermount: Permission denied'). So --ram no longer redirects the
flatpak dirs — it builds on disk as before. PyInstaller workpath, AppImage
AppDir and TMPDIR/PIP_CACHE_DIR still go to tmpfs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
flatpak-builder hardlinks between its state dir and the build dir, so they must
share a filesystem. With only the build dir on tmpfs it errored ('state dir not
on the same filesystem as the target dir'). Move .flatpak-builder to tmpfs as
well via FLATPAK_STATEDIR; its download cache no longer persists across --ram
runs, which is the accepted trade for the tmpfs speedup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trap INT/TERM around the parallel wait recursively kills each job's process
tree (subshell + grandchildren: podman container, flatpak-builder, pyinstaller),
then exits 130 — the EXIT trap frees the tmpfs scratch. Verified: SIGINT leaves
no orphan processes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Redirect the per-channel build scratch to /dev/shm and skip UPX, a big win when
building from a USB stick / SD card (I/O-bound on flash):
- TMPDIR + PIP_CACHE_DIR -> tmpfs
- PyInstaller: --workpath -> tmpfs (PYI_WORKPATH); UPX off via TESTIUM_NO_UPX
- Flatpak: build dir + ostree repo -> tmpfs (FLATPAK_BUILDDIR/REPODIR); the
.flatpak-builder download cache stays on disk
- AppImage: bind-mount a tmpfs dir at the in-container AppDir path
(APPIMAGE_APPDIR_TMPFS)
Scratch is freed on exit. Each build.sh honors the env vars with on-disk
defaults, so behavior is unchanged without --ram. With --ram, prefer --serial
on RAM-limited machines (flatpak+appimage are ~1 GB each).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Serial prep (venv tool installs + flatpak runtimes + wheel, which the AppImage
depends on), then the four heavy builds run concurrently. The shared venv is
only written during prep, so the parallel builds (read-only on the venv) don't
race on pip. Per-step logs under dist/.build-logs/; failing logs are printed.
--serial falls back to one-at-a-time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The unquoted build-command was parsed by YAML as a dict because of the
':all: ' colon-space, so flatpak-builder ran an empty module and pygls was
never installed into the bundle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make `testium lsp` (and the testium_assist editor extension that spawns it)
work from every distribution channel: source, wheel, PyInstaller, Flatpak,
AppImage.
Two enablers:
1. Declarative ACTIONS registry. The TestItemActions parents (console, plot,
json_rpc) now declare their nested actions as a class attribute
`ACTIONS = {yaml_key: class}`, mirroring PARAMS. The base __init__ seeds
action_classes from type(self).ACTIONS; register_actions() is kept only as
an imperative escape hatch. lsp/schema.py reads ACTIONS directly, dropping
the inspect.getsource/AST walk that returned no actions in a frozen
PyInstaller build (no .py source on disk).
2. pygls bundled per channel. Kept as the pyproject [lsp] extra (lean
`pip install testium`), layered into each full-app channel:
- build_env.sh installs pygls into test/tmp/.venv (source run + PyInstaller
build env)
- AppImage installs the wheel as `…whl[lsp]`
- Flatpak adds a python3-lsp network-pip module (matches the manifest's
global --share=network)
- PyInstaller .spec collect_submodules(pygls/lsprotocol) + hiddenimports for
the lazily-imported lsp/lsp.server/lsp.schema
test/validation/lsp_smoke.py (run by run.sh before the suite) enforces both
per channel: `<channel> schema` must keep console/plot/json_rpc actions and
`<channel> lsp` must answer an initialize request without reporting pygls
missing. Verified for source mode; the other channels need a rebuild to verify.
DESIGN.md updated (declarative section + new "Language server across channels"
subsection + Recent fixes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds two new LSP features that share the schema with completion:
- textDocument/hover — when the cursor is on the item-type word of a
step line (`- sleep:`), the server renders the same Markdown doc
used by the completion item, listing required/optional params.
Other words (string values, YAML keys other than item types) don't
trigger the popup.
- textDocument/documentSymbol — the outline view now contains one
entry per step, nested by leading-dash indentation so container
items (group, parallel, cycle, console, plot, json_rpc) display
their children as a subtree. Each symbol's `detail` shows the
YAML `name:` field if present nearby — found via a small forward
scan, no YAML parsing yet.
Action item types (console open/close/…, plot open/close/…, json_rpc
query/receive/…) are accepted by hover and outline too, so the
outline doesn't stop at the parent.
Markdown rendering is now shared by completion and hover via
`_render_item_markdown(cmd, entry)`; both surfaces show the same
description regardless of how the user reached it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds two new testium CLI subcommands:
testium schema dump the JSON-shaped schema of every test item type
(PARAMS merged with the common ones, console / plot /
json_rpc actions nested under their parent's "actions"
block). Zero runtime dependencies — usable by any editor
that speaks the YAML JSON Schema extension to get static
completion straight away.
testium lsp start a pygls LSP server over stdio. First feature:
completion of test item type names when the user starts
a new step (`- |`). The completion item carries the
item's display name and a hover doc listing its required
and optional non-common parameters.
pygls is declared as an optional extra ([project.optional-dependencies]
lsp = ["pygls>=1.3"]) so the core install isn't enlarged for users who
don't need the server. The import compatibility shim picks
pygls.lsp.server.LanguageServer (pygls 2.x) first and falls back to
pygls.server.LanguageServer (pygls 1.x).
The subcommands intercept argv[1] before argparse runs so they don't
share the GUI/batch flag surface.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Brings DESIGN.md in sync with the v0.2 changes:
- new section describing the PARAMS/ParamSet/Param descriptor on every
TestItem subclass and the unknown/missing-param diagnostics;
- rewrites the Flatpak section so it matches the flatpak-spawn --host
pipeline instead of the obsolete LD_LIBRARY_PATH/apply_host_lua_paths
injection;
- documents the --mode flag in the validation suite section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrates the remaining test items to the ParamSet/Param declaration
introduced in d0721af:
- dialogs: image, question, value, choices, tested_references
- actions: check, run, report
- console: parent + open/read_until actions
- py_func / lua_func
- containers: group, parallel + parallel_branch, unittest
- complex: cycle (sub-block exit_condition documented in
EXIT_CONDITION_PARAMS), git
- runtime_plot: parent + open/close/periodic/last_value actions
- json_rpc: parent + query/receive actions
Items intentionally without PARAMS (and therefore not validated) are
those whose body is the unstructured user value: console write/writeln,
plot add/export, and the json_rpc/console open & close actions. Same
for the internally-instantiated TestItemUnittestElement which passes
dict_item=None.
Behavior on valid .tum files is unchanged (validation suite source
mode: SUCCESS). Typos on declared params now surface as warnings
listing the accepted names; missing required params surface as load-
time errors with file context.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
