Make the suite run cleanly on Windows.
Product code:
- __init__.py: force UTF-8 on stdout/stderr. The Windows console code
page (cp1252) cannot encode the box-drawing/accented characters the
runner prints, which crashed the parent capture_stdout thread. Only
the stream encoders are reconfigured; the locale default used to read
cp1252 config files is left untouched.
- report_export_junit/html: open the report file with encoding="utf-8"
(XML/HTML are UTF-8) instead of the platform default, matching the
txt/json exporters.
Validation:
- run.bat: source mode now sets up its own venv and runs testium from
src\ directly instead of delegating to the project run.bat (which
launches the GUI and drops its arguments). Installs the fake_exporter
entry-point plugin (report_plugin) and the [lsp] extra, and runs the
same lsp_check.py pre-flight as run.sh.
- jsonrpc/test.tum: launch the echo server via "$(python_bin)" instead
of "python3" (the Microsoft Store stub on Windows).
- post_execution.py: write the JUnit XML with encoding="utf-8".
- restore items/run/sub_pass.tum and sub_fail.tum, deleted by mistake in
d97d00c "removed test logs".
- term console via flatpak-spawn --host so host venvs resolve (bins.host_console_command)
- QSettings sync() before subprocess kill in choices/tested-refs dialogs
- console regression test: fails on the in-sandbox 0.2.1 console
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Manual (modes.rst) and README: install the extension from Open VSX in
VSCodium/Cursor/etc., and as a .vsix by hand in Microsoft VSCode; note
that testium must be on PATH or set via testium.serverPath.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Set the app id via setDesktopFileName so the window stops inheriting the
launcher's class ("python3" under the AppImage), which is what GNOME was
keying the wrong icon off. On native Wayland the task-bar icon comes from
an installed desktop file matched to the app id (setWindowIcon is
ignored there), so on Linux drop an idempotent desktop entry + 256px icon
under ~/.local/share. Flatpak keeps its own id/desktop; Windows / macOS
use the window icon. No-op off Linux.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
load_test_recursively expanded nested lists and included 'sequence'
entries by splicing each into the step list and rebuilding the whole
list every time (O(n^2)). The list branch also rebuilt after an in-place
splice, duplicating entries when a nested list held more than one item.
Replace both with a single linear _flatten_actions pass. Build phase
~12% faster at 6k items; the real fix is the duplication (a nested
2-element list now yields a,b,c,d not a,b,c,c,d). Validation suite
identical (post-exec SUCCESS, same verdicts/tracebacks).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Base the TUM loaders (and the param-file load) on yaml.CLoader when
PyYAML is built with libyaml, falling back to the pure-Python Loader
otherwise. Same ParserError/ScannerError, same custom !include
constructors. YAML parse time ~8x lower; validation suite identical
(same verdicts, same 8 expected-fail tracebacks, post-exec SUCCESS).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Shared jinja Environment + compiled-template cache keyed on (path, mtime,
size), and render to an in-memory StringIO instead of a temp file.
Behaviour unchanged (validation suite passes); template time -10..40x,
total load -20..30% on template-heavy trees.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Generator + in-process harness timing the real loader's three stages and
template/YAML call counts, across tunable profiles. cases/ git-ignored;
see test/benchmark/README.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The wheel channel was installed without [lsp], so 'testium lsp' reported pygls
missing. Install testium[lsp] in the wheel venv — validating the wheel's
language-server-capable form, like the AppImage installs ...whl[lsp].
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
modes.rst: new 'Language server (editor support)' section — testium lsp /
testium schema, the testium_assist client, the [lsp] extra. tum_syntax.rst:
a note on parameter validation (unknown-key warning, missing-required error).
Builds clean (sphinx html, no RST errors).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The 'main' merge (bd1cd03) brought a more detailed AppImage entry (with the
libfuse2 requirement); my earlier branch had added a simpler one, leaving two.
Keep main's, drop the duplicate, retain the language-server note.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
README: add the AppImage release, an 'Editor support' section (testium lsp /
schema, [lsp] extra, the testium_assist client), note the LSP ships in every
channel. DESIGN.md: document build_all parallelism, --ram tmpfs mode and the
Ctrl+C job-tree kill. release_note.txt: 0.2 entries for the language server and
the build_all parallel/--ram work.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Previously the reaping loop waited on jobs in array order, so a finished build's
OK/FAILED line was delayed until the loop reached its PID (e.g. appimage done
but unreported while flatpak still ran). Use 'wait -n -p' to print each result
as soon as that build finishes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
flatpak-builder mounts its state dir via rofiles-fuse; FUSE mounts fail on
/dev/shm ('fusermount: Permission denied'). So --ram no longer redirects the
flatpak dirs — it builds on disk as before. PyInstaller workpath, AppImage
AppDir and TMPDIR/PIP_CACHE_DIR still go to tmpfs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
flatpak-builder hardlinks between its state dir and the build dir, so they must
share a filesystem. With only the build dir on tmpfs it errored ('state dir not
on the same filesystem as the target dir'). Move .flatpak-builder to tmpfs as
well via FLATPAK_STATEDIR; its download cache no longer persists across --ram
runs, which is the accepted trade for the tmpfs speedup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trap INT/TERM around the parallel wait recursively kills each job's process
tree (subshell + grandchildren: podman container, flatpak-builder, pyinstaller),
then exits 130 — the EXIT trap frees the tmpfs scratch. Verified: SIGINT leaves
no orphan processes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Redirect the per-channel build scratch to /dev/shm and skip UPX, a big win when
building from a USB stick / SD card (I/O-bound on flash):
- TMPDIR + PIP_CACHE_DIR -> tmpfs
- PyInstaller: --workpath -> tmpfs (PYI_WORKPATH); UPX off via TESTIUM_NO_UPX
- Flatpak: build dir + ostree repo -> tmpfs (FLATPAK_BUILDDIR/REPODIR); the
.flatpak-builder download cache stays on disk
- AppImage: bind-mount a tmpfs dir at the in-container AppDir path
(APPIMAGE_APPDIR_TMPFS)
Scratch is freed on exit. Each build.sh honors the env vars with on-disk
defaults, so behavior is unchanged without --ram. With --ram, prefer --serial
on RAM-limited machines (flatpak+appimage are ~1 GB each).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Serial prep (venv tool installs + flatpak runtimes + wheel, which the AppImage
depends on), then the four heavy builds run concurrently. The shared venv is
only written during prep, so the parallel builds (read-only on the venv) don't
race on pip. Per-step logs under dist/.build-logs/; failing logs are printed.
--serial falls back to one-at-a-time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The unquoted build-command was parsed by YAML as a dict because of the
':all: ' colon-space, so flatpak-builder ran an empty module and pygls was
never installed into the bundle.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Make `testium lsp` (and the testium_assist editor extension that spawns it)
work from every distribution channel: source, wheel, PyInstaller, Flatpak,
AppImage.
Two enablers:
1. Declarative ACTIONS registry. The TestItemActions parents (console, plot,
json_rpc) now declare their nested actions as a class attribute
`ACTIONS = {yaml_key: class}`, mirroring PARAMS. The base __init__ seeds
action_classes from type(self).ACTIONS; register_actions() is kept only as
an imperative escape hatch. lsp/schema.py reads ACTIONS directly, dropping
the inspect.getsource/AST walk that returned no actions in a frozen
PyInstaller build (no .py source on disk).
2. pygls bundled per channel. Kept as the pyproject [lsp] extra (lean
`pip install testium`), layered into each full-app channel:
- build_env.sh installs pygls into test/tmp/.venv (source run + PyInstaller
build env)
- AppImage installs the wheel as `…whl[lsp]`
- Flatpak adds a python3-lsp network-pip module (matches the manifest's
global --share=network)
- PyInstaller .spec collect_submodules(pygls/lsprotocol) + hiddenimports for
the lazily-imported lsp/lsp.server/lsp.schema
test/validation/lsp_smoke.py (run by run.sh before the suite) enforces both
per channel: `<channel> schema` must keep console/plot/json_rpc actions and
`<channel> lsp` must answer an initialize request without reporting pygls
missing. Verified for source mode; the other channels need a rebuild to verify.
DESIGN.md updated (declarative section + new "Language server across channels"
subsection + Recent fixes).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adds two new LSP features that share the schema with completion:
- textDocument/hover — when the cursor is on the item-type word of a
step line (`- sleep:`), the server renders the same Markdown doc
used by the completion item, listing required/optional params.
Other words (string values, YAML keys other than item types) don't
trigger the popup.
- textDocument/documentSymbol — the outline view now contains one
entry per step, nested by leading-dash indentation so container
items (group, parallel, cycle, console, plot, json_rpc) display
their children as a subtree. Each symbol's `detail` shows the
YAML `name:` field if present nearby — found via a small forward
scan, no YAML parsing yet.
Action item types (console open/close/…, plot open/close/…, json_rpc
query/receive/…) are accepted by hover and outline too, so the
outline doesn't stop at the parent.
Markdown rendering is now shared by completion and hover via
`_render_item_markdown(cmd, entry)`; both surfaces show the same
description regardless of how the user reached it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds two new testium CLI subcommands:
testium schema dump the JSON-shaped schema of every test item type
(PARAMS merged with the common ones, console / plot /
json_rpc actions nested under their parent's "actions"
block). Zero runtime dependencies — usable by any editor
that speaks the YAML JSON Schema extension to get static
completion straight away.
testium lsp start a pygls LSP server over stdio. First feature:
completion of test item type names when the user starts
a new step (`- |`). The completion item carries the
item's display name and a hover doc listing its required
and optional non-common parameters.
pygls is declared as an optional extra ([project.optional-dependencies]
lsp = ["pygls>=1.3"]) so the core install isn't enlarged for users who
don't need the server. The import compatibility shim picks
pygls.lsp.server.LanguageServer (pygls 2.x) first and falls back to
pygls.server.LanguageServer (pygls 1.x).
The subcommands intercept argv[1] before argparse runs so they don't
share the GUI/batch flag surface.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Brings DESIGN.md in sync with the v0.2 changes:
- new section describing the PARAMS/ParamSet/Param descriptor on every
TestItem subclass and the unknown/missing-param diagnostics;
- rewrites the Flatpak section so it matches the flatpak-spawn --host
pipeline instead of the obsolete LD_LIBRARY_PATH/apply_host_lua_paths
injection;
- documents the --mode flag in the validation suite section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Migrates the remaining test items to the ParamSet/Param declaration
introduced in d0721af:
- dialogs: image, question, value, choices, tested_references
- actions: check, run, report
- console: parent + open/read_until actions
- py_func / lua_func
- containers: group, parallel + parallel_branch, unittest
- complex: cycle (sub-block exit_condition documented in
EXIT_CONDITION_PARAMS), git
- runtime_plot: parent + open/close/periodic/last_value actions
- json_rpc: parent + query/receive actions
Items intentionally without PARAMS (and therefore not validated) are
those whose body is the unstructured user value: console write/writeln,
plot add/export, and the json_rpc/console open & close actions. Same
for the internally-instantiated TestItemUnittestElement which passes
dict_item=None.
Behavior on valid .tum files is unchanged (validation suite source
mode: SUCCESS). Typos on declared params now surface as warnings
listing the accepted names; missing required params surface as load-
time errors with file context.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds utils/param_decl.py with Param/ParamSet/kind descriptors. TestItem
declares COMMON_PARAMS (name, doc, condition, key, skipped, …) and a
new _validate_declared_params() method that warns on unknown keys and
errors on missing required ones — opt-in per subclass (skipped while
PARAMS is None to keep the migration incremental).
Migrates sleep, let, msg_dialog, note_dialog as pilots. Behavior is
unchanged for any well-formed .tum; typos like 'timeoot' on a sleep
item now produce a clear WARN listing the accepted parameters.
The descriptor intentionally carries no Python type information —
parameter values that are $(…) / <|…|> expressions only acquire their
effective type after expansion, so a static type would be misleading.
Per-param post-expansion validators stay opt-in via validate=lambda.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The Pre-built releases section had no AppImage entry. Add one with
the per-distro libfuse2 package names and an APPIMAGE_EXTRACT_AND_RUN
escape hatch for hosts that no longer ship FUSE 2 by default.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Inside a Flatpak the host /usr/bin/git is reachable at
/run/host/usr/bin/git but linked against host glibc/zlib, which the
sandbox cannot load (libz-ng.so.2 missing). gitpython resolves git
eagerly on import and crashed the whole validation run.
Install a tiny shell wrapper under /tmp at module load
(``exec flatpak-spawn --host git "$@"``) and point gitpython at it via
GIT_PYTHON_GIT_EXECUTABLE so test_version / test_modifs work in
flatpak mode. No-op outside Flatpak.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
py_func, lua_func and the run item now reach host binaries through
`flatpak-spawn --host` instead of trying to load them under the
sandbox runtime (which fails with a glibc ABI mismatch). Adds
`--talk-name=org.freedesktop.Flatpak` to the manifest, stages the
/app/lib/testium tree under /tmp so the host can read it, and drops
the dead `_FLATPAK_HOST_DIRS` / lib-injection code paths that the
new approach makes obsolete.
Validation suite gains a `--mode source|wheel|pyinstaller|flatpak|
appimage` flag so the same item set can run against every packaging
channel; per-mode report file names avoid clobbering.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Skip steps whose dist/ artifact already exists; add --clean/-c to
force a full rebuild. Install sphinx/linuxdoc, build, pyinstaller
in their respective steps instead of upfront. Auto-add flathub
remote and install missing Flatpak SDK/runtime deps before step 4.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The v0.1.2 fix that forced Qt's non-native dialog for the "open test"
dialog only covered one call site. The same XDG-portal-vs-sibling-files
problem applies to every other QFileDialog in the GUI (save report,
log file path, default report/log dirs in preferences, python/lua
interpreter pickers).
Extracted a single ``file_dialog.options()`` helper in main_win/ and
threaded it through every getOpenFileName / getSaveFileName /
getExistingDirectory call in main_win/. Outside Flatpak the helper
returns an empty Options(), so the native dialog stays in use on
KDE / GNOME / Windows / macOS.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
eval_proc was started before -d/GUI defines reached gd, so
``-d python_bin=...`` and the GUI ``python_bin`` preference were
silently ignored by the very subprocess that runs ``<| ... |>`` evals
(and only took effect for later items once the discovery cache had
already been seeded with the system interpreter). apply_overrides() is
now applied before eval_process_init(), and bins._resolve()'s cache is
keyed by (name, override) so a later param.yaml change re-resolves on
the next lookup.
The validation suite now ships a wrapper (run.sh / run.bat) that
creates a dedicated venv in the system temp dir and pins it via
``-d python_bin=...``. A new ``venv`` item asserts the override took
effect for both eval_proc and py_func paths, with a
``sys.prefix != sys.base_prefix`` marker to catch the case where the
override happens to be a system interpreter (path-equality alone would
miss it, the venv's ``bin/python3`` being a symlink to the host).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Two new steps per language: function returning nothing and function
returning explicit nil/None. Both tagged $(test)_PASS — they would
have failed before the lua nil fix (Lua side reported nil result as
error). Python side already worked but is covered for parity.
_handle_request was using the 1st pcall return as the error
discriminator, so any Lua function returning nothing (e.g. long_wait
in the example) was reported as failed. Discriminate on the 2nd
return (err) instead, and encode nil result as cjson.null so the
returned_value field stays present in the JSON-RPC response.
console.read_until polls a should_stop callback in 0.2s chunks across
all protocols. py_func/lua_func override stop() to tear down the worker
and wake the parent RPC wait. json_rpc adapters honor should_stop too.
Engaged leaf steps now report FAILURE on stop (sleep no-dialog was
silently SUCCESS).
