François
d4889c2a2e
py_func, lua_func and the run item now reach host binaries through `flatpak-spawn --host` instead of trying to load them under the sandbox runtime (which fails with a glibc ABI mismatch). Adds `--talk-name=org.freedesktop.Flatpak` to the manifest, stages the /app/lib/testium tree under /tmp so the host can read it, and drops the dead `_FLATPAK_HOST_DIRS` / lib-injection code paths that the new approach makes obsolete. Validation suite gains a `--mode source|wheel|pyinstaller|flatpak| appimage` flag so the same item set can run against every packaging channel; per-mode report file names avoid clobbering. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Validation
This directory contains the testium validation suite. A single set of
items (items/), fixtures and post-processing (post_execution.py) is
re-used across every packaging channel.
Running the suite
./test/validation/run.sh # default mode = source
./test/validation/run.sh --mode wheel
./test/validation/run.sh --mode pyinstaller
./test/validation/run.sh --mode flatpak
./test/validation/run.sh --mode appimage
On Windows (only source, wheel, pyinstaller are supported):
test\validation\run.bat --mode pyinstaller
Pass clean as the first argument to recreate the validation venv
from scratch (useful after a system Python upgrade):
./test/validation/run.sh clean --mode flatpak
Any extra arguments after the mode flag are forwarded to testium.
Modes
| Mode | What it launches | Prerequisite |
|---|---|---|
source |
python3 src/testium via the project's run.sh |
none — works straight out of the repo |
wheel |
python -m testium inside a dedicated wheel venv |
./build_all.sh produced dist/testium-<v>-py3-none-any.whl |
pyinstaller |
dist/testium-<v> (frozen binary) |
./build_all.sh produced the PyInstaller binary |
flatpak |
flatpak run --command=testium org.testium.Testium |
the Flatpak bundle is installed (flatpak install --user dist/testium-<v>.flatpak) |
appimage |
dist/Testium-<v>-x86_64.AppImage |
./build_all.sh produced the AppImage |
Each mode writes its results to a distinct report file
(validation-<mode>.sqlite / validation-<mode>-<item>.xml), so you
can run several modes in a row without clobbering previous reports.
How python_bin is pinned
Every test-execution subprocess (inline <| ... |> evaluation,
py_func, cycle, post_execution, …) is routed through a dedicated
venv at ${TMPDIR:-/tmp}/testium-validation-venv. The venv is created
with --system-site-packages so existing system packages stay visible,
then junit-xml is pip-installed for post_execution.py.
This is a host venv. In every mode (including Flatpak) the
test-execution subprocesses end up running on the host — directly for
source/wheel/pyinstaller/appimage, and via flatpak-spawn --host for
Flatpak — so the same venv works across modes. The wheel mode
additionally creates a separate testium-wheel-venv-<v> to hold the
installed wheel; that one is only used to launch testium itself.
What is checked
The venv item under items/venv/ asserts that the validation venv is
actually being used:
python_binis set in the global dict.- The eval subprocess (used for
<| ... |>expressions) hassys.executable == python_bin,sys.prefix == dirname(dirname(python_bin)), andsys.prefix != sys.base_prefix(i.e. is actually inside a venv). - A
py_funcsubprocess passes the same three checks.
These checks use abspath/normpath rather than realpath on
purpose: the venv's bin/python3 is a symlink to the host interpreter,
so realpath would map both venv and non-venv interpreters to the same
target. sys.prefix != sys.base_prefix is the venv-specific marker
that distinguishes the two cases.