flatpak: run host interpreters via flatpak-spawn; validation --mode flag
py_func, lua_func and the run item now reach host binaries through `flatpak-spawn --host` instead of trying to load them under the sandbox runtime (which fails with a glibc ABI mismatch). Adds `--talk-name=org.freedesktop.Flatpak` to the manifest, stages the /app/lib/testium tree under /tmp so the host can read it, and drops the dead `_FLATPAK_HOST_DIRS` / lib-injection code paths that the new approach makes obsolete. Validation suite gains a `--mode source|wheel|pyinstaller|flatpak| appimage` flag so the same item set can run against every packaging channel; per-mode report file names avoid clobbering. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
@@ -1,34 +1,67 @@
|
||||
# Validation
|
||||
|
||||
This directory contains the testium validation suite.
|
||||
This directory contains the testium validation suite. A single set of
|
||||
items (`items/`), fixtures and post-processing (`post_execution.py`) is
|
||||
re-used across every packaging channel.
|
||||
|
||||
## Running the suite
|
||||
|
||||
```sh
|
||||
./test/validation/run.sh # Linux
|
||||
test\validation\run.bat # Windows
|
||||
./test/validation/run.sh # default mode = source
|
||||
./test/validation/run.sh --mode wheel
|
||||
./test/validation/run.sh --mode pyinstaller
|
||||
./test/validation/run.sh --mode flatpak
|
||||
./test/validation/run.sh --mode appimage
|
||||
```
|
||||
|
||||
The wrapper creates a dedicated Python venv in the system temp dir
|
||||
(`${TMPDIR:-/tmp}/testium-validation-venv` on Linux, `%TEMP%\testium-validation-venv`
|
||||
on Windows), using `--system-site-packages` so existing system packages
|
||||
stay visible. The validation suite is then run with that venv pinned as
|
||||
`python_bin`. Every test-execution subprocess (inline `<| ... |>`
|
||||
evaluation, `py_func`, `cycle`, `post_execution`, ...) runs inside the
|
||||
venv, while testium itself keeps running in the project's own
|
||||
environment.
|
||||
On Windows (only `source`, `wheel`, `pyinstaller` are supported):
|
||||
|
||||
Pass `clean` as the first argument to recreate the venv from scratch
|
||||
(useful after a system Python upgrade):
|
||||
```bat
|
||||
test\validation\run.bat --mode pyinstaller
|
||||
```
|
||||
|
||||
Pass `clean` as the **first** argument to recreate the validation venv
|
||||
from scratch (useful after a system Python upgrade):
|
||||
|
||||
```sh
|
||||
./test/validation/run.sh clean
|
||||
./test/validation/run.sh clean --mode flatpak
|
||||
```
|
||||
|
||||
Any extra arguments after the mode flag are forwarded to testium.
|
||||
|
||||
## Modes
|
||||
|
||||
| Mode | What it launches | Prerequisite |
|
||||
|---------------|-------------------------------------------------------------|------------------------------------------------------------------|
|
||||
| `source` | `python3 src/testium` via the project's `run.sh` | none — works straight out of the repo |
|
||||
| `wheel` | `python -m testium` inside a dedicated wheel venv | `./build_all.sh` produced `dist/testium-<v>-py3-none-any.whl` |
|
||||
| `pyinstaller` | `dist/testium-<v>` (frozen binary) | `./build_all.sh` produced the PyInstaller binary |
|
||||
| `flatpak` | `flatpak run --command=testium org.testium.Testium` | the Flatpak bundle is installed (`flatpak install --user dist/testium-<v>.flatpak`) |
|
||||
| `appimage` | `dist/Testium-<v>-x86_64.AppImage` | `./build_all.sh` produced the AppImage |
|
||||
|
||||
Each mode writes its results to a distinct report file
|
||||
(`validation-<mode>.sqlite` / `validation-<mode>-<item>.xml`), so you
|
||||
can run several modes in a row without clobbering previous reports.
|
||||
|
||||
## How `python_bin` is pinned
|
||||
|
||||
Every test-execution subprocess (inline `<| ... |>` evaluation,
|
||||
`py_func`, `cycle`, `post_execution`, …) is routed through a dedicated
|
||||
venv at `${TMPDIR:-/tmp}/testium-validation-venv`. The venv is created
|
||||
with `--system-site-packages` so existing system packages stay visible,
|
||||
then `junit-xml` is pip-installed for `post_execution.py`.
|
||||
|
||||
This is a **host** venv. In every mode (including Flatpak) the
|
||||
test-execution subprocesses end up running on the host — directly for
|
||||
source/wheel/pyinstaller/appimage, and via `flatpak-spawn --host` for
|
||||
Flatpak — so the same venv works across modes. The wheel mode
|
||||
additionally creates a separate `testium-wheel-venv-<v>` to hold the
|
||||
installed wheel; that one is only used to launch testium itself.
|
||||
|
||||
## What is checked
|
||||
|
||||
The `venv` item under `items/venv/` asserts that the venv is actually
|
||||
being used:
|
||||
The `venv` item under `items/venv/` asserts that the validation venv is
|
||||
actually being used:
|
||||
|
||||
* `python_bin` is set in the global dict.
|
||||
* The eval subprocess (used for `<| ... |>` expressions) has
|
||||
|
||||
@@ -1,61 +1,131 @@
|
||||
@echo off
|
||||
SETLOCAL EnableExtensions
|
||||
SETLOCAL EnableExtensions EnableDelayedExpansion
|
||||
|
||||
REM Runs the testium validation suite with a dedicated Python venv used
|
||||
REM by every py_func / cycle / inline-eval subprocess. testium itself
|
||||
REM keeps running in the project's own environment; the validation venv
|
||||
REM only isolates *test execution*.
|
||||
REM Runs the testium validation suite against any installable channel of
|
||||
REM testium on Windows (source, wheel, pyinstaller).
|
||||
REM
|
||||
REM test\validation\run.bat [clean] [extra testium args]
|
||||
REM Usage:
|
||||
REM test\validation\run.bat [clean] [--mode MODE] [extra testium args]
|
||||
REM
|
||||
REM Requires the project venv to already exist (run the project's
|
||||
REM run.bat once first, or any other testium install method).
|
||||
REM clean remove the validation venv before recreating it
|
||||
REM (must be the first argument; useful after a Python upgrade)
|
||||
REM
|
||||
REM --mode MODE which testium build to validate. One of:
|
||||
REM source (default) project's run.bat (src\testium)
|
||||
REM wheel dist\testium-<v>-py3-none-any.whl
|
||||
REM pyinstaller dist\testium-<v>.exe (or dist\testium-<v>)
|
||||
REM
|
||||
REM Every test-execution subprocess runs in a dedicated host venv under
|
||||
REM %TEMP%\testium-validation-venv (created with --system-site-packages,
|
||||
REM then junit-xml is pip-installed for post_execution.py).
|
||||
REM
|
||||
REM The report file is suffixed with the mode so consecutive runs in
|
||||
REM different modes don't overwrite each other.
|
||||
|
||||
SET "SCRIPT_DIR=%~dp0"
|
||||
SET "PROJECT_DIR=%SCRIPT_DIR%..\.."
|
||||
REM Venv in the user temp dir (Windows equivalent of /tmp).
|
||||
SET "VENV_DIR=%TEMP%\testium-validation-venv"
|
||||
SET "PROJECT_VENV=%PROJECT_DIR%\test\tmp\testium_venv"
|
||||
IF "%SCRIPT_DIR:~-1%"=="\" SET "SCRIPT_DIR=%SCRIPT_DIR:~0,-1%"
|
||||
SET "PROJECT_DIR=%SCRIPT_DIR%\..\.."
|
||||
SET /P VERSION=<"%PROJECT_DIR%\src\VERSION"
|
||||
|
||||
REM ---------- arg parsing ----------------------------------------------------
|
||||
|
||||
SET "MODE=source"
|
||||
SET "CLEAN=0"
|
||||
IF /I "%~1"=="clean" (
|
||||
rmdir /s /q "%VENV_DIR%"
|
||||
SET "CLEAN=1"
|
||||
SHIFT
|
||||
)
|
||||
|
||||
REM Locate a host Python.
|
||||
SET "PYTHON_EXE=python"
|
||||
SET "EXTRA="
|
||||
:PARSE_ARGS
|
||||
IF "%~1"=="" GOTO ARGS_DONE
|
||||
IF /I "%~1"=="--mode" (
|
||||
SET "MODE=%~2"
|
||||
SHIFT
|
||||
SHIFT
|
||||
GOTO PARSE_ARGS
|
||||
)
|
||||
SET "EXTRA=!EXTRA! "%~1""
|
||||
SHIFT
|
||||
GOTO PARSE_ARGS
|
||||
:ARGS_DONE
|
||||
|
||||
REM ---------- locate host python ---------------------------------------------
|
||||
|
||||
SET "PYTHON_EXE="
|
||||
py --version >nul 2>&1
|
||||
IF %ERRORLEVEL% EQU 0 (
|
||||
SET "PYTHON_EXE=py"
|
||||
goto :PYTHON_FOUND
|
||||
GOTO PYTHON_FOUND
|
||||
)
|
||||
python --version >nul 2>&1
|
||||
IF %ERRORLEVEL% EQU 0 (
|
||||
SET "PYTHON_EXE=python"
|
||||
goto :PYTHON_FOUND
|
||||
GOTO PYTHON_FOUND
|
||||
)
|
||||
echo ERROR : Python could not be found on this system.
|
||||
echo ERROR: Python could not be found on this system.
|
||||
exit /b 1
|
||||
|
||||
:PYTHON_FOUND
|
||||
|
||||
REM ---------- validation venv -------------------------------------------------
|
||||
|
||||
SET "VENV_DIR=%TEMP%\testium-validation-venv"
|
||||
IF "%CLEAN%"=="1" IF EXIST "%VENV_DIR%" rmdir /s /q "%VENV_DIR%"
|
||||
|
||||
IF NOT EXIST "%VENV_DIR%" (
|
||||
echo Creating validation venv at %VENV_DIR%
|
||||
%PYTHON_EXE% -m venv --system-site-packages "%VENV_DIR%"
|
||||
IF %ERRORLEVEL% NEQ 0 (
|
||||
IF !ERRORLEVEL! NEQ 0 (
|
||||
echo ERROR while creating the validation venv.
|
||||
exit /b 1
|
||||
)
|
||||
call "%VENV_DIR%\Scripts\pip" install --quiet --upgrade pip
|
||||
call "%VENV_DIR%\Scripts\pip" install --quiet junit-xml
|
||||
)
|
||||
|
||||
SET "VENV_PYTHON=%VENV_DIR%\Scripts\python.exe"
|
||||
|
||||
IF NOT EXIST "%PROJECT_VENV%" (
|
||||
echo ERROR : project venv not found at %PROJECT_VENV%. Run the project run.bat once first.
|
||||
REM ---------- shared "tail" forwarded to every launcher -----------------------
|
||||
REM Reports are stamped with the mode so successive runs don't clobber each other.
|
||||
|
||||
SET "TAIL=-b -d "python_bin=%VENV_PYTHON%" -d "validation_report_file=validation-%MODE%" -- "%SCRIPT_DIR%\main.tum"%EXTRA%"
|
||||
|
||||
REM ---------- per-mode launcher ----------------------------------------------
|
||||
|
||||
echo -- validation mode: %MODE%
|
||||
|
||||
IF /I "%MODE%"=="source" GOTO MODE_SOURCE
|
||||
IF /I "%MODE%"=="wheel" GOTO MODE_WHEEL
|
||||
IF /I "%MODE%"=="pyinstaller" GOTO MODE_PYI
|
||||
echo ERROR: unknown --mode '%MODE%'. Expected: source ^| wheel ^| pyinstaller.
|
||||
exit /b 1
|
||||
|
||||
:MODE_SOURCE
|
||||
call "%PROJECT_DIR%\run.bat" %TAIL%
|
||||
exit /b %ERRORLEVEL%
|
||||
|
||||
:MODE_WHEEL
|
||||
SET "WHEEL=%PROJECT_DIR%\dist\testium-%VERSION%-py3-none-any.whl"
|
||||
IF NOT EXIST "%WHEEL%" (
|
||||
echo ERROR: wheel not found at %WHEEL% -- run build_all.sh first.
|
||||
exit /b 1
|
||||
)
|
||||
SET "WHEEL_VENV=%TEMP%\testium-wheel-venv-%VERSION%"
|
||||
IF "%CLEAN%"=="1" IF EXIST "%WHEEL_VENV%" rmdir /s /q "%WHEEL_VENV%"
|
||||
IF NOT EXIST "%WHEEL_VENV%" (
|
||||
echo Creating wheel venv at %WHEEL_VENV%
|
||||
%PYTHON_EXE% -m venv --system-site-packages "%WHEEL_VENV%"
|
||||
call "%WHEEL_VENV%\Scripts\pip" install --quiet --upgrade pip
|
||||
call "%WHEEL_VENV%\Scripts\pip" install --quiet "%WHEEL%"
|
||||
)
|
||||
"%WHEEL_VENV%\Scripts\python.exe" -m testium %TAIL%
|
||||
exit /b %ERRORLEVEL%
|
||||
|
||||
call "%PROJECT_VENV%\Scripts\activate"
|
||||
python "%PROJECT_DIR%\src\testium" -b -d "python_bin=%VENV_PYTHON%" -- "%SCRIPT_DIR%main.tum" %*
|
||||
:MODE_PYI
|
||||
SET "PYI_BIN=%PROJECT_DIR%\dist\testium-%VERSION%.exe"
|
||||
IF NOT EXIST "%PYI_BIN%" SET "PYI_BIN=%PROJECT_DIR%\dist\testium-%VERSION%"
|
||||
IF NOT EXIST "%PYI_BIN%" (
|
||||
echo ERROR: PyInstaller binary not found in %PROJECT_DIR%\dist -- run build_all.sh first.
|
||||
exit /b 1
|
||||
)
|
||||
"%PYI_BIN%" %TAIL%
|
||||
exit /b %ERRORLEVEL%
|
||||
|
||||
@@ -1,47 +1,143 @@
|
||||
#!/bin/bash
|
||||
# Runs the testium validation suite with a dedicated Python venv used by
|
||||
# every py_func / cycle / inline-eval subprocess (i.e. everything that
|
||||
# goes through ``bins.python_bin()``). testium itself keeps running in
|
||||
# the project's own environment — the validation venv only isolates
|
||||
# *test execution*.
|
||||
# Runs the testium validation suite against any installable channel of
|
||||
# testium (source, wheel, pyinstaller, flatpak, appimage).
|
||||
#
|
||||
# ./test/validation/run.sh [clean] [extra testium args]
|
||||
# Usage:
|
||||
# ./test/validation/run.sh [clean] [--mode MODE] [extra testium args]
|
||||
#
|
||||
# ``clean`` (optional, must be the first arg) removes the venv before
|
||||
# recreating it; this is the way to refresh the venv after a system
|
||||
# Python upgrade.
|
||||
# clean remove the validation venv before recreating it
|
||||
# (must be the first argument; useful after a Python upgrade)
|
||||
#
|
||||
# --mode MODE which testium build to validate. One of:
|
||||
# source (default) src/testium via project run.sh
|
||||
# wheel dist/testium-<v>-py3-none-any.whl
|
||||
# pyinstaller dist/testium-<v>
|
||||
# flatpak installed org.testium.Testium
|
||||
# appimage dist/Testium-<v>-*.AppImage
|
||||
#
|
||||
# Every test-execution subprocess (inline <| ... |>, py_func, cycle,
|
||||
# post_execution, ...) runs in a dedicated host venv under
|
||||
# /tmp/testium-validation-venv. That venv is shared across modes —
|
||||
# even Flatpak reaches it via flatpak-spawn --host. The validation venv
|
||||
# is created with --system-site-packages so existing system packages
|
||||
# (PySide6, lxml, ...) stay visible, then junit-xml is pip-installed
|
||||
# for post_execution.py.
|
||||
#
|
||||
# The report file is suffixed with the mode (e.g. validation-flatpak.sqlite)
|
||||
# so consecutive runs in different modes don't overwrite each other.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_PATH="$(readlink -f "$0")"
|
||||
SCRIPT_DIR="$(realpath "$(dirname "$SCRIPT_PATH")")"
|
||||
PROJECT_DIR="$(realpath "$SCRIPT_DIR/../..")"
|
||||
# Venv lives in the system temp dir so it stays out of the project tree
|
||||
# (and is naturally cleaned up by tmpfiles/reboot on most distros).
|
||||
VENV_DIR="${TMPDIR:-/tmp}/testium-validation-venv"
|
||||
VERSION="$(cat "$PROJECT_DIR/src/VERSION")"
|
||||
|
||||
# ---------- arg parsing -------------------------------------------------------
|
||||
|
||||
MODE=source
|
||||
|
||||
if [ "${1:-}" = "clean" ]; then
|
||||
rm -rf "$VENV_DIR"
|
||||
CLEAN=1
|
||||
shift
|
||||
else
|
||||
CLEAN=0
|
||||
fi
|
||||
|
||||
EXTRA=()
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--mode)
|
||||
MODE="$2"
|
||||
shift 2
|
||||
;;
|
||||
--mode=*)
|
||||
MODE="${1#--mode=}"
|
||||
shift
|
||||
;;
|
||||
*)
|
||||
EXTRA+=("$1")
|
||||
shift
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ---------- validation venv ---------------------------------------------------
|
||||
|
||||
VENV_DIR="${TMPDIR:-/tmp}/testium-validation-venv"
|
||||
if [ "$CLEAN" -eq 1 ]; then
|
||||
rm -rf "$VENV_DIR"
|
||||
fi
|
||||
|
||||
if [ ! -d "$VENV_DIR" ]; then
|
||||
echo "Creating validation venv at $VENV_DIR"
|
||||
# --system-site-packages so we don't have to reinstall pyside6, lxml
|
||||
# & friends just to support the validation helpers. We still pip
|
||||
# install junit-xml below because it is the one dep that does *not*
|
||||
# ship as a system package on most distros and is required by
|
||||
# post_execution.py.
|
||||
python3 -m venv --system-site-packages "$VENV_DIR"
|
||||
"$VENV_DIR/bin/pip" install --quiet --upgrade pip
|
||||
"$VENV_DIR/bin/pip" install --quiet junit-xml
|
||||
fi
|
||||
|
||||
VENV_PYTHON="$VENV_DIR/bin/python3"
|
||||
|
||||
# Delegate to the project's run.sh so testium itself still runs in the
|
||||
# project venv (with pyside6, gitpython, ...). ``-d python_bin=...``
|
||||
# pins every test-execution subprocess to the validation venv.
|
||||
exec "$PROJECT_DIR/run.sh" -b \
|
||||
# ---------- per-mode launcher -------------------------------------------------
|
||||
|
||||
case "$MODE" in
|
||||
source)
|
||||
CMD=("$PROJECT_DIR/run.sh")
|
||||
;;
|
||||
wheel)
|
||||
WHEEL="$PROJECT_DIR/dist/testium-${VERSION}-py3-none-any.whl"
|
||||
if [ ! -f "$WHEEL" ]; then
|
||||
echo "ERROR: wheel not found at $WHEEL — run ./build_all.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
WHEEL_VENV="${TMPDIR:-/tmp}/testium-wheel-venv-${VERSION}"
|
||||
if [ "$CLEAN" -eq 1 ]; then
|
||||
rm -rf "$WHEEL_VENV"
|
||||
fi
|
||||
if [ ! -d "$WHEEL_VENV" ]; then
|
||||
echo "Creating wheel venv at $WHEEL_VENV"
|
||||
python3 -m venv --system-site-packages "$WHEEL_VENV"
|
||||
"$WHEEL_VENV/bin/pip" install --quiet --upgrade pip
|
||||
"$WHEEL_VENV/bin/pip" install --quiet "$WHEEL"
|
||||
fi
|
||||
CMD=("$WHEEL_VENV/bin/python" -m testium)
|
||||
;;
|
||||
pyinstaller)
|
||||
PYI_BIN="$PROJECT_DIR/dist/testium-${VERSION}"
|
||||
if [ ! -x "$PYI_BIN" ]; then
|
||||
echo "ERROR: PyInstaller binary not found at $PYI_BIN — run ./build_all.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
CMD=("$PYI_BIN")
|
||||
;;
|
||||
flatpak)
|
||||
if ! flatpak info --user org.testium.Testium &>/dev/null \
|
||||
&& ! flatpak info --system org.testium.Testium &>/dev/null; then
|
||||
echo "ERROR: org.testium.Testium is not installed." >&2
|
||||
echo " flatpak install --user $PROJECT_DIR/dist/testium-${VERSION}.flatpak" >&2
|
||||
exit 1
|
||||
fi
|
||||
CMD=(flatpak run --command=testium org.testium.Testium)
|
||||
;;
|
||||
appimage)
|
||||
APPIMAGE=$(ls -1t "$PROJECT_DIR/dist"/Testium-"${VERSION}"-*.AppImage 2>/dev/null | head -1)
|
||||
if [ -z "$APPIMAGE" ] || [ ! -x "$APPIMAGE" ]; then
|
||||
echo "ERROR: no AppImage for version $VERSION under $PROJECT_DIR/dist — run ./build_all.sh first." >&2
|
||||
exit 1
|
||||
fi
|
||||
CMD=("$APPIMAGE")
|
||||
;;
|
||||
*)
|
||||
echo "ERROR: unknown --mode '$MODE'. Expected: source|wheel|pyinstaller|flatpak|appimage." >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
# ---------- launch ------------------------------------------------------------
|
||||
|
||||
echo "-- validation mode: $MODE"
|
||||
echo "-- launch: ${CMD[*]}"
|
||||
|
||||
exec "${CMD[@]}" -b \
|
||||
-d "python_bin=$VENV_PYTHON" \
|
||||
-- "$SCRIPT_DIR/main.tum" "$@"
|
||||
-d "validation_report_file=validation-$MODE" \
|
||||
-- "$SCRIPT_DIR/main.tum" "${EXTRA[@]}"
|
||||
|
||||
Reference in New Issue
Block a user