validation: dedicated venv + fix python_bin override timing
eval_proc was started before -d/GUI defines reached gd, so ``-d python_bin=...`` and the GUI ``python_bin`` preference were silently ignored by the very subprocess that runs ``<| ... |>`` evals (and only took effect for later items once the discovery cache had already been seeded with the system interpreter). apply_overrides() is now applied before eval_process_init(), and bins._resolve()'s cache is keyed by (name, override) so a later param.yaml change re-resolves on the next lookup. The validation suite now ships a wrapper (run.sh / run.bat) that creates a dedicated venv in the system temp dir and pins it via ``-d python_bin=...``. A new ``venv`` item asserts the override took effect for both eval_proc and py_func paths, with a ``sys.prefix != sys.base_prefix`` marker to catch the case where the override happens to be a system interpreter (path-equality alone would miss it, the venv's ``bin/python3`` being a symlink to the host). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
@@ -1,10 +1,43 @@
|
||||
# Validation
|
||||
|
||||
This directory contains the necessary material to run the testium validation.
|
||||
This directory contains the testium validation suite.
|
||||
|
||||
Here is the documentation on how to configure the validation, run it and check that the
|
||||
results are correct.
|
||||
## Running the suite
|
||||
|
||||
# Tests
|
||||
```sh
|
||||
./test/validation/run.sh # Linux
|
||||
test\validation\run.bat # Windows
|
||||
```
|
||||
|
||||
TBD
|
||||
The wrapper creates a dedicated Python venv in the system temp dir
|
||||
(`${TMPDIR:-/tmp}/testium-validation-venv` on Linux, `%TEMP%\testium-validation-venv`
|
||||
on Windows), using `--system-site-packages` so existing system packages
|
||||
stay visible. The validation suite is then run with that venv pinned as
|
||||
`python_bin`. Every test-execution subprocess (inline `<| ... |>`
|
||||
evaluation, `py_func`, `cycle`, `post_execution`, ...) runs inside the
|
||||
venv, while testium itself keeps running in the project's own
|
||||
environment.
|
||||
|
||||
Pass `clean` as the first argument to recreate the venv from scratch
|
||||
(useful after a system Python upgrade):
|
||||
|
||||
```sh
|
||||
./test/validation/run.sh clean
|
||||
```
|
||||
|
||||
## What is checked
|
||||
|
||||
The `venv` item under `items/venv/` asserts that the venv is actually
|
||||
being used:
|
||||
|
||||
* `python_bin` is set in the global dict.
|
||||
* The eval subprocess (used for `<| ... |>` expressions) has
|
||||
`sys.executable == python_bin`, `sys.prefix == dirname(dirname(python_bin))`,
|
||||
and `sys.prefix != sys.base_prefix` (i.e. is actually inside a venv).
|
||||
* A `py_func` subprocess passes the same three checks.
|
||||
|
||||
These checks use `abspath`/`normpath` rather than `realpath` on
|
||||
purpose: the venv's `bin/python3` is a symlink to the host interpreter,
|
||||
so `realpath` would map both venv and non-venv interpreters to the same
|
||||
target. `sys.prefix != sys.base_prefix` is the venv-specific marker
|
||||
that distinguishes the two cases.
|
||||
|
||||
1
test/validation/items/venv/param.yaml
Normal file
1
test/validation/items/venv/param.yaml
Normal file
@@ -0,0 +1 @@
|
||||
no_param: Null
|
||||
53
test/validation/items/venv/test.tum
Normal file
53
test/validation/items/venv/test.tum
Normal file
@@ -0,0 +1,53 @@
|
||||
# venv test: assert that the dedicated validation venv is the python
|
||||
# being used for every test-execution subprocess (eval_proc / py_func /
|
||||
# cycle / ...). The venv path is pinned by ``-d python_bin=...`` in
|
||||
# test/validation/run.sh (or run.bat).
|
||||
#
|
||||
# We use ``abspath``/``normpath`` rather than ``realpath`` on purpose:
|
||||
# the venv's ``bin/python3`` is a symlink to the host python, so
|
||||
# realpath would map every venv interpreter to the same system path and
|
||||
# the comparison would silently pass even *without* a venv.
|
||||
# ``sys.prefix != sys.base_prefix`` is the venv-specific marker that
|
||||
# catches that case.
|
||||
|
||||
- check:
|
||||
name: python_bin is set in the global dict
|
||||
key: $(test)_PASS
|
||||
values:
|
||||
- <| bool(r"$(python_bin)") |>
|
||||
|
||||
- check:
|
||||
name: eval_proc subprocess runs in the validation venv
|
||||
key: $(test)_PASS
|
||||
values:
|
||||
- <| os.path.normpath(os.path.abspath(sys.executable)) == os.path.normpath(os.path.abspath(r"$(python_bin)")) |>
|
||||
|
||||
- check:
|
||||
name: eval_proc sys.prefix matches python_bin venv root
|
||||
key: $(test)_PASS
|
||||
values:
|
||||
- <| os.path.normpath(os.path.abspath(sys.prefix)) == os.path.dirname(os.path.dirname(os.path.normpath(os.path.abspath(r"$(python_bin)")))) |>
|
||||
|
||||
- check:
|
||||
name: eval_proc is actually inside a venv (sys.prefix != sys.base_prefix)
|
||||
key: $(test)_PASS
|
||||
values:
|
||||
- <| os.path.normpath(os.path.abspath(sys.prefix)) != os.path.normpath(os.path.abspath(sys.base_prefix)) |>
|
||||
|
||||
- py_func:
|
||||
name: py_func subprocess runs in the validation venv
|
||||
key: $(test)_PASS
|
||||
file: $(test_path)$(psep)verify_venv.py
|
||||
func_name: check_sys_executable
|
||||
|
||||
- py_func:
|
||||
name: py_func sys.prefix matches python_bin venv root
|
||||
key: $(test)_PASS
|
||||
file: $(test_path)$(psep)verify_venv.py
|
||||
func_name: check_sys_prefix_in_venv
|
||||
|
||||
- py_func:
|
||||
name: py_func is actually inside a venv
|
||||
key: $(test)_PASS
|
||||
file: $(test_path)$(psep)verify_venv.py
|
||||
func_name: check_is_venv
|
||||
62
test/validation/items/venv/verify_venv.py
Normal file
62
test/validation/items/venv/verify_venv.py
Normal file
@@ -0,0 +1,62 @@
|
||||
import os
|
||||
import sys
|
||||
|
||||
import py_func.tm as tm
|
||||
|
||||
|
||||
def _norm(p):
|
||||
# normpath + normcase, without resolving symlinks. realpath() would
|
||||
# follow the venv's ``python3`` symlink to ``/usr/bin/python3.X`` and
|
||||
# defeat the comparison.
|
||||
return os.path.normcase(os.path.normpath(os.path.abspath(p)))
|
||||
|
||||
|
||||
def _venv_dir():
|
||||
# python_bin is at ``<venv>/(bin|Scripts)/python*`` so the venv root
|
||||
# is two levels above the executable.
|
||||
exe = tm.gd("python_bin", "")
|
||||
if not exe:
|
||||
return ""
|
||||
return os.path.dirname(os.path.dirname(_norm(exe)))
|
||||
|
||||
|
||||
def check_sys_executable():
|
||||
"""py_func subprocess: sys.executable must match the configured python_bin."""
|
||||
expected = _norm(tm.gd("python_bin", ""))
|
||||
actual = _norm(sys.executable)
|
||||
if expected and actual == expected:
|
||||
return True
|
||||
return (
|
||||
-1,
|
||||
f"sys.executable={actual!r} differs from python_bin={expected!r}",
|
||||
)
|
||||
|
||||
|
||||
def check_sys_prefix_in_venv():
|
||||
"""py_func subprocess: sys.prefix must match the venv root derived
|
||||
from python_bin (two levels up from the executable)."""
|
||||
venv = _venv_dir()
|
||||
if not venv:
|
||||
return (-1, "python_bin is not set in the global dict")
|
||||
actual = _norm(sys.prefix)
|
||||
if actual == venv:
|
||||
return True
|
||||
return (
|
||||
-1,
|
||||
f"sys.prefix={actual!r} is not the validation venv {venv!r}",
|
||||
)
|
||||
|
||||
|
||||
def check_is_venv():
|
||||
"""py_func subprocess: confirm we are inside a venv, i.e. sys.prefix
|
||||
differs from sys.base_prefix. This catches the case where python_bin
|
||||
happens to be a system interpreter and the path-based check would
|
||||
pass trivially."""
|
||||
actual = _norm(sys.prefix)
|
||||
base = _norm(sys.base_prefix)
|
||||
if actual != base:
|
||||
return True
|
||||
return (
|
||||
-1,
|
||||
f"sys.prefix == sys.base_prefix == {actual!r}: not running in a venv",
|
||||
)
|
||||
61
test/validation/run.bat
Normal file
61
test/validation/run.bat
Normal file
@@ -0,0 +1,61 @@
|
||||
@echo off
|
||||
SETLOCAL EnableExtensions
|
||||
|
||||
REM Runs the testium validation suite with a dedicated Python venv used
|
||||
REM by every py_func / cycle / inline-eval subprocess. testium itself
|
||||
REM keeps running in the project's own environment; the validation venv
|
||||
REM only isolates *test execution*.
|
||||
REM
|
||||
REM test\validation\run.bat [clean] [extra testium args]
|
||||
REM
|
||||
REM Requires the project venv to already exist (run the project's
|
||||
REM run.bat once first, or any other testium install method).
|
||||
|
||||
SET "SCRIPT_DIR=%~dp0"
|
||||
SET "PROJECT_DIR=%SCRIPT_DIR%..\.."
|
||||
REM Venv in the user temp dir (Windows equivalent of /tmp).
|
||||
SET "VENV_DIR=%TEMP%\testium-validation-venv"
|
||||
SET "PROJECT_VENV=%PROJECT_DIR%\test\tmp\testium_venv"
|
||||
|
||||
IF /I "%~1"=="clean" (
|
||||
rmdir /s /q "%VENV_DIR%"
|
||||
SHIFT
|
||||
)
|
||||
|
||||
REM Locate a host Python.
|
||||
SET "PYTHON_EXE=python"
|
||||
py --version >nul 2>&1
|
||||
IF %ERRORLEVEL% EQU 0 (
|
||||
SET "PYTHON_EXE=py"
|
||||
goto :PYTHON_FOUND
|
||||
)
|
||||
python --version >nul 2>&1
|
||||
IF %ERRORLEVEL% EQU 0 (
|
||||
SET "PYTHON_EXE=python"
|
||||
goto :PYTHON_FOUND
|
||||
)
|
||||
echo ERROR : Python could not be found on this system.
|
||||
exit /b 1
|
||||
|
||||
:PYTHON_FOUND
|
||||
|
||||
IF NOT EXIST "%VENV_DIR%" (
|
||||
echo Creating validation venv at %VENV_DIR%
|
||||
%PYTHON_EXE% -m venv --system-site-packages "%VENV_DIR%"
|
||||
IF %ERRORLEVEL% NEQ 0 (
|
||||
echo ERROR while creating the validation venv.
|
||||
exit /b 1
|
||||
)
|
||||
call "%VENV_DIR%\Scripts\pip" install --quiet --upgrade pip
|
||||
call "%VENV_DIR%\Scripts\pip" install --quiet junit-xml
|
||||
)
|
||||
|
||||
SET "VENV_PYTHON=%VENV_DIR%\Scripts\python.exe"
|
||||
|
||||
IF NOT EXIST "%PROJECT_VENV%" (
|
||||
echo ERROR : project venv not found at %PROJECT_VENV%. Run the project run.bat once first.
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
call "%PROJECT_VENV%\Scripts\activate"
|
||||
python "%PROJECT_DIR%\src\testium" -b -d "python_bin=%VENV_PYTHON%" -- "%SCRIPT_DIR%main.tum" %*
|
||||
47
test/validation/run.sh
Executable file
47
test/validation/run.sh
Executable file
@@ -0,0 +1,47 @@
|
||||
#!/bin/bash
|
||||
# Runs the testium validation suite with a dedicated Python venv used by
|
||||
# every py_func / cycle / inline-eval subprocess (i.e. everything that
|
||||
# goes through ``bins.python_bin()``). testium itself keeps running in
|
||||
# the project's own environment — the validation venv only isolates
|
||||
# *test execution*.
|
||||
#
|
||||
# ./test/validation/run.sh [clean] [extra testium args]
|
||||
#
|
||||
# ``clean`` (optional, must be the first arg) removes the venv before
|
||||
# recreating it; this is the way to refresh the venv after a system
|
||||
# Python upgrade.
|
||||
|
||||
set -e
|
||||
|
||||
SCRIPT_PATH="$(readlink -f "$0")"
|
||||
SCRIPT_DIR="$(realpath "$(dirname "$SCRIPT_PATH")")"
|
||||
PROJECT_DIR="$(realpath "$SCRIPT_DIR/../..")"
|
||||
# Venv lives in the system temp dir so it stays out of the project tree
|
||||
# (and is naturally cleaned up by tmpfiles/reboot on most distros).
|
||||
VENV_DIR="${TMPDIR:-/tmp}/testium-validation-venv"
|
||||
|
||||
if [ "${1:-}" = "clean" ]; then
|
||||
rm -rf "$VENV_DIR"
|
||||
shift
|
||||
fi
|
||||
|
||||
if [ ! -d "$VENV_DIR" ]; then
|
||||
echo "Creating validation venv at $VENV_DIR"
|
||||
# --system-site-packages so we don't have to reinstall pyside6, lxml
|
||||
# & friends just to support the validation helpers. We still pip
|
||||
# install junit-xml below because it is the one dep that does *not*
|
||||
# ship as a system package on most distros and is required by
|
||||
# post_execution.py.
|
||||
python3 -m venv --system-site-packages "$VENV_DIR"
|
||||
"$VENV_DIR/bin/pip" install --quiet --upgrade pip
|
||||
"$VENV_DIR/bin/pip" install --quiet junit-xml
|
||||
fi
|
||||
|
||||
VENV_PYTHON="$VENV_DIR/bin/python3"
|
||||
|
||||
# Delegate to the project's run.sh so testium itself still runs in the
|
||||
# project venv (with pyside6, gitpython, ...). ``-d python_bin=...``
|
||||
# pins every test-execution subprocess to the validation venv.
|
||||
exec "$PROJECT_DIR/run.sh" -b \
|
||||
-d "python_bin=$VENV_PYTHON" \
|
||||
-- "$SCRIPT_DIR/main.tum" "$@"
|
||||
Reference in New Issue
Block a user