v-and-v/testium
2
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 7 Wiki Activity

validation: dedicated venv + fix python_bin override timing

Browse Source 
eval_proc was started before -d/GUI defines reached gd, so
``-d python_bin=...`` and the GUI ``python_bin`` preference were
silently ignored by the very subprocess that runs ``<| ... |>`` evals
(and only took effect for later items once the discovery cache had
already been seeded with the system interpreter). apply_overrides() is
now applied before eval_process_init(), and bins._resolve()'s cache is
keyed by (name, override) so a later param.yaml change re-resolves on
the next lookup.

The validation suite now ships a wrapper (run.sh / run.bat) that
creates a dedicated venv in the system temp dir and pins it via
``-d python_bin=...``. A new ``venv`` item asserts the override took
effect for both eval_proc and py_func paths, with a
``sys.prefix != sys.base_prefix`` marker to catch the case where the
override happens to be a system interpreter (path-equality alone would
miss it, the venv's ``bin/python3`` being a symlink to the host).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
Foué
2026-05-19 08:19:57 +02:00
parent 6f832cd67b
commit 4d8cafb5a0
10 changed files with 321 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files

43
test/validation/README.md
View File

@@ -1,10 +1,43 @@
# Validation
This directory contains the necessary material to run the testium validation.
This directory contains the testium validation suite.
Here is the documentation on how to configure the validation, run it and check that the
results are correct.
## Running the suite
# Tests
```sh
./test/validation/run.sh # Linux
test\validation\run.bat # Windows
```
TBD
The wrapper creates a dedicated Python venv in the system temp dir
(`${TMPDIR:-/tmp}/testium-validation-venv` on Linux, `%TEMP%\testium-validation-venv`
on Windows), using `--system-site-packages` so existing system packages
stay visible. The validation suite is then run with that venv pinned as
`python_bin`. Every test-execution subprocess (inline `<| ... |>`
evaluation, `py_func`, `cycle`, `post_execution`, ...) runs inside the
venv, while testium itself keeps running in the project's own
environment.
Pass `clean` as the first argument to recreate the venv from scratch
(useful after a system Python upgrade):
```sh
./test/validation/run.sh clean
```
## What is checked
The `venv` item under `items/venv/` asserts that the venv is actually
being used:
* `python_bin` is set in the global dict.
* The eval subprocess (used for `<| ... |>` expressions) has
`sys.executable == python_bin`, `sys.prefix == dirname(dirname(python_bin))`,
and `sys.prefix != sys.base_prefix` (i.e. is actually inside a venv).
* A `py_func` subprocess passes the same three checks.
These checks use `abspath`/`normpath` rather than `realpath` on
purpose: the venv's `bin/python3` is a symlink to the host interpreter,
so `realpath` would map both venv and non-venv interpreters to the same
target. `sys.prefix != sys.base_prefix` is the venv-specific marker
that distinguishes the two cases.