francois
4d8cafb5a0
eval_proc was started before -d/GUI defines reached gd, so ``-d python_bin=...`` and the GUI ``python_bin`` preference were silently ignored by the very subprocess that runs ``<| ... |>`` evals (and only took effect for later items once the discovery cache had already been seeded with the system interpreter). apply_overrides() is now applied before eval_process_init(), and bins._resolve()'s cache is keyed by (name, override) so a later param.yaml change re-resolves on the next lookup. The validation suite now ships a wrapper (run.sh / run.bat) that creates a dedicated venv in the system temp dir and pins it via ``-d python_bin=...``. A new ``venv`` item asserts the override took effect for both eval_proc and py_func paths, with a ``sys.prefix != sys.base_prefix`` marker to catch the case where the override happens to be a system interpreter (path-equality alone would miss it, the venv's ``bin/python3`` being a symlink to the host). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
44 lines
1.5 KiB
Markdown
44 lines
1.5 KiB
Markdown
# Validation
|
|
|
|
This directory contains the testium validation suite.
|
|
|
|
## Running the suite
|
|
|
|
```sh
|
|
./test/validation/run.sh # Linux
|
|
test\validation\run.bat # Windows
|
|
```
|
|
|
|
The wrapper creates a dedicated Python venv in the system temp dir
|
|
(`${TMPDIR:-/tmp}/testium-validation-venv` on Linux, `%TEMP%\testium-validation-venv`
|
|
on Windows), using `--system-site-packages` so existing system packages
|
|
stay visible. The validation suite is then run with that venv pinned as
|
|
`python_bin`. Every test-execution subprocess (inline `<| ... |>`
|
|
evaluation, `py_func`, `cycle`, `post_execution`, ...) runs inside the
|
|
venv, while testium itself keeps running in the project's own
|
|
environment.
|
|
|
|
Pass `clean` as the first argument to recreate the venv from scratch
|
|
(useful after a system Python upgrade):
|
|
|
|
```sh
|
|
./test/validation/run.sh clean
|
|
```
|
|
|
|
## What is checked
|
|
|
|
The `venv` item under `items/venv/` asserts that the venv is actually
|
|
being used:
|
|
|
|
* `python_bin` is set in the global dict.
|
|
* The eval subprocess (used for `<| ... |>` expressions) has
|
|
`sys.executable == python_bin`, `sys.prefix == dirname(dirname(python_bin))`,
|
|
and `sys.prefix != sys.base_prefix` (i.e. is actually inside a venv).
|
|
* A `py_func` subprocess passes the same three checks.
|
|
|
|
These checks use `abspath`/`normpath` rather than `realpath` on
|
|
purpose: the venv's `bin/python3` is a symlink to the host interpreter,
|
|
so `realpath` would map both venv and non-venv interpreters to the same
|
|
target. `sys.prefix != sys.base_prefix` is the venv-specific marker
|
|
that distinguishes the two cases.
|