validation: dedicated venv + fix python_bin override timing
eval_proc was started before -d/GUI defines reached gd, so ``-d python_bin=...`` and the GUI ``python_bin`` preference were silently ignored by the very subprocess that runs ``<| ... |>`` evals (and only took effect for later items once the discovery cache had already been seeded with the system interpreter). apply_overrides() is now applied before eval_process_init(), and bins._resolve()'s cache is keyed by (name, override) so a later param.yaml change re-resolves on the next lookup. The validation suite now ships a wrapper (run.sh / run.bat) that creates a dedicated venv in the system temp dir and pins it via ``-d python_bin=...``. A new ``venv`` item asserts the override took effect for both eval_proc and py_func paths, with a ``sys.prefix != sys.base_prefix`` marker to catch the case where the override happens to be a system interpreter (path-equality alone would miss it, the venv's ``bin/python3`` being a symlink to the host). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
61
test/validation/run.bat
Normal file
61
test/validation/run.bat
Normal file
@@ -0,0 +1,61 @@
|
||||
@echo off
|
||||
SETLOCAL EnableExtensions
|
||||
|
||||
REM Runs the testium validation suite with a dedicated Python venv used
|
||||
REM by every py_func / cycle / inline-eval subprocess. testium itself
|
||||
REM keeps running in the project's own environment; the validation venv
|
||||
REM only isolates *test execution*.
|
||||
REM
|
||||
REM test\validation\run.bat [clean] [extra testium args]
|
||||
REM
|
||||
REM Requires the project venv to already exist (run the project's
|
||||
REM run.bat once first, or any other testium install method).
|
||||
|
||||
SET "SCRIPT_DIR=%~dp0"
|
||||
SET "PROJECT_DIR=%SCRIPT_DIR%..\.."
|
||||
REM Venv in the user temp dir (Windows equivalent of /tmp).
|
||||
SET "VENV_DIR=%TEMP%\testium-validation-venv"
|
||||
SET "PROJECT_VENV=%PROJECT_DIR%\test\tmp\testium_venv"
|
||||
|
||||
IF /I "%~1"=="clean" (
|
||||
rmdir /s /q "%VENV_DIR%"
|
||||
SHIFT
|
||||
)
|
||||
|
||||
REM Locate a host Python.
|
||||
SET "PYTHON_EXE=python"
|
||||
py --version >nul 2>&1
|
||||
IF %ERRORLEVEL% EQU 0 (
|
||||
SET "PYTHON_EXE=py"
|
||||
goto :PYTHON_FOUND
|
||||
)
|
||||
python --version >nul 2>&1
|
||||
IF %ERRORLEVEL% EQU 0 (
|
||||
SET "PYTHON_EXE=python"
|
||||
goto :PYTHON_FOUND
|
||||
)
|
||||
echo ERROR : Python could not be found on this system.
|
||||
exit /b 1
|
||||
|
||||
:PYTHON_FOUND
|
||||
|
||||
IF NOT EXIST "%VENV_DIR%" (
|
||||
echo Creating validation venv at %VENV_DIR%
|
||||
%PYTHON_EXE% -m venv --system-site-packages "%VENV_DIR%"
|
||||
IF %ERRORLEVEL% NEQ 0 (
|
||||
echo ERROR while creating the validation venv.
|
||||
exit /b 1
|
||||
)
|
||||
call "%VENV_DIR%\Scripts\pip" install --quiet --upgrade pip
|
||||
call "%VENV_DIR%\Scripts\pip" install --quiet junit-xml
|
||||
)
|
||||
|
||||
SET "VENV_PYTHON=%VENV_DIR%\Scripts\python.exe"
|
||||
|
||||
IF NOT EXIST "%PROJECT_VENV%" (
|
||||
echo ERROR : project venv not found at %PROJECT_VENV%. Run the project run.bat once first.
|
||||
exit /b 1
|
||||
)
|
||||
|
||||
call "%PROJECT_VENV%\Scripts\activate"
|
||||
python "%PROJECT_DIR%\src\testium" -b -d "python_bin=%VENV_PYTHON%" -- "%SCRIPT_DIR%main.tum" %*
|
||||
Reference in New Issue
Block a user