fix(windows): UTF-8 console + self-sufficient validation wrapper

Make the suite run cleanly on Windows.

Product code:
- __init__.py: force UTF-8 on stdout/stderr. The Windows console code
  page (cp1252) cannot encode the box-drawing/accented characters the
  runner prints, which crashed the parent capture_stdout thread. Only
  the stream encoders are reconfigured; the locale default used to read
  cp1252 config files is left untouched.
- report_export_junit/html: open the report file with encoding="utf-8"
  (XML/HTML are UTF-8) instead of the platform default, matching the
  txt/json exporters.

Validation:
- run.bat: source mode now sets up its own venv and runs testium from
  src\ directly instead of delegating to the project run.bat (which
  launches the GUI and drops its arguments). Installs the fake_exporter
  entry-point plugin (report_plugin) and the [lsp] extra, and runs the
  same lsp_check.py pre-flight as run.sh.
- jsonrpc/test.tum: launch the echo server via "$(python_bin)" instead
  of "python3" (the Microsoft Store stub on Windows).
- post_execution.py: write the JUnit XML with encoding="utf-8".
- restore items/run/sub_pass.tum and sub_fail.tum, deleted by mistake in
  d97d00c "removed test logs".

README.md

@@ -129,6 +129,22 @@ A VSCode / VSCodium client extension (`testium_assist`) wraps `testium lsp`;
 the schema is built from testium itself, so new item types and parameters
 appear in the editor on the next testium upgrade with no client change.
 
+It is published on [Open VSX](https://open-vsx.org/extension/testium/testium-assist),
+so in **VSCodium, Cursor, Windsurf, Theia and code-server** it installs from the
+Extensions view (search `testium-assist`) or with
+`codium --install-extension testium.testium-assist`.
+
+**Microsoft VSCode** does not list Open VSX extensions, so install the `.vsix`
+by hand — download it from the Open VSX page above, then *Extensions → ⋯ →
+Install from VSIX…* or:
+
+```sh
+code --install-extension testium-assist-0.1.0.vsix
+```
+
+The extension runs `testium lsp`, so `testium` must be on the `PATH` (otherwise
+point the `testium.serverPath` setting at the binary/AppImage).
+
 ## Troubleshooting
 
 ### `wl_proxy_marshal_flags` symbol error

doc/manual/sphinx/source/modes.rst

@@ -67,3 +67,36 @@ dependencies:
     :caption: enable the language server for a wheel / source install
 
     pip install 'testium[lsp]'
+
+Installing the VSCode / VSCodium extension
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The *testium_assist* client extension is published on `Open VSX
+<https://open-vsx.org/extension/testium/testium-assist>`_, the registry used by
+VSCodium, Cursor, Windsurf, Eclipse Theia and code-server. In those editors,
+open the Extensions view and search ``testium-assist``, or install it from the
+command line:
+
+.. code-block:: text
+    :caption: install in VSCodium and other Open VSX editors
+
+    codium --install-extension testium.testium-assist
+
+Microsoft *VSCode* uses a different marketplace that does not list Open VSX
+extensions, so install the packaged ``.vsix`` by hand. Download it from the
+Open VSX page linked above, then either choose *Extensions* → *⋯* →
+*Install from VSIX…* in the UI, or run:
+
+.. code-block:: text
+    :caption: install the .vsix in Microsoft VSCode
+
+    code --install-extension testium-assist-0.1.0.vsix
+
+The extension launches ``testium lsp``, so the ``testium`` command must be on
+the ``PATH``. If *testium* is installed elsewhere — a specific binary or an
+AppImage — point the ``testium.serverPath`` setting at it instead.
+
+Once installed, open a ``.tum`` file: completion of item types, hover
+documentation and the outline view become available. If nothing happens, check
+that no ``files.associations`` entry forces ``*.tum`` to another language (it
+must stay the ``tum`` language the extension provides).

release_note.txt

@@ -1,3 +1,9 @@
+version 0.2.2
+==============
+- Flatpak sandbox issue fixed for term console. Now a term console is
+  exactly like a host console.
+- Persistence fix of dialogs in case of flatpak.
+
 version 0.2.1
 ==============
 - Faster test loading, especially for large tests built from jinja

src/VERSION

@@ -1 +1 @@
-0.2.1
+0.2.2

src/testium/__init__.py

@@ -11,6 +11,16 @@ sys.path.append(os.path.abspath(ourpath.parent))
 import interpreter.utils.constants as cst
 
 def main():
+    # Force UTF-8 on stdout/stderr so the runner's output survives a legacy
+    # console code page (Windows cp1252 can't encode box-drawing/accented
+    # chars). Only the stream encoders change; the locale default used for
+    # config files is untouched.
+    for _stream in (sys.stdout, sys.stderr):
+        try:
+            _stream.reconfigure(encoding="utf-8")
+        except (AttributeError, ValueError):
+            pass  # no stdout (frozen GUI) or non-reconfigurable stream
+
     # Subcommand dispatch (must run *before* argparse so neither 'schema' nor
     # 'lsp' has to share the GUI/batch flag surface). The subcommands also
     # skip the multiprocessing 'spawn' setup which is only meaningful for the

src/testium/api/termconsole.py

@@ -81,9 +81,13 @@ class TermConsole(Console):
                                    bufsize=0)
 
         else:
-            self.term = pexpect.spawn(  shell_cmd,
-                                        echo=False,
-                                        cwd=self.ppath)
+            # In Flatpak this returns a `flatpak-spawn --host` wrapper so the
+            # console behaves like a host shell (matching py_func / lua_func /
+            # run); elsewhere it's the chosen command unchanged.
+            from interpreter.utils import bins
+            argv = bins.host_console_command(shell_cmd, self.ppath)
+            self.term = pexpect.spawn(argv[0], args=argv[1:],
+                                      echo=False, cwd=self.ppath)
 
         self.q = BytesStore()
         self.t = threading.Thread(target=self.enqueue_output)

src/testium/interpreter/test_items/dialog_choices_files/choices_dialog.py

@@ -221,6 +221,11 @@ def main(args, conn=None):
 
     if conn:
         settings.setValue(SettingsLastChoices, result)
+        # Flush before sending: the parent terminates this subprocess as soon
+        # as it reads the result, so the QSettings destructor never runs and
+        # the write would race the kill (lost under Flatpak — see the
+        # tested-references dialog for the full rationale).
+        settings.sync()
         conn.send([result, success])
         conn.close()
     else:

src/testium/interpreter/test_items/tested_references_files/tested_refs_dialog.py

@@ -76,6 +76,12 @@ def main(args, conn=None):
 
     if conn:
         settings.setValue(SettingsLastReference, result)
+        # Flush to disk *before* handing the result back: as soon as the parent
+        # receives it on the pipe it terminates this subprocess (SIGTERM, no
+        # handler), so the QSettings destructor never runs. Without sync() the
+        # write races the kill and is lost — reliably so under Flatpak, where
+        # the .conf is atomically renamed on the slower ~/.var/app overlay.
+        settings.sync()
         conn.send([result, success])
         conn.close()
     else:

src/testium/interpreter/test_report/report_export_html.py

@@ -14,7 +14,7 @@ class ReportExportHTML(rpe.ReportExport):
         self.prepareFile()
         self.create_base()
         self.process_tests()
-        with open(self._file_name, 'w') as f:
+        with open(self._file_name, 'w', encoding="utf-8") as f:
             f.write(lxml.html.tostring(self.root, pretty_print=True).decode())
 
     def testsIterate(self, row):

src/testium/interpreter/test_report/report_export_junit.py

@@ -20,7 +20,7 @@ class ReportExportJUnit(rpe.ReportExport):
 
         ts = TestSuite(repname, test_cases=self.test_cases,
                        hostname=tm.gd('host_ip'))
-        with open(self._file_name, 'w') as f:
+        with open(self._file_name, 'w', encoding="utf-8") as f:
             TestSuite.to_file(f, [ts])
 
     def testsIterate(self, row):

src/testium/interpreter/utils/bins.py

@@ -19,6 +19,7 @@ Public API
 
 import atexit
 import os
+import shlex
 import shutil
 import subprocess
 import tempfile
@@ -177,6 +178,27 @@ def flatpak_host_spawn(interp_bin, cmd_args, host_cwd, extra_env=None):
     return spawn
 
 
+def host_console_command(shell_cmd, cwd):
+    """Build the argv to start *shell_cmd* as an ordinary interactive console.
+
+    *shell_cmd* is the command the caller chose (a string — shell-split — or
+    an argv list); the choice is preserved verbatim.
+
+    Outside Flatpak the command is returned unchanged. Inside Flatpak a bare
+    spawn would run in the sandbox under the runtime python3, so a host venv
+    (``/path/venv/bin/python3 -m mod``) can't see its pip deps. We simply run
+    it on the host with ``flatpak-spawn --host`` so it behaves like any other
+    terminal: flatpak-spawn passes the current environment through unchanged
+    and the shell (sourced venv, profile, …) sets things up as the user wants.
+    No env forwarding or scrubbing — the launcher's leaked PYTHONPATH points at
+    /app paths absent on the host, so it's inert there.
+    """
+    argv = shlex.split(shell_cmd) if isinstance(shell_cmd, str) else list(shell_cmd)
+    if not _in_flatpak():
+        return argv
+    return ["flatpak-spawn", "--host", f"--directory={cwd}", *argv]
+
+
 def _which_host_flatpak(name):
     """Resolve a binary name (or absolute path) on the host via flatpak-spawn.
 

test/validation/items/console/test.tum

@@ -94,6 +94,17 @@
 {% endif %}
         - read_until: {expected: endOfCmd, timeout: 1, process_result: "'Hello' in r'''$(result)''' and 'PASS' in r'''$(result)''' "}
 
+{% if os == "Linux" %}
+- console:
+    name: Console runs on host (not the Flatpak sandbox)
+    doc: Regression guard for the 0.2.1 Flatpak bug (term console spawned inside the sandbox instead of on the host). /.flatpak-info exists only inside the sandbox, so the host-only marker is emitted (and matched by read_until) ONLY when the shell really runs on the host. On a broken Flatpak the marker never appears, read_until times out and the item FAILS. The marker is built at runtime ($M) so it is never present in the command line itself. Passes on every other channel.
+    console_name: term
+    key: $(test)_PASS
+    steps:
+        - writeln: 'test -e /.flatpak-info && M=SANDBOX || M=HOST; echo "console_host_check_$M"'
+        - read_until: {expected: console_host_check_HOST, timeout: 5}
+{% endif %}
+
 - console:
     name: Console closure
     execute_on_stop: true

test/validation/items/jsonrpc/test.tum

@@ -20,7 +20,7 @@
             console_name: jrpces
             key: $(test)_PASS
             steps:
-                - writeln: python3 {{include_directory}}/jrpc_echo_server.py -c {{include_directory}}/jrpces.ini
+                - writeln: '"$(python_bin)" {{include_directory}}/jrpc_echo_server.py -c {{include_directory}}/jrpces.ini'
                 - read_until: {expected: ready, timeout: 5}
 
         - console:

test/validation/post_execution.py

@@ -89,7 +89,7 @@ def exec():
         junit_report = report.replace(".sqlite", f"-{test}.xml")
         print(junit_report)
         _prepare_file_to_save(junit_report)
-        with open(junit_report, "w") as f:
+        with open(junit_report, "w", encoding="utf-8") as f:
             f.write(TestSuite.to_xml_string([ts]))
 
     # cleanup

test/validation/run.bat

@@ -89,6 +89,13 @@ REM Reports are stamped with the mode so successive runs don't clobber each othe
 
 SET "TAIL=-b -d "python_bin=%VENV_PYTHON%" -d "validation_report_file=validation-%MODE%" -- "%SCRIPT_DIR%\main.tum"%EXTRA%"
 
+REM The report-exporter plugin (items\report_plugin) is a pip entry-point
+REM package. It must live in the *testium* environment, so it is installed into
+REM the source/wheel venvs below. A frozen PyInstaller binary cannot see
+REM externally-installed plugins, so report_plugin is expected to be skipped
+REM there (same as Linux pyinstaller mode).
+SET "FAKE_EXPORTER=%SCRIPT_DIR%\fake_exporter"
+
 REM ---------- per-mode launcher ----------------------------------------------
 
 echo -- validation mode: %MODE%
@@ -100,8 +107,25 @@ echo ERROR: unknown --mode '%MODE%'. Expected: source ^| wheel ^| pyinstaller.
 exit /b 1
 
 :MODE_SOURCE
-call "%PROJECT_DIR%\run.bat" %TAIL%
-exit /b %ERRORLEVEL%
+REM Run testium from src\ in a dedicated venv set up here. We do NOT delegate to
+REM the project's run.bat: that one launches the GUI and does not forward its
+REM arguments, so the suite would never run head-less.
+SET "TESTIUM_VENV=%PROJECT_DIR%\test\tmp\testium_venv"
+IF NOT EXIST "%TESTIUM_VENV%" (
+    echo Creating testium venv at %TESTIUM_VENV%
+    %PYTHON_EXE% -m venv "%TESTIUM_VENV%"
+    IF !ERRORLEVEL! NEQ 0 (
+        echo ERROR while creating the testium venv.
+        exit /b 1
+    )
+    call "%TESTIUM_VENV%\Scripts\pip" install --quiet --upgrade pip
+    call "%TESTIUM_VENV%\Scripts\pip" install --quiet -r "%PROJECT_DIR%\src\requirements.txt"
+    REM language-server extra so `testium lsp` works from source (lsp_check.py)
+    call "%TESTIUM_VENV%\Scripts\pip" install --quiet "pygls>=1.3"
+)
+call "%TESTIUM_VENV%\Scripts\pip" install --quiet -e "%FAKE_EXPORTER%"
+SET CMD="%TESTIUM_VENV%\Scripts\python.exe" "%PROJECT_DIR%\src\testium"
+GOTO LAUNCH
 
 :MODE_WHEEL
 SET "WHEEL=%PROJECT_DIR%\dist\testium-%VERSION%-py3-none-any.whl"
@@ -115,10 +139,13 @@ IF NOT EXIST "%WHEEL_VENV%" (
     echo Creating wheel venv at %WHEEL_VENV%
     %PYTHON_EXE% -m venv --system-site-packages "%WHEEL_VENV%"
     call "%WHEEL_VENV%\Scripts\pip" install --quiet --upgrade pip
-    call "%WHEEL_VENV%\Scripts\pip" install --quiet "%WHEEL%"
+    REM install with the [lsp] extra so the wheel channel is validated in its
+    REM language-server-capable form (pulls pygls), matching `pip install testium[lsp]`.
+    call "%WHEEL_VENV%\Scripts\pip" install --quiet "%WHEEL%[lsp]"
 )
-"%WHEEL_VENV%\Scripts\python.exe" -m testium %TAIL%
-exit /b %ERRORLEVEL%
+call "%WHEEL_VENV%\Scripts\pip" install --quiet -e "%FAKE_EXPORTER%"
+SET CMD="%WHEEL_VENV%\Scripts\python.exe" -m testium
+GOTO LAUNCH
 
 :MODE_PYI
 SET "PYI_BIN=%PROJECT_DIR%\dist\testium-%VERSION%.exe"
@@ -127,5 +154,22 @@ IF NOT EXIST "%PYI_BIN%" (
     echo ERROR: PyInstaller binary not found in %PROJECT_DIR%\dist -- run build_all.sh first.
     exit /b 1
 )
-"%PYI_BIN%" %TAIL%
+SET CMD="%PYI_BIN%"
+GOTO LAUNCH
+
+REM ---------- launch ----------------------------------------------------------
+
+:LAUNCH
+echo -- launch: %CMD%
+
+REM LSP check (this exact channel): `schema` must keep its nested actions and
+REM `lsp` must answer initialize. Mirrors run.sh; aborts the run on failure.
+echo -- LSP check (%MODE%)
+"%VENV_PYTHON%" "%SCRIPT_DIR%\lsp_check.py" %CMD%
+IF !ERRORLEVEL! NEQ 0 (
+    echo ERROR: LSP check failed for mode %MODE%.
+    exit /b 1
+)
+
+%CMD% %TAIL%
 exit /b %ERRORLEVEL%