Solve a tuple error in Lua and unify Lua and Python tests

- term console via flatpak-spawn --host so host venvs resolve (bins.host_console_command)
- QSettings sync() before subprocess kill in choices/tested-refs dialogs
- console regression test: fails on the in-sandbox 0.2.1 console

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

README.md

@@ -129,6 +129,22 @@ A VSCode / VSCodium client extension (`testium_assist`) wraps `testium lsp`;
 the schema is built from testium itself, so new item types and parameters
 appear in the editor on the next testium upgrade with no client change.
 
+It is published on [Open VSX](https://open-vsx.org/extension/testium/testium-assist),
+so in **VSCodium, Cursor, Windsurf, Theia and code-server** it installs from the
+Extensions view (search `testium-assist`) or with
+`codium --install-extension testium.testium-assist`.
+
+**Microsoft VSCode** does not list Open VSX extensions, so install the `.vsix`
+by hand — download it from the Open VSX page above, then *Extensions → ⋯ →
+Install from VSIX…* or:
+
+```sh
+code --install-extension testium-assist-0.1.0.vsix
+```
+
+The extension runs `testium lsp`, so `testium` must be on the `PATH` (otherwise
+point the `testium.serverPath` setting at the binary/AppImage).
+
 ## Troubleshooting
 
 ### `wl_proxy_marshal_flags` symbol error

doc/manual/sphinx/source/modes.rst

@@ -67,3 +67,36 @@ dependencies:
     :caption: enable the language server for a wheel / source install
 
     pip install 'testium[lsp]'
+
+Installing the VSCode / VSCodium extension
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The *testium_assist* client extension is published on `Open VSX
+<https://open-vsx.org/extension/testium/testium-assist>`_, the registry used by
+VSCodium, Cursor, Windsurf, Eclipse Theia and code-server. In those editors,
+open the Extensions view and search ``testium-assist``, or install it from the
+command line:
+
+.. code-block:: text
+    :caption: install in VSCodium and other Open VSX editors
+
+    codium --install-extension testium.testium-assist
+
+Microsoft *VSCode* uses a different marketplace that does not list Open VSX
+extensions, so install the packaged ``.vsix`` by hand. Download it from the
+Open VSX page linked above, then either choose *Extensions* → *⋯* →
+*Install from VSIX…* in the UI, or run:
+
+.. code-block:: text
+    :caption: install the .vsix in Microsoft VSCode
+
+    code --install-extension testium-assist-0.1.0.vsix
+
+The extension launches ``testium lsp``, so the ``testium`` command must be on
+the ``PATH``. If *testium* is installed elsewhere — a specific binary or an
+AppImage — point the ``testium.serverPath`` setting at it instead.
+
+Once installed, open a ``.tum`` file: completion of item types, hover
+documentation and the outline view become available. If nothing happens, check
+that no ``files.associations`` entry forces ``*.tum`` to another language (it
+must stay the ``tum`` language the extension provides).

doc/manual/sphinx/source/test_items/let_test_item.rst

@@ -9,9 +9,9 @@ This element is of the following form:
     - let:
         name: Let Item
         values:
-            key1: value1
-            key2: value2
-            key3: <| $(variable)[$(loop_index)] |>
+            - key1: value1
+            - key2: value2
+            - key3: <| $(variable)[$(loop_index)] |>
 
 The ``let`` element is used to set values in the global directory.
 

doc/manual/sphinx/source/tum_syntax.rst

@@ -51,8 +51,8 @@ The parameter file can be specified in the `.tum` file root:
     :caption: configuration files definition in the main `.tum` test file
 
     config_file:
-        config1.yaml
-        config2.yaml
+        - config1.yaml
+        - config2.yaml
 
     main:
         name: Test example

release_note.txt

@@ -1,3 +1,9 @@
+version 0.2.2
+==============
+- Flatpak sandbox issue fixed for term console. Now a term console is
+  exactly like a host console.
+- Persistence fix of dialogs in case of flatpak.
+
 version 0.2.1
 ==============
 - Faster test loading, especially for large tests built from jinja

src/VERSION

@@ -1 +1 @@
-0.2
+0.2.2

src/testium/api/termconsole.py

@@ -81,9 +81,13 @@ class TermConsole(Console):
                                    bufsize=0)
 
         else:
-            self.term = pexpect.spawn(  shell_cmd,
-                                        echo=False,
-                                        cwd=self.ppath)
+            # In Flatpak this returns a `flatpak-spawn --host` wrapper so the
+            # console behaves like a host shell (matching py_func / lua_func /
+            # run); elsewhere it's the chosen command unchanged.
+            from interpreter.utils import bins
+            argv = bins.host_console_command(shell_cmd, self.ppath)
+            self.term = pexpect.spawn(argv[0], args=argv[1:],
+                                      echo=False, cwd=self.ppath)
 
         self.q = BytesStore()
         self.t = threading.Thread(target=self.enqueue_output)

src/testium/interpreter/test_items/dialog_choices_files/choices_dialog.py

@@ -221,6 +221,11 @@ def main(args, conn=None):
 
     if conn:
         settings.setValue(SettingsLastChoices, result)
+        # Flush before sending: the parent terminates this subprocess as soon
+        # as it reads the result, so the QSettings destructor never runs and
+        # the write would race the kill (lost under Flatpak — see the
+        # tested-references dialog for the full rationale).
+        settings.sync()
         conn.send([result, success])
         conn.close()
     else:

src/testium/interpreter/test_items/test_item_console.py

@@ -344,7 +344,7 @@ class TestItemConsoleReadUntil(TestItemConsoleAction):
     def execute(self):
         cons = self.get_console()
         ru = self._prms.expanse(self._read_until)
-        read_timeout = int(self._prms.getParam("timeout", default=-1, processed=True))
+        read_timeout = float(self._prms.getParam("timeout", default=-1, processed=True))
         mute = self._prms.getParam("mute", default=False, processed=True)
         if read_timeout < 0:
             read_timeout = None

src/testium/interpreter/test_items/tested_references_files/tested_refs_dialog.py

@@ -76,6 +76,12 @@ def main(args, conn=None):
 
     if conn:
         settings.setValue(SettingsLastReference, result)
+        # Flush to disk *before* handing the result back: as soon as the parent
+        # receives it on the pipe it terminates this subprocess (SIGTERM, no
+        # handler), so the QSettings destructor never runs. Without sync() the
+        # write races the kill and is lost — reliably so under Flatpak, where
+        # the .conf is atomically renamed on the slower ~/.var/app overlay.
+        settings.sync()
         conn.send([result, success])
         conn.close()
     else:

src/testium/interpreter/utils/bins.py

@@ -19,6 +19,7 @@ Public API
 
 import atexit
 import os
+import shlex
 import shutil
 import subprocess
 import tempfile
@@ -177,6 +178,27 @@ def flatpak_host_spawn(interp_bin, cmd_args, host_cwd, extra_env=None):
     return spawn
 
 
+def host_console_command(shell_cmd, cwd):
+    """Build the argv to start *shell_cmd* as an ordinary interactive console.
+
+    *shell_cmd* is the command the caller chose (a string — shell-split — or
+    an argv list); the choice is preserved verbatim.
+
+    Outside Flatpak the command is returned unchanged. Inside Flatpak a bare
+    spawn would run in the sandbox under the runtime python3, so a host venv
+    (``/path/venv/bin/python3 -m mod``) can't see its pip deps. We simply run
+    it on the host with ``flatpak-spawn --host`` so it behaves like any other
+    terminal: flatpak-spawn passes the current environment through unchanged
+    and the shell (sourced venv, profile, …) sets things up as the user wants.
+    No env forwarding or scrubbing — the launcher's leaked PYTHONPATH points at
+    /app paths absent on the host, so it's inert there.
+    """
+    argv = shlex.split(shell_cmd) if isinstance(shell_cmd, str) else list(shell_cmd)
+    if not _in_flatpak():
+        return argv
+    return ["flatpak-spawn", "--host", f"--directory={cwd}", *argv]
+
+
 def _which_host_flatpak(name):
     """Resolve a binary name (or absolute path) on the host via flatpak-spawn.
 

src/testium/lua_func/handle.lua

@@ -56,9 +56,17 @@ function handle.func_call(params)
     if err == nil then
         print(string.format("Function executed from '%s'", pfile))
         utils.log("func_call function found '%s', '%s'", file, fname)
-        succ, ret = pcall(func, unpack(prms))
+        err_res = {pcall(func, unpack(prms))}
         utils.log("func_call returned '%s', '%s'", tostring(succ), tostring(ret))
 
+        -- manage tuple ouput of a lua function
+        succ =  table.remove(err_res, 1)
+        if #err_res > 1 then
+            ret = err_res
+        else
+            ret = unpack(err_res)
+        end
+
         if succ then
             res = ret
         else

src/testium/main_win/desktop_integration.py

@@ -0,0 +1,95 @@
+"""Install a desktop entry + icon under the user's data dir so desktop shells
+show the testium icon in the task bar / dock.
+
+On a native Wayland session GNOME takes a window's task-bar icon from the
+``.desktop`` file whose name (or ``StartupWMClass``) matches the window
+``app_id`` — ``QGuiApplication.setWindowIcon`` is ignored there. The portable
+channels (source checkout, PyInstaller binary, AppImage) install no system
+desktop file, so we drop an idempotent one in ``~/.local/share``. The window
+``app_id`` is set to ``testium`` (see ``QApplication.setDesktopFileName`` in
+``testium_win``), which is exactly this file's base name.
+
+Flatpak ships its own ``org.testium.Testium.desktop`` and keeps its own app id,
+so the caller skips this integration there.
+"""
+import os
+import sys
+
+from PySide6.QtCore import Qt
+from PySide6.QtGui import QPixmap
+
+# Must match QApplication.setDesktopFileName(...) for the GUI, and is used as
+# both the desktop-file base name and the StartupWMClass.
+APP_ID = "testium"
+
+
+def _launch_command():
+    """Best-effort Exec= for the menu entry. Not needed for icon matching, but
+    makes the entry actually launchable when possible."""
+    appimage = os.environ.get("APPIMAGE")
+    if appimage:
+        return f'"{appimage}"'
+    if getattr(sys, "frozen", False):
+        return f'"{os.path.abspath(sys.executable)}"'
+    argv0 = os.path.abspath(sys.argv[0]) if sys.argv and sys.argv[0] else ""
+    if argv0 and os.path.exists(argv0):
+        return f'"{os.path.abspath(sys.executable)}" "{argv0}"'
+    return f'"{os.path.abspath(sys.executable)}" -m testium'
+
+
+def ensure_desktop_entry():
+    """Create (or refresh) ~/.local/share icon + desktop entry. Best-effort:
+    any failure is swallowed so it can never take the GUI down.
+
+    Freedesktop-only: a no-op off Linux (Windows / macOS use the window icon)."""
+    if not sys.platform.startswith("linux"):
+        return
+    try:
+        data_home = os.environ.get("XDG_DATA_HOME") or os.path.join(
+            os.path.expanduser("~"), ".local", "share"
+        )
+        icon_dir = os.path.join(data_home, "icons", "hicolor", "256x256", "apps")
+        app_dir = os.path.join(data_home, "applications")
+        icon_path = os.path.join(icon_dir, f"{APP_ID}.png")
+        desktop_path = os.path.join(app_dir, f"{APP_ID}.desktop")
+
+        os.makedirs(icon_dir, exist_ok=True)
+        os.makedirs(app_dir, exist_ok=True)
+
+        # Icon: render the bundled Qt resource to a PNG once. Requires a live
+        # QGuiApplication (the caller creates it before calling us).
+        if not os.path.isfile(icon_path):
+            pixmap = QPixmap(u":/black/testium_logo.png")
+            if not pixmap.isNull():
+                pixmap = pixmap.scaled(
+                    256, 256,
+                    Qt.AspectRatioMode.KeepAspectRatio,
+                    Qt.TransformationMode.SmoothTransformation,
+                )
+                pixmap.save(icon_path, "PNG")
+
+        # Absolute Icon= path so the shell resolves it without an icon-cache
+        # refresh; StartupWMClass lets X11 / XWayland match too.
+        desktop = (
+            "[Desktop Entry]\n"
+            "Type=Application\n"
+            "Name=Testium\n"
+            "Comment=Test sequencer\n"
+            f"Icon={icon_path}\n"
+            f"Exec={_launch_command()} %f\n"
+            "Terminal=false\n"
+            f"StartupWMClass={APP_ID}\n"
+            "Categories=Utility;Development;\n"
+        )
+
+        # Write only when missing or changed, to avoid needless menu churn.
+        current = None
+        if os.path.isfile(desktop_path):
+            with open(desktop_path, "r") as fh:
+                current = fh.read()
+        if current != desktop:
+            with open(desktop_path, "w") as fh:
+                fh.write(desktop)
+    except Exception:
+        # Desktop integration is a nicety, never a hard requirement.
+        pass

src/testium/main_win/test_file_manager.py

@@ -51,14 +51,18 @@ class TestFileManager:
         w.disconnect_signals()
         # Snapshot user-selected checkboxes and fold state so they survive a
         # reload of the same file (same logic as session-restore through prefs).
+        # checkList works only if show_checkboxes is True
         previous_check_list = w.treeTests.getCheckList()
         previous_fold_list = w.treeTests.getFoldList()
         previous_count = w.treeTests.getItemCount()
         self.clear_process()
-        if self.load(file_name) and w.test_service is not None:
-            if w.treeTests.getItemCount() == previous_count:
-                w.treeTests.restoreCheckList(previous_check_list, w.test_service)
+        if self.load(file_name) and \
+            w.test_service is not None and \
+            w.treeTests.getItemCount() == previous_count:
+                if prefs.settings.show_checkboxes :
+                    w.treeTests.restoreCheckList(previous_check_list, w.test_service)
                 w.treeTests.restoreFoldList(previous_fold_list)
+
         w.reconnect_signals()
 
     def _make_progress(self, w):

src/testium/main_win/testium_win.py

@@ -678,6 +678,24 @@ def MainWin(
     debug=False,
 ):
     app = QApplication(sys.argv)
+    # Application identity so desktop shells (GNOME, ...) show the testium
+    # icon in the task bar / dock instead of a generic one. On Wayland this
+    # sets the surface app_id; on X11/XWayland it sets WM_CLASS, so the window
+    # stops inheriting the launcher's class (e.g. "python3" under the AppImage,
+    # which is what GNOME was keying the wrong icon off) and the window icon
+    # below is used as the fallback. In Flatpak the id must be the Flatpak app
+    # id so it matches the installed desktop file.
+    app.setApplicationName("Testium")
+    app.setApplicationDisplayName("Testium")
+    app.setDesktopFileName(os.environ.get("FLATPAK_ID", "testium"))
+    app.setWindowIcon(QIcon(u":/black/testium_logo.png"))
+    # On native Wayland the task-bar icon comes from an installed desktop file
+    # matched to the app_id, not from setWindowIcon(). Flatpak ships its own;
+    # for the other Linux channels drop an idempotent one under ~/.local/share.
+    # Windows / macOS use the window icon set above, so this is Linux-only.
+    if sys.platform.startswith("linux") and not os.environ.get("FLATPAK_ID"):
+        from main_win.desktop_integration import ensure_desktop_entry
+        ensure_desktop_entry()
     ui = MainWindow(
         test_file,
         config_files,

test/validation/items/console/test.tum

@@ -84,7 +84,18 @@
         - read_until: {expected: HelloConsole, timeout: 1, mute: true}
 
 - console:
-    name: Console read_until muted
+    name: Console read_until float timeout
+    console_name: term
+    key: $(test)_PASS
+    steps:
+        - writeln: echo "HelloConsole"
+{% if os == "Windows" %}
+        - read_until: {expected: echo "HelloConsole", timeout: 0.2}
+{% endif %}
+        - read_until: {expected: HelloConsole, timeout: 0.2}
+
+- console:
+    name: Console read_until process result
     console_name: term
     key: $(test)_PASS
     steps:
@@ -94,6 +105,17 @@
 {% endif %}
         - read_until: {expected: endOfCmd, timeout: 1, process_result: "'Hello' in r'''$(result)''' and 'PASS' in r'''$(result)''' "}
 
+{% if os == "Linux" %}
+- console:
+    name: Console runs on host (not the Flatpak sandbox)
+    doc: Regression guard for the 0.2.1 Flatpak bug (term console spawned inside the sandbox instead of on the host). /.flatpak-info exists only inside the sandbox, so the host-only marker is emitted (and matched by read_until) ONLY when the shell really runs on the host. On a broken Flatpak the marker never appears, read_until times out and the item FAILS. The marker is built at runtime ($M) so it is never present in the command line itself. Passes on every other channel.
+    console_name: term
+    key: $(test)_PASS
+    steps:
+        - writeln: 'test -e /.flatpak-info && M=SANDBOX || M=HOST; echo "console_host_check_$M"'
+        - read_until: {expected: console_host_check_HOST, timeout: 5}
+{% endif %}
+
 - console:
     name: Console closure
     execute_on_stop: true

test/validation/items/let/test.tum

@@ -11,8 +11,8 @@
         - let:
             name:  Let it be
             values:
-                it: $(loop_param)
-                be: <| $(loop_param) == $(it) |>
+                - it: $(loop_param)
+                - be: <| $(loop_param) == $(it) |>
 
 - loop:
     name: Cycle iterating on list

test/validation/items/lua_func/lua_func.lua

@@ -12,12 +12,12 @@ function module.assertparam(param)
 end
 
 function module.checkglobal(param)
-    local res = tm.gd(param)
-    return res
+    assert(param=='test parameter')
+    return 0
 end
 
 function module.checkglobal2(index)
-    return tm.gd("lua_data_to_be_returned")[index]
+    return tm.gd("data_to_be_returned")[index+1]
 end
 
 function module.should_not_be_called(param)
@@ -53,7 +53,7 @@ function module.return_nothing()
     -- Returns no value: ret is nil but no error.
 end
 
-function module.return_explicit_nil()
+function module.return_explicit_none()
     return nil
 end
 

test/validation/items/lua_func/param.yaml

@@ -1,6 +1,6 @@
 skipped_test_item: ['skipped_checkglobal']
 
-lua_data_to_be_returned:
+data_to_be_returned:
   - 1
   - {a: 1, b: 2}
   - ["a", 1, 2.1, True]

test/validation/items/lua_func/test.tum

@@ -1,7 +1,15 @@
 - let:
     name: lua_func test constants,
     values:
-      lua_func test parameter: test parameter lua_func
+      - func_test_parameter: test parameter
+
+- lua_func:
+    name: pass lua_func
+    key: $(test)_PASS
+    file: $(test_path)$(psep)lua_func.lua
+    func_name: assertparam
+    param:
+        - true
 
 - lua_func:
     name: fail lua_func
@@ -12,7 +20,7 @@
         - false
 
 - lua_func:
-    name: fail lua_func with expected result FAIL
+    name: fail lua_func with expected result "FAIL"
     key: $(test)_PASS
     file: $(test_path)$(psep)lua_func.lua
     func_name: assertparam
@@ -62,35 +70,7 @@
     file: $(test_path)$(psep)lua_func.lua
     func_name: checkglobal
     param:
-        - lua_func test parameter
-    expected_result: $(lua_func test parameter)
-
-- lua_func:
-    name: global param lua_func 1
-    key: $(test)_PASS
-    file: $(test_path)$(psep)lua_func.lua
-    func_name: checkglobal2
-    param:
-        - 1
-    expected_result: ($(lua_data_to_be_returned))[0]
-
-- lua_func:
-    name: global param lua_func 2
-    key: $(test)_PASS
-    file: $(test_path)$(psep)lua_func.lua
-    func_name: checkglobal2
-    param:
-        - 2
-    expected_result: ($(lua_data_to_be_returned))[1]
-
-- lua_func:
-    name: global param lua_func 3
-    key: $(test)_PASS
-    file: $(test_path)$(psep)lua_func.lua
-    func_name: checkglobal2
-    param:
-        - 3
-    expected_result: ($(lua_data_to_be_returned))[2]
+        - $(func_test_parameter)
 
 - let:
     name:  python2func
@@ -98,88 +78,189 @@
     values:
         - py: $(test_path)$(psep)lua_func.lua
 
+- lua_func:
+    name: global param int
+    key: $(test)_PASS
+    file: $(test_path)$(psep)lua_func.lua
+    func_name: checkglobal2
+    param:
+        - 0
+    expected_result: ($(data_to_be_returned))[0]
+
+- lua_func:
+    name: global param dict
+    key: $(test)_PASS
+    file: $(test_path)$(psep)lua_func.lua
+    func_name: checkglobal2
+    param:
+        - 1
+    expected_result: ($(data_to_be_returned))[1]
+
+- lua_func:
+    name: global param list
+    key: $(test)_PASS
+    file: $(test_path)$(psep)lua_func.lua
+    func_name: checkglobal2
+    param:
+        - 2
+    expected_result: ($(data_to_be_returned))[2]
+
+
+
+- lua_func:
+    name: global param lua_func
+    key: $(test)_PASS
+    file: $(py)
+    func_name: checkglobal
+    param:
+        - $(func_test_parameter)
 
 - lua_func:
     name: skipped_checkglobal
+    key: $(test)_PASS
     file: $(test_path)$(psep)lua_func.lua
     func_name: should_not_be_called
     param:
-        - $(test parameter)
+        - $(func_test_parameter)
 
 - lua_func:
     name: skipped true
+    key: $(test)_FAIL
     file: $(test_path)$(psep)lua_func.lua
-    func_name: checkglobal
+    func_name: echo
     skipped: true
     param:
-        - $(test parameter)
+        - "skipped"
 
 - lua_func:
     name: skipped 1
+    key: $(test)_FAIL
     file: $(test_path)$(psep)lua_func.lua
-    func_name: checkglobal
+    func_name: echo
     skipped: 1
     param:
-        - $(test parameter)
+        - "skipped"
 
 - group:
     name: Function results check
     steps:
         - group:
-            name: Function result failure
+            name: Functions result
             steps:
                 - lua_func:
-                    name: int failure
+                    name: int
                     key: $(test)_PASS
                     file: $(test_path)$(psep)lua_func.lua
                     func_name: echo
                     param: [-1]
                 - lua_func:
-                    name: float failure
+                    name: float 
                     key: $(test)_PASS
                     file: $(test_path)$(psep)lua_func.lua
                     func_name: echo
-                    param: [-1.3]
+                    param: [-20.3]
                 - lua_func:
-                    name: String failure
+                    name: String 
                     key: $(test)_PASS
                     file: $(test_path)$(psep)lua_func.lua
                     func_name: echo
                     param: [ "FAIL" ]
                 - lua_func:
-                    name: Tuple int,str failure
-                    key: $(test)_PASS
-                    file: $(test_path)$(psep)lua_func.lua
-                    func_name: tuple_return
-                    param: [ -1, "Got a failure" ]
-        - group:
-            name: Functions result success
-            steps:
-                - lua_func:
-                    name: int success
-                    key: $(test)_PASS
-                    file: $(test_path)$(psep)lua_func.lua
-                    func_name: echo
-                    param: [0]
-                - lua_func:
-                    name: float success
-                    key: $(test)_PASS
-                    file: $(test_path)$(psep)lua_func.lua
-                    func_name: echo
-                    param: [0.3]
-                - lua_func:
-                    name: String success
-                    key: $(test)_PASS
-                    file: $(test_path)$(psep)lua_func.lua
-                    func_name: echo
-                    param: [ "Something that is not only strictly FAIL" ]
-                - lua_func:
-                    name: Tuple int,str success
+                    name: Tuple int,str 
                     key: $(test)_PASS
                     file: $(test_path)$(psep)lua_func.lua
                     func_name: tuple_return
                     param: [ 0, "OK" ]
-
+        - group:
+            name: Functions result expected
+            steps:
+                - lua_func:
+                    name: int expected
+                    key: $(test)_PASS
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [18]
+                    expected_result: 18
+                - lua_func:
+                    name: float expected
+                    key: $(test)_PASS
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [0.3]
+                    expected_result: 0.3
+                - lua_func:
+                    name: String expected
+                    key: $(test)_PASS
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [ "Something" ]
+                    expected_result: Something
+                - lua_func:
+                    name: Tuple int,str expected
+                    key: $(test)_PASS
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: tuple_return
+                    param: [ 0, "OK" ]
+                    expected_result: [0, "OK"]
+                - lua_func:
+                    name: small list expected
+                    key: $(test)_PASS
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [ [-23] ]
+                    expected_result: [-23]
+                - lua_func:
+                    name: big list expected
+                    key: $(test)_PASS
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [ [-23, 17, 67] ]
+                    expected_result: [-23, 17, 67]
+        - group:
+            name: Function result not expected
+            steps:
+                - lua_func:
+                    name: int not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [18]
+                    expected_result: 17
+                - lua_func:
+                    name: float not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [0.3]
+                    expected_result: 0.5
+                - lua_func:
+                    name: String not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [ "Something" ]
+                    expected_result: Nothing
+                - lua_func:
+                    name: Tuple int,str not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: tuple_return
+                    param: [ 0, "OK" ]
+                    expected_result: [0, "OUPS"]
+                - lua_func:
+                    name: small list not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [ [-23] ]
+                    expected_result: [-22]
+                - lua_func:
+                    name: big list not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)lua_func.lua
+                    func_name: echo
+                    param: [ [-23, 17, 67] ]
+                    expected_result: [-23, 16, 67]
 - lua_func:
     name: delgd test
     key: $(test)_PASS
@@ -193,40 +274,39 @@
     func_name: return_nothing
 
 - lua_func:
-    name: function returning explicit nil should succeed
+    name: function returning explicit None should succeed
     key: $(test)_PASS
     file: $(test_path)$(psep)lua_func.lua
-    func_name: return_explicit_nil
+    func_name: return_explicit_none
 
 - group:
     name: context_id tests
     steps:
         - lua_func:
-            name: set context value
+            name: set serializable value
             key: $(test)_PASS
             file: $(test_path)$(psep)lua_func.lua
             func_name: set_context_value
-            context_id: lua_ctx_test
             param:
-                - hello lua
-            expected_result: hello lua
+                - hello context
+            expected_result: hello context
         - lua_func:
-            name: get context value (same context_id)
+            name: get serializable value (same context_id)
             key: $(test)_PASS
             file: $(test_path)$(psep)lua_func.lua
             func_name: get_context_value
-            context_id: lua_ctx_test
-            expected_result: hello lua
+            context_id: ctx_test
+            expected_result: hello context
         - lua_func:
-            name: get context value (no context_id, from main gd)
+            name: get serializable value (no context_id, from main gd)
             key: $(test)_PASS
             file: $(test_path)$(psep)lua_func.lua
             func_name: get_context_value
-            expected_result: hello lua
+            expected_result: hello context
         - lua_func:
-            name: get context value (different context_id)
+            name: get serializable value (different context_id)
             key: $(test)_PASS
             file: $(test_path)$(psep)lua_func.lua
             func_name: get_context_value
-            context_id: lua_ctx_other
-            expected_result: hello lua
+            context_id: ctx_other
+            expected_result: hello context

test/validation/items/py_func/param.yaml

@@ -1 +1,6 @@
 skipped_test_item: ['skipped_checkglobal']
+
+data_to_be_returned:
+  - 1
+  - {a: 1, b: 2}
+  - ["a", 1, 2.1, True]

test/validation/items/py_func/py_func.py

@@ -16,8 +16,8 @@ def checkglobal(param):
     assert param=='test parameter'
     return 0
 
-def checkglobal2():
-    return tm.gd("py_func test parameter")
+def checkglobal2(index):
+    return tm.gd("data_to_be_returned")[index]
 
 def should_not_be_called(param):
     raise

test/validation/items/py_func/test.tum

@@ -1,7 +1,7 @@
 - let:
     name: py_func test constants,
     values:
-      py_func test parameter: test parameter
+      - func_test_parameter: test parameter
 
 - py_func:
     name: pass py_func
@@ -70,7 +70,7 @@
     file: $(test_path)$(psep)py_func.py
     func_name: checkglobal
     param:
-        - $(py_func test parameter)
+        - $(func_test_parameter)
 
 - let:
     name:  python2func
@@ -79,11 +79,32 @@
         - py: $(test_path)$(psep)py_func.py
 
 - py_func:
-    name: global param py_func 2
+    name: global param int
     key: $(test)_PASS
-    file: $(py)
+    file: $(test_path)$(psep)py_func.py
     func_name: checkglobal2
-    expected_result: $(py_func test parameter)
+    param:
+        - 0
+    expected_result: ($(data_to_be_returned))[0]
+
+- py_func:
+    name: global param dict
+    key: $(test)_PASS
+    file: $(test_path)$(psep)py_func.py
+    func_name: checkglobal2
+    param:
+        - 1
+    expected_result: ($(data_to_be_returned))[1]
+
+- py_func:
+    name: global param list
+    key: $(test)_PASS
+    file: $(test_path)$(psep)py_func.py
+    func_name: checkglobal2
+    param:
+        - 2
+    expected_result: ($(data_to_be_returned))[2]
+
 
 
 - py_func:
@@ -92,104 +113,162 @@
     file: $(py)
     func_name: checkglobal
     param:
-        - $(py_func test parameter)
+        - $(func_test_parameter)
 
 - py_func:
     name: skipped_checkglobal
+    key: $(test)_PASS
     file: $(test_path)$(psep)py_func.py
     func_name: should_not_be_called
     param:
-        - $(py_func test parameter)
+        - $(func_test_parameter)
 
 - py_func:
     name: skipped true
+    key: $(test)_FAIL
     file: $(test_path)$(psep)py_func.py
-    func_name: checkglobal
+    func_name: echo
     skipped: true
     param:
-        - $(py_func test parameter)
+        - "skipped"
 
 - py_func:
     name: skipped 1
+    key: $(test)_FAIL
     file: $(test_path)$(psep)py_func.py
-    func_name: checkglobal
+    func_name: echo
     skipped: 1
     param:
-        - $(py_func test parameter)
+        - "skipped"
 
 - py_func:
     name: FunctionItem test
+    key: $(test)_PASS
     file: $(test_path)$(psep)py_func.py
     func_name: ValidationTest
     param:
-        - $(py_func test parameter)
+        - $(func_test_parameter)
 
 - group:
     name: Function results check
     steps:
         - group:
-            name: Function result 1
+            name: Functions result
             steps:
                 - py_func:
-                    name: int failure
+                    name: int
                     key: $(test)_PASS
                     file: $(test_path)$(psep)py_func.py
                     func_name: echo
                     param: [-1]
-                    expected_result: -1
                 - py_func:
-                    name: float failure
+                    name: float 
                     key: $(test)_PASS
                     file: $(test_path)$(psep)py_func.py
                     func_name: echo
-                    param: [-1.3]
-                    expected_result: -1.3
+                    param: [-20.3]
                 - py_func:
-                    name: String failure
+                    name: String 
                     key: $(test)_PASS
                     file: $(test_path)$(psep)py_func.py
                     func_name: echo
                     param: [ "FAIL" ]
-                    expected_result: FAIL
                 - py_func:
-                    name: Tuple int,str failure
+                    name: Tuple int,str 
                     key: $(test)_PASS
                     file: $(test_path)$(psep)py_func.py
                     func_name: tuple_return
-                    param: [ -1, "Got a failure" ]
-                    expected_result: [-1, "Got a failure"]
+                    param: [ 0, "OK" ]
         - group:
-            name: Functions result 2
+            name: Functions result expected
             steps:
                 - py_func:
-                    name: int success
+                    name: int expected
                     key: $(test)_PASS
                     file: $(test_path)$(psep)py_func.py
                     func_name: echo
-                    param: [0]
-                    expected_result: 0
+                    param: [18]
+                    expected_result: 18
                 - py_func:
-                    name: float success
+                    name: float expected
                     key: $(test)_PASS
                     file: $(test_path)$(psep)py_func.py
                     func_name: echo
                     param: [0.3]
                     expected_result: 0.3
                 - py_func:
-                    name: String success
+                    name: String expected
                     key: $(test)_PASS
                     file: $(test_path)$(psep)py_func.py
                     func_name: echo
-                    param: [ "Something that is not only strictly FAIL" ]
-                    expected_result: Something that is not only strictly FAIL
+                    param: [ "Something" ]
+                    expected_result: Something
                 - py_func:
-                    name: Tuple int,str success
+                    name: Tuple int,str expected
                     key: $(test)_PASS
                     file: $(test_path)$(psep)py_func.py
                     func_name: tuple_return
                     param: [ 0, "OK" ]
                     expected_result: [0, "OK"]
-
+                - py_func:
+                    name: small list expected
+                    key: $(test)_PASS
+                    file: $(test_path)$(psep)py_func.py
+                    func_name: echo
+                    param: [ [-23] ]
+                    expected_result: [-23]
+                - py_func:
+                    name: big list expected
+                    key: $(test)_PASS
+                    file: $(test_path)$(psep)py_func.py
+                    func_name: echo
+                    param: [ [-23, 17, 67] ]
+                    expected_result: [-23, 17, 67]
+        - group:
+            name: Function result not expected
+            steps:
+                - py_func:
+                    name: int not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)py_func.py
+                    func_name: echo
+                    param: [18]
+                    expected_result: 17
+                - py_func:
+                    name: float not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)py_func.py
+                    func_name: echo
+                    param: [0.3]
+                    expected_result: 0.5
+                - py_func:
+                    name: String not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)py_func.py
+                    func_name: echo
+                    param: [ "Something" ]
+                    expected_result: Nothing
+                - py_func:
+                    name: Tuple int,str not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)py_func.py
+                    func_name: tuple_return
+                    param: [ 0, "OK" ]
+                    expected_result: [0, "OUPS"]
+                - py_func:
+                    name: small list not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)py_func.py
+                    func_name: echo
+                    param: [ [-23] ]
+                    expected_result: [-22]
+                - py_func:
+                    name: big list not expected
+                    key: $(test)_FAIL
+                    file: $(test_path)$(psep)py_func.py
+                    func_name: echo
+                    param: [ [-23, 17, 67] ]
+                    expected_result: [-23, 16, 67]
 - py_func:
     name: delgd test
     key: $(test)_PASS

test/validation/items/run/sub_fail.tum

@@ -1,7 +0,0 @@
-main:
-    name: run sub-test (always fail)
-    steps:
-        - check:
-            name: fail
-            values:
-                - false

test/validation/items/run/sub_pass.tum

@@ -1,7 +0,0 @@
-main:
-    name: run sub-test (always pass)
-    steps:
-        - check:
-            name: pass
-            values:
-                - true

test/validation/main.tum

@@ -31,7 +31,11 @@ main:
 
 {% for item in items %}
                 # item test
-                - let: {name: {{ item }} test constants, values: {test: {{ item }}, test_path: items/$(test)}}
+                - let: 
+                    name: {{ item }} test constants
+                    values: 
+                        - test: {{ item }}
+                        - test_path: items/$(test)
                 - group:
                     name: {{ item }} test
                     steps: