Add context_id to py_func and lua_func for shared persistent subprocess

- py_func and lua_func items accept a context_id parameter; items sharing
  the same id reuse the same subprocess for the duration of the test run
- Subprocess-side tm.setgd/tm.gd use a local fallback dict for non-JSON-
  serializable values (py_func only); serializable values reach the main
  process global dict and are accessible from any test item or subprocess
- Shared subprocess engines are cleaned up in process.py finally block
- LuaProcessBase gains is_alive() (was missing, broke all lua_func items)
- Validation tests cover serializable sharing across different context ids,
  non-serializable sharing within the same context_id, and cross-item access
- RST documentation updated for both py_func and lua_func items

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

doc/manual/sphinx/source/test_items/lua_func_test_item.rst

@@ -39,11 +39,14 @@ The ``lua_func`` test item is of the form:
 Beside common test items attributes, lua_func item has specific attribute, some of which being mandatory.
 
 * ``file``: the script file name that contains the function to be executed.
-  Only python script format is supported.
+  Only Lua script format is supported.
 * ``func_name``: The function name to be executed.
 * ``param``: This is a list of parameters that are passed to the function
   in the order they are presented in the script. These parameters are not
   mandatory and are highly dependent of the function prototype.
+* ``context_id``: Optional. When set, all ``lua_func`` items sharing the same
+  ``context_id`` value run inside the same persistent Lua subprocess for the
+  duration of the test. See :ref:`lua_func context<sec_lua_func_context>` for details.
 
 .. code-block:: yaml
     :caption: ``lua_func`` test item example of usage
@@ -56,16 +59,71 @@ Beside common test items attributes, lua_func item has specific attribute, some
             - $(my_param)
 
 The result of the function (after eventual post treatment) is stored in the global
-variable named ``pfn_<func_name>``
+variable named ``lfn_<item_name>``
 (See :ref:`global variables<sec_global_variables>` for more detail
 on how to access to global variables from test items and scripts).
 
 In the example above, the global variable ``$(lfn_activity)``
 would be created at the end of the item execution. It would contain the resulting
-value of the funcToBeExecuted python function.
+value of the methodName Lua function.
 
 The ``lua_func`` will always result ``PASS``, except if the called function raises
-and exception or if the ``expected_result`` attribute is used.
+an exception or if the ``expected_result`` attribute is used.
+
+.. _sec_lua_func_context:
+
+Sharing state between ``lua_func`` calls
+------------------------------------------
+
+Each ``lua_func`` item without a ``context_id`` runs in a dedicated subprocess that
+is started and stopped around the call. Module-level variables are not preserved
+between two such items.
+
+Inside a ``lua_func`` script, the ``tm`` module exposes ``tm.setgd`` and ``tm.gd``
+to read and write the testium global dictionary of the test process. Values stored
+this way are accessible from any subsequent test item without requiring a shared
+subprocess.
+
+.. code-block:: lua
+    :caption: sharing a value via the global dictionary
+
+    local tm = require("tm")
+    local module = {}
+
+    function module.produce(val)
+        tm.setgd("my_shared_value", val)
+        return val
+    end
+
+    function module.consume()
+        return tm.gd("my_shared_value")
+    end
+
+    return module
+
+When ``context_id`` is set, all ``lua_func`` items that share the same identifier
+reuse the same persistent subprocess. This allows Lua-side state (upvalues, module
+cache) to be retained across calls beyond what ``tm.setgd`` persists.
+
+.. code-block:: yaml
+    :caption: ``lua_func`` items sharing a persistent subprocess
+
+    - lua_func:
+        name: produce value
+        file: my_script.lua
+        func_name: produce
+        context_id: my_context
+        param:
+            - hello
+
+    - lua_func:
+        name: consume value
+        file: my_script.lua
+        func_name: consume
+        context_id: my_context
+        expected_result: hello
+
+The shared subprocess is automatically stopped at the end of the test run.
 
 **Lua Interpreter environment setup**
 

doc/manual/sphinx/source/test_items/py_func_test_item.rst

@@ -89,6 +89,9 @@ some of which being mandatory.
 * ``param``: This is a list of parameters that are passed to the function
   in the order they are presented in the script. These parameters are not
   mandatory and are highly dependent of the function prototype.
+* ``context_id``: Optional. When set, all ``py_func`` items sharing the same
+  ``context_id`` value run inside the same persistent Python subprocess for the
+  duration of the test. See :ref:`py_func context<sec_py_func_context>` for details.
 
 .. code-block:: yaml
     :caption: ``py_func`` test item example of usage
@@ -111,6 +114,86 @@ value of the funcToBeExecuted python function.
 The ``py_func`` will always result ``PASS``, except if the called function raises
 and exception or if the ``expected_result`` attribute is used.
 
+.. _sec_py_func_context:
+
+Sharing state between ``py_func`` calls
+------------------------------------------
+
+Each ``py_func`` item without a ``context_id`` runs in a dedicated subprocess that
+is started and stopped around the call. State cannot be shared between two such
+items using module-level variables.
+
+Two mechanisms are available to share data across calls:
+
+**Using the testium global dictionary**
+
+Inside a ``py_func`` script, the ``tm`` module exposes ``tm.setgd`` and ``tm.gd``
+to read and write the testium global dictionary of the test process. Values stored
+this way are accessible from any subsequent test item (including other ``py_func``
+items) without requiring a shared subprocess.
+
+.. code-block:: python
+    :caption: sharing a serializable value via the global dictionary
+
+    import py_func.tm as tm
+
+    def produce(val):
+        tm.setgd("my_shared_value", val)
+        return val
+
+    def consume():
+        return tm.gd("my_shared_value", None)
+
+Values stored with ``tm.setgd`` must be JSON-serializable (str, int, float, list,
+dict, bool, None). Non-serializable values (objects, connections, file handles…)
+are handled transparently by the local fallback described below.
+
+**Using a shared persistent subprocess (``context_id``)**
+
+When ``context_id`` is set, all ``py_func`` items that share the same identifier
+reuse the same subprocess. The subprocess is kept alive until the end of the test.
+
+This is required for non-JSON-serializable objects (e.g. a socket connection, a
+device handle). Calling ``tm.setgd`` with such a value stores it inside the
+subprocess local dictionary instead of sending it to the main process. It can then
+be retrieved with ``tm.gd`` from any subsequent call that runs in the same subprocess.
+
+.. code-block:: python
+    :caption: sharing a non-serializable object via ``context_id``
+
+    import py_func.tm as tm
+
+    class _Connection:          # not JSON-serializable
+        def __init__(self):
+            self.value = "open"
+
+    def open_connection():
+        tm.setgd("conn", _Connection())   # stored locally in the subprocess
+        return "ok"
+
+    def use_connection():
+        conn = tm.gd("conn")              # retrieved from the subprocess local dict
+        return conn.value
+
+.. code-block:: yaml
+    :caption: ``py_func`` items sharing a persistent subprocess
+
+    - py_func:
+        name: open connection
+        file: my_script.py
+        func_name: open_connection
+        context_id: my_context
+        expected_result: ok
+
+    - py_func:
+        name: use connection
+        file: my_script.py
+        func_name: use_connection
+        context_id: my_context
+        expected_result: open
+
+The shared subprocess is automatically stopped at the end of the test run.
+
 **Python Interpreter environment setup**
 
 Some global variables have an impact on the ``py_func`` test item behavior:

src/py_func/tm.py

@@ -1,19 +1,26 @@
 
+import json
 import sys
 from py_func.handle import FuncHandler
 from lib.tum_except import ETUMRuntimeError
 from lib.api import SUPPORTED_API
 
 thismodule = sys.modules[__name__]
-# Shared FuncHandler instance used to forward API calls. Remains None
-# until `_init_api` is invoked.
 _func_call_thread = None
 
+# Local storage for non-JSON-serializable values
+_local_dict = {}
+
+
+def _is_json_serializable(value):
+    try:
+        json.dumps(value)
+        return True
+    except (TypeError, ValueError):
+        return False
+
 
 ###############################################################################
-# Dynamically create module-level functions for each supported API name.
-# Each generated function shares the implementation of `api_call` but
-# has a distinct name used as the remote action identifier.
 def _make_api(name):
     def _wrapper(*params):
         if _func_call_thread is not None:
@@ -31,21 +38,62 @@ def _make_api(name):
     return _wrapper
 
 for k in SUPPORTED_API:
-    setattr(thismodule, k, _make_api(k))
+    if k not in ('gd', 'setgd', 'delgd'):
+        setattr(thismodule, k, _make_api(k))
+
+
+###############################################################################
+# gd/setgd/delgd with local-dict fallback for non-serializable values
+
+def gd(*params):
+    key = params[0] if params else None
+    if key is not None and key in _local_dict:
+        return _local_dict[key]
+    if _func_call_thread is not None:
+        res = _func_call_thread.call("gd", params)
+        if "result" in res:
+            return res["result"]
+        elif "error" in res:
+            raise ETUMRuntimeError(f"api call to 'tm.gd' failed with error '{res['error']}'")
+        else:
+            raise ETUMRuntimeError("api call failure in jrpc client to be reported to testium support team.")
+    raise ETUMRuntimeError("api not initialized")
+
+
+def setgd(*params):
+    key = params[0] if params else None
+    value = params[1] if len(params) > 1 else None
+    if key is not None and not _is_json_serializable(value):
+        _local_dict[key] = value
+        return None
+    if _func_call_thread is not None:
+        res = _func_call_thread.call("setgd", params)
+        if "result" in res:
+            return res["result"]
+        elif "error" in res:
+            raise ETUMRuntimeError(f"api call to 'tm.setgd' failed with error '{res['error']}'")
+        else:
+            raise ETUMRuntimeError("api call failure in jrpc client to be reported to testium support team.")
+    raise ETUMRuntimeError("api not initialized")
+
+
+def delgd(*params):
+    key = params[0] if params else None
+    if key is not None and key in _local_dict:
+        del _local_dict[key]
+        return None
+    if _func_call_thread is not None:
+        res = _func_call_thread.call("delgd", params)
+        if "result" in res:
+            return res["result"]
+        elif "error" in res:
+            raise ETUMRuntimeError(f"api call to 'tm.delgd' failed with error '{res['error']}'")
+        else:
+            raise ETUMRuntimeError("api call failure in jrpc client to be reported to testium support team.")
+    raise ETUMRuntimeError("api not initialized")
+
 
 def _init_api(host, port, timeout):
-    """Start and initialize the remote function handler.
-
-    Starts a ``FuncHandler`` bound to ``port``, runs it and blocks until
-    it signals readiness.
-
-    Args:
-        port: port number or identifier passed to ``FuncHandler``.
-
-    Returns:
-        The initialized ``FuncHandler`` instance assigned to
-        ``_func_call_thread``.
-    """
     global _func_call_thread
     _func_call_thread = FuncHandler(host, port, timeout=timeout)
     return _func_call_thread
@@ -53,17 +101,10 @@ def _init_api(host, port, timeout):
 
 ###############################################################################
 def _remote_print(*values):
-    """Forward print-like output to the remote handler.
-
-    If a ``_func_call_thread`` is available, this function calls the
-    handler with action name ``"print"`` and the provided values. Errors
-    during forwarding are ignored because printing is best-effort.
-    """
     if _func_call_thread is not None:
         try:
             _func_call_thread.call("print", values)
         except:
-            # Best-effort: ignore forwarding failures
             pass
 
 

src/testium/interpreter/process.py

@@ -275,6 +275,13 @@ Is the python exec path correct ?"""
                     # Stop python eval execution process
                     eval_proc.stop()
                     eval_proc.join()
+                    # Stop shared func context engines (keep_context_id)
+                    for engine in tm.gd("_py_func_contexts", {}).values():
+                        engine.stop()
+                        engine.join()
+                    for engine in tm.gd("_lua_func_contexts", {}).values():
+                        engine.stop()
+                        engine.join()
 
             except Exception as e:
                 print_exception(e)

src/testium/interpreter/test_items/test_item_lua_func.py

@@ -12,10 +12,13 @@ from interpreter.utils.lua_func_exec import LuaFuncExecEngine
 from interpreter.utils.api_srv import api_request
 from interpreter.utils.constants import TestItemType as cst
 
+_LUA_FUNC_CONTEXTS_KEY = "_lua_func_contexts"
+
 
 class TestItemLuaFunc(TestItem):
     """lua_func item usage.
     func file: func_file.lua, func_name: func, param: [$(variable1), [1, 2, 3], true]
+    Optional: context_id: <id>  — share a persistent process with other lua_func items using the same id.
     """
 
     def __init__(self, dict_item, parent=None, status_queue=None, filename=""):
@@ -27,14 +30,26 @@ class TestItemLuaFunc(TestItem):
             self.file_name = self._prms.getParam("file", required=True)
             self.func_name = self._prms.getParam("func_name", required=True)
             self.params = self._prms.getParamAll("param")
+            self._context_id = self._prms.getParam("context_id", default=None, processed=False)
         except:
             raise ETUMSyntaxError(
                 f"The '{self.cmd()}' test item named '{self.name()}' (child of '{self.parent.name()}') has a missing or wrong parameter",
                 self.seqFilename(),
             )
-        # Lua functions call subprocess initialization
         self._lua_func_proc = LuaFuncExecEngine(tm.gd("lua_bin", ""), api_request, 10)
 
+    def _get_engine(self):
+        """Return (engine, persistent). If context_id is set, use a shared persistent engine."""
+        if self._context_id is None:
+            return self._lua_func_proc, False
+
+        ctx_id = self._prms.expanse(self._context_id)
+        contexts = tm.gd(_LUA_FUNC_CONTEXTS_KEY, {})
+        if ctx_id not in contexts:
+            contexts[ctx_id] = LuaFuncExecEngine(tm.gd("lua_bin", ""), api_request, 10)
+            tm.setgd(_LUA_FUNC_CONTEXTS_KEY, contexts)
+        return contexts[ctx_id], True
+
     @test_run
     def execute(self):
         self.result.set(
@@ -48,22 +63,25 @@ class TestItemLuaFunc(TestItem):
             print("Parameters list:")
             print(textwrap.indent(pprint.pformat(pl), " |"))
 
-            self._lua_func_proc.start()
+            engine, persistent = self._get_engine()
-            if not self._lua_func_proc.wait_ready(10):
+
-                raise ETUMRuntimeError(
+            if not engine.is_alive():
-                    f"""Impossible to start the external lua execution process.
+                engine.start()
+                if not engine.wait_ready(10):
+                    raise ETUMRuntimeError(
+                        f"""Impossible to start the external lua execution process.
 Is the lua path correct ?
 lua_bin = {tm.gd("lua_bin", "no lua path defined")}
 Are "lua-sockets" and "lua-cjson" installed ?
 Is the lua environnment well defined in the "LUA_PATH" and "LUA_CPATH" variables ?"""
-                )
+                    )
 
             try:
-                success, ret = self._lua_func_proc.func_call(self.file_name, self.func_name, pl)
+                success, ret = engine.func_call(self.file_name, self.func_name, pl)
             finally:
-                # Stops lua function execution process
+                if not persistent:
-                self._lua_func_proc.stop()
+                    engine.stop()
-                self._lua_func_proc.join()
+                    engine.join()
 
             if success == TestValue.SUCCESS:
                 self.result.set(TestValue.SUCCESS)
@@ -73,7 +91,6 @@ Is the lua environnment well defined in the "LUA_PATH" and "LUA_CPATH" variables
                 print("Returned value:")
                 print(textwrap.indent(pprint.pformat(res), " |"))
 
-                # The result of the func test item is put in global dir and result
                 tm.setgd("lfn_" + self._name, res)
                 self.result.value = res
 
@@ -88,5 +105,5 @@ Is the lua environnment well defined in the "LUA_PATH" and "LUA_CPATH" variables
             traceback.print_exception(*sys.exc_info())
             self.result.set(
                 TestValue.FAILURE,
-                'Unrecoverable "py_func" item error from {}'.format(self.func_name),
+                'Unrecoverable "lua_func" item error from {}'.format(self.func_name),
             )

src/testium/interpreter/test_items/test_item_py_func.py

@@ -12,10 +12,13 @@ from interpreter.utils.py_func_exec import PyFuncExecEngine
 from interpreter.utils.api_srv import api_request
 from interpreter.utils.constants import TestItemType as cst
 
+_PY_FUNC_CONTEXTS_KEY = "_py_func_contexts"
+
 
 class TestItemPyFunc(TestItem):
     """py_func item usage.
     func file: func_file.py, func_name: func, param: [$(variable1), [1, 2, 3], true]
+    Optional: context_id: <id>  — share a persistent process with other py_func items using the same id.
     """
 
     def __init__(self, dict_item, parent=None, status_queue=None, filename=""):
@@ -27,6 +30,7 @@ class TestItemPyFunc(TestItem):
             self.file_name = self._prms.getParam("file", required=True)
             self.func_name = self._prms.getParam("func_name", required=True)
             self.params = self._prms.getParamAll("param")
+            self._context_id = self._prms.getParam("context_id", default=None, processed=False)
         except:
             raise ETUMSyntaxError(
                 f"The '{self.cmd()}' test item named '{self.name()}' (child of '{self.parent.name()}') has a missing or wrong parameter",
@@ -34,6 +38,18 @@ class TestItemPyFunc(TestItem):
             )
         self._py_func_proc = PyFuncExecEngine(tm.gd("python_bin", ""), api_request, 10)
 
+    def _get_engine(self):
+        """Return (engine, persistent). If context_id is set, use a shared persistent engine."""
+        if self._context_id is None:
+            return self._py_func_proc, False
+
+        ctx_id = self._prms.expanse(self._context_id)
+        contexts = tm.gd(_PY_FUNC_CONTEXTS_KEY, {})
+        if ctx_id not in contexts:
+            contexts[ctx_id] = PyFuncExecEngine(tm.gd("python_bin", ""), api_request, 10)
+            tm.setgd(_PY_FUNC_CONTEXTS_KEY, contexts)
+        return contexts[ctx_id], True
+
     @test_run
     def execute(self):
         self.result.set(
@@ -47,20 +63,23 @@ class TestItemPyFunc(TestItem):
             print("Parameters list:")
             print(textwrap.indent(pprint.pformat(pl), " |"))
 
-            # start the process for executing external python
+            engine, persistent = self._get_engine()
-            self._py_func_proc.start()
+
-            if not self._py_func_proc.wait_ready():
+            if not engine.is_alive():
-                raise ETUMRuntimeError(
+                engine.start()
-                    f"""Impossible to start the external python execution process.
+                if not engine.wait_ready():
+                    raise ETUMRuntimeError(
+                        f"""Impossible to start the external python execution process.
 Is the python path correct ?
 python_bin = {tm.gd("python_bin", "no python path defined")}"""
-                )
+                    )
+
             try:
-                success, ret = self._py_func_proc.func_call(self.file_name, self.func_name, pl)
+                success, ret = engine.func_call(self.file_name, self.func_name, pl)
             finally:
-                # Stops python function execution process
+                if not persistent:
-                self._py_func_proc.stop()
+                    engine.stop()
-                self._py_func_proc.join()
+                    engine.join()
 
             if success == TestValue.SUCCESS:
                 self.result.set(TestValue.SUCCESS)
@@ -70,7 +89,6 @@ python_bin = {tm.gd("python_bin", "no python path defined")}"""
                 print("Returned value:")
                 print(textwrap.indent(pprint.pformat(res), " |"))
 
-                # The result of the func test item is put in global dir and result
                 tm.setgd("pfn_" + self._name, res)
                 self.result.value = res
 

src/testium/interpreter/utils/lua_process.py

@@ -221,6 +221,11 @@ class LuaProcessBase:
             return self._rpc.wait_ready(timeout)
         return False
 
+    def is_alive(self):
+        if self._rpc is not None:
+            return self._rpc.is_alive()
+        return False
+
     def stop(self):
         """
         Stops the RPC client.

test/validation/items/lua_func/lua_func.lua

@@ -32,5 +32,13 @@ function module.tuple_return(first, second)
     return first, second
 end
 
+function module.set_context_value(val)
+    tm.setgd("_lua_ctx_test_value", val)
+    return val
+end
+
+function module.get_context_value()
+    return tm.gd("_lua_ctx_test_value")
+end
 
 return module

test/validation/items/lua_func/test.tum

@@ -179,3 +179,36 @@
                     file: $(test_path)$(psep)lua_func.lua
                     func_name: tuple_return
                     param: [ 0, "OK" ]
+
+- group:
+    name: context_id tests
+    steps:
+        - lua_func:
+            name: set context value
+            key: $(test)_PASS
+            file: $(test_path)$(psep)lua_func.lua
+            func_name: set_context_value
+            context_id: lua_ctx_test
+            param:
+                - hello lua
+            expected_result: hello lua
+        - lua_func:
+            name: get context value (same context_id)
+            key: $(test)_PASS
+            file: $(test_path)$(psep)lua_func.lua
+            func_name: get_context_value
+            context_id: lua_ctx_test
+            expected_result: hello lua
+        - lua_func:
+            name: get context value (no context_id, from main gd)
+            key: $(test)_PASS
+            file: $(test_path)$(psep)lua_func.lua
+            func_name: get_context_value
+            expected_result: hello lua
+        - lua_func:
+            name: get context value (different context_id)
+            key: $(test)_PASS
+            file: $(test_path)$(psep)lua_func.lua
+            func_name: get_context_value
+            context_id: lua_ctx_other
+            expected_result: hello lua

test/validation/items/py_func/py_func.py

@@ -27,3 +27,23 @@ def echo(param):
 
 def tuple_return(first, second):
     return first, second
+
+def set_context_value(val):
+    tm.setgd("_py_ctx_test_value", val)
+    return val
+
+def get_context_value():
+    return tm.gd("_py_ctx_test_value", None)
+
+
+class _NotSerializable:
+    def __init__(self, val):
+        self.val = val
+
+def set_ns_value(val):
+    tm.setgd("_py_ctx_ns_value", _NotSerializable(val))
+    return val
+
+def get_ns_value():
+    obj = tm.gd("_py_ctx_ns_value", None)
+    return obj.val if obj is not None else None

test/validation/items/py_func/test.tum

@@ -189,3 +189,51 @@
                     func_name: tuple_return
                     param: [ 0, "OK" ]
                     expected_result: [0, "OK"]
+
+- group:
+    name: context_id tests
+    steps:
+        - py_func:
+            name: set serializable value
+            key: $(test)_PASS
+            file: $(test_path)$(psep)py_func.py
+            func_name: set_context_value
+            param:
+                - hello context
+            expected_result: hello context
+        - py_func:
+            name: get serializable value (same context_id)
+            key: $(test)_PASS
+            file: $(test_path)$(psep)py_func.py
+            func_name: get_context_value
+            context_id: ctx_test
+            expected_result: hello context
+        - py_func:
+            name: get serializable value (no context_id, from main gd)
+            key: $(test)_PASS
+            file: $(test_path)$(psep)py_func.py
+            func_name: get_context_value
+            expected_result: hello context
+        - py_func:
+            name: get serializable value (different context_id)
+            key: $(test)_PASS
+            file: $(test_path)$(psep)py_func.py
+            func_name: get_context_value
+            context_id: ctx_other
+            expected_result: hello context
+        - py_func:
+            name: set non-serializable value
+            key: $(test)_PASS
+            file: $(test_path)$(psep)py_func.py
+            func_name: set_ns_value
+            context_id: ctx_ns_test
+            param:
+                - hello ns
+            expected_result: hello ns
+        - py_func:
+            name: get non-serializable value (same context_id)
+            key: $(test)_PASS
+            file: $(test_path)$(psep)py_func.py
+            func_name: get_ns_value
+            context_id: ctx_ns_test
+            expected_result: hello ns