v-and-v/testium
2
0
Fork 0
Code Issues 5 Pull Requests 1 Actions Packages Projects Releases 7 Wiki Activity

Compare commits

base: v-and-v:tum_schema
Branches Tags
v-and-v:main v-and-v:small_improvements v-and-v:perf/load-time v-and-v:feature/lsp v-and-v:refactor/item-params v-and-v:tum_schema v-and-v:Start_Stop_Pause v-and-v:ai_integration v-and-v:text_no_pyside v-and-v:f1_variables v-and-v:feature/func-keep-context v-and-v:refactor/ui-modularization
v-and-v:v0.2.2 v-and-v:v0.2.1 v-and-v:v0.2 v-and-v:v0.1.3 v-and-v:v0.1.2 v-and-v:v0.1.1 v-and-v:v0.1
..
compare: v-and-v:dd584c906403cc71c6639f6b45dc37be174b6065
Branches Tags
v-and-v:main v-and-v:small_improvements v-and-v:perf/load-time v-and-v:feature/lsp v-and-v:refactor/item-params v-and-v:tum_schema v-and-v:Start_Stop_Pause v-and-v:ai_integration v-and-v:text_no_pyside v-and-v:f1_variables v-and-v:feature/func-keep-context v-and-v:refactor/ui-modularization
v-and-v:v0.2.2 v-and-v:v0.2.1 v-and-v:v0.2 v-and-v:v0.1.3 v-and-v:v0.1.2 v-and-v:v0.1.1 v-and-v:v0.1

2 Commits
tum_schema ... dd584c9064

Author SHA1 Message Date
francois
dd584c9064 gui: bypass XDG portal for all file/dir dialogs in Flatpak 
The v0.1.2 fix that forced Qt's non-native dialog for the "open test"
dialog only covered one call site. The same XDG-portal-vs-sibling-files
problem applies to every other QFileDialog in the GUI (save report,
log file path, default report/log dirs in preferences, python/lua
interpreter pickers).

Extracted a single ``file_dialog.options()`` helper in main_win/ and
threaded it through every getOpenFileName / getSaveFileName /
getExistingDirectory call in main_win/. Outside Flatpak the helper
returns an empty Options(), so the native dialog stays in use on
KDE / GNOME / Windows / macOS.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-19 08:32:14 +02:00
francois
4d8cafb5a0 validation: dedicated venv + fix python_bin override timing 
eval_proc was started before -d/GUI defines reached gd, so
``-d python_bin=...`` and the GUI ``python_bin`` preference were
silently ignored by the very subprocess that runs ``<| ... |>`` evals
(and only took effect for later items once the discovery cache had
already been seeded with the system interpreter). apply_overrides() is
now applied before eval_process_init(), and bins._resolve()'s cache is
keyed by (name, override) so a later param.yaml change re-resolves on
the next lookup.

The validation suite now ships a wrapper (run.sh / run.bat) that
creates a dedicated venv in the system temp dir and pins it via
``-d python_bin=...``. A new ``venv`` item asserts the override took
effect for both eval_proc and py_func paths, with a
``sys.prefix != sys.base_prefix`` marker to catch the case where the
override happens to be a system interpreter (path-equality alone would
miss it, the venv's ``bin/python3`` being a symlink to the host).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-05-19 08:19:57 +02:00
34 changed files with 376 additions and 4001 deletions
Expand all files Collapse all files

21
DESIGN.md
View File

@@ -114,11 +114,20 @@ To add a new API call usable from subprocesses:
### External interpreter resolution (`bins.py`)
`src/testium/interpreter/utils/bins.py` — single source of truth for the paths to the external Python and Lua interpreters used by subprocesses.
- `python_bin()` / `lua_bin()` : resolve once, cache in memory. User can override via the `python_bin` / `lua_bin` global dict keys (typically populated from the YAML config). Falls back to discovery on PATH (candidates: `python3`/`python` and `lua`/`lua5.5`/`lua5.4`/`lua5.3`/`lua5.2`/`lua5.1`).
- `python_bin()` / `lua_bin()` : resolve and cache. The cache is keyed by `(name, override)` so that a later change to `gd[python_bin]` (typically when a `param.yaml` sets the key) triggers a re-resolution on the next lookup instead of returning the stale auto-discovered path. Falls back to discovery on PATH (candidates: `python3`/`python` and `lua`/`lua5.5`/`lua5.4`/`lua5.3`/`lua5.2`/`lua5.1`).
- `ensure(*names)` : called by `TestSet._validate_runtime_deps()` at test load. Always requires `python` (the eval engine always runs); requires `lua` only if a `lua_func` item is in the tree. Fails fast with a clear error citing tried candidates and override key.
Engines (`PyProcessBase`, `LuaProcessBase`, `EvalExecEngine`) call `bins.python_bin()`/`bins.lua_bin()` themselves — call sites never pass an explicit binary path.
#### Override-timing contract (`apply_overrides`)
`bins.python_bin()` is called for the **first** time inside `eval_process_init()` (the long-lived inline-`<| … |>` subprocess), which happens **before** the YAML param files are loaded. To make `-d python_bin=…` and the GUI `python_bin` preference take effect for `eval_proc` itself, `process.py:run()` applies them to gd **before** `eval_process_init()` via the `apply_overrides()` helper extracted from `update_global()`. The post-load `update_global()` call then re-applies the same overrides (after `prepare_global()` clears gd), keeping the gd value in sync with the cached resolution.
| Override source | `eval_proc` | `py_func` / `cycle` / `post_exec` |
|---|---|---|
| `-d python_bin=…` (CLI) | ✅ | ✅ |
| GUI `python_bin` preference | ✅ | ✅ |
| `python_bin: …` in `param.yaml` | ❌ (eval_proc already started) | ✅ (cache re-resolves on key change) |
## Key files
| Path | Role |
@@ -261,12 +270,18 @@ Both Flatpak and AppImage export `TESTIUM_VERSION` from a launcher (Flatpak: lau
- `unittest` item: renamed from `unittest_file`.
- GUI test tree: check and fold state preserved across same-file reloads.
- Licence: EUPL-1.2.
- Interpreter override timing: `apply_overrides()` extracted from `update_global()` and called by `process.py:run()` before `eval_process_init()`, so `-d python_bin=…` / GUI prefs reach `bins.python_bin()` on its first lookup. `bins._resolve()` cache is now keyed by `(name, override)` so later `param.yaml` changes are picked up by subsequently constructed engines.
## Validation tests
Located in `test/validation/`. Run with `-b` flag:
Located in `test/validation/`. Two entry points:
```
./run.sh -b -- test/validation/main.tum
./test/validation/run.sh # wrapper — uses a dedicated venv (see below)
./run.sh -b -- test/validation/main.tum # direct — testium's own python is used for test execution
```
The `run.sh` / `run.bat` wrappers create a dedicated Python venv at `${TMPDIR:-/tmp}/testium-validation-venv` (Linux) or `%TEMP%\testium-validation-venv` (Windows), with `--system-site-packages` + `pip install junit-xml`, and run the suite with `-d python_bin=…` so every test-execution subprocess (eval_proc, py_func, cycle, post_exec) runs inside the venv. testium itself keeps running in the project's own environment. `clean` as the first argument recreates the venv.
The `venv` item (`test/validation/items/venv/`) asserts that the override actually took effect: `python_bin` is set, `sys.executable` matches it, `sys.prefix == dirname(dirname(python_bin))`, and `sys.prefix != sys.base_prefix` (the last marker catches the case where `python_bin` happens to be a system interpreter, which path-equality alone would miss because the venv's `bin/python3` is a symlink to the host). Both `eval_proc` (inline `<| … |>`) and `py_func` paths are exercised.
Parallel item tests: `test/validation/items/parallel/test.tum`
## Dependencies

19
release_note.txt
View File

@@ -1,3 +1,22 @@
version 0.1.3
==============
- Stop interrupts engaged blocking steps (console, py_func, lua_func,
json_rpc, sleep) within ~200 ms instead of waiting for the step
to finish.
- GUI Start / Stop / Pause flow simplified.
- lua_func: a function returning nil is no longer reported as a failure.
- ``-d python_bin=...`` and the GUI ``python_bin`` preference now reach
the eval subprocess (used to be silently ignored). ``param.yaml`` can
also override ``python_bin`` for py_func / cycle / post_exec.
- Validation suite: ``test/validation/run.sh`` (and ``run.bat``)
runs the suite inside a dedicated venv in the system temp dir.
- build_all.sh: ``release_note.txt`` and the user manual copied into
``dist/``; warning if the file has no entry for the version being built.
- Flatpak: every GUI file/directory dialog (open test, save report, log
path, default report/log dirs, python/lua interpreter pickers) now
bypasses the XDG document portal — the v0.1.2 fix was only on the
"open test" dialog.
version 0.1.2
==============
- Flatpak: opening a test from the GUI now correctly finds its companion

73
schema/test.tum
View File

@@ -1,73 +0,0 @@
config_file:
- premier
- saluT
main:
name: Main file
steps:
- group:
name: Test
doc: Une peitite documentation
no_fail: true
steps:
- let:
values:
- my_var: <| ${salut} |>
- check:
values:
- <| ${salut} |>
- dialog_message:
question: c'est quoi?
- lua_func:
file: c'est quoi?
func_name: c'est quoi?
- console:
console_name: cons_1
steps:
- open:
protocol: telnet
terminal_path: ijfeifj
- read_until : {expected: "tutu", timeout: -4.5, mute: true}
- write: something
- writeln: tutu
- close :
- json_rpc:
name: JSONRPC console Query
doc: JSONRPC console Query not waiting (only send)
console:
name : jsonrpc_server
prompt: "@@>"
timeout: 1
version: "2.0"
steps:
- query:
method: echo
params:
- Hello world
- [0, 1, 2, 3]
id: 3095372
no_wait: true
- json_rpc:
name: JSONRPC console Reception
doc: JSONRPC console reception of the previous request
console: {name : jsonrpc_server}
timeout: 1
steps:
- receive:
id: 3095372
timeout: 0.5
report:
enabled: true
log_stored: true
export:
junit:
path: $(validation_report_path)
file_name: $(validation_report_file).junit
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)

96
schema/test_schema/check.tum
View File

@@ -1,96 +0,0 @@
config_file:
- param.yaml
- items/check/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: check test constants
values:
- test: check
- test_path: items/$(test)
- group:
name: check test
steps:
- sequence:
steps:
- py_func:
file: $(test_path)$(psep)check.py
func_name: echo
key: $(test)_PASS
name: Dummy_int
param:
- 2
- py_func:
file: $(test_path)$(psep)check.py
func_name: echo
key: $(test)_PASS
name: Dummy_str
param:
- my taylor is rich
- check:
key: $(test)_PASS
name: Check condition on existing variable (PASS)
values:
- <| $(pfn_Dummy_int) > 1 |>
- check:
key: $(test)_FAIL
name: Check condition on existing variable (FAIL)
values:
- <| "tailor" in "$(pfn_Dummy_str)" |>
filename: /home/renish/workspace/testium/code/test/validation/items/check/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

453
schema/test_schema/common.tum
View File

@@ -1,453 +0,0 @@
config_file:
- param.yaml
- items/common/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: common test constants
values:
test: common
test_path: items/$(test)
- group:
name: common test
steps:
- sequence:
steps:
- group:
name: Results
steps:
- sequence:
steps:
- group:
name: Expected Result
steps:
- py_func:
expected_result: true
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Return True expect True
param:
- true
- py_func:
expected_result: false
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_FAIL
name: Return True expect False (must
fail)
param:
- true
- py_func:
expected_result: None
file: $(test_path)$(psep)results$(psep)results.py
func_name: return_none
key: $(test)_PASS
name: Return None expect None
- py_func:
expected_result: PASS
file: $(test_path)$(psep)results$(psep)results.py
func_name: return_none
key: $(test)_PASS
name: Return None expect PASS
- py_func:
expected_result: 14
file: $(test_path)$(psep)results$(psep)results.py
func_name: return_none
key: $(test)_FAIL
name: Return None expect 14 (must fail)
- group:
name: Expected Result Last test result
steps:
- py_func:
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: result is 28
param:
- 28
- py_func:
expected_result: $(last_step_result)
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: check that the last test result
is 28
param:
- 28
- group:
name: Expected result Failure raised issue
steps:
- py_func:
file: $(test_path)$(psep)results$(psep)results.py
func_name: raise_issue
key: $(test)_FAIL
name: Raise an issue (must fail)
param:
- $(str_example)
- py_func:
expected_result: FAIL
file: $(test_path)$(psep)results$(psep)results.py
func_name: raise_issue
key: $(test)_PASS
name: Raise an issue and expected the
test to be FAIL
param:
- $(str_example)
- py_func:
expected_result: FAIL
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_FAIL
name: Return a String expect a FAILURE
(must fail)
param:
- $(str_example)
- group:
name: process result
steps:
- py_func:
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Process result equal String
param:
- $(str_example)
process_result: '''$(str_example)'' ==
''$(result)'''
- py_func:
expected_result: true
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_FAIL
name: Process result string in the result
(must fail)
param:
- $(str_example)
process_result: '''44'' in ''$(result)'''
- py_func:
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Save the result in a global variable
param:
- 44
store_result: process_result_value
- py_func:
expected_result: $(process_result_value)
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Check the saved global variable
param:
- 44
- py_func:
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: store_result with process_result
param:
- $(str_example)
process_result: '''$(result)''.upper()'
store_result: upper_str_example
- py_func:
expected_result: $(upper_str_example)
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Check store_result with process_result
param:
- $(str_example)
process_result: '''$(result)''.upper()'
- let:
key: $(test)_PASS
name: "store_result on let item (None\
\ value \u2192 stores PASS)"
store_result: let_store_result
values:
- dummy: 0
- py_func:
expected_result: $(let_store_result)
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Check store_result on let stores
PASS
param:
- PASS
- py_func:
expected_result: FAIL
file: $(test_path)$(psep)results$(psep)results.py
func_name: return_none
key: $(test)_FAIL
name: "store_result on failing test (None\
\ value \u2192 stores FAIL)"
store_result: none_fail_store_result
- py_func:
expected_result: $(none_fail_store_result)
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Check store_result on failing test
stores FAIL
param:
- FAIL
- py_func:
expected_result: FAIL
file: $(test_path)$(psep)results$(psep)results.py
func_name: return_none
key: $(test)_PASS
name: "store_result with no_fail (None\
\ value \u2192 stores real FAIL, not\
\ forced PASS)"
no_fail: true
store_result: none_nofail_store_result
- py_func:
expected_result: $(none_nofail_store_result)
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Check store_result with no_fail
stores real FAIL
param:
- FAIL
- py_func:
file: $(test_path)$(psep)results$(psep)results.py
func_name: return_none
key: $(test)_FAIL
name: Process result when result is None
(must fail)
process_result: $(result) is None
- group:
name: no_fail result
steps:
- py_func:
expected_result: false
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Return True expect False but no_fail=True
no_fail: true
param:
- true
- py_func:
expected_result: false
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_FAIL
name: Return True expect False but no_fail=False
(must fail)
no_fail: false
param:
- true
- py_func:
expected_result: false
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_PASS
name: Return True expect False but no_fail
expansed
no_fail: <| bool(0) == False |>
param:
- true
- py_func:
expected_result: false
file: $(test_path)$(psep)results$(psep)results.py
func_name: echo
key: $(test)_FAIL
name: Return True expect False but no_fail
expansed (must fail)
no_fail: <| bool(1) == False |>
param:
- true
filename: /home/renish/workspace/testium/code/test/validation/items/common/results/test.tum
- group:
name: Conditional
steps:
- sequence:
steps:
- loop:
doc: This loop illustrate the way to exit on
a condition.
exit_condition:
value: <| $(pfn_Echo function) > 3 |>
key: $(test)_PASS
name: Infine loop with conditional exit
steps:
- sleep:
name: small wait
timeout: 0.2
- py_func:
file: $(test_path)$(psep)conditional$(psep)conditional.py
func_name: echo
key: $(test)_PASS
name: Echo function
param:
- $(loop_param)
stop_on_failure: false
- let:
key: $(test)_PASS
name: let
values:
- conditional_exec: <| random.randint(1, 2) |>
- console:
condition: <| $(conditional_exec) == 1 |>
console_name: consname
doc: Opening the console
key: $(test)_PASS
name: Console creation
steps:
- open:
protocol: terminal
terminal_path: $(test_directory)
- writeln: echo "terminal loaded"
- console:
condition: <| $(conditional_exec) == 1 |>
console_name: consname
key: $(test)_PASS
name: Console read_until with timeout
steps:
- read_until:
expected: terminal loaded
timeout: 5
- console:
condition: <| $(conditional_exec) == 1 |>
console_name: consname
key: $(test)_PASS
name: Console write
steps:
- writeln: echo 0
- sleep:
condition: <| $(conditional_exec) == 1 |>
name: sleep item
timeout: 1
- console:
condition: <| $(conditional_exec) == 1 |>
console_name: consname
key: $(test)_PASS
name: Console read_until immediate
steps:
- read_until:
expected: '0'
timeout: 0
- console:
condition: <| $(conditional_exec) == 1 |>
console_name: consname
key: $(test)_PASS
name: Console read_until immediate (2)
steps:
- read_until:
expected: $(terminal_prompt)
timeout: 0
- console:
condition: <| $(conditional_exec) == 1 |>
console_name: consname
key: $(test)_PASS
name: Console closure
steps:
- close: consname
- sleep:
condition: <| $(conditional_exec) == 2 |>
name: sleep item
timeout: 1
filename: /home/renish/workspace/testium/code/test/validation/items/common/conditional/test.tum
- group:
name: Various syntax robustness
steps:
- sequence:
steps:
- sleep:
key: $(test)_PASS
timeout: 0.2
- sleep:
key: $(test)_PASS
name: null
timeout: 0.2
- sleep:
doc: null
key: $(test)_PASS
name: Empty "doc:" declared (must PASS)
timeout: 0.2
filename: /home/renish/workspace/testium/code/test/validation/items/common/syntax_robustness/test.tum
- group:
name: Helper lib functions
steps:
- py_func:
file: $(test_path)$(psep)helper_lib.py
func_name: check_os
key: $(test)_PASS
name: OS
param:
- $(os)
- py_func:
file: $(test_path)$(psep)helper_lib.py
func_name: check_get_main_dir
key: $(test)_PASS
name: get_main_dir
- py_func:
file: $(test_path)$(psep)helper_lib.py
func_name: check_timestamp_as_sec_conversion
key: $(test)_PASS
name: timestamp_as_sec conversion
- py_func:
file: $(test_path)$(psep)helper_lib.py
func_name: check_timestamp
key: $(test)_PASS
name: timestamp and timestamp_as_sec
filename: /home/renish/workspace/testium/code/test/validation/items/common/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

164
schema/test_schema/console.tum
View File

@@ -1,164 +0,0 @@
config_file:
- param.yaml
- items/console/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: console test constants
values:
- test: console
- test_path: items/$(test)
- group:
name: console test
steps:
- sequence:
steps:
- console:
console_name: term
doc: Opening the console
key: $(test)_PASS
name: Console creation
steps:
- open:
protocol: terminal
terminal_path: $(test_directory)
- writeln: echo "endOfOpen"
- console:
console_name: term
key: $(test)_PASS
name: Console read_until with timeout
steps:
- read_until:
expected: endOfOpen
timeout: 5
- console:
console_name: term
key: $(test)_PASS
name: Console write
steps:
- writeln: echo 0
- sleep:
name: sleep item
timeout: 1
- console:
console_name: term
key: $(test)_PASS
name: Console read_until immediate
steps:
- read_until:
expected: '0'
timeout: 0
- console:
console_name: term
key: $(test)_PASS
name: Console write
steps:
- writeln: echo "HelloConsole"
- console:
console_name: term
key: $(test)_FAIL
name: Console read_until fail
steps:
- read_until:
expected: Something never prints
timeout: 1
- console:
console_name: term
key: $(test)_PASS
name: Console write
steps:
- writeln: echo "HelloConsole"
- console:
console_name: term
key: $(test)_PASS
name: Console read_until no_fail
steps:
- read_until:
expected: Something never prints
no_fail: true
timeout: 1
- console:
console_name: term
key: $(test)_PASS
name: Console read_until muted
steps:
- writeln: echo "HelloConsole"
- read_until:
expected: HelloConsole
mute: true
timeout: 1
- console:
console_name: term
key: $(test)_PASS
name: Console read_until muted
steps:
- writeln: echo "HelloConsole is PASS" && echo "endOfCmd"
- read_until:
expected: endOfCmd
process_result: '''Hello'' in r''''''$(result)''''''
and ''PASS'' in r''''''$(result)'''''' '
timeout: 1
- console:
console_name: term
execute_on_stop: true
key: $(test)_PASS
name: Console closure
steps:
- close: term
filename: /home/renish/workspace/testium/code/test/validation/items/console/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

106
schema/test_schema/cycle.tum
View File

@@ -1,106 +0,0 @@
config_file:
- param.yaml
- items/cycle/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
terminal_prompt: $(linux_prompt)
psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: cycle test constants
values:
- test: cycle
- test_path: items/$(test)
- group:
name: cycle test
steps:
- sequence:
steps:
- loop:
iterator: 10
key: $(test)_PASS
name: Cycle number of loops
steps:
- py_func:
file: $(test_path)$(psep)cycle.py
func_name: donothing
name: do nothing
- loop:
iterator:
- 12
- 20
- 30
key: $(test)_PASS
name: Cycle iterating on list
steps:
- py_func:
file: $(test_path)$(psep)cycle.py
func_name: checkloopparam
name: check loop param
param:
- $(loop_param)
- loop:
exit_condition:
file: $(test_path)$(psep)cycle.py
func_name: exitcondition
key: $(test)_PASS
name: Infinite loop with exit condition
steps:
- py_func:
file: $(test_path)$(psep)cycle.py
func_name: donothing
name: do nothing
filename: /home/renish/workspace/testium/code/test/validation/items/cycle/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

135
schema/test_schema/dialogs.tum
View File

@@ -1,135 +0,0 @@
config_file:
- param.yaml
- items/dialogs/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: dialogs test constants
values:
test: dialogs
test_path: items/$(test)
- group:
name: dialogs test
steps:
- sequence:
steps:
- dialog_image:
auto_result: ok
condition: $(validation_dialogs)
filename: $(test_path)$(psep)IMG_20140213_171455.jpg
key: $(test)_PASS
name: dialog image PASS
question: click ok if you see the image
- dialog_image:
auto_result: cancel
condition: $(validation_dialogs)
filename: $(test_path)$(psep)IMG_20140213_171455.jpg
key: $(test)_FAIL
name: dialog image FAIL
question: click cancel
- dialog_references:
auto_result: ok
condition: $(validation_dialogs)
key: $(test)_PASS
name: dialog_reference PASS
question: click ok
- dialog_references:
auto_result: cancel
condition: $(validation_dialogs)
key: $(test)_FAIL
name: dialog_reference FAIL
question: click cancel
- dialog_value:
auto_result: ok
auto_value: '123'
condition: $(validation_dialogs)
key: $(test)_PASS
name: dialog_value PASS
question: enter 123 and click ok
- dialog_value:
auto_result: ok
condition: $(validation_dialogs)
key: $(test)_FAIL
name: dialog_value empty FAIL
question: enter nothing and click ok
- dialog_value:
auto_result: cancel
condition: $(validation_dialogs)
key: $(test)_FAIL
name: dialog_value canceled FAIL
question: enter nothing and click cancel
- dialog_message:
auto_result: ok
condition: $(validation_dialogs)
key: $(test)_PASS
name: dialog_message PASS
question: click ok
- dialog_question:
auto_result: 'yes'
condition: $(validation_dialogs)
key: $(test)_PASS
name: dialog_question PASS
question: click yes
- dialog_question:
auto_result: 'no'
condition: $(validation_dialogs)
key: $(test)_FAIL
name: dialog_question FAIL
question: click no
filename: /home/renish/workspace/testium/code/test/validation/items/dialogs/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

130
schema/test_schema/expanse.tum
View File

@@ -1,130 +0,0 @@
config_file:
- param.yaml
- items/expanse/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: expanse test constants
values:
test: expanse
test_path: items/$(test)
- group:
name: expanse test
steps:
- sequence:
steps:
- check:
key: $(test)_PASS
name: Check variables expansion is correct (PASS)
values:
- <| $(expanse_index) == 1 |>
- <| $(expanse_table)[$(expanse_index)] == 9012 |>
- <| $(expanse_eval) == True |>
- let:
key: $(test)_PASS
name: Dynamic variables expansion
values:
- expanse_select: <|"$(expanse_select)".replace("o", "a")|>
- expanse_index: $(expanse_index_$(expanse_select))
- expanse_table: $(expanse_table_$(expanse_select))
- expanse_eval: <|$(expanse_index) == 1|>
- check:
key: $(test)_PASS
name: Check variables expansion is correct (PASS)
values:
- <| $(expanse_index) == 0 |>
- <| $(expanse_table)[$(expanse_index)] == "abcd" |>
- <| $(expanse_eval) == False |>
- let:
key: $(test)_PASS
name: Complex variables expansion
values:
- var1: expanse
- var2: var
- var3: bla
- var4: blo
- expanse_var_bla: 3
- expanse_blo_var: 5
- expanse_complex: <|<|$(expanse_$(var2)_$(var3))*6|>
+ <|4*$($(var1)_$(var4)_$(var2))|>|>
- check:
key: $(test)_PASS
name: Check complex variables expansion is correct (PASS)
values:
- <| $(expanse_complex) == 38 |>
- let:
key: $(test)_PASS
name: Variables expansion in object
values:
- expanse_key: b
- expanse_var: 3
- expanse_var_2: 6
- expanse_object:
- $(expanse_key): <|2**3|>
a: $(expanse_var_2)
- <|"bla".replace("a", "o")|>:
- <|$(expanse_var)*$(expanse_var_2)|>
- 25
- check:
key: $(test)_PASS
name: Check complex variables expansion is correct (PASS)
values:
- '<| $(expanse_object) == [{"a": 6, "b": 8}, {"blo": [18,
25]}] |>'
filename: /home/renish/workspace/testium/code/test/validation/items/expanse/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

82
schema/test_schema/git.tum
View File

@@ -1,82 +0,0 @@
config_file:
- param.yaml
- items/git/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: git test constants
values:
test: git
test_path: items/$(test)
- group:
name: git test
steps:
- sequence:
steps:
- git:
key: $(test)_PASS
name: Testium repo
repo: $(test_directory)
- git:
key: $(test)_PASS
name: Testium repo
repo:
- $(test_directory)
- $(test_directory)
filename: /home/renish/workspace/testium/code/test/validation/items/git/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

129
schema/test_schema/include.tum
View File

@@ -1,129 +0,0 @@
config_file:
- param.yaml
- items/include/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: include test constants
values:
test: include
test_path: items/$(test)
- group:
name: include test
steps:
- sequence:
steps:
- sequence:
steps:
- py_func:
file: $(test_path)$(psep)include.py
func_name: ValidationTest
key: $(test)_PASS
name: My first include test
param:
- $(test parameter)
filename: /home/renish/workspace/testium/code/test/validation/items/include/inc
no template/my first include.tum
- sequence:
steps:
- py_func:
file: $(test_path)$(psep)include.py
func_name: ValidationTest
key: $(test)_PASS
name: My first include test
param:
- $(test parameter)
filename: /home/renish/workspace/testium/code/test/validation/items/include/inc
no template/my first include.tum
- sequence:
steps:
- py_func:
file: $(test_path)$(psep)include.py
func_name: ValidationTest
key: $(test)_PASS
name: My second include test
param:
- My second include test parameter
filename: /home/renish/workspace/testium/code/test/validation/items/include/inc
with template/my second include.tum
- sequence:
steps:
- py_func:
file: $(test_path)$(psep)include.py
func_name: ValidationTest
key: $(test)_PASS
name: My second include test
param:
- My second include test parameter
filename: /home/renish/workspace/testium/code/test/validation/items/include/inc
with template/my second include.tum
- let:
name: Declare param for inclusion
values:
- inc: Dali
- Dali_inc: Dalida
- sequence:
steps:
- let:
name: Test param inclusion 1
values:
- inclusion: $($(inc)_inc)
filename: /home/renish/workspace/testium/code/test/validation/items/include/inc
with template/my_3d_include.tum
filename: /home/renish/workspace/testium/code/test/validation/items/include/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

80
schema/test_schema/isolation.tum
View File

@@ -1,80 +0,0 @@
config_file:
- param.yaml
- items/isolation/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: isolation test constants
values:
test: isolation
test_path: items/$(test)
- group:
name: isolation test
steps:
- sequence:
steps:
- py_func:
expected_result: true
file: $(test_path)$(psep)check_isolation.py
func_name: check_isolation
key: $(test)_PASS
name: py_func/lua_func do not depend on testium internals
param:
- $(testium_path)
filename: /home/renish/workspace/testium/code/test/validation/items/isolation/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

490
schema/test_schema/jsonrpc.tum
View File

@@ -1,490 +0,0 @@
config_file:
- param.yaml
- items/jsonrpc/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: jsonrpc test constants
values:
test: jsonrpc
test_path: items/$(test)
- group:
name: jsonrpc test
steps:
- sequence:
steps:
- console:
console_name: jrpces
doc: check if jrpc_echo_server.py is available
key: $(test)_PASS
name: json rpc echo server
steps:
- open:
protocol: terminal
- read_until:
expected: $(terminal_prompt)
no_fail: true
timeout: 1
- writeln: test -f /home/renish/workspace/testium/code/test/validation/items/jsonrpc/jrpc_echo_server.py
&& echo JRPC_OK
- read_until:
expected: JRPC_OK
no_fail: true
timeout: 2
- group:
condition: <| 'JRPC_OK' in r'''$(cn_json rpc echo server)''' |>
name: jsonrpc tests
steps:
- console:
console_name: jrpces
key: $(test)_PASS
name: Start the json rpc echo server
steps:
- writeln: python3 /home/renish/workspace/testium/code/test/validation/items/jsonrpc/jrpc_echo_server.py
-c /home/renish/workspace/testium/code/test/validation/items/jsonrpc/jrpces.ini
- read_until:
expected: ready
timeout: 5
- console:
console_name: jsonrpc_server
doc: Opening the RAW TCP console
key: $(test)_PASS
name: Open the raw tcp Console
skipped: $(skip_tcp)
steps:
- open:
protocol: rawtcp
tcp_host: localhost
tcp_port: 4321
- json_rpc:
console:
name: jsonrpc_server
key: $(test)_PASS
name: JSONRPC console Query waiting for reception
skipped: $(skip_tcp)
steps:
- query:
method: echo
params:
- Hello World
- a: 1
b: hello
timeout: 1
- json_rpc:
console:
name: jsonrpc_server
key: $(test)_PASS
name: JSONRPC console Query not waiting (only send)
skipped: $(skip_tcp)
steps:
- query:
id: 3095372
method: echo
no_wait: true
params:
- a: -1
b: olleh
- World Hello
timeout: 1
- sleep:
name: Small delay for the test
skipped: $(skip_tcp)
timeout: 1
- json_rpc:
console:
name: jsonrpc_server
key: $(test)_PASS
name: JSONRPC console Reception
skipped: $(skip_tcp)
steps:
- receive:
id: 3095372
timeout: 1
- console:
console_name: jsonrpc_server
doc: Opening the RAW TCP console
execute_on_stop: true
key: $(test)_PASS
name: Close the raw tcp console
skipped: $(skip_tcp)
steps:
- close: null
- json_rpc:
key: $(test)_PASS
name: JSONRPC UDP query waiting for reception
steps:
- open: null
- query:
method: echo
name: echo
params:
- Hello World
- a: 1
b: hello
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 8765
server: localhost
snd_port: 4323
- json_rpc:
key: $(test)_FAIL
name: Failing JSONRPC UDP query waiting for reception
(returning an error)
steps:
- open: null
- query:
method: echo2
params:
- Hello World
- a: 1
b: hello
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 8765
server: localhost
snd_port: 4323
- json_rpc:
key: $(test)_PASS
name: JSONRPC UDP query waiting for reception of
an expected error
steps:
- open: null
- query:
expected_result:
code: -32000
message: function not found
method: echo2
params:
- Hello World
- a: 1
b: hello
- close: null
timeout: 1
udp:
rcv_port: 8765
server: localhost
snd_port: 4323
- json_rpc:
doc: 'Failing JSONRPC UDP query waiting for reception
and checking result
and timeout elapses (wrong udp port)
'
key: $(test)_FAIL
name: Failing UDP JSONRPC query timeout elapses (wrong
udp port)
steps:
- open: null
- query:
method: echo
params:
- Hello World
- a: 1
b: hello
timeout: 0.5
- close: null
timeout: 1
udp:
rcv_port: 48393
server: localhost
snd_port: 4326
- json_rpc:
key: $(test)_PASS
name: JSONRPC UDP query not waiting (only send)
steps:
- open: null
- query:
id: 3095372
method: echo
no_wait: true
params:
- a: -1
b: olleh
- World Hello
timeout: 1
udp:
rcv_port: 8765
server: localhost
snd_port: 4323
- sleep:
name: Small delay for the test
timeout: 1
- json_rpc:
key: $(test)_PASS
name: JSONRPC UDP Reception
steps:
- receive:
id: 3095372
- close: null
timeout: 1
udp:
rcv_port: 8765
server: localhost
snd_port: 4323
- json_rpc:
key: $(test)_PASS
name: JSONRPC UDP query not waiting (only send)
steps:
- open: null
- query:
id: 3095372
method: echo2
no_wait: true
params:
- a: -1
b: olleh
- World Hello
timeout: 1
udp:
rcv_port: 8765
server: localhost
snd_port: 4323
- sleep:
name: Small delay for the test
timeout: 1
- json_rpc:
key: $(test)_FAIL
name: Failing JSONRPC UDP Reception (returning an
error)
steps:
- receive:
id: 3095372
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 8765
server: localhost
snd_port: 4323
- json_rpc:
doc: JSONRPC UDP query waiting for reception and
checking result
key: $(test)_PASS
name: UDP JSONRPC query waiting and checking
steps:
- open: null
- query:
expected_result:
- - Hello World
- a: 1
b: hello
- {}
method: echo
params:
- Hello World
- a: 1
b: hello
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 48393
server: localhost
snd_port: 4323
- json_rpc:
doc: JSONRPC UDP query waiting for reception and
checking result
key: $(test)_FAIL
name: Failing UDP JSONRPC query waiting and checking
steps:
- open: null
- query:
expected_result:
- []
- {}
method: echo
params:
- Hello World
- a: 1
b: hello
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 48393
server: localhost
snd_port: 4323
- json_rpc:
doc: JSONRPC UDP query not waiting, with the purpose
to check the result at reception
key: $(test)_PASS
name: UDP JSONRPC query not waiting (for checking)
steps:
- open: null
- query:
id: 3095372
method: echo
no_wait: true
params:
- a: -1
b: olleh
- World Hello
timeout: 1
udp:
rcv_port: 9876
server: localhost
snd_port: 4323
- sleep:
name: Small delay for the test
timeout: 1
- json_rpc:
doc: JSONRPC UDP Reception and checking result
key: $(test)_PASS
name: UDP JSONRPC reception checking
steps:
- receive:
expected_result:
- - a: -1
b: olleh
- World Hello
- {}
id: 3095372
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 9876
server: localhost
snd_port: 4323
- json_rpc:
doc: JSONRPC UDP Reception and checking result
key: $(test)_FAIL
name: Failing UDP JSONRPC reception checking
steps:
- receive:
expected_result:
- - a: -1
b: ollhe
- World Hello
- {}
id: 3095372
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 9876
server: localhost
snd_port: 4323
- json_rpc:
doc: 'JSONRPC UDP query waiting for reception and
checking result with
replacing $(result) and evaluating string.
'
key: $(test)_PASS
name: UDP JSONRPC query waiting and evaluating result
steps:
- open: null
- query:
expected_result: 1
method: echo
params:
- Hello World
- a: 1
b: hello
process_result: $(result)[0][1]['a']
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 48393
server: localhost
snd_port: 4323
- json_rpc:
doc: 'JSONRPC UDP query waiting for reception and
checking result with
replacing $(result) and evaluating string.
'
key: $(test)_FAIL
name: Failing UDP JSONRPC query waiting and evaluating
result
steps:
- open: null
- query:
expected_result: $(result)[0][1]['a'] == 0
method: echo
params:
- Hello World
- a: 1
b: hello
timeout: 1
- close: null
timeout: 1
udp:
rcv_port: 48393
server: localhost
snd_port: 4323
- console:
console_name: jrpces
doc: check if the jsonrpc echo server is installed
execute_on_stop: true
key: $(test)_PASS
name: Stop json rpc echo server
steps:
- close:
protocol: terminal
filename: /home/renish/workspace/testium/code/test/validation/items/jsonrpc/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

157
schema/test_schema/let.tum
View File

@@ -1,157 +0,0 @@
config_file:
- param.yaml
- items/let/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: let test constants
values:
- test: let
- test_path: items/$(test)
- group:
name: let test
steps:
- sequence:
steps:
- loop:
iterator: 10
key: $(test)_PASS
name: Cycle number of loops
steps:
- py_func:
file: $(test_path)$(psep)let.py
func_name: donothing
name: do nothing
- let:
name: Let it be
values:
- be: <| $(loop_param) == $(it) |>
- it: $(loop_param)
- loop:
iterator:
- 12
- 20
- 30
key: $(test)_PASS
name: Cycle iterating on list
steps:
- py_func:
file: $(test_path)$(psep)let.py
func_name: checkloopparam
name: check loop param
param:
- $(loop_param)
- let:
name: Let it be
values:
- it: $(loop_param)
- be: <| $(loop_param) == $(it) |>
- let:
key: $(test)_PASS
name: Get time
values:
- loop_t0: $(ts_start_Cycle iterating on list)
- loop_t1: $(ts_end_Cycle iterating on list)
- loop_duration: $(ts_duration_Cycle iterating on list)
- let:
key: $(test)_PASS
name: Get parameter file value
values:
- test_overwrite_me: <| $(overwrite_me) == True |>
- py_func:
file: $(test_path)$(psep)let.py
func_name: checkGlobalDic
name: Check global dic pass
param:
- test_overwrite_me
- true
- let:
key: $(test)_PASS
name: Overwrite parameter file value
values:
- overwrite_me: false
- py_func:
expected_result: $(overwrite_me) == False
file: $(test_path)$(psep)let.py
func_name: checkGlobalDic
key: $(test)_FAIL
name: Check global dic fail
param:
- overwrite_me
- true
- py_func:
expected_result: fail
file: $(test_path)$(psep)let.py
func_name: checkGlobalDic
key: $(test)_PASS
name: Check global dic fail
param:
- overwrite_me
- true
- let:
key: $(test)_PASS
name: Evaluate Overwriting parameter file value
values:
- test_overwrite_me: <| "$(overwrite_me)" == True |>
- check:
key: $(test)_PASS
name: Check Overwriting parameter file value
values:
- <| $(test_overwrite_me) == False |>
filename: /home/renish/workspace/testium/code/test/validation/items/let/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

297
schema/test_schema/lua_func.tum
View File

@@ -1,297 +0,0 @@
config_file:
- param.yaml
- items/lua_func/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: lua_func test constants
values:
test: lua_func
test_path: items/$(test)
- group:
name: lua_func test
steps:
- sequence:
steps:
- let:
name: lua_func test constants,
values:
lua_func test parameter: test parameter lua_func
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: assertparam
key: $(test)_FAIL
name: fail lua_func
param:
- false
- lua_func:
expected_result: FAIL
file: $(test_path)$(psep)lua_func.lua
func_name: assertparam
key: $(test)_PASS
name: fail lua_func with expected result FAIL
param:
- false
- lua_func:
expected_result: FAIL
file: $(test_path)$(psep)lua_func.lua
func_name: assertparam
key: $(test)_FAIL
name: pass lua_func with expected result FAIL
param:
- true
- lua_func:
expected_result: -1
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: expected -1
param:
- -1
- lua_func:
expected_result: 354848436 - 354848437
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: expected eval
param:
- -1
- lua_func:
expected_result: '[-1, ''a'', {''toto'': ''tata''}]'
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: expected table
param:
- - -1
- a
- toto: tata
- lua_func:
expected_result: $(lua_func test parameter)
file: $(test_path)$(psep)lua_func.lua
func_name: checkglobal
key: $(test)_PASS
name: global param lua_func
param:
- lua_func test parameter
- lua_func:
expected_result: ($(lua_data_to_be_returned))[0]
file: $(test_path)$(psep)lua_func.lua
func_name: checkglobal2
key: $(test)_PASS
name: global param lua_func 1
param:
- 1
- lua_func:
expected_result: ($(lua_data_to_be_returned))[1]
file: $(test_path)$(psep)lua_func.lua
func_name: checkglobal2
key: $(test)_PASS
name: global param lua_func 2
param:
- 2
- lua_func:
expected_result: ($(lua_data_to_be_returned))[2]
file: $(test_path)$(psep)lua_func.lua
func_name: checkglobal2
key: $(test)_PASS
name: global param lua_func 3
param:
- 3
- let:
key: $(test)_PASS
name: python2func
values:
- py: $(test_path)$(psep)lua_func.lua
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: should_not_be_called
name: skipped_checkglobal
param:
- $(test parameter)
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: checkglobal
name: skipped true
param:
- $(test parameter)
skipped: true
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: checkglobal
name: skipped 1
param:
- $(test parameter)
skipped: true
- group:
name: Function results check
steps:
- group:
name: Function result failure
steps:
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: int failure
param:
- -1
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: float failure
param:
- -1.3
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: String failure
param:
- FAIL
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: tuple_return
key: $(test)_PASS
name: Tuple int,str failure
param:
- -1
- Got a failure
- group:
name: Functions result success
steps:
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: int success
param:
- 0
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: float success
param:
- 0.3
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: echo
key: $(test)_PASS
name: String success
param:
- Something that is not only strictly FAIL
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: tuple_return
key: $(test)_PASS
name: Tuple int,str success
param:
- 0
- OK
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: test_delgd
key: $(test)_PASS
name: delgd test
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: return_nothing
key: $(test)_PASS
name: function returning nothing should succeed
- lua_func:
file: $(test_path)$(psep)lua_func.lua
func_name: return_explicit_nil
key: $(test)_PASS
name: function returning explicit nil should succeed
- group:
name: context_id tests
steps:
- lua_func:
context_id: lua_ctx_test
expected_result: hello lua
file: $(test_path)$(psep)lua_func.lua
func_name: set_context_value
key: $(test)_PASS
name: set context value
param:
- hello lua
- lua_func:
context_id: lua_ctx_test
expected_result: hello lua
file: $(test_path)$(psep)lua_func.lua
func_name: get_context_value
key: $(test)_PASS
name: get context value (same context_id)
- lua_func:
expected_result: hello lua
file: $(test_path)$(psep)lua_func.lua
func_name: get_context_value
key: $(test)_PASS
name: get context value (no context_id, from main
gd)
- lua_func:
context_id: lua_ctx_other
expected_result: hello lua
file: $(test_path)$(psep)lua_func.lua
func_name: get_context_value
key: $(test)_PASS
name: get context value (different context_id)
filename: /home/renish/workspace/testium/code/test/validation/items/lua_func/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

414
schema/test_schema/parallel.tum
View File

@@ -1,414 +0,0 @@
config_file:
- param.yaml
- items/parallel/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: parallel test constants
values:
test: parallel
test_path: items/$(test)
- group:
name: parallel test
steps:
- sequence:
steps:
- parallel:
branches:
- name: Branch A
steps:
- let:
name: Set A done
values:
- branch_a_done: true
- name: Branch B
steps:
- let:
name: Set B done
values:
- branch_b_done: true
key: $(test)_PASS
name: Both branches pass
sync: all
- check:
key: $(test)_PASS
name: Both branches ran
values:
- <| $(branch_a_done) == True |>
- <| $(branch_b_done) == True |>
- parallel:
branches:
- name: Pass branch
steps:
- let:
name: Set pass flag
values:
- pass_branch_ran: true
- name: Fail branch
steps:
- py_func:
expected_result: fail
file: $(test_path)$(psep)parallel.py
func_name: sleep_func
name: Raise exception
param:
- 0
key: $(test)_PASS
name: One branch fails
no_fail: true
sync: all
- check:
key: $(test)_PASS
name: Pass branch still ran
values:
- <| $(pass_branch_ran) == True |>
- let:
name: Reset slow flag
values:
- slow_done: false
- parallel:
branches:
- name: Fast branch
steps:
- let:
name: Fast done
values:
- fast_done: true
- name: Slow branch
steps:
- py_func:
file: $(test_path)$(psep)parallel.py
func_name: sleep_func
name: Sleep 2s
param:
- 2
- let:
name: Slow done
values:
- slow_done: true
key: $(test)_PASS
name: sync any - first wins
sync: any
- check:
key: $(test)_PASS
name: Fast branch ran, slow branch was stopped
values:
- <| $(fast_done) == True |>
- <| $(slow_done) == False |>
- let:
name: Reset sync flag
values:
- sync_flag: ''
- waiter_ran: false
- parallel:
branches:
- name: Setter branch
steps:
- py_func:
file: $(test_path)$(psep)parallel.py
func_name: sleep_func
name: Sleep 0.3s then set flag
param:
- 0.3
- let:
name: Set sync flag
values:
- sync_flag: ready
- name: Waiter branch
steps:
- let:
name: Got flag
values:
- waiter_ran: true
wait_for:
condition: <| "$(sync_flag)" == "ready" |>
timeout: 10
key: $(test)_PASS
name: wait_for synchronization
sync: all
- check:
key: $(test)_PASS
name: Waiter branch ran after flag was set
values:
- <| $(waiter_ran) == True |>
- parallel:
branches:
- name: Sleep A
steps:
- sleep:
name: Sleep 1s A
timeout: 1
- name: Sleep B
steps:
- sleep:
name: Sleep 1s B
timeout: 1
key: $(test)_PASS
name: Timing test
sync: all
- let:
name: Capture parallel duration
values:
- parallel_duration: $(ts_duration_Timing test)
- check:
key: $(test)_PASS
name: Duration < 1.8s (would be 2s if sequential)
values:
- <| float("$(parallel_duration)") < 1.8 |>
- let:
name: Reset N flags
values:
- n_a: false
- n_b: false
- n_c: false
- n_d: false
- parallel:
branches:
- name: NA
steps:
- let:
name: set n_a
values:
- n_a: true
- name: NB
steps:
- let:
name: set n_b
values:
- n_b: true
- name: NC
steps:
- let:
name: set n_c
values:
- n_c: true
- name: ND
steps:
- let:
name: set n_d
values:
- n_d: true
key: $(test)_PASS
name: Four branches
sync: all
- check:
key: $(test)_PASS
name: Four branches all set their flag
values:
- <| $(n_a) == True |>
- <| $(n_b) == True |>
- <| $(n_c) == True |>
- <| $(n_d) == True |>
- let:
name: Reset nested flags
values:
- outer_x: false
- inner_x_1: false
- inner_x_2: false
- parallel:
branches:
- name: Outer X
steps:
- let:
name: set outer_x
values:
- outer_x: true
- parallel:
branches:
- name: Inner X1
steps:
- let:
name: set inner_x_1
values:
- inner_x_1: true
- name: Inner X2
steps:
- let:
name: set inner_x_2
values:
- inner_x_2: true
name: Inner parallel
sync: all
- name: Outer Y
steps:
- sleep:
name: brief sleep
timeout: 0
key: $(test)_PASS
name: Outer parallel
sync: all
- check:
key: $(test)_PASS
name: Nested parallel set all flags
values:
- <| $(outer_x) == True |>
- <| $(inner_x_1) == True |>
- <| $(inner_x_2) == True |>
- let:
name: Reset waiter timeout flag
values:
- waiter_timeout_ran: false
- parallel:
branches:
- name: Quick branch
steps:
- sleep:
name: brief sleep
timeout: 0
- name: Doomed waiter
steps:
- let:
name: should not run
values:
- waiter_timeout_ran: true
wait_for:
condition: <| "never" == "ready" |>
timeout: 1
key: $(test)_PASS
name: wait_for timeout
no_fail: true
sync: all
- check:
key: $(test)_PASS
name: Doomed waiter never ran its steps
values:
- <| $(waiter_timeout_ran) == False |>
- parallel:
branches:
- name: ok branch
steps:
- let:
name: noop
values:
- noop_var: 1
- name: broken branch
steps:
- py_func:
expected_result: fail
file: $(test_path)$(psep)parallel.py
func_name: sleep_func
name: Forced fail
param:
- 0
key: $(test)_FAIL
name: One branch really fails
sync: all
- let:
name: Reset branch condition flag
values:
- cond_branch_ran: false
- other_branch_ran: false
- parallel:
branches:
- condition: <| "always" == "false" |>
name: Skipped branch
steps:
- let:
name: should not run
values:
- cond_branch_ran: true
- name: Other branch
steps:
- let:
name: ran
values:
- other_branch_ran: true
key: $(test)_PASS
name: Condition-skipped branch
sync: all
- check:
key: $(test)_PASS
name: Skipped condition branch did not run
values:
- <| $(cond_branch_ran) == False |>
- <| $(other_branch_ran) == True |>
- let:
name: Reset loop counters
values:
- loop_count_a: 0
- loop_count_b: 0
- loop:
iterator: 3
name: Loop wrapping parallel
steps:
- parallel:
branches:
- name: LA
steps:
- let:
name: bump A
values:
- loop_count_a: <| int("$(loop_count_a)")
+ 1 |>
- name: LB
steps:
- let:
name: bump B
values:
- loop_count_b: <| int("$(loop_count_b)")
+ 1 |>
name: Per-iteration parallel
sync: all
- check:
key: $(test)_PASS
name: Both branches ran 3 times
values:
- <| int("$(loop_count_a)") == 3 |>
- <| int("$(loop_count_b)") == 3 |>
filename: /home/renish/workspace/testium/code/test/validation/items/parallel/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

137
schema/test_schema/plot.tum
View File

@@ -1,137 +0,0 @@
config_file:
- param.yaml
- items/plot/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: plot test constants
values:
test: plot
test_path: items/$(test)
- group:
name: plot test
steps:
- sequence:
steps:
- group:
condition: <| $(validation_dialogs) and not tm.text_mode()
|>
name: Plot test
steps:
- plot:
key: $(test)_PASS
name: Open the plot
plot_name: Mon Plot
steps:
- open:
log_path: $(validation_report_path)
- plot:
key: $(test)_PASS
name: Add periodic to the plot
plot_name: Mon Plot
steps:
- periodic:
eval: '{"periodic": $(result)}'
file: $(test_path)$(psep)plot.py
func_name: random_value
period: 1
- sleep:
dialog: true
name: sleep
timeout: 3
- loop:
iterator: 10
name: Add of other data in the plot
steps:
- plot:
key: $(test)_PASS
name: Add to the plot
plot_name: Mon Plot
steps:
- add:
value1: $(loop_index)
value2: $(loop_index)+2
- sleep:
name: sleep between values
timeout: 1
- py_func:
file: $(test_path)$(psep)plot.py
func_name: LastValues
key: $(test)_PASS
name: last plot values
param:
- Mon Plot
- plot:
execute_on_stop: true
key: $(test)_PASS
name: Export
plot_name: Mon Plot
steps:
- export: $(validation_report_path)/plot_export.pdf
- export: $(validation_report_path)/plot_export.csv
- plot:
execute_on_stop: true
key: $(test)_PASS
name: Close the plot
plot_name: Mon Plot
steps:
- close:
timeout: 2
wait_dialog_exit: true
filename: /home/renish/workspace/testium/code/test/validation/items/plot/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

326
schema/test_schema/py_func.tum
View File

@@ -1,326 +0,0 @@
config_file:
- param.yaml
- items/py_func/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: py_func test constants
values:
test: py_func
test_path: items/$(test)
- group:
name: py_func test
steps:
- sequence:
steps:
- let:
name: py_func test constants,
values:
py_func test parameter: test parameter
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: assertparam
key: $(test)_PASS
name: pass py_func
param:
- true
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: assertparam
key: $(test)_FAIL
name: fail py_func
param:
- false
- py_func:
expected_result: FAIL
file: $(test_path)$(psep)py_func.py
func_name: assertparam
key: $(test)_PASS
name: fail py_func with expected result "FAIL"
param:
- false
- py_func:
expected_result: FAIL
file: $(test_path)$(psep)py_func.py
func_name: assertparam
key: $(test)_FAIL
name: pass py_func with expected result FAIL
param:
- true
- py_func:
expected_result: -1
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: expected -1
param:
- -1
- py_func:
expected_result: 354848436 - 354848437
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: expected eval
param:
- -1
- py_func:
expected_result: '[-1, ''a'', {''toto'': ''tata''}]'
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: expected table
param:
- - -1
- a
- toto: tata
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: checkglobal
key: $(test)_PASS
name: global param py_func
param:
- $(py_func test parameter)
- let:
key: $(test)_PASS
name: python2func
values:
- py: $(test_path)$(psep)py_func.py
- py_func:
expected_result: $(py_func test parameter)
file: $(py)
func_name: checkglobal2
key: $(test)_PASS
name: global param py_func 2
- py_func:
file: $(py)
func_name: checkglobal
key: $(test)_PASS
name: global param py_func
param:
- $(py_func test parameter)
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: should_not_be_called
name: skipped_checkglobal
param:
- $(py_func test parameter)
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: checkglobal
name: skipped true
param:
- $(py_func test parameter)
skipped: true
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: checkglobal
name: skipped 1
param:
- $(py_func test parameter)
skipped: 1
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: ValidationTest
name: FunctionItem test
param:
- $(py_func test parameter)
- group:
name: Function results check
steps:
- group:
name: Function result 1
steps:
- py_func:
expected_result: -1
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: int failure
param:
- -1
- py_func:
expected_result: -1.3
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: float failure
param:
- -1.3
- py_func:
expected_result: FAIL
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: String failure
param:
- FAIL
- py_func:
expected_result:
- -1
- Got a failure
file: $(test_path)$(psep)py_func.py
func_name: tuple_return
key: $(test)_PASS
name: Tuple int,str failure
param:
- -1
- Got a failure
- group:
name: Functions result 2
steps:
- py_func:
expected_result: 0
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: int success
param:
- 0
- py_func:
expected_result: 0.3
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: float success
param:
- 0.3
- py_func:
expected_result: Something that is not only
strictly FAIL
file: $(test_path)$(psep)py_func.py
func_name: echo
key: $(test)_PASS
name: String success
param:
- Something that is not only strictly FAIL
- py_func:
expected_result:
- 0
- OK
file: $(test_path)$(psep)py_func.py
func_name: tuple_return
key: $(test)_PASS
name: Tuple int,str success
param:
- 0
- OK
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: test_delgd
key: $(test)_PASS
name: delgd test
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: return_nothing
key: $(test)_PASS
name: function returning nothing should succeed
- py_func:
file: $(test_path)$(psep)py_func.py
func_name: return_explicit_none
key: $(test)_PASS
name: function returning explicit None should succeed
- group:
name: context_id tests
steps:
- py_func:
expected_result: hello context
file: $(test_path)$(psep)py_func.py
func_name: set_context_value
key: $(test)_PASS
name: set serializable value
param:
- hello context
- py_func:
context_id: ctx_test
expected_result: hello context
file: $(test_path)$(psep)py_func.py
func_name: get_context_value
key: $(test)_PASS
name: get serializable value (same context_id)
- py_func:
expected_result: hello context
file: $(test_path)$(psep)py_func.py
func_name: get_context_value
key: $(test)_PASS
name: get serializable value (no context_id, from
main gd)
- py_func:
context_id: ctx_other
expected_result: hello context
file: $(test_path)$(psep)py_func.py
func_name: get_context_value
key: $(test)_PASS
name: get serializable value (different context_id)
- py_func:
context_id: ctx_ns_test
expected_result: hello ns
file: $(test_path)$(psep)py_func.py
func_name: set_ns_value
key: $(test)_PASS
name: set non-serializable value
param:
- hello ns
- py_func:
context_id: ctx_ns_test
expected_result: hello ns
file: $(test_path)$(psep)py_func.py
func_name: get_ns_value
key: $(test)_PASS
name: get non-serializable value (same context_id)
filename: /home/renish/workspace/testium/code/test/validation/items/py_func/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

87
schema/test_schema/sleep.tum
View File

@@ -1,87 +0,0 @@
config_file:
- param.yaml
- items/sleep/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: sleep test constants
values:
test: sleep
test_path: items/$(test)
- group:
name: sleep test
steps:
- sequence:
steps:
- sleep:
condition: $(validation_dialogs)
dialog: true
key: $(test)_PASS
name: Sleep timeout with dialogs
timeout: 3
- sleep:
key: $(test)_PASS
name: Sleep timeout without dialog
timeout: 3.0
- sleep:
key: $(test)_PASS
name: Sleep timeout in textual format
skipped: true
timeout: 1h 3m 2s
filename: /home/renish/workspace/testium/code/test/validation/items/sleep/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

84
schema/test_schema/unittest.tum
View File

@@ -1,84 +0,0 @@
config_file:
- param.yaml
- items/unittest/param.yaml
main:
name: Testium validation suite
steps:
- group:
name: Test preparation
steps:
- let:
condition: <| "$(os)" == "Linux" |>
name: Set test variables for Linux
values:
- terminal_prompt: $(linux_prompt)
- psep: /
- let:
condition: <| "$(os)" == "Windows" |>
name: Set test variables for Windows
values:
- terminal_prompt: $(windows_prompt)
- psep: \
- group:
name: Group of tests
steps:
- let:
name: unittest test constants
values:
test: unittest
test_path: items/$(test)
- group:
name: unittest test
steps:
- sequence:
steps:
- unittest:
key: $(test)_PASS
name: Unittest item
test_file: /home/renish/workspace/testium/code/test/validation/items/unittest/unittest.py
test_method: test_01_pass
- unittest:
key: $(test)_FAIL
name: Unittest item
test_file: /home/renish/workspace/testium/code/test/validation/items/unittest/unittest.py
test_method:
- test_04_disabled
- test_03_fail
filename: /home/renish/workspace/testium/code/test/validation/items/unittest/test.tum
- sequence:
steps:
- report:
export:
- text:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.txt
- html:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.html
- junit:
key: $(test)_PASS
path: $(validation_report_path)$(psep)$(test)_PASS.junit
name: Expected PASS $(test) test
- report:
export:
- text:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.txt
- html:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.html
- junit:
key: $(test)_FAIL
path: $(validation_report_path)$(psep)$(test)_FAIL.junit
name: Expected FAIL $(test) test
filename: /home/renish/workspace/testium/code/test/validation/items/report.tum
report:
enabled: true
export:
junit:
file_name: $(validation_report_file).junit
path: $(validation_report_path)
sqlite:
file_name: $(validation_report_file).sqlite
path: $(validation_report_path)
log_stored: true

532
schema/tum.json
View File

@@ -1,532 +0,0 @@
{
"$schema": "https://json-schema.org/draft/2020-12/schema",
"$defs": {
"config_file":{
"desciption": "The list of the configuration files",
"type": "array",
"items": {
"type": "string"
}
},
"items":{
"properties": {
"name": { "type": "string" },
"stop_on_failure":{ "type": "boolean" },
"execute_on_stop":{ "type": "boolean" },
"skipped":{ "type": "boolean" },
"no_fail":{ "type": "boolean" },
"doc":{ "type": "string" },
"key":{ "type": "string" },
"report":{ "$ref": "#/$defs/report_export" },
"condition":{ "type": "string" },
"process_result":{ "type": "string" },
"expected_result":{ },
"store_result":{ "type": "string" }
}
},
"steps" : {
"required": ["steps"],
"properties": {
"version": { "type": "string" },
"steps": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"properties":{
"let": { "$ref": "#/$defs/let" },
"check": { "$ref": "#/$defs/check" },
"dialog_question": { "$ref": "#/$defs/dialog" },
"dialog_message": { "$ref": "#/$defs/dialog" },
"dialog_note": { "$ref": "#/$defs/dialog" },
"dialog_value": { "$ref": "#/$defs/dialog" },
"dialog_image": { "$ref": "#/$defs/dialog" },
"lua_func": { "$ref": "#/$defs/func" },
"py_func": { "$ref": "#/$defs/func" },
"console": { "$ref": "#/$defs/console" },
"sleep": { "$ref": "#/$defs/sleep" },
"json_rpc": { "$ref": "#/$defs/json_rpc" },
"group": { "$ref": "#/$defs/group" },
"sequence": { "$ref": "#/$defs/sequence" },
"report": { "$ref": "#/$defs/report" },
"loop": { "$ref": "#/$defs/loop" },
"git": { "$ref": "#/$defs/git" },
"unittest": { "$ref": "#/$defs/unittest" }
}
}
}
}
},
"cons_open" : {
"type":"object",
"allOf":
[
{ "$ref": "#/$defs/items" },
{
"properties": {
"protocol" : {
"enum": ["telnet", "ssh", "serial", "rawtcp", "terminal"]
},
"telnet_host" : {"type": "string" },
"telnet_port" : {"type": "number" },
"ssh_host": {"type": "string" },
"ssh_user": {"type": "string" },
"ssh_pwd": {"type": "string" },
"serial_port": {"type": "string" },
"serial_baudrate": {"type": "integer" },
"buffered": {"type": "boolean" },
"tcp_host" : {"type": "string" },
"tcp_port" : {"type": "number" },
"terminal_path": {"type":"string" },
"shell": {"type":"string" }
},
"required": ["protocol"]
}
]
},
"cons_read": {
"type":"object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties": {
"protocol" : {
"enum": ["telnet", "ssh", "serial", "rawtcp", "terminal"]
},
"expected" : {"type": "string" },
"timeout": {"type": "number" },
"mute": {"type": "boolean" }
},
"required":["expected"]
}
]
},
"console" : {
"description": "The let items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"required": ["steps", "console_name"],
"properties":{
"console_name": {"type":"string" },
"steps": {
"type": "array",
"items": {
"type": "object",
"properties":{
"open": {
"$ref": "#/$defs/cons_open"
},
"write": { "type": "string" },
"writeln": { "type": "string" },
"read_until": {
"$ref": "#/$defs/cons_read"
},
"close": {
"anyOf": [
{"type": "null"},
{"type": "string"}
]
}
},
"additionalProperties": false
}
}
}
}
]
},
"json_rpc_console": {
"type":"object",
"properties": {
"name" : {"type": "string" },
"prompt" : {"type": "string" }
},
"required":["name"],
"additionalProperties": false
},
"json_rpc_udp": {
"type":"object",
"properties": {
"server" : {"type": "string" },
"udp_snd_port" : {"type": "integer" },
"udp_rcv_port" : {"type": "integer" },
"bufsize" : {"type": "integer" }
},
"required":["name"],
"additionalProperties": false
},
"json_rpc_query": {
"type":"object",
"properties": {
"method" : { "type": "string" },
"params" : { "type": "array" },
"id" : {"type": "integer" },
"no_wait" : {"type": "boolean" }
},
"required":["method"],
"additionalProperties": false
},
"json_rpc_receive": {
"type":"object",
"properties": {
"id" : {"type": "integer" },
"timeout" : {"type": "number" }
},
"required":["id"],
"additionalProperties": false
},
"json_rpc" : {
"description": "The json_rpc items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"required": ["steps"],
"properties":{
"udp" : { "$ref": "#/$defs/json_rpc_udp" },
"console" : { "$ref": "#/$defs/json_rpc_console" },
"timeout": {"type": "number" },
"version": {"enum": ["1.0", "2.0"]},
"steps": {
"type": "array",
"items": {
"type": "object",
"additionalProperties": false,
"properties":{
"open": { "type": "null" },
"query": { "$ref": "#/$defs/json_rpc_query" },
"receive": { "$ref": "#/$defs/json_rpc_receive" },
"writeln": { "type": "string" },
"read_until": { "$ref": "#/$defs/cons_read"
},
"close": { "type": "null" }
}
}
}
}
}
]
},
"group" : {
"description": "The group items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{ "$ref": "#/$defs/steps" }
]
},
"sequence" : {
"description": "The sequence items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{ "$ref": "#/$defs/steps" },
{
"properties": {
"filename": { "type": "string" }
},
"required": ["filename"]
}
]
},
"loop" : {
"description": "The loop items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{ "$ref": "#/$defs/steps" },
{
"properties": {
"iterator": { },
"exit_condition": {
"description": "the posibility to stop the loop",
"type": "object",
"additionalProperties": false,
"properties": {
"time" : {"type":"number"},
"value" : {"type":"number"},
"file" : {"type":"string"},
"func_name" : {"type":"string"},
"eval" : {"type":"string"}
},
"dependentRequired": {
"file" : ["func_name"]
}
}
}
}
]
},
"main" : {
"description": "The Main items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{ "$ref": "#/$defs/steps" },
{
"properties": {
"version": { "type": "string" }
}
}
]
},
"sleep" : {
"description": "Sleep for X time [secondes",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties":{
"dialog": { "type": "boolean" },
"timeout": { "type": "number" }
},
"required": ["timeout"]
}
]
},
"let" : {
"description": "The let items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties":{
"values": {
"anyOf": [
{
"type": "array",
"items": {"type": "object" }
},
{ "type":"object" }
]
}
},
"required": ["values"]
}
]
},
"check" : {
"description": "The let items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties":{
"values": {
"type": "array",
"items": {"type": "string" }
}
},
"required": ["values"]
}
]
},
"git" : {
"description": "The git items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties":{
"repo": { "type": "string" }
},
"required": ["repo"]
}
]
},
"unittest" : {
"description": "The unittest items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties":{
"test_file": { "type": "string" },
"test_method": {
"anyOf": [
{"type":"string"},
{
"type":"array",
"items": {"type":"string"}
}
]
}
},
"required": ["test_file"]
}
]
},
"dialog" : {
"description": "The let items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties":{
"question": { "type": "string" },
"auto_result": { "type": "string" },
"auto_value": { "type": ["number", "string"] },
"filename" : {"type": "string"},
"reference" : {
"type":"array",
"items": { "type":"string"}
}
},
"required": ["question"]
}
]
},
"dialog_choice" : {
"description": "The one choi items",
"type": "object",
"additionalProperties": false,
"properties":{
"name": { "type": "string" },
"description": { "type": "string" },
"icon": { "type": "string"},
"choices": {
"type":"array",
"items": { "$ref": "#/$defs/dialog_choice" }
}
},
"required": ["name", "description"]
},
"dialog_choices" : {
"description": "The dialog choices items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties":{
"name": { "type": "string" },
"question": { "type": "string" },
"icon": { "type": "string"},
"choices": {
"type":"array",
"items": { "$ref": "#/$defs/dialog_choice" }
}
},
"required": ["question"]
}
]
},
"func" : {
"description": "The py_fun and lua_func items",
"type": "object",
"unevaluatedProperties": false,
"allOf": [
{ "$ref": "#/$defs/items" },
{
"properties":{
"file": { "type": "string" },
"func_name": { "type": "string" },
"context_id": { "type": "string" },
"param" : {
"anyOf": [
{"type": "array"},
{"type": "object"}
]
}
},
"required": ["file", "func_name"]
}
]
},
"report_export": {
"type": "object",
"properties": {
"path": {"type":"string" },
"file_name": {"type":"string" },
"pattern": {"type":"string" },
"key": {"type":"string" }
}
},
"report":{
"desciption": "The list of the configuration files",
"type": "object",
"additional_properties":false,
"properties": {
"enabled": {"type":"boolean" },
"log_stored": {"type":"boolean" },
"export": {
"anyOf": [
{
"type": "object",
"properties":{
"html": { "$ref": "#/$defs/report_export" },
"sqlite": { "$ref": "#/$defs/report_export" },
"junit": { "$ref": "#/$defs/report_export" }
}
},
{
"type": "array",
"items": {
"type":"object",
"properties":{
"html": { "$ref": "#/$defs/report_export" },
"sqlite": { "$ref": "#/$defs/report_export" },
"junit": { "$ref": "#/$defs/report_export" }
}
}
}
]
}
}
}
},
"type": "object",
"properties": {
"config_file" : { "$ref": "#/$defs/config_file" },
"main": { "$ref": "#/$defs/main" },
"report" : { "$ref": "#/$defs/report" }
},
"required" : ["main"],
"additionalProperties": false
}

14
src/testium/interpreter/process.py
View File

@@ -16,6 +16,7 @@ from interpreter.utils.test_init import (
env_init,
prepare_global,
update_global,
apply_overrides,
set_standard_gd_keys,
test_run_init,
test_run_header,
@@ -210,6 +211,19 @@ class TestProcess(Process):
env_init()
# Apply GUI defaults and CLI defines to the global dict
# *before* eval_proc starts: bins.python_bin() reads
# ``python_bin`` from gd on its very first call (during
# eval_process_init) and caches the result. Without this,
# ``-d python_bin=...`` and the GUI ``python_bin`` preference
# would only take effect for items spawned *after* the cache
# was already populated with the auto-discovered interpreter,
# i.e. they would silently be ignored for eval_proc itself.
# _load_initial_params re-applies the same overrides after
# ``prepare_global()`` clears gd, so the gd value stays in
# sync with the cached path.
apply_overrides(self.__defs, self.__gui_defaults)
# Creation of the python evaluation process for loading of the complete test
eval_proc = eval_process_init(api_request, 10, test_dir)
eval_proc.start()

16
src/testium/interpreter/utils/bins.py
View File

@@ -202,16 +202,24 @@ _SPECS = {
"lua": ("Lua 5.1+", "lua_bin", _LUA_CANDIDATES, _is_lua51),
}
# Cached per (name, override) so that runtime changes to gd[gd_key] —
# e.g. ``python_bin`` set from a YAML config file loaded *after*
# eval_proc has already resolved its own interpreter — are picked up by
# the next lookup instead of returning the stale, auto-discovered path.
# Long-lived subprocesses (eval_proc) keep whatever they captured at
# construction time, but every new PyProcessBase / FuncExecEngine spawned
# afterwards sees the current override.
_resolved = {}
def _resolve(name):
if name in _resolved:
return _resolved[name]
display, gd_key, candidates, validator = _SPECS[name]
override = tm.gd(gd_key, "") or ""
cached = _resolved.get(name)
if cached is not None and cached[0] == override:
return cached[1]
path = ""
if override:
# Absolute path: accept as-is (user knows exactly what they want).
@@ -239,7 +247,7 @@ def _resolve(name):
path = p
break
_resolved[name] = path
_resolved[name] = (override, path)
return path

20
src/testium/interpreter/utils/test_init.py
View File

@@ -165,11 +165,14 @@ def env_init():
_constants_init()
def update_global(config_files, defines, gui_defaults, silent=False):
"""Global dict updated with the content of the config file and a dict provided.
this function returns the resulting dict.
def apply_overrides(defines, gui_defaults):
"""Push GUI defaults then CLI defines into the global dict.
Extracted from update_global so it can be called *before* eval_proc
starts: interpreter overrides (python_bin, lua_bin) must be visible
to bins.python_bin() on its first lookup, which happens during
eval_process_init.
"""
# GUI preferences applied first
for k, v in gui_defaults.items():
try:
val = ast.literal_eval(v)
@@ -177,7 +180,6 @@ def update_global(config_files, defines, gui_defaults, silent=False):
val = v
tm.setgd(k, val)
# Then command line defines
for k, v in defines.items():
try:
val = ast.literal_eval(v)
@@ -185,6 +187,14 @@ def update_global(config_files, defines, gui_defaults, silent=False):
val = v
tm.setgd(k, val)
def update_global(config_files, defines, gui_defaults, silent=False):
"""Global dict updated with the content of the config file and a dict provided.
this function returns the resulting dict.
"""
# GUI preferences applied first, then command line defines
apply_overrides(defines, gui_defaults)
# Then the configuration files
# load global dic before test item
_feed_gd_with_params(config_files, silent)

21
src/testium/main_win/file_dialog.py Normal file
View File

@@ -0,0 +1,21 @@
"""Helpers for Qt file/directory dialogs.
In Flatpak the native QFileDialog goes through the XDG document portal,
which returns ``/run/user/UID/doc/.../<file>`` and only exposes the
selected file — sibling files (param.yaml, scripts, recent paths in
preferences, ...) are unreachable. Forcing Qt's own non-native dialog
makes it walk the real filesystem mounted via ``--filesystem=home``
and return a regular path.
"""
import os
from PySide6.QtWidgets import QFileDialog
def options():
"""Default ``QFileDialog`` options for the current runtime."""
opts = QFileDialog.Options()
if os.path.isfile("/.flatpak-info"):
opts |= QFileDialog.Option.DontUseNativeDialog
return opts

9
src/testium/main_win/preference_win/preference_win.py
View File

@@ -3,6 +3,7 @@ from PySide6.QtWidgets import QDialog, QFileDialog
from PySide6.QtGui import QFont
from main_win.preference_win.preference_core_win import Ui_preferenceWindow
from main_win import file_dialog
import interpreter.utils.settings as prefs
@@ -193,6 +194,7 @@ class PrefWindow(QDialog):
self,
caption="Select the default report directory",
dir=self.ui.editDefaultReportPath.text(),
options=file_dialog.options(),
)
if path:
self.ui.editDefaultReportPath.setText(path)
@@ -203,6 +205,7 @@ class PrefWindow(QDialog):
self,
caption="Select the default log directory",
dir=self.ui.editDefaultLogPath.text(),
options=file_dialog.options(),
)
if path:
self.ui.editDefaultLogPath.setText(path)
@@ -213,6 +216,7 @@ class PrefWindow(QDialog):
self,
caption="Select the python interpreter",
dir=self.ui.editPythonPath.text(),
options=file_dialog.options(),
)
if path:
self.ui.editPythonPath.setText(path)
@@ -220,7 +224,10 @@ class PrefWindow(QDialog):
@Slot()
def on_butLuaPath_pressed(self):
path, _ = QFileDialog.getOpenFileName(
self, caption="Select the lua interpreter", dir=self.ui.editLuaPath.text()
self,
caption="Select the lua interpreter",
dir=self.ui.editLuaPath.text(),
options=file_dialog.options(),
)
if path:
self.ui.editLuaPath.setText(path)

11
src/testium/main_win/test_file_manager.py
View File

@@ -9,6 +9,7 @@ from PySide6.QtWidgets import QApplication, QFileDialog, QProgressDialog
from interpreter.process import TestProcess
from interpreter.utils.test_ctrl import TestSetController
from main_win.test_controller_service import TestControllerService
from main_win import file_dialog
import interpreter.utils.settings as prefs
from runtime.tum_except import ETUMFileError, ETUMRuntimeError
@@ -212,17 +213,9 @@ class TestFileManager:
d = ""
if w.testFile is not None:
d = os.path.dirname(w.testFile)
# In Flatpak the native dialog goes through the XDG document portal,
# which returns /run/user/UID/doc/.../test.tum and only exposes the
# selected file — sibling files (param.yaml, .py, etc.) are unreachable.
# Force Qt's own dialog, which walks the real filesystem mounted via
# --filesystem=home and returns a regular path with sibling access.
options = QFileDialog.Options()
if os.path.isfile("/.flatpak-info"):
options |= QFileDialog.Option.DontUseNativeDialog
file_name, _ = QFileDialog.getOpenFileName(
w, "Open the test file", d,
"testium file (*.tum);;All Files (*)", options=options
"testium file (*.tum);;All Files (*)", options=file_dialog.options()
)
if file_name:
self.reload(file_name)

7
src/testium/main_win/testium_win.py
View File

@@ -37,6 +37,7 @@ from interpreter.utils.icons import icon_prefix
from main_win.test_run.outlog import OutLog
from main_win.test_run.test_run import ThreadTestStatus
from main_win import file_dialog
import interpreter.utils.settings as prefs
from runtime.stdout_redirect import stdio_redir
import api.testium as tm
@@ -484,7 +485,8 @@ class MainWindow(QMainWindow, Ui_MainWindow):
else:
initialPath = None
fileName, _ = QFileDialog.getSaveFileName(
self, "Path to Log file", initialPath, "Log Files (*.log);;All Files (*)"
self, "Path to Log file", initialPath, "Log Files (*.log);;All Files (*)",
options=file_dialog.options(),
)
if fileName:
shutil.copy(self.logFileName, fileName)
@@ -525,7 +527,8 @@ class MainWindow(QMainWindow, Ui_MainWindow):
else:
initialPath = None
fileName, _ = QFileDialog.getSaveFileName(
self, "Path to log file", initialPath, "Log Files (*.log);;All Files (*)"
self, "Path to log file", initialPath, "Log Files (*.log);;All Files (*)",
options=file_dialog.options(),
)
if fileName:
self.editLogFilePath.setText(fileName)

43
test/validation/README.md
View File

@@ -1,10 +1,43 @@
# Validation
This directory contains the necessary material to run the testium validation.
This directory contains the testium validation suite.
Here is the documentation on how to configure the validation, run it and check that the
results are correct.
## Running the suite
# Tests
```sh
./test/validation/run.sh # Linux
test\validation\run.bat # Windows
```
TBD
The wrapper creates a dedicated Python venv in the system temp dir
(`${TMPDIR:-/tmp}/testium-validation-venv` on Linux, `%TEMP%\testium-validation-venv`
on Windows), using `--system-site-packages` so existing system packages
stay visible. The validation suite is then run with that venv pinned as
`python_bin`. Every test-execution subprocess (inline `<| ... |>`
evaluation, `py_func`, `cycle`, `post_execution`, ...) runs inside the
venv, while testium itself keeps running in the project's own
environment.
Pass `clean` as the first argument to recreate the venv from scratch
(useful after a system Python upgrade):
```sh
./test/validation/run.sh clean
```
## What is checked
The `venv` item under `items/venv/` asserts that the venv is actually
being used:
* `python_bin` is set in the global dict.
* The eval subprocess (used for `<| ... |>` expressions) has
`sys.executable == python_bin`, `sys.prefix == dirname(dirname(python_bin))`,
and `sys.prefix != sys.base_prefix` (i.e. is actually inside a venv).
* A `py_func` subprocess passes the same three checks.
These checks use `abspath`/`normpath` rather than `realpath` on
purpose: the venv's `bin/python3` is a symlink to the host interpreter,
so `realpath` would map both venv and non-venv interpreters to the same
target. `sys.prefix != sys.base_prefix` is the venv-specific marker
that distinguishes the two cases.

1
test/validation/items/venv/param.yaml Normal file
View File

@@ -0,0 +1 @@
no_param: Null

53
test/validation/items/venv/test.tum Normal file
View File

@@ -0,0 +1,53 @@
# venv test: assert that the dedicated validation venv is the python
# being used for every test-execution subprocess (eval_proc / py_func /
# cycle / ...). The venv path is pinned by ``-d python_bin=...`` in
# test/validation/run.sh (or run.bat).
#
# We use ``abspath``/``normpath`` rather than ``realpath`` on purpose:
# the venv's ``bin/python3`` is a symlink to the host python, so
# realpath would map every venv interpreter to the same system path and
# the comparison would silently pass even *without* a venv.
# ``sys.prefix != sys.base_prefix`` is the venv-specific marker that
# catches that case.
- check:
name: python_bin is set in the global dict
key: $(test)_PASS
values:
- <| bool(r"$(python_bin)") |>
- check:
name: eval_proc subprocess runs in the validation venv
key: $(test)_PASS
values:
- <| os.path.normpath(os.path.abspath(sys.executable)) == os.path.normpath(os.path.abspath(r"$(python_bin)")) |>
- check:
name: eval_proc sys.prefix matches python_bin venv root
key: $(test)_PASS
values:
- <| os.path.normpath(os.path.abspath(sys.prefix)) == os.path.dirname(os.path.dirname(os.path.normpath(os.path.abspath(r"$(python_bin)")))) |>
- check:
name: eval_proc is actually inside a venv (sys.prefix != sys.base_prefix)
key: $(test)_PASS
values:
- <| os.path.normpath(os.path.abspath(sys.prefix)) != os.path.normpath(os.path.abspath(sys.base_prefix)) |>
- py_func:
name: py_func subprocess runs in the validation venv
key: $(test)_PASS
file: $(test_path)$(psep)verify_venv.py
func_name: check_sys_executable
- py_func:
name: py_func sys.prefix matches python_bin venv root
key: $(test)_PASS
file: $(test_path)$(psep)verify_venv.py
func_name: check_sys_prefix_in_venv
- py_func:
name: py_func is actually inside a venv
key: $(test)_PASS
file: $(test_path)$(psep)verify_venv.py
func_name: check_is_venv

62
test/validation/items/venv/verify_venv.py Normal file
View File

@@ -0,0 +1,62 @@
import os
import sys
import py_func.tm as tm
def _norm(p):
# normpath + normcase, without resolving symlinks. realpath() would
# follow the venv's ``python3`` symlink to ``/usr/bin/python3.X`` and
# defeat the comparison.
return os.path.normcase(os.path.normpath(os.path.abspath(p)))
def _venv_dir():
# python_bin is at ``<venv>/(bin|Scripts)/python*`` so the venv root
# is two levels above the executable.
exe = tm.gd("python_bin", "")
if not exe:
return ""
return os.path.dirname(os.path.dirname(_norm(exe)))
def check_sys_executable():
"""py_func subprocess: sys.executable must match the configured python_bin."""
expected = _norm(tm.gd("python_bin", ""))
actual = _norm(sys.executable)
if expected and actual == expected:
return True
return (
-1,
f"sys.executable={actual!r} differs from python_bin={expected!r}",
)
def check_sys_prefix_in_venv():
"""py_func subprocess: sys.prefix must match the venv root derived
from python_bin (two levels up from the executable)."""
venv = _venv_dir()
if not venv:
return (-1, "python_bin is not set in the global dict")
actual = _norm(sys.prefix)
if actual == venv:
return True
return (
-1,
f"sys.prefix={actual!r} is not the validation venv {venv!r}",
)
def check_is_venv():
"""py_func subprocess: confirm we are inside a venv, i.e. sys.prefix
differs from sys.base_prefix. This catches the case where python_bin
happens to be a system interpreter and the path-based check would
pass trivially."""
actual = _norm(sys.prefix)
base = _norm(sys.base_prefix)
if actual != base:
return True
return (
-1,
f"sys.prefix == sys.base_prefix == {actual!r}: not running in a venv",
)

61
test/validation/run.bat Normal file
View File

@@ -0,0 +1,61 @@
@echo off
SETLOCAL EnableExtensions
REM Runs the testium validation suite with a dedicated Python venv used
REM by every py_func / cycle / inline-eval subprocess. testium itself
REM keeps running in the project's own environment; the validation venv
REM only isolates *test execution*.
REM
REM test\validation\run.bat [clean] [extra testium args]
REM
REM Requires the project venv to already exist (run the project's
REM run.bat once first, or any other testium install method).
SET "SCRIPT_DIR=%~dp0"
SET "PROJECT_DIR=%SCRIPT_DIR%..\.."
REM Venv in the user temp dir (Windows equivalent of /tmp).
SET "VENV_DIR=%TEMP%\testium-validation-venv"
SET "PROJECT_VENV=%PROJECT_DIR%\test\tmp\testium_venv"
IF /I "%~1"=="clean" (
rmdir /s /q "%VENV_DIR%"
SHIFT
)
REM Locate a host Python.
SET "PYTHON_EXE=python"
py --version >nul 2>&1
IF %ERRORLEVEL% EQU 0 (
SET "PYTHON_EXE=py"
goto :PYTHON_FOUND
)
python --version >nul 2>&1
IF %ERRORLEVEL% EQU 0 (
SET "PYTHON_EXE=python"
goto :PYTHON_FOUND
)
echo ERROR : Python could not be found on this system.
exit /b 1
:PYTHON_FOUND
IF NOT EXIST "%VENV_DIR%" (
echo Creating validation venv at %VENV_DIR%
%PYTHON_EXE% -m venv --system-site-packages "%VENV_DIR%"
IF %ERRORLEVEL% NEQ 0 (
echo ERROR while creating the validation venv.
exit /b 1
)
call "%VENV_DIR%\Scripts\pip" install --quiet --upgrade pip
call "%VENV_DIR%\Scripts\pip" install --quiet junit-xml
)
SET "VENV_PYTHON=%VENV_DIR%\Scripts\python.exe"
IF NOT EXIST "%PROJECT_VENV%" (
echo ERROR : project venv not found at %PROJECT_VENV%. Run the project run.bat once first.
exit /b 1
)
call "%PROJECT_VENV%\Scripts\activate"
python "%PROJECT_DIR%\src\testium" -b -d "python_bin=%VENV_PYTHON%" -- "%SCRIPT_DIR%main.tum" %*

47
test/validation/run.sh Executable file
View File

@@ -0,0 +1,47 @@
#!/bin/bash
# Runs the testium validation suite with a dedicated Python venv used by
# every py_func / cycle / inline-eval subprocess (i.e. everything that
# goes through ``bins.python_bin()``). testium itself keeps running in
# the project's own environment — the validation venv only isolates
# *test execution*.
#
# ./test/validation/run.sh [clean] [extra testium args]
#
# ``clean`` (optional, must be the first arg) removes the venv before
# recreating it; this is the way to refresh the venv after a system
# Python upgrade.
set -e
SCRIPT_PATH="$(readlink -f "$0")"
SCRIPT_DIR="$(realpath "$(dirname "$SCRIPT_PATH")")"
PROJECT_DIR="$(realpath "$SCRIPT_DIR/../..")"
# Venv lives in the system temp dir so it stays out of the project tree
# (and is naturally cleaned up by tmpfiles/reboot on most distros).
VENV_DIR="${TMPDIR:-/tmp}/testium-validation-venv"
if [ "${1:-}" = "clean" ]; then
rm -rf "$VENV_DIR"
shift
fi
if [ ! -d "$VENV_DIR" ]; then
echo "Creating validation venv at $VENV_DIR"
# --system-site-packages so we don't have to reinstall pyside6, lxml
# & friends just to support the validation helpers. We still pip
# install junit-xml below because it is the one dep that does *not*
# ship as a system package on most distros and is required by
# post_execution.py.
python3 -m venv --system-site-packages "$VENV_DIR"
"$VENV_DIR/bin/pip" install --quiet --upgrade pip
"$VENV_DIR/bin/pip" install --quiet junit-xml
fi
VENV_PYTHON="$VENV_DIR/bin/python3"
# Delegate to the project's run.sh so testium itself still runs in the
# project venv (with pyside6, gitpython, ...). ``-d python_bin=...``
# pins every test-execution subprocess to the validation venv.
exec "$PROJECT_DIR/run.sh" -b \
-d "python_bin=$VENV_PYTHON" \
-- "$SCRIPT_DIR/main.tum" "$@"