A CLI tool to render and execute Robot Framework tests using Jinja templating. Combining Robot's language agnostic syntax with the flexibility of Jinja templating allows dynamically rendering a set of test suites from the desired infrastructure state expressed in YAML syntax.
$ nac-test --help
Usage: nac-test [OPTIONS]
A CLI tool to render and execute Robot Framework tests using Jinja templating.
Additional Robot Framework options can be passed at the end of the command to
further control test execution (e.g., --variable, --listener, --loglevel).
These are appended to the pabot invocation. Pabot-specific options and test
files/directories are not supported and will result in an error.
╭─ Options ────────────────────────────────────────────────────────────────────╮
│ * --data -d PATH Path to data YAML files. │
│ [env var: NAC_TEST_DATA] │
│ [required] │
│ * --templates -t DIRECTORY Path to test templates. │
│ [env var: │
│ NAC_TEST_TEMPLATES] │
│ [required] │
│ * --output -o DIRECTORY Path to output directory. │
│ [env var: │
│ NAC_TEST_OUTPUT] │
│ [required] │
│ --filters -f DIRECTORY Path to Jinja filters. │
│ [env var: │
│ NAC_TEST_FILTERS] │
│ --tests DIRECTORY Path to Jinja tests. │
│ [env var: NAC_TEST_TESTS] │
│ --include -i TEXT Selects the test cases by │
│ tag (include). │
│ [env var: │
│ NAC_TEST_INCLUDE] │
│ --exclude -e TEXT Selects the test cases by │
│ tag (exclude). │
│ [env var: │
│ NAC_TEST_EXCLUDE] │
│ --processes INTEGER Number of parallel │
│ processes for test │
│ execution (pabot │
│ --processes option), │
│ default is max(2, cpu │
│ count). │
│ [env var: │
│ NAC_TEST_PROCESS] |
| --render-only Only render tests without │
│ executing them. │
│ [env var: │
│ NAC_TEST_RENDER_ONLY] │
│ --dry-run Dry run flag. See robot │
│ dry run mode. │
│ [env var: │
│ NAC_TEST_DRY_RUN] │
│ --verbosity -v [DEBUG|INFO|WARNING|ERR Verbosity level. │
│ OR|CRITICAL] [env var: │
│ NAC_VALIDATE_VERBOSITY] │
│ [default: WARNING] │
│ --version Display version number. │
│ --help Show this message and │
│ exit. │
╰──────────────────────────────────────────────────────────────────────────────╯
All data from the YAML files (--data option) will first be combined into a single data structure which is then provided as input to the templating process. Each template in the --templates path will then be rendered and written to the --output path. If the --templates path has subfolders, the folder structure will be retained when rendering the templates.
After all templates have been rendered Pabot will execute all test suites in parallel and create a test report in the --output path. The --skiponfailure non-critical argument will be used by default, meaning all failed tests with a non-critical tag will show up as "skipped" instead of "failed" in the final test report.
Python 3.10+ is required to install nac-test. Don't have Python 3.10 or later? See Python 3 Installation & Setup Guide.
nac-validate can be installed in a virtual environment using pip or uv:
# Using pip
pip install nac-test
# Using uv (recommended)
uv tools install nac-testThe following Robot libraries are included with nac-test:
- RESTinstance
- robotframework-requests
- robotframework-jmespath
- robotframework-jsonlibrary
- robotframework-pabot for parallel test execution
Any other libraries can of course be added via pip or uv.
Values in YAML files can be encrypted using Ansible Vault. This requires Ansible (ansible-vault command) to be installed and the following two environment variables to be defined:
export ANSIBLE_VAULT_ID=dev
export ANSIBLE_VAULT_PASSWORD=Password123
ANSIBLE_VAULT_ID is optional, and if not defined will be omitted.
The !env YAML tag can be used to read values from environment variables.
root:
name: !env VAR_NAMEdata.yaml located in ./data folder:
---
root:
children:
- name: ABC
param: value
- name: DEF
param: valuetest1.robot located in ./templates folder:
*** Settings ***
Documentation Test1
*** Test Cases ***
{% for child in root.children | default([]) %}
Test {{ child.name }}
Should Be Equal {{ child.param }} value
{% endfor %}
After running nac-test with the following parameters:
nac-test --data ./data --templates ./templates --output ./testsThe following rendered Robot test suite can be found in the ./tests folder:
*** Settings ***
Documentation Test1
*** Test Cases ***
Test ABC
Should Be Equal value value
Test DEF
Should Be Equal value value
As well as the test results and reports:
$ tree -L 1 tests
tests
├── log.html
├── output.xml
├── pabot_results
├── report.html
├── test1.robot
└── xunit.xmlCustom Jinja filters can be used by providing a set of Python classes where each filter is implemented as a separate Filter class in a .py file located in the --filters path. The class must have a single attribute named name, the filter name, and a classmethod() named filter which has one or more arguments. A sample filter can be found below.
class Filter:
name = "filter1"
@classmethod
def filter(cls, data):
return str(data) + "_filtered"Custom Jinja tests can be used by providing a set of Python classes where each test is implemented as a separate Test class in a .py file located in the --tests path. The class must have a single attribute named name, the test name, and a classmethod() named test which has one or more arguments. A sample test can be found below.
class Test:
name = "test1"
@classmethod
def test(cls, data1, data2):
return data1 == data2Special rendering directives exist to render a single test suite per (YAML) list item. The directive can be added to the Robot template as a Jinja comment following this syntax:
{# iterate_list <YAML_PATH_TO_LIST> <LIST_ITEM_ID> <JINJA_VARIABLE_NAME> #}
After running nac-test with the data from the previous example and the following template:
{# iterate_list root.children name child_name #}
*** Settings ***
Documentation Test1
*** Test Cases ***
{% for child in root.children | default([]) %}
{% if child.name == child_name %}
Test {{ child.name }}
Should Be Equal {{ child.param }} value
{% endif %}
{% endfor %}
The following test suites will be rendered:
$ tree -L 2 tests
tests
├── ABC
│ └── test1.robot
└── DEF
└── test1.robotA similar directive exists to put the test suites in a common folder though with a unique filename.
{# iterate_list_folder <YAML_PATH_TO_LIST> <LIST_ITEM_ID> <JINJA_VARIABLE_NAME> #}
The following test suites will be rendered:
$ tree -L 2 tests
tests
└── test1
├── ABC.robot
└── DEF.robotAn additional directive exists to render a single test suite per (YAML) list item in chunks, which is useful for handling large datasets by splitting them across multiple template files. This is a variant of iterate_list that would still create separate folders.
Note: This directive is experimental and may change in future versions. It is not subject to semantic versioning guarantees.
{# iterate_list_chunked <YAML_PATH_TO_LIST> <LIST_ITEM_ID> <JINJA_VARIABLE_NAME> <OBJECT_PATH> <CHUNK_SIZE> #}
All objects under the OBJECT_PATH will be counted and if their number is greater than the specified chunk size, the list will be split into multiple test suites with suffix _2, _3, etc.
Consider the following example:
---
root:
children:
- name: ABC
param: value
nested_children:
- name: Child1
param: value
- name: Child2
param: value
- name: Child3
param: value
- name: DEF
param: value
nested_children:
- name: Child1
param: valueAfter running nac-test with this data from the previous and the following template:
{# iterate_list_chunked root.children name child_name nested_children 2 #}
*** Settings ***
Documentation Test1
*** Test Cases ***
{% for child in root.children | default([]) %}
{% if child.name == child_name %}
Test {{ child.name }}
Should Be Equal {{ child.param }} value
{% for nested_child in child.nested_children | default([]) %}
Test {{ child.name }} Child {{ nested_child.name }}
Should Be Equal {{ nested_child.param }} value
{% endfor %}
{% endif %}
{% endfor %}
Objects from the nested_children path will be counted and if their number is greater than the specified chunk size (2), the list will be split into multiple test suites with suffix _002, _003, etc. The following test suites will be rendered:
$ tree -L 2 tests
tests
├── ABC
│ ├── test1_001.robot
│ └── test1_002.robot
└── DEF
└── test1_001.robotIt is possible to include and exclude test cases by tag names with the --include and --exclude CLI options. These options are directly passed to the Pabot/Robot executor and are documented here.
By default, nac-test (via pabot) executes test suites (i.e., each robot file) in parallel. The number of parallel processes can be controlled via the --processes option.
However, suite-level parallelization may be inefficient for test suites containing multiple long-running test cases (e.g., >10 seconds each). If your test cases are independent and can run concurrently, you can enable test-level parallelization by adding the following metadata to the suite's settings:
*** Settings ***
Metadata Test Concurrency TrueNote: This approach benefits only long-running tests. For short tests, the scheduling overhead and log collection may offset any performance gains.
Tip: The Test Concurrency metadata is case-insensitive (test concurrency, TEST CONCURRENCY, etc.).
Implementation: nac-test checks the rendered robot files for the Metadata setting and instruct pabot to run each test within the respective suite in parallel (using pabot's --testlevelsplit --orderingfile ordering.txt arguments). You can inspect the ordering.txt file in the output directory.
This behaviour can be disabled when setting the environment variable NAC_TEST_NO_TESTLEVELSPLIT.