Feature request: Tests dependant on other tests

[verdog]

Is your feature request related to a problem? Please describe.
I am using avocado to test software that interfaces with 3rd party software. The 3rd party software sometimes comes with optional features implemented and sometimes without. I would like to write avocado tests that are smart enough to notice when the 3rd party software is missing a feature and skip tests for my software.

It's simple enough to write a function to check for the 3rd party feature and e.g. run it in setUp, but it starts to break down when you have multiple requirements. e.g. in addition to checking that the 3rd party software implements a feature, I would also like to sometimes check for an internet connection, the status of a dependent system daemon, filesystem, etc.

So I would like a solution that's more composable than a list of function in setUp. I would like to model the check as a test itself, and have dependent tests that only run if the first test passes, and have a build-system style solver that solves the test execution order for me (e.g. how you write dependencies in a Makefile and it schedules the build for you).

Describe the solution you'd like
I would like a way to mark a test (or function) as a dependency and a way to annotate a test as being dependent on it. In order for the test to run, the test needs to pass or the function needs to return true. I'd like to compose these requirements as needed for each test and have avocado figure out the test execution order for me. I run my tests in parallel whenever possible and would like to tell avocado "run the suite with a maximum parallelism of n tests, respecting dependencies".

Describe alternatives you've considered
The functions in setUp mentioned above.

I have searched the avocado documentations for this and see there is some notion of dependencies already, but they seem to only be about external packages.

Additional information
I did a brief search of the currently open issues for something like this but didn't find anything. If there's a duplicate issue, feel free to close this one and link it.

[richtja]

Hi @verdog, thank you for reaching us. IIUIC your proposal is very similar to #5205 and details about it are described in BP004 DOT files. Unfortunately, we have other priorities, and we don't have resources for implementation right now. Therefore, it is located in our long-term-backlog.

But the implementation is more than welcome, if you would be interested, we can help you with explanation of the current implementation of requirement functionality in Avocado

[clebergnu]

Hi @verdog,

Besides the comments from @richtja, I would add that you can emulate the test dependency by writing custom jobs. Altough Avocado's Job API does not completely live outside the avocado.core namespace, it's been intended for general use for a while.

You could have a suite with one or more prereqs, execute it, record its status and conditionally execute another one. A simple example:

#!/usr/bin/env python3

import sys

from avocado.core.job import Job

PREREQ_CONFIG = {
    "resolver.references": ["/bin/true"],
}

TESTS_CONFIG = {
    "resolver.references": ["/bin/uname"],
}

with Job.from_config(PREREQ_CONFIG) as prereq_job:
    prereq_result = prereq_job.run()
    if not prereq_result and (
        prereq_job.result.tests_total == prereq_job.result.passed
    ):
        with Job.from_config(TESTS_CONFIG) as tests_job:
            sys.exit(tests_job.run())

Let me know if that can help somehow for now.

[clebergnu]

One more idea: the prereq job could set some variables and the tests job could always run, while the tests themselves could be skipped by decorators based on the value of those variables.

[verdog]

Thanks for the replies!

Hi @verdog, thank you for reaching us. IIUIC your proposal is very similar to #5205 and details about it are described in BP004 DOT files. ...

Yes, that is very similar. The functionality is what I want, but I would like to mention that I don't want to write a DOT file. To me, that seems like a more complex way to specify dependencies than is needed. I would like to just decorate tests directly in the python source file (or write something in the doc comment) and have avocado build and check the graph for cycles for me.

Other random thought - my tests could benefit from dependencies that do the same thing not getting combined into a single graph node 100% of the time. e.g. tests that run mktemp. Those tests run the same command, but the command result is different each time. I would assume/hope that with the dependency graph, there would be a way to actually keep those separate mktemps separate and not combine them all. Maybe with a simple flag in the config like --always or something. (I don't know the actual syntax for avocado requirements, adjust as needed)

Besides the comments from @richtja, I would add that you can emulate the test dependency by writing custom jobs....

One more idea: the prereq job could set some variables and the tests job could always run, while the tests themselves could be skipped by decorators based on the value of those variables.

This is interesting and is getting at what I want to do. Some things I'm wondering:

1. Instead of /bin/true, could I use a test class?
2. Could I embed the job within a test class test_ method? Currently, I have a suite of 100+ tests and would like to add dependencies to maybe ~10% of them. I would like them to all have the same test_ interface.

But the implementation is more than welcome, if you would be interested, we can help you with explanation of the current implementation of requirement functionality in Avocado

There is a small but non-zero chance that I could carve out some time to do this at my day job since I'm using avocado there and it would benefit my team. I'll have to get back to you on that!

[richtja]

Thanks for the replies!

Hi @verdog, thank you for reaching us. IIUIC your proposal is very similar to #5205 and details about it are described in BP004 DOT files. ...

Yes, that is very similar. The functionality is what I want, but I would like to mention that I don't want to write a DOT file. To me, that seems like a more complex way to specify dependencies than is needed. I would like to just decorate tests directly in the python source file (or write something in the doc comment) and have avocado build and check the graph for cycles for me.

Other random thought - my tests could benefit from dependencies that do the same thing not getting combined into a single graph node 100% of the time. e.g. tests that run mktemp. Those tests run the same command, but the command result is different each time. I would assume/hope that with the dependency graph, there would be a way to actually keep those separate mktemps separate and not combine them all. Maybe with a simple flag in the config like --always or something. (I don't know the actual syntax for avocado requirements, adjust as needed)

Unfortunately, avocado doesn't support such feature and always combine same dependencies into a single graph node. But feel free to open a feature request, and we can discuss how to make it possible.

Besides the comments from @richtja, I would add that you can emulate the test dependency by writing custom jobs....
One more idea: the prereq job could set some variables and the tests job could always run, while the tests themselves could be skipped by decorators based on the value of those variables.

This is interesting and is getting at what I want to do. Some things I'm wondering:

1. Instead of `/bin/true`, could I use a test class?

Yes you can "resolver.references" value is a string that can be resolved into (interpreted as) one or more tests by the Avocado Test Resolver. More info in Test References doc. This means that you can specify things like this:
Class:

PREREQ_CONFIG = {
    "resolver.references": ["./foo.py:FooClass"],
}

Specific test within class:

PREREQ_CONFIG = {
    "resolver.references": ["./foo.py:FooClass.test_foo"],
}

2. Could I embed the job _within_ a test class `test_` method? Currently, I have a suite of 100+ tests and would like to add dependencies to maybe ~10% of them. I would like them to all have the same `test_` interface.

You can, but you have to think about avocado job as avocado run command. Therefore, doing it in such way it would be same as calling avocado within avocado test and that can cause troubles like separated results, separated config, missing console output etc.

Probably better solution is to configure your whole test run in Job API and create multiple Jobs (with and without dependencies)

But the implementation is more than welcome, if you would be interested, we can help you with explanation of the current implementation of requirement functionality in Avocado

There is a small but non-zero chance that I could carve out some time to do this at my day job since I'm using avocado there and it would benefit my team. I'll have to get back to you on that!