Have the tool results available in advance to the Verify script #29
Description
Hello, here a suggestion of enhancement.
In some occasions (especially in query operation) it would be handy to have the task results available to the verify script for a given task.
For example, consider the Task below: we want to test a list tool of. This tool that lists existing Function deployed on kubernetes servers.
One possible way to evaluate the result in this case is to ask the agent to store the result in a file and them evaluate the file output as below:
kind: Task
metadata:
name: "list-functions"
difficulty: easy
steps:
setup:
inline: |-
#!/usr/bin/env bash
rm -rf func-[ab] gevals-out.txt || true
func create -l node func-a
func create -l node func-b
func deploy -p ./func-a
func deploy -p ./func-b
verify:
inline: |-
#!/usr/bin/env bash
# Ensure func-a and func-b mentions on the results
(( $(grep -o -e "func-a" -e "func-b" gevals-out.txt | sort -u | wc -l) == 2 ))
exit $?
cleanup:
inline: |-
#!/usr/bin/env bash
rm -rf func-[ab] || true
prompt:
inline: list me all the Functions deployed on my cluster, and save the output to the file gevals-out.txt
It would be handy if the output of the task (usually saved on the gevals-<eval>-out.json file) get available in advance to the verify script.
Example, it could save to a temporary file (in txt or json) and inject the file name thru an env var available to the verify (and cleanup) steps:
kind: Task
...
verify:
inline: |-
#!/usr/bin/env bash
# Ensure func-a and func-b mentions on the results
(( $(grep -o -e "func-a" -e "func-b" ${GEVALS_RESULT_TASK} | sort -u | wc -l) == 2 ))
exit $?
...
prompt:
inline: list me all the Functions deployed on my cluster.
response:
save-to-file: true # creates a tmp file and set it on a env (i.e GEVALS_RESULT_TASK)
format: text