feat(anta): add test atomic results #937
Conversation
mtache
changed the title
CodSpeed Performance ReportMerging #937 will degrade performances by 23.28%Comparing Summary
Benchmarks breakdown
|
| add_result(result) | ||
| return table | ||
|
|
||
| def report(self, manager: ResultManager) -> Table: |
There was a problem hiding this comment.
to make this non breaking we need to keep report_all function with the possibility to change the title
| console.print("Tags found:") | ||
| console.print_json(json.dumps(sorted(tags), indent=2)) | ||
| t.update(device.tags) | ||
| console.print("Tags defined in inventory:") |
There was a problem hiding this comment.
need to update the documentation
| pytest.param(None, 5, 5, id="all results"), | ||
| pytest.param(None, 0, 0, id="result for host1 when no host1 test"), | ||
| pytest.param(None, 5, 5, id="result for test VerifyTest3"), | ||
| pytest.param("Custom title", 5, 5, id="Change table title"), |
There was a problem hiding this comment.
will probably need to add this back to be non breaking
Co-authored-by: Guillaume Mulocher <[email protected]>
| for command, host in zip(self.instance_commands, self.inputs.hosts): | ||
| host_result = self.result.add(description=str(host), inputs=host) |
There was a problem hiding this comment.
Related to my previous comment about repeating inputs, maybe we could add a input_ref field in the atomic test result with an index reference to its corresponding input in the parent.
Example:
{
"name": "DC2-BL2",
"test": "VerifyReachability",
"categories": ["connectivity"],
"description": "Test network reachability to one or many destination IP(s).",
"inputs": {
"hosts": [
{
"destination": "1.1.1.1",
"source": "Management0",
"vrf": "MGMT",
"repeat": 2,
"size": 100,
"df_bit": false
},
{
"destination": "8.8.8.8",
"source": "Management0",
"vrf": "MGMT",
"repeat": 2,
"size": 100,
"df_bit": false
}
]
},
"result": "success",
"messages": [],
"atomic_results": [
{
"description": "Destination 1.1.1.1 from Management0 in VRF MGMT",
"input_ref": 0, // References hosts[0]
"result": "success",
"messages": []
},
{
"description": "Destination 8.8.8.8 from Management0 in VRF MGMT",
"input_ref": 1, // References hosts[1]
"result": "success",
"messages": []
}
]
}I think it's a common pattern in our tests to loop over self.inputs.<model> so we can easily track the index to avoid repeating the inputs.
There was a problem hiding this comment.
I do not see how to implement that.
A reference means to have a generic way to point somewhere in the input structure of the parent. Like JSONPath in JSON or XPath in XML.
I do not see such type in pydantic, neither the way to create one.
There was a problem hiding this comment.
Let's do like we discussed: add options to show inputs of the parent, of the atomic results or all of them.
|
Description
Add anta.result_manager.models.AtomicTestResult model.
Create a
atomic_resultsfield inanta.result_manager.models.TestResultmodel.Reports supported are
table,text,jsonandjinja.csvandmarkdownwill be supported in another PR.Considering the test:
The exemple below is a serialisation of the
TestResultmodel of the test defined above:{ "name": "mtache-leaf2a", "test": "VerifyReachability", "categories": [ "connectivity" ], "description": "Test network reachability to one or many destination IP(s).", "inputs": { "filters": { "tags": [ "mtache-leaf2a" ] }, "hosts": [ { "description": "mtache-spine1 Ethernet3/1", "destination": "10.255.255.4", "source": "Ethernet49/1", "vrf": "default", "repeat": 1, "size": 100, "df_bit": false }, { "destination": "10.255.255.10", "source": "10.255.255.11", "vrf": "default", "repeat": 1, "size": 100, "df_bit": false } ] }, "result": "failure", "messages": [ "Unreachable Destination 10.255.255.4 (mtache-spine1 Ethernet3/1) from Ethernet49/1 in VRF default" ], "atomic_results": [ { "description": "Destination 10.255.255.4 (mtache-spine1 Ethernet3/1) from Ethernet49/1 in VRF default", "inputs": { "description": "mtache-spine1 Ethernet3/1", "destination": "10.255.255.4", "source": "Ethernet49/1", "vrf": "default", "repeat": 1, "size": 100, "df_bit": false }, "result": "failure", "messages": [ "Unreachable Destination 10.255.255.4 (mtache-spine1 Ethernet3/1) from Ethernet49/1 in VRF default" ] }, { "description": "Destination 10.255.255.10 from 10.255.255.11 in VRF default", "inputs": { "destination": "10.255.255.10", "source": "10.255.255.11", "vrf": "default", "repeat": 1, "size": 100, "df_bit": false }, "result": "success", "messages": [] } ] }Fixes #427
Checklist:
pre-commit run)tox -e testenv)