Von: Neil Williams [mailto:neil.williams@linaro.org] Gesendet: Dienstag, 12. Juni 2018 17:26 An: Tim Jaacks tim.jaacks@garz-fricke.com Cc: lava-users@lists.linaro.org Betreff: Re: [Lava-users] Specifying metadata for test cases
On 12 June 2018 at 16:05, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
Hello everyone,
I know from the LAVA documentation how to add metadata to jobs and test suites. When I look at test results, I see that test cases have metadata, too. E.g. https://validation.linaro.org/results/testcase/9759970 shows the following metadata:
case: linux-linaro-ubuntu-lscpu definition: 0_smoke-tests-lxc result: pass
That data is retrieved directly from the test case. case is the string passed to lava-test-case as the test case ID. definition is the currently running test definition (so an integral part of how the test case is reported) and then there is the result.
Actions can supply metadata, e.g. https://validation.linaro.org/results/testcase/9759948
This metadata comes from the Python3 code running the action and covers a range of dynamic data which is not otherwise recorded. This data is not modifiable.
Test suites gain metadata from the operation of the test job itself. The content of the Lava Test Shell Definition is not used, other than to identify the test defiinition.
Why can metadata be defined, when it is not used at all? Metadata from inline test shell definitions can be read out via the REST API, metadata from external test shell definitions can't. What's the point in defining metadata, if you cannot use it afterwards?
The user-controllable metadata needs to be in the Test Job Submission, to then appear in the top level results:
e.g. in
https://validation.linaro.org/results/1882533
path: health-checks/dragonboard-820c.yaml source: https://git.linaro.org/lava-team/refactoring.git
come from the test job submission.
Is there a possibility to add custom metadata to test cases?
Only as a URL: https://validation.linaro.org/static/docs/v2/writing-tests.html#recording-te...
Why this limitation? Is there any reason, why it has to be an URL? Why not allow any string to be appended to a test case?
Once the test job has completed, bug links are also available for test jobs, test suites and test cases.
Extra data can be recorded as separate test cases using the test case name, e.g.
https://staging.validation.linaro.org/results/testcase/5557198
This seems like a dirty hack to me, quite contrary to the „keep your data clean“ principle.
(Although I really should tweak that to drop the Python3 string handling artifacts in the test shell scripts. It should read code-version-2018.5+12261.50e347aca )
We are considering new features for the Lava Test Shell Definition but changes won't be made any time soon, it's likely to be 2019 before we have a clear plan for that work.
Are there any news on this? What new features are planned?
Sorry for re-activating such an old thread, but the topic is still relevant for us. We would like to append metadata to test cases and test suites, and there does not seem to be a useful way to achieve this with LAVA. Our actual goal is to include LAVA test results in the release notes for our OS releases. For this purpose, we need some kind of human-readable description of our tests, which shall be stored alongside the results.
As far as I can see from the database layout, metadata is a data field of test cases. Would you accept code contributions which add functionality for adding custom metadata to this field?
However, test suites do not have such a field. Instead, actions have metadata, which are not stored in the database but in dedicated metadata yaml files under /var/lib/lava-server/default/media/job-output/. Why is that? Why not add a metadata database field for actions/test suites as well?
Do you have any ideas of how to achieve our goal using LAVA?
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.comhttp://www.garz-fricke.com/ WE MAKE IT YOURS! [cid:image001.jpg@01D4C949.2759D020] Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun
[cid:image004.png@01D4C949.A7C7C5F0]
