On Mon, 17 Dec 2018 at 11:15, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 11:12, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 17 Dec 2018 at 12:01, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 10:58, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
At NXP, we would like the result app to report regressions/progressions between 2 tests. We run CTS/VTS job on Android 9, and as we add some features to the OS (DRM for example), we want to make sure we get the expected behavior. For that, we need to know exactly which test cases results are different between the two last jobs. CTS runs about 400 000-500 000 test cases, it's kind of heavy to process manually.
For example, the result table could show 2 tables, one would be the same it is atm, and the other would show the list of test cases results that are different from the previous job.
What do you think about this ? I think this could be useful to everyone.
Axel,
Did you check https://qa-reports.linaro.org/android-lkft/aosp-master-tracking/ ? It's doing exactly what you're asking for. Sources are here: https://github.com/Linaro/squad.
Hi Milosz,
I didn't know about this tool, looks great. So, is it possible to integrate it to my existing LAVA instance ?
yes, sure. Have a look at the docs here: https://squad.readthedocs.io/en/latest/ci.html#id1 There is also a ML for SQUAD here: https://lists.linaro.org/mailman/listinfo/squad-dev
milosz
Axel: this is (partially) covered in the LAVA docs: https://master.lavasoftware.org/static/docs/v2/custom-result-handling.html#c...
The LAVA Results App is not intended to be a fully featured CI reporting tool for every team. LAVA concentrates on running the test jobs and providing an API for other tools to do the specific work required for specific teams. LAVA is only one part of your CI, it cannot ever hope to meet everyone's reporting needs, so it is focused only on the most generic visualisation and a full API to expose the data to other services.
KernelCI developed their frontend using this API to meet the needs of kernel.org developers. Other teams have developed other frontends using the LAVA API, linking that in with their internal build systems etc. to provide a complete overview of their CI instead of just the bit of the CI which LAVA sees.
Squad was developed as a frontend using the same API to meet the needs of developers within Linaro and member companies. It's more generic than KernelCI.
Squad should certainly be investigated for what you describe. Every development team needs something slightly different but spend the time looking into Squad before spending time writing another frontend. Frontends don't get integrated "into" the LAVA instance, a frontend sits alongside (e.g. in a separate VM or as a separate virtual host) and uses the LAVA API to present the results data in the ways you configure.
Milosz: If you have time to submit an MR against doc/v2/custom-result-handling.rst to include a paragraph or two on Squad and how it can help other teams customise their CI results (including but not limited to LAVA), that would be very welcome.
Thanks.
Regards, Axel
milosz
Should I submit an issue to https://git.lavasoftware.org/lava/lava/ ?
Best regards, Axel
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users