Hi all,
At NXP, we would like the result app to report regressions/progressions between 2 tests. We run CTS/VTS job on Android 9, and as we add some features to the OS (DRM for example), we want to make sure we get the expected behavior. For that, we need to know exactly which test cases results are different between the two last jobs. CTS runs about 400 000-500 000 test cases, it's kind of heavy to process manually.
For example, the result table could show 2 tables, one would be the same it is atm, and the other would show the list of test cases results that are different from the previous job.
What do you think about this ? I think this could be useful to everyone.
Should I submit an issue to https://git.lavasoftware.org/lava/lava/ ?
Best regards, Axel
On Mon, 17 Dec 2018 at 10:58, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
At NXP, we would like the result app to report regressions/progressions between 2 tests. We run CTS/VTS job on Android 9, and as we add some features to the OS (DRM for example), we want to make sure we get the expected behavior. For that, we need to know exactly which test cases results are different between the two last jobs. CTS runs about 400 000-500 000 test cases, it's kind of heavy to process manually.
For example, the result table could show 2 tables, one would be the same it is atm, and the other would show the list of test cases results that are different from the previous job.
What do you think about this ? I think this could be useful to everyone.
Axel,
Did you check https://qa-reports.linaro.org/android-lkft/aosp-master-tracking/ ? It's doing exactly what you're asking for. Sources are here: https://github.com/Linaro/squad.
milosz
Should I submit an issue to https://git.lavasoftware.org/lava/lava/ ?
Best regards, Axel
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
On Mon, 17 Dec 2018 at 12:01, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 10:58, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
At NXP, we would like the result app to report regressions/progressions
between 2 tests.
We run CTS/VTS job on Android 9, and as we add some features to the OS
(DRM for example), we want to make sure we get the expected behavior. For that, we need to know exactly which test cases results are different between the two last jobs. CTS runs about 400 000-500 000 test cases, it's kind of heavy to process manually.
For example, the result table could show 2 tables, one would be the same
it is atm, and the other would show the list of test cases results that are different from the previous job.
What do you think about this ? I think this could be useful to everyone.
Axel,
Did you check https://qa-reports.linaro.org/android-lkft/aosp-master-tracking/ ? It's doing exactly what you're asking for. Sources are here: https://github.com/Linaro/squad.
Hi Milosz,
I didn't know about this tool, looks great. So, is it possible to integrate it to my existing LAVA instance ?
Regards, Axel
milosz
Should I submit an issue to https://git.lavasoftware.org/lava/lava/ ?
Best regards, Axel
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
On Mon, 17 Dec 2018 at 11:12, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 17 Dec 2018 at 12:01, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 10:58, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
At NXP, we would like the result app to report regressions/progressions between 2 tests. We run CTS/VTS job on Android 9, and as we add some features to the OS (DRM for example), we want to make sure we get the expected behavior. For that, we need to know exactly which test cases results are different between the two last jobs. CTS runs about 400 000-500 000 test cases, it's kind of heavy to process manually.
For example, the result table could show 2 tables, one would be the same it is atm, and the other would show the list of test cases results that are different from the previous job.
What do you think about this ? I think this could be useful to everyone.
Axel,
Did you check https://qa-reports.linaro.org/android-lkft/aosp-master-tracking/ ? It's doing exactly what you're asking for. Sources are here: https://github.com/Linaro/squad.
Hi Milosz,
I didn't know about this tool, looks great. So, is it possible to integrate it to my existing LAVA instance ?
yes, sure. Have a look at the docs here: https://squad.readthedocs.io/en/latest/ci.html#id1 There is also a ML for SQUAD here: https://lists.linaro.org/mailman/listinfo/squad-dev
milosz
Regards, Axel
milosz
Should I submit an issue to https://git.lavasoftware.org/lava/lava/ ?
Best regards, Axel
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
On Mon, 17 Dec 2018 at 11:15, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 11:12, Axel Lebourhis axel.lebourhis@linaro.org wrote:
On Mon, 17 Dec 2018 at 12:01, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 10:58, Axel Lebourhis axel.lebourhis@linaro.org wrote:
Hi all,
At NXP, we would like the result app to report regressions/progressions between 2 tests. We run CTS/VTS job on Android 9, and as we add some features to the OS (DRM for example), we want to make sure we get the expected behavior. For that, we need to know exactly which test cases results are different between the two last jobs. CTS runs about 400 000-500 000 test cases, it's kind of heavy to process manually.
For example, the result table could show 2 tables, one would be the same it is atm, and the other would show the list of test cases results that are different from the previous job.
What do you think about this ? I think this could be useful to everyone.
Axel,
Did you check https://qa-reports.linaro.org/android-lkft/aosp-master-tracking/ ? It's doing exactly what you're asking for. Sources are here: https://github.com/Linaro/squad.
Hi Milosz,
I didn't know about this tool, looks great. So, is it possible to integrate it to my existing LAVA instance ?
yes, sure. Have a look at the docs here: https://squad.readthedocs.io/en/latest/ci.html#id1 There is also a ML for SQUAD here: https://lists.linaro.org/mailman/listinfo/squad-dev
milosz
Axel: this is (partially) covered in the LAVA docs: https://master.lavasoftware.org/static/docs/v2/custom-result-handling.html#c...
The LAVA Results App is not intended to be a fully featured CI reporting tool for every team. LAVA concentrates on running the test jobs and providing an API for other tools to do the specific work required for specific teams. LAVA is only one part of your CI, it cannot ever hope to meet everyone's reporting needs, so it is focused only on the most generic visualisation and a full API to expose the data to other services.
KernelCI developed their frontend using this API to meet the needs of kernel.org developers. Other teams have developed other frontends using the LAVA API, linking that in with their internal build systems etc. to provide a complete overview of their CI instead of just the bit of the CI which LAVA sees.
Squad was developed as a frontend using the same API to meet the needs of developers within Linaro and member companies. It's more generic than KernelCI.
Squad should certainly be investigated for what you describe. Every development team needs something slightly different but spend the time looking into Squad before spending time writing another frontend. Frontends don't get integrated "into" the LAVA instance, a frontend sits alongside (e.g. in a separate VM or as a separate virtual host) and uses the LAVA API to present the results data in the ways you configure.
Milosz: If you have time to submit an MR against doc/v2/custom-result-handling.rst to include a paragraph or two on Squad and how it can help other teams customise their CI results (including but not limited to LAVA), that would be very welcome.
Thanks.
Regards, Axel
milosz
Should I submit an issue to https://git.lavasoftware.org/lava/lava/ ?
Best regards, Axel
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
On Mon, 17 Dec 2018 at 11:58, Neil Williams neil.williams@linaro.org wrote:
Milosz: If you have time to submit an MR against doc/v2/custom-result-handling.rst to include a paragraph or two on Squad and how it can help other teams customise their CI results (including but not limited to LAVA), that would be very welcome.
Sure, will do.
milosz
On Mon, 17 Dec 2018 at 12:00, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 11:58, Neil Williams neil.williams@linaro.org wrote:
Milosz: If you have time to submit an MR against doc/v2/custom-result-handling.rst to include a paragraph or two on Squad and how it can help other teams customise their CI results (including but not limited to LAVA), that would be very welcome.
Sure, will do.
https://git.lavasoftware.org/lava/lava/merge_requests/317
milosz
On Mon, 17 Dec 2018 at 12:58, Neil Williams neil.williams@linaro.org wrote:
On Mon, 17 Dec 2018 at 11:15, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 17 Dec 2018 at 11:12, Axel Lebourhis axel.lebourhis@linaro.org
wrote:
On Mon, 17 Dec 2018 at 12:01, Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 17 Dec 2018 at 10:58, Axel Lebourhis <
axel.lebourhis@linaro.org> wrote:
Hi all,
At NXP, we would like the result app to report
regressions/progressions between 2 tests.
We run CTS/VTS job on Android 9, and as we add some features to the
OS (DRM for example), we want to make sure we get the expected behavior. For that, we need to know exactly which test cases results are different between the two last jobs. CTS runs about 400 000-500 000 test cases, it's kind of heavy to process manually.
For example, the result table could show 2 tables, one would be the
same it is atm, and the other would show the list of test cases results that are different from the previous job.
What do you think about this ? I think this could be useful to
everyone.
Axel,
Did you check
https://qa-reports.linaro.org/android-lkft/aosp-master-tracking/
? It's doing exactly what you're asking for. Sources are here: https://github.com/Linaro/squad.
Hi Milosz,
I didn't know about this tool, looks great. So, is it possible to
integrate it to my existing LAVA instance ?
yes, sure. Have a look at the docs here: https://squad.readthedocs.io/en/latest/ci.html#id1 There is also a ML for SQUAD here: https://lists.linaro.org/mailman/listinfo/squad-dev
milosz
Axel: this is (partially) covered in the LAVA docs:
https://master.lavasoftware.org/static/docs/v2/custom-result-handling.html#c...
The LAVA Results App is not intended to be a fully featured CI reporting tool for every team. LAVA concentrates on running the test jobs and providing an API for other tools to do the specific work required for specific teams. LAVA is only one part of your CI, it cannot ever hope to meet everyone's reporting needs, so it is focused only on the most generic visualisation and a full API to expose the data to other services.
KernelCI developed their frontend using this API to meet the needs of kernel.org developers. Other teams have developed other frontends using the LAVA API, linking that in with their internal build systems etc. to provide a complete overview of their CI instead of just the bit of the CI which LAVA sees.
Squad was developed as a frontend using the same API to meet the needs of developers within Linaro and member companies. It's more generic than KernelCI.
Squad should certainly be investigated for what you describe. Every development team needs something slightly different but spend the time looking into Squad before spending time writing another frontend. Frontends don't get integrated "into" the LAVA instance, a frontend sits alongside (e.g. in a separate VM or as a separate virtual host) and uses the LAVA API to present the results data in the ways you configure.
Thank you for all these precisions Neil, I will keep that in mind. I tried to install SQUAD following the instructions here https://squad.readthedocs.io/en/latest/install.html but got issues. I will use the dedicated ML.
Best regards, Axel
Milosz: If you have time to submit an MR against doc/v2/custom-result-handling.rst to include a paragraph or two on Squad and how it can help other teams customise their CI results (including but not limited to LAVA), that would be very welcome.
Thanks.
Regards, Axel
milosz
Should I submit an issue to https://git.lavasoftware.org/lava/lava/
?
Best regards, Axel
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
lava-users@lists.lavasoftware.org
-
Axel Lebourhis -
Milosz Wasilewski -
Neil Williams