On Mon, 17 Dec 2018 at 11:15, Milosz Wasilewski
<milosz.wasilewski@linaro.org> wrote:
>
> On Mon, 17 Dec 2018 at 11:12, Axel Lebourhis <axel.lebourhis@linaro.org> wrote:
> >
> >
> >
> > On Mon, 17 Dec 2018 at 12:01, Milosz Wasilewski <milosz.wasilewski@linaro.org> wrote:
> >>
> >> On Mon, 17 Dec 2018 at 10:58, Axel Lebourhis <axel.lebourhis@linaro.org> wrote:
> >> >
> >> > Hi all,
> >> >
> >> > At NXP, we would like the result app to report regressions/progressions between 2 tests.
> >> > We run CTS/VTS job on Android 9, and as we add some features to the OS (DRM for example), we want to make sure we get the expected behavior. For that, we need to know exactly which test cases results are different between the two last jobs. CTS runs about 400 000-500 000 test cases, it's kind of heavy to process manually.
> >> >
> >> > For example, the result table could show 2 tables, one would be the same it is atm, and the other would show the list of test cases results that are different from the previous job.
> >> >
> >> > What do you think about this ? I think this could be useful to everyone.
> >>
> >> Axel,
> >>
> >> Did you check https://qa-reports.linaro.org/android-lkft/aosp-master-tracking/
> >> ? It's doing exactly what you're asking for. Sources are here:
> >> https://github.com/Linaro/squad.
> >
> >
> > Hi Milosz,
> >
> > I didn't know about this tool, looks great. So, is it possible to integrate it to my existing LAVA instance ?
>
> yes, sure. Have a look at the docs here:
> https://squad.readthedocs.io/en/latest/ci.html#id1
> There is also a ML for SQUAD here:
> https://lists.linaro.org/mailman/listinfo/squad-dev
>
> milosz

Axel: this is (partially) covered in the LAVA docs:
https://master.lavasoftware.org/static/docs/v2/custom-result-handling.html#custom-result-handling

The LAVA Results App is not intended to be a fully featured CI
reporting tool for every team. LAVA concentrates on running the test
jobs and providing an API for other tools to do the specific work
required for specific teams. LAVA is only one part of your CI, it
cannot ever hope to meet everyone's reporting needs, so it is focused
only on the most generic visualisation and a full API to expose the
data to other services.

KernelCI developed their frontend using this API to meet the needs of
kernel.org developers. Other teams have developed other frontends
using the LAVA API, linking that in with their internal build systems
etc. to provide a complete overview of their CI instead of just the
bit of the CI which LAVA sees.

Squad was developed as a frontend using the same API to meet the needs
of developers within Linaro and member companies. It's more generic
than KernelCI.

Squad should certainly be investigated for what you describe. Every
development team needs something slightly different but spend the time
looking into Squad before spending time writing another frontend.
Frontends don't get integrated "into" the LAVA instance, a frontend
sits alongside (e.g. in a separate VM or as a separate virtual host)
and uses the LAVA API to present the results data in the ways you
configure.

Milosz: If you have time to submit an MR against
doc/v2/custom-result-handling.rst to include a paragraph or two on
Squad and how it can help other teams customise their CI results
(including but not limited to LAVA), that would be very welcome.

Thanks.

> >
> > Regards,
> > Axel
> >
> >>
> >> milosz
> >>
> >> >
> >> > Should I submit an issue to https://git.lavasoftware.org/lava/lava/ ?
> >> >
> >> > Best regards,
> >> > Axel
> >> >
> >> >
> >> > _______________________________________________
> >> > Lava-users mailing list
> >> > Lava-users@lists.lavasoftware.org
> >> > https://lists.lavasoftware.org/mailman/listinfo/lava-users
>
> _______________________________________________
> Lava-users mailing list
> Lava-users@lists.lavasoftware.org
> https://lists.lavasoftware.org/mailman/listinfo/lava-users



--

Neil Williams
=============
neil.williams@linaro.org
http://www.linux.codehelp.co.uk/