Hello,
in order to validate that lava-server and lava-dispatcher are not regressing from one version to another, we added a lot of unit tests that are run every time a commit is pushed under review on https://review.linaro.org/ Before each release, we also run integrations tests on real hardware.
However, we don't have access to most of the hardware currently supported by LAVA. Which mean that we cannot fully tests that LAVA is not regressing.
For this reason, we created a project called meta-lava (currently available at https://framagit.org/ivoire/meta-lava)
When run, meta-lava will: 1/ build a docker image for lava-server and lava-dispatcher * install lava-server and lava-dispatcher Debian packages from the production repo * pull the sources from git and overwrite installed sources 2/ start the containers in a specific network (meta-lava-net) 3/ wait for lava-slave to connect to the master 4/ launch a bunch of tests 5/ compare the results against the expected results
In order to run the tests (as the meta-lava slave is *not* connected to any real hardware), we use a program called DummySys ( https://framagit.org/ivoire/DummySys) that will: * print the exact lines that lava is expecting * expect the exact inputs that lava is supposed to send
This allow to test that lava will always behave exactly the same, for a given type of board, from one release to another. We can also simulate hardware failures, kernel crashes, ...and check that LAVA will continue to report the error correctly.
meta-lava is currently running health-check for 24 device types (see https://framagit.org/ivoire/meta-lava/tree/master/tests/health-checks) and also checking for some common errors (like auto-login failures) or use cases (doing a GET on a given url when a job is finished).
However without some help from the lava community we won't be able to add every boards that you care about nor all your specific use cases. So if you are interested, please raise your hand and contact me.
Thanks