On Fri, 18 Jan 2019 at 16:04, Denis HUMEAU denis.humeau@st.com wrote:

Hello,

A request from some Lava users internally.

We have 3 boot stages, TF-A, U-boot, and kernel.

Is there a way, in a Lava job, to test that these components’ versions are the expected ones?

Depends entirely on how the version strings are to be retrieved.

In most cases, you don't want to simply parse these out of the serial console stream because if anything has gone wrong, it is too late to do anything about it.

So the solution would be to use the interactive test action support to *ask* the device for the version and control that in the test job match strings. This will give you a test result and you can define a mismatch as a test job failure, preventing the rest of the test job from continuing and producing misleading data. Of course, for the kernel version, you can use a standard POSIX test action and lava-test-case / lava-test-raise which also allows you to use test action parameters and do the validation of the version string in the test action itself. This is akin to uname -a | | assert blah

That would mean, as far as I understand, not testing the embedded software itself, but the Lava job log…

I wouldn't recommend that for the reasons above. Parsing the job log is too late and there is nothing you can usefully do when you do spot a problem.

Equally, there are requirements here for how you create the templates of the test job submissions. Your build system will create URLs which contain a build number or date but not usually anything which maps to the version string being output at runtime. Make sure this is covered by adapting your build scripts to retrieve and export that version string as a piece of metadata. I'd recommend creating an ASCII file alongside the binary - it should contain the version string and it could also contain the md5sum and/or sha256sum of the binary file itself, probably along with a timestamp from the build system. For extra points, make it a JSON or YAML file format and retrieve it in the scripts which then generate the LAVA test job submission YAML. Embed the version string into the test job submission (if only as metadata) and use the checksums alongside the URLs to LAVA. (Not enough real-world test jobs use checksums IMHO, it is particularly useful to know that what LAVA downloaded is what the build system created. Personally, I think all LAVA downloads should have a checksum provided by the test writer and modern build scripts which populate metadata into the test job submission are certainly capable of this.)

Otherwise, you won't be able to map build #56 to 2015.07-00752-g79062d5 which then leads to a lot of wasted time when it comes to working out which build number you want to deploy to the device to achieve a rollback from a broken build.

However, if you merely want to add these version strings to the metadata, then a custom frontend tool could be used to create retrospective data.