lava-users July 2017

lava-users@lists.lavasoftware.org

19 participants
24 discussions

by Robert Berger

Hi, On 2017-07-04 15:18, Agustin Benito Bethencourt wrote: > Dear CIP friends, > > please check below. This is a use case we will meet in a few > weeks/months. It is important to see others walking the same route. My simple tests are running now thanks to the help of the nice people from the #linaro-lava chat. 1) Crazy me decided to use upstream u-boot 2017.05 instead of running some ancient version from 2014;) 1.1) which happens to have a different AUTOBOOT_PROMPT than the one Lava expects "Press SPACE to abort autoboot in %d seconds\n" instead of "Hit any key to stop autoboot" so (since) I would like to be able to stay as close as possible to upstream LAVA 2017.06 I patched u-boot[1]. Note that this could be fixed with LAVA as well - interrupt_prompt: {{ interrupt_prompt|default('Hit any key to stop autoboot') }} 1.2) also the SYS_PROMPT of upstream u-boot is different than the one expected by LAVA and again I made a u-boot patch[2]. Note that this could be fixed with LAVA as well - {% set bootloader_prompt = bootloader_prompt|default('U-Boot') %} 2) After some searching it turned out that LAVA set some funny variables in u-boot which made my kernel crash. (Crazy me decided to use a 4.9.x uImage without baked in address). Adding this: {% set base_high_limits = false %} to my bbb03.jinja2 file fixed it. ... obviously ;) Regards, Robert [1] https://github.com/RobertBerger/meta-mainline/blob/pyro-training-v4.9.x/u-b… [2] https://github.com/RobertBerger/meta-mainline/blob/pyro-training-v4.9.x/u-b…

6 years, 9 months

Re: [Lava-users] [Linaro-validation] Further steps with Zephyr tests

by Paul Sokolovsky

Hello Milosz, Thanks for routing this thread to lava-users - when I made initial post to linaro-validation, I check my archive and so that e.g. Neil posts there frequently, but I missed that it's not official LAVA list. On Mon, 3 Jul 2017 22:25:31 +0100 Milosz Wasilewski <milosz.wasilewski(a)linaro.org> wrote: [] > > So, I'm not exactly sure about build-only tests on real HW boards. > > The "default" idea would be that they should run, but I imagine in > > reality, some may need to be filtered out. But then blacklisting > > would be better approach than whitelisting. And I'm not sure if > > Zephyr has concept of "skipped" tests which may be useful to handle > > hardware variations. (Well, I actually dunno if LAVA supports > > skipped tests!) > > As far as I can tell they acutely run on the board, but usually output > just 'Hello world!' or sth similar. As we discussed with Kumar, this > is still OK. What Kumar requested (and I still didn't deliver) is that > whenever the LAVA test job completes, the test should be considered > 'passed'. So we wouldn't have to do any parsing of patterns. I'm not > sure if that will work, but it's worth to try. Hmm, I wonder what would be criteria for being "failed" for such tests... Anyway, thanks for sharing - I'm not familiar with all Zephyr tests/samples myself, will keep in mind such issues when looking into them. [] > > more boards will be installed in the Lab and stability of them > > improves (so far they seem to be pretty flaky). > > > > You're absolutely right. This is a pretty big task to work on and IMHO > requires someone to work full time at least for couple of weeks. The > second part is also true, the boards don't behave as they should. I > guess Dave can elaborate more on that. I can only see the result - > boards (frdm-kw41z) don't run the tests they're requested. Matt Hart actually showed me a ticket on that, so at least it's confirmed/known issue in works. But even with arduino_101 and frdm_k64f, I hit cases more than once when board(s) were stuck for extended time, but still were routed jobs to (which either failed or timed out). So, there may be problem with health checks, which either don't run frequently enough or aren't robust enough. arduino_101 is all the lone one, so if something happens to it, there's no backup. Etc, etc. [] > > So, the problem, for starters, is how to make LAVA *feed* the > > input, as specified in the test definition (like "2+2") into a > > board. > > Right. What I proposed was coding all the inputs in the test itself. Well, that would require bunch of legwork, but the biggest problem is that it wouldn't test what's actually required. E.g. both JerryScript and MicroPython Zephyr ports are actually interactive apps working over serial connection. And functional testing of them would be feeding something over this serial connection and checking that results are as expected. I'll keep in mind idea of "builtin" tests though. Thanks! -- Best Regards, Paul Linaro.org | Open source software for ARM SoCs Follow Linaro: http://www.facebook.com/pages/Linaro http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

6 years, 9 months

Re: [Lava-users] [Linaro-validation] Further steps with Zephyr tests

by Milosz Wasilewski

I was too quick to hit reply. CCing lava-users for comments from LAVA team. milosz On 3 July 2017 at 21:50, Paul Sokolovsky <paul.sokolovsky(a)linaro.org> wrote: > Hello Milosz, > > I appreciate getting at least some response ;-). Some questions however > could use a reply from LAVA team, I guess. > > On Mon, 3 Jul 2017 13:34:49 +0100 > Milosz Wasilewski <milosz.wasilewski(a)linaro.org> wrote: > > [] > >> > jobs submit a number of tests to LAVA (via >> > https://qa-reports.linaro.org/) for the following boards: >> > arduino_101, frdm_k64f, frdm_kw41z, qemu_cortex_m3. Here's an >> > example of cumulative test report for these platforms: >> > https://qa-reports.linaro.org/lite/zephyr-upstream/tests/ >> > >> > That's really great! (Though the list of tests to run in LAVA seems >> > to be hardcoded: >> > https://git.linaro.org/ci/job/configs.git/tree/zephyr-upstream/submit_for_t…) >> > >> >> It is, as I wasn't really sure what to test. The build job needs to >> prepare the test templates to be submitted to LAVA. In case of zephyr >> each tests is a separate binary. So we end up with the number of file >> paths to substitute in the template. Hardcoding was the easiest thing >> to get things running. But I see no reason why it wouldn't be changed >> with some smarter code to discover the binaries. The problem with this >> approach is that some of these tests are just build time. They have no >> meaning when running on the board and need to be filter out somehow. > > I see, that makes some sense. But thinking further, I'm not entirely > sure about "build only" tests. Zephyr's sanitycheck test has such > concept, but I'd imagine it comes from the following reasons: a) > sanitycheck runs tests on QEMU, which has very bare hardware support, > so many tests are not runnable; b) sanitycheck can operate on "samples", > not just "tests", as sample can be interactive, etc. it makes sense to > only build them, not run. > > So, I'm not exactly sure about build-only tests on real HW boards. The > "default" idea would be that they should run, but I imagine in reality, > some may need to be filtered out. But then blacklisting would be better > approach than whitelisting. And I'm not sure if Zephyr has concept of > "skipped" tests which may be useful to handle hardware variations. > (Well, I actually dunno if LAVA supports skipped tests!) > > Anyway, these are rough ideas for the future. I've spent couple of > weeks of munging with LITE CI setup, there're definitely some > improvements, but also a Pandora box of other ideas and improvements to > make. I'm wrapping up for now, but hope to look again in some time > (definitely hope to look before the Connect, so we can discuss further > steps there). In the meantime, I hope that more boards will be > installed in the Lab and stability of them improves (so far they seem > to be pretty flaky). > > [] > >> > - test: >> > monitors: >> > - name: foo >> > start: "" >> > end: Hello, ZJS world! >> > pattern: (?P<result>(PASS|FAIL))\s-\s(?P<test_case_id>\w+)\. >> > >> > So, the "start" substring is empty, and perhaps matches a line >> > output by a USB multiplexer or board bootloader. "End" substring is >> > actually the expected single-line output. And "pattern" is unused >> > (dunno if it can be dropped without def file syntax error). Is >> > there a better way to handle single-line test output? >> >> You're making a silent assumption that if there is a matching line, >> the test is passed. In case of other tests (zephyr unit tests), it's >> not the case. The 'start' matches some line which is displayed when >> zephyr is booting. End matches the line which is displayed after all >> testing is done. The pattern follows the unit test pattern. > > Thanks, but I'm not sure I understand this response. I don't challenge > that Zephyr unittests need this support, or the way they're handled. > LITE however needs to test more things than "batch" Zephyr unittests. I > present another usercase which albeit simple, barely supported by LAVA. > (That's a question to LAVA team definitely.) > >> > Well, beyond a simple output matching, it would be nice even for the >> > initial "smoke testing" to actually make some input into the >> > application and check the expected output (e.g., input: "2+2", >> > expected output: "4"). Is this already supported for LAVA "v2" >> > pipeline tests? I may imagine that would be the same kind of >> > functionality required to test bootloaders like U-boot for Linux >> > boards. >> >> I didn't use anything like this in v2 so far, but you're probably best >> off doing sth like >> >> test 2+2=4 PASS. >> >> than you can easily create pattern that will filter the output. In >> case of zephyr pattern is the only way to filter things out as there >> is no shell (?) on the board. > > So, the problem, for starters, is how to make LAVA *feed* the > input, as specified in the test definition (like "2+2") into a board. > > As there were no reply from LAVA team (I may imagine they're busy with > other things), I decided to create a user story in Jira for them, as I > couldn't create a LAVA-* ticket, I created it as > https://projects.linaro.org/browse/LITE-175 . Hopefully that won't go > unnoticed and LAVA team would get to it eventually. > >> >> milosz > > Thanks! > > -- > Best Regards, > Paul > > Linaro.org | Open source software for ARM SoCs > Follow Linaro: http://www.facebook.com/pages/Linaro > http://twitter.com/#!/linaroorg - http://www.linaro.org/linaro-blog

6 years, 9 months

OE/Yocto testcase

by Robert Berger

Hi, I succeeded in running the example Lava tests[1][2] on a Beagle Bone Black in my lab. Now I would like to test a core-image-minimal I built with Yocto and cooked up something like this[3][4]. The result is unfortunately this[5]. I guess one of the problems I have is this: lava-test: # ls -l /lava-38/ export SHELL=/bin/sh /lava-38/bin/lava-test-runner /lava-38/0ls: cannot access '/lava-38/': No such file or directory lava-test: # export SHELL=/bin/sh lava-test: # /lava-38/bin/lava-test-runner /lava-38/0 -sh: /lava-38/bin/lava-test-runner: No such file or directory Regards, Robert [1] https://validation.linaro.org/static/docs/v2/examples/test-jobs/standard-ar… [2] https://validation.linaro.org/static/docs/v2/examples/test-jobs/standard-ar… [3] https://github.com/RobertBerger/git.linaro.org-lava-team-refactoring/blob/m… [4] https://github.com/RobertBerger/git.linaro.org-qa-test-definitions/blob/mas… [5] https://pastebin.com/14pjxhiN

6 years, 9 months

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users July 2017