- lava-devel - op-lists.linaro.org

Design meeting minutes from 2019-02-27

by Steve McIntyre

Hi folks, We held our regular weekly design meeting today via Hangout. Summary of discussion: 1. [Steve] Layoffs in Linaro affecting the team 2. [Dean] A user has noticed that their jobs running the same process multiple times in a test shell have noticed that the latter iterations take much longer than earlier iterations, even though they should take the same amount of time. 1. They noticed the "Listened to connection for namespace '<NAMESPACE>' done" message appeared a lot. 2. shell.py has the following noted around this debug log: 1. # With an higher timeout, this can have a big impact on 2. # the performances of the overall loop. 1. Is there any known issues with the read feedback checks? 2. Is there any reason why this step would take longer over the course of a job? If so, is there anything we can do to mitigate this? 3. Is there any settings we can tweak to adjust performance? 4. Any further information that might help us investigate this further? 5. Maybe a pexpect problem? Changes in this area happened in 20l8.7, Dean is using 2018.5 3. [Rémi] lavafed labs 1. Neil’s lab is off 2. ARM? In process, but may take a while - needs IT involvement to open up ports etc. 3. [Rémi] Contact lava users: 1. Collabora? [done] 2. Baylibre? [done] 3. ST? 1. [Rémi] Add matt’s lab 2. [Steve] Can set up some stuff if needed (Mustang? BBB? Panda? Maybe grab old boards from Neil?) 4. [Rémi] LAVA 2019.2 release 1. When ? 2. Start the process Thu 28th, but we're not going to get it all done then 3. Expect to finish Monday 4th? 4. Need to document what functional tests we're doing manually for now (list was in Neil's head!) 5. We want to get to the point where lavafed etc. make this obsolete 1. Will need to actually work out useful tests for all the devices! The LAVA design meeting is held weekly, every Wednesday at 13:00 to 14:00 UTC using Google Hangouts Meet: https://meet.google.com/qre-rgen-zwc Feel free to comment here or join us directly in the meeting. Cheers, -- Steve McIntyre steve.mcintyre(a)linaro.org <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs

6 years, 2 months

1
1
0 0

Design meeting, 13th Feb 2019

by Neil Williams

Starting lava-run in a dedicated container =============================== https://git.lavasoftware.org/lava/lava/issues/114 Work has already been done for the device support in this area. The intention is that the admins can create static udev rules which add the device to the correct container. To achieve this, the name of the container needs to be made available to the udev rule. The plan will be for lava-slave to create a file in /var/cache/lava-slave/. The file will be named according to the device hostname and will contain the container name plus some other useful data, e.g. the job ID. the udev rule can then parse this file to know which container to use. The udev rule would be triggered on each ADD. If lava-slave is run in a docker, /var/cache/lava-slave/ would need to be made available to that docker as a volume. (LAVA specifies the name of the container in advance.) Test job to control the image to be used ----------------------------------------------------- The LAVA documentation will need to recommend using official LAVA Software Community Project docker images, some teams will want to build & use images based on those to help include tools which take a long time to install / build. LAVA will not be able to check the provenance of the images being used, this is a test writer problem. LAVA will need to clearly output the docker image being executed and retain that in the permanent test job log output or result metadata. lava-slave will need to handle "latest" URLs and turn it into a reproducible ID using docker inspect to get the image ID. This is to be done by passing an argument to a new lava-run option. LAVA already outputs the version of lava-dispatcher (lava-run) in use (and other tools) and this will continue with docker. Admins will continue to control certificates, e.g. ZMQ Capabilities may need to be added too. LAVA runs the container with --rm (possibly with --force too). Releases and milestones =================== We have created a 2019.03 milestone which is expected to contain the work on lava-run in a separate container above. We have moved a number of issues and merge requests from 2019.02 into 2019.03 (or sometimes into .05) to get to a feasible number of changes to release 2019.02. Adding env variables to the test shell ============================= In combination with https://git.lavasoftware.org/lava/lava/issues/228 we will be looking at making device dictionary elements and some environment variables available inside the Lava-Test Test Definition 1.0 overlays using a test shell helper. Test writers are advised not to rely on device-specific information unless essential. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 2 months

2
1
0 0

IPv6 and LXC?

by Neil Williams

We've come across a problem with LXC test jobs on a network which only supports IPv6 but it's hard to replicate (this network is at a conference, just for a few days). Has anyone already looked at IPv6 and LXC? Are there other IPv6 issues with LAVA? I've found this guide but to be able to document this, the content needs to come from someone who can replicate and test the actual problems. https://techoverflow.net/2018/06/06/routing-public-ipv6-addresses-to-your-l… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 2 months

2
1
0 0

Design meeting minutes from 2019-01-30

by Steve McIntyre

Hi folks, We held our regular weekly design meeting yesterday via Hangout. Brief summary of discussion: * [Stevan] Further discsion about the permissions model code + He followed up on -devel with more details [1] * [Steve] Sprint - where are we up to? + Hanging fire for now, waiting for OK from Linaro management on budget etc. [1] https://lists.lavasoftware.org/pipermail/lava-devel/2019-January/000017.html The LAVA design meeting is held weekly, every Wednesday at 13:00 to 14:00 UTC using Google Hangouts Meet: https://meet.google.com/qre-rgen-zwc Feel free to comment here or join us directly in the meeting. Cheers, -- Steve McIntyre steve.mcintyre(a)linaro.org <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs

6 years, 3 months

1
0
0 0

LAVA authorization restructuring

by Stevan Radakovic

Dear all, I'd like to propose a couple of designs and thoughts on the authorization subject and get the feedback from the community and LAVA core team in the process. Link to the issue: https://git.lavasoftware.org/lava/lava/issues/201 As stated before, the main reason behind the authorization revamp is that checking the device types accessible to the specific user is not optimized in the slightest and does not scale. The specific device type authorization is in some cases also used in device authorization which also adds to the complexity. Now, while the problems at hand can be addressed directly to mitigate the scalability, it'd be also smart to do something about the django-restricted-resource library. Why? Well, django already has a perfectly sound authentication model which can also be used for a per-object access with a little effort so essentially it means much less complexity, code and all the benefits that go along with these. There's two approaches here: write our own authentication backend and use already existing django-guardian project (available as debian package) which is well maintained. I think having our own backend is slightly better solution just because there's no need to add more complexity with a third-party package then we need and it seems to me that our needs can be addressed with a small code base. Once that's in place, the proposal is to address the device-type authorization with a cached value (currently, if a user can access any device from a specific device type, it can access the device type as well), meaning that we store the device-type visibility as a separate permission automagically so that the check for that can be performed without checking all the device permission in that device type. We'd also remove the complexity in the various frontend views regarding the device/device type visibility without changing the behavior. All comments/ideas welcome. Cheers, -- Stevan Radaković | LAVA Engineer Linaro.org <www.linaro.org> │ Open source software for ARM SoCs

6 years, 3 months

3
4
0 0

Re: [Lava-devel] Lava Dispatchers running in a container with USB passthrough

by Dave Pigott

I replied to Dean on this in an email thread, but thought I’d capture the information in the public forum. I’ve got a device passthrough script that doesn’t require the docker container to be run in privileged mode: https://git.linaro.org/lava/lava-lab.git/tree/shared/lab-scripts/docker/pas… <https://git.linaro.org/lava/lava-lab.git/tree/shared/lab-scripts/docker/pas…> I can force a trigger event by setting up an appropriate udev rule, e.g. I have the following in my test rig /etc/udev/rules.d/100-lava-docker.rules: ACTION=="add", ATTR{serial}=="21F6C6B800314249", RUN+="/root/docker/passthrough -d 21F6C6B800314249 -i lava-dispatcher-01-2018-11" ACTION=="add", ATTR{serial}=="28B114C800334FA2", RUN+="/root/docker/passthrough -d 28B114C800334FA2 -i lava-dispatcher-02-2018-11" Once set up, just do: udevadm control --reload-rules This will then allow hot plug for fastboot type devices. My bash script for running up a docker dispatcher is as follows: #!/bin/bash set -e set -x docker run \ -v /dev/bus/usb:/dev/bus/usb \ -v /dev:/dev \ -v /dev/serial:/dev/serial \ -e "DISPATCHER_HOSTNAME=--hostname=lava-dispatcher-01-2018.11" \ -e "LOGGER_URL=tcp://172.16.1.1:5555 <tcp://172.16.1.1:5555>" \ -e "MASTER_URL=tcp://172.16.1.1:5556 <tcp://172.16.1.1:5556>" \ --net lavanet --ip 172.16.2.1 \ --name lava-dispatcher-01-2018-11 \ -it lavasoftware/amd64-lava-dispatcher:2018.11 With this, static serial type devices will just be available - i.e. Serial devices, Arm Energy Probes etc. I am in the process of writing start scripts in Python that hides all the nasty work. I actually got this working using the Python docker library yesterday, now working on making it friendly. (sorry if I’m preaching to the choir here, just trying to give as much info as possible!) Hope this helps Dave ---------------- Dave Pigott LAVA Lab Lead Linaro Ltd t: (+44) (0) 1223 400063

6 years, 3 months

1
0
0 0

Design meeting minutes from 2019-01-23

by Steve McIntyre

Hi folks, We held our regular weekly design meeting today via Hangout. Brief summary of discussion: * [Stevan] Publish the sprint topics to the -devel mailing list? + Will do once we have the new date worked out * [Stevan] Access rights code investigation + He'll follow up on -devel with more details * [Dean B] Post about the docker passthrough stuff on -devel + Just happened - see https://lists.lavasoftware.org/pipermail/lava-devel/2019-January/000010.html * [Steve] 2019.1 Release planned for Thu 24th + Rémi and Steve doing the release, Neil advising/observing The LAVA design meeting is held weekly, every Wednesday at 13:00 to 14:00 UTC using Google Hangouts Meet: https://meet.google.com/qre-rgen-zwc Feel free to comment here or join us directly in the meeting. Cheers, -- Steve McIntyre steve.mcintyre(a)linaro.org <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs

6 years, 3 months

1
0
0 0

Lava Dispatchers running in a container with USB passthrough

by Dean Birch

Dear All, We've been looking into running some lava dispatchers in containers from the official lava dispatcher images, and we've been running into issues when interfacing these with real DUTs. This is mostly to do with USB passthrough from the physical server to the dispatcher container. Without any USB device passthrough, lava cannot interface with the DUT. We are under the impression there is some scripts or code (which may still be a work in progress) that may help us with this situation. We're wondering if we can get our hands on this and test this against our devices and see if it covers all our use cases. Our use cases include a mix of static devices (present all the time), dynamic devices (only present under some circumstances, like the DUT is powered on). We also have cases where the device will be present on job startup, but renumerates on certain conditions, which means we may need to combine techniques for static and dynamic devices. We'd also like to test different usages of these devices within containers after the passthrough happens to check that each bit of tooling can work correctly (adb/fastboot/mounting filesystems/etc). Thanks, Dean

6 years, 3 months

1
0
0 0

Design meeting minutes from 2019-01-16

by Steve McIntyre

Hi folks, We held our regular weekly design meeting today via Hangout. Brief summary of discussion: * LAVA 2019.01 release + Tentatively planned for 24th January + Neil normally does the functional release tests, may need to take those over if he's still ill. + What should be included? Quick discussion of open Merge Requests: - !336 Milosz still testing - !333 - merge, looks good - !331 - merge, definitely needed - !328 - looks good - !324 - needs review - !323, !321 will wait - !309 - check and review - !302 - review - !295 will wait for discussion - !294 needs more work * BKK19 presentations? + lavafed/meta-lava "LAVA community enabled testing" + LAVA user forum Already requested + Bootloader testing (with Loïc Poulain) Would be great * Inviting community to the design meeting? + We've been talking about doing this for a while, but never actually got round to it. It's time we did! See below for details. + Agreed to send mail to lava-devel each week with summary of the minutes, with a link to the HO for the meeting itself [Steve] + Mail to -announce describing this [Steve] + If we ever overflow what hangouts will deal with, worry about it then + If we ever need private meetings, we can have those as needed * Access rights improvement Stevan is looking into revamping how access controls are implemented, to make things simpler/cleaner/faster. He'll add a ticket at git.lavasoftware.org and mail the -devel list with more details. * Decoupling scheduler from the frontend We were planning on discussing this at the sprint this week before it was postponed. Hoping to re-organise it soon... The LAVA design meeting is held weekly, every Wednesday at 13:00 to 14:00 UTC using Google Hangouts Meet: https://meet.google.com/qre-rgen-zwc Feel free to comment here or join us directly in the meeting. Cheers, -- Steve McIntyre steve.mcintyre(a)linaro.org <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs

6 years, 3 months

2
1
0 0

Linaro & LAVA at FOSDEM 2019

by Neil Williams

https://fosdem.org/2019/ (No registration is necessary for FOSDEM - if you can get to Brussels, you are welcome to just turn up on site. Avoid trying to park near the site unless you get there very early, use public transport.) https://fosdem.org/2019/practical/ https://fosdem.org/2019/stands/ If you haven't been to FOSDEM before, note that there will be a huge number of people attending FOSDEM (well over 8,000 are expected), so the only real chance of meeting up is to have a known meeting point. Hopefully, you'll have a chance to get to some of the 700 sessions which have been arranged over the 2 days. Linaro will have a stand at FOSDEM 2019 in the AW building. The stand will be on the ground floor, along the corridor from AW120 near MicroPython and PINE64. Various Linaro and LAVA people will be around the stand during the event, so this can act as a focal point for anyone wanting to talk about Linaro and or LAVA during FOSDEM. https://fosdem.org/2019/schedule/room/aw1120/ The AW building is near the car park, across from Janson and the H building. Various Linaro and LAVA people have been routinely attending FOSDEM for a number of years. If you are able to get there, we will be happy to see you. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 3 months

1
0
0 0

Require build procedure for lavacli

by Dasar, Yashwanth

Dear Lava Dev team, I cloned the LAVA source from git@github.com:Linaro/lava.git and able to build. From the said repo, I dont get "lavacli", "lava-coordinator" and "lavapdu-daemon". Please provide more information on source repos and build procedure. Thank you, -Yash

6 years, 3 months

2
1
0 0

Default patterns and fixup dicts in Lava-Test Test Definition 1.0

by Neil Williams

https://git.lavasoftware.org/lava/lava/issues/179 If your Lava-Test Test Definition 1.0 YAML files explicitly use a parse: block (like: https://git.linaro.org/qa/test-definitions.git/tree/automated/linux/ltp/ltp…) then this will remain supported in Definition 1.0. If you use the monitors or interactive test actions, this does not affect you at all. If you rely on LAVA to create a TestCase based on a command in the Lava-Test Test Definition just echoing "pass" or "fail", then this is the Default Pattern and this change will directly affect those test jobs. The current Default Pattern and Fixup are lifted directly from V1 (https://git.lavasoftware.org/lava/lava/blob/master/lava_common/constants.py…): # V1 compatibility DEFAULT_V1_PATTERN = "(?P<test_case_id>.*-*)\\s+:\\s+(?P<result>(PASS|pass|FAIL|fail|SKIP|skip|UNKNOWN|unknown))" DEFAULT_V1_FIXUP = { "PASS": "pass", "FAIL": "fail", "SKIP": "skip", "UNKNOWN": "unknown", } We've recently updated the documentation to drop mention of the default pattern support for the following reasons: * It has always been problematic to encode a Python regular expression in YAML. Failures are difficult to debug and patterns are global for the entire test operation. * The move towards more portable test definitions puts the emphasis on parsing the test output locally on the DUT using a customised parser. This has further advantages: * The pattern does not have to be mangled into YAML * The pattern can be implemented by a language other than Python * The pattern can change during the operation of the test shell, e.g. a different pattern may be required for setup than for the test itself. We are now starting to plan for Lava-Test Test Definition 2.0 with an emphasis on requiring portable test scripts and removing more of the lava_test_shell Test Helper scripts. Full information on 2.0 will be available early in 2019. As a first step, the generally unhelpful Default Pattern and Default Fixup dict are likely to be removed. If you need this support, the pattern can be added to your Lava-Test Test Definition 1.0 YAML files. In the next release, it is proposed that unless an explicit pattern is specified in the Lava-Test Test Definition 1.0 YAML file, then no pattern will be implemented. Processes which echo "pass" or "fail" would be ignored and no TestCase would be created. Let us know if there are any thoughts or problems on this proposal. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 4 months

1
0
0 0

Migrations and releases

by Neil Williams

When a new release is being planned, it is useful to give admins some notice if the release includes a database migration. Releases without migrations can be applied with minimal downtime (just the time needed to unpack the files and restart the daemons). LAVA is also planning changes to support scaling and high availability, these changes may involve creating multiple dockers accessing the same external database. This provides for no downtime when upgrading to a new LAVA release, each docker could be restarted independently, jobs can continue to be submitted etc.. However, as soon as the release includes even one database migration, scheduled downtime becomes unavoidable and services trying to submit jobs would get a HTTP502 error. All docker containers would have to be stopped before the database is migrated and then updated containers started which use the new code to match the effect of the migrations. To help admins with future releases, the plan is to "batch up" migrations into a single release and then give warning on lava-announce when work starts on that release (not at the point of announcing the release itself). Merge requests which include database changes will have the database changes label applied: https://git.lavasoftware.org/lava/lava/merge_requests?label_name%5B%5D=Data… Issues which are expected to require database changes will have the same label: https://git.lavasoftware.org/lava/lava/issues?label_name%5B%5D=Database+cha… Issues and merge requests with the label will be put into the same milestone and the milestone will mention that database changes are expected: https://git.lavasoftware.org/groups/lava/-/milestones/2?milestone%5Btitle%5… When developers are planning changes which include database migrations, please make sure that this is declared (on this list and in all issues or merge requests), the relevant label is attached and a suitable milestone is set. Avoid merging any MR which has a milestone ahead of the next release. Milestones are tracked here: https://git.lavasoftware.org/groups/lava/-/milestones One suggestion for developers would be to host the change which includes one or more migrations in a dedicated docker container and keep your localhost on the master branch. Comments? Problems? -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 5 months

1
0
0 0

Testing...

by Steve McIntyre

Cheers, -- Steve McIntyre steve.mcintyre(a)linaro.org <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs

6 years, 6 months

1
0
0 0

Design notes from 17th October 2018

by Neil Williams

https://git.lavasoftware.org/lava/ci-images/merge_requests/4 I've had problems getting Docker in Docker working on arm64. As the second main architecture for LAVA, I was looking at how to build arm64 docker images inside GitLab CI - using the same methods as the amd64 builds. A number of issues resulted: The DinD base images are complex and it's not obvious how to build an arm64 docker image which can run in privileged mode and build other docker images inside it - certainly not within our existing GitLab CI. The available arm64 hardware has lots of cores but is slow at single threaded operations. I manually built the arm64 docker images to run the unit tests outside of the GitLab CI, even with -n 18 support to use 18 of the 24 cores, it is still too slow to use on every change to every merge request. That then means that automatically building the arm64 docker images is less of an issue - we can simply run some arm64 tests when a git tag is pushed. The overall issue will be kept under review but for now, we'll be fine with manually rebuilding arm64 dockers from time to time and running arm64 tests only on tags. So the WIP for ci-images has been abandoned and the WIP on lava will be lower priority. Other merge requests were discussed too: https://git.lavasoftware.org/lava/lava/merge_requests/103 The XML_RPC support will gain documentation on how to use the allow_none setting in XMLRPC clients to help keep the newer XML_RPC function output consistent. https://git.lavasoftware.org/lava/lava/merge_requests/14 The skip_timeout support will be tweaked to use a boolean flag in the timeout block of the test action and the contribution guide updated to show a distinction that tags should relate to user/device data and elements relating directly to the code should use the structures instead. https://git.lavasoftware.org/lava/lava/merge_requests/12 Excessive messages will be changed to handle the trim inside the exception handler. These merge requests will get updates in the next few days. https://git.lavasoftware.org/lava/lava/merge_requests?scope=all&utf8=%E2%9C… There are a number of merge requests tagged with the 2018.10 milestone - these are the ones we are trying to get merged before making a new upstream release of LAVA. Finally, a reminder, this list is intended for developers of LAVA to support discussion of changes before or after a merge request is opened. Please abide by the Code of Conduct and Contribution Guide. https://www.lavasoftware.org/code-of-conduct.html https://master.lavasoftware.org/static/docs/v2/contribution-intro.html -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 6 months

1
0
0 0

Testy test 2

by Steve McIntyre

Cheers, -- Steve McIntyre steve.mcintyre(a)linaro.org <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs

6 years, 7 months

1
0
0 0

2025

2024

2023

2022

2021

2020

2019

2018

lava-devel