lava-users April 2026

lava-users@lists.lavasoftware.org

3 participants
2 discussions

LAVA 2026.04

by Remi Duraffort

Hi folks, The 2026.04 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Security fixes ## Private job information leaked in device table The device table was displaying the submitter and description of the currently running job even when the user did not have permission to view that job. Now users without view permission will only see that the device is "Running" a job, without any job details. # Breaking changes ## LXC support dropped LXC support has been removed. It has been unmaintained and deprecated for several years. Users of LXC jobs must migrate to [Docker-based alternatives]( https://lava.readthedocs.io/en/latest/user/advanced-tutorials/running-arbit… ). ## Database migrations squashed (upgrade path limited) `lava_results_app` migrations have been squashed to remove non-portable PostgreSQL-specific SQL and Python. The squashed migration is now the new initial migration. **Direct upgrades from LAVA 2020.09 or earlier are no longer supported.** The oldest supported upgrade path is from LAVA 2020.12 (the version shipped with Debian 11 Bullseye). Administrators running older instances must upgrade to at least 2020.12 before upgrading to this release. ## Worker auto-registration disabled by default Worker auto-registration is now disabled by default due to security implications. If `WORKER_AUTO_REGISTER_NETMASK` is set and a HTTP proxy is in use, LAVA sees the proxy's IP address (possibly localhost), which could allow unauthorized workers to register and obtain tokens. To re-enable auto-registration, explicitly configure `WORKER_AUTO_REGISTER_NETMASK` in your settings. ## Debian 11 support As of 2026.04 release Debian 11 (Bullseye) is no longer supported. Administrators are requested to migrate to at least Debian 12 (Bookworm). # LAVA Server ## REST API New endpoints for managing groups and users are available in the v0.2 REST API: * `GET/POST /api/v0.2/groups/` * `GET/PUT/PATCH/DELETE /api/v0.2/groups/<id>/` * `GET/POST /api/v0.2/users/` * `GET/PUT/PATCH/DELETE /api/v0.2/users/<id>/` ## XMLRPC API Fix device owner not being updated when calling `lavacli devices update --user`. The `physical_owner` field was not being updated by the API call. # LAVA Worker ## Test services LAVA now supports running test services on the worker. Test services are Docker Compose-based containers started on the LAVA worker, enabling custom services for test definitions. For example, LAVA can now start an OTA server that a DUT under test can connect to. Job definition example: ```yaml - test: services: - name: my-service compose: version: "3" services: ota-server: image: my-ota-server:latest ports: - "8080:8080" definitions: - repository: https://example.com/tests.git from: git path: testdefs/ota.yaml name: ota-test ``` ## Expected test results When an expected test list is provided, any test case not in that list now results in a **fail** (previously it was only logged). This makes it easier for regression-detection tools to spot unexpected test cases that should be investigated. ## GRUB ## Overriding `net_commands` Device dictionaries can now override `net_commands` to set custom GRUB network commands. This is useful for debugging network boot issues (e.g. adding `net_ls_addr`): ```jinja2 {% set net_commands = ['net_ls_addr', 'dhcp'] %} ``` ### Error detection LAVA will now automatically catch a large set of error raised by Grub while trying to boot a kernel. The following error patterns are now matched: * `missing (.*) symbol.` * `couldn't send network packet` * `destination unreachable` * `couldn't autoconfigure` * `timeout reading` * `file '(.*)' not found` * `no such device` * `disk '(.*)' not found` * `unknown filesystem` ## udev: kernel netlink fallback for containers The udev monitoring code now falls back to kernel netlink if no udev events are received. This allows containerized dispatchers to detect USB devices without requiring `--network=host` or `CAP_NET_ADMIN`, since containers don't propagate udev events across namespaces. sysfs attributes are used as a fallback for `ID_SERIAL_SHORT` when udev properties are unavailable. ## LAVA_JOB_ID environment variable `LAVA_JOB_ID` is now available in the `lava-run` environment, accessible to `user_commands` the same way as secrets declared in the job definition. This is useful for running commands that are job-specific. # Documentation The documentation has been again vastly improved with changes to: * Admin docs: hardware recommendations, typical deployment topologies, LDAP configuration, first steps, recommended security settings * Developer docs: contribution guide, code of conduct, guide for adding new actions, new device type guide, developer first steps * User docs: job debugging guide, job results, LAVA auth, user permissions, common LAVA use cases # CI More pylint checks enabled and code brought into compliance: `W1510`, `W0611`, `W1115`, `E0307`, `E0611`, `E1136`. The minimum checked Python version has been increased to **3.11**. Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

2 weeks, 4 days

Issue: Log processor bottleneck and deploy-flasher timeouts on high-frequency output (LAVA 2026.02 )

by pawanKumar

Dear LAVA Community, I am reaching out to report a regression (or significant change in behavior) regarding how the LAVA dispatcher handles high-frequency terminal output and escape sequences during flashing operations. Background: After upgrading from LAVA 2022.06 to a recent version LAVA 2026.02 , our flashing jobs—which utilize mfgtoolcli (NXP) to write images to i.MX6/8 devices—have begun failing consistently with a deploy-flasher timeout. The Issue: The flashing tool outputs a high-frequency progress bar using carriage returns (\r) and ANSI escape codes (e.g., \e[1F, \e[2K). 1. Log Bloat: In the older version, these were handled gracefully. In the current version, every escape sequence is captured as a new log entry, resulting in logs exceeding 5MB for a single flash. 2. Dispatcher Lag: The dispatcher appears to bottleneck while processing this flood of data. This "processing lag" causes the internal action timer to hit the default 500s limit, even if the physical flashing process succeeds locally on the worker. 3. Timeout Overrides: We’ve observed that deploy-flasher often ignores the timeout values specified in the Job YAML, defaulting to 500 seconds unless explicitly overridden in the Device Dictionary. Error Examples: finish programming rootfs-a>> [1F [2K 1 100% [============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================= ... continues >] Donedeploy-flasher timed out after 500 seconds>> uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.243-0-ged48c51>> >> Wait for Known USB Device Appear... [?25l [1F [1F [1F [1F [1F [1F1:241312 1/ 0 [ ] >> [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [ Steps we have attempted without success: - Setting TERM=dumb and piping to cat (the tool continues to output sequences). - Using tr -d '\r' (reduces lines but the data volume still triggers the timeout). - Increasing timeouts in the Job YAML (often ignored by the sub-action). Question: Has there been a change in the dispatcher's log-collection priority or buffering logic that would cause this bottleneck? Are there recommended "best practices" for handling tools that force interactive progress bars in the newer LAVA architecture? Best Regards Pavan Kumar

3 weeks, 6 days

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users April 2026