lava-users

lava-users@lists.lavasoftware.org

2 participants
954 discussions

LAVA 2026.10

by Remi Duraffort

Hi folks, The 2026.10 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Migrations ## New migrations 4 database migrations are applied by this release: * lava_scheduler_app.0071_lava_log_entry * lava_scheduler_app.0072_testjob_pool_pattern * lava_scheduler_app.0073_testjob_view_permission_and_device_id_index * lava_scheduler_app.0074_testjob_add_missing_permissions `0071_lava_log_entry` moves the Device, Worker and TestJob history from Django's `LogEntry` table to dedicated LAVA tables. On instances with a long history, this migration can take some time as every log entry is copied. `0073_testjob_view_permission_and_device_id_index` adds an index on `(actual_device, -id)`. On PostgreSQL, the index is built concurrently so job submission is not blocked on large instances. # Distributions This release drops support for Django 4.2 and requires **Django 5.2** or later. Debian 12 (bookworm) is no longer supported. On Debian 13 (trixie), `python3-django` has to be installed from trixie-backports. The docker images have been updated accordingly. # Breaking changes Job cancel, fail, resubmit and toggle_favorite in the web UI, job cancel in the REST API, remote-artifact token delete, and the saved query and chart endpoints (delete, toggle published, omit, include, refresh, reorder, per-user settings, and removing a query from a chart) all answered GET. The hardening fix (introduced in https://gitlab.com/lava/lava/-/merge_requests/3516) made the state-changing endpoints `POST`-only. `GET` on these endpoints now returns 405 instead of acting. chart_query_order_update and settings_update returned 403 before. **Clients must use `POST`**. # Documentation ## Security The security documentation has been extended: * a [threat model]( https://gitlab.com/lava/lava/-/blob/master/doc/threat-models/THREAT_MODEL.m…) for LAVA covering assets, entry points, threats and recommended mitigations * a [vulnerability handling page]( https://lava.readthedocs.io/en/latest/admin/vulnerability-handling.html) documenting how security issues are reported and handled under the Linaro VDP * the [SECURITY.md]( https://gitlab.com/lava/lava/-/blob/master/SECURITY.md?ref_type=heads) file now covers the EU Cyber Resilience Act duties: supported-versions policy, safe harbor statement and PSIRT reporting duties ## lavacli MCP Document the lavacli MCP command with an example of usage. # Device-types ## New device-types * frdm-imxrt1186 * frdm-mcxe31b * frdm-mcxn947 * mimxrt700-evk ## NXP MCU Devices based on `base-nxp-mcu.jinja2`: * support the `flasher` deploy method when `flasher_deploy_commands` is defined * support the `minimal` boot method * can override `boot_character_delay` and `test_character_delay` # Dispatcher ## lava-signal: kmsg When the kernel console is on the same serial line as the test shell, the kernel can print a message in the middle of a LAVA signal, corrupting it and making the job fail. LAVA now detects such corruption and suggests to use `lava-signal: kmsg`. With `kmsg`, every signal is written to `/dev/kmsg`. The kernel prints each record as a whole, so a kernel message can only appear before or after a signal, never inside it. ```yaml - test: definitions: - name: kselftest from: git repository: https://github.com/Linaro/test-definitions path: automated/linux/kselftest/kselftest.yaml lava-signal: kmsg ``` See the [test action documentation]( https://lava.readthedocs.io/en/latest/technical-references/job-definition/a… ). ## Cleanup order The cleanup actions are now run in reversed order. This ensures that the DUT is always powered off before the rest of the cleanup, even when an exception is raised during the pipeline execution. This is required, for instance, when using `usbg-ms` as the DUT should be powered off before disabling `usbg-ms` on the worker. ## Secrets Fix a regression where the secret file was not created in the overlay. ## Multinode The multinode protocol now reads and sends the full messages over the socket. Previously, a partial `recv` or `send` could corrupt large messages. ## Power When a reboot is skipped because there is no connection, a warning is logged and the skip is recorded in the job results. # Coordinator lava-coordinator has been hardened: * read and send the full messages * do not crash when a client drops the connection or sends invalid data * add a per-request timeout and a payload limit * fix `lava-wait-all --role` * use the same `--log-file` and `--level` arguments as the other daemons * exit cleanly on ctrl+c and set the exit code # Server ## Multinode pool pattern A multinode job can now require every device of the group to belong to the same *pool* of devices. A pool is described by a device tag matching a glob pattern: ```yaml protocols: lava-multinode: pool_pattern: "lab-*" roles: server: device_type: qemu count: 1 client: device_type: bcm2711-rpi-4-b count: 1 ``` With devices tagged `lab-paris` or `lab-lyon`, the group will either run entirely on `lab-paris` devices or entirely on `lab-lyon` devices, but never on a mix of both. This is useful when the devices of a group have to be physically connected together (network switch, power supply, USB hub, ...). See the [protocols documentation]( https://lava.readthedocs.io/en/latest/technical-references/job-definition/p… ). ## view_testjob permission The `view_testjob` permission was never created in the database, so it was impossible to grant it. A user holding this permission can see every test job, including private jobs and jobs older than `PUBLIC_JOB_WINDOW_DAYS`. It is not granted to any group by default. Grant it in the Django admin group editor or with the XML-RPC API. See [the authorization documentation]( https://lava.readthedocs.io/en/latest/technical-references/authorization.ht… ). ## REST API performances Listing jobs with `/api/v0.2/jobs/` was materializing every visible job ID in memory. On large instances this was using hundreds of MB per request and taking 20-25s. The planner issue has been fixed at the root with a new index and by filtering on the device foreign key directly. ## Log entries Device, Worker and TestJob history is now stored in dedicated LAVA tables instead of Django's `LogEntry` table, which cannot be indexed. This fixes slow device and worker pages. The worker page now displays both the worker transitions and the transitions of its devices in two separate tables. ## Health checks A device dictionary extending a base template (like `base-uboot.jinja2`) made the scheduler look for the health check of the base template. The scheduler now looks for `<device-type>.yaml` first, then for the extended template. ## Job logs * Add `scheduler.jobs.logs_count` XML-RPC API returning the number of log lines of a job. Combined with `scheduler.jobs.logs`, it allows to fetch the end of the logs. * The REST API does not send `Content-Length` anymore for running jobs, which was truncating the log of running jobs. * Job result exports (CSV and YAML) are streamed without caching the full result set in memory. ## Security fixes * fix a stored XSS in device and worker health logs * fix `SO_PEERCRED` check in lava-dispatcher-host which was rejecting every peer, root included # Packaging ## Apache Static files are now served directly by Apache instead of being proxied to gunicorn, freeing gunicorn workers. ## Debian packages * `python3-guestfs` is now a recommended dependency as the `debugfs` implementation does not require it * `iproute2` is now a required dependency * `python3-bpfcc` is now a recommended dependency ## Local development A Makefile has been added to run LAVA locally using uv. Run `make help` to list the available targets. ```shell make init make run ``` Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

4 days, 19 hours

Proposal to drop XML-RPC API

by Milosz Wasilewski

Hi, Currently LAVA server offers 2 APIs: XML-RPC and REST. Most of the features of XML-RPC should be available using REST API. Dropping XML-RPC would free up resources for maintaining other parts of the code. One of the prominent use cases for XML-RPC is lavacli [1]. Before the XML-RPC API is dropped the tool needs to be migrated to the REST API. If removal of XML-RPC might cause an issue for you, please respond to this thread. There is no immediate plan for removal yet and we're collecting requirements/features that might be missing from the REST API. [1] https://gitlab.com/lava/lavacli Best Regards, Milosz

5 days, 17 hours

dropping queries and charts

by Milosz Wasilewski

Hi, There is a plan to drop Results->Queries and Results->Charts form the LAVA UI. The feature wasn't maintained for a very long time and there seems to be no one actively using it. 2026.10 will be the last release carrying these 2 options. Please respond if the plan is causing an issue for you. Best Regards, Milosz

5 days, 17 hours

LAVA 2026.09

by Remi Duraffort

Hi folks, The 2026.09 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Migrations ## New migrations 3 database migrations are applied by this release: * lava_scheduler_app.0068_testjob_metadata * lava_scheduler_app.0069_testjob_metadata_gin_index * lava_scheduler_app.0070_increase_tag_name_max_length These migrations should not create any downtime, as they can run while the instance is running. However, if you are running a large LAVA deployment, be aware that 0069_testjob_metadata_gin_index can take a lot of time on instance with many TestJobs. ## Squashed migrations Squash most lava_scheduler_app migrations into one. This will speedup the initial migration for new instances. # Distributions This release is the last one to support django 4.2. The following version will support django 5.2 and later. The following release will drop support for Debian 12 (old stable) and add support to Debian 14 (testing). For Debian 13, the use of django from stable-backport will be required. # Documentation ## AI/ML policy An AI/ML policy <https://lava.readthedocs.io/en/latest/developer/methodology/contribute.html…> has been added to the project contributions guide. * contributions with AI-created code require an "Assisted-by: LLM" line before `Signed-off-by`; tool names stay optional and contributor-supplied, so the project advertises no vendor * Signed-off-by must be written by a human; AI tools must not generate or prompt it * contributors remain fully responsible: review, verification, GPLv2+/OSD compatibility, and third-party licensing in LLM output all fall on the submitter * prohibit leaking confidential data (deployment data, device credentials, tokens, customer jobs) to third-party services * LLM use stays optional; no contributor is disadvantaged for declining ## SECURITY.md Document the LAVA security posture <https://gitlab.com/lava/lava/-/blob/master/SECURITY.md?ref_type=heads>. Security issues should be sent to Linaro Product Security Incident Response Team. # Device-types ## New device-types * rename imx91-11x11-frdm-imx91s to imx91-11x11-frdm-s * imx93w-evk * imx93w-frdm * imx941-15x15-lpddr4-evk * imx942-15x15-lpddr4-evk * imx943-orangebox * imx95-15x15-lpddr5-frdm * imx95-19x19-frdm-pro ## imx8/9 Adjust load addresses to allow for bigger kernels. Default size for kernel can be too small with some defconfigs. Mainly reduce dtb area to allow larger kernels ## Musca Add support for `flasher` deploy method and `minimal` boot. # Dispatcher ## Artifact download Allow to specify the filename of an artifact during deployment. The file gets its name from the last part of the URL path. Some URLs have no name there. A redirect endpoint saves every image as `download`, so they overwrite each other. Add an optional filename. It is the name we download, so the compression suffix goes the same way as it does for a name from the URL. It must be a plain file name, so a job cannot write outside the download directory. ```yaml rootfs: url: https://example.com/download?id=42 filename: rootfs.tar.xz compression: xz ``` ## Extra files for tftp Specifies a dictionary of additional files to make available on the TFTP server. This may be used to support board specific requirements for additional firmware, test scripts, or anything else that does not fit into one of the above file types. ```yaml - deploy: to: tftp extra_files: test_scripts: url: http://example.com/test_scripts.itb ``` See the tftp documentation <https://lava.readthedocs.io/en/latest/technical-references/job-definition/a…> . ## Job tags Export job tags as `LAVA_JOB_TAGS` environment variable. See injected variables in the documentation <https://lava.readthedocs.io/en/latest/technical-references/job-definition/e…> . ## multi-part cpio archives Modern mkinitramfs (v0.146+) concatenates multiple cpio archives into a single initramfs file: an uncompressed cpio with kernel modules followed by a compressed cpio with the rest of the initrd. Allow LAVA to make use of such cpio archives. # Server ## gevent support to gunicorn Allow to use gevent worker class with gunicorn. By default gunicorn is using gthread, switching to gevent is a matter of adding `GUNICORN_WORKER_CLASS=gevent` to the docker environment variables. ## Public job window Allow to make private job pages older than a given threshold. This is intended for public LAVA instances being enumerated by crawlers, where the scraped pages are the same pages people use and only the age of the reference job distinguishes the traffic. Update the settings to specify the public job window. Every jobs older will only be visible to authenticated users: ```yaml PUBLIC_JOB_WINDOW_DAYS: 30 ``` See the private instance documentation <https://lava.readthedocs.io/en/latest/technical-references/authorization.ht…> . ## SQLite LAVA server can now use sqlite as database backend. This is a good way to try LAVA quickly using uv. To start a full LAVA instance locally, just run: ```shell export DATABASE_URL=sqlite:///db.sqlite uv run --frozen --extra server -- manage.py migrate & uv run --frozen --extra server -- manage.py lava-publisher --log-file - -u $USER -g $USER & uv run --frozen --extra server -- manage.py lava-scheduler --log-file - -u $USER -g $USER & uv run --frozen --extra server -- manage.py runserver & uv run --frozen --extra server -- manage.py shell & sudo uv run --frozen --extra dispatcher -- lava_dispatcher/worker.py --url http://localhost:8000 --log-file - --token WORKER_TOKEN --ws-url http://localhost:8001/ws/ & ``` ## TestJob metadata TestJob metadata used to be saved into the database and queryable from the web interface. The support has been broken since 2020 and fixed in this release. You can import metadata for existing TestJob using `lava-server manage jobs backfill-metadata`. You can then filter TestJob by metadata parameter in the rest api with `metadata__<key>` query string: ``` /api/v0.2/jobs/?metadata__build_id=1234 /api/v0.2/jobs/?metadata__branch__startswith=release/ ``` Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

1 month, 1 week

LAVA 2026.07

by Remi Duraffort

Hi folks, The 2026.07 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Database migation 3 database migrations are applied by this release: * lava_scheduler_app.0065_alter_devicetype_health_frequency_default * lava_scheduler_app.0066_blacklist_to_jsonfield * lava_scheduler_app.0067_testjob_requested_device_and_worker These migrations should not create any downtime, as they can run while the instance is running. However, if you are running a large LAVA deployment, be aware that 0066 rewrites the notification table in place, so it needs temporary disk space of about that table's size and the table is unavailable while it runs. 0067 may briefly stall the scheduler. The new columns themselves are added instantly, but the indexes that come with them block writes to the job table until the migration commits. Generally this is pretty quick, a few seconds even on millions of jobs. # Device-types ## New device-types * stm32wba65i-dk1 * imx952-frdm ## QEMU: overlay drive interface The `qemu` and `kvm` templates were attaching the `lava-guest.qcow2` test overlay to an IDE drive (`if=ide`), which only works on x86 machines. On other architectures QEMU rejected the drive, so only boot-only tests could run. The interface is now selected from the architecture: * `ide` is kept on x86 for backward compatibility * `virtio` is used everywhere else (aarch64, riscv, ppc, ...). # Breaking changes ## qdl: `overlay_path` removed `overlay_path` became a no-op in the `qdl` deploy action after the `CompressOverlay` rework, and has been removed from the deploy schema. Jobs still setting it will now fail validation and must simply drop the key: ```yaml - deploy: to: qdl rootfs_image: "disk-sdcard.img2" qcomflash: url: 'https://example.com/qcomflash.tar.gz' apply-overlay: true ``` # Server ## Target a specific device or worker A job can now request one exact device or restrict scheduling to the devices attached to one worker. The device must belong to the requested `device_type`, and the worker must exist on the instance: ```yaml device_type: qemu device: qemu-01 worker: worker-01 ``` When `device` is used the scheduler will only run the job on that device, and will not fall back to another device of the same type. Likewise `worker` restricts the job to devices of the requested type attached to that worker. If both are given, the selected device must also be attached to the selected worker. For multinode jobs, `device` and `worker` are specified inside each `lava-multinode` role. A role-level `device` constraint requires that role to use `count: 1`. ## Require login on specific paths Allow requiring login for specific URL paths. `REQUIRE_LOGIN` gates a whole instance, which is too broad for otherwise public instances. The query UI, for example, is open to anonymous users and can trigger expensive database queries, making large instances easy to overload with indiscriminate crawlers. The new `REQUIRE_LOGIN_PATHS` setting takes a list of URL path prefixes, relative to the mount point. Each prefix covers itself and everything below it: ```yaml REQUIRE_LOGIN_PATHS: ["results/query", "results/chart"] ``` Requests under a configured prefix get the same enforcement as `REQUIRE_LOGIN`, including the standard exemptions and API token passthrough; everything else stays open. An invalid configuration raises `ImproperlyConfigured` at startup. ## Custom default health-check frequency Device-types were always created with a routine health-check frequency of 24 hours. Admins can now change that default for an instance: ```yaml HEALTH_FREQUENCY_HOURS: 168 ``` The value is validated when the application starts: non-integer or non-positive values are rejected with `ImproperlyConfigured`. ## Secrets in multinode jobs A top level `secrets` section was silently dropped when submitting a multinode job. It is now copied to every child job of the multinode job, like the other shared sections. ## Notification blacklist stored as JSON `Notification.blacklist` moved from a PostgreSQL-only `ArrayField` to a `JSONField` so that it works on every database backend. A data migration converts the existing `varchar[]` column to `jsonb` in place, preserving the stored values. The conversion only runs on PostgreSQL instances where the column is still an array, and is a no-op on fresh installs and on SQLite. ## Bug fixes * **Login redirects**: `LOGIN_URL` is now computed with the mount point prefix, and `LOGIN_PATH` was replaced by a `login_path()` classmethod reading the current setting at call time. With `MOUNT_POINT=lava/`, the login page exemption check was failing and redirects were landing outside the instance. * **Compressed logs**: gunicorn used the `sendfile` kernel call when the file object of a `FileResponse` exposes `fileno()`. For LZMA files this returned the underlying file descriptor, so responses contained truncated, still compressed data. The LZMA `fileno()` is now overridden to raise `io.UnsupportedOperation`. * **Empty settings file**: `yaml.safe_load()` returns `None` for an empty or comment-only file, and calling `.items()` on it crashed LAVA at startup. An empty `settings.conf`, `settings.yaml` or `settings.d/*.yaml` now simply contributes no setting. * **ConfigFile**: the parser has been reworked on top of `shlex` instead of regexes, fixing a file handle leak, restoring key validation and reporting line numbers starting at 1. # Dispatcher ## user_commands: lists of commands `do` and `undo` in `user_commands` accepted a single command only, unlike `power_on_command` and `power_off_command`. Both keys now also accept a list of commands, run in the order they are listed: ```jinja {% set user_commands = {'set_boot_to_qspi': {'do': ['webrelay --host 192.168.0.15 --port 1234 --relay 1 on', 'webrelay --host 192.168.0.15 --port 1234 --relay 2 on'], 'undo': ['webrelay --host 192.168.0.15 --port 1234 --relay 2 off', 'webrelay --host 192.168.0.15 --port 1234 --relay 1 off']}} %} ``` Empty commands (and lists containing an empty command) are rejected at validation time instead of silently running nothing. The device schema has been aligned with the implementation. ## Devices always powered off The power cleanup now runs even when `CLEANUP_TIMEOUT` has already been used up by the previous cleanup steps, so boards are no longer left powered on after a job. ## Faster ext4 overlay Two improvements to the `debugfs` based overlay introduced in 2026.05: * Only the parent directory is resolved for non-directory tar members, the basename being appended afterwards. Directory components are cached while each file leaf is unique, so every file previously cost two uncacheable `debugfs` stat calls. Building a large overlay took ~40 seconds for ~11 seconds of actual writes. * Partition copies use a large `dd` block size. Behaviour change: a file whose destination is an existing symlink in the image now replaces it instead of writing through it, which matches `tar` and `libguestfs` `tar_in` semantics. ## Bug fixes * **vexpress**: most exceptions raised while deploying are now reported as `InfrastructureError` rather than job failures. * **Overlay environment**: a debug message is logged when `LAVA_DEVICE_HOSTNAME` or `LAVA_DEVICE_TYPE` are not exported because the device configuration lacks the `hostname` or `devicetype` field. * **Device-type templates**: variables unused by any base template have been removed. # Packaging ## sysusers.d The Debian packages now ship `sysusers.d` configuration files instead of creating users from hand-written maintainer scripts. This declarative configuration also allows `/usr`-only image based systems to be created, and `/etc` to be recreated on boot after a factory reset. It does not add a hard dependency on systemd: debhelper generates the dependencies so that non-systemd and non-Linux Debian builds keep working. ## SQLite for local development `DATABASE_URL` is now supported in the development settings and the model no longer uses PostgreSQL-specific fields, so a development instance can run on SQLite. The uv based development workflow is documented. # CI ## pre-commit The CI now runs a `pre-commit` job in the analyze stage, and most of the checks have been moved to hooks: `ruff` (check and format), `codespell`, `pyupgrade`, `django-upgrade`, `bandit`, `pylint`, `mypy` and the standard file checks (large files, merge conflicts, private keys, TOML and YAML syntax, ...). The `pyrefly` and `ty` hooks are enabled experimentally. The now redundant `codespell`, `pylint`, `ruff` and SAST/`semgrep` CI jobs and their runner scripts have been removed, and `black` and `isort` have been dropped from the development dependencies in favour of `ruff`. ## Supply chain An SBOM is generated and scanned for vulnerabilities in CI, and the `uv.lock` file has been refreshed (closes CVE-2026-32274). ## Type checking `lava_dispatcher/action.py` is now type checked. This required using `isinstance` checks in `Action.parsed_command`, waiting on the `subprocess.Popen` object rather than on `PexpectPopenSpawn`, and using `subprocess.run` in `Action.run_command`. The asymmetric `errors` getter/setter of `Action` and `Protocol` has been replaced by an `errors_add` method, since type checkers require a getter and a setter to use the same type. Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

2 months, 3 weeks

LAVA 2026.06

by Remi Duraffort

Hi folks, The 2026.06 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Device-types ## New device-types * lenovo-500e-gen5-rev3-nissa * imx937-frdm * imx941-19x19-lpddr4-evk * imx942-19x19-lpddr4-evk * tegra234-3737-0000+3701-0000 * devices deriving from the new `base-qdl` template (including qcs6490-rb3gen2, which now supports minimal, u-boot and fastboot boot on top of qdl flashing) ## Renamed and removed device-types * `e850-96` renamed to `exynos850-e850-96` to follow the device tree name. * `juno-r2` dropped in favour of `juno`. The `juno-r2` template was an empty alias with no overrides, and a single `juno` device-type covers all Juno revisions. # Breaking changes ## VLANd support dropped The VLANd code has been finally removed from LAVA. This feature has been unmaintained for years and was no longer used by any active deployment. # Dispatcher ## QDL deploy and boot LAVA now supports [`qdl`](https://github.com/linux-msm/qdl/), the tool used to flash Qualcomm devices. The typical device life cycle is: * **deploy to qdl**: download a tarball containing all artifacts and add the LAVA overlay * **boot to qdl**: flash the board using the tarball contents. This step can be repeated to flash the various storage types (spinor, nvme, ufs, emmc, ...) * boot using another method depending on the software build (minimal, u-boot, fastboot, ...) `boot-to-qdl` also detects the installed `qdl` version and enables options accordingly (for example `--skipblock`, which is only available on v2.7 and above). The minimal version required to work with modern boards is 2.0. `qdl` is not yet packaged in Debian, so it must be available on the dispatcher or inside the container. It has been added to the suggested packages of `lava-dispatcher`. Example job flashing and booting a `qcs6490-rb3gen2`: ```yaml device_type: qcs6490-rb3gen2 job_name: qcs6490 qdl boot timeouts: job: minutes: 210 visibility: public actions: - deploy: to: qdl rootfs_image: "disk-sdcard.img2" overlay_path: "/" qcomflash: url: 'https://example.com/qcomflash.tar.gz' apply-overlay: true timeout: minutes: 200 - boot: method: qdl firehose_program: "prog_firehose_ddr.elf" rawprogram: "rawprogram0.xml" patch: "patch0.xml" storage: "ufs" path: "." timeout: minutes: 5 - boot: method: minimal prompts: - root@debian timeout: minutes: 3 ``` The device dictionary should include the right configuration to enter EDL mode. ```jinja {% set board_qdl_id = 'BA9B2FEB' %} {% set qdl_enter_commands = ['/usr/local/bin/tac-api.py --serial DP05DD00 --command bootToEDL'] %} ``` ## rclone download support A new `rclone` download backend allows downloading images via [rclone](https://rclone.org/), giving access to 70+ cloud storage backends (S3, Google Drive, Azure, SFTP, ...) through a unified URL scheme: ```yaml - deploy: to: tmpfs images: rootfs: url: rclone://s3remote/bucket/images/rootfs.img.gz compression: gz ``` Configuration can be provided from environment variable or the secret block: ```yaml secrets: rclone_env: RCLONE_CONFIG_S3REMOTE_TYPE: "s3" RCLONE_CONFIG_S3REMOTE_PROVIDER: "AWS" RCLONE_CONFIG_S3REMOTE_ACCESS_KEY_ID: "AKIAIOSFODNN7EXAMPLE" RCLONE_CONFIG_S3REMOTE_SECRET_ACCESS_KEY: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" RCLONE_CONFIG_S3REMOTE_REGION: "us-east-1" actions: - deploy: to: tmpfs images: rootfs: url: rclone://s3remote/bucket/images/rootfs.img.gz compression: gz ``` ## Docker login support LAVA is now able to pull docker image from private docker registry using username and password credentials provided in the job definition. To pull images from a password protected registry add a login section with the registry domain name, username and password. ```yaml - test: docker: image: example.com/my-adb-image login: registry: example.com user: foobar password: my_password ``` ## Test definition tarballs: strip-components Tarballs fetched via `from: url` (such as GitHub and GitLab archives) often wrap their contents in a top-level directory. LAVA now support a new optional `strip-components` parameter (passed through to `tar --strip-components`), so the test definition path can be given relative to the repository root. This can be used with: ```yaml - test: definitions: - name: smoke-tests from: url repository: https://github.com/Linaro/test-definitions/archive/2019.03.tar.gz compression: gz strip-components: 1 path: automated/linux/smoke/smoke.yaml ``` ## Faster tarball extraction Test definition tarballs are now unpacked using the `tar` binary instead of the pure-Python `tarfile` module. This is significantly faster and also allows unpacking Zstd-compressed archives. ## flasher: skip image download Add an option to the `flasher` deploy action to not download an image but to make it available in the substitution. ```yaml - deploy: to: flasher images: direct_url_image: url: https://example.com/image.img.xz download: false ``` The image is not downloaded; the URL from the job is passed directly to the substitution. This avoids extra downloads and uploads when the flasher device can fetch the image itself. ## Device metadata in the shell environment LAVA test action now export the device hostname and device type as environment variables: * `LAVA_DEVICE_HOSTNAME` * `LAVA_DEVICE_TYPE` ## fastboot: pre-os-command always run `pre-os-command` is now always run on `boot-to-fastboot`, including when the action uses a Docker container. This fixes booting boards (such as RB1) using fastboot with an NFS-mounted rootfs. ## Bug fixes * **lava-test-runner**: use KMSG in the runner when the test definition uses it, so the `EXIT` signal cannot arrive before `ENDRUN` and cause a run to be wrongly marked as failed. * **Overlay path**: `CompressOverlay` now stores the overlay under the full `lava_test_results_dir` instead of just its basename, fixing a path mismatch when the variable is overridden in the context. * **QEMU overlay**: the guest overlay is now created as `ext2` again, so DUTs with older kernels can mount it (a recent `e2fsprogs` enables `metadata_csum_seed` by default). * **vexpress**: default `test_character_delay` to `boot_character_delay` so tests are paced like boot and deploy, helping slow consoles (such as Juno's). # Server ## Notifications: remote artifact tokens Notification callbacks can now resolve secret tokens using the `RemoteArtifactsAuth` table in addition to the legacy XML-RPC `AuthToken` table. Previously only XML-RPC tokens could be used as callback secrets, which required administrative privileges to set. Regular users can now manage their own GitLab/API tokens for job notifications without administrative intervention. # Packaging ## PEP 621 project metadata The build now uses PEP 621 compliant project metadata and `setuptools-scm` for automatic version derivation from git tags. Dependencies are organised by component (dispatcher, server, dispatcher-host, coordinator) so only what is needed can be installed, with a `full` extra pulling in everything: ```sh uv sync --extra full --extra test pip install lava[server] ``` Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

3 months, 2 weeks

LAVA 2026.05

by Remi Duraffort

Hi folks, The 2026.05 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # New debugfs implementation for generating mountable images Instead of using `libguestfs` LAVA uses `debugfs` command from `e2fsprogs`. This should in theory result in fewer bugs compared to `libguestfs`. To fallback to `libguestfs` implementation is in place if `debugfs` command is not found. An optional `overlay_backend` key in the deploy stage allows explicit selection of the backend (`auto`, `e2fsprogs`, or `guestfs`): ```yaml - deploy: overlay_backend: e2fsprogs images: rootfs: url: https://example.com/rootfs.ext4 format: ext4 overlays: lava: true ``` # Device-types ## New device-types * r9a09g057-rzv2h-evk * imx952-15x15-evk # Server ## Job cancellation tracking Admin log entries are now recorded when a job is cancelled, making it easier to track who cancelled a job and when. ## Database New indexes on `state` and `health` columns improve performance when filtering jobs. The deprecated `index_together` on `TestJob` has been replaced with dedicated per-column indexes. ## 500 error responses Server 500 errors now return JSON instead of HTML when the client sends an `Accept: application/json` header (as `lava-worker` and `lava-run` now do). This prevents raw HTML from appearing in job logs. ## UI improvements Enable search for device-type and all workers pages. Also order tags by name (case insensitive) in the devices page. ## Future proofing A lot of changes has been done recently to remove deprecated and old django and python features from the LAVA source code. This was possible due to the removal of support for Debian 10. Multiple fixes for Debian 13 are also part of this LAVA release: * Fixed `text_offset` not being part of device type schema even thought it was used by some u-boot devices. * Fixed REST API pages having invalid layout on Debian 13. * Fixed various deprecation warnings. * Fixed LDAP settings not working on Debian 13. * Fix OIDC settings not working on Debian 13. # Dispatcher ## Docker test action Docker test actions no longer require the boot action to stop at a shell prompt. This allows booting minimal systems without a proper shell and running tests exclusively from the Docker test shell on the worker. ## fastboot ### fastboot autodetect When using fastboot auto-detect feature, LAVA will now run fastboot in a container if that's requested in the job definition. Prio to this release, LAVA was ignoring the docker configuration for running the auto-detection step. ### DUT power sequence LAVA is now downloading and preparing image prior to powering DUT on. This aligns fastboot deployment with the typical download-then-deploy pattern used by the other deployment methods in LAVA. ## Test services Environment variables from the device dictionary and job definition are now written to a `.env` file in the test service repository. Docker Compose reads this file automatically, making it easy to pass per-device information to service containers. Device dictionary example: ```jinja2 {% set environment = { 'BOARD_QDL_ID': 'D902AA38'} %} ``` Docker Compose example to expose the variable to the container: ```yaml services: srv1: environment: - BOARD_QDL_ID ``` ## u-boot `#conf@XXX` is no longer required for every device when using the `bootm` command with FIT images. The config suffix is only appended when `bootm_device_tree_config` is set in the device dictionary. Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

4 months, 3 weeks

Issue: "Secret filtering" feature leads to "Unable to parse invalid logs" error when using empty secret

by armand.beneteau＠iot.bzh

Hello all, I am reporting here an issue we observed, in my company, that has started with the "2026.02" release, and more accurately, with the "Secret filtering" feature. Starting from this release, we've noticed that some of our jobs logs were not displayed in the UI. Instead we just had the following error message: "Unable to parse invalid logs: This is maybe a bug in LAVA that should be reported.". After looking for the source of the problem, we found out that the corresponding temporary logs files (example: /var/lib/lava/dispatcher/worker/tmp/2/logs.yaml) were full of "[MASKED]" string. This leads to a problem during the YAML parsing done a bit further by Lava. And this is at this moment that we understood that the problem happens when one of our "secrets" variable was empty. So, I guess this is a case that you did not cover in your tests, and this is why I wanted to warn you. Best regards, Armand Beneteau

5 months, 1 week

LAVA 2026.04

by Remi Duraffort

Hi folks, The 2026.04 tag has been pushed to master on gitlab.com/lava/lava. .deb packages have been built in GitLab CI and are published at https://apt.lavasoftware.org/release Docker images for amd64 and arm64 have been built in GitLab CI and are available from https://registry.gitlab.com/ and https://hub.docker.com/u/lavasoftware Changes in this release ================== # Security fixes ## Private job information leaked in device table The device table was displaying the submitter and description of the currently running job even when the user did not have permission to view that job. Now users without view permission will only see that the device is "Running" a job, without any job details. # Breaking changes ## LXC support dropped LXC support has been removed. It has been unmaintained and deprecated for several years. Users of LXC jobs must migrate to [Docker-based alternatives]( https://lava.readthedocs.io/en/latest/user/advanced-tutorials/running-arbit… ). ## Database migrations squashed (upgrade path limited) `lava_results_app` migrations have been squashed to remove non-portable PostgreSQL-specific SQL and Python. The squashed migration is now the new initial migration. **Direct upgrades from LAVA 2020.09 or earlier are no longer supported.** The oldest supported upgrade path is from LAVA 2020.12 (the version shipped with Debian 11 Bullseye). Administrators running older instances must upgrade to at least 2020.12 before upgrading to this release. ## Worker auto-registration disabled by default Worker auto-registration is now disabled by default due to security implications. If `WORKER_AUTO_REGISTER_NETMASK` is set and a HTTP proxy is in use, LAVA sees the proxy's IP address (possibly localhost), which could allow unauthorized workers to register and obtain tokens. To re-enable auto-registration, explicitly configure `WORKER_AUTO_REGISTER_NETMASK` in your settings. ## Debian 11 support As of 2026.04 release Debian 11 (Bullseye) is no longer supported. Administrators are requested to migrate to at least Debian 12 (Bookworm). # LAVA Server ## REST API New endpoints for managing groups and users are available in the v0.2 REST API: * `GET/POST /api/v0.2/groups/` * `GET/PUT/PATCH/DELETE /api/v0.2/groups/<id>/` * `GET/POST /api/v0.2/users/` * `GET/PUT/PATCH/DELETE /api/v0.2/users/<id>/` ## XMLRPC API Fix device owner not being updated when calling `lavacli devices update --user`. The `physical_owner` field was not being updated by the API call. # LAVA Worker ## Test services LAVA now supports running test services on the worker. Test services are Docker Compose-based containers started on the LAVA worker, enabling custom services for test definitions. For example, LAVA can now start an OTA server that a DUT under test can connect to. Job definition example: ```yaml - test: services: - name: my-service compose: version: "3" services: ota-server: image: my-ota-server:latest ports: - "8080:8080" definitions: - repository: https://example.com/tests.git from: git path: testdefs/ota.yaml name: ota-test ``` ## Expected test results When an expected test list is provided, any test case not in that list now results in a **fail** (previously it was only logged). This makes it easier for regression-detection tools to spot unexpected test cases that should be investigated. ## GRUB ## Overriding `net_commands` Device dictionaries can now override `net_commands` to set custom GRUB network commands. This is useful for debugging network boot issues (e.g. adding `net_ls_addr`): ```jinja2 {% set net_commands = ['net_ls_addr', 'dhcp'] %} ``` ### Error detection LAVA will now automatically catch a large set of error raised by Grub while trying to boot a kernel. The following error patterns are now matched: * `missing (.*) symbol.` * `couldn't send network packet` * `destination unreachable` * `couldn't autoconfigure` * `timeout reading` * `file '(.*)' not found` * `no such device` * `disk '(.*)' not found` * `unknown filesystem` ## udev: kernel netlink fallback for containers The udev monitoring code now falls back to kernel netlink if no udev events are received. This allows containerized dispatchers to detect USB devices without requiring `--network=host` or `CAP_NET_ADMIN`, since containers don't propagate udev events across namespaces. sysfs attributes are used as a fallback for `ID_SERIAL_SHORT` when udev properties are unavailable. ## LAVA_JOB_ID environment variable `LAVA_JOB_ID` is now available in the `lava-run` environment, accessible to `user_commands` the same way as secrets declared in the job definition. This is useful for running commands that are job-specific. # Documentation The documentation has been again vastly improved with changes to: * Admin docs: hardware recommendations, typical deployment topologies, LDAP configuration, first steps, recommended security settings * Developer docs: contribution guide, code of conduct, guide for adding new actions, new device type guide, developer first steps * User docs: job debugging guide, job results, LAVA auth, user permissions, common LAVA use cases # CI More pylint checks enabled and code brought into compliance: `W1510`, `W0611`, `W1115`, `E0307`, `E0611`, `E1136`. The minimum checked Python version has been increased to **3.11**. Rgds -- Rémi Duraffort Principal Tech Lead LAVA Tech Lead Automation Software Team Linaro

6 months

Issue: Log processor bottleneck and deploy-flasher timeouts on high-frequency output (LAVA 2026.02 )

by pawanKumar

Dear LAVA Community, I am reaching out to report a regression (or significant change in behavior) regarding how the LAVA dispatcher handles high-frequency terminal output and escape sequences during flashing operations. Background: After upgrading from LAVA 2022.06 to a recent version LAVA 2026.02 , our flashing jobs—which utilize mfgtoolcli (NXP) to write images to i.MX6/8 devices—have begun failing consistently with a deploy-flasher timeout. The Issue: The flashing tool outputs a high-frequency progress bar using carriage returns (\r) and ANSI escape codes (e.g., \e[1F, \e[2K). 1. Log Bloat: In the older version, these were handled gracefully. In the current version, every escape sequence is captured as a new log entry, resulting in logs exceeding 5MB for a single flash. 2. Dispatcher Lag: The dispatcher appears to bottleneck while processing this flood of data. This "processing lag" causes the internal action timer to hit the default 500s limit, even if the physical flashing process succeeds locally on the worker. 3. Timeout Overrides: We’ve observed that deploy-flasher often ignores the timeout values specified in the Job YAML, defaulting to 500 seconds unless explicitly overridden in the Device Dictionary. Error Examples: finish programming rootfs-a>> [1F [2K 1 100% [============================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================= ... continues >] Donedeploy-flasher timed out after 500 seconds>> uuu (Universal Update Utility) for nxp imx chips -- libuuu_1.4.243-0-ged48c51>> >> Wait for Known USB Device Appear... [?25l [1F [1F [1F [1F [1F [1F1:241312 1/ 0 [ ] >> [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [1F [ Steps we have attempted without success: - Setting TERM=dumb and piping to cat (the tool continues to output sequences). - Using tr -d '\r' (reduces lines but the data volume still triggers the timeout). - Increasing timeouts in the Job YAML (often ignored by the sub-action). Question: Has there been a change in the dispatcher's log-collection priority or buffering logic that would cause this bottleneck? Are there recommended "best practices" for handling tools that force interactive progress bars in the newer LAVA architecture? Best Regards Pavan Kumar

6 months, 1 week

Jump to page:

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-users