Hello Everyone,
for more than 5 years now, the LAVA gitlab server has been hosted and
maintained by Linaro in the Cambridge lab.
We recently decided to migrate the LAVA project source code to gitlab.com
as gitlab is providing licenses for OSS projects like LAVA (and the server
is getting older).
The migration has been finished today. Every projects are now in the LAVA
namespace at https://gitlab.com/lava/
For instance:
* lava: https://gitlab.com/lava/lava/
* lavacli: https://gitlab.com/lava/lavacli/
* lavafed: https://gitlab.com/lava/lavafed/
* meta-lava: https://gitlab.com/lava/meta-lava/
* ...
You need to change the git configuration to point to gitlab.com instead of
git.lavasoftware.org.
As the migration process does not migrate the pipeline data, if you have
open MRs, please rebase and update them.
If you need access, please create a username on gitlab.com and request
access.
Best regards
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.10 tag has been pushed to master on gitlab.com/lava/lava.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://registry.gitlab.com/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Device-types
## New device-types
New supported devices:
* acer-chromebox-cxi5-brask
* acer-cp514-2h-1130g7-volteer
* acer-cp514-2h-1160g7-volteer
* dell-latitude-5300-8145U-arcada
* hifive-unmatched-a00
# lava-docker-worker
Add a systemd unit file to start multiple lava-docker-worker instances.
```shell
systemctl start lava-docker-worker@<instance-01>
systemctl start lava-docker-worker@<instance-02>
```
# Scheduler race condition
A scheduler race condition has been fixed in this release. This race
condition has been affecting LAVA for many years. This was causing multiple
issues like:
* #620: Cannot access Multinode Job with Sub ID
* #622: One of Multinode Job Stuck in Scheduling status forever
* A device is set to reserved state without a job assigned to it
# Packages
Publish debian packages to [lava-apt-repo](
https://gitlab.com/lava/lava-apt-repo/) gitlab page.
# XMLRPC API
## Jobs filtering
Extend the `scheduler.jobs.list` endpoint to filter jobs that take longer
than N minutes.
When there are some network issues or changes on the worker side, jobs that
handled by the worker may not updated to finish state even after job
timeout reached.
## Inactive users
Disable token for inactive users. This is already the case for the REST
API. Extend this behavior to the XMLRPC API.
Rgds
--
Rémi Duraffort
Principal Tech Lead
LAVA Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.06 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Upgrading
## Database migrations
Three database migrations are present in this release:
* lava_results_app.0019_update_query_contenttype
* lava_scheduler_app.0057_dt_permissions_worker_master_version
* lava_scheduler_app.0058_add_testjob_view_performance_indexes
The three migrations can be safely ran before upgrading and restarting
gunicorn.
If you already applied the two first migrations, you should revert them by
calling:
```shell
lava-server manage migrate lava_results_app 0018_drop_buglink
lava-server manage migrate lava_scheduler_app 0056_testjob_queue_timeout
```
Then you can safely call
```shell
lava-server manage migrate
```
# Device-types
## New device-types
New supported devices:
* acer-n20q11-r856ltn-p1s2-nissa
* r8a779m1-ulcb
# Docker images
The LAVA container images are now based on Debian (12) Bookworm.
# Docker caching
When using a docker container in a job action, by default, LAVA dispatcher
will call `docker image pull` to get the latest version of the image.
In order to drop the `pull`, ones can add `local: true`. Starting from this
release, when using `local: true`, if and only if the image is not present
locally, LAVA dispatcher will pull the image first.
If the image is already available on the worker, then LAVA dispatcher will
never force a refresh.
# Job metadata
Allow to pass any kind of objects into the job metadata:
```
metadata:
api:
name: example.com
url: https://example.com/api/
version: latest
node_id: 12345
```
# Kernel log parsing
Allow to ignore kernel errors when parsing the boot log.
In order to use this, add `ignore_kernel_messages: True` to the boot action.
# lava-docker-worker
The LAVA project is currently migrating to [gitlab](https://gitlab.com/lava/).
In order to ease the transition, lava-docker-worker is now pulling docker
images from `registry.gitlab.com` and fallback to `hub.lavasoftware.org`.
# Performances
## lava-run logs
When failing to push logs from `lava-run` to the server, wait some time
before retrying. Prior to this version, `lava-run` was DOSing the server in
case of error.
## lava-server
The LAVA server SQL queries has been improved a lot and should make the web
interface way faster.
# Prometheus
A prometheus endpoint is now available at `/v1/prometheus`. To configure
prometheus, add:
```yaml
- job_name: "lava"
metrics_path: "/v1/prometheus"
static_configs:
- targets: ["https://lavaserver.example.com"]
```
To export metrics about private devices, you should provide basic
authentication with username and token (same as XMLRPC API).
# Version format
The development version string format is now `<tag>.dev<commits>` instead
of `<tag>.<commits>.g<hash>` to follow python recommendations that are now
enforced by Debian 12.
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.05 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Relicense LAVA to GPLv2 or later
Before this patch, LAVA code was licensed under GPLv2 (or later) or AGPLv3,
depending on the python module. This release relicense all the code to
GPLv2 or later.
Approval from the relevant contributors has been requested and granted
before hand.
# Migration to gitlab.com
Most of the LAVA projects has been migrated to [gitlab.com](
https://gitlab.com/lava) in the `lava` namespace. The remaining projects
(including LAVA) should be migrated before the end of July.
# Deploy to UUU
Connect device before flashing images. The DUT serial output is now
included in job logs. This is useful to retrieve error message from boot
image raised during flash steps
# lava-docker-worker
Don't use cache when building the dispatcher docker image
This is to make sure the latest dependencies defined in Dockerfile are
built into the custom image, typically when the depends are pull from git
repo or downloaded from internet.
Also add `--use-cache` to allow to override the default behavior.
# LAVAFed rework
The lavafed project has been reworked bringing a better test overage on
real devices. The project is mostly using staging.validation.linaro.org
instance for running on physical devices. If you are willing to
participate, please provide access to you staging instances.
# LAVA server performances
The performances of the LAVA server service has been improved by reworking
the SQL queries when enforcing the authorization model. This should improve
the performances on most pages.
# XML-RPC API
Add support to delete device and worker:
```
scheduler.devices.delete
scheduler.workers.delete
```
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.03 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
## New device-types
New supported devices:
* hp-14b-na0052xx-zork
* imx8mm-innocomm-wb15-evk
* rk3588-rock-5b
## Deploy
### Apply overlay
Fix applying overlay to a custom directory.
The test directory, which defaults to `/lava-%s` (where `%s` gets replaced
by the `job ID`), can be overridden like this:
```yaml
context:
lava_test_results_dir: /var/lib/lava-%s
```
### nbd
Handle `nbd-server` running as non-root.
On Debian, nbd-server is running as nbd by default, thus preventing the
image owned by root to be mounted as rw. So LAVA should detect the
nbd-server user and chown image to it.
## Boot
### Fastboot
LAVA is now raising `InfrastructureError` on `fastboot-boot-commands`
timeout.
Timeouts caused by the DUT not entering fastboot mode or by faults in the
connection to the DUT while running pre-boot fastboot commands should be
treated as infrastructure errors.
### Nodebooter
Add support for nodebooter service for booting grub based devices. In order
to enable the support, add to the device-type template:
```yaml
{% set has_nodebooter = True %}
```
Nodebooter action will start the nodebooter container, configure it, add
the DUT via API to nodebooter and reset the device to be netbooted by
nodebooter.
### uuu
Serial availability is now skipped when the first command is `bcu: reset
usb` or when uuu block contain only bcu protocol.
## Healthz
Add a new endpoint at `/v1/healthz/`. This endpoint checks that the
connection to the database is working properly and returns either 200 or
500. This can be used by reverse proxy to check the status of the gunicorn
process.
## lava-docker-worker
Fix `lava-docker-worker` systemd service dependencies. The service should
depend on both `containerd` and `docker`.
Prior to this version, system reboot would take a lot of time waiting
lava-docker-worker to stop.
## XMLRPC
### encoding
Fix the encoding of the response from the server. This will fix encoding
issues with some specific characters.
### Device(-type)s permissions
A new set of api endpoints has been added to manage devices and
device-types permissions:
* `scheduler.devices.perms(add|delete|list)`
* `scheduler.device_typess.perms(add|delete|list)`
### Users and groups
A new set of api endpoints has been added to mange users and groups:
* `auth.groups`: (add|delete|list|show) groups
* `auth.groups.perms` (add|delete|list) permissions of a specific group
* `auth.users` (add|delete|list|show|update) users
* `auth.users.groups` (add|delete|list) groups of a specific user
* `auth.users.perms` (add|delete|list) permissions of a specific user
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.02 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
## New device-types
New supported devices:
* acer-cbv514-1h-34uz-brya
* acer-chromebox-cxi4-puff
* acer-cp514-3wh-r0qs-guybrush
* b-u585i-iot02a
* stm32mp15x-eval
## Deploy and boot methods
### FVP
Under some rare circumstances some FVP model can enter an infinite loop of
printing again and again the same lines. This was preventing LAVA to cleanup
correctly and was leaking a running model inside it's docker container.
Starting from this release, LAVA will cleanup the container correctly.
### uuu
Distinguish between normal timeout and abnormal exit when running serial
download check.
## lava-dispatcher-host package
Add support to install and use `lava-dispatcher-host` on Ubuntu 20.10 and
22.04. Also fix device sharing on Debian Bookworm.
## lava-publisher
lava-publisher is now listening on /ws/v1/healthz endpoint and returns a 200
status code. This can be used to check lava-publisher health.
## LAVA events
websocket events are not authenticated. When connecting to the websocket
stream, you will only receive events for devices that you can access. Use
the
latest lavacli release to send authenticated websocket requests.
## Performances
The performance of the server part has been improved on multiple pages.
Loading
the corresponding pages should be way quicker now.
## PostgreSQL
Starting from 2023.02, LAVA will set a default value of 600 to
`CONN_MAX_AGE`,
enabling persistent database connection. This is recommended by Django and
should improve the performances.
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.01 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
## New device-types
New supported devices:
* hp-x360-14a-cb0001xx-zork
* imx6ulz-lite-evk
* imx93-9x9-qsb
* jh7100-starfive-visionfive-v1
* mt8195-cherry-tomato-r2
## Security issues
Multiple security issues has been found and fixed in this release. The
details will be shared in some weeks in order to give time for admins to
upgrade.
## Debian support
The support for Debian Buster has been dropped as Debian Buster does not
provide support for the latest pyyaml versions.
This release also add support for Debian Bookworm.
## Action timeouts
Allow defining a timeout for an individual action in the block section.
This allows to override all the actions in a block at once, while still
being
able to control single action timeouts within the same block.
```yaml
actions:
- deploy:
timeout:
minutes: 5
timeouts:
http-download:
minutes: 1
```
## Interactive test action
In interactive test action, you can now user the server IP and Job ID in
test
commands. The variables are `SERVER_IP` and `JOB_ID`.
This can be used with:
```yaml
- test:
interactive:
- name: network
prompts: ["=> ", "/ # "]
script:
- command: ping {SERVER_IP}
```
## Settings
A new set of environment variable is taken into account when loading
settings.
Every variables prefixed by `LAVA_YAML_SETTINGS_` will be yaml loaded and
added
to the django settings.
For instance to define `Debug` to `True`:
```shell
LAVA_YAML_SETTINGS_DEBUG=true
```
This will be useful when deploying using docker-compose or k8s.
## Test definitions
This release drop the TarRepoAction test definition action.
This feature was never used in LAVAv2 and has been replaced by the
UrlRepoAction since 2022.09.
Instead of providing the tar as base64 encoded in the job definition (that
can
be huged), the user can provide a url to the tar that will be downloaded,
decompressed (if needed) and extracted.
## XMLRPC API
Lava uses XMLRPC for its system management endpoints, which is secured via
HTTP
basic authentication. This adds `system.set_user_groups` taking a string
email
address for the user, and an array of string group names which will be the
exact set of groups of which the user is a member.
This endpoint requires a token from a user with staff permissions in order
to
access it.
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
We have recently fixed some serious security issues on LAVA server.
CVE-2022-44641: Recursive XML entity expansion
Users with valid accounts can submit a specially crafted XML document via the
XMLRPC that causes a recursive XML entity expansion, consuming large amounts
of resources and eventually cause a Denial of Service on the LAVA server.
This problem was found, and the fix provided, by Igor Ponomarev from
Collabora. The fix has been released in 2022.11, with the following patch:
https://git.lavasoftware.org/lava/lava/-/commit/1bee0f8957741582c2bed800974…
CVE-2022-45132: Code execution in jinja templates
A specially crafted jinja2 template can be submitted to a publicly accessible
REST API endpoint without any authentication and cause a remote command
execution as the same user that is running the LAVA server web application.
This problem was found, and the fix provided, by Igor Ponomarev from
Collabora. The fix has been released in 2022.11.1, with the following patch:
https://git.lavasoftware.org/lava/lava/-/commit/ab17e8304f10c7c0fe912067f2e…
We strongly recommend that administrators upgrade to the 2022.11.1
release immediately, or failing that, at least apply the patches linked
above locally to their lava server.
Hi folks,
The 2022.11 tag has been pushed to master on git.lavasoftware.org
<http://git.lavasoftware.org/>.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Device-types
## New device-types
New supported devices:
* kv260
* sm8350-hdk
* asus-CM1400CXA-dalboz
## imx8
* separate common configure of 8u series to imx8u-common and add new device
type imx8ulp-9x9-evk
# LAVA dispatcher
* Modifying sparse rootfs is now fully supported.
* Add ava and base-edk2 device types
* schema.deploy.fvp: add the optional uniquify param
# Bug fixes
* Fix filenames when overlaying tar files
* Add missing OIDC setting keys to common settings
* share/requires.py: fix building for debian -backports and -security suites
* reprepro-release: don't trigger on debian/* tags
* Fix a huge performance issue when parsing kernel boot log
* schema: allow to use auto_login in depthcharge boot action
* kernel messages: fix match for login prompts
* device-type: add bcu_board_name for some missed imx boards
* fvp: raise a JobError when escaping is required
Thanks,
--
Stevan Radaković | Senior Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs
