Hi folks,
The 2024.03 tag has been pushed to master on gitlab.com/lava/lava.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://registry.gitlab.com/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Device-types
## New device-types
* mt8195-cherry-tomato-r3
## aaeon-UPN-EHLX4RE-A10-0864
Allow to set `kernel_start_message` variable.
## bcm2837-rpi-3-b
Fix the DT load addresses that was potentially overwriting u-boot ATAGs.
See [rpi-3-32b fix](
https://gitlab.com/lava/lava/-/commit/afa28d5ae128a591414517edd72652800a72b…
)
## FVP
Automatically retry failures when activating the ARMLM license.
# APIs
## Rest API
The REST API v1 has been removed. The API has been superseded by v2 for a
long time now.
At the same time, a new version of the API (v3) has been started. This is
still an on going work to improve the API performances and reliability.
## XMLRPC
### Devices
Allow to update a device `physical_owner` field from the API. Calling
`lavacli devices update --user <username>` will update the field.
### Workers
Validate yaml syntax before updating the worker configuration.
# docker
## lava-dispatcher
Use QEMU from bookworm-backports, upgrading from 2.7 to 8.2.1.
# Services
## lava-dispatcher-host
Improve stability by handling more potential exceptions.
## lava-publisher
Improve lava-publisher stability by handling multiple potential connection
errors.
## lava-run
When sending logs over https fails because the payload is too large,
lava-run will automatically reduce the size of the payload before retrying.
Before this version, lava-run would retry with the exact same payload (or
sometime even larger one) and will be unlikely to ever succeed.
When failing to send logs to the server, lava-run will also record the
error to stderr. lava-worker will then forward these messages to the
server. This will be recorded as failure_comment.
## lava-server-gunicorn
Switch gunicorn default worker from eventlet to gthread.
With eventlet, the event notification system (based on ZMQ socket) will
need to create a new socket for each connection. If the publisher service
is unreachable, the server will consume all the file descriptors and crash
after some time.
gthread worker allows to reuse the same zmq socket for multiple requests
while still allowing requests to live longer than the worker timeout.
Regarding the performances, the apache bench test results shown that
gthread is able to handle more concurrent connection than eventlet in most
use cases.
## lava-server cli
A new argument has been added to the `lava-server jobs rm` command:
`--logs-only`. When this parameter is set, the command will only remove the
logs from the filesystem while keeping the TestJob objects in the database.
Rgds
--
Rémi Duraffort
Principal Tech Lead
LAVA Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2024.01 tag has been pushed to master on gitlab.com/lava/lava.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://registry.gitlab.com/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Upgrading
## Database migrations
This release includes one migration in lava_scheduler_app. Remote artifact token
max length is increased from 100 to 200 for super-long API tokens.
# Device-types
## New device-types
New supported devices:
* stm32mp157a-dhcor-avenger96
* meson-g12b-a311d-libretech-cc
* meson-sm1-s905d3-libretech-cc
* imx93-11x11-evk-pmic-pf0900
* avh(Arm Virtual Hardware)
* imx95-19x19-evk
* dell-latitude-3445-7520c-skyrim
* mt8186-corsola-steelix-sku131072
## NXP
Download addresses for i.MX8 are updated to match those shipped in the
firmware for i.MX8MP-EVK which has been continuing to work happily. The old
addresses no longer give enough room to avoid them overwriting each other.
# LAVA publisher
IPV6 is supported in ZMQ events now. Adding the following line in server
settings enables IPV6 support.
```yaml
EVENT_IPV6: true
```
# LAVA scheduler
* Improve device's Jinja template performance by using a shared Jinja
environment. The template cache will be shared between HTTP requests and no
longer spend time to compile templates for every request. For example, when a
dispatcher downloads job from the server the template has to be rendered with
a job's context.
* Record worker health transitions. Worker online and offline state changes are
logged and displayed now.
# REST API
The `/device/validate` endpoint now requires "add_device" permission.
# Deploy and boot methods
## avh
New AVH deploy and boot methods are added for integrating Arm Virtual Hardware
in LAVA.
Example:
```yaml
device_type: avh
job_name: avh-rpi4b-deploy-boot-example
timeouts:
job:
minutes: 60
priority: medium
visibility: public
secrets:
# The value should be the name of user's remote artifact token
avh_api_token: remote_artifact_token_name
actions:
- deploy:
to: avh
options:
model: rpi4b
api_endpoint: https://app.avh.arm.com/api
project_name: "Default Project"
timeout:
minutes: 30
images:
rootfs:
# Disk image.
url: https://example.com/rpi4b/nand
format: ext4
# root partition for LAVA test overlay.
root_partition: 1
# Partition for custom overlays.
partition: 1
overlays:
modules:
compression: xz
format: tar
path: /
url: https://example.com/rpi4b/modules.tar.xz
kernel:
url: https://example.com/rpi4b/kernel
dtb:
url: https://example.com/rpi4b/devicetree
- boot:
method: avh
timeout:
minutes: 20
prompts:
- "pi@raspberrypi:"
- "root@raspberrypi:"
auto_login:
login_prompt: "login:"
username: pi
password_prompt: 'Password:'
password: raspberry
login_commands:
- sudo su
```
## fvp
Added buffering to the model stdout and stderr. Without this buffering, the
model sometimes hangs when stdout or stderr buffers are full.
## fastboot
DUT will be powered off when fastboot timeout. On some devices, the fastboot
command will never return and is not killable. This might block lava-run
forever. Powering-off the DUT will release fastboot command.
## uuu
* Fixed individual action timeout not being respected in
boot-corrupt-boot-media.
* Disconnect device after uuu boot action to flush the output buffer.
## download
Allow to turn off kisscache at job level.
Example:
```yaml
- deploy:
timeout:
minutes: 5
to: tmpfs
images:
rootfs:
use_cache: false
```
# LAVA dispatcher
## Git clones
Added 6 times retries for git clone function. It could be helpful for
infrastructure level one time failure.
## DUT output
In case it is needed by post-processing out of LAVA, single `\r` character in
pexpect output is not stripped off anymore.
## Test definition parameters
Fixed test definition parameter `0` being converted to the empty string. LAVA
dispatcher converts both key values `None` and `0` to `''` before. Now only
`None` is converted.
Example:
```yaml
- test:
timeout:
minutes: 5
definitions:
- repository: http://git.linaro.org/lava-team/lava-functional-tests.git
from: git
path: lava-test-shell/smoke-tests-net.yaml
name: smoke-tests
parameters:
USE_NC: 0
```
`USE_NC: 0` was converted to `USE_NC=''` in the `run.sh` shell script before. It
is now converted to `USE_NC='0'`.
Rgds
--
Chase Qi
Senior Engineer
Automation Software Team
Linaro
_______________________________________________
lava-announce mailing list -- lava-announce(a)lists.lavasoftware.org
To unsubscribe send an email to lava-announce-leave(a)lists.lavasoftware.org
%(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
Hello Everyone,
for more than 5 years now, the LAVA gitlab server has been hosted and
maintained by Linaro in the Cambridge lab.
We recently decided to migrate the LAVA project source code to gitlab.com
as gitlab is providing licenses for OSS projects like LAVA (and the server
is getting older).
The migration has been finished today. Every projects are now in the LAVA
namespace at https://gitlab.com/lava/
For instance:
* lava: https://gitlab.com/lava/lava/
* lavacli: https://gitlab.com/lava/lavacli/
* lavafed: https://gitlab.com/lava/lavafed/
* meta-lava: https://gitlab.com/lava/meta-lava/
* ...
You need to change the git configuration to point to gitlab.com instead of
git.lavasoftware.org.
As the migration process does not migrate the pipeline data, if you have
open MRs, please rebase and update them.
If you need access, please create a username on gitlab.com and request
access.
Best regards
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.10 tag has been pushed to master on gitlab.com/lava/lava.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://registry.gitlab.com/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Device-types
## New device-types
New supported devices:
* acer-chromebox-cxi5-brask
* acer-cp514-2h-1130g7-volteer
* acer-cp514-2h-1160g7-volteer
* dell-latitude-5300-8145U-arcada
* hifive-unmatched-a00
# lava-docker-worker
Add a systemd unit file to start multiple lava-docker-worker instances.
```shell
systemctl start lava-docker-worker@<instance-01>
systemctl start lava-docker-worker@<instance-02>
```
# Scheduler race condition
A scheduler race condition has been fixed in this release. This race
condition has been affecting LAVA for many years. This was causing multiple
issues like:
* #620: Cannot access Multinode Job with Sub ID
* #622: One of Multinode Job Stuck in Scheduling status forever
* A device is set to reserved state without a job assigned to it
# Packages
Publish debian packages to [lava-apt-repo](
https://gitlab.com/lava/lava-apt-repo/) gitlab page.
# XMLRPC API
## Jobs filtering
Extend the `scheduler.jobs.list` endpoint to filter jobs that take longer
than N minutes.
When there are some network issues or changes on the worker side, jobs that
handled by the worker may not updated to finish state even after job
timeout reached.
## Inactive users
Disable token for inactive users. This is already the case for the REST
API. Extend this behavior to the XMLRPC API.
Rgds
--
Rémi Duraffort
Principal Tech Lead
LAVA Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.06 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Upgrading
## Database migrations
Three database migrations are present in this release:
* lava_results_app.0019_update_query_contenttype
* lava_scheduler_app.0057_dt_permissions_worker_master_version
* lava_scheduler_app.0058_add_testjob_view_performance_indexes
The three migrations can be safely ran before upgrading and restarting
gunicorn.
If you already applied the two first migrations, you should revert them by
calling:
```shell
lava-server manage migrate lava_results_app 0018_drop_buglink
lava-server manage migrate lava_scheduler_app 0056_testjob_queue_timeout
```
Then you can safely call
```shell
lava-server manage migrate
```
# Device-types
## New device-types
New supported devices:
* acer-n20q11-r856ltn-p1s2-nissa
* r8a779m1-ulcb
# Docker images
The LAVA container images are now based on Debian (12) Bookworm.
# Docker caching
When using a docker container in a job action, by default, LAVA dispatcher
will call `docker image pull` to get the latest version of the image.
In order to drop the `pull`, ones can add `local: true`. Starting from this
release, when using `local: true`, if and only if the image is not present
locally, LAVA dispatcher will pull the image first.
If the image is already available on the worker, then LAVA dispatcher will
never force a refresh.
# Job metadata
Allow to pass any kind of objects into the job metadata:
```
metadata:
api:
name: example.com
url: https://example.com/api/
version: latest
node_id: 12345
```
# Kernel log parsing
Allow to ignore kernel errors when parsing the boot log.
In order to use this, add `ignore_kernel_messages: True` to the boot action.
# lava-docker-worker
The LAVA project is currently migrating to [gitlab](https://gitlab.com/lava/).
In order to ease the transition, lava-docker-worker is now pulling docker
images from `registry.gitlab.com` and fallback to `hub.lavasoftware.org`.
# Performances
## lava-run logs
When failing to push logs from `lava-run` to the server, wait some time
before retrying. Prior to this version, `lava-run` was DOSing the server in
case of error.
## lava-server
The LAVA server SQL queries has been improved a lot and should make the web
interface way faster.
# Prometheus
A prometheus endpoint is now available at `/v1/prometheus`. To configure
prometheus, add:
```yaml
- job_name: "lava"
metrics_path: "/v1/prometheus"
static_configs:
- targets: ["https://lavaserver.example.com"]
```
To export metrics about private devices, you should provide basic
authentication with username and token (same as XMLRPC API).
# Version format
The development version string format is now `<tag>.dev<commits>` instead
of `<tag>.<commits>.g<hash>` to follow python recommendations that are now
enforced by Debian 12.
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.05 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Relicense LAVA to GPLv2 or later
Before this patch, LAVA code was licensed under GPLv2 (or later) or AGPLv3,
depending on the python module. This release relicense all the code to
GPLv2 or later.
Approval from the relevant contributors has been requested and granted
before hand.
# Migration to gitlab.com
Most of the LAVA projects has been migrated to [gitlab.com](
https://gitlab.com/lava) in the `lava` namespace. The remaining projects
(including LAVA) should be migrated before the end of July.
# Deploy to UUU
Connect device before flashing images. The DUT serial output is now
included in job logs. This is useful to retrieve error message from boot
image raised during flash steps
# lava-docker-worker
Don't use cache when building the dispatcher docker image
This is to make sure the latest dependencies defined in Dockerfile are
built into the custom image, typically when the depends are pull from git
repo or downloaded from internet.
Also add `--use-cache` to allow to override the default behavior.
# LAVAFed rework
The lavafed project has been reworked bringing a better test overage on
real devices. The project is mostly using staging.validation.linaro.org
instance for running on physical devices. If you are willing to
participate, please provide access to you staging instances.
# LAVA server performances
The performances of the LAVA server service has been improved by reworking
the SQL queries when enforcing the authorization model. This should improve
the performances on most pages.
# XML-RPC API
Add support to delete device and worker:
```
scheduler.devices.delete
scheduler.workers.delete
```
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.03 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
## New device-types
New supported devices:
* hp-14b-na0052xx-zork
* imx8mm-innocomm-wb15-evk
* rk3588-rock-5b
## Deploy
### Apply overlay
Fix applying overlay to a custom directory.
The test directory, which defaults to `/lava-%s` (where `%s` gets replaced
by the `job ID`), can be overridden like this:
```yaml
context:
lava_test_results_dir: /var/lib/lava-%s
```
### nbd
Handle `nbd-server` running as non-root.
On Debian, nbd-server is running as nbd by default, thus preventing the
image owned by root to be mounted as rw. So LAVA should detect the
nbd-server user and chown image to it.
## Boot
### Fastboot
LAVA is now raising `InfrastructureError` on `fastboot-boot-commands`
timeout.
Timeouts caused by the DUT not entering fastboot mode or by faults in the
connection to the DUT while running pre-boot fastboot commands should be
treated as infrastructure errors.
### Nodebooter
Add support for nodebooter service for booting grub based devices. In order
to enable the support, add to the device-type template:
```yaml
{% set has_nodebooter = True %}
```
Nodebooter action will start the nodebooter container, configure it, add
the DUT via API to nodebooter and reset the device to be netbooted by
nodebooter.
### uuu
Serial availability is now skipped when the first command is `bcu: reset
usb` or when uuu block contain only bcu protocol.
## Healthz
Add a new endpoint at `/v1/healthz/`. This endpoint checks that the
connection to the database is working properly and returns either 200 or
500. This can be used by reverse proxy to check the status of the gunicorn
process.
## lava-docker-worker
Fix `lava-docker-worker` systemd service dependencies. The service should
depend on both `containerd` and `docker`.
Prior to this version, system reboot would take a lot of time waiting
lava-docker-worker to stop.
## XMLRPC
### encoding
Fix the encoding of the response from the server. This will fix encoding
issues with some specific characters.
### Device(-type)s permissions
A new set of api endpoints has been added to manage devices and
device-types permissions:
* `scheduler.devices.perms(add|delete|list)`
* `scheduler.device_typess.perms(add|delete|list)`
### Users and groups
A new set of api endpoints has been added to mange users and groups:
* `auth.groups`: (add|delete|list|show) groups
* `auth.groups.perms` (add|delete|list) permissions of a specific group
* `auth.users` (add|delete|list|show|update) users
* `auth.users.groups` (add|delete|list) groups of a specific user
* `auth.users.perms` (add|delete|list) permissions of a specific user
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.02 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
## New device-types
New supported devices:
* acer-cbv514-1h-34uz-brya
* acer-chromebox-cxi4-puff
* acer-cp514-3wh-r0qs-guybrush
* b-u585i-iot02a
* stm32mp15x-eval
## Deploy and boot methods
### FVP
Under some rare circumstances some FVP model can enter an infinite loop of
printing again and again the same lines. This was preventing LAVA to cleanup
correctly and was leaking a running model inside it's docker container.
Starting from this release, LAVA will cleanup the container correctly.
### uuu
Distinguish between normal timeout and abnormal exit when running serial
download check.
## lava-dispatcher-host package
Add support to install and use `lava-dispatcher-host` on Ubuntu 20.10 and
22.04. Also fix device sharing on Debian Bookworm.
## lava-publisher
lava-publisher is now listening on /ws/v1/healthz endpoint and returns a 200
status code. This can be used to check lava-publisher health.
## LAVA events
websocket events are not authenticated. When connecting to the websocket
stream, you will only receive events for devices that you can access. Use
the
latest lavacli release to send authenticated websocket requests.
## Performances
The performance of the server part has been improved on multiple pages.
Loading
the corresponding pages should be way quicker now.
## PostgreSQL
Starting from 2023.02, LAVA will set a default value of 600 to
`CONN_MAX_AGE`,
enabling persistent database connection. This is recommended by Django and
should improve the performances.
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
Hi folks,
The 2023.01 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
## New device-types
New supported devices:
* hp-x360-14a-cb0001xx-zork
* imx6ulz-lite-evk
* imx93-9x9-qsb
* jh7100-starfive-visionfive-v1
* mt8195-cherry-tomato-r2
## Security issues
Multiple security issues has been found and fixed in this release. The
details will be shared in some weeks in order to give time for admins to
upgrade.
## Debian support
The support for Debian Buster has been dropped as Debian Buster does not
provide support for the latest pyyaml versions.
This release also add support for Debian Bookworm.
## Action timeouts
Allow defining a timeout for an individual action in the block section.
This allows to override all the actions in a block at once, while still
being
able to control single action timeouts within the same block.
```yaml
actions:
- deploy:
timeout:
minutes: 5
timeouts:
http-download:
minutes: 1
```
## Interactive test action
In interactive test action, you can now user the server IP and Job ID in
test
commands. The variables are `SERVER_IP` and `JOB_ID`.
This can be used with:
```yaml
- test:
interactive:
- name: network
prompts: ["=> ", "/ # "]
script:
- command: ping {SERVER_IP}
```
## Settings
A new set of environment variable is taken into account when loading
settings.
Every variables prefixed by `LAVA_YAML_SETTINGS_` will be yaml loaded and
added
to the django settings.
For instance to define `Debug` to `True`:
```shell
LAVA_YAML_SETTINGS_DEBUG=true
```
This will be useful when deploying using docker-compose or k8s.
## Test definitions
This release drop the TarRepoAction test definition action.
This feature was never used in LAVAv2 and has been replaced by the
UrlRepoAction since 2022.09.
Instead of providing the tar as base64 encoded in the job definition (that
can
be huged), the user can provide a url to the tar that will be downloaded,
decompressed (if needed) and extracted.
## XMLRPC API
Lava uses XMLRPC for its system management endpoints, which is secured via
HTTP
basic authentication. This adds `system.set_user_groups` taking a string
email
address for the user, and an array of string group names which will be the
exact set of groups of which the user is a member.
This endpoint requires a token from a user with staff permissions in order
to
access it.
Rgds
--
Rémi Duraffort
Principal Tech Lead
Automation Software Team
Linaro
We have recently fixed some serious security issues on LAVA server.
CVE-2022-44641: Recursive XML entity expansion
Users with valid accounts can submit a specially crafted XML document via the
XMLRPC that causes a recursive XML entity expansion, consuming large amounts
of resources and eventually cause a Denial of Service on the LAVA server.
This problem was found, and the fix provided, by Igor Ponomarev from
Collabora. The fix has been released in 2022.11, with the following patch:
https://git.lavasoftware.org/lava/lava/-/commit/1bee0f8957741582c2bed800974…
CVE-2022-45132: Code execution in jinja templates
A specially crafted jinja2 template can be submitted to a publicly accessible
REST API endpoint without any authentication and cause a remote command
execution as the same user that is running the LAVA server web application.
This problem was found, and the fix provided, by Igor Ponomarev from
Collabora. The fix has been released in 2022.11.1, with the following patch:
https://git.lavasoftware.org/lava/lava/-/commit/ab17e8304f10c7c0fe912067f2e…
We strongly recommend that administrators upgrade to the 2022.11.1
release immediately, or failing that, at least apply the patches linked
above locally to their lava server.
