Hi folks,
The 2021.10 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Device-types
## New device-types
New supported devices:
* arduino-nano-33-ble
* imx8mp-ab2
* meson-sm1-odroid-c4
* pc-k10n78
## mt8192-asurada
mt8192-asurada as been renamed mt8192-asurada-rev1 to match the device-tree
name.
## FVP
In order to fix some issues with the secondary consoles that should be read
at all time, LAVA is now using a specific wrapper called `lava-outerr`.
The wrapper script consumes the sub-process std(out|err) as fast as
possible, keep the data in a buffer and written to std(out|err) whenever
possible.
## nrf52-nitrogen
Allow to set connect_before_flash. The value is False for backward
compatibility. You can set it in the device dictionary:
```jinja
{% set connect_before_flash = true %}
```
# External authentication
Since [2021.03](2021.03#authentication), LAVA can delegate the
authentication to GitLab using the `django-allauth` module.
This release add the possibility to use any supported [provider](
https://django-allauth.readthedocs.io/en/latest/providers.html).
In order to use it, add this to the configuration:
```yaml
AUTH_SOCIALACCOUNT: "{'gitlab':{'GITLAB_URL': 'https://gitlab.example.com
'}}"
```
See `django-allauth` documentation for the right parameters.
For background compatibility, `AUTH_GITLAB_URL` and `AUTH_GITLAB_SCOPE` are
still supported.
# Interactive tests
When running tests with LAVA test shell, LAVA will regularly read the
connections from the non-active namespaces. In the logs, this is called
"feedbacks". This is important for some device-types like fvp models.
Interactive tests are now also regularly reading the non-actives
connections to fix some issues with the fvp device-type.
# kexec
The kexec boot method is now able to crash the kernel to test the kdump
feature.
The job definition will look like:
```yaml
boot:
method: kexec
kernel: /home/vmlinux
dtb: /home/dtb
initrd: /home/initrd
command: /sbin/kexec
on_panic: true
boot_message: "Starting crashdump kernel"
prompts:
- 'root@debian:~#'
```
LAVA will use `--load-panic` instead of `--load` and will call `echo c >
/proc/sysrq-trigger` to force a kernel crash.
# Overlays
It's now possible to add single files to a given root file system (or
initrd) with the overlays feature.
The job definition should look like:
```yaml
rootfs:
image_arg: "-initrd {rootfs}"
url: "https://example.com/rootfs.cpio.zst"
compression: zstd
format: cpio.newc
overlays:
onefile:
url: https://example.com/file
format: file
path: /hello-world
```
# qemu and docker
The qemu-system packages included in the lava-dispatcher docker image are
now coming from bullseye-backports.
Rgds
--
Rémi Duraffort
LAVA and Tux Architect
Linaro
Hi folks,
The 2021.09 tag has been pushed to master on git.lavasoftware.org
<http://git.lavasoftware.org>.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release <https://apt.lavasoftware.org/release>
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/ <https://hub.lavasoftware.org/>
and
https://hub.docker.com/u/lavasoftware
<https://hub.docker.com/u/lavasoftware>
Changes in this release
==================
# Upgrading
## Database migrations
No DB migrations in this release.
# Device-types
## New device-types
New supported devices:
* Cypress cy8ckit-064s0s2-4343w
* qrb5165-rb5
* mt8192-asurada
## NXP
Make lpcxpresso55s69 support flasher deploy and minimal boot method for
trusted
firmware M.
# LAVA dispatcher
## Overlay
When using the lava_test_results_dir the compress overlay used to add
the whole
path to the tarball. This resulted in qemu guestfs mount to have double
path of
the lava_test_results_dir as a LAVA overlay home directory. This issue
is now
fixed by having the tarball contain only to bottom level directory from the
lava_test_results_dir parameter.
# Docker
## LAVA docker worker
Support providing a custom dockerfile so that users can provide the
container
with additional package installations or custom scripts they might require.
# Bug fixes
* Show template error clearly when using sync_to_lava feature.
* download: add a 60s timeout to every http request
Thank you,
--
Stevan Radaković | LAVA Senior Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs
Hi folks,
The 2021.08 tag has been pushed to master on git.lavasoftware.org
<http://git.lavasoftware.org>.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release <https://apt.lavasoftware.org/release>
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/ <https://hub.lavasoftware.org/>
and
https://hub.docker.com/u/lavasoftware
<https://hub.docker.com/u/lavasoftware>
Changes in this release
==================
# Upgrading
## Database migrations
Added queue_timeout field to TestJob model.
# Device-types
## New device-types
New supported devices:
* asus-C436FA-Flip-hatch
* dell-latitude-5400-sarien
* gemini-ssi1328
* i945gsex-qs
* imx8mp-ddr4-evk
* intel-ixp42x-welltech-epbx100
* jh7100-beaglev-starlight
* hp-x360-14-G1-sona
* sm8150-mtp
* sm8250-mtp
* sm8350-mtp
## apertis
New deployment type for apertis is now available. Apertis is a Debian
derivative with images that typically do not have bash and often the
rootfs is
mounted read-only as well.
## jacuzzi
Device type acer-mt8183-cp311-3h-jacuzzi renamed to
mt8183-kukui-jacuzzi-juniper-sku16 to fit the device tree name.
## depthcharge
base-depthcharge device type template now supports flasher method and also a
minimal boot.
## qemu
Add cpu and machine context options for i386 arch.
## i.mx
Default ```shutdown_message``` was changed to 'Restarting system'
## pyocd
Allow to set the pyocd flash tool command. Example:
```yaml
{% set pyocd_command = "pyocd" %}
```
Specifically, for nitrogen board, allow to customize flash and boot cmds by
introducing pyocd_options to pyocd parameters. Defaults remain the same as
before.
## uuu
New option is now supported, ```uuu_power_off_before_corrupt_boot_media```
which will power down the device before the power up.
# API
## REST API
Test definition validation is now part of the LAVA REST API v2.
# LAVA server
## Scheduler UI
Add maintenance column to scheduler index view.
# LAVA dispatcher
## lava_test_shell
```ip``` command is now supported in lava test shell as in some yocto
releases
there is no /sbin/ifconfig but rather only /sbin/ip.
# Docker
## test docker action
When waiting for docker shell prompt, check it didn't die if there's
some error
in docker command line.
Allow to add arbitrary docker options through the device type. Example:
```yaml
actions:
deploy:
...
boot:
methods:
pyocd:
...
test:
methods:
docker:
global_options:
- --debug
options:
- --privileged
-
--mount=type=bind,source=/dev/disk/by-id,destination=/dev/disk/by-id
-
--mount=type=bind,source=/dev/serial/by-id,destination=/dev/serial/by-id
- {{ docker_shell_extra_arguments }}
```
## LAVA docker
debian:bullseye-slim is now used as base image.
# LXC
## lxc-mocker
Added support to use overlay for LXC rootfs. In order to provide support of
running multiple jobs at the same time in LXC mocker use a overlay
filesystem
to create a unique rootfs per LXC. This functionality can be activated using
LXC_MOCKER_USE_OVERLAY env variable which is disabled by default.
# Bug fixes
* Fix the documentation for "django migrations" setup command.
* Fix environment warnings during debian package upgrade
* Fix file permissions for jh7100-beaglev-starlight.jinja2 and
stm32mp157c-lxa-mc1.jinja2
* docs: fix dead link for XMLRPC methods
* pycocd: fix crash if pre_os_command is not defined
* kernel messages: fix order of messages
* scheduler_app: always select related submitter of current_job
* qemu: Don't disable the vland block in qemu
* qemu: lava_test_results_dir is now used as a mountpoint for guestfs
* debian: depends on uglifyjs instead of node-uglify
* schema: fix docker.network_from type
* download: fix crash when content-length == 0
* lava_rest_app: lock the job when calling cancel inside a transaction
* XMLRPC: fix TOCTOU crash
* docker: set dokerfile version to 1.2
* Job error: Failed to erase old recovery image is now and
Infrastructure error
Best regards,
--
Stevan Radaković | LAVA Senior Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs
Hi folks,
The 2021.05 tag has been pushed to master on git.lavasoftware.org
<http://git.lavasoftware.org>.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release <https://apt.lavasoftware.org/release>
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/ <https://hub.lavasoftware.org/>
and
https://hub.docker.com/u/lavasoftware
<https://hub.docker.com/u/lavasoftware>
Changes in this release
==================
# Device-types
## New device-types
New supported devices:
* n1sdp
* stm32l562e-dk
* avenger96
* BeagleBone Enhanced
* asus-C523NA-A20057-coral
* asus-C433TA-AJ0005-rammus
* hp-x360-12b-n4000-octopus
## xilinx-zcu102
Fixed broken device type template.
## lpcxpresso55s69
Add support for "pyocd" boot method.
## u-boot
Add more uboot error messages encoutered on kernelci labs.
# LAVA server
## docker
### psql config generation
When running every services in the same container, generate the
configuration.
### Set environment for qemu in docker
When using qemu in docker, environment are not passed to it.
For the moment, only QEMU_AUDIO_DRV is usefull for qemu, so we pass only
it for limiting the length of the command line.
# Bug fixes
* lava-server: Remove console logs from profile template.
* doc: reminder to add the site to Chosen sites
* postinst: lava-scheduler is again a lava service
* security: Support security fix in pyyaml 5.4
* debian: lava-lxc-mocker: don't provide lxc
* lava_rest_app: test_api: don't subclass pathlib.Path
Thank you,
--
Stevan Radaković | LAVA Senior Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs
Hi folks,
The 2021.04 tag has been pushed to master on git.lavasoftware.org
<http://git.lavasoftware.org>.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release <https://apt.lavasoftware.org/release>
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/ <https://hub.lavasoftware.org/>
and
https://hub.docker.com/u/lavasoftware
<https://hub.docker.com/u/lavasoftware>
Changes in this release
==================
# Device-types
## New device-types
New supported device:
* kontron-pitx-imx8m
## imx8
The base_high_limits attribute default value is now false.
## lpcxpresso55s69
Add support for "pyocd" boot method.
## docker
### image dependencies
Two new apache2 mods are now enabled in lava-server docker image:
* mod_rewrite
* mod_ssl
Enabling these two modules by default will make docker based https
deployment
and upgrade easier.
The new dependency has been added to the docker image:
python3-django-allauth.
This makes gitlab SSO possible in the docker deployment.
### package versions
The following packages are now installed from the buster backports and
will thus
have a higher installation version:
* qemu-system-arm
* qemu-system-mips
* qemu-system-misc
* qemu-system-ppc
* qemu-system-x86
## FVP
Instead of launching a long-running container to just get the model version,
LAVA now does a one-off docker run call. This avoids leaving that container
running in the background.
# LAVA server
## notifications
A custom query argument named ```dependency_query``` is added for
notification
criteria section. If specified, the notifications will only be sent if
all the
jobs from the query satisfy the criteria condition specified in criteria
status
field. Example code:
```yaml
notify:
criteria:
status: finished
dependency_query: testjob__description__icontains__somestring
callback:
url: https://git.lavasoftware.org/api/v4/projects/1232/jobs/11113/play
method: POST
```
## configuration
LAVA now makes it possible to set the database, secret key and
ALLOWED_HOSTS for
each environment based on environment variables. It is useful for other use
cases as well, such as LAVA deployments based on containers.
Variables that can be set via env variables:
* DATABASE_URL
* INSTANCE_NAME
* SECRET_KEY
* ALLOWED_HOSTS
## Worker token API
It is now possible to set the worker token via the lava-server manage
command.
## Debian packaging
Downgraded postgresql and apache2 to a Recommends for lava-server,
making LAVA
setup more flexible.
## Master/worker version mismatch
Added a new setting variable ALLOW_VERSION_MISMATCH to disable the version
check.
Note: users should be aware of potential side effects of version mismatch.
# LAVA dispatcher
## boot/pyocd
A new option is available in pyocd boot action, ```connect_before_flash```.
This will allow to connect to device before flashing and capture serial
output
appearing immediately after flashing.
## failure retry interval
By default, individual action would be retried after 1 second, but you
can now
specify ``failure_retry_interval`` to increase the interval between retries.
```yaml
- deploy:
failure_retry_interval: 10
```
## overlay
LAVA is now adding env variables (from env-dut.yaml) to the secrets file in
overlay.
# Bug fixes
* lava-server manage sync: show device type when devices come back from
retired
* dispatcher: download: catch stderr when decompression fails
* doc: authentication: add notes to GitLab auth integration
* testdef: do not crash if a test step is None
* tar test definition: fix compression support
* containers: fix duplicate mount point for container share
Thank you,
--
Stevan Radaković | LAVA Senior Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs
Hi folks,
Due to an issue with authorization in REST API in 2021.03 release, we're
putting out a 2021.03.post1 hotfix.
The 2021.03.post1 tag has been pushed to master on git.lavasoftware.org
<http://git.lavasoftware.org>.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release <https://apt.lavasoftware.org/release>
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/ <https://hub.lavasoftware.org/>
and
https://hub.docker.com/u/lavasoftware
<https://hub.docker.com/u/lavasoftware>
Changes in this release
==============
# REST API
Issue where non-admin users could not submit jobs to unrestricted
devices introduced in 2021.03 is now fixed.
# FVP
## Add namespace to the feedback channel.
Feedback logs will now have an additional field indicating the namespace.
## Allow to specify the docker network.
Similar to the docker device type, allow to specify the docker network.
The use case is network isolation on a host running concurrent FVP devices.
Cheers,
--
Stevan Radaković | LAVA Senior Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs
Hi folks,
The 2021.03 tag has been pushed to master on git.lavasoftware.org
<http://git.lavasoftware.org>.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release <https://apt.lavasoftware.org/release>
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/ <https://hub.lavasoftware.org/>
and
https://hub.docker.com/u/lavasoftware
<https://hub.docker.com/u/lavasoftware>
Changes in this release
==================
# Upgrading
## Security issues
A security issue has been discovered and fixed in LAVA:
* removing/updating jobs via API without authentication
Every version of LAVA (since 2018) is affected.
We strongly advise to upgrade your instances to LAVA 2021.03.
# Device-types
## New device-types
New supported devices:
* imx6sl
* imx6ull
* imx8dxl-phantom-mek
* stm32mp157c-lxa-mc1
* meson-axg-s400
* meson-gxl-s905x-p212
## docker
Docker container can now run on a remote machine with the following option:
```yaml
methods:
docker:
options:
remote: {{ docker_remote|default('') }}
```
## depthcharge
A depthcharge support for x86 architecture has been added as part of the
base-depthcharge.jinja2 template.
## qemu
It is now possible to change machine type for m68k qemu jobs.
## mason
mason templates now support using booti via use_mainline_uboot flag.
# Device management
## sync command
Additional options have been added to the ``sync`` management command:
* physical_owner
* physical_group
* group_device_permissions
Example device dictionary snippet:
```jinja
{% set sync_to_lava = {
"device_type": "qemu",
"worker": "worker-1",
"physical_owner": "user1",
"physical_group": "group1",
"group_device_permissions": [
["change_device", "group1"],
["view_device", "group1"],
["submit_to_device", "group1"],
]
}
%}
```
# Authentication
## Gitlab integration
GitLab auth provider from allauth has been added as the new authentication
backend. For this the django-allauth package needs to be installed manually.
The following line needs to be added to the settings file:
```yaml
AUTH_GITLAB_URL: "https://gitlab.example.com"
```
For more information please refer to
https://docs.lavasoftware.org/lava/authentication.html
# LAVA dispatcher
## Ping interval
Time between two pings to the server is now configurable via
``--ping-interval`` argument.
# Docker
## options
LOGLEVEL setting can now be overridden by setting an environment variable on
the container.
# Bug fixes
* udev.py: Fix wait_device_board_id test
* rcar-gen3-common: fix kernel/dtb address
* dispatcher: fix JLink boot method to support LPC55S
* downloads: Add export of dynamic_data on postprocess
* job: do use sub_id with urls in templates
* apply_overlay: do not crash when the overlay is mentioned but missing
* schema: Fix docker image regexp
* map_scanned_results: try to keep most of the data when results is too
large
* docker: upgrade sentry_sdk to 1.0.0
* qemu: Do not search qemu binary when docker is in use
* rest api: Updating device dict should not check for global user
permission.
* lava-server manage wait: make the command more resilient
Thanks,
--
Stevan Radaković | LAVA Senior Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs
Hi folks,
The 2021.01 tag has been pushed to master on git.lavasoftware.org
<http://git.lavasoftware.org>.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release <https://apt.lavasoftware.org/release>
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/ <https://hub.lavasoftware.org/>
and
https://hub.docker.com/u/lavasoftware
<https://hub.docker.com/u/lavasoftware>
Changes in this release
==================
# Device-types
## New device-types
New supported devices:
* Musca S
* imx8dxl-ddr3l-evk
* hp-11A-G6-EE-grunt x86_64 Chromebook
## FVP
Multiple console.
Allow for reading from multiple FVP consoles. All output from additional
consoles is directed to LAVA feedback log.
FVP boot now method allows to define additional regexes to match more than
one console.
This is done with ``feedbacks`` keyword:
```yaml
console_string: 'terminal_0: Listening for serial connection on
port (?P<PORT>\d+)'
feedbacks:
- '(?P<NAME>terminal_1): Listening for serial connection on port
(?P<PORT>\d+)'
- '(?P<NAME>terminal_2): Listening for serial connection on port
(?P<PORT>\d+)'
- '(?P<NAME>terminal_3): Listening for serial connection on port
(?P<PORT>\d+)'
```
Feedbacks will be read twice during boot process (before matching login
prompt) and periodically during test-shell.
## docker
Mappings in static_info are now correctly added to the container as well.
## cmsis_dap
As part of job log, LAVA now logs contents of DAPLink virtual USB disk and
contents of DETAILS.TXT (if any).
Add action parameter "post_umount_delay". This tells LAVA to wait given
number of seconds after the umount command.
# Docker actions
## downloads namespace
Docker test shell action can now specify custom downloads namespace.
* if user specify downloads-namespace, use it
* if user not specify downloads-namespace, fall back to `namespace` to mount
the downloads dir.
# LAVA dispatcher
## support uniqify parameter in deploy to: downloads
This will cause the downloaded files to be available under ${image_key}/,
instead of at the root of the downloads directory. It allows jobs to
download more than one image with the same filename.
# Bug fixes
* REST API: __in filtering now works correctly for `choice` fields.
* REST API: fix field names crash in RelatedFilters due to django filters
upgrade
* lxc-mocker: avoid `apt upgrade` on lxc-create
* lava_dispatcher: add wait_device_board_id in device_info for fastboot
Thanks,
--
Stevan Radaković | LAVA Engineer
Linaro.org <www.linaro.org> │ Open source software for ARM SoCs
Hi folks,
The 2020.10 tag has been pushed to master on git.lavasoftware.org.
.deb packages have been built in GitLab CI and are published at
https://apt.lavasoftware.org/release
Docker images for amd64 and arm64 have been built in GitLab CI and
are available from
https://hub.lavasoftware.org/
and
https://hub.docker.com/u/lavasoftware
Changes in this release
==================
# Upgrading
No database migrations are included in this release.
# Device-types
## New device-types
New supported devices:
* imx8dx-mek
* imx8mq-zii-ultra-zest
# Secret management for deploy action URLs
We now support custom headers in the GET requests for deploy action URLs
(images, kernel, ramdisk etc) in the job definition [schema](
https://docs.lavasoftware.org/lava/pipeline-schema.html#job-submission-sche…
).
This mainly serves for passing the authentication tokens for the services
which
require one (ie. gitlab).
LAVA now also supports per user token management (found in both admin app
and
in the Profile page). Every entry will have a `token name` and the token
itself.
If the `headers` entry in the job definition contains the existing token
name
in the list of submitters list of tokens, the system will replace it with
the
token value in the header when downloading artifacts from the deploy
section.
The token values will never be visible in the job definition and also the
user
submitting the job will only ever be able to use own tokens.
Example:
```yaml
images:
kernel:
image_arg: '-kernel {kernel} -append "console=ttyAMA0,115200
root=/dev/ram0 debug verbose"'
url:
https://git.lavasoftware.org/api/v4/projects/61/jobs/112514/artifacts
headers:
"PRIVATE_TOKEN": my_private_gitlab_token
sha256sum:
c45f07a9c0ebd0c06a9dcb9676d7aada63967e0733d072f9e529b697a8a7c6ca
```
# XMLRPC and rest APIs
## Job and worker token
Super users can now get Job and Worker token from the rest api endpoints.
The field is called `token`.
```json
{
"hostname": "staging01.lavalab",
"health": "Active",
"state": "Online",
...
"token": "<worker_token>"
}
```
For XMLRPC, only the worker token is available.
# Notifications and celery
Sending notifications for large jobs can take a large amount of CPU and
time. For instances that uses a lot of notifications, admins can delegate
this work to a celery worker.
In order to use this new feature, admins should:
* pick and install a celery broker (rabbitmq, redis, ...)
```shell
sudo apt-get install redis-server python2-redis
```
* update the settings
```yaml
CELERY_TASK_ALWAYS_EAGER: false
CELERY_BROKER_URL: "redis://localhost:6379/0"
```
* restart `lava-server-gunicorn` and activate `lava-celery-worker`
```shell
sudo service lava-server-gunicorn restart
sudo systemctl enable lava-celery-worker
sudo service lava-celery-worker start
```
The use of `celery` and `lava-celey-worker` is optional. By default, LAVA
will continue to send notification like in previous releases.
## DUT environment
Environment variables can be set on the DUT by updating the configuration
file. The configuration could be stored in:
* `/etc/lava-server/dispatcher.d/{name}/env-dut.yaml`
* `/etc/lava-server/env.dut.yaml`
The filename where not consistent. Starting from this release, the
configuration files are:
* `/etc/lava-server/dispatcher.d/{name}/env-dut.yaml`
* `/etc/lava-server/env-dut.yaml`
* `/etc/lava-server/env.dut.yaml`
`/etc/lava-server/env.dut.yaml` is **deprecated** and will be dropped in a
future version.
# Crashes
Some crashes where fixed in the release:
* `docker test action` crash when using test definition dependencies
* `lava-dispatcher-host` crash in race condition
* `lava_scheduler_app` crash with some really long log lines
* `lava-server manage linaro_ldapuserlist` crash after python3-ldap upgrade
* `lava-worker` crash with version mismatch
* `lava-worker` crash when `result.yaml` is empty
Thanks
--
Rémi Duraffort
LAVA Architect
Linaro
