- lava-announce - op-lists.linaro.org

LAVA 2017.11 release

by Neil Williams

See https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html 2017.11 is the second release on the roadmap to the removal of V1. This is the single largest change ever made to the LAVA packages. * All dashboard URLs are permanently disabled in lava-server. * All devices which were not enabled for V2 are now hidden and unusable. * All V1 code has been removed from lava-dispatcher. * All V1 documentation has been removed from lava-server-doc. 2017.11 disables all access to any V1 test data - including V1 only devices. The data itself remains in the database but is not accessible via any API. If you have chosen to keep an archive of V1 test data and have not yet done so, you should do this before installing this upgrade. At the very least, you **must** have a backup of your postgresql database **before** you upgraded to 2017.11 in order to make a V1 test data archive later. 2017.11 requires that **all** workers and each associated master are upgraded to 2017.11 for test jobs to continue running. This is because there have been very large changes to the directories and file layout within both lava-server and lava-dispatcher associated with the removal of the V1 components. The LAVA software team have been checking ahead for the final stage of the roadmap and plans are on track, as described. http:// and https:// changes in django ====================================== A reminder, if you are accessing an instance using http:// (whether localhost or not), then the changes described in the documentation **must** be applied for the instance to operate correctly. This is a security change in Django, not a change in LAVA. https://staging.validation.linaro.org/static/docs/v2/pipeline-debug.html#us… Symptoms include HTTP500 errors but can also include the inability to login to an instance using LDAP or local django accounts, depending on your local apache configuration. Changes in 2017.11 ================== Changes in the 2017.11 release are described below, including the short git commit hash. Use the git commit hash to go directly to that commit using a URL like https://git.linaro.org/lava/lava-server.git/commit/?id=b19b9648a for lava-server or https://git.linaro.org/lava/lava-dispatcher.git/commit/?id=1451e3a4 for lava-dispatcher. Where the changelog message includes a story id like LAVA-552, that id can be added to the base URL: https://projects.linaro.org/browse/ to find out more about the change itself. e.g. https://projects.linaro.org/browse/LAVA-552 All LAVA stories are publicly available without requiring a login. 2017.11 also includes large changes to the packaging of both lava-server and lava-dispatcher. The prompts formerly used to configure a V1 remote worker have been removed. Packaging is maintained at https://github.com/Linaro/pkg-lava-server and https://github.com/Linaro/pkg-lava-dispatcher Packages for 2017.11 have been uploaded to Debian unstable and also the LAVA repositories (for jessie-backports and stretch-backports). lava-server =========== a799fc6c Fix links back to the main instance 5b9656b6 Fix broken plural in header b21f8c8f Fix crash in job_section_log c9526f01 Document the LOG_SIZE_LIMIT setting in settings.conf 63b57c94 Remove lava-mount-masterfs 75b403f4 Remove add_device script 8ddea823 Add description about the disc space requirement for pg_upgradecluster. 5c9dceb2 Remove TemporaryDevice imports 9b21ea48 Removed unused Worker functions 2accb4c3 Matching change to templates for directory change 1d384c4a Update docs regarding device-dictionary lava-server manage command. 32472fac Port 77fd10e2 to scheduler.devices.get_dictionary 114bbfa5 LAVA-1093 - Add a management command to drop all materialized views 8bc9d83a Update dashboard XML-RPC API help for api_version bb090e96 LAVA-552 - add system.api_version support 77fd10e2 LAVA-1092 support passing a context to device_config de7ad3aa Remove dashboard_app views and static files. 4bc68531 Remove templates from dashboard_app. 1a618c27 Remove unused functions d50deca7 api: limit the number of jobs returned 59880e14 Fix HTTP 500 on devicetype health history a39fecf5 Remove unused resources (css, js and images) a32ac3b6 Rename base-bootstrap and content-bootstrap 7c108de1 job: print lava.job result in case of failure b51c419f Remove old templates a9139c1f Use bootstrap template 7915f18f Remove unused resources 7331901a scheduler.api.jobs.show: add failure_comment 71233bff Fix print syntax for python3 compatibility 936d5eca Remove unused tags 4ebb2a39 Remove unused sources 21a7eb18 Save more sql queries by caching values 4d0d867e Remove non pipeline jobs support a6648dbd Retain docs on disabling V1 workers 8eac0d74 Revert "LAVA-950 set the master identity" 68573b1a Remove V1 dependencies a8151559 timing: handle new start/end line format 26cc3ad0 Remove unnecessary js lib beautify. c44009fa Simplify job view e1da85d4 Fix loading of description.yaml 7b41f7ef Add documentation for secondary media writer parameters 1b4d3dd5 LAVA-1081 migrate instance name to settings f3dc1987 Fix up typos from earlier changes faa24f59 LAVA-1079 - remove HIDE_V1 and HIDE_V2 doc options 942b84c3 Remove unused functions and imports e61a4831 LAVA-1056 Drop V1 documentation 126dce0c Fix crash introduced by 42451c9cc f0b28214 Remove v1 codebase 31bb1bf0 Fix notification exception when query is used for data comparison. a6cff700 Remove last references to lava_dispatcher v1 043690d9 Remove references to lava_dispatcher v1 4c9e493c Remove dashboard_app traces from rest of the codebase. c371b8ca drop link to removed page 1b1b638e LAVA-876 - Return empty data stubs for dashboard API calls. 919c10c2 Remove links to dashboard_app in the scheduler_app 1483e9fb Remove leftover from job wizard f7a1c976 docker: allow passing extra options 42451c9c Use all_jobs_with_custom_sort when applicable cc4ffd64 Fix broken links in example test job 0ca5d75c Drop the migration status page. b37fe944 Add a device-type config for the rpi-3b in armv7 32bit mode 9e904a18 Revert "LAVA-876 - Remove access to Dashboard" f07e0950 Fix warning regarding naive datetime e8348425 Remove more references to dashboard_app 103a44be LAVA-876 - Remove access to Dashboard d33c9c98 LAVA-876 - Remove access to Dashboard b6fc46aa Revert "LAVA-1038 add a settings to archive the instance" f75a333d Do not run dashboard test anymore d9163445 Remove references to v1 jobs addb61d9 bug 3268: fix lava-master crash with invalid logs febf91b1 Improve advice on api/help ac8b7df3 fix pep8 error 64025cf3 LAVA-1072 test all template connection syntax 10d5fbe1 Adjust U-Boot load addresses for tegra124 devices to allow bigger kernels f30ef936 LAVA-1065 - Remove dashboard_app urls server wide. 096a2f1c LAVA-1049 - Allow for .yml as well as .yaml for healthchecks 918f0bc7 Fix logic so addldapuser and mergeldapuser work for all LDAP configs 36fcb3c2 Documentation changes for multiple uarts b1de8acf Update instructions for migrating postgres 0ff08373 LAVA-1053 Results limit does not work for queries f9e64aa1 Fix SQL request storm when listing jobs 2d9c4757 No need to save after get_or_create or Create() 8988cb36 LAVA-896 fix level in result export 219fa3dd LAVA-1073 - support for a second UART on a device c934bbbe master: fix database reconnection 4f0e0e62 LAVA-950 set the master identity 316d77fb Expand the device integration guidance. lava-dispatcher =============== d91a5b99 Revert "Stop parsing kernel messages when the end of a panic has been found" 1451e3a4 LAVA-1098 unassigned variable in read_feedback 437d57ba Rationalise the devices directory structure bdffb3bf Add power control to pyocd d9553680 Add feedback check to test shell 06588ddf Receive the udev device directly from the udev rule d811b2a8 Fix license 65d6c6a6 Move arduino101 dfu example urls to images.v.l.o 61cff41c Only log when feedback content was found 0ba35392 Move juno removable URLs to images.validation.linaro.org cb983d97 Cleanup after pytest runs 997265c2 LAVA-1086 add handler to listen to feedback 29a60e38 Add ConfigObj dependency for TFTP check 70bd3fab Fix prospector warnings 6edb8ef9 Fix missing import and other pylint issues 6f7f5f05 Remove lava-lxc-device-* scripts which are unused. 578b054b Use py.test 65c03eca docker: allow passing extra options 7b7d920a device-info should be yaml ef8f7d4c Remove deprecated function 165e14f8 Fix linger period: set it to 10s instead of 5ms 49c500ac Fix double crash when the process crash early 7dbdc2b6 Add test case for secondary deployment writer tool parameters 969459c6 LAVA-1069 - migrate dispatcher device configs 9e8e5258 Use optargs to extend ci-run for python3 e1c1561b Fix python3 breakage in unit test 69277753 Read feedback when finalising connections 0fb44088 LAVA-1067 Refactor fastboot flash operations 9ac71872 Add missing import 94430958 Ensure finalize always finalises protocols 9d712364 Add option to not uniquify secondary image download paths cdfd5909 Fix master cert handling for lava-run and lxc helper 73548f7a Rename dd params to tool for removable media 1802c3e8 Handle master and slave certs as filenames only ed306477 Do not rely on SysV runlevel to ensure container is ready. 75b47c6a Remove loop mounts via guestfs, while applying overlay to sparse image. cd1152e3 Support archlinux and slackware rootfs 6fc024a5 Remove deprecated modules ddf5d2fc Remove v1 code 5561be23 LAVA-1074 - second UART support 758a41a6 Allow configurable writes for secondary storage deployment fb75db4a Add flag to disable uniquifying the download paths -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 1 month

1
0
0 0

2017.10 Production release

by Neil Williams

Disabling V1 test job submissions ================================= **IMPORTANT** 2017.10 starts the roadmap to the removal of V1 by DISABLING all V1 test job submissions. Queued V1 test jobs will continue to run but once 2017.10 is installed, all attempts to submit a V1 test job will be rejected. This includes health checks for devices which do not yet have a V2 health check. (So those devices would go offline - remove the health check job from the database entry of the device-type to get the device back online if there are queued jobs.) Before installing this upgrade, check the Migration Status page on the particular instance. e.g. https://staging.validation.linaro.org/scheduler/migration Notification on installation ============================ By default, installing 2017.10 will raise a notice, reminding you of this change: The 2017.10 release **disables ALL V1 test job submissions**. Queued test jobs will continue to run but if you proceed, only V2 submissions will be accepted by this instance. . If you want to create an archive of the existing V1 test data, install this version but do not allow further upgrades of LAVA packages. . See https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html If you are deploying this change using configuration management, you may need to define this environment variable to allow the installation to proceed: APT_LISTCHANGES_FRONTEND=none Window for creating a V1 archive ================================ Once installed, admins can choose to wait until all queued V1 test jobs have finished and then use a backup of the postgres database to create a read-only V1 archive. Install 2017.10 on the archive instance and then restore the backup database. All dashboard objects in 2017.10 are read-only. Admins will need to ensure that all devices on the archive instance match those on the original instance and then change all devices to Retired status in the django admin interface. See also https://staging.validation.linaro.org/static/docs/v2/pipeline-server.html#s… Archive repository support ========================== 2017.10 will be preserved in a dedicated LAVA repository in case an archive has to be reconstructed from a database backup after 2017.11 is released. http://images.validation.linaro.org/v1-archive-repo/ lava-server =========== 03fec0ea Fix out of date stretch reference 2c8a88d8 Include non https in title for security defaults configuration doc. 56185add Mark required parameters in left over deploy action reference. b1de8acf Update instructions for migrating postgres 2b845967 Fix issues in the documentation build. f9e64aa1 Fix SQL request storm when listing jobs 95b12c01 Fix missing import 0a63c2f3 Adjust U-Boot load addresses for imx6q-sabrelite to allow bigger kernels 3a4cddf4 LAVA-341 Unicode issues in tables and queries 5bff2fd5 Mark required parameters in deploy action using an asterisk. 00b5044a LAVA-1035 Force all Dashboard objects into read-only mode for users 7a006f64 Change load addresses for Jetson TK1 to allow bigger kci kernels ca5e6834 Update commands for eMMC boot of OE on X15 0d17e797 Add docker device-type configuration f0c2107c Remove doc references to lava-lxc-device-* commands. 715c6309 LAVA-771 - Support dragonboard-820c with lxc 7a4adb03 Fix the hierarchy of md5sum and sha256sum in nbdroot index. 576661f4 Make clear when to use lxc:// 7c68e21b Document download deploy and explain about lxc:// url scheme. 8f4d7419 Rewrite fastboot deploy action reference doc. 11f5c222 archive: do not allow to force hc 9536450d logger: drop level and action name 771d0d93 Fix a typo in deploy to name. 0a2404fa Update command requires an alias argument e7a704a1 Add V1 EOL summary to the index page for all docs cdaa6376 LAVA-1003 do not lose logging messages f09b6beb LAVA-1045 - document the archive setting 483fb1a7 LAVA-1038 add a settings to archive the instance 866afbf4 migration: v1 health-check are not used anymore 896db15b LAVA-1033 remove v1 job wizard 2eac055b LAVA-1043 fix handling of large log files aef7d7b8 Add device-type disco-l475-iot1 399b9dbc Use the CDumper to export results c9dc6a14 LAVA-1032 reject v1 submission at api level 65394986 Drop lookups to ActionData from testcase export 80d1a9fa Add support for the NextThing CHIP Pro 21c8fc52 Only write logs to output.yaml 01df9ce0 Improve job listing by fetching more data 436c220e Decrease SQL queries when listing test results 445b9c66 Add support for reading django settings for limiting LDAP group access a33b9c9f Only set the job status if lava-run crashed 581b616f Fix boot action doc for fastboot and lxc boot methods. 08393dbb Use new logging API 33a73bb3 device-types: panda: handle bigger kernels 746fe4c9 Allow to override prompt in device configuration 5932f9d1 LAVA-1048 Extend X15 to typical U-Boot support affaa35d Fix error message in daemonise.py e24db160 Fix lambda syntax 8fb16105 v2: Do not fall back to db for health-checks bceecc64 Do not submit new health checks for v1 only device lava-dispatcher =============== 1b129396 Remove unnecessary calls to get_udev_devices. 55cc9a0d Replace LxcAddDeviceAction with LxcCreateUdevRuleAction. 9cf899df Include vendor and product id if available, while writing udev rule. 5eea8b61 Add missing requirement for magic binding 75b47c6a Remove loop mounts via guestfs, while applying overlay to sparse image. 1f285ecf Ensure the image is Android sparse image before acting on it. 44b8436a Add missing dependency on pytz b953a562 Append to the udev rules and avoid overwriting. 44380b3e Add a new device type: docker 336719bf Fix broken link in hikey960 unit test. 1703dc94 Remove unused signals after introduction of udev based device addition. 7f2f049e Always don't assume a ZMQ handler for logging. 9fa3995e Fix a typo in log message. 9cab91d0 LAVA-1046 allow to send logs from lxc udev scripts 37c7e863 log: allow to add a linger when closing socket dd4f6b09 LAVA-1040 - Initial boot only support for artifact conversion in LXC 7ed0674c logger: remove level and action name 70fb64c4 lava-run: output_dir should be an absolute path 8f31f584 Always revert directory changes 40ca2c11 Add a timezone aware log message at job start 79325b7b Ensure a retry sets a failed result b8527df2 Debug log message to know image files are copied to LXC. 3665e840 Allow fastboot calls to fail and return the log 7de0f887 Fix bug #3007 - Unable to reboot target between tests on hikey-hi6220 dc5e5b18 Send a bad status only when lava-run crashed 18607373 Write description.yaml when all logs where sent to lava-master e185d967 Move most exception handling to lava-run 57829892 Move signal handling to lava-run d9d39463 Provide useful output when no classes accept an action 1c45a4ad Setup logging as early as possible af4401f0 LAVA-1048 extend X15 support for U-Boot 12711359 Raise an error instead of setting self.errors in run() -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 2 months

1
0
0 0

2017.9 Production release

by Neil Williams

The 2017.9 release is now available from the lava-repositories. To support Stretch, the package available from the lava-repositories now has a version suffix matching the suite for which the binaries were built. So instances using jessie-backports will see 2017.9-1+jessie and instances using stretch-backports will see 2017.9-1+stretch when using the lava repositories. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… lava-server | 2017.9-1+jessie | jessie-backports lava-dispatcher | 2017.9-1+jessie | jessie-backports lava-server | 2017.9-1+stretch | stretch-backports lava-dispatcher | 2017.9-1+stretch | stretch-backports Additional packages from jessie-backports ========================================= $ sudo apt -t jessie-backports install python-pexpect python-setproctitle python-ptyprocess Make sure these packages are updated from jessie-backports before installing 2017.9. Versions in stretch are already sufficient. jessie-backports "sloppy" ========================= This release has been uploaded to Debian unstable. After a period of time, an upload will be made into jessie-backports-sloppy which is the place for backports from buster to the previous stable release. An upload to stretch-backports will be made at the same time. To install the new release from jessie-backports-sloppy, a suitable apt source will need to be added - LAVA still needs some packages from jessie-backports. e.g. to use the bytemark Debian mirror, add this line: deb http://mirror.bytemark.co.uk/debian jessie-backports-sloppy main Jessie continues to be the preferred release for LAVA but we are planning to move to Stretch in October. We don't anticipate any problems with lava-server or lava-dispatcher on Stretch as this has been tested frequently. Django changes for localhost ============================ Django upstream have changed some of the security defaults and this has an effect when using http://localhost - 2017.9 contains documentation on which settings are required to get localhost working again. https://staging.validation.linaro.org/static/docs/v2/pipeline-debug.html#ch… Reminder about the migration status page ======================================== During the time 2017.9 is installed, all instances need to get to a similar state of device migration as staging has currently: https://staging.validation.linaro.org/scheduler/migration * All active devices migrated to V2 support. * All active V2 devices exclusive to V2. * All active devices have migrated to V2 healthchecks * All active devices have healthchecks enabled. The important phrase here is 'active'. Health checks can be disabled for a complete device-type in the django admin interface for that type, that does not affect completion of the migration status. Retired devices are not counted at all. During the changes to remove V1, various of the checks which are done for the migration status page may also be required to be able to start or complete the next stage of the removal of V1 - including checks which could block the installation of a future release if not completed in time. More details on this will be announced as and when the checks are created, based on the testing done during development. The roadmap to remove V1 begins now =================================== https://lists.linaro.org/pipermail/lava-announce/2017-September/000037.html **2017.9 is the LAST release which will allow V1 test jobs to be submitted and executed.** As of 2017.10, V1 test job submissions will be DISABLED and the process continues to the removal of access to V1 data in 2017.11 and the removal of V1 test data in 2017.12. This means that if an instance is running V1 test jobs after the 2017.10 release, the only support available will be how to migrate to V2. lava-dispatcher =============== 255e1dc2 Allow to use lava-run from the command line 5e3e4df8 parser: fix exception with unknown action b167715b Fix exception message 75ef463a Add transfer_overlay support to fastboot 4fcdbb0c LAVA-1020 - Allow branch to be specified for git d067a6e4 Remove FIXME about unused feature test-case-deps. 93947b9d Fix job tmp_dir NoneType error. cf0095ca Extend NBD unit tests for template update ca72b0cc Honour reboot to bootloader irrespective of power state existence. b210b297 LAVA-1009 - Improve cleanup of temp files and containers 70fb8c09 Add debug log at the right place for lxc-destroy command execution. 51938bf5 Stop executing shell commands twice. 60c39bae Create device-info-file consumed by udev rules in job's tmp_dir. 46a9789d LAVA-1012 - Handle udev paths outside /dev/bus/usb c6e06463 LAVA-998 - Default to shallow clones in overlay b9cc1da8 LAVA-1001 - raise TestError on invalid signal call 4a8241e1 Use check_output instead of Popen from subprocess module. 7024de4c Move to permanent URLs for better unit test support 4f32f4e2 Fix reference to smoke test 927bb33a LAVA-1011 - Check for ENOSPACE after adding overlay to sparse image 47608877 Add udev rule file contents to debug message. 15d8adea Add more debug messages for udev rules addition on the fly. 1b2c9252 Ensure auto-login-action can execute protocols 44755d5a CTT-441 - lava-lxc-device-* don't work properly d55526c4 Add option to enable IPv6 connections 07839bdf Raise a JobError when lava_test_results_dir is undefined 3f9c87f5 Remove uneeded test 003990f7 Add setproctitle as a python installation requirement to setup.py d04dcf58 Fix prospector warnings c2601cbe Extend pipeline_reference readme 115d84ad Create a clean script to replace lava-dispatch for v2 jobs 80bfa16e device: remove 'hostname' and 'target' b663e427 Do not crash when cleanup raises and exception fcbff4a6 Hikey 960 support for OE and AOSP bab3c6f1 Replace kernel.txt with a normal boot and add kernel-panic.txt c61c67bc Fix value of self.results['success'] in pipeline boot action 6939c5ea Add pre-power command support to LXC protocol 98ede789 LAVA-996 - preserve empty parameters in the overlay 9d33ced5 Use encoding to pexpect.spawn for python3 support ce2dbd83 Stop parsing kernel messages when the end of a panic has been found b4ae5b4d lava-slave: Pretty print the configuration cf22ed83 Extend secondary media to mustang 3e746b47 Add network block device boot support lava-server =========== 6d1ff5c5 Improve the docs on lava-test-reference and metadata 2f11898f LAVA-997 - Invalidate primary connections setup when power is configured. ccfb3961 b2260: allow to set bootloader_prompt in device config 4a1b8f5f Prevent mistaken fixup of device reservation 5093f40c LAVA-1022 - V2 support for IFC6410 1f18eada LAVA-1021 - Document specifying branch for git 36513ec3 Change default source code URL after systems change da49e558 Remove doc for unimplemented feature test-case-deps. 61dfd204 Document settings to enable authentication in http://localhost 5cbf5b11 Allow vcs checks to skip monitor test actions b85baa75 NBD kernel bootarg fixups c089adb7 Add failure comment message via async update on the job detail page. 5ca25620 Improve V1 doc build for Debian reproducibility dccd1588 LAVA-998 - Default to shallow clones in overlay 250f27dd Fix warning of linaro.img not found. 938b3da1 Fix typos and warnings in deploy to nbd doc. 90c9ed74 LAVA-1013 improve performance of result export 0af80d19 LAVA-1008 Group visibility check 080dd10d LAVA-1007 Prevent NoneType crash in job duration b2d90962 Fix typos in manage users which talk about devices 07e5fa19 Fix reference to smoke test 59340f70 Expand advice on good commit messages. cfae2faa Skip some dashboard tests if django is too new 68b24fcc Revert "Move action fields templates to table files." c9f65507 Add log file argument to dispatcher-master command. dabaf74c LAVA-1002 - fix group label editing in query details page. 7398abd4 CTT-441 - lava-lxc-device-* don't work properly f1cb7faf Decrease the number of SQL queries to list the groups 984f3b02 doc: fix example of zmq_client usage f864696d Documentation build error tweaks 9aeff523 XMLRPC authentication: fix wrong logic 93572067 Basic integration advice for U-Boot devices 5b2d39d7 Support PyOCD for KW41Z and remove CMSIS boot method 3f8c8a93 Fix a typo in persistent container configuration doc. d334642c Rename xmlrpc module to api 9f947433 Add ssh deployment support for secondary connections. ce9f0da6 Document how to combine LXC and MultiNode e2be8567 Allow documentation to be built with python3 cc6e9696 Add notes on writing new unit tests c40ac0da Document option for device_type qemu to choose host audio handling 97eac4b6 Update lava-self documentation for change in output 7e0f6eca Fix PipelineDevice usage after last dispatcher commit 37d4eba4 Fix a typo - repeated usb_vendor_id parameter in device_info example. d4a80fc1 Improve the validation message when a role is not found in every 1a5aec5f Add a seealso for fresh installs of lava-tool 4707ff98 Check that the Debian package has been installed 72d30834 Mention cookie specific options for reverse proxy 716424a3 api: fix scheduler.jobs.submit's docstring 6dc1a966 Add sudo for the apache config copy command in V2 worker setup doc. ae39d57b Add docs on calling protocols from actions a49cc9c6 Expand the device integration guidance. 1fd5e964 Add Integration Stories for supported devices 620dcbf1 Move smoke-tests into functional test repository ad2f57df device-type: update at91-sama5d4_xplained addresses b6734844 device-type: update at91-sama5d2_xplained addresses c4ead793 device-type: update sama53d addresses 1204f81c Skip links to secondary connections device-types 8b9068a3 Add Metadata output to verbose notifications. 2da8c076 LAVA-357 - Allow override of the nfsvers in base_nfs_root_args 5926b9a8 Catch TemplateError in notifications 98b3da07 Initial improvements for the Queries and Charts docs 092f56a5 device-type: update armada-xp-linksys-mamba addresses 0fdf33a8 device-type: update armada-388-gp addresses 5af8f640 device-type: update armada-370-db addresses 9f4c975f XMLRPC: self.user return an AnonymousUser instead of None d7b34e2d Add an helper command to switch to a dev setup 9bedff1e Add option to enable listening on IPv6 8ff7c4f1 Extend secondary media docs f132af4e Add support for secondary media for mustang 1fecd500 Integration of Hikey 960 b27f60c4 LAVA xnbd protocol support and documentation for nbd boot -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 3 months

1
0
0 0

Roadmap to the removal of LAVA V1

by Neil Williams

The LAVA software team are happy to announce the roadmap for the removal of LAVA V1 support, test data and source code. The next LAVA production release will be 2017.9 which will be the last release with functioning V1 support. There are no changes to V1 in 2017.9, so it will continue working as it does currently for this release. https://projects.linaro.org/browse/LAVA-994 tracks the work for 2017.9 Any instance where V1 test jobs need to continue running must NOT upgrade beyond 2017.9. There will be no future updates of V1 for any reason and no support for anything related to V1 other than to migrate V1 test jobs and devices to V2. Blocking V1 test job submissions with 2017.10 ============================================= The 2017.10 release will *disable* the submission of all V1 test jobs either from the UI or using XML-RPC or lava-tool. If any devices still have V1 health checks, those health checks will no longer run. Any queued V1 test jobs will run. Only V2 test jobs will be allowed to be submitted. https://projects.linaro.org/browse/LAVA-1026 tracks the work for 2017.10 If you want to keep a read-only archive of your V1 test data, then 2017.10 is the last chance to create this archive. Any instance where the V1 data should remain visible must NOT upgrade beyond 2017.10. Removing access to V1 URLs in 2017.11 ===================================== All URLs related to V1 will be removed or disabled in the 2017.11 release - the Dashboard drop-down menu will disappear and any bookmarks to dashboard URLs would give a 404. The Dashboard XML-RPC API will become no-ops returning no data. This means that bundles and bundle streams will not be able to be downloaded or viewed. Filters, Image Reports and Image Reports 2.0 will all become unavailable. V1 documentation will be deleted in this release - the V2 docs will be promoted as the LAVA documentation, removing references to V1. The V1 source code for lava-dispatcher will be deleted at this point. There will be some changes to the directory structure of the remaining code, so if you are maintaining out of tree patches, you will need to fix some merge conflicts. Elements of the web UI which distinguish between V1 and V2 support will also be disabled and removed. This will include removing access to viewing or downloading the log files of V1 test jobs. https://projects.linaro.org/browse/LAVA-1028 tracks the work for 2017.11 DELETION of V1 test data in 2017.12 =================================== This has been highlighted many times in LAVA announcements, Connect sessions, in the documentation and in all support enquiries. It is finally happening with 2017.12 Any instance which installs 2017.12 will directly cause the deliberate and permanent deletion of ALL V1 TEST DATA for that instance. Bundles, BundleStreams, Filters, Image Reports, Image Reports 2.0 - everything in the database which relates to V1 will be deleted. Alongside the removal of the test data, the V1 source code will also be deleted from the lava-server codebase. *This is a one-way process*. If there is no usable backup for the instance, there will be NO way to restore any V1 data after this point. After restoring from backup, if 2017.12 or any later release is installed, the V1 data will again be deleted. https://projects.linaro.org/browse/LAVA-1029 tracks the work for 2017.12 -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 3 months

1
0
0 0

Installing LAVA on Stretch

by Neil Williams

There are delays getting packages into stretch-backports, as expected. In the meantime, this is a reminder of how to use backports: first start with stable. So when installing LAVA on Stretch, even if what you want is the latest release from production-repo or staging-repo (currently 2017.7), then your first step is to install 2016.12 from Stretch. # apt -y install lava-server # apt -y install vim # wget http://images.validation.linaro.org/production-repo/production-repo.key.asc # apt-key add production-repo.key.asc # vim /etc/apt/sources.list.d/lava.list # # edit the file to specify: deb http://images.validation.linaro.org/production-repo jessie-backports main Yes, that is jessie-backports - we don't have packages in stretch-backports at this time. # apt update # apt upgrade If you specify backports too early, you'll get a range of other packages from backports - you don't actually need jessie-backports or stretch-backports from Debian at this time. Packages for jessie-backports are built on jessie. Packages for stretch-backports are built on stretch. It's the same source code in each case. Right now, there aren't any problems with installing from jessie-backports on stretch - particularly if you install lava-server from stretch in the first place so that the majority of your packages come from stretch. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 4 months

1
0
0 0

Stretch upgrades, UWSGI and Apache

by Neil Williams

During testing, we've seen that upgrades to Stretch from Jessie can have an issue if UWSGI is still left from a previous installation: apache2: Syntax error on line 146 of /etc/apache2/apache2.conf: Could not open configuration file /etc/apache2/mods-enabled/uwsgi.load: No such file or directory Action 'restart' failed. The Apache error log may have more information. dpkg: error processing package lava-server (--install): subprocess installed post-installation script returned error exit status 1 Setting up lava-server-doc (2017.7-1~bpo9+1) ... Setting up lava-dev (2017.7-1~bpo9+1) ... Processing triggers for systemd (232-25) ... Processing triggers for man-db (2.7.6.1-2) ... Errors were encountered while processing: lava-server The problem is because of a bug in the uwsgi packages. The uwsgi module is not cleanly removed from apache. Run this command to fix: $ sudo a2dismod uwsgi Then complete the install: $ sudo apt-get -f install Fresh Stretch installs should not be affected as 2016.12 in Stretch does not use uwsgi. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 5 months

1
0
0 0

2017.7 release

by Neil Williams

The 2017.7 release is available from the lava-repositories for jessie-backports and has been uploaded to Debian unstable. Support for stretch-backports is pending. (The production-repo/jessie-backports can be used in the meantime.) https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… jessie-backports "sloppy" ================== This release has been uploaded to Debian unstable. The period after a Debian stable release is turbulent in unstable and the LAVA packages may not migrate into buster (testing) smoothly which will delay uploads to the official backports archive (for jessie backports and stretch backports). To update jessie-backports, the upload will need to be the version from buster, rebuilt for jessie and sent to jessie-backports-sloppy. Once that is done, I'll post again to the this list about how that becomes available to admins running Jessie systems. Jessie continues to be the preferred release for LAVA until such time as we've done more testing on Stretch. We don't anticipate any problems with lava-server or lava-dispatcher on Stretch as this has been tested frequently. However, third party packages and scripts still need testing with real instances using Stretch. V1 support in buster =============== Some unit tests in the V1 support have had to be disabled due to bug fixes in django 1.11 LTS in buster and sid affecting the atomic transaction handling. The remote worker design in V1 has always had issues with atomicity and these unit tests have failed intermittently for a while. This is one of the many reasons to drive all instances to V2. These tests are now failing consistently with 1.11, so have been disabled if django 1.11 or newer is detected. V1 is no longer run on any of our development instances (running buster or sid), only in production instances (running jessie) which are in preparation for the migration. We can only assume that the failure of the unit tests would make V1 unreliable on buster or sid. However, with the pending removal of V1, there will not be any fixes for any problems discovered when using V1 with django 1.11 or newer. LDAP support in buster and sid ======================= python-django-auth-ldap has changed dependencies, from python-ldap to python-pyldap. When upgrading python-django-auth-ldap, python-ldap will be removed and python-pyldap installed. Instances *must* restart the lava-server-gunicorn service after this change or users will experience HTTP 500 errors. lava-server manage check --deploy ========================= This release adds some checks to the django system checks. When admins update health checks or device-type templates, it is worth running the deployment check which does a quick sanity check on the instance. This can help to spot invalid health check submissions amongst other issues. Note that on development instances, some issues will always be reported - e.g. when Debug is set to True for django using /etc/lava-server/settings.conf Also, for developers, some of the improved security features in django may get in the way of signing in to http://localhost by enforcing requirements for https:// which is unusual on localhost. Check the django documentation - to enable localhost, you may need to disable at least these security defaults by adding to /etc/lava-server/settings.conf: "CSRF_COOKIE_SECURE": false, "SESSION_COOKIE_SECURE": false, As a result of these changes, installing 2017.7 can generate a prompt to update /etc/lava-server/settings.conf. In general, keep your existing configuration but look at the differences and migrate the changes across. Remember to restart lava-server, lava-master and lava-server-gunicorn services whenever /etc/lava-server/settings.conf is modified. Skipping 2017.8 ============ There won't be a full 2017.8 production release of lava-server or lava-dispatcher as a number of the team are away at the annual Debian conference, DebConf17 in Montreal. https://debconf17.debconf.org/ If you are going to be there, let us know. lava-server changes ============== a9390fc9a Remove debug hacking session from doc examples 13edd8f12 Skip certain V1 unit tests on django 1.11 and later 0577447fc Update query script for python3 and to output CSV 0b3b0beb7 device-type: addresses fix for armada-3720-espressobin ed1e402e5 device-type: addresses fix for armada-3720-db c382a5aa0 device-type: fix at91sam9m10g45ek 54a3b7a46 Apply change for distinct test case links to all results 1824a18fb Adding uefi-menu tftp method 71aac817e LAVA-363 - set the scheme in api/help from settings 2c840debc base-uboot.jinja2: don't load DTB if append_dtb is set 1b13a0e3e Fix extra keyword in api/help view 2882529bc Provide an example script for using queries 7f4cfe569 logs: continue to poll the log when canceling 25eaab176 XMLRPC: add offline_info argument to list endpoint 38f7570e3 XMLRPC: fix inconsistency between code and doc 8c134839d LAVA-972 test case should use ID in URL. 99725606c Remove references to OpenID and Crowd 95b959602 XMLRPC: add a function to download job definition ebf80c266 XMLRPC: fix crash when device.worker_host is None b1ec14c89 LAVA-861 separate base-fastboot.jinja2 template c4fd79d57 Remove unused file 7c066195a Remove deprecated django configuration 520ac8778 XMLRPC: remove unused function 3b8ab7d61 XMLRPC: do not raise an error when canceling twice cce1bdea6 XMLRPC: allow to force health check for a device 218408db6 Handle the new log level called "input" b49d9fb96 Adding uefi parameters for vexpress 4a52ba861 Allow overriding django system check values from settings as documented d2067a5f3 LAVA-987 fix log parsing bbc39489c results_app: API: add the 'run_query' function 8746edb6b results_app: API: add the 'make_custom_query' function 4fd429ac2 Add note on apache2 for V2-only workers 641e61dcb Explain the handling of measurements and Decimal bf8f53bf5 Count going offline as busy in overview table 06febd188 Expand on essential roles, results and sync 815d41b52 Tighten advice on prompt strings in docs 351f6fe41 Add documentation for login_commands in auto_login 5a7d4be95 Avoid warning for health checks of retired devices. 5713f5295 Document transfer_overlay deployment differences 8f04c6097 Fix typo in multinode api documentation. fe2093994 device-types: add meson-gxl-s905x-khadas-vim 853ee80e3 Move action fields templates to table files. 12de946eb Allow bare except and override pep8 dd4faacaf Enforce schema rules for metadata 352049204 Add missing check for null key|value in metadata b77731b31 device-types: rcar-gen3-common: handle bigger kernels d2acb94a5 LAVA-983 - Helper for DeviceStateTransition b4208116c LAVA-980 - Fix django deployment warnings 778fd4a8b Fix link to glossary term af85e5225 LAVA 931 - documentation. 212e98a3a LAVA-976 Use the Django system check framework e9c7ad477 Adding commands for booting whatever the precanned defaults are f78c58811 Added device type for Cavium ThunderX lava-dispatcher changes ================= d8dabe63 Switch the print to a test in unit test d1e22b67 Revert "Replace invalid characters in test_case_id." b3b4f5ac Update the example to a working test job with prompts. 504ec2bc LAVA-777 last log line should be lava.job result ad55acb9 Ensure overlay unpacking is clearly logged 2e7c9598 Add pipeline_ref update for test skipped by lavabot ed936cd3 Simplify the power control actions 78dcc1b0 Fix dispatcher configuration 89880ce1 LAVA-96 Check for errors if a tftp file exceeds 4G 6fd2702e LAVA-989 add extra logging for actions df86ed30 LAVA-990 add logging for VLANd + MultiNode support bbfb1817 Stop the unit test outputting noise c7839a98 Add a new log level for the DUT inputs 2502b8af Destroy LXC containers properly. 957734cf Add a unit test for Vland actions 68468abe Handle a bad url cleanly. 713d4e6e LAVA-645 - summarise slave logging 96a5effe Do not raise exceptions in validate f45106fb Catch errors in testdef from handler cleanly f34f41e2 LAVA-984 - LXC protocol timeout to report job result e31722b5 Allow ci-run with updated pep8 175d8e84 LAVA-931 - MultiNode lava-sync should report results fd944543 EOF in a test shell is an InfrastructureError 429691d9 Support power control for CMSIS devices 049c0227 lava-slave: add the right configuration for the host 1e92bf38 lxc: improve error message b2f25cf0 Add support for booting from UEFI shell 67a1e1c6 ramdisk: only pass to bootx if has u-boot header f4a4abd9 LAVA-967: check namespaces usage -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 5 months

1
0
0 0

2017.6 release

by Neil Williams

As with previous releases since 2016.12, the 2017.6 release is only available from the lava repositories: https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… However, Debian Stretch is due to be released very soon. 2017.6 may be the first release to be uploaded to Debian unstable since the stretch freeze started with 2016.12. There may be a short delay whilst unstable settles down after the usual round of extra uploads. Once packages migrate into the new testing (buster) and stretch-backports becomes available, packages will be uploaded for stretch-backports and jessie-backports. Device dictionaries moved to the filesystem =========================================== LAVA-757 - all existing device dictionaries will be migrated to the filesystem (/etc/lava-server/dispatcher-config/devices/) during the installation of the 2017.6 production release. The dependency on python-django-kvstore has been dropped. This means that there is no longer a need to update the device dictionary in the database, simply install the new file into the directory and the next test job to be assigned to that device will use the new configuration. Unlike device-types, there are no default device dictionaries provided by LAVA, so a fresh install will have no device dictionary files in /etc/lava-server/dispatcher-config/devices/ The documentation has been updated. There is now no need for lava-server manage device-dictionary, so this command has been dropped. Instead, admins who have permissions to run lava-server manage can simply copy a device-dictionary .jinja2 file into /etc/lava-server/dispatcher-config/devices During installation, this migration will take place: Applying lava_scheduler_app.0027_device_dict_onto_filesystem Database entries for all device dictionaries are replaced with files. If a device can support V2 but has no device dictionary, it will be marked as Skipped. This change means that lava-server no longer depends on the python-django-kvstore package. Feedback support with namespaces ================================ Devices may output messages to the serial console as a result of operations performed within an LXC and this output is now logged as part of the test job log file as 'feedback'. See the documentation for more information on feedback output. Architecture is optional for LXC ================================ Some LXC deployments support identifying the deployment architecture based on the host system. Some instances may use workers of different architectures and test jobs should be portable between instances. Therefore, the arch: field of the lava-lxc protocol has been made optional. Note that Fedora and Ubuntu deployments still need the architecture to be specified for x86 machines to distinguish between 32bit and 64bit x86 deployments. The architecture value can still be specified in test jobs but the test jobs will be more portable if omitted. Action names use hyphens, not underscore ======================================== If you are using action-specific timeouts in your test jobs or in customised device-type templates, note that the names of some actions have *changed*. This is to make the usage of hyphens and underscores consistent. Actions with names containing underscores are now invalid and all underscores in names have been replaced with hyphens. 'pdu_reboot' becomes 'pdu-reboot'. lava-server changes =================== daf1e2a lava-master: fix init script b6b5fe7 Reduce device configuration dynamic connections ebf4463 Fix a crash in the migration when the db is empty cdfeb12 Fix index out of range exception 33092ef Adding usb deployment method to vexpress device template 3257ee2 XML-RPC: remove unused imports 55dd014 XML-RPC: fix crash when called by anonymous users a3d4718 XML-RPC: allow every user to get templates/dict ddaf693 Add documentation of feedback support. eaae89a Set a default connection timeout for lava-test-shell f305e47 publisher: fix umask c25686f logrotate: lava-master should be owned by lavaserver 2da8e74 logs: except more exceptions 14f5f6c Update docs for changes in 2017.6 a486d12 Tidy up of debian support docs b05712d Preserve original comments when resubmitting 19bd2db XML-RPC: add an api to manage aliases 30973a0 Log an error when saving device configuration fails 763239d Use --arch for fedora LXC example 8f26102 XML-RPC: add a method to show tag details 19a69f7 fix location of example test job files 480a6f6 XML-RPC: add an api for handling jobs 356667d Update installation docs for Stretch release da74ae5 XML-RPC: add a decorator for is_superuser check 49f370c XML-RPC: add a clean api for device-types and aliases c6d503d commands/devices: extend devices command to add tags 1655dbf Move set commands outside body block 87b8c01 XML-RPC: add a clean api for workers and tags 351c16b XML-RPC: add a clean api for devices c4dec1f device-types: add bcm2837-rpi-3-b 9ca0e0d Adding combined vexpress device template 67f531f Make lava_scheduler_app.api a module d3021a3 xmlrpc: add a system.version function 3ef0157 linaro_django_xmlrpc: remove unused properties e994852 Use render shortcut instead of the loader e51ca21 linaro_django_xmlrpc: rework the application 4bbe17c LAVA-959 run lava-master as lavaserver user b283288 Fix migrations from Debian Jessie to Stretch and ensure smooth upgrade. bc25204 LAVA 329 - Add old job URL to metadata when resubmitting a job. 271723d LAVA-782 - Change error type for incorrect handling of custom queries. 368903e notify: Don't send verbose mails when the job is cancelled 71c514d Fix broken link in unit test. 1f48674 LAVA-955 - Remove 'arch' parameter from lxc protocol. 3b36ba0 Add auto_login test case in test_pipeline unit test e23bb17 Fix typo which causes XML-RPC error when saving device dict 9a82aee XML-RPC: allow to have sub_modules b0c7a5b Document reboot and apply-overlay elements for fastboot deploy method. 8a93049 LAVA-935 - Provide feedback output from connections d71f267 Fix typos in proxy configuration documentation. c7227b3 Fix rebase error in grub hikey test 9253497 Only the master should create log directories 1ee736b Allow every u-boot devices to use minimal boot e816c17 LAVA-832 allow to manage users from the command line eb83cbe Add grub interrupt prompt and interrupt character to constants ee79de2 LAVA-937 show job visibility and fix group visibility 73b6d86 Prevent key error in metadata handling. f89e840 Improve job output migration helper eb1ad02 device dict: do not show exclusive flag 3900276 Restore date-based subdirectories 1faa9f0 LAVA-855 - ART CI: Intel NUC device integration for LAVA v2 d195b01 Allow to revert (dummy) the migration 35f0802 Fix device dict export da7ebe2 device dict: fix web rendering 4a3daa9 Allow override of root for installed mustang. 12c0508 Expand zmq_client to handle publisher socket 34f3625 Trivial whitespace changes in glossary 6246a5d Improve data export docs 2af8e72 Allow for non-Const values in device dictionary 59742fe Extend examples to use the DNS support for NFS c514296 Fix change in import paths for V1 and V2 scheduling 765901e Add notes on advanced features for submission fd0905c Remove deprecated commands 0237b85 Use CommandError whenever possible a1cc08b Improve device-type templates 05123cc Missing conditional in device-type alias handling 17af14e Move commands to lava_server app 5bab54d Remove reference to unavailable boot method 'fastboot-boot-image'. 78e9048 Add mediatek dt for v2 49f27fa Allow in-place YAML include in V2 job submission. c2be83f Grub support for HiKey ae4d477 LAVA-757 Move device dictionaries to file system 52eb1e1 Remove unused model JobPipeline 9d0a5ae results: make the link unique 55bf39f Add a callback_url functionality to notification schema. 0324915 Add login_commands to the auto_login schema ce6d0f7 Don't reload template from disk on each access lava-dispatcher changes ======================= 38da5d8 LAVA-974: Fix init script and allow to set hostname 0fd687e Don't use os.path.realpath() on device names d84dab7 Modifying vemsd deploy action to use ResetDevice d21385e testdef: get 'revision' from the right variable 5ea16fe Rework test shell feedback timeouts 510c4e5 Declare the feedback timeout b994757 Use pyudev device links as well as device nodes e609c22 Fix namespace usage in power commands. 39e2e7e Read from feedback connection during long operations 7be926a Allow tar.gz firmware bundles 20e66c1 overlay: Raise JobError if root_partition is None 4e27fcd LAVA-935 - Provide feedback output from connections 831c71c Fix bug #3032 - Running an LXC/Hikey job without a test action ... 84e7378 Fix namespace issue in NFS deployment action f1f8d71 Add login_prompt valid test case 7dc48aa Issue lxc-stop before destroying a container. 3e1cb07 Changing udev checks to only look for FS label d08db39 Fix #3010: set logging timezone to UTC 1b664e5 LAVA-955 - Remove 'arch' parameter from lxc protocol. 8a4dbc7 Increase LXC finalize timeout to 90 seconds similar to multinode. 3823c49 Use one debug log call to log the same kind of message in lxc protocol. 5db313c Creating vemsd mount points ea03ef1 Fixing boot test issues using NFS deploy action d1f4a30 Fix rebase error with grub prompt constant b2aef8b device-types: add sun6i-a31-app4-evb1 8ceca07 Make grub use interrupt prompt and character from constants b495a35 device-types: add ttyAT0 to Atmel devices to support multiv7_defconfig 3a89027 device-types: fix at91sam9x25ek to use bootz 9e00895 Set download max_retries in the constructor 959480c Add minimal boot action 31fd021 User command: do not clean if it was not executed 26572a2 Fix keys and label that were outdated 67dd60e Remove unused functions 0c100ff Remove boot-result common data 342039a Fix RetryAction inside a RetryAction 9ce430f Fix check on LXC for default usage. 0576754 Fix typo: Use pixel job create function for testing pixel device. e7a4422 Fix typo: remove visibility added twice. 9e67b6d LAVA-855 - ART CI: Intel NUC device integration for LAVA v2 dd5d874 Ensure all deployments can add LXC devices except QEMU e72d490 Fix pipeline refs for change in monitor retry cb57d2b LAVA-942 UEFI Menu and command lists e57013b Start adb daemon before attempting adb commands. ac389ad lava-slave: remove tmp dir when END_OK is received 632a0ff Fix device configuration 0cffce4 Fix validation of boot auto_login parameters and add a unit test 7e40701 Allow unix line separators for some UEFI devices 095798a Avoid calling cleanup in validate f98dade Allow updating all pipeline_refs at a time. 31b62dd LAVA-928 - HiKey issue when switching between AOSP and OE 51f40ef Make soft reboot optional LAVA-846 4d43d02 HiKey Grub EFI support 874f889 Avoid logging the same message twice. 3be27df lxc protocol: simplify the tests 75126ff Do not set self.errors when Action.run fail ce32bfb Do not use self.errors in Action.run() a5d9eba Fix detection of missing ssh_host value in validate 234eef9 Fix calling of protocols after LXC change 477a36b tests: make ShellCommand.logger a DummyLogger 62867f4 Use images.validation.linaro.org files for unit tests 1868936 Silence logging in more unit tests 609c783 Fix download action name b170197 More tweaks to silence messages from the unit tests 29efa26 Adjust for pep8 checks in jessie 0733f89 Make sure the test_character_delay is used for all commands in test shell 78d2509 Account for empty environment string. ea09b19 Take namespace into account when counting test stages 734688c Declare namespace of the test suite in results 60561c2 Parser: remove unused context 49a6b92 Drop noisy info log message. 8ed8467 lava-test-monitor: use TestMonitorRetry action 4fd5d4a Drop unused imports and unused variables 678b216 Rework the removable action to allow sd cards 3b42122 Fix bug #2975 after qemu-nfs introduction 7462ea6 Expand LXC support to add devices from all sybsystems b853865 Fix missing check for u-boot commands parameters 8293693 Replace invalid characters in test_case_id. a94e0f2 Fix preseed/late_command appending 9647d55 Run login_commands if provided in auto_login db03483 Fix Action names (use - instead of _) 9e836e6 Add a Command Action bbd4ba2 Make the parser stricter about the block names 65b3a56 Allow to pass integers to run_command b41326b Raise JobError instead of NotImplementedError 073f3b3 update gitignore 00f5d33 log: limit the length of lines send other zmq 3f99cc3 Add missing super call to WaitUSBDeviceAction 5b2b074 LAVA-889 Fix handling of multiple test blocks 5436658 Fix indent typo 3bfdcdb Add support for use_xip 8df17dd Add support for append_dtb 3c110ca Remove unused imports 4847bf1 Put device rebooting info message at the correct place. 6b85382 Fix call to lava-lxc protocol for pre-os-command 176a4d4 Add new utility function infrastructure_error_multi_paths bdb4d12 Fix test_lxc_api unit test dc008eb Export the full version string. e9df320 LAVA-920 - Workaround ptable issue in firmware 7b4096c Allow LXC protocol to call pre-os-command b33d04b Fix OE image support for HiKey. d714a9b Add missing test shell helpers daebfad Extend secondary connection fix to support primary 9518e74 Add missing import e97801d Add optional deployment complete message list to secondary media deployments fae0ced Move unnecessary constants into base jinja template. 18f745b device-types: add kirkwood-db-88f6282.conf 815e518 device-types: add at91sam9m10g45ek.conf 6ca0731 device-types: add at91rm9200ek.conf 003b53d device-types: add at91-sama5d4_xplained.conf 9b2371c device-types: add armada-3720-db.conf 5a76733 device-types: add armada-xp-gp.conf 8b05093 device-types: add kirkwood-openblocks_a7.conf f5801d3 device-types: add alpine-v2-evp.conf 19e2027 device-types: add sama5d34ek.conf 49da704 device-types: add armada-385-db-ap.conf ed7aa6b device-types: add armada-370-db.conf 9a10432 device-types: add at91-sama5d2_xplained.conf ddb34ce device-types: add alpine-db.conf f7642f5 device-types: add armada-375-db.conf 0d70c7d device-types: add at91sam9x35ek.conf a642d45 device-types: add armada-xp-db.conf 2a5ff9e device-types: add armada-7040-db.conf 1a43999 device-types: add orion5x-rd88f5182-nas.conf ad855a6 device-types: add armada-xp-linksys-mamba.conf 58308ca device-types: add armada-388-gp.conf 6a2898b device-types: add armada-370-rd.conf cd1b4f4 device-types: add armada-398-db.conf 51925fb device-types: add sun8i-a83t-allwinner-h8homlet-v2.conf 4976d06 device-types: add sun8i-a33-sinlinx-sina33.conf 681480d device-types: add sama5d35ek.conf db74d60 device-types: add sun5i-r8-chip.conf 40736b9 device-types: add sama5d36ek.conf 433be0a device-types: add imx6q-nitrogen6x.conf ed4c82e device-types: add at91sam9x25ek.conf ce7a488 device-types: add at91sam9261ek.conf f87cdb4 device-types: add armada-xp-openblocks-ax3-4.conf 634442c device-types: add armada-388-clearfog.conf -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 6 months

1
0
0 0

LAVA, Django, Stretch and python3

by Neil Williams

This is a long mail but there is a lot to cover... Stretch ====== Debian Stretch is expected to be released on June 17 2017: https://lists.debian.org/debian-devel-announce/2017/05/msg00002.html As has been mentioned on this list before - admins MUST check the apt sources for all machines *NOW* to ensure that all your Debian apt sources are pointing at 'jessie' NOT 'stable' if you are to be able to manage the upgrade yourself. What you want in /etc/apt/sources.list and /etc/apt/sources.list.d/* is: deb http://mirror.bytemark.co.uk/debian jessie main deb http://mirror.bytemark.co.uk/debian jessie-backports main (The mirror you choose is up to you.) If you find any apt sources referencing 'stable' or 'testing' CHANGE THESE NOW or those systems could automatically upgrade to Stretch or Buster respectively on 17th June. As with all such releases, production machines are recommended to allow some time for final testing before doing the upgrade. The Cambridge LAVA lab is expected to upgrade from Jessie to Stretch during August 2017. The LAVA software team have been doing testing of this upgrade for some time, without issues. However, real machines with extra packages and local admin configuration cannot be tested in this way, so each lab should take some time to plan when and how to do the upgrade. (Hint: have a backup and check that it can be restored correctly before starting.) https://validation.linaro.org/static/docs/v2/admin-backups.html There has been a fix for lava-server in stretch to smooth the upgrade for those users who have not installed lava-server from jessie-backports. Those already running 2017.* are not affected by this change and can continue using jessie-backports and the LAVA repositories as before. Stretch will release with lava-server 2016.12-2 and python-django 1:1.10.7-2 Once Stretch is released, Debian testing will be called 'buster', Debian stable will be 'stretch' and Debian oldstable will be 'jessie'. At a later date, jessie-backports will be opened again and a new stretch-backports will become available. New LAVA production releases will be uploaded to unstable to migrate into testing (buster) and then be backported to stretch-backports and jessie-backports. The LAVA repositories will be updated as well. Support for jessie-backports will continue after stretch-backports support is added but cannot continue indefinitely. At some point, all instances will need to have moved to Stretch to continue getting updates of LAVA. (Prior to the process of releasing Debian Buster, the jessie packages will be dropped from the archive, including jessie-backports.) Changes in LAVA repositories ======================= 2017.6 will be released in the same way as previous releases in 2017 - to the LAVA repository for use on jessie using jessie-backports. At a later point, 2017.6 will be uploaded to Debian unstable for migration into testing (buster). Only developers should install that version as Debian unstable can be problematic in the first few weeks after a Debian stable release. There are changes in how the lava packages are built on jessie versus on stretch, so the lava repositories will be split. I'll be updating the documentation in 2017.6 to cover the changes required to help admins use the new suites. There will be a new jessie-backports suite and a new stretch-backports suite. The existing 'sid' or unstable suite will have no uploads after 2017.6. This will be covered again in the 2017.6 release email and again with 2017.7. For now, no changes are required. Once the new suites are announced, admins who choose to upgrade to Stretch would use the stretch-backports suite from the LAVA repository. deb https://images.validation.linaro.org/production-repo stretch-backports main Those who will upgrade later, would use jessie-backports from the LAVA repository. deb https://images.validation.linaro.org/production-repo jessie-backports main The purpose of these suites is to allow admins to choose to upgrade to the latest production release for jessie or stretch without waiting for the migration of the packages into Debian testing and the subsequent uploads to backports. The packages are otherwise the same. Django ====== Longer term, there have been a lot of discussions within Debian about managing django releases. The plan is that only django LTS releases will be available in Debian unstable, testing and therefore stable after the Stretch release. Non-LTS releases will only be available in Debian experimental. This means that lava-server will migrate from one django LTS to the next django LTS with future Debian releases. The LAVA software team will manage how deprecation cycles within django affect lava-server. This does have the issue that the version of django in Debian stable will be towards the end of the upstream support cycle but it is better than having a non LTS django release in Debian stable. Synchronising releases across multiple upstream teams is not feasible. The next django LTS release after 1.8 is 1.11 which is expected to be in the next Debian stable release - buster. https://www.djangoproject.com/download/ Python3 ======== The last django LTS release to support python2.7 will be 1.11 LTS which retains upstream security and data loss bug fix support until at least April 2020. The LAVA software team have already been testing lava-dispatcher V2 with python3. Once the V1 code is removed from lava-server, testing will proceed for lava-server too. We fully expect to be able to support lava-dispatcher V2 and lava-server V2 on python2.7 and python3 for a limited time. There will never be python3 support in lava-dispatcher V1 or lava-server V1. Python2.7 support for lava-dispatcher V2 and lava-server V2 will be removed at some point prior to the release of the next django LTS after 1.11 which is 2.2 LTS, due in April 2019. Once django 2.0 (non-LTS) arrives in Debian experimental, the LAVA software team will begin testing lava-dispatcher V2 and lava-server V2 with python3. For some period in 2018, both python2 and python3 will be supportable but only when building local developer packages. LAVA production releases will stay on python2.7 during testing with python3. Once we are happy that python3 support is ready, there will be an email to this list and the following production release will provide only python3 packages. This change will be transparent for most instances, the upgrade will simply pull in a lot of python3 packages and the python2 packages will be left unused. Reviews and contributions to lava-dispatcher V2 already run the unit tests with python3 and once the V1 code is removed, this will start to happen with the lava-server code as well. LAVA can move to python3 at any time once the V1 code is removed - all the necessary support is already available in Jessie. LAVA will have to move to python3 before django 2.2LTS is the only version of django released in Debian stable. Details of how this will be managed are yet to be defined. We expect to continue parallel support of python2.7 and python3 for a limited time. It is likely that support for jessie-backports would need to cease before support for both python2.7 and python3 is included, just to manage to matrix of builds and tests. LAVA can support python3 using django 1.11LTS but must drop support for python2.7 to support django 2.2LTS. Summary ======= 17th June 2017 - expected date of Debian Stretch release August 2017 - migration of Cambridge lab to Debian Stretch instead of Jessie November/December 2017 - removal of V1 codebase Q1 2018 - testing with django 1.11 LTS Q2 2018 - testing with python3, likely end of support for jessie-backports Q3/4 2018 - add python3 support Q1 2019 - drop python2.7 support. April 2019 - release of django 2.2LTS -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 6 months

1
0
0 0

2017.5 production release

by Neil Williams

jessie-backports ============ New package update required from jessie-backports. sudo apt -t jessie-backports install python-voluptuous This package is used to check the submission schema of V2 test jobs. It is already installed for V2 support but 2017.5 requires an updated version which already exists in jessie-backports. For a smooth upgrade ensure this package is upgraded as above before installing 2017.5. As with previous releases since 2016.12, the 2017.5 release is only available from the lava repositories: https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… New device-type templates =================== 2017.5 includes a lot of new device-type templates and a number of V1 device-type configurations. These have been contributed to LAVA, as part of the support for KernelCI, by community members. Many thanks to all those who have contributed device-type support for this release. Action names and effects on timeouts =========================== 2017.5 makes the individual names of actions in the source code consistent across all classes and deployments. Where previously power_off was available as an action name which could be specified in timeouts and protocols, that action name now uses a hyphen instead of the underscore, power-off. The same change has been made for all classes across the V2 lava-dispatcher codebase. Future changes will only be accepted using a hyphen to separate words, not underscores or other characters. This change affects V2 jinja2 templates as well as V2 test jobs (if those jobs override the action timeout or call an action using protocols. The full list of changes is below: lava-server ======== 8ec2d6736 Fix jinja2 templates for default string handling eb39ec0fd Extend bbb template check for ssh_host support. 55a251b87 Fix documentation example test job and remove unused 7769be718 Update docs for change in submit behaviour. c10bf6197 Add more index items and detail on namespaces b9686e5b1 Mark V1 XML-RPC functions as deprecated. a0ac9894b Prevent health check warning when disabled 78f796eee more silencing of unit test logging 064bfabb9 Add a template for frdm-kw41z and delete a duplicate for k64f a08df108d Silence logging in more unit tests 97dbd95b1 Extend power-off timeout for b2260 0e39966d4 device-types: add Hardkernel meson8b-odroidc1 board b6151b003 Add support for aliases in device-type management 15589b8d1 Fix some typos in development documentation. f476a351c device-types: base-uboot: use run bootcmd 6911b8502 Expand notes on reviews cbbdcfcf4 Expand notes on code analysis around reviews 970e52230 Drop confirmation page on job submit for V2 jobs. 4a5050691 Device commands are allowed to be lists cc60805db Adjust hikey template to allow target_mac and ip support 9d4878c40 Avoid forcing the date path immediately 68497b557 Remove the character_delay block override for d03 5aeed4aef Tags: fix HTML syntax errors a0ae8a62d Update doc for adding a pipeline worker. 2f6287e71 Add Raspberry pi devices 69a8f40c4 Add collection of Exynos 4 and 5 devices e31953519 Add more Tegra124 based devices b843bfaec Add more r-car generation 3 devices 24d21cdb0 Add a note on https repositories and apt-transport-https package. c3b852798 Expand notes on portability c57c9f111 Fix doc to explain unprivileged containers and DUT interaction. 2018f705c Show the requested device tags in the job log page 96999537e Fix Action names (use - instead of _) f0a272b4b Extend recent job support to requested device type 9040de9ca Add XML-RPC call to obtain job level metadata a04086dd7 api: add get_recent_jobs_for_device 8bbd298f7 Set the documented flash_cmds_order for hikey c9b9453d1 Migrate many U-boot devices to v2 configuration 613d17fb5 Tweak the developer workflow to skip devices/ c59cc6c31 Add a unit test for some of the new UBoot support. e43e244b0 Remove unused imports and unused variables. e2b3784f4 Fix pep8 error af3cc6d42 Schema: Allow boolean variables in parameters 4d5861607 Allow is_valid check to operate correctly. 294bb62bd ensure device_type is checked b8ce5ba49 Add a note on developer branches 11725bc3e doc: fix a small gap about test suites 6280b94d7 Add "sd" for removable media 21a5ecf6e Add "command" action to schema and device template 8db43278a Add schema validation for test/monitor/name in job definition. 0a5b1eaa3 tweak gitignore e3f003f63 templates: remove duplicated blocks f1331ef2e Exclude retired devices from Device Health table 790b39b9d master: use yaml.CLoader that is way faster 749a1081c Add notes on load balancing different bootloaders 26b822d67 Add note on how pyudev replaces / with _ cc9e4ea2c Allow to override U-boot bootcmd command 9a007665c Fix 500 when output.yaml is invalid b3c2d162b Make it easier to spot incomplete test jobs fe6af63a7 Improve job and device schema validation 4a568e261 Fix directory and file permissions f33661ae4 env: fix comments about default values fd6fe12b6 Fix scheduling when putting a device into looping 99b35ba79 Export the full lava-server version da87efaa6 base-uboot.jinja2: add support for append_dtb and use_xip babaef51d lava_scheduler_app: api: Add pipeline information to get_device_status abe787872 Add a note on installing lava-dev 2b6ff1dca MASTER_CERT of lava-master should use secret key d9e7e2c08 add recipients in notifications.yaml 44eb1b75b Move job outputs to sub-dirs based on submit-time 839b3ff19 Create directories with 0o755 by default 9459aae36 lava-master: call job.output_dir to get the path 8c9af4897 Fix description for devices and workers 541478930 Fix health-check tests by testing None and '' 68b33d572 Use job.output_dir whenever possible d93df5f4f Add a management command to remove old jobs bfd57121d Move unnecessary constants into base jinja template. lava-dispatcher =========== 3be27df4 lxc protocol: simplify the tests a5d9ebad Fix detection of missing ssh_host value in validate 234eef91 Fix calling of protocols after LXC change 477a36b7 tests: make ShellCommand.logger a DummyLogger 62867f49 Use images.validation.linaro.org files for unit tests 1868936c Silence logging in more unit tests 609c783c Fix download action name b1701978 More tweaks to silence messages from the unit tests 29efa263 Adjust for pep8 checks in jessie 0733f89f Make sure the test_character_delay is used for all commands in test shell 78d2509f Account for empty environment string. ea09b19a Take namespace into account when counting test stages 734688c5 Declare namespace of the test suite in results 49a6b92c Drop noisy info log message. 4fd5d4af Drop unused imports and unused variables 678b216c Rework the removable action to allow sd cards 3b42122a Fix bug #2975 after qemu-nfs introduction 7462ea62 Expand LXC support to add devices from all sybsystems b853865f Fix missing check for u-boot commands parameters 8293693c Replace invalid characters in test_case_id. a94e0f24 Fix preseed/late_command appending db03483a Fix Action names (use - instead of _) 9e836e64 Add a Command Action bbd4ba25 Make the parser stricter about the block names 65b3a563 Allow to pass integers to run_command b41326bc Raise JobError instead of NotImplementedError 073f3b34 update gitignore 00f5d33f log: limit the length of lines send other zmq 5b2b0746 LAVA-889 Fix handling of multiple test blocks 3bfdcdb6 Add support for use_xip 8df17dd7 Add support for append_dtb 3c110cae Remove unused imports 176a4d49 Add new utility function infrastructure_error_multi_paths dc008eb5 Export the full version string. daebfad1 Extend secondary connection fix to support primary fae0ced5 Move unnecessary constants into base jinja template. 18f745bc device-types: add kirkwood-db-88f6282.conf 815e518d device-types: add at91sam9m10g45ek.conf 6ca0731f device-types: add at91rm9200ek.conf 003b53dc device-types: add at91-sama5d4_xplained.conf 9b2371cf device-types: add armada-3720-db.conf 5a767331 device-types: add armada-xp-gp.conf 8b050938 device-types: add kirkwood-openblocks_a7.conf f5801d38 device-types: add alpine-v2-evp.conf 19e20276 device-types: add sama5d34ek.conf 49da704f device-types: add armada-385-db-ap.conf ed7aa6b1 device-types: add armada-370-db.conf 9a104327 device-types: add at91-sama5d2_xplained.conf ddb34cea device-types: add alpine-db.conf f7642f58 device-types: add armada-375-db.conf 0d70c7dd device-types: add at91sam9x35ek.conf a642d451 device-types: add armada-xp-db.conf 2a5ff9e4 device-types: add armada-7040-db.conf 1a439995 device-types: add orion5x-rd88f5182-nas.conf ad855a62 device-types: add armada-xp-linksys-mamba.conf 58308ca1 device-types: add armada-388-gp.conf 6a2898bf device-types: add armada-370-rd.conf cd1b4f47 device-types: add armada-398-db.conf 51925fb1 device-types: add sun8i-a83t-allwinner-h8homlet-v2.conf 4976d068 device-types: add sun8i-a33-sinlinx-sina33.conf 681480d5 device-types: add sama5d35ek.conf db74d60e device-types: add sun5i-r8-chip.conf 40736b95 device-types: add sama5d36ek.conf 433be0a6 device-types: add imx6q-nitrogen6x.conf ed4c82e6 device-types: add at91sam9x25ek.conf ce7a488e device-types: add at91sam9261ek.conf f87cdb49 device-types: add armada-xp-openblocks-ax3-4.conf 634442ca device-types: add armada-388-clearfog.conf -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 7 months

1
0
0 0

Changing action names in V2

by Neil Williams

For 2017.5, we are planning on changing all of the action names (self.name) to use hyphens to separate words instead of the current mix of hyphens and underscores. This is to make future support more consistent. All V2 templates currently in upstream will pick up these changes as part of the review process. Any V2 templates which are not upstream and which reference timeouts for specific actions may need adaptation, however, note that 2017.5 also moves a lot of the timeout settings to base.jinja2 anyway. If you have test jobs which reference specific actions, these will need changes too. We're implementing a change in the dispatcher codebase which verifies that underscore is not used in the self.name of any class inheriting from Action, so all code reviews will also need updating. https://review.linaro.org/#/c/19027/ -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 7 months

1
0
0 0

2017.4.post1 hotfix release (lava-dispatcher only)

by Neil Williams

This interim release includes a small set of fixes aimed at the LKFT instance and HiKey support using LXC. If your instance uses the pre_os_command support for fastboot with LXC (not just pre_power_command), then note that this support now needs to be called from within the LXC protocol. See https://lkft.validation.linaro.org/scheduler/job/885/definition#defline77 for an example. Changes: lava-dispatcher (2017.4.post1-1) unstable; urgency=medium . * Production hotfix release for LKFT support. 3f99cc3 Add missing super call to WaitUSBDeviceAction 5436658 Fix indent typo 4847bf1 Put device rebooting info message at the correct place. 6b85382 Fix call to lava-lxc protocol for pre-os-command bdb4d12 Fix test_lxc_api unit test e9df320 LAVA-920 - Workaround ptable issue in firmware 7b4096c Allow LXC protocol to call pre-os-command b33d04b Fix OE image support for HiKey. d714a9b Add missing test shell helpers 9518e74 Add missing import -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 7 months

1
0
0 0

2017.4 production release

by Neil Williams

V2 healthcheck changes ====================== https://lists.linaro.org/pipermail/lava-announce/2017-March/000027.html This support has now been delivered in 2017.4 using the lava-server manage migrate-health-checks command. If used, the --clean option will remove the old definition from the database for the device-type. https://staging.validation.linaro.org/static/docs/v2/healthchecks.html#inde… There is also a new migration status page to help admins track which devices still need to be migrated to V2. Branding message ================ The branding support now includes a message which can be added to the home page for the instance. For example, a short message to describe the main purpose of that instance. Reminder about the LAVA repositories ==================================== Until Debian Stretch has been released and the new testing suite (called buster) is open for new uploads, new production releases of LAVA will only be available from the images.validation.linaro.org/production-repo repository. See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… See also: https://lists.linaro.org/pipermail/lava-announce/2016-December/000023.html Changes ====== A longer list of changes this time as 2017.3 was skipped to allow for the Linaro Connect BUD17 conference. lava-server =========== 7e36443dd Restore UEFI boot method for HiKeys. 6679d548e Fix dragoboard410c boot sequence. 5082adc80 Move anchoring fix for navbar to base template. 187c8520a Allow to override U-boot tftp command 0d28de652 Handle OSError as well as IOError in metadata store be04868ff Update migration status page for disabled healthchecks 5f689e930 Improve job page ed4dee272 Expand the docs on arbitrary device_info elements 35a7378e0 Remove references to deprecated health_check_job 5e5e6890c Add a link to the migration page 84f88a4fc Allow to override U-boot usb command 043bfeea4 LAVA-916 - Restore adb connectivity with hikey in V2 4211de051 Don't force {BOOTX} macro in U-boot template 571d71313 Don't force dhcp in U-boot template b0184585c Add a section on which file is what in /etc/ 9701ad6a0 Adding timeouts to Juno template b2b8d8851 Add content on developer workflow e36889a81 Revert change for debug filter button c06ed4a22 Fix bug 2927 - typo in OPTIONS for $MASTER_CERT cfa0c02c5 Fix navbar problem with in-page anchors. bd554a2c4 Add docs on boot commands. 624c79ee1 Fix bug #2925 - LDAP Configuration demo mismatch distro.py 1ddde696b Another parent timeout needed for qemu c3f356c0b Allow boot_qemu_image action timeout to be overriden b24fe09af Use standard paths to keep the unit tests running 7433bb4e4 Add note on installing developer builds a0e7756c7 Add a migration status page ce59699ad Tweak the device type docs adf670e7d Update copyright year for the docs to 2017 b2428ad8c Small updates to the docs for test development 6610ce36f Allow to import token from another instance d57e5217b Cleanups for advanced installation docs 0119fc57f Fix job submission page 0c4e68769 Dedupe the index page and add more entries ec0aa40ad Improve device page 912c174b1 add notes on the new lava-tool features a662f0ed6 Add a standard NFS test job for panda e80ed1e13 Update docs on transfer_overlay b0ae30b56 Update migration for job_status_trigger for backwards compatibility. f28af7fbe Fix mustang doc examples and add uefi-menu a755b4b12 LAVA-768 allow to disable hc for a device type 7a05b7941 Add support for a branding message b18e8d293 log: add a link to the top and to the doc 0fd8c985e fix typo in rst header 75d411f14 Do not send a spurious POST submit request d7f8d6322 Update docs for change in behaviour of log file UI acd068173 LAVA-112: display status and role for sub jobs bf144047f Check V2 devices have valid config before reservation. 0a2da3f92 Skip retired and obsolete device types c106b17c0 LAVA-913 - Documentation for lxc persistence 14c733d30 Add another index entry for the example first job definition b8629e0e3 Expand simple admin to include administration outline 43bb9da87 V2 jinja2 template for armada-8040-db ca05a93d0 Add a command to migrate health checks to the fs ad55286d6 doc: fix label 0b17a0496 Add qemu NFS docs a8bf524f4 Update of doc examples 7bc903e70 LAVA-736: move health checks to the file system 1de7ea5b3 LAVA-912 - Document verbose for lxc protocol 643c23efc Remove deprecated (and unused) functions 91cff154b Add notes on when to add more workers 69df43a50 LAVA-904: Improve job page layout 3e3075f2f Reorganise the scheduler app test files 639600952 LAVA-910 allow iteration if vland tag check fails 1ebe9d217 Improve readability of testcase results b7792c56e fix typo in index syntax 143b5f742 Allow boot_interface override b85bc07a6 Update default interrupt_* in hi6220-hikey device type jinja2 template a561326bc Add a test case for secondary connections d6a441c16 Add Acer Chromebook 13 CB5-311 7ded576cb Add Rada Rock 2 Square 65e2b1e22 Add Renesas R-Car Starter Kit Pro 61edf0180 Add Samsung Chromebook 2 068005a9a Add i.mx6q sabre lite board a84a509e4 Allow u-boot-interrupt timeouts to be overridden b7a23dbfd Fix test job timeouts and fix doc c94238613 Ensure extra_kernel_args is used from job context 77693f080 Clean up error handling in is_deprecated_json method. 8b5fa8130 Add a 'full docs' link to the front page 9ab49a8e9 Allow unit tests to run without kvm module e47c5c005 Change job_status_trigger field type to a non-deprecated one. 6ec411a55 Tweak the authentication configuration docs 7ee493bb1 Tweak the authentication configuration docs baa9f5ebb Update references for a permanent location. da79c4be7 Add available architectures for qemu 'arch' validation. 65e4c35ec Prevent intermittent unit test failure. 7f9b8760d LAVA-215 QEMU NFS support (server side) 3f2f414e2 LAVA-619 document changes for bootz to zimage 8bbbd425f Ensure missing font is available for packaging. 0f2d8a4ff X-axis attribute in charts now use NamedTestAttribute table. 93cc20aa6 Fix export custom queries feature. a8d6cab43 Display results sorted naturally by key in job logs. e7aa614c5 Update the default length of the job result table. e2c07c1ac Fix doc example YAML syntax to read from file ce59d6b94 Fix sphinx build errors in manpage 63d5f6fa5 Add a check on all templates in source tree d9eb3e1f3 Ensure character delays are supported by overdrive 75efcb00b Allow repeating test cases 93d620a53 Explain how connection-namespace parameter works. 9041b0dc0 Begin a section on common YAML syntax errors. 6521cc0f8 LAVA-894 - Document lava lxc device add / wait command ee3e5964f Fix documentation - punctuations, links, bullets. 988d89a34 Add server side support for mustang EFI Grub. 525d9ed47 Fix code blocks in Boot Action documentation. 106f7f4fc LAVA-890 - Allow DUT to stay in Android OS 887175d82 Remove unused images 9f9e8d304 Fix unknown test case handling in tables bff6ea8ff LAVA-888 - Download button in Device Dictionary page 3314966de LAVA-887 - Introduce fastboot sequence in device type templates 1a1be1ff2 LAVA-867 - Allow fastboot options in device dictionary d876d1463 Remove legacy nexus deploy document from V2 docs. 2750804af Updating Juno jinja2 template to accommodate vemsd deploy action 2b6d3d277 Deprecate "pipeline-worker" in favor of "workers" 587bc8931 Add a "details" command to "device-type" and "workers" 96761e533 Add a command to manage devices 3b2f27ee0 Encode maintenance state machine in unit tests 730b57b81 Do not put into maintenance retired devices c0adfd2c2 Add a command to go into maintenance 49abd8bf7 Fix state transition when OFFLINING a device f337d7d1c Go from OFFLINING to RUNNING when a job exists 39b50ec15 Extend docs on simplistic testing 8e56315eb Workers: allow to set properties 944caa4fd pylint improvements c42629220 Use continue in a loop, not return 61b6886f9 add notes on debugging multinode as admin 2ed1f35c7 Device tables: add more fields to select_related d95ad2d5b job logs: Highlight all failing results ace40b01a Generate less database queries 7fcbd7c68 Allow to use last version of Django Debug Toolbar 61069528d LAVA-862 - Integrate Google Pixel into LAVA V2 e7925e7a1 Update docs on publishing and attaching to tests da99f08f0 Fix a typo in nexus5x device template used for unit test. ba5a106db More stdout changes to clean up the test output c84741460 Skip panda_lxc template check if lxc not installed 6a0a73d36 Fix a typo in documentation. 79ea7cedd Fix looping bug in template 466f1506a LAVA-841 - Document device_info attribute d420b9d5f LAVA-856 - Integrate nexus5x into LAVA V2 de33008ec Autologin doc update. lava-dispatcher =============== 7aac7ae5 Fix problem with self.host on secondary connections 530ca366 Ban / in test case names, replace with underscore. 31ee0012 If allow_fail is not set, ensure the command fails e53a280f Remove old device_type config files which are unused. 80fb3997 LAVA-916 - Restore adb connectivity with hikey in V2 c57a102b Add debug output to the LXC device operations. 7deea0fb Add missing calls to super 341ac593 Writing connection info to namespace data 3ee2af87 Upgrade the boot commands output to info a1a87daa Fix bug in checking return value of action run_command output. 604323b0 Add a unit test for primary ssh connections 561f78a6 LAVA-914 - Action run_command should return log on allow_fail a8fb0d9d device-types: add Armada 8040 DB board 98fc1d4d LAVA-911 transfer overlay to the rootfs e85af0a2 Report versions of critical software on the worker bdc97bd5 LAVA-901 - Allow lxc debug output if requested in job 93aab7ab Add systemd packages as default for debian and ubuntu templates. d01f01a3 Allow params for install git repos support eb821dc8 testdefs: rename the extra results keys f262b21c testdefs: fix TOCTOU issues with test definitions 07affb55 Print warnings in validate and not in populate 3de0aac9 Make empty revision message more friendly 685eea49 Do not populate append when using tmpfs GuestFS 22a7ba93 Add missing imports a1baf85e Add missing JobError import. 9e8c3c2f Improve validate and job summary b1100080 Catch error in secondary connections 7428a9ae Improve git error logs 949b3fcd Rework the timeout settings in job and device conf 6933ac63 download: improve error message 304c39f7 LAVA-852 - LXC fastboot support for persistent devices in V2 fc28aafe Add an error type to the job result 522946b8 Make '/sys/class/misc/kvm' a constant. f9a1b381 tests: use the DummyLogger whenever needed ee11c796 tests: use permanent urls 538ff35f Context architecture validation for qemu. bd88adab Check for enabled kvm module on workers. 28c66090 Fix bug #2898 - Job doesn't exit on fastboot-deploy failure 2879c019 Handle qcow2 conversion errors 7daf65d1 Action: use LAVAError base exception class 603fd049 shell: remove unused exception handling da97077d Add missing import and remove uneeded one beccc4da LAVA-215 QEMU NFS support (dispatcher) c2e3ee34 Use DummyLogger for grub and add .target() ed3519d2 Get uboot header length from utils.constants 7df2904d If no kernel type given in deploy, use the boot type until deprecated 82237860 Ensure the deployment data shell is exported. 108ef4a3 Add the commit-id to the lava testdef results 1e3dd3ce run: improve error handling 41b08257 lava-dispatch: use the new LAVAError base class 171e8b65 device-types: add Allwinner A23 EVB board 5898747b device-types: add OrangePi PC board 4ffb4c1e Simplify lava-test-shell by removing stdout.log a5222296 validate: improve error handling c2df0b20 Remove the message from cleanup() 78c334c2 Simplify test/shell.py check_patterns signals. 68f1c822 test-runner: remove redundant functions and files 3fe2105e Fix bug #2888 - adb root command from job breaks connection to device 22157cd2 Add a base class for LAVA exceptions 44af91c3 Support using UEFI Menu to load Grub using PXE b667f2fd Use LAVA exceptions 1690ed95 Pylint updates for deploy actions e538b5d8 LAVA-890 - Allow DUT to stay in Android OS bbcf76fd Raise a TestError when the test installer fails 0d45d8fa test-runner: use the current shell 27593024 Blacklist brltty on workers e2593966 Allow reusing of serial connection from another namespace. cfe9821b LAVA-867 - Allow fastboot options in device dictionary d5094f28 Fix unknown usage command in lava-test-reference 381eca8d LAVA-881 - Sequence list for fastboot pipeline e84c00b1 action: be consitent when printing result duration 82d7f82f Print the result of a test run only when avaiable 1d09ece0 Catch python3 errors to clean up output 833d7207 python3: only show requests logs above warnings 5fb1e8a0 rpcinfo: also capture the output 496c7181 Sanitise default mkimage_arch and kernel handling a6838ecb LAVA-862 - Integrate Google Pixel into LAVA V2 d2f968e3 Follow up to stabilize dragonboard 410c. 8ab18e05 rpcinfo check needs to specifically check NFS service 61777bb0 Refactor wait usb device action to stabilize dragonboard 410c. e7649e64 Remove references to device_path in device configuration. 3fd11d81 LAVA-856 - Integrate nexus5x into LAVA V2 d970298d Adding new action for monitoring USB mass storage devices via udev 94876fd7 pyocd: allow multiple images to be flashed 26528b0c Detect bootloader resets. 09650ab1 Adding a Versatile Express Firmware deployment action 605fe529 Detect errors during auto login. 00306b79 Job: log the job result as a logger.results -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 8 months

1
0
0 0

Moving V2 health checks out of the database

by Neil Williams

2017.4 will deliver support for migrating V2 health checks out of the database and into /etc/lava-server/dispatcher-config/health-checks/ on the master. This has several benefits for admins. * Health checks from 2017.4 onwards will be based on the device configuration, not the device type. This applies to V1 devices and V2 devices. * Multiple instances can all have the same V2 health checks controlled in VCS using configuration management tools. * V2 devices of the same device-type can have different V2 health checks according to the Jinja2 template specified in the extends field of the device dictionary. Submissions to such devices would need to be managed through device tags. * Devices which are V1 only (is_pipeline is False in the django admin interface) will continue to use V1 health checks from the database, based on the device-type. * Devices which support V2 (whether these devices are exclusive to V2 or not) will only use V2 health checks from the filesystem, based on the device dictionary of each device. This does mean that devices which are currently both V1 and V2 will need to have a V2 health check from 2017.4 onwards or the device will no longer run a health check at all. Admins may choose to submit and monitor regular V1 testjobs until all test jobs for those devices are V2 but those test jobs will not be able to take the device offline automatically. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 8 months

1
0
0 0

Issues with python-lockfile

by Neil Williams

There is a bug in python-lockfile available from jessie-backports: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=858835 This affects fresh installs of LAVA when LAVA is installed directly from jessie-backports without first being installed in jessie. The underlying problem is with apt which pulls in the updated version of packages which have a backport available even if the requested package does not explicitly ask for the backport of that package. The problem shows up with lava-coordinator first - the package fails to install cleanly because the daemon cannot be started correctly. To fix this problem, simply instruct apt to install python-lockfile from jessie, not from jessie-backports: apt install python-lockfile=1:0.8-2 lava-coordinator will then install correctly. This change will persist over later upgrades as apt does not consider jessie-backports as upgrades of jessie - except in the case where the package is not yet installed. Other daemons also now need to be restarted: $ sudo service lava-server restart $ sudo service lava-master restart $ sudo service lava-slave restart Future builds for the staging-repo and production-repo will force only version 1:0.8-2 of python-lockfile as these builds are done on and for jessie. Systems already running Stretch or unstable do not seem to be affected and can safely use python-lockfile at version 1:0.12.2-2 - this is a particular problem inside jessie-backports. However, this does mean that until this bug is fixed, users running Stretch or unstable should build their own developer packages instead of using the LAVA repositories on images.validation.linaro.org. We currently do not have capacity to build separate packages for jessie and stretch. (LAVA does not recommend using Stretch or unstable for production systems until Stretch has been released and stretch-backports becomes available.) -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 8 months

1
0
0 0

2017.2 production release

by Neil Williams

Reminder about the LAVA repositories ==================================== Until Debian Stretch has been released and the new testing suite (called buster) is open for new uploads, new production releases of LAVA will only be available from the images.validation.linaro.org/production-repo repository. See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… See also: https://lists.linaro.org/pipermail/lava-announce/2016-December/000023.html 2017.2 release ============== More updates to the jinja2 templates in this release and some new templates. diffstat output: lava_scheduler_app/tests/device-types/apq8016-sbc-uboot.jinja2 | 3 lava_scheduler_app/tests/device-types/arduino101.jinja2 | 5 lava_scheduler_app/tests/device-types/b2260.jinja2 | 8 - lava_scheduler_app/tests/device-types/base-uboot.jinja2 | 26 ++-- lava_scheduler_app/tests/device-types/base.jinja2 | 11 +- lava_scheduler_app/tests/device-types/d02.jinja2 | 4 lava_scheduler_app/tests/device-types/d03.jinja2 | 1 lava_scheduler_app/tests/device-types/frdm-k64f.jinja2 | 30 +++++ lava_scheduler_app/tests/device-types/hi6220-hikey.jinja2 | 7 - lava_scheduler_app/tests/device-types/highbank.jinja2 | 2 lava_scheduler_app/tests/device-types/juno-uboot.jinja2 | 1 lava_scheduler_app/tests/device-types/mustang-uefi.jinja2 | 3 lava_scheduler_app/tests/device-types/nexus10.jinja2 | 1 lava_scheduler_app/tests/device-types/nexus4.jinja2 | 1 lava_scheduler_app/tests/device-types/nexus9.jinja2 | 1 lava_scheduler_app/tests/device-types/nrf52-nitrogen.jinja2 | 9 + lava_scheduler_app/tests/device-types/nucleo-l476rg.jinja2 | 23 ++++ lava_scheduler_app/tests/device-types/nxp-k64f.jinja2 | 1 lava_scheduler_app/tests/device-types/nxp-ls2088.jinja2 | 12 ++ lava_scheduler_app/tests/device-types/overdrive.jinja2 | 1 lava_scheduler_app/tests/device-types/r8a7791-porter.jinja2 | 11 ++ lava_scheduler_app/tests/device-types/stm32-carbon.jinja2 | 11 +- lava_scheduler_app/tests/device-types/x15.jinja2 | 55 ++++++++++ lava_scheduler_app/tests/device-types/x86.jinja2 | 1 24 files changed, 178 insertions(+), 50 deletions(-) Changes in the 2017.2 release are described below, including the short git commit hash, include: Use the git commit hash to go directly to that commit using a URL like https://git.linaro.org/lava/lava-server.git/commit/?id=266b2e633 Commit messages including a reference to LAVA-NNN relate to projects.linaro.org append the reference to https://projects.linaro.org/browse/ e.g. https://projects.linaro.org/browse/LAVA-854 lava-server - https://git.linaro.org/lava/lava-server.git/ ----------- 9df8b354b Remove whitespace from case names 9354cd585 Uboot parameters fixes 297f27900 Fix documentation for auto_login 64b549e9a Fix group visibility and hidden device types b211434de Document the use of a list for bug 2870 eb8a7bec3 LAVA-854 - Device type template for x15 ad81f33fb Add device template for renesas r8a7791-porter board 8c4d44ae4 Add device template for nucleo-l476rg 112ec6487 Remove send_char, and fix some broken character_delay settings 8f615cd6b Update reviewers handling 0fe482ffb b2260: remove duplicate ip_args from extra_kernel_args a72c86732 LAVA-847 protect metadata store IOError 53a3c778e If a text_offset is used, quote it to ensure it is a string. 51351e18c Add documentation of the dispatcher-config support 7a1bfb649 Tidy up the Device-type templates for IoT devices 0b5de4dcb Allow lava_scheduler_app to unittest logs to stdout bc807def4 Fix sizelimit warning handling for V2 13b691615 Summarise job timeouts with example and notes b5c2ea1a7 Improve notes on portability. 65ae7d644 Fix internal server error due to unpacking TestJobUser. 83e8fa075 Migration doc improvements 9eb145050 Add a unit test for the ethaddr support in b2260 91eb07a36 Creating and restoring backups for V2 479a6295f Add support for setting ethaddr 52b5cf06d Add triage documentation for power failures 526a6e298 LAVA-840 - Remove documentation for USB_DEVICE_WAIT 3415d5319 Allow .py files in examples/sources to be packaged be33389bf Allow notifications for jobs in Running status. 4d6afd607 Fix blacklisted bug in notifications. 58b48a8f0 Catch all errors during resubmission. 40b010238 Tag admin clean support must return name 14b15f40b LAVA-734 compatibility docs dd18c26e1 scheduler: do not cascade deletion for some objects 5415eafa7 Do not create unused TestJobUser 2f3cd8389 Bug 2796. Add submitter username to job_details API. 08cd6c01f Revert "master: set the master identity" be28c8c7b LAVA-814 - Explore pyudev for usb device wait 9150c9973 LAVA-832: Rework device-types management command 70e81493e LAVA-762 link level in timing to pipeline definition c276df83f LAVA-728 extend ZMQ example to submit and wait f6a0456a1 LAVA-731 document remote worker issues 41c67ae3b Add device template for NXP-ls2088 ee8964c7b Remove spurious call to save() 2a79a622d Ensure base_ip_args can be overridden affa68b23 Add a command to add and list workers 10e6ea970 tokens: allow format as CSV b0b5a1d30 Remove unused config value LAVA_CONFIG_VERSION 69138ac89 Add a command to manage tokens e09beb1a4 Remove unused custom commands 905a0f9be Fix multinode job definition page bb0fdc7a8 job logs: Highlight failing results e1a125aa8 master: set the master identity b7cb28e49 Fix device schema and test for arduino template d411416bf LAVA-707 update result case and metadata views 15fc5c62f LAVA-140 - update running job calculation 9fcf92cfd LAVA-515 restore line numbers to YAML definitions b7266712b LAVA-824 skip unused device types in running table de6a141aa Expand docs on namespace c54a5e262 events: fix documentation lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/ --------------- ba779908 testdef: set "set +x" to avoid duplicating SIGNALS ae2ab424 Enforce a download order to produce predictable pipelines 55a39b92 Remove unnecessary boot action in fastboot x15 81b178fc Add a unit test for db410c pipeline generation. aa96dd75 LAVA-839 - Integrate X15 into LAVA V2 14acd2e2 Fixup u-boot prompts sanity 829fa380 Support CMSIS-DAP devices that do not reset after being flashed d09a7a74 Simplify send_char support 031ba8e8 Ensure dispatcher.yaml is available for admins aa6142dd Detect pdu controlled device using availability of power command. c54891b9 Properly isolate stdout and stderr in unit tests 2d8b3110 Ensure lava-test-reference is available for release 555b6be0 Skip one test if lxc-info is not installed fc42bd89 Fix support for persistent NFS 57c96b4b Remove duplicate prompt handling. b0d39204 LAVA-844 allow string and dict for install: git-repos 76998b75 LAVA-830 Use the configured TFTP directory 1f01283f Partially revert a1658c by using a DEALER socket 9045292b Allow boot commands in the job definition af34dfcb Extend the timeout unit tests to boot actions 9135c5aa Add support for CMSIS-DAP IoT devices and cleanup PyOCD 96dcc7d7 Only match udev events when device is added a1658c7e slave: use a ROUTER socket to connect to master 35937826 Fix prospector warnings e4e2fb97 etc/lava-slave: fix documentation of WARN log level b45cb005 Remove unsed imports 93a282bf Job: log known error status when leaving ee404039 lava-dispatch: allow to run without zmq logging 26daca49 lava-dispatch: work around a variable clash c62e1ee9 LAVA-814 - Explore pyudev for usb device wait 989cd97b LAVA-827 ensure the case name is a valid URL 570c1b1e Do not reset the timeout before each connection -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 10 months

1
0
0 0

2017.1 production release

by Neil Williams

Reminder about the LAVA repositories =========================== Until Debian Stretch has been released and the new testing suite (called buster) is open for new uploads, new production releases of LAVA will only be available from the images.validation.linaro.org/production-repo repository. See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… See also: https://lists.linaro.org/pipermail/lava-announce/2016-December/000023.html 2017.1 release =========== This release includes substantial updates to the jinja2 templates. If you have made local changes to these files, you will be prompted during installation. /etc/lava-server/dispatcher-config/device-types/apq8016-sbc-uboot.jinja2 /etc/lava-server/dispatcher-config/device-types/arndale.jinja2 /etc/lava-server/dispatcher-config/device-types/b2120h410.jinja2 /etc/lava-server/dispatcher-config/device-types/base-uboot.jinja2 /etc/lava-server/dispatcher-config/device-types/base.jinja2 /etc/lava-server/dispatcher-config/device-types/beaglebone-black.jinja2 /etc/lava-server/dispatcher-config/device-types/cubietruck.jinja2 /etc/lava-server/dispatcher-config/device-types/highbank.jinja2 /etc/lava-server/dispatcher-config/device-types/juno-uboot.jinja2 /etc/lava-server/dispatcher-config/device-types/mustang.jinja2 /etc/lava-server/dispatcher-config/device-types/panda.jinja2 /etc/lava-server/dispatcher-config/device-types/qemu.jinja2 Changes in the 2017.1 release are described below, including the short git commit hash, include: Use the git commit hash to go directly to that commit using a URL like https://git.linaro.org/lava/lava-server.git/commit/?id=266b2e633 lava-server - https://git.linaro.org/lava/lava-server.git/ -------------- 266b2e633 LAVA-795 - Expose lava-publisher event socket settings. f4b76e48e Fix ordereddict failure 9d3ae86cc Use dpkg-query which is available on all systems 080c3184f Update the device-type context help page 46a055a3b Fixup some jinja2 templates cecefbfef Timing: improve the description 5bef1d877 LAVA-821 - add support for lava-test-reference 3b26fb234 LAVA-820 record lava-server version in job metadata ff71a42a6 Append to existing metadata store results 0c9abf643 Fix RST whitespace typo. 05b96af3c Merge timing page into the job page edc4431f1 Restore access to chart Action buttons 69709881e Improve the timing page and port to the new log format a00c02aa8 LAVA-535 Make devicetags case-insensitive 36617b955 LAVA-743 - handling complex test operations 50084f1b5 Omit summary button for V2 definitions. 5c8bf4643 Upgrade the dispatcher v2 parser interface. 22409d543 master: send the dispatcher config to the slave lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/ ---------------------- 64ae9131 Report the bootloader load addresses used 2085ffb8 Skip parsing of kernel messages for ssh connections 00fa2d24 Improve allow_fail fault handling dce680f3 Simplify the connection session handling ea18594a Use dpkg-query which is available on all systems 2cc64b70 Reinstate git version_tag in version.py 5c05bdae Improve logging 50236943 LAVA-821 - lava-test-reference support a118e943 Add missing wait definition in base class 2dc955f6 Allow timeout during Linux Kernel message parsing 0404a419 LAVA-819 - record lava-dispatcher version in logs 542effdf Raise an Infra error when download fail in run() 29a232c1 Fix Download retries crashes 12e5789f monitor: make test a bit faster a90a5184 Fix computation of action duration ffd1d176 Fix retry logic (partial revert of 7bbd706d) 36ca3c21 LAVA-818 - errors should set the job as Incomplete f5ac0328 Skip generation of invalid test_case_id 9e5e56c6 Re-enable support for nose tests bb89772f Merge and re-use BootloaderCommandOverlay and BootloaderCommandAction f3864c2f job: fix cleanup when validation failed 8dd450c5 run_actions: clean error messages and exceptions ec44db64 Properly compute the timeout f3058a12 atexit is not needed anymore for cleanup 7bbd706d Improve error handling and reporting a5e0e678 Fix issue when downloading two files with the same name 0d3773ea Call finalize directly in job.cleanup af608eb2 Remove unused setter, properties and __call__ f946a21d validate: do not extend action.errors 4504aeb9 slave: use the dispatcher configuration 925019c8 Use the dispatcher config to get the dispatcher ip -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 11 months

1
0
0 0

2016.12 production release

by Neil Williams

Enable the LAVA repository ==================== 2016.12 will be the last chance to update of lava-server and lava-dispatcher into Debian Stretch before the release as Debian 9.0. This means that users of jessie-backports should now enable the LAVA production repository - see https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://lists.linaro.org/pipermail/lava-announce/2016-November/000019.html Until Debian makes the 9.0 release, jessie-backports cannot receive updates of the LAVA releases in 2017, so all updates will only be available from the LAVA repositories. Shortly after the release, a new stretch-backports suite will become available and jessie-backports will be opened again. In accordance with the Debian freeze policy, it will take 10 days for the new uploads to migrate into stretch, not the previous 5, so an upload to jessie-backports will be made at that time. Note: this may be an issue for armhf or arm64 workers - if you are using such workers, let us know so that we can prioritise the changes required to build lava-dispatcher for those architectures. Updates from jessie-backports ====================== A few updates are required from jessie-backports with 2016.12: lxc and python-pyudev The lxc update ensures that jessie and stretch containers can be correctly created on jessie. The pyudev support enables lava-dispatcher to monitor udev activity for certain operations. These updates can be applied before installing 2016.12, if desired: $ sudo apt install -t jessie-backports lxc python-pyudev lava-tool ====== A new version of lava-tool is also available, including a new way of handling the authentication tokens and the ability to use shortcuts (like "local" instead of "http://localhost/RPC2"). https://tracker.debian.org/pkg/lava-tool https://tracker.debian.org/news/825902 The new version of lava-tool is available via the LAVA staging and production repositories and the package available in jessie-backports will be updated in due course. Unfortunately, it proved not to be possible to export the existing authentication secrets into the new backend or to support both backends simultaneously. All users will need to re-add all their authentication tokens to use lava-tool after this upgrade. The change has several benefits, including the ability to list and remove authentication configurations using the new backend and the removal of a whole host of problems related to using the old backend in environments where lava-tool was not running in a full terminal. In addition, this new backend allows lava-tool to support shortcuts which prevent problems with repetitively entering long and complex URLs. On installing the new upgrade, users will see the NEWS message: https://github.com/Linaro/pkg-lava-tool/blob/master/debian/NEWS lava-tool has changed the backend for token storage. This means that existing authentications will not be usable to this new version. This change has been made to fix persistent problems with the python-keyring support, including: * DBus errors when used over SSH * Interaction with gnome-keyring causing authentication failure * Inability to list or remove authentications added to the keyring * Inability to work with other keyring solutions. The new backend is able to list and remove authentications. The new support also removes the need for a default password on the user keyring, so lava-tool will no longer pause waiting for password entry. 2016.12 release ============ Changes in the 2016.12 release are described below, including the short git commit hash, include: Use the git commit hash to go directly to that commit using a URL like https://git.linaro.org/lava/lava-server.git/commit/?id=b19b9648a lava-server - https://git.linaro.org/lava/lava-server.git/ ----------- 541ba7b0a Enable ssh secondary connections on juno-uboot 2d6bb2fcc Handle errors which cause invalid description data b19b9648a Fix template_mismatch check to use extends d7f71eae3 Update ipxe x86 nfs to use base_ip_args fa6b8b3ab Add a note on release-notes on lava-announce d007d1819 Fix a crash when a device does not have a device dict d46d6e4fd Protect against invalid parameter submissions 431fd2fc0 Allow pipeline jobs to be used with XML-RPC job_output f3bf855ba Add Copy to clipboard functionality to MultiNode 19d814e0c Fix some failing tests due to new namespace changes 86950e985 LAVA-745 - expose device_path for all types 083d67d54 Add device-type template for arduino101 fd99c576e Fix UI submission error. 708a38a23 Add SSH support for overdrive e838de3ba Update Linaro lab links to actual site d8e127b74 Allow flash_cmds_order to be set 92eefe4aa Adding extra boot commands for secondary media on Juno 74310353c LAVA-792 - add shorthash to changelog 8f860a1e9 Keep result blocks on a single line with MultiNode 3b1d6654e LAVA-499 - Device type template for dragonboard-410c 5c46ea4ab Provide a default fastboot flash commands order. a61ab2501 Download results XMLRPC support. 347fb3fd3 Allow to forward the messages to additional sockets 4d0753c3a Allow validate_pipeline devices to filter by device type name. a859e5ede publisher: improve logging and fix logfile owner f0002efb1 Fix sphinx indentation error 9f9bd71b9 Correctly expose TestSet data in results c6f9b1df8 Handle decimal conversion error in test case 1769d8b6b Expand hints on new device-types eec1ca1d0 master: expect a protocol version with HELLO messages cd8d109f7 master: add the job id to the log messages a0ac2d66a Add absolute_url to job_details API. 951ab9f68 master: device configuration is empty for secondary media c0575503d master: improve error reporting when reading env file f551daa49 Add notes on unattended upgrades 1d2e25d1d Allow override of nfsroot options, not arguments edaad5419 Update docs to reflect reality of from: support 09503a482 Manually revert d31a8462: errors are already saved 3516b856d add some notes on packages needed for LXC usage b5ee513bb Rename 'usb-showup' to 'usb-device-wait' 406832582 Drop misleading hostname from device metadata 6a6ee6f4d Handle change to git.linaro.org frontend d85f99e1b Fix crash when description.yaml is empty 37a94aafc Fix prospector warnings 080499d18 Remove references to wsgi 267d22302 Paste job definition to clipboard as text and not as HTML. ef09a9698 add a link to the device tag glossary entry 08bd011a4 Update aliases migration for review change 7010775b6 Move metadata extra content to files 95ed9a3ba Expand notes on debugging test jobs and test cases 6d772bb0d Update for pycodestyle pep8 failures dd58a1182 LAVA-798 - lookup device-types by alias 104feb593 Enable LXC for existing device-types. 376a80021 LAVA-797 - Document changing USB_SHOW_UP_TIMEOUT in job 3e8dbfeca all_devices() api call to do one big query instead of many small ones. d27f7a4d5 Fix for Debian reproducible builds. ac8bb8e2a Fix bug #2596 - No result collected for multinode jobs 7b3adad0a All jobs will have a definition, so show it by default. 1e6992f6b Set boot_character_delay in x86 template 5cc0a4746 Extend x86 unit test to check character_delay support ad4f8780f Combine interfaces block for juno and vland bb2ecf716 Allow changing the timeout for u-boot-interrupt 19de59978 Add stm32-carbon jinja2 lava-dispatcher - https://git.linaro.org/lava/lava-dispatcher.git/ --------------- 19fd79a9 Update preseed support for namespace changes ef352abf Update zephyr sample jobs to use LAVA functional test urls 1301258a LAVA-498 - Support dragonboard-410c with lxc 1e0b60d2 Ensure pyudev version is >= 0.21 7d6a8134 Update the DFU boot code for namespace changes 161e8b47 Ensure installer tests share the final connection 567ddf0d remove duplicate call e49bd971 Fix namespace error in flash_cmds_order retrieval. cb04549d Set connection for secondary media deployments dcd75e7b Use correct suffix for secondary media deployments 2a7b386f Declare the original ramdisk as compressed fd73e937 Support DFU devices 6aa7280c LAVA-504 - Dynamic fastboot partition support in pipeline 9228901a LAVA-809 - Show HiKey kernel boot messages 774c7557 Extend namespace fixes for secondary connections dc5bec22 Action: Always call super() in validate() 9b983cdf GitRepoAction: do not call validate twice bd112aab Make namespaces common to all dynamic data operations 2e83e2ca Prevent invalid measurements being reported 1768c181 Fix testlevels for multi-test LXC jobs 0a49da8e Give namespaces to all actions and support simpler code flow 2423a123 slave: send the protocol version with HELLO message 983d3775 Ensure fixupdict is checked in test_case_result 08f79918 slave: writing an empty string to a file is valid c6f382ef Move cleanup_actions from Action to Job bc56ca24 Remove unused Action, Pipeline and Job members 1256f31e Run.sh should use set -x a726be8f Allow install-steps to use the git-repo directly b0ca553a Create a useful exception message 077aca26 slave: don't crash when lava-dispatch can't start 543ed4af Honor USB device wait timeout and add device before returning connection 0e07089f Fix problem with duplicate test shells failing f14747c1 Use dnf instead of yum on Fedora version greater than or equal to 22. efdaabc1 More fix for usage of LXC with a device. 2d188baa Limit repetitive kernel message log entries c3455a11 Allow newline line separator to be overriden per bootloader 01a482f6 Fix for Debian reproducible builds. 62c07465 Indicate that character delay is in milliseconds c1f79680 Remove hardcoded character delay in ipxe e43552b3 Fixing secondary media deployment e885c881 device-types: add Renesas r8a7795 Salvator-X board 1374289b device-types: add Amlogic meson-gxbb-odroidc2 -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

8 years, 11 months

1
0
0 0

Migration to V2

by Neil Williams

Everyone using LAVA should be aware of V1 and V2 and that V1 is going to be turned off. The plan for how to do that has been discussed within the team and within Linaro. The most recent presentation on the migration can be seen here: http://connect.linaro.org/resource/las16/las16-503/ (Note: the video will auto-start, be aware.) The slides are available on slideshare: http://www.slideshare.net/linaroorg/las16503-lava-v2-migration The dates for each step in the migration are not finalised but will occur during 2017, this is advance notice. The highlights of the migration are: 0: During 2017, regular announcements will be made on this list before each major change. In addition, the regular release announcements will include details of the major changes in that release. 1: The first major change will be that a new release of lava-server and lava-dispatcher will *disable* V1 submissions entirely. Any queued V1 test jobs will become invalid upon this upgrade, 2: A later release will remove V1 documentation and start removing the V1 source code. 3: A release will be made which *forcibly deletes V1 test jobs, bundles, filters and image reports*. This is required to complete the removal of the V1 source code. 4: The final step of the migration is a release containing no V1 source code at all. Only V2 support will remain. This is essential as the V1 source code is currently blocking some useful additions to V2 as well as plans for future development. The data cannot remain accessible without the V1 source code. There is not and can not be any support for converting V1 data to V2. Please take this chance to read up on the V2 documentation, including how Results, Queries and Charts replace the deprecated V1 support from filters and image reports. Work has started upstream on making sure the migration runs smoothly, especially when deleting large numbers of V1 test cases. Remember: Once the first major step is released, upgrades will stop running any V1 test jobs. Subsequent releases will include changes which force the deletion of your V1 test data. If you want to preserve your V1 data, consider setting up a read-only reference instance which is then pinned at a particular release of LAVA to prevent deletion of the V1 data. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years

1
0
0 0

Problem with QEMU 2.7 from jessie-backports

by Neil Williams

There is a known bug with QEMU 2.7 which is affecting installations of LAVA using QEMU from jessie-backports. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=840853 The problem manifests in the apply-overlay-guest action because QEMU is unable to process the GuestFS operations due to a missing file which is in the seabios 1.9 package but not in seabios 1.7 - so V2 QEMU test jobs hang at this point. If you are using QEMU from jessie (version 1:2.1+dfsg-12+deb8u6) then there is no problem and nothing needs to be done. If you are using QEMU from jessie-backports (version 1:2.7+dfsg-3~bpo8+2 currently) then there are two simple remedies. 0: You can find the updated seabios package in the LAVA production repository. See https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… If you have the production repository already available, you can simply apt update and apt upgrade to install the new seabios package. 1: If you are not using aarch64 QEMU tests or the QEMU UEFI support for LuvOS, then the version of qemu in jessie will work just fine. To downgrade, specify the version string to apt: $ sudo apt install qemu-system-x86=1:2.1+dfsg-12+deb8u6 qemu-system-common=1:2.1+dfsg-12+deb8u6 It is also worth downgrading other qemu packages at the same time. Check your current install with: $ dpkg -l | grep qemu The relevant bug has been re-opened and a backport of seabios has been uploaded. An updated backport of qemu has also been requested which would require the installation of the backported seabios 1.9. So if this bug is not troubling you immediately, you can simply wait for seabios 1.9 to be available in jessie-backports and install it. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years

1
0
0 0

2016.11 production release

by Neil Williams

2016.11 includes a move away from libapache2-mod-uwsgi and libapache2-mod-wsgi to gunicorn support. This has been done to make it possible for lava-server and lava-dispatcher to migrate into Stretch ahead of the Debian 9 Stretch release. (see https://lists.linaro.org/pipermail/lava-announce/2016-November/000019.html). This should allow us to upload 2016.11 to jessie-backports in due course. See: https://lists.linaro.org/pipermail/lava-announce/2016-October/000017.html for details of the changes for gunicorn support. In particular, if you are using apache, before installing this upgrade, ensure that these commands have been run on the master: $ sudo a2enmod proxy $ sudo a2enmod proxy_http $ sudo service apache2 restart These commands are safe to run before upgrading to 2016.11 but will need to be run before 2016.11 will actually run. This release includes all changes in master since the 2016.9 release. The release has been uploaded to Debian unstable, it should appear on the official mirrors shortly. It has also been uploaded to the production repo. We are anticipating that there will be an interim release ahead of 2016.12 for additional LXC support, more details will follow via this list. lava-server (2016.11-1) unstable; urgency=medium . * New production release aimed at Stretch. Release details available at: https://projects.linaro.org/browse/LAVA-789?filter=12078 * Drop dependency on libapache2-mod-uwsgi and libapache2-mod-wsgi. Replaced by dependency on gunicorn. * Require sphinx 1.4 or later for correct theme support. * Downgrade linaro-image-tools and fuse to Recommends * Update compat version. * Highlights of upstream changes: Note about removing adb and fastboot packages in LXC admin doc. LAVA-789 - Document LXC support in V2 LAVA-788 - link to sub_jobs_list from results Expand user notification documentation. Move jinja template tests to unittest Adding target interface entries to juno-uboot.jinja2 Add docs on notifications Update home page documentation links Improve documentation on metadata and job_name Fix up glossary page Tweak docs handling standard test jobs Example jobs need auto_login support Fix missing create_device_database ref Port the documentation change for essential roles Tweak the doc build options Add notes on multiple hacking sessions Update debugging docs for tmp file changes LAVA-65 document recording measurements LAVA-780 stop referring to wheezy images Avoid crash in master on early failure Remove beautify.js and fix yaml URL submission. Update screenshot for cancel button fix Fix visibility of the Cancel and Admin buttons Update docs to reinstate web UI submission Add a helper to add devices from cmd line Remove old link about LAVA packaging which contains stale information. LAVA-749 - Add actions column to results pages. Ensure lava-server-gunicorn restarts cleanly results: do not crash if the description is empty Job submission UI for v2. Add copy to clipboard button for job definition. Turn all action durations into result measurements Updating juno-uboot.jinja2 Add a log file for gunicorn logs. Fix the LAVA logo Ensure the V2 layout is included on jessie. bootstrap: do not use the minified CSS Results: fix HTML syntax Use sensible default for interrupt string in HiKey jinja. LAVA-749 - Improve query pages Fix unit test to have unique test definition names Add pre_* commands to base.jinja2 that will be applied conditionally. Remove unused dependendies on lava_dispatcher Fix a variable redefinition in list comprehension Update contents and organise toctree Allow changing the timeouts for auto-login-action Add documentation on QEMU standard kernel tests Remove old mention of JSON master: set the failure_comment Add a template for highbank using base-uboot Remove hardcoded values from base Fix profile page error due to wrong device health history url reference. metadata: fix a crash if the description is invalid Cover lava-coordinator in changes for V1 workers LAVA-522 Link in existing docs on replacing vmgroups Extend the qemu options docs Fix index links and debugging doc links LAVA-735 describe anonymous access to JIRA qemu.jinja2: use qemu-system-i386 for i386 Extend docs for disabling V1 on a master. Fix HTTP 500 on query views when a group is used. LAVA-765 V1 MultiNode sub_id correction Fix jinja2 syntax issues in base-uboot Fix sub_id assignment in V2 Improved debugging docs Add content for growing your lab. Improve first device and first job pages Update apache docs for gunicorn Update hacking sessions and lava-network docs Enable vland for overdrive Fix uWSGI configuration file Add notes on removing V1 from dispatchers Add a helper to add device types from cmd line DeviceType admin: fix has_health_check computation Replace ip=dhcp with {{ base_ip_args }} timing: add the corresponding timeout Fix typo in the title of the result download link Add ip=dhcp for juno with a base default Fix page to show health history of my devices and not my device type. Fix error on 'Devices Health History' page in django 1.10 Fix user profile page error in django 1.10 Create a base uboot jinja template Use Gunicorn instead of uWSGI wsgi: use the Django public API index: add the missing title Mark support for LAVA in Debian testing. Expand the context schema for extra_options Add the new LAVA logo to the front page of the V2 docs Warn if a pipeline device does not have a template master: don't generate the description Add nxp-k64f jinja2 Add nrf52-nitrogen jinja2 Add drafts of new logos with SVG. Extend base timeouts and convert to minutes Handle inline definitions using only install steps job details: removed unused template variable master: don't print too many logs in debug Fix build errors and simplify index pages Add support for overdrive device type Fix bad lookups into the glossary Switch ASCII art to the existing SVG Update dispatcher actions Improve timing warning message Document the principles of a CI Loop master: remove the need for the ERROR message Expand job_details doc string with available keys Replace 1.10 deprecated get_field_by_name with get_field method. Add support for extra QEMU options LAVA-747 - Report parameters as result metadata Switch from using png to svg for the architecture diagram Add a page to display the pipeline timings Expand glossary and add sections on parameters Add D03 device type to pipeline Add an introduction to results in LAVA. Add docs on the standard test jobs Simplify and clarify the "first install" docs LAVA-584 - expand for sub-types of device-types LAVA-706 - codebase structure Port submission priority support to V2 Adopt a similar contents layout to the django docs Device dictionary output Extend qemu jinja template for cortex-m3 Pipeline log: use a better icon for download Update bootstrap from 3.1.1 to 3.3.7 Fix relative url for suite results Add nbdroot as optional parameter to the schema Update qemu/kvm templates to allow overriding of guestfs interface LAVA-93 mark some roles as essential lava-dispatcher (2016.11-1) unstable; urgency=medium . * New production release aimed at Stretch. Release details available at: https://projects.linaro.org/browse/LAVA-789?filter=12078 * Update compat * Highlights of upstream changes: Ensure namespace is available to the test shell. cpio output is not an error Downgrade message match failure to a warning Fix up usage of LXC with a device Only attempt lxc-destroy if the container exists based on exitstatus. Drop ShellLogger __del__ command Improve accepts for monitored QEMU Strategy needs to check methods, not device_type Support parallel (pbzip2) compressed bzip2 files Allow vendor U-Boot builds using Ctrl interrupts Improve cpio extract command line and error message slave: do not send None as error code slave: send a SIGINT to lava-dispatch to cancel Making target IP and MAC addresses available in lava test shell TestShell: allow multiple test shells Fix kernel message detection for panic testdef: remove unused common data Fix the logic regarding the pipeline building shell: make testdef_index common data a list device/target: allow multiple boot control chars Add 943907AEVAL1F to pyocd deploy supported devices Check if container exists before destroying. Do not generate results.yaml Fix uefi_menu pipeline for fastboot. slave: move tmp files to /var/ Add device to lxc only when it is available. device-types: add Amlogic meson8b-odroidc1 board LAVA-755 - use UUID for guest, not label. slave: send an error code when err is not empty Fix git-repos reference in inline test definition. Allow a different bootloader interrupt character Make pipeline logging python3 compatible slave: fix a crash under python3 device-types: add Amlogic meson-gxbb-p200 board device-types: add TI da850-lcdk bootloader: add ramdisk_raw option Detect invalid characters in testdefinition name Fix timeout inheritance from job definition blocks Set the right permis when creating the ramdisk dir ci-run: exclude the .eggs directory tests: Close the file descriptors Fix root directory of V2 apache config LAVA-760 - Trim LXC log output Require a summary and description to validate Additional changes for removable media usage Fix extra_options to always be a list of strings Do not set kernel type for all 'kernel' YAML keys Action: re-raised the exception without modification LAVA-748 ensure test definition names are unique Change sample jobs urls to images.v.l.o Allow testdefinitions without run steps Generate the pipeline description on the slave slave: fix the END message parameters Use the QEMU extra options where available slave: resend the END message if the ack was lost slave: send the error file along with the END message Move tmp files to /var/lib/lava/dispatcher/job_id/ Only extract the nfsrootfs once. Use the uboot interrupt prompt from templates Improve LXC HiKey support. Don't end test shell at invalid testcase name SignalMatch: simplify and fix exception message Shell: improve report formatting Shell: factorize handling of measurements RetryAction: log errors Shell: factorize signal handling Move override and log_results functions into Action Initial support for PyOCD Use \r\n for newlines Allow guest drive interface to be specified allow-modify functionality added -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 1 month

1
0
0 0

About LAVA and Debian 9 Stretch

by Neil Williams

The 2016.11 release is currently in final preparation and is targeted at Debian 9 Stretch. We've tried to get the documentation and support up to a point where users of Stretch have a usable platform before later upgrading via stretch-backports. I expect to get 2016.11 into jessie-backports about 5 days after that release. A few reminders on how the Debian release process (which starts in January 2017) will affect things, most of which are the same as during the jessie release process. More details will be given on this list as the Debian release process continues. 0: New lava-server and lava-dispatcher uploads to Debian won't migrate into Stretch once the freeze starts. This means that these will not be available in jessie-backports either as the presence of the same version of the package in Stretch is required for an upload to jessie-backports. I expect to get 2016.11 and 2016.12 into jessie-backports. 1: Once Stretch is released, stretch-backports will be opened and uploads to jessie-backports can resume. Debian makes the release when the release is ready, so there are only estimates of when Stretch will actually be released. The key information is that the freeze to prepare the release starts in January 2017. 2: I expect to keep jessie-backports updated at least for the first few releases. This list will carry more information about when there may have to be changes in this area. Essentially it depends on movement within Debian after the Stretch release. 3: During this period, uploads will remain available on images.validation.linaro.org built for jessie - lava-dispatcher will only be available for amd64 from those repositories. If you are using armhf or arm64 for the dispatcher, please let us know. Use the production-repo to get updates of lava-server and lava-dispatcher during the Stretch release process until jessie-backports is open again. 4: At some point after the release, the production-repo will start to be built on and for Stretch. stretch-backports and jessie-backports will need to be open before this change is made. More details of that will appear on this list before it happens. 5: Once Stretch becomes Debian stable, Jessie becomes Debian oldstable. Debian testing gains a new name: Buster. Get used to the names. 6: It is worth checking *now* that all your apt sources on all your machines (including VMs) specify jessie and not stable. Otherwise, as soon as you do an apt upgrade after the Stretch release, your machine will start the upgrade from jessie to stretch. Make sure you are ready for that upgrade and that it doesn't happen until you are ready for it. Also check the images you use in test jobs and the build tools you use to prepare those images. 7: We have been building images based on Stretch ahead of the release for test jobs using QEMU and similar. We have also updated the documentation in the next release to use stretch images. 8: At some point after the Stretch release, there will come a point where support for jessie is dropped and uploads to jessie-backports will cease. The upgrade to stretch will be required at some point to continue getting updates to LAVA. Prepare for it ahead of time and everything will go more smoothly. Further details of the timing will be announced on this list. 9: The LAVA software team have been testing with stretch for some time and will continue to do so. We do not expect any issues with LAVA packages when upgrading from jessie to stretch. Nonetheless, all admins are recommended to prepare for the new Debian release and to perform the upgrade under their own control. Make sure your other services will work after those machines upgrade to Debian 9 Stretch. More information: https://release.debian.org/stretch/freeze_policy.html -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 1 month

1
0
0 0

Update to build-dependencies of lava-server on jessie

by Neil Williams

$ sudo apt-get install -t jessie-backports python-sphinx To solve some problems with the build of the documentation, we have had to move to the version of python-sphinx available in jessie-backports. (Essentially, our documentation is now so large that it is causing problems for the theme we've chosen and we've had to make some changes to keep improve the navigation within the pages.) If you are building lava-server in sbuild, there is an addition option required to sbuild: --build-dep-resolver=aptitude The build script in the lava-dev package uses this option already. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 1 month

1
0
0 0

Switching from uWSGI to Gunicorn - configuration changes

by Neil Williams

There have been problems with using uWSGI for lava-server, including preventing updates of 2016.6-2~bpo8+1 in jessie-backports. The LAVA software team have completed a change which will be included in the 2016.11 release which removes the need for the libapache2-mod-uwsgi, libapache2-mod-wsgi and uwsgi-core packages. The replacement is the gunicorn package. This does involve changes to the apache configuration of all sites which upgrade to 2016.11. Ahead of installing 2016.11, two apache modules need to be enabled. https://staging.validation.linaro.org/static/docs/v2/installing_on_debian.h… $ sudo a2enmod proxy $ sudo a2enmod proxy_http $ sudo service apache2 restart These will not affect older installations. Then install 2016.11 and your instance will continue to operate. If you have made changes to /etc/apache2/sites-available/lava-server.conf then the changes to the upstream file will need to be merged into your changes. If you choose to remove libapache2-mod-uwsgi after completing the install, be aware that there is a Debian bug which may affect you. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=839666 If your apache server fails to restart after removing the libapache2-mod-uwsgi package, you will need to workaround this bug by executing: $ sudo a2dismod uwsgi $ sudo apache2ctl restart -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 1 month

1
0
0 0

2016.9 production release

by Neil Williams

2016.9 includes a rewrite of each of the daemons used by LAVA, lava-server, lava-master and lava-slave. When installing the upgrade, admins need to accept the maintainer's version of the scripts in /etc/init.d/ for these daemons. Where previously changes were made in the scripts in /etc/init.d/, the same parameters can now be changed in associated configuration files to prevent these conflicts in the future. /etc/init.d/lava-slave - change /etc/lava-dispatcher/lava-slave /etc/init.d/lava-master - change /etc/lava-server/lava-master /etc/init.d/lava-server - change /etc/default/lava-server The prompt would look something like this: Configuration file '/etc/init.d/lava-server' ==> Modified (by you or by a script) since installation. ==> Package distributor has shipped an updated version. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** lava-server (Y/I/N/O/D/Z) [default=N] ? y See https://validation.linaro.org/static/docs/v2/pipeline-server.html#configuri… References in the changelogs, commit history and announcements which start with LAVA- followed by a number relate to the planning stories which the LAVA team use for new features. The LAVA planning has now been opened up to anonymous access by appending the reference to a stub URL: http://projects.linaro.org/browse/, e.g. http://projects.linaro.org/browse/LAVA-62 The planning stories do not support comments or changes other than by the LAVA software team. If you have comments on the stories, please use the lava-users mailing list. lava-server changes Show measurements and units of testcase detail page Drop lxc schema - no support for alternatives Configure logging for linaro_django_xmlpc calls Fix link to lava-tool context-help Fix the "Debian-based distribution" link grammar Improve monitor test type, allow multiple tests LAVA-494 allow searching for device tags in tables Add support for LXC with multinode Fix broken ref link for unit_test Fix NoReverseMatch from ImageReports2.0 editing. Fix typo in ./share/validate.py --hostname Update hikey jinja2 templates for LXC Fix occasionally failing multinode tests. Add a configuration file for lava-master Update lava-slave and lava-master docs LAVA-374 - drop versiontools Tidy up some build errors and reformat consistently. Improve documentation describing inline test definitions Major improvements for the "Writing MultiNode" page Add docs for the lava-vland-names change Using BOOTX in Juno bootcmds Correcting default uboot commands in juno device template Document the code locations for developers Ensure multinode description is generated correctly Allowing overrides to connection and action timeouts master: Fix log rotations Remove filtering by filter id for available testcases in image reports. Init: remove the right lock file and simplify Adding uboot jinja2 templates for all Juno flavours LAVA-739 - declare sub_id and job.id for multinode master: don't validate jobs on the master select_device only accepts pipeline jobs Remove legacy SyntaxHighlighter from job definition pages. lava-master: simplify log file handling Fix title disappearance on image reports. Ignore retired devices in validate call Don't add anchors to logs for running jobs Don't crash when log contains invalid test names Schema: accept 'parse' for inline test definitions Use libYAML when loading logs LAVA queries use cases. Bug link fix for url regex. Add LogEntry for BugLinks. Update the init scripts Remove the entry point as it's no longer needed Ensure test-case name is valid Extend vland unit tests to cover assignment Expand on the device dictionary and templates Expand the notes on using jessie-backports Add a Contents page Rewrite lava-server command line from scratch events: set umask to a restrictive value Bug links port. LAVA-200 - track admin actions made in the UI Add device tags support to V2 Allow the monitor test type Update qemu command line options for arm64 lava-dispatcher changes Catch the edge case where serial line corruption means we miss the testrun Add support for alternative Test actions Update manpage of lava-slave for configuration change. Allow configuring the slave from file git: use -C instead of --git-dir Allow setting the envrionment on most systems Handle exceptions from subprocess from compression download: don't expand the path when retrying Grab finalize from the root pipeline Log exception as strings and not objects LAVA-740 - export vlan names and interface names Reopen log file when rotating the logs LAVA-699 - Support hi6220-hikey board with lxc Init: remove the right lock file and simplify pipeline:vland: raise JobError if switch_id or port_id is None Remove redundant assignation Allow logging at the validate stage Force the shell to produce an initial prompt Handle errors if the testrun start is omitted. Pipeline parser: sort and reverse in one call logs: remove unused support for local logs slave: put all tmp files into the same directory Power: improve logging message Exclude patterns from description Port lava-slave to Python3 ApplyOverlayTftp: remove duplicated untar_file ExportDeviceEnvironment: only source valid shell_file's add LEDE to available distributions Always log the command to be run beforehand. There is currently a delay on uploading lava-server to jessie-backports due to problems with uwsgi in Debian. A backport will be made available once these problems are fixed. In the meantime, the same build as will later appear in jessie-backports is available in the production-repo: https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 3 months

1
0
0 0

Configuration file updates in 2016.9

by Neil Williams

A number of the configuration files which manage the daemons used by lava-server and the lava-slave daemon for lava-dispatcher are being updated in 2016.9. This has implications for those who install the upgrade, especially if this is normally done by automation. Configuration file '/etc/init.d/lava-slave' ==> Modified (by you or by a script) since installation. ==> Package distributor has shipped an updated version. What would you like to do about it ? Your options are: Y or I : install the package maintainer's version N or O : keep your currently-installed version D : show the differences between the versions Z : start a shell to examine the situation The default action is to keep your current version. *** lava-slave (Y/I/N/O/D/Z) [default=N] ? y Installing new version of config file /etc/init.d/lava-slave ... dpkg will check the checksums of the init files against the current and previous packages and halt installation with the above prompt in case of doubt. Admins may need to install the update manually (directly on the command line). The recommendation is to accept the package maintainer's version but all admins may want to check the difference between the two. Changes in this release include fixing the log rotate handling of lava-master using reload and a rewrite of the lava-server CLI script to drop the use of entrypoints which changes the syntax of the daemon options. The most common reason for init files to be modified is to change the loglevel. This can be done in /etc/default/lava-server: # Additional options that are passed to the Daemon. DAEMON_OPTS="loglevel=debug" -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 3 months

1
0
0 0

Community access to LAVA planning

by Neil Williams

LAVA has long been developed in the open as far as the git repositories and bug reports are concerned. However, the git commit history can often include references to LAVA-NNN and these references have previously been obscure. Anonymous access has now been granted to all LAVA-* references, so here's some context on what and where. https://projects.linaro.org/projects/LAVA This is a JIRA instance and we refer to issues in JIRA as cards or stories. All stories in LAVA are now visible to the community, so you can map any of the LAVA-NNN references directly to the actual URL by prepending projects.linaro.org/browse/LAVA-NNN, e.g. https://projects.linaro.org/browse/LAVA-734 It has not been possible to open the actual organisation of the stories (KANBAN) and some elements do not provide the same links as when viewed internally but hopefully some of those issues can be addressed in the future. For now, community members can follow any explicit LAVA-NNN reference from the git history or elsewhere to the actual story and view the complete description and all comments on that story. Most stories also provide a direct link to the review which implemented that story within the Comments section. Merged reviews provide information in the "Included In" menu about which branches and tags include that commit. Tags match the production release of the same name as the tag, so this allows everyone to know which releases contain which changes. Stories which are currently status Open may simply be ideas which still need to be scoped. Some of these will be dropped without being implemented as use cases and code development moves on. Dates in stories are very rough estimates, usually longer than it will actually require. If community members have comments or questions on particular stories, please use the lava-users mailing list. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 3 months

1
0
0 0

2016.8 production release

by Neil Williams

This release moves to requiring django 1.8 from jessie-backports. (django1.7 is no longer supported, django1.8 is the LTS release) See https://lists.linaro.org/pipermail/lava-announce/2016-June/000010.html As part of this release, OpenID and therefore Launchpad login support has been removed - the openid support for django cannot support django1.8. See https://lists.linaro.org/pipermail/lava-announce/2016-June/000011.html This release also includes support for running lava-server with the recently released django1.10 which will soon be available in Debian Stretch (current testing). 2016.8 includes a rewrite or large sections of the documentation and the new logging support for LAVA V2. 2016.8 includes the changes planned for 2016.7, so this is a larger set of changes than previous releases. lava-server changes lava-master: use also get_env_string for multinode Handle unrecognised result messages. Publisher: drop privileges at startup Add lava-publisher init scripts Update mustang jinja template Similar jobs feature. Django1.10 fixes v2: include a 'secrets' field in the job def Adding device-type templates for juno Fix a deprecation warning with render_to_string events: add more details and use a useful username Ensure failed health checks go directly to offline. Faster loading of yaml logs Add a u-boot-commands timeout just for panda Improve error handling in result metadata use job.id inside a not job.is_multinode conditional Fix multinode link from definition back to the job. Allow parentheses in test case names Allow the d02 debian installer grub device to be overridden in device-dictionary Allow for creating devices already offline. Device state transition validation. Open context-sensitive help in a new browser tab Unavailable qemu command should not fail unit tests. LAVA-719 - support branding of source and bugs URL Ensure logging to django logs is info or higher Fix e1d66f to use pk when not multinode. Create and display measurements with units result: don't crash when parsing an invalid result Implement notification blacklist. results: handle skip result Simple notification list. Add 'name' to testcase export. Use the right syntax for character delays First device configuration for ST b2120h410 Fix bug #2278 - inconsistent multinode job id / alias usage Fix HTTP500 by allowing for + in test case names Show job sub_id for multinode jobs. Implement IRC notifications. Fix bug #2263 - parameters and params reference in job def and test def LAVA-708 - Device path should be a list Fix a crash when viewing a query for the first time result: show the metadata as a list (and sublists) Rename conflicting notification properties. log: don't show 'extra' result data log: add a link to each line using AnchorJS log: skip broken strings log: add an icon for the download button log: add link from the result page back to the log Results: improve admin page TestResults: order by job_ids then name Simplify a bit the result page Remove unnecessary loading of django-tables log: redirect complete_log to the job_detail page log: add a link to the result page for each result log: fix HTML syntax errors log: improve rendering of errors and exceptions Update load addresses for larger multiplatform kernels Fix result table log: fix a bug when the page is reloaded Protect from admin error in health check submission Fix default value for device_path to be None and not 'None'. Add missing device_path to nexus jinja templates. log: adapt the result parser to the new log stream mustang UBoot needs 32bit header logs: update job status and device information log: change the arrow when clicking on the affix Fix handling of context with multinode Fix metadata handling for multinode and dynamic connections Fix hidden-device-type listings in JobTable job: remove redudant information Events: add a monitoring thread Initial notifications for v2. lava-master: save the logs in output.yaml job: add a new template for the new log format LAVA-262 Allow admins to expire user accounts log: better formatting of tracebacks Remove support for Django < 1.8 Improve scheduler debug with device details. . Documentation updates Add links and notes to developer branch guide Add notes on making Lava Test Shell portable Add notes on running lava-server unit tests Add timeout documentation. Update the developer guide Document the 'secrets' dictionary Ensure V2 documentation examples are available. update local user account image tidy up api docs Remove multinode use cases tidy up the writing-multinode page expand simple-admin for admin roles tidy up hidden toctree listings for previous/next markup Update chapters for theme Switch to the bootstrap theme updates for multinode and simple administration Major update to the docs for writing multinode tests move all examples into one directory and add test definitions move lava tool issues to a separate file fold the FAQ into the lava-tool docs update the multinode use cases port the mustang example to a separate yaml file use rst macros for see also Add publishing API ref doc initial content for a results intro Move doc yaml to a directory WIP rewrite of the multinode doc Start thinking about how to grow a lab Re-org some early admin stuff Split out the completed YAML jobs Query omit documentation updates. Fix documentation for test definition name handling add instruction for -t jessie-backports move example YAML to an rtsi for easier checking add notes on setting up the first device and device type fix whitespace in migration example Update the scheduling ordering with links Add notes on LAVA being developer focused Update other examples for deploy change fixup deploy action add example of first qemu V2 device start the pipeline design page Minor wording tweaks Rework the hacking session doc expand notes on first installation tweaks and updates for writing tests Fix definition link to log for pipeline Updates for test repositories update multinode docs for V2 fix build messages and errors update examples of params support and custom scripts for parsing complete fixme in advanced-installation add background on CI and LAVA add notes and images for first job submission and results explain the first job and tidy up the example YAML Clean up health check docs add notes for first job Significant cleanup of wording around lava-test-shell Add lots of code-block:: yaml directives Add details of features and architecture. Add content to the what-is section lava-dispatcher changes Only require conversion parameters if using conversion. Support some basic kernel conversion tasks Export the 'secrets' dictionary to the overlay uboot: only log self.errors when it's not empty Enabling NFS only deployment Fix connection_timeout handling for lava-test-shell validate needs to set errors, not raise Fixes for multinode without needing ACK Device environment: fix a crash if env_dut is None Unavailable qemu command should not fail unit tests. Use character_delays instead of character-delays Tests: Use the name defined in the job definition Show when a revision is being applied for VCS Fix bug #2263 - parameters and params reference in job def and test def Improve logging and use lazy logging Log more test results metadata LAVA-708 - Device path should be a list Support kickstart installations on grub, and centos distro type Allow for a prefix with nfsrootfs Finalize: send the job status with the right level log: send the full exception untouched beaglebone-black: update load addresses for larger multiplatform kernels Take into account, device_path that is unset. Fix YAML formatting of wait string log change the result logging format Add sample nexus9 device dictionary. LAVA-701 - Support vendor flashing using fastboot. utils/messages.py: fix out of index error Add support for fixup dictionary and patterns Ensure guest command is available in multinode Stop V2 test shells needing an ACK from the dispatcher logging: add datetime and a better structure Upgrade the failure test case for lavabot First batch of VExpress changes Handle ValueError while waiting To enable jessie-backports, simply copy your existing apt source for jessie and change jessie to jessie-backports, then run apt update. Installations from backports are not automatic, you'll need to tell apt to select jessie-backports for django1.8, as outlined in previous mails to this list: $ sudo apt install -t jessie-backports python-django python-django-common python-django-tables2 Once 2016.8 migrates into Stretch and is then backported to jessie-backports itself, apt will be able to handle the extra dependencies. https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… The production repo also has this release. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 4 months

1
0
0 0

Skipping 2016.7

by Neil Williams

Due to resource shortages in the team, conference time and a few problematic fixes which still need testing, we will have to skip the delayed 2016.7 release and move all changes to the 2016.8 release, due at the end of the first week of August 2016. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 5 months

1
0
0 0

OpenID support to cease with 2016.7

by Neil Williams

OpenID has continued to be available for lava-server on jessie as long as lava-server stayed on django1.7 but the OpenID django support does not operate with django later than 1.7. To continue with upstream django security and bug fix support, lava-server needs to move to requiring django1.8 which means that OpenID can no longer be supported in jessie. Support for OpenID in unstable and testing was removed when django1.8 support first arrived. Once lava-server 2016.7 arrives in jessie-backports, it will conflict with the python-django-auth-openid package - this means that to install the 2016.7 upgrade from jessie-backports, apt will first cause the removal of python-django-auth-openid. (Allowing python-django-auth-openid to remain on the system will corrupt subsequent django operations, causing the package installation to fail during the database migrations.) OpenID support in lava-server is also due to be removed for 2016.7, so the documentation and settings will no longer reference OpenID. LDAP support, Debian SSO and local django accounts will remain available. https://packages.debian.org/stable/python-django-auth-openid https://wiki.debian.org/DebianSingleSignOn -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 5 months

1
0
0 0

Django1.8, Django-Tables and lava-server 2016.7

by Neil Williams

Further to the notice about the availability of django1.8 support: https://lists.linaro.org/pipermail/lava-announce/2016-April/000007.html The 2016.7 release of lava-server will be moving to requiring django1.8 from jessie-backports and will no longer work with django1.7 available in jessie. This has been done because the 1.7 tree is no longer receiving upstream security or bug fix support. lava-server installations newer than 2016.2 can use django1.8 already, as long as python-django-tables2 is updated at the same time. So you are invited to upgrade to django1.8 at any time prior to upgrading to 2016.7. To add jessie-backports, copy the apt source for your favourite Debian mirror and paste that as a new line with jessie updated to jessie-backports. e.g. for the http://mirror.bytemark.co.uk/debian mirror: deb http://mirror.bytemark.co.uk/debian jessie main deb-src http://mirror.bytemark.co.uk/debian jessie main becomes deb http://mirror.bytemark.co.uk/debian jessie main deb http://mirror.bytemark.co.uk/debian jessie-backports main deb-src http://mirror.bytemark.co.uk/debian jessie main $ sudo apt update $ sudo apt install -t jessie-backports python-django python-django-tables2 $ sudo apache2ctl restart $ sudo service lava-server restart It is imperative that python-django-tables2 is updated at the same time as python-django - the old version of django will not work with new tables and the old version of tables will not work with new django. Backports do not automatically replace packages you already have installed, so you remain in control of which backports get installed onto your systems and can run apt dist-upgrade without getting extra upgrades. Backports related to lava-server are also fully supported and are the recommended (soon to be required) way of installing updates for lava-server. After 2016.7, updates of lava-server will not install on jessie without django and django-tables being updated from jessie-backports. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 6 months

1
0
0 0

2016.6 production release

by Neil Williams

lava-server changes * Update V1 docs for Ubuntu changes - lava-server no longer migrates into Ubuntu and was removed from Xenial. * Drop heartbeat support * Prevent scheduler ValueError in reservation * scheduler: reduce the number of SQL queries * Expose DISALLOWED_USER_AGENTS to handle search bots * Add a page for listing Pipeline Devices. * Add Auth support in REST API for more functions * Remove device status glyphicons everywhere, since heartbeat is dead. * Create metadata on the number of test definitions * Remove the need for extensions * Remove deprecated lava_projects * Update docs for guestfs and resulting issues. * Enable job definition metadata. * dispatcher-master: support zmq CURVE encryption * Add documentation on using ZMQ curve lava-dispatcher changes * Load the overlay as an extra QEMU drive to prevent need for loopback devices. * Support delays between sending characters for all actions * LXC support for fastboot and adb and drop the need for adb or fastboot to be installed. * Remove adb related code * Port V2 unit tests only to python3 * ZMQ: Support encryption of the logs sent to the server * Add support for QEMU Debian Installer tests * Update standards version (no changes) * Remove adb and fastboot from recommends and add python-guestfs as a dependency. To use LXC with LAVA 2016.6 (V2 only), the lxc package itself needs to be installed from jessie-backports (the lava-dispatcher package has a version requirement in the Recommends which enforces this, if lxc is installed). Additionally, for the new version of LXC to work correctly, the dispatcher itself needs to be rebooted to apply the LXC updates to the kernel boot arguments. To enable jessie-backports, simply copy your existing apt source for jessie and change jessie to jessie-backports, then run apt update. Installations from backports are not automatic, you'll need to tell apt to select jessie-backports for the LXC package: apt install -t jessie-backports lxc https://validation.linaro.org/ has already been updated to 2016.6. Packages have been uploaded to Debian (unstable) and will appear on the mirrors in due course. Once these versions migrate into stretch, backports will be made to jessie-backports. https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… The production repo also has this release. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 6 months

1
0
0 0

New GuestFS dependency added in 2016.6 for V2

by Neil Williams

The 2016.6 release which is currently in preparation adds some important dependencies which some admins may prefer to install ahead of the upgrade. Debian has the concept of Dependencies which must be installed and Recommends which are optional but expected to be useful by most users of the package in question. The Recommends for GuestFS can be omitted from the installation if admins desire but this needs to be done ahead of the upgrade to 2016.6. V2 in 2016.6 adds support for GuestFS which removes the need for loopback support for QEMU devices. GuestFS supports a wide range of filesystem utilities and other virtualisation support tools, not all of which will be of interest to LAVA. (Notably, libguestfs Recommends a mail-transport-agent and the default MTA in Debian is exim. With the default exim4 config, this simply adds a local MTA with no external support.) Debian supports opting out of Recommends when installing packages, so if admins have concerns about extra packages being installed on the dispatchers (e.g. if using ARMv7 dispatchers or simply to reduce the complexity of the install) then Recommends can be omitted for the installation of these dependencies: The option is: $ sudo apt --no-install-recommends install python-guestfs We'll add a note to the installation docs for this too. With the move to V2 having a dumb dispatcher, the dependencies themselves do need to be installed to allow the server to dictate what jobs the dispatcher can run without the dispatcher failing due to missing support. If later LAVA upgrades need any of the packages that would have been installed as Recommends, LAVA will add an explicit dependency on just those packages and any others will remain optional. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 6 months

1
0
0 0

2016.4 backports uploaded - a note on django and django-tables

by Neil Williams

Updates of existing backports are not usually announced in this way but there is a note to consider, so I'm covering that here. Independent of the update of the lava-server and lava-dispatcher backports to 2016.4, admins using jessie-backports do need to be aware of a issue which is still pending related to django and django-tables. A backport of django-tables is pending, waiting for these two packages to also be accepted into jessie-backports: https://ftp-master.debian.org/new/django-haystack_2.4.1-1~bpo8%2B1.html https://ftp-master.debian.org/new/python-fudge_1.1.0-1~bpo8%2B1.html These are required before django-tables can build inside jessie-backports and therefore before I can upload a build of django-tables for jessie-backports. Irrespective of the version of LAVA being used, if django itself is installed from jessie-backports, the version of django-tables in jessie will cause a HTTP500. The fix is to either downgrade django back to the version in jessie or to pull in the version of python-django-tables2 from *testing*.: https://packages.debian.org/stretch/python-django-tables2. It is this upstream version which is to be backported to jessie-backports. Once these last two dependencies clear the queue for backports, I'll make the backport of django-tables and then admins can have django and django-tables from jessie-backports which will work. i.e. the installed django needs to match up with the installed django-tables - either both from jessie or, once available, both from jessie-backports. This isn't a problem in LAVA as such, it affects LAVA and I'll have the fix available just as soon as the queue clears within Debian. Finally, LAVA is likely to move to using django from jessie-backports once this issue is fixed so that instances gain the security fixes in the django 1.8 release - at that point, LAVA will ensure that the updated django-tables is pulled in at the same time. LAVA has been running with django 1.8 (and django1.9) for developers for some time, testing with 1.8 will continue in the meantime. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 7 months

1
0
0 0

2016.4 production release

by Neil Williams

lava-server changes: Add support for python-django-debug-toolbar Deleting V1 filters now cascade delete image chart filters. Reduce the number of SQL queries used on common pages. Improve scheduler iterative performance. Add support for deleting unused tokens Stop runaway healthchecks in V2. Migrate option_list to argparse for django 1.8 and later. Allow authentication with result export in V2 Drop references to Ubuntu beyond 2016.9.post1 Implement omitting individual results from queries in V2 Indicate omitted results and allow including them back in. Add a management command for refreshing queries Change V1 measurement field to be float only. Clean up top-level documentation Introduce limit to queries in V2. * Suggest python-django-debug-toolbar * Refresh all V2 queries during package postinst to ensure materialized views are available. lava-dispatcher changes: Allow for adb not being available on some arches Support ramdisks that are not compressed Move the ramdisk header requirements into the device template Fix calling the Bzr unit tests Use tar flags from deployment data when unpacking overlay kvm-aarch32: introduce device type for 32-bit guest on 64-bit KVM host kvm-aarch64: expose a virtual random number generator (RNG) to the guest Provide an apache2 config for V2 workers Initial support for device using GRUB bootloader in V2 Add in the ability to halt the V1 master image before powering off No longer require ssh options or identity_file Don't use -p port in SCP options https://validation.linaro.org/ will be updated in due course (probably at the start of next week). Packages have been uploaded to Debian (unstable) and will appear on the mirrors in due course. Once these versions migrate into stretch, backports can be made to jessie-backports. https://tracker.debian.org/pkg/lava-server https://tracker.debian.org/pkg/lava-dispatcher https://qa.debian.org/developer.php?email=pkg-linaro-lava-devel%40lists.ali… The production repo also has this release. https://validation.linaro.org/static/docs/v2/installing_on_debian.html#lava… https://validation.linaro.org/static/docs/v1/installing_on_debian.html#lava… Information will also appear at http://releases.linaro.org/components/lava/ in due course. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 8 months

1
0
0 0

Deprecating the old dispatcher

by Neil Williams

The 2016.2 release marks the start of the migration to the pipeline dispatcher. The previous dispatcher code, the documentation for that code and the support for JSON job formats are now *deprecated*. As a result, the corresponding lava-server code has also been *deprecated*, this includes Bundles, Filters and Image Reports (original and 2.0). The deprecated Dashboard is replaced by Results with Queries and Charts which have been ported to the data produced by the pipeline testjobs. Future development within LAVA will be based solely on the pipeline dispatcher and associated server side objects. Support for the deprecated dispatcher will be maintained for future releases only during the migration and *will be removed* once the migration is complete. The migration is expected to take most of 2016. Individual devices and particular items of job support will be gradually disabled until the migration completes for all devices and all jobs. The migration process involves: 0: Adding pipeline support for devices and deployment methods 1: Migrating user submissions and automated submissions to the pipeline support. For the Linaro LAVA instances, this will happen within the particular teams inside Linaro. 2: Making selected devices exclusive to the pipeline so that JSON jobs can no longer be submitted for those devices. At this point, support for JSON jobs which rely on those devices will cease. 3: Once all devices are exclusive, reject all JSON submissions. 4: Disable the old scheduler daemon on the LAVA instance. At some point, probably in 2017 - a month or two after JSON submissions cease - the migration will complete by executing: 5: Removal of code support for the old dispatcher in lava-dispatcher and lava-tool codebases and associated documentation. 6: Removal of code support for database objects specific to the old dispatcher, like Bundle, in lava-server with associated documentation. This will involve the deletion of the Bundle data as well as image reports, filters and BundleStreams. TestJob objects which are not pipeline jobs will also be deleted. 7: Removal of the rest of the code which is still dependent upon or only used to support deprecated objects and functions. 8: Modification of code which only exists to isolate the deprecated objects from the pipeline objects. e.g. newly created jobs or devices will default to pipeline support. LAVA instances outside Linaro will need to manage their own migrations during 2016 if updates are to be applied during 2017. It is not possible to retain the database objects without the deprecated code, so owners of individual instances may choose to create an archive instance which has no online devices, accepts no submissions, just provides access to a snapshot of the data at the time that JSON submissions ceased. To maintain access to the archived data, this instance must not upgrade to LAVA releases after the archive is created and the original devices should persist in the database but be kept in the Offline state. Devices should also have submissions restricted to a single administrator account for which no API token exists. More information on the details of the migration and the state of support for jobs using the old dispatcher will be announced on this mailing list. As a reminder - lava-announce is a read-only list. Posts to this list are only made by the LAVA software team. Replies need to be directed to linaro-validation(a)lists.linaro.org - your email client should do this for you, using the Reply-To header added by mailman. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

9 years, 10 months

1
0
0 0

Issues with Django OpenID Authentication

by Neil Williams

Please reply to linaro-validation(a)lists.linaro.org Please let us know if you are using OpenID authentication with LAVA. Newer versions of django will make it impossible to support django-openid-auth in Debian unstable and testing. The version of django-openid-auth in Jessie can continue to be used, so we would like to know how many users want to continue with this support. OpenID as a protocol has been dying for some time and Linaro has moved over to LDAP, which is fine if LDAP is already available. The time pressure for this change is coming from the schedule to get the latest django and the latest lava packages into Ubuntu Xenial 16.04LTS which means that support needs to be implemented in the 2015.12 or 2016.1 LAVA releases. This is why this is quickly following the trusty change. We have been aware of the issues with django-openid-auth for some time, it was only when we had completed the move of the Cambridge lab to LDAP that changes involving django-openid-auth could be considered. If you are using OpenID authentication (e.g. using Launchpad or Google OpenID), please let us know. If you would like to see some other forms of authentication supported, also let us know. We can investigate Python Social Auth (http://psa.matiasaguirre.net/), if there is interest. If we don't hear from users who want django-openid-auth support for use on Debian Jessie, we will drop django-openid-auth support from all lava builds. This will leave LDAP and local Django accounts in 2015.12. If anyone has experience of other django authentication modules, also let us know. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years

1
0
0 0

Freezing support for LAVA on Ubuntu Trusty 14.04LTS

by Neil Williams

See also: https://lists.linaro.org/pipermail/lava-announce/2015-November/000002.html which was also sent to these lists (except linaro-dev). So far, nobody has come forward as a Trusty user. The only Trusty instance of which we are aware is already due to migrate to Debian Jessie. The LAVA software team are now applying updates which will freeze LAVA software support for Ubuntu Trusty at 2015.9 for lava-dispatcher and 2015.9.post1 for lava-server due to the complexities of supporting both django1.6 and the current django1.7 in Jessie and django1.8, possibly django1.9 by the time Debian Stretch is released. The last packages for Ubuntu Trusty 14.04LTS will be: lava-server 2015.9.post1 lava-dispatcher 2015.9 Once these changes are applied, the Debian packaging used to build future versions of LAVA packages will prevent builds against django1.6 and prevent installation if django1.6 is found, in order to prevent database corruption. This means that Trusty users will not be able to use the results of the dispatcher refactoring. Ubuntu Xenial Xerus - which is planned to be the 16.04LTS in April 2016 - is expected to pick up LAVA software releases from Debian up until the 2016.1 release (possibly 2016.2) and is also expected to be using django1.8. The next Debian stable release (Stretch), for which no date has yet been set, may use django1.9. Initial attempts at migrating a test instance from Trusty to django1.7 did not go well and the migration from Trusty to Xenial cannot be supported by the LAVA software team - the recommendation is to go directly from 2015.9 on Trusty to the same version available for Debian Jessie but there will still be work to be done to prepare and implement the migration which will be instance-dependent. Documentation is being added to assist with this migration but there will remain risks of data loss which will need to be managed for each instance. It is imperative that anyone using Trusty has an up to date backup of the postgresql database dump before considering any migration. If the existing data is to be dropped, a new install on Debian Jessie is recommended. It is not possible for the LAVA software team to support all versions of django from 1.6 to 1.9 - particular problems are known when going from django1.6 to django1.7 as the methods to migrate the lava-server database changed fundamentally in django1.7. Notes are being added to the documentation on the trusty branch based on 2015.9 to be released within lava-server 2015.9.post1 and to the documentation in the master branch (which will go into 2015.12). All future builds of LAVA software will now be made and uploaded only to Debian and releases.linaro.org. So far, nobody has come forward who is willing to maintain packaging for LAVA software on any distribution other than Debian. As the refactoring proceeds, we expect that it will become easier to package LAVA for other distributions but the migration to the refactoring must be complete first. Everyone interested in or using LAVA is encouraged to subscribe to the lava-announce mailing list which is low volume and only used for substantial changes like this. https://lists.linaro.org/mailman/listinfo/lava-announce See also https://validation.linaro.org/static/docs/support.html -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years

1
0
0 0

Calling all Trusty users of LAVA

by Neil Williams

This is a call to *all* users of LAVA on Trusty - please let us know who you are, what you're needs are and whether you are able to migrate to Debian Jessie instead of going from Trusty 14.04LTS to Xenial Xerus 16.04LTS. If you are using LAVA on any release of Ubuntu older than Trusty, the only advice from the LAVA team is to immediately migrate to Debian Jessie. If you are using LAVA on Wily Werewolf or Vivid Vervet, you should also consider testing the migration to Xenial and comparing with a migration to Debian Jessie, as below. LAVA is looking at a fix for the 2015.9 release but Django1.9 is in beta release already. There are issues with django1.8 which are in development. Currently, we are proposing that this update will be applied to 2015.9 and made into a frozen release. The master branch of LAVA will continue to develop and will need to use more features only available in Django1.7 and later. Documentation relating to installing Trusty would then be deprecated and removed in subsequent releases from the master branch. Equally, future database migrations on the master branch would no longer provide south support, using the django migration support provided by django1.7 and later. So these releases would not be built for Trusty - leaving only the frozen branch. Due to the complexity of supporting django1.6, it is unlikely that updates will be available for the frozen branch once this happens.. The changes in the lava-server due to the ongoing refactoring will mean that users of Trusty will be unable to migrate to pipeline support until the server has also been migrated to Xenial 16.04LTS. In addition to this, there is concern that migrating from Trusty and django1.6 all the way to django1.8 or possibly django1.9 in Ubuntu Xenial 16.04LTS is going to be problematic and the LAVA team will be unable to assist in most cases. The alternative is for someone with a reasonably complex lab running Ubuntu to take up a role as tester of the frozen branch *and* responsibility for patches which can maintain trusty support and migration to Xenial 16.04LTS. The problem then will be that it will be a very large transition when 16.04LTS actually becomes available - only for the same lag to start all over again. I'm unsure when Xenial will close the window for migrations from Debian into Xenial - I expect that the 2015.12 release of LAVA will migrate, I expect that 2016.1 will migrate too but I cannot be sure about 2016.2 or 2016.3. That migration is completely outside the control of the LAVA software team. https://launchpad.net/ubuntu/xenial/+source/lava-server Everyone considering staying on Ubuntu is advised to try a migration to Xenial *now* - in a VM, with and without a recent backup of your database and logs. Xenial currently has 2015.11. Also compare with a migration to Debian Jessie by dumping and reimporting the database. In each case, ensure that the permissions on /var/lib/lava-server and sub-directories are retained from the original. Please talk to us and test out what you are going to do. https://validation.linaro.org/static/docs/support.html -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years

1
0
0 0

Skipping 2015.11 availability on Trusty

by Neil Williams

It has become impractical to maintain support for the latest production releases on Ubuntu Trusty 14.04LTS due to behaviour required by the server side changes needed by the dispatcher refactoring. These are important changes and are impractical to backport to the version of python-django available in Trusty. I have tried working around the issues but it involves pulling more dependencies from Jessie or Vivid and in my tests, the existing installation became confused and failed to install due to problems with the database migrations. The LAVA software team do not have the resources to maintain backports of the complete dependency chain in Ubuntu, it is a much more manageable set in Debian Jessie (where the same migration went without issues on multiple machines). The changes in 2015.11 are almost exclusively related to the dispatcher refactoring and therefore there is expected to be limited impact on current installations on Trusty. For the next release, we will investigate code changes which omit the new support for Trusty builds but this will have the effect of preventing those installs from working with the pipeline and the new YAML job submissions. The documentation for Trusty will also be updated, if necessary as a hotfix to 2015.9 just for Trusty. If the code changes do not work cleanly, then the current 2015.9 (and the documentation hotfix to follow) would become the last LAVA releases for Ubuntu Trusty 14.04LTS. Users of Ubuntu Trusty 14.04LTS are reminded that a new LTS is due for release in April 2016 but that the LAVA software team are not using Trusty any longer. Any issues with the migration from Trusty to the new LTS are outside of our control. Migrating an instance from Ubuntu to Debian would involve dumping and re-importing the database - bear in mind that Trusty has postgres9.3 and Jessie has 9.4, so care will be needed to migrate the postgresql cluster correctly - there is documentation for that migration in the lava-server docs. It is not recommended to attempt to migrate the OS itself from Ubuntu to Debian (or vice-versa), a reinstall is a better option, so make a backup and ensure that the backup works. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 1 month

1
0
0 0

New mailing lists and refactoring changes

by Neil Williams

After discussions at the LAVA Users Forum at Connect SFO15 [0], the LAVA team have arranged for some changes in the mailing lists to make it easier to announce and manage the upcoming changes from the ongoing refactoring. https://lists.linaro.org/mailman/listinfo/lava-users - a discussion list amongst users for users, with some admins listening in and helping out. Users in this sense includes test writers and administrators of LAVA instances outside Linaro. It is particularly aimed at those running LAVA in places where Linaro is not necessarily aware of the install. I'm not asking for LAVA instances to declare themselves, it is free software after all, it is an offer of assistance and a request that users take advantage of the support available to manage the implications of the refactoring for their own use cases. Please feel free to initiate threads and discuss issues and problems on the lava-users list. As a reminder, for those who use LAVA but have not recently been to a Connect, the dispatcher is being refactored and this had led to advancements and modifications in the lava-server as well as a completely re-written job submission format. LAVA is retaining compatibility *only* with the Lava Test Shell Definitions (the YAML files people are currently using) and there can be no automated way of converting existing JSON job submissions to the new job submission format (which uses YAML to allow for comments, amongst other improvements). The timetable for these changes is expected to cover most of 2016. After https://validation.linaro.org/ has migrated to the new code, the current dispatcher code will start to be removed from the upstream in 2017. One result of these changes is that as the Linaro production instance migrates away from the current dispatcher codebase, there is a likelihood of some breakage in some parts of the current dispatcher. (This is one of the primary reasons why the dispatcher is being refactored.) The refactoring introduces a lot of benefits, including much more robust communication between the workers and the master, removal of configuration on the workers so that admins only change things in one place, a lot of new methods within the dispatcher to support new types of test and a much cleaner, more modular, codebase for future development. Also, the Linaro Validation Mailman List <linaro-validation(a)lists.linaro.org> is to be renamed (keeping linaro-validation as an alias) to lava-devel for consistency with lava-users and lava-announce. There will be a separate email to the linaro-validation list about these changes shortly. These new mailing lists will be included in the documentation included in next release of the LAVA software (2015.10). Documentation of the new code is already in the master branch of lava-server and installed via the lava-server-doc package and the initial code for the refactoring is being developed in parallel with the current code. This list will be used to announce when the refactoring codebase is ready for admins to begin migrating tests and test writers to the new code and methods - currently, the new dispatcher is developer only. At SFO15, there was also an update on the state of the refactoring. [1] lava-announce will continue to be a low volume list. Reply-To has been set to the linaro-validation mailing list (transparently migrating to lava-devel once the linaro-validation alias is in place). The LAVA developers would ask that everyone running LAVA would subscribe to at least the lava-announce mailing list, to help with the migration to the new support. Videos will be available for most meetings at SFO15 in due course. The slides are already available. [0] https://sfo15.pathable.com/meetings/302656 [1] https://sfo15.pathable.com/meetings/303074 -- Neil Williams ============= Tech Lead, LAVA software team. neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

10 years, 2 months

1
0
0 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-announce