lava-announce

lava-announce@lists.lavasoftware.org

136 discussions

by Neil Williams

Opening up LAVA development ============================ The LAVA software project has been developed in the open from its very beginning: Free Software licensing, public discussion spread across IRC and email, and open access to the software development repositories. The roadmap and planning behind LAVA development has also become increasingly open over time, with the use of publicly archived mailing lists and anonymous access to LAVA issue tracking. A community of users and contributors has steadily grown up around LAVA. With the completion of the migration to the V2 codebase, the Linaro LAVA software team is now moving to a more open development model. We invite new and existing contributors to be directly involved in planning future development of the LAVA software project. To this end, Linaro has created the new LAVA Software Community Project (LSCP) to host and foster LAVA development with more input and control available to the community. This removes the need for contributors to register with Linaro, and provides much more flexibility for LAVA Software development. The existing Linaro lab in Harston, UK, will remain entirely within Linaro, providing test infrastructure for Linaro and its members. The sole focus of the new community project will be LAVA software itself. This will allow the project to focus on priorities beyond just what Linaro needs, for example easier integration of devices and new/different use cases. The Linaro engineers working directly on LAVA software have migrated into a newly-formed Linaro LAVA Software Team and will act as the founding maintainers and contributors for the upstream LAVA Software Community Project. The Linaro LAVA Software Team will still retain access to the Linaro lab staging instance (https://staging.validation.linaro.org/) for functional testing using devices available within the Linaro lab. A new home - https://www.lavasoftware.org The LAVA Software Community Project (LSCP) has a new home at https://www.lavasoftware.org. We have set up initial services there including: https://www.lavasoftware.org/ - the main website https://git.lavasoftware.org/ - a GitLab instance to which we have moved upstream LAVA development https://master.lavasoftware.org/ - a LAVA master service, available for development use Existing LAVA contributors will need to reconfigure their git checkouts accordingly. The new GitLab instance does not share credentials with the existing git.linaro.org service; new accounts will be needed - see the introductory contribution guide https://master.lavasoftware.org/static/docs/v2/contribution-intro.html and the full https://master.lavasoftware.org/static/docs/v2/contribution.html for more details. The former location on git.linaro.org will be archived in due course. New repositories exist for the nightly LAVA builds at https://apt.lavasoftware.org/debian/ (using our existing apt signing key) and the documentation is being updated to match. The primary location for test images used in LAVA software functional testing will now be https://files.lavasoftware.org/; a process to add more files to extend the functional tests will follow soon. URLs using http://images.validation.linaro.org/ will continue to be available for a transitional period. New mailing lists will also be available soon on the lavasoftware.org site. There will be a separate announcement about this shortly; subscriptions are not yet open on the new site. Contributing to the LSCP ======================== The LAVA Software Community Project has published a Contribution Guide https://master.lavasoftware.org/static/docs/v2/contribution.html; all contributors will be expected to adhere to this when submitting issues or merge requests. These guidelines are designed to clarify the requirements for contributions, to make contributing more efficient for all involved. The contribution guide also includes details of how to switch from the old development process to the new. We have also adopted a Code of Conduct for the LSCP ( https://www.lavasoftware.org/code-of-conduct.html). All participants in the LAVA Software Community Project will be expected to communicate and behave appropriately when working with us. We request that contributors with currently unmerged Gerrit reviews on https://review.linaro.org/#/q/status:open+lava/lava should move these across into merge requests in the new system at https://git.lavasoftware.org/lava/lava. We will offer help here on request, but will not automatically migrate them ourselves. All the outstanding LAVA JIRA stories have been converted into GitLab issues at https://git.lavasoftware.org/lava/lava/boards. We intend to use the GitLab workflow for easy integration between CI, issues and merge requests. At some point in the near future, Linaro will disable the LAVA projects on Gerrit (https://review.linaro.org/) and JIRA ( https://projects.linaro.org/). A new way of contributing to LAVA is also under development. A federated functional testing system for LAVA will allow multiple labs to contribute to distributed functional testing of LAVA Software, by running specially designed test jobs on a much wider range of hardware. More details of this will be announced in due course. Please ensure that you abide by the Code of Conduct and please read the contribution guide https://master.lavasoftware.org/static/docs/v2/contribution.html alongside this announcement. Discussion of our new project structure is very welcome: please post to the existing lava-users mailing list at lava-users(a)lists.linaro.org. Please note that some details of the LSCP are not yet finalised, so not every question may be answered at this time. LSCP governance ================ Options for the future governance of the LAVA Software Community Project are being prepared - more details will follow. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 3 months

testy test 1

by Steve McIntyre

Cheers, -- Steve McIntyre steve.mcintyre(a)linaro.org <http://www.linaro.org/> Linaro.org | Open source software for ARM SoCs

6 years, 3 months

Developer builds and git tag

by Neil Williams

To improve the functional test framework, we're looking at a mechanism based on docker and this has had a knock-on effect on how we use git tags. Previously, tags were made against the release branch. To get this new support working, we've moved the 2018.7 tag from the release branch to the master branch. This does have one effect when using the developer build scripts: dch: fatal error at line 1092: New version specified (2018.4.post1-382-gf2f1af5d8-1) is less than the current version number (2018.5.post1-2)! Use -b to force. This is due to the tag issue. To fix, use git pull --tag: neil@sylvester:lava (master)$ git pull --tag remote: Enumerating objects: 1, done. remote: Counting objects: 100% (1/1), done. remote: Total 1 (delta 0), reused 0 (delta 0) Unpacking objects: 100% (1/1), done. >From gitlinaro:lava/lava t [tag update] 2018.7 -> 2018.7 Already up to date. Developer builds will then proceed. Note that we've changed the version syntax slightly too. lava_2018.7-17-gf2f1af5d8-1 See https://staging.validation.linaro.org/static/docs/v2/debian.html#local-vers… -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 4 months

LAVA 2018.7 production release

by Neil Williams

This release has not been uploaded to Debian due to an ongoing transition relating to Python3.7 - see https://tracker.debian.org/pkg/lava and https://release.debian.org/transitions/html/python3.7.html The release is available via the LAVA repositories. A docker container for lava-dispatcher (only) is also available: https://hub.docker.com/r/linaro/lava-dispatcher-production-stretch-amd64/ta… LAVA changes ============ 6978ad899 Revert "Make primary key non-editable" a6c34e83f Output a validation error for missing commands: 839a14d07 Create but not edit objects in admin interface d54b73918 Simplify and rationalise log permissions 0892c0929 Skip deployment_data test if simg2img not installed 34cac8588 worker: add a link to the admin page f4c85dd27 Cosmetics baf22688e lava-slave: cosmetics 1d660d9bd Add further fixes for table searching 5903343d4 Rationalise deployment_data for Python3 e298f5acd api: log when updating device/worker health 5572ad267 Exclude Retired devices from warning messages d874e845c LAVA-1381 CLI to copy an existing device 0736d9d13 Add rk3328-rock64 device type 35594371c Device: set default health to HEALTH_MAINTENANCE 7974b5398 slave: allow to kill the process cleanly 664977da8 Fix notification token after 1a13b85e 2b7e5768c Prevent admins for creating some objects 6383230a6 Make primary key non-editable f770b0455 Add docs on adding devices in offline state faa650080 Remove lxc_cmd_prefix from Action class c0356957b Extend .gitignore for .pytest_cache/ 2e124e609 Fix crash when loader yaml description f037069b3 Make handle_testcase local 138f35d75 Fix bzr tests 425fcaefc Address the Meta data directly bad4d83c8 device-types: add meson-gxm-khadas-vim2 5ca3ec4f4 Fix crash after 210140b0 d5c05e3c5 dt: improve the warning when hc are disabled 0216c20cc Do not crash when description.yaml is not readable 01faa5d41 Port 41747ab2 to android lava-test-runner 7099b83a1 Add a section on metalava 779a3c826 Add coverage default configuration ec8c147bd device-types: convert tegra124-nyan-big to Depthcharge 60c137be0 Remove unused sources 7ce3b9d1b device-types: Remove no-op block f4c557554 Add a few more notes on best practices 9bb6ee3ce Fix crash introduced by 792b63ad ee379d343 Validate timeout format inside test definition fa46c7d72 Use reverse relations to break some cyclic-imports 07889e548 LAVA-419 remove google analytics b89a51557 Allow for 'os' to be optional in deploy actions 5b539c6e2 LAVA-952 per dispatcher configuration 3244fd57f master: remove unused options 1a13b85e7 Move notifications to a specific file ed44797dc Move django signal handlers to the right module 35ff9e2aa test: remove dependency on instance.conf 30e59f1eb Fix log reading when end < start ede84bb8f Loosen the check on health check visibility f38471b6e Raise JobError if commands do not exist 3a6e4895a Some pylint fixes 296ba75cc Outline how to relate a test result to source 8d7afd663 LAVA-960 - remove device-type from device config 8d7de532c Update development notes for new unit tests 8865e27cf Drop functions expecting a JSON job definition ced34d7dd Fix Python3 support for csv in XMLRPC 3adc5434d Allow to download only some part of the logs a0312f682 Fix visibility in pending_jobs_by_device_type 210140b0f Remove support for the old v2 format (output.txt) 09288494c Fix lava-logs crashes with invalid test cases a76b3c14c LAVA-1370 Fix device state when removing TestJob 0e5d01ee7 Update the lava-run example in the docs 792b63ad3 LAVA-1368 provide access to the callback data 59d345080 LAVA-1350 document qemu support in buster b667d371c LAVA-1367 - drop pretty printing of device dictionary bbca0922f Remove post_only and use require_POST c7ede8881 LAVA-1365 - drop call to remote JS 8ca4aae3b Fix undefined variables 92d592816 scheduler.get_pipeline_device_config: fix crash e005377e3 adb: add missing parameter to super().run 7f00e7de7 logs: write log lines before calling TestJob.save ef9664aad Link device hostname and device type on devicedict page. 35f251612 Fix misplaced continue, where a return was intended. 4f6ae7e78 Add sdm845-mtp fastboot template support 77fab7605 Add partition:0 to db820c device-type flash order. eaee79db9 Dont send an os.linesep when interrupting grub d23ee01f8 Log if udev passes a path outside /dev f53cd7e9c Update the warning about over-simplifying integration a5bf77f17 Remove references to python2 and use python3 e9367c747 By default, dict.get() default value is None 5704e19a3 Remove support for django before 1.10 305f019d4 Add a warning to over-simplifying the integration b528871ca Extend test_recovery to check the -bl template bae4d6cad Remove duplicated or unused imports 28049de48 Move debian_package_(version|arch) to lava_common 40fcc1d18 Silence known pytest warnings 804bf8715 Make return statement consistent 37a137c32 Fix exception formating c0f800991 reduce is part of functools in python3 d3531269d Add missing super().run call 2be843f25 Remove python2 code abb90c380 Fix pytest for lava_scheduler_app 6cae769ac remove unused function a87ad6d51 Test lava_common.Timeout class fd8d4a2e9 Remove export PS1 from ssh connections support e824fd392 Add notes on dependencies needed for unit tests b6d2bd601 Set the shell prompt in all cases after 5d7c2090 9bd270263 Allow for existing LAVA_DB_NAME containing hyphens ca7179686 dateutil is missing from server requirements 0679aa426 LAVA-1338 Handle STARTTC, ENDTC and TESTCASE 5d7c20906 LAVA-1120 - drop export PS1 lava-test 9b7ec7120 Use yaml.safe_dump whenever possible a475c825f Fix crash with old job definitions 087d8ad73 LAVA-1359 Fix lava-publisher shutdown 00747fbe8 Add missing requirement for guestfs in lava-dispatcher 7db4042df Update cirun to avoid sudo ecafe79b5 LAVA-1361 Fix timeout computation for some actions e11c8c9f9 Fix job timeout name 9b64f68fd Add example jinja2 files to released tarball. 80703dd4b LAVA-1360 - Investigate Charts Add Query autocompletion bee1d82d3 Fix order of the log lines when validate fails b1af0e771 LAVA-1357 - drop include: functionality 5cb96b87c Finalize: ignore exception but log the error f8a2b520c Fix typo in lava-slave Jobs SQLITE 1cf21c2ed Remove unnecessary Python 2 imports a1278b6df Remove "args" from Action.run() 306e25d83 Add a glossary entry for BMC d623748e3 Move import at the begining of the module cce095f89 Check user rights when returning dt template d4df5f3d6 Separate the server and dispatcher requirements 15519d598 Add missing import f5d60ba5c timing: fix parsing with some timeouts 24a1293e2 make device and heathcheck_job pages similar 69c13d3c1 Fix crash when canceling on device without worker 141aeb176 Show an error if a device does not have a worker 4e289f0e3 device-types: add sun50i-h5-libretech-all-h3-cc 9bde1032e device-types: add sun8i-h3-libretech-all-h3-cc 3872951be device-types: add sun8i-h2-plus-libretech-all-h3-cc 934de4f20 No need to test that a dir exists with os.makedirs 1411a019d Use random.SystemRandom() when applicable c43ac7034 Use CLoader when applicable 4039281ea LAVA-1260 - using Jinja2 templating 48874353a Improve indexing of lava-target-storage b21810690 Set out plan for functional testing 325309ba9 Fix outdated comments relating to V1 support. e0a3aaefe Use contextlib.suppress when applicable 0bbe4d381 Check tag support for multiple devices per role c4ebcfc97 Remove bare except 9aaa115e3 Fix build errors in docs due to removed sections. 661cda0b5 Allow ci-run just for server 0c9cacae3 Remove unicode from yaml definition 2c1b3481d Allow server unit tests to run without nfs-kernel-server eed71633a LAVA-1354 specify the tests directories for pytest 12fa843aa Improve invalid_template check 218eb19e9 Allow for lava-server without lava-dispatcher a2fbd0c5b Add lava-lxc-mocker version to log output. 4d02ab538 Skip wait if no kernel_start_message a2a6715f6 Apply DRY to X15 jinja2 template 306bf6bb5 Fix lxc-create command of lava-lxc-mocker to parse options correctly. 2965e7427 Use '-q' option whenever apt-get is called. 69a96a268 lava/lxc-mocker: lxc-attach support -v (set var) 371790242 Remove django python2 compatibility code 047912932 Remove __future__ import f6813e535 Remove python2 specific code ec2e64b57 Use apt-get instead of apt. bd4d0b736 context_help is a string not a tuple 63bb7c5cc Remove dummy constructor 316268eca Only use "is" when comparing against None 75300f938 Raise an infra error when guestfs fails to start 8a9f75db2 LAVA-1305 - catch errors in vexpress mcc a53748c34 Revert "NFS defaults: vers=3" 6ea49f7fb LAVA-1349 fix state machine when canceling and hc 830df97a0 skip some tests if rpcinfo is not available 95036757e Fix crash when env-dut.yaml is defined 50e347aca Change spawnwindowsize to spawn_maxread 74e000108 LAVA-1351 fix addldapuser user_search encoding 3cd50f3ab LAVA-1349 Cancelled health check should not repeat a72ecbf3e LAVA-1347 allow instances to disable admin Delete 3b88c861c Drop unused setup files. 4110a04a9 Add support for just running the clean task acd3ccf05 Fix idempotency of decompress_file 93dc46bc5 Allow adding static_devices with no children 93bb55ad8 device-type: khadas-vim: use default uboot prompt 26cd076ab device-types: meson-gx-common: interrupt w/ctrl-c 70db0802c Expand the documentation on setup commands 91d99739d NFS defaults: vers=3 ff1af7f6d Fix: Duplicated Wait for USB 7473b94bc LAVA-1342 - improve docs on character delays 60039c38c LAVA-973 - Create a packaging script for permissions and user 934fdb6b9 Fix map_scanned_results() when test status string is invalid c0382aeab device-types: meson8b-odroidc1: interrupt with ctrl-c 731e8df49 device-types: base: allow interrupt_ctrl_list for uboot e737ced93 Split the scheduler API Help by sections d5ef2f9a6 LAVA-909 convert JSON submissions to YAML 7f3ce8960 LAVA-1262 - document script side effects a37a30f7b Don't utf8 encode the log data twice ac6245249 LAVA-1333 fix visibility check for health checks 6ec3ba863 LAVA-1305 - catch errors in vexpress mcc 94c0991d8 Only setup a database once when using pytest 8801f8d26 LAVA-1320 - Add unit test for new minus-lxc jobs 332a40862 Allow limiting the pexpect spawn_window_size e71ed3760 LAVA-1277 Document lavacli e504d0666 LAVA-1299 - clarify Bad and Maintenance health 41747ab2f Use /dev/kmsg for LAVA test signals, if requested 7c9d1a0f4 LAVA-836 publish events from a test shell 576aeb896 LAVA-939 Integrate cc3220SF board b7dbb12d9 device-types: Introduce sunxi-common -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 5 months

Upcoming changes in 2018.7

by Neil Williams

Advance notice of some changes coming in 2018.7 {% set device_type = 'foo' %} is going away =========================================== The purpose of any item in the device dictionary or device-type template is to provide information to lava-run to operate the test job. As far as lava-run is concerned, the device_type is completely meaningless. Device type is a database object, a component of scheduling and administration. So in 2018.7, all device-type Jinja2 templates have been updated to move this to a comment and the schema has been updated to remove the device_type field. If you have local Jinja2 templates which use {% set device_type = 'foo' %}, this needs to be changed to: {# device_type = 'foo' #} or if your template uses: device_type: foo This needs to be removed or commented out: # device_type: foo Any mention of device-type in a device dictionary will be ignored. This also means that deploying 2018.7 will involve changes to most upstream device-type templates. If you use an admin tool like salt, ansible, puppet etc. then it is worth making this change ahead of time. In the Harston lab, we create a code review against the git repo containing the device-type templates. When the admin is ready to upgrade a particular instance, the changes from the review are applied before the package upgrade is started. This prevents dpkg interrrupting the upgrade with prompts about changed configuration files whilst keeping the device-type templates under version control. Database configuration is now upstream ====================================== Previous releases have relied on the Debian packaging to do the work of setting up the PostgresQL database and managing database migrations during installation when a database already exists. We took this upstream so that we could manage the process in Python instead of shell, partly because the psql commands require some complex quoting which is very difficult to read and understand in shell. The other advantage is to have more eyes on the database setup code and to have formal code review of changes. In due course, it may also be helpful with the use of LAVA with docker. Availability ============ 2018.7 is likely to be available on or after 24th July 2018. At that point, Debian will still be handling the migration to Python3.7 which has caused issues in a range of Python packages with new keywords. https://tracker.debian.org/pkg/lava On request, we will not be uploading 2018.7 directly to Debian unstable until this transition is complete. As far as we can tell at the moment, there are no changes required within the LAVA codebase but there are a lot of Python dependencies which are yet to be fixed. This will also delay the arrival of 2018.7 into buster and stretch-backports in Debian. 2018.7 will be available from the LAVA repositories on images.validation.linaro.org once the release is ready. I'll make a separate announcement with the full list of changes at that time. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 5 months

Re: [Lava-announce] [Lava-users] How to add pub key to download source code after LAVA create the container

by Neil Williams

On Fri, 6 Jul 2018 at 09:41, Yuan, ZhanghuiX <zhanghuix.yuan(a)intel.com> wrote: > Hi LAVA mainters, > > > > Nice to meet you. > > I want to add my pub key after LAVA installed container. > Unless you configure LXC to allow this (and that is outside the scope of this list), there is no remote access to the container. Adding the public key will not make it possible to SSH into the container. Hacking sessions are not supported in the LXC. Think of the LXC as a transparent layer on the worker. It is not a separate virtual machine. What are you trying to achieve? If you want to access the rootfs of an LXC, you can do so by logging in to the worker and using the standard LXC support. If you want to push files from the LXC to a remote location, you can use the publishing support: https://staging.validation.linaro.org/static/docs/v2/publishing-artifacts.h… This is not a LAVA constraint, this is a reality of using LXC. > But I don’t which parameters should I need add in my job xx.yaml file. > > Could you give me some help and advice. > > > > > > >>>>>>>>>>>>>>> Some part information of my xx.yaml file: > <<<<<<<<<<<<<<<<<<<<<<<<<< > > > > actions: > > - deploy: > > timeout: > > minutes: 300 > > to: lxc > > os: ubuntu > > packages: [python, wget, unzip, python-pexpect, python-serial, > openssh-server] > > > > - boot: > > prompts: > > - '[root(a)(.*) /]#' > > timeout: > > minutes: 300 > > method: lxc > > > > - test: > > timeout: > > minutes: 300 > > gefinitionsg > > - run: > > steps: > > lava-test-shell: echo "------------------------------------- > debug 1 -----------------------" > > lava-test-shell: wget --no-proxy -q > http://otcpkt.bj.intel.com/downloads/pubkey/sys_oak.zip > > lava-test-shell: mkdir -p ~/.ssh > > lava-test-shell: unzip sys_oak.zip -d ~/.ssh > > definitions: > > - repository: ssh:// > sys_oak@git-amr-4.devtools.intel.com:29418/pk_osi_test-source.git > > from: git > > path: <device>/<test_type>_lxc.yaml > > name: <test_type>-<device> > > params: > > BUILD_NUMBER: <build_number> > > IMAGE_URL: <image_url> > > PRODUCT: <build_name> > > IRC_USER: "sys_oak" > > PUB_KEY: "ssh-rsa > AAAAB3NzaC1yc2EAAAADAQABAAABAQDCDJvT5UPE*************************** > > > > >>>>>>>>>>>>>>> Some part information of my xx.yaml file: > <<<<<<<<<<<<<<<<<<<<<<<<<< > > > > > > > > > > > > Best Regards > > Zhanghui Yuan > > OTC Production Kernel Integration Test > _______________________________________________ > Lava-users mailing list > Lava-users(a)lists.linaro.org > https://lists.linaro.org/mailman/listinfo/lava-users > -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 5 months

Changes to running unit tests

by Neil Williams

After the security hotfix, a number of other security changes are being made. One which will affect developers running unit tests involves the permissions of /etc/lava-server/instance.conf Each installation of lava-server will set the permissions of this file to 0o640 and the owner to the LAVA_DB_USER configuration value (by default, named lavaserver). This is to protect production instances as this file contains the database password. -rw-r----- 1 lavaserver lavaserver 181 Jun 25 08:59 /etc/lava-server/instance.conf Developers need to ensure that the user running the unit tests is now part of this group, e.g. $ sudo adduser <username> lavaserver Otherwise, a permission error will be raised when trying to create the devel database used by the lava-server unit tests: PermissionError: [Errno 13] Permission denied: '/etc/lava-server/instance.conf' See also https://review.linaro.org/#/c/26068/ -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 6 months

2018.5.post1 Security hot fix

by Neil Williams

2018.5.post1 ============ During routine development, a new security scanning tool (bandit) was used on the LAVA codebase. Three security problems were found relating to the Job Submit UI and the loading of YAML files through XMLRPC. The problems date back to 2013, possibly earlier, so all releases of LAVA are affected. Fixes were developed and have now been released. https://review.linaro.org/#/c/25917/ Remove the ability to paste URLs in the submit page https://review.linaro.org/25918 Use requests instead of urlopen https://review.linaro.org/25919 Use yaml.safe_load when parsing user data Thanks to Remi Duraffort for identifying and fixing the issues. Note: These changes are not trivial to backport to previous releases. It is possible but some familiarity with the codebase will be required. We have packed a lot of changes into the time since the end of the migration and we are hoping to have a more stable time ahead. The LAVA software team recommend that all instances look to upgrade to 2018.5.post1. Our apologies for these problems. We are NOT aware of any exploits using these issues but now that the problems are public, it is prudent to apply the available fixes before anything happens. We expect to make more use of bandit and similar tools in future. CVE's have been requested but we don't have the CVE numbers back at this time. The production repo now carries these changes as 2018.5.post1-1+stretch An upload to Debian unstable will follow in due course. (The Debian security team were notified once we had a fix.) An upload to Debian Stretch to update 2016.12-1 is being prepared. -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 6 months

2018.5 Production release

by Neil Williams

This is the first full release from the combined source tree. The expected direction of the LAVA codebase is to add more devices and the work of device integration is and has always been difficult. Merging the codebase removes an unnecessary hurdle in integrating new devices and smoothes the process of ensuring that existing integrations are maintained by generating dispatcher test files directly from the server templates. (This has been a problem in previous releases - static device configuration files in lava_dispatcher unit tests hid problems arising from changes to the templates in lava_scheduler_app. The unit tests are now more representative of real test job processing.) As a validation framework, our own unit test and CI support is a key technical component. lava_scheduler_app and lava_dispatcher modules will remain tightly integrated, using ZMQ to allow for physical separation. Combining the tree also makes it easier to update the documentation each time a new device is integrated. The new lava-common binary package allows lava-server to be installed without needing lava-dispatcher - this will reduce the number of dependencies required on instances with multiple workers where there are no devices attached to the master. The default remains to install lava-dispatcher alongside lava-server - use the --no-install-recommends option, as per the updated documentation. There are changes required in Debian for this release and those will take some time. See https://ftp-master.debian.org/new.html and https://ftp-master.debian.org/new/lava_2018.5-1.html (It is not necessary for every package in that queue to be processed before the LAVA upload is processed, some packages are held back for specific reasons.) Once the LAVA upload clears NEW, the old lava-dispatcher and lava-server source packages will be removed from Debian. One key advantage of this change within Debian is that all lava packages will migrate together - this will prevent a recurrence of the recent problem when lava-dispatcher was delayed arriving in backports but lava-server was accepted, causing installation failures. Both lava-server and lava-dispatcher binary packages depend on the same version of lava-common. When upgrading, lava-common should be added automatically. If not, use dist-upgrade or list the packages explicitly to let apt add the new package. e.g. $ sudo apt upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following NEW packages will be installed: lava-common The following packages will be upgraded: lava-dev lava-dispatcher libprocps6 procps Once lava-common is installed, if a master has no devices attached, admins can choose to remove lava-dispatcher and the extra dependencies can also be removed. *Check the list of packages to be removed on each instance* for packages required by other tools on the master, e.g. local admin scripts and similar. $ sudo apt update $ sudo apt upgrade or to list packages explicitly: # master with devices (may also have workers): $ sudo apt update $ sudo apt install lava-server lava-dispatcher # or for workers: $ sudo apt install lava-dispatcher # or for master without devices: $ sudo apt install lava-server $ sudo apt remove lava-dispatcher $ sudo apt-get --purge autoremove lava changes ============ 4a4bde2a1 LAVA-999 Export master and logger config 6f50e1145 LAVA-1315 add option to update existing users ff8d9f882 Update device-types to match my lab config 809ae34ea Use InfrastructureError for uefi menu timeout e52d83dc6 Stop the Overlay tests duplicating the base class 099f19599 LAVA-1310 separate the template unit tests 198283abb Drop Python2 xmlrpc support 13fa6bdef Fix support for scheduler.jobs.definition a40fe8664 Fix query results export to CSV. b18e74ca9 Create lava_common for Timeout and Exceptions ece1a46d4 Generate device configuration from templates e2d0f4b36 LAVA-1324 QEMU usage of KVM accelerator 59e0ab091 Extend timeouts in example test jobs 4269cc55a Merge branch 'master' into coordinator 7f49853f0 Complete fastboot sequence cannot be governed in device dictionary. 50d69d971 Update instructions for merged codebase e2de099f4 LAVA-1316 Instant scheduling edd48e0b1 signals: always add health_check field 1bbaec1e2 Allow to create health-check directly e0cddeb5e Create admin log entries when looping 2489bcbc8 Add auto-login to fastboot sequence of hi6220 device types. 13a4b5923 Drop debug line to update pipeline_reference da933a151 Add support for optionally running using pytest-3 51c81f659 Test decompression during download streams 3d2541d00 docker: always run with --rm 1cdd275aa Fix ResetDevice comment to match what it actually does. db2775001 Allow to sleep after the usb serial appeared 391595bcc LAVA-1309 - pre-* commands should be called with minus-lxc d524354f7 Add an helper to list all keys of an action namespace 9bfe50d84 Expand command lists in device dictionaries 0ff09a569 LAVA-1173 - allow for matching parents of nodes 3b5f47bf7 Make device_info validate errors unique dad039adb Allow for multiple udev rules in a single file 80ca5346a LAVA-1203 - Allow multiple callback URLs in notify action 120181cb6 ci-run should stop on first error bed5fcb20 Allow dispatcher unit tests to generate device config a3cc054f3 Implementation of recovery mode deploy and boot 47b184796 Print the url when validating file:// download 8d1ffad00 dispatcher: raise a TestError for invalid regexp fb39a9da8 Factorize compatiblity code 4511d4de6 monitor: log the start message c18752652 Fix broken unit test 84769209a Exclude Retired devices from invalid_template check 534f1a9b8 LAVA-1300 - fail if static devices cannot be found 235f17d35 Add the end time to the Recent job errors table e18d27c5a Update templates for changes from Harston lab fa0d9125f Update second UART example jobs 600fc7b4e ci-run: Remove some of the unused args and clean it up for jenkins 36fee805b Revert "Remove unused and obsolete requirements.txt" 2e051cb6d Remove any generated top level init.py 2060d7851 LAVA-1298 fix references to old repositories 1978274d1 Treat kernel panics as a job failure d920b7825 Remove unused and obsolete requirements.txt ed9039e87 LAVA-1297 - Incorporate changes to run jobs without LXC fabd4a44f Port to python3 and fix prospector warning e98737c55 LAVA-1286 Raise JobError if 'params' is not a dict 5af7a7f74 compression: do not use shell=True a4e1ac09b device-types: add meson-gxbb-nanopi-k2 2e84699d8 Port classes to python3 only 6a02322a1 device-types: add meson-gxl-s905x-libretech-cc 968873081 publisher: do not duplicate logs 05d119426 Send the wait-all role reply 80a912d7e Add a ./ci-run script 02a83fed4 Fix waitall with a role support 33644d89e Port to new daemon support with systemd service a1d9f99e3 Explicitly allow wait-all to occur before send 8b2e3feb0 Add a status check script a4cfb1b90 LAVA-489 Python3 support 1c570a0f8 add the gitreview file for lava-coordinator 210e4b7cf bump version -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 7 months

Re: [Lava-announce] [Lava-users] Merging the LAVA code source

by Neil Williams

On 24 April 2018 at 14:09, Zoran S <zoran.stojsavljevic.de(a)gmail.com> wrote: > > Sorry, but this was only a notification of a decision already taken and > implemented by > > the development team. The change itself was not and is not up for > discussion on this list. > > Sorry to be (brutal) honest with you and Linaro. ;-) > > I (sincerely) expect in The Future you/Linaro will be MUCH (much) more > serious with some responsibilities to this list, and to your/Linaro > Lava architecture? > This is a user list. It is not a development list. I remain unconvinced that you, personally, understand the LAVA architecture and design let alone our aims and objectives. > > Or... I expect out there that will be some more serious > people/companies with some more serious testing environment. I hope > for this to happen (competition is All Good, don't you think)! ;-) > > Do we have agreement here (I truly hope so)?! :-) > Not really. LAVA is absolutely serious about automation validation. I disagree with all the rest of your assumptions. > > Zoran Stojsavljevic > > On Tue, Apr 24, 2018 at 10:02 AM, Neil Williams > <neil.williams(a)linaro.org> wrote: > > On 24 April 2018 at 08:21, Zoran S <zoran.stojsavljevic.de(a)gmail.com> > wrote: > >> > >> > Not true. The master produces the configuration which is fed to the > >> > dispatcher - the two are very directly tied > >> > at the level of what devices can support and how that support is used > by > >> > test jobs. Changes and fixes in the > >> > device-type templates will change how the dispatcher uses the code > >> > support to control the device. > >> > >> With all due respect, Sir, I would not agree/would disagree to by > >> your/Linaro's currently presented architecture. > > > > > > Sorry, but this was only a notification of a decision already taken and > > implemented by the development team. The change itself was not and is > not up > > for discussion on this list. > > > >> > >> On Mon, Apr 23, 2018 at 4:06 PM, Neil Williams < > neil.williams(a)linaro.org> > >> wrote: > >> > On 23 April 2018 at 14:54, Zoran S <zoran.stojsavljevic.de(a)gmail.com> > >> > wrote: > >> >> > >> >> > lava-server and lava-dispatcher have been merged into a single > source > >> >> > repository - lava > >> >> > >> >> If you ask me, this is the (very) questionable decision. Indeed!? > >> >> > >> >> lava-server is the front-end manager of the whole Lava test > >> >> environment, while lava-dispatcher is the back-end. > Connected/splitted > >> >> by ZMQ protocol. And... They should be separately maintained > >> > > >> > > >> > Not true. The master produces the configuration which is fed to the > >> > dispatcher - the two are very directly tied at the level of what > devices > >> > can > >> > support and how that support is used by test jobs. Changes and fixes > in > >> > the > >> > device-type templates will change how the dispatcher uses the code > >> > support > >> > to control the device. The two have always been maintained by the same > >> > team > >> > and in a tightly integrated manner. > >> > > >> >> > >> >> , since > >> >> they represent (very) different things. Essentially, they do (very) > >> >> different tasks. They are, after all, apples and oranges. > >> > > >> > > >> > The bulk of the expected work from here on is Device Integrations. > Each > >> > new > >> > device integration typically involves changes to the dispatcher to > >> > support > >> > new deployment / boot methods as well as new templates for the device > >> > configuration. Combining the codebase allows those changes to be more > >> > easily > >> > verified because the output of the device configuration templates can > be > >> > fed > >> > directly to the lava-dispatcher code changes in the dispatcher unit > >> > tests. > >> > > >> > Previously, this has resulted in static device configuration files in > >> > the > >> > dispatcher unit tests and that has caused problems. > >> > > >> >> > >> >> > >> >> In other words, if you update/buy advanced/more expensive apples, you > >> >> also force testers to update for nuthin'/buy for the loss the same > >> >> oranges, they had before?! > >> > > >> > > >> > Not true. the packages are built from a single source package. That > does > >> > not > >> > mean that the binary packages need to be updated on the actual > >> > instances. > >> > > >> > Up until now, lava-server has always depended on the latest version of > >> > lava-dispatcher. That specific dependency is being removed as part of > >> > this > >> > change. > >> > > >> > Nevertheless, the principle remains that the lava-master and > lava-slave > >> > should be updated together so that the device configuration is in line > >> > with > >> > the dispatcher support. See compatibility settings in the > documentation. > >> > > >> > https://lava.codehelp.co.uk/static/docs/v2/simple-admin.html#index-5 > >> > > >> > > >> > > >> >> > >> >> > >> >> Is it the wise decision??? > >> >> > >> >> Two cents worth lamenting/analysis, > >> >> Zoran > >> >> _______ > >> >> > >> >> On Thu, Apr 19, 2018 at 3:41 PM, Neil Williams > >> >> <neil.williams(a)linaro.org> > >> >> wrote: > >> >> > This is for particular note for developers as it changes the way > that > >> >> > reviews happen. > >> >> > > >> >> > If you've noticed a few reviews being abandoned, this is why. > >> >> > > >> >> > lava-server and lava-dispatcher have been merged into a single > source > >> >> > repository - lava > >> >> > > >> >> > lava-coordinator will follow in time, to ease the update of > >> >> > lava-coordinator > >> >> > to Python3. > >> >> > > >> >> > This will, in future, allow easier testing of device integrations > by > >> >> > allowing the lava_scheduler_app unit tests to be linked to the > >> >> > lava_dispatcher unit tests and have a single review which adds both > >> >> > sides of > >> >> > the device support. > >> >> > > >> >> > There will be a lot of testing, as normal, so staging will be > moving > >> >> > to > >> >> > packages built from the new source repository tree. > >> >> > > >> >> > The old lava-server.git and lava-dispatcher,git repositories will > >> >> > become > >> >> > read-only and will get no further code changes. All changes will be > >> >> > done > >> >> > in > >> >> > lava.git > >> >> > > >> >> > https://git.linaro.org/lava/lava.git/ > >> >> > > >> >> > The documentation will be updated over the next few days. > >> >> > > >> >> > -- > >> >> > > >> >> > Neil Williams > >> >> > ============= > >> >> > neil.williams(a)linaro.org > >> >> > http://www.linux.codehelp.co.uk/ > >> >> > > >> >> > _______________________________________________ > >> >> > Lava-announce mailing list > >> >> > Lava-announce(a)lists.linaro.org > >> >> > https://lists.linaro.org/mailman/listinfo/lava-announce > >> >> > > >> >> _______________________________________________ > >> >> Lava-users mailing list > >> >> Lava-users(a)lists.linaro.org > >> >> https://lists.linaro.org/mailman/listinfo/lava-users > >> > > >> > > >> > > >> > > >> > -- > >> > > >> > Neil Williams > >> > ============= > >> > neil.williams(a)linaro.org > >> > http://www.linux.codehelp.co.uk/ > > > > > > > > > > -- > > > > Neil Williams > > ============= > > neil.williams(a)linaro.org > > http://www.linux.codehelp.co.uk/ > -- Neil Williams ============= neil.williams(a)linaro.org http://www.linux.codehelp.co.uk/

6 years, 8 months

← Newer
1
...
5
6
7
8
9
10
11
...
14
Older →

Jump to page:

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

lava-announce