Hi,
I made a first attempt to add REST API for some LAVA objects (jobs, devices and device types). The API is very rudimentary but already solves some basic job filtering issues. However I'm not sure how to add dependencies. There are 2 packages required for this code to work properly. Pip packages are named 'djangorestframework' and 'djangorestframework-filters'. Former has a corresponding debian package 'python3-djangorestframework' but the later does not. Any hints how to add proper dependencies to LAVA?
The code can be found here: https://git.lavasoftware.org/mwasilew/lava/tree/rest_api
milosz
On Fri, 28 Sep 2018 at 22:42, Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
https://master.lavasoftware.org/static/docs/v2/development-intro.html#develo...
"""If your change introduces a dependency on a new python module, always ensure that this module is available in Debian by searching the Debian package lists. If the module exists but is not in the current stable release of Debian, it can be backported but be aware that this will delay testing and acceptance of your change. It is expressly not acceptable to add a dependency on a python module which is only available using pypi or pip install. Introducing such a module to Debian can involve a large amount of work - talk to us before spending time on code which relies on such modules or which relies on newer versions of the modules than are currently available in Debian testing."""
This would be a blocker for this merge request. The packaging work would have to be done first before the CI could work. The package which does exist in Debian can be added to the requirements as part of the merge request:
https://git.lavasoftware.org/lava/lava/tree/master/share/requirements/debian
The files for stretch, buster and unstable need the package to be listed. (The syntax for this is recent, it still needs a tweak in the docs.)
e.g. PyYAML: name: python3-yaml
Python module name: name: Debian package name for Python3
If you need a specific version, that can be specified, follow the examples. Note that stretch needs to be supportable with that version.
See the allauth merge request for an example: https://git.lavasoftware.org/lava/lava/merge_requests/3/diffs
Pip cannot be supported directly. (Mixing virtualenv and packaging is not acceptable, it introduces version conflicts in the dependency chain.)
Adding a package to Debian involves a maintenance burden for that package which the team would have to be able to accept for the lifetime of at least Debian Buster (due for release Q2 2019, supported until ~2022). Django packages tend to be more complex than dependencies added for lava-dispatcher - security fixes et al. It's not impossible, we have added django-restricted-resource, but it depends on the expected usage (how many other packages and users in Debian are going to start using it and filing bugs etc.) and how complex it is to work with the package.
If there is a way to avoid using this module, it should be explored. If it needs to be added, there will be a delay until it's available (it would need to get through to stretch-backports to be part of a production release). Adding new python packages to Debian involves adding them as Python3 and for django modules, with Django moving to Django2.0 soon, Python2 support is not worth adding. If you have other use cases for this package, it is probably worth adding the package to Debian.
On Sat, 29 Sep 2018 at 09:11, Neil Williams neil.williams@linaro.org wrote:
Is there any reason why lava is stuck with debian? Why not package it as pip package and make distro agnostic?
It's about filtering this time. This module is not mandatory, but without it API is only half as useful. I'll provide it as separate patch for those brave who will install it separately.
milosz
packages to Debian involves adding them as Python3 and for django modules, with Django moving to Django2.0 soon, Python2 support is not worth adding. If you have other use cases for this package, it is probably worth adding the package to Debian.
On Mon, 1 Oct 2018 at 09:14, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
This is covered in the docs. LAVA has dependencies which cannot be provided by pip and it has functionality which pip cannot support.
That cannot be supported - LAVA can't accept merge requests which depend on people making their own patches.
On Mon, 1 Oct 2018 at 09:20, Neil Williams neil.williams@linaro.org wrote:
Could you point me to the docs?
I didn't say it will be a merge request. The main part of the REST API can be supported as there is a debian package for it "python3-djangorestframework". Filtering part will be a separate branch "for the brave ones"
milosz
On Mon, 1 Oct 2018 at 10:03, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
https://master.lavasoftware.org/static/docs/v2/first-installation.html#softw...
Also https://master.lavasoftware.org/static/docs/v2/debian.html#developing-lava-o... and https://master.lavasoftware.org/static/docs/v2/development-intro.html#develo...
I can't recommend maintaining a permanent fork. It isn't that much work, probably, to get this package into Debian.
On Mon, 1 Oct 2018 at 10:24, Neil Williams neil.williams@linaro.org wrote:
OK, so from what I read here, there are 2 reasons listed: 1. dependencies not available in pip (guestfs) There already is a python package available for it here: http://download.libguestfs.org/python/ as described in the bug: https://bugzilla.redhat.com/show_bug.cgi?id=1075594.
2. daemons and system-wide configuration is correctly updated I'm not sure what exactly that means but most likely can be done for example with ansible role that corresponds to proper lava version. This would allow to have parts of lava in containers and have easier development process.
I'm not signing up to be a package maintainer (been there once long time ago). I can ask the author, but if the package wasn't created so far I don't have much hopes. Beside that, the branch would be just 1 patch long.
milosz
On Mon, 1 Oct 2018 at 10:46, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
It turns out debian isn't any better in managing dependencies here. In order not to implement all filtering again I tried using django-filter package. Unfortunately the version that is currently in stretch (0.13.0-1) has a bug [1] that is fixed in 1.0.2. So buster version (1.1.0) is OK. There is no fix in stretch-backports. Any hints what to do in such case?
[1] https://github.com/carltongibson/django-filter/issues/673
milosz
On Wed, 3 Oct 2018 at 10:24, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
That's an unhelpful description of the situation - there's been no request to put this change into stretch-backports. Stretch is a stable release, changes don't go in without a reason. Other people using this package in Stretch have obviously not had problems with the version in Stretch.
We can request an upload to stretch-backports of the fixed version in buster. This is how Debian works - packages in testing (currently buster) can go into stable-backports (currently stretch-backports), once that work is requested.
That gives us a change of limited impact which can be properly staged, validated and implemented and which won't change underneath us later.
[1] https://github.com/carltongibson/django-filter/issues/673
milosz
On Wed, 3 Oct 2018 at 10:44, Neil Williams neil.williams@linaro.org wrote:
maybe there is no one using it. It fails as soon as you include it into django settings.
milosz
On Wed, 3 Oct 2018 at 10:50, Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Works fine for me - as long as django settings are set up correctly using only packages from Stretch:
root@lxc-stretch-security:/# lava-server manage shell Python 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516] on linux2 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole)
import django_filters
root@lxc-stretch-security:/# dpkg -l python-django-filters python-django lava-server|cut -c -80 Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-=====================-=================-============-======================= ii lava-server 2016.12-3 all Linaro Automated Valida ii python-django 1:1.10.7-2+deb9u2 all High-level Python web d ii python-django-filters 0.13.0-1 all filter Django QuerySets
If you're using current LAVA or using django from backports, then that would likely be the problem.
Anyway, we can see about a rebuild of the version from buster targetted at stretch-backports.
On Wed, 3 Oct 2018 at 11:13, Neil Williams neil.williams@linaro.org wrote:
It is: https://git.lavasoftware.org/mwasilew/lava/-/jobs/3085 It fails with lava unit tests.
milosz
To close this topic of REST API. There are now 3 branches for REST API in LAVA:
1. very basic one that works out of the box https://git.lavasoftware.org/mwasilew/lava/tree/rest_api_minimal There is a pull request to merge that into LAVA master but currently fails on publishing step. There is a ticket to resolve this as it's the problem with CI rather than the code: https://git.lavasoftware.org/lava/lava/issues/122
2. filtering with django-filters https://git.lavasoftware.org/mwasilew/lava/tree/rest_api_filters Since django-filters from stretch doesn't work with django from stretch-backports and request to include more recent version of django-filters to backports went nowhere [1], there is a separate branch that works on buster. It will work on stretch but requires either manual package installation or installing django-filters from pip (do at your own risk). Details are described in README
3. nested routes API + related filters https://git.lavasoftware.org/mwasilew/lava/tree/rest_api_plus This branch adds v0.2 of the API that replaces the TestJob simple routes with nested routes. So it is possible to do /jobs/<job_id>/suites/<suite_id>/tests/<test_id> queries (or any subpath of it). Please note that this feature has poor performance. It's not clear whether this can be fixed due to access restriction model that is implemented in LAVA. It also allows filtering in the extended paths. This branch also offers related filters as supported in djangorestframework-filters module. This branch is for adventurous users only. It requires installing packages that are not available in debian. README file was updated with installation instructions. Again, do it at your own risk. It didn't brake anything for me, but you never know what might happen in production instance.
[1] https://lists.debian.org/debian-backports/2018/10/msg00004.html
milosz.
lava-users@lists.lavasoftware.org
-
Milosz Wasilewski -
Neil Williams