On 2016-06-10 04:35 AM, Neil Williams wrote:
On 10 June 2016 at 00:15, Konrad Scherer Konrad.Scherer@windriver.com wrote:
Hello,
For the 2016.4 release I had created a custom LAVA extension to add 2 commands and 2 xmlrpc methods. The extension mechanism was very convenient. All I had to do was install an egg on the system and restart the lava service. When I upgraded to 2016.6 the extension did not work due to commit b6fd045cc2b320ed34a6fefd713cd0574ed7b376 "Remove the need for extensions".
I was not able to find a way to add my extension to INSTALLED_APPS and register the xmlrpc methods without modifying settings/common.py and urls.py. I looked through common.py and distro.py and could not find support in settings.conf for extensions. I also looked for a local_settings import which is referenced on the Internet as a common way to extend django, but did not find it. If there is a way to extend LAVA without modifying the LAVA python code, please let me know and I will be happy to send in a documentation patch.
It would have been nice if functionality such as the extension mechanism, which is part of the external interface of LAVA, had gone through a deprecation cycle. A reworked demo app showing how the new extension mechanism works would have also been helpful.
Sorry about that - the extensions support was removed amongst a lot of other changes to (start to) streamline the codebase and remove historical or legacy code, particularly at the lower levels where it affects all operations. It's part of the ongoing work for LAVA V2, including the upcoming move to require django 1.8. The current migration is complex and code sometimes needs to be removed or adapted. Extensions cannot be tested as part of the development of LAVA, so it is preferable for changes to be made inside the LAVA codebase and extensions have therefore been removed. There wasn't a deprecation cycle for this step, so our apologies for that.
The migration to LAVA V2 increases the number and types of methods available to export data from LAVA - XML-RPC, REST api and ZMQ. The intention is that customisation will take place in custom frontends which abstract the generic LAVA data to make it relevant for particular teams. If there are extra calls which need to be exported, these need to be contributed to the LAVA codebase so that these can be tested and maintained.
We've had significant problems when lava has been mixed with pip installs or third-party eggs - introducing instability, failures and errors. The use of any non-packaged code with lava, other than through the existing API, is not supported.
Please talk to us about what the commands and methods were trying to achieve - we are open to having more calls available as long as the code being used can be properly tested within the codebase.
Thank you for the detailed response. The extension code was complex and I understand why it was removed. Ideally the LAVA API would accommodate my requirements so it may help to understand why I wrote the extension and what it currently does:
1) A command to create a pipeline worker and a set of qemu devices. Extending settings.conf to be able to add an app to INSTALLED_APPS would be the simplest way to enable this.
But why did I create the command? I am running a Mesos[1] cluster with 50+ x86 servers and am experimenting with using the cluster to run qemu/lxc tests. The idea is that the LAVA server will be connected to both hardware in a lab and to the mesos cluster and users can run tests for qemu devices on demand using available mesos cluster resources. My proof of concept is working, i.e. a custom mesos scheduler polls the LAVA server for submitted qemu jobs and starts lava workers dynamically to run the jobs. The trend is to use cluster managers like Mesos[1], Nomad[2] or Kubernetes[3] to utilize hardware resources more efficiently. My long term plan is to share the mesos cluster resources between services like LAVA, Jenkins and Marathon[4]. We can discuss the details of this if you are interested.
2) A command that integrates LAVA with our internal lab management software. Unfortunately LAVA does not own the lab devices and when a user reserves a lab target using this management service, this command is used to mark the device offline in LAVA and back online when the lab target is unreserved.
3) XMLRPC methods to mark devices online and offline. I noticed that when a pipeline worker disconnects from the server, its attached devices are not transitioned offline and the scheduler will attempt to reserve and run jobs on these devices. With the ephemeral workers this does not work, so I added the xmlrpc methods to enable and disable only one qemu device per worker when the worker starts and stops.
My custom xmlrpc methods are currently unauthenticated which is unacceptable for a production deployment. My only choice currently is to manually create and share tokens across all the workers which not ideal. Ideally there would be a way for a "manager" process to create and revoke tokens for an ephemeral worker to access a specific device for a limited time. This would require some remote methods to manage tokens. I was not aware of the REST or ZMQ API and will have to investigate that further.
Fundamentally LAVA assumes it owns the devices and that makes it hard to integrate into existing infrastructure. I would be interested in any ideas you have that could make this easier.
I also have Dockerfiles to create lava and lava-worker docker images that I have been using for my experiment. It makes it very easy to setup development instances of lava-server and test upgrades, etc. If you are interested, I can put them on Github. Let me know what would be the best way to share them.
Thank you for your time. I look forward to working with you to make LAVA better.
[1]: http://mesos.apache.org/ [2]: https://www.nomadproject.io/ [3]: http://kubernetes.io/ [4]: https://mesosphere.github.io/marathon/