On Tue, Jul 09, 2019 at 05:04:00PM +0100, Dean Birch wrote:
Hi All,
We've previously had an issue on our LAVA instance where it stopped responding to workers and stopped dispatching jobs when it finished running large job definition (around 25000 lines in the definition, around 1000 deploy/boot/test actions). I've been looking into reproducing this safely in a development environment, and I've got a few observations and questions about how the situation could be improved.
Ah, you're seeing this when the large job *finishes*, not when it's starting up? That's useful data - I think we were all looking at startup!
The lava-master process appears be stuck processing the job results, and takes a painstakingly long time to finish this and send an ACK for END_OK. During this processing, the master doesn't respond to worker pings, and doesn't schedule other jobs. Tracking a bit deeper, it seems that the vast majority of time (I've never seen it finish as I have always restarted the lava services before it finishes) in the walk_actions and build_action functions of the lava_results_app/dbutils.py file: https://git.lavasoftware.org/lava/lava/blob/2019.05.post1/lava_results_app/ dbutils.py#L401 https://git.lavasoftware.org/lava/lava/blob/2019.05.post1/lava_results_app/ dbutils.py#L354
What options is there to mitigate this issue? Some ideas below: - Could we optimize the build_action function? There are a few Django model/db queries in build_action, could some results be queried once and cached? With an obscenely large job, would this even give us enough savings to make the time invested in safely optimizing this worth it?
Oh, wow. build_action looks like it could be really slow. Do you know how many testdata items your job is iterating through?
- What are the implications of not having created ActionData objects for a job? Does this mean that no options will be available in the "Pipeline ↓" drop-down on the job page for quick navigation? Could we optionally abort after a certain amount of these (and make it configurable per LAVA instance)? - Should/could the handling of the results be forked off, so lava-master can continue to schedule more jobs and respond to worker pings, but slowly the ActionData objects can be populated? I'm unsure if you have to be on a special thread to write to Django models. Even if this could be done, would any weird behaviours occur on the slave side as it will still be waiting for the ACK for END_OK from the master?
Any guidance on how to proceed with this would be appreciated! I'm happy to place this and some more details in as a LAVA issue on git.lavasoftware.org if this is easier to track and discuss.
I think that would be useful, yes! :-)
Cheers,