On 12 May 2017 at 00:47, Don Brown don.brown@codethink.com wrote:
Hi Everyone,
This week one of my teammates discovered that storage was very low on our LAVA Server. After he investigated, he found that /var/lib/lava/dispatcher gradually increases in size. He realized that when a test is run, the files are accumulated in /var/lib/lava/dispatcher/slave/tmp for each job. However, they are never removed.
Does LAVA have a setting or does it have some kind of automation that will remove tests, say, after X days or by some other criteria or, do we need to remove them manually?
We are looking at how best to handle these files at the moment. There is a use case for having these files available for debugging purposes (at least for some test jobs) but it hasn't been fully scoped. We will put out a full announcement on this problem next week.
For now, those files can be removed if there are storage concerns on the worker. Admins are free to keep files for selected test jobs or remove them all, it will not affect future functioning of the worker to remove the files.
The principle area of long term storage for test jobs is in /var/lib/lava-server/default/media/ on the master and we have changed that to organise the files by date based sub-directories to make it easier for admins to archive the data.
There are other issues here as well relating to management of test definition repositories - it is surprising how much space is taken up by a single clone of git://git.linaro.org/qa/test-definitions.git. That leads to a possibly bloated overlay being transferred to the device which may be using a ramdisk or tmpfs.
So, yes, there is a problem here on workers and we will get fixes and documentation into place for 2017.6. In the meantime, unless admins want to keep those files for debugging, all data in /var/lib/lava/dispatcher/slave/tmp can be removed from workers.