Hi Jorgen,
On Mon, Aug 13, 2018 at 02:23:34AM +0000, Jorgen Peddersen wrote:
I'm trying to use a situation where a large bulk of similar jobs are being sent to several devices of the same device type. These devices require the 'transfer_overlay' option in order for the tests to work due to their setup. We currently are using a single master and worker on the same server.
It seems that the same transfer overlay file location (i.e., /var/lib/lava/ dispatcher/tmp/overlay-1.3.2.5.tar.gz) is used regardless of the job id and that some of our jobs are experiencing a collision trying to use different copies of that file at the same time. Thus, one of our jobs is deleting the transfer overlay file before another job tries to delete it and the second gets errors.
This is the error LAVA produces when viewing the job in the GUI:
File "/usr/bin/lava", line 11, in <module> load_entry_point('lava-tool==0.25', 'console_scripts', 'lava')() File "/usr/lib/python2.7/dist-packages/lava/tool/dispatcher.py", line 150, in run raise SystemExit(cls().dispatch(args)) File "/usr/lib/python2.7/dist-packages/lava/tool/dispatcher.py", line 140, in dispatch return command.invoke() File "/usr/lib/python2.7/dist-packages/lava/dispatcher/commands.py", line 247, in invoke self.args.validate) File "/usr/lib/python2.7/dist-packages/lava/dispatcher/commands.py", line 91, in run_pipeline_job exitcode = job.run() File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/job.py", line 340, in run self.cleanup(self.connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/job.py", line 375, in cleanup self.pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 649, in cleanup self.internal_pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 649, in cleanup self.internal_pipeline.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/action.py", line 251, in cleanup child.cleanup(connection) File "/usr/lib/python2.7/dist-packages/lava_dispatcher/pipeline/actions/boot/__init__.py", line 458, in cleanup os.unlink(self.url) OSError: [Errno 2] No such file or directory: '/var/lib/lava/dispatcher/tmp/overlay-1.3.2.5.tar.gz'
We are working on the 2017.7 build of LAVA, so if this has been changed in a later build, please let me know. If this has not been updated, is there another way to change things so that each job uses a different name and/or location for the overlay file?
2017.7 is quite old, and as far as I can see there was a fix for this issue some time ago, in commit 004371066fe7f13cb75d0d344852ad1863aece22. That fix was included for the first time in the 2018.4 release.
Please try upgrading...
Cheers,