Hello,
I have a LAVA job with long running test, so I put a timeout of 180 minutes in the test action itself[1].
However, but the job times out after 3989 seconds (~ 66 min)[2].
Looking closer at the "Timing" section of the job, I see that lava-test-shell indeed has a timeout of ~3989 seconds, but I have no idea where that number comes from. That's neither the 10 minutes in the default "timeouts" section, nor the 180 minutes I put in the "test" action.
Hmm, after almost pushing send on this, I now seeing that in the "timeouts" sections, the whole job has a timeout of 70 minutes. So, I assume that means an absolute max, even if one of the actions puts a higher timeout?
So I guess this email now turns into a feature request rather than a bug report.
Maybe LAVA should show a warning at the top of the job if any of the actions has a timeout that's longer than the job timeout.
Kevin
[1] http://lava.baylibre.com:10080/scheduler/job/60374/definition#defline89 [2] http://lava.baylibre.com:10080/scheduler/job/60374#results_694663
On Fri, 30 Nov 2018 at 22:14, Kevin Hilman khilman@baylibre.com wrote:
Hello,
I have a LAVA job with long running test, so I put a timeout of 180 minutes in the test action itself[1].
As you note later, the over-arching job timeout was set at 70 minutes, so that is the maximum the entire test job can be running.
However, but the job times out after 3989 seconds (~ 66 min)[2].
We're aware that this isn't easy to track from the timeout error, it's not an easy thing to fix at runtime because timeouts happen inside exception handlers and without access to the full runtime context of the job.
Looking closer at the "Timing" section of the job, I see that lava-test-shell indeed has a timeout of ~3989 seconds, but I have no idea where that number comes from. That's neither the 10 minutes in the default "timeouts" section, nor the 180 minutes I put in the "test" action.
Hmm, after almost pushing send on this, I now seeing that in the "timeouts" sections, the whole job has a timeout of 70 minutes. So, I assume that means an absolute max, even if one of the actions puts a higher timeout?
Yes.
http://lava.baylibre.com:10080/static/docs/v2/timeouts.html#job-timeouts
So I guess this email now turns into a feature request rather than a bug report.
Maybe LAVA should show a warning at the top of the job if any of the actions has a timeout that's longer than the job timeout.
That we can look at, yes - if a single action timeout in the test job definition exceeds the specified total job timeout.
https://git.lavasoftware.org/lava/lava/issues/177
The job timeout will still trigger if there are two or more actions with long timeouts which, together, could exceed the job timeout if both actions take longer than usual, but that is not for LAVA to decide.
Kevin
[1] http://lava.baylibre.com:10080/scheduler/job/60374/definition#defline89 [2] http://lava.baylibre.com:10080/scheduler/job/60374#results_694663
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Neil Williams neil.williams@linaro.org writes:
On Fri, 30 Nov 2018 at 22:14, Kevin Hilman khilman@baylibre.com wrote:
Hello,
I have a LAVA job with long running test, so I put a timeout of 180 minutes in the test action itself[1].
As you note later, the over-arching job timeout was set at 70 minutes, so that is the maximum the entire test job can be running.
However, but the job times out after 3989 seconds (~ 66 min)[2].
We're aware that this isn't easy to track from the timeout error, it's not an easy thing to fix at runtime because timeouts happen inside exception handlers and without access to the full runtime context of the job.
Looking closer at the "Timing" section of the job, I see that lava-test-shell indeed has a timeout of ~3989 seconds, but I have no idea where that number comes from. That's neither the 10 minutes in the default "timeouts" section, nor the 180 minutes I put in the "test" action.
Hmm, after almost pushing send on this, I now seeing that in the "timeouts" sections, the whole job has a timeout of 70 minutes. So, I assume that means an absolute max, even if one of the actions puts a higher timeout?
Yes.
http://lava.baylibre.com:10080/static/docs/v2/timeouts.html#job-timeouts
So I guess this email now turns into a feature request rather than a bug report.
Maybe LAVA should show a warning at the top of the job if any of the actions has a timeout that's longer than the job timeout.
That we can look at, yes - if a single action timeout in the test job definition exceeds the specified total job timeout.
https://git.lavasoftware.org/lava/lava/issues/177
The job timeout will still trigger if there are two or more actions with long timeouts which, together, could exceed the job timeout if both actions take longer than usual, but that is not for LAVA to decide.
Yeah, that's fine, and I can see that would be harder to calculate, but at least if a single action has a timeout that exceeds the job maximum, that should be a pretty obvious red flag that the submitter did something that cannot work.
Thanks,
Kevin
Hello Kevin,
I agree that the timeout computation can be a bit difficult to grab. For each action in the pipeline, LAVA will compute a remaining time depending on the time since the start of the job and the timeouts (the job timeout, the action timeout, ...).
That why you can see something like 3989 seconds remaining. It's more or less computed as: time.time() - job.start + min(timeouts))
Le ven. 30 nov. 2018 à 23:14, Kevin Hilman khilman@baylibre.com a écrit :
Hello,
I have a LAVA job with long running test, so I put a timeout of 180 minutes in the test action itself[1].
However, but the job times out after 3989 seconds (~ 66 min)[2].
Looking closer at the "Timing" section of the job, I see that lava-test-shell indeed has a timeout of ~3989 seconds, but I have no idea where that number comes from. That's neither the 10 minutes in the default "timeouts" section, nor the 180 minutes I put in the "test" action.
Hmm, after almost pushing send on this, I now seeing that in the "timeouts" sections, the whole job has a timeout of 70 minutes. So, I assume that means an absolute max, even if one of the actions puts a higher timeout?
So I guess this email now turns into a feature request rather than a bug report.
Maybe LAVA should show a warning at the top of the job if any of the actions has a timeout that's longer than the job timeout.
Kevin
[1] http://lava.baylibre.com:10080/scheduler/job/60374/definition#defline89 [2] http://lava.baylibre.com:10080/scheduler/job/60374#results_694663
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
lava-users@lists.lavasoftware.org
-
Kevin Hilman -
Neil Williams -
Remi Duraffort