On Mon, 11 Feb 2019 at 14:53, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
-----Ursprüngliche Nachricht----- Von: Neil Williams neil.williams@linaro.org Gesendet: Montag, 11. Februar 2019 15:00 An: Tim Jaacks tim.jaacks@garz-fricke.com Cc: lava-users@lists.lavasoftware.org Betreff: Re: [Lava-users] Multinode pipeline synchronisation / timeouts
On Mon, 11 Feb 2019 at 11:46, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
-----Ursprüngliche Nachricht----- Von: Neil Williams neil.williams@linaro.org Gesendet: Montag, 11. Februar 2019 11:38 An: Tim Jaacks tim.jaacks@garz-fricke.com Cc: lava-users@lists.lavasoftware.org Betreff: Re: [Lava-users] Multinode pipeline synchronisation / timeouts
On Mon, 11 Feb 2019 at 10:29, Tim Jaacks tim.jaacks@garz-fricke.com wrote:
Hello everyone,
I am having problems with timeouts when using the LAVA multinode protocol. Assume the following minimal pipeline with two nodes (device = DUT, remote = some kind of external hardware interfacing with the DUT):
deploy: role: device
boot: role: device
deploy: role: remote
boot: role: remote
test: role: remote
test: role: device
What I would expect: The device is booted first, then the remote is booted.
No. That is a fundamental misunderstanding of MultiNode. One MultiNode test job submission creates multiple LAVA test jobs. Those test jobs are scheduled only when all nodes defined in the target_group are ready to start. When scheduled, all test jobs start immediately.
Once any of the test jobs has got to a Lava Test Definition 1.0 shell, then that shell can use the MultiNode API to send a message declaring that it is ready and then use the same API to wait for whatever other nodes are required.
Afterwards, the tests run on both nodes, being able to interact with each other. The pipeline model seems to be implemented in a way that each node has its own pipeline. This kind of makes sense, because the tests of course have to run simultaneously.
However, in my case booting the device takes a lot more time than booting the remote. This makes the 'test' stage on the remote run a lot earlier than the 'test' stage on the device.
My problem: How do I define a useful timeout value for the 'test' stage on the remote?
This has nothing to do with timeouts. This is about the MultiNode API. https://master.lavasoftware.org/static/docs/v2/multinodeapi.html#mul tinode-api
I am already using the MultiNode API and my test cases are working in general. I also understood that a MultiNode job submission creates sub-jobs for each node.
Are you using lava-send and lava-wait? Have you ensured that the message sent by one node is different to the message sent by any other node?
Messages are cached by the coordinator, so both nodes can lava-send and then lava-wait. The timeout for the test action containing the lava-wait on the fastest node must allow for all the time required for the slower node to get to the point of sending it's message, as well as however long it will take to execute it's own test action.
Maybe I have not made clear enough what my problem is. A 'test' stage has to have a timeout value, right? And this timeout value is defined in the test job description, so it has to be known before the job is submitted.
The test action timeout needs to allow for all the operations within that test action to occur, this includes any time spent waiting.
Let's assume a useful timeout value for my test is 1 minute. That means: if both test stages (on the device and on the remote) start at the same time, the test should be completed in less than 1 minute, otherwise some kind of failure can be assumed.
BUT: in reality, the two test stages do NOT start at the same time. There is no way to predict or influence the point of time a stage starts.
Separate the synchronisation test shell from the other actions - as recommended in the docs. The test shell which waits for the other node to be ready uses the lava-multinode protocol timeout to determine how long to wait.
The MultiNode API needs to be used to enforce that the two test stages DO start at the same time.
Whichever node is fastest sends a signal that it's ready with lava-send and then uses lava-wait to halt further execution until a similar message is sent by the other node. When the second node sends it's signal, the first node continues operation and all subsequent test actions now run as if both nodes had started at the same time.
Then the total timeout for all test definitions within the test action needs to cover the time required for the waiting and for the action itself. This can be calculated from existing single-node test jobs in some cases or by trial and error.
If my deploy and boot stages on the device take 3 minutes, while they take only 1 minute on my remote, the test stage on the remote times out before the test stage on the device has even started.
The test stage timeout needs to include the expected time required for the synchronisation calls.
I created a simple test job as an example: https://pastebin.com/Gtk1xZ6N There are two nodes. One of them sends a signal to the other one via the MultiNode API. But before, the first node runs an additional test stage containing a sleep. If this stage is removed, the job terminates successfully. If it is left in there, the test stage on node2 times out, even though I did not change any of the timeout values.
Of course it will timeout, the action has been extended beyond it's timeout. 180 seconds > 2 minutes.
- test: role: node1 timeout: minutes: 2 definitions:
- repository: metadata: format: Lava-Test Test Definition 1.0 name: sleep
You must change the timeout values if you materially change the amount of time that the test will normally take to execute. By adding 3 minutes to the execution time, you must increase the timeout for that same action by at least 3 minutes.
Okay, that was a quick-and-dirty mistake. I wanted to make the sleep 90 seconds, so that it is within the action timeout.
Do I miss something here? What is the supposed way to handle this?
MultiNode is complex. You must consider the synchronisation to be part of the test action - it must be included in the total timeout. LAVA cannot tell if the test action is just sleeping or working.
Obviously I have to take the boot time difference between the two nodes into account. This seems counter-intuitive to me, since the timeout value should affect the actual test only.
What happens if I use an image on the device which takes even a lot more time to boot?
You extend the lava-multinode protocol timeout and you extend the timeout of the test action of the faster role to allow for the time that will be consumed waiting for the other role.
Or if I insert more testcases on the device which do not need the remote before? In both cases I would have to adjust the timeout value for the remote 'test' stage.
Yes. That is how the synchronisation works. One side waits and one side sends the message. The time required for the wait must be included in the timeout of the action doing the waiting. The timeout set in the protocol is how long each wait can be. Test actions may wait several times for different services and the test action timeout must take this into account. The timeout applies to everything done by the test action, not just any specific test definition.
Thanks, that is what I thought. And I indeed find this counter-intuitive and it seems like a design compromise to me. An action timeout should be affected only by the action for which it is defined.
I don't follow.
You've defined an operation within the action which takes longer than you have defined for that action, so yes, the action timeout is determined by the time taken for the action to complete all the operations that you have defined.
A single test action can include multiple test definitions. The test action timeout must allow for all those definitions to complete.
definitions: - repository: foo # takes 3 minutes - repository: bar # takes 25 minutes - repository: baz # take 30 seconds
The initial recommended timeout for the test action would therefore be 30 minutes. Any less than that and you're likely to get intermittent timeout failures.
Timeouts do NOT exist to artificially speed up test jobs. There needs to be a balance between a timeout which is too aggressive and a timeout which leaves the devices blocked on a dead job for too long.
Also, on average, a complete set of actions takes 27 minutes and 14 seconds, a timeout of 30 minutes is reasonable if the dispersion is fairly tight. If however there is a large variation between values within the grouping behind that average, then the actual timeout for all test jobs in that group could easily need to be 1 hour.
If that second test definition above, bar, is actually a MultiNode synchronisation, then that still comes under the same rule. The MultiNode protocol timeout is entirely separate. It might need to be 15 or 20 minutes for test jobs with a large disparity between the boot sequences of different nodes. It still gets taken into account in the test action timeout. Equally, the protocol timeout may only be 5 minutes but the test definition does many wait and synchronisation operations, starting up various services.
Timeouts cannot necessarily be predicted in advance, especially with Multinode there is a period of optimisation involving trial and error to get the best timeouts. It may require running a hundred test jobs or more to get the right balance.
However, I just found out that I can simply remove the action timeout on the remote node completely (I thought it was mandatory, but obviously it is not). This at least serves as a workaround for my specific case.
No. That just puts in the default timeout of 30 seconds.
Is this a design compromise? Or are there any possibilities of synchronizing pipeline stages on different nodes? I am thinking of some mechanism like "do not start 'test' stage on remote before 'boot' stage on device has finished".
You are thinking of the LAVA MultiNode API.
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun
Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun
--
Neil Williams
neil.williams@linaro.org http://www.linux.codehelp.co.uk/
Mit freundlichen Grüßen / Best regards Tim Jaacks DEVELOPMENT ENGINEER Garz & Fricke GmbH Tempowerkring 2 21079 Hamburg Direct: +49 40 791 899 - 55 Fax: +49 40 791899 - 39 tim.jaacks@garz-fricke.com www.garz-fricke.com WE MAKE IT YOURS!
Sitz der Gesellschaft: D-21079 Hamburg Registergericht: Amtsgericht Hamburg, HRB 60514 Geschäftsführer: Matthias Fricke, Manfred Garz, Marc-Michael Braun