On 24 April 2018 at 09:08, Quentin Schulz <quentin.schulz@bootlin.com> wrote:

Hi Neil,

I think there was a global misunderstanding from a poorly choice of
words. When I was saying "available device" I meant a device that isn't
in Maintenance or Retired. If the device is idle, running a job,
scheduled to run a job, etc... I consider it "available". Sorry for the
misunderstanding.

Currently, "Maintenance" is considered as available for submission & for scheduling. This is to support maximum uptime and minimal disruption to CI loops for temporary work happening on devices.

We are looking at clarifying this soon.

I know this has been a long and complex thread. Thank you for sticking with the discussion, despite the complexity and terminology. 

 

On Mon, Apr 23, 2018 at 12:54:03PM +0100, Neil Williams wrote:
> On 23 April 2018 at 11:21, Quentin Schulz <quentin.schulz@bootlin.com>
> wrote:
>
> > Hi Neil,
> >
> > Thanks for your prompt answer.
> >
> > On Fri, Apr 20, 2018 at 07:56:29AM +0100, Neil Williams wrote:
> > > On 19 April 2018 at 20:11, Quentin Schulz <quentin.schulz@bootlin.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I've encountered a deadlock in my LAVA server with the following
> > scheme.
> > > >
> > > I have an at91rm9200ek in my lab that got submitted a lot of multi-node
> > > > jobs requesting an other "board" (a laptop of type dummy-ssh).
> > > > All of my other boards in the lab have received the same multi-node
> > jobs
> > > > requesting the same and only laptop.
> > > >
> > >
> > > That is the source of the resource starvation - multiple requirements of
> > a
> > > single device. The scheduler needs to be greedy and grab whatever
> > suitable
> > > devices it can as soon as it can to be able to run MultiNode. The primary
> > > ordering of scheduling is the Test Job ID which is determined at
> > submission.
> > >
> >
> > Why would you order test jobs without knowing if the boards it depends
> > on are available when it's going to be scheduled? What am I missing?
> >
>
> To avoid the situation where a MultiNode job is constantly waiting for all
> devices to be available at exactly the same time. Instances frequently have
> long queues of submitted test jobs, a mix of single node and MultiNode. The
> MultiNode jobs must be able to grab whatever device is available, in order
> of submit time, and then wait for the other part to be available.
> Otherwise, all devices would run all single node test jobs in the entire
> queue before any MultiNode test jobs could start. Many instances constantly
> have a queue of single node test jobs.
>

That's understood and expected.

>
> >
> > > If you have an imbalance between the number of machines which can be
> > > available and then submit MultiNode jobs which all rely on the starved
> > > resource, there is not much LAVA can do currently. We are looking at a
> > way
> > > to reschedule MultiNode test jobs but it is very complex and low
> > priority.
> > >
> > > What version of lava-server and lava-dispatcher are you running?
> > >
> >
> > lava-dispatcher                   2018.2.post3-1+jessie
> > lava-server                       2018.2-1+jessie
> > lava-coordinator                 0.1.7-1
> >
>
> (You need to upgrade to Stretch - there will be no fixes or upgrades
> available for Jessie. All development work must only happen on Stretch. See
> the lava-announce mailing list archive.)
>

Thanks, we'll have a look into this.

>
> >
> > > What is the structure of your current lab?
> > >
> > > MultiNode is complex - not just at the test job synchronization level but
> > > also at the lab structure / administrative level.
> > >
> >
> > I have two machines. One acting as LAVA server and one as LAVA slave.
> > The LAVA slave is handling all boards in our lab.
> >
> > I have one laptop (an actual x86 laptop for which we know the NIC driver
> > works reliably at high (~1Gbps) speeds) that we use for MultiNode jobs
> > (actually requesting the laptop and one board at the same time only) to
> > test network. This laptop is seen as a board by LAVA, there is nothing
> > LAVA-related on this board (it should be seen as a device).
> >
> >
> Then you need more LAVA devices to replicate the role played by the laptop.
> Exactly one device for each MultiNode test job which can be submitted at
> any one time. Then use device tags to allocate one of the "laptop" devices
> to each of the other boards involved in the MultiNode test jobs.
>
> Alternatively, you need to manage both the submissions and the device
> availability.
>
> Think of just the roles played by the devices.
>
> There are N client role devices (not in Retired state) and there are X
> server role devices where the server role is what the laptop is currently
> doing.
>
> You need to have N == X to solve the imbalance in the queue.
>
> If N > 1 (and there are more than one device-type in the count 'N') then
> you also need to use device tags so that each device-type has a dedicated
> pool of server role devices where the number of devices in the server role
> pool exactly matches the number of devices of the device-type using the
> specified device tag.
>
>
> > >
> > > >
> > > > I had to take the at91rm9200ek out of the lab because it was behaving.
> > > >
> > >
> > >
> > >
> > > > However, LAVA is still scheduling multi-node jobs on the laptop which
> > > > requires the at91rm9200ek as the other part of the job, while its
> > status
> > > > is clearly Maintenance.
> > > >
> > > >
> > > A device in Maintenance is still available for scheduling - only Retired
> > is
> > > excluded - test jobs submitted to a Retired device are rejected.
> > >
> >
> > Why is that? The device is explicitely in Maintenance, which IMHO tells
> > that the board shouldn't be used.
> >
>
> Not for the scheduler - the scheduler can still accept submissions and
> queue them up until the device comes out of Maintenance.
>
> This prevents test jobs being rejected during certain kinds of maintenance.
> (Wider scale maintenance would involve taking down the UI on the master at
> which point submissions would get a 404 but that is up to admins to
> schedule and announce etc.)
>
> This is about uptime for busy instances which frequently get batches of
> submissions out of operations like cron. The available devices quickly get
> swamped but the queue needs to continue accepting jobs until admins decide
> that devices which need work are going to be unavailable for long enough
> that the resulting queue would be impractical. i.e. when the length of time
> to wait for the job exceeds the useful window of the results from the job
> or when the number of test jobs in the queue exceeds the ability of the
> available devices to keep on top of the queue and avoid ever increasing
> queues.
>

I guess that's an implementation choice but I'd have guessed the
scheduler was first looping over idle devices to then schedule the
oldest job in the queue for this device type.

But my understanding is that the scheduler rather sets an order when
jobs are submitted that isn't temperable with. Is that correct?

The order is priority, submit_time and then target_group.

Changing ordering on-the-fly and backing out from certain states is the subject of https://projects.linaro.org/browse/LAVA-595 - that is the work I've already described as low priority, large scale and complex.

 

You do have the ability to set the Priority of new test jobs for submission which want to use the laptop in a MultiNode test job along with a device which is NOT the at91rm9200ek. You will need to cancel the test job involving the at91rm9200ek which is currently scheduled. (Other test jobs for the at91rm9200ek in the Queue can be left alone provided that these test jobs have a lower Priority than the jobs you want to run on other devices.)

When the scheduler comes back around, it will find a new test job with higher Priority which wants to use the laptop with a hikey device or whatever and the at91rm9200ek will be ignored. It's not perfect because you would then need to either keep that Priority pipe full or cancel the submitted test jobs for the at91rm9200ek.

>
> >
> > > Once a test job has been submitted, it will be either scheduled or
> > > cancelled.
> > >
> >
> > Yes, that's understood and that makes sense to me. However, for "normal"
> > jobs, if you can't find a board of device type X that is available, it
> > does not get scheduled, right? Why can't we do the same for MultiNode
> > jobs?
> >
>
> Because the MultiNode job will never find all devices in the right state at
> the same time once there is a mixed queue of single node and MultiNode jobs.
>
> All devices defined in the MultiNode test job must be available at exactly
> the same time. Once there are single node jobs in the queue, that never
> happens.
>
> A is running
> B is Idle
> MultiNode submitted for A & B
> single node submitted for A
> single node submitted for B
>
> scheduler considers the queue - MultiNode cannot start (A is busy), so move
> on and start the single node job on B (because the single node test job on
> B may actually complete before the job on A finishes, so it is inefficient
> to keep B idle when it could be doing useful stuff for another user).
>
> A is running
> B is running
>
> no actions
>
> A completes and goes to Idle
> B is still running
>
> and so the pattern continues for as long as there are any single node test
> jobs for either A or B in the queue.
>
> The MultiNode test job never starts because A and B are never Idle at the
> same time until the queue is completely empty (which *never* happens in
> many instances).
>
> So the scheduler must grab B while it is Idle to prevent the single node
> test job starting. Then when A completes, the scheduler must also grab A
> before that single node test job starts running.
>
> A is running
> B is Idle
> MultiNode submitted for A & B
> single node submitted for A
> single node submitted for B
>
> B is transitioned to Scheduling and is unavailable for the single node test
> job.
>
> A is running
> B is scheduling
>
> no actions
>
> A completes and goes to Idle
> B is scheduling
>
> Scheduler transitions A into scheduling - that test job can now start.
>
> (Now consider MultiNode test jobs covering a dozen devices in an instance
> with a hundred mixed devices and permanent queues of single node test jobs.)
>
> The scheduler also needs to be very fast, so the actual decisions need to
> be made on quite simple criteria - specifically, without going back to the
> database to find out about what else might be in the queue or trying to
> second-guess when test jobs might end.
>

That is understood as well for devices that are idle, running or
scheduled to run. The point I was trying to make was why schedule a job
for a device that is in Maintenance (what I meant by the poorly chosen
"available" word).

Is that because one the job is submitted it's ordered by the scheduler
and then run by the scheduler in the given order and the jobs are not
discriminated against the device Maintenance status?

The problem you have is not that the device is in Maintenance but that the other device(s) in the MultiNode test job are Idle. Therefore, those jobs get scheduled because there is no reason not to do so.

If the submission was to be rejected when all devices of the requested device-type are in Maintenance, that is a large change which would negatively impact a lot of busy instances.

We do need to clarify these states but essentially, Maintenance is a manual state change which has the same effect as the automatic state change to Bad. That is as far as it goes currently.

 

>
> >
> > > Now, until I put the at91rm9200ek back in the lab, all my boards are
> > > > reserved and scheduling for a multi-node job and thus, my lab is
> > > > basically dead.
> > > >
> > > >
> > > The correct fix here is to have enough devices of the device-type of the
> > > starved resource such that one of each other device-type can use that
> > > resource simultaneously and then use device-tags to match up groups of
> > > devices so that submitting lots of jobs for one type all at the same time
> > > does not simply consume all of the available resources.
> > >
> > > e.g. four device-types - phone, hikey, qemu and panda. Each multinode job
> > > wants a single QEMU with each of the others, so the QEMU type becomes
> > > starved, depending on how jobs are submitted. If two hikey-qemu jobs are
> > > submitted together, then 1 QEMU gets scheduled, waiting for the hikey to
> > > become free after running the first job. If each QEMU has device-tags,
> > then
> > > the second hikey-qemu job will wait not only for the hikey but will also
> > > wait for the one QEMU which has the hikey device tag. This way, only
> > those
> > > jobs would then wait for a QEMU device. There would be three QEMU
> > devices,
> > > one with a device tag like "phone", one with "hikey" and one with
> > "panda".
> > > If another panda device is added, another QEMU with the "panda" device
> > tag
> > > would be required. The number of QEMU devices required is the sum of the
> > > number of devices of each other device-type which may be required in a
> > > MultiNode test job.
> > >
> > > This is a structural problem within your lab.
> > >
> > > You would need one "laptop" for each other device-type which can use that
> > > device-type in your lab. Then each "laptop" gets unique a device-tag .
> > Each
> > > test job for at91rm9200ek must specify that the "laptop" device must have
> > > the matching device tag. Each test job for each other device-type uses
> > the
> > > matching device-tag for that device-type. We had this problem in the
> > > Harston lab for a long time when using V1 and had to implement just such
> > a
> > > structure of matched devices and device tags. However, the need for this
> > > disappeared when the Harston lab transitioned all devices and test jobs
> > to
> > > LAVA V2.
> > >
> >
> > I strongly disagree with your statement. A software problem can often be
> > dealt with by adding more resources but I'm not willing to spend
> > thousands on something that can be fixed on the software side.
> >
>
> We've been through these loops within the team for many years and have
> millions of test jobs which demonstrate the problems and the fix. I'm
> afraid you are misunderstanding the problem if you think that there is a
> software solution for a queue containing both MultiNode and single node
> test jobs - other than the solution we now use in the LAVA scheduler. The
> process has been tried and tested over 8 years and millions of test jobs
> across dozens of mixed use case instances and has proven to be the most
> efficient use of resources across all those models.
>
> Each test job in a MultiNode test is considered separately - if one or more
> devices are Idle, then those are immediately put into Scheduling. Only when
> all are in Scheduling can any of those jobs start. The status of other test
> jobs in the MultiNode group can only be handled at the point when at least
> one test job in that MultiNode group is in Scheduling.
>

I think there is a global misunderstanding due to my bad choice of
words. I understand and I'm convinced there are no other ways to deal
with what you explained above.

>
> >
> > Aside from a non-negligeable financial and time (to setup and maintain)
> > effort to buy a board with a stable and reliable NIC for each and every
> > board in my lab, it just isn't our use case.
> >
> > If I would do such a thing, then my network would be the bottleneck to
> > my network tests and I'd have to spend a lot (financially and on time or
> > maintenance) to have a top notch network infrastructure for tests I
> > don't care if they run one after the other. I can't have a separate
> > network for each and every board as well, simply because my boards often
> > have a single Ethernet port, thus I can't separate the test network from
> > the lab network for, e.g. images downloading that are part of the
> > booting process, hence I can't do reliable network testing even by
> > multiplying "laptop" devices.
> >
> > I can understand it's not your typical use case at Linaro and you've
> > dozens and dozens of the same board and a huge infrastructure to handle
> > the whole LAVA lab and maybe people working full-time on LAVA, the lab,
> > the boards, the infrastructure. But that's the complete opposite of our
> > use case.
> >
> > Maybe you can understand ours where we have only one board of each
> > device type, being part of KernelCI to test and report kernel booting
> > status and having occasional custom tests (like network) on upstream or
> > custom branches/repositories. We sporadically work on the lab, fixing
> > the issues we're seeing with the boards but that's not what we do for a
> > living.
> >
>
> I do understand and I personally run a lab in much the same way. However,
> the code needs to work the same way in that lab as it does in the larger

Of course.

> labs. It is the local configuration and resource availability which must
> change to suit.
>
> For now, the best thing is to put devices into Retired so that submissions
> are rejected and then you will also have to manage your submissions and
> your queue.
>

Can't we have a "Maintenance but I don't know when it's coming back so
please still submit jobs but do not schedule them" option :D ?

This is exactly what you already have - but it's not actually what you mean.

The problem is that you're thinking of the state of the at91rm9200ek when what matters to the scheduler is the state of the laptop device *in isolation*. The state of the at91rm9200ek only matters AFTER the laptop has been assigned.

What you mean is:

"I don't know when ANY of the devices of device-type A are going to be ready to start a test job, so do not allow ANY OTHER device of ANY OTHER device-type to be scheduled either IF that device-type is listed in a MultiNode test job which ALSO requires a device of this device-type".

(The reason it's ANY is that if 10 test jobs are submitted all wanting at91rm9200ek and laptop, then if you had 10 laptops and 1 at91rm9200ek, those 10 laptops would also go into Scheduled - that is the imbalance we talked about previously.)

It is a cross-relational issue.

Correlating across the MultiNode test job at the scheduling state is likely to have a massive impact on the speed of the scheduler because:

0: a device in Maintenance or Bad is NOT currently available to be scheduled - that's the whole point.

1: the other device(s) in the MultiNode group DO get scheduled because those were in Idle

2: asking the scheduler to check the state of all devices of all device-types mentioned in a MultiNode job when considering whether to schedule any other device in that same MultiNode job is going to make the scheduler SLOW.

So what we do is let the Idle (laptop) device go into a waiting state and let the scheduler move the at91rm9200ek device into scheduling *only* when a at91rm9200ek device becomes available for scheduling - as two completely separate decisions. Then, the relational work is done by the database when the lava-master makes the query "which test jobs are available to start NOW". This is much more efficient because we are looking at jobs where all devices in the target_group are in state SCHEDULED. The database can easily exclude test jobs which are in state SUBMITTED (the at91rm9200ek jobs) and a simple check on target_group shows that the MultiNode test job is not ready to start. That can all be done with a single database query using select_related and other ORM niceties.

Let's describe this with roles:

role:

  client

  device-type: at91rm9200ek

role:

  server

  device-type: laptop

If the at91rm9200ek is in Maintenance and there are no other devices of device-type at91rm9200ek in state Idle, then nothing will get scheduled for at91rm9200ek.

However, when a MultiNode test job is submitted for 1 at91rm9200ek and 1 or more "laptop" device(s), then there is no reason to stop scheduling the laptop device in state Idle without scrabbling through the test job definition and working out (again and again, every time the scheduler loops through the queue) which device-types are requested, which devices of those types are available and what to do next.

The problem is NOT the state of the at91rm9200ek - Maintenance or Bad, it makes no difference. The problem for the SCHEDULER is that the laptop device is Idle and requested by a test job with a sufficiently low submit_time (and high enough Priority) that it is first in the queue.

The problem at the SUBMISSION stage is that the only decision available is whether to allow the test job asking for at91rm9200ek & laptop onto the queue or whether to refuse it outright. Currently, a refusal is only implemented if all devices of at least one device-type specified in the test job are in Retired state.

After many, many rounds of testing, test jobs, discussions going on over several years we came to the decision that in your situation - where there is a dire shortage of a resource used by multiple MultiNode test jobs, that the only thing that was safe for the SCHEDULER to do was to allow the Idle device to be scheduled and let the test job wait for resources to become available, either by moving the other device out of Maintenance or providing extra hardware for the Idle device.

 

> We're looking at what the Maintenance state means for MultiNode in
> https://projects.linaro.org/browse/LAVA-1299 but it is not acceptable to
> refuse submissions when devices are not Retired. Users have an expectation
> that devices which are being fixed will come back online at some point - or
> will go into retired. There is also

I agree.

> https://projects.linaro.org/browse/LAVA-595 but that work has not yet been
> scoped. It could be a long time before that work starts and will take
> months of work once it does start.
>
> The problem is a structural one in the physical resources available in your
> local lab. It is a problem we have faced more than once in our own
> instances and we have gone down all the various routes until we've come to
> the current implementation.
>
>
> >
> > We also work actively on the kernel and thus, we take boards (which we
> > own only once) out of the lab to work on it and then put it into the
> > lab once we've finished working. This is where we put it in Maintenance
> > mode as, IMHO, Retired does not cover this use case.
> > This "Maintenance" can take seconds, days or months.
> >
> > For me, you're ignoring an issue that is almost inexistent in your case
> >
>
> It is an issue which has had months of investigation, discussion and
> intervention in our use cases. We have spent a very long time going through
> all of the permutations.
>

I understand the scheduler is a critical part of the software that had
your attention for a long time and appropriate testing, no doubt.

>
> > because you've dealt with it by adding as much resource as you could to
> > make the probability to happen to be close to zero. That does not mean
> > it does not exist. I'm not criticizing the way to deal with it, I'm just
> > saying this way isn't a path we can take personally.
> >
>
> Then you need to manage the queue on your instance in ways that allow for
> your situation.
>
>
> >
> > >
> > >
> > > > Let me know if I can be of any help debugging this thing or testing a
> > > > possible fix. I'd have a look at the scheduler but you, obviously
> > > > knowing the code base way better than I do, might have a quick patch on
> > > > hand.
> > > >
> > >
> > > Patches would be a bad solution for a structural problem.
> > >
> > > As a different approach, why do you need MultiNode with a "laptop" type
> > > device in the first place? Can the test jobs be reconfigured to use LXC
> > > which does not use MultiNode? What is the "laptop" device-type doing that
> > > cannot be done in an LXC? LXC is created on-the-fly, one for each device,
> > > when the test job requests one. This solved the resource starvation
> > problem
> > > with the majority of MultiNode issues because the work previously done in
> > > the generic QEMU / "laptop" role can just as easily be done in an LXC.
> > >
> >
> > We're testing Gigabit NICs can actually handle Gbps transfers. We need a
> > fully available Gbps NIC for each and every test we do to make the results
> > reliable and consistent.
> >
>
> Then as that resource is limited, you must create a way that only one test
> job of this kind can ever actually run at a time. That can be done by
> working at the stage prior to submission or it can be done by changing the
> device availablity such that the submission is rejected. Critically, there
> must also be a way to prevent jobs entering the queue if one of the
> device-types is not available. That can be easily determined using the
> XML-RPC API prior to submission. Once submitted, LAVA must attempt to run

That's a "lot" of complexity to deal with on our side but that's indeed
a way to do it. I'd have to make sure only one device has a MultiNode
job in queue and monitor it to send the next one.

> the test job as quickly as possible, under the expectation that devices
> which have not been Retired will become available again within a reasonable
> amount of time. If that is not the case then those devices should be
> Retired. (Devices can be brought out of Retired as easily as going in, it
> doesn't have to be a permanent state, nothing is actually deleted from the
> device configuration.)
>

Hum... I'm just wondering. What about a device that was submitted a
MultiNode job but got Retired since then?

Well spotted - that is left to the admins to manage. There is a story outstanding to cancel all test jobs submitted,scheduled or running for devices which are transitioned into Retired.

 

Now I'm wondering what's the difference between Retired and Maintenance
except that it does not accept job submission?

The difference only shows if ALL devices of the device-type are in the specified state.

Retired - no submissions allowed. No test jobs in the queue will be scheduled. Running test jobs will be left to complete. Submitted jobs are currently unchanged if the state changes to Retired.

Maintenance - submissions are allowed. No test jobs in the queue will be scheduled. Running test jobs will be left to complete. Submitted jobs are unchanged if the state changes to Maintenance.

So the only distinction between Retired and Maintenance at the moment is submissions.

 

>
> >
> > > What you are describing sounds like a misuse of MultiNode resulting in
> > > resource starvation and the fix is to have enough of the limited resource
> > > to prevent starvation - either by adding hardware and changing the
> > current
> > > test jobs to use device-tags or relocating the work done on the starved
> > > resource into an LXC so that every device can have a dedicated
> > "container"
> > > to do things which cannot be easily done on the device.
> > >
> >
> > Neither of those options are possible in our use case.
> >
> > I understand the MultiNode scheduler is complex and low priority.
> > We've modestly contributed to LAVA before, we're not telling you to fix
> > it ASAP but rather to help or guide us to fix this issue in a way it
> > could be accepted in the upstream version of LAVA.
> >
> > If you still stand strong against a patch or if it's a lengthy complete
> > rework of the scheduler, could we have at least a way to tell for how
> > long a test have been scheduled (or for how long a board has been
> > reserved for a test that is scheduled)?
>
>
> That data is already available in the current UI and over the XML-RPC API
> and REST API.
>
> Check for Queued Jobs and the scheduling state, also the job_details call
> in XML-RPC. There are a variety of ways of getting the information you
> require using the existing APIs - which one you use will depend on your
> preference and current scripting.
>
> Rather than polling on XML-RPC, it would be better for a monitoring process
> to use ZMQ and the publisher events to get push notifications of change of
> state. That lowers the load on the master, depending on how busy the
> instance actually becomes.
>
>
>
> > That way we can use an external
> > tool to monitor this and manually cancel them when needed. Currently, I
> > don't think there is a way to tell since when the job was scheduled.
> >
>
> Every test job has a database object of submit_time created at the point
> where the job is created upon submission.
>

Submit_time isn't really an option if it's telling what its name is
telling because I can have jobs in queue for days. (Multiple network
tests for each and every board and also time-consuming tests (e.g.
crypto) that have the same priority).

I'll have a look at what you've offered above, thanks.

Thanks for having taken the time to answer my question,

Quentin

--

Neil Williams
=============
neil.williams@linaro.org
http://www.linux.codehelp.co.uk/