Re: [Lava-users] Deadlock in scheduler

26 Apr 2018


      On 25 April 2018 at 11:30, Quentin Schulz quentin.schulz@bootlin.com
wrote:
...
Hi Neil,
On Tue, Apr 24, 2018 at 10:10:32AM +0100, Neil Williams wrote:
...
On 24 April 2018 at 09:08, Quentin Schulz quentin.schulz@bootlin.com
wrote:
...
Hi Neil,
I think there was a global misunderstanding from a poorly choice of
words. When I was saying "available device" I meant a device that isn't
in Maintenance or Retired. If the device is idle, running a job,
scheduled to run a job, etc... I consider it "available". Sorry for the
misunderstanding.
Currently, "Maintenance" is considered as available for submission & for
scheduling. This is to support maximum uptime and minimal disruption to
CI
...
loops for temporary work happening on devices.
We are looking at clarifying this soon.
I know this has been a long and complex thread. Thank you for sticking
with
...
the discussion, despite the complexity and terminology.
Thanks for taking the time to answer those questions, much appreciated.
...
...
On Mon, Apr 23, 2018 at 12:54:03PM +0100, Neil Williams wrote:
...
On 23 April 2018 at 11:21, Quentin Schulz <
quentin.schulz@bootlin.com>
...
...
...
wrote:
...
Hi Neil,
Thanks for your prompt answer.
On Fri, Apr 20, 2018 at 07:56:29AM +0100, Neil Williams wrote:
...
On 19 April 2018 at 20:11, Quentin Schulz <
quentin.schulz@bootlin.com>
...
...
...
wrote:
> Hi all,
>
> I've encountered a deadlock in my LAVA server with the
following
...
...
...
...
scheme.
...
>
I have an at91rm9200ek in my lab that got submitted a lot of
multi-node
...
...
...
> jobs requesting an other "board" (a laptop of type dummy-ssh).
> All of my other boards in the lab have received the same
multi-node
...
...
...
...
jobs
...
> requesting the same and only laptop.
>
That is the source of the resource starvation - multiple
requirements of
...
...
a
...
single device. The scheduler needs to be greedy and grab whatever
suitable
...
devices it can as soon as it can to be able to run MultiNode. The
primary
...
...
...
ordering of scheduling is the Test Job ID which is determined at
submission.
...
Why would you order test jobs without knowing if the boards it
depends
...
...
...
...
on are available when it's going to be scheduled? What am I
missing?
...
...
...
...
To avoid the situation where a MultiNode job is constantly waiting
for
...
...
all
...
devices to be available at exactly the same time. Instances
frequently
...
...
have
...
long queues of submitted test jobs, a mix of single node and
MultiNode.
...
...
The
...
MultiNode jobs must be able to grab whatever device is available, in
order
...
of submit time, and then wait for the other part to be available.
Otherwise, all devices would run all single node test jobs in the
entire
...
...
...
queue before any MultiNode test jobs could start. Many instances
constantly
...
have a queue of single node test jobs.
That's understood and expected.
...
...
...
If you have an imbalance between the number of machines which
can be
...
...
...
...
...
available and then submit MultiNode jobs which all rely on the
starved
...
...
...
resource, there is not much LAVA can do currently. We are
looking at
...
...
a
...
...
way
...
to reschedule MultiNode test jobs but it is very complex and low
priority.
...
What version of lava-server and lava-dispatcher are you running?
lava-dispatcher                   2018.2.post3-1+jessie
lava-server                       2018.2-1+jessie
lava-coordinator                 0.1.7-1
(You need to upgrade to Stretch - there will be no fixes or upgrades
available for Jessie. All development work must only happen on
Stretch.
...
...
See
...
the lava-announce mailing list archive.)
Thanks, we'll have a look into this.
...
...
...
What is the structure of your current lab?
MultiNode is complex - not just at the test job synchronization
level but
...
...
...
also at the lab structure / administrative level.
I have two machines. One acting as LAVA server and one as LAVA
slave.
...
...
...
...
The LAVA slave is handling all boards in our lab.
I have one laptop (an actual x86 laptop for which we know the NIC
driver
...
...
works reliably at high (~1Gbps) speeds) that we use for MultiNode
jobs
...
...
...
...
(actually requesting the laptop and one board at the same time
only) to
...
...
...
...
test network. This laptop is seen as a board by LAVA, there is
nothing
...
...
...
...
LAVA-related on this board (it should be seen as a device).
Then you need more LAVA devices to replicate the role played by the
laptop.
...
Exactly one device for each MultiNode test job which can be
submitted at
...
...
...
any one time. Then use device tags to allocate one of the "laptop"
devices
...
to each of the other boards involved in the MultiNode test jobs.
Alternatively, you need to manage both the submissions and the device
availability.
Think of just the roles played by the devices.
There are N client role devices (not in Retired state) and there are
X
...
...
...
server role devices where the server role is what the laptop is
currently
...
...
...
doing.
You need to have N == X to solve the imbalance in the queue.
If N > 1 (and there are more than one device-type in the count 'N')
then
...
...
...
you also need to use device tags so that each device-type has a
dedicated
...
...
...
pool of server role devices where the number of devices in the server
role
...
pool exactly matches the number of devices of the device-type using
the
...
...
...
specified device tag.
...
...
>
> I had to take the at91rm9200ek out of the lab because it was
behaving.
...
...
...
>
> However, LAVA is still scheduling multi-node jobs on the laptop
which
...
...
...
> requires the at91rm9200ek as the other part of the job, while
its
...
...
...
...
status
...
> is clearly Maintenance.
>
>
A device in Maintenance is still available for scheduling - only
Retired
...
...
is
...
excluded - test jobs submitted to a Retired device are rejected.
Why is that? The device is explicitely in Maintenance, which IMHO
tells
...
...
...
...
that the board shouldn't be used.
Not for the scheduler - the scheduler can still accept submissions
and
...
...
...
queue them up until the device comes out of Maintenance.
This prevents test jobs being rejected during certain kinds of
maintenance.
...
(Wider scale maintenance would involve taking down the UI on the
master
...
...
at
...
which point submissions would get a 404 but that is up to admins to
schedule and announce etc.)
This is about uptime for busy instances which frequently get batches
of
...
...
...
submissions out of operations like cron. The available devices
quickly
...
...
get
...
swamped but the queue needs to continue accepting jobs until admins
decide
...
that devices which need work are going to be unavailable for long
enough
...
...
...
that the resulting queue would be impractical. i.e. when the length
of
...
...
time
...
to wait for the job exceeds the useful window of the results from
the job
...
...
...
or when the number of test jobs in the queue exceeds the ability of
the
...
...
...
available devices to keep on top of the queue and avoid ever
increasing
...
...
...
queues.
I guess that's an implementation choice but I'd have guessed the
scheduler was first looping over idle devices to then schedule the
oldest job in the queue for this device type.
But my understanding is that the scheduler rather sets an order when
jobs are submitted that isn't temperable with. Is that correct?
The order is priority, submit_time and then target_group.
Changing ordering on-the-fly and backing out from certain states is the
subject of https://projects.linaro.org/browse/LAVA-595 - that is the
work
...
I've already described as low priority, large scale and complex.
You do have the ability to set the Priority of new test jobs for
submission
...
which want to use the laptop in a MultiNode test job along with a device
which is NOT the at91rm9200ek. You will need to cancel the test job
involving the at91rm9200ek which is currently scheduled. (Other test jobs
for the at91rm9200ek in the Queue can be left alone provided that these
test jobs have a lower Priority than the jobs you want to run on other
devices.)
When the scheduler comes back around, it will find a new test job with
higher Priority which wants to use the laptop with a hikey device or
whatever and the at91rm9200ek will be ignored. It's not perfect because
you
...
would then need to either keep that Priority pipe full or cancel the
submitted test jobs for the at91rm9200ek.
...
...
...
...
Once a test job has been submitted, it will be either scheduled
or
...
...
...
...
...
cancelled.
Yes, that's understood and that makes sense to me. However, for
"normal"
...
...
jobs, if you can't find a board of device type X that is
available, it
...
...
...
...
does not get scheduled, right? Why can't we do the same for
MultiNode
...
...
...
...
jobs?
Because the MultiNode job will never find all devices in the right
state
...
...
at
...
the same time once there is a mixed queue of single node and
MultiNode
...
...
jobs.
...
All devices defined in the MultiNode test job must be available at
exactly
...
the same time. Once there are single node jobs in the queue, that
never
...
...
...
happens.
A is running
B is Idle
MultiNode submitted for A & B
single node submitted for A
single node submitted for B
scheduler considers the queue - MultiNode cannot start (A is busy),
so
...
...
move
...
on and start the single node job on B (because the single node test
job
...
...
on
...
B may actually complete before the job on A finishes, so it is
inefficient
...
to keep B idle when it could be doing useful stuff for another user).
A is running
B is running
no actions
A completes and goes to Idle
B is still running
and so the pattern continues for as long as there are any single node
test
...
jobs for either A or B in the queue.
The MultiNode test job never starts because A and B are never Idle
at the
...
...
...
same time until the queue is completely empty (which *never* happens
in
...
...
...
many instances).
So the scheduler must grab B while it is Idle to prevent the single
node
...
...
...
test job starting. Then when A completes, the scheduler must also
grab A
...
...
...
before that single node test job starts running.
A is running
B is Idle
MultiNode submitted for A & B
single node submitted for A
single node submitted for B
B is transitioned to Scheduling and is unavailable for the single
node
...
...
test
...
job.
A is running
B is scheduling
no actions
A completes and goes to Idle
B is scheduling
Scheduler transitions A into scheduling - that test job can now
start.
...
...
...
(Now consider MultiNode test jobs covering a dozen devices in an
instance
...
...
...
with a hundred mixed devices and permanent queues of single node test
jobs.)
...
The scheduler also needs to be very fast, so the actual decisions
need to
...
...
...
be made on quite simple criteria - specifically, without going back
to
...
...
the
...
database to find out about what else might be in the queue or trying
to
...
...
...
second-guess when test jobs might end.
That is understood as well for devices that are idle, running or
scheduled to run. The point I was trying to make was why schedule a job
for a device that is in Maintenance (what I meant by the poorly chosen
"available" word).
Is that because one the job is submitted it's ordered by the scheduler
and then run by the scheduler in the given order and the jobs are not
discriminated against the device Maintenance status?
The problem you have is not that the device is in Maintenance but that
the
...
other device(s) in the MultiNode test job are Idle. Therefore, those jobs
get scheduled because there is no reason not to do so.
If the submission was to be rejected when all devices of the requested
device-type are in Maintenance, that is a large change which would
negatively impact a lot of busy instances.
We do need to clarify these states but essentially, Maintenance is a
manual
...
state change which has the same effect as the automatic state change to
Bad. That is as far as it goes currently.
That's where I was missing the last piece of the puzzle. The scheduler
actually only looks at the status of the device it's trying to schedule
a job for and not for all the devices part of this job.
In my mind, scheduling a job for an Idle device that requires an other
board which is in Maintenance was actually "scheduling a job with a
device in Maintenance" which is not what Maintenance was stated to do.
There is a slight but important nuance here for MultiNode jobs that
isn't obvious.
...
...
...
...
...
Now, until I put the at91rm9200ek back in the lab, all my boards
are
...
...
...
...
...
> reserved and scheduling for a multi-node job and thus, my lab
is
...
...
...
...
...
> basically dead.
>
>
The correct fix here is to have enough devices of the
device-type of
...
...
the
...
...
...
starved resource such that one of each other device-type can use
that
...
...
...
...
...
resource simultaneously and then use device-tags to match up
groups
...
...
of
...
...
...
devices so that submitting lots of jobs for one type all at the
same
...
...
time
...
...
...
does not simply consume all of the available resources.
e.g. four device-types - phone, hikey, qemu and panda. Each
multinode job
...
...
...
wants a single QEMU with each of the others, so the QEMU type
becomes
...
...
...
...
...
starved, depending on how jobs are submitted. If two hikey-qemu
jobs
...
...
are
...
...
...
submitted together, then 1 QEMU gets scheduled, waiting for the
hikey to
...
...
...
become free after running the first job. If each QEMU has
device-tags,
...
...
then
...
the second hikey-qemu job will wait not only for the hikey but
will
...
...
also
...
...
...
wait for the one QEMU which has the hikey device tag. This way,
only
...
...
...
...
those
...
jobs would then wait for a QEMU device. There would be three QEMU
devices,
...
one with a device tag like "phone", one with "hikey" and one with
"panda".
...
If another panda device is added, another QEMU with the "panda"
device
...
...
tag
...
would be required. The number of QEMU devices required is the
sum of
...
...
the
...
...
...
number of devices of each other device-type which may be
required in
...
...
a
...
...
...
MultiNode test job.
This is a structural problem within your lab.
You would need one "laptop" for each other device-type which can
use
...
...
that
...
...
...
device-type in your lab. Then each "laptop" gets unique a
device-tag
...
...
.
...
...
Each
...
test job for at91rm9200ek must specify that the "laptop" device
must
...
...
have
...
...
...
the matching device tag. Each test job for each other device-type
uses
...
...
the
...
matching device-tag for that device-type. We had this problem in
the
...
...
...
...
...
Harston lab for a long time when using V1 and had to implement
just
...
...
such
...
...
a
...
structure of matched devices and device tags. However, the need
for
...
...
this
...
...
...
disappeared when the Harston lab transitioned all devices and
test
...
...
jobs
...
...
to
...
LAVA V2.
I strongly disagree with your statement. A software problem can
often
...
...
be
...
...
dealt with by adding more resources but I'm not willing to spend
thousands on something that can be fixed on the software side.
We've been through these loops within the team for many years and
have
...
...
...
millions of test jobs which demonstrate the problems and the fix. I'm
afraid you are misunderstanding the problem if you think that there
is a
...
...
...
software solution for a queue containing both MultiNode and single
node
...
...
...
test jobs - other than the solution we now use in the LAVA
scheduler. The
...
...
...
process has been tried and tested over 8 years and millions of test
jobs
...
...
...
across dozens of mixed use case instances and has proven to be the
most
...
...
...
efficient use of resources across all those models.
Each test job in a MultiNode test is considered separately - if one
or
...
...
more
...
devices are Idle, then those are immediately put into Scheduling.
Only
...
...
when
...
all are in Scheduling can any of those jobs start. The status of
other
...
...
test
...
jobs in the MultiNode group can only be handled at the point when at
least
...
one test job in that MultiNode group is in Scheduling.
I think there is a global misunderstanding due to my bad choice of
words. I understand and I'm convinced there are no other ways to deal
with what you explained above.
...
...
Aside from a non-negligeable financial and time (to setup and
maintain)
...
...
...
...
effort to buy a board with a stable and reliable NIC for each and
every
...
...
...
...
board in my lab, it just isn't our use case.
If I would do such a thing, then my network would be the
bottleneck to
...
...
...
...
my network tests and I'd have to spend a lot (financially and on
time
...
...
or
...
...
maintenance) to have a top notch network infrastructure for tests I
don't care if they run one after the other. I can't have a separate
network for each and every board as well, simply because my boards
often
...
...
have a single Ethernet port, thus I can't separate the test network
from
...
...
the lab network for, e.g. images downloading that are part of the
booting process, hence I can't do reliable network testing even by
multiplying "laptop" devices.
I can understand it's not your typical use case at Linaro and
you've
...
...
...
...
dozens and dozens of the same board and a huge infrastructure to
handle
...
...
...
...
the whole LAVA lab and maybe people working full-time on LAVA, the
lab,
...
...
...
...
the boards, the infrastructure. But that's the complete opposite
of our
...
...
...
...
use case.
Maybe you can understand ours where we have only one board of each
device type, being part of KernelCI to test and report kernel
booting
...
...
...
...
status and having occasional custom tests (like network) on
upstream or
...
...
...
...
custom branches/repositories. We sporadically work on the lab,
fixing
...
...
...
...
the issues we're seeing with the boards but that's not what we do
for a
...
...
...
...
living.
I do understand and I personally run a lab in much the same way.
However,
...
...
...
the code needs to work the same way in that lab as it does in the
larger
...
...
Of course.
...
labs. It is the local configuration and resource availability which
must
...
...
...
change to suit.
For now, the best thing is to put devices into Retired so that
submissions
...
are rejected and then you will also have to manage your submissions
and
...
...
...
your queue.
Can't we have a "Maintenance but I don't know when it's coming back so
please still submit jobs but do not schedule them" option :D ?
This is exactly what you already have - but it's not actually what you
mean.
...
The problem is that you're thinking of the state of the at91rm9200ek when
what matters to the scheduler is the state of the laptop device *in
isolation*. The state of the at91rm9200ek only matters AFTER the laptop
has
...
been assigned.
What you mean is:
"I don't know when ANY of the devices of device-type A are going to be
ready to start a test job, so do not allow ANY OTHER device of ANY OTHER
device-type to be scheduled either IF that device-type is listed in a
MultiNode test job which ALSO requires a device of this device-type".
(The reason it's ANY is that if 10 test jobs are submitted all wanting
at91rm9200ek
and laptop, then if you had 10 laptops and 1 at91rm9200ek, those 10
laptops
...
would also go into Scheduled - that is the imbalance we talked about
previously.)
That's exactly it.
...
It is a cross-relational issue.
Correlating across the MultiNode test job at the scheduling state is
likely
...
to have a massive impact on the speed of the scheduler because:
0: a device in Maintenance or Bad is NOT currently available to be
scheduled - that's the whole point.
1: the other device(s) in the MultiNode group DO get scheduled because
those were in Idle
2: asking the scheduler to check the state of all devices of all
device-types mentioned in a MultiNode job when considering whether to
schedule any other device in that same MultiNode job is going to make the
scheduler SLOW.
So what we do is let the Idle (laptop) device go into a waiting state and
let the scheduler move the at91rm9200ek device into scheduling *only*
when
...
a at91rm9200ek device becomes available for scheduling - as two
completely
...
separate decisions. Then, the relational work is done by the database
when
...
the lava-master makes the query "which test jobs are available to start
NOW". This is much more efficient because we are looking at jobs where
all
...
devices in the target_group are in state SCHEDULED. The database can
easily
...
exclude test jobs which are in state SUBMITTED (the at91rm9200ek jobs)
and
...
a simple check on target_group shows that the MultiNode test job is not
ready to start. That can all be done with a single database query using
select_related and other ORM niceties.
Let's describe this with roles:
role:
  client
  device-type: at91rm9200ek
role:
  server
  device-type: laptop
If the at91rm9200ek is in Maintenance and there are no other devices of
device-type at91rm9200ek in state Idle, then nothing will get scheduled
for
...
at91rm9200ek.
However, when a MultiNode test job is submitted for 1 at91rm9200ek and 1
or
...
more "laptop" device(s), then there is no reason to stop scheduling the
laptop device in state Idle without scrabbling through the test job
definition and working out (again and again, every time the scheduler
loops
...
through the queue) which device-types are requested, which devices of
those
...
types are available and what to do next.
The problem is NOT the state of the at91rm9200ek - Maintenance or Bad, it
makes no difference. The problem for the SCHEDULER is that the laptop
device is Idle and requested by a test job with a sufficiently low
submit_time (and high enough Priority) that it is first in the queue.
The problem at the SUBMISSION stage is that the only decision available
is
...
whether to allow the test job asking for at91rm9200ek & laptop onto the
queue or whether to refuse it outright. Currently, a refusal is only
implemented if all devices of at least one device-type specified in the
test job are in Retired state.
After many, many rounds of testing, test jobs, discussions going on over
several years we came to the decision that in your situation - where
there
...
is a dire shortage of a resource used by multiple MultiNode test jobs,
that
...
the only thing that was safe for the SCHEDULER to do was to allow the
Idle
...
device to be scheduled and let the test job wait for resources to become
available, either by moving the other device out of Maintenance or
providing extra hardware for the Idle device.
Understood, thanks for the full explanation.
...
...
...
We're looking at what the Maintenance state means for MultiNode in
https://projects.linaro.org/browse/LAVA-1299 but it is not
acceptable to
...
...
...
refuse submissions when devices are not Retired. Users have an
expectation
...
that devices which are being fixed will come back online at some
point -
...
...
or
...
will go into retired. There is also
I agree.
...
https://projects.linaro.org/browse/LAVA-595 but that work has not
yet
...
...
been
...
scoped. It could be a long time before that work starts and will take
months of work once it does start.
The problem is a structural one in the physical resources available
in
...
...
your
...
local lab. It is a problem we have faced more than once in our own
instances and we have gone down all the various routes until we've
come
...
...
to
...
the current implementation.
...
We also work actively on the kernel and thus, we take boards
(which we
...
...
...
...
own only once) out of the lab to work on it and then put it into
the
...
...
...
...
lab once we've finished working. This is where we put it in
Maintenance
...
...
...
...
mode as, IMHO, Retired does not cover this use case.
This "Maintenance" can take seconds, days or months.
For me, you're ignoring an issue that is almost inexistent in your
case
...
...
...
...
It is an issue which has had months of investigation, discussion and
intervention in our use cases. We have spent a very long time going
through
...
all of the permutations.
I understand the scheduler is a critical part of the software that had
your attention for a long time and appropriate testing, no doubt.
...
...
because you've dealt with it by adding as much resource as you
could to
...
...
...
...
make the probability to happen to be close to zero. That does not
mean
...
...
...
...
it does not exist. I'm not criticizing the way to deal with it, I'm
just
...
...
saying this way isn't a path we can take personally.
Then you need to manage the queue on your instance in ways that
allow for
...
...
...
your situation.
...
...
> Let me know if I can be of any help debugging this thing or
testing a
...
...
...
> possible fix. I'd have a look at the scheduler but you,
obviously
...
...
...
...
...
> knowing the code base way better than I do, might have a quick
patch on
...
...
...
> hand.
>
Patches would be a bad solution for a structural problem.
As a different approach, why do you need MultiNode with a
"laptop"
...
...
type
...
...
...
device in the first place? Can the test jobs be reconfigured to
use
...
...
LXC
...
...
...
which does not use MultiNode? What is the "laptop" device-type
doing
...
...
that
...
...
...
cannot be done in an LXC? LXC is created on-the-fly, one for each
device,
...
...
...
when the test job requests one. This solved the resource
starvation
...
...
...
...
problem
...
with the majority of MultiNode issues because the work previously
done in
...
...
...
the generic QEMU / "laptop" role can just as easily be done in an
LXC.
...
...
...
We're testing Gigabit NICs can actually handle Gbps transfers. We
need
...
...
a
...
...
fully available Gbps NIC for each and every test we do to make the
results
...
...
reliable and consistent.
Then as that resource is limited, you must create a way that only one
test
...
job of this kind can ever actually run at a time. That can be done by
working at the stage prior to submission or it can be done by
changing
...
...
the
...
device availablity such that the submission is rejected. Critically,
there
...
must also be a way to prevent jobs entering the queue if one of the
device-types is not available. That can be easily determined using
the
...
...
...
XML-RPC API prior to submission. Once submitted, LAVA must attempt
to run
...
...
That's a "lot" of complexity to deal with on our side but that's indeed
a way to do it. I'd have to make sure only one device has a MultiNode
job in queue and monitor it to send the next one.
...
the test job as quickly as possible, under the expectation that
devices
...
...
...
which have not been Retired will become available again within a
reasonable
...
amount of time. If that is not the case then those devices should be
Retired. (Devices can be brought out of Retired as easily as going
in, it
...
...
...
doesn't have to be a permanent state, nothing is actually deleted
from
...
...
the
...
device configuration.)
Hum... I'm just wondering. What about a device that was submitted a
MultiNode job but got Retired since then?
Well spotted - that is left to the admins to manage. There is a story
outstanding to cancel all test jobs submitted,scheduled or running for
devices which are transitioned into Retired.
That's a way to deal with it.
...
...
Now I'm wondering what's the difference between Retired and Maintenance
except that it does not accept job submission?
The difference only shows if ALL devices of the device-type are in the
specified state.
Retired - no submissions allowed. No test jobs in the queue will be
scheduled. Running test jobs will be left to complete. Submitted jobs are
currently unchanged if the state changes to Retired.
Maintenance - submissions are allowed. No test jobs in the queue will be
scheduled. Running test jobs will be left to complete. Submitted jobs are
unchanged if the state changes to Maintenance.
So the only distinction between Retired and Maintenance at the moment is
submissions.
Understood.
I understand in the current implementation of the scheduler, it's too
costly (or at least assumed to be) to check for the status of all
devices of a MultiNode job.
However, could we have something like a timeout for "scheduling" jobs?
e.g. after X minutes/hours after a job has been scheduled for a device,
if that job hasn't been running, move it back to the queue and
"unschedule it" and retry later. That way, the device isn't stuck
forever (well, until a board is put back into Idle mode).
We have considered that, it would have to be part of a wider change because
only the test writer of the *running* test job or the admin of the device
in Maintenance is going to be able to determine how long is "enough".
(Depending on whether the latency is due to the other device(s) being in
state Running or state Maintenance respectively.) We have many jobs in our
labs which (usefully) run for more than a day which is unusual compared to
a lot of labs feeding data to KernelCI. It also means adding data to the
objects in the Queue that the data should be ignored for a configurable
time because once it's back on the queue, it is at or very close to the
start of the queue and the scheduler would likely just put it back straight
into the state it just left. The scheduling timeout may even need to take
into account the total size of the Queue so that the scheduling process
itself does not get bogged down. It's a very tight loop currently and needs
to stay fast.
It all gets very complex, very quickly and that's on top of the current
complexity. (Don't forget, we have to check the VLANd criteria as well at
this one point of assigning devices. MultiNode test jobs are not always
limited to 2 devices either, ten or more is possible and labs can easily
have 50 device types and over 100 devices.)
All of this only applies when there is no practical / affordable way to
solve the resource problem by adding more devices.
So far, that has been the working solution for other instances of this
issue.
Overall, there are so many factors here that it may just turn out to be
something which is best managed by humans. The current workarounds aren't
pretty and could be made smoother. (At each point, the settings need to be
applied to all devices of the relevant device-type which are currently
"available" at the start of the process - so that information needs to be
retained and reinstated at the end.)
0: Set Retired to exclude the device(s) from having more jobs added to the
Queue and exclude existing MultiNode jobs from being scheduled on the other
devices in the group.
1: Submit test jobs to use the Idle device(s) with higher Priority than the
jobs waiting for the Retired device(s).
2: Cancel any scheduled jobs for the Retired device(s) to free up the other
device in the MultiNode group.
3: Return the Retired device(s) to Unknown when it is ready to start
running test jobs again.
As an option:
2a: Set the Retired device to restricted submissions (like the admin group)
temporarily to ensure that when it does come back to Idle it doesn't mangle
the test jobs in the Queue through simple admin errors. Set the owner of
the device(s) and clear the Public checkbox in the Django admin interface.
The lava-server manage CLI can be used for this too.
2b: Once admin test jobs have run successfully, restore the Public setting
to run the jobs in the Queue and ensure the health is set to Good (as
presumably the admin will at least have run a health check).
...
Quentin
...
...
...
...
...
What you are describing sounds like a misuse of MultiNode
resulting
...
...
in
...
...
...
resource starvation and the fix is to have enough of the limited
resource
...
...
...
to prevent starvation - either by adding hardware and changing
the
...
...
...
...
current
...
test jobs to use device-tags or relocating the work done on the
starved
...
...
...
resource into an LXC so that every device can have a dedicated
"container"
...
to do things which cannot be easily done on the device.
Neither of those options are possible in our use case.
I understand the MultiNode scheduler is complex and low priority.
We've modestly contributed to LAVA before, we're not telling you
to fix
...
...
...
...
it ASAP but rather to help or guide us to fix this issue in a way
it
...
...
...
...
could be accepted in the upstream version of LAVA.
If you still stand strong against a patch or if it's a lengthy
complete
...
...
...
...
rework of the scheduler, could we have at least a way to tell for
how
...
...
...
...
long a test have been scheduled (or for how long a board has been
reserved for a test that is scheduled)?
That data is already available in the current UI and over the
XML-RPC API
...
...
...
and REST API.
Check for Queued Jobs and the scheduling state, also the job_details
call
...
...
...
in XML-RPC. There are a variety of ways of getting the information
you
...
...
...
require using the existing APIs - which one you use will depend on
your
...
...
...
preference and current scripting.
Rather than polling on XML-RPC, it would be better for a monitoring
process
...
to use ZMQ and the publisher events to get push notifications of
change
...
...
of
...
state. That lowers the load on the master, depending on how busy the
instance actually becomes.
...
That way we can use an external
tool to monitor this and manually cancel them when needed.
Currently, I
...
...
...
...
don't think there is a way to tell since when the job was
scheduled.
...
...
...
...
Every test job has a database object of submit_time created at the
point
...
...
...
where the job is created upon submission.
Submit_time isn't really an option if it's telling what its name is
telling because I can have jobs in queue for days. (Multiple network
tests for each and every board and also time-consuming tests (e.g.
crypto) that have the same priority).
I'll have a look at what you've offered above, thanks.
Thanks for having taken the time to answer my question,
Quentin
--
Neil Williams
neil.williams@linaro.org
http://www.linux.codehelp.co.uk/
-- 

Neil Williams
=============
neil.williams@linaro.org
http://www.linux.codehelp.co.uk/

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

Re: [Lava-users] Deadlock in scheduler

Neil Williams