Dear Sir/Madam,
I'm new to LAVA. I'm still in initial stage to use lava in our farm. I use customized board with Qualcomm's chip.
Recently I met a question, a very simple job, sometimes when I start to operate uboot, the serial have chance to be closed. They I set retry in lava job, let it retry a lots of times, but looks never successfully open the serial again.
But, after job finish, I quickly open the serial in manual, it's ok!
I search your code a lot, and found in shell.py there is one comments, I want to know is this the reason that I failed to retry for serial? When will you fix it?
*# FIXME: deliberately closing the connection (and starting a new one) needs to be supported.*
*Cheers,*
*Peter*
Peter,
On Mon, 18 May 2020 at 09:52, Peter Lucas peterlucas073@gmail.com wrote:
Dear Sir/Madam,
I'm new to LAVA. I'm still in initial stage to use lava in our farm. I use customized board with Qualcomm's chip.
Recently I met a question, a very simple job, sometimes when I start to operate uboot, the serial have chance to be closed. They I set retry in lava job, let it retry a lots of times, but looks never successfully open the serial again.
I don't quite understand your use case. Do you manually close the serial connection? Could you describe your setup (hardware connections) in a bit more detail?
milosz
But, after job finish, I quickly open the serial in manual, it's ok!
I search your code a lot, and found in shell.py there is one comments, I want to know is this the reason that I failed to retry for serial? When will you fix it?
# FIXME: deliberately closing the connection (and starting a new one) needs to be supported.
Cheers, Peter _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Hello, Milosz
It's a very simple hardware connection, for example, the serial port is "/dev/ttyUSB0", I set "telnet xxx 9999" in device, then I use uboot/nfs in lava job.
1. Sometimes when I submit my job to lava, it's just ok.
2. But sometimes when I submit my job to lava, in uboot operation, for example, "tftpboot xxx image", it will tells me "Connection closed", I don't know why. But when I submit a job to lava at once, then it's ok this time!
3. I guess some issue with something, I'm not sure it's lava issue or my board issue. Then I set a failure_retry: 10, to let lava retry it for me. Strange, never success. But if manual retry it in hand, it's high probability that I could retry successfully.
On Mon, May 18, 2020 at 4:56 PM Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
Peter,
On Mon, 18 May 2020 at 09:52, Peter Lucas peterlucas073@gmail.com wrote:
Dear Sir/Madam,
I'm new to LAVA. I'm still in initial stage to use lava in our farm. I
use customized board with Qualcomm's chip.
Recently I met a question, a very simple job, sometimes when I start to
operate uboot, the serial have chance to be closed. They I set retry in lava job, let it retry a lots of times, but looks never successfully open the serial again.
I don't quite understand your use case. Do you manually close the serial connection? Could you describe your setup (hardware connections) in a bit more detail?
milosz
But, after job finish, I quickly open the serial in manual, it's ok!
I search your code a lot, and found in shell.py there is one comments, I
want to know is this the reason that I failed to retry for serial? When will you fix it?
# FIXME: deliberately closing the connection (and starting a new one)
needs to be supported.
Cheers, Peter _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
On Mon, 18 May 2020 at 10:08, Peter Lucas peterlucas073@gmail.com wrote:
Hello, Milosz
It's a very simple hardware connection, for example, the serial port is "/dev/ttyUSB0", I set "telnet xxx 9999" in device, then I use uboot/nfs in lava job.
do you run ser2net on your worker? How does the setting for ser2net look like?
Sometimes when I submit my job to lava, it's just ok.
But sometimes when I submit my job to lava, in uboot operation, for example, "tftpboot xxx image", it will tells me "Connection closed", I don't know why. But when I submit a job to lava at once, then it's ok this time!
'connection closed' is sometimes hard to track. For example in our LAB these errors (at least some of them) were caused by poor quality cables. So it might be worth looking at your hardware.
milosz
- I guess some issue with something, I'm not sure it's lava issue or my board issue. Then I set a failure_retry: 10, to let lava retry it for me. Strange, never success. But if manual retry it in hand, it's high probability that I could retry successfully.
On Mon, May 18, 2020 at 4:56 PM Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Peter,
On Mon, 18 May 2020 at 09:52, Peter Lucas peterlucas073@gmail.com wrote:
Dear Sir/Madam,
I'm new to LAVA. I'm still in initial stage to use lava in our farm. I use customized board with Qualcomm's chip.
Recently I met a question, a very simple job, sometimes when I start to operate uboot, the serial have chance to be closed. They I set retry in lava job, let it retry a lots of times, but looks never successfully open the serial again.
I don't quite understand your use case. Do you manually close the serial connection? Could you describe your setup (hardware connections) in a bit more detail?
milosz
But, after job finish, I quickly open the serial in manual, it's ok!
I search your code a lot, and found in shell.py there is one comments, I want to know is this the reason that I failed to retry for serial? When will you fix it?
# FIXME: deliberately closing the connection (and starting a new one) needs to be supported.
Cheers, Peter _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Yes, I use ser2net, like next:
9999:telnet:0:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT
If do it manual, I never see any issue. But if do it in lava, although not always, I still have chance to see the connection closed. Maybe 1 times for every 100 tries.
As I said, after this happen, I retry open serial port in manual, it's ok. Or I just resubmit a job again to lava, it's ok. But from time to time, there will be 1 times "connection closed", I'm lost why it only happen in LAVA? Anyway I can workaround it? I replaced several cables, still sometimes saw it.
On Mon, May 18, 2020 at 5:12 PM Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Mon, 18 May 2020 at 10:08, Peter Lucas peterlucas073@gmail.com wrote:
Hello, Milosz
It's a very simple hardware connection, for example, the serial port is
"/dev/ttyUSB0", I set "telnet xxx 9999" in device, then I use uboot/nfs in lava job.
do you run ser2net on your worker? How does the setting for ser2net look like?
Sometimes when I submit my job to lava, it's just ok.
But sometimes when I submit my job to lava, in uboot operation, for
example, "tftpboot xxx image", it will tells me "Connection closed", I don't know why. But when I submit a job to lava at once, then it's ok this time!
'connection closed' is sometimes hard to track. For example in our LAB these errors (at least some of them) were caused by poor quality cables. So it might be worth looking at your hardware.
milosz
- I guess some issue with something, I'm not sure it's lava issue or my
board issue. Then I set a failure_retry: 10, to let lava retry it for me. Strange, never success. But if manual retry it in hand, it's high probability that I could retry successfully.
On Mon, May 18, 2020 at 4:56 PM Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
Peter,
On Mon, 18 May 2020 at 09:52, Peter Lucas peterlucas073@gmail.com
wrote:
Dear Sir/Madam,
I'm new to LAVA. I'm still in initial stage to use lava in our farm.
I use customized board with Qualcomm's chip.
Recently I met a question, a very simple job, sometimes when I start
to operate uboot, the serial have chance to be closed. They I set retry in lava job, let it retry a lots of times, but looks never successfully open the serial again.
I don't quite understand your use case. Do you manually close the serial connection? Could you describe your setup (hardware connections) in a bit more detail?
milosz
But, after job finish, I quickly open the serial in manual, it's ok!
I search your code a lot, and found in shell.py there is one
comments, I want to know is this the reason that I failed to retry for serial? When will you fix it?
# FIXME: deliberately closing the connection (and starting a new one)
needs to be supported.
Cheers, Peter _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
On Mon, 18 May 2020 at 10:18, Peter Lucas peterlucas073@gmail.com wrote:
Yes, I use ser2net, like next:
9999:telnet:0:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT
If do it manual, I never see any issue. But if do it in lava, although not always, I still have chance to see the connection closed. Maybe 1 times for every 100 tries.
As I said, after this happen, I retry open serial port in manual, it's ok. Or I just resubmit a job again to lava, it's ok. But from time to time, there will be 1 times "connection closed", I'm lost why it only happen in LAVA? Anyway I can workaround it? I replaced several cables, still sometimes saw it.
It looks like your experience is similar to our LAB. We also see connection closed sometimes and it's really hard to tell why. We have slightly lower frequency - usually below 0.5% but it still happens. I can only ask to report the reason if you find it. I believe the reason is usually short enough that you're not able to tell manually after the fact. The only outcome is broken connection. After that you can connect again after a very short amount of time. So it will be pretty hard to find manually. You can try to enable saving ser2net logs to the file. Maybe this helps?
milosz
On Mon, May 18, 2020 at 5:12 PM Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
On Mon, 18 May 2020 at 10:08, Peter Lucas peterlucas073@gmail.com wrote:
Hello, Milosz
It's a very simple hardware connection, for example, the serial port is "/dev/ttyUSB0", I set "telnet xxx 9999" in device, then I use uboot/nfs in lava job.
do you run ser2net on your worker? How does the setting for ser2net look like?
Sometimes when I submit my job to lava, it's just ok.
But sometimes when I submit my job to lava, in uboot operation, for example, "tftpboot xxx image", it will tells me "Connection closed", I don't know why. But when I submit a job to lava at once, then it's ok this time!
'connection closed' is sometimes hard to track. For example in our LAB these errors (at least some of them) were caused by poor quality cables. So it might be worth looking at your hardware.
milosz
- I guess some issue with something, I'm not sure it's lava issue or my board issue. Then I set a failure_retry: 10, to let lava retry it for me. Strange, never success. But if manual retry it in hand, it's high probability that I could retry successfully.
On Mon, May 18, 2020 at 4:56 PM Milosz Wasilewski milosz.wasilewski@linaro.org wrote:
Peter,
On Mon, 18 May 2020 at 09:52, Peter Lucas peterlucas073@gmail.com wrote:
Dear Sir/Madam,
I'm new to LAVA. I'm still in initial stage to use lava in our farm. I use customized board with Qualcomm's chip.
Recently I met a question, a very simple job, sometimes when I start to operate uboot, the serial have chance to be closed. They I set retry in lava job, let it retry a lots of times, but looks never successfully open the serial again.
I don't quite understand your use case. Do you manually close the serial connection? Could you describe your setup (hardware connections) in a bit more detail?
milosz
But, after job finish, I quickly open the serial in manual, it's ok!
I search your code a lot, and found in shell.py there is one comments, I want to know is this the reason that I failed to retry for serial? When will you fix it?
# FIXME: deliberately closing the connection (and starting a new one) needs to be supported.
Cheers, Peter _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
Ok, if LAVA can't handle it currently with a re-connect, we choose to tolerate it, thanks for the information.
On Mon, May 18, 2020 at 7:24 PM Milosz Wasilewski < milosz.wasilewski@linaro.org> wrote:
On Mon, 18 May 2020 at 10:18, Peter Lucas peterlucas073@gmail.com wrote:
Yes, I use ser2net, like next:
9999:telnet:0:/dev/ttyUSB0:115200 8DATABITS NONE 1STOPBIT
If do it manual, I never see any issue. But if do it in lava, although
not always, I still have chance to see the connection closed. Maybe 1 times for every 100 tries.
As I said, after this happen, I retry open serial port in manual, it's
ok. Or I just resubmit a job again to lava, it's ok.
But from time to time, there will be 1 times "connection closed", I'm
lost why it only happen in LAVA? Anyway I can workaround it?
I replaced several cables, still sometimes saw it.
It looks like your experience is similar to our LAB. We also see connection closed sometimes and it's really hard to tell why. We have slightly lower frequency - usually below 0.5% but it still happens. I can only ask to report the reason if you find it. I believe the reason is usually short enough that you're not able to tell manually after the fact. The only outcome is broken connection. After that you can connect again after a very short amount of time. So it will be pretty hard to find manually. You can try to enable saving ser2net logs to the file. Maybe this helps?
milosz
On Mon, May 18, 2020 at 5:12 PM Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
On Mon, 18 May 2020 at 10:08, Peter Lucas peterlucas073@gmail.com
wrote:
Hello, Milosz
It's a very simple hardware connection, for example, the serial port
is "/dev/ttyUSB0", I set "telnet xxx 9999" in device, then I use uboot/nfs in lava job.
do you run ser2net on your worker? How does the setting for ser2net
look like?
Sometimes when I submit my job to lava, it's just ok.
But sometimes when I submit my job to lava, in uboot operation,
for example, "tftpboot xxx image", it will tells me "Connection closed", I don't know why. But when I submit a job to lava at once, then it's ok this time!
'connection closed' is sometimes hard to track. For example in our LAB these errors (at least some of them) were caused by poor quality cables. So it might be worth looking at your hardware.
milosz
- I guess some issue with something, I'm not sure it's lava issue or
my board issue. Then I set a failure_retry: 10, to let lava retry it for me. Strange, never success. But if manual retry it in hand, it's high probability that I could retry successfully.
On Mon, May 18, 2020 at 4:56 PM Milosz Wasilewski <
milosz.wasilewski@linaro.org> wrote:
Peter,
On Mon, 18 May 2020 at 09:52, Peter Lucas peterlucas073@gmail.com
wrote:
Dear Sir/Madam,
I'm new to LAVA. I'm still in initial stage to use lava in our
farm. I use customized board with Qualcomm's chip.
Recently I met a question, a very simple job, sometimes when I
start to operate uboot, the serial have chance to be closed. They I set retry in lava job, let it retry a lots of times, but looks never successfully open the serial again.
I don't quite understand your use case. Do you manually close the serial connection? Could you describe your setup (hardware connections) in a bit more detail?
milosz
But, after job finish, I quickly open the serial in manual, it's
ok!
I search your code a lot, and found in shell.py there is one
comments, I want to know is this the reason that I failed to retry for serial? When will you fix it?
# FIXME: deliberately closing the connection (and starting a new
one) needs to be supported.
Cheers, Peter _______________________________________________ Lava-users mailing list Lava-users@lists.lavasoftware.org https://lists.lavasoftware.org/mailman/listinfo/lava-users
lava-users@lists.lavasoftware.org
-
Milosz Wasilewski -
Peter Lucas