Hi,
I am trying to use transfer_overlay as the fs on my DUT is read-only, but the /data/ and /tmp/ directories are writeable to some extent. I have a working method to use wget, and it works when I use it manually to download the overlay tarball over http, both on my DUT and on the worker device. However, the LAVA test itself always returns the error 'Network Unreachable' What are the possible reasons for this?
Best regards, Michael
Hi,
I have reached a (non-preferred) solution. I modified the source code in /usr/lib/python3/dist-packages/lava_dispatcher/actions/boot/__init__.py under the http if section (approx. line 717 onwards) to have actual character delay instead of using the class property character_delay (which when I looked for it on git online it was simply 0). I also split up the download command I submitted through the test job as I had quite a unique one due to the circumstances explained already.
If anyone has suggestions that don't require me to modify the code, please let me know here. Otherwise, feel free to ask questions for particulars (not that I can be of much use, I just experimented heavily despite my inexperience with LAVA).
Best regards, Michael
Hello,
Le lun. 26 févr. 2024 à 04:46, Michael Peddie michael.peddie@gallagher.com a écrit :
Hi,
I am trying to use transfer_overlay as the fs on my DUT is read-only, but the /data/ and /tmp/ directories are writeable to some extent. I have a working method to use wget, and it works when I use it manually to download the overlay tarball over http, both on my DUT and on the worker device. However, the LAVA test itself always returns the error 'Network Unreachable'
Is the network up-and-running when LAVA tries to download the overlay? You can prepend the transfer_overlay command by some code to wait for systemd to reach network-up target (don't remember the exact name).
Rgds
What are the possible reasons for this?
Best regards, Michael _______________________________________________ lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
At the time of the original post, no it was not, but by the time of my update I did figure that much out and fixed that issue. However I still would get that problem from time to time, or (for unknown reasons) I would get a 404 error. The problem I was facing afterwards (and still am) was instead that telnet (or LAVA, I am unsure which is the issue here) sends the commands over serial on top of one another, or on top of the output of a previous command. This sometimes (not always) causes them not to work, and thus requires intervention or the re-run of the test. Another issue is that just sometimes characters are missing, or completely rearranged within the submitted command. With some modifications of the file mentioned in my update (posted on 05/03), and some character delay option setting in the ser2net config file, I have increased the success rate from the initial 24% to 32%, which is still far from what I want.
I am unsure what needs to be done to get this working properly, and have been considering trying a multinode set up instead. I don't know if this would fix any of the issues though.
Best regards, Michael
Hi Michael,
this reminds me of the U-Boot problem I had: https://lists.lavasoftware.org/archives/list/lava-users@lists.lavasoftware.o...
I also first worked around this with an (unreasonably) high character delay. Maybe this helps you in tracking down the problem?
Best regards, Stefan
On 2024-03-18 02:13, Michael Peddie wrote:
At the time of the original post, no it was not, but by the time of my update I did figure that much out and fixed that issue. However I still would get that problem from time to time, or (for unknown reasons) I would get a 404 error. The problem I was facing afterwards (and still am) was instead that telnet (or LAVA, I am unsure which is the issue here) sends the commands over serial on top of one another, or on top of the output of a previous command. This sometimes (not always) causes them not to work, and thus requires intervention or the re-run of the test. Another issue is that just sometimes characters are missing, or completely rearranged within the submitted command. With some modifications of the file mentioned in my update (posted on 05/03), and some character delay option setting in the ser2net config file, I have increased the success rate from the initial 24% to 32%, which is still far from what I want.
I am unsure what needs to be done to get this working properly, and have been considering trying a multinode set up instead. I don't know if this would fix any of the issues though.
Best regards, Michael _______________________________________________ lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
Hi Stefan,
Thank you for showing me that post. It has more or less confirmed an idea I had as to why it may not be working. I had suspected it may be lava matching to the previous prompt instead of the current, which at the time seemed both plausible and yet ridiculous, but it is the best explanation for overlapping commands. The particular delay I found most effective until now wasn't as crazy as 100ms, but several delays for commands at approximately 5ms each.
There is not much I can do to change the test or device configuration for the prompt, as my device simply uses '#' as the prompt for commands. There is something I have noticed though, that seems unique to telnet, which is that there is some kind of an "echo", where there is an empty prompt after every command execution. minicom does not have this, nor does ssh. Iirc, this can be disabled in interactive tests. Is there a way to disable this for regular tests or in general? I am not sure it will be effective at solving the issue, but it is worth a shot.
If you want to see logs, I will paste the part of the plain log here where the failure happens: - {"dt": "2024-03-24T20:37:41.827319", "lvl": "debug", "msg": "Setting prompt string to ['# ']"} - {"dt": "2024-03-24T20:37:41.827995", "lvl": "debug", "msg": "Forcing a shell prompt, looking for ['# ']"} - {"dt": "2024-03-24T20:37:41.828663", "lvl": "input", "msg": "\n"} - {"dt": "2024-03-24T20:37:41.879484", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:37:41.880437", "lvl": "debug", "msg": "expect-shell-connection: Wait for prompt ['# '] (timeout 00:05:00)"} - {"dt": "2024-03-24T20:37:41.881144", "lvl": "debug", "msg": "Waiting using forced prompt support (timeout 00:02:30)"} - {"dt": "2024-03-24T20:37:41.881966", "lvl": "target", "msg": ""} - {"dt": "2024-03-24T20:37:41.901778", "lvl": "target", "msg": ""} - {"dt": "2024-03-24T20:37:41.902572", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:37:41.903598", "lvl": "debug", "msg": "end: 2.4 expect-shell-connection (duration 00:00:00) [common]"} - {"dt": "2024-03-24T20:37:41.904420", "lvl": "debug", "msg": "start: 2.5 overlay-unpack (timeout 00:03:38) [common]"} - {"dt": "2024-03-24T20:37:41.905174", "lvl": "input", "msg": "rm overlay-1.4.2.5.tar.gz\n"} - {"dt": "2024-03-24T20:37:42.006073", "lvl": "target", "msg": "# rm overlay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:37:42.008397", "lvl": "debug", "msg": "Sending with 100 millisecond of delay"} - {"dt": "2024-03-24T20:37:42.009156", "lvl": "input", "msg": "cd / ; tar -C /tmp/ -xzf /data/wgetcurl.tar.gz ; cd /tmp/ ; sleep 2\n"} - {"dt": "2024-03-24T20:37:52.131475", "lvl": "target", "msg": "cd / ; tar -C /tmp/ -xzf /data/wgetcurl.tar.gz ; cd /tmp/ ; sleep 2"} - {"dt": "2024-03-24T20:37:52.233013", "lvl": "target", "msg": "rm overlay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:37:52.234357", "lvl": "target", "msg": "rm overlay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:37:52.235107", "lvl": "target", "msg": "crm: can't remove 'overlay-1.4.2.5.tar.gz': No such file or directory"} - {"dt": "2024-03-24T20:37:52.236086", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:37:52.236652", "lvl": "target", "msg": "# d / ; tar -C /tmp/ -xzf /data/wgetcurl.tar.gz ; cd /tmp/ ; sleep 2"} - {"dt": "2024-03-24T20:37:52.237197", "lvl": "target", "msg": "cd / ; tar -C /tmp/ -xzf /data/wgetcurl.tar.gz ; cd /tmp/ ; sleep 2"} - {"dt": "2024-03-24T20:37:52.238553", "lvl": "debug", "msg": "Sending with 100 millisecond of delay"} - {"dt": "2024-03-24T20:37:52.239190", "lvl": "input", "msg": "/tmp/wgetcurl/wget http://10.60.2.209/tmp/505/compress-overlay-mubx369r/overlay-1.4.2.5.tar.gz%... - {"dt": "2024-03-24T20:38:06.419274", "lvl": "target", "msg": "/tmp/wgetcurl/wget http://10.60.2.209/tmp/505/compress-overlay-mubx369r/overlay-1.4.2.5.tar.gz%... - {"dt": "2024-03-24T20:38:06.522664", "lvl": "debug", "msg": "Sending with 100 millisecond of delay"} - {"dt": "2024-03-24T20:38:06.523485", "lvl": "input", "msg": "tar -C /tmp -xzf overlay-1.4.2.5.tar.gz\n"} - {"dt": "2024-03-24T20:38:12.435400", "lvl": "target", "msg": "tar -C /tmp -xzf overlay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:38:12.536924", "lvl": "target", "msg": "/tmp/wgetcurl/wget http://10.6# "} - {"dt": "2024-03-24T20:38:12.537783", "lvl": "target", "msg": "# 0.2.209/tmp/505/compress-overlay-mubx369r/overlay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:38:12.538566", "lvl": "target", "msg": "/t[ 70.070000] ttyS ttyS0: 1 input overrun(s)"} - {"dt": "2024-03-24T20:38:12.539238", "lvl": "target", "msg": "mp/wgetcurl/wget http:/10.60ay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:38:12.539887", "lvl": "target", "msg": "ta--2024-03-24 20:37:47-- ftp://http//10.60ay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:38:12.540534", "lvl": "target", "msg": " => '10.60ay-1.4.2.5.tar.gz'"} - {"dt": "2024-03-24T20:38:12.541168", "lvl": "target", "msg": "Resolving http... r -C /tmp -xzf overlay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:38:12.541804", "lvl": "target", "msg": "tar -C /tmp -xzf overlay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:38:12.542495", "lvl": "target", "msg": ""} - {"dt": "2024-03-24T20:38:12.543693", "lvl": "debug", "msg": "end: 2.5 overlay-unpack (duration 00:00:31) [common]"} - {"dt": "2024-03-24T20:38:12.544612", "lvl": "debug", "msg": "start: 2.6 export-device-env (timeout 00:03:08) [common]"} - {"dt": "2024-03-24T20:38:12.545549", "lvl": "debug", "msg": "end: 2.6 export-device-env (duration 00:00:00) [common]"} - {"dt": "2024-03-24T20:38:12.546567", "lvl": "info", "msg": "end: 2 minimal-boot (duration 00:01:52) [common]"} - {"dt": "2024-03-24T20:38:12.547476", "lvl": "info", "msg": "start: 3 lava-test-retry (timeout 00:05:00) [common]"} - {"dt": "2024-03-24T20:38:12.548381", "lvl": "debug", "msg": "start: 3.1 lava-test-shell (timeout 00:05:00) [common]"} - {"dt": "2024-03-24T20:38:12.549129", "lvl": "debug", "msg": "Using namespace: common"} - {"dt": "2024-03-24T20:38:12.549894", "lvl": "input", "msg": "#\n"} - {"dt": "2024-03-24T20:38:12.650937", "lvl": "target", "msg": "#"} - {"dt": "2024-03-24T20:38:12.652301", "lvl": "debug", "msg": "lava-test-shell: Wait for prompt ['# '] (timeout 00:05:00)"} - {"dt": "2024-03-24T20:38:12.653278", "lvl": "debug", "msg": "Using /tmp/lava-505"} - {"dt": "2024-03-24T20:38:12.653913", "lvl": "input", "msg": "export SHELL=/bin/sh\n"} - {"dt": "2024-03-24T20:38:12.755166", "lvl": "target", "msg": "export SHELL=/bin/sh"} - {"dt": "2024-03-24T20:38:12.756527", "lvl": "target", "msg": "#"} - {"dt": "2024-03-24T20:38:12.757552", "lvl": "target", "msg": "#"} - {"dt": "2024-03-24T20:38:12.758430", "lvl": "target", "msg": ""} - {"dt": "2024-03-24T20:38:12.759326", "lvl": "target", "msg": "export SHELL=/bin/sh"} - {"dt": "2024-03-24T20:38:12.785528", "lvl": "target", "msg": "export SHELL=/bin/sh"} - {"dt": "2024-03-24T20:38:12.786460", "lvl": "target", "msg": ""} - {"dt": "2024-03-24T20:38:16.893918", "lvl": "target", "msg": "failed: Temporary failure in name resolution."} - {"dt": "2024-03-24T20:38:16.894892", "lvl": "target", "msg": "wget: unable to resolve host address 'http'"} - {"dt": "2024-03-24T20:38:16.958057", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:38:16.959242", "lvl": "input", "msg": ". /tmp/lava-505/environment\n"} - {"dt": "2024-03-24T20:38:17.060570", "lvl": "target", "msg": "# tar -C /tmp -xzf o. /tmp/lava-505/environment"} - {"dt": "2024-03-24T20:38:17.061874", "lvl": "input", "msg": "/tmp/lava-505/bin/lava-test-runner /tmp/lava-505/0\n"} - {"dt": "2024-03-24T20:38:17.163565", "lvl": "target", "msg": "/tmp/lava-505/bin/lava-test-runner /tmp/lava-505/0"} - {"dt": "2024-03-24T20:38:17.164670", "lvl": "info", "msg": "Test shell timeout: 10s (minimum of the action and connection timeout)"} - {"dt": "2024-03-24T20:38:17.167644", "lvl": "target", "msg": "verlay-1.4.2.5.tar.gz"} - {"dt": "2024-03-24T20:38:17.168419", "lvl": "target", "msg": ". /tmp/lava-505/environmenttar: can't open 'overlay-1.4.2.5.tar.gz': No such file or directory"} - {"dt": "2024-03-24T20:38:17.169109", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:38:17.169753", "lvl": "target", "msg": "# #"} - {"dt": "2024-03-24T20:38:17.170442", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:38:17.171078", "lvl": "target", "msg": "# export SHELL=/bin/sh"} - {"dt": "2024-03-24T20:38:17.171719", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:38:17.172355", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:38:17.172989", "lvl": "target", "msg": ". /tmp/la/tmp/lava-505/bin/lava-test-runner /tmp/lava-505/0va-505/environment"} - {"dt": "2024-03-24T20:38:17.173633", "lvl": "target", "msg": "-sh: .: can't open '/tmp/lava-505/environment': No such file or directory"} - {"dt": "2024-03-24T20:38:17.174326", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:38:17.174951", "lvl": "target", "msg": "# "} - {"dt": "2024-03-24T20:38:17.207624", "lvl": "target", "msg": "/tmp/lava-505/bin/lava-test-runner /tmp/lava-505/0"} - {"dt": "2024-03-24T20:38:17.223874", "lvl": "target", "msg": "-sh: /tmp/lava-505/bin/lava-test-runner: not found"} - {"dt": "2024-03-24T20:38:17.239721", "lvl": "target", "msg": "# "}
After that I cancelled the test as I didn't want or need to wait for it to time out, since intervention would be required to get the test to finish, and I am just trying to get it working. As of this reply that is the latest test I have run. If there is anything else needed to help diagnose the issue and come up with a solution, please let me know so I can do my best to provide more information.
Best regards, Michael
Some looking into telnet says that in regards to the CRLF setting, it will use only CR if that is what it receives, but if it gets an LF, it will use CRLF instead regardless of if there is a CR preceding. I am not sure if this has any impact on LAVA and it's ability to send commands in their entirety (as I have had issues with missing characters, out of order commands/characters, miss-timed commands, and others not covered by those).
Hi Michael,
same there... if you use '# ' as prompt on your device, then use '# $' as prompt string in LAVA.
I had the same prompt, but I do change the default prompt string through a LAVA overlay, which puts a '/etc/profile.d/prompt.sh' to my device which has 'export PS1="\u@\h:\w$ "' as content - thus I have the "usual" prompt which cannot be confused so easily.
Best regards
Stefan
On 2024-03-24 22:02, Michael Peddie wrote:
Hi Stefan,
Thank you for showing me that post. It has more or less confirmed an idea I had as to why it may not be working. I had suspected it may be lava matching to the previous prompt instead of the current, which at the time seemed both plausible and yet ridiculous, but it is the best explanation for overlapping commands. The particular delay I found most effective until now wasn't as crazy as 100ms, but several delays for commands at approximately 5ms each.
There is not much I can do to change the test or device configuration for the prompt, as my device simply uses '#' as the prompt for commands. There is something I have noticed though, that seems unique to telnet, which is that there is some kind of an "echo", where there is an empty prompt after every command execution. minicom does not have this, nor does ssh. Iirc, this can be disabled in interactive tests. Is there a way to disable this for regular tests or in general? I am not sure it will be effective at solving the issue, but it is worth a shot.
If you want to see logs, I will paste the part of the plain log here where the failure happens: [...]
After that I cancelled the test as I didn't want or need to wait for it to time out, since intervention would be required to get the test to finish, and I am just trying to get it working. As of this reply that is the latest test I have run. If there is anything else needed to help diagnose the issue and come up with a solution, please let me know so I can do my best to provide more information.
Best regards, Michael _______________________________________________ lava-users mailing list -- lava-users@lists.lavasoftware.org To unsubscribe send an email to lava-users-leave@lists.lavasoftware.org %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s
Hi Stefan,
Thank you. I have since tried that, and it seems to have fixed the commands running on top of each other issue, but there are still other issues present that I can't seem to fix. There must be some issue with telnet directly, as during manual testing without LAVA, I ran some simple commands to check file contents on my DUT, and for no reason there were characters in the command that just did not get sent over and caused the command to fail, as it was no longer a valid command (for example, "cat" became "ct"). It's possible it's an issue with ser2net, but I have no idea.
Though I don't expect to be able to get much help on it, I did try using minicom as an alternative to telnet, as I found it never gave that issue (during manual testing, similar to what I mentioned before with telnet), I ran into the problem of "No cursor motion capability (cm)", and nothing I did fixed it. If there is a work around or proper solution for this please let me know.
Best regards, Michael
lava-users@lists.lavasoftware.org
-
Michael Peddie -
Remi Duraffort -
Stefan