On 25 April 2016 at 15:22, Neil Williams neil.williams@linaro.org wrote:
On 25 April 2016 at 14:56, Daniel Thompson daniel.thompson@linaro.org wrote:
On 25/04/16 14:14, Dave Pigott wrote:
I’ve compared the “working” and “non-working” kernel logs and there are only two differences I can find.
- In the non working one it has to run an fsck:
https://validation.linaro.org/scheduler/job/845472/log_file#L_349_223
- I can’t imagine that this is causing the problem
- A little later on, you see it tries to bring the network interfaces up:
I was curious about this.
I looked at the kernel traces and was interested to see that the working version has an smsc911x message coming out as soon as the network adapter is used. On close inspection it looks to me like in the working case the smsc911x driver compiled into the kernel, whilst for the non-working version smsc911x is loaded as a module. Worse the module is not loaded until well after we have started trying to bring up the network.
- Working one:
https://validation.linaro.org/scheduler/job/845552/log_file#L_285_9
[ 0.000000] Linux version 3.10.63.0-1-linaro-lt-vexpress64 (buildslave@x86-64-07) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #1ubuntu1~ci+150113192220 SMP Tue Jan 13 19:25:14 UTC 2015
Wrong link, should be:
https://validation.linaro.org/scheduler/job/845552/log_file#L_281_8
Linux version 4.4.0-rc2+ (leoy@leoy-linaro) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #73 SMP PREEMPT Sat Apr 23 23:11:46 CST 2016
Always check the kernel boot prior to the message: INFO: System is in test image now, performing basic user space tests.
- Non working one:
https://validation.linaro.org/scheduler/job/845472/log_file#L_351_10
[ 0.000000] Linux version 4.4.0-rc2+ (leoy@leoy-linaro) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #73 SMP PREEMPT Sat Apr 23 23:11:46 CST 2016
Wrong link, should be: https://validation.linaro.org/scheduler/job/845472/log_file#L_279_8
Linux version 4.4.0-rc2+ (leoy@leoy-linaro) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #73 SMP PREEMPT Sat Apr 23 23:11:46 CST 2016
So it is the same kernel booting each time and the logs also include the checksums of the downloaded files, those match in each job.
It's one of the problems of V1 - there are a lot of kernel boots involved in a juno testjob. You also need to check the reported firmware versions and other messages. It can be best to download both log files, strip out the other kernel boot logs and compare the results that way.
https://validation.linaro.org/scheduler/job/845472/log_file#L_296_1
<LAVA_DISPATCHER>2016-04-24 09:11:19 AM ERROR: Userspace Error: image prompt not found.
So it tries again: https://validation.linaro.org/scheduler/job/845472/log_file#L_347_8 Linux version 4.4.0-rc2+ (leoy@leoy-linaro) (gcc version 4.9.2 20140904 (prerelease) (crosstool-NG linaro-1.13.1-4.9-2014.09 - Linaro GCC 4.9-2014.09) ) #73 SMP PREEMPT Sat Apr 23 23:11:46 CST 2016347.9 [ 0.000000] Boot CPU: AArch64 Processor [410fd030]
In each of those logs, I see the smsc911x message:
https://validation.linaro.org/scheduler/job/845552/log_file#L_283_152
https://validation.linaro.org/scheduler/job/845472/log_file#L_349_152
As spotted already, the problem is not that the driver isn't loading, it's that the state is inconsistent:
https://validation.linaro.org/scheduler/job/845552/log_file#L_285_9 Configuring network interfaces... [ 8.449499] smsc911x 18000000.ethernet eth0: SMSC911x/921x identified at 0xffffff8000120000, IRQ: 29285.10 udhcpc (v1.24.1) started 285.11 Sending discover... 285.12 Sending discover... 285.13 Sending select for 10.7.0.20... 285.14 Lease of 10.7.0.20 obtained, lease time 3600 285.15 /etc/udhcpc.d/50default: Adding DNS 10.0.0.2285.16 done.
https://validation.linaro.org/scheduler/job/845472/log_file#L_351_10 Configuring network interfaces... ifup: interface lo already configured 351.11 ifup: interface eth0 already configured 351.12 done.351.13 Starting OpenBSD Secure Shell server: sshd