On 18/12/17 15:06, Neil Williams wrote:
he On 18 December 2017 at 14:44, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 18/12/17 11:45, Neil Williams wrote:
On 14 December 2017 at 09:47, Guillaume Tucker < guillaume.tucker@collabora.com> wrote:
On 07/12/17 17:16, Neil Williams wrote:
On 7 December 2017 at 16:20, Guillaume Tucker <
guillaume.tucker@collabora.com> wrote:
A change was sent a while ago to add support for the Coreboot /
Depthcharge bootloader which is used on Chromebook devices. This is useful in particular to avoid having to install U-Boot on Chromebook devices. See this Gerrit review here for previous history:
https://review.linaro.org/#/c/15203/I'm now opening this case again to try and get this resolved, there seem to be several issues with the original patch that would need to be clarified. Also, some things might have changed since then in LAVA or Coreboot which could potentially lead to a different approach - any feedback on this would be welcome.
Thanks for picking this up.
You're welcome. I've now uploaded a new version which generates the command line file but not the FIT image, it expects the kernel image to be already in this format. Still the same Gerrit number:
https://review.linaro.org/#/c/15203/I've also made a patch to add the rk3288-veyron-jaq as a "depthcharge" device type:
https://review.linaro.org/#/c/22992/So as a next step, it would be convenient to find a way to have the FIT image generated as part of the LAVA job with a given kernel image, dtb, maybe the .its file and optionally a ramdisk.
For the reference:
http://git.denx.de/?p=u-boot.git;a=blob;f=doc/uImage.FIT/howto.txt;hb=master
To start with, I understand that running mkimage on the
dispatcher is not a valid thing to do, it should receive a
FIT (flattened image tree) kernel image ready to be booted. This complicates things a bit for projects like kernelci.org where only a plain kernel image is built and ramdisks are served separately, but it's fair enough to say that LAVA is not meant to be packaging kernel images on the fly.
We've come up with a method in the meantime, although it does mean
using LXC but that makes it completely generic. It's principally designed for boards which need to munge a kernel and other files into an image to be transferred to the device using tools like fastboot. This is how KernelCI will be able to submit boot tests on devices like HiKey and db410c. Sadly, the example test job is suffering because the db410c devices have a different problem which is keeping them offline. Matt has been looking into this.
https://staging.validation.linaro.org/scheduler/job/203317/definition
https://staging.validation.linaro.org/static/docs/v2/actions -deploy.html#index-25
Thanks for the pointers, seems worth investigating.
On the other hand, creating the FIT image is a similar process to that of uImage, which is currently being done directly on the dispatcher:
https://git.linaro.org/lava/lava-dispatcher.git/tree/lava_dispatcher/actions/deploy/prepare.py#n79
So would it make sense to add some code there to support FIT?
What is an example command line to mkimage to do this?
mkimage -D "-I dts -O dtb -p 2048" -f rk3288-veyron-jaq.its arch/arm/boot/vmlinuz
No --arch argument? Is this operation agnostic about 32bit vs 64bit or armv7 vs armv8?
The CPU arch is also part of the .its file, here's an example:
https://people.collabora.com/~gtucker/tmp/rk3288-veyron-jaq-ramdisk.its
-p 2048 - the position presumably comes from the device configuration too, just like a load address?
This is the dtc -p padding option, not mkimage -p position for external data. It's actually more related to the kernel image and what the bootloader does to it than any platform in particular. So I think a large enough padding value like 2048 should work with all cases. It may still make sense to have a template variable to override this on some devices.
(mkimage_arch is an available value, so it would seem sensible to use it if it's supportable. Someone, somewhere will create a 64bit machine with 32bit firmware - as happened with the mustang - so avoid any assumptions here.)
Sure.
Are any external configuration files required?
Everything should be in the .its file, and it should also be possible to generate it on the fly using a template and the LAVA device properties (kernel load address etc...). If this proves to not be flexible enough in practice, then I suppose the .its file could be downloaded although I think we should avoid doing this if we can.
Building the .its file in the per-job temporary directory from data already available in the device configuration sounds ideal.
Great.
The LXC support remains available if particular tests need customised handling or options (or if a patched version of mkimage is needed).
The FIT format has been around for long enough for it to work with standard mkimage, I would hope. But yes, if some LAVA labs use an older distro with an incompatible version then LXC would be a way to keep the version of mkimage under control. It would however make the test definitions a bit more complicated and depthcharge-specific, which is something we would prefer to avoid for kernelci.org in particular.
So I'll see how it goes with running mkimage on the dispatcher and with various versions of mkimage.
Then I believe creating the command line file in LAVA should be
fine, although it probably makes more sense to have both the FIT
image and cmdline file generated by the same build system. In any case, both files would need to be served from the dispatcher TFTP server to the target device running Coreboot / Depthcharge.
That bit is fine, the problem is why this cannot use the existing
temporary paths which all the other TFTP devices use. Is it just to do some mangling of the files?
This is resolved now with the version I sent yesterday.
That makes this review much better, thanks.
Great, thanks for confirming.
So the idea was basically to have an option in Coreboot /
Depthcharge to interactively tell it where to find these files
for the current job to run, say:
<JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/vmlinuz <JOB_NUMBER>/tftp-deploy-<RANDOM>/kernel/cmdlineIt looks like the current patch in Gerrit relies on this location to be hard-coded in the bootloader, which works fine for a private development set-up but not for LAVA.
That makes very little sense because the whole point of TFTP is that
everything after the SERVER_IP is just a relative path from the TFTP base directory which is handled by the TFTP daemon itself.
Ditto.
To recap, my understanding is that the "depthcharge" boot support
code in LAVA would need to:
* maybe create the cmdline file with basically the kernel command line split up with one argument per lineAlternatively, do whatever operations are required in a test shell in
the LXC and then pass those files to the device - entirely within the test shell support.
That, or maybe run mkimage on the dispatcher like for uImage...
The cmdline file is now generated on the dispatcher.
* or just download the cmdline file along with the vmlinuz FITThe ready-made FIT kernel image is now downloaded with the
version I sent yesterday.
* place both the cmdline and vmlinuz FIT files in the job'sTFTP directory on the dispatcher* turn on the device and open the serial console... * interactively pass at least the path to the job TFTP directory on the serial console (and if possible the server IP address as well, and maybe even the individual file names rather than hard-coded vmlinuz and cmdline)Isn't this equivalent to what U-Boot already does with TFTP?
Almost. This part is now all implemented in the last patch I sent. One thing though is that the NFS rootfs parameters are stored in the kernel cmdline file and not set interactively in the bootloader shell.
How can these be extended by test writers? We do see requests to add arguments to the NFS parameters but adding options to the kernel command line itself is all but essential for most testing.
This can be done using the {{ extra_kernel_args }} template variable, see the other change to add base-depthcharge.jinja2:
https://review.linaro.org/#/c/22992/1/lava_scheduler_app/tes ts/device-types/base-depthcharge.jinja2
If anything more special ever needs to be done with some parameters such as inserting some IP address, it can be done in DepthchargeCommandOverlay where the command line file is generated.
That's what I wanted to check, OK. Generating the command line is the right way to do it.
Great.
The only command sent is to start the tftpboot with the server IP and the relative paths to the kernel and cmdline files.
On this topic, the changes to add the tftpboot command in Depthcharge are still under review:
https://chromium-review.googlesource.com/c/chromiumos/platfo rm/depthcharge/+/451382
So I think it would actually be wiser to not merge base-depthcharge.jinja2 until the review above has been merged in case the command line syntax needs to be adjusted.
OK.
The one remaining issue is that this support (like NBD) doesn't have supported devices available in the staging instance, so we need to do as much validation as possible in the unit tests to ensure that future changes elsewhere do not cause issues with depthcharge. e.g. Do as much of the command line generation as possible in validate() so that this can be checked in the unit test. Then use the substitute support to replace {KERNEL} or {LOAD_ADDR} type fields in the run() function to allow for the temporary locations. Only you know how much variability there may be in the final command lines - each variant should be checked, that if the incoming device configuration is correct (static, so easy to check), then the validate() produces a command line which is precisely what is expected, prior to running substitute. We won't be able to run functional tests in the run up to a production release, we need the unit test to have coverage which is as wide as possible.
Understood.
In addition to adding good unit test coverage, I'll see if I can keep a spare Chromebook outside of our lab to use with my dev set-up as I'm using now (i.e. laptop running latest LAVA). It won't be automated like on your staging instance but would be better than nothing if I can just run some tests each month.
* look for a bootloader message to know when the kernel startsto load and hand over to the next action (login...)Done as well, I've now got the veyron-jaq device booting fine
with NFS rootfs. There was an issue with adding a ramdisk to the FIT image as it was to big to boot on the device, will investigate this part to add "ramdisk" boot commands.
Seems to be a perennial problem with U-Boot - the initial load addresses are always too conservative and as soon as people start building fully functional kernels / ramdisks, there isn't enough space between the load addresses.
I've fixed this now, it's a slightly different issue here. There's a build-time configuration in Depthcharge to reserve some memory to receive the FIT image. The default value is fine for Chromebook products which don't have a ramdisk. I've extended it and now it gets the full FIT with ramdisk included. I still need to fix a few more things though as it's not quite booting with ramdisk yet...
Best wishes, Guillaume