How to debug "Starting kernel ..." of Bullseye debian-installer?

[mossroy]

Debian Bullseye will ship with a kernel 5.10, which should support Olinuxino A64 boards (at least for most features, see https://linux-sunxi.org/Linux_mainlining_effort).

I tried the debian-installer (daily images from https://d-i.debian.org/daily-images/arm64/daily/netboot/SD-card-images/ , I tested version of March 1st 2021). The installation works, but :

• The board does not restart by itself at the end of the installation (it is stuck on a line saying it's asking the board to reboot, after all steps are finished). It is minor
• On startup, it is stuck on "Starting kernel ...". It is of course a blocker

The kernel unfortunately does not give any feedback on what's wrong.
I tried adding "earlyprintk loglevel=7" boot parameters in u-boot before startup (in the u-boot prompt, and even by re-generating a modified boot.scr with mkimage), but did not manage to have debug information on startup.

Anybody would know how to have more information? Or has succeeded in installing Debian Bullseye on a Olinuxino A64 board?
The answer might be somewhere in the commits of https://github.com/OLIMEX/linux-olimex/ or https://github.com/OLIMEX/olimage
Maybe some DTB or DTB overlays need to be added and configured ? Should we open a bug on Debian side?

Excerpt from serial console :

Found U-Boot script /boot.scr
2226 bytes read in 1 ms (2.1 MiB/s)
## Executing script at 4fc00000
26990448 bytes read in 1198 ms (21.5 MiB/s)
28249 bytes read in 5 ms (5.4 MiB/s)
27081515 bytes read in 1205 ms (21.4 MiB/s)
Booting Debian 5.10.0-3-arm64 from mmc 0:1...
## Flattened Device Tree blob at 4fa00000
  Booting using the fdt blob at 0x4fa00000
  Loading Ramdisk to 4862c000, end 49fffb2b ... OK
  Loading Device Tree to 0000000048622000, end 000000004862be58 ... OK

Starting kernel ...

[mossroy]

I still have the same behavior with a recent daily Debian image (2021-04-27).

Do you know how to debug that, and what is the right way to report this issue?

[JohnS]

Looks like a software issue and thus a good question to ask on the linux-sunxi ML.

John

[mossroy]

Looks like a software issue and thus a good question to ask on the linux-sunxi ML.

Thanks for the hint. I'll try to discuss that on their IRC.

I'm wondering if it could not come from the fact that I have the A64-OLinuXino-2Ge8G-IND version. Did anybody else test the bullseye debian-installer with another Olinuxino A64 board?

I tried to switch to another dtb (in the boot partition of the installed debian), using the sun50i-a64-olinuxino-2Ge8G.dtb file found in Olimex images, but with no luck so far

[mossroy]

I've been advised on IRC to set "earlycon" to bootargs environment variable in u-boot (instead of earlyprintk)

It indeed gives a bit more info :

Starting kernel ...

[    2.889410] sun50i-de2-bus 1000000.bus: Error couldn't map SRAM to device
[    3.186735] sun50i-a64-pinctrl 1c20800.pinctrl: Couldn't get bank PB regulator
[    3.193983] sun50i-a64-pinctrl 1c20800.pinctrl: request() failed for pin 40
[    3.200947] sun50i-a64-pinctrl 1c20800.pinctrl: pin-40 (1c28000.serial) status -517
[    3.208604] sun50i-a64-pinctrl 1c20800.pinctrl: could not request pin 40 (PB8) from group PB8  on device 1c20800.pinctrl
[    3.219465] dw-apb-uart 1c28000.serial: Error applying setting, reverse things back
[    3.227746] sun50i-a64-pinctrl 1c20800.pinctrl: Couldn't get bank PB regulator
[    3.234989] sun50i-a64-pinctrl 1c20800.pinctrl: request() failed for pin 40
[    3.241953] sun50i-a64-pinctrl 1c20800.pinctrl: pin-40 (1c28000.serial) status -517
[    3.249610] sun50i-a64-pinctrl 1c20800.pinctrl: could not request pin 40 (PB8) from group PB8  on device 1c20800.pinctrl
[    3.260470] dw-apb-uart 1c28000.serial: Error applying setting, reverse things back

[mossroy]

I've tried a few other dtb/dtbo combinations using the files found in Olimex images, like :

setenv bootargs earlycon
setenv fdtfile allwinner/sun50i-a64-olinuxino-2Ge8G.dtb
setenv fdtoverlays /dtbs/5.10.0-6-arm64/allwinner/sun50i-a64-i2c0.dtbo

It sometimes changed the log a bit, but did not make the board start

[JohnS]

I suppose you might try earlycon with a working kernel and report to the ML what changed.

You may be asked to bisect from git...

John

[mossroy]

In fact, these u-boot variable values sometimes let me start debian bullseye. There must be a random aspect that makes the startup not reliable.
I had to modify the boot.scr file to remove the "quiet" kernel parameter to see what was going on (following advice from linux-sunxi IRC)

There are currently some issues, though :

• the board does not always start (I'd say it starts around once over 4 times). It seems to be related to mmc (I tested with 2 different SD-cards)
• the board does not manage to reboot : it stops but never restarts (I have to do it manually)
• the hardware crypto works with aes-xts, but not with aes-cbc
• if it needs to regenerate the initramfs, it fails with "Unsupported platform 'Olimex A64-Olinuxino-2Ge8G-IND'"

But there are good news, too :

• the default governor is "schedutil" and does not seem to suffer from the slowness described in https://www.olimex.com/forum/index.php?topic=7976.0
• all the sensors are working : it does not suffer from https://www.olimex.com/forum/index.php?topic=8142.0

[mossroy]

People from linux-sunxi seem to be interested to have more feedback from Olimex (it's at least what apritzel said on IRC).

About what Olimex did in their dtb/dtbo (and why), about the patches applied on the kernel (they know the github repo is public, but would need to understand the reasons behind the commits, at least on the timer jump issue) etc. So that all this might be applied in upstream kernel at some point, instead of being olimex-specific changes that need to be maintained by Olimex.

They would also be interested in more Olimex feedback and tests (not only about code).

I'd be glad to help on that, but can't do it alone.

Wouldn't it be much easier for everybody if the boards were supported by a future version of debian-installer?

[mossroy]

I tested to install the latest 5.10.23 kernel of Olimex on the SD-card generated by debian-installer (instead of the 5.10.0-6-arm64 provided by debian, based on 5.10.28), using the dtb/dtbo from Olimex too. I had to cheat in boot.scr to do that.

The board always starts properly (even if there is always a kernel stacktrace at drivers/clocksource/arm_arch_timer.c:364 during startup), and I did not see any issue with MMC.
The board reboots normally (most of the time : it happened to not reboot, like with debian kernel)
Hardware crypto works for both aes-cbc and aes-xts, but same issue with initramfs (it's probably a problem on my side)
But of course the performance governor is used, and the sensors are missing

NB : I reverted all that, it was just for testing purpose

[mossroy]

I found this related bug report on debian : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=975490