oLinuXino Lime - kernel crash (?) after poweroff command

[roed]

oLinuXino Lime. SD card image taken from here.
Fragment of serial console output after sudo poweroff command:

Code Select Expand

[  725.773280] reboot: Power down
[  725.776405] ------------[ cut here ]------------
[  725.781051] WARNING: CPU: 0 PID: 1 at drivers/i2c/i2c-core.h:41 i2c_transfer+0x93/0x9c
[  725.788960] No atomic I2C transfer handler for 'i2c-0'
[  725.794092] Modules linked in: rfkill evdev axp20x_adc at24 industrialio lima gpu_sched sun4i_drm_hdmi sun4i_ss uio_pdrv_genirq uio cpufreq_dt sch_fq_codel sun4i_ts usb_f_acm u_serial g_serial libcomposite ip_tables x_tables sun4i_gpadc
[  725.815081] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 5.6.14-olimex #081742
[  725.822555] Hardware name: Allwinner sun7i (A20) Family
[  725.827811] [<c010db45>] (unwind_backtrace) from [<c010a19d>] (show_stack+0x11/0x14)
[  725.835563] [<c010a19d>] (show_stack) from [<c09bd10b>] (dump_stack+0x6f/0x7c)
[  725.842797] [<c09bd10b>] (dump_stack) from [<c011b897>] (__warn+0xb7/0xb8)
[  725.849681] [<c011b897>] (__warn) from [<c011bb43>] (warn_slowpath_fmt+0x5f/0x74)
[  725.857169] [<c011bb43>] (warn_slowpath_fmt) from [<c07b242f>] (i2c_transfer+0x93/0x9c)
[  725.865178] [<c07b242f>] (i2c_transfer) from [<c07b2473>] (i2c_transfer_buffer_flags+0x3b/0x50)
[  725.873882] [<c07b2473>] (i2c_transfer_buffer_flags) from [<c06c3cc7>] (regmap_i2c_write+0x13/0x24)
[  725.882931] [<c06c3cc7>] (regmap_i2c_write) from [<c06c0949>] (_regmap_raw_write_impl+0x459/0x534)
[  725.891892] [<c06c0949>] (_regmap_raw_write_impl) from [<c06c11c9>] (regmap_write+0x29/0x3c)
[  725.900333] [<c06c11c9>] (regmap_write) from [<c06ca4e3>] (axp20x_power_off+0x23/0x30)
[  725.908256] [<c06ca4e3>] (axp20x_power_off) from [<c0136a29>] (__do_sys_reboot+0xf5/0x168)
[  725.916526] [<c0136a29>] (__do_sys_reboot) from [<c0101001>] (ret_fast_syscall+0x1/0x62)
[  725.924610] Exception stack(0xdf511fa8 to 0xdf511ff0)
[  725.929667] 1fa0:                   4321fedc be97eb48 fee1dead 28121969 4321fedc 565f3d00
[  725.937846] 1fc0: 4321fedc be97eb48 be97eb44 00000058 be97eb48 be97eb44 fffff000 be97eb4c
[  725.946019] 1fe0: 00000058 be97eabc a7e3cd65 a7dbbbe6
[  725.951070] ---[ end trace f77db84b51badb33 ]---

Is this normal?
According to this discussion I think master_xfer_atomic implementation is missing in i2c-mv64xxx.c driver.

[LubOlimex]

Can you tell me which board exactly do you have and what is the hardware revision so we can test it here?

Thanks,
Lub/OLIMEX

[LubOlimex]

Notice the focal releases are still experimental. If you experience problems with focal try the buster images.

[roed]

Board: A20-oLinuXino-Lime, Rev. E.

Normal behavior on Bionic and Buster images.

[lpg]

I have the same kernel warning during poweroff (but not during a reboot) with a custom bootable image (based on olimex forks of u-boot and linux) on a A20-OLinuxino-Lime2.

Code Select Expand

[  782.333855] sun7i-dwmac 1c50000.ethernet eth0: Link is Down
[  782.821361] printk: systemd-shutdow: 24 output lines suppressed due to ratelimiting
[  782.937779] systemd-shutdown[1]: Syncing filesystems and block devices.
[  784.528892] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[  784.553351] systemd-journald[152]: Received SIGTERM from PID 1 (systemd-shutdow).
[  784.587451] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[  784.620260] systemd-shutdown[1]: Unmounting file systems.
[  784.640329] [322]: Remounting '/' read-only in with options '(null)'.
[  784.701479] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[  784.713205] systemd-shutdown[1]: All filesystems unmounted.
[  784.713254] systemd-shutdown[1]: Deactivating swaps.
[  784.717876] systemd-shutdown[1]: All swaps deactivated.
[  784.717960] systemd-shutdown[1]: Detaching loop devices.
[  784.748230] systemd-shutdown[1]: All loop devices detached.
[  784.748274] systemd-shutdown[1]: Detaching DM devices.
[  784.827914] reboot: Power down
[  784.827944] printk: enabled sync mode
[  784.835012] printk: console [ttyS0]: printing thread stopped
[  784.953715] ------------[ cut here ]------------
[  784.958355] WARNING: CPU: 0 PID: 1 at drivers/i2c/i2c-core.h:41 i2c_transfer+0x108/0x114
[  784.966469] No atomic I2C transfer handler for 'i2c-0'
[  784.971601] Modules linked in: evdev axp20x_battery axp20x_ac_power axp20x_adc pinctrl_axp209 industrialio at24 lima gpu_sched sunxi_cedrus(C) sun4i_ts sun4i_codec hwm
on v4l2_mem2mem videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common videodev sun4i_ss libdes mc uio_pdrv_genirq uio cpufreq_dt usb_f_acm u_serial g_seri
al libcomposite ip_tables x_tables
[  785.004331] CPU: 0 PID: 1 Comm: systemd-shutdow Tainted: G        C        5.10.36-rt34+ #1
[  785.012765] Hardware name: Allwinner sun7i (A20) Family
[  785.017986] Backtrace:
[  785.020435] [<c0a9f188>] (dump_backtrace) from [<c0a9f550>] (show_stack+0x20/0x24)
[  785.028020]  r7:00000029 r6:600f0193 r5:00000000 r4:c0fafbe4
[  785.033672] [<c0a9f530>] (show_stack) from [<c0aa2f40>] (dump_stack+0xa4/0xb8)
[  785.040899] [<c0aa2e9c>] (dump_stack) from [<c0124334>] (__warn+0xfc/0x114)
[  785.047874]  r7:00000029 r6:00000009 r5:c07f1e0c r4:c0cd26cc
[  785.053526] [<c0124238>] (__warn) from [<c0a9fca4>] (warn_slowpath_fmt+0xa4/0xc0)
[  785.061016]  r7:00000029 r6:c0cd26cc r5:c0f077c8 r4:c0cd276c
[  785.066668] [<c0a9fc04>] (warn_slowpath_fmt) from [<c07f1e0c>] (i2c_transfer+0x108/0x114)
[  785.074856]  r9:00000032 r8:c102efc0 r7:c83b94c1 r6:00000001 r5:c14fdd30 r4:c83d54d0
[  785.082589] [<c07f1d04>] (i2c_transfer) from [<c07f1e74>] (i2c_transfer_buffer_flags+0x5c/0x80)
[  785.091296]  r6:c83b94c1 r5:00000002 r4:c0f077c8
[  785.095908] [<c07f1e18>] (i2c_transfer_buffer_flags) from [<c06b409c>] (regmap_i2c_write+0x24/0x40)
[  785.104966]  r5:00000001 r4:00000002
[  785.108538] [<c06b4078>] (regmap_i2c_write) from [<c06ae9e4>] (_regmap_raw_write_impl+0x5ec/0x91c)
[  785.117503]  r5:00000001 r4:c8388200
[  785.121075] [<c06ae3f8>] (_regmap_raw_write_impl) from [<c06aed9c>] (_regmap_bus_raw_write+0x88/0xb0)
[  785.130302]  r10:00000058 r9:c0f117b8 r8:e39d6300 r7:00000080 r6:00000032 r5:c06a9f60
[  785.138122]  r4:c8388200
[  785.140653] [<c06aed14>] (_regmap_bus_raw_write) from [<c06ae0dc>] (_regmap_write+0x7c/0x180)
[  785.149186]  r7:c8388200 r6:00000080 r5:00000032 r4:c8388200
[  785.154837] [<c06ae060>] (_regmap_write) from [<c06afb04>] (regmap_write+0x4c/0x6c)
[  785.162504]  r9:c0f117b8 r8:e39d6300 r7:fee1dead r6:00000080 r5:00000032 r4:c8388200
[  785.170236] [<c06afab8>] (regmap_write) from [<c06bed84>] (axp20x_power_off+0x3c/0x48)
[  785.178165]  r7:fee1dead r6:00000000 r5:4321fedc r4:c0f077c8
[  785.183818] [<c06bed48>] (axp20x_power_off) from [<c010ae38>] (machine_power_off+0x34/0x38)
[  785.192172] [<c010ae04>] (machine_power_off) from [<c014ec34>] (kernel_power_off+0x7c/0x80)
[  785.200525] [<c014ebb8>] (kernel_power_off) from [<c014ed80>] (__do_sys_reboot+0x148/0x1f4)
[  785.208878] [<c014ec38>] (__do_sys_reboot) from [<c014ee9c>] (sys_reboot+0x18/0x1c)
[  785.216543]  r9:c14fc000 r8:c0100264 r7:00000058 r6:00000000 r5:00000000 r4:043d6954
[  785.224275] [<c014ee84>] (sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x5c)
[  785.231937] Exception stack(0xc14fdfa8 to 0xc14fdff0)
[  785.236991] dfa0:                  043d6954 00000000 fee1dead 28121969 4321fedc e39d6300
[  785.245164] dfc0: 043d6954 00000000 00000000 00000058 bee76c68 00000000 bee76c00 043d56f8
[  785.253334] dfe0: 00000058 bee76b5c aedaad45 aed2c746
[  785.258381] ---[ end trace 0000000000000002 ]---

I found another topic mentionning this warning on post #14.

The warning does not seem related to loaded modules because even when there are not any loaded module mentionned in the debug, the warning appears.

Because I did not see this warning using Olimage I was wondering how it was solved.

If it can be usefull, some informations from u-boot:

Code Select Expand

U-Boot SPL 2021.04-00009-gafdc7e9cb7-dirty (Jun 02 2021 - 11:21:25 +0200)
DRAM: 1024 MiB
CPU: 912000000Hz, AXI/AHB/APB: 3/2/2
Trying to boot from MMC1


U-Boot 2021.04-00009-gafdc7e9cb7-dirty (Jun 02 2021 - 11:21:25 +0200) Allwinner Technology

CPU:  Allwinner A20 (SUN7I)
Model: Olimex A20-OLinuXino-LIME2
I2C:  ready
DRAM:  1 GiB
MMC:  mmc@1c0f000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment


[LubOlimex]

I can confirm the kernel panic during poweroff, it also happens here. I've notified the developers to look into it.

[vazhnov]

The same with fresh Armbian images. Board: Olimex A20-OLinuXino-LIME2.

Old image "Armbian_21.05.1_Lime2_buster_current_5.10.34.img" works fine and does poweroff.

Code Select Expand

_     _                  ____
| |   (_)_ __ ___   ___  |___ \
| |   | | '_ ` _ \ / _ \   __) |
| |___| | | | | | |  __/  / __/
|_____|_|_| |_| |_|\___| |_____|

Welcome to Armbian 21.05.1 Buster with Linux 5.10.34-sunxi

System load:   6%           Up time:       24 min Local users:   2
Memory usage:  7% of 998M   IP:        192.168.1xx.yy
CPU temp:      43°C           Usage of /:    35% of 3.5G
Battery:       60% discharging
RX today:      12 KiB

root@lime2-buster:~# cat /proc/cmdline
root=UUID=1c515dac-00f8-4ffd-85f7-36c362e6b710 rootwait rootfstype=ext4 console=ttyS0,115200 console=tty1 hdmi.audio=EDID:0 disp.screen0_output_mode=1920x1080p60 consoleblank=0 loglevel=1 ubootpart=952f5bc3-01 ubootsource=mmc usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u   sunxi_ve_mem_reserve=0 sunxi_g2d_mem_reserve=0 sunxi_fb_mem_reserve=16 cgroup_enable=memory swapaccount=1

root@lime2-buster:~# uname -a
Linux lime2-buster 5.10.34-sunxi #21.05.1 SMP Thu May 6 20:13:21 UTC 2021 armv7l GNU/Linux


Fresh "Armbian_22.02.1_Lime2_bullseye_current_5.15.25.img" works fine, but has kernel panic when doing poweroff:

No atomic I2C transfer handler for 'i2c-1'
...
mv64xxx: I2C bus locked, block: 1, time_left: 0

Also, I created my first fresh build with https://github.com/armbian/build , "Armbian_22.08.0-trunk_Lime2_bullseye_current_5.15.57_minimal.img", and it has the same issue.

I found this [BUG] sun4i: axp209: no atomic i2c transfer handler but there is no solution.

Workaround

Use older Linux kernel.

Armbian has possibility to create image with "legacy" kernel version. I used these options:

Code Select Expand

./compile.sh BOARD=lime2 BRANCH=legacy RELEASE=bullseye BUILD_MINIMAL=yes BUILD_DESKTOP=no KERNEL_ONLY=no KERNEL_CONFIGURE=prebuilt COMPRESS_OUTPUTIMAGE=sha,gpg,img

With this image, my Olimex A20-OLinuXino-LIME2 has no problem with powering off.

[markus2330]

I get the same error message but also during runtime. From time to time the i2c blocks/locks up/doesn't reset on one of my devices out of the blue (probably the fault of a Mod-IO2 connected, which needs a power-off afterwards). If this situation occurs (it does every few days) on any attempt of i2c communication I get

Code Select Expand

EIO. The kernel messages issued are the same, so I thought it is the same issue:

Code Select Expand

[   15.725811] i2c i2c-2: mv64xxx: I2C bus locked, block: 1, time_left: 0
[ 2705.668308] i2c i2c-2: mv64xxx: I2C bus locked, block: 1, time_left: 0
[ 3715.536689] i2c i2c-2: mv64xxx_i2c_fsm: Ctlr Error -- state: 0x2, status: 0x0, addr: 0x21, flags: 0x0
[ 3718.621965] i2c i2c-2: mv64xxx: I2C bus locked, block: 1, time_left: 0

Any plans that the atomic i2c transfer handler gets implemented?

Is there also some way to install the old kernel without creating a new image?

[markus2330]

I could improve i2c locks at runtime for me by reducing the i2c bus speed. The overlay is part of https://opensesame.libelektra.org/ (files/i2c_2_clock_freq_overlay.dtbo see also doc/I2C.md).