oLinuXino Lime - kernel crash (?) after poweroff command

Started by roed, August 23, 2020, 10:02:29 PM

Previous topic - Next topic

roed

oLinuXino Lime. SD card image taken from here.
Fragment of serial console output after sudo poweroff command:

[  725.773280] reboot: Power down
[  725.776405] ------------[ cut here ]------------
[  725.781051] WARNING: CPU: 0 PID: 1 at drivers/i2c/i2c-core.h:41 i2c_transfer+0x93/0x9c
[  725.788960] No atomic I2C transfer handler for 'i2c-0'
[  725.794092] Modules linked in: rfkill evdev axp20x_adc at24 industrialio lima gpu_sched sun4i_drm_hdmi sun4i_ss uio_pdrv_genirq uio cpufreq_dt sch_fq_codel sun4i_ts usb_f_acm u_serial g_serial libcomposite ip_tables x_tables sun4i_gpadc
[  725.815081] CPU: 0 PID: 1 Comm: systemd-shutdow Not tainted 5.6.14-olimex #081742
[  725.822555] Hardware name: Allwinner sun7i (A20) Family
[  725.827811] [<c010db45>] (unwind_backtrace) from [<c010a19d>] (show_stack+0x11/0x14)
[  725.835563] [<c010a19d>] (show_stack) from [<c09bd10b>] (dump_stack+0x6f/0x7c)
[  725.842797] [<c09bd10b>] (dump_stack) from [<c011b897>] (__warn+0xb7/0xb8)
[  725.849681] [<c011b897>] (__warn) from [<c011bb43>] (warn_slowpath_fmt+0x5f/0x74)
[  725.857169] [<c011bb43>] (warn_slowpath_fmt) from [<c07b242f>] (i2c_transfer+0x93/0x9c)
[  725.865178] [<c07b242f>] (i2c_transfer) from [<c07b2473>] (i2c_transfer_buffer_flags+0x3b/0x50)
[  725.873882] [<c07b2473>] (i2c_transfer_buffer_flags) from [<c06c3cc7>] (regmap_i2c_write+0x13/0x24)
[  725.882931] [<c06c3cc7>] (regmap_i2c_write) from [<c06c0949>] (_regmap_raw_write_impl+0x459/0x534)
[  725.891892] [<c06c0949>] (_regmap_raw_write_impl) from [<c06c11c9>] (regmap_write+0x29/0x3c)
[  725.900333] [<c06c11c9>] (regmap_write) from [<c06ca4e3>] (axp20x_power_off+0x23/0x30)
[  725.908256] [<c06ca4e3>] (axp20x_power_off) from [<c0136a29>] (__do_sys_reboot+0xf5/0x168)
[  725.916526] [<c0136a29>] (__do_sys_reboot) from [<c0101001>] (ret_fast_syscall+0x1/0x62)
[  725.924610] Exception stack(0xdf511fa8 to 0xdf511ff0)
[  725.929667] 1fa0:                   4321fedc be97eb48 fee1dead 28121969 4321fedc 565f3d00
[  725.937846] 1fc0: 4321fedc be97eb48 be97eb44 00000058 be97eb48 be97eb44 fffff000 be97eb4c
[  725.946019] 1fe0: 00000058 be97eabc a7e3cd65 a7dbbbe6
[  725.951070] ---[ end trace f77db84b51badb33 ]---

Is this normal?
According to this discussion I think master_xfer_atomic implementation is missing in i2c-mv64xxx.c driver.

LubOlimex

Can you tell me which board exactly do you have and what is the hardware revision so we can test it here?

Thanks,
Lub/OLIMEX
Technical support and documentation manager at Olimex

LubOlimex

Notice the focal releases are still experimental. If you experience problems with focal try the buster images.
Technical support and documentation manager at Olimex

roed

Board: A20-oLinuXino-Lime, Rev. E.

Normal behavior on Bionic and Buster images.

lpg

I have the same kernel warning during poweroff (but not during a reboot) with a custom bootable image (based on olimex forks of u-boot and linux) on a A20-OLinuxino-Lime2.

[  782.333855] sun7i-dwmac 1c50000.ethernet eth0: Link is Down
[  782.821361] printk: systemd-shutdow: 24 output lines suppressed due to ratelimiting
[  782.937779] systemd-shutdown[1]: Syncing filesystems and block devices.
[  784.528892] systemd-shutdown[1]: Sending SIGTERM to remaining processes...
[  784.553351] systemd-journald[152]: Received SIGTERM from PID 1 (systemd-shutdow).
[  784.587451] systemd-shutdown[1]: Sending SIGKILL to remaining processes...
[  784.620260] systemd-shutdown[1]: Unmounting file systems.
[  784.640329] [322]: Remounting '/' read-only in with options '(null)'.
[  784.701479] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[  784.713205] systemd-shutdown[1]: All filesystems unmounted.
[  784.713254] systemd-shutdown[1]: Deactivating swaps.
[  784.717876] systemd-shutdown[1]: All swaps deactivated.
[  784.717960] systemd-shutdown[1]: Detaching loop devices.
[  784.748230] systemd-shutdown[1]: All loop devices detached.
[  784.748274] systemd-shutdown[1]: Detaching DM devices.
[  784.827914] reboot: Power down
[  784.827944] printk: enabled sync mode
[  784.835012] printk: console [ttyS0]: printing thread stopped
[  784.953715] ------------[ cut here ]------------
[  784.958355] WARNING: CPU: 0 PID: 1 at drivers/i2c/i2c-core.h:41 i2c_transfer+0x108/0x114
[  784.966469] No atomic I2C transfer handler for 'i2c-0'
[  784.971601] Modules linked in: evdev axp20x_battery axp20x_ac_power axp20x_adc pinctrl_axp209 industrialio at24 lima gpu_sched sunxi_cedrus(C) sun4i_ts sun4i_codec hwm
on v4l2_mem2mem videobuf2_dma_contig videobuf2_memops videobuf2_v4l2 videobuf2_common videodev sun4i_ss libdes mc uio_pdrv_genirq uio cpufreq_dt usb_f_acm u_serial g_seri
al libcomposite ip_tables x_tables
[  785.004331] CPU: 0 PID: 1 Comm: systemd-shutdow Tainted: G        C        5.10.36-rt34+ #1
[  785.012765] Hardware name: Allwinner sun7i (A20) Family
[  785.017986] Backtrace:
[  785.020435] [<c0a9f188>] (dump_backtrace) from [<c0a9f550>] (show_stack+0x20/0x24)
[  785.028020]  r7:00000029 r6:600f0193 r5:00000000 r4:c0fafbe4
[  785.033672] [<c0a9f530>] (show_stack) from [<c0aa2f40>] (dump_stack+0xa4/0xb8)
[  785.040899] [<c0aa2e9c>] (dump_stack) from [<c0124334>] (__warn+0xfc/0x114)
[  785.047874]  r7:00000029 r6:00000009 r5:c07f1e0c r4:c0cd26cc
[  785.053526] [<c0124238>] (__warn) from [<c0a9fca4>] (warn_slowpath_fmt+0xa4/0xc0)
[  785.061016]  r7:00000029 r6:c0cd26cc r5:c0f077c8 r4:c0cd276c
[  785.066668] [<c0a9fc04>] (warn_slowpath_fmt) from [<c07f1e0c>] (i2c_transfer+0x108/0x114)
[  785.074856]  r9:00000032 r8:c102efc0 r7:c83b94c1 r6:00000001 r5:c14fdd30 r4:c83d54d0
[  785.082589] [<c07f1d04>] (i2c_transfer) from [<c07f1e74>] (i2c_transfer_buffer_flags+0x5c/0x80)
[  785.091296]  r6:c83b94c1 r5:00000002 r4:c0f077c8
[  785.095908] [<c07f1e18>] (i2c_transfer_buffer_flags) from [<c06b409c>] (regmap_i2c_write+0x24/0x40)
[  785.104966]  r5:00000001 r4:00000002
[  785.108538] [<c06b4078>] (regmap_i2c_write) from [<c06ae9e4>] (_regmap_raw_write_impl+0x5ec/0x91c)
[  785.117503]  r5:00000001 r4:c8388200
[  785.121075] [<c06ae3f8>] (_regmap_raw_write_impl) from [<c06aed9c>] (_regmap_bus_raw_write+0x88/0xb0)
[  785.130302]  r10:00000058 r9:c0f117b8 r8:e39d6300 r7:00000080 r6:00000032 r5:c06a9f60
[  785.138122]  r4:c8388200
[  785.140653] [<c06aed14>] (_regmap_bus_raw_write) from [<c06ae0dc>] (_regmap_write+0x7c/0x180)
[  785.149186]  r7:c8388200 r6:00000080 r5:00000032 r4:c8388200
[  785.154837] [<c06ae060>] (_regmap_write) from [<c06afb04>] (regmap_write+0x4c/0x6c)
[  785.162504]  r9:c0f117b8 r8:e39d6300 r7:fee1dead r6:00000080 r5:00000032 r4:c8388200
[  785.170236] [<c06afab8>] (regmap_write) from [<c06bed84>] (axp20x_power_off+0x3c/0x48)
[  785.178165]  r7:fee1dead r6:00000000 r5:4321fedc r4:c0f077c8
[  785.183818] [<c06bed48>] (axp20x_power_off) from [<c010ae38>] (machine_power_off+0x34/0x38)
[  785.192172] [<c010ae04>] (machine_power_off) from [<c014ec34>] (kernel_power_off+0x7c/0x80)
[  785.200525] [<c014ebb8>] (kernel_power_off) from [<c014ed80>] (__do_sys_reboot+0x148/0x1f4)
[  785.208878] [<c014ec38>] (__do_sys_reboot) from [<c014ee9c>] (sys_reboot+0x18/0x1c)
[  785.216543]  r9:c14fc000 r8:c0100264 r7:00000058 r6:00000000 r5:00000000 r4:043d6954
[  785.224275] [<c014ee84>] (sys_reboot) from [<c0100060>] (ret_fast_syscall+0x0/0x5c)
[  785.231937] Exception stack(0xc14fdfa8 to 0xc14fdff0)
[  785.236991] dfa0:                  043d6954 00000000 fee1dead 28121969 4321fedc e39d6300
[  785.245164] dfc0: 043d6954 00000000 00000000 00000058 bee76c68 00000000 bee76c00 043d56f8
[  785.253334] dfe0: 00000058 bee76b5c aedaad45 aed2c746
[  785.258381] ---[ end trace 0000000000000002 ]---

I found another topic mentionning this warning on post #14.

The warning does not seem related to loaded modules because even when there are not any loaded module mentionned in the debug, the warning appears.

Because I did not see this warning using Olimage I was wondering how it was solved.

If it can be usefull, some informations from u-boot:
U-Boot SPL 2021.04-00009-gafdc7e9cb7-dirty (Jun 02 2021 - 11:21:25 +0200)
DRAM: 1024 MiB
CPU: 912000000Hz, AXI/AHB/APB: 3/2/2
Trying to boot from MMC1


U-Boot 2021.04-00009-gafdc7e9cb7-dirty (Jun 02 2021 - 11:21:25 +0200) Allwinner Technology

CPU:  Allwinner A20 (SUN7I)
Model: Olimex A20-OLinuXino-LIME2
I2C:  ready
DRAM:  1 GiB
MMC:  mmc@1c0f000: 0
Loading Environment from FAT... *** Warning - bad CRC, using default environment

LubOlimex

I can confirm the kernel panic during poweroff, it also happens here. I've notified the developers to look into it.
Technical support and documentation manager at Olimex

vazhnov

#6
The same with fresh Armbian images. Board: Olimex A20-OLinuXino-LIME2.

Old image "Armbian_21.05.1_Lime2_buster_current_5.10.34.img" works fine and does poweroff.

_     _                  ____
| |   (_)_ __ ___   ___  |___ \
| |   | | '_ ` _ \ / _ \   __) |
| |___| | | | | | |  __/  / __/
|_____|_|_| |_| |_|\___| |_____|

Welcome to Armbian 21.05.1 Buster with Linux 5.10.34-sunxi

System load:   6%           Up time:       24 min Local users:   2
Memory usage:  7% of 998M   IP:        192.168.1xx.yy
CPU temp:      43°C           Usage of /:    35% of 3.5G
Battery:       60% discharging
RX today:      12 KiB

root@lime2-buster:~# cat /proc/cmdline
root=UUID=1c515dac-00f8-4ffd-85f7-36c362e6b710 rootwait rootfstype=ext4 console=ttyS0,115200 console=tty1 hdmi.audio=EDID:0 disp.screen0_output_mode=1920x1080p60 consoleblank=0 loglevel=1 ubootpart=952f5bc3-01 ubootsource=mmc usb-storage.quirks=0x2537:0x1066:u,0x2537:0x1068:u   sunxi_ve_mem_reserve=0 sunxi_g2d_mem_reserve=0 sunxi_fb_mem_reserve=16 cgroup_enable=memory swapaccount=1

root@lime2-buster:~# uname -a
Linux lime2-buster 5.10.34-sunxi #21.05.1 SMP Thu May 6 20:13:21 UTC 2021 armv7l GNU/Linux


Fresh "Armbian_22.02.1_Lime2_bullseye_current_5.15.25.img" works fine, but has kernel panic when doing poweroff:

Quote from: undefinedNo atomic I2C transfer handler for 'i2c-1'
...
mv64xxx: I2C bus locked, block: 1, time_left: 0

Also, I created my first fresh build with https://github.com/armbian/build , "Armbian_22.08.0-trunk_Lime2_bullseye_current_5.15.57_minimal.img", and it has the same issue.

I found this [BUG] sun4i: axp209: no atomic i2c transfer handler but there is no solution.

Workaround

Use older Linux kernel.

Armbian has possibility to create image with "legacy" kernel version. I used these options:
./compile.sh BOARD=lime2 BRANCH=legacy RELEASE=bullseye BUILD_MINIMAL=yes BUILD_DESKTOP=no KERNEL_ONLY=no KERNEL_CONFIGURE=prebuilt COMPRESS_OUTPUTIMAGE=sha,gpg,img

With this image, my Olimex A20-OLinuXino-LIME2 has no problem with powering off.

markus2330

I get the same error message but also during runtime. From time to time the i2c blocks/locks up/doesn't reset on one of my devices out of the blue (probably the fault of a Mod-IO2 connected, which needs a power-off afterwards). If this situation occurs (it does every few days) on any attempt of i2c communication I get
EIO. The kernel messages issued are the same, so I thought it is the same issue:

[   15.725811] i2c i2c-2: mv64xxx: I2C bus locked, block: 1, time_left: 0
[ 2705.668308] i2c i2c-2: mv64xxx: I2C bus locked, block: 1, time_left: 0
[ 3715.536689] i2c i2c-2: mv64xxx_i2c_fsm: Ctlr Error -- state: 0x2, status: 0x0, addr: 0x21, flags: 0x0
[ 3718.621965] i2c i2c-2: mv64xxx: I2C bus locked, block: 1, time_left: 0

Any plans that the atomic i2c transfer handler gets implemented?

Is there also some way to install the old kernel without creating a new image?

markus2330

#8
I could improve i2c locks at runtime for me by reducing the i2c bus speed. The overlay is part of https://opensesame.libelektra.org/ (files/i2c_2_clock_freq_overlay.dtbo see also doc/I2C.md).