Date suddenly changed to 2114

Started by mossroy, October 16, 2019, 12:42:30 pm

Previous topic - Next topic

mossroy

I have 3 A64-OLinuXino-2Ge8G-IND devices, using them as headless debian servers.

Yesterday, one of them became unreachable from the local network.
After using the debug cable, I saw that its date had suddenly changed to 2114 October 7, 05:20:08.
Manually fixing the date resolved the issue. For some reason, this wrong date made a few services to fail (even after a reboot) : at least NetworkManager and MariaDB.

I had the stack below in the logs, at this precise moment.
I'm using ftp://staging.olimex.com/Allwinner_Images/a64-olinuxino/linux/1.latest_images/buster/images/Armbian_5.92.1_Olinuxino-a64_Debian_buster_next_5.2.5.7z , with all current updates. Kernel 5.2.5.

What happened? Is it a hardware or software issue?
It's very annoying as it can not be fixed through the network.

[2115903.263561] rcu: INFO: rcu_sched self-detected stall on CPU
[2115903.263579] rcu:   2-...!: (3 ticks this GP) idle=302/0/0x1 softirq=20234276/20234277 fqs=0
[2115903.263583]        (t=750639755227 jiffies g=72093289 q=1)
[2115903.263589] rcu: rcu_sched kthread starved for 750639755227 jiffies! g72093289 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x402 ->cpu=0
[2115903.263591] rcu: RCU grace-period kthread stack dump:
[2115903.263595] rcu_sched       I    0    10      2 0x00000028
[2115903.263602] Call trace:
[2115903.263619]  __switch_to+0xb4/0x1b8
[2115903.263628]  __schedule+0x1f4/0x4a0
[2115903.263632]  schedule+0x28/0x98
[2115903.263638]  schedule_timeout+0x90/0x3a0
[2115903.263647]  rcu_gp_kthread+0x714/0x968
[2115903.263653]  kthread+0x124/0x128
[2115903.263657]  ret_from_fork+0x10/0x1c
[2115903.263662] Task dump for CPU 0:
[2115903.263665] swapper/0       R  running task        0     0      0 0x0000002a
[2115903.263669] Call trace:
[2115903.263673]  __switch_to+0xb4/0x1b8
[2115903.263678]  0xffff000010df8000
[2115903.263680] Task dump for CPU 1:
[2115903.263683] swapper/1       R  running task        0     0      1 0x0000002a
[2115903.263688] Call trace:
[2115903.263692]  __switch_to+0xb4/0x1b8
[2115903.263695]  0xffff000010df8000
[2115903.263697] Task dump for CPU 2:
[2115903.263699] swapper/2       R  running task        0     0      1 0x0000002a
[2115903.263704] Call trace:
[2115903.263709]  dump_backtrace+0x0/0x140
[2115903.263714]  show_stack+0x14/0x20
[2115903.263720]  sched_show_task+0xf4/0x128
[2115903.263726]  dump_cpu_task+0x40/0x50
[2115903.263731]  rcu_dump_cpu_stacks+0xc8/0x118
[2115903.263735]  rcu_sched_clock_irq+0xf8/0x7e8
[2115903.263740]  update_process_times+0x2c/0x58
[2115903.263747]  tick_sched_handle.isra.5+0x30/0x48
[2115903.263751]  tick_sched_timer+0x48/0x98
[2115903.263756]  __hrtimer_run_queues+0xfc/0x218
[2115903.263760]  hrtimer_interrupt+0xf8/0x2d0
[2115903.263766]  arch_timer_handler_phys+0x28/0x40
[2115903.263771]  handle_percpu_devid_irq+0x80/0x140
[2115903.263777]  generic_handle_irq+0x24/0x38
[2115903.263782]  __handle_domain_irq+0x5c/0xb0
[2115903.263786]  gic_handle_irq+0x58/0xa8
[2115903.263789]  el1_irq+0xb8/0x140
[2115903.263793]  arch_cpu_idle+0x10/0x18
[2115903.263797]  do_idle+0x1e0/0x2c0
[2115903.263801]  cpu_startup_entry+0x20/0x28
[2115903.263807]  secondary_start_kernel+0x190/0x1d0
[2115903.263810] Task dump for CPU 3:
[2115903.263812] swapper/3       R  running task        0     0      1 0x0000002a
[2115903.263817] Call trace:
[2115903.263821]  __switch_to+0xb4/0x1b8
[2115903.263824]  0xffff000010df8000

LubOlimex

Technical support and documentation manager at Olimex

mossroy

October 20, 2019, 01:03:24 pm #2 Last Edit: October 20, 2019, 07:28:27 pm by mossroy
Many thanks for this detailed answer.

To sum up :
- It comes from a hardware issue in the Allwinner A64 chip, and affects all boards using the A64 (not only from Olimex)
- Allwinner is informed but doesn't care much
- A software workaround is implemented in kernel 5.3, and might be backported in previous kernel versions if necessary. But it involves recompiling the kernel in any case.

Now that the kernel 5.3 is officially released, when will you release a new version of your images in ftp://staging.olimex.com/Allwinner_Images/a64-olinuxino/linux/1.latest_images/ ? And/or a kernel update in http://repository.olimex.com ?

LubOlimex

Technical support and documentation manager at Olimex

mossroy

Well, we're now Soon™ + one month...

I saw that kernel 5.3 has been released for a13 : https://olimex.wordpress.com/2019/11/06/new-mainline-linux-images-with-kernel-5-3-8-for-a13-olinuxino-and-a13-som-are-uploaded/

Any update for a kernel >=5.3 for A64 that fixes the date issue?