IRC #olimex 2015-02-24

[01:01:45] <adj_> nabblet, i'm sure that the type of JST connector is in olimex irc log
[01:04:42] <adj_> http://mail.olimex.com/irc?date=2014-11-11
[01:15:45] <lukas2511> in the end i bought those: http://www.ebay.de/itm/JST-2-0mm-PH-2-Pin-Connector-with-Wire-x-20-Sets-/181441351432 they fit but are backwards (red/black - can be easily fixed)
[11:27:11] <TJvV> ping Tsvetan
[11:39:26] <nabblet> adj_: nice one! And it was actually you again who helped back then :)
[11:41:55] <TJvV> anyone happen to be well versed in u-boot?
[11:42:42] <TJvV> got an issue where the new Rev. G boards with the Samsung modules fail to find the kernel half the time because of CRC failure
[11:43:12] <TJvV> (A13-OLinuXinio board)
[13:31:42] <ssvb> TJvV: which version of u-boot is that?
[13:34:53] <ssvb> TJvV: you can try to compile the mainline u-boot yourself, and apply tweaks to the A13-OLinuXino_defconfig file http://git.denx.de/?p=u-boot.git;a=blob;f=configs/A13-OLinuXino_defconfig;h=806d5b7bd5346f93518494393324581c2809e0cc;hb=HEAD
[13:35:20] <ssvb> TJvV: first try to change CONFIG_DRAM_EMR1 to 4
[13:36:14] <ssvb> TJvV: also add the "+S:CONFIG_DRAM_TIMINGS_DDR3_800E_1066G_1333J=y" line at the end there
[13:37:08] <ssvb> TJvV: and tweak CONFIG_DRAM_CLK up and down in 24MHz steps
[13:37:50] <ssvb> TJvV: once you get the kernel booting without problems, move on to http://linux-sunxi.org/Hardware_Reliability_Tests#DRAM
[13:38:52] <TJvV> ssvb: ssvb U-Boot SPL 2015.04-rc2-00015-g1320112
[13:39:34] <TJvV> I'm not 100% sure on the configuration options, I just got the .bin from Olimex
[13:40:01] <ssvb> TJvV: looks good, it means that you can recompile and replace u-boot without any need to apply other changes to the SD card
[13:40:27] <TJvV> I'm still investigating, but it seems Rev. F with the hynix memory gives different memory problems
[13:40:55] <TJvV> haven't seen that revision fail on finding the kernel, but it does give a kernel oops
[13:41:55] <ssvb> TJvV: Rev. F with hynix memory also using the mainline u-boot or something else?
[13:42:27] <ssvb> TJvV: if both boards fail, could it be some other problem? for example your PSU?
[13:42:43] <TJvV> same uSD, same peripherals
[13:43:32] <ssvb> TJvV: can you recompile u-boot yourself?
[13:44:44] <TJvV> I've got the sources and found the A13-OLinuXino_defconfig; am I correct in assuming they shipped it without other options configured?
[13:45:07] <ssvb> TJvV: I'm afraid that we have no clue
[13:45:15] <TJvV> fun fact: Rev. F with samsung memory seems to work
[13:46:26] <ssvb> but the whole point is is that the users of this board would benefit from it being properly supported in the mainline u-boot
[13:46:48] <ssvb> instead of the third-party u-boot forks
[13:48:21] <TJvV> I'll give the lima-memtester a go on the boards that will boot
[13:48:21] <ssvb> TJvV: so please forget about the existing olimex binary and try to compile u-boot yourself :)
[13:50:08] <ssvb> btw, can you run https://github.com/maxnet/a10-meminfo on it first?
[13:50:58] <ssvb> is should retrieve the olimex dram settings and we can compare them with the mainline u-boot defaults
[13:51:05] <TJvV> ok
[13:51:18] <TJvV> doesn't matter that it specifies a10?
[13:51:58] <ssvb> a10/a13/a20 have minor variations of the same dram controller
[13:55:38] <TJvV> will report back in a min with the meminfo output of the different boards
[13:55:51] <ssvb> thanks
[14:06:03] <TJvV> it's almost like the Rev G's know I want to test them, they all refuse to start up now :P
[14:09:10] <TJvV> just hit a different Oops with Rev G; alg: hash: Failed to load transform for md4-generic; kernel BUG at crypto/api.c:109
[14:09:55] <TJvV> it's funny how many issues I'm hitting here...
[14:11:52] <ssvb> are you sure that you have a reliable power supply?
[14:12:15] <TJvV> using the DC adapter
[14:12:58] <TJvV> anyways, got the meminfos from Rev E (hynix), Rev F (samsung) and Rev G (samsung)
[14:13:13] <TJvV> want me to try the lima-memtester on the Rev G while it's up?
[14:15:14] <TJvV> according to diff all three are identical
[14:16:26] <ssvb> can you pastebin them?
[14:16:39] <TJvV> in a min
[14:16:55] <ssvb> also I assume that it is from u-boot 2015.04-rc2, right?
[14:17:05] <TJvV> yep
[14:17:08] <ssvb> ok
[14:17:19] <TJvV> these are done with the one Olimex gave me
[14:17:29] <TJvV> not the self-compiled version yet
[14:17:50] <ssvb> that's fine
[14:18:58] <ssvb> and running lima-memtester should just verify whether this dram setup is reliable, because lima-memtester is a bit more though than just booting the system
[14:19:57] <TJvV> http://pastebin.com/hfRaUfKf
[14:22:00] <ssvb> ok, I see, so olimex is still using the allwinner magic dram settings there
[14:22:13] <TJvV> sure looks like it
[14:23:34] <ssvb> so would you try the defconfig tweaks that I suggested earlier?
[14:23:45] <TJvV> not yet
[14:25:52] <TJvV> ok, is there a way to include an existing configuration into a u-boot configs file?
[14:26:14] <TJvV> would rather include and then override than mess with the existing config
[14:31:44] <ssvb> TJvV: you would edit the config, confirm that it works better for you and submit a patch for u-boot
[14:32:31] <TJvV> for now I copied the A13_Olinuxino_defconfig and edited the copy; will let you know what I find out
[14:33:02] <ssvb> thanks!
[14:33:04] <TJvV> should probably be easier to diff too
[14:33:29] <TJvV> what's the best way to determine it works better?
[14:33:54] <ssvb> if it boots instead of failing to load the kernel, then it is better
[14:34:15] <TJvV> it's still a bit random on whether or not it boots
[14:34:18] <ssvb> if it boots and able to pass the lima-memtester test, then it is likely to be reliable
[14:35:50] <TJvV> hmm
[14:35:51] <ssvb> the readme at https://github.com/ssvb/lima-memtester/ explains it
[14:36:07] <TJvV> getting invalid arguments on ioctl calls to /dev/fb0
[14:36:59] <ssvb> do you get a rotating cube anyway? or it does not start?
[14:37:12] <TJvV> getting interleaved rotating cube and terminal sayin invalid arguments
[14:37:39] <ssvb> can you pastebin these messages?
[14:38:30] <ssvb> but if the cube is rotating, then the test works more or like as expected
[14:39:28] <ssvb> if hardware problems are detected, then it will either stop spinning or you will see a red glowing background picture
[14:40:45] <ssvb> I will be offline by an hour or so, but will check the irc logs
[14:41:07] <TJvV> http://pastebin.com/aY7wFReb
[14:42:31] <ssvb> hmm, that's strange
[14:44:09] <TJvV> checking my kernel config for possible ioctl/fb configurations
[16:09:32] <adj_> nabblet, thats the reason because I remembered it, I don't have all the logs in my head! ;)
[16:45:28] <TJvV> ssvb: welcome back
[16:46:09] <TJvV> did some more testing and it seems that the 2G image I obtained from Olimex does (seem to) work on the boards that wouldn't boot
[16:46:47] <TJvV> it does have an earlier version of u-boot than the one in my own image
[16:46:49] <ssvb> TJvV: thanks, that's good to know
[16:46:54] <ssvb> got a meminfo log from it?
[16:47:05] <TJvV> not yet
[16:47:27] <ssvb> the meminfo log might explain the differences
[16:48:45] <TJvV> will have it in a min, in the mean time I going through the changes I made to everything even remotely relevant to start-up and/or memory
[16:50:38] <TJvV> is it possible to determine with which configuration options u-boot was compiled?
[16:52:22] <TJvV> seeing quite some changes compared to the others
[16:52:40] <TJvV> this image has a CAS latency of 7 instead of 9
[16:54:40] <ssvb> cool, can you pastebin it?
[16:54:53] <TJvV> the dram_tprX values are different as well and emr2 is set to 8 instead of 10
[16:55:02] <TJvV> sure, just give me a min :)
[16:56:24] <TJvV> http://pastebin.com/Q0h60SrJ
[17:18:57] <ssvb> TJvV: these timings look like http://git.denx.de/?p=u-boot.git;a=blob;f=board/sunxi/dram_timings_sun4i.h;h=29b934da639c86b1830d1e26b7a751b6b227a21a;hb=HEAD#l22
[17:19:02] <ssvb> DDR3-1066F @408MHz, timings: 7-6-6-16 */
[17:21:30] <ssvb> TJvV: it means that they have added "+S:CONFIG_DRAM_TIMINGS_DDR3_1066F_1333H=y" to defconfig
[17:22:25] <ssvb> TJvV: you can still try to change emr1 to 4 and experiment with dram overclocking headroom on this board, using lima-memtester as a reliability verification tool
[17:22:49] <TJvV> I actually have that in my own version of u-boot, without success
[17:23:24] <TJvV> maybe I should grab the u-boot version they used in their image
[17:23:43] <TJvV> 2015.04-rc1-dirty
[17:28:28] <TJvV> the +S:CONFIG part that is, I removed the EMR1 setting
[17:28:41] <ssvb> TJvV: oh, also be sure to set the old kernels compatibility option if you are building u-boot yourself
[17:29:21] <ssvb> TJvV: CONFIG_OLD_SUNXI_KERNEL_COMPAT=y
[17:30:22] <TJvV> got it via menuconfig :)
[17:30:35] <ssvb> if you are building u-boot from the latest upstream git, it will even not allow you to boot the old kernel without this option set
[17:31:05] <ssvb> TJvV: have you run menuconfig twice (once for the SPL and once for the main u-boot binary)?
[17:31:55] <ssvb> TJvV: it is always safer to edit the defconfig :)
[17:33:11] <TJvV> consider it added :)
[17:33:44] <TJvV> just to be on the safe side, with or without +S: ?
[17:33:56] <ssvb> with +S:
[17:34:11] <TJvV> good
[17:34:45] <ssvb> this config option needs to be applied to the SPL, because that's where PLL5 is configured
[17:38:56] <TJvV> seems it still takes me to kernel oops land... will pastebin my config
[17:40:00] <TJvV> http://pastebin.com/z4Zi48QD
[17:42:52] <ssvb> TJvV: can you also pastebin the log with the kernel oops?
[17:44:34] <ssvb> TJvV: also with the working u-boot from olimex, after the system is booted, you can try to run "dmesg | grep buck" and check the dcdc3 voltage
[17:45:07] <ssvb> TJvV: if they have increased the dcdc3 voltage, this could explain better reliability
[17:51:18] <TJvV> ssvb: it's a bit difficult to capture the oopses in the kernel, but I think they have the same problem as u-boot when the kernel CRC fails
[17:53:13] <ssvb> yes, the problem is that the dram is unreliable and may fail any time
[17:53:45] <ssvb> TJvV: don't you get these oopses on the serial console on your PC?
[17:54:02] <TJvV> yeah, but minicom spaces them out really weird
[17:54:21] <ssvb> TJvV: try screen instead, it is much nicer than minicom
[17:54:33] <ssvb> "screen /dev/ttyUSB0 115200"
[17:54:51] <ssvb> or whatever /dev/tty*
[17:57:15] <TJvV> think I've got a nice one here
[17:59:25] <TJvV> http://pastebin.com/Kwy4UyPt
[18:02:25] <ssvb> TJvV: "axp20_buck3: 700 <--> 3500 mV at 1200 mV" <- this is very low voltage
[18:02:44] <TJvV> also noticed the incomplete constraints messages?
[18:03:12] <ssvb> TJvV: I think, u-boot should configure dcdc3 to at least 1.25V
[18:03:49] <ssvb> TJvV: but this can be overrided by fex
[18:04:26] <ssvb> TJvV: so knowing the dcdc3 voltage value from the working image would be very interesting
[18:05:19] <TJvV> just checked, it showed the same 1400 and 1200 mV, right before crashing....
[18:06:15] <ssvb> TJvV: are you sure that you are using the right defconfig when building u-boot? because 1.2V is the reset default, which might be there if you don't configure AXP209
[18:06:57] <TJvV> like I said, I hijacked the A13-OLinuXino_defconfig, one would assume it would suit the board
[18:07:45] <TJvV> also got the AXP209_POWER defined in my custom defconfig
[18:08:19] <ssvb> TJvV: please double check this, if you have AXP209_POWER defined, then u-boot should set dcdc3 to 1.25V
[18:08:48] <ssvb> TJvV: and also check your script.bin and fex
[18:09:21] <ssvb> TJvV: if the script.bin file enforces dcdc3 to 1.2V, then the reliability goes south
[18:09:49] <TJvV> the fex sets it to 1400 and 1200, in both images
[18:10:25] <TJvV> in the image I got from Olimex, I had to generate the fex from the .bin and haven't changed it afterwards
[18:12:03] <ssvb> dcdc3 at 1.2V is very low, and you are expected to have troubles using it
[18:12:38] <ssvb> TJvV: is the properly working new image from olimex also using 1.2V for dcdc3?
[18:12:47] <TJvV> that is what it said
[18:13:12] <TJvV> now writing a fresh version of the image to exclude any mixups
[18:14:16] <ssvb> that's weird, then I'm just curious why it does not fail, this looks like a miracle :)
[18:16:16] <TJvV> I'm great at attracting such miracles
[18:17:12] <ssvb> it's a good idea to increase the dcdc3 voltage in fex to 1250 or even 1300
[18:17:40] <TJvV> update: newly written 2G image from olimex: dcdc2 = 1400, dcdc3 = 1200
[18:18:04] <TJvV> and I have to go soon, but will be able to pick up on this tomorrow
[18:18:20] <TJvV> will check if things go better at 1250
[18:19:16] <ssvb> TJvV: can you try a10-meminfo from this repository? https://github.com/ssvb/a10-dram-tools
[18:19:39] <TJvV> dcdc2 @ 1400 is good?
[18:19:43] <ssvb> TJvV: one more possible explanation is that olimex is using a very low MBUS clock speed to keep up with low DCDC3 voltage
[18:19:54] <ssvb> TJvV: dcdc2 is used for the CPU
[18:20:41] <ssvb> TJvV: you can in fact also configure the MBUS clock speed in the defconfig
[18:21:43] <ssvb> TJvV: "+S:CONFIG_DRAM_MBUS_CLK=204"
[18:21:54] <ssvb> TJvV: instead of the default 300MHz
[18:22:16] <TJvV> failed on the first try @1250
[18:22:24] <ssvb> hmm
[18:22:50] <TJvV> second try gives kernel panic
[18:23:51] <TJvV> to clarify: first try failed at CRC
[18:24:30] <ssvb> that's purely u-boot related, script.bin is taken into use much later
[18:24:38] <ssvb> please try lower MBUS clock speed
[18:24:47] <ssvb> this would probably help
[18:26:59] <TJvV> have to go now; will check MBUS speed and dram-tools tomorrow
[18:27:11] <TJvV> thank you for your help so far
[18:27:34] <ssvb> you are welcome