Olimex Support Forum

Microcontrollers => ESP32 => Topic started by: justlikeef on November 21, 2022, 02:39:51 AM

Title: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on November 21, 2022, 02:39:51 AM
I have two boards that I successfully flashed esphome to, but after running for several minutes, they will no longer boot and will not flash.

one of them crashes immediately after boot:
ets Jul 29 2019 12:21:46

rst:0x1 (POWERON_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:2
load:0x3fff0018,len:4
load:0x3fff001c,len:1044
load:0x40078000,len:10124
load:0x40080400,len:5828
entry 0x400806a8
[I][logger:258]: Log initialized
[C][ota:469]: There have


and the other does this over and over:
rst:0x10 (RTCWDT_RTC_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff
ets Jul 29 2019 12:21:46

rst:0x10 (RTCWDT_RTC_RESET),boot:0x1f (SPI_FAST_FLASH_BOOT)
invalid header: 0xffffffff
invalid header: 0xffffffff
invalid header: 0xffffffff



Attempting to flash either gives one of several errors:

A fatal error occurred: Failed to connect to ESP32: No serial data received.
Invalid head of packet (0x0D): Possible serial noise or corruption.




I have tried various baud rates and placing various combinations of the 10mf cap and 2K resister between en and 3.3 and en and ground.  I don't have the capability of replacing the surface mount resister, but I can work with someone to troubleshoot.

I have replaced the USB cable and tried a powered hub.  Putting a scope on the 5V pin looks pretty stable.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: LubOlimex on November 22, 2022, 09:02:10 AM
Software downloaded usually doesn't affect the ability to program via the USB. Try another USB port, try another software for programming. Maybe re-install the CH340T drivers, you can find suitable ones at the product page.

You don't need to solder around the EN, if you wish to have a boot button, we have provided a way to transform user button BUT to boot button. Unsolder resistor R47 and solder it on the pads of resistor R49. Refer to the schematic in the "Buttons" area to understand better.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on November 22, 2022, 11:52:32 PM
I have tried on multiple machines and they behave the same.

I don't have the capability of soldering the SMD components.  I am using the headers.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: LubOlimex on November 24, 2022, 09:07:47 AM
But did you try different software tool?

Maybe try with Arduino for ESP32, since it is well detailed how to install. Just try to upload anything to see if the upload works.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on November 28, 2022, 12:35:44 AM
I have tried the esphome web tool and the command line esptool.  I get a similar response from both.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: LubOlimex on November 28, 2022, 10:28:35 AM
Now that I look your older replies about forcing bootloader mode without soldering:

- you can force bootloader mode by connecting GPIO0 to GND (and if that doesn't work also connect GPIO2 to GND); this should act as having pressed BOOT button

- leave EN as it is, in this design it is down to GPIO0 state not to EN.

Both GPIO0 and GPIO2 can be found at EXT1, GPIO0 is pin #5. GPIO2 is pin #7. GND is pin #3.

Remember to leave forced bootloader mode you need to disconnect these connection afterwards.

Try to force this mode and if that allows for programming.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on November 30, 2022, 05:21:07 AM
I didn't have time to look at them today.  Will do so tomorrow EST.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on December 05, 2022, 03:20:53 AM
Finally figured out what is going on.  I have 5 boards that are now dead under the same circumstances.

They run on POE (ran one untouched for more than a day) until I solder headers on them. After soldering the headers on and plugging the ethernet cable back in, they run for a while and then stop responding.  The charge and power light stay on, but the link and activity light go off.

To troubleshoot the issue, I turned POE off on the switch and plugged the network and USB cables in.  If I let the board sit for a while (I am assuming something is cooling off), it will boot, and the link and activity light will come on for somewhat less than a second, then go back off, repeatedly.  After a while, the serial console starts printing garbage, then eventually locks up.  Let the board sit for a while, and the process will repeat.  If you don't let it sit (unplug the USB cable and then immediately plug it back in), there is nothing printed to the serial console.

Device logs (https://drive.google.com/file/d/1DUslVr4IZjdUQhz4dcpM7HEkE1pd3rGd/view?usp=sharing)
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on December 05, 2022, 03:33:42 AM
pictures  (https://photos.app.goo.gl/S78RzcGSDXStVKHE7)of one of the board with headers soldered on.

Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: LubOlimex on December 05, 2022, 01:18:35 PM
Interesting, so the problem appears after soldering the headers? Maybe you put too much solder and it went below and caused some short-circuit? Or maybe you heat the solder for too long and accidentally damaged some component?

There is nothing attached to the headers between the moment they were soldered and when the moment that appeared?

What happens if you desolder and remove the header?

Also you said you connect USB after failure happens on PoE, but were you careful not to have both USB powering and PoE powering attached to the ESP32-PoE at the same time?
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on December 05, 2022, 11:54:46 PM
Quote from: LubOlimex on December 05, 2022, 01:18:35 PMInteresting, so the problem appears after soldering the headers? Maybe you put too much solder and it went below and caused some short-circuit? Or maybe you heat the solder for too long and accidentally damaged some component?

Not out of the question, but I've soldered boards for 35 years and not had a consistent problem like this.

There are some things VERY close to the copper pads.  My guess is a bridge between GPIO0 and D9 or GPIO3 or 4 and the via that is in the middle of the GPIO4 label.  I'll find out for sure once I remove them.  Does a solder bridge to either of those make sense?

Quote from: LubOlimex on December 05, 2022, 01:18:35 PMThere is nothing attached to the headers between the moment they were soldered and when the moment that appeared?

Correct

Quote from: LubOlimex on December 05, 2022, 01:18:35 PMWhat happens if you desolder and remove the header?

Next on my list. 

Quote from: LubOlimex on December 05, 2022, 01:18:35 PMAlso you said you connect USB after failure happens on PoE, but were you careful not to have both USB powering and PoE powering attached to the ESP32-PoE at the same time?

Correct.
Quote from: justlikeef on December 05, 2022, 03:20:53 AMTo troubleshoot the issue, I turned POE off on the switch and plugged the network and USB cables in.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: LubOlimex on December 06, 2022, 08:35:45 AM
Notice something - these headers were meant to be soldered on the other side of the board (at the bottom). The idea was that these connectors can be used to place the board on top of another board. But we've also had headers installed like you did, on the top side for a customer that requested it, and it also worked fine. So the way you placed them on top shouldn't be an issue.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on December 06, 2022, 10:07:21 PM
I've added pictures (https://photos.app.goo.gl/S78RzcGSDXStVKHE7) of the board after removing the headers. I see no evidence of solder bridging or damaged components.  I have plugged it back into POE to see how it behaves now.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on December 07, 2022, 03:57:57 AM
The problem is on the EXT1 side.  Removing the EXT2 side made no difference.  Once I removed the EXT1 side, the problem goes away.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: LubOlimex on December 07, 2022, 04:24:53 PM
It looks like a there might have been lose solder forming a connection between one of the pads of the EXT1 connector and nearby components. Under the plastic of the connector. Around GPIO0, GPIO1, GPIO2 pads - things are not so clear, looks like traces of left out solder. Probably connection between GPIO0 and D9 or GPIO0/GPIO1 and the big C25. But D2 and R16 are also possible.

I tested it today. Got the same revision I board and tested it with the Ethernet code from the Arduino IDE package for ESP32 (File -> Examples -> Ethernet -> ETH_LAN8720). It connects to Google every ten seconds and reports something like:

13:15:58.483 -> connecting to google.com
13:15:58.671 -> HTTP/1.1 301 Moved Permanently
13:15:58.671 -> Location: http://www.google.com/
13:15:58.671 -> Content-Type: text/html; charset=UTF-8
13:15:58.671 -> Cross-Origin-Opener-Policy-Report-Only: same-origin-allow-popups; report-to="gws"
13:15:58.671 -> Report-To: {"group":"gws","max_age":2592000,"endpoints":[{"url":"https://csp.withgoogle.com/csp/report-to/gws/other"}]}
13:15:58.671 -> Date: Wed, 07 Dec 2022 11:15:57 GMT
13:15:58.671 -> Expires: Fri, 06 Jan 2023 11:15:57 GMT
13:15:58.718 -> Cache-Control: public, max-age=2592000
13:15:58.718 -> Server: gws
13:15:58.718 -> Content-Length: 219
13:15:58.718 -> X-XSS-Protection: 0
13:15:58.718 -> X-Frame-Options: SAMEORIGIN
13:15:58.718 ->
13:15:58.718 -> <HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
13:15:58.718 -> <TITLE>301 Moved</TITLE></HEAD><BODY>
13:15:58.718 -> <H1>301 Moved</H1>
13:15:58.718 -> The document has moved
13:15:58.718 -> <A HREF="http://www.google.com/">here</A>.
13:15:58.718 -> </BODY></HTML>
13:15:58.718 -> closing connection

So I left it working for a while. Then soldered EXT1 header, and left it working for an hour, it is still working fine. Then tried to reprogram it and it still programs fine. Here are the pictures on how I soldered it:

https://imgur.com/a/XU5qBs9

I believe you put too much solder and it flew under the connector and caused un invisible connection under the plastic.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on December 08, 2022, 12:24:25 AM
As you can see from the picture, my boards are red, so they are at least from a different run.  I'm wondering if there is something about these red boards.  Several people that are having power or similar stability problems have posted pictures of the red boards.

Please let me know what you would like clearer pictures of.  I see no evidence of solder bridging at all.  I will solder headers on to a new board and pull the plastic strip of if that is a question.

The board I pulled the headers off of has been running for 24 hours now without incident.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on December 08, 2022, 12:37:22 AM
Here (https://photos.app.goo.gl/Kv5txF1pVBWJfDrj7) is a closeup of the headers on a board having this problem.  I am testing it now to see how long it lasts.
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: LubOlimex on December 08, 2022, 08:23:02 AM
Good that you removed the plastic that would clear things up. I was going to suggest that. Let me know how testing goes.

I should have mentioned that we run prototypes before release in green silk screen (for in house testing and keeping), the color of the silk screen doesn't matter. Revision matters and the revision of the board that I used is the same as yours. We are currently at revision K of the boards and I couldn't find any older revisions made for customers (since we are out of revision I).
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: justlikeef on December 31, 2022, 11:13:16 PM
I ended up with a solder bridge on one board which caused the some strangeness after a few minutes.  That's the one that resolved when I removed the headers.

I also found that the BH1750 light sensors that I was using were burning out after a few minutes of use and causing the exact same symptoms.  Replacing the sensors with another has completely resolved the issue.

Here's to Murphy...
Title: Re: ESP32-POE will not reliably boot and unable to flash
Post by: LubOlimex on January 03, 2023, 09:03:18 AM
Glad these issues were resolved successfully back in 2022. Let's hope no such weird bugs would pop out in 2023!