A long tale with a happy ending...
When we launched Inky pHAT around six months ago, we were delighted with the reception that it got. It was the first widely available three-colour e-paper display add-on for the Raspberry Pi, and people were taken immediately by its beautiful display, and the cute artwork and example code designed by Lydia and Phil.
Our usual setup for functionally testing pHATs and HATs during development (after the detailed electrical testing that Niko and Jon carry out) is a Raspberry Pi 3 with Mini Black HAT Hack3r, or now pHAT Stack, attached via a ribbon cable. This was exactly how we’d been testing Inky pHAT.
Our circuitry on Inky pHAT mirrored what was on the development board that we'd got from the manufacturer, and we'd followed the protocol information provided by the manufacturer for driving the display.
Our usual development cycle is to plan and prototype the boards on breadboards if possible, then design the board layout in Eagle while Paul and/or Lydia work on the silkscreen, order a prototype panel from Eurocircuits, and put together and test several of these prototypes. Phil then roughs out some test software, and Niko, Jon, and Phil test the hardware to make sure that everything works as it should. Assuming no hitches, then we'll order a full batch of production panels and go into production. If we come across any problems with the prototypes, then we'll redesign as necessary, go through another spin of prototypes, and repeat that process until we're 100% happy to proceed to production.
Everything had gone swimmingly with Inky pHAT, but within weeks of launch, things got... murky...
A Brief Aside on E-paper Displays
E-paper displays work similarly to real paper and ink in a lot of ways. The displays have two or more coloured pigments that can be pulled up to the top of the display by applying voltages across the display. This process of pulling the pigments around is done sequentially for each pigment, pulling them all down, then pulling each one up to the top in the parts of the display that you want.
The first e-paper displays, like those in Kindle e-readers, used just black and white pigments (the white is a distinct pigment rather than just the colour of the display). E-paper displays with more than two colours, like the three colour red, black, and white one we use on Inky pHAT, are still relatively new technology, but you'll see them in use in some large retailers as price tags on display units that can be updated over the air.
If you look closely, the displays have a honeycomb mesh covering the display that is used to move the pigments into particular areas. The original Inky pHAT display has electrodes in one spot at the bottom right corner of the display, when mounted on Inky pHAT. The white rubber potting compound at the top right of the display covers the chip that's talked to via SPI by the Pi.
Blurred lines
The first reports that we got of damage to the Inky pHAT displays varied from single black lines horizontally across the display, to larger blurry horizontal groups of lines. This damage seemed to be permanent, and would get progressivley worse the more the displays were refreshed.
As soon as we saw these issues, we began investigating. We set up banks of five Inky pHATs on pHAT Stacks connected to Pi Zero Ws, refreshing continuously , and were able to replicate the damage seen. The code that Phil wrote, wrote the refresh cycle number to the display so that we could keep track of when the damage was appearing.
There was little pattern to how the damage appeared. Five Inky pHATs, all being driven by the same Pi and updated at the same time would show damage after different numbers of refreshes, some would degrade rapidly, some would degrade slowly.
The displays are rated for 100,000 cycles, according to the datasheet, so the fact that damage was appearing after significantly fewer cycles was obviously not normal.
The only consistent pattern was that there was a 100% failure rate with these displays. Because the displays took around 15 seconds to refresh, and damage was only appearing consistently after several thousand refreshes, investigating these problems took... a while!
After being sure that i) the damage was irreversible and, ii) it didn't seem to be an isolated problem, we took the decision to halt sales of Inky pHAT, on July 12th, two to three weeks after launch.
Investigations Continue...
Our aim was to get Inky pHAT back on sale as soon as possible, so we came up with a few hypotheses about what the problem might be caused by, ways in which to test these, and ways to fix these potential problems.
Our hypotheses included:
- electromagnetic interference from the Wi-Fi and Bluetooth on the Pi
- some sort of photo-electric effect on the display circuitry
- abnormal voltages being applied to the display via the board circuitry
- temperature effects
For each of these hypotheses, we tested them in scientific fashion, with three banks of five displays, each bank testing an individual variable.
For instance, for the electromagnetic interference hypothesis, we had five Pi Zeros (no-W, without Wi-Fi and Bluetooth), five Zero Ws connected to Wi-Fi and pinging an IP address constantly, and five Zero Ws with Wi-Fi enabled and a piece of ESD foil shielding on the underside of the Inky pHAT boards.
This first test tantalised us, because the non-W Pi Zeros only began to show damage after the Pi Zero Ws, but all fifteen did eventually succumb.
Electromagetic interference
To eliminate the possible effect of electromagnetic interference, the tests from here on in used non-W Pi Zeros.
We hypothesised that there might be a photoelectric effect after realising that we'd been carrying out all of our tests thus far next to a west-facing window that got a significant amount of sunlight. To test this one, we simply covered up one bank of the Zeros. They all failed again, so we ruled out photoelectric effect.
Interestingly, we found that a flashgun fired next to the display circuitry completely destroys the display in a much more dramatic fashion than the damage we'd been seeing!
(Note that the damaged lines had appeared before the more dramatic damage on the right hand side, and after more than 3 refresh cycles!)
Photoelectric effect
To test the voltages being applied to the display, Phil and Niko scoped the signals being sent and compared them to those from the dev. board. There were some voltage spikes on Inky pHAT that we didn't see with the dev. board, and we noticed that the inductor on the dev. board was a slightly higer-specced one than on Inky, so we changed this out on another bank of five Inky pHATs. Although they did survive slightly longer, they all failed again.
Abnormal voltages
The MCU chip on the e-paper displays has a built-in temperature sensor that can compensate for ambient temperature differences, so we mulled the possibility that temperature might have a damaging effect on the way that the displays were being driven.
Remember that we said that we were carrying out these tests next to a window? Oddly, we'd noticed that the displays behaved really weirdly within seconds of the window being opened! The images would look washed out, there would be a sort of checkerboard effect on some, and some would even show the lines of damage that we'd already been seeing almost instantly. Stranger still, the "damage" seemed to be transient, and would disappear once the displays were back at a stable room temperature.
We did notice that the damage on our test displays would often appear during the night, and when we checked first thing in the morning there would be several showing damage that had been fine the previous evening.
Our Best Guess
Our best guess was that running the displays at a lower temperature for a prolonged period would cause the transient damage that we saw with sudden temperature drops to become permanent, and that once that initial damage had occured then the displays were past the point of no return and would continue to degrade.
To sum up our findings:
- temperature had a definite effect, although we weren't certain whether it caused permanent damage
- electromagnetic interference from Wi-Fi and Bluetooth had no effect, as far as we could see
- extreme photoelectric shocks to the display circuitry caused dramatic damage
- abnormal spiky voltages caused damage to appear more quickly, but removing them did not prevent damage
We also tried some alternate software, provided by the display manufacturer, with whom we'd been in contact throughout our testing. This didn't help either.
Solution: New Everything!
After several weeks of testing, trying to find a solution that would allow us to continue using the same Inky pHAT boards and displays, we decided that the best approach was to switch to an alternate display from a different manufacturer.
New Display
We'd found another red, black, and white e-paper display of the same size and with the same pinout as the previous one, but with a different MCU and electrode layout on the display (the two electrodes were separated at the top right and bottom right of the display respectively). We ordered some of these new displays to test, and went through the same process as with the original displays, prototyping, testing, etc.
New PCB
We redesigned the power circuitry on Inky pHAT board to use a larger inductor to protect against spiky voltages, and Phil completely re-wrote the Inky pHAT library to support the new display.
New Software
The new displays use an almost completely different method to drive them in software, with a look-up table that allows you to tweak the way that the different colours are driven. We spent a significant amount of time playing with these to get the image to look as good as possible.
We also managed to take advantage of the fact that the busy pin, used by the displays to tell the Pi that they're ready (or not), used different logic between the first and second displays. This meant that, on initialising the display in software, we could reset it and then look at the state of this pin to determine which display was being used.
The end result is that the library should function identically whether you're using either display and automagically detect the display type and use the correct protocol for that display.
Re-releasing Inky pHAT into the Wild!
Needless to say, we tested these new displays exhaustively! The really good news is that they passed these stress tests with flying colours!
After being completely satisfied that these new displays, boards, and software were as solid as could be, we put a small number (around 70) of Inky pHATs back on sale on December 6th, with the intention that this first new batch would allow us to further check for any unforeseen issues in the wild.
We're expecting a much larger batch of new displays to arrive around the end of December, so we should have a significantly larger batch on sale in the New Year.
We're sorry it's taken so long, and we hope that you understand from everything written above why it's taken a while to get Inky pHAT back on sale, but we hope that you fall in love with Inky pHAT all over again!
And, of course, we'll be more than happy to replace any damaged first generation Inky pHATs with new ones. Please get in touch with support@pimoroni.com.