“EC Card 2”, a closed-case debugging (CCD) Expansion Card for the Framework Laptop

DISCLOSURE

This expansion card prototype was funded by Framework Computer through their Expansion Card Developer Program.

Closed-Case Debugging (CCD)⌗

A day will dawn in every person’s life where they do something completely stupid. For some of us, that something will take the form of bricking (or very nearly bricking) our daily carry laptop.

A few of those somethings will be caused by hacking our laptop’s embedded controller¹, and a few of those will be caused by a compiler upgrade² (more on this later). If you find yourself facing either of these two things you might be able to recover – or at least diagnose! – your machine the old fashioned way.

Closed-Case Debugging (CCD) is pretty straightforward: you get access to a debug port without cracking open your machine. That debug port on the “device under test” (DUT) might be accessible through a headphone jack, four weird magnetic pins on the side, a USB A port with some questionable standards compliance, etc.

Chromebooks all³ have one! Some phones do, too.

In short, it’s so that you don’t have to do this:

Framework Laptop 13⌗

On the Framework Laptop 13⁴ the rear right USB C port doubles as a CCD port, offering UART access to the embedded controller. It also just so happens that the embedded controller is very chatty and provides a full debug console.

This debug port works any time the embedded controller is on, which means that you can use it to debug the power sequence from another computer while your DUT is “turned off.”

Connect CC1 and CC2 through one 5.1kΩ resistor each to GND.

SBU1 and SBU2 will switch over to EC TX and EC RX respectively.

Debug port parameters:

• 3.3V TTL
• 115200 8N1
• No flow control

This configuration is only usable in one orientation.

A	1	2	3	4	5	6	7	8	9	10	11	12
signal	GND	nc	nc	nc	CC1 -5.1kΩ-> GND	nc	nc	SBU1 EC TX	nc	nc	nc	GND
signal	GND	nc	nc	nc	SBU2 EC RX	nc	nc	CC2 -5.1kΩ-> GND	nc	nc	nc	GND
B	12	11	10	9	8	7	6	5	4	3	2	1

receptacle pinout

Unlike the aforementioned “USB A port with questionable standards compliance,” this is entirely compliant: the SBU (or Side-Band Use) pins are allocated for exactly this purpose! Okay, they’re actually allocated for an implementation-defined purpose. That’s the point. For this implementation, they’re allocated for this purpose. Don’t at me.

Existing solutions⌗

The existence of dedicated CCD in phones, laptops, and other devices begat a number of devices to fill the need. Among those is Google’s Servo (now on v4). New Servo boards are almost unobtainable by the general public. They are, I’m told, highly flexible.

I do not own a Servo, so I cannot comment on its effectiveness.

You could also use a simple USB C breakout (make sure to get one that breaks out every pin and doesn’t include any CC resistors out-of-the-box).

EC Card 2⌗

Not satisfied with the unavailability of a Servo and annoyed that I’d bought a couple USB C breakouts that, while they broke out SBU1/2 and CC1/2, couldn’t be coaxed into working properly⁵, I set out to design my own CCD breakout.

Because it’s for the Framework Laptop, I wanted to make it an expansion card. Furthermore, I wanted it to be totally self-contained and not require an external USB<->UART interface.

I settled on an FT230X and started with Framework’s Expansion Card reference design.

And wow was it ever ugly! My first pass at laying out a board is not at all worth sharing, but suffice it to say that I trusted the auto-router too much and hell, I didn’t even lay a ground plane. That was EC Card 1. It never reached production thanks to the fine folks over at Framework telling me that I had done a bad job and should feel bad⁶.

EC Card 2 turned out better.

I chose PCBway for prototype production and assembly⁷. They would not assemble the USB C connectors, however, so I did that on my own.

Combined with some expansion card shells 3D-printed at the library, it looks quite nice!

The design is open source at DHowett/framework-ec-debug-card (HowettNET GitLab) .

When you’re using it, you get serial output directly from the embedded controller.

The staggered pin header you see in the original layout was inspired by SparkFun’s locking headers. I intended for it to retain a 2.54mm pin header in place as a debug probe, but it doesn’t hold. It is still useful in that it lets you repurpose the board as a simply FTDI carrier or access SBU1/SBU2 without USB serial.

Known issues⌗

There are some!

In addition to that:

• The microUSB port hangs over the edge of the card, and it is sharp!
• There’s a minor power leakage from the pull-up on RX by the FT230X into the EC, which can power it when it is not supposed to be powered. The MEC1521 seems to be guarded against this causing any damage, but it does mean that on occasion the EC starts up and sends over a bunch of garbage before you click the power button.
• FTDI chips are great, but maybe a bit over-featured and expensive.

A recovery story⌗

I authored a guide on hacking your laptop’s embedded controller, and so I led at least two people down a path that resulted in them rendering their laptops unbootable. That sucked. As far as I can tell, it was entirely due to some undefined behavior² that lay at the bottom of a well waiting for a well-meaning person to fall in.

I stumbled into this issue in exactly the same way anybody stumbles into anything: blindly. I reflashed my EC, rebooted, pressed power and pop. Nothing. A whirr of fans, and then all the lights went out.

The debug console told me that it was crashing only after I pressed the power button. The EC was running fine until then. And then, it was panicing with a watchdog failure.

Now I lucked out, because I had been working on bringing the RW firmware⁸ online. I could just reboot into a working copy… but those folks on the forum weren’t quite so lucky.

With access to the EC debug console in hand, and them with some access to UART as well, I built a payload that would overwrite the miscompiled code in memory as well as a build system to automatically produce that payload⁹.

It worked. It worked! Both of them recovered their machines.

However: I sent one of them an EC Card 2, and that didn’t work. Drat!

v3 plans?⌗

I don’t know if I’m going to make more of these, but if I do I’d like to fix a couple annoyances.

• nrp suggested using a USB C port instead of microUSB, so that it could be placed in a USB C expansion card shell.
• I should set back the microUSB port to address the 1mm overhang where the sharp edges stick off the edge of the laptop and can cut your hand to ribbons.
• Look at the voltage leak from the technician machine to the DUT, if possible¹⁰.

Makers: e-mail me!⌗

I still have some of these, as yet unassembled. If you’re into EC hacking and feel like you could benefit from one, reach out to me with a description of your project!