Expansion interface: rules

Before Christmas Kevin and I discussed the expansion header rule-set that I posted last time. I ended up giving him a little clarification. What follows is a summary. Keep in mind that this is a work in progress. Once we finish the new expansion routing for rev C we will post a summary and use case examples.

Figure 1. A View of the MEC8-RA Expansion Header on Titan.

Trace Lengths and Matching All signal trace lengths must be matched to within 20mils and may not exceed 2000mils. Additionally each differential pair in the bank will be routed as a differential pair each pair's N and P net will be length tuned within 5mils. This tuning is intentionally over-kill to reserve as much margin as possible for the routing on modules.

TX and RX pair assignments All LVDS transmit pairs must be connected to the same side of the expansion connector. Figure 2 shows and mapping that meets this rule. This rule is included because the Samtec MEC8-RA connector as longer pins on one side of the connector. Figure 3 was taken from the MEC8-RA's STEP file. My rough measurements show that the top pins are about 100mils longer than the bottom pins. We can avoid accounting for this difference by routing the ECP5's true LVDS transmitter pairs on one side (labelled as pads A and B) and the receiver pairs (labelled as pads C and D) on the other side.

Figure 2. An TX/RX Mapping Example (TX on the Long Side, RX on the Short Side).

Figure 3. A Top and Bottom Pin Pair from the MEC8-RA.

DQ Pin assignments Each DQ group must be matched to the same set of diff pairs; Within the each DQ group the DQS pair must be on a fixed pin. Figure 4 shows an example DQ mapping. This rule means that the individual DQ pair assignments are not fixed and may be routed differently to ease routing.

Figure 4. An Example Mapping of the Two DQ Groups from an ECP5 Bank.

Other Fixed Pin Assignments The bank's VREF net and PCLK on C/D pads must be routed to fixed pins.

Expansion interface

We are finalizing our plans for rev C of Titan now. Over the last week Kevin has been working on our current change list: layout adjustments to ease manufacture, tighter tuning of the expansion header traces, and the addition of a 1.2V power supply for the SerDes (PCIe) transmitter and receiver. Kevin is working in his fork of the Titan project. Once he completes his changes I'm going to make a few adjustments to the DDR3 layout. When we are ready to order we will pull the changes back into the master branch of Titan on the C-E-S organization Github page.

While Kevin is updating the layout, I've been focused on validated our changes. Initially I was working on using the Diamond's Reveal embedded logic analyzer to identify the PCIe issues. Now that I know that there isn't a PCIe core in the samples we have, I'm focusing my efforts on validating the DDR3 memory design and the external interfaces.

Figure 1. Titan Rev B with Two Expansion Modules Installed.

Our expansion headers bring out a single bank of the ECP5 FPGA to a single connector. Now that we've had more time to consider possible expansion modules it has become clearer how we should organize the connections from the FPGA's I/O banks to the connectors. We are trying to balance keeping the layout simple and clean with our desire to standardize the expansion header pinout to make our future modules interchangeable from one Titan header to another.

Each FPGA bank that we have connected to an expansion header can be configured as 32 single-ended GPIOs, 16 differential GPIOs, two DDR data groups, or 7:1 geared interfaces (ie. CameraLink). Lattice's ECP5 High-Speed I/O Interface guide describes the interface possibilities. One specific interface that we are studying that has applications with SDR is the Linear Technology LTC217x quad ADC.

This leads me to my list of features that need to be the same from one expansion header to another. These assignments are based on the expansion differential pair nomenclature used on the rev B Titan schematic (see Figure 2).

Figure 2. Titan Expansion Header (showing our simple nomenclature).

Titan Expansion Header Features

• One PCLK pair must assigned to a fixed diff pair (ie. DO_P/N)
• Each of the two DQS pairs must be assigned to a fixed diff pair (ie D8_P/N and D12_P/N)
• Each DQ group must be matched to the same set of diff pairs (ie D0/1/2/3/9/10/11_P/N)
• VREF must be assigned to a fixed net (ie. D1_P)
• All LVDS transmit pairs must be connected to the same side of the expansion connector (ie. D0/1/2/3/4/5/6/7_P/N)
• All trace lengths must be matched to within 20mils and may not exceed 2000mils.

Now we'll see if I've given Kevin an impossible set of requirements. If I have I'm sure that I'll hear about it.

Stupid mistakes

Through my career I've had the pleasure of working with some intelligent and talented engineers and scientists. Along the way I've learned to listen when I'm given advice. I might decide not to heed their advice, but I always listen. The two most memorable nuggets of advice that I've received are:

1. Don't be a candy ass
2. Don't sweat the stupid mistakes, it's the smart mistakes that should worry you

Tonight I'm trying hard to remind myself of #2. The principle is simple; We all make silly mistakes, overlook things, or just forget something. As long as we identify our mistakes quickly and move on it shouldn't be a problem. The bad mistakes are ones where you focus on something for a long time and come to the wrong conclusion. These smart mistakes often mean that you really don't understand the problem.

While debugging our two rev B boards, we've seen several successes. Our external expansion headers are working, the power rails all work, and the USB programming circuit all work. Unfortunately we were not able to get the PCI Express interface to operate. We found three issues that prevented PCIe from working:

1. From testing on the Lattice ECP5 PCI Express card, I showed that my implementation would only enumerate the PCIe interface on Diamond 3.2, not 3.3. This isn't too big a deal since I can just go back to 3.2 for now.
2. We did not include LDOs for the SerDes (PCIe) transmitter and receiver. We were able to repurpose an LDO on Titan to provide the 1.2V as required by the ECP5 SERDES/PCS Usage Guide. This LDO wire-mod can be seen in the LED blinky video (below) on the left side of the ECP5. The mod worked well, and I verified that the noise level at the decoupling caps nearest to the ECP5 was similar to that measured on the Lattice's ECP5 card.
3. The third issue keeping PCIe from working on Titan can be seen in the blinky video. Watch it again at 720p or higher as see if you can find it. I recommend watching it in full screen.

Did you see it? The engineering samples that we used on the rev B boards were part number LFE5U-85F-8BG381IES. I noticed this when I was building a DDR3 test project tonight. The correct part number is LFE5UM-85F-8BG381IES. What does the M mean? SerDes. We are testing parts without the SerDes (PCIe) interface included.

Second build of rev B

As Kevin mentioned yesterday, we were able to build a second rev B Titan using the ECP5 that we reclaimed from our rev A build. I'm still working to get the PCI Express interface to enumerate, but the build went well as the video above shows.

We've found a few clear errors in the PCB that we need to fix and we've already started some preliminary work on rev C. I'll outline what we found as soon as we have a more complete list.