DesignCon 2016

I purchased my pass for DesignCon 2016 today! Tomorrow is the last day to get the Super Early Bird Special. I'm excited about the trip, and hopefully I'll see some of you out there!

Rev C ordered!

Rev C of titan has been released on github, and I ordered the PCB prototypes tonight.

I'm working on an inventory of the components that I have before I order new parts for the build. Thanks to Kevin for a quick review of the PCB.

Figure 1. Rev C of Titan

Rev C almost ready!

I've (finally) completed my changes for rev C of Titan. The token is officially passed to Kevin for review. I'll post more about the changes on this rev soon. For now, enjoy:

Figure 1. Candidate Rev C.

DDR3 re-check

I decided to double check the DDR3 routing before finishing rev C of Titan, and I found a few length tuning issues to correct. The updates are committed on github: https://github.com/jsloan256/titan/tree/Rev-C.

Figure 1. Superfluous Picture of Titan's Updated DDR3 Routing.

Expansion interface: looking for problems

Before I dive into final tuning I decided to look for any routing issues with the expansion interface. Figures 1-6 show the issues that I found.  I'm going to shift the routing around a bit to try to limit long runs where traces are run closely in parallel.

Figure 1. Tight Routing Between P3 and P5.

Figure 2. Long Parallel Routing.

Figure 3. Too Tight Length Loop.

Figure 4. Too Tight Length Loop.

Figure 5. Long Parallel Routing.

Figure 6. Too Tight Length Loop and Long Parallel Routing.

Expansion interface: rough tuning complete

I finally found some cycles to work on Titan today. I was able to complete the rough tuning of the expansion interface. I still need to tune the length a bit more, match the differential pairs, and I have a bit gold plating left (by gold plating I mean adding a little bit of beautification polish). The good news is that I had enough room to match the expansion trace lengths; It was not an easy task with the minimum and maximum trace lengths differing by as much as 2.5 inches.

Figure 1. Rough Tuning Complete.

Motivation

I let myself get behind on the layout because we didn't know when we would get more ECP5s. Guess what showed up yesterday? Looks like I need to get busy!

Figure 1. The Item.

Expansion interface: the tuning begins

I finished connecting all of the expansion nets tonight. The current route is committed on my Titan Github page. The principle trade-off was that I had to use some long traces. Figure 1 shows the current state of the route. I've also listed the minimum and maximum trace lengths for each DDR data group in Table 1. As I start tuning the trace lengths, I may re-route some nets to make the design easier to tune (both by balancing layer usage and shortening the longest routes).

Figure 1. Expansion Route (All Nets Connected, Pre-tuning).

Expansion Header P3			Expansion Header P5
A_DQ0 min	1239.646		B_DQ0 min	1718.488
A_DQ0 max	3388.985		B_DQ0 max	2966.341
A_DQ1 min	1059.547		B_DQ1 min	1248.02
A_DQ1 max	1604.663		B_DQ1 max	3499.98

Table 1. Net Minimum and Maximums.

Expansion interface: the depature

My attempts to adjust the expansion header routing by changing pin assignments minimally to make the two headers pinout compatible has turned into an unbounded problem. I decided to change tack. I wiped the current routing, assigned the nets manually, and defined a few groupings to direct the routing.

Within each group I've allowed myself varying level of routing flexibility depending on how critical the net locations are. So far it is going well, hopefully I'll have time to complete the routing this weekend analyze the results. A summary of the working groupings are listed below in priority order:

DQ Group 0 & 1   Each expansion header is connected to a bank that contains two DSQ groups. One of these DQ groups contains the PCLK inputs for the bank as well as the VREF net; This DQS is designated as DQ0 and routed to the long side of the expansion header

PCLK, DQS0, & DQS1   The PCLK input and data strobe (DQ0 & DQ1) differential pairs are assigned to fixed locations that cannot be changed. These nets are routed first and are only allowed layer transitions at the ECP5 and at the expansion header (never more than two). Figure 1 shows my initial route of these nets.

DQ0 pairs 0 & 1 / DQ1 pairs 0 & 1   The first two data pairs adjacent to the data strobes must be "true LVDS TX" pairs from the ECP5 (identified as A/B pairs). They are labeled DQ0-0, DQ0-1, DQ1-0, and DQ1-1. Any A/B pair from the same DQ group may be used. I was able to route these pairs only using vias at the ECP5 and/or the expansion header, like the highest priority pairs (see Figure 2). I'm not sure if we should allow more or not, but since the routing went well I don't have to worry about that today.

DQ0 pairs 2, 3, & 4   The last three differential pairs for each data group (DQ0-2, DQ0-3, DQ0-4, DQ1-2, DQ1-3, DQ1-4) are all routed to ECP5 C/D pairs. Each set of three pairs within the same DQ group (ie. DQ0-2, DQ0-3, & DQ0-4) may be switched to each routing. I have not yet routed these pairs on Titan, but from my examination of the current state of the route (Figure 2) I've concluded that I will have to allow additional via transitions to route all of these pairs (hopefully no more than one additional via).

VREF & D0   The final two nets in the DQ0 data group will be routed as single ended nets (not differential). This is because the VREF net is on a diff pair true net on one bank and on a compliment net on the other bank. There is no common routing that will allow these nets to be differential and have the VREF pin on the same expansion header pin.

D1, D2, D3, & D4   Since the DQ1 data group does not contain a PCLK input or VREF, it has four "extra" nets to route. These will all be routed as singled ended nets.

Figure 1. Top Priority Nets Routed.

Figure 2. Second Priority Nets Routed.

Expansion interface: the re-route

Over the holiday I've been considering how painful it will be to re-route the expansions headers on Titan to meet (or approximate) the rules that I defined last week. The answer: Really, really, painful.

The current route is layer efficient and has very few layer transitions (both fantastic qualities). Unfortunately, this natural matching of an ECP5 bank to an expansion connector places DQ groups/strobes, PCLK inputs, and the VREF all over the place. Table 1 shows my analysis of the pinout comparison between expansion headers P3 and P5. I realize that this table is a little busy, but I was trying to look at several different factors at the same time. I considered posting the color version, but it is a rather terrifying to behold.

P5 Function			P3 Function							P3 Function			P5 Function
DQ	Pair	Oth	DQ	Pair	Oth	Top Side Net	Pin #		Bottom Side Net	DQ	Pair	Oth	DQ	Pair	Oth
						3.3V	2	1	V_I/O
						3.3V	4	3	V_I/O
						GND	6	5	GND
89	C		41	C	PCLK	D0_P (S0)	8	7	D8_P (S16)	41	A		89	A
89	D		41	D	PCLK	D0_N (S1)	10	9	D8_N (S17)	41	B		89	B
						GND	12	11	GND
89	A		41	A	DQS	D1_P (S2)	14	13	D9_P (S18)	41	C		89	C
89	B		41	B	DQS	D1_N (S3)	16	15	D9_N (S19)	41	D		89	D
						GND	18	17	GND
89	A		17	A	DQS	D2_P (S4)	20	19	D10_P (S20)	17	A		89	A	DQS
89	B		17	B	DQS	D2_N (S5)	22	21	D10_N (S21)	17	B		89	B	DQS
						GND	24	23	GND
53	C	PCLK	41	C		D3_P (S6)	26	25	D11_P (S22)	17	C		89	C
53	D	PCLK	41	D		D3_N (S7)	28	27	D11_P (S22)	17	D		89	D
						GND	30	29	GND
						GND	32	31	GND
53	C		41	A		D4_P (S8)	34	33	D12_P (S24)	17	C		53	C
53	D		41	B		D4_N (S9)	36	35	D12_N (S25)	17	D		53	D
						GND	38	37	GND
53	C		17	A		D5_P (S10)	40	39	D13_P (S26)	17	C		89	C
53	D		17	B		D5_N (S11)	42	41	D13_N (S27)	17	D		89	D
						GND	44	43	GND
53	A		41	C	VREF	D6_P (S12)	46	45	D14_P (S28)	17	A		53	A
53	B		41	D		D6_N (S13)	48	47	D14_N (S29)	17	B		53	B
						GND	50	49	GND
53	A	DQS	41	A		D7_P (S14)	52	51	D15_P (S30)	17	C		53	A
53	B	DQS	41	B		D7_N (S15)	54	53	D15_N (S31)	17	D		53	B	VREF
						GND	56	55	GND
						n/c	58	57	n/c
						n/c	60	59	n/c

Table 1. P3 & P5 Pin Mappings.

Figures 1 and 2 show the scale of the problem. Figure 2 especially shows just how clean the current route is. I spent quite awhile looking at different pinouts where where only the most critical nets were routed to the same connector pin. The scale of the change quickly becomes so severe that it doesn't look possible without adding quite a few layer transitions (bad for signal quality) and possibly using more PCB layers (bad for PCB cost).

Figure 1. The Spaghetti Monster.

Figure 2. Routing Layers for the Expansion Interface.

I only have one knob left to twist: function. I have a hunch that only routing about half of the nets as differential pairs will make it much easier to route the rest as single-ended nets. I have a few use-cases in mind. Hopefully I'll find a way to map them to a rational pin mapping.

Reading this you might wonder why I'm taking so much time with this re-planning of the expansion interface. The first is simple: I believe that an intelligent pin assignment will ease the design of modules. The second is all in the schedule: We have our first order of ECP5 FPGA's on the way, but the lead time is long enough for us to give a careful review of the entire design before placing our next PCB order. We intentionally side-stepped the tricky problem of the expansion header design in our first two PCB revs. Now that the rest of the design is stable (or nearly stable), it's time to reconsider this part of the design.

Expansion interface: rules

Before Christmas Kevin and I discussed the expansion header rule-set that I posted last time. I ended up giving him a little clarification. What follows is a summary. Keep in mind that this is a work in progress. Once we finish the new expansion routing for rev C we will post a summary and use case examples.

Figure 1. A View of the MEC8-RA Expansion Header on Titan.

Trace Lengths and Matching All signal trace lengths must be matched to within 20mils and may not exceed 2000mils. Additionally each differential pair in the bank will be routed as a differential pair each pair's N and P net will be length tuned within 5mils. This tuning is intentionally over-kill to reserve as much margin as possible for the routing on modules.

TX and RX pair assignments All LVDS transmit pairs must be connected to the same side of the expansion connector. Figure 2 shows and mapping that meets this rule. This rule is included because the Samtec MEC8-RA connector as longer pins on one side of the connector. Figure 3 was taken from the MEC8-RA's STEP file. My rough measurements show that the top pins are about 100mils longer than the bottom pins. We can avoid accounting for this difference by routing the ECP5's true LVDS transmitter pairs on one side (labelled as pads A and B) and the receiver pairs (labelled as pads C and D) on the other side.

Figure 2. An TX/RX Mapping Example (TX on the Long Side, RX on the Short Side).

Figure 3. A Top and Bottom Pin Pair from the MEC8-RA.

DQ Pin assignments Each DQ group must be matched to the same set of diff pairs; Within the each DQ group the DQS pair must be on a fixed pin. Figure 4 shows an example DQ mapping. This rule means that the individual DQ pair assignments are not fixed and may be routed differently to ease routing.

Figure 4. An Example Mapping of the Two DQ Groups from an ECP5 Bank.

Other Fixed Pin Assignments The bank's VREF net and PCLK on C/D pads must be routed to fixed pins.

Expansion interface

We are finalizing our plans for rev C of Titan now. Over the last week Kevin has been working on our current change list: layout adjustments to ease manufacture, tighter tuning of the expansion header traces, and the addition of a 1.2V power supply for the SerDes (PCIe) transmitter and receiver. Kevin is working in his fork of the Titan project. Once he completes his changes I'm going to make a few adjustments to the DDR3 layout. When we are ready to order we will pull the changes back into the master branch of Titan on the C-E-S organization Github page.

While Kevin is updating the layout, I've been focused on validated our changes. Initially I was working on using the Diamond's Reveal embedded logic analyzer to identify the PCIe issues. Now that I know that there isn't a PCIe core in the samples we have, I'm focusing my efforts on validating the DDR3 memory design and the external interfaces.

Figure 1. Titan Rev B with Two Expansion Modules Installed.

Our expansion headers bring out a single bank of the ECP5 FPGA to a single connector. Now that we've had more time to consider possible expansion modules it has become clearer how we should organize the connections from the FPGA's I/O banks to the connectors. We are trying to balance keeping the layout simple and clean with our desire to standardize the expansion header pinout to make our future modules interchangeable from one Titan header to another.

Each FPGA bank that we have connected to an expansion header can be configured as 32 single-ended GPIOs, 16 differential GPIOs, two DDR data groups, or 7:1 geared interfaces (ie. CameraLink). Lattice's ECP5 High-Speed I/O Interface guide describes the interface possibilities. One specific interface that we are studying that has applications with SDR is the Linear Technology LTC217x quad ADC.

This leads me to my list of features that need to be the same from one expansion header to another. These assignments are based on the expansion differential pair nomenclature used on the rev B Titan schematic (see Figure 2).

Figure 2. Titan Expansion Header (showing our simple nomenclature).

Titan Expansion Header Features

• One PCLK pair must assigned to a fixed diff pair (ie. DO_P/N)
• Each of the two DQS pairs must be assigned to a fixed diff pair (ie D8_P/N and D12_P/N)
• Each DQ group must be matched to the same set of diff pairs (ie D0/1/2/3/9/10/11_P/N)
• VREF must be assigned to a fixed net (ie. D1_P)
• All LVDS transmit pairs must be connected to the same side of the expansion connector (ie. D0/1/2/3/4/5/6/7_P/N)
• All trace lengths must be matched to within 20mils and may not exceed 2000mils.

Now we'll see if I've given Kevin an impossible set of requirements. If I have I'm sure that I'll hear about it.

Stupid mistakes

Through my career I've had the pleasure of working with some intelligent and talented engineers and scientists. Along the way I've learned to listen when I'm given advice. I might decide not to heed their advice, but I always listen. The two most memorable nuggets of advice that I've received are:

1. Don't be a candy ass
2. Don't sweat the stupid mistakes, it's the smart mistakes that should worry you

Tonight I'm trying hard to remind myself of #2. The principle is simple; We all make silly mistakes, overlook things, or just forget something. As long as we identify our mistakes quickly and move on it shouldn't be a problem. The bad mistakes are ones where you focus on something for a long time and come to the wrong conclusion. These smart mistakes often mean that you really don't understand the problem.

While debugging our two rev B boards, we've seen several successes. Our external expansion headers are working, the power rails all work, and the USB programming circuit all work. Unfortunately we were not able to get the PCI Express interface to operate. We found three issues that prevented PCIe from working:

1. From testing on the Lattice ECP5 PCI Express card, I showed that my implementation would only enumerate the PCIe interface on Diamond 3.2, not 3.3. This isn't too big a deal since I can just go back to 3.2 for now.
2. We did not include LDOs for the SerDes (PCIe) transmitter and receiver. We were able to repurpose an LDO on Titan to provide the 1.2V as required by the ECP5 SERDES/PCS Usage Guide. This LDO wire-mod can be seen in the LED blinky video (below) on the left side of the ECP5. The mod worked well, and I verified that the noise level at the decoupling caps nearest to the ECP5 was similar to that measured on the Lattice's ECP5 card.
3. The third issue keeping PCIe from working on Titan can be seen in the blinky video. Watch it again at 720p or higher as see if you can find it. I recommend watching it in full screen.

Did you see it? The engineering samples that we used on the rev B boards were part number LFE5U-85F-8BG381IES. I noticed this when I was building a DDR3 test project tonight. The correct part number is LFE5UM-85F-8BG381IES. What does the M mean? SerDes. We are testing parts without the SerDes (PCIe) interface included.

Second build of rev B

As Kevin mentioned yesterday, we were able to build a second rev B Titan using the ECP5 that we reclaimed from our rev A build. I'm still working to get the PCI Express interface to enumerate, but the build went well as the video above shows.

We've found a few clear errors in the PCB that we need to fix and we've already started some preliminary work on rev C. I'll outline what we found as soon as we have a more complete list.

Another picture of rev B

Two more pictures from PCB-Pool just arrived. These show the surface finish of the PCBs being built:

Figure 1. Titan rev B Top Side (Surface Finish).

Figure 2. Titan rev B Bottom Side (Surface Finish).