Monitoring TLPs

So far; so good. I implemented the first part of my TLP master plan. The test that I ran is rather simple: I have a version of the titan_wiggle FPGA project with the Lattice's Reveal logic analyzer installed and connected to the PCIe core's receive TLP interface (Fig 1). Titan is installed in a PC running Linux and Shaun's pyhwtest utility.

Figure 1. Block Diagram of the Titan FPGA Design Using Reveal to Capture TLP messages.

pyhwtest is a great little utility. With it I can access the memory space on a PCIe card using python. Everything from simple read/writes to DMA transactions work. With pyhwtest I don't have to write a kernel driver. To use it, all that is required is to find the base address for the BAR that I want to access. Listing 1 (below) shows the output of lspci: BAR0 is mapped to 0xDC000000.

 [root@localhost src]# lspci -d dead:* -v  
 04:00.0 Non-VGA unclassified device: Indigita Corporation Device beef (rev 06)  
     Flags: bus master, fast devsel, latency 0, IRQ 10  
     Memory at dc000000 (32-bit, non-prefetchable) [size=256]  
     Memory at d8000000 (32-bit, non-prefetchable) [size=64M]  
     Capabilities: [50] Power Management version 3  
     Capabilities: [70] MSI: Enable- Count=1/1 Maskable- 64bit+  
     Capabilities: [90] Express Endpoint, MSI 00  
     Capabilities: [100] Device Serial Number 00-00-00-00-00-00-00-00  
 [root@localhost src]#  

Listing 1. Linux Terminal Output Showing the Output from lspci.

Listing 2 shows the commands required to write 0x12345678 to BAR0 (address 0xDC000000), and Figure 2 shows the Reveal capture that results. It's a good sign that the data written in pyhwtest shows up in the capture. I built a simple spreadsheet to decode the TLP packet (Table 1). From here I'll start thinking about how best to decode the TLP packets in the FPGA and the best way to handle data flow to and from the PC.

 [root@localhost refresh_test]#  
 [root@localhost refresh_test]# python -i titan_test.py -b 0xdc000000  
 Base address: 0xdc000000L  
 >>> hwtest.writelw(BASE, le32_to_be(0x12345678))  
 >>>  
 [root@localhost refresh_test]#  

Listing 2. Python Terminal Output Showing a Data Write to the Base Address of BAR0.

Figure 2. Reveal Capture of the Data Write TLP Message.

H0	R	FMT		Type					R	TC			R				TD	EP	Attr		R		Length
	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	1
H1	Requester ID																Tag								Last BE				First BE
	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	0	1	1	1	1
H2	Address																															R
	1	1	0	1	1	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0
D0	Data
	0	0	0	1	0	0	1	0	0	0	1	1	0	1	0	0	0	1	0	1	0	1	1	0	0	1	1	1	1	0	0	0

Table 1. Decode of Captured RX TLP Message.

Pushing TLPs

Now that the physical interfaces on Titan have shown to work, the fun part begins. I've given some thought to the firmware framework that Titan needs. In the simplest terms, I want to be able to control and test all of Titan's interfaces from a PC (via PCIe). Developing this firmware will require interfacing to Lattice's PCIe core, the DDR3 core, and a bit of logic glue here and there. I also want to be able to simulate the entire design. A complete simulation environment will allow me to "crack open" the hood and locate bugs much faster than with hardware-only testing.

Lattice's PCIe core presents TLP (transaction layer packets) to the user side of the PCIe core. In past I've used higher level bridges to avoid dealing with TLPs directly. Xilinx has cores available that provide a bus interface to the user (ie. AXI or PLB). Lattice doesn't have a higher level core, but they do have an example of a TLP to Wishbone bridge in the firmware for the ECP3 and ECP5 Versa cards.

While I was tempted to try the TLP to Wishbone bridge, I decided that building firmware to consume TLPs directly had two advantages: It will likely be smaller in the FPGA, and it will give me a chance to understand PCIe transactions better than I have before. How can I pass up a chance to dive a little deeper into PCIe?

The implementation plan is summarized in the Figure 1 block diagram. Then idea is to build two state machines to control the flow of TLP messages between the PCIe core and the registers or FIFOs. One state machine will handle the RX (receive) TLP messages while the second will handle the TX (transmit) messages. The registers will be used for simple interfaces such as the GPIO pins. The FIFOs will handle the buffering and clock domain transition required to interface to DDR3.

Figure 1. Block Diagram of the Titan FPGA Design Connected to a PC.

To simulate the PCIe core, I'm building a testbench where a second PCIe core is instantiated as a stand-in for the PC. I got the idea to connect two PCIe cores together from the Lattice PCIe Endpoint User's Guide. Using a core to test itself would be a bad idea if I was designing the PCIe core itself. Since I'm just implementing a user-side interface, I'm OK trusting that Lattice meets the PCIe SIG standards. Initially this will allow me to write some simple state machines to control the simulation PCIe (Figure 2). Eventually I'll abstract the control interface to the simulation-side PCIe core to allow higher level interface; This will likely be implemented in SystemVerilog, but I'm also considering a python-based interface using MyHDL.

Figure 2. Block Diagram of the Titan FPGA Design Connected to Another PCIe Core Acting as a BFM

So, where to start? I've been reviewing documentation of the structure of TLPs. While TLP isn't overly complex, a complete implementation to handle every possible transaction will take awhile to build. To speed things up a bit I decided to focus my initial work on the most relevant TLPs. My plan is to install Titan in a PC running Linux and use the pyhwtest to send write and read messages. Inside the FPGA, Lattice's Reveal analyzer will be instantiated so I can capture the TLPs received for various commands from Linux. See Figure 3.

I'll use the captured data to design simple synthesizable FSMs (finite state machines) to decode and act on the TLP messages captured in Reveal. Once I have that working, I'll design some simulation FSMs to generate the same messages as the Reveal captured messages. Together, these FSMs will comprise a starting point for the BFM (bus functional model) and Titan simulation of the PCIe link.

Figure 3. Block Diagram of the Titan FPGA Design Using Reveal to Capture TLP messages.

PCIe enumeration

Figure 1 doesn't look like much, does it? This simple output from lspci means that the PCI Express interface on the ECP5 is being enumerated on the host. Big step!

Figure 1. Output from lspci

Rev C bring up

I built the first rev C of Titan earlier this week, and I've been working to bring up the board. This has been the cleanest build yet. It did take about five hours to build by hand, but my process is getting better with each build.

So far I've confirmed that all of the power rails are operational, the on-board USB programming circuit works, and the FPGA can be programmed. The LED counter didn't run when I loaded titan_wiggle, so I'll need to debug a little tonight.

Figure 1. First Rev C.

Rev C almost ready!

I've (finally) completed my changes for rev C of Titan. The token is officially passed to Kevin for review. I'll post more about the changes on this rev soon. For now, enjoy:

Figure 1. Candidate Rev C.

DDR3 re-check

I decided to double check the DDR3 routing before finishing rev C of Titan, and I found a few length tuning issues to correct. The updates are committed on github: https://github.com/jsloan256/titan/tree/Rev-C.

Figure 1. Superfluous Picture of Titan's Updated DDR3 Routing.

Expansion interface: looking for problems

Before I dive into final tuning I decided to look for any routing issues with the expansion interface. Figures 1-6 show the issues that I found.  I'm going to shift the routing around a bit to try to limit long runs where traces are run closely in parallel.

Figure 1. Tight Routing Between P3 and P5.

Figure 2. Long Parallel Routing.

Figure 3. Too Tight Length Loop.

Figure 4. Too Tight Length Loop.

Figure 5. Long Parallel Routing.

Figure 6. Too Tight Length Loop and Long Parallel Routing.

Expansion interface: rough tuning complete

I finally found some cycles to work on Titan today. I was able to complete the rough tuning of the expansion interface. I still need to tune the length a bit more, match the differential pairs, and I have a bit gold plating left (by gold plating I mean adding a little bit of beautification polish). The good news is that I had enough room to match the expansion trace lengths; It was not an easy task with the minimum and maximum trace lengths differing by as much as 2.5 inches.

Figure 1. Rough Tuning Complete.

Motivation

I let myself get behind on the layout because we didn't know when we would get more ECP5s. Guess what showed up yesterday? Looks like I need to get busy!

Figure 1. The Item.

Expansion interface: the tuning begins

I finished connecting all of the expansion nets tonight. The current route is committed on my Titan Github page. The principle trade-off was that I had to use some long traces. Figure 1 shows the current state of the route. I've also listed the minimum and maximum trace lengths for each DDR data group in Table 1. As I start tuning the trace lengths, I may re-route some nets to make the design easier to tune (both by balancing layer usage and shortening the longest routes).

Figure 1. Expansion Route (All Nets Connected, Pre-tuning).

Expansion Header P3			Expansion Header P5
A_DQ0 min	1239.646		B_DQ0 min	1718.488
A_DQ0 max	3388.985		B_DQ0 max	2966.341
A_DQ1 min	1059.547		B_DQ1 min	1248.02
A_DQ1 max	1604.663		B_DQ1 max	3499.98

Table 1. Net Minimum and Maximums.

Expansion interface: the depature

My attempts to adjust the expansion header routing by changing pin assignments minimally to make the two headers pinout compatible has turned into an unbounded problem. I decided to change tack. I wiped the current routing, assigned the nets manually, and defined a few groupings to direct the routing.

Within each group I've allowed myself varying level of routing flexibility depending on how critical the net locations are. So far it is going well, hopefully I'll have time to complete the routing this weekend analyze the results. A summary of the working groupings are listed below in priority order:

DQ Group 0 & 1   Each expansion header is connected to a bank that contains two DSQ groups. One of these DQ groups contains the PCLK inputs for the bank as well as the VREF net; This DQS is designated as DQ0 and routed to the long side of the expansion header

PCLK, DQS0, & DQS1   The PCLK input and data strobe (DQ0 & DQ1) differential pairs are assigned to fixed locations that cannot be changed. These nets are routed first and are only allowed layer transitions at the ECP5 and at the expansion header (never more than two). Figure 1 shows my initial route of these nets.

DQ0 pairs 0 & 1 / DQ1 pairs 0 & 1   The first two data pairs adjacent to the data strobes must be "true LVDS TX" pairs from the ECP5 (identified as A/B pairs). They are labeled DQ0-0, DQ0-1, DQ1-0, and DQ1-1. Any A/B pair from the same DQ group may be used. I was able to route these pairs only using vias at the ECP5 and/or the expansion header, like the highest priority pairs (see Figure 2). I'm not sure if we should allow more or not, but since the routing went well I don't have to worry about that today.

DQ0 pairs 2, 3, & 4   The last three differential pairs for each data group (DQ0-2, DQ0-3, DQ0-4, DQ1-2, DQ1-3, DQ1-4) are all routed to ECP5 C/D pairs. Each set of three pairs within the same DQ group (ie. DQ0-2, DQ0-3, & DQ0-4) may be switched to each routing. I have not yet routed these pairs on Titan, but from my examination of the current state of the route (Figure 2) I've concluded that I will have to allow additional via transitions to route all of these pairs (hopefully no more than one additional via).

VREF & D0   The final two nets in the DQ0 data group will be routed as single ended nets (not differential). This is because the VREF net is on a diff pair true net on one bank and on a compliment net on the other bank. There is no common routing that will allow these nets to be differential and have the VREF pin on the same expansion header pin.

D1, D2, D3, & D4   Since the DQ1 data group does not contain a PCLK input or VREF, it has four "extra" nets to route. These will all be routed as singled ended nets.

Figure 1. Top Priority Nets Routed.

Figure 2. Second Priority Nets Routed.

Expansion interface: the re-route

Over the holiday I've been considering how painful it will be to re-route the expansions headers on Titan to meet (or approximate) the rules that I defined last week. The answer: Really, really, painful.

The current route is layer efficient and has very few layer transitions (both fantastic qualities). Unfortunately, this natural matching of an ECP5 bank to an expansion connector places DQ groups/strobes, PCLK inputs, and the VREF all over the place. Table 1 shows my analysis of the pinout comparison between expansion headers P3 and P5. I realize that this table is a little busy, but I was trying to look at several different factors at the same time. I considered posting the color version, but it is a rather terrifying to behold.

P5 Function			P3 Function							P3 Function			P5 Function
DQ	Pair	Oth	DQ	Pair	Oth	Top Side Net	Pin #		Bottom Side Net	DQ	Pair	Oth	DQ	Pair	Oth
						3.3V	2	1	V_I/O
						3.3V	4	3	V_I/O
						GND	6	5	GND
89	C		41	C	PCLK	D0_P (S0)	8	7	D8_P (S16)	41	A		89	A
89	D		41	D	PCLK	D0_N (S1)	10	9	D8_N (S17)	41	B		89	B
						GND	12	11	GND
89	A		41	A	DQS	D1_P (S2)	14	13	D9_P (S18)	41	C		89	C
89	B		41	B	DQS	D1_N (S3)	16	15	D9_N (S19)	41	D		89	D
						GND	18	17	GND
89	A		17	A	DQS	D2_P (S4)	20	19	D10_P (S20)	17	A		89	A	DQS
89	B		17	B	DQS	D2_N (S5)	22	21	D10_N (S21)	17	B		89	B	DQS
						GND	24	23	GND
53	C	PCLK	41	C		D3_P (S6)	26	25	D11_P (S22)	17	C		89	C
53	D	PCLK	41	D		D3_N (S7)	28	27	D11_P (S22)	17	D		89	D
						GND	30	29	GND
						GND	32	31	GND
53	C		41	A		D4_P (S8)	34	33	D12_P (S24)	17	C		53	C
53	D		41	B		D4_N (S9)	36	35	D12_N (S25)	17	D		53	D
						GND	38	37	GND
53	C		17	A		D5_P (S10)	40	39	D13_P (S26)	17	C		89	C
53	D		17	B		D5_N (S11)	42	41	D13_N (S27)	17	D		89	D
						GND	44	43	GND
53	A		41	C	VREF	D6_P (S12)	46	45	D14_P (S28)	17	A		53	A
53	B		41	D		D6_N (S13)	48	47	D14_N (S29)	17	B		53	B
						GND	50	49	GND
53	A	DQS	41	A		D7_P (S14)	52	51	D15_P (S30)	17	C		53	A
53	B	DQS	41	B		D7_N (S15)	54	53	D15_N (S31)	17	D		53	B	VREF
						GND	56	55	GND
						n/c	58	57	n/c
						n/c	60	59	n/c

Table 1. P3 & P5 Pin Mappings.

Figures 1 and 2 show the scale of the problem. Figure 2 especially shows just how clean the current route is. I spent quite awhile looking at different pinouts where where only the most critical nets were routed to the same connector pin. The scale of the change quickly becomes so severe that it doesn't look possible without adding quite a few layer transitions (bad for signal quality) and possibly using more PCB layers (bad for PCB cost).

Figure 1. The Spaghetti Monster.

Figure 2. Routing Layers for the Expansion Interface.

I only have one knob left to twist: function. I have a hunch that only routing about half of the nets as differential pairs will make it much easier to route the rest as single-ended nets. I have a few use-cases in mind. Hopefully I'll find a way to map them to a rational pin mapping.

Reading this you might wonder why I'm taking so much time with this re-planning of the expansion interface. The first is simple: I believe that an intelligent pin assignment will ease the design of modules. The second is all in the schedule: We have our first order of ECP5 FPGA's on the way, but the lead time is long enough for us to give a careful review of the entire design before placing our next PCB order. We intentionally side-stepped the tricky problem of the expansion header design in our first two PCB revs. Now that the rest of the design is stable (or nearly stable), it's time to reconsider this part of the design.