DDR3 data group tuning

I moved half of the upper byte to layer 4 and tuned the tracks. I was shooting to tune each byte to within 100mils, but I ended up beating that without really trying. The lower byte is tuned within 32 mils (see Table 1) and the upper byte is tuned within 21 mils (Table 2).

Figures 1 and 2 (below) show the routes on layers 4 and 6. All of the tracks in the data groups except for the DQS differential pairs are shown; They are routed on layers 1 and 3.

	Pre-tuned length (mils)	Post-tuned length (mils)
D0	859	1205
D1	855	1185
D2	839	1178
D3	934	1210
D4	1027	1204
D5	1187	1187
D6	934	1194
D7	1016	1202
DM0	802	1202
DQS0p	1200	1200
DQS0n	1194	1194

Table 1. Pre and Post Track Lengths for the Lower Byte.

	Pre-tuned length (mils)	Post-tuned length (mils)
D8	962	962
D9	861	981
D10	877	982
D11	975	972
D12	746	961
D13	868	978
D14	803	970
D15	953	973
DM0	888	974
DQS1p	983	983
DQS1n	924	973

Table 2. Pre and Post Track Lengths for the Upper Byte.

Figure 1. Tuned Data Tracks on Layer 6 (Upper Byte and Half of Lower Byte).

Figure 2. Tuned Data Tracks on Layer 4 (Half of Lower Byte).

The PCB layout shown in this post by Custom Embedded Solutions, LLC. is licensed under a Creative Commons Attribution 4.0 International License.

DDR3 data routing for x16 parts

I tried to length match each byte pair last night and immediately ran into troubles. While I had left room to allow for tuning wiggles, it turns out that the mismatch is too great. Figure 1 shows the location of the lower and upper bytes in the route. My goal was to route both of the data bytes on the same layer.

Figure 1. Single Layer Data Route with Bytes Annotated.

Typically, six layer DDR3 routes require four signal layers, one ground plane, and one power plane. So far, I've assumed that if I only used a single DDR3 part I could drop the number of signal layers from four to three by avoiding the connection of control/address signals between each DDR3 IC.

Everything was going great until I started to tune the data bytes. The problem was with the lower byte (D0 to D7). Since the Lattice ECP5 requires that the same DQS group (data byte, strobe, and byte enable) go to the same DQS group on the ECP5, my route groups the bytes together. This is fine for the upper byte, but half of the lower byte (D1, D3, D5, D7, & DM0) have to swing wide to the left to escape the DDR3 IC without crossing the upper byte. This swing horribly mismatches the trace lengths within the byte:

• D0 length: 877 mils
• D1 length: 1371 mils
• D2 length: 861 mils
• D3 length: 1532 mils
• D4 length: 956 mils
• D5 length: 1619 mils
• D6 length: 892 mils
• D7 length: 1460 mils

The average mismatch between odd and even lower byte bits is 599 mils. Figures 2 and 3 show D0 before and after tuning. Since I could only remove 291 mils of mismatch, I'm doubtful that I will be able to tune the lower byte as routed.

Figure 2. Non-tuned D0 Route (Length = 877 mils)

Figure 3. Tuned D0 Route (Length = 1168 mils)

This leaves me with a few ideas about how to proceed:

1. Use a fourth signal layer to route the lower byte. While this doesn't meet my goal for such a simple DDR3 design, it looks like a pretty standard solution.
2. Try to more aggressively select ECP5 DQS groups to route the entire x16 data set. This may be possible because the DQS groups are stacked roughly in pairs of two from the top to the bottom of the ECP5 (per side). This approach might work, but it works heavily against my address/control routing method where I bring all address/control nets in on the top layer.
3. Switch to two x8 DDR3 ICs. With each byte coming from a different IC I can add more vertical space between the parts to ease routing

After taking to Kevin, I've decided to go with option 1 as long as the impact on the rest of the layout is minimal.


The PCB layout shown in this post by Custom Embedded Solutions, LLC. is licensed under a Creative Commons Attribution 4.0 International License.

DDR3 data lines connected

I just finished routing both DDR3 data bytes from the memory IC (left side of Figure 1) to the FPGA (right side). Using 0.125mm traces (4.92mils) made the route easy. I only had to move a few of the address/control nets. The net reassignments were required because the Lattice ECP5 requires that all data bytes be routed to the same data group (named LDQ17, LDQ41, LDQ53, or LDQ89 on the left side of the ECP5).

Next I will tune the trace lengths of the address/control and data bytes. I'm going to start with a ruleset that was developed for a 1333 Mbps DDR3 design. Since the ECP5's maximum data rate is 800 Mbps this ruleset should be overkill. If the route becomes too difficult I'll reconsider the rules.

For now my goal is to get a route that is good enough so that I have something to simulate. Here's hoping that I'll have a simulation ready design soon.

Figure 1. Data Connection (2 Bytes) from DDR3 IC to FPGA.

The PCB layout shown in this post by Custom Embedded Solutions, LLC. is licensed under a Creative Commons Attribution 4.0 International License.

Lattice ECP5 PCI Express Card

I received the Lattice Semiconductor ECP5 PCI Express Development Kit today! A few gratuitous unboxing photos follow:

Figure 1. The Box.

Figure 2. Lattice ECP5 PCI Express Development Card (Front Side).

Figure 3. Lattice ECP5 PCI Express Development Card (Back Side).

New address route

I finished connecting the DDR3 to the FPGA (thru the series resistors) using 0.125mm (4.92mil) traces. The route looks very clean and there is plenty of room left to tune the traces lengths. I'm going to do the initial route of the data lines next and then I'll start tuning. Figure 1 shows the new route with both routing layers shown at the same time. Figures 2 and 3 show the old, fat, route for reference.

This route was fast and I expect the data bytes to be just as quick.

Figure 1. New Top and Inner Layer DDR3 Route with Thin Traces

Figure 2. Top Layer DDR3 Route with Fat Traces.

Figure 3. Inner Layer DDR3 Route with Fat Traces.

The PCB layout shown in this post by Custom Embedded Solutions, LLC. is licensed under a Creative Commons Attribution 4.0 International License.

Series termination layout

While reconsidering the trace size and termination topology for the address and control bus (see Fat, fat traces), it became apparent that I needed to keep my layout as tight as possible for the series resistors. The resistors needed to be packed tightly to allow them to be placed as close as possible to the driver (the FGPA) and to limit trace length differences to each resistor.

The first solution that came to mind was the termination part used on Lattice's ECP3 Versa development card. They used a CTS Corp RT2402B7-TR7 (Figures 1-3). They are designed to be used for common termination applications such as DDR3, VME64, PCIX, etc. I always try to limit the number of new components that I have to include in CES' database, but this part certainly fits the bill as a compact termination solution.

As I studied it a bit more, I made this list:

• For series termination, the C Series looks like the right option. Figure 2 shows an RT2402 part used in a series termination application.
• 18 series resistors are in the RT2402 (C Series), this is four short of the 22 resistors that I need.
• In in smallest (1.0mm) package, the RT2402 is 4.0mm by 9.0mm. When I tried to replicate the arrangement and trace pattern in Figure 3 using discrete 0402 resistors, my pattern was approximately 4.0mm by 10.0mm (Figure 4).

While the RT2402 looks like a fantastic part for many applications, I think that well-placed discrete resistors will work well for me.

Figure 1. CTS Corp. RT2402 (from the RT2402 datasheet)

Figure 2. Series Termination with the RT2402 (from the RT2402 datasheet)

Figure 3. Escape Routing with the 1.0mm RT2402 (from the RT2402 datasheet)

Figure 4. My Series Resistor Routing Inspired by the RT2402.

The PCB layout shown in this post by Custom Embedded Solutions, LLC. is licensed under a Creative Commons Attribution 4.0 International License.

Fat, fat traces

As I mentioned in the last post, I decided to route the DDR3 memory using a series R tree architecture. My initial analysis showed that I needed to use very thick traces to get a 50 Ohm impedance using PCB-Pool's default stack-up. I didn't complete the route, but I got close enough that it looked possible to connect everything. Several problems became apparent:

• In Figure 1, the control traces on the lower left hand corner connecting the DDR3 IC and the series resistors had to fan out in a wide pattern to the left of the DDR3 IC. This makes the overall memory layout large given the size of the components. Since the control traces have to be length matched, the large trace distance difference between the traces connecting to the top versus the bottom resistors implies that the top traces will need a lot of tuning 'wiggles'.
• Normally the control nets (left side of the DDR3 IC in Figures 1 and 2) are routed vertically between the DDR3's fanout vias. The wide fanout that the large traces required will make re-use of this layout in designs with multiple DDR3 ICs unlikely
• The data bus routes on the right side of the DDR3 IC on the inner layer and the bottom layer (Figures 2 and 3) will be even more challenging to tune. I met my goal of routing a byte per layer, but the traces are so close that there is no tuning room left.

When I re-read Micron's design guide for point to point designs (TN-41-13) this morning, I realized that I overlooked a critical note on pages 15-16: "The tree with series R provides acceptable signal quality without having to provide VTT power when there is a mismatch between the driver and transmission line. The tree without series R option generally provides acceptable signal quality without having to provide VTT power when there is a minimal mismatch between the driver and transmission line".

My route was trying to both match the transmission line impedance (thick traces) and use series resistors. According to TN-41-13 this is overkill. I should pick one strategy or the other, not both. Since PCB-Pool's pricing is so attractive, I'm going to stick with the series resistors and use much smaller trace widths. I should be able to simulate the layout and validate if this approach will work.

Figure 1. Top Layer DDR3 Route

Figure 2. Inner Layer DDR3 Route.

Figure 3. Bottom Layer DDR3 Route.

The PCB layout shown in this post by Custom Embedded Solutions, LLC. is licensed under a Creative Commons Attribution 4.0 International License.

DDR3 Control Architectures

With Kevin working on the main Titan layout, I've been focusing on the DDR3 layout and building a simple FPGA project to validate our pin selection. I expect to post the FPGA project to my github account: https://github.com/jsloan256. Be warned, I'm still learning to use git and posting testing code; I don't guarantee that anything works. When I reach a milestone I'll post about it here and link to the github commit.

The DDR3 design on Titan will use a simple point to point topology. Unlike more traditional module DDR3 designs like one would find on a PC motherboard, point to point designs have both the DDR3 memory IC and the controller (CPU, FPGA, etc.) on the same PCB. The lack of connectors and shorter traces in point to point design allow some simplification of the DDR3 routing rules. I'm still finalizing my DDR3 rules, but I've found the Micron design guide for point to point designs (TN-41-13) to be very helpful.

Figures 1 thru 4 below show Micron's recommended address/control routing topologies. Since I'm only planning to use a single DRAM, the difference between fly-by and tree architectures is negligible. I settled on the topology on Figure 1 (tree architecture with series R) mostly because it doesn't require a VTT power rail and the signal quality should be better with the series R than without (Figure 2).

I've started routing and have hit some hurdles. I'll post some pictures of the route next time and talk about what I'm trying to fix.

Figure 1. Tree Architecture with series R and Vtt pull-up (from TN-41-13)

Figure 2. Tree Architecture with no R (from TN-41-13)

Figure 3. Tree Architecture with series R (from TN-41-13)

Figure 4. Fly-by Architecture with Vtt pull-up (from TN-41-13)

Impedance and PCB-Pool, Part 2

In my last post I talked about about the challenges of routing 50 Ohm traces using PCB-Pool's prototype stackup. In this post I'm going to talk a bit more about using the MMTL Electromagnetic Simulator.

With MMTL you can model a PCB stack-up by drawing objects that represent each layer of either a conductor or a dielectric (insulator). Figure 1 shows an example of a model for the outer layer traces. The top yellow rectangle represents copper traces on an outer layer (ie. the top or bottom layer). The green space below the copper trace is the 0.2mm dielectric layer that separates the top layer from the first inner layer. The blue object below the dielectric is the nearest inner plane. In this case we use the nearest inner plane as a ground plane.

Figure 1. MMTL Model of the Top Two Layers with a 0.2mm Dielectric.

One of MMTL's best features is the ability to sweep a parameter. Below are the results from a 100 iteration sweep of the trace width. I used this output to determine that 0.366mm (14.4mil) wide traces were required on the top layer to achieve 50 impedance traces. Next week I'll post routing examples from my ECP5 DDR3 route.

____ # iterations:  100 ___
Write: "C:/Program Files (x86)/tnt-1.2.2/examples/pcbpooltop.xsctn.temp"
----------------------------------------
::Top width = 9.5e-005
::TopR0   Impedance = 90.4255
____________________________________________________
::Top width = 9.80808080808e-005
::TopR0   Impedance = 89.4904
____________________________________________________
::Top width = 0.000101161616162
::TopR0   Impedance = 88.5816
____________________________________________________
::Top width = 0.000104242424242
::TopR0   Impedance = 87.6974
____________________________________________________
::Top width = 0.000107323232323
::TopR0   Impedance = 86.8369
____________________________________________________
::Top width = 0.000110404040404
::TopR0   Impedance = 85.9987
____________________________________________________
::Top width = 0.000113484848485
::TopR0   Impedance = 85.1817
____________________________________________________
::Top width = 0.000116565656566
::TopR0   Impedance = 84.385
____________________________________________________
::Top width = 0.000119646464646
::TopR0   Impedance = 83.6075
____________________________________________________
::Top width = 0.000122727272727
::TopR0   Impedance = 82.8486
____________________________________________________
::Top width = 0.000125808080808
::TopR0   Impedance = 82.1071
____________________________________________________
::Top width = 0.000128888888889
::TopR0   Impedance = 81.3827
____________________________________________________
::Top width = 0.00013196969697
::TopR0   Impedance = 80.6743
____________________________________________________
::Top width = 0.000135050505051
::TopR0   Impedance = 79.9815
____________________________________________________
::Top width = 0.000138131313131
::TopR0   Impedance = 79.3036
____________________________________________________
::Top width = 0.000141212121212
::TopR0   Impedance = 78.6399
____________________________________________________
::Top width = 0.000144292929293
::TopR0   Impedance = 77.7666
____________________________________________________
::Top width = 0.000147373737374
::TopR0   Impedance = 77.1325
____________________________________________________
::Top width = 0.000150454545455
::TopR0   Impedance = 76.5111
____________________________________________________
::Top width = 0.000153535353535
::TopR0   Impedance = 75.9021
____________________________________________________
::Top width = 0.000156616161616
::TopR0   Impedance = 75.3048
____________________________________________________
::Top width = 0.000159696969697
::TopR0   Impedance = 74.719
____________________________________________________
::Top width = 0.000162777777778
::TopR0   Impedance = 74.1442
____________________________________________________
::Top width = 0.000165858585859
::TopR0   Impedance = 73.58
____________________________________________________
::Top width = 0.000168939393939
::TopR0   Impedance = 73.0262
____________________________________________________
::Top width = 0.00017202020202
::TopR0   Impedance = 72.4823
____________________________________________________
::Top width = 0.000175101010101
::TopR0   Impedance = 71.948
____________________________________________________
::Top width = 0.000178181818182
::TopR0   Impedance = 71.4231
____________________________________________________
::Top width = 0.000181262626263
::TopR0   Impedance = 70.9073
____________________________________________________
::Top width = 0.000184343434343
::TopR0   Impedance = 70.4002
____________________________________________________
::Top width = 0.000187424242424
::TopR0   Impedance = 69.9016
____________________________________________________
::Top width = 0.000190505050505
::TopR0   Impedance = 69.4113
____________________________________________________
::Top width = 0.000193585858586
::TopR0   Impedance = 68.9291
____________________________________________________
::Top width = 0.000196666666667
::TopR0   Impedance = 68.4546
____________________________________________________
::Top width = 0.000199747474748
::TopR0   Impedance = 67.9876
____________________________________________________
::Top width = 0.000202828282828
::TopR0   Impedance = 67.5281
____________________________________________________
::Top width = 0.000205909090909
::TopR0   Impedance = 67.0756
____________________________________________________
::Top width = 0.00020898989899
::TopR0   Impedance = 66.6302
____________________________________________________
::Top width = 0.000212070707071
::TopR0   Impedance = 66.1915
____________________________________________________
::Top width = 0.000215151515152
::TopR0   Impedance = 65.7594
____________________________________________________
::Top width = 0.000218232323232
::TopR0   Impedance = 65.3337
____________________________________________________
::Top width = 0.000221313131313
::TopR0   Impedance = 64.9143
____________________________________________________
::Top width = 0.000224393939394
::TopR0   Impedance = 64.501
____________________________________________________
::Top width = 0.000227474747475
::TopR0   Impedance = 64.0936
____________________________________________________
::Top width = 0.000230555555556
::TopR0   Impedance = 63.6921
____________________________________________________
::Top width = 0.000233636363636
::TopR0   Impedance = 63.2961
____________________________________________________
::Top width = 0.000236717171717
::TopR0   Impedance = 62.9058
____________________________________________________
::Top width = 0.000239797979798
::TopR0   Impedance = 62.5208
____________________________________________________
::Top width = 0.000242878787879
::TopR0   Impedance = 62.141
____________________________________________________
::Top width = 0.00024595959596
::TopR0   Impedance = 61.7664
____________________________________________________
::Top width = 0.000249040404041
::TopR0   Impedance = 61.3968
____________________________________________________
::Top width = 0.000252121212121
::TopR0   Impedance = 61.0322
____________________________________________________
::Top width = 0.000255202020202
::TopR0   Impedance = 60.6723
____________________________________________________
::Top width = 0.000258282828283
::TopR0   Impedance = 60.3171
____________________________________________________
::Top width = 0.000261363636364
::TopR0   Impedance = 59.9665
____________________________________________________
::Top width = 0.000264444444445
::TopR0   Impedance = 59.6204
____________________________________________________
::Top width = 0.000267525252525
::TopR0   Impedance = 59.2787
____________________________________________________
::Top width = 0.000270606060606
::TopR0   Impedance = 58.9413
____________________________________________________
::Top width = 0.000273686868687
::TopR0   Impedance = 58.6081
____________________________________________________
::Top width = 0.000276767676768
::TopR0   Impedance = 58.279
____________________________________________________
::Top width = 0.000279848484849
::TopR0   Impedance = 57.954
____________________________________________________
::Top width = 0.000282929292929
::TopR0   Impedance = 57.6328
____________________________________________________
::Top width = 0.00028601010101
::TopR0   Impedance = 57.3156
____________________________________________________
::Top width = 0.000289090909091
::TopR0   Impedance = 57.0022
____________________________________________________
::Top width = 0.000292171717172
::TopR0   Impedance = 56.6925
____________________________________________________
::Top width = 0.000295252525253
::TopR0   Impedance = 56.3863
____________________________________________________
::Top width = 0.000298333333333
::TopR0   Impedance = 56.0839
____________________________________________________
::Top width = 0.000301414141414
::TopR0   Impedance = 55.7849
____________________________________________________
::Top width = 0.000304494949495
::TopR0   Impedance = 55.4893
____________________________________________________
::Top width = 0.000307575757576
::TopR0   Impedance = 55.1971
____________________________________________________
::Top width = 0.000310656565657
::TopR0   Impedance = 54.9083
____________________________________________________
::Top width = 0.000313737373738
::TopR0   Impedance = 54.6227
____________________________________________________
::Top width = 0.000316818181818
::TopR0   Impedance = 54.3402
____________________________________________________
::Top width = 0.000319898989899
::TopR0   Impedance = 54.061
____________________________________________________
::Top width = 0.00032297979798
::TopR0   Impedance = 53.7847
____________________________________________________
::Top width = 0.000326060606061
::TopR0   Impedance = 53.5116
____________________________________________________
::Top width = 0.000329141414142
::TopR0   Impedance = 53.2413
____________________________________________________
::Top width = 0.000332222222222
::TopR0   Impedance = 52.974
____________________________________________________
::Top width = 0.000335303030303
::TopR0   Impedance = 52.7095
____________________________________________________
::Top width = 0.000338383838384
::TopR0   Impedance = 52.4479
____________________________________________________
::Top width = 0.000341464646465
::TopR0   Impedance = 52.189
____________________________________________________
::Top width = 0.000344545454546
::TopR0   Impedance = 51.9328
____________________________________________________
::Top width = 0.000347626262626
::TopR0   Impedance = 51.6793
____________________________________________________
::Top width = 0.000350707070707
::TopR0   Impedance = 51.4285
____________________________________________________
::Top width = 0.000353787878788
::TopR0   Impedance = 51.1802
____________________________________________________
::Top width = 0.000356868686869
::TopR0   Impedance = 50.9345
____________________________________________________
::Top width = 0.00035994949495
::TopR0   Impedance = 50.6912
____________________________________________________
::Top width = 0.00036303030303
::TopR0   Impedance = 50.4504
____________________________________________________
::Top width = 0.000366111111111
::TopR0   Impedance = 50.2121
____________________________________________________
::Top width = 0.000369191919192
::TopR0   Impedance = 49.976
____________________________________________________
::Top width = 0.000372272727273
::TopR0   Impedance = 49.7424
____________________________________________________
::Top width = 0.000375353535354
::TopR0   Impedance = 49.5111
____________________________________________________
::Top width = 0.000378434343435
::TopR0   Impedance = 49.282
____________________________________________________
::Top width = 0.000381515151515
::TopR0   Impedance = 49.0552
____________________________________________________
::Top width = 0.000384595959596
::TopR0   Impedance = 48.8305
____________________________________________________
::Top width = 0.000387676767677
::TopR0   Impedance = 48.608
____________________________________________________
::Top width = 0.000390757575758
::TopR0   Impedance = 48.3876
____________________________________________________
::Top width = 0.000393838383839
::TopR0   Impedance = 48.1693
____________________________________________________
::Top width = 0.000396919191919
::TopR0   Impedance = 47.9532
____________________________________________________
::Top width = 0.0004
::TopR0   Impedance = 47.739
____________________________________________________
Set the display values to the data prior to the simulations.

Impedance and PCB-Pool

I've been looking into low cost options for prototyping PCBs for some time now. I originally used OSH Park and I was pleased with their prices and build quality, but for more complicated designs I kept running across electrical errors even when I met their design guidelines (post). In my casting about for another low cost PCB prototyping house I ran across PCB-Pool. What excited me about PCB-Pool was that they offered both flying-lead electrical testing and free metal stencils. Hopefully this will avoid the problems that I had with my dense designs from OSH Park.

Kevin and I have been trying to address our biggest concerns with PCB-Pool: Can we really route a 0.8mm BGA and meet their design rules and can we rationally route impedance controlled designs using their PCB stack-up. Kevin spoke with an engineer at PCB-Pool and got a good answer for the first question: They said that we can route using slightly smaller traces than their rules allow and we can expect it to work (and the electrical testing will confirm it). I've been looking into impedances.

Figure 1. PCB Stackup from PCB-Pool Specifications

To calculate the impedance for traces on the top layer I used the MMTL Electromagnetic Simulator. Unfortunately PCB-Pool's stack-up isn't ideal for the 50 Ohm traces needed for PCI Express and DDR3. I would usually design a custom stack-up so that traces somewhere between 4 and 6 mils would have a characteristic impedance of 50 Ohms. PCB-Pool's thicker pre-preg layers means that 50 Ohm traces will need to be 14.4mils thick. With traces that wide we will have to neck them down to escape route from the ICs. Hopefully the neck-downs won't degrade the signal quality too much. As we test and simulate more I will post if I have any interesting results.

So many blogs!

Kevin and I have decided to move our regular postings about life, the universe, and work on Titan to our personal blogs. We are going to do bigger roll-up posts on the company blog (CES) twice a month. Hopefully this will make it easier to follow our individual status as well as our trains of thought.

I'm going to cross-post a few recent CES posts here to provide a little context before I push ahead.