Rev C1

I received bityExpress-C1 yesterday. This PCB is the OSH Park rebuild of rev C with a little more separation between the inner-plane polygons. My initial inspection showed no plane shorts.

This is where things get hard. I'd love to know how well this card works, but with my pivot to Lattice I'm not sure that spending any more on this path makes sense. If I can stay strong I'll just do some more electrical testing to confirm that my latest OSH Park ruleset worked.

bityExpress-C1

It's cloudy out

After I decided to try Lattice's Diamond toolset, I had to find a place to install it. Recently I've been using Linux boxes to run the Xilinx and Altera toolsets and I've have very good success using the boxes remotely. This means either a graphical VNC connection or just SSH'ing into the box and running the FPGA tools from the command line.

Lattice presents an interesting opportunity. I've been running Ubuntu/Debian for quite a while and I'm comfortable with them. Unfortunately Lattice Diamond is built for RHEL (Red Hat Enterprise Linux) and I don't have a box set aside to install RHEL on. I've gone down the path of installing toolsets on unsupported Linux distributions; It usually works but it can be very time consuming.  My solution: Amazon EC2 (Amazon Elastic Compute Cloud).

I set up a AWS Free Usage Tier account a few months ago and this was the perfect excuse to start using it.  They had a free machine image that runs 64bit RHEL 6.4.  The free system's specs are pretty meager, but should be fast enough to see if this is worth pursuing.  I can always switch to a faster instance if I like it.

It took me a few hours to set the image up, learn how to enable a VNC session, and install Lattice Diamond.  If anyone is interested I'll consider writing up a tutorial.

Lattice

I've been reconsidering my FPGA selection lately. With my current interest in developing low-cost PCI Express designs, the $80 or so that I'm shelling out for each Altera Cyclone V GX is becoming a pricey way to work on my home-brew designs. Based on Digikey pricing, the cheapest Altera Cyclone V GX is ~$80, the cheapest Xilinx Spartan-6 25T is ~$50, and the Lattice ECP3 family capable of PCIe starts at around $30.

$30 is enough to entice me to test drive Lattice's development tools and see how mature they are.

Shorts

I heard back from OSH Park about the shorts on bityExpress-C. My board meets their design rules and the shorts are a fab error. That said, they are recommending larger polygon to polygon (plane) spacing on internal layers (2 to 3 mils above the clearance spec). I'm concerned that they may not be specifying some rules that they should be; I've decided to look into rule sets from some other PCB prototype houses to look for common themes.

Vendor	Track Width / Gap	Smallest Drill	Via Annular Ring	Component Annular Ring	Board Edge to Copper	Drill to Copper Gap	Plane to Plane Clearance	Solder Mask Silver	Solder Mask Expansion
OSH Park	5	10	4 (5*)	ns	ns	ns	ns (7-8*)	ns	ns (4*)
PCB-POOL	5	8	5	8	12	12	ns	4	3

ns = not specified, * = guess based on experience

PCB-Pool's rule listing is more complete than OSH Park's. Given how similar they are I'm inclined to generalize my prototyping rule set in Altium to work for both vendors.

PCB-Pool's Drill to Copper Gap caught my attention. This rule defines how far copper on an internal layer must be from a drill with a different net.  With the bityExpress-C ruleset, the 5 mil annular ring rule plus the 5 mil clearance rule meant that my PCB had a 10 mil Drill to Copper Gap where the PCB-Pool rule is 12 mils. This is especially interesting because if I increase my Plane to Plane Clearance (polygons in this case) to the 7 mils that OSH Park recommended, I also meet PCB-Pool's Drill to Copper Gap rule.

All of this makes me wonder if OSH Park was right that the bityExpress-C shorts are inter-plane shorts. It's possible that the shorts are really due to inaccurate drills hitting planes. I'll give this some more thought. Perhaps I can drill out the vias on one shorted net and see if I can clear the short.

OSH Park wanted to re-run the PCB so I updated it to rev C1. The only change made was to bump the polygon clearance rule to 7 mils. I'm already working on some plans for rev D (which will most likely be built at PCB-Pool). I'll post about my rev D plans soon.

PCB-Pool pricing

It looks like the delta between OSH Park and PCB-Pool is less than I originally thought.  I noticed that if I select any file format on PCB-Pool's website other than GERBER the price drops another $25.63.  This makes bityExpress-C cost $48.33 more at PCB-Pool rather than $73.96.

I updated my comparison chart to show the reduced PCB-Pool pricing.  If the number of PCBs doesn't matter, this means that at 10 square inches PCB-Pool clearly wins.  Below 10 square inches my selection of prototyping vendor will depend on if the design is complex enough to require full electrical testing.

Vendor	Clearance	Drill	Layer	Finish	Silk	E Testing
OSH Park	5	10	4	ENIG	Top/bottom	No
PCB-POOL	5	8	4	ENIG	None	Yes

PCB Specs used in the comparison

Reconsidering the PCB

As much as I love OSH Park's prices, I'm starting to reconsider using them for more complicated designs.  If it turns out that the bityExpress-C error is related to PCB manufacture (rather than design), then I will have gone thru two PCB spins without determining if the PERSTn misconnection on bityExpress-A was the problem.  This hasn't been too bad since I've learned a lot about home-brew PCB assembly during these spins, but looking forward OSH Park's cost advantage starts to look less appealing.

I've been casting about since I discovered the shorts on bityExpress-C.  Most prototype PCB houses don't support 5 mil traces/clearances and 8 mil drills, but I found one that does: PCB-Pool.  Their pricing model is a bit more complicated than OSH Park's simple $10 per square inch for a four layer PCB.  They charge a flat fee based on the design's complexity up to 16 square inches.  If the design is small you can get as many boards as it takes to fill up 16 square inches.  For comparison I used PCB-Pool's online price calculator to generate a quantity one price for various PCB sizes from 0.5 square inches to 25 square inches:

Vendor	Clearance	Drill	Layer	Finish	Silk	E Testing
OSH Park	5	10	4	ENIG	Top/bottom	No
PCB-POOL	5	8	4	ENIG	Top	Yes

PCB Specs used in the comparison

For bityExpress-C, OSH Park costs $55.50 for 3 PCBs.  At PCB-Pool it would cost $155.37 for 2 PCBs (or $159.96 for 3).  At first glance $99.87 extra looks excessive, but if I drop the top silkscreen the price drops to $140.51.  PCB-Pool also provides a free metal stencil.  Since I paid $11.05 from OSH Stencil that brings the price difference down to $73.96.

So the question becomes is the free electrical testing at PCB-Pool worth $73.96.  For me the answer is yes.  I spent twice that amount on the FPGAs that I installed on bityExpress-B and bityExpress-C.

A swing and a miss

While checking the bityExpress-C board that I assembled, I found that most of the power rails were shorted to ground.  I inspected the PCB before assembly, but I did not check for power shorts.  After finding the shorts on the assembled board I checked my two remaining un-populated PCBs and found the same shorts.

I re-ran the DRC in Altium (my EDA tool) and it passes.  I opened a service ticket at OSH Park.  If this does turn out to be another error by OSH Park I may have to find another PCB house.

Building bityExpress-C

Assembly and reflow of my first bityExpress-C board went well last night.  I'll test it later today and see how well it really went.  Below is a video of the process:

Finished bityExpress-C sn #1 assembly

bityExpress-C PCB inspection

I grabbed a few pictures of the three new bityExpress-C PCBs.  These boards were built with a 4 mil solder mask expansion.  Vias have an 10 mil drill with 5 mil annular rings.  This is more conservative than the 4 mil annular ring vias that I used on bityExpress-B.

The boards look good overall.  Board #3 has the worst solder mask alignment, but it is acceptable.  I'll need to measure it more closely, but based on my initial inspection it looks like I may have to stick with a solder mask expansion of 4 mils.  If I had used a 2 mil expansion this card may have had solder mask on the BGA pads.

The via drills look appropriately sized and well aligned.  On to assembly!

bityExpress-C sn #1 BGA solder mask alignment

bityExpress-C sn #2 BGA solder mask alignment

bityExpress-C sn #3 BGA solder mask alignment

bityExpress-C sn #1 QFN solder mask alignment

bityExpress-C sn #2 QFN solder mask alignment

bityExpress-C sn #3 QFN solder mask alignment

bityExpress-C sn #1 drill/annular ring

bityExpress-C sn #2 drill/annular ring

bityExpress-C sn #3 drill/annular ring

Rev C has arrived

It seemed like it took forever to get bityExpress-C in, but it was almost exactly one month.

Why can you sit still?

I was building some boards that were similar to bityExpress last week.  The boards had SMT components on both sides and the back-side build went perfectly.  When I ran the top sides (which included a similar FPGA), everything looked good until I inspected the cards closely.  The solder on all of the SMT components flowed well and everything centered up nicely except the FPGA BGA.  At first I assumed that I had placed the parts badly, but when I spoke with my friendly neighborhood failure analysis expert he zeroed in on a different cause:  a too-rapid cooling profile.

His theory is that if the BGA and PCB are allowed to cool at different rates, the stresses applied to the cooling solder can cause the part to twist before the solder solidifies.  Since my current profile involves opening the toaster oven door 90 seconds after all of the solder joints have flowed, I think that his theory is plausible.  I should have my bityExpress-C boards in this weekend so we'll see if I have better luck with the BGA soldering with a slower cooling profile.

Not one direction, a new direction

I decided to change up bityExpress a bit given what I've learned on the last few board builds.   The upgrades include:

• The entire design is now single-sided SMT.  This should be much easier to paste and assemble myself.  I'll only need to order one stencil, and I won't have to worry reflowing the board twice or deal with back-side components preventing the board from being level while I paste the top-side.  There may be a SI (signal integrity) price to be paid for moving the decoupling capacitors a little farther away from the FPGA's power pins, but this layout still exceeds Altera's guidelines. I'd also like to start testing my SI rules-of-thumb.
• I've removed the JTAG programmer and switching power supplies that I've already validated.  They really only added cost and assembly time at this point.  I'll consider re-adding them if I ever decide to build the board in larger volumes.
• The board is now much more rectangular than it was.  This should make it fit better inside OSH Stencil's jigs;  Pasting should be easier and faster.

3D model of bityExpress-C

I ordered the updated card from OSH Park today and was assigned to today's panel!

Drills

I heard back from OSH Park about these boards.  They said that it looks like the fab house made a mistake on my PCBs and they are refunding my order (for bityExpress-B boards).  They also confirmed that my via size was valid and it should have worked.  That's great news and gives me more confidence in OSH Park as a good place to get low cost four layer PCBs.

I decided against re-ordering rev B.  Even though my PCB met specs, I'm going to look at some risk mitigation ideas and move on to rev C.

Maybe it wasn't me

I started inspecting my two remaining bityExpress-B PCBs and found something disturbing.

Before I built the rev B PCB I noticed that OSH Park had reduced their minimum drill size from 13mils to 10mils and their minimum annular ring size from 7mils to 4mils.  I dropped the size of all of my vias on bityExpress to the smaller size (18mil total via size vs 27mils).  At this smaller size I can start considering more interesting additions to the card for later revisions.

I never really thought twice about it until I started inspecting the cards today.  Based on what happened on my first rev B build, I decided to check for continuity on all critical nets to the FPGA before loading the components and I found that TDI was open!  That means that my problem on the first card might not have been my soldering but with the board itself.  Looking at the PCB under the microscope showed that the annular rings are very thin around the vias.  It looks like the 10mil drills were actually drilled at something larger (13mils?).

A via from the gerbers.  The inner circle is the drill hole and the annular ring is the additional radius beyond the drill.

A via from the bityExpress-B PCB.  The drill is a much larger proportion of the total via diameter on the PCB

Double, Double Toil and Trouble

I dare say that when one's FPGA starts to foam at the mouth, things have gotten a bit out of hand.

It all started when I tested the bityExpress-B board from Sunday's build.  The on-board JTAG programmer was unable to identify the FPGA.  I tried Altera's JTAG debugger app and it turned out that the FPGA was reporting his device type to the programmer correctly, but no data from programmer was making it thru the FPGA.  This implied that the TCK, TMS, & TDO pins were all working correctly, but the TDI pin (data from the CPLD to the FPGA) was  not connected.  After I verified TDI was not shorted to anything nearby, I concluded that the BGA soldering must be the issue.

I tried reflowing the topside of the PCB again and ended up breaking the FPGA to programmer connection even more (no more device type response).  My last desperate measure was to dump a heck of a lot of flux under the BGA and reflow it one last time.  That's when things really went south.  I cleaned the board, powered it up, and started checking the power rails again.  After being on for a few minutes I noticed a frothy white substance coming from under the BGA.  It turns out that the flux I used was conductive and I did not fully clean the underside of the BGA:  Failure was catastrophic.

I spent some time reviewing my build process, footprint design, and solder paste stencil.  I've decided to build another backside myself and take the board back to local assembly house to load the FPGA.  This way I can validate the design.  If that works I'll try to build the third board myself and work on my process.

The big one

I built the top side of my bityExpress-B board last night.  I had to stack some PCBs under my stencil frames to account for the components that I loaded previously on the back side of the board.  The paste worked well everywhere except U4 (the on-board programmer's CPLD).  Since I can bypass the programmer I went ahead and loaded the components and baked it.

All of the solder joints looked good except... you guessed it... U4.  It had at least 3 shorts.  I cleared those and tested.  So far the power supplies all work and I can program the CPLD and the FTDI FT245R.  I stopped when the on-board programmer failed to recognize the FPGA.  I'm betting that is related to solder issues on U4.  I'll try a bit longer to resolve U4's issues and if I fail I'll just bypass it and use an external programmer to check the FPGA.

Ready for paste

Paste looks good (except for U4)

Bakin'

Backside build

I ran the backside of bityExpress-B using my toaster oven on Monday night.  I was planning on running the backside for all three PCBs and pick the best on to run again with the topside components, but I changed my mind when I saw how well the backside ran the first card.



Solder paste applied:

Parts placed:

After reflow:

Now I'm just waiting on my Altera Cyclone V FPGA to arrive before I build the topside.

Why did I ever trust the paste?

After looking again (and getting some advice), I've decided that the real problem was that the paste was too old.  When solder paste ages it looses flux due to its volatility.  When I heated the PCB last night it smoked much less than when I ran the bityBGA boards (a clue I missed).  Today's inspection settles it:

Oh, thermal mass

I used my rev B stencil to apply some paste to an extra rev A PCB to test my ability to solder bityExpress.  Unfortunately I had poor results.   I couldn't get the paste hot enough to flow evenly.

I'm not sure if the issue is with the larger thermal mass of this card or if I need to adjust the IR lens a bit more.  I was discouraged enough to look into toasters.

Back on track

I ordered rev B of my simple Cyclone V PCIe card today.  I spent quite a bit of time trying to understand why the PCIe interface wouldn't function and I couldn't come to a definitive conclusion.  The misconnection of PERSTn (PCIe reset) seems like the best candidate.  I used the Altera BFM (bus functional model) to verify that my firmware works in simulation, but misconnecting PERSTn in the simulation did not cause the simulation to fail.

I decided to go ahead and roll another rev to address the issues that I'm aware of and we'll see how it goes.  My list of candidate problems (corrected on rev B) is below.  If rev B works I may modify a working card to reintroduce as many of the rev A errors as possible to see what the culprit was:

• PERSTn was connected to a general purpose I/O rather than nPERSTL1
• GXB_RX_L0 (PCIe data receive) and REFCLK0L (PCIe reference clock input) each had their polarity reversed on rev A.  The PCIe spec requires devices to support polarity inversion to simplify routing but I cannot find any clear documentation from Altera confirming that they comply with the spec.
• The unused reference clock inputs (REFCLK1L and REFCLK2L) were left floating and not tied to ground as the Altera documentation requires.
• VCCBAT was tied to 3.3V rather than something in the 1.2V to 3.0V range that Altera requires.

Layer 1 check-plot from OSH Park