Transit Note #39
MBTA: Wonderland Packaging

Andre DeHon

Original Issue: February 1991

Last Updated: Mon Nov 8 14:59:14 EST 1993

Abstract:

Wonderland, the 64 processor MBTA machine, uses Transit stack packaging to effect an extremely compact computational structure. By packaging the computational nodes in the stack along with the network, the entire logic portion of the machine including network, memory, and processors will fit comfortably in about half of a cubic foot (). This note describes how this structure is achieved.

Introduction

Using the Transit Dual Sided Pad Grid Array 372 (DSPGA372) packages and Button Board 372 (BB372) connectors, it is possible to package a multistage network in an efficient, dense, three-dimensional structure (tn33). The Transit stack packaging scheme using these components provides sufficient cooling, alignment, and vertical interconnect to make such a three-dimensional structure viable.

Building a large multiprocessor around such a network, we are left with the challenge of packaging the machine's processing nodes (including memory) and connecting these to the network. One option would be to package the nodes in semi-conventional packaging separate from the stacked network. In this case, it would be necessary to bring numerous signals out of the network stack to connect them to the processor. For even a small machine like Wonderland, this would require bringing on the order of wires out of the network stack and cabling them to the processing nodes. To avoid losing significant performance at this interface, such interconnect would need to be extremely short; additionally, quiet, controlled impedance cabling to these nodes would be a necessity.

Alternately, we can package the nodes in the same stack structure. This allows us to easily connect the nodes to the network. The same features of the DSPGA372 and BB372 that allowed efficient interconnection of the network can be utilized to connect the nodes to the network. The same cooling system can be used to cool all components. The entire structure becomes one dense stack. Interconnection distances can be kept short minimizing transit time on wires and clock-skew.

Packaging the nodes in the same stack structure as the network is not without its challenges. We must be able to package major components in pad grid array packages. A few random components can be added, but the height of such components must fit within the tight spacing between printed circuit boards provided by DSPGA372 and BB372. The size and geometry of the node is tightly constrained by the network.

The remainder of this note discusses how we intend to cope with these problems to construct a complete multiprocessor based on the Transit stack packaging strategy.

Basic Strategy

The basic strategy is to package each node in a quadrant over 4 routing components in the network. In this manner, four nodes are packaged in each stack layer. This requires 16 layers of nodes to contain all 64 nodes. We can package half of these layers above and below the network, requiring stacks of 8 nodes over each quadrant on each side of the network. We can make use of symmetry and rotation to allow a single node board design to tap off four distinct network connections from the through signals running through the node. Thus we only need two distinct flavors of node boards to allow each node in a stack of 8 to access a unique network connection.

Each node may contain 4 components packaged in DSPGA372 packages, as well as, surface mount components in the space between DSPGA372 packages. Cooling for node components in DSPGA372 packages is identical to router cooling (tn33) as they exist in the same component columns as the routers. Surface mount components packaged on the node are air cooled by flow across the stack. Alignment occurs just as for network components. A bolt through the middle of each node column also serves to keep the nodes structurally in place. Clock distribution is aided by the fact that any major components is aligned in one of 16 columns. Adequate and uniform pressure is maintained across the entire stack by rigid plates on the ends with bolts to hold the structure together.

Figure shows a compact side-view of the Wonderland stack. This view is basically the same from all four sides with the exception that the top and bottom boards only overhang on one side. Figure shows this structure expanded to make composition clearer.

Difficulties

The node boards will be approximately square. Each node board is thus contstrained to be approximately square. This will be a tight fit for the node logic.

The 80960CA currently only comes in a square PGA package. This package is completely incompatible with the stack structure. The package is too large and too tall; it is also not amenable to the stack cooling strategy. We would like to consider repackaging 80960CA/B die in DSPGA372 (or compatible) packages. The die for the 80960CA-16S V594 version of the 80960 is just too large to fit in the cavity of the current DSPGA372.

Note: The assumption that we can repackage the 80960 into a DSPGA372 compatible package is the biggest (and to first order, only) assumption made by the following description. If we cannot do anything about the packaging of the 80960, it will almost certainly be the show-stopper for this packaging scheme.

The network interface and bus controller must be packaged in DSPGA372 components. The glue logic and memory must be packaged in low-profile, surface mount packages and packed very densely.

Possibilities

The logic is tight, but there are a number of new packages which will make much of this feasible. If we can get the SRAM memory in 128K8, gull-wing surface mount components with 50 mil lead spacing, memory should not be a problem to place on the node. TSOP packages would be a viable alternative, if we can get sufficiently fast memory packaged in that form factor. TI's SSOP packages offer 25 mil pitch surface mount components with 16-bit wide bus options. This packaging option for bus logic should allow us to keep the area consumed by random logic down to the available space.

We do have the freedom to package our custom components, the network interface (tn31) and the bus controller [DS90c], in DSPGA372 packages. As described in (tn18) we can integrate all 4 network interfaces into a single component. With the 80960, bus controller, and network interface each consuming one DSPGA372 on the node board, the fourth DSPGA372 remains uncommitted. If we can find a way to use this package for glue logic, it will further ease our tight spacing requirements on the node.

Constituent Printed Circuit Boards

The Wonderland stack is composed of the following printed circuit boards:

(1) Clock/T-Station Board
(1) Termination/Power Board
(4) unique network boards
(32) A-flavor node boards
(32) B-flavor node boards

Node Boards

Figure shows a diagram of a stack node board. The network interface, bus controller, and 80960 each reside in DSPGA372 packages aligned with routing components in the network. The fourth DSPGA372 package will either remain blank, simply providing through interconnect, or house some glue logic. A 250 mil hole in the center of the board provides for structural alignment. Components reside on the top of their respective printed circuit board node and, generally, underneath another node board. The live contacts for the node components mostly reside on the top of the board. Most of the contacts do not need to (and should not) connect to the bottom of the printed circuit boards. The corners of the board may be notched as shown to accommodate the center 250 mil hole for a bolt holding the stack together.

Each DSPGA372 provides 76 dedicated through vias (19 per quadrant). Each node requires 36 signals from the network plus the two initialization lines ( HINIT and SINIT) and clock. Each A-board taps off the 36 through via signals on the ``top'' and ``right'' quadrants of the network interface's package. Similar each B-board taps off the 36 through via signals on the ``bottom'' and ``left'' quadrants. The initialization and clock lines may be shared between A- and B-boards. All through vias connect straight through each node board. In this manner, the 8 nodes in a stack each get their unique network connection by using all 4 rotations of A- and B-boards. Figure shows a cross-section of the Wonderland stack looking down on a node layer.

Network Boards

Routing boards are configured as shown in Figure . Routing is contained to the square portion of the board. The tab only serves to bring the 16 temperature sensor signals from the router out to the edge of the board where they can be cabled elsewhere as appropriate. The top board does not need a temperature tab since no routing component rest upon it. The routing components and screw holes align with those shown for the nodes in Figure .

Clock/T-Station Board

The topmost board in the stack houses the T-Station host interface (tn20), the clock drives, and the clock buffers (Figure ). The ECL source clock is generated and buffered here to provide a separate buffered output driver for each of the 16 columns of components. The traces are fanned out on this board in roughly equal length runs to each of columns to provide minimal skew clocks to all components in the stack (see (tn37) for detailed information the the clocking strategy). The initialization lines ( for RN1 and and for the nodes) are also run in roughly equal length traces on this layer. For most the the square area above the network, the only traces running are the clock and initialization traces. Also, the only contact pads requiring vias are those for clock and initialization.

One quadrant of the board, does have through connections. The lower-right quadrant has pads to tap signals onto and off of one node board. This allows T-Station to connect directly to a boot node for input and output from Wonderland.

T-Station and the ECL clocks and buffers are packaged on the tab that extends along the right side of this board. This tab is outside of the stack allowing it to be cooled by normal air flow and removing any height restrictions on the the components.

The temperature sensing signals from the routing boards (and perhaps from the nodes) may be brought here also. In this manner, T-Station can also serve to periodically sample the temperatures of the components and send the data to a host computer for collection.

Termination/Power Board

The bottommost board (Figure ) provides series termination for the clock signals and provides the power connection for the entire stack. The only connections actually made to this board are the signals requiring termination and the power connections.

Through Interconnections

There are three distinct regions in the stack where the through vias connect straight through boards and components (see Figure ).

From the top surface of the top outer routing board through the top of the stack, through vias run all the way through the nodes to provide network connections for the 32 nodes on top of the network.
From the bottom surface of the top outer routing board through the top surface of the bottom outer routing board, through vias allow network connections to begin or end on the appropriate side of the network for the associated node.
From the bottom surface of the bottom outer routing board through the bottom of the stack, through vias run through the nodes to provide network connections for the 32 nodes below the network.

As such, the top and bottom outer routing boards do not have continuous through vias whereas all other boards do.

Provisions for Additional Memory

In the event that we need to add additional memory to some or all nodes, we cannot simply add more memory on the node cards due to space limitations. However, if we are careful with the positioning of signals, it should be possible to construct a memory card which can be placed between nodes in the stack. The memory card can connect to the top of the node components (and hence the important node signals) to interconnect with the nodes. Such a card would not need to interfere with the normal propagation of network connections up the stack. The memory card would simply increase the height of the component stack.

Transit Note #39
MBTA: Wonderland Packaging

Abstract:

Introduction

Basic Strategy

Difficulties

Possibilities

Constituent Printed Circuit Boards

Node Boards

Network Boards

Clock/T-Station Board

Termination/Power Board

Through Interconnections

Provisions for Additional Memory

See Also...

References

Transit Note #39 MBTA: Wonderland Packaging

Abstract:

Introduction

Basic Strategy

Difficulties

Possibilities

Constituent Printed Circuit Boards

Node Boards

Network Boards

Clock/T-Station Board

Termination/Power Board

Through Interconnections

Provisions for Additional Memory

See Also...

References

Transit Note #39
MBTA: Wonderland Packaging