Network Interface
Andre DeHon
Original Issue: August 1990
Last Updated: Mon Nov 8 13:46:56 EST 1993
The network interface component provides an interface between an RN1 based network and the processor and memory on an MBTA node. The network interface can be configured either as a network input ( net-in), dealing with all network traffic destined for the node, or as a network output ( net-out), dealing with traffic originating from the node. This note describes the interface to and the composition and behavior of the network interface component.
The network interface is designed to handle most primitive operations directly without need for intervention from the processor. For more complicated operation, it simply serves as an interface between the processor and network, acting under the direct control of the processor.
The network interface handles virtually all of the low-level issues of communication. It is intended to handle the portions of the network interface which must be implemented in hardware and are well understood now. The network interface is especially intended to handle operations which need to be implemented efficiently in hardware in order to obtain a reasonable level of performance. To this end, the network interface handles:
The network interface component is shown in Figure .
(tn25) shows how network interfaces are integrated into an MBTA node.
Table
describes the network interface's data and control
signals. Table
summarizes the pin requirements for this
component.
The network interface component needs to be synchronized to both the network and the node. To keep the frequency of the node in line with that of the network, the network interface provides NODE_CLK_OUT which is used as the source for clocking the node. NODE_CLK_OUT runs at half the frequency of the network clock ( NCLK). To avoid skew problems with the node, the signals which interfrace with the node are synchronized to the input clock, NODE_CLK, which is presumed to be result of buffering NODE_CLK_OUT so it can be fanned out to all components on the node requiring clocks. See (tn37) for further details on MBTA clocking strategy.
The processor communicates with each network interface as a memory
mapped device. Table shows the relevant communication
offset addresses within each network interface's designated memory region.
Each network interface has its own memory region assigned by the node bus
controller (tn30).
is used to inform the network
interfaces that they are being addressed by the processor. The bits 7:4 of
the address are used to distinguish which network interface is being
addressed. W/
is used to indicate whether the processor
wishes to transfer data to or from the network interface. All
communications directly between the processor and a network interface takes
place on the low data bus (D<31:0>); as such all network interface
addresses are at least double word aligned.
The processor will only address the network interface during the processor's designated memory cycle. The processor will never communicate with a network interface during a borrowed memory cycle.
Table lists the internal registers in the network
interface. Many of these can be read or changed through addresses shown in
Table
. Each register is described elsewhere in this
document where its function is relevant.
Note that Table marks the 3rd lowest nibble (bits 11:8)
as
. These bits specify which network part is being addressed by the
memory operation. Table
summarizes the meanings of the
various values of
.
The network interface uses the node's SRAM for the source and
destination of data sent over the network. The processor tells the network
interface where to put incoming data from an ROP network operation
or from a remote read by setting the out_buf_ptr. Similarly, the processor tells the network interface where to
find outgoing data by setting the interface's in_buf_ptr. Once
set, these pointers remain in effect until changed. As such, it is only
necessary for the processor to respecify a pointer when a target memory
location changes. These pointers should not be changed while the network
interface is performing an operation which uses them to reference memory.
The remote_address register specifies the destination address on the remote node for raw write operations. This register is only used when the interface is configured as a network output and performing a network write operation.
The processor tells the network interface to perform an operation by writing to the OPERATION address. All transactions are initiated, aborted, continued, or stopped by issuing an operation.
The general format of an operation is:
LEN FUNCTION
OP
DST
Each portion of the operation word is one byte in length. DST specifies the destination node for the specified operation. LEN specifies the length in words of the operation. OP specifies the operation to be performed at the remote node. FUNCTION specifies a function to be performed by the local node. Operations do not always use all of the fields of the operation.
The function is the most basic part of the operation command. It is the
only field which is interpreted for all operations. Functions are
summarized in Table .
NETOP launches a network operation as described in (tn19) and (tn21). This operation is the only operation which interprets the high three bytes of the operation command. NETOPNOTURN launches a remote operation which does not turn the network and thus does not get reply or status information.
ENDNOW instructs the network interface to close an open connection that the processor does not wish to use. RESPONDFINAL and RESPOND initiate remote operation responses. RESPONDFINAL closes the connection after sending data whereas RESPOND turns the connection around again for another reply.
ABORT instructs net-out to drop the current operation and return to its idle state as soon as possible.
OP specifies the network operation to be performed. Operation
encodings are shown in Table and described further in
(tn21).
LEN specifies the length in words of data to be transferred during a network operation. This is needed by read, write, and ROP operations.
DST specifies the destination node for the network message.
The network interface has a number of configurable options. It is possible
to specify the number of dummy cycles between real network data by setting
dummy-cycles. The number of retransmissions net-out will attempt is
specified by retries. The number of network stages can be selected
by setting stages. The node number is configured by setting
node-number. Figure
shows the composition of the
configuration register. Individual portions of the word cannot be set
independently. To change just part of the configuration, read the
configuration, reset the desired bits, and write the configuration back.
The unused bits in the configuration word are available for other
configuration options which may come up during design and prototyping.
Space is specifically left next to the number of dummy cycles so this
parameter can be expanded if early experience with MBTA indicates the
number of allocated bits is insufficient.
The status_ptr points to the memory location for the status buffer. Net-out will place the result of each network retransmission in successive double words in memory starting at the address stored in status_ptr. The status_ptr has no use when the interface is configured as a network input. For each failed network attempt, two words are written to memory. The first is the status word which indicates what caused the connection to fail. The second word is the bad checksum word which indicates how the checksum was corrupted when a checksum fails.
When a connection is successfully opened, net-out will write a status word to memory after the operation completes. An ROP which turns the turns the network more than once from the forward direction will only store the status from the final turn -- connection-wise, this data should be identical to that acquired on the first turn.
That is, the status word contains a sequence of two bit status indications
from each network stage and a two bit status indication from the
net-in interface at the destination node (Section ).
The status bit pairs occur in the order they are received
from the network as shown.
is either the status bits
from the net-in network interface at the far end of the network or
the status bits from the stage in the network where the connection failed.
Failed connections may thus have less status bit pairs than complete
connections. Tables
shows the meaning of each pair of
status bits. The lowest-order bit of the status word, FAIL,
indicates whether or not the connection was successful. If FAIL is
set, the connection failed; if FAIL is clear, the connection was
successfully made. The processor needs only to look at FAIL if it
only wants to know if a particular connection attempt was successful. The
bits in between
and FAIL are currently unused and
are undefined.
If the connection failed because the network interface received an inconsistent checksum at some stage in the network, the bad checksum word for that connection attempt will include both the first failed status-checksum pair and the expected checksum for this pair. The status word will indicate where in the network the bad checksum occurred. If the operation failed for some reason other than a bad checksum, the bad checksum is undefined. The bad checksum word is formatted as follows:
Reading STATE will return the state of the network interface. The format and meaning of this word will be defined as the component is implemented. Some subset of these bits should indicate what the interface is expecting from the processor. This may also be useful for keeping the processor in synch with the component. The state as a whole should be useful in diagnostic testing.
The state should probably indicate at least the following:
When errors occur such that the network interface is forced to signal the
processor that an error has occurred using its
line, the processor should be able to determine the error by reading STATE.
The current list of possible errors is shown in Table
.
N.B. All of the errors shown so far are essentially fatal. When one
of these errors occurs, either the processor and the network interface are
in inconsistent states or there is a bug in the source program. The
assertion of indicates that such an error has
occurred; the processor should halt and signal the error to the host so the
source of the error can be located and debugged.
The address ACK is used to check the successful completion of an operation. As such it is used in two slightly different ways depending on the network operation performed. After any operation which turns the network around for an acknowledgment but not for data ( i.e. noop, reset, or write), it indicates whether or not the ack returned indicated the success or failure of the operation. After any operation which sends a response over the network ( i.e. read and ROP), it indicates whether or not the reply checksum was correct. Reading ACK gives the processor an entire word, but only the lowest two bit are important. The second lowest bit indicates whether or not the ack or final checksum has been received. This bit is cleared at the beginning of an operation and is set when the ack arrives. The lowest bit is only valid when this second lowest bit is set. The lowest bit indicates the state of the actual success or failure of the operation. When set, the operation succeeded ( i.e. the ack was true or the checksum was valid); when cleared, the operation failed ( i.e. the ack was false or the checksum was invalid).
Each network interface will be counting the number of dummy cycles so it will know when to send and expect real data over the network. Each emulation cycle is composed of eight real network cycles and hence 8 sets of dummy cycles. The end of cycle counter keeps track of the number of dummy cycles and the real network cycles. Dummy cycles count from 0 modulo the configured number of dummy cycles plus one. The dummy cycle counter is incremented every node cycle. The phase counter counter increments every network cycle and counts from 0 modulo 8. Each reset of the phase counter denote a node cycle and hence increments the dummy counter. The end of cycle counter is formatted as:
The two network outputs used in an MBTA node function logically as a single network output interface which randomly selects between network ports for transmissions. RND_IN and RND_OUT are used to select the output port, and hence the associated net-out, for a particular transmission attempt
When the processor initiates a network transaction, it writes the operation generically to net-out. Both network outputs receive the operation. They both xor RND_IN and RND_OUT together. If the result of the xor is the same as the network interface's UNIT designation, the network interface handles the network transmission. In this manner, exactly one net-out attempts to transmit the network transaction.
If the previous attempt to open a connection fails, another attempt must be
made to open the connection. The network outputs need, once again, to
randomly select a network port. The net-out which made the failed
connection attempt, asserts to indicate that
retransmission is necessary. The other net-out does nothing except
wait for the next operation or retransmission. On the network cycle
following the assertion of
, both net-outs xor
RND_IN and RND_OUT together and select which network
output will handle the retransmission. The assertion of
also signals the idle net-out to increment its retries counter.
After receiving a TURN byte, net-in transmits STATUS
and CHECKSUM much like RN1. The checksum is calculated identically
to RN1's checksum and similarly segregated into bytes (tn26). The
lowest two bits of STATUS, which RN1 uses for connection information,
are slight different for net-in. The encoding of these bits is shown
in Table .
The use of the 0b11 encoding makes this slightly inconsistent with RN1's status indication. It is possible to fail to make a connection through net-in for any of the following reasons:
Each network interface has an opportunity to access memory once every
eight real network cycles. Since the memory is 64 bits wide, this is just
frequently enough to transfer data at the full network data rate when
necessary. During an eight network cycle memory round, each logical
network interface has a designated access cycle on each shared bus (
i.e. address and data busses). The portion of the round belonging to
each logical network interface is shown in Figure .
When a network interface wants to use memory during its access cycle, it
asserts the want bus ( WB) signal during its designated WB cycle prior to
presenting the data to be read or written. Along with asserting WB,
the network interface should assert the appropriate word write enables
(<1:0>). When writing to either or both words of the
specified memory location, the appropriate word write enable should be
asserted. For memory reads, both word write enable should be deasserted.
The host bus controller (tn30) deals with turning the WB and
<1:0> signals into the appropriate enables for the SRAM
memory. The network interface does not support byte writes. Both WB
and
<1:0> should be asserted only during the network
interface part's respective cycle on the write enable bus.
Node memory operation timing differs somewhat when there are no dummy
cycles from when there are dummy cycles. With no dummy cycles, the network
interface will generally be performing back to back memory cycles in the
pipelined fashion required by the node bus. Figure shows
what the bususe from a single network interface looks like. This pattern
of usage is repeats as necessary for each memory interaction. As mentioned
above each network interface has its own designated cycle for use of the
data and address busses so each network interface uses this pattern
appropriately out of phase with its peers. During the R/W Addr cycle
the address of the next read data or the previous write data is presented
(see (tn25)).
When dummy cycles are present, each network interface only references
memory during the beginning of the each emulation cycle. As such, it is
not possible to optimize back to back memory cycles. Instead, within the
two node cycles following the beginning of the cycle, each network
interface performs a complete read or write operation. The processor is
then free to steal cycles during the remained of the emulation cycle
knowing that the network interfaces will not require use of the node busses
until the beginning of the next emulation cycle. Figure
shows the end of cycle and bus timing when dummy cycles are present. As
noted, the point at which the EC signal is asserted with respect to
the phase of a network interface depends on the network interface.
For network-input 0 (which is
out of phase with the processor
and hence the only unit from which the processor will be stealing bus
cycles), EC is asserted during its designated address phase one
node cycle before the network input uses its address bus. This allows the
bus controller adequate warning so that the address bus will be available
if the network interface wishes to perform a read operation.
The network traffic generated and received by the network interfaces conforms to the network transactions defined in (tn19) and (tn21). Message checksums are generated by the network interface. All other data contained in messages is provided by the node processor or obtained from memory at the locations indicated by the processor.
In this section the following conventions will be used to distinguish required and optional processor operations:
It is always optional to specify new buffer pointers. Checking acknowledgments is never required, but always recommended.
All the sequences in this section concentrate on the i/o operations between the processor and the network interface. Intervening computation by the processor is categorically omitted.
Only network outputs will actually originate network operations. This section briefly describes the way the processor uses net-out to issue network transactions.
Checking the success of a network operation is not explicitly shown in the sequences which follow. In general, the processor will want to read the net-out's STATE to check on its progress and perhaps look at the status words in memory. When a network output fails to successfully open within the configured number of retires, the network output will cease to attempt retransmission. The processor should recognize this occurrence when it checks the state of the network output.
A noop sequence proceeds as:
A reset sequence proceeds as:
A read sequence proceeds as:
A write sequence proceeds as:
An ROP sequence proceeds as:
When configured as a network input, the network interface will autonomously handle all of the incoming low-level network transactions described in (tn19) and [DeH90c] except for the ROP, remote operation, transaction which implicitly requires the processor's control.
These transactions require no node resources.
When a NOOP network transaction is received, net-in drops the
connection after returning its status and checksum bytes. See
Section for information on the status and checksum bytes.
Upon receiving a RESET network transaction, net-in initiates a
reset sequence for the node processor and holds the network connection
open. Net-in asserts for the appropriate number of
cycles (see (tn25)) then releases
to effect a
processor reset. When net-in first powers up,
should
be asserted. After performing the reset sequence, net-in monitors
the
signal. If the processor succeeds in booting,
net-in returns an ack_t; otherwise it returns an ack_f
(see (tn21)).
Net-in can directly handle the raw memory transactions described in
(tn21). This along with the RESET transaction allow the node to
be booted over the network without EPROMs (tn19) [DeH90b]. The
node bandwidth is sufficient to handle these raw operations at the full
network data rate (see Section ). The format of data received
and transmitted over the network during any of these transactions is given
in (tn21).
Upon receiving a read transaction, net-in returns the requested words at the emulation rate ( i.e. one word per emulation cycle). Following the last word, net-in sends a forward checksum byte on the data transmitted before closing the connection.
Write transactions are handled similar to read operations. One word is written into memory each emulation cycle. When the network is turned around following the transmission of the write data, net-in transmits an ack to indicate whether or not the write completed successfully. ack_f may occur for any of the following reasons:
On incoming write operations, the checksum on the address and length fields of the message must be correct before net-in will write any data to memory. This checksum is necessary to guarantee that random portions of a node's memory are not trashed by transmission errors (tn21).
In addition to autonomous transactions network inputs must handle remote
operations ( ROP's) so the processor can respond accordingly.
When an ROP is received, net-in places the contents of the
message in memory at the address specified by the out_buf_ptr.
The processor recognizes the arrival of the ROP by checking on the
state of net-in. Once received, net-in will hold the
connection open sending idle cycles over the network until the
processor sets up a response. During an ROP, the network can be
turned around as many times as the software requires. Once the initial
message is received, ROPs are handled in much the same way as
net-out handles ROPs (see Section ).
An ROP sequence proceeds as: