The Nostrum Protocol Stack and Suggested Services Provided by ...

December 6, 2002 1:51 pmNostrum - II.II.THE NOSTRUM ARCHITECTURE AND BACKBONEThe purpose of this section is to describe Nostrum architecture, an in-house developedplatform for SoC design using the Network on chip for strategy for communication.The Nostrum concept is to provide a platform with versatile communicationcapabilities, which can be reused for a large number of SoC designs.Some Terminology - Processes & ResourcesToday’s integrated digital systems can be seen to consist of computation and communication.Communication from our viewpoint is just the job of getting the right informationfrom one application to another, where an application is anything thatperforms computation. In order to prove the Nostrum architecture/backbone conceptcomputation is, in this document defined as, anything that performs/implementsa function. A functional unit like this is called a Process (P). One or moreProcesses defines a Resource (R). Examples of different Resources are Processor Elements,Memory Elements, I/O Elements etc.Any communication between Processes with a Resource is defined as internalResource communication; whereas communication between Resources, i.e. Processesresiding in different Resources, is considered to be inter-Resource communication.R 1P P 1 1 P 2P 3R 4P 5P 9R 2FIGURE 2.1Process/Resource MappingOne alternative view of the Resources is to see them as 100k-1M gates playgroundsfor the end-user for free utilisation. The purpose of the Nostrum backbone is to providea reliable communication platform for these Resources.The Idea Behind the Nostrum ConceptIn order to alleviate the designer from the cumbersome task of designing the communicationbetween different processes the Nostrum backbone offers a general butfixed packet switched communication platform, which can be reused for a largenumber of SoC designs. This communication platform is organised as a network overwhich the Processes communicate.DEFINITION 2.1DEFINITION 2.2Nostrum architecture An in-house developed platform for SoC design usingthe Network on chip for strategy for on chip communicationNostrum backbone A layered communication platform used within the Nostrumarchitecture for on chip communicationThe network can both be used for message passing between Resources as well as fordata distribution. However, from here on, both, true messages and any data distributionwill be referred to as Messages. The message passing between the Resources ispacket based, i.e. the Message is segmented and put into packets that are sent over5

December 6, 2002 1:51 pm Nostrum - 2.2more Processes are mapped onto a Resource.SNIRFIGURE 2.3Switch/Network Interface / Resource relationsAll Resources are equipped with a Network Interface (NI), connected to the Switch, tohandle the communication between the Resource core and the network. The NI handlesall the communication protocols needed to make the network as transparent aspossible to the Resource.However the Nostrum is not inherently dependent of the 2D-mesh topology, otherpossibilities might include folded torus, fat-trees etc. [11][12]. We have opted for a2D-mesh topology for four main reasons.• First, as analysed by Culler [11], (1) low dimension topologies are favoured whenwiring and interconnects carry a significant cost, (2) there is a high bandwidthbetween Switches, and (3) the delay caused by Switches is comparable to theinter-switch delay. This is the case for VLSI implementations on the 2-dimensionalsurface of a chip and practically rules out higher dimension topologies.• Second, we assume that all applications that we have in mind, e.g. telecom equipmentand terminals, multi-media devices, consumer electronics etc., exhibit ahigh degree of locality in the communication pattern. This is in stark contrast tothe objective of traditional parallel computer designed to minimise latency forarbitrary communication patterns.• Third, we assume a heterogeneous set of Resources with nodes on the peripheryoften specialised for I/O tasks. Hence, we rejected torus topologies in favour of amesh, open at the chip borders to simplify layout and wiring.• Fourth, the main difference between a “general” network, with a more or less randomstructure, and a regular 2D-mesh network is the addressing. In a generalnetwork, the lack of ordered structure raises the need for routing tables. Thetables are needed because the actual location of the destination, or the route to it,might not fully be known on beforehand. Due to the topology of the 2D-mesh, theaddressing of the Resources will become very easy. It is always known on beforehandwhere the Resource is physically/logically located. This property makes thedesign of the Switches simple and the need for routing tables is reduced or eliminated.2.2 An introduction to the Nostrum BackboneThe purpose of the Nostrum backbone is to provide a reliable communication infrastructurefor SoCs. The backbone is a part of the Nostrum concept and provides aversatile communication platform where the designer can explore and choose from aset of implementations with different levels of reliability, complexity of service etc.The trade-off cost is analysed and presented in terms of power, timing, reliability etc.7

December 6, 2002 1:51 pmNostrum - III.The backbone has been developed with a set of different communication protocols inmind (OCP, MPI, UDP, IP etc.)[23] [13]. Consequently, the backbone can be used forboth single-message passing between Resources (datagram based communication) aswell as for stream oriented data distribution (virtual circuit). The message passingbetween the Resources is packet based, i.e. the message is segmented and put intopackets that are sent over the network. Segmentation is needed because the possiblemismatch between message size and the packet payload size, i.e. the message mighthave to be sent as several packets due to large size.In order to make the Resources benefit from the backbone they are all equipped witha Network Interface (NI) to offer a uniform communication interface between theResource core and the network. The NI handles all the communication protocolsneeded to make the network as transparent as possible to the Resource. On top ofNostrum ProtocolStackApplicationRNINetworkInterfaceResourcesNetworkApplicationRNINetworkInterfaceCustom ProtocolStackFIGURE 2.4The Application/RNI/NI Interfacesthe NI resides the Resource Network Interface (RNI) (Figure 2.4). The RNI is the customhardware/software used to connect the Nostrum backbone protocol stack withthe communication protocols used by the Resource. The RNI handles all the applicationspecific communication issues. Some examples• In the case of the application being a memory the RNI could act as an arbiter orDMA.• If the application being a processor e.g. an ARM core [14] the RNI could act as abridge between e.g. the AMBA bus and the NI.• Also the RNI can act as a bridge between already proposed standards for communicationand the Nostrum, e.g. the VSI Alliance [1]The boundary between the NI and RNI is, however, not regulated within the Nostrumarchitecture. The Nostrum protocol stack can be implemented with an ‘arbitrary’depth, i.e. if the application only requests a very basic functionality the stack can bemore or less shallow.III.RELATED AREAS AND DEFINITIONS3.1 The Layered Communication ApproachIn order to make different Processes communicate, a standard needs to be definedfor the format of communication. Communication in our case means that regardlesswhat data format is used and regardless the physical and logical location of the8

December 6, 2002 1:51 pm Nostrum - 3.2transmitter and receiver of the communicating Processes, they will be able to understandeach other.DEFINITION 3.1 Messages The information/data exchanged by Processes are called MessagesThese intentions put a lot of complex requirements on the mechanism to implementcommunication. These requirements are quite diverse; the process communicationmechanism needs to be able to deal with different data formats, different priorities,the interaction with the underlying network etc.3.2 TerminologyIn order to implement this diverse functionality a vertical set of layers has beendefined, with each layer providing the layer above and below with a set of services.The driving force behind the layered approach is that the interface and the set ofservices a layer provides to the layers above and below is static whereas the implementationof these services is dynamic.DEFINITION 3.2DEFINITION 3.3Layer A Layer is an abstract notation used for grouping related functionalitiesin a “vertical” manner where the Layer’s position corresponds to the abstractionlevel usedService A service is defined as an agreed functionality which the particularlayer has to provideThis layered approach to communication to great extent enhances the possibilities ofperforming simulations and implementations of the different layers in a multitude ofdesign/model/specification languages; this due to flexibility of implementation. Italso alleviates changes in the protocol stack; i.e. it makes it very easy to change theimplementation of one or several layers, without affecting the others.Layer N + 1 Layer N + 1Layer NSAPSAPLayer NSAPSAPEntityEntityPeer ProtocolEntityEntityLayer N - 1SAPSAPLayer N - 1SAPSAPFIGURE 3.1Overview Over the Layered Approach with SAPsThese services a layer offer could further be divided into Functions implemented bythat particular Layer.DEFINITION 3.4Function A Function is a concrete call for a services a layer providesThe Functions within a Layer are collected into groups called Entities. These Entities,9

December 6, 2002 1:51 pm Nostrum - 3.3within the same layer communicate with their peers using one or more Protocols, i.e.rules for interpretation of information communicated within that Layer. These unitsof information/of agreed data format used to implement the Protocol are called ProtocolData Units (PDUs).DEFINITION 3.5DEFINITION 3.6Entities Entities are defined as a group of Functions that a layer could performin order to implement its servicesA Protocol Data Unit - PDU A Protocol Data Unit (PDU) is the agreed dataformat within a Layer, i.e. used by peersDEFINITION 3.7 Protocol A Protocol is the interpretation of the Protocol Data Unit within aLayerIn order to provide these services interfaces to the upper and lower Layers has to bedefined. These interfaces are known as Service Access Points (SAPs). As long as aLayer agrres with its SAP and implements its services, “any” implementation may beused.DEFINITION 3.8Service Access Point - SAP A Service Access Point (SAP) is the communicationinterface between adjacent LayersWhen describing the different layers the term payload is used for the data transmittedby the current layer as part of the service provided to the upper layers and theheader is the additional information added/needed by this particular layer in order toimplement its services. The payload might contain headers used by subsequent layers.A PDU consists of header and payload unique to that particular layer.3.3 Switching TechniquesIn a communication system there are several possibilities when it comes to transmittingmessages between parties. In order to classify these different possibilities anomenclature based on the work of Irvine et al. is used [15]. Mainly, there exists twopossible methods for arranging point to point communication through a network:connection-oriented and connection-less. In the connection-oriented case, a path isset up between the transmitter and the receiver; whereas in the connection-lesscase, the information is encapsulated into units called datagrams, or packets whichcan make their way through the network without an explicit connection is beingestablished. Traditional telecommunication networks have been connection-oriented,while computer networks have been connection-less.One further partition can be made when describing the communication in networks:circuit switching and packet switching. Circuit switching is equivalent to having adedicated path set up between transmitter and receiver and the information is sentas a stream. Circuit switching is always connection-oriented. Since the Nostrumbackbone is packet switched, circuit switching will not further be described.Packet switching can be either connection-oriented or connection-less. The connection-lesspacket switching is employed when a small amount of information is to besent; setting up a connection to transfer this information would be inefficient due tohigh initial cost. The connection-less packet switching has the advantage that thenetwork is only used when information is to be sent. The connection-less packetswitching can further be divided into message switching and cell switching.10

December 6, 2002 1:51 pm Nostrum - 3.3Message switching is also known as store-and-forward; due to that the whole messageis treated as one packet of arbitrary length which is transferred between theSwitches, buffered until the full message is received, and then transmitted to thenext Switch.In cell switching, all packets have the same size. This common format reduces theneed for buffering in the Switches since a packet can be forwarded immediately.However, since all packets are equipped with a header, the message might be sent inseveral packets and the utilisation of the network will be less.CommunicationPacket switchingCircuit switchingconnection-lessConnection-orientedCell-switchingMessage switchingDatagram switchingVirtual CircuitFIGURE 3.2Deflective routing Wormhole routingCommunication NomenclatureTDMAThere are two main categories of cell switching techniques: datagram switching andvirtual circuit.In a datagram switching-based system, all packets are considered as separate unitsand will be routed through the network as such. This means that different packets ofthe same message may take different routes through the system and possibly arrivedisordered. To order packets/keep the order of packets, a sequence number isappended to the packet’s header. The main benefit of this approach is the ability offast adaptation to changes in the network, i.e. congestion, broken links etc. In theNostrum backbone a deflective routing (hot-potato) [16] approach is chosen to implementthe datagram switching. Deflective routing uses no explicit buffers in theSwitches but relies on that packets can be sent immediately in a “sufficiently” gooddirection in the network, hence the nickname hot-potato routing. The reason forchoosing this approach is the little need for hardware.In the virtual circuit cell switching approach a path, between the transmitter andreceiver, is set up prior to transmission. All the packets transmitted will follow thispredestined path during transmission. Consequently packets can not arrive out oforder and no explicit sequence number is therefore needed. After transmission, thepath is possibly torn down, this since only a limited number of virtual paths are possible.In all other aspects, this approach is very similar to the connection-orientedpoint-to-point communication. In the Nostrum backbone two different variants ofthe general virtual circuit is used. Wormhole routing and Time-Division MultipleAccess (TDMA).In Wormhole routing [17] one additional requirement has been put on transmissioncompared to the general virtual circuit approach. The requirement is that the packetsare sent out continuously. When the first packet in a message arrives to a Switch,11

December 6, 2002 1:51 pm Nostrum - 3.4a routing decision is taken and the packet is routed in that direction. All the immediatelysubsequent packets belonging to this message will be routed in the same direction.The last packet, belonging to the message, tears the connection down. Theobserved behaviour of this will be that of a worm of packets, trailing its way fromtransmitter to receiver. Since the routing decision is only taken dynamically for thefirst packet but not for the rest this approach resides both under the datagram andthe virtual circuit category.In the case of the virtual circuit TDMA, a path between the transmitter and thereceiver is set up prior to transmission. Once the path is set up, the transmitter/receiver pair is guaranteed/subscribed to a fraction of the full bandwidth availableover that particular path. The fraction is implemented, as the TDMA suggests, as atime-slot when the transmitter/receiver pair has unique access to that particularpath. A time-slot in the Nostrum backbone is one network transmission clock cycleand can vary from one to four. A network transmission clock cycle is the time it takesto transmit a packet from one Switch to another neighbouring Switch. The TDMAchannel can be set up either statically or dynamically i.e. either, during start-up orin run-time.3.4 The OSI ModelAs a starting point for the Layers employed, the OSI model is used as a basis. Thismodel is well described in James Irvine et al.’s “Data Communications and Networks[21][15] and responsibilities of the different layers are described briefly below.Application (Layer 7): The application layer provides the user interface to the communicationsystem. Examples include electronic mail, distributed databases andnetwork operating systems.Presentation: The presentation layer provides a common format to the applicationlayer. It allows processes to communicate even if they use different representations.For example, different web browsers provide different user interfaces- different applications- but all understand the HTML page format, the JPEG image format etc.These file formats are examples of presentation layer protocols, while the browsersthemselves reside in the application layer. Services provided by the presentationlayer therefore include things like file transfer, virtual terminal protocols, compression,code transformation and encryption.Session: The session layer breaks down the services sent from the presentation layerinto basic data flows which can be sent over the system. It manages interactionsbetween pairs of communication application processes. ‘Sessions’ can be short orlong, involving one or many message interactions. Services provided to the presentationlayer include session establishment/termination, authentication/accounting,synchronisation, data transfer and exception reporting.Transport: The transport layer provides the initial establishment of communicationchannels, the transfer of data and the final release of the channel. For data transfer,it interfaces with the network layer to ensure an error-free virtual point-to-point connection,where the communications arrive in the correct order. In the case whencomputers have multi-tasking capabilities, the transport layer ensures that the communicationgoes to the correct process and performs any multiplexing or de-multiplexingrequired.12

December 6, 2002 1:51 pmNostrum - IV.Network: The network layer interconnects data link communication paths into a globalnetwork that connects all open systems, i.e the routing of the communicationfrom the source node to the destination node. It provides the rules for routing packetsthrough the network, and for flow control where necessary, to ensure that not tomany packets enter the system.Data Link: This layer ensures reliable communication over the physical layer. Theprotocol determines the structure of data, i.e. frame or packet size, and deals withaspects such as flow control, error detection, and error recovery.Physical: This layer is concerned with the physical aspects of the communicationlink - mechanical and electrical - and provides the means to transit bits of dataacross a continuous communication path. The protocol for this layer sets detailssuch as cable sizes, loss and frequency characteristics, connector types, pin arrangements,voltage levels, and transmission coding.IV.THE NOSTRUM PROTOCOL STACKIntroductionThe overall purpose with the Nostrum Protocol Stack used within the Nostrum backboneis to aid in the design of communication on chip.The communication backbone is divided in several layers with different responsibilities.These responsibilities are defined from a conceptual perspective where similarresponsibilities are grouped into the same layer. These layers have a one-to-one correspondenceto the Nostrum Protocol Stack.Communication is based on the assumption that you are either designing an applicationor designing functionality in the protocol stack. In both cases you will requirea set of services from the communication backbone, e.g transmit packet or suchlike.User of ServiceSAPPacket Transmissionfunctionf_transmitPacket(dpid, data)f_packetReceived(dpid, data)NostrumFIGURE 4.1Overview of User of Service versus the Nostrum backboneThe broker of these services are called a Service Access Point (SAP). A service is implementedby a set of Functions. Between the User of Service and the Service AccessPoint there exist a contract. The contract is mutual, i.e the User of Service is notonly employing the service of getting something done, we also implement the functionsrequested from the SAP in the opposite direction!EXAMPLE 4.1 In Figure 4.1 Overview of User of Service versus the Nostrum backbone. Theservice is the one of Packet Transmission and the functions offered by the SAP is the13

December 6, 2002 1:51 pmNostrum - IV.f_transmitPacket. The function that the User of Service needs to be able to respond to is thef_packetReceived. Both these functions have data dpid (destination process id) as arguments/outputs.function f_transmitMessage(dpid, data, MSN) f_MessageReceived(dpid, data, MSN)SAPMessage TransmissionUser/Provider of ServiceSAPPacket Transmissionfunction f_transmitPacket(dpid, data) f_packetReceived(dpid, data)Nostrum ApplicationFIGURE 4.2Overview Over User of Service and the Application versus the Nostrum backboneIn the case of designing functionality in the protocol stack the User of Service is alsoa provider of services to the layer above, as depicted in Figure 4.1 Overview of User ofService versus the Nostrum backbone. From the figure we can see that there are actuallyfour kinds of services that exist. Offered and required services both exist in theup-stream and the down-stream data flow in the Nostrum protocol stack.The services offered and requested from the SAP below should all be treated as servicesoffered and requested from the Nostrum backbone. This regardless where in theprotocol stack the User of Service reside.The “opposite” of the services offered and requested by the Nostrum backbone arethe services offered and requested by the Application above. In the same manner aseverything below was Nostrum now everything above is Application and as beforethis regardless where in the protocol stack, you as the User of Service reside.DEFINITION 4.1 Service Offered to ApplicationThe service that the User of Service isrequired to provide in order to fully utilise the SAP contract to the Application aboveDEFINITION 4.2 Service Required from ApplicationThe service the Application above mustoffer to the User of Service belowDEFINITION 4.3 Service Offered by NostrumThe service the SAP/Nostrum offers to theUser of Service, i.e. the service the SAP below offers to the User of Service above inthe Nostrum protocol stackDEFINITION 4.4 Service Required by NostrumThe service the User of Service above mustoffer to the Nostrum in order to make the SAP fulfil its promises14

December 6, 2002 1:51 pmNostrum - IV.In order to get these definitions right one has to think from the User/Provider ofService’s perspective.By defining the services in this hierarchical manner a fully functioning communicationplatform can be built in a structured and parallel way.ClocksWith each of the SAP every layer provide these might be the a clock associated. Thepurpose of the clock is to enable the synchronous clock driven model[20]. Usuallythe clocks are used to synchronise the function calls and set up deadlines for whendata/function calls gets stale. Also the clocks gives the possibility of describing executiontime for different function implementations. An example of this behaviour isthe implementation of the Data Link Layer Clock (LLC). The LLC decides the speed atData Link Layer ClockUser of ServicePacket TransmissionfunctionCall() functionCall()NostrumData Link Layer ClockFunction calls/ Sent frames 1 2 3FIGURE 4.3The Data Link Layer Clockwhich frames are accepted from the SAP. On every raising edge of the LLC one frameis moved from the output of a Switch to the corresponding input of the Switch in theother end of the data channel.CommentsThe following discussion might be omitted in the final documentThe speed at which packets are sent out is decided by the Network Layer Clock. Theratio between the Network Layer Clock and the Data Link Layer Clock decides the15

December 6, 2002 1:51 pmNostrum - IV.duration of packets in terms of Data Link Layer Clock ticks.Data Link Layer ClockPacket Duration = 1Network Layer ClockPacket Duration = 2Sent PacketsNetwork Layer ClockSent Packets1 2 3 4 5 61 1 2 2 3 3FIGURE 4.4= HeaderIn the most basic set-up of the network, the duration of a packet is one Data LinkLayer Clock tick. This means that the Network Layer Clock is running at the samespeed as the Data Link Layer Clock, i.e. the Network / Data Link Layer Clock ratio isone. This ratio is from here on referred to as the packet Duration. Ifthepacket Durationis, e.g., two, this naturally means that one packet is allowed to occupy two NetworkLayer Clock clock-cycles. The reception and emission of packets are at thisstage handled in a Store & Forward manner [CULLER - PARALLEL COMPUTER...] p.756 ff. As suggested by [CULLER - PARALLEL COMPUTER...] this could be handledin several different ways, further investigation is therefore needed.Even though a packet, normally, only occupies one Data Link Layer Clock cycle, itsometimes is advisable to extend the packet Duration to more than one Data LinkLayer Clock cycle.The reason for letting one packet occupy more than one Network Clock cycle is theimproved Header/Payload ratio. In the most basic setup with a packet Duration ofone, the Header/Payload ratio is only approximately 50%. This under the assumptionmade in.The layersA standard protocol stack for a Network-on-Chip platform allows the separate, independentdevelopment and validation of Resources, communication network andapplications as long as they comply with the defined protocols. This clear separationfacilitates the systematic reuse of Resources, communication infrastructure andapplication features.As a starting point for the layers employed in the Nostrum backbone protocol stack,the OSI model [15] is used as a basis as proposed by others [5][6][7][9]. With thiswell-known layered model in perspective the Nostrum backbone is described interms of the proposed terminology. However, these layers only exist as a conceptualaid in the design process, once the design is set the layers might be collapsed at alogic level in order to enhance the possibility of hardware implementation optimisations.Within the concept of the Nostrum backbone resides the three “compulsory” layers;the Network, the Data Link, and the Physical Layer.16

December 6, 2002 1:51 pm Nostrum - 4.1Together, one of the most basic service these three layer provide is the service ofdelivering packets. Please note that the purpose of this brief description of the layersbelow is to define the abstraction level where the respective layers are active. Thesuggested services works as in inspiration how it could work. A more detailed discussingis found in the next chapter Section V - Suggested Services Provided by theNostrum Backbone, we also present the suggested services in more detail.4.1 Physical LayerThe lowest layer is the Physical Layer (PL). The purpose of the PL, from a simulationpoint of view, in the Nostrum protocol stack is to give the possibility of error introduction.The spatial and temporal patterns of these errors are extracted from modelsof the physical implementation of the hardware. The errors are introduced at bitlevel.If no errors are introduced the layer is transparent.Suggested ServiceThe layer’s responsibility is to move a word from the output of aSwitch to the corresponding input of the next with the possibility of introducingerrors.Protocol Data UnitThe Physical Layer’s PDU is the word. The word is an array of bits,in the current implementation the word size is 128 bits.CommentSince the Physical Layer is the bottom layer it has no Nostrum backbone torely on and therefore no service requests from the backbone needs to be handled.The only services are the ones offered to the Application, i.e. the Data Link Layer4.2 Data Link LayerThe Data Link Layer (LL) provides the reliable transfer of information across thephysical link.Suggested ServiceThe LL is responsible for the sending of frames with the necessarysynchronisation, error control, and flow control. In the standard definition of theservices the layer is only concerned with the synchronisation of the transmission offrames.Protocol Data UnitExcept for the propagated frame information about detected/correctederrors can be set up-stream. The PDU of the extended services in the DataLink Layer is the Error Correction Codes (ECCs) appended on the frame.CommentIn order to fit the SAP of the PL the frames of the LL might get segmented orconcatenated. The reason for making such option available is to give the opportunityof adopting to changes in the physical implementation without actually affect theupper layers. It also gives the possibility to elaborate different bit-widths contraphysical clock speeds, e.g. is it beneficial to use a double-sized bus and half thespeed?4.3 Network LayerThe Network Layer’s (NL) responsibility is to provide the transport of packets fromthe transmitting Resource through the network to the receiving Resource. The NL hasthe responsibility of how a packet should be routed.The NL also has the responsibility for management and mapping of the physical/logicaladdresses to the Process identifiers associated with the different Processes in the17

December 6, 2002 1:51 pm Nostrum - 4.4different Resources. This management involves the mapping/re-mapping of the Process-Resourcepairs, either at system set-up or during run-time.Suggested ServicesWithin the standard definition of the services, two different services,handling data transmission, can be requested. The first service is the datagramservice. Any packet sent to the NL will get routed dynamically through the network.The second service is the virtual circuit switching service. On request, a virtual circuitbetween the transmitter and the receiver is set up.Protocol Data Unit PDU.SPid MSN DPid messageTransportf_transmitPacket(data, DPid, SPid, MSN)SPidMSNDPidmessageMidpayloadMidMidMidPSN 3PSN 2PSN 1payload 3payload 2payload 1TL-PDUDPiddataNetworkf_transmitPacket(data, DPid)DAddr = f(DPid)Priority DAddr payloadpacketProtectionSchemedataData Linkf_transmitFrame(data, protectionScheme)ECCframeframedataPhysicalf_transmitWord(data, errorModel)wordwordPossible error introductionErrorModelFIGURE 4.5The Nostrum Protocol Stack and the SAPs for the Datagram Service Implementation4.4 Transport LayerThe Transport Layer (TL) handles the establishment of communication. In the case ofthe virtual circuit it issues an establishment of a connection and handles the transferof data, i.e. it ensures a virtual point-to-point connection, where the messages aresegmented into packets which are sent via the NL. At the destination the packets areassembled, ordered and the message is delivered and the connection is torn down. Itmay also provide the service of traffic control, i.e. the load of the network is moni-18

December 6, 2002 1:51 pm Nostrum - 4.5tored and overload of the network is hence avoided.Suggested ServicesThere are two kinds of standard services that the TL provides, datagramdelivery of messages and virtual circuit transmission of data.In the case of the datagram-message delivery service a message gets delivered as fastas possible to the destination Process decided by the DPid. In the case of anextended service implementation the additional services of lost packet detection/correctionand priority message delivery is also available.Virtual circuit transmission differs from the datagram approach by that a virtualpath needs to be set up prior to transmission. The choice of the virtual circuit isbased on the tolerance to latency due to the cost of connection set up. Once issued aNetwork Layer call for the virtual service set up is done, when the connection isestablished a guaranteed bandwidth exists between the transmitter and receiver.Even though not implemented there exists an inherent possibility of multi-cast virtualcircuit since the connection is based on full duplex; i.e. the virtual path is twoway.Protocol Data UnitIn the case of a datagram transmission the TL segments the messageinto datagrams (packets). The first packet in the message carries the DPid sothat the receiver will be able to determine the source of the message. The DPid is alsoforwarded to the SAP of the NL. Every packet gets tagged by a unique Message Id(Mid) together with a Packet Sequence Number (PSN) which is necessary for propermessage recovery at the destination. The last packet also gets the Last Packet (LP) bitset.The protocol for the virtual circuit service is similar to the datagram protocol. Thevery difference lies in that a connection needs to be established. Even not present inFigure 4.5 there exist additional data fields in the SAP dedicated for virtual path setup. The request for the establishment of a virtual circuit is performed by sending aDestination and Source Process id virtual circuit request. Once granted an acknowledgementwill be sent back and the TL is now transmitting any message with thededicated Destination and Source Process id pair over the connection.4.5 Session LayerThe purpose of the Session Layer (SL) is the administration of the send- and receivebuffersin order to handle the interfaces between the network and the application ata higher level. This might also include traffic profiling in order to employ the rightservice from the TL.V. SUGGESTED SERVICES PROVIDED BY THE NOSTRUM BACKBONEThe purpose of this chapter is to give a concrete proposal of services that could beincluded in the standard repertoire of the Nostrum backbone. However, as statedbefore, these suggestions should be treated as firm but not fixed and should undergocareful review and changes and additions are most likely to be made.The presentation of the layers and the services is pretty much a reprint of the previouschapter Section IV - The Nostrum Protocol Stack with incremental details added.Even though the different services in inherently dependent on the physical layout of19

December 6, 2002 1:51 pm Nostrum - 5.1the Nostrum architecture the Section 5.1 - The NoC Layering - “Physical” Interpretationmight serve as a good reminder of the respective realms where the different servicesreside.Data LinkPhysicalNetworkTransportNetwork InterfaceResourceSSSSRRRRSS S SR R R RS S S SFIGURE 5.1 The NoC Layering - “Physical” InterpretationSome AssumptionsThe presentation of the different layers and the services is based on the followingassumptions• The network is an n × n 2D mesh network with no torus, i.e. no wrap-around• The size of the n × n network is assumed to be at most 32 × 32• No Packet in the network will have to travel more than twice the diameter of thenetwork, i.e 2 × 2 × n = 2 × 64• Absolute addressing is used; if relative addressing is to be used, one additionaladdress bit needs to be added in each dimension, i.e. two bits for an n × n network5.1 The Physical LayerThe lowest layer is the Physical Layer (PL); the layer’s responsibility is to move a wordfrom the output of a Switch to the corresponding input of the next. The purpose ofthe Physical Layer is to enable error insertion in simulation. If no errors are insertedthe layer is transparent. The domain of the Physical Layer is depicted in Figure 5.220

December 6, 2002 1:51 pm Nostrum - 5.1The Physical Layer DomainResourceTransportNetworkData LinkPhysicalData LinkNetworkPhysicalTransportResourceFIGURE 5.2The Physical Layer Domain5.1.1 The Service of Word Transmission - StandardThe layer’s responsibility is to move a word from the output of one Switch to the correspondinginput of the next.f_transmitWord(word [transmission time])Purpose:Parameters:To transmit a word from the output of one Switch to the corresponding input ofthe next.word - The word to transmit[transmission time] - The transmission time is optionally specifiedTime:The SAP accepts a function call any time and executes it immediately. Thetime for transmission is optionally specified, i.e. the time before af_wordReceived call is issued in the receiving Switch.Service Required from ApplicationThe Application is required to accept af_wordReceived function call at any time. Unanswered function calls will result inloss of data.f_wordReceived(data)PurposeService Offered by NostrumNone. The Physical Layer is the lowest layer, i.e. no Nostrumbelow!Service Required by NostrumNone. The Physical Layer is the lowest layer, i.e. no Nostrumbelow!Protocol Data UnitThe Physical Layer’s PDU is the word. The word is an array of bits,21

December 6, 2002 1:51 pm Nostrum - 5.1in the current implementation the word size is 128 bits.Physicalf_transmitWord(data)Service Offeredf_wordReceived(data)Service RequiredFIGURE 5.3The Behaviour of the Physical Layer5.1.2 The Service of Word Transmission - ErroneousThe layer’s responsibility is to move a word from the output of a Switch to the correspondinginput of the next with the possibility of introducing errors. The SAP acceptsa function call any time and executes it immediately. The time for transmission isoptionally specified.Also the possibility of error introduction is supported. A suggested error model fieldmight consist of errorModel = {none | model1(param1, param2,...) |model2(param1, param2,... ). The model and parameters could be any and is currentlyunder investigation[22].f_transmitWord(data, errorModel, [transmission time])Service Required from ApplicationThe Application is required to accept af_wordReceived function call at any time. Unanswered function call will result in lossof data.Also an additional field, errorIntroduced = {true | false}, for marking whetherthe data was corrupt or not has been introduced.f_wordReceived(data, errorIntroduced)Service Offered by NostrumNone. The Physical Layer is the lowest layer, i.e. no Nostrumbelow!Service Required by NostrumNone. The Physical Layer is the lowest layer, i.e. no Nostrumbelow!Protocol Data UnitThe Physical Layer’s PDU is the word. The word is an array of bits,in the current implementation the word size is 128 bits. In addition to the word PDUsome formats for the error creation, possibilities might be needed?.Physicalf_transmitWord(data, errorType)Service Offeredf_wordReceived(data, errorIntroduced)Service RequiredPossible Error IntroductionFIGURE 5.4The Behaviour of the Physical Layer22

December 6, 2002 1:51 pm Nostrum - 5.25.2 The Data Link LayerThe second lowest layer is the Data Link Layer (LL); the layer’s responsibility is tomove a frame from the output of a Switch to the corresponding input of the next, i.e.the transfer of information across the physical link. The possibility of error correctionand detection is introduced. The LL is responsible for the sending of frames withnecessary synchronisation.The frames of the Data Link Layer might get segmented or concatenated in order tofit the SAP of the Physical Layer. The reason for making such option available is to beable to adopt to changes in the physical implementation without a actually affect theupper layers. It also gives the possibility to elaborate different bit-widths contraphysical clock speeds, e.g is it beneficial to use a double-sized bus as half the speed?The domain of the Data Link Layer is depicted in Figure 5.5 The Data Link LayerDomainResourceTransportNetworkData LinkPhysicalData LinkNetworkPhysicalTransportResourceFIGURE 5.5The Data Link Layer Domain5.2.1 The Service of Frame Transmission - StandardThe layer’s responsibility is to move a frame from the output of a Switch to the correspondinginput of the next. The Data Link Layer (LL) provides the transfer of informationacross the physical link. The LL is responsible for the sending of frames withnecessary synchronisation. The SAP accepts a function call any time and executeson next raising Data Link Layer clock. The time for transmission is decided by implementationand measured in LL clock ticks.f_transmitFrame(data)Service Required from ApplicationThe Application is required to accept af_frameReceived function call at any raising LL clock. Unanswered function calls willresult in loss of data.f_frameReceived(data)Service Offered by NostrumThe service needed from the Nostrum backbone is the oneof Error Free Word Transmission. The function in particular isf_transmitWord(data)Service Required by NostrumIn order to be able to request the service of Error Free23

December 6, 2002 1:51 pm Nostrum - 5.2Word Transmission we must implement the function off_wordReceived(data)Protocol Data UnitThe Data Link Layer’s PDU is the frame. The frame is an array ofbits, in the current implementation the frame size is 128 bits.Data LinkPhysicalf_transmitFrame(data)f_frameReceived(data)Service OfferedService RequiredUser/Provider of Servicef_transmitWord(data)f_wordReceived(data)Service OfferedService RequiredNostrumData Link Layer ClockFIGURE 5.6The Behaviour of the Data Link Layer5.2.2 The Service of Frame Transmission - Single Bit Error DetectionThe layer’s responsibility is to move a frame from the output of a Switch to the correspondinginput of the next Switch. The Data Link Layer (LL) provides the transfer ofinformation across the physical link. The LL is responsible for the sending of frameswith necessary synchronisation.In addition to the standard service In the Single Bit Error Detection Service any singlebit errors occur in the protected field of the data an error will be signalled.The SAP accepts a function call any time and executes on next raising Data LinkLayer clock. The time for transmission is decided by implementation and measuredin LL clock ticks.Also the possibility of error detection is introduced. Any single bit error will bedetected in the protectionScheme(start, end) field. The start and end parametersdenotes the start and end position of the protected field. If protected is left out errordetection is omitted. As well as the model and parameters for generating errors isunder investigation the scheme for error detection is under investigation as well[22].f_transmitFrame(data [ protectionScheme(start, end) ])Service Required from ApplicationThe Application is required to accept af_frameReceived function call at any raising LL clock. Unanswered function calls willresult in loss of data. In addition to the standard Service of Frame Transmission andextra date field of errorDetected is added. The format of errorDetected = {true |false} (boolean).f_wordReceived(data, errorDetected)Service Offered by NostrumThe service needed from the Nostrum backbone is the one24

December 6, 2002 1:51 pm Nostrum - 5.3of Service of Word Transmission - Erroneous. The function in particular isf_transmitWord(data, errorModel [, transmission time])Service Required by NostrumIn order to be able to request the service of Word Transmission-Erroneous we must implement the function off_wordReceived(data, errorIntroduced))Note that we are not required to deal with the information errorIntroduced this fieldonly serves as a reference for analysis of the error detection implemented.Protocol Data UnitThe Data Link Layer’s PDU is the frame. The frame is an array ofbits, in the current implementation the frame size is 128 bits. In addition to these128 bits, additional bits will be needed for the error protection scheme used. Theexact number is not known but will depend on size of the protected field[22].CommentsThis is a section for additional comments around the topic of error detection.This part might be omitted in the final document.What errors should be detected?• No detection - trivial!• Errors in the full frame, i.e an error will be detected anywhere in the frame. This isthe safest on since it ensures that packets will be sent and delivered flawless bythe network. One drawback that is easy to spot is the computational overheadrequired. Computational overhead means more time, more area, and more power.• Detection of errors which potentially might reside in the protection field in theframe. This field may correspond to important information in the header part of apacket at the Network layer. This since a potential error, in the Header, is moredangerous to the network, i.e. a miss-routed data might cause larger damage to acritical application. The designer might also consider a lighter Header protectionin order to save time, area, or speed.If errors are detected, what’s the counter-action?• Removal of packets without notifying upper layers• Re-sending of packets. Either by buffering packets in the Switches or in the NetworkInterface, i.e. should the errors be detected/repaired by the Data Link Layeror the Lower Transport Layer?• ...5.3 The Network LayerThe next layer is the Network Layer. This layer is responsible for the, error free,transport of packets from the transmitting Resource through the network to thereceiving Resource. The Network Layer has the responsibility of how a packet shouldbe routed.The domain of the Network Layer is depicted in Figure 5.5 The Data Link Layer25

December 6, 2002 1:51 pm Nostrum - 5.3DomainResourceTransportData LinkPhysicalNetworkData LinkPhysicalNetworkTransportResourceFIGURE 5.7The Network Layer Domain5.3.1 The Service of Packet Transmission - StandardThe Standard service provide the transport of packets from the transmittingResource through the network to the receiving Resource. The Network Layer has theresponsibility of how a packet should be routed. The standard service provide noerror detection/correction capabilities and therefore the standard service of FrameTransmission can be utilised, i.e. no errors are introduced.The time for transmission is decided by implementation together with the characteristicsof the network (load, topology etc.) and is measured in NL clock ticks. The SAPaccepts a function call f_transmitPacket(data, pid) any time and executes on nextraising Network Layer clock IF there is network capacity available! If the networkcurrently is congested a counter f_networkCongestion() function call is raised andthe packet has to be re-transmitted, i.e. a new f_transmitPacket(data, pid) functioncall must be made. The function call of f_transmitPacket(data, dpid) have twodata fields; data which is the data that the Application wants to transmit togetherwith a dpid (Destination Process identifier) which identifies the destination of thepacket. Inside the network layer the dpid will be translated into a physical addressthat is used in the routing in the network.f_transmitPacket(data, pid)Service Required from ApplicationThe Application is required to accept af_packetReceived(data) function call at any raising LL clock. Unanswered functioncalls will result in loss of data.f_packetReceived(data, pid)Also the function of f_packetTransmitted() needs to be implemented. This functionis raised any time any f_transmitPacket call is successful, i.e. the packet could besent. The function has no parameters.f_transmitPacket()Service Offered by NostrumThe service needed from the Nostrum backbone is the oneof Frame Transmission - Standard. The function in particular is26

December 6, 2002 1:51 pm Nostrum - 5.3f_transmitFrame(data)Service Required by NostrumIn order to be able to request the service of The Serviceof Packet Transmission - Standard we must implement the function off_frameReceived(data)required by the Frame Transmission - Standard serviceNetworkf_transmitPacket(data, pid)Service OfferedUser/Provider of Servicef_packetTransmitted()f_packetReceived(data, pid)Service RequiredNetwork LayerClockf_transmitFrame(data)f_frameReceived(data)Data LinkService OfferedService RequiredNostrumData LinkLayer ClockFIGURE 5.8The Behaviour of the Network LayerProtocol Data UnitThe Data Link Layer’s PDU consists of the packet and some contoldata fields. The PDU is an array of bits, in the current implementation the PDU sizeis 128 bits. Further the PDU is divided into the following fieldsdata - 98 bits, the payload, i.e. the data that is sent from the source to destination.dpid - 10 bits, the Destination Process id. This tag is used to identify the receivingprocess.logic address - 10 bits. The logic address used for the routing, i.e. the addressused to determine where the destination process physically is locatedpriority field - 10 bits. This field is used in order to enhance routing.How these different fields are used is up the designer of the service. One suggestedapproach is to see the packet as a carrier of information over the network. Within thepacket the data fields of data, pid, logic address and a Hop Counter (see commentsection) is present. Other information which is used to enhance the performane ofthe network may reside in the priority field.Several different proposal have been made how to use this field, e.g. for• Hop Counter (See Comment section)• Proximity Congestion Awareness[19] implementation, i.e the field is used foravoiding congesting by transmission of local congestion The Proximity CongestionAwareness (PCA) is an incoming signal to the Switch from all its surroundingneighbours. The signal is used to signal the degree of congestion in this direction.27

December 6, 2002 1:51 pm Nostrum - 5.3• Empty packet indication• As a pure priority set by the application at the f_transmitPacket(...) functioncall• As channel selector in the case of a virtual channel implementationIn the implementation of the PCA the priority field could be interpreted in the followingwayHop Counter - 5 bitsPCA information - 4 bitsEmpty packet indicator - 1 bitCommentsThis is a section for additional comments around the topic of routing. Thispart might be omitted in the final document.The Destination Address is the “obvious” physical destination address whereas theHop-Counter is a counter tracking the number of hops a packet has done, i.e. theHop-Counter is increased by one for every hop taken. The higher Hop-Counter, thehigher priority that packet will get in case of a competitive situation in the SwitchIn our current implementation ten bits have been used for the Destination and theSource Address each, whereas seven bits have been allocated for the Hop-Counter.The reason for choosing seven bit is that we assume that the maximum hops apacket has to undergo under any circumstances is twice the diameter of the network,i.e. 2×2n.5.3.2 The Service of Pid to Logic Address Mapping Administation - StandardThe Standard service of Pid to Logic Address Mapping provides the user with a set offunction for manipulating the mapping of Pids to Logic Addresses i.e. where a processis currently running. Suggestedf_getSelfAddressf_mapPidtoAddressf_sendPidandAddresstoAddressf_sendPidandAddresstoPidf_getAddressf_getPid(5.3.3 The Service of Network Diagnostics - Standardf_getPCA V?f_snoop ?Network Layer - ControlBetween the Switches some control signals might be needed in order to secure thetraffic flow.packets Not Accepted One of these control signals is the Packets Not Accepted signal.The Packets Not Accepted signal is a signal used for signalling whether a Switchaccepts packets from one of its neighbouring Switches the next transmit round. Thereason for raising this signal could be on of the following:• The Switch’s corresponding Resource wants to transmit a Message on a too heav-28

December 6, 2002 1:51 pmNostrum - VI.ily loaded network. The raising of this signal guarantees that a Message can besent out the next transmission round.• The Switch is broken or unavailable due to multiple sized Resources, see• A priority Message path might have been set up - this is however not definedwithin the original NoC Backbone set-up. For more information seeIn the current implementation one bit has been allocated for the Packets NotAccepted signal.VI.SOME GUIDELINES WHEN DESIGNING FOR NOSTRUMThe purpose of this chapter is to give some general guidelines when designing for theNostrum. These guidelines apply both to the end application designer as well as tothe designer of functionality in the protocol stack. In both cases you as the designerare the User/Provider of Services. Now there exist three main questions.Q1. Where in the protocol stack does the functionality that I intend to implement fitin?Q2. What are the service that I want to offer to the application above?Q3. What are the services that I need/require in order to implement my services?A1. The answer to the question one is most easily found by looking at the suggestedresponsibilities in Section IV - The Nostrum Protocol Stack. The servicesdefined here are suggestions and should be treated as guidelines for further development.However, the Protocol Data Unit used gives a clear hint of what kind of servicesthis layer is intended to provide.A2. The answer to question two must be the easiest AND the trickiest one... It’seasy in the sense that you wanted to implement some kind of service and that’s thereason for reading this, i.e. this is the answer that you thought you already had ;)On the other hand, to formulate this in a comprehensive way will be hard :( but don’tdespair. My tip is to formulate it in terms of function calls. These function calls29

December 6, 2002 1:51 pmNostrum - VII.should alone suffice when implementing the service intended.function f_transmitMessage(dpid, data, MSN) f_MessageReceived(dpid, data, MSN)SAPMessage TransmissionUser/Provider of ServiceSAPPacket Transmissionfunction f_transmitPacket(dpid, data) f_packetReceived(dpid, data)Nostrum ApplicationFIGURE 6.1Overview of User/Provider of Service and the Application versus the Nostrum backboneA3. The partial answers to the last question should hopefully be at hand afteranswering question two. Now it is time to see what’s available; hopefully somebodyalready have implemented the service and the corresponding functions in the Nostrumprotocol stack. Remember, everything that is below the current layer is Nostrumand everything (if there exist anything, that is) above the current layer isApplication!If the service you need is available - use it! If not, its time to either change your servicedefinition or start to implement the services that you need at a lower level.VII. REFERENCES[ 1- VSI] Virtual Socket Interface Alliance, Virtual Component Interface Standard Version2, March 2000. http://www.vsi.org[ 2- SONICS] Sonics Inc., http://www.sonicsinc.com[ 3- GUERRIER-A GENERIC...] P. Guerrier and A. Greiner. A generic architecture for onchippacket-switched interconnections. In Proc. of DATE 2000, March 2000.[ 4- BENINI-POWERING...] L. Benini and G. DeMicheli. Powering networks on chip. InProc. of the 14th Int. Symposium on System Synthesis, Oct. 2001.[ 5- BENINI-NOCS: A NEW SOC...] L. Benini and G. DeMicheli. Networks on chip: A newSoC paradigm. IEEE Computer, 35(1): p. 70 ff., Jan. 2002[ 6- KTH-VTT02 - NOC ARC] Shashi Kumar, Axel Jantsch, Juha-Pekka Soininen, MarttiForsell, Mikael Millberg, Johnny Öberg, Kari Tiensyrjä, and Ahmed Hemani. A network on30

December 6, 2002 1:51 pmNostrum - VII.chip architecture and design methodology. In Proceedings of IEEE Computer Society.Annual Symposium on VLSI, April 2002.[ 7- GOOSSENS - COMBINING BEST-EFFORT...] K. Goossens et al., Networks on Silicon:Combining Best-Effort and Guaranteed Services, DATE 2002, March 2002.[ 8- DALLY - ROUTE PACKETS...] W. J. Dally and B. Towles. Route packets, not wires:On-chip interconnection networks. In Proc. of DAC 2001, June 2001.[ 9- SGROI - ... SOC INTERCONNECT WOES...] M. Sgroi et al. Addressing the system-ona-chipinterconnect woes through communication-based design. In Proc. of DAC 2001,June 2001.[ 10- WINGARD - MICRONETWORK...] Drew Wingard, “MicroNetwork-Based Integrationof SoCs”, In Proc. of DAC 2001, June 2001.[ 11- CULLER - PARALLEL COMPUTER...] David E. Culler and Jaswinder Pal Singh,“Parallel Computer Architecture - a Hardware Software Approach”, 1999, MorganKaufmann Publishers, Inc., ISBN 1-55860-343-3[ 12- LEISERSON - FAT TREES] C. E. Leiserson. Fat Trees: Univ. Networks for HardwareEfficient Supercomputing. IEEE Computer, p 892 ff. vol. c-34, No 10, Oct. 1985.[ 13- MPI] Message Passing Interface Forum. MPI: A Message Passing Interface Standard.Version 1.1, http://www.mpi-forum.org, 1995[ 14- FURBER - ARM] Steve Furber. ARM: system-on-chip architecture. Second edition2000, Addison-Wesley Publishing Company, ISBN 0-201-67519-6[ 15- IRVINE - DATA COMMUNICATIONS...] J. Irvine and D. Harle, “Data Communicationsand Networks - an Engineering Approach”, 2002, John Wiley & Sons, ISBN 047180872 5[ 16- FEIGE - HOT-POTATO] Feige, U.; Raghavan, P. Exact analysis of hot-potato routing.Foundations of Computer Science. In Proc. of the 33rd Annual Symposium on, p. 553 -562, 1992[ 17- DALLY - DEADLOCK-FREE MESSAGE] W. Dally, C. Seitz, Deadlock-free MessageRouting in Multiprocessor Interconnection Networks, IEEE Trans. on Computers, vol. C-36, no. 5, p. 547-553, May 1987.[ 18- HEGDE - ENERGY EFFICIENCY] R. Hegde and N. Shanbhag, Toward AchievingEnergy Efficiency in Presence of Deep Sub-micron Noise, IEEE Trans. VLSI Systems, p.379-391, Aug. 2000[ 19- NILSSON - PCA] lErland Nilsson et al. Load distribution with Proximity CongestionAwareness in a Network on Load distribution with Proximity Congestion Awareness in aNetwork on Chip, Submitted to DATE 2003[ 20- THID - NOCSIM] Rikard Thid et al. A Simulator for the Nostrum Network on ChipArchitecture, Submitted to DATE 2003, Authors Suppressed[ 21- STALLINGS1994] William Stallings, “Data and Computer Communications” -31

December 6, 2002 1:51 pmNostrum - VII.Fourth Edition, 1994, Prentice-Hall International, ISBN 0-13-326828-4[ 22- ZIMMER2002] Heiko Zimmer’s erroneous theses ;)[ 23- OCP] Open Core Protocol Specification, OCP-IP Association. http://www.ocpip.org/home[ 24- ]32