Converged network for industrial IoT

One of the major challenges to the implementation of the industrial internet of things (IIoT) is the convergence of information technology (IT) and operational technology (OT) networks. Currently, these networks exist in separate domains. Limited communication in each direction is possible via dedicated gateways. Converging these networks is a key enabler for cyber physical systems in which nodes interact with each other using the IIoT.

The current architecture for controlling factory automation is hierarchical. Enterprise resource planning (ERP) applications at the highest level provide an integrated management and automation of business core processes, progressing down to manufacturing execution systems (MES) that control the manufacturing process. Programmable logic controller (PLC) systems execute the automation tasks using connected industrial devices like electric drives, sensors or I/Os which reside at the lowest level of the hierarchy. This is often called the ‘automation pyramid’ (Figure 1), illustrating the broad amount of devices at the bottom and high performance computers at the top. Layers of the pyramid illustrate the hierarchies.

Figure 1: Automation Pyramid with typical assignment of networks to hierarchies

Converged networks

Different layers in the pyramid have different network requirements. While higher layers need high bandwidth and flexible network topologies, lower layers need deterministic behaviour and the capability to transport samples in constant intervals with low packet delay variation. This leads to multiple networks that work side-by-side.

A converged network will address several challenges that currently exist in today's disparate network architecture. There is more transparency, all data from all hierarchies can be made accessible to every element in the factory without translation in between.

There is less network planning, as more flexible topologies enable easier changes. Another benefit is less capex, with the reduction of cabling, and of gateways between networks with different protocols. There is also an opex reduction with less network administration.

The increased bandwidth avoids being limited to one network speed.

Finally, it is optimised for M2M and is ready for networking between machines with a common data model, like OPC UA, across the factory.

This convergence is provided by time sensitive networking (TSN). TSN is a set of IEEE 802 sub-standards which, when implemented, enable deterministic communication over Ethernet networks, while keeping the benefits of communication in best effort manner, on that same network.

Time sensitive networking

TSN introduces different traffic classes that share the same link. TSN network configuration reserves resources for streams with deterministic time characteristics. TSN, therefore, enables one common network to be implemented which supports multiple communication standards.

This introduces several improvements over standard Ethernet. Standard Ethernet communication is not time-aware. It distributes the data over the entire bandwidth of the link with packets queued in order for transmission. TSN implements time awareness with scheduled traffic at configured offsets in cyclic intervals. That follows a schedule that is distributed by a network configuration controller. Further TSN features introduce filtering and policing for TSN streams, seamless redundancy and supports cyclic data transmission while also providing pre-emption for higher priority packets.

Figure 2: IEEE TSN standards

TSN is defined by a set of IEEE 802.1 standards which specify the implementation (Figure 2).  As of September 2017, four of these standards are adopted, while the remainder is still within the Task and Working Group stages. 

These standards are implemented over Ethernet (IEEE 802.3 physical layer) and support star, chain, ring and mixed topologies and are not limited to a specific data rate. Industrial applications will use 100Mbit and 1Gbit data rates mainly. Therefore, TSN provides the convergence between the IT and OT networks. This convergence reduces the cost of implementing the network, significantly lowering the cost of ownership and operation.

TSN implementation

Correctly implementing TSN requires a solution which can provide a low latency and deterministic response at TSN end points and TSN bridges. Many applications solve this challenge by combining a processor and an FPGA connected together over a high-speed link such as PCIe. This two-chip solution not only increases occupied board space, power consumption, development time and cost but also prevents a holistic integrated solution from being developed. As the design is segmented between two devices, this also increases the complexity of verification.

Increasingly, IIoT solutions providers are using All Programmable Zynq-7000 and Zynq UltraScale+ MPSoC devices to implement their solutions. These devices provide a combination of processing system (PS) and programmable logic (PL) enabling the implementation of acquisition, control and processing applications. They offer the ability to interface and control a range of sensors, actuators, motors and other application-specific interfaces.

They are also able to implement complex processing at the edge, for example machine learning, sensor fusion, image processing and real-time analytics.

Other benefits are scalability of the number of network interfaces and security, with the device and system secure in terms of information assurance, anti-tamper and trust.

The support of any-to-any interfacing and the ability to couple PS and PL make the Zynq-7000 and Zynq UltraScale+ MPSoC devices suitable to implement TSN next to a user application.

Timing issues

Xilinx's 1G/100M TSN sub-system LogiCORE IP consists of FPGA logic for MAC, TSN bridge and TSN endpoint. The TSN design with dedicated logic resources guarantees that the timing behaviour is strictly deterministic. Software which runs in the SoC's PS is for network synchronisation, initialisation, and for the interfacing with network configuration controllers for stream reservation. The software is designed for running on Petalinux and will be published for Yocto builds.

Figure 3: TSN IP in Xilinx Zynq-7000 or Zynq UltraScale+ MPSoC

The LogiCORE IP also comes with an optional integrated time-aware L2 switch that creates the chain or tree topology that is required in many industrial applications without allocating another port at an external TSN switch. Seamless redundancy (P802.1CB) also requires the additional port. The complete IP is illustrated in Figure 3. The user can freely configure before synthesis whether or not the switch shall be integrated.

Once instantiated, the TSN IP core provides individual AXI streaming ports for each traffic class. Scheduled traffic, reserved traffic and best effort traffic are supported. The AXI streaming ports connect to infrastructure that is introduced by Xilinx's Vivadodesign suite. AXI Lite is used for the configuration of the TSN blocks.

Xilinx provides a ready-to-use implementation for evaluation purposes that includes direct memory access (DMA) separately for each of the traffic classes. This evaluation system can be used as is to test the behaviour between Xilinx components as well as in combination with third party equipment or with protocol analysers. Figure 4 shows a block diagram of the evaluation system.

As programmable logic is flexible, it also provides the ability to update the IP core as the TSN standards and market segment specific conformance tests progress, while devices with fixed hardware implementations (like custom ASICS and application specific standard products or ASSPs) lack the option to introduce functional changes along the TSN evolution.

To demonstrate the TSN IP core, Xilinx developed a demonstration application for both the ZCU102 and ZC702 development boards, featuring devices from the Zynq UltraScale+ MPSoC and Zynq-7000 families, respectively. When two of these boards are connected (Figure 5) it enables network traffic to be transmitted and received, enabling validation of TSN networking capabilities.

Figure 5: TSN Evaluation System (i.e. for Zynq UltraScale+ MPSoC)

TSN support network

To further support TSN deployment and applications, Xilinx is a member of the Time Sensitive Networking testbed at the Industrial Internet Consortium (IIC). Participation within this testbed provides the ability to perform vendor interoperability testing, along with testing of high performance and latency critical applications. This testing can occur both formally on one of the two permanent testbeds located in either the USA or Europe.

To enable deployment of the cyber physical systems for Industry 4.0 and IIoT, there needs to be convergence between IT and OT networks. TSN provides the ability to converge these networks, offering significant advantages in network connectivity, scalability and cost of deployment and ownership. Implementing TSN within an All Programmable Zynq-7000 or Zynq UltraScale+ MPSoC device provides the user with a single chip solution, which can also provide the processing capability in the PS and PL to perform the IIoT application at the edge.