Create Scalable Solutions to Improve Deterministic Behavior of Industrial Systems with TSN Offerings from Intel
IoT and Industry 4.0
Internet of Things (IoT) technologies are driving changes in Industry 4.0, known as the fourth industrial revolution, by reshaping systems architecture across all domains in manufacturing. This change has allowed manufacturers to optimize their processes, create new revenue streams, and scale their business models. However, changes in Industry 4.0 presents new challenges that require various manufacturers (food and beverage industry, oil and gas, utility providers, etc.) and industrial equipment vendors (OEMs and ODMs) to improve existing real-time compute capabilities and the vectors of time synchronization (time sync) and timeliness across devices. For example, next generation automation controllers must process various workloads such as video stream and control traffic in parallel, provide for deep learning capabilities all in one machine, and simultaneously communicate with the other controllers in the factory’s network in a timely manner.
TSN and Real-time Compute from Intel
Developers who face both time sync (coordinated events) and timeliness (timely events) challenges with industrial applications such as robotic arms and motion controllers, can use TSN together with real-time features on Intel® architecture to synchronize networks of devices for improving the deterministic behavior of industrial systems. As compared with existing real-time Ethernet protocols, combining TSN with real-time features on Intel® architecture can enable customers to precisely control time-sensitive cycles.
TSN, a collection of IEEE* standards, defines the protocols for how time-sensitive data is transmitted over Ethernet networks. Real-time features on Intel® architecture, including Cache Allocation Technology (CAT), PCIe* PTM (Precision Time Management), and end-to-end virtual channels, can help to optimize the processing of data packets. TSN defines rules for time sync and timeliness to optimally transmit data between systems, whereas the real-time features on Intel® architecture provide these vectors within a system.
Both these vectors can supplement one another to provide the level of determinism required by industrial systems to enhance time sync (for example, audio and video frames being aligned or robotic arms moving in unison) as well as timeliness (deterministic, predictable behavior).
Industry Standards for Real-time Applications Over Ethernet
Historically, industrial communication systems have leveraged real-time protocols such as CAN*, Profibus* and Modbus*. As industrial applications evolved to require high speed, the last two decades have seen an increase in push towards Ethernet based real-time communication.
However, IEEE 802.3, the standard specification for Ethernet, is non-deterministic and therefore not appropriate for real-time applications which require a certain level of determinism (a shared concept of time among devices or systems in a network). The need to incorporate a real-time element into Ethernet has led to the development of many different industrial Ethernet standards such as PROFINET*, EtherCAT*, Sercos*, EtherNet/IP*, CCLink-IE*, Modbus TCP* and Ethernet PowerLink*.
These standards share similar requirements and market segments but their implementations and ecosystems differ. Most of these standards have a parent organization that’s guided by a main market player who drives development of each standard. When a factory operator buys a PLC (based on the PROFINET standard), they are often required to have other PLCs on the PROFINET bus, enabling factories to be connected but also requiring the factory operator to buy parts from vendors that support PROFINET. As a result, manufacturers often have proprietary solutions and stakeholders of industrial value chains can find it difficult to agree on a particular technology. Also, end users and device manufacturers are faced with multiple solutions that they must evaluate. To address these challenges, the IEEE 802.1 Working Group developed TSN to enable deterministic communication over Ethernet.
Time-Sensitive Networking (TSN)
To standardize deterministic communication between industrial networks, TSN defines a set of standards for how time-sensitive data is transmitted over Ethernet networks. Traditionally, Information Technology (IT) controls both computer and data-related network traffic in manufacturing facilities, whereas Operations Technology (OT) governs the network traffic for the operation of industrial control systems. OT and IT are often independent networks with different objectives and requirements. OT establishes and maintains control processes with physical impact, such as manufacturing floors and production environments but IT creates, transmits, stores, and secures data.
TSN enables convergence of these OT and IT infrastructures to be shared across networks, enabling time-sensitive traffic (guaranteed delivery) to co-exist with best-effort traffic (non-guaranteed delivery). Many current solutions meeting these requirements are based on a control hierarchy in which multiple, rigid bus layers are created and optimized to meet the requirements for specific tasks. Each layer has varying levels of latency, bandwidth, and Quality of Service (QoS), making interoperability challenging and flexible data connection changes virtually impossible. With greater interdependence among industrial systems, TSN can play an important role in enabling precise collaboration among automation components and provide interoperable and scalable solutions factories. As an example, consider the aeronautics industry where many vendors often work together to deliver a final solution to end customers. A use case could be an aircraft manufacturer outsourcing different features of its cockpit design solution to various avionics manufacturers.
Migrating to Ethernet-based communication protocols can increase bandwidth, reduce wiring complexity and reduce costs. A common networking layer leads to greater openness for IoT innovations. The capabilities of TSN are incorporated at the data link layer (Layer 2) of the Open System Interconnection (OSI) model as shown in Figure 1. For encapsulating TSN functionality, data link layer enhances the mechanism to deal with transmission errors, regulate the flow of data, and provide a well-defined interface to the network layer. TSN can be used to transport different high-level industrial protocols such as OPC Unified Architecture* (OPC UA)— a machine to machine communication protocol for industrial automation developed by the OPC Foundation*. Additionally, Figure 1 shows an OSI model with different industrial automation protocols that use the same infrastructure.
TSN addresses various needs such as reliability, bounded low latency, time synchronization and resource management. These capabilities are realized through TSN sub-standards (for example, IEEE 802.1AS, IEEE 802.1Qbv, etc.) and customers can determine which standards to implement based on their needs.
Components of TSN and Key IEEE* Sub-standards
Figure 2 shows the key components of TSN: time synchronization, traffic scheduling, and system configuration. The sections below describe these components and highlight the main TSN sub-standards that feed into them.
Industrial systems that communicate with each other in real time need a shared understanding of time to agree on corrective actions, recognize each other’s state, and cooperate together.
For example, in a high-speed conveyor application with multiple systems, packaged components, such as canned food items, travel along a conveyor belt at constant speed. The systems in this example rely on time synchronization to properly handle a defective item. The first system detects the presence of each component, analyzes it for defects, and then updates the second system about the state of that component. Based on those inputs, the second system can then take timely action (make a quick decision to pass or reject a component). As industrial systems become more connected, the need for a shared understanding of time becomes increasingly important.
In 2002, the IEEE 1588-2002 standard was created to define Precision Time Protocol (PTP) for synchronizing clocks throughout a network. PTP devices exchange Ethernet messages to synchronize network nodes to a common time reference by defining clock master selection, negotiation algorithms, clock rate matching, and adjustment mechanisms. The IEEE 802.1ASrev project is working to create a profile of the IEEE 1588 PTP synchronization protocol for TSN. This profile will enable clock synchronization compatibility between different TSN devices. 802.1ASrev also addresses support for fault tolerance and multiple active synchronization masters.
Figure 2. Components of Time-Sensitive Networking
Example of Linux Time Synchronization Software
Linux* peer-to-peer (p2p) is a software implementation of PTP, based on the IEEE 1588 standard. Linux p2p makes use of the Linux kernel, which allows customers to take advantage of the flexibility that is inherent to the Linux Application Programming Interfaces (APIs). Linux p2p is free and available for download at Linux PTP Project.
At the core of 802.1AS synchronization is timestamping. During a p2p message exchange from the 802.1AS-capable MAC, the p2p Ethertype triggers the sampling of local real-time counter values, resulting in the time synchronization of all p2p nodes. Figure 3 shows a typical system connected through the PTP protocol. PCIe* devices can further use the PCIe PTM Engineering Change Notice (ECN) to enhance time synchronization of PCIe devices.
Traffic scheduling allows for different traffic classes to coexist with competing priorities on the same network. IEEE 802.1Qbv and IEEE 802.1Qbu work together to help manage this co-existence. To illustrate this concept, imagine a hydro-electric power plant with several turbines converting mechanical energy into electrical energy. The sensors connected to these turbines monitor speed and temperature and transfer these data points to a central system for observing the health of the turbines. Within the central system, a lot of data is produced in the network not only from IT traffic (e.g., emails, applications) but also from the sensors. In this example, IEEE 802.1 Qbv and IEEE 802.1Qbu can prioritize different traffic classes and enables time-sensitive data from sensors to be routed to the central system through the network to prevent errors that cause accidents, derive actionable insight such as proper coordination among different motors etc.
Devices or systems with IEEE 802.1Qbv can prioritize TSN Ethernet frames on a schedule, while non-TSN Ethernet frames (IT traffic) to be transmitted on a best-effort basis around the TSN frames. The 802.1Qbv standard defines up to eight queues per port for forwarding traffic where each frame is assigned to a queue based on a QoS priority. To control the flow of queued traffic from a TSN enabled switch, this standard defines a time-aware shaper (TAS) mechanism that blocks all ports except one based on a time schedule, preventing delays during scheduled transmission. Put simply, a gate in front of each queue opens at a specific point in time for time-sensitive traffic over standard (non-TSN) Ethernet packets.
In conjunction with IEEE 802.1 Qbv, IEEE 802.1 Qbu stops the transmission of long, non-critical frames to prioritize time-sensitive traffic, addressing the problem of transmission hogging. A major challenge for the timely transfer of critical messages is the presence of legacy traffic sharing the same network, where an individual frame can be 1200 bytes (1.2kB) long. Once a packet travels down a wire, it will block the wire from other packets until the end of the packet is reached. For example, a network with a bandwidth of 100 Mbps and a typical packet size of 1.2 kB can block the network for about 120 ms (1.2 kB/100 Mbps).
To counter this issue, IEEE defined two standards, IEEE 802.1 Qbu, to preempt frames, and IEEE 802.3br, to intersperse express traffic. These standards, which build upon the TAS feature in 802.1Qbv, allow devices to interrupt the transmission of non-TSN Ethernet frames (often legacy traffic) to prioritize high-priority frames, while allowing the remainder of the interrupted frame to be sent later.
The hydro-electric power plant example described in the beginning of this section can risk packets being dropped due to cut cables or malfunctioning devices. By inserting duplicate frames at the sender and then discarding those duplicates at the receiver, IEEE 802.1CB can help a system to recover from dropped Ethernet frames or single-point failures such as cut cables or broken switches in a TSN network. This standard inserts redundant copies of the same messages in parallel over separate paths through the network. Traditionally, Transmission Control Protocol (TCP) and Spanning Tree Protocol (STP) have provided these capabilities. However, both approaches don’t guarantee determinism.
Figure 3. Time Synchronization between Systems
IEEE 802.1Qat and IEEE 802.1Qcc are key standards that define the system configuration of TSN networks1. Figure 5 shows the system architecture of a TSN network, highlighting the system configuration elements of TSN. In the previous example of a hydro-electric power plant, the overall network components (turbines, bridges, controllers, etc.) can use IEEE 802.1Qat to communicate their QoS requirements, such as traffic class and data rate, to each other. After the QoS related messages are communicated, IEEE 802.1Qcc provides a software model to configure these components to meet those requirements.
IEEE 802.1Qat enhances Stream Reservation Protocol (SRP) which defines a plug-and-play configuration mechanism to configure or modify stream reservations. A stream is unidirectional flow of data from a Talker to one or more Listeners. IEEE 802.1Qat enhances SRP by adding support for more streams, enhanced stream characteristics, and a User Network Interface (UNI) for routing and reservations.
Figure 4. IEEE 802.1Qbv
802.1 Qci protects against faulty or malicious endpoints and switches by isolating faults to specific regions in the network. It works at the incoming port of the switch (forwarding engine) in order to protect the outgoing queues from being flooded with frames. In this process, the data packets are checked to ensure that they correspond to a reserved data
To meet the needs of industrial markets beyond professional audio or video, the IEEE 802.1 TSN task group is defining new configuration models, the first of which are specified in IEEE 802.1Qcc. SRP uses a distributed configuration approach, where network bandwidth reservations are established by propagating requests and responses through the entire network. IEEE 802.1Qcc adds a fully centralized configuration model, where Talkers and Listeners send their streams requirements to a Centralized User Configuration (CUC) entity.
With knowledge of the application’s end station stream QoS requirements, a CUC can then communicate those requirements to a Centralized Network Configuration (CNC) entity. With knowledge of the entire network’s stream requirements, the CNC performs calculations to determine if stream QoS requirements for a given application can be met in the TSN-enabled network and how to meet them.
In summary, the CUC is responsible for configuring “users” of the network (Talkers and Listeners) and the CNC is responsible for configuring the TSN-enabled network.
This amendment specifies synchronized cyclic en-queuing and queue draining procedures, managed objects, and extensions to existing protocols that enable bridges and end stations to synchronize their transmission of frames to achieve zero congestion loss and deterministic latency. This allows deterministic delays through a bridged network to be easily calculated regardless of network topology. This is an improvement of the existing techniques that provides much simpler determination of network delays, reduces delivery jitter, and simplifies provision of deterministic services across a bridged LAN. IEEE 802.1Qch collects packets according to their traffic class and forwards them in one cycle. This cyclic enqueuing and queue draining procedure gives a defined upper boundary for latency and enables time-controlled communication in conformity to other 802.1 standards. Essentially this is a simple way to use TSN if controlled timing is desired but reducing latency isn’t highly important.
IEEE 802.1Q is not a TSN sub-standard but it defines bridge managed objects for enabling the configuration of TSN bridges; it describes these objects using multiple data-modeling languages (data formats), such as Management Information Base (MIB)and the YANG. This standard also assumes the use of a Network Management Protocol such as the Simple Network Management Protocol (SNMP), Network Configuration Protocol (NETCONF), and RESTCONF to remotely configure bridge managed objects. NETCONF was developed and standardized by The Internet Engineering Task Force (IETF) to provide mechanisms for installing, manipulating, and deleting the configuration of network devices. This protocol uses XML-based data encoding for configuration data and protocol messages.
IETF recommends YANG for bridge managed objects and YANG-compatible Network Management Protocols, such as NETCONF.
Table 1. Key IEEE TSN Standards
|IEEE 802.1ASRev, IEE 1588||Timing and Synchronization||Defines a protocol to precisely synchronize clocks for automation systems that communicate using Ethernet.|
|IEEE 802.1Qbv||Enhancements for Scheduled Traffic||Enables TSN Ethernet frames to be transmitted on a schedule (guaranteed), while allowing non-TSN frames to be transmitted on a best-effort basis (no guarantee). Each frame is assigned a queue based on QoS priority.|
|IEEE 802.1Qbu||Frame Preemption||Enables frame pre-emption to interrupt the transmission of long frames in favor of high priority frames.|
|IEEE 802.1CB||Frame replication and Elimination for Reliability||Provides for capabilities to recover from dropped Ethernet frames or broken switches in a TSN network by inserting duplicating frames at the sender and then discarding the duplicate at the destination.|
|IEEE 802.1Qcc||Enhancements and Performance Improvements||Supports more streams and improved description of stream characteristics. Also includes support for Layer 3 streaming as well as UNI (User Network Interface) for routing and reservations.|
|IEEE 802.1Qci||Per Stream Filtering and Policing||Protects against faulty and/or malicious endpoints and switches. Checks data packets to ensure that they match to the reserved data stream at the other end.|
|IEEE 802.1Qch||Cycling Queuing and Forwarding||Removes packet delays due to topology and number of nodes in the network. Enforces a fixed packet processing delay for each switch based on a particular traffic class.|
Figure 5. TSN System Architecture
Industrial TSN System Architecture
A TSN network can have multiple components depending upon the complexity of the network. The four main components that comprise most TSN solutions include: end devices, bridges, a CNC, and a CUC. Figure 5 loosely depicts a typical TSN system architecture.
End devices are source and destinations components—also known as talkers and listeners—that run applications which require deterministic communication.
Bridges are network switches that schedule and transmit Ethernet frames based on TSN standards.
The CNC is a centralized component that configures network resources on behalf of TSN applications (users). It calculates the network schedule and distributes these parameters to the infrastructure components (Ethernet switches). The CNC application is provided by the vendor of the TSN bridges
The CUC discovers and configures application (user) resources in end stations. It exchanges information with the CNC in order to configure TSN features on behalf of its end stations
TSN Use Cases
TSN standards drive multiple IOT use cases in various industrial markets segments that can benefit from time synch and timeliness capabilities enabled through this technology.
TSN creates a system where smart, hyper-connected devices and infrastructure of manufacturing machines, transportation systems, and the electrical grid will embed sensing, processing, control, and analysis capabilities. In the use cases below, we highlight key industrial market segments poised to gain value from TSN implementation.
Industrial automation applications—motion control, machine-to-machine (M2M) communication, robotics, etc.— use TSN to improve time sensitive processes. For example, TSN can be used to control high-speed motion processes, such as voltage or current regulation in the renewable energy industry. Other manufacturing examples include consumer packaged goods and electronic components where standards, such as IEEE 802.1ASRev, can help robots to better synchronize with one another as they move parts across a supply chain floor.
TSN can help improve factory functions such as quality control, predictive maintenance, and production flow monitoring. Manufacturers who implement these standards can enable their industrial automation applications to process raw data in a timely manner, decrease latency, and provide a base for advanced manufacturing (where data is flexible and shared between layers of the control system).
The subsections below highlight use cases where TSN can be applied to solve challenges in industrial automation—motion control, robotics, and machine to machine communication.
Motion control applications have strict delay requirements to ensure that real-time data transmissions can support workload demands. Motion control spans various industrial market segments (discrete industries, process industries, power industry, etc.) and supports dedicated applications such as PLC controllers. Other use cases include controlling the velocity or position of a mechanical device—hydraulic pumps, linear actuators, or electric motors. As the automation industry consolidates its operations, motion controllers need to process more workloads, resulting in a greater need for increased bandwidth and information transparency between different levels in the factory.
For example, next generation PLC machines require response times to be in the low microsecond range (32.5 µs to 125 µs). TSN was developed to accommodate these developments and represents the next step in the evolution of dependable and standardized industrial communication technology. TSN standards such as 802.1Qbv allow the specification of QoS which enables time sensitive traffic to efficiently navigate through networks. According to a MarketsandMarkets research report, the market for motion control is expected to reach $22.84 billion by 20222. The main drivers for this adoption will be metal and machinery manufacturing, as industry leaders look to improve speed and accuracy along with increased production.
An industrial robot is a programmable, mechanical device used in place of a person to perform dangerous or repetitive tasks with a high degree of accuracy. Based on the operating environment, industrial robots can be classified as fixed (robotic arms), mobile (autonomous guide vehicles) and collaborative (pick and place robots). A key challenge in robotics is the absence of a standard communication protocol. Robotic manufacturers must support many customized protocols, which can lead to increased integration times and costs. Since modern robotics integrates artificial intelligence (AI), machine vision, and predictive maintenance into one system, there is a need for sensors and actuators to stream high bandwidth data in real time. A common solution is to use a specific channel for real-time control (such as EtherCAT and PROFINET) and a separate one for higher bandwidth communications (TCP/UDP). For applications that generate high bandwidth traffic (100 megabytes(MB) to 1 gigabyte(GB)), using two separate communication channels becomes inefficient. TSN provides a shared communication channel for high bandwidth traffic and real-time control traffic.
Machine-to-Machine (M2M) Communication
M2M communication—two machines communicating without human interaction—is reinventing manufacturing by enabling data to be shared across different control and analytical applications to derive superior operational efficiencies. TSN enhances M2M communication by connecting previously un-connected proprietary controllers. This is made possible through a network of TSN machines
Figure 6. Production Cell
(TSN-compliant end points) connected through TSN-enabled switches. M2M communication can enable remote management, operation of equipment/devices through cellular point-to-point connections. Figure 6 shows a production cell with a supervisory PLC coordinating communication across four different TSN machines. Through a central configuration mechanism, IEEE 802.1Qcc allows the management of different components and defines a standard UNI for communication among them. Another example of M2M communication can be PLCs communicating with other controllers, conveyor belts, and other control equipment (at the same network layer) to regulate or monitor the production of a product
Power and Energy
Communication networks play an important role in the exchange of information and data for power and energy applications in production plants. Electrical substations are the main components of these networks. A substation is often one of many subsidiary stations within an electric power system. Substations have many functions; they control and monitor the switch yard, record data, and protect power equipment via monitoring. Modern substations communicate via IEC 61850, a standard for communication at electrical substations. Introducing redundancy measures—Parallel Redundancy Protocol and High Availability Seamless Redundancy—can prevent data loss of time-sensitive traffic. However, redundancy measures are not enough to guarantee the on-time delivery of data. Even in IEC 61850 networks, single events and data transfers within a substation can greatly increase the amount of network traffic.
Figure 7. Universal WellPad Controller
Considering the large number of simultaneous communications within a substation, the availability of bandwidth becomes critical. The challenge for utility companies, such as Arizona’s Salt River Project*, is to ensure network availability for critical data streams in case of network congestion. TSN addresses this problem by providing deterministic, reliable and secure communication at the network layer and allows a variety of high layer protocols (OPC-UA*, PROFINET, EtherCAT etc.) at the management layer. To optimize performance and the cost of production, TSN can also provide cloud-based services for system users to access real-time data from power plants, such as turbines.
Oil and Gas
Industrial Ethernet is used as the communication standard for monitoring systems used in oil exploration and production. Ethernet is essential at every step of oil production—upstream, midstream, and downstream. TSN can play an important role in enabling real time applications in the oil and gas industry by providing deterministic communication for process and control networks associated with surface production facilities. Oil and gas facilities require reliable, robust and high-capacity communication networks that can operate over wide geographical areas under critical and harsh environmental conditions.
Figure 7 shows Universal WellPad Controller (UWC), an industry initiative that currently uses control software enabled with TSN for close loop control. Intel is using off-the-shelf hardware, with TSN enabled, together with open-source software from multiple vendors to securely monitor and control onshore production wells and surface production facilities.
TSN Products From Intel
Intel offers a portfolio of TSN products that include discrete and integrated Ethernet controllers as well as FPGAs. To educate customers about TSN, Intel has established testbeds (using the Intel® Ethernet Controller I210 and Intel® FPGAs) at various sites for showcasing the implementation of TSN sub-standards such as IEEE 802.1Qbv. Intel is also working with the ecosystems governed by open alliances such as Avnu Alliance*, Industrial Internet Consortium (IIC)*, and the International Electrotechnical Commission (IEC)* to define these standards for addressing the pain points of the industrial ecosystem.
Intel® Ethernet Controller I210
The Intel® Ethernet Controller I210 is a discrete network adapter for use in real time Ethernet applications3 that can be used as a TSN end point. It supports TSN standards such as IEEE802.1AS and IEEE 802.1Qbv (through TSN reference software) for industrial applications. TSN reference software is a C-based application that demonstrates how to configure TSN by using the ‘tc’ utility (see figure 9).
This network adapter can support a speed of up to 1 Gb/s, pre-fetch Ethernet frames (ahead of their specified transmission time) from system memory (DRAM), and store this data in the transmission buffer. The network adapter also supports Launch Time, a concept for handling Ethernet packets which uses the “earliest time first” traffic-shaping algorithm to deterministically transmit frames. If developers combine Launch Time with the “time aware priority” scheduling algorithm and VLAN tag packets, they can reduce irregularities in transmitting Ethernet frames. Manufacturers interested in testing TSN solutions can add this hardware to a base board and start evaluating TSN for their factories.
Open Source Initiative
Intel contributes to the Linux project by developing kernel interfaces that can be used by applications that require timely transmission (Tx) of scheduled packets. These kernel interfaces are comprised of three new components—the ‘SO_TXTIME’ API, earliest txtime first (ETF) scheduler (qdisc), and the Time Aware Priority Shaper (TAPRIO) scheduler. In addition, Intel also supports the PREEMPT-RT patch which is basic requirement for real-time applications.
The ‘SO_TXTIME’ API enables sockets to be used for time-based Tx and configures control-message parameters. ETF provides time-based admission control and "earliest deadline first" dequeue mode, while TAPRIO can execute a pre-defined schedule.
PREEMPT-RT- In the case of Linux, the standard kernel does not provide real-time capabilities. However, with the real-time preemption patch (PREEMPT-RT), it is possible to achieve real-time computing capabilities. The PREEMPT-RT patches tries to minimize the amount of kernel code that is non-preemptable4. The main benefit is that it is possible to use standard Linux tools and libraries without requiring specific real-time APIs. Also, because Linux is widely used and supported, this helps to keep the OS updated with new technologies and features, something which is often a problem in smaller projects due to resource limitations.
Figure 8. Linux* Networking Stack
The key features that the PREEMPT_RT Linux patch provides include:
- Spinlocks Spinlocks are a locking mechanism that is used by two or more competing processes to solve the problem of processes waiting for each other to finish while trying to access a given section of code. PREEMPT_ RT enables sleep spinlocks where preemption is allowed.
- rt_mutex A mutual exclusion object (mutex) is a program object that manages access to shared resources, such as file access, for multiple programs. A real-time mutex (rt_mutex) implements priority inheritance to avoid priority inversion, shortening the block of high-priority tasks which protect shared resources. Priority inheritance allows well-designed applications to use userspace locks in critical parts of a high priority thread, without losing determinism.
- High resolution timers Allows precisely timed scheduling and removes the dependency of timers on the periodic scheduler tick.
SO_TXTIME: This API is enabled through the setsockopt() system call and allows applications to request time-based transmission. The packets for timed transmission should be sent using sendmsg(), with a control-message header of type SCM_TXTIME.
ETF: This algorithm provides time-based scheduling per transmission queue. It allows applications to control the instant when a packet should be dequeued from the traffic control layer into the netdevice. ETF qdisc relies on both the ‘SO_TXTIME’ API and ‘SCM_TXTIME’ control-message header in each packet field to configure the behavior of time-dependent sockets and drops any packets with a txtime in the past or if a packet’s transmission time expires. This scheduling strategy exposes a feature known as "Launch Time" (or "Time-Based Scheduling") which makes it possible to shape traffic for TSN applications. ETF sorts packets from multiple sockets by the earliest transmission time and then sends them to the network interface controller (NIC). It also enables packets to be offloaded to the hardware (if the NIC supports it), providing more time accuracy.
TAPRIO: This algorithm schedules traffic classes according to a pre-generated time sequence. The TAPRIO qdisc configures a sequence of gate states, where each gate state allows outgoing traffic (for a subset of traffic classes) to pass through, allowing network administrators to configure schedules for traffic classes.
To accelerate time to market for TSN enabled systems, Intel is also driving FPGA-based TSN endpoints and switches. OEM, ODM, and industrial equipment manufacturers can use Intel FPGAs to be the first to market, increasing their return on investment and making their products suited for industrial IoT. As new TSN standards are approved or when existing standards are altered, developers can quickly reconfigure Intel FPGAs, ensuring that devices support the latest TSN functionality.
Reasons for implementing TSN on Intel FPGAs for use in industrial systems include5:
- FPGA-based designs are reprogrammable: FPGAs can be reprogrammed to adapt to evolving standards, enabling customers to increase efficiency and expand the capabilities of their current solutions.
- Consolidate and accelerate workloads: Growth in network traffic has created challenges in the transfer and scaling of data packets, requiring a new set of compute capabilities in the systems. To accelerate system performance, developers can optimize their systems by having the CPU offload work to an Intel FPGA.
- Flexible I/O: Intel FPGAs allow for TSN implementation along with other Industrial Ethernet protocols on one device.
- Achieve functional safety and security: As TSN connects previously unconnected systems, functional safety and security should be considered. Intel FPGAs, tools and IP are certified to IEC61508 safety standards.
Next Generation Intel Products with TSN
Intel plans to develop products with integrated TSN support to enable networking features for traffic scheduling as well as shaping and prioritizing packet flows.
Details of the plan for developing these products include:
- Enhancing real-time compute in its silicon, by providing capabilities for end-to-end virtual channels, CAT technologies for L2 and LLC caches, PCIe PTM support, and instructions to optimize cache management.
- Demonstrating some of the most popular communication middleware such as OPC-UA (to map over TSN using the newly introduced Pub/Sub extension to the spec) by providing open-source reference stacks.
- Engaging with standard bodies like IEEE/IEC to define network configuration delivery mechanisms.
Figure 9 shows the reference architecture for Linux TSN controller stack. All user-space stack including OPC UA stack, ‘tc’ utility, linuxptp, OpenAvnu stack and IOTG TSN Ref SW interacts with the Linux kernel via the socket interface. TSN Ref SW contains scripts that demonstrates how to configure the TSN configuration by using ‘tc’ utility and send traffic to hardware via the socket interface. Specifically, the LaunchTime capability in I210 and the TSN Ref SW sample app contains code to show how to use the socket interface and set the per-packet transmit time in the SO_TXTIME field of the socket’s control message.
Through its engagement with the open-source community and investments in building open-source infrastructure (schedulers, sockets), Intel expects to enable customers to create flexible and scalable solutions. With the infrastructure for TSN, customers can easily port their applications and reduce the total cost of ownership. To reduce latencies as packet traverses through different layers, Intel plans to standardize data and configuration paths further enhancing timely processing of packets. The kernel and its networking stack forms the hardware abstraction layer for the underlying hardware platform and TSN controllers. This abstraction layer lets customers continue to develop and innovate without the need to use resources to manage the connectivity of low-level device drivers.
Figure 9. Linux TSN Controller Software Architecture (for reference only)
Additionally, because of the abstraction layer at the application level, customers can innovate and add value to their solutions without the need to fine-tune their hardware. Intel is providing this functionality through TSN reference software that helps emulate IEEE 802.1Qbv functionality for I210 controller.
Hitachi TSN Testbed
To highlight the practical aspects of TSN, Intel is partnering with Hitachi* to explore TSN technology in Hitachi’s next generation products. Intel and Hitachi are co-developing a TSN testbed to support Hitachi’s technology requirements that include real-time compute and analytics workloads.
A TSN testbed is a basic prototype of two automation controllers that communicate with each other to demonstrate TSN features and vendor interoperability.
Figure 10 shows a simple setup of the TSN testbed in which two TSN-enabled endpoints based on an Intel® Ethernet Controller I210 are connected through a Cyclone® V FPGA to demonstrate real-time communication. The testbed includes components such as real-time Linux patches for TSN enhancements and reference software that includes technical support from Intel.
Figure 10. Sample Testbed Setup
Adopting TSN is a gradual process as it is often complex and expensive to update a factory’s infrastructure. Intel offers a suite of products (such as network adapters, FPGAs, and next generation products) that can help customers adopt TSN. The decision to implement TSN can depend on whether a project is greenfield or brownfield.
The term “brownfield” refers to projects with an existing infrastructure. In such cases, the Intel Ethernet Controller I210 can be plugged into a customer’s base board design to make end devices compliant with the IEEE 802.1AS standard; this allows customers to cost-effectively evaluate TSN. Additionally, customers can use the traffic control interface module to enable the IEEE 802.1Qbv standard, see the TSN Network Configuration block in Figure 9. For connecting TSN-compliant devices within a network, customers can also use the bridge (switch) functionality of Intel FPGAs. Customers can also evaluate TSN for brownfield projects by working with vendors who have developed a proxy feature that connects TSN networks to non-TSN networks. This proxy feature can be used by customers (for example, PROFINET* users) who plan to retain their original manufacturing infrastructure and applications. However, this may increase the overhead in terms of operational expenses. Brownfield projects vary greatly from greenfield projects in terms of opportunities for technology implementations in the factory. The term “greenfield” refers to projects where customers are developing a new infrastructure.
For such opportunities, Intel’s next generation products not only provide TSN capabilities but include features such as functional safety, real time compute capabilities, in band ECC, etc. Using next generation products enable customers to get more features at lower BOM costs and the product can be used for multiple applications. For example, robotics applications require real-time capability but for certain robot types such as cobots, who interact with humans, functional safety is paramount.
Intel also plans to provide developer tools such as Linux* Yocto with PREEMPT-RT kernel patches along with support for standard kernel TSN API’s. In addition, Intel also plans to provide reference software for TSN middleware stacks – OPC-UA (Client/Server, PubSub), Linuxptp (802.1AS, 1588), demo applications etc. Intel also plans to provide a software tool kit that will enable customers to configure their platforms for real-time workloads without the need for customers to know the details of the underlying platforms. Such features provide extensive capabilities for customers to design their greenfield projects as well as reduce their development time and time to market.
Intel FPGAs further provide flexibility to be used as an end-point or switch in greenfield projects. Moreover, various off-the-shelf FPGA-based products (available through Intel’s partners) provide a variety of options for speed, implementing TSN standards, and network software packages. As these standards evolve, Intel FPGAs can be quickly re-configured to ensure that the latest TSN functionality is supported in the device, allowing customers to fully benefit from TSN features.
Market studies indicate that as TSN drives efficiency in factory operations, it will become the dominant technology for real-time connectivity in the next decade6. TSN can support high-speed transmission rates such as 1 Gb/s or 5 Gb/s, a key advantage over traditional industrial Ethernet networks which are typically defined for 100 Mb/s.
Developers can start using Intel developed kernel interfaces (SO_TXTIME, ETF, TAPRIO) to improve their applications that require real-time communication. Figure 10 highlights these sockets API’s that are part of the abstraction layer that Intel plans to provide for customers. Product Managers at OEM/ ODMs are encouraged to start designing products with TSN along with high level abstraction layers such as OPC-UA to design TSN compliance products that cater to the needs of Industry 4.0 paradigm. Manufacturing companies can more readily adopt TSN in their factories by encouraging machine builders and equipment providers (OEM and ODMs) to supply them with TSN-compliant products. Manufacturers can then use these TSN-compliant products to evaluate their current applications or software stack, enabling them to determine how to upgrade their infrastructure.
Intel has started to establish OT testbeds at various Intel sites to educate customers on TSN and its benefits. As standards are being developed, these testbeds provide an ideal ground for customers to understand TSN technology and apply it to their specific technical requirements. Additionally, they can influence the definition and development of these standards.
Combining TSN standards with the real-time features on Intel® architecture can provide customers with flexible, scalable computing for supporting a wide range of industrial applications. Customers can contact their Intel Account Managers to discuss solutions for implementing TSN in their next generation products.
- To learn more about Intel’s TSN reference software visit GitHub* IOTG TSN Reference Software.
- To design with products from Intel, visit the Resource Design Center.
- Avnu Alliance’s Theory of Operation
- Motion Control Market worth 22.84 Billion USD by 2022
- Intel® Ethernet Controller I210 Datasheet
- Technical details of PREEMPT_RT patch
- Industrial Automation
- Time-Sensitive Networking Market worth 606.0 Million USD by 2024
You may not use or facilitate the use of this document in connection with any infringement or other legal analysis concerning Intel products described herein. You agree to grant Intel a non-exclusive, royalty-free license to any patent claim thereafter drafted which includes subject matter disclosed herein.
No license (express or implied, by estoppel or otherwise) to any intellectual property rights is granted by this document.
All information provided here is subject to change without notice. Contact your Intel representative to obtain the latest Intel product specifications and roadmaps.
The products described may contain design defects or errors known as errata which may cause the product to deviate from published specifications. Current characterized errata are available on request.
Copies of documents which have an order number and are referenced in this document may be obtained by calling 1-800-548-4725 or by visiting: http://www.intel.com/design/literature.html.
Intel technologies' features and benefits depend on system configuration and may require enabled hardware, software or service activation. Performance varies depending on system configuration. No computer system can be absolutely secure. Check with your system manufacturer or retailer or learn more at [most relevant URL to your product].
No computer system can be absolutely secure.