DétacherÉpinglerFermer

TwinCAT Machine Learning:

Scalable, open and in real time

Artificial intelligence seamlessly integrated at control level

Artificial intelligence (AI) is a very successful method of automating processes, as it imitates the cognitive abilities of humans. There are many different ways to imitate these skills, with the most successful approach in recent years being machine learning (ML), which is based on processing large sample data sets. The basic idea of machine learning is to learn solutions for specific tasks using representative example data.

As a specialist in PC-based control, Beckhoff integrates this technology into the TwinCAT 3 control software, recognizing that a successful method for automating processes has to be understood as an integral part of a machine’s control behavior. This is why Beckhoff provides a modular range of hardware and software to integrate AI models directly into the PLC.

See below to discover ideal applications for AI in industrial automation and explore the products and workflows offered by Beckhoff to enable successful adoption of AI applications on a broad scale – and not just by AI experts.

AI applications in industrial automation

The use of AI methods is particularly promising when the variance of the incoming data is difficult to predict. If, for example, the quality of wooden boards or agricultural/biological products is to be evaluated visually, the large variance of these natural products must be included in the evaluation. AI methods are highly promising and can therefore often be found in applications. The idea is that the solution to a task is learned from examples rather than applying a static sequence of algorithms to a previously devised solution.

Application examples from the field of image processing are always easy to grasp. This approach can also be applied to time series and other data. If a product is manufactured in a machine, AI can be used to process any metrologically recorded data points that describe the product in a representative manner (a kind of fingerprint of the manufactured product). This data can then be used to predict estimates or classifications.

The potential applications are incredibly diverse and cannot be described in full here. The McKinsey market research institute, among other entities, has singled out the following as the top fields of application for AI in the industry:

  • machine-integrated quality checking
  • reducing waste quantities
  • collaborative and context-aware robotics
  • machine optimization
  • predictive maintenance

From data to application

Simplified representation of the workflow from data collection and training to the integration of a trained AI model into a production environment
Simplified representation of the workflow from data collection and training to the integration of a trained AI model into a production environment

Beckhoff offers an open system workflow without a lock-in effect for the entire cycle, from data collection and model training through to the execution of the learned model in a productive environment. With its open system, Beckhoff makes it possible to meet specific requirements using toolboxes and functions from the TwinCAT modular system. This also applies to existing system infrastructures that are not based on Beckhoff products.

See below to discover more about the possibilities that Beckhoff can offer you along the workflow outlined on the right.

Each application and also each IT infrastructure places different demands on the method of collecting machine data: SQL or noSQL, file-based, local or remote, limited port releases, cloud-based data lake, and many more. A variety of established TwinCAT products are available for all of these scenarios, including the TF6420TwinCAT 3 Database Server, the TF3300TwinCAT 3 Scope Server, the TF3500TwinCAT 3 Analytics Logger, and the TF6720TwinCAT 3 IoT Data Agent, to name just a few. For image data, TwinCAT Vision even provides an entire product family for image acquisition, image (pre)processing, and image storage.

The concept of machine learning is based on learning relationships using representative examples. This is why it is essential to review and tidy up the recorded data to create a clean data set. As a general rule (with the exception of applications such as anomaly detection), the sample data must be annotated (labeled) before model training can be carried out successfully.

For automation and process experts

The TE3850TwinCAT 3 Machine Learning Creator allows even non-AI experts to develop efficient and high-quality AI applications. The software automatically creates AI models based on data sets. These AI models can be optimized in terms of their accuracy and latency to ensure they run efficiently on Beckhoff Industrial PCs with TwinCAT products. The generated models can also still be used as standardized ONNX models beyond the Beckhoff product range. For use with TwinCAT products, a PLCopen XML with IEC 61131-3 code is created in addition to the ONNX file, which describes the complete AI pipeline and can be imported seamlessly into TwinCAT.

For AI experts

The TE3850TwinCAT 3 Machine Learning Creator is also suitable for AI experts to accelerate and standardize the AI development process. The open ONNX export allows experts to read the created AI model into any framework for analysis or refinement.

AI experts can also train AI models completely without the TwinCAT 3 Machine Learning Creator. The training can take place in frameworks such as PyTorch, TensorFlow, and SciKit-Learn®. Ultimately, only the ONNX export functionality of the framework is required. This means that there are no limits to AI model development.

If a trained AI model is available in the form of an ONNX file, Beckhoff offers the option of loading and executing these models on the control computer in the PLC. This makes the AI application part of the control application. The advantages of this integration are as follows:

  • no additional (AI-specific) hardware
  • no additional interfaces or subsystems
  • maintenance of a single central system (updates, security, log-ins, etc.)
  • synchronous and transparent availability of all machine data in the control computer
  • cost-efficient and scalable solution

A wide variety of different AI models are now available on the market. Some can be executed very efficiently even with small computing resources, while others require larger computing capacities in order to obtain a result within an adequate time. Beckhoff offers a flexible and scalable portfolio to suit all requirements.

Hardware

For a large number of applications, the CPU-based execution of AI models is entirely sufficient. For these cases, a suitable hardware platform can be selected from the scalable portfolio of Beckhoff Industrial PC and Embedded PC components that meets the requirements for execution speed (latency).

In some cases, it is necessary to use hardware accelerators due to the achievable execution speed. With the C6043 ultra-compact Industrial PC, Beckhoff offers a scalable solution including an NVIDIA GPU. Both the Intel® CoreTM-i CPU and the NVIDIA GPU can be selected for this device, resulting in cost-efficient solutions in this segment, too.

Software

Beckhoff offers several options adapted to the specific requirements for loading and executing trained AI models via the TwinCAT PLC. The options can essentially be divided into two groups:

  • execution of the model in the TwinCAT runtime process
  • execution of the model as a server process

The following table shows the respective properties:

Inference in the TwinCAT runtime Inference server
Hard real-time execution Near real-time execution (separate process)
Execution on standard x64 CPUs AI-optimized hardware options – e.g., NVIDIA GPUs
Selected highly optimized AI models Support for almost all AI models
Standard PLC function block for seamless integration Standard PLC function block for seamless integration
Interoperability through ONNX support Interoperability through ONNX support
Bundled into one TwinCAT license Local and (multi-)remote client connections
Priority scheduling

The ‘Execution of the model in the TwinCAT runtime process’ group contains the following TwinCAT products, which are bundled together in one license:

  • TF3800TwinCAT 3 Machine Learning Inference Engine
  • TF3810TwinCAT 3 Neural Network Inference Engine
  • TF7810TwinCAT 3 Vision Neural Network

The TwinCAT products in the ‘Execution of the model as a server process’ group are as follows:

  • TF3820TwinCAT 3 Machine Learning Server
  • TF3830TwinCAT 3 Machine Learning Server Client

ML models are capable of improving through training with larger sets of data. Likewise, general conditions can change gradually or spontaneously when the machine is being operated. This means you can update your trained ML models during the machine’s normal runtime – i.e., without stopping the machine, without recompilation, and also completely remotely through the standard IT infrastructure. Regardless of which TwinCAT product you choose to execute the AI models, a new AI model can be transferred to the control computer and reloaded via PLC functions.

What’s more, you can also operate your training environment remotely or locally on the industrial PC within the operating system context, allowing you to retrain, exchange, and load models with TwinCAT in close proximity to the process.

Application examples for artificial intelligence in control systems

Virtual machines provide flexible execution environments for containerized edge applications.
Virtual machines provide flexible execution environments for containerized edge applications.

In the discrete production of metallic workpieces, the geometric shape is often a key quality feature. In addition to metric measurement methods to assess a workpiece quantitatively, qualitative statements (such as the classic categorization into OK and non-OK) are often sufficient.

A representative data set of approx. 200 images was recorded and saved using the TwinCAT Vision library. The data was annotated as OK and non-OK, whereby various different error patterns were summarized together as non-OK. With the TE3850 TwinCAT 3 Machine Learning Creator, an image classification model could be trained based on this data set, which can predict whether a workpiece is OK or not OK in more than 95% of the cases considered – without any AI expert knowledge.

Retrofitting existing machines
Retrofitting existing machines

Automation in the food industry contributes to the efficient and resource-saving supply of a wide variety of foods. One challenge is the automated sorting of foodstuffs, as these have a high natural variance compared to artificially produced products. In the context of eggs, for example, these should automatically be sorted into the categories OK, dirty, and broken. For this purpose, 200 images were taken with these three classes and annotated. With the TE3850 TwinCAT 3 Machine Learning Creator, it was possible to create an AI model that can correctly classify an egg in more than 90% of the cases considered. Using the explainability methods for AI models included in the product, it was easy to find out that misclassifications occurred especially in marginal areas from OK to dirty. This made it immediately clear what measures needed to be taken to improve the model: Either provide more sample data in the boundary area between OK and dirty, or define the boundary more cleanly by revising the existing annotations.

All data from the machine park is consolidated and pre-processed.
All data from the machine park is consolidated and pre-processed.

Wind turbines are a key component in the transition to renewable energies. They supply clean, electrical energy, which they obtain from the kinetic energy of the wind. Knowing both the wind direction and the wind speed is crucial for the efficiency of the system. The rotor attached to the nacelle is aligned with the wind direction according to the wind direction. As for the pitch of the rotor blades, this is adjusted according to the wind speed so that the turbine is operated as constantly as possible at its rated output.

Wind direction tracking and pitch adjustment are relatively slow, which means that the future wind direction and speed have to be estimated in order to move the turbine predictively to the optimum orientation.

Based on wind data collected from real wind turbines, an AI model was created that is able to estimate wind direction and wind speed values 10 to 20 seconds in the future with an acceptable margin of error. This is based entirely on past wind values. The created model can be easily integrated into TwinCAT with the TF3810 TwinCAT 3 Neural Network Inference Engine.

Representation of the core width as one of the quality criteria for the core wrapping
Representation of the core width as one of the quality criteria for the core wrapping

A mechanical bolt anchor essentially comprises the bolt, a washer, a hexagon nut, and a metal sleeve. The frictional forces between the sleeve and the wall of the drill hole ensure sufficient adhesion during use. To apply the normal forces required for the holding force to the drill hole, the sleeve is expanded with the drill hole via the conical head of the metal bolt.

The project, led by R&D engineer Robin Vetsch as part of the OST’s Bachelor of Science in Systems Engineering, focused on the enclosure process, whereby the preformed, punched sleeve encloses the conical neck of the bolt anchor. Only the existing machine data was to be used for quality control – i.e., no installation of additional sensors.

Until this point, the quality of the sleeve around the bolt was mostly checked manually using a test gauge. Now, however, it has been demonstrated that each enclosure can be classified into three different categories (under-enclosed, acceptable, over-enclosed) within the quality specifications. The geometric key data of the enclosed sleeve (sleeve width, height, and opening) should also be predicted with a regression. The 100% inspection of the enclosure process is designed to detect trends or deviations at an early stage.

Lu Peng (left), project manager from Tianjin FengYu, and Wan Pinlei (center), system application engineer from Beckhoff China – with the CX51xx Embedded PC as the hardware core of the TwinCAT Machine Learning solution – and Xie Shaowei, technical support engineer from Beckhoff China (right)
Lu Peng (left), project manager from Tianjin FengYu, and Wan Pinlei (center), system application engineer from Beckhoff China – with the CX51xx Embedded PC as the hardware core of the TwinCAT Machine Learning solution – and Xie Shaowei, technical support engineer from Beckhoff China (right)

Instant noodles can be found in just about every food store in China. In a bid to reduce the number of products with packaging errors and the associated customer complaints, a large Chinese producer of instant noodles decided to turn to Beckhoff control technology including TwinCAT Machine Learning. This made it possible to perform intelligent and reliable real-time inspection of the packaging quality.

In the first step, the sensor data was acquired via EL1xxx and EL3xxx EtherCAT digital and analog input terminals along with TE1300 TwinCAT 3 Scope View Professional. The AI model was then trained via the Open Source Framework Scikit Learn, and the model description file was generated from it. The necessary pre-processing of the sensor data was implemented in the controller using TF3600 TwinCAT 3 Condition Monitoring. In the next step, the corresponding model description file was deployed to a CX51xx Embedded PC, which runs the AI model in real-time with the help of the TF3800 TwinCAT 3 Machine Learning Inference Engine and outputs the inference results for the detection of faulty products via an EL2xxx EtherCAT digital output terminal. The system openness in particular – a great advantage of the Beckhoff control technology – bore real fruit here, as this could be integrated into the existing third-party main controller of the production line with no great effort.

Products

TE3850 | TwinCAT 3 Machine Learning Creator New

TE3850 | TwinCAT 3 Machine Learning Creator

The TwinCAT 3 Machine Learning Creator automatically creates AI models based on data sets. These AI models can be optimized in terms of their accuracy and latency to ensure they run efficiently on Beckhoff Industrial PCs with TwinCAT products. The generated models can also still be used as standardized ONNX models beyond the Beckhoff product range. For use with TwinCAT products, a PLCopen XML with IEC 61131-3 code is created in addition to the ONNX file, which describes the complete AI pipeline and can be imported seamlessly into TwinCAT.

TF3800 | TwinCAT 3 Machine Learning Inference Engine

TF3800 | TwinCAT 3 Machine Learning Inference Engine

The TF3800 TwinCAT 3 Function is a high-performance execution module (inference engine) for trained, conventional machine learning algorithms.

TF3810 | TwinCAT 3 Neural Network Inference Engine

TF3810 | TwinCAT 3 Neural Network Inference Engine

The TF3810 TwinCAT 3 Function is a high-performance execution module (inference engine) for trained neural networks.

TF3820 | TwinCAT 3 Machine Learning Server

TF3820 | TwinCAT 3 Machine Learning Server

The TF3820 TwinCAT 3 Machine Learning Server is a high-performance service for executing trained AI models with the option of using hardware accelerators.

TF3830 | TwinCAT 3 Machine Learning Server Client New

TF3830 | TwinCAT 3 Machine Learning Server Client

The TwinCAT 3 Machine Learning Server includes a connection to a local client as standard (local TwinCAT runtime). If (possibly further) TwinCAT runtimes need remote access to a TwinCAT 3 Machine Learning Server, these runtimes must each be equipped with a license for the TF3830 TwinCAT 3 Machine Learning Client.

TF7800 | TwinCAT 3 Vision Machine Learning

TF7800 | TwinCAT 3 Vision Machine Learning

TwinCAT 3 Vision Machine Learning provides an integrated machine learning (ML) solution for vision-specific use cases. Both the training and the implementation of the machine learning models take place in real time, and they even help machines to learn sophisticated data analyses automatically. This can be used to replace complex, manually created program constructs.

TF7810 | TwinCAT 3 Vision Neural Network New

TF7810 | TwinCAT 3 Vision Neural Network

TwinCAT 3 Vision Neural Network provides an integrated machine learning (ML) solution for vision-specific use cases. The implementation of the machine learning models takes place in real time. With the help of these models, complex data analyses can be learned automatically. This means that complex, manually created program constructs can be replaced.

C6043 | Ultra-compact Industrial PC with NVIDIA® GPU New

C6043 | Ultra-compact Industrial PC with NVIDIA® GPU

The C6043 extends the series of ultra-compact industrial PCs to include a high-performance device with a built-in slot for powerful graphics cards. With the latest Intel® Core™ processors and highly parallelizing NVIDIA® graphics processors, the PC becomes the perfect central control unit for ultra-sophisticated applications. These include applications with high demands on 3D graphics or deeply integrated Vision and AI program blocks with minimal cycle times. The Beckhoff TwinCAT 3 control software is capable of mapping this as a fully integrated solution – without any additional software or interfaces. With the additional freely assignable PCIe compact module slot, the C6043 can be flexibly expanded with supplementary functions.

C6675 | Control cabinet Industrial PC

C6675 | Control cabinet Industrial PC

The C6675 industrial PC is equipped with top-performance components with Intel® Celeron®, Pentium® or Core™ i3/i5/i7/i9 of the latest generation on an ATX motherboard.. The housing and cooling concept adopted from the C6670 also enables the use of a GPU accelerator card, among other things. A total of 300 W is available for full-length plug-in cards. Applications in the field of machine learning or vision can thus be realized in an industrial environment.