Sensor Technology as the Interface between Man and Machine

With a rattle and a clatter, a gate creaks open in back. Endless racks stretch to just under to the ceiling, at a height of 13 meters. Between the racks winds a maze of narrow, sometimes crooked aisles. In one corner stands a tall stack of packaging material; in the other there a tower of pallet cages. Warehouse workers and order pickers hurry through the aisles. This is a typical scene in many warehouses and distribution centers. Warehouses are busy hubs for efficiently moving and handling goods, which makes them a very hectic and hard-to-manage environment.

Nonetheless, project manager and automation specialist Peter Krumbholz sees this environment as the perfect seedbed for the next stage of automation. Krumbholz has now been working for KION Group for 13 years—six of which were spent in advance development—and has played a key role in automating order picking processes for years as a project manager in the KI.FABRIK and other internal KION research projects.

He sees the hustle and bustle of a warehouse as “an outstanding foundation for truly autonomous machine operation,” because many of the processes in this environment are unpredictable. For machines, this means that they “have to make their own decisions in a short amount of time,” says Krumbholz, which presents a real challenge. Sensors play a particularly important role in overcoming this.

“In the warehouse of the future, everything revolves around sensors,” adds Alexander Billiet, Manager Solution Design Technical Engineering at KION subsidiary Dematic: “Sensors are the eyes and ears of autonomous intralogistics.” Without sensors, forklift trucks, robots and other machines cannot perceive what is going on around them, which means that they cannot react to changes in this environment. Sensors are therefore an essential requirement for multiple systems and trucks to work together in the same hall. In the future, they will also become increasingly aware of each other and be able to interact, says Billiet.

This particularly applies to mixed human-machine operation. “Until now, it’s been more a form of co-existence: Man and machine are often separate from one another, in terms of space and time, and hardly ever interact,” explains Peter Krumbholz. This should change in the future thanks to (even) better sensor technology and systems, and genuine human/machine collaboration. Krumbholz believes that the potential of this kind of “genuine” collaboration is vastly underestimated.

People could fill gaps in processes that are too much for the machines to handle and vice versa. He believes that this would make warehouses more efficient and economical. And Dematic expert Alexander Billiet is even more convinced, adding: “The really cool thing is that we can increase the level of safety using sensors, without having to make any concessions in terms of productivity. Man and machine can both complete their respective jobs without hindering or colliding with each other. The robots detect the humans’ movements and take them into account accordingly.”

Much of this is already a reality. Stereo cameras enable 3D imaging and create a depth or distance value for each pixel, which enables Automated Guided Vehicles (AGVs) to form a precise picture of the space around them. The scan rates (i.e. the frequency with which a sensor delivers values) are already high today. But when it comes to the intralogistics of tomorrow, this technology cannot be viewed in isolation.

After all, it’s not enough for sensors to deliver more data in a higher frequency and quality; the data also needs to be processed. “Ultimately, many different disciplines and technologies come together to enable the use of modern sensors in practice,” says Krumbholz: “An interdisciplinary team is required to make the project a success.”

Interconnection is also a prerequisite and having a high level of connectivity in the warehouse and high transfer rates—with 5G for example—enables the sensor data to be analyzed and assessed in real time. “Until now, the data processing technology has been situated in the machines themselves. Today, data is taken from a combination of forklift trucks, wearables and the infrastructure. That’s why it’s now migrating to the cloud,” explains Krumbholz. This means the data is no longer processed in the truck itself, but externally in a “data lake” from where it can be accessed by the respective company.

“Data is the new gold, if you will,” says Peter Krumbholz. But what does he mean by this? “Companies that master sensor technology can use it to train neural networks and artificial intelligence. The more high-quality data there is available, the better products and solutions can be provided to the customer, which ultimately increases revenue,” explains Krumbholz. Research projects like the ARIBIC project, started in 2021 by the software company LeddarTech, the Karlsruhe Institute of Technology (KIT), the STARS Lab at the University of Toronto and KION, therefore produce huge volumes of images and data, which are used to train neural networks.

In a process known as “labeling,” countless photos of an object are shown to the system, from a range of directions and perspectives and in a variety of quality and lighting conditions. This is repeated until the system is able to recognize the object itself and knows, for example, whether it’s a forklift truck or a pallet. The training phase involves a lot of work but is a key element in autonomous interaction between man and machine.

The interplay between AI, cloud technology and sensor technology is giving rise to something new, which is greater than the sum of its individual parts. The ultimate goal is “machine vision.” “This would enable the technology to make its own decisions,” explains Alexander Billiet from Dematic, adding, “but we’re not quite there yet.”

And yet, at the moment, it’s already about more than just increased automation and this relates to the subtle difference between automation and autonomy. “Automation generally refers to repeated processes that run without self-adaptation. Autonomy, on the other hand, refers to systems that react to their environment and its influences,” explains project manager Krumbholz. Hybrid systems currently dominate day-to-day warehouse processes as they are used both for automated and autonomous solutions.

One example of this is the iGO Neo order picker from KION subsidiary STILL. The forklift truck largely acts autonomously in mixed operation, but does, however, rely on safety-certified laser scanners to comply with safety standards. The Machine Directive which applies Europe-wide and the relevant standards derived from it require AGVs to meet a level of safety that AI-assisted imaging is yet to meet.

In the meantime, that is not deterring KION Group constructors and engineers from integrating machine vision into safety-certified systems and having both systems work together. Krumbholz: “Certain AGVs use machine vision to move through the warehouse, while the certified laser scanners have the last word in crucial moments and prevent collisions.” He believes that it won’t be long before machine vision can attain the required safety level on its own.

While automated processes certainly require a neat and manageable warehouse environment, the warehouse of the future will be more chaotic and dynamic. According to Peter Krumbholz, autonomous trucks will soon be able to move around this challenging environment by themselves, thanks to sensors, cloud technology and AI. Krumbholz is convinced that there will still be some role for us to play in the future, however: “Humans will continue to be indispensable in the intralogistics of the future.”