When Trains Learn to See

Author -

FEV etamax

Published -

Reading time -

6 mins

For about 200 years, trains have become faster, safer, more efficient and increasingly connected. Modern train control systems know a great deal about the state of railway operations: which routes are safely set, whether monitored track sections are reported clear, and how train movements must be protected.

What these systems traditionally know much less about is what is happening outside this controlled information environment. A person near the track, an unexpected obstacle or vegetation protruding into the train’s path are still largely left to the driver to notice and assess. As railway operations become more automated, some of this capability must move into the technical system.

That is what environmental perception is for. The first step is to equip trains with cameras, radar and LiDAR, but what happens next? After all, seeing is not the same as understanding.

Infographic showing the historical evolution of trains from steam locomotives to high-speed rail, highlighting how modern environmental perception enables trains to understand unexpected surroundings.
Figure 1: Evolution of trains – From powerful steam engines to intelligent environment perception („Understand the Unexpected“).

From sensor data to useful information

Each sensor perceives the environment differently. Cameras provide rich visual information, LiDAR provides precise spatial measurements, while radar can robustly measure object range and relative velocity. Combining complementary sensor properities can provide a more robust representation of the environment than relying on a single sensing principle.

An environmental perception system needs to identify the relevant track, detect and classify objects, determine where they are in relation to the train and track them as the situation develops. Information from several sensors can then be fused into a consistent representation of the surroundings. This is the point where raw sensor data becomes useful information, something an operator or another vehicle function can actually work with.

One environmental perception system developed by FEV together with Wabtec illustrates this chain. The system uses camera and LiDAR data to determine the ego track, detect switches and identify objects around the locomotive. Detected objects are classified according to their position, for example whether they are within the foul volume [CB1.1]or right-of-way area. Sensor fusion is used to improve detection and range measurement. The resulting information can then be provided to the driver and automated control functions.

Infographic demonstrating the four-step AI perception workflow (Sense, Fuse, Understand, Inform Action) for environmental perception, combining camera, LiDAR, and radar data for real-time track and obstacle detection.
Figure 2: AI Perception: From Raw Sensor Data to Railway Environment Understanding

The challenge of uncertainty

Perception systems do not operate with the same kind of information as conventional railway signalling. Conventional signalling deals largely with predefined system states whose safety properties can be specified and verified. Environmental perception has to interpret a scene that was not completely specified beforehand. That introduces uncertainty.

Consider the visual recognition of a switch position. A perception system may independently observe how a switch appears to be positioned. This, however, is fundamentally different from obtaining safety-related information about the switch state from the signalling system.

A perception algorithm may express confidence about an individual observation, for example through a confidence score. “I am 90% confident that the switch leads to the right.” But a confidence score for an individual observation is not the same as demonstrating the safety integrity of the function as a whole. Being quite sure about an observation is therefore not the same as knowing that a function is safe.

So how can this uncertainty be handled? One answer may be better perception, but additional independent information from infrastructure or other sensing principles may be more effective than simply improving the perception algorithm.

This is also why adding ever more capable AI is not automatically the best solution. If the perception problem can be simplified or additional deterministic information can be made available, this may lead to a system that is easier to understand and verify. AI becomes particularly valuable where the variability of the environment makes purely deterministic approaches impractical.

For safety-relevant applications, that distinction becomes crucial. A system that warns a driver about an obstacle has a different responsibility than one that directly initiates a safety-critical reaction. Moving from assistance towards automated intervention therefore requires not only better detection performance, but a clear understanding of system boundaries, uncertainties and the consequences of incorrect or missing information.

Do not wait for automated trains

It is tempting to discuss perception mainly as one component of a future fully automated train. That misses much of its near-term value.

Perception can already support assistance functions in which the operational concept deliberately keeps the human in the decision loop. Obstacle detection and collision warnings can provide additional information to a driver. Remote operators can use onboard cameras and connectivity features to supervise and control locomotives. Environmental information can also be combined with other vehicle and infrastructure data to improve situational awareness.

This allows perception capabilities to be introduced gradually and for specific operational purposes rather than waiting for a fully automated system.

The operational environment matters here as well. Highly automated operation is already established in controlled environments such as many automated metro systems. Open mainline operation creates a very different problem. Between these extremes are applications such as shunting yards and closed industrial operations, where speeds and operating environments are more constrained. These applications can provide realistic opportunities to introduce perception and automated functions.

And sometimes the right conclusion will still be that automation does not provide enough benefit for a particular operation. Technical feasibility alone is not a sufficient reason to automate a crucial train control function. There also has to be an operational benefit.

Useful perception

Environmental perception does not have the same purpose in every railway application. In depots, shunting areas or industrial railways, it may be sufficient to determine whether the relevant section of track is clear and safe to proceed. This limits the perception task to the information actually required by the operational function.

In tram and regional rail operations, perception may have to cope with road traffic, pedestrians, level crossings and other situations in which objects can legitimately approach or cross the track. For high-speed mainline operation, the challenge changes again: at high speeds, it is not only important whether an obstacle can be detected reliably, but whether it can be detected early enough for the subsequent function to take meaningful action.

Here, other applications such as remote operation, platform monitoring or safe departure may provide more immediate value.

This is also where FEV.io can support railway operators and manufacturers: starting with the operational problem rather than a particular sensor, algorithm or level of automation. This can mean defining what information a function actually needs, developing and integrating perception solutions, or assessing how perception can be introduced into safety-relevant railway systems. The goal is not automation for its own sake, but a solution that provides a real operational benefit and can be integrated into the railway system in a technically sound way.