A comparison of vision sensing technologies in autonomous systems
Sensors enable modern life. They are the eyes and the ears, and so much more, of electronic systems. The data we get from sensors is often raw but that is the nature of sensing; to provide an analog of the real world. Sensor data will only ever be an interpretation of the thing being sensed. Even our own five senses work this way.
The key here isn’t to get as close to the real thing as is absolutely possible but as close as is absolutely necessary. Knowing what to ignore can be just as important as knowing what to look for. It is also why autonomous systems use a variety of sensing modalities. Each one provides the right kind of data in the right way.
Active and passive sensing
There are many ways to categorize sensor modalities, but let’s start with a high-level bifurcation. Passive sensing is by far the most common; all inertial sensors are passive, and every smartphone has several inertial sensors. Inertial measurement units (IMUs) are mostly MEMS-based devices with multiple sensors. And they are tiny, measuring just millimeters. As they are so small, they can be integrated into almost anything. IMUs normally operate in a “dead reckoning” mode. That means they detect the relative change in a parameter and aggregate that change to provide an absolute measurement.
IMUs aren’t the only example of passive sensing used today. Image sensors are possibly even more widely utilized than IMUs. Image sensors detect incident light without emitting a source, which makes them passive. Passive infrared (PIP) sensors are the same. All these passive sensors are used in autonomous systems in different ways.
An active sensor works by emitting its own source signal and measuring how it changes, normally by detecting a reflection. Both radar and lidar are active, but so too is ultrasound. In autonomous systems, active sensing is now seen as an essential part of the sensor mix. We could consider GPS as being halfway between active and passive, as the sensor detects an emitted signal, but doesn’t generate it.
Sensors and the electromagnetic spectrum
The big three sensor modalities used in autonomous systems today are radar, lidar and image sensors. On the surface there is a lot of overlap in the data they provide. They are all predominantly used for object detection, as opposed to location or navigation. That is precisely why they are so interesting today, because location and navigation are largely passive activities, which can be implemented using passive sensing technologies. They rely on historic, immutable data such as the position of roads and junctions.
Object detection and collision avoidance, by its nature, is unpredictable. It requires technologies that can identify and classify objects and obstacles in real time and typically without having to refer to historic data.
We can group all three technologies under the loose heading of vision systems because their function is to provide visibility of objects that are relevant to the autonomous system. Anything that isn’t relevant can be safely disregarded.
Radar, lidar and image sensors can be compared in various ways, but let’s start with spectrum. They all work by detecting energy in the electromagnetic (EM) spectrum. Radar systems can operate from around 5 MHz to 300 GHz. Higher frequencies deliver higher resolution. In autonomous automotive systems the frequency range now most used is 76 GHz to 81 GHz.
Lidar uses energy in the part of the spectrum occupied by light, at frequencies in the 100s of THz. The wavelengths reach into the part of the spectrum visible to humans. As with radar, higher frequencies deliver higher resolution.
Image sensors predominantly operate in the visible part of the EM spectrum, although many can now detect in near IR, which improves low-light performance. Image sensors are clearly the most like our own eyes and so provide data that is familiar, often in color. Autonomous systems designed to interpret image sensor data can make use of data generated by other image sensors. This is a big advantage when applying artificial intelligence and machine learning, thanks to the large number of images now available for training purposes.
Unlike radar and lidar, the resolution of an image sensor is not dependent on the frequency of operation. It is dependent on the design of the image sensor itself, but also hugely reliant on the quality of the optics used. An image sensor is largely useless without a lens. The choice of lens will also influence the field of view. The pixel size and number of pixels in the sensor array will determine resolution, while the pixel design will define overall performance.
In many ways, an image sensor is much more complex than the transmitter and receiver used in radar and lidar systems. However, this complexity is amortized across the various parts of the semiconductor industry, which has enabled the technology to develop massively over several decades. This has driven performance up while pushing costs down. The same market dynamics haven’t existed in the markets where radar and lidar are strongest, but that is changing thanks to the huge interest growing around autonomous systems.
Ranging is the key to autonomous systems
If there wasn’t a good reason to use either radar or lidar instead of image sensors, then those market dynamics may never have moved on. The secret sauce is in their active nature and the “R” in their names, which stands for ranging. By emitting a controlled signal, reflections can be deciphered to reveal distance. This is what drove the development of radar and why lidar has emerged as its potential successor.
Both radar and lidar detect reflected energy. In a radar system, the amount of energy reflected will indicate the distance to the reflective object. A shift in the signal’s frequency, or the Doppler effect, will also indicate if it is moving closer to or farther away from the transmitter. Of course, in an autonomous mobile system, such as a car, the changing position of the transmitter will also need to be factored in.
With lidar, the energy is in the form of photons. Range is calculated by measuring the time it takes for the reflected energy to be detected. This is the Time-of-Flight (ToF) principle and is well established in ranging technologies. Some smartphones now use ToF to provide features from virtual tape measures to augmented reality.
Detecting range with image sensors is also possible. Our own two eyes allow us to perceive range because they are separated by a small distance and so have a slightly different perspective on the observed scene. Autonomous systems also use this trick to infer distance based on small differences in two instances of the same scene.
Determining distance while mobile
Another significant difference between image sensors and range sensors is position. As outlined above, image sensors infer everything, nothing in the scene really has inherent data. The big benefit of that is the image sensor system doesn’t really care about where it is in space or its orientation.
With active systems such as radar and lidar, the position and orientation of the transmitter/receiver are fundamental to their operation. With a relatively small aperture, the signal transmitted is highly directional and the received signal highly sensitive to that direction.
This means the overall radar/lidar system must always understand where it is looking. For lidar, this can be further complicated, as these systems often capture data in 360 degrees. They achieve this through rotation. Spinning around at high speed, the transmitter and receiver take advantage of the signal travelling at the speed of light for fast operation but must be acutely aware of the angle of rotation during operation.
Both systems must also understand the direction of travel and velocity if they are to provide useful data. This is implicit in an active system but much less relevant in an image sensor system.
Drawbacks and compromises
All three systems have their relative strengths and weaknesses. The weather can impact all three modalities, particularly rain, but radar performs fine in fog and works just as well in deep darkness as it does on bright sunny days. Lidar has higher resolution than radar and, importantly, that resolution works in all three axes. Even the smallest difference in surface contours can be picked up. This means lidar is the best technology for generating three-dimensional images.
Image sensors are designed to provide two-dimensional data. Organic brains understand perspective, and distance can be inferred by people. It isn’t quite so natural for image systems to understand the difference between near and far in a two-dimensional image.
Cost is another consideration. Image sensors are highly cost-sensitive and therefore cost-effective. Radar is becoming more common, having cut its automotive teeth on adaptive cruise control. Lidar is the relative newcomer and has been seen as the most expensive of all three modalities. That may be true, but the price gap is closing quickly.
The best of all sensing modalities
With its high resolution and robust nature, lidar has been touted as the best of all three modalities. There is some conjecture over if or when lidar will become the dominant technology. There are good reasons to choose just one sensor technology. Cost would be one of those reasons, but system complexity and serviceability would also factor. It would be simpler to standardize on a single sensor type, but only if it could provide the necessary data under all operating conditions.
It isn’t clear if lidar meets that criterium. It is more certain that radar doesn’t. There is potential for image sensors to take the lead and some car manufacturers are moving in that direction. Without the buy-in from the automotive industry it seems unlikely that lidar will reach the kind of market penetration needed to fuel its virtuous cycle of cost reduction and performance improvements. In this respect image sensor technology is many generations ahead of lidar, but still has inherent limitations that can’t be ignored.
Vision sensing technology at a glance
| Ranging Resolution | Field of View | System Complexity | Scene Dependency | Weather Dependency | Cost | |
|---|---|---|---|---|---|---|
| Image Sensors | None | Good, but dependent on optics | High | Works best in good light levels | Works best in good weather | Low |
| Radar | Good | Narrow | Medium - Low | Works well in any light level | Affected by rain | Medium |
| Lidar | Best | Narrow but steerable | Medium - High | Works well in most light levels | Affected by rain, fog, smoke | High |
When it comes to vision sensing technology for autonomous applications, each technology has its strengths and weaknesses.