Irisys Experts - Vector 4D Q&A

The ‘Irisys Expert Interviews’ are a series of interviews designed to cover key themes in Irisys technology and applications.

In this interview, Nick Stogdale chats with Tim Wheatley, Irisys CTO, and Neil Johnson, Irisys Chief Software Architect, about the development and capabilities of the Vector 4D people counter.

NS: Vector 4D is Irisys’ innovative ‘Time-of-Flight’ people counter. Let’s start out with a description of ‘Time of Flight’ technology in layman’s language. Tim, can you describe to a man in the street what it is we are talking about here?

TW: Time of Flight (ToF) technology is a way of building up an ‘image’ of a scene based on distance. So every point in the scene is represented, not as an image of the scene at that point, but by a measurement of the distance from the detector to that point in the scene. ToF basically works by flooding the scene with pulses of light and detecting the reflections from the scene. The round trip time taken for the light being emitted to being received is used to work out, very accurately, the distance to every point in the scene. In fact, ToF technology has been used for many years for highly accurate distance measuring in laser tape measures, for example.

"Very reliable, very robust"

So, fundamentally, the ToF technology in Vector 4D provides a very reliable, very robust way of directly working out the distance to every point in a scene instantaneously and building up a picture from that.

NS: That makes sense, so following on from that description, why is the Time of Flight technology in Vector 4D ideal for people counting?

NJ: I think that Time of Flight (ToF ) is a game-changer for any machine vision application. Rather than having an image of the scene in which you have to infer where objects are, where their boundaries are and what type of object each is, you are directly and accurately measuring the shape and depth of the objects in the scene. The processing task becomes much simpler, so you can search the scene for objects, identify the different objects and where they sit relative to one another to a very high level of accuracy. This is much harder with classical video imaging technologies. Also with the Time of Flight detector, the depth accuracy and the errors generated are far smaller than you might get in with something like a stereo video system.

With a stereo algorithm for instance, you are reliant on matching two images to understand the depth of the scene, there will be textures and patterns, and there will be blank areas in the scene that are very hard for those kinds of algorithms to deal with and that can cause errors. Time of Flight is a much more reliable way of measuring, giving you direct measurement of the depth at each point rather than having to estimate it from very complicated algorithms. You are doing far less work to get the depth information itself, so the accuracy is greater and then it’s much easier to process that data, find and track people within the scene and ultimately to very accurately count people and obviously derive high-quality analytics from there.

NS: So the people counting algorithms in the Vector 4D are familiar to us from our extensive experience of thermal people counting?

NJ: One thing that we have always paid a lot of attention to at Irisys is building algorithm sets that are state-of-the-art in machine vision but applied in a bespoke way to our requirements. This is what we did with the thermal detectors. We looked at best practice within the world of computer vision at the time and the way tracking algorithms were constructed, and then we looked at the data we had and the peculiarities of the thermal system and we built an algorithm set which was optimal for that task.

"State-of-the-art machine vision"

With the Vector 4D Time of Flight detector we’ve been back to the literature and refreshed ourselves on the best techniques that are currently being used to track people using depth data. Again, we have applied the Irisys approach to that. Rather than using a standard set of algorithms or a generic tracker that’s designed to track from all sorts of different positions and all sorts of different objects, we’ve built something that is very specific to our task.

NS: If I were to look at the competitive space, as an external observer, I’d say that stereo technology seems to dominate. So what makes Irisys think that Time-of-Flight is the way to go, as this would almost appear to be against the main technology direction at the moment?

TW: Well you have to consider that stereo detection is probably a 10-year-old technology. The reality is that when stereo people counters first came out, there were not that many viable ways of doing 3D imaging available at an economic cost so that was the only option you had. The reality is that the 3D imaging world has moved on a long way over the last 10 years, a lot of applications for 3D imaging have emerged fully autonomous cars, gesture recognition for computers and tablets. There are many applications where high-quality 3D imaging is really driving technology now and Time of Flight really is a technology whose time has arrived.

There has been a lot of development in the last 5 years, a number of chips have become available and the price is dropping to a point that makes the benefits of Time of Flight available for mainstream people counting devices. There are a number of benefits of using Time of Flight for Vector 4D, it works in complete darkness, it will give you more accurate distance measurements than stereo and possibly one of the biggest benefits is the fact that you are not wasting a huge amount of processing power actually working out what the depth field looks like, its available to you directly from the sensor.

NS: Perfect, so Neil, we’ve been talking about using the depth field for people counting what other information can we draw out of the Vector 4D that might be interesting for other retail analytics?

NJ: I think there are a number of ways in which the data can be used to provide a richer set of information. Firstly, it’s extremely accurate for tracking and will give highly accurate people counting as a consequence. This kind of technology opens up your ability to reason in more complex ways about what people are doing around scenes.

"More advanced analytics"

So, for example, if you can follow an individual accurately for a prolonged period of time possibly over a number of sensors, then you can start to look in more detail at the behavioral aspects of what they are doing. You can start to use more complex algorithms to infer their behavior. You can understand how they are interacting with the scene, with products, messaging, advertising and so on. So, there is a whole area of more advanced analytics, which the Vector 4D technology will open up.

NS: Also, you can accurately measure the height of people?

NJ: Yes absolutely. Height is measured directly rather than being estimated as it is in other technologies, so you get a highly accurate height measurement. So, you can look at the height, you can look at the distributions of heights in different scenarios. You will get a ‘height histogram’ showing the height range for people coming into a store directly from Vector 4D. You can filter your counts according to height limits that are intelligently set, and you can look at the demographics of people using the store. So, yes absolutely, there is height information available from Vector 4D of a very high quality. One of the other advantages of ToF over stereo is that it doesn’t need to see texture in the scene, so it doesn’t need to see edges or objects in order to measure distance. One of the benefits of that is that we can see the floor and we can work out how far the floor is away from the detector so the Vector 4D can work out its own height.

NS: Tim, so what other information can you draw out from the Vector 4D ToF detector? One of the things that thermal technology is not good at is when people stop or linger it is difficult for the thermal detector to keep track of them. This is not a problem for counting at a doorway, but it does limit applications in dwell monitoring. How is Vector 4D ToF different in that respect?

TW: The Time of Flight technology in Vector 4D has no problem with stationary people, indeed it will keep a target acquired in the scene pretty well indefinitely. So applications become possible with Vector 4D where you want to understand people’s dwelling behavior, for example, around supermarket deli counters, advertising signage, product displays, product on shelves and so on. Understanding how people move around and dwell in more detail is directly available from the Vector 4D. In fact, you will get a ‘dwell histogram’, showing the range of time that people have been stationary in a location, directly from Vector 4D, opening up a whole range of different applications.

NS: Excellent. How exciting is it to be working on a whole new technology?

NJ: Extremely exciting. We have been working for a long time with the thermal technology and that has been an interesting journey. We have progressed through the story from single detectors to multiplexing detectors together, from people counting in retail to tracking over wide areas, right through to specialist algorithms for queuing at supermarket checkouts where we are the undoubted market leader.To get our teeth into something fundamentally new and very cutting edge is very exciting. As Tim has said, Time of Flight has now ‘come of age’.

"Time of Flight is a game changer"

ToF is a game-changer in terms of the type of algorithm you can develop and the kinds of reasoning you can do with the image. Suddenly a lot of the really hard problems of computer vision disappear and you can focus your efforts on the next level of algorithms, on the analytics and working out what people are doing within the scene. For Vector 4D it is a question of how far we can take this technology and this opens up some innovative and interesting ideas for the future.

NS: Excellent. It’s true Tim, there’s been a buzz around the place since we started this work and you can feel it.

TW: Yes, yes absolutely. For example, just think about the idea about being able to use ToF technology for other behavioral analysis work. I think in some respects it’s wrong to call Vector 4D a ‘people counter’ because I think that does it a huge disservice. It’s really a ‘behavioral analytics tool’ and it happens to be that we have chosen people crossing a line as the first bit of behavior that we have chosen to analyze. In fact the system and the technology has the ability to produce way more sophisticated information about how the targets are moving in the field of view and we are only just beginning that journey.

NS: Presumably, you could extend that thinking to our other business areas. For one, Smart Building applications comes to mind. Other things that we have always been good at in the Queue Management side of the business, will we be bringing the Vector 4D device into those areas as well?

TW: Yes indeed. I think in Smart Buildings there are lots of potential applications beyond just the basic people counting and occupancy-monitoring type of function. Looking at meeting room occupancy, for example, you can measure and understand the number of people who are in a meeting room at any particular moment in time. For Queue Management, the high performance of the Vector 4D in dwell and stationary targets will be of huge benefit there in opening up some new applications.

NS: Very good. So Neil, not everything can be perfect! What do you think will be the ‘edge effects’ here, where will Vector 4D ToF have potential issues?

NJ: In a sense that is a difficult question because at the moment Vector 4D is behaving extremely well in all the environments that we have used it, which is good! There is potential for ToF or active illumination technologies like this to be affected by sunlight and so on but we are working extremely hard to make sure that the device performs very well in all scenarios.

"The detection technology is highly advanced"

The detection technology is highly advanced now and we go to great lengths to test its performance over all conditions that might be encountered. We have test environments that we have built to replicate the sunniest climates to make sure that we have pre-empted those scenarios and the device can deal with that. This is why I say it’s difficult to see how the boundaries are going to go. As we always do at Irisys - we have built this thing properly from the ground up. We’ve done the physics, we’ve done the experimental science, we’ve thoroughly understood the technology we are working with and built the platform to be robust and able to deal with all of the situations that we can expect it to encounter.

NS: What is the future for the thermal technology, Irisys has been world leaders in that field for 15 years we are not proposing to stop manufacturing thermal detectors?

TW: No absolutely not, thermal remains and will remain a key part of Irisys’ product portfolio. We firmly believe that thermal makes a very good, robust, stable people counter and we really see the benefit of the Vector 4D product sitting alongside a ‘Vector Thermal’ in a product portfolio.

You will be able to make the decision about which detector you want to use at the shop doorway rather than at the beginning of the roll out as is previously been the case. No, Thermal is definitely going to continue to be a major part of our portfolio. Let’s face it it’s installed in over 500,000 locations, the market indicates that it’s a robust and reliable technology!

NS: OK, on reliability - we know almost everything there is to be known about the thermal people counter and its reliability is second to none. Are you working to replicate this reliability with the Vector 4D device, and there is no reason to assume that the reliability will be anything other than excellent?

TW: Oh absolutely yes. This comes down to what Neil was saying about the fact that we have approached this very much from the ground up, we have done all the physics, we’ve done all of the detailed simulation, thermal simulation, reliability simulations of the system, the LEDs and the processor.

Reliability is our reputation

We’ve gone about this in a very thorough way as we do with all of our products and the reality is our reputation as having the most reliable product out there doesn’t come from luck - it comes from engineering and knowledge and we are using the same knowledge and the same understanding in the engineering of the Vector 4D product.

NS: So there is no possibility that the LED’s which are in the active illumination system, could be a potential reliability weak spot in the Vector 4D?

TW: No, every electronic component has a lifetime and that lifetime is generally degraded by operating temperature. LEDs are no different, they are not special in any way, and they are no different to the processor or the other components. It is important that you manage the thermal conditions of any piece of electronics in order to give it decent longevity. So yes, we have modelled the thermal management to ensure that we maintain the temperature of the device at an acceptable level. This is a key part of our reliability work and it would be true of our thermal detector as well as our Time of Flight detector. There is really no difference in reliability with Vector 4D.

SEE VECTOR 4D IN ACTION - SCHEDULE A DEMO