Algorithm for shoe-based blind assistance system

A shoe that tells you where to go: The Lower Austrian company Tec-Innovation has developed an intelligent shoe for detecting obstacles. The shoe, known as InnoMake, has recently been put on the market as an approved medical device and is intended to make the personal mobility of blind and visually impaired people safer.

“Ultrasonic sensors on the toe of the shoe detect obstacles up to four meters away. The wearer is then warned by vibration and/or acoustic signals. This works very well and is already a great help to me personally,” says Markus Raffer, one of the founders of Tec-Innovation and himself visually impaired.

The team led by Raffer and his founding partner Kevin Pajestka determined early in the development phase that two pieces of advanced information are extremely important to usability: the nature of an obstacle and its directional path, especially if it is downward facing, such as holes or stairs. “Not only is the warning that I am facing an obstacle relevant, but also the information about what kind of obstacle I am facing. Because it makes a big difference whether it’s a wall, a car or a staircase,” says Raffer.

AI recognizes walkable areas

After an active search, Tec-Innovation was able to win over Graz University of Technology (TU Graz) as a cooperation partner in 2016. Since then, the Institute of Computer Graphics and Vision has been working on a camera-based addition to the first version of the product, as computer scientist Friedrich Fraundorfer explains: “We have developed state-of-the-art deep-learning algorithms modeled on neural networks that can do two main things after detecting and interpreting the content of the image. They use camera images from the foot perspective to determine an area that is free of obstacles and thus safe to walk on. And they can recognize and distinguish objects.” The algorithms trained using machine learning can already be operated on a specially designed mobile system. Thanks to the latest powerful special processors, mobile use of the complex AI algorithms is now also possible. “This can be attributed to the enormous processor development of the past few years,” adds Fraundorfer’s colleague David Schinagl. The TU Graz algorithm is patented and has been transferred to Tec-Innovation.

Merging data into a navigation map

Tec-Innovation is now working on integrating the system into a prototype; the camera plus processor must be robustly and comfortably integrated into the shoe. Friedrich Fraundorfer and his team at TU Graz, in turn, have already turned to the next stage as the logical continuation of the project. They want to combine the information collected while wearing the shoe in terms of swarm knowledge into a kind of street view navigation map for visually impaired people. “As it currently stands, only the wearer benefits in each case from the data the shoe collects as he or she walks. It would be much more sustainable if this data could also be made available to other people as a navigation aid,” says Fraundorfer.

For the conception and prototypical implementation of such a street view map for blind and visually impaired persons, a funding application is currently being submitted to the Austrian Research Promotion Agency FFG. Because of the enormous challenges it faces, this type of navigation support is still in the distant future. According to Fraundorfer, the biggest sticking points are the continuously needed updating and expansion of the map, linking it with previous data, and the IT connection of the shoe system. For the researcher at TU Graz, however, one thing is certain: “We will definitely continue to stay on the topic. After all, in our highly innovative world, an alternative to the 70-plus-year-old cane for the blind must also be possible.”