A Survey of Mobile Computing for the Visually Impaired

# A Survey of Mobile Computing for the Visually Impaired

Martin Weiss \AndMargaux Luck \AndRoger Girgis \AndChris Pal \AndJoseph Paul Cohen \And
Montreal Institute for Learning Algorithms, Université de Montréal
Polytechnique Montréal, McGill University
###### Abstract

The number of visually impaired or blind (VIB) people in the world is estimated at several hundred millionbourne2017magnitude []. Based on a series of interviews with the VIB and developers of assistive technology, this paper provides a survey of machine-learning based mobile applications and identifies the most relevant applications. We discuss the functionality of these apps, how they align with the needs and requirements of the VIB users, and how they can be improved with techniques such as federated learning and model compression. As a result of this study we identify promising future directions of research in mobile perception, micro-navigation, and content-summarization.

A Survey of Mobile Computing
for the Visually Impaired

Martin Weiss Margaux Luck Roger Girgis Chris Pal Joseph Paul Cohen Montreal Institute for Learning Algorithms, Université de Montréal Polytechnique Montréal, McGill University

\@float

noticebox[b]Preprint. Work in progress.\end@float

## 1 Introduction

The World Health Organization estimates that there are 253 million visually impaired or blind (VIB) people in the world bourne2017magnitude []. These people typically use a white cane to help them navigate the world, enabling them to detect uneven floors and static obstacles. Guide dogs can help VIB people detect and avoid dynamic obstacles such as cars, bicycles, and pedestrians. Nevertheless, common tasks such as sorting mail and medications, reading, separating laundry, selecting items on a menu, or navigating unknown environments still present a tremendous challenge.

Recent advances in machine-learning and mobile technology are starting to pave the way towards a higher degree of autonomy for the visually impaired through the development of software that can identify objects, extract text, or describe scenes. Organizations like Aira.io and BeMyEyes or initiatives like VizWiz:Social brady2013visual [] provide mobile applications that connect VIB users to sighted volunteers or employees through asynchronous image captioning and live video calls to handle more complex tasks.

This paper presents the existing mobile application landscape for the visually impaired. We discuss currently available features, the successes and failures of those technologies, and the various machine-learning models they employ. We conclude by presenting our vision for the creation of better tools and where research should focus to help people with visual impairments.

## 2 Features of Assistive Tools

Visually impaired or blind people would greatly benefit from increased autonomy. The software thus aims to provide them with opportunities and abilities that rival those of the sighted. This requires a robust set of features that includes object detection, robust image captioning, text extraction, semantic understanding, and micro-navigation. Figure 1 shows the conceptual framework for most assistive tools for the VIB. This should be used to guide the further development of tools and explore what is conceptually missing.

### 2.1 Perception

General object detection is necessary for daily activities like the localization and identification of furniture, computers, and food or drug containers. While a common feature, its actual effectiveness varies greatly across apps. For example, BlindTool Cohen2015 [] (2015) has the ability to quickly detect objects using an on-device neural network, but is restricted to only a thousand classes. A problem with all current object detectors is generalization. For example, apps like NantMobile Money Reader, LookTel Money Reader, Eye Notes, and others deliver high performance identification on a range of currencies, but they fail to generalize to newly distributed versions of bills.

Specific object detection refers to the task of identifying specific objects rather than general classes. These primarily include either objects familiar to an individual user such as their keys, car, wallet, or even friends, or those that are common knowledge such as famous landmarks or individuals. For example, Talking Goggles can recognize a wide variety of books, movies, and celebrities, whereas SeeingAI can fine-tune a device’s object detection model on relevant user-generated images so that the user can recognize unique classes. These functionalities are a fundamental step towards higher-order tasks such as micro-navigation and scene description.

Depth: Thanks to stereo vision, the sighted can estimate the depth of their surroundings quickly and reliably. The app Seeing With Sound [10.1109/10.121642, ] provides a proxy for this ability by translating an image into sound that the user can interpret. By moving the device around, the user can view the scene from different directions and determine the approximate depth. The challenge lies in acquiring a suitable depth map through for instance a stereo camera or extrapolating from a single camera’s output, and to then communicate its complexity to the user. One potential approach is to simply present a dynamic 3D representation of the scene to the user but no such solution has been developed yet.

Text extraction technology enables common tasks such as reading labels, price tags, documents, or pieces of mail and making sense of linear formats. The VIB user is prompted to position the text in front of the camera and then listen to audio feedback to center or scan the full text. Apps based on optical character recognition (OCR) such as KNFB Reader are rich in features. They allow the user to navigate by line, sentence, word, or individual characters in a wide variety of languages. Apps like Minerva, SeeingAI, and EnvisionAI also offer on-device OCR, although it is not their core feature. All currently available apps struggle to parse visually oriented layouts.

### 2.2 Content Summarization

Image captioning solutions use natural language to describe a scene. Querying a semantic representation of the environment can yield contextually relevant information to the VIB user. Unfortunately, available applications like BlindSight which perform the image captioning task on-device suffer from low accuracy, whereas apps like SeeingAI and EnvisionAI which offload the task to a server suffer from slow response times and sub-optimal interaction patterns. The lack of semantic and contextual representations of information in neural networks presents a major obstacle for designers of assistive technologies that provide image captions, except for BeMyEyes and VizWiz which crowd-source captions from human volunteers or employees.

Text summaries require a semantic understanding of the contents and context of a document. Significant amounts of prior information are often needed. Several apps, like Summly, Trimit, and WrapItUp have advanced technology for the summarization of news articles, papers, and other documents. However, we have failed to find any app which contains both text summarization and OCR functionality, a combination necessary for the interpretation of physical documents. These technologies cannot yet reliably extract, summarize, or make sense of tabular data.

Sentiment analysis is a necessity for inter-personal interaction. While the tone and semantic content of spoken words carries significant amounts of information, VIB people must rely solely on these. Computer vision-assisted analysis can identify non-verbal cues such as facial expressions and posture, but gestures and significant eye movements remain challenging to identify. Apps like SeeingAI, EyeCYou and Minerva achieve this task off-device mainly by using APIs such as SkyBiometry and Microsoft Azure cognitive services.

We define the micro-navigation task as that of traversing the last hundred meters to a destination. Aira.io solves this problem by pairing sighted professionals with VIB users through smart-glasses and synchronous video chat. For some VIB people, hiring professional assistants is prohibitively expensive, with plans ranging from $.50-$1.00 per minute. Additionally, smart-glasses often lack cosmetic acceptability, are expensive, and have limited battery life. Automating this task will require robust localization, path-planning, intersection crossing and obstacle avoidance.

Intersection Crossing is considered the most difficult and risky aspect of independent navigation for the VIB people shen2008mobile []. To safely accomplish this task the user must be aware of the spatial configuration of the intersection, the intersection’s signalling pattern, the correct orientation to follow while crossing, and the time when it is safe to begin crossing. While previous research has tackled these problems individually, we are not aware of an available mobile application capable of assisting with all the tasks involved in this challenge.

Obstacle Avoidance is primarily handled by the traditional assistive tools, such as guide-dogs or canes. While previous work has proposed solutions to tackle obstacle avoidance using machine learning and vision Poggi2016AWM [], their solution remains restricted to the lab setting as they require specialized hardware that is not commercially available. Options like the iOS BeAware app use a combination of beacons planted in the environment and bluetooth and WiFi. We are not aware of a viable commercial mobile application that tackles obstacle avoidance.

## 3 Technical Details

There are several obvious drawbacks to the apps we just presented: they can be expensive, lack cross-platform or multi-language support, have high latency, and are not always available. This is partly due to the fact that internet access is not always available raising the question of local versus remote processing (on- or off-device). Also, for some specific features, embedded models are needed to guarantee privacy. To achieve the trade-off between privacy, accuracy, autonomy and accessibility those apps use light models with specialized architecture and also machine learning techniques like one-shot object detection, model compression, and federated learning.

### 3.1 Specialized Architectures and One Shot Object Detection

Object detection models like Faster R-CNN ren2015faster [] that are two-stage networks including a region proposal network and a Fast R-CNN girshick2015fast [] work well but are slow.

One-shot object detectors like YOLO redmon2016you [], SSD liu2016ssd [], SqueezeDet wu2017squeezedet [] and DetectNet tao2016detectnet [] are faster and more suitable for mobile. They require only a single pass through the neural network using fixed grid detectors that allow the prediction of all bounding boxes at the same time. Those models are composed of a body network, usually pre-trained on a large image classification dataset like ImageNet, that acts as a feature extractor and a head network that detects objects.

However, those networks can be too large for mobile deployment. Instead, it will be preferred to use an architecture specifically designed for mobile like SqueezeNet iandola2016squeezenet [] that uses fire modules, SEP-NET li2017sep [] that uses filter convolutions and pattern residual blocks, MobileNet V1 howard2017mobilenets [] that is based on depth-wise separable convolutions or MobileNetV2 sandler2018mobilenetv2 [] that adds linear bottleneck and expansion convolution to MobileNetV1Howard2017MobileNetsEC []. SDDLite sandler2018mobilenetv2 [] has also been proposed as an object detection model and uses separable convolutions.

### 3.2 Model Compression

Over the past few years deep neural networks have received attention for their state-of-the-art accuracy on tasks in computer vision and natural language processing, but these models are often too heavy and energy-hungry making them impossible to run or are very slow on mobile device. Therefore, strategies for model compression in mobile device have been developed into a large area of research.

For example, Deep Compression proposed by Han et al. (2015) han2015deep [] reduces the size of a network using 3 independent processes: pruning, trained quantization and Huffman encoding. Pruning transforms the original dense network into a sparse one by applying a certain threshold on the weight values to retain only the relevant connections. Once the network is compressed it can be fine-tuned. Low-rank factorization is a computationally intensive compression method that uses tensor decomposition to determine the relevant parameters denton2014exploiting []. Trained quantization consists in han2015deep [] of applying clustering over the original or pruned weights of the network. Then, the weights are replaced with the centroids of their cluster permitting weight sharing. Other methods of quantization exist including neural network binarization courbariaux2016binarized [] and trained ternary quantization zhu2016trained []. The major drawback of these methods is that they are based on simple matrix approximations that can result in a reduction of accuracy. Huffman encoding is a loss-less data compression technique using sparse matrix indices han2015deep [].

Knowledge distillation can also be used to compress deep models. Those approaches shift knowledge from a larger teacher network into a smaller student network by learning the output class distributions using a softened softmax bucilua2006model []; romero2014fitnets []. Hinton et al. (2015) hinton2015distilling [] also proposed an ensemble of models composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse. However, those models can only be applied to classification through a softmax loss function.

### 3.3 Federated Learning

Federated learning is a way of training that allows for learning from data stored on a swarm of mobile devices without sacrificing privacy. In federated learning systems, a subset of users can identify a failing class, label training images, retrain their model locally, and share their improved weights to a central server where a federated learning algorithm produces a better, averaged model mcmahan2016communication [].

Models tailored for the visually impaired can be created through the use of federated learning systems. As previous research has explored macleod2017understanding []; salisbury2017toward [], the needs of blind users are fundamentally different than those of sighted ones as they do not have means of knowing the accuracy of a given description. We are aware of only one public visual question answering dataset, VizWiz gurari2018vizwiz [], intended to assist the visually impaired community. More subtly problematic is the disparity between the image captioning datasets used to train these models, such as MS COCO lin2014microsoft [], and the needs of this community who require descriptions that contain specific details. Federated learning allows user communities to retrain their models selecting exactly the kind of tasks they want to improve without sharing their private images to a server.

## 4 Where should the field go next?

Micro-navigation Einsiedler2012IndoorMN [] is an active area of research but has not yet matured. Existing solutions require models unsuitable for use on edge devices or specialized sensing hardware. Fundamental research on vision-and-language navigation anderson2018vision [], outdoor navigation Foell:2014:MUB:2694768.2694770 [], scene segmentation Alvarez:2012:RSS:2403272.2403301 [], and visual question answering Antol2015VQAVQ [] are necessary before these models can deliver value to VIB people.

Advances in computer vision and natural language models, novel model-compression techniques, and increasing mobile processing power are already enabling the first on-device solutions for content summarization and navigation. Tasks like text extraction, color identification, ambient light sensing, and object recognition are being performed on-device to take advantage of increased performance from lower latency, better data privacy, and independence from unreliable mobile coverage; tasks that still require larger models and more processing power must offload the user’s data and processing to powerful servers which reduces autonomy and incurs service fees. As technologies improve and model accuracies increase, it is important to continue pushing processing towards edge devices to take advantage of their inherent gains in latency, privacy, and accessibility.

## References

• [1] Jose M. Alvarez, Theo Gevers, Yann LeCun, and Antonio M. Lopez. Road scene segmentation from a single image. In Proceedings of the 12th European Conference on Computer Vision - Volume Part VII, ECCV’12, Berlin, Heidelberg, 2012. Springer-Verlag.
• [2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, 2018.
• [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. 2015 IEEE International Conference on Computer Vision (ICCV), 2015.
• [4] Rupert RA Bourne, Seth R Flaxman, Tasanee Braithwaite, Maria V Cicinelli, Aditi Das, Jost B Jonas, Jill Keeffe, John H Kempen, Janet Leasher, Hans Limburg, et al. Magnitude, temporal trends, and projections of the global prevalence of blindness and distance and near vision impairment: a systematic review and meta-analysis. The Lancet Global Health, 2017.
• [5] Erin Brady, Meredith Ringel Morris, Yu Zhong, Samuel White, and Jeffrey P Bigham. Visual challenges in the everyday lives of blind people. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2013.
• [6] Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 2006.
• [7] Joseph Paul Cohen. BlindTool – A mobile app that gives a "sense of vision" to the blind with deep learning, 2015.
• [8] Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint arXiv:1602.02830, 2016.
• [9] Emily L Denton, Wojciech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in neural information processing systems, 2014.
• [10] Jens Einsiedler, Oliver Sawade, Bernd Schäufele, Marcus Witzke, and Ilja Radusch. Indoor micro navigation utilizing local infrastructure-based positioning. 2012 IEEE Intelligent Vehicles Symposium, 2012.
• [11] Stefan Foell, Gerd Kortuem, Reza Rawassizadeh, Marcus Handte, Umer Iqbal, and Pedro Marrón. Micro-navigation for urban bus passengers: Using the internet of things to improve the public transport experience. In Proceedings of the First International Conference on IoT in Urban Space, URB-IOT ’14, ICST, Brussels, Belgium, Belgium, 2014. ICST (Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering).
• [12] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, 2015.
• [13] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. arXiv preprint arXiv:1802.08218, 2018.
• [14] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
• [15] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
• [16] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
• [17] Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, 2017.
• [18] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.07360, 2016.
• [19] Zhe Li, Xiaoyu Wang, Xutao Lv, and Tianbao Yang. Sep-nets: Small and effective pattern networks. arXiv preprint arXiv:1706.03912, 2017.
• [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision. Springer, 2014.
• [21] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European conference on computer vision. Springer, 2016.
• [22] Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell. Understanding blind people’s experiences with computer-generated captions of social media images. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. ACM, 2017.
• [23] H Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, et al. Communication-efficient learning of deep networks from decentralized data. arXiv preprint arXiv:1602.05629, 2016.
• [24] Peter BL Meijer. An experimental system for auditory image representations. IEEE transactions on biomedical engineering, 1992.
• [25] Matteo Poggi and Stefano Mattoccia. A wearable mobility aid for the visually impaired based on embedded 3d vision and deep learning. 2016 IEEE Symposium on Computers and Communication (ISCC), 2016.
• [26] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
• [27] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, 2015.
• [28] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014.
• [29] Elliot Salisbury, Ece Kamar, and Meredith Ringel Morris. Toward scalable social alt text: Conversational crowdsourcing as a tool for refining vision-to-language technology for the blind. Proceedings of HCOMP 2017, 2017.
• [30] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
• [31] Huiying Shen, Kee-Yip Chan, James Coughlan, and John Brabyn. A mobile phone system to find crosswalks for visually impaired pedestrians. Technology and disability, 2008.
• [32] A Tao, J Barker, and S Sarathy. Detectnet: Deep neural network for object detection in digits. Parallel Forall, 2016.
• [33] Bichen Wu, Forrest N Iandola, Peter H Jin, and Kurt Keutzer. Squeezedet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving. In CVPR Workshops, 2017.
• [34] Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv:1612.01064, 2016.
You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters