Smart Cap is an assistant for visually impaired which narrates the description of the scene.
There are about 285 million visually impaired people in the world. They are not able to experience the world the way we do. Smart cap aims to provide this missing experience for them. The system uses state of the art deep learning techniques from Microsoft Cognitive Services for image classification and tagging.
The smart cap aims bring the beautiful world as a narrative to the visually impaired. The narrative is generated by converting the scenes in front of them to text which describes the important objects in the scene. Examples of text include ‘A group of people playing a game of football’, ‘yellow truck parked next to the car’, a bowl of salad kept on the table’. For the first prototype of the system, one line along with some keywords are played as an audio to the users but in the later versions, a detailed description would be added as the feature.
The architecture of the system includes Amazon Echo, Dragonboard 410c/Raspberry Pi and online computer vision API’s.
A webcam which is retrofitted into a regular cap is connected to the Dragonboard/Raspberry Pi. The code given here runs on Dragonboard 410c/Raspberry Pi. The function of the code is to capture the image from the webcam and send it to Microsoft API’s for recognition task. The response is then inserted to DynamoDB. When the user asks Alexa to describe the scene, the Alexa Skills Kit triggers Amazon Lambda function to fetch the data from the database (DynamoDB). The correct text is the played as an audio on the Alexa device.
The project was awarded by Qualcomm and Amazon.