A Device For Assisting Blind And Visually Impaired Individuals Using Augmentative And Alternative Communication.

Abstract

The present invention is related to a wearable device (100) that assists blind and visually impaired individuals by converting visual information into audio. The device comprises a camera or sensor (110) to capture visual data, a voice recognition module (120) for receiving commands, and a facial recognition system (130) to identify known or unknown individuals. It includes an object detection module (140), an obstacle detection system (150), and features for color (160) and currency (170) detection. A text-to-voice system (180) reads aloud text, while a voice-to-text system (185) converts speech into text. An ultrasonic sensor (190) enhances spatial awareness, and a server connectivity module (195) provides real-time processing with minimal latency. The device enables independent navigation by offering users auditory information, empowering them to identify obstacles, recognize faces, and perform daily tasks with ease.

Specification

Description:The invention is related to a wearable device is designed to assist blind and visually impaired individuals. Fig 1 illustrates a schematic of a wearable device (100). This wearable device integrates advanced technologies to assist blind and visually impaired individuals by providing real-time audio feedback based on visual input. With its combination of facial recognition, object and obstacle detection, color and currency identification, and text reading, it offers a comprehensive solution for independent navigation and interaction with the world around them. The voice recognition and voice-to-text systems further enhance the user experience, making the device a versatile and essential tool for improving the quality of life for individuals with visual impairments.
The wearable device (100) is an advanced assistive technology designed to aid blind and visually impaired individuals by providing real-time audio feedback about their surroundings. The device incorporates a variety of modules and sensors to gather, process, and convert visual information into a form that the user can easily understand. By combining these different features, the device enables greater independence for users in navigating their environment and performing everyday tasks.
At the heart of the device is the camera or sensor (110), which continuously captures visual information from the user's surroundings. This module functions as the eyes of the device, collecting images and data about objects, people, and other elements in the environment. The visual information gathered is then processed and interpreted by various systems within the device, allowing it to provide audio feedback to the user.
The voice recognition module (120) enables seamless interaction between the user and the device. Users can control the device through simple voice commands, allowing them to activate specific features like facial recognition, object detection, or text reading. This hands-free control is critical for blind and visually impaired individuals, as it allows them to interact with the device without needing to navigate complex interfaces. The device's ability to understand spoken commands ensures a user-friendly experience, enabling greater autonomy in its use.
One of the core functionalities of the device is the facial recognition system (130), which identifies known individuals in the user's surroundings. By comparing real-time visual data with stored facial information, the system can recognize people like family members, friends, or caregivers. When a known person is identified, the device announces their name through audio feedback. If the person is not recognized, the device will inform the user that the individual is unknown. This feature is highly beneficial in social interactions, allowing users to recognize people around them, even in crowded or unfamiliar environments.
The object detection module (140) plays a crucial role in enhancing the user's understanding of their surroundings. It identifies objects in the vicinity and provides audio descriptions, helping users become aware of important items or landmarks. This module is especially helpful in situations where users need to locate specific objects, such as furniture, doorways, or personal belongings, allowing them to navigate both indoor and outdoor environments more effectively.
Safety is a key consideration for individuals with visual impairments, and the device's obstacle detection system (150) addresses this need by identifying potential hazards. This system detects obstacles like vehicles, animals, or traffic signals, alerting the user to their presence through audio warnings. By recognizing these dangers in advance, the device helps users avoid accidents and make safer decisions while moving through public spaces, enhancing their overall confidence in independent mobility.
To further assist with daily tasks, the device includes a color detection feature (160) that identifies and announces the colors of objects. This is particularly useful for tasks like choosing clothing, sorting items, or identifying colored objects in the environment. By converting color information into spoken descriptions, the device enables users to perform color-dependent tasks with greater ease.
The currency detection module (170) adds another layer of functionality, allowing users to identify and handle money independently. When the user presents a currency note in front of the camera, the device captures its image and determines its denomination, then announces the value aloud. This feature is essential for financial independence, enabling users to manage cash transactions confidently without needing assistance from others.
Another valuable feature is the text-to-voice conversion system (180), which reads aloud printed text from images captured by the camera. This functionality allows users to access a wide range of information, such as reading signs, books, bills, or product labels. By converting written text into speech, the device makes it easier for users to engage with printed materials in everyday life.
The voice-to-text conversion system (185) provides users with the ability to transcribe their spoken words into text. This feature is particularly useful for creating notes, sending messages, or interacting with text-based applications. It gives users the ability to communicate in written form without requiring visual input, further broadening the scope of tasks they can perform independently.
The ultrasonic sensor (190) enhances the user's spatial awareness by measuring the distance between the user and nearby objects or individuals. This sensor is particularly helpful in navigation, as it provides real-time feedback about how far away obstacles or people are, helping the user move through spaces with confidence.
Finally, the server connectivity module (195) enables the device to process the captured visual data through an internet connection. This module connects the device to a remote server, where data can be processed and analyzed with minimal latency, ensuring that users receive real-time audio feedback. This ensures that the user experience is seamless and responsive, with minimal delay in processing visual information.
To ensure seamless operation and optimal functionality, the wearable device (100) for assisting blind and visually impaired individuals is built on a structured connectivity framework. At the core of this framework lies the Central Processing Unit (CPU) (200), which serves as the brain of the device. The CPU handles communication between all modules and ensures the efficient flow of data. It processes inputs from various sensors and modules, distributing tasks like facial recognition, object detection, or text-to-voice conversion to the relevant subsystems. By managing the overall data flow, the CPU guarantees real-time responses, thus enhancing the user experience. The device's camera or sensor (110) is a pivotal component, directly linked to the CPU. The camera captures visual data from the user's surroundings and relays it to the CPU, which distributes this data to other modules for further analysis. For example, visual data from the camera is sent to the facial recognition system (130) for identifying known individuals, or to the object detection module (140) for recognizing and reporting objects in the environment. The text-to-voice conversion system (180) also uses the camera data to convert any captured text, such as signs or printed materials, into audio feedback, enabling users to "hear" visual text content. To interact with the device, users rely on the voice recognition module (120), which is also connected to the CPU. This module captures the user's voice commands and transmits them to the CPU, where they are processed to activate the appropriate functionalities. Whether it's requesting the identification of nearby objects, detecting obstacles, or inquiring about the color of an item, the CPU executes the necessary tasks by activating modules like the obstacle detection system (150), color detection feature (160), or currency detection module (170), depending on the command issued. For environmental awareness, the device integrates an ultrasonic sensor (190) that measures the distance to nearby objects or individuals. This sensor continuously communicates with the CPU, feeding real-time spatial data to ensure the user is alerted to potential hazards or obstacles. The obstacle detection system (150), which includes sensors for vehicles and traffic signals, further enhances safety by scanning the environment for imminent dangers and notifying the user via the audio output system.
To assist with complex tasks like facial recognition or object identification, the device includes a server connectivity module (195). This module enables communication between the CPU and a remote server, offloading heavy data processing tasks that may require additional computational power. By transmitting captured data to a server for processing and then receiving the processed output, the CPU ensures that tasks are completed with minimal latency, maintaining real-time feedback for the user.
Additionally, the device incorporates a wireless communication module that facilitates data transfer between the device and remote servers, further enabling offsite processing of visual data. The battery unit powers all components, including the CPU, camera, sensors, and communication modules, ensuring the device can operate for extended periods. A connectivity port allows for easy charging and data transfer to external devices.
The audio output system serves as the primary interface for delivering feedback to the user. Once the CPU processes the data-whether it's identifying objects, detecting obstacles, recognizing individuals, or interpreting colors-the information is relayed to the user through audio alerts or voice outputs. This system ensures that all processed data reaches the user in an accessible and understandable format, enhancing independent navigation and interaction with the environment.
Figure 2 illustrates the method for providing audio information to blind and visually impaired individuals begins by receiving a voice command (step (a)) through a voice recognition module (120). The user issues verbal instructions, and the device recognizes and processes the commands to activate various functions. Upon detecting the voice command, the system activates a camera or sensor (110) (step (b)) to capture visual information from the user's surroundings. This visual data forms the basis for further processing by the device.
Next, the captured visual information is sent to a facial recognition system (130) (step (c)), where it is analyzed to identify known individuals or label any unknown individuals present in the environment. The method also incorporates an object detection module (140) (step (d)), which processes the visual data to identify objects within the user's vicinity, reporting their presence for enhanced awareness. Additionally, the system includes an obstacle detection system (150) (step (e)), which scans for potential hazards such as vehicles and traffic signals, issuing timely audio alerts for imminent dangers.
The method further enhances user interaction by performing color detection (160) (step (f)), allowing the system to recognize and vocalize the colors of nearby objects. In the case of handling money, the currency detection module (170) (step (g)) captures images of currency notes and vocalizes their denominations, aiding the user in financial transactions. Similarly, the system activates a text-to-voice conversion system (180) (step (h)) to read aloud any text that appears in the captured images, such as signs or printed materials, providing auditory access to written information.
To facilitate note-taking or messaging, the method employs a voice-to-text conversion system (185) (step (i)), which transcribes spoken input from the user into text. This enables the user to create text-based content through voice commands. Additionally, the device uses an ultrasonic sensor (190) (step (j)) to measure distances to nearby objects, helping to enhance the user's spatial awareness by providing feedback about their proximity to obstacles or other individuals.
The method also includes communicating with a server (195) (step (k)) to process the visual data remotely, ensuring minimal latency when delivering audio feedback. This server-based processing offloads intensive computational tasks, enabling real-time responses and accurate analysis of visual information. Finally, the processed audio information is delivered back to the user in real-time (step (l)), allowing for seamless navigation and interaction with the environment through audio outputs. This method provides a comprehensive solution for assisting blind and visually impaired individuals in interpreting and interacting with their surroundings.
To develop a comprehensive system that assists visually impaired individuals through a wearable smart assistive device, it is essential to integrate advanced technologies like object detection with various supportive features. This device's core functionality revolves around real-time feedback about the user's surroundings, including obstacle detection and object recognition, delivered through audio cues. By utilizing deep learning frameworks such as YOLOv4 and YOLOv5, the system enhances the capacity to identify and localize multiple objects in dynamic environments, which is crucial for enabling safe navigation.
Object Detection Fundamentals: Object detection is a vital aspect of computer vision that focuses on accurately identifying and localizing objects within an image. This process encompasses two primary tasks: object localization and object classification. Object localization involves determining the position of an object within an image by drawing a bounding box around it, while object classification identifies what the object is. Combining these tasks allows the system to recognize multiple objects simultaneously and provide meaningful contextual information about them. By implementing YOLOv4 and YOLOv5 algorithms, the system can track moving objects and identify static obstacles in various environments, thus facilitating visually impaired users' navigation through obstacles and interaction with their surroundings.
Primary Objectives of the System: The primary objective of the proposed system is to create a robust object recognition mechanism tailored to the specific needs of blind and visually impaired individuals. This includes identifying obstacles in the user's path to ensure safe navigation, enabling users to discern the types of objects in their environment, and calculating the distance from detected objects. Providing real-time audio feedback about identified objects is also crucial, as it facilitates user-friendly interaction, enhances independence, and fosters confidence among users.
Essential Software and Hardware Resources: To achieve these outcomes, the system requires specific software and hardware resources. The preferred operating system is Windows due to its compatibility with various programming tools and libraries. Python 3.9 is selected as the programming language for its simplicity and the extensive libraries available for image processing and machine learning, such as OpenCV and YOLO frameworks. The hardware requirements include a system with at least an Intel i3 processor, 8 GB of RAM, and a minimum of 512 GB of hard disk space to ensure smooth processing and real-time performance. Additionally, input devices like cameras and microphones are essential for capturing environmental data, while speakers provide the necessary audio output for conveying information to users.
Implementation Steps: The implementation involves several key steps. First, dataset preparation is critical, utilizing the COCO dataset, which is a large-scale object detection dataset containing images with labeled objects. This dataset provides diverse examples of various objects, enhancing the model's ability to generalize and perform well in real-world scenarios. Next, the object detection models are trained using the YOLOv4 and YOLOv5 frameworks. This training includes configuring hyper parameters, running training algorithms, and fine-tuning the models for optimal performance. Once trained, the models are deployed in a live environment where they process video feeds from the camera, detecting objects in each frame and localizing and classifying them in real-time.
Audio Feedback and User Interaction: A critical component of the system is the integration of libraries like pyttsx3 to provide audio feedback based on detected objects. The system should also incorporate a voice recognition feature, enabling users to interact with the device through spoken commands. This enhances user engagement and allows for customization of the system's settings. For instance, users might request specific information about their surroundings or ask the device to alert them to certain types of objects, fostering a more interactive experience.
Safety and Additional Features: Safety considerations must also be integrated into the system design. Features such as a built-in emergency alert system can be included, which notifies a predetermined contact or emergency services if a user encounters a dangerous situation or requires assistance. The device can also utilize GPS technology to provide location-based assistance, such as navigating users through unfamiliar environments or alerting them to nearby hazards.
User-Centric Design: Finally, user-centric design principles should guide the overall development of the device. It is crucial to involve visually impaired individuals in the design and testing phases to ensure that the device meets their unique needs effectively. Feedback from users can inform adjustments and improvements, leading to a more intuitive and user-friendly experience.
By integrating these various aspects, the proposed wearable smart assistive device can empower visually impaired individuals, significantly improving their ability to navigate their environments with confidence and independence. The combination of robust object detection, real-time audio feedback, interactive features, and a focus on user experience creates a comprehensive solution tailored to enhancing the lives of those with visual impairments.
, Claims:We claim
1. A wearable device (100) for assisting blind and visually impaired individuals, the device comprising:
(a) a camera or sensor (110) for capturing visual information;
(b) a voice recognition module (120) for receiving user voice commands;
(c) a facial recognition system (130) for identifying known individuals and labeling unknown individuals;
(d) an object detection module (140) for identifying and reporting the presence of objects within the user's environment;
(e) an obstacle detection system (150) for recognizing potential hazards, including vehicles and traffic signals;
(f) a color detection feature (160) for identifying and announcing the colors of objects;
(g) a currency detection module (170) for recognizing and vocalizing the denominations of currency notes;
(h) a text-to-voice conversion system (180) for reading aloud text from images captured by the camera;
(i) a voice-to-text conversion system (185) for transcribing spoken input into text for note-taking or messaging;
(j) an ultrasonic sensor (190) for measuring the distance to objects or individuals; and
(k) a server connectivity module (195) for processing captured visual data and providing output with minimal latency via an internet connection.
2. The method of claim 1, wherein the facial recognition module is configured to:
(a) store and retrieve pre-trained facial data from a database;
(b) compare real-time captured facial images with stored data to identify known individuals;
(c) output the name of the recognized person or an indication of "unknown" if no match is found in the database.

3. The wearable as claimed in claim 1, further comprising:
(a) a wireless communication module, enabling data transfer between the device and a remote server for processing and receiving visual recognition data;
(b) a battery unit, powering the camera, sensors, processing unit, and communication modules, with sufficient capacity for prolonged use;
(c) a connectivity port, allowing for physical connection to external devices for charging or data transfer.
4. The device as claimed in claim 1, wherein the camera module is mounted on the front of the wearable device, and the device is designed to be worn as a headset or goggles, positioning the camera for an unobstructed view of the user's surroundings.
5. A method (200) for providing audio information to blind and visually impaired individuals, comprising the steps of:
(a) receiving a voice command from a user through a voice recognition module;
(b) activating a camera or sensor in response to the detected voice command to capture visual information from the user's environment;
(c) processing the captured visual information using a facial recognition system to identify known individuals or label unknown individuals;
(d) utilizing an object detection module to analyze the visual information and identify objects within the user's surroundings;
(e) implementing an obstacle detection system to assess the environment for potential hazards and provide audio alerts for imminent dangers, such as vehicles or obstacles;
(f) performing color detection to recognize and vocalize the colors of objects in the vicinity;
(g) engaging a currency detection module to capture images of currency notes and vocalize their denominations;
(h) activating a text-to-voice conversion system to read aloud any text identified in captured images, such as signs or printed materials;
(i) employing a voice-to-text conversion system to transcribe user voice inputs into text for notes or messages;
(j) utilizing an ultrasonic sensor to measure distances to nearby objects, enhancing the user's spatial awareness;
(k) communicating with a server to process captured visual data and provide audio feedback with minimal latency; and
(l) delivering the processed audio information back to the user in real-time, enabling independent navigation and interaction with the environment.

Documents