AR-BASED OBJECT RECOGNITION SYSTEM FOR ASSISTING VISUALLY IMPAIRED INDIVIDUALS

Abstract

An augmented reality (AR) based system is designed to assist visually impaired individuals by recognizing objects in real-time and providing auditory descriptions. The system utilizes wearable AR devices equipped with cameras to capture the user's environment. Advanced computer vision algorithms, including YOLO (You Only Look Once) for real-time object detection and ResNet (Residual Neural Network) for image classification, process the visual data to identify objects, obstacles, and points of interest. Natural Language Processing (NLP) techniques, such as BERT (Bidirectional Encoder Representations from Transformers), generate contextual descriptions of the detected items. The system then delivers customized audio feedback to the user via spatial' audio, describing .the type, location, and relevant characteristics of detected items. Key features include real-time object recognition, directional audio cues, and customizable settings to adapt to user preferences and environmental contexts. By enhancing situational awareness and providing timely information, this AR-based system aims to improve mobility, safety, and independence for visually impaired individuals in their daily lives. The invention also incorporates privacy controls, continuous learning capabilities using federated learning, and integration with existing assistive technologies to ensure a comprehensive and user-friendly solution.

Specification

FIELD OF INVENTION
The present invention pertains to the field of augmented reality (AR) systems and assistive technologies for individuals with visual impairments. More specifically, it relates to an innovative AR-based object recognition system designed to enhance the day-to-day experiences of visually impaired users by providing real-time information about objects in their environment. The system employs advanced computer vision and machine learning algorithms to recognize, classify, and label objects. These technologies are integrated into a wearable device, such as smart glasses, or a mobile device equipped with an AR interface. The invention provides users with immediate feedback through audio cues (spoken descriptions of objects), haptic signals, or other sensory outputs that effectively convey information about the recognized objects and their spatial locations. This AR-based system can recognize common objects like household items, signs, vehicles, people, and obstacles, offering essential navigational support and situational awareness. By using deep learning models, the system learns from various environments and enhances its recognition capabilities, adapting to new objects or settings over time. This invention aims to increase independence and improve the quality of life for visually impaired individuals, allowing them to better navigate public and private spaces, engage in social activities, and perform tasks with greater ease.

SUMMARY OF INVENTION
This invention is an augmented reality (AR) system designed to assist visually impaired individuals by recognizing objects in real-time and providing auditory descriptions. The system uses wearable AR devices, like smart glasses with cameras, to capture the user's environment.
YOLO and ResNet algorithms handle real-time object detection and classification, while BERT processes contextual descriptions of the identified objects. The system delivers customized audio feedback through spatial audio, describing the type, location, and characteristics of nearby objects. Key features include real-time recognition^ directional audio cues, and customizable settings. It also incorporates federated learning for continuous. improvement, privacy controls, and integration with existing assistive technologies, improving mobility and independence for visually impaired users.

DETAILED DESCRIPTION OF INVENTION
This invention presents an Augmented Reality (AR) system specifically designed to assist visually impaired individuals by recognizing objects in real-time and providing auditory . feedback to enhance their independence and safety. The system incorporates multiple advanced technologies in computer vision, deep learning, natural language processing (NLP), and spatial audio, delivering a comprehensive assistive solution. 1. System Architecture and Components The AR system consists of the following core components: • Wearable AR Device: The user wears a device, such as AR-enab led glasses or a headset, equipped with high-resolution cameras and microphones. These devices capture real­time video and audio from the user's surroundings. • Processing Unit: This is either integrated within the wearable device or connected to a smartphone or other computing device. It is responsible for executing the object recognition and processing algorithms. • Audio Output: Users receive auditory feedback via bone-conduction speakers, in-ear headphones, or spatial audio systems, ensuring that the feedback is clear and non­intrusive.
2. Object Recognition and Detection The system utilizes advanced computer vision algorithms to analyze and interpret the visual input in real time. Specifically, the system employs: • YOLO (You Only Look Once): This real-time object detection algorithm allows the system to detect multiple objects simultaneously with high accuracy and speed. YOLO identifies objects in the user's field of view and maps their spatial positions. • ResNet (Residual Neural Network): ResNet is used for image classification, allowing . the system to accurately classify the detected objects into specific categories (e.g., furniture, vehicles, people). ResNet's deep learning architecture ensures that the system can handle complex object recognition tasks with high precision.


3. Natural Language Processing (NLP) and Contextual Feedback Once objects are detected and classified, the system employs NLP techniques to generate contextual and informative descriptions for the user: *
• BERT (Bidirectional Encoder Representations from Transformers): This transformer­based NLP model processes the detected objects and contextualizes their roles in the environment. For example, if the system identifies a "chair" near a "table," BERT can infer that the scene represents a dining area or workspace. These contextual clues help provide more meaningful information to the user. 4. Customized Audio Feedback The system delivers personalized auditory feedback through various means: • Spatial Audio: The feedback is presented using directional sound, so the user can locate objects based on the direction of the sound cues. For instance, if an object is on the user's right, the audio will play from the corresponding side. • Descriptive Audio: Once objects are recognized, the system provides clear descriptions such as, "There is a chair 2 feet in front of you" or "A vehicle is approaching from your left."
• Contextual Feedback: In addition to object recognition, the system informs users about obstacles or points of interest, including crosswalks, doorways, or stairs. The system can also adjust descriptions based on environmental factors, such as lighting or time of day, and offer recommendations (e.g., "It's dark outside, be cautious of uneven terrain"). 5. Customizable Settings and User Preferences The system allows users to personalize their experience through customizable settings: • Object Recognition Focus: Users can prioritize certain object types (e.g., only detecting vehicles and obstacles while walking in a city). • Audio Feedback Settings: Feedback can be tailored for verbosity or detail, enabling concise notifications ("obstacle ahead") or more elaborate descriptions. • Environmental Adaptation: The system automatically adjusts to different environments such as outdoor, indoor, or crowded places by fine-tuning the object recognition and feedback frequency. 6. Continuous Learning and Adaptability To ensure the system remains up-to-date and adaptable, it employs:
Federated Learning: This allows the system to continuously improve without compromising user privacy. Federated learning enables the AR device to learn from user interactions and object recognition events locally, sending only anonymized updates to the central model. This ensures the system improves its accuracy over time and adapts to new environments, while maintaining robust privacy controls. • Personalized Learning: The system adapts to the specific needs and habits of the user, learning which types of objects or feedback are most relevant. For instance, the system may recognize that a particular user often navigates busy urban areas and will optimize its detection of vehicles, crosswalks, and pedestrians. 7. Privacy and Data Security Since the system captures and processes visual data from the user's environment, strong privacy controls are integrated:
• Data Encryption: All data is securely encrypted, both in transmission and at rest, ensuring that sensitive visual and auditory information is protected. • Local Processing: Where possible, object detection and processing are carried out on the device itself to minimize the need to transmit data externally. • User Control: The system provides users with full control over what data is shared and allows them to disable certain features (e.g., cloud updates) to maximize privacy. 8. Integration with Existing Assistive Technologies This AR-based system can be integrated with existing assistive technologies to provide a holistic solution:
• White Canes: The AR system complements traditional white canes by providing advanced object recognition, enhancing, the user's awareness of their environment beyond tactile feedback.
• Screen Readers and Navigation Apps: The system can interface with navigation apps and screen readers, providing seamless transitions from digital to physical navigation (e.g., guiding users from a GPS-based location to identifying obstacles in real time).

WE CLAIM
1. An augmented reality (AR) system for assisting visually impaired individuals, comprising a wearable device with a camera, real-time object detection using YOLO, object classification using ResNet, and an audio output system that provides spatial auditory feedback for enhanced navigation. 2. Natural Language Processing (NLP), including BERT, generates contextual descriptions of detected objects based on their relationships and enviroment. 3. The proposed system has featuring customizable spatial audio cues that allow users to personalize object recognition settings and audio feedback for different environments and tasks. 4. The system employing federated learning for continuous adaptation and improvement of object recognition while ensuring privacy through local processing and encryption of user data.

Documents