Breakthrough AI transforms how people with visual impairments experience the world, giving them tools to discover, understand, and experience the beauty of unfamiliar places like never before.
Study: AI system facilitates people with blindness and low vision in interpreting and experiencing unfamiliar environments. Image credit: Angel Santana Garcia/Shutterstock.com
A team of researchers from China developed an artificial intelligence (AI)-driven system that can potentially help visually impaired individuals explore, understand, and relish unfamiliar environments surrounding them. The study is published in the Nature Portfolio Journal Artificial Intelligence.
Background
Exploring natural environments, such as parks, has a significant positive impact on physical and mental health. However, people with low vision or blindness are often excluded from these benefits because appropriate assistive aids are not available to help them proactively engage with them.
Existing assistive solutions developed to guide visually impaired individuals primarily focus on providing functional assistance, such as navigation and obstacle avoidance, allowing them to engage with nature passively.
Visually impaired individuals often feel helpless while exploring unfamiliar environments. This usually means they rely on family members, friends, or volunteers for assistance, which impairs their ability to actively explore and understand unfamiliar environments, as well as to remember and communicate with other visually impaired individuals about their journey.
A team of China-based researchers developed an AI-driven System named VIPTour to offer visually impaired individuals a sense of independence in unfamiliar environments.
How does VIPTour function?
VIPTour is an AI-driven system containing a set of lightweight, portable, consumer-grade devices (a camera and a smartphone) and a novel deep-learning algorithm network called FocusFormer. Efficient multisensory interaction techniques, such as audio and hierarchical tactile interaction, drive the interaction between visually impaired users and the VIPTour system.
FocusFormer considers aesthetics, freshness (novelty), and basic needs (including navigation and safety) as the main factors in extracting meaningful information from complex, unfamiliar environments and excluding redundant visual details. This reduces the cognitive load on visually impaired users.
FocusFormer transforms vast amounts of information into a structured, sparse, and hierarchical personalized graph. Based on this well-structured graph, FocusFormer interacts with visually impaired users through a smartphone application, understands their preferences, and provides personalized assistance through an adapter.
It is trained with thousands of public tourism videos from sighted tourists in a self-supervised manner, which is beneficial for effectively reducing aesthetic bias.
The VIPTour system also has options for recording, storing, and sharing experiences, facilitating emotional communications among visually impaired individuals, and promoting the exchange of knowledge and experiences within their social networks.
VIPTour’s core technical innovation lies in its multi-attention FocusFormer network. This approach utilizes a background subnetwork to filter out commonly seen objects, an attraction subnetwork to identify highlights, a freshness subnetwork to discover novel features, and a needs subnetwork trained on surveys conducted with visually impaired participants. These subnetworks combine to select, rank, and present the most relevant information for each user.
The VIPTour system also utilizes a BLV-in-the-Loop Adapter, which updates its recommendations in real-time based on individual user feedback, such as “likes” and “dislikes,” thereby enabling personalization.
User opinion about VIPTour
The VIPTour system was tested on 33 individuals with blindness or low vision, and self-reported emotional experiences were collected for analysis.
Regarding assistive performance, the study found that the VIPTour system effectively helped visually impaired individuals actively explore and thoroughly understand unfamiliar environments, empowered them with accurate and long-lasting recollections, and enabled them to communicate with their peers.
By extensively analyzing self-reported experiences, the study found that the participants using VIPTour successfully achieved a 67.9% increase in positive emotional response, a 94.7% increase in arousal, a 772.73% increase in cognitive mapping accuracy, and a 200% increase in long-term memory accuracy.
In user evaluations, the VIPTour system’s usability scores were consistently above 80 out of 100, comparable to or better than those of other assistive tools for visually impaired individuals.
Physiological measures, including electrodermal activity and heart rate variability, showed significant improvements with VIPTour use, indicating enhanced emotional engagement.
Study significance
The study highlights the potential uses of the AI-driven VIPTour system in providing visually impaired individuals with an enjoyable and memorable experience while actively exploring unfamiliar environments. These experiences can significantly boost their emotional state and improve their overall quality of life.
Existing evidence suggests that presenting organized and engaging information can enhance a person’s pleasure level and facilitate deeper memory retention. Humans have a natural tendency to process well-structured and meaningful information, which makes their experiences more enjoyable and memorable.
This human tendency may be explained by the concept of cognitive fluency, which indicates that clear and organized information presentation reduces the cognitive load on individuals. Subsequently, this helps them channel mental resources towards understanding and integrating the content. This improved processing fluency induces a positive response, as individuals perceive the information more pleasantly.
Furthermore, the interaction between novel and familiar information influences the effect of organized and interesting information on memory. Novel information stimulates curiosity and enhances attention, while familiar information provides cognitive comfort and coherence.
Presenting the information in a structured and engaging way can balance novelty and familiarity, which helps maintain individuals’ interest and engagement.
The self-supervised training of FocusFormer with thousands of unlabeled public tourism videos has effectively captured cognitive fluency, revealing the statistical relationships between different concepts in tourism scenes. This approach eliminates potential bias in tour preference labeling and trains the model to extract only relevant contextual information.
These personalized design considerations of FocusFormer have enabled the VIPTour system to successfully model the desired cognitive fluency, thereby improving the tourism experience for visually impaired individuals.
It is worth noting that VIPTour’s impact depends on the quality of the underlying AI techniques, such as object detection and semantic graph generation. Future improvements in these methods could further enhance the system’s performance.
Download your PDF copy now!
Journal reference:
