Archived - Helios Headgear Uses Intel® RealSense™ Technology to Empower the Visually Impaired

The Intel® RealSense™ SDK has been discontinued. No ongoing support or updates will be available.

Our Work and Motivation

The HELIOS* project is focused on enhancing and complementing human sensory functions with cutting-edge vision technologies.

A study published by the World Health Organization reveals an estimated 285 million visually impaired people worldwide: 39 million blind and 246 million with low vision.

We believe it is very important to improve mobility, safety, and access to knowledge for people with sight deficiencies.

Using Computer Vision, Artificial Intelligence, and Intel® RealSense™ technology, we are working on innovative solutions to help visually impaired individuals overcome several challenges they face on a daily basis. Our approach centers on the development of smart headgear to assist with partial or complete vision loss.

HELIOS at IDF16 San Francisco

HELIOS Headgear Models and Features

The HELIOS headgear provides a series of accessibility features for visually impaired individuals, empowering them to perform actions and tasks with more ease and confidence.

HELIOS Light

HELIOS Light addresses individuals suffering from low vision, using AR/VR technology to enhance the user’s level of visual perception, capitalizing on the Intel RealSense RGB and depth data streams to provide an adaptable vision aid for performing a variety of daily tasks.

HELIOS Light 3D representation
HELIOS Light 3D representation

HELIOS Touch

HELIOS Touch addresses individuals suffering from severe or complete blindness. It uses our HTI interface to translate visual data to the user via haptic signals, granting nearby environment localization and obstacle avoidance capabilities.

HELIOS Touch 3D representation
HELIOS Touch 3D representation

A key feature of HELIOS is to give the user better awareness of the nearby environment, substantially improving freedom of movement and safety.

Understanding non-Braille text is another important functionality. HELIOS can read the content of books, magazines, or other printed material, such as restaurant menus.

Furthermore, HELIOS delivers a new layer of context for interpersonal interaction by recognizing friendly faces and social cues.

Hardware breakdown

Intel® RealSense™ Technology

Intel® RealSense™ cameras have RGB-D capability and versatile sensors that provide HELIOS with high-quality depth and RGB streams. Their features, performance, and small form factor make them optimal for integration into the HELIOS headgear.

Intel® RealSense™ camera R200. Learn more about it in this article
Intel® RealSense™ camera R200. Learn more about it in this article.

The Razer* Stargazer, which is a third-party version of the Intel® RealSense™ camera SR300.
The Razer* Stargazer, which is a third-party version of the Intel® RealSense™ camera SR300.

HTI* Haptic interface

HTI is a hardware component of HELIOS Touch, developed by our team. It is designed to translate visual data into haptic feedback, providing the user with an extra layer of information, in a precise, non-intrusive form.

HTI test board
HTI test board

Open Source Virtual Reality

The Razer OSVR Hacker Development Kit is a highly customizable platform for Virtual and Augmented Reality. It is an ideal off-the-shelf component for HELIOS Light given its open source, extendable nature, and compelling hardware design.

Razer OSVR HDK
Razer OSVR HDK

Intel Next Unit of Computing

Intel’s latest generation of small form factor PCs provides a robust platform for running HELIOS software components in real-time, with emphasis on performance, power efficiency, and portability.

Intel® NUC
Intel® NUC

Software. Intel® RealSense™ SDK

The Intel RealSense SDK is a central piece of HELIOS's software components. Out-of-the-box it facilitates access to high-frame-rate RGB, and depth and IR streams and provides a comprehensive set of computer vision algorithms for tasks such as person tracking, facial recognition, or 3D mapping. The SDK ships with a great set of sample projects and extensive online documentation

The following code sample reveals the key components for developing a Text-to-Speech module with RealSense and UWP (Universal Windows Platform):

        public async void StartRealSenseStreaming()

        {

            Status streamingStatus;



            // Set RealSense sample reader and bind SetOcrFrame event

            SampleReader sampleReader = SampleReader.Activate(senseManager);

            sampleReader.SampleArrived += SetOcrFrame;



            // Set RGB stream profile and device info filter

            Dictionary<StreamType, PerceptionVideoProfile> profiles = new Dictionary<StreamType, PerceptionVideoProfile>();

            profiles[StreamType.STREAM_TYPE_COLOR] = ColorProfile;

            sampleReader.EnableStreams(profiles);

            readers.Add(sampleReader);

            if (currentRealSenseDevice != null)

                senseManager.CaptureManager.FilterByDeviceInfo(currentRealSenseDevice.DeviceInfo);



            // Set streaming status message

            if ((streamingStatus = await senseManager.InitAsync()) == Intel.RealSense.Status.STATUS_NO_ERROR)

            {

                if ((streamingStatus = senseManager.StreamFrames()) == Intel.RealSense.Status.STATUS_NO_ERROR)

                {

                    StatusMessage = "Streaming started";

                }

                else

                {

                    StatusMessage = "Failed to stream: " + streamingStatus.ToString();

                }

            }

            else

            {

                StatusMessage = "Initialization failed: " + streamingStatus.ToString();

            }



            IsStreaming = true;

        }



        private void SetOcrFrame(Object module, SampleArrivedEventArgs args)

        {

            // Setting current frame for OCR processing

            Sample sample = args.Sample;

            if (sample == null) return;



            var localOcrFrame = sample.Color;

            if (localOcrFrame == null) return;



            lock (sample)

            {

                ocrFrame = localOcrFrame.SoftwareBitmap;

            }

        }
       

        private async void TextToSpeech()

        {

            // setup OCR engine for English

            OcrEngine ocrEngine = OcrEngine.TryCreateFromLanguage(new Language("en"));



            // recognize text from the RealSense OcrFrame

            var ocrResult = await ocrEngine.RecognizeAsync(RealSense.OcrFrame);



            if (!String.IsNullOrEmpty(ocrResult.Text))

            {

                // setup speech synthesizer

                var voice = SpeechSynthesizer.AllVoices;

                using (SpeechSynthesizer speechSynthesizer = new SpeechSynthesizer())

                {

                    speechSynthesizer.Voice = voice.First(v => v.Gender == 0);

                    var voiceStream = await speechSynthesizer.SynthesizeTextToStreamAsync(ocrResult.Text);



                    // setup playback of voice synthesis

                    PlaybackVoice(voiceStream);

                }

            }

        }

Testing and validation

Mihai Leoveanu was born with severe sight deficiency, but that didn’t stop him from becoming the outstanding person he is today.

Mihai has a strong can-do attitude and is one of the best students in his graduating class. He is currently working on his Master’s Thesis that proposes a series of Accessibility improvements for the Targoviste Royal Court historical site. These improvements would allow visually impaired tourists to have a richer visiting experience.

Mihai is the first person to review the capabilities of our headgear.

Mihai field testing HELIOS
Mihai field testing HELIOS

Mihai reading with HELIOS
Mihai reading with HELIOS

During the experimentation process, Mihai provided observations for each feature of HELIOS he was using. He seamlessly adopted the new information sources, and in a matter of minutes he was successfully using the headgear to gain a more accurate description of his surroundings.

Conclusion

The results from development and testing are very positive. Functionalities like environment perception and non-Braille reading are much easier for the user. With further development, HELIOS has the potential to make a real difference for individuals with sight impairment, becoming a true complement to their senses.

About the Authors

Silviu-Tudor Serban, Cristian Dragomir, and Andrei Nistor are Intel RealSense technology experts with backgrounds in Computer Vision, Artificial Intelligence, Software Development, and IoT. Find out more at Helios Vision and Intel Devmesh.

Para obtener información más completa sobre las optimizaciones del compilador, consulte nuestro Aviso de optimización.