State of the Field

Using Unity, Oculus, and Leap Motion for Rehabilitation in Stroke Survivors

As the field of Extended Reality (XR) continues to rapidly develop, emerging technologies are creating opportunities for novel applications of Augmented Reality (AR) and Virtual Reality (VR). The use of XR in various scientific fields, such as the introduction of interactive surgical simulation in the training of medical professionals, indicates the great potential it has for the advancement of STEM. Recently, VR has become a popular area of investigation for inclusion in therapies. First, it has become a widely-used alternative to traditional treatments for PTSD. Through exposure therapy, veterans are subject to a virtual environment with their triggers and gradually become desensitized to the war and trauma situations that contributed to the development of their disorder. Similarly, in the past year, VR treatments have been developed for other traumas, phobias, and mental illnesses. The most popular VR treatments, other than for PTSD, tend to be treatments for arachnophobia, acrophobia, ornithophobia, and concrete phobias. As a student of cognitive neuroscience and computer science, I became particularly interested in using XR technology for treating neurological disorder.

Stroke is the leading cause of long-term disability in adults. The majority of stroke survivors experience upper extremity (UE) deficits, particularly stroke-induced hemiparesis, which significantly limits fine motor control. For 55–75% of stroke survivors with motor control impairments, UE deficits will continue well beyond the time of stroke. Fortunately, through experience-dependent neuroplasticity, it is possible to rehabilitate the damaged brain and ameliorate UE deficits. This project involved significant research on traditional or non-immersive stroke therapies, previous studies involving stroke survivors or motor control rehabilitation, and VR therapies in general. Traditional therapies involve repetition of various hand exercises, such as pinching of the fingers or flexing of the hand. While there are already various explorations of XR in the rehabilitation of stroke survivors, the majority are non-immersive, i.e. using 2D computer displays. In this final project, I sought to develop the prototype of a VR experience, which could serve to demonstrate the benefits of using immersive VR technology to exercise fine motor control. By offering various options for practicing motor skills from one centralized place, users have a more enjoyable experience doing repetitive hand exercises, which serves to increase the time spent doing therapy and the allocation of neural resources to the therapy. With more neural resources being allocated to the immersive experience and more time being invested in these more stimulating exercises, the therapy can actually become more effective in facilitating these motor skills. This makes VR therapy for hemiparesis a potentially more salient alternative to traditional therapies.

My project was developed using Unity, for use with Oculus Rift and LeapMotion. By mounting the LeapMotion on the Oculus headset, the program is able to track and render the user’s hands in real time, so that the user can actively interact with objects in the virtual environment with their hands.

First, I developed a central scene, the Welcome Scene, from which a user could select an exercise option and be transported to a new scene where they practice that skill. For the purposes of my project, I only needed the functionality of navigating between scenes from this central location. In a final prototype, I would want to store a username and user history (i.e. previous scores, high scores, time spent on therapy, etc.).

Next, I developed a scene for each skill. The first of these was Cube, a basic cube stacking scene, which I started with in order to explore how to build and structure such a scene. Here, there are three cube game objects, and the user can use their hands to pick up and stack the cubes, via pinching or grabbing. Pinching and grabbing are both typical of stroke therapies. In order to return home and try a new exercise, the user can turn to the right and touch the Home object, which will take them back to the Welcome Scene.

Then, after our discussion with a Professor of Dance at Brown University, I wanted to bring in more “embodied” action. He suggested doing something involving balance. Thus, I decided to create another new scene, called Balance, which requires the user to balance a ball on their hand. This requires dynamic flexing of the fingers, which is a popular and robust exercise in traditional therapies for stroke survivors. Once again, when the user is ready to try a new exercise, they can turn to the right and touch the Home object.

Finally, I wanted to include a more ecologically valid scene. That is, in cognitive neuroscience, certain environments are considered more representative of daily lived experience. Things that a person might actually do, such as brushing their teeth or picking up a fork to eat, are both more ecologically valid than actions carried out in lab environments, like selecting responses based on presented stimuli. My ecologically valid scene was the Darts scene, in which the user has an infinite supply of darts to throw at a dart board, using a pinching motion. This scene is the best example of the type of exercise I would want to include in a final product of this prototype. Because there are points associated with darts, I could include a reward system, which would award the user different point values based on their progress. Then, I could keep track of scores and maintain a high score leaderboard to provide positive reinforcement learning and encourage the user to play this game more often, benefitting the therapy. I did try developing this system, but it proved a much more difficult task than expected, so I decided to omit it for the purposes of my prototype. This scene, however, still demonstrates the power of using immersive VR relative to traditional therapies without such reward structures.

Above, you can find a link to a short video that follows the user’s view as they navigate the experience. They start at the Welcome Scene and navigate through each exercise scene, practicing each skill before moving onto the next. This would be the general workflow of this therapy.

There are several limitations to the project as of now. First, LeapMotion does tend to glitch, especially when it loses sight of the user’s hands, which can disrupt the experience, interfere with the sense of immersion, and frustrate the user. These are downsides in that they might lead to the therapy not being practiced as often by the subject, which would decrease its effectiveness. As the technologies involved advance and become more easily integrated over time, however, this will become less of a concern. Next, the actual implementation of a game, even as simple as throwing darts, is a complex process, and it will involve much more time and resources devoted to working on animation, story development, and user interactions. Because I am not an expert coder, I was not able to attain the level of detail I would want in a final product to be fully used by stroke survivors, but the prototype demonstrates the efficacy and potential of this structure for therapy.

In the future, I would want more time and resources to flesh out the exercises and make them more ecologically valid, i.e. more animation/more detailed environments and more exercises such as pinching a paintbrush to draw or playing a virtual keyboard. In addition, I would want to include scores. As mentioned earlier, points create a reward structure that reinforces behavior, so this would greatly increase the likeliness of the user practicing the exercises more often. It could also provide a measurement of their progress in the therapies over time. This would be a straightforward addition, although requiring some technical skill. Finally, I am hoping to get my PI into the lab, so she can check out my prototype as a Doctor in Cognitive Neuroscience. Unfortunately, she was unavailable for the past week due to a chaotic finals period and a brain research conference, but I would appreciate this guidance as my project is certainly one which depends on scientific principles.

Overall, I had a lot of fun thinking about, planning, and researching this project, and I am fascinated with the intersection of VR and cognitive neuroscience. There is great potential for exploring, entertaining, and manipulating the mind and brain in XR, and I am glad that these past two semesters have provided me with a strong foundation in the field. Can’t wait to see how this powerful field continues to develop in coming years!

Selected Citations:

LeBlanc S, Paquin K, Carr K, Horton S (2013) Non-immersive Virtual Reality for Fine Motor Rehabilitation of Functional Activities in Individuals with Chronic Stroke: A Review. Aging Sci 1: 105. doi:10.4172/2329–8847.1000105

Usoh, M., Catena, E., Arman, S. & Slater, M. Using presence questionnaires in reality. Presence-Teleoper. Virtual Environ. 9, 497–503 (2000).

Related posts:

Data science is easily one of the most sought-after skills in the market today. According to a CNBC article, ‘data scientist’ is among the top 10 professions of 2017 with respect to factors such as…

Kubernetes clusters have a lot of moving parts — and so does each application running on a cluster. With frequent application and environment updates, the state of every cluster can change rapidly…

I had never heard of Zimbalist Williams until I received a DM from one of his fans. It took 5 minutes of research for me to become a fan. If you give me five minutes of your time, I am going to try…