dana blogs math: Warrior for the Blind

"Warrior 2" is an example of a yoga pose that blind people can now learn with Kyle Rector's yoga game. Model: Katie Thomas. Photographer: Karlie Sherman Photography. Used with permission.

Why should sighted people have all the fun? Kyle Rector, one of the young participants in this year’s Heidelberg Laureate Forum, doesn’t think they should.

As part of her dissertation research on “eyes-free technology” at the University of Washington, with professors Julie Kientz and Richard Ladner, Rector has written a computer program that enables blind people to learn yoga from a computer. A Microsoft Kinect “watches” the movements of a blind yoga student performing, say, a Warrior 2 pose. If the student makes a mistake, the computer program provides verbal corrections. If she does the pose right, the computer tells her that, too.

Rector’s project is a perfect convergence of three of her biggest interests: exercise, disabilities, and computer science. She loves exercising: she rides a bike, she has run a marathon (although it caused her to injure her knee), and yes, she does yoga. “I’m not a yoga instructor, though,” she says.

She also has a cousin who is nearly deaf, so she grew up very aware of the challenges faced by people with disabilities. “When I came to visit UW as a potential graduate school, I was excited to find out that Richard Ladner, one of my advisors, does research on accessibility,” she says.

Finally, Rector’s interest in computer science is, of course, the passion that has brought her to the Heidelberg Laureate Forum. The event will give her a chance to network with people whom she would otherwise have known only as a name in a textbook. “I still can’t believe that I’m going to be around that many Turing Award winners and Abel Prize winners,” she says. “It’s still a shock.”

Curiously, Rector’s career in computer science almost didn’t happen. If it hadn’t been for her mother’s sharp eyes, as well as the patient mentoring of her undergraduate advisor, she might have chosen a different path.

About ten years ago, when Rector was 17, her mother brought home some information about a summer program at MIT called the Women in Technology program. Rector thought she had no chance of getting accepted, so she threw the application into the recycling bin. “My mom found it there, fished it out and said, ‘You’re going to apply,’” Rector says. “So I applied and got in. I couldn’t believe it.”

In the month-long program at MIT, she took courses in math, electrical engineering and computer science. “I actually despised computer science the first time around,” she says. The reason was that most of her fellow campers already knew how to program computers, and she didn’t. “When you have these people writing circles around you, you feel as if you’re stupid,” she says. So when she went to college at Oregon State University, she had every intention of majoring in electrical engineering, not computer science.

But at Oregon State she met Margaret Burnett, a computer science professor who specializes in human-computer interaction and who also has a strong interest in the connection between gender and technology. Burnett was her adviser for a project on the differences between the ways that males and females use Microsoft Excel. Gradually Rector began to warm up to computer science.

“It took her two years to persuade me to add a computer science major,” she says. “I finally realized that if I like the research, and the classes are getting more enjoyable, and now I’m starting to love the programming, then I probably just like computer science. I also saw that there was a bigger meaning to computer science. All of these struggles were worth something because you’re actually building something useful that matters to people.”

So far the yoga game is the most ambitious “something that matters” that she has built. It’s still a work in progress. Most notably, the game doesn’t actually have a “start” command yet, because she has been too busy getting the core of the program to work. But what she’s already accomplished in a short time (from the fall of 2012 to the spring of 2013) is quite impressive.

Perhaps the simplest part is teaching the computer to recognize when a student isn’t doing a pose right. As it turns out, this part is very mathematical; in fact, it’s high-school geometry. Consider a typical pose, such as the Warrior 2 pose. You are supposed to extend your arms straight out to the sides, pivot one foot by 90 degrees and the other by 45 degrees, and bend one knee so that it forms a right angle, with the knee directly above the ankle. All of these concepts have to do with angles.

The Kinect has two cameras that can identify the spatial coordinates of each one of 20 joints in the human body. Rector’s program then calculates the angles at each joint, using the law of cosines. In the Warrior 2 pose, for example, the arms are supposed to form a right angle with the torso. If the Kinect sees them forming an angle of less than 80 degrees or greater than 100 degrees, then Rector’s software will alert the student to raise or lower her arms until the angle falls within the targeted range.

As it turned out, the mathematics was the easy part. The next step was teaching the computer how to explain what a student was doing wrong. She asked five yoga instructors for tips on how they communicated with students, and experimented with different wording. “I learned that metaphors were really helpful,” she says. “For example, I tell the students to stretch their arms to the side like a tightrope walker, or keep their feet parallel as if they were on skis.” Sometimes she was surprised by what worked and what didn’t. With blind people, telling them to reach toward the sky was not very effective. They weren’t used to thinking that way. But telling them to reach up as if they were getting something off of a high shelf—that immediately clicked.

After programming in the verbal instructions, she tested the yoga game herself, deliberately making every possible mistake. This turned out to be a not entirely smart thing to do, because she aggravated her sore knee in the process. But it did give her the confidence to test the program on other volunteers, only three months after she began the project. “By the time they tried it, it was safe,” she reassures me.

There was one more problem: Who would try it? “It’s not so easy to find people who have low vision, who are able to commute to where you are, and who are also interested in performing yoga for maybe the first time,” she says. She finally lined up 16 volunteers, for which she had to travel around the state of Washington. “It was exhausting, because each study took an hour, and there was one day when I ran seven participants,” she said.

But the results were very gratifying. Thirteen of the 16 students liked the “exergame” and many of them wanted to know when they could buy it! That surprised Rector, who (just for the record) has no plans yet to make the yoga game a commercial project. She will, in all likelihood, make the yoga program available for free to the volunteers and to the yoga instructors who helped her out, once it is stable and user-friendly enough. She’s not interested in selling it because software products for blind people tend to be very expensive, and she doesn’t want to add to their financial burden.

I have a hunch, though. If she really wanted to make money—which she clearly doesn’t—there’s another clientele that might find her product useful: yoga students with normal eyesight. After all, how are you supposed to know if you’re doing the poses right when you’re practicing at home, far from the instructor’s benevolent gaze?

For the time being, Rector is more interested in living the life of an academic than that of an entrepreneur: traveling to conferences, collaborating with and learning from interesting people. This spring she went to conferences in Paris and San Francisco. She spent the summer doing a research internship at HP Labs in Palo Alto. After the Heidelberg Laureate Forum, her whirlwind schedule will continue with a trip to the Grace Hopper Celebration of Women in Computing in Minneapolis, where she will present her yoga research. Clearly, for this long-distance runner and long-distance traveler, slowing down is not an option!

Click below to watch Kyle Rector’s YouTube video demonstrating the yoga “exergame.”

This blog post originates from the official blog of the 1st Heidelberg Laureate Forum (HLF) which takes place in Heidelberg, Germany, September 22 – 27, 2013. 40 Abel, Fields, and Turing Laureates will gather to meet a select group of 200 young researchers.

Dana Mackenzie is a member of the HLF blog team. Please find all his postings on the HLF blog.