University of Minnesota Duluth computer science professor, Arshia Khan and her students are programming humanoid robots to recognize people and communicate with them on subjects that match their interests.

As far as Khan's aware, no one else has programmed Pepper, the world's first social humanoid robot designed by Japan-based SoftBank Robots, to act as what she calls "COVID gossip bots." It's all in the name of helping the elderly, especially during the pandemic when many are experiencing even higher levels of isolation.

"We are using the concept that socialization often happens with some sort of gossip," she said. "We want to take that gossip element and use it in a positive manner. So what we're doing is we are creating an imaginary persona that is very similar to the person we are working with."

Ideally, each of the three humanoid robots Khan and her students are programming will be able to embody several different personalities to match the person they're communicating with.

Newsletter signup for email alerts

Not only can the robots talk and read facial expressions, they can also offer medication and monitor the vitals of people wearing sensors. The data the sensors pick up — blood pressure levels, movement, the heart's electrical signal and physiological changes like stress — is sent to a "very fast" Nvidia processor that's locked away in a room on the UMD campus.

"Tons of data is coming into this hub," Khan said "That's where the data is coming in and being analyzed and processed."

The data is also password protected to prevent access to anyone's private information. Additionally, all identifying elements have been removed from the data.

Prior to the pandemic, Khan said, there was more resistance to robotics and the role they can play in enhancing health care. She hopes that when the virus passes the acceptance of robots remains.

"Because of science fiction, people think robots can do a lot more than they can really do. And so there's that fear that they're going to hurt somebody or try to take over the world, or even concerns about ethics," Khan said. "Technology is not that advanced that the robot can behave like a human. We are far from that. Whatever we programmed it to do, it can do. It cannot do more than that."

Graduate studies professor at UMD's Swenson College of Science and Engineering Arshia Kahn plays a demonstration on a robot called Pepper on Wednesday, Oct. 28. (Tyler Schank / tschank@duluthnews.com)
Graduate studies professor at UMD's Swenson College of Science and Engineering Arshia Kahn plays a demonstration on a robot called Pepper on Wednesday, Oct. 28. (Tyler Schank / tschank@duluthnews.com)

Collaborating across disciplines

In all of her projects, Khan brings in people from other disciplines to consult with, including a philosophy professor who collaborates with Khan on classroom discussion surrounding ethics as well as professionals with the U of M Medical School and health experts at Essentia Health and St. Luke's in Duluth.

"This work cannot be done in a silo," Khan said. "It has to be done with people from a lot of different disciplines."

Katie Benziger, a cardiologist at Essentia, acts as a medical consultant to Khan's research. Her role in the robotics research is to make sure the artificial intelligence is correctly monitoring vital signs and diagnosing high blood pressure.

She's optimistic about the future of robotics in medicine and she stressed that she doesn't foresee robotics replacing people, but rather aiding and enhancing what health care workers can do.

"I think some people probably just feel like the patient-physician relationship is so coveted and they don't want anything to disrupt that," Benziger said. "But the problem is there are not enough physicians for the number of patients who are aging. Baby boomers are getting older and ... especially in these rural areas, there's not enough specialists. There's not enough primary care docs to see everybody."

Robots are commonly used in surgeries because they can do complex maneuvers in small spaces, therefore stepping in where the human hand cannot and increasing surgical precision, all while a surgeon oversees every step of the way, Benziger said.

"What's unique about (Khan's) robots is how they're programmed to learn. A lot of robots have a skillset and that's just what they do. They're not able to adapt," she said. "With the conversation that (the robots) have with the person, they're able to learn the person's name or their grandchildren's name or where they live and be able to continue that as a conversation as if it were a real person."

Computer science student Reilly Moberg demonstrates how Pepper can act as nurse triage Wednesday, Oct. 28, at the University of Minnesota Duluth. (Tyler Schank / tschank@duluthnews.com)
Computer science student Reilly Moberg demonstrates how Pepper can act as nurse triage Wednesday, Oct. 28, at the University of Minnesota Duluth. (Tyler Schank / tschank@duluthnews.com)

Dementia friendly

The COVID bots are part of a larger project Khan's working on called the "dementia-friendly living space," which uses sensors placed around the room to help improve the quality of life for those living with dementia.

Using artificial intelligence, Khan said, there are many ways in which robotics can be used to monitor and prevent symptoms of dementia. For example, sensors in a room can track when someone starts to wander away.

They can also engage those affected with dementia in a common practice of treatment called reminiscence therapy, which uses the various senses to help people remember events, people and places from their lives.

"That is really difficult nowadays during the pandemic, but the robot can do the same job without bringing in the concern of infection," Khan said. "Human beings can get tired of doing that over a period of time, but the robot will never get tired. It can keep repeating the same thing over and over."

Pepper, a robot designed to help those in assisted living homes, raises her arms as she says, "Go Bulldogs," Wednesday, Oct. 28, during a demonstration at UMD. (Tyler Schank / tschank@duluthnews.com)
Pepper, a robot designed to help those in assisted living homes, raises her arms as she says, "Go Bulldogs," Wednesday, Oct. 28, during a demonstration at UMD. (Tyler Schank / tschank@duluthnews.com)


Approval process

Since Khan and her students are still in the process of programming the robots, they are not currently in use anywhere. She does have a couple dementia facilities in Duluth and Superior interested in using the robots in their assisted living spaces when they're ready.

Both facility staff and the families of the residents needed to be willing to accept the robots into the center.

Before that can happen Khan needs to receive a few different approvals to use the robots in real-life settings for research. First, the University of Minnesota's Office of the Vice President of Research needs to approve the research. That approval process, which can take months, includes ensuring that research participants will be protected and ethical standards are upheld.

The socializing robots will not, for example, have the ability to take information from person to person. However, the robots can gossip about the imaginary personas of the other robots, such as Baxter, a larger robot that can help people get out of bed after surgery.

"Now that would be a disaster," said Khan, imagining a situation in which robot Pepper could spread real, human gossip around an assisted living facility.