Startup uses AI chatbot to provide mental health advice, then realizes it’s ‘weird’

A mental health nonprofit is being criticized for using an AI chatbot as an “experiment” to provide support to people seeking advice and for experimenting with the technology on real people.

“We provided mental health support to around 4,000 people – using GPT-3. Here’s what happened,” Rob Morris, co-founder of mental health nonprofit Koko, tweeted Friday. “Messages composed by AI (and supervised by humans) were rated significantly higher than those written by humans themselves (p<0.001). Response times decreased by 50%, to well under 'one minute... [but] once people learned that the messages were co-created by a machine, it didn't work. The simulated empathy feels weird, empty. Morris, who is a former Airbnb data scientist, noted that AI was used in more than 30,000 posts.

In his video demo which he posted in a follow-up Tweet, Morris is shown engaging with the Koko bot on Discord, where he asks GPT-3 to respond to a negative message someone wrote about himself in trouble. “We are making it easier to help others and with GPT-3 we are making it even easier to be efficient and effective as a provider of help. … This is a very short article, and yet AI alone in a few seconds wrote a really nice articulate response here,” Morris said in the video.

In the same Tweet thread, Morris said AI-composed posts scored significantly higher than human-written ones, and response rates dropped by 50% with the help of AI. Yet, he said, when people learned that the posts were written with an AI, they felt disturbed by the “simulated empathy”.

Koko uses Discord to provide peer-to-peer support for people in mental health crisis and those seeking advice. The whole process is guided by a chatbot and is rather clunky. In a test conducted by Motherboard, a chatbot asks you if you are looking for help with “Dating, Friendships, Work, School, Family, Eating Disorders, LGBTQ+, Discrimination or Other”, asks you to write what is your problem, mark your “most negative thought” about the problem, then send this information to someone else on the Koko platform.

In the meantime, you are asked to help others going through a crisis; in our test, we were asked to choose from four responses to a person who said they had trouble loving themselves: “You are NOT a loser; I’ve been there; Sorry to hear that :(; Other ” and personalize the message with a few additional phrases.

On Discord, Koko promises that it “connects you with real people who really understand you. Not therapists, not counselors, just people like you.”

Ethicists, experts and AI users seemed alarmed by Morris’ experiment.

“While it’s hard to judge the merits of an experiment based on a tweet thread, there were a few red flags that stood out to me: leading with large numbers without context in advance , running the “experiment” through a peer support app with no mention of a consent process or ethical review, and insinuating that people don’t like a chatbot in their mental health care was something something new and surprising, Elizabeth Marquissenior UX researcher at MathWorks and PhD student at the University of Michigan, Motherboard told.

Emily M. Bender, a linguistics professor at the University of Washington, told Motherboard that there is great potential for harm in relying on AI to treat mental health patients. “Large language models are programs that can generate plausible-sounding text given their training data and an input prompt. They have no empathy, no understanding of the language they produce, and no understanding of the situation they find themselves in. But the text they produce sounds plausible and people are therefore likely to assign meaning to it. To throw something like this into sensitive situations is to take unknown risks. A key question to ask is: who is responsible if the AI ​​makes harmful suggestions? In this context, does the company deploying the experiment impose all the responsibility on the members of the community who choose the AI ​​system? »

After the initial backlash, Morris posted updates on Twitter and told Motherboard, “Users were actually told that the posts were co-written by humans and machines from the start. The message they received said “written in collaboration with kokobot”, which they could decide whether to read or not. Koko users correspond with our bot all the time and they were introduced to this concept during the integration.

Screenshot 2023-01-09 at 6.18.39 PM.png

“It seems people misinterpreted that line: ‘when they realized the posts were a bot…'” Morris said. had been co-written by humans and machines from the start. The message they received said “written in collaboration with kokobot”, which they could decide whether to read or not. Koko users correspond with our bot and they were introduced to this concept during onboarding.

“They rated these (AI/human) posts more favorably than those written solely by humans. However, and here’s the nuance: as you start to pick up the flavor of these posts over time (at least for me), you may start to see ones that haven’t been edited by the provider. ‘assistance. You start seeing what appears to be from the bot only, unfiltered. It changes the dynamic in my opinion,” he added.

Morris also told Motherboard and tweeted that this experiment is exempt from informed consent, which would require the company to provide each participant with a written document regarding the possible risks and benefits of the experiment, in order to decide whether they wish to participate. He claimed that Koko did not use any personal information and did not intend to publish the study publicly, which would dispense the experiment from needing informed consent. This suggests that the experiment did not receive any formal approval process and was not overseen by an Institutional Review Board (IRB), which is required for all research experiments involving human subjects and animals. access to personally identifiable information.

“Each individual must provide consent when using the service. If this were an academic study (which it is not), it would fall under an ‘exempt’ category of research,” a he declared. “It imposed no additional risk on users, no deception, and we do not collect any personally identifiable information or protected health information (no email, phone number, IP address, name of ‘user, etc.). In fact, previous research we have done along these lines, but with more complexity, was exempt.”

“This experience highlights a series of overlapping ethical issues. The study does not appear to have been reviewed by an institutional review board, and deception of potentially vulnerable people should always raise red flags in research,” Luke Stark, the assistant professor in the Faculty of Information and Media Studies (FIMS) at Western University in London, Ont., told Motherboard. “That the system is successful in formulating routine answers to mental health issues is not surprising when we realize that it relies on many such answers formulated in the past by therapists and counselors and available on the web. It is unethical to mislead research participants for no good reason, whether or not using prompts provided by a natural language model.

“Anything presented as mental health support is clearly context sensitive and not one to be experienced without careful ethical review, informed consent, etc.” Bender told Motherboard. “Whether [experiments] must be carried out, a clear research question must be explored and an ethical review of the study before its initiation, in accordance with well-established principles of protection of human subjects. These review processes balance the benefits to society against the potential harm to research subjects.

Both Bender and Marquis agree that if AI were to be used for psychological purposes, affected communities, people with lived mental health experiences, community advocates, and mental health experts must be key players in the process. development process, rather than just anonymous users or data subjects.

For Morris, Koko’s primary goal is to create more accessible and affordable mental health services for the underserved. “We pulled the feature anyway and I wanted to unravel the concern as a thought piece, to help reign in the excitement about replacing therapists with gpt3,” he told Motherboard.

“I think everyone wants to help. It seems that people have identified insufficient mental health care resources as a problem, but rather than working to increase resources (more funding for training and hiring mental health workers), technologists want to find a shortcut. And because GPT-3 and its ilk can produce plausible text on any topic, they may look like a solution,” Bender said.

“From the real need for more accessible mental health resources to AI being a relatively cheap and scalable way to make money in the right application, there are myriad reasons why researchers and practitioners of AI might want to use AI for psychological purposes,” Marquis said. “Computer science curricula are just beginning to teach ethics, and human-centered AI is a relatively new field of study, so some might not see the warnings until too late. Perhaps less charitably , it can be handy to ignore the warnings about AI non-personification when you’re in a field that values ​​efficiency and advancement above all else.

Comments are closed.