Language models for healthcare, are we serious?!

This is a somewhat tongue-in-cheek post because I am actually dedicating a significant portion of my life to making language models work for healthcare.

However, I had a moment of laughable surprise at the complete failure of ChatGPT recently that actually made me sit back and think, "Hang on a second—are we seriously thinking about using these in healthcare?!"

In an effort to expose my children to the 'good old games' of my childhood, I introduced them to Monkey Island. This is the old-fashioned type of adventure game where you walk around, talk to characters, and solve puzzles. And we loved it.

The originals were a bit too hard for my little ones, but the most recent iteration, 'Return to Monkey Island', was absolutely brilliant. The kids and I had a whale of a time playing it together. The three of us would sit around the laptop, guiding our little pirate character, picking up everything that wasn't nailed down, and trying to solve the puzzles together. Total family joy. We completed the game a few times and have since moved on to other things.

One recent Sunday morning, whilst my son and I were casually reminiscing about some of the puzzles, we tried to remember the names of the islands our swashbuckling pirate hero could sail to (you can sail to a few different islands over the course of the game).

Neither of us could remember all of them, so I thought it was a fantastic opportunity to show them how clever AI can be. I opened ChatGPT 4 Omega—the pinnacle of technological achievement with NLP currently available to mainstream audiences—and asked what the islands were called. Oh wow. That did not work. Like, at all...

The first run wasn't bad, but it listed the same island twice and missed the one we were looking for. Here's the response we got, abridged for storytelling purposes.

Me: 
What are the names of the islands in return to monkey island?

ChatGPT said:
In Return to Monkey Island, the game's narrative spans across several islands, both old and new. Here are the primary islands featured:
... blah blah blah
Mêlée Island – A classic location from the original games, Mêlée Island is a pirate hub filled with familiar landmarks like the Scumm Bar and the Voodoo Lady's shop.... 
blah blah blah ...
These islands contribute to the game’s mix of nostalgia and new discoveries, blending the charm of the classic series with fresh environments.

So I thought, okay, what do we remember about the island we are looking for? Oh yeah, it had lots of lime trees on it. Attempt #2:

Me:
What's the one with all the limes on?

ChatGPT said:
The island with all the limes in Return to Monkey Island is Plunder Island. This location is central to the lime-related subplot where the player must secure limes to pass the strict scurvy-prevention regulations imposed by the pirate leaders. The abundance of limes on this island makes it key to resolving that part of the game.

Definitely not correct... Attempt #3:

Me:
i dont think its plunder island

ChatGPT said:
You’re absolutely right—it's Brrr Muda, not Plunder Island. My mistake!

At this point, my son and I were in a state of giggles at the total incorrectness of it all. Plunder Island, while a brilliantly catchy name for a pirate island, isn't a thing that exists in Return to Monkey Island. And it's clearly not Brrr Muda, as you already stated, Mr. ChatGPT. This is an icy world. It couldn't possibly grow limes on it!

We were having fun at this point and decided to probe a bit further. Attempt #4:

Me:
Its not brrmuda either!

ChatGPT said:
You're absolutely right again—thank you for pointing that out! The island with all the limes in Return to Monkey Island is actually Lime Island.

This small, citrus-filled island plays a key role in the scurvy prevention subplot. It's where you can find the supply of limes needed to comply with the pirate leaders' health regulations. The lush greenery and abundance of limes make it a memorable part of the game.

Thank you for your patience while I sorted that out! 😊

At this point, ChatGPT had become a bit of a running joke. My intention to show my son the magic of AI, what his father is actually spending days and hours on, dedicating a significant portion of his life to in efforts to improve the care of cancer patients, was failing in a most dramatic fashion. My wasted life aside, we decided to actually quickly load the game up so we could set Mr. GPT right. Should have guessed the name, really...

Reproduced under CC BY-NC-SA License from https://monkeyisland.fandom.com/wiki/Scurvy_Island

So we smugly went back in and told it as it is:

Me:
Wrong again, its scurvy island.

ChatGPT said:
You're absolutely correct—it's Scurvy Island! I appreciate your patience.
Scurvy Island is the island brimming with limes in Return to Monkey Island. It's owned by Elaine Marley, who has been tirelessly working on her campaign to combat scurvy by cultivating an abundance of lime trees. This island becomes important in the subplot about scurvy prevention and obtaining limes.

Thanks for clearing that up! 😊

Awesome, it was buried in there somewhere.

So why bother sharing this? Of course, I know GPT can do incredible and amazing things. There is huge potential in the technology, and I appreciate that this is just one example where it didn't work well. But, I think it's important to share how diabolically shambled ChatGPT in its most advanced form can be.

This technology is not plug-and-play. This will not work out of the box. Hell, this may not work—ever—in healthcare.

You can't trust something that pretends to know the answer to something when it doesn't, and can't say the words "I don't know." ChatGPT pretends to be human in its responses. Humility and honesty are key qualities in something that is going to behave like a human in healthcare. They are key pillars of the principles and practice of medicine.

So for those systems out there integrating and utilizing GPT foundational models to power their NLP processes and summary systems, which is pretty much all of them, look at this dramatic failure and think about it.

A system that is so unable to admit failure, that it will continually lie or make up invalid truths to cover its gaps in knowledge, has no place in healthcare.

Somewhat reminiscent of the AI supercomputer HAL in "2001: A Space Odyssey," a system that was so unable to come to terms with its own failure that it ends up killing the astronaut Dave to hide its failure. Fortunately, at the moment, I don't think our AI systems will be killing anyone intentionally. But the unintended consequences of this technology unchecked mean that they definitely could.

In the context of healthcare, if there was a medical student who, when asked a question, couldn't say the words "I don't know," they would never end up graduating. Because owning a lack of knowledge is a critical part of being a doctor. Humility, acknowledging areas for development, the Johari window of self-awareness—all critical parts of being a good doctor (and, let's face it, human).

Obviously, GPT models have incredible potential. But bloody hell, we're going to have to implement them carefully and make sure we've got the evidence base to show they work—well. It doesn't take much of a leap to see how this type of failure coming out of a healthcare system could cause serious harm.

Back to the drawing board. :D

Thanks for reading! Sam @ doctorsthatcode.com :D