A new study by The Permanente Medical Group (part of Kaiser Permanente) offers confidence that ambient voice transcriptions can save time and improve the patient experience by generating clinical notes about patient visits. This article summarizes the key findings in that study.

What We’re Looking For in Ambient Voice Transcription

Ambient voice transcription has been covered extensively on this web site in articles and video interviews. It’s an application of modern generative AI techniques to relieve doctors of a major administrative burden and demand on their time. The software records what the doctor and patient say, then uses generative AI to produce a note in the proper format.

The doctor, of course, has to edit the note for accuracy. Many vendors claim a high accuracy rate. But one always has to check results for irrelevant and false information (hallucinations) and omissions.

So clinicians will have many questions about ambient voice transcriptions. Are they risky? Do they improve or degrade the physician/patient interaction? Do they actually save time, and do they do so enough to justify their cost? This study provides encouraging data on some of these questions.

Evaluating This Study

TPMG, as most readers know, is the largest medical group in the United States, with nearly 9,800 physicians and 44,000 nurses and other staff. After deploying the Nabla ambient voice solution in October 2023, the organization quickly built up a large database of AI-generated clinical notes to run through their study.

Note that this study was carried out by the team at TPMG who installed the ambient voice solution. A study by an independent organization would be more authoritative.  But results were strong enough for us to pay attention.

To evaluate AI-generated notes, researchers used the standard PDQI-9 metric. Note that this metric has been criticized for inconsistent results and inaccuracy. Furthermore, the researchers modified the PDQI-9 by removing one domain or scale, and adding two others. It’s hard to tell how the changes might reflect the measure’s accuracy.

Still, results were impressive enough in the accuracy of transcriptions to make the study worthwhile. For instance, transcripts achieved, on average, 48 out of a possible 50 points on the “free from bias,” “synthesized,” “internally consistent,” and “succinct” domains.

The researchers added surveys to verify anecdotal evidence that patients like the AI solution. Although that survey had a small number (21) of respondents, it’s good to know that 71% of patients “reported they spent more time speaking with their physician” and 81% “reported that their physician spent less time looking at the computer screen than in their previous visits.”

The EHR also provided useful statistics, because the organization could track how much time doctors logged into it. Analysis showed that physicians who used the AI note generator spent less time in the EHR, especially between 7:00 PM and 7:00 AM. This statistic illustrates how digitization can facilitate the collection of useful data (although at the cost of revealing data on individual users). But I couldn’t tell from the article whether the time decrease was enough to make a big difference in the lives of the doctors.  That’s something to consider analyzing more as they continue to study the benefits of ambient clinical voice.

To wrap up, we can ask whether this study adequately backed up claims in the article that an AI-based, ambient voice transcription service can “reduce documentation burden” and “enhance physician–patient encounters.”  No doubt we’re just at the start of evaluating the impact of ambient clinical voice in healthcare and more studies are on their way to really understand these two impacts.

It says something that TMPG has almost 1000 physicians using ambient voice for more than 100 patient encounters in only 10 weeks since the initial implementation.  Usage may be the most important number to watch when it comes to the impact of ambient clinical voice, since doctors aren’t likely to use this technology if they don’t enjoy using it and find it beneficial.

I recommend reading the article, which has other interesting observations on how to facilitate adoption of new technologies.