How AI meeting notes work: from audio to summary

An AI note taker does four jobs in a row. It records the audio, turns speech into text, works out who said what, and then writes a summary. Each job is a separate step, and a mistake in one step carries into the next.

What you're dealing with

Before the how, a few terms. You'll meet them in the settings of nearly every tool.

• Transcript: the full written text of what was said, word for word.
• Speech recognition: the software step that makes the transcript. It's also called speech-to-text.
• Diarisation: the step that labels who spoke when. You'll also see it spelled with a z.
• Summary: a shorter text written from the transcript.
• Minutes: the official record of what was decided. A summary isn't the same thing, and turning a transcript into minutes is its own job.

Step one: capturing the audio

The tool gets sound from your meeting. That might be a call platform, a phone app, or a microphone in the room. Whatever the source, the quality of this audio sets the ceiling for everything after it. What you can do about it is in what to set up before the meeting.

Step two: speech becomes text

Speech recognition is described as a field concerned with methods that "translate spoken language into text or other interpretable forms"¹. In a meeting tool, that means your audio goes in and a block of text comes out.

This step struggles with the same things a human listener does. One source notes that speech is distorted by background noise, echoes, and recording characteristics¹. It also says that when people speak spontaneously, recognition must deal with disfluencies such as "uh" and "um", false starts and laughter¹. Meetings are full of all of that.

So a wrong word in the transcript usually isn't the software being careless. It's the audio being hard.

Step three: working out who said what

A transcript with no names is hard to use. Diarisation fixes that by splitting the audio into turns for each speaker. One definition calls it the process of partitioning an audio stream into segments according to the identity of each speaker².

The same source says it combines two jobs: finding the points where the speaker changes, and grouping together the segments that sound like the same person². Notice what that doesn't say. Grouping by voice gives you "Speaker A" and "Speaker B". Diarisation can be combined with speaker recognition systems to give a speaker's true identity, and how a given tool assigns names varies.

If two people talk over each other, or two voices sound alike, the labels can swap. That matters when you later assign tasks.

Step four: the summary

Most tools hand the transcript to a language model. A language model is software trained on large amounts of text so it can write new text in response to a prompt.

Summarising comes in two broad styles. An extractive summary pulls sentences straight out of the original. An abstractive one writes new wording. The abstractive style is described as generating new text that did not exist in the original text³. Check how your own tool describes its summaries.

Here's the catch. Writing new wording means the model can also write something that wasn't said. Chatbots built on language models may embed plausible-sounding random falsehoods within their generated content⁴. In meeting notes, that could be a wrong date, an invented decision, or a task given to the wrong person.

What to do with this

• Treat the summary as a draft. Read it against what you remember, and against the transcript for anything that matters.
• Check names, numbers, dates and decisions first. Those are the details that cause trouble when they're wrong.
• Keep the transcript until the notes are checked. Where recordings and transcripts are stored covers what happens to it.
• Tell people before you record. The consent basics point you to the rules.

If a step goes wrong, fix it at that step. A poor transcript can't be saved by a better summary prompt.