- AST turns sound into an image and recognises 527 kinds of sound from AudioSet.
- CLAP compares sound with a text description.
- On the Mac mini M6's graphics chip a minute of sound takes about 10 s, on the processor about 29 s.
- AST called a low impact a heartbeat, and CLAP matched 4 of 6 effects to the right description.
The eleventh part of the series is about models that listen. When I rate a reel, the sound tells me as much as the picture: is there music, am I speaking over it, does a cut have a transition effect, does the reel start with an impact. In my reel studio two models do this: AST, which recognises 527 kinds of sound from AudioSet, and CLAP, which compares sound with a text description. I test them on three of my own reels on a Mac mini M6 with 32 GB, once on the processor and once on the graphics chip.
How a model hears
AST, the Audio Spectrogram Transformer, does something surprisingly simple with sound. It turns it into a spectrogram, an image with time on the horizontal axis, pitch on the vertical axis and brightness as loudness. Then it looks at this image just as a photo model looks at a photo.
AST is a classifier: it assigns each piece of sound to categories fixed in advance and says how confident it is about each of them. The version I use is fine-tuned on AudioSet, a large set of recordings from Google. People labelled its sounds with categories such as speech, electronic music, applause or a door slam. The model on my disk knows exactly 527 categories.
CLAP works differently. According to the model card it was trained on pairs of sound and text, so it turns both into numbers in a shared space. It has no fixed list of categories: you write “fast transition whoosh” or “calm lo-fi music” and get a number that says how well the recording matches the description. The authors trained the larger_clap_general version on general sounds, music and speech.
| AST AudioSet | CLAP larger_clap_general | |
|---|---|---|
| Type | classifier (527 categories) | sound and text in a shared space |
| Question it answers | what can be heard, according to the AudioSet list | how well the sound matches my description |
| Role in the studio | speech, music, singing and effects over time | music style and mood |
| Licence | BSD-3-Clause | Apache 2.0 |
How sound helps in rating a reel
The sound stage in the studio combines three sources:
- AST judges one-second analysis windows, moved every half second. These judgements give the shares of speech, music, singing and speech over music, plus the places where effects appear: transition whooshes, impacts and interface sounds.
- Loudness jump analysis shows whether the sound has an accent where there is a cut and whether the reel starts with a strong hit.
- CLAP describes the music style if there is enough music.
Every analysis window and every loudness jump give numbers, from which 18 sound features are made. Together with 20 picture features from the frame model, that makes 38 features of the reel.
I rejected a few approaches after trials on real reels. The PANNs CNN14 event detection model found no effect at all in a recording with speech, not even an artificial whoosh I added on purpose. CLAP used for detecting effects labelled speech as a “ding” sound. What remained were AST categories and loudness analysis, which behaved predictably.
What came out of it? On 47 of my own reels the sound stage took a median of 16.1 s per reel. The sound features then go into the model that predicts reel results, so it is worth measuring sound in every reel.
What I measure and how
| Mac mini M6 (Mac18,5) | |
|---|---|
| Processor | Apple M6, 12 CPU cores: 2 Super, 4 Performance, 6 Efficiency |
| Graphics | 12 GPU cores, Metal 4 |
| Memory | 32 GB shared by the processor and graphics, 170 GB/s bandwidth |
| Storage | 512 GB SSD (494 GB formatted) |
| System | macOS 27.0.1 |
The test has two parts.
In the first I run the studio’s whole sound stage on three of my own reels (26 s, 47 s and 90 s). I do it once on the processor and once on the Mac’s graphics chip through MPS, three runs each. The first run includes loading the models.
In the second part I run AST and CLAP on six sounds from my effects kit under the CC0 licence: transition whooshes, impacts, a riser and a click. I want to see what the models call them. AST pads short sounds with silence to about 10 s, so its confidence is lower for them. For AST I save the three most likely categories, and for CLAP the similarity of each sound to six descriptions.
I take the measurement on a Mac with no other heavy programs running. I recorded a separate single run in a real Terminal window.
Results: processor or graphics chip
| Processor | Graphics chip (MPS) | |
|---|---|---|
| 26 s reel | 16.8 s / 12.4 s | 4.8 s / 4.3 s |
| 47 s reel | 22.1 s / 22.1 s | 8.0 s / 8.0 s |
| 90 s reel | 43.1 s / 43.2 s | 15.2 s / 15.2 s |
| Per minute of video (median) | about 28.7 s | about 10.1 s |
| Highest memory use by the process (graphics chip memory may not be fully counted) | 4.7 GB (whole test) | 4.7 GB (whole test) |
CLAP kicks in only in the windows where AST hears music. In the test that happened only in the 26 s reel, while AST alone analysed the 47 s and 90 s reels. The first run on the first reel took longer, because it includes loading the models. For the next reels the models were already in memory, so the first run and the median barely differ.
Time grows evenly with the length of the recording, because a minute of sound almost always costs the same. The reel in which CLAP also worked took 29.0 s per minute on the processor, and the others 28.5–28.7 s.
The Mac’s graphics chip runs this stage 2.8–2.9 times faster than the processor: about 10 s instead of 29 s per minute of video. The MPS result matches the project, in which the sound stage took 10.5 s per minute on 47 of my reels. The studio picks the graphics chip by default, and this number confirms it.
Other observations:
- music types and speech probability were identical on both devices to the fourth decimal place, so the graphics chip does not change the result,
- on 32 GB this stage is light: the whole process took at most 4.7 GB,
- in my reels the average speech probability is 0.81–0.87, and the music types according to AST are close to zero (0.003 at most), even in the reel with a quiet music bed under my voice.
Results: what the models call the effects
1st category 2nd category 3rd category Transition, fast chop (fs-127197) Whoosh 0.18 Sound effect 0.09 Bang 0.02 Transition, woosh (fs-683096, listen above) Explosion 0.30 Whoosh 0.12 Burst, pop 0.08 Riser (fs-867126, listen above) Whoosh 0.21 Music 0.12 Sound effect 0.10 Low impact (fs-830856, listen below) Heartbeat 0.67 Heart murmur 0.14 Throbbing 0.11 Low explosion (Kenney) Heart murmur 0.16 Heartbeat 0.12 Sound effect 0.08 Click, plop (fs-447910, listen below) Sound effect 0.30 Burping 0.04 Speech 0.04
Best description Description matching the category Hit Transition, fast chop whoosh transition 0.170 whoosh transition 0.170 yes Transition, woosh whoosh transition 0.170 whoosh transition 0.170 yes Riser whoosh transition 0.314 rising tension riser 0.183 no (3rd place) Low impact (fs-830856) whoosh transition 0.185 deep cinematic impact 0.096 no (3rd place), but see below Low explosion (Kenney) deep cinematic impact 0.174 deep cinematic impact 0.174 yes Click, plop short pop click 0.205 short pop click 0.205 yes
The other two descriptions, “a man speaking” and “upbeat electronic music”, did not win for any sound. That is a good sign, because the effects contain neither speech nor music.
What AST showed:
- it recognises a transition whoosh, though with low confidence: “Whoosh” is in the top three for both transitions (0.18 and 0.12) and for the riser, but for fs-683096 “Explosion” wins,
- short, low sounds sound like a heart to it: the low whoosh fs-830856 got “Heartbeat” with confidence 0.67, and the low explosion from the Kenney kit “Heart murmur” 0.16. My guess is that in AudioSet low pulsing most often comes with a heartbeat,
- the AudioSet list has no riser, so the model picks the nearest “Whoosh”,
- the click got the general “Sound effect” category.
One detail changes the verdict on CLAP. Sound fs-830856, which my kit keeps as a low impact, is described on Freesound as a whoosh. So CLAP, which picked “whoosh transition” for it, has a point here. Counting strictly, CLAP matched 4 of 6 sounds to the description of their category in my kit.
The best cosine similarities reach only 0.17–0.31, though. That is why in the studio I do not treat them as a sure label; I only compare them with each other within one reel.
Local or API
There is no obvious equivalent for sound recognition among popular paid cloud services. You can send a recording to a multimodal model and ask about effects, but the result is text, not a judgement for every half second.
According to Gemini’s pricing, audio in Gemini 3.1 Flash-Lite costs USD 0.50 per million tokens. A minute is 1,920 tokens according to the docs, about USD 0.001. Gemini will point to moments in MM:SS format, but it will not give a confidence for 527 categories every half second.
What matters more to me is that my features are numbers over time: where an effect is and what the accent on a cut is. A language model does not return such a timeline reliably. AST on your own computer gives it for free, in 4–15 s per reel on the graphics chip.
What to watch out for
AST was trained on AudioSet, recordings from YouTube, so it recognises sounds typical of such videos best. A riser, a popular effect from editors’ libraries, has no category of its own in the AudioSet ontology, so the model calls it by the nearest known sound.
A transition whoosh has one category: “Whoosh, swoosh, swish”. That is why in the studio’s features I group it with related categories (“Zing”, “Arrow”) and look at changes over time rather than a single label.
CLAP, in turn, reacts to the exact words of a description: “whoosh” and “swoosh transition” can give different results. It is worth checking descriptions on a few sounds before basing a decision on them. I write about how I make such effects myself in the part on Stable Audio Open.
How to use sound analysis in your own editing
The simplest use, which works from day one, is checking three things in every finished reel before publishing:
- Is there a clear sound accent in the first 3 seconds? AST and loudness analysis tell you whether the reel starts with a hit, an effect or a strong word, or with silence.
- Does the music drown out the speech? The share of speech-over-music parts and the level of the music bed in them are easy to calculate from AST’s results.
- How do the cuts sound? On my reels the effect of an accent on a cut is uncertain. That is why I measure it rather than add a click to every cut.
The studio calculates these three numbers automatically and shows them in the reel report. They do not replace your ears, but they point to where it is worth listening again. Thanks to such a report I do not have to listen to whole reels in search of a problem.
Verdict
AST and CLAP are a cheap and fast way to add a sound layer to video analysis. AST answers “what can be heard here”, and CLAP answers “does it sound the way I wanted”. Both stay in the reel studio, each in its own role.
I build video and sound analysis on your own computer, fitted to your content, as part of AI automations and agents.

