New on GitHub
Sources 33Updates always the freshest
0 Newsletter
Animated illustration for: AST and CLAP: a model hears what happens in a reelAST AudioSetCLAP
Scale · Image, video & voice · Comparison · 12 min read

AST and CLAP: a model hears what happens in a reel

I recognise sounds in my reels with the AST and CLAP models on a Mac mini: speech, music and effects. Time on the processor and graphics chip, and mistakes.

In short
  • AST turns sound into an image and recognises 527 kinds of sound from AudioSet.
  • CLAP compares sound with a text description.
  • On the Mac mini M6's graphics chip a minute of sound takes about 10 s, on the processor about 29 s.
  • AST called a low impact a heartbeat, and CLAP matched 4 of 6 effects to the right description.

The eleventh part of the series is about models that listen. When I rate a reel, the sound tells me as much as the picture: is there music, am I speaking over it, does a cut have a transition effect, does the reel start with an impact. In my reel studio two models do this: AST, which recognises 527 kinds of sound from AudioSet, and CLAP, which compares sound with a text description. I test them on three of my own reels on a Mac mini M6 with 32 GB, once on the processor and once on the graphics chip.

How a model hears

AST, the Audio Spectrogram Transformer, does something surprisingly simple with sound. It turns it into a spectrogram, an image with time on the horizontal axis, pitch on the vertical axis and brightness as loudness. Then it looks at this image just as a photo model looks at a photo.

AST is a classifier: it assigns each piece of sound to categories fixed in advance and says how confident it is about each of them. The version I use is fine-tuned on AudioSet, a large set of recordings from Google. People labelled its sounds with categories such as speech, electronic music, applause or a door slam. The model on my disk knows exactly 527 categories.

CLAP works differently. According to the model card it was trained on pairs of sound and text, so it turns both into numbers in a shared space. It has no fixed list of categories: you write “fast transition whoosh” or “calm lo-fi music” and get a number that says how well the recording matches the description. The authors trained the larger_clap_general version on general sounds, music and speech.

Two sound models in my studio
AST AudioSetCLAP larger_clap_general
Typeclassifier (527 categories)sound and text in a shared space
Question it answerswhat can be heard, according to the AudioSet listhow well the sound matches my description
Role in the studiospeech, music, singing and effects over timemusic style and mood
LicenceBSD-3-ClauseApache 2.0

How sound helps in rating a reel

The sound stage in the studio combines three sources:

  • AST judges one-second analysis windows, moved every half second. These judgements give the shares of speech, music, singing and speech over music, plus the places where effects appear: transition whooshes, impacts and interface sounds.
  • Loudness jump analysis shows whether the sound has an accent where there is a cut and whether the reel starts with a strong hit.
  • CLAP describes the music style if there is enough music.

Every analysis window and every loudness jump give numbers, from which 18 sound features are made. Together with 20 picture features from the frame model, that makes 38 features of the reel.

I rejected a few approaches after trials on real reels. The PANNs CNN14 event detection model found no effect at all in a recording with speech, not even an artificial whoosh I added on purpose. CLAP used for detecting effects labelled speech as a “ding” sound. What remained were AST categories and loudness analysis, which behaved predictably.

What came out of it? On 47 of my own reels the sound stage took a median of 16.1 s per reel. The sound features then go into the model that predicts reel results, so it is worth measuring sound in every reel.

What I measure and how

Test hardware: my Mac mini (data from system_profiler and Apple's website)
Mac mini M6 (Mac18,5)
ProcessorApple M6, 12 CPU cores: 2 Super, 4 Performance, 6 Efficiency
Graphics12 GPU cores, Metal 4
Memory32 GB shared by the processor and graphics, 170 GB/s bandwidth
Storage512 GB SSD (494 GB formatted)
SystemmacOS 27.0.1

The test has two parts.

In the first I run the studio’s whole sound stage on three of my own reels (26 s, 47 s and 90 s). I do it once on the processor and once on the Mac’s graphics chip through MPS, three runs each. The first run includes loading the models.

In the second part I run AST and CLAP on six sounds from my effects kit under the CC0 licence: transition whooshes, impacts, a riser and a click. I want to see what the models call them. AST pads short sounds with silence to about 10 s, so its confidence is lower for them. For AST I save the three most likely categories, and for CLAP the similarity of each sound to six descriptions.

I take the measurement on a Mac with no other heavy programs running. I recorded a separate single run in a real Terminal window.

A recording of a single run in a real Terminal. The numbers in the tables come from a separate measurement.

Results: processor or graphics chip

The sound stage (AST + CLAP) on the Mac mini M6: first run / median of 3 (measured 9 Nov 2026)
ProcessorGraphics chip (MPS)
26 s reel16.8 s / 12.4 s4.8 s / 4.3 s
47 s reel22.1 s / 22.1 s8.0 s / 8.0 s
90 s reel43.1 s / 43.2 s15.2 s / 15.2 s
Per minute of video (median)about 28.7 sabout 10.1 s
Highest memory use by the process (graphics chip memory may not be fully counted)4.7 GB (whole test)4.7 GB (whole test)

CLAP kicks in only in the windows where AST hears music. In the test that happened only in the 26 s reel, while AST alone analysed the 47 s and 90 s reels. The first run on the first reel took longer, because it includes loading the models. For the next reels the models were already in memory, so the first run and the median barely differ.

Time grows evenly with the length of the recording, because a minute of sound almost always costs the same. The reel in which CLAP also worked took 29.0 s per minute on the processor, and the others 28.5–28.7 s.

The Mac’s graphics chip runs this stage 2.8–2.9 times faster than the processor: about 10 s instead of 29 s per minute of video. The MPS result matches the project, in which the sound stage took 10.5 s per minute on 47 of my reels. The studio picks the graphics chip by default, and this number confirms it.

Other observations:

  • music types and speech probability were identical on both devices to the fourth decimal place, so the graphics chip does not change the result,
  • on 32 GB this stage is light: the whole process took at most 4.7 GB,
  • in my reels the average speech probability is 0.81–0.87, and the music types according to AST are close to zero (0.003 at most), even in the reel with a quiet music bed under my voice.

Results: what the models call the effects

Sound 1 from my CC0 kit: a deep transition whoosh.

Sound 2: a riser.

AST: the three most likely AudioSet categories for 6 sounds from my CC0 kit
1st category2nd category3rd category
Transition, fast chop (fs-127197)Whoosh 0.18Sound effect 0.09Bang 0.02
Transition, woosh (fs-683096, listen above)Explosion 0.30Whoosh 0.12Burst, pop 0.08
Riser (fs-867126, listen above)Whoosh 0.21Music 0.12Sound effect 0.10
Low impact (fs-830856, listen below)Heartbeat 0.67Heart murmur 0.14Throbbing 0.11
Low explosion (Kenney)Heart murmur 0.16Heartbeat 0.12Sound effect 0.08
Click, plop (fs-447910, listen below)Sound effect 0.30Burping 0.04Speech 0.04

Sound 3: a low impact that AST took for a heartbeat with confidence 0.67.

Sound 4: a click (a plop pitched down an octave), simply a “sound effect” for AST.

CLAP: which of 6 English descriptions best fits each sound (cosine similarity from −1 to 1, higher means closer to the description)
Best descriptionDescription matching the categoryHit
Transition, fast chopwhoosh transition 0.170whoosh transition 0.170yes
Transition, wooshwhoosh transition 0.170whoosh transition 0.170yes
Riserwhoosh transition 0.314rising tension riser 0.183no (3rd place)
Low impact (fs-830856)whoosh transition 0.185deep cinematic impact 0.096no (3rd place), but see below
Low explosion (Kenney)deep cinematic impact 0.174deep cinematic impact 0.174yes
Click, plopshort pop click 0.205short pop click 0.205yes

The other two descriptions, “a man speaking” and “upbeat electronic music”, did not win for any sound. That is a good sign, because the effects contain neither speech nor music.

What AST showed:

  • it recognises a transition whoosh, though with low confidence: “Whoosh” is in the top three for both transitions (0.18 and 0.12) and for the riser, but for fs-683096 “Explosion” wins,
  • short, low sounds sound like a heart to it: the low whoosh fs-830856 got “Heartbeat” with confidence 0.67, and the low explosion from the Kenney kit “Heart murmur” 0.16. My guess is that in AudioSet low pulsing most often comes with a heartbeat,
  • the AudioSet list has no riser, so the model picks the nearest “Whoosh”,
  • the click got the general “Sound effect” category.

One detail changes the verdict on CLAP. Sound fs-830856, which my kit keeps as a low impact, is described on Freesound as a whoosh. So CLAP, which picked “whoosh transition” for it, has a point here. Counting strictly, CLAP matched 4 of 6 sounds to the description of their category in my kit.

The best cosine similarities reach only 0.17–0.31, though. That is why in the studio I do not treat them as a sure label; I only compare them with each other within one reel.

Local or API

There is no obvious equivalent for sound recognition among popular paid cloud services. You can send a recording to a multimodal model and ask about effects, but the result is text, not a judgement for every half second.

According to Gemini’s pricing, audio in Gemini 3.1 Flash-Lite costs USD 0.50 per million tokens. A minute is 1,920 tokens according to the docs, about USD 0.001. Gemini will point to moments in MM:SS format, but it will not give a confidence for 527 categories every half second.

What matters more to me is that my features are numbers over time: where an effect is and what the accent on a cut is. A language model does not return such a timeline reliably. AST on your own computer gives it for free, in 4–15 s per reel on the graphics chip.

What to watch out for

AST was trained on AudioSet, recordings from YouTube, so it recognises sounds typical of such videos best. A riser, a popular effect from editors’ libraries, has no category of its own in the AudioSet ontology, so the model calls it by the nearest known sound.

A transition whoosh has one category: “Whoosh, swoosh, swish”. That is why in the studio’s features I group it with related categories (“Zing”, “Arrow”) and look at changes over time rather than a single label.

CLAP, in turn, reacts to the exact words of a description: “whoosh” and “swoosh transition” can give different results. It is worth checking descriptions on a few sounds before basing a decision on them. I write about how I make such effects myself in the part on Stable Audio Open.

How to use sound analysis in your own editing

The simplest use, which works from day one, is checking three things in every finished reel before publishing:

  • Is there a clear sound accent in the first 3 seconds? AST and loudness analysis tell you whether the reel starts with a hit, an effect or a strong word, or with silence.
  • Does the music drown out the speech? The share of speech-over-music parts and the level of the music bed in them are easy to calculate from AST’s results.
  • How do the cuts sound? On my reels the effect of an accent on a cut is uncertain. That is why I measure it rather than add a click to every cut.

The studio calculates these three numbers automatically and shows them in the reel report. They do not replace your ears, but they point to where it is worth listening again. Thanks to such a report I do not have to listen to whole reels in search of a problem.

Verdict

AST and CLAP are a cheap and fast way to add a sound layer to video analysis. AST answers “what can be heard here”, and CLAP answers “does it sound the way I wanted”. Both stay in the reel studio, each in its own role.

I build video and sound analysis on your own computer, fitted to your content, as part of AI automations and agents.

Frequently asked questions

How is AST different from CLAP?
AST is a classifier: for each piece of sound it says how well it fits each of 527 fixed AudioSet categories. CLAP has no fixed list; it compares sound with any text description you give it, so you can ask about things outside AudioSet.
Can the model tell speech from music in a reel?
Yes, that is one of the easiest tasks for AST. In my studio its results become the shares of speech, music, singing and speech over music across the whole reel.
Why do I not use CLAP to detect sound effects?
Because in my trial it confused speech with a "ding" sound. For detecting effects, AST categories and sudden loudness jumps worked better, and I kept CLAP for describing music style and judging how well a sound matches a description.
Does sound analysis work on the processor?
Yes, both models run on the processor alone, but more slowly than on the Mac's graphics chip. On the Mac mini M6 a minute of sound takes about 29 s on the processor and about 10 s on the graphics chip.

Sources

Image, video & voice

Related