HT Demucs: A Separated Vocal Still Needs a Listening Test

Imagine you have a stereo master of a song you made years ago. The session files are gone, but you want to bring the lead vocal into a quieter arrangement. A separation tool gives you a file called ‘vocals’. In the old mix it sounds convincing. Against the quieter arrangement, a splash of cymbal appears at the end of a phrase. What sounded like a recovered track is better understood as an estimate that needs auditioning.

Hybrid Transformer Demucs is a useful example of how AI tackles this problem. The research asks how to infer musical sources from a finished mixture. For a creator, the next question is practical: what can you do with an estimate once the rest of the arrangement changes?

What the model is trying to separate

In the Demucs project’s four-source setup, the outputs are vocals, drums, bass and other accompaniment. That last category matters. A file called ‘other’ is a group of sounds, not a promise of separate tracks for the keyboard, guitar and strings. The repository also documents an experimental six-source variant and reports unwanted spill and artefacts in its piano output.

An original vocal recording and an estimated vocal stem answer different questions. The recording preserves what was captured in the session. The estimate is the system’s reconstruction of a target source using the available mix. If the original session is available, it is a better reference for checking what the model removed, kept or altered.

Why the research used two views of the audio

Alexandre Défossez’s Hybrid Demucs paper, first posted in November 2021, combined a waveform representation, which follows the audio signal over time, with a spectrogram representation, which shows how its frequency content changes over time. The architecture lets the two representations contribute to the separation.

Simon Rouard, Francisco Massa and Défossez extended that approach in Hybrid Transformers for Music Source Separation, first posted in November 2022 and presented at ICASSP in 2023. The transformer compares information within each representation and across the two. The authors used this architecture to investigate whether a wider span of musical context helps source separation.

The training result is more informative than the model name. The authors reported weak performance when the transformer model was trained only on MUSDB. With 800 additional training songs, it beat Hybrid Demucs trained on the same data by 0.45 dB on their signal-to-distortion measure. The MUSDB18 project provides mixes alongside isolated reference parts. That gives researchers a comparison unavailable when the only surviving file is your own master.

The comparison supports a particular result under a particular training and evaluation setup. It does not give an acceptance threshold for a sparse remix, a vocal exposed in an advert, or a section with an unusually dense backing arrangement. Those are different listening jobs.

Listen for what stayed and what went missing

In the hypothetical remix, the obvious test is whether another instrument remains in the vocal. Give equal attention to what the separation may have taken away. A line can be intelligible while losing a breath, a soft consonant or the decay at the end of a note. Decide which details matter for the new production.

Try passages where the target has to compete with something else. A chorus with drums and doubled vocals is a different test from a quiet verse. Also check a gap in the singing: a brief instrumental sound there may tell you more about the extraction than a loud sustained note.

Compare at a comfortable, similar listening level. Listen to the original mix, the estimated stem on its own, then the stem against the intended new backing. A sound that is easy to miss in one arrangement can become distracting in another. These are proposed listening checks, not reported outcomes from running the model on a particular recording.

Give the estimate a specific job

A rough arrangement guide and a vocal for public release need different review decisions. For a guide, you might accept some residue if you can follow the phrasing and timing. For a quiet production with the voice in front, a short burst of spill can be enough to choose another take or record the line again.

• Choose a passage: include a quiet line, a busy section and a gap where the target should be absent.
• Describe the issue: note the time and what you hear, such as a drum hit in the vocal gap or a clipped word ending.
• Check the destination: audition the proposed edit inside the actual arrangement and listen to the final export before signing it off.

This is a different task from generating an arrangement from a melody. MaryChuks.com’s MusicGen article explains what to check when a melody guides generation. Here the source mix already exists. The question is how well the estimate preserves the part you need while leaving the rest behind.

Keep the original and label the new material

Keep the master unchanged and identify the new file as an estimated stem. Record the model or service, the settings you know, and any edits made afterwards. A collaborator can then tell whether the file came from the original session or from a separation pass, and can return to the master if the new arrangement exposes a problem.

That distinction makes room for useful creative work. An imperfect estimate can still help you rehearse a part, sketch an arrangement or decide whether a re-recording is worthwhile. Choose the passages and uses it can support by listening to the result in the place it will actually be heard.

Sources

Défossez, Hybrid Spectrogram and Waveform Source Separation (first posted 5 November 2021); Rouard, Massa and Défossez, Hybrid Transformers for Music Source Separation (first posted 15 November 2022; ICASSP 2023); Demucs project documentation and MUSDB18 dataset documentation.