Does a Cloned Voice Stay the Same? Scoring Qwen3-TTS With ECAPA

Written with AI assistance. Every number comes from my own runs; if you spot an error, tell me and I'll fix it.

I had Qwen3-TTS 1.7B read two long narrations, one in Korean and one in English, and scored every sentence chunk against the voice sample using ECAPA speaker embeddings. In these two runs, English held steady and Korean wandered more: the English narration had a median similarity of 0.861 and no chunk below 0.75, while the Korean one had a median of 0.806, with 10 of its 51 chunks under 0.75.

The practical lesson is short. Don't assume a cloned voice stays the same across a long script. Score every chunk against the sample, re-read the low ones, and keep the closest take. Below is how I did that, plus what the numbers can and can't tell you.

Why a cloned voice drifts in long narration

A local text-to-speech (TTS) model rarely reads a 20-minute script in one go. You split it into sentence chunks, generate each one, and stitch the results together. Each chunk is its own generation, so the voice can shift a little from one chunk to the next.

Both narrations used Qwen3-TTS 1.7B with Base-model voice cloning. For each language channel I generated a voice sample once. After that, every chunk was read by cloning that same sample. I wanted to know how closely each chunk actually matched it.

How do you measure whether a TTS voice stays consistent?

I used a speaker embedding. That's a vector a model computes from a speech clip, designed to capture who is talking rather than what's being said. Two clips of the same person should give vectors pointing in roughly the same direction.

To compare two embeddings I used cosine similarity. Higher means more alike, and 1.0 means the vectors point in exactly the same direction. The embedding model was speechbrain's spkrec-ecapa-voxceleb, a pretrained ECAPA speaker recognition model. I ran it on the CPU.

Here's a minimal sketch of the scoring step. It isn't my exact script, just its general structure, using speechbrain's pretrained interface. In older speechbrain versions the import lives under speechbrain.pretrained.

import torch
import torchaudio
from speechbrain.inference.speaker import EncoderClassifier

encoder = EncoderClassifier.from_hparams(
    source="speechbrain/spkrec-ecapa-voxceleb",
    run_opts={"device": "cpu"},
)

def embed(path):
    signal, sr = torchaudio.load(path)
    if sr != 16000:  # the VoxCeleb model expects 16 kHz audio
        signal = torchaudio.functional.resample(signal, sr, 16000)
    signal = signal.mean(dim=0, keepdim=True)  # mono
    return encoder.encode_batch(signal).squeeze()

def similarity(path_a, path_b):
    a, b = embed(path_a), embed(path_b)
    return torch.nn.functional.cosine_similarity(a, b, dim=0).item()

Next comes the retry loop. generate_chunk is a placeholder for the Qwen3-TTS voice-clone call. The usage for cloning from a reference clip is in the Qwen3-TTS repository (https://github.com/QwenLM/Qwen3-TTS), and the details depend on how you've installed it.

THRESHOLD = 0.7
MAX_REREADS = 3

def read_chunk(text, reference_wav):
    # reference_wav is the language channel's voice sample:
    # the Korean sample for Korean chunks, the English one for English.
    best_path, best_score = None, -1.0
    for attempt in range(1 + MAX_REREADS):
        path = generate_chunk(text, reference_wav)  # Qwen3-TTS clone call
        score = similarity(path, reference_wav)
        if score > best_score:
            best_path, best_score = path, score
        if score >= THRESHOLD:
            break
    return best_path, best_score

In plain terms: generate the chunk, score it against the sample, and stop once the score reaches 0.7. If it doesn't, re-read it up to three times and keep whichever take came closest.

The numbers

These scores are for the final chunks that went into each narration, after any re-reads.

One caution before comparing the columns. Each language was scored against its own voice sample, with a different script and a different number of chunks. So the gap between them isn't proof that the language itself is the cause.

The lowest final Korean chunk was 0.705, just above the 0.7 re-read threshold. I didn't log individual attempts, so I can't tell how many chunks needed re-reads.

In these two runs, English was the tighter of the two. Its worst chunk (0.808) scored higher than the Korean median (0.806). The Korean scores also covered a wider range, from 0.705 to 0.897.

The 0.75 row is just the cutoff I used for counting in this write-up. It's not part of the re-read loop. Ten Korean chunks cleared 0.7 but stayed under 0.75, and no English chunk did.

What these scores don't tell you

There's one narration per language, so one video each. I didn't test why Korean scored lower, and with this sample I can't separate the effect of the language from the script or the voice sample.

I also didn't record scores before re-reading, or how many retries happened, so I can't say how much the loop helped. And I never scored an unrelated speaker, so I don't know where "clearly a different voice" falls on this scale.

The biggest limit is that similarity is only a rough stand-in for "same speaker." It says nothing about pronunciation, intonation, or whether a sentence sounds natural. A chunk could score high and still mispronounce a word.

Generation speed, from a separate test

These figures come from a different record, my project's progress tracker, not from the two narrations above. They cover Korean only, from a single test.

In that test, Qwen3-TTS 1.7B turned 252 Korean characters into 34.6 seconds of audio. Generation took 97 seconds, about 2.8x real time, and used 4.9 GB of VRAM. The tracker marks Korean TTS as done but notes that a listening check is still needed.

I didn't measure how much time the ECAPA scoring and re-reads added to the narrations.

Checklist

What I actually did:

1. Generate the voice sample once per language channel and clone that same file for every chunk.
2. Split the script into sentence-sized chunks.
3. Score each chunk against its channel's sample with spkrec-ecapa-voxceleb on the CPU.
4. Re-read any chunk below 0.7, up to three times, and keep the closest take.
5. Report the lowest, median and highest score, plus how many chunks fall under 0.75.

Suggestions I haven't tested yet:

• Log every attempt's score, not just the final one.
• Use 0.75 as a second line for flagging chunks to review.
• Score a few clips from a different speaker to get a baseline.
• Listen to the low-scoring chunks, since the score won't catch pronunciation problems.

FAQ

How do I check if a cloned TTS voice stays consistent across a long narration?

I scored each sentence chunk against the voice sample using cosine similarity between ECAPA speaker embeddings from speechbrain's spkrec-ecapa-voxceleb, run on the CPU.

What ECAPA similarity scores did Qwen3-TTS 1.7B get?

After re-reads, the English narration (108 chunks) ranged from 0.808 to 0.908 with a median of 0.861. The Korean narration (51 chunks) ranged from 0.705 to 0.897 with a median of 0.806.

What threshold did I use for re-reading a chunk?

Any chunk scoring under 0.7 was re-read up to three times, and I kept the take closest to the voice sample.

Does a high speaker similarity score mean the audio sounds good?

No. Similarity only roughly indicates whether it's the same speaker. It doesn't measure pronunciation or intonation, so you still need to listen.

Does Korean drift more than English in Qwen3-TTS?

In my two runs, Korean scored lower. But each language used a different voice sample and script, with one video each, so this doesn't show a language effect.

How much VRAM does Qwen3-TTS 1.7B use?

In a separate Korean-only test, it used 4.9 GB of VRAM and took 97 seconds to produce 34.6 seconds of audio from 252 characters.