Home → Blog → How to build a batch transcription pipeline for an audio archive

How to build a batch transcription pipeline for an audio archive

One recording is a single line of code. Ten thousand recordings is a different problem: something will fail halfway, the process will be killed, half the files will already be done, and you will need to know which half. This is the shape of a batch pipeline that survives all of that.

What breaks when you go from one file to ten thousand

A loop over a folder works fine until the archive gets big. Then four things go wrong:

• Something always fails. Over ten thousand requests, a handful will hit a network blip or a 5xx. Without retries the loop dies on file 4 312.
• The process gets killed. A deploy, an OOM, a laptop lid. If progress lives only in memory, you start from zero and pay for the same minutes twice.
• Sequential is slow. A file takes a couple of seconds; ten thousand files sequentially is most of a day. Concurrency turns that into an hour.
• The bill is invisible. Per-minute billing means the cost is decided by the audio you send, and you find out afterwards.

Every point below exists to fix one of those four.

Step 1. Make progress durable

The single most important decision: the record of what is done must survive the process. A directory of result files is enough — the presence of the output is the state.

import pathlib

SRC = pathlib.Path("recordings")
OUT = pathlib.Path("transcripts")
OUT.mkdir(exist_ok=True)

def pending():
    for p in sorted(SRC.glob("*.mp3")):
        if not (OUT / (p.stem + ".txt")).exists():
            yield p

Now the job is idempotent: run it, kill it, run it again — it picks up exactly where it stopped and never pays for the same file twice. For bigger archives an SQLite table with a status column works the same way and gives you queries; the principle does not change.

Write the output atomically, or a process killed mid-write leaves a truncated file that looks finished:

tmp = out.with_suffix(".part")
tmp.write_text(text, encoding="utf-8")
tmp.rename(out)          # atomic: either no new file, or a complete one

Step 2. Run several files at once

Transcription is network-bound waiting, not local CPU work, so threads are the right tool and a thread pool is all you need:

import os
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI

client = OpenAI(base_url="https://voicesscribe.com/v1", api_key=os.environ["VS_KEY"], max_retries=0)

def transcribe(path):
    with path.open("rb") as f:
        return client.audio.transcriptions.create(model="whisper-1", file=f).text

with ThreadPoolExecutor(max_workers=4) as pool:
    for path, text in zip(pending(), pool.map(transcribe, pending())):
        (OUT / (path.stem + ".txt")).write_text(text, encoding="utf-8")

Pick max_workers to match the requests-per-second limit on your plan, not the number of cores. Going wider than the limit does not speed anything up — it just converts successful requests into 429s.

Step 3. Retry the failures, not the successes

Two classes of error need opposite handling. A 5xx or a timeout is worth retrying; a 400 on a corrupted file will fail identically forever.

import time
from openai import APIStatusError, APIConnectionError

def with_retry(path, attempts=3):
    for n in range(attempts):
        try:
            return transcribe(path)
        except APIConnectionError:
            pass                                  # network blip — retry
        except APIStatusError as e:
            if e.status_code == 429 or e.status_code >= 500:
                pass                              # rate limit or server error — retry
            else:
                raise                             # 400/401 — retrying will not help
        time.sleep(2 ** n)                        # 1s, 2s, 4s
    raise RuntimeError(f"failed after {attempts}: {path}")

The exponential backoff matters for 429 specifically: retrying immediately keeps you over the limit. Note max_retries=0 on the client above — otherwise the SDK retries too and the two policies fight each other.

Do not let one bad file stop the run. Log it, move on, and deal with the failures list at the end — it is usually two or three files out of thousands.

Step 4. Prepare the audio before sending it

This is where the bill is actually decided. Two rules do most of the work:

• Convert once, upfront. Mono at 16 kHz is what the model uses internally, and it uploads far faster than the original wav.
• Skip what has no speech. Silence still costs minutes.

# convert the whole archive in one pass
find recordings -name '*.wav' -print0 |
  xargs -0 -P4 -I{} ffmpeg -loglevel error -i {} -ac 1 -ar 16000 -b:a 48k {}.mp3

Files over 25 MB have to be split anyway; with mono 48 kbit/s that threshold is about an hour of audio, so most archives stop needing splitting at all after conversion.

Step 5. Know the cost before you start

Billing is per minute of audio, so the total is knowable in advance — measure the archive instead of guessing:

ffprobe -v error -show_entries format=duration -of csv=p=0 file.mp3

import subprocess

def minutes(path):
    out = subprocess.run(["ffprobe", "-v", "error", "-show_entries",
                          "format=duration", "-of", "csv=p=0", str(path)],
                         capture_output=True, text=True).stdout
    return float(out) / 60

total = sum(minutes(p) for p in SRC.glob("*.mp3"))
print(f"{total:.0f} min, {total * RATE:.2f} at the current rate")

Run this before the pipeline, not after. It takes a minute, it costs nothing, and it is the difference between a planned spend and a surprise.

Step 6. Watch the run

A long batch needs three numbers visible: done, failed, and rate. A counter printed every hundred files is enough:

done = failed = 0
for path in pending():
    try:
        save(path, with_retry(path))
        done += 1
    except Exception as e:
        failed += 1
        print("FAIL", path.name, e)
    if (done + failed) % 100 == 0:
        print(f"{done} done, {failed} failed")

The client area shows the same run from the other side — minutes consumed, requests, and the per-request history — which is the quickest way to confirm the pipeline is doing what you think it is doing.

What not to build

• A message broker for ten thousand files. A folder plus a thread pool handles that scale. Reach for a queue when the work arrives continuously, not when it is a one-off archive.
• Your own rate limiter. Set the pool size below the plan limit and you never hit one.
• A resume-position counter. Storing “I stopped at file 4312” breaks the moment the input order changes. Derive state from what exists on disk instead.

Frequently asked questions

Related reading

• How to transcribe a call recording to text — A step-by-step guide to turning phone call recordings into text through an API in minutes: Python and C# samples, response formats and the errors you will meet.
• Speech-to-text pricing: what you actually pay for — How transcription billing works, which parts of your audio cost money, when self-hosting a GPU is cheaper than an API, and how to estimate your own monthly spend.
• How to improve speech-to-text accuracy: eight practical fixes — Eight changes that actually move transcription quality: audio preparation, language hints, the prompt parameter, chunking, formats, and how to measure word error rate.