Deep learning

Speech & audio recognition

call transcription, keyword spotting, sound events

Speech to text, or labels for audio.

Linear algebra

Calculus & gradients

Optimisation

Foundations

Spectrograms are Fourier transforms; the rest is the same deep-learning foundation as images.

Frame the problem

Frame

Streaming or batch. Word error rate or a downstream metric. Which languages and accents.

Sourcing & signal

Labeling & data collection

Exploring the data

Data & labels

Transcription labels are expensive; pretrained encoders make a small set go far. Check microphone and environment diversity.

Modalities → vectors

Audio networks

Represent

Spectrograms or MFCCs for classical pipelines. Raw waveform into a pretrained encoder otherwise.

Split the data

Split

By speaker.

Dumb baseline

Pretrained audio encoders

Seq2seqbackground

Your own recogniser

CTC over a 1D CNN or RNN when you must train from scratch.

CTC & speech recognition

RNN / LSTM / GRU

Model

Run Whisper or wav2vec zero-shot and measure. For classification, pretrained embeddings plus logistic regression.

Fine-tuning a pretrained model

Fine-tuning & LoRA

Train

Fine-tune the encoder on domain audio.

Classification metrics

Error analysis & significance

Evaluate offline

Word error rate by accent, noise level, and speaker. Per-class F1 for events.

Evaluate online

Nothing unusual here.

Inference optimisation

Serving & release

Ship & monitor

Streaming needs chunking and latency budgets. Quantise for edge.