Clinical Speech ML Pipeline
Mobile speech capture, FastAPI ingestion, and acoustic analysis pipeline.
- Status
- Archived · Kong Labs, Purdue
- Role
- Undergraduate Research Assistant
- Timeline
- Feb–Aug 2025
- Stack
- Python · Expo · FastAPI · Whisper · Librosa · FFmpeg · Postgres · asyncpg
01Problem
Capturing and analysing clinical speech data requires a pipeline that handles mobile audio recording, reliable upload, audio preprocessing, transcription, and structured persistence — without requiring expensive clinic-side infrastructure.
Capture environments vary across devices, rooms, and microphones. Normalisation provides a consistent preprocessing step, but does not eliminate those differences or establish that acoustic measures are comparable across participants.
02Mobile recording flow
An Expo mobile app handles speech capture on iOS and Android. Sessions are recorded as audio files and uploaded to the FastAPI backend via multipart HTTP upload.
The app sends session metadata (participant ID, session type, recording parameters) alongside the audio payload. The backend stages each upload before passing it to the analysis pipeline.
03Upload pipeline and normalisation
The FastAPI backend receives multipart uploads and stages audio files for processing. FFmpeg normalises each file — adjusting loudness, sample rate, and channel format — before it enters the analysis pipeline.
Consistent preprocessing makes the analysis path easier to inspect and repeat; it cannot remove all effects of the capture environment. The described backend uses asyncpg for database access and connection pooling.
04Transcription and acoustic analysis
The prototype combines Whisper transcription with exploratory acoustic analysis using Librosa. These outputs support investigation of recordings, rather than clinically validated measurements of speech or health.
These metrics are stored per-session in Postgres alongside the transcript. The combination supports downstream analysis of speech patterns across sessions and participants.
05Persistence and analytics
Postgres stores session metadata, upload records, transcripts, and per-metric acoustic scores. An analytics endpoint aggregates metrics across sessions for a given participant — producing summary statistics for review.
Health check endpoints cover database connectivity and pipeline service availability.
06Limitations
Access controls not established. This prototype case study does not establish production participant or researcher authentication and authorization. The linked organization page supplies institutional context, not public evidence of those implementation details.
No privacy or compliance guarantees. This description does not establish clinical data-handling, consent, or regulatory compliance. It is a research prototype, not a clinical tool.
Exploratory acoustic analysis. Acoustic outputs are not clinically validated measures, and no diagnostic conclusions should be drawn from them.
Prototype-stage evaluation. The pipeline has not been validated for reliability at scale or against clinical ground truth. Performance claims from other research contexts should not be attributed to this implementation.
07What I learned
A consistent preprocessing path makes intermediate results easier to inspect. Capture conditions still matter, so preprocessing choices and analysis assumptions need to remain visible when interpreting outputs.
Separating upload, normalisation, and analysis stages made each independently testable and debuggable. A monolithic pipeline would have obscured which stage produced unexpected results.