Personal Project2026
SpeechVal
Multilingual TTS Evaluation Toolkit · Personal Project
A production-oriented reference implementation for evaluating interpersonal warmth in multilingual TTS and speech-to-speech output — deterministic acoustic measurements as evidence, not a hand-written definition of warmth. Designed and built solo, from the acoustic feature pipeline through calibration and the human-annotation loop.
Python
FastAPI
NumPy
SciPy
Pydantic
Docker
OpenRouterWhisperThe Workspace
A view per capability, each one built to show the evidence behind a verdict rather than the verdict alone.
Upload & Evaluate
Drag in a clip, set the locale and context, and pick which members of the judge council should run. The result is never a single number: multidimensional scores arrive with a per-judge scorecard beside the mean they collapse into, acoustic diagnostics, reliability, and a run history kept locally in the browser.
Evidence, Not a Guess
A Contract, Not a Number
Every measurement carries its own receipt
Each acoustic measurement is a value/unit/valid/coverage/reason_invalid/config_hash record. Missing data is always null with a stated reason — never a silent zero standing in for "unknown."
Deterministic First
Acoustics before opinions
Pitch, timing, intensity, spectral, phonation, clipping, VAD, and response-latency measurements run through Praat/Parselmouth with a documented autocorrelation fallback, versioned and hashed into every result.
Never Invents a Score
No calibrator, no verdict
Without an injected judge and a validated human calibrator for the target locale, the API returns acoustic evidence with status=partial. A provider's raw confidence is never published as the product's calibrated confidence.
Provider-Neutral Judging
Three Ways to Judge
Audio, transcript, or context — behind one interface
Provider-neutral judges run audio-only, transcript-only, and contextual evaluation with positive-first/negative-first pole-order checks and repeat reliability, so a judge that's just agreeing with the ordering it was given gets caught.
A Careful OpenRouter Adapter
One fixed model, no silent fallback
The OpenRouter judge sends a canonical PCM16 WAV to a named audio-capable model and rejects any response served by a different model ID. Data collection defaults to deny, with an explicit zero-retention flag for endpoints that support it.
Strict, Silent Parsing
Failures are logged by field path only
The local JSON parser reports failures by field path and never echoes the model's raw output back into an error or a log line, so a malformed judge response can't leak into observability tooling.
Human-in-the-Loop Calibration
Locale-Specific, Full Stop
No implicit language fallback
Temperature scaling and conformal intervals are fit per locale from human ratings. An en-US artifact is never silently reused for pt-BR just because both are configured.
Speaker-Level Holdouts
Splits respect the speaker
Calibration splits are group-aware — by speaker, not by sample — so a calibrator is never validated on the same voice it was trained on.
Agreement Before Trust
Reliability is computed, not assumed
Inter-rater agreement is computed across every sample with two or more human ratings; samples with fewer raters carry no reliability information and are excluded rather than padded.
Catching a Judge That's Gaming It
Four perturbations a human barely notices
Gain shifts, background noise, and a telephone-band filter should barely move a human's perception of warmth. If a judge's score swings by 0.5 or more on those clips anyway, the case is flagged gaming_suspected — evidence it's reacting to a channel artifact, not the construct it claims to measure.
This project must not be used to infer stable traits, health, personality, ethnicity, age, gender, or regional identity from a person's voice.
