Overview
This project explores how explainable user interfaces can help language analysts pick the right Speech-to-Text (STT) model in increasingly complex workflows. It was a year-long, research-driven team project with the Laboratory for Analytic Sciences (LAS), a translational research lab that brings together government, academic, and industry partners.
We looked at how explainable AI (XAI), human–AI collaboration, and interface design can bridge advanced machine learning and the needs of non-technical experts, anticipating a future where analysts choose from a large, specialized toolkit of STT models every day.
Scenario Videos
Riley, a fictional analyst at the Nixon Historical Audio Task Force, is piloting new STT models to find conversations about the Apollo missions. The videos show how layered explanations support her choices without breaking her momentum.
Design Problem
Specialized STT models are multiplying, each tuned to different audio conditions, languages, and goals. That gives analysts more power, but also a new problem: choosing the right model for each audio file, quickly and confidently. Analysts are experts in content and context, not machine learning, and current tools assume technical literacy they don't need to have.
Without clear guidance, analysts fall back on trial and error, default models, or opaque scores, which leads to inefficiency, mistrust, and cognitive overload.
How might an explainable user interface help language analysts confidently select the most appropriate STT model, without increasing cognitive overload?
We treated model selection as part of the workflow, not a separate technical task. The interface should surface just enough explanation, at the right moment, to support action. The goal is not maximal transparency but calibrated trust.
Design Process
Our process moved back and forth between understanding analyst workflows, framing decision challenges, exploring interaction models, and validating choices through feedback, so every decision about explainability and trust stayed grounded in real analytic constraints.
Key insight: layered explanations
Research distinguishes local explanations, which justify a recommendation in the user's current context, from global explanations, which reveal how the system behaves overall. Trust grows when the two are paired. Layered explanations let us stay transparent without overwhelming analysts, who can go as deep as their role and moment require.
Research foundation
We surveyed commercial and research STT tools and found a consistent gap between system-level metrics and what analysts need. A parallel literature review on XAI, recommendation systems, and interpretability showed that effective explanation depends less on technical depth than on structure, timing, and relevance to the task.
From insight to interface structure
We turned the What–Why–How framework into a UI built around the selection workflow: a glanceable, plain-language summary of suggested models; a middle layer explaining task-relevant factors such as audio characteristics and model strengths; and an optional deep layer with technical logic and performance metrics for those who want to verify.
Persona & user journey map
Concept Exploration
Low-fidelity exploration tested how three levels of explanation could fit into the existing workflow, trying different visualizations, interaction patterns, and placements to see how each affected speed and clarity.
Idea one: list-based
While a model is in use, the system suggests alternatives based on performance and community usage, in a concise list.
Idea two: visual comparison
The system compares the current model with top alternatives across key metrics, so strengths and weaknesses are quick to spot.
Idea three: task-focused expansion
Based on the analyst's current activity, the system suggests not only better models but also tasks that may suit the audio better.
Early feedback and a pivot
Radar charts and dense comparative graphics proved hard to read quickly and added friction. Analysts preferred familiar patterns they could scan and compare fast, so we shifted from visual novelty to legibility, restraint, and task alignment.
Mid-fidelity: two directions
System-guided discovery. The interface detects audio characteristics in the selected cut and presents a ranked list of models from two sources: system recommendations based on performance and audio analysis, and community recommendations from peer usage. Suited to time-sensitive triage.
Analyst-guided exploration. Analysts mark what they hear, such as background noise, cadence issues, or semantic inconsistency, and get tailored recommendations as comparable model cards, with community notes as trust signals. Suited to deliberate decisions.
Survey & Evaluation
We surveyed 28 language analysts across senior, intermediate, and junior levels with varying AI familiarity, to learn how they read recommendations, what explanations they rely on, and how that changes with expertise. Most agreed the interface was intuitive, the recommendations understandable, and that they felt confident finding a suitable model.
A clear pattern emerged: analysts trusted community-validated signals, such as peer usage, task fit, and shared experience, more than system scores. Three responses captured what guided the next iteration:
“If there is user feedback to crowdsource, wouldn't that ALWAYS trump the system suggestions? Users are going to be more influenced by other users rather than a nebulous system suggestion.”
Senior Analyst P10
“The ability to immediately jump to keywords and have a provided STT transcript is an incredibly valuable tool that would speed up my workflow considerably.”
Intermediate Analyst P05
“Model description and tags are helpful but don't explain how audio is processed. It would be helpful to have examples of original vs processed clip and what the model is NOT good at.”
Senior Analyst P19
Hi-Fi Design: Layered Explanation System
Instead of adding metrics, we refined the interface around three directions from the survey: transparency through three levels of explanation, adaptability to real analyst workflows, and trust through community insight alongside system recommendations.
Adaptability to real workflows
Transcript sampling, audio playback, and side-by-side comparison let analysts check a recommendation inside their existing audio flow. Model selection lives in the audio player and task context, so it barely interrupts the work.
Trust through community
Peer usage frequency, task-specific success, and qualitative feedback sit beside the system's own scores, so analysts can calibrate trust through both system intelligence and shared experience.
Transparency through layers
At a glance, analysts see plain-language recommendations and key suitability signals. On demand, they can expand to learn why a model is suggested, and at the deepest level how it performs on technical metrics and training context.
Insights & Takeaways
- Explanations should be layered, so they serve generalists and experts without privileging either.
- Trust is calibrated socially: analysts rely on peer-validated experience over abstract metrics.
- Explainable interfaces should be responsive, matching explanation depth to the analyst's moment in the workflow.