Building automated speech recognition has become one of the most resource-intensive arms races in modern computing. For years, the industry standard has centered on scaling foundational transformer models across hundreds of thousands of hours of speech data to drive Word Error Rates closer to zero. Yet, as speech-to-text engines have become ubiquitously integrated into virtual meeting software, streaming platforms, and classroom tools, a fundamental limitation has become increasingly obvious to anyone relying on them for daily communication. While modern speech models are remarkably adept at transcribing vocabulary, they remain largely oblivious to the emotional tone, cadence, and urgency that give spoken language its actual meaning.

In high-stakes technical environments—such as engineering sprint retrospectives, architectural design reviews, or advanced university STEM lectures—spoken dialogue is rarely delivered as flat prose. A slight upward inflection can turn an apparent statement of fact into a skeptical question; a sudden drop in vocal pitch can signal a serious warning about a code vulnerability; and an urgent delivery can differentiate a critical design constraint from a casual suggestion. When standard automated speech recognition strips these acoustic cues away, leaving behind an uninflected block of text at the bottom of a display, it creates what Human-Computer Interaction (HCI) researchers call an intention gap. For deaf and hard-of-hearing professionals, the result is continuous cognitive strain, forcing them to guess speaker intent while rapidly darting their visual attention between slides, physical demonstrations, and disconnected caption windows.

At the University of California, Irvine and Virginia Tech, HCI researcher Dr. Sunday David Ubur has taken an alternative engineering path to address this challenge. Rather than treating speech recognition solely as a sequence-to-sequence text translation problem, Dr. Ubur approaches the challenge as an asynchronous distributed systems and spatial computing problem. Across several years of published laboratory work, his research has centered on constructing multimodal machine learning pipelines and spatial augmented reality frameworks capable of detecting and rendering non-verbal vocal cues in real time, without creating computational latency or overwhelming the user's working memory.

"The core issue with modern transcription isn't simply that words get misheard," Dr. Ubur explained in an interview. "The deeper problem is that speech-to-text algorithms throw away almost all of the acoustic information that carries human intent. Our goal was never to pile more visual noise onto the user. It was to find an architectural method to preserve the critical vocal cues that conventional captions discard, while keeping the user's cognitive workload as low as possible."

The first major systems bottleneck Dr. Ubur tackled was low-latency multi-stream inference within his platform, Interpretive Caption. Capturing vocal nuance in live conversation requires running deep acoustic feature extraction alongside continuous speech recognition without introducing perceptible delay or thread blocking. To solve this, Dr. Ubur architected a parallel pipeline that bifurcates incoming audio streams into two asynchronous processing tracks. One branch routes the audio through speech recognition transformers to maintain lexical transcription accuracy, while the second branch feeds raw acoustic waveforms into a specialized deep acoustic neural network fine-tuned across standard benchmark vocal datasets. This dual-pipeline setup extracts prosodic features directly from the audio signal, mapping vocal delivery across distinct emotional and paralinguistic categories in real time while preserving synchronization with the live speaker.

However, extracting deep acoustic signals in real time solved only half the engineering equation; the more difficult challenge lay in interface design and user cognitive load. Early experimental systems across the accessibility sector often made the mistake of cluttering displays with bright banners, large animated avatars, or intrusive graphical tags that quickly overwhelmed users during demanding analytical tasks. Drawing upon established cognitive models of working memory, Dr. Ubur structured his system around an on-demand progressive disclosure interface. By segmenting incoming captions into short, readable phrase units and transmitting them over asynchronous WebSockets, the architecture presents clean, continuous text while embedding lightweight single-letter tokens adjacent to relevant phrases. These tokens remain visually subordinate to the primary transcript, expanding into plain-language tooltips only when a user intentionally hovers over or focuses on them, ensuring that the interface never obstructs the flow of technical information.

Dr. Ubur applied a similar empirical rigor to immersive spatial computing within his EasyCaption framework. For years, teams developing consumer mixed-reality applications operated under the intuitive assumption that rendering expressive, animated digital avatars alongside text captions would create a more engaging experience for deaf users. To evaluate whether this design choice actually supported comprehension, Dr. Ubur ran a study inside virtual-reality head-mounted displays that combined electroencephalography (EEG) brain-wave monitoring with measures of user attention, engagement, and eye gaze. The resulting data indicated that animated virtual avatars reduced sustained attention and engagement—and slightly increased user stress—compared with cleaner caption layouts presented without avatars. The findings gave interface designers concrete empirical evidence that assistive systems in virtual environments should prioritize clean geometry and customizable density over decorative visual animations.

To reduce the split-attention penalty in physical environments such as lecture halls and collaborative laboratories, Dr. Ubur extended his captioning work into three-dimensional spatial augmented reality built with Unity and mixed-reality toolkits. Rather than requiring users to look away from a speaker down to a mobile device or laptop screen, his augmented reality prototype anchors floating, head-stabilized, emotionally annotated captions within the user's natural field of view, near the speaker. In preliminary evaluations, this AR-based approach improved comprehension and reduced cognitive effort compared with standard plain-text captions.

Dr. Ubur subsequently extended these spatial architectures into collaborative group settings through a speaker-aware captioning interface presented at the ACM CHI Conference on Human Factors in Computing Systems. In multi-speaker group discussions, failing to identify conversational turn-taking often leads to immediate communicative breakdown. By integrating multi-speaker audio diarization with confidence-gated affect flags and an on-demand contextual repair module driven by language models, the platform allows users to follow fast-paced group discussions and recover missed conversational context without losing track of the live dialogue.

The significance of Dr. Ubur's work has increasingly resonated beyond individual software prototypes, appearing at premier academic and technical venues including the ACM Conference on Human Factors in Computing Systems (CHI), the ACM SIGACCESS Conference on Computers and Accessibility (ASSETS), and the IEEE Conference on Virtual Reality and 3D User Interfaces (IEEE VR). The Rochester Institute of Technology's National Technical Institute for the Deaf (NTID) also invited him to present his research on AI-driven accessible captioning. His research path—progressing from competitive international fellowships to doctoral work at Virginia Tech's Center for Human-Computer Interaction, and on to his current postdoctoral research at the University of California, Irvine, where he leads independent research initiatives in accessible computing and classroom informatics—reflects a sustained focus on bridging the divide between theoretical artificial intelligence and practical accessibility engineering.

Beyond system development, Dr. Ubur's scholarship examines how the field itself evaluates accessibility. In an autoethnographic account of his transition from secondary school to university in Nigeria—undertaken without formal disability accommodations or a Deaf community—Dr. Ubur and his co-authors introduce the concept of self-accommodation: the access work displaced onto the disabled student when no institutional system exists to provide it, captured in the study's central image, “my body became the access infrastructure.” In a separate narrative review synthesizing dozens of peer-reviewed studies, he found that real-time speech-to-text tools still largely fail to convey affective nuance, and that emotion-aware captioning within immersive extended-reality environments remains a persistent, under-addressed gap—an evidence-based case for building affect-responsive, user-centered captioning rather than plain-text transcription alone.

"Accessibility is frequently treated as an optional feature or a compliance checkbox that gets tacked on at the end of a software cycle," Dr. Ubur noted. "In reality, when you engineer systems for edge-case environments and users with distinct sensory constraints, you are forced to solve the hardest latency, cognitive load, and interface problems first. The resulting architectures end up being faster, more resilient, and far more usable for everyone."

By approaching accessible communication through the rigorous lens of low-latency systems engineering, neuro-ergonomics, and spatial computing, Dr. Ubur's research demonstrates that speech technology still has significant room to evolve. As intelligent interfaces and spatial headsets become standard tools across classrooms and workplaces, the systems that truly succeed will not just be those that transcribe raw vocabulary, but those that capture and convey the full context of human dialogue.

This article was published under HackerNoon's Business Blogging program.