WhisperNER: Unified Open Named Entity and Speech Recognition

Integrating named entity recognition (NER) with automatic speech recognition (ASR) can significantly enhance transcription accuracy and enrich its content. We introduce WhisperNER, a novel model that facilitates joint speech transcription and entity recognition. WhisperNER supports opentype NER, enabling recognition of various entities during inference. Building on recent advancements in open NER research, we augment a large synthetic dataset with synthetic speech samples. This approach enables us to train WhisperNER on numerous examples with various NER tags. During training, the model is prompted with NER labels and optimized to produce the transcribed utterance alongside the corresponding tagged entities. For evaluation, we generate synthetic speech for commonly used NER benchmarks and annotate existing ASR datasets with open NER tags. Our experiments show that WhisperNER outperforms natural baselines in both out-of-domain open-type NER and supervised fine-tuning.