4.3 Azure AI Speech Service
Key Takeaways
- Azure AI Speech provides four capabilities: speech-to-text (STT), text-to-speech (TTS), speech translation, and speaker recognition, all via the Speech SDK and REST APIs.
- Speech-to-text supports real-time and batch transcription, custom speech for domain vocabulary, and speaker diarization to separate speakers.
- Text-to-speech uses 600+ neural voices across 150+ languages/locales, supports SSML for fine control, and offers gated custom neural voice.
- Speech translation runs recognize -> translate -> (optionally) synthesize, so it can output translated text or translated speech.
- Speaker verification is 1:1 (confirm a claimed identity); speaker identification is 1:many (determine who an unknown speaker is).
Quick Answer: Azure AI Speech provides four core capabilities: speech-to-text (convert audio to text), text-to-speech (convert text to audio), speech translation (translate spoken language), and speaker recognition (identify who is speaking). All capabilities are available via REST APIs and the Speech SDK.
What Is Azure AI Speech?
Azure AI Speech is a cloud-based service that provides a comprehensive set of speech-related AI capabilities. It handles the audio side of NLP — converting between speech and text, translating spoken language, and identifying speakers. Where Azure AI Language works on text, Azure AI Speech works on the spoken word.
Core Capabilities
1. Speech-to-Text (STT)
What it does: Converts spoken audio into written text (transcription).
| Feature | Description |
|---|---|
| Real-time transcription | Transcribe audio as it is spoken (live captioning, voice commands) |
| Fast / batch transcription | Transcribe pre-recorded audio files (meeting recordings, podcasts) |
| Custom speech | Train on your vocabulary and acoustic environment for better accuracy |
| Multi-language | Supports 100+ languages and dialects |
| Speaker diarization | Separate speakers in a conversation ("Speaker 1:", "Speaker 2:") |
Use cases: live meeting captioning, voice-controlled apps, call-center transcription for quality analysis, accessibility captioning for deaf or hard-of-hearing users, and dictation.
2. Text-to-Speech (TTS)
What it does: Converts written text into natural-sounding speech audio using neural voices.
| Feature | Description |
|---|---|
| Neural voices | 600+ natural-sounding prebuilt voices across 150+ languages and locales |
| Custom neural voice | Create a unique voice for your brand (limited/gated access, requires responsible-AI review) |
| SSML support | Fine-tune pronunciation, pitch, rate, and emphasis using Speech Synthesis Markup Language |
| Visemes | Synchronized mouth shapes for animated avatars |
Use cases: screen readers and accessibility tools, audiobook generation, Interactive Voice Response (IVR) systems, voice-enabled chatbots, e-learning narration, and public-announcement systems.
Two voice tiers worth knowing: standard neural voices (the broad prebuilt catalog) and newer high-definition (HD) / Dragon HD voices that add greater expressiveness and emotional range. For the exam, the headline fact is simply that TTS uses neural voices that sound human, not the older robotic concatenative voices.
3. Speech Translation
What it does: Translates spoken audio from one language to another, in near real time.
| Feature | Description |
|---|---|
| Speech-to-text translation | Spoken language -> translated text |
| Speech-to-speech translation | Spoken language -> translated spoken audio |
| Multi-language | Many languages supported for speech translation |
| Real-time | Near-instant translation for live conversations |
How it works — three steps:
- Recognize the spoken input (speech-to-text)
- Translate the recognized text to the target language
- Synthesize the translated text (text-to-speech) — only for speech-to-speech translation
Use cases: real-time translation in international meetings, travel assistance (speak in your language, output in the local language), multilingual customer service, and cross-language collaboration tools.
4. Speaker Recognition
What it does: Identifies or verifies a person based on their unique voice characteristics.
| Mode | Description | Matching |
|---|---|---|
| Speaker verification | Confirm a claimed identity | "Is this person who they say they are?" (1:1) |
| Speaker identification | Identify an unknown speaker | "Who is this person?" (1:many) |
Use cases: voice-based authentication (banking, security), meeting transcription (who said what), call-center caller identification, and smart-speaker personalization.
Custom Speech Models
Azure AI Speech can be customized for better accuracy in specific environments:
| Customization | What It Does | When to Use |
|---|---|---|
| Custom speech-to-text | Train on your domain vocabulary and acoustic conditions | Medical, legal, or technical terminology |
| Custom neural voice | Create a unique branded voice | Brand identity, specific character voices |
| Pronunciation assessment | Evaluate speech pronunciation accuracy | Language-learning applications |
A common exam scenario: a hospital wants accurate transcription of drug names and clinical terms. The default model struggles with rare vocabulary, so the answer is custom speech — train on the domain words, not switch services.
Azure AI Speech vs. Azure AI Language
| Aspect | Azure AI Speech | Azure AI Language |
|---|---|---|
| Input | Audio / speech | Text |
| Primary tasks | STT, TTS, speech translation, speaker recognition | Sentiment, NER, key phrases, CLU |
| Focus | Converting between audio and text | Understanding text meaning |
The two are often chained: Speech converts a caller's audio to text -> Language analyzes intent and sentiment -> a response is generated -> Speech converts the response back to audio. Knowing this pipeline answers many "which combination of services?" questions.
Common Exam Scenarios
| Scenario | Capability |
|---|---|
| Add live captions to a webinar | Speech-to-text (real-time) |
| Read web content aloud for accessibility | Text-to-speech |
| Build an IVR phone menu with a natural voice | Text-to-speech (often with SSML) |
| Translate a spoken conversation between two people | Speech translation |
| Confirm a caller is the account holder by voice | Speaker verification |
| Transcribe a podcast archive overnight | Speech-to-text (batch) |
| Improve recognition of legal jargon | Custom speech |
A subtle distinction the exam likes: speech translation (audio in, translated text or audio out) lives in Azure AI Speech, whereas Azure AI Translator only handles text. If the input is spoken, think Speech; if the input is written, think Translator.
On the Exam: Speech handles audio-to-text and text-to-audio conversions; Language analyzes text. Speech translation (audio) is different from Azure AI Translator (text). Speaker verification is 1:1; speaker identification is 1:many.
Which Azure AI Speech capability converts spoken audio into written text?
A company wants to create an accessibility feature that reads web page content aloud to visually impaired users. Which Azure AI Speech capability should they use?
A banking app wants to verify a customer's identity by comparing their voice to a stored voice sample. Which Azure AI Speech capability is needed?
How does speech translation work in Azure AI Speech?
A hospital needs Azure AI Speech to accurately transcribe rare drug names and clinical terms that the default model frequently mis-hears. What is the best approach?
Match each Azure AI Speech capability to its use case:
Match each item on the left with the correct item on the right