4.6 Question Answering and Conversational AI

Quick Answer: Custom Question Answering (CQA) replaces QnA Maker and builds knowledge bases from URLs, files, manual pairs, and chit-chat. It supports multi-turn follow-up prompts, active learning, a confidenceScoreThreshold, and a defaultAnswer fallback. To combine FAQ answering with action intents, use the orchestration workflow in CLU, then surface the bot via Azure Bot Service.

Custom Question Answering vs. QnA Maker

On the Exam: QnA Maker was retired; new builds and migrations target Custom Question Answering. A frequent distractor pairs "QnA Maker" with Azure Cognitive Search — CQA still uses a search index under the hood, but you provision and manage it through the Language resource.

Knowledge Base Sources

Each answer can carry alternate questions (synonymous phrasings), metadata filters (e.g. product=widget), and follow-up prompts.

Querying with Thresholds and Fallback

query_body = {
  "question": "What is your return policy?",
  "top": 3,
  "projectName": project, "deploymentName": "production",
  "confidenceScoreThreshold": 0.5,          # below this -> no answer
  "includeUnstructuredSources": True}
r = requests.post(f"{ep}/language/:query-knowledgebases?api-version=2023-04-01",
                  headers=headers, json=query_body).json()
for a in r["answers"]:
    print(a["answer"], round(a["confidenceScore"], 2), a["source"])

If no answer clears confidenceScoreThreshold, the project's defaultAnswer is returned — tune the threshold up to reduce wrong answers, down to increase coverage.

Multi-Turn Conversations

Follow-up prompts chain Q&A pairs so a broad question narrows over several turns:

User: How do I return a product?
Bot:  Returns must start within 30 days. In-store or online?
User: Online
Bot:  Log in -> Order History -> Return -> print label. Want refund timing?

Each prompt links a parent answer to a child Q&A pair, forming a conversation tree. The client passes the prior qnaId as context so the service follows the correct branch.

Active Learning

Active learning watches real traffic: when users phrase a question in a way that partially matches several pairs, the service clusters those queries and suggests alternate questions to the editor. The editor accepts or rejects; accepted phrasings become alternate questions, raising future match rates — no full retrain needed.

Orchestration Workflow

A single bot often must both answer FAQs and perform actions. The orchestration workflow, authored in CLU, is a top-level router:

[User input] -> [Orchestration model]
   |- intent BookFlight -> CLU flight model
   |- intent CompanyFAQ -> Custom Question Answering KB
   |- intent ChatGPT     -> Azure OpenAI / LLM
   '- None               -> default response

The orchestrator predicts which connected project should handle the utterance and forwards it. This is the canonical pattern to combine intent recognition with knowledge-base answering in one experience.

Deployment

A deployed knowledge base is exposed through Azure Bot Service, which adds channels — Microsoft Teams, Web Chat, Direct Line, Slack, Facebook — without rewriting the bot. The bot calls the CQA/CLU prediction endpoints using its own credentials.

Common Trap: Active learning suggests alternate phrasings from live queries — it does not retrain a deep model, auto-deploy, or translate. And orchestration is the answer whenever a scenario needs a single assistant that handles both questions (route to CQA) and commands (route to CLU); a lone CLU or lone CQA project cannot do both well.

Sources, Precise Answers, and Tuning

A strong knowledge base mixes sources deliberately. Importing a public FAQ URL bootstraps dozens of pairs automatically, files such as product PDFs add depth, manual pairs handle edge cases the documents omit, and a chit-chat personality keeps the bot from sounding robotic when users greet it or thank it. Custom Question Answering also offers precise answering, which extracts a short span from a longer passage so the bot returns a sentence rather than a wall of text; this is one of the headline improvements over the retired QnA Maker and a common exam talking point.

Tuning revolves around the confidence score threshold and the default answer. Raising the threshold makes the bot answer only when it is sure, trading coverage for accuracy and pushing borderline questions to the fallback; lowering it does the opposite. The default answer is your safety net for everything below the bar, and writing it to invite escalation such as offering to connect the user to support is better practice than leaving the generic placeholder. Metadata filters let one knowledge base serve multiple products by tagging pairs and filtering at query time.

Deployment and the Bigger Picture

Once a project is trained and deployed, Azure Bot Service wraps it so the same logic reaches Microsoft Teams, Web Chat, Direct Line, and more without rewriting the bot. In modern designs the orchestration workflow can even route some utterances to Azure OpenAI for open-ended generation while keeping deterministic FAQ answers in the knowledge base, blending retrieval with generation.

For the exam, anchor on these mappings: QnA Maker is retired in favor of Custom Question Answering, follow-up prompts power multi-turn dialogs, active learning improves matching from real traffic, the orchestration workflow routes between intents and Q&A, and Azure Bot Service publishes to channels.