3.1 Computer Vision Concepts and Tasks
Key Takeaways
Computer vision enables AI to interpret and extract information from images and videos — it is how machines "see."
Image classification assigns a single label to an entire image (e.g., "cat" or "dog"); it answers "What is in this image?"
Object detection identifies AND locates objects within an image using bounding boxes — it answers "Where are objects in this image?"
Semantic segmentation classifies every pixel in an image, creating a detailed map of what is where — used in autonomous driving.
OCR (Optical Character Recognition) extracts text from images, converting visual text into machine-readable text.
Quick Answer: Computer vision enables machines to interpret images and videos. The key tasks are image classification (label an entire image), object detection (find and locate objects with bounding boxes), semantic segmentation (classify every pixel), OCR (extract text from images), and facial analysis (detect and analyze faces). Each task answers a different question about the visual content.
What Is Computer Vision?
Computer vision is a field of AI that enables computers to interpret and extract meaningful information from visual data — images and videos. It is how machines "see" and understand the visual world.
Computer vision powers applications you use every day:
- Unlocking your phone with your face
- Depositing checks by photographing them
- Self-driving cars navigating roads
- Security cameras detecting intruders
- Medical imaging detecting tumors
- Barcode and QR code scanning
Core Computer Vision Tasks
Understanding the differences between these tasks is critical for the AI-900:
1. Image Classification
What it does: Assigns a single label to an entire image.
Question it answers: "What is in this image?"
Output: A class label with a confidence score.
| Input Image | Predicted Label | Confidence |
|---|---|---|
| Photo of a golden retriever | "Dog" | 95.2% |
| Photo of a tabby cat | "Cat" | 98.7% |
| Chest X-ray | "Pneumonia" | 87.3% |
| Satellite image | "Urban" | 91.5% |
Use cases:
- Medical image screening (disease vs. healthy)
- Quality inspection in manufacturing (defective vs. acceptable)
- Content moderation (appropriate vs. inappropriate)
- Wildlife monitoring (species identification)
- Plant disease detection (healthy vs. infected)
2. Object Detection
What it does: Identifies AND locates multiple objects within an image using bounding boxes.
Question it answers: "What objects are in this image, and where are they?"
Output: A list of detected objects, each with a class label, confidence score, and bounding box coordinates (x, y, width, height).
| Input Image | Detected Objects |
|---|---|
| Street scene | Car (0.97) at [120, 200, 300, 180], Person (0.93) at [450, 150, 80, 200], Dog (0.85) at [600, 350, 60, 50] |
| Grocery shelf | Apple (0.99) at [50, 100, 80, 80], Banana (0.96) at [200, 100, 120, 40] |
Use cases:
- Autonomous vehicles (detect pedestrians, cars, signs)
- Retail analytics (count products on shelves)
- Security and surveillance (detect people, vehicles)
- Medical imaging (locate tumors, lesions)
- Wildlife monitoring (count and locate animals)
3. Semantic Segmentation
What it does: Classifies every pixel in an image, creating a detailed map of the scene.
Question it answers: "What class does every pixel in this image belong to?"
Output: A pixel-by-pixel classification map where each pixel is assigned a category.
Use cases:
- Autonomous driving (road, sidewalk, car, pedestrian, building — for every pixel)
- Medical imaging (precise tumor boundaries)
- Land use mapping (forest, water, urban, agricultural — from satellite images)
- Augmented reality (separate foreground from background)
4. OCR (Optical Character Recognition)
What it does: Extracts text from images, converting visual text into machine-readable text.
Question it answers: "What text is in this image?"
Output: Extracted text with position information.
Use cases:
- Reading documents (contracts, forms, letters)
- License plate recognition
- Receipt and invoice digitization
- Sign and label reading
- Handwriting recognition
- Business card scanning
5. Facial Detection and Analysis
What it does: Detects human faces in images and optionally analyzes attributes.
Question it answers: "Are there faces in this image, and what can you tell about them?"
Output: Face locations and optional attributes.
| Attribute | Description |
|---|---|
| Face location | Bounding box around each face |
| Age | Estimated age (limited; needs Microsoft approval) |
| Head pose | Rotation angles (pitch, roll, yaw) |
| Glasses | Whether the person wears glasses |
| Facial landmarks | Key points (eyes, nose, mouth, jaw) |
Important: Microsoft has retired the Face API capabilities that infer emotion and gender, and limits age, smile, facial hair, hair and makeup to approved use cases, due to responsible AI concerns. The current Face API focuses on face detection, verification, and identification — not emotion or gender inference.
Comparison Table: Key Computer Vision Tasks
| Task | Input | Output | Question Answered |
|---|---|---|---|
| Image Classification | Whole image | Single label + confidence | "What is this image?" |
| Object Detection | Whole image | Multiple objects + bounding boxes | "Where are objects in this image?" |
| Semantic Segmentation | Whole image | Per-pixel classification map | "What is every pixel in this image?" |
| OCR | Image with text | Extracted text + positions | "What text is in this image?" |
| Facial Detection | Image with faces | Face locations + attributes | "Where are faces and what do they look like?" |
On the Exam: The most common question type is: "A company needs to [scenario]. Which computer vision task should they use?" Focus on the output: single label = classification, located objects = detection, pixel map = segmentation, extracted text = OCR, face analysis = facial detection.
A retail company wants to count the number of each product type on store shelves by identifying and locating products in photographs. Which computer vision task should they use?
Image classification
OCR
Semantic segmentation
Object detection
Which computer vision task classifies every pixel in an image?
Image classification
Object detection
Semantic segmentation
Facial detection
A bank wants to automatically read account numbers and amounts from paper checks that customers photograph with their phones. Which computer vision task is needed?
Image classification
Object detection
OCR (Optical Character Recognition)
Semantic segmentation
What is the key difference between image classification and object detection?
Image classification assigns one label to the whole image; object detection locates multiple objects with bounding boxes
Image classification works with color images; object detection works with black and white
Object detection is faster than image classification
Image classification uses deep learning; object detection uses traditional algorithms
Match each computer vision task to its primary output:
Match each item on the left with the correct item on the right
Image classification
Object detection
Semantic segmentation
OCR
Facial detection
Image Classification vs. Object Detection vs. Segmentation: The Granularity Ladder
The three location-aware vision tasks form a ladder of increasing detail. Pinning down where each sits is one of the most reliable points on the exam:
| Granularity | Task | Output | "Where" Information |
|---|---|---|---|
| Whole image | Image classification | One label | None — just what the image is |
| Region | Object detection | Boxes + labels | Bounding box per object |
| Pixel | Semantic segmentation | Pixel map | Exact shape outline of each region |
A quick scenario test: "Is there a tumor in this scan?" → classification. "Where is the tumor?" (a box) → detection. "What is the exact boundary of the tumor?" (precise outline) → segmentation.
OCR: Two Things It Can Read
OCR (optical character recognition) handles both printed and handwritten text and returns not only the characters but their position on the page, so downstream systems can map text back to where it appeared. Typical inputs are photos of signs, scanned forms, license plates, receipts, and handwritten notes. When a question adds structure — "extract the invoice number and total" rather than just "read the text" — that crosses from plain OCR into Azure AI Document Intelligence, which understands fields and tables, not just characters.
Facial Detection vs. Recognition (and the Responsible-AI Line)
Facial detection simply finds that a face is present and returns its bounding box and landmarks. Facial recognition goes further to verify (is this the same person? 1:1) or identify (who is this among a known group? 1:many). The exam treats these as distinct because recognition carries higher privacy and surveillance risk.
Important correction to legacy material: Microsoft has retired the Face service's ability to infer emotion and gender (age is now limited to approved use cases) because facial appearance does not reliably indicate those attributes and the inferences can be biased. Older study guides that list "emotion detection" as a current Face capability are out of date. The current service focuses on detection, verification, identification, and liveness (anti-spoofing).
Video and the Edge
Computer vision extends to video. Microsoft's video offering for scene detection, transcription, and face tracking across frames is Azure AI Video Indexer. Azure AI Vision's Spatial Analysis feature, which counted people in live camera feeds on IoT Edge devices, was retired on March 30, 2025.
On the Exam: Decide by the question's verb and output. "Label the image" = classification; "locate and count objects" = object detection; "outline every region precisely" = segmentation; "read the text" = OCR; "find/verify a person" = Face. Spatial Analysis, the retired Azure AI Vision people-counting feature, is no longer a current answer.
Sections you finish are checked off in the contents.