8.2 AI Computer Vision for Unattended & Remote Interfaces

8.2 AI Computer Vision for Unattended & Remote Interfaces

Core Concept: UiPath AI Computer Vision (CV) is an artificial-intelligence-driven automation technology engineered to recognize, parse, and interact with user interfaces purely through visual appearance. Utilizing deep-learning convolutional neural networks (CNNs) paired with advanced Optical Character Recognition (OCR), Computer Vision interprets screens human-like: identifying UI elements (text boxes, buttons, checkboxes, tables) semantically based on their visual morphology rather than relying on underlying operating system accessibility trees or Document Object Models (DOM).

In an ideal enterprise automation environment, every application exposes a rich accessibility API—such as Microsoft UI Automation (UIA), Microsoft Active Accessibility (MSAA), or an accessible browser DOM—allowing software robots to extract precise, deterministic XML selectors. However, enterprise automations frequently operate in restrictive environments where accessibility trees are completely inaccessible. When automating over Virtual Desktop Infrastructure (VDI), secure remote terminal sessions, or legacy graphical runtimes, robots receive only a flat stream of graphical pixels. AI Computer Vision bridges this divide, delivering high-speed, resilient automation without requiring native selectors.

---

1. The Computer Vision Paradigm vs. Classic Image Automation

Prior to AI Computer Vision, automating remote desktop environments relied on classic image automation and basic OCR. These legacy techniques are notoriously brittle in enterprise production:

The Failure Modes of Classic Image Automation

• Pixel-by-Pixel Template Matching: Classic image activities (such as Click Image or Find Image) utilize normalized cross-correlation algorithms. If a single pixel shifts, or if display resolution, font anti-aliasing, color palettes, or rendering themes differ between the developer's laptop and the unattended runner VDI, template matching fails.
• DPI and Scaling Breakdowns: When an unattended robot logs into a remote virtual machine with a display scaling factor of 125% or 150% while the automation was recorded at 100%, classic image matching suffers catastrophic selector faults.
• Lack of Semantic Understanding: A classic image search has no concept of what a "text input field" or "checkbox" represents; it merely evaluates whether an array of RGB pixel values matches a reference bitmap.

How Deep-Learning AI Computer Vision Operates

UiPath AI Computer Vision replaces template matching with deep-learning neural networks trained on millions of enterprise UI screenshots across Windows, macOS, Linux, web frameworks, and terminal emulators. The neural network understands UI semantics:

1. Visual Element Classification: The CV engine analyzes visual contours, shadows, borders, and margins to classify components into functional types: Button, InputField, Checkbox, RadioButton, Dropdown, Table, or Icon.
2. Visual OCR Fusion: Simultaneous OCR processing (using UiPath Screen OCR, Google Cloud OCR, or OmniPage) extracts text labels, numbers, and symbols across the entire viewport.
3. Relational Context Mapping: The engine automatically correlates extracted text labels with adjacent input fields (e.g., recognizing that the text "First Name" positioned directly above or to the left of an empty rectangular box serves as the semantic anchor for that input).
4. Display Invariance: Because the neural network identifies high-level features and morphological patterns, it operates with complete invariance across font anti-aliasing shifts, minor color theme updates, and resolution variations.

---

2. Target Environments & Use Cases

Computer Vision is the definitive automation strategy for scenarios where native UI trees cannot be extracted:

---

3. The CV Screen Scope Architecture

Computer Vision reaches workflows in two ways. In the Modern experience, Computer Vision is one of the Unified Target methods of ordinary activities such as Click and Type Into, ranked after selectors and switched on per project or per element. The dedicated CV Screen Scope container and its CV activities are the older, CV-only way of working, and they are still useful for screens where no selector works at all.

+-----------------------------------------------------------------------------------+
|  CV SCREEN SCOPE                                                                  |
|  Target: Citrix Client - SAP GUI Window                                           |
|  Server Endpoint: UiPath cloud service, AI Center, or a local CV server         |
|  API Key: ********************                                                    |
|                                                                                   |
|  +-----------------------------------------------------------------------------+  |
|  | [Step 1: Capture Viewport Screenshot]                                       |  |
|  |                      │                                                      |  |
|  |                      ▼                                                      |  |
|  | [Step 2: Transmit to CV Server & Run Neural Network + OCR]                  |  |
|  |                      │                                                      |  |
|  |                      ▼                                                      |  |
|  | [Step 3: Generate & Store Local Visual Tree Cache in RAM]                   |  |
|  |                      │                                                      |  |
|  |                      ▼                                                      |  |
|  | [Child CV Activities Execute Instantly Against Local Cache]                 |  |
|  | - CV Type Into 'Username' (uses Anchor 'User ID:')                          |  |
|  | - CV Type Into 'Password' (uses Anchor 'Password:')                         |  |
|  | - CV Click 'Log On' Button                                                  |  |
|  +-----------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------+

Server Connection Options

The CV models run on a server that the robot calls:

1. UiPath cloud service. Robots connected to Automation Cloud can use UiPath's hosted Computer Vision service. Outbound HTTPS access from the robot machine is required.
2. AI Center or on-premises deployments. Organizations with data-residency requirements can host the Computer Vision model themselves and point the activities at that endpoint, supplying an API key where the setup requires one.
3. Computer Vision Local Server. UiPath documents a separately installed local server for running Computer Vision inside the corporate network.

Whichever option you use, screenshots of the target window are sent to that endpoint, so confirm the choice with your security team before automating sensitive screens.

Image Caching Mechanics: Why CV Is Fast

A common misconception is that Computer Vision makes expensive cloud or neural network queries for every individual click or keystroke. The CV Screen Scope eliminates this overhead through Visual Tree Caching:

• Upon entering the CV Screen Scope, the activity captures a single, full screenshot of the specified target window or bounding area.
• This screenshot is sent once to the CV engine along with the OCR request.
• The neural network detects and classifies every element across the entire screen, returning a structured JSON visual tree containing the bounding boxes, element types, and OCR text coordinates.
• This visual tree is stored in local volatile memory (RAM) as a cache.
• All subsequent child activities inside the scope—such as ten consecutive CV Type Into and CV Click activities—execute in milliseconds because they query the local in-memory cache directly without taking new screenshots or initiating network calls.

---

4. Core Computer Vision Activities

UiPath Studio provides a dedicated suite of activities tailored specifically for the Computer Vision engine:

---

5. Anchor-Based Targeting in Computer Vision

In real-world enterprise applications, interfaces frequently contain multiple identical or unlabeled input fields. For instance, a customer creation form might feature three empty text boxes arranged vertically for Home Phone, Mobile Phone, and Work Phone. If a robot searches purely for an InputField, it cannot determine which box corresponds to which phone number.

Computer Vision resolves this ambiguity using Visual Anchors:

• Visual Pairing: During element indication, the developer selects the target control (e.g., the empty text box) and links it to a distinct visual anchor (e.g., the static text label "Mobile Phone:").
• Spatial Relationship Modeling: The CV engine calculates relative spatial vectors (e.g., Target is positioned 15 pixels to the right of Anchor or Target is positioned 8 pixels below Anchor).
• Multi-Anchor Fallbacks: Developers can configure multiple anchors for a single target (e.g., anchoring an input box to both a field label on the left and a section header above). If one anchor shifts or is partially occluded, the secondary anchor guarantees deterministic element resolution.
• Fuzzy Label Matching: If an application's text label undergoes minor capitalization or typographical changes (e.g., "Mobile:" vs. "Mobile Phone:"), CV's OCR fuzzy matching maintains target identification without throwing runtime exceptions.

+---------------------------------------------------------------------+
|  ANCHOR-BASED TARGETING LOGIC                                       |
|                                                                     |
|  [Anchor: Text Label]              [Target: InputField]             |
|  +--------------------+            +-----------------------------+  |
|  | "Invoice Number:"  | ---------> | [                           ] |  |
|  +--------------------+  Relative  +-----------------------------+  |
|                          Spatial                                    |
|                          Vector                                     |
|                                                                     |
|  The CV Engine identifies "Invoice Number:" via OCR, locates the    |
|  closest InputField along the horizontal axis, and routes the click/ |
|  typing action directly into the target bounding box.               |
+---------------------------------------------------------------------+

---

6. Refreshing and Tuning CV Scopes

While the visual caching mechanism delivers exceptional execution speed, it introduces a critical design requirement: managing stale screen states.

The Stale Cache Dilemma and CV Refresh

Because the CV Screen Scope analyzes the interface once upon container entry, any user interaction that alters the visual structure of the screen renders the cached visual model obsolete. Examples include:

• Clicking a dropdown menu that displays an overlay list of selectable options.
• Clicking a "Next Page" button that loads a new batch of records into a table.
• Clicking an expandable accordion panel or tab control that renders new input fields.
• Submitting a form that prompts a modal confirmation dialog.

If an automation attempts to execute a child CV activity against an element that appeared after the scope was initialized, the robot will query the stale cache, fail to find the element, and throw an element-not-found error.

To resolve this, developers use the CV Refresh activity:

• When CV Refresh executes, it instructs the parent CV Screen Scope to capture a fresh screenshot of the target window.
• The new screenshot is transmitted to the CV engine and re-analyzed.
• The in-memory visual tree cache is immediately updated with the new controls and text labels, allowing subsequent CV activities to interact with the updated screen seamlessly.

Scope Tuning Best Practices

1. Optimize OCR Engine Selection:
   • UiPath Screen OCR: Specifically optimized for digital screen fonts, anti-aliased text, and low-contrast UI displays. Default and recommended for virtually all VDI and remote desktop automations.
   • OmniPage OCR / Google Cloud OCR: Ideal when remote applications display scanned document previews, PDF attachments, or non-standard character sets.
2. Configure Scope Informer & Accuracy:
   • Adjust accuracy settings (a scale of 0.0 to 1.0) based on environmental stability. Lowering accuracy helps with heavily compressed video streams, while raising it prevents false-positive matches on crowded interfaces.
3. Isolate Scope Boundaries:
   • Avoid setting the CV Screen Scope to monitor an entire 4K multi-monitor desktop. Constrain the scope's target selector to the specific application window or sub-panel being automated. This minimizes image payload sizes, reduces network transmission latency, and accelerates neural inference times.