7.2 Anchors, Relative Elements & UI Automation Activities

7.2 Anchors, Relative Elements & UI Automation Activities

Core Concept: In dynamic web applications and enterprise ERP portals, individual UI elements frequently lack unique, permanent identifiers. An input field might be rendered simply as <input class='form-control' type='text'> with no name, unique ID, or accessibility tag. Anchors solve this fundamental automation challenge by establishing relative spatial and structural relationships between a volatile target element and one or more stable nearby reference elements (such as text labels, column headers, or icons).

In the Classic Design Experience, anchoring required wrapping action activities inside a cumbersome Anchor Base container, which supported only a single anchor and frequently failed when responsive web designs shifted layouts. The Modern Design Experience incorporates native multi-anchor targeting directly into every UI activity's descriptor. Combined with a redesigned activity suite centered around Use Application/Browser, modern activities deliver deterministic execution, self-contained verification, and unified application lifecycle management.

---

1. Anchors in Modern UI Automation

An Anchor is a stable UI element on the screen that helps the robot identify a target element that cannot be uniquely or reliably identified on its own.

+-------------------------------------------------------------------------+
| FORM CONTAINER                                                          |
|                                                                         |
|   [Anchor: Stable Label]                 [Target: Dynamic Input Field]   |
|   +-----------------------+              +---------------------------+  |
|   |  "First Name:"        | ───────────► |  <input type='text' />    |  |
|   |  (tag='LABEL', id='lbl')             |  (No ID, class='input')   |  |
|   +-----------------------+   Spatial    +---------------------------+  |
|                               Vector                                    |
+-------------------------------------------------------------------------+

How Anchoring Works Internally:

1. Descriptor Binding: When indicating an element, Studio automatically searches for nearby static text or structural landmarks and recommends candidates as anchors (highlighted in blue, while the target is highlighted in green).
2. Relative Spatial Vector: The framework records the relative spatial vector between the anchor and the target (horizontal distance, vertical distance, angle, and relative quadrant).
3. Runtime Triangulation: During execution, the robot locates the stable anchor first (using strict, fuzzy, or image targeting). Once the anchor is verified, the robot projects the spatial vector to locate the target field within a calculated bounding region, ignoring duplicate input fields located elsewhere on the page.

Modern Anchors vs. Classic Anchor Base

---

2. Multi-Anchor Targeting Mechanics

While a single anchor suffices for simple linear forms, complex enterprise applications—such as SAP GUI tables, Salesforce record grids, and multi-column financial forms—often present multiple elements sharing identical labels. In these scenarios, Multi-Anchor Targeting provides deterministic triangulation.

+-------------------------------------------------------------------------+
| TABULAR INVOICE MATRIX                                                  |
|                                                                         |
|                          [Anchor 2: Column Header]                      |
|                          +-----------------------+                      |
|                          |  "Unit Price"         |                      |
|                          +-----------------------+                      |
|                                      │                                  |
|                                      │ Vertical Vector                  |
|  [Anchor 1: Row Identifier]          ▼                                  |
|  +-----------------------+       +---------------------------+          |
|  |  "Item SKU: A-4910"   | ────► |  [Target: Cell Input]     |          |
|  +-----------------------+       |  $149.50                  |          |
|      Horizontal Vector           +---------------------------+          |
+-------------------------------------------------------------------------+

Multi-Anchor Scenarios:

• Tabular and Grid Intersections: In an editable data grid, targeting a specific cell by its row identifier (Anchor 1: "Item SKU: A-4910") and column header (Anchor 2: "Unit Price"). The intersection of both vectors pinpoints the exact cell, even if rows are sorted dynamically or columns rearranged.
• Duplicated Section Labels: In an onboarding form containing both "Billing Address" and "Shipping Address" sections, both sections feature a field labeled "Postal Code". By binding the input field to both "Postal Code" (Anchor 1) and "Billing Address" (Anchor 2), the robot disambiguates the billing field from the shipping field.
• Visual Icon + Text Pairing: Binding a dynamic action link to both an adjacent descriptive label and a static corporate logo or status badge icon.

---

3. The Modern UI Activity Suite

Modern UI Automation redesigns fundamental interaction activities to operate seamlessly inside the Use Application/Browser container.

Use Application/Browser Scope Container

The Use Application/Browser activity is the foundational container for modern web and desktop automations. It establishes a managed execution context for all child activities, eliminating the need for classic Open Browser, Attach Window, or Kill Process sequences.

Key Operational Properties:

• Open Directive:
  • Always: Launches a new application process or browser window every time the container executes.
  • IfNotOpen (Default): Checks if an instance matching the selector is already running. If found, attaches to the existing window; if not found, launches a fresh instance.
  • Never: Assumes the target application is already open and running. If no matching window is discovered within the timeout, throws an exception without attempting launch.
• Close Directive:
  • Always: Terminates the application or closes the browser tab immediately upon exiting the container scope.
  • IfOpenedByAppBrowser (Default): Tracks whether the application was launched by this specific activity instance. If the robot launched it, the container closes it upon completion; if the robot attached to a pre-existing session, it leaves the window open.
  • Never: Leaves the application running regardless of how it was initiated.
• Window attach mode:
  • Single window: Child activities interact strictly with the specific window indicated.
  • Application: Child activities can interact with any window, popup, or child dialog spawned by the application process.

---

Modern Action Activities

1. Click

The modern Click activity executes mouse click events against a Unified Target. Beyond standard coordinates, it features built-in verification:

• Click type: Single, Double, Down, or Up (enabling custom drag-and-drop sequences).
• Mouse button: Left, Right, or Middle.
• Key modifiers: Alt, Ctrl, Shift, or Win (e.g., executing Ctrl + Click to select multiple list items).
• Cursor motion type: Instant (default, teleports cursor directly) or Smooth (simulates realistic human-like cursor acceleration curves, essential for fraud-detection systems and anti-bot protections).

2. Type Into

Automates keyboard input with deterministic text handling:

• Text vs. SecureText: Accepts either standard String input or encrypted System.Security.SecureString data directly from credential assets, preventing plaintext passwords from exposing in memory or logs.
• Empty field: Chooses how existing text is removed before typing:
  • None: Keeps the existing content.
  • Single line (End, Shift+Home, Del): The default; clears a single-line field.
  • Multi line (Ctrl+A, Del): Clears a multi-line field.
  • Empty field, Click before typing, and Delay between keys cannot be used with the Simulate input mode.
• Click before typing: None, Single (the default), or Double, which ensures the field has focus before typing.
• Activate before typing: Focuses and brings the target window to the foreground if hardware input is required.

3. Hover

Moves the cursor over a target element to trigger hover-dependent UI behaviors (such as CSS :hover states, JavaScript mouse-over navigation menus, or floating tooltips).

• Requirement: Must be executed using Hardware Events or Chromium API. Simulate mode does not support mouse hover because it lacks cursor position emulation.

4. Check/Uncheck

Enforces a deterministic state (Checked or Unchecked) on checkboxes, radio buttons, and toggle switches.

• Deterministic Operation: Unlike a raw Click activity—which inverts the current state—Check/Uncheck inspects the element's accessibility state first. If a checkbox is already checked and the activity is set to Check, it performs no action, preventing accidental unchecking.

---

4. Scrolling, Synchronization & Element State Verification

Enterprise web pages frequently utilize virtualized scrolling or lazy loading, where DOM elements do not exist in the visual viewport until scrolled into view.

Automated Auto-Scrolling

Modern UI activities inside Use Application/Browser incorporate automated auto-scrolling. If a target element is located outside the active viewport, the execution engine automatically dispatches mouse-wheel or DOM scroll commands to bring the target into view prior to evaluating selectors or image templates. Developers can fine-tune this behavior via the Scroll to element property.

Check App State Activity

In Classic automation, checking if an element existed required dragging in an Element Exists activity, assigning the output to a boolean variable, and building an If statement. If an unexpected error banner appeared, complex exception wrappers were needed.

The modern Check App State activity consolidates element synchronization into a structured, dual-branch control flow:

+-------------------------------------------------------------------------+
| CHECK APP STATE: "Search Results Table"                                 |
| Target: Unified Target [Timeout: 10s]                                    |
|                                                                         |
|   ├── [Branch: Target appears]                                          |
|   │   - Executes immediately when the element is detected on screen     |
|   │   - Performs data extraction, row processing, or downloads          |
|   │                                                                     |
|   └── [Branch: Target does not appear]                                  |
|       - Executes if timeout elapses without element detection           |
|       - Performs fallback search, error logging, or recovery logic       |
+-------------------------------------------------------------------------+

Check App State can wait for an element to appear (branches Target appears and Target does not appear) or to disappear (branches Target disappears and Target does not disappear), for example waiting for a loading spinner to vanish before continuing.

Built-in Verify Execution

Many modern action activities (Click, Type Into) feature a Verify execution property section. Developers specify an expected UI change that must occur as a direct consequence of the action:

• Verification Target: An element that should appear (e.g., a "Success" notification banner), disappear (e.g., a modal dialog), or change its text.
• Verification Behavior: The activity waits up to the verification Timeout (10 seconds by default) for the expected change. If Retry is selected, the action is repeated, followed by the check, for the duration of the activity timeout; if the change still does not happen, the activity throws an exception.

---

5. Comprehensive Activity Mapping & Comparison

The following reference matrix outlines the direct progression from Classic activities to their Modern UI Automation counterparts: