Section 3.1: Ingestion Methods and Service Graph Connectors

Ingestion Methods and Service Graph Connectors

1. Introduction to ServiceNow CMDB Data Ingestion

Data ingestion is the lifecycle foundation of the Configuration Management Database (CMDB). Without reliable and standardized methods to populate Configuration Items (CIs) and their relationships, the CMDB quickly becomes a repository of stale, duplicate, or inaccurate information. ServiceNow offers two primary pathways for importing data into the CMDB: legacy Import Sets and modern Service Graph Connectors (SGCs). In the Certified Implementation Specialist - Data Foundations (CIS-DF) exam, a core focus is understanding the operational behavior, configuration requirements, and platform governance trade-offs of these two approaches.

2. Legacy Import Sets and Transform Maps

Legacy data ingestion relies on Import Sets and Transform Maps (sys_transform_map). When importing data using this method, records are first loaded from a source system (such as JDBC, LDAP, CSV, or XML) into a flat import set staging table that extends the sys_import_set_row base table. Once the staging table is populated, a Transform Map executes to determine how fields from the staging table map to fields on target CMDB tables (e.g., mapping u_hostname to name on the cmdb_ci_server class).

The critical architectural vulnerability of legacy Transform Maps is that they bypass the Identification and Reconciliation Engine (IRE) by default. Instead, standard Transform Maps rely on basic "coalesce" fields (e.g., mapping by hostname or serial number) to decide whether to insert a new record or update an existing one. This bypass creates several risks:

• Data Duplication: If different integrations use slightly different coalesce criteria, duplicate CIs will be created for the same physical asset.
• Data Overwrites and Corruption: Legacy transform maps do not respect data source authority or reconciliation rules. A lower-integrity data source (such as a manual spreadsheet import) can easily overwrite attributes populated by a high-integrity source (such as ServiceNow Discovery), corrupting the "Golden Record."
• Lack of Class Hierarchy Logic: Transform maps write directly to target tables without evaluating parent-child class hierarchies or validating dependent relationships.

To force a legacy Transform Map to use the IRE, implementation specialists must write complex, custom scripts within the Transform Map (e.g., using onBefore scripts) that intercept the data and call the SNC.IdentificationEngineScriptableApi scriptable API manually. This custom scripting is prone to errors, hard to maintain, and negatively impacts platform upgradeability.

3. Service Graph Connectors (SGC)

To establish a modern, governed, and standardized data ingestion framework, ServiceNow introduced Service Graph Connectors (SGCs). Service Graph Connectors are pre-packaged, certified integrations developed in collaboration with leading third-party vendors (such as Microsoft SCCM, AWS, Azure, Microsoft Intune, Dynatrace, and SolarWinds).

SGCs are designed specifically to ingest data into the CMDB while ensuring absolute alignment with the Common Service Data Model (CSDM). SGCs deliver several enterprise benefits:

• ServiceNow Certified and Maintained: SGCs are vetted and certified by ServiceNow, meaning they are guaranteed to work seamlessly across major platform upgrades without breaking custom integrations.
• Native IRE Enforcement: SGCs never bypass the IRE. They are built to route all ingested payloads directly through the IRE API, ensuring that every import respects identification rules, reconciliation rules, and data source precedence.
• CSDM-Compliant Schema Mapping: Data is automatically mapped to the correct classes and relationship structures out-of-the-box. For example, virtual machine relationships to host hardware are mapped exactly as defined in the CSDM Design and Manage Technical Services domains.
• High-Performance Ingestion: SGCs utilize IntegrationHub ETL and Robust Transform Engine (RTE) frameworks to process data in optimized batches, significantly reducing CPU usage and database lock times.

4. Robust Transform Engine (RTE) and IntegrationHub ETL

Under the hood, Service Graph Connectors do not use standard Transform Maps. Instead, they are powered by the Robust Transform Engine (RTE). While standard Transform Maps map data on a flat, row-by-row basis, the RTE processes data in sequential, structured operations (e.g., clean, transform, map, associate).

The RTE defines entities and mappings in the following system tables:

• sys_rte_eb_definition (Entity Definition)
• sys_rte_eb_entity (RTE Entity)
• sys_rte_eb_field (RTE Field)
• sys_rte_eb_mapping (Entity Mapping)

The RTE processes raw staging table records and transforms them into a nested, structured JSON payload. This payload contains the identified CIs, their attributes, and their relationships in the exact format required by the IRE. The RTE then passes this JSON payload to the IRE for final validation and write operations.

IntegrationHub ETL (Extract, Transform, Load) is the low-code, user-friendly graphical interface used to view, build, and customize RTE definitions. Within IntegrationHub ETL, administrators can:

• Preview raw staging table data and see how it looks when mapped.
• Perform clean-up operations (such as regex replaces, casing changes, or splits) without writing JavaScript.
• Map source columns to target CMDB classes.
• Define CI-to-CI relationships (such as "Runs on" or "Hosted on") visually.
• Test and preview the resulting IRE JSON payload before active deployment.

5. Metadata Mapping: The sys_object_source Table

When data is processed through the IRE, the engine tracks the relationship between the external source and the internal CI in the sys_object_source table. This table acts as a critical metadata ledger and records:

• name: The name of the discovery source (e.g., SG-SCCM or ServiceNow).
• id: The unique key of the asset in the external system, stored as the source native key.
• target_sys_id: The sys_id of the matching CI in the CMDB.

When a Service Graph Connector executes, the IRE performs a lookup on the sys_object_source table matching the incoming source name and native key. If a match is found, the IRE retrieves the target_sys_id of the existing CI, bypassing the evaluation of the identification rules entirely. This shortcut increases ingestion throughput and prevents duplicate records when multiple sources report on the same physical device.