3.5 Technical and Business Catalogs, Connections & Lineage

3.5 Technical and Business Catalogs, Connections & Lineage

A catalog answers two different questions: How can an engine read this data? and What does this data mean, who owns it, and may I use it? The AWS Glue Data Catalog primarily answers the technical question. Amazon SageMaker Catalog in SageMaker Unified Studio adds business discovery and governed collaboration.

AWS Glue Data Catalog as technical metadata

The Glue Data Catalog stores catalogs, databases, tables, columns, partition keys, table properties, and connections. Athena, Redshift Spectrum, EMR, Glue, and compatible engines can use it as a metastore. A table definition points to data; the catalog does not contain the S3 dataset itself.

A crawler connects to a source, applies custom classifiers before built-in classifiers, infers schemas, and creates or updates tables and partitions. Treat inference cautiously. Mixed files can cause columns to become strings or separate tables, and a crawler cannot decide whether a customer identifier is legally sensitive or whether a metric is approved.

Connections store reusable source properties and, for VPC sources, network configuration. The job still needs routes, security groups, DNS, credentials, and data permissions. Test a connection from the environment that will use it rather than assuming a successful catalog entry proves reachability.

Synchronizing partitions

Choose one method deliberately:

Do not confuse an S3 prefix with a registered partition. A query engine prunes catalog partitions only when the table metadata and query predicate align. For high-cardinality layouts, partition projection can avoid millions of Glue API lookups.

Schema evolution and conversion

Adding nullable columns is usually safer than renaming or changing types. Define producer compatibility rules, preserve the raw version, and test old and new readers. Glue table updates describe a schema; they do not rewrite existing files.

For heterogeneous relational migrations, current AWS guidance recommends DMS Schema Conversion to assess and convert supported schema and code objects. DMS data movement and schema conversion are separate concerns, and unsupported code still needs manual remediation.

Amazon SageMaker Catalog

A technical name such as daily_sales_agg_v3 is difficult for another team to trust. In SageMaker Unified Studio, a project can inventory an asset, add a business name, description, owner, README, glossary terms, and structured metadata forms, then publish the latest asset version to the SageMaker Catalog. Publication makes the asset discoverable beyond the project; updating the inventory later does not automatically publish the new version.

Consumers search the catalog and request a subscription. When approved, access is granted to the requesting project's execution role, commonly through Lake Formation for governed tables. Approval is not merely a UI label—it must produce data-plane access for the execution identity. Revocation must remove that grant.

Technical and business catalogs complement each other:

• Glue supplies schema, location, partitions, and engine integration.
• SageMaker Catalog supplies shared language, curation, ownership, discovery, and subscription workflow.
• Lake Formation enforces governed table access.

Data lineage

Lineage should connect a source dataset to a job, a job run, and its output dataset. It supports two investigations:

1. Root cause: Which upstream input or job produced a bad column?
2. Impact: Which downstream tables, dashboards, or models depend on the field I will change?

SageMaker Unified Studio lineage is compatible with OpenLineage events and can visualize datasets, jobs, and job runs. Current integrations can capture lineage from Glue and EMR Spark jobs, with column-level detail when supported. The graph is evidence only if run identities, source versions, and output versions are accurate.

Do not equate CloudTrail with data lineage. CloudTrail records API activity; it can show that a job or catalog API was called, but it does not by itself express the semantic mapping from source column to transformed output column.

Governance checklist

For each production asset, record a stable identifier, technical location, schema, owner, business definition, sensitivity, freshness expectation, quality status, lineage, and access policy. Separate draft inventory from published approved assets, and automate checks for missing owners or stale metadata. A catalog with thousands of uncurated tables is an inventory, not a trusted data product marketplace.