12.1 Centralized Data Lake Governance & Access Control with AWS Lake Formation

Centralized Data Lake Governance & Access Control with AWS Lake Formation

Traditional data lake security models rely on a combination of Amazon S3 bucket policies, IAM policies, and KMS key permissions. As data lakes scale to thousands of tables and millions of objects, managing prefix-level access across multiple analytics tools becomes operationally unmaintainable. IAM policies hit maximum size limits, and bucket policies lack the ability to enforce column- or row-level permissions natively without duplicating datasets.

AWS Lake Formation solves these governance challenges by providing a centralized authorization layer on top of the AWS Glue Data Catalog and Amazon S3. Instead of defining broad permissions at the S3 object prefix level, Lake Formation enforces fine-grained access control (FGAC) at the database, table, column, row, and cell levels.

---

Lake Formation Architecture & Registration

To manage an Amazon S3 path with AWS Lake Formation, the location must first be registered with Lake Formation. When an S3 path is registered, Lake Formation registers the underlying IAM role (AWSServiceRoleForLakeFormationDataAccess) that possesses read/write privileges to that S3 location.

Data Location Registration & Initial Policy Migration

1. Register Data Location: Administrators register S3 storage locations (e.g., s3://my-company-datalake/analytics/) with Lake Formation via RegisterDataLocation.
2. Revoke Default Permissions: By default, new AWS Glue catalog objects grant ALL permissions to IAMAllowedPrincipals. To transition to Lake Formation governance, these default grants must be revoked so that IAM policies no longer bypass Lake Formation evaluations.
3. Data Location Permissions: Principals must be granted DATA_LOCATION_ACCESS permissions on the registered S3 path to create databases or tables pointing to those locations.

# Register an S3 bucket path with Lake Formation using AWS CLI
aws lakeformation register-data-location    --resource-arn arn:aws:s3:::company-data-lake-prod/analytics/    --use-service-linked-role

---

Credential Vending Mechanics

Lake Formation does not modify physical S3 objects or re-encrypt files on disk. Instead, it acts as a credential vending service for integrated query engines such as Amazon Athena, Amazon Redshift Spectrum, Amazon EMR (with Lake Formation integration), and AWS Glue ETL.

How Credential Vending Works

1. A user submits a SQL query through an integrated engine (e.g., Athena).
2. The query engine requests data access authorization from AWS Lake Formation on behalf of the user's IAM principal.
3. Lake Formation checks its permission database (evaluating named resource grants or LF-tags, column exclusions, and row filters).
4. If authorized, Lake Formation vends short-lived credentials for the registered S3 location.
5. The integrated query engine fetches the objects and enforces the authorized table, column, row, or cell filters. The credentials themselves are not byte-range credentials that independently encode row and column filters.

Because end users and IAM roles never receive direct s3:GetObject permissions on the raw S3 bucket, they cannot bypass business rules or access unapproved columns.

---

Fine-Grained Access Control (FGAC) & Data Cell Filters

AWS Lake Formation enables three levels of fine-grained access control:

• Column-Level Security: Specifies explicit column inclusion or exclusion lists. For example, financial analysts can query customer_id and lifetime_value, but the credit_card_number column is omitted from query planning.
• Row-Level Security: Uses SQL predicate expressions to restrict which rows a principal can view (e.g., region = 'us-east-1' or department = 'marketing').
• Cell-Level Security: Combines column projection and row-level SQL expressions into a single policy entity called a Data Cell Filter.

Creating and Applying a Data Cell Filter

A Data Cell Filter is defined on a target Glue Data Catalog table and granted to specific IAM users, roles, or SAML groups.

{
  "DataCellFilter": {
    "TableCatalogId": "123456789012",
    "DatabaseName": "financial_db",
    "TableName": "customer_transactions",
    "Name": "us_analyst_transactions_filter",
    "ColumnNames": ["transaction_id", "customer_id", "amount", "transaction_date"],
    "RowFilter": {
      "FilterExpression": "country_code = 'US' AND is_fraudulent = false"
    }
  }
}

When a Data Analyst executes SELECT * FROM financial_db.customer_transactions in Athena, Lake Formation applies us_analyst_transactions_filter. The engine executes the query as if non-permitted columns (ssn, tax_id) and non-matching rows (country_code != 'US') do not exist in the database.

---

Tag-Based Access Control (LF-TBAC)

Managing access via Named Resources (explicit grants on DatabaseA.TableB) becomes cumbersome when managing hundreds of databases and thousands of tables. Tag-Based Access Control (LF-TBAC) provides a scalable governance model using policy abstractions.

LF-TBAC Concepts & Inheritance

1. Define LF-Tags: Administrators construct key-value tag taxonomies in Lake Formation (e.g., Environment: [Dev, Staging, Prod], Confidentiality: [Public, Restricted, Secret], Department: [Finance, HR, Engineering]).
2. Assign LF-Tags to Catalog Resources: Tags are attached to Glue Databases, Tables, or individual Columns. Tags follow strict inheritance rules: Table tags inherit from Database tags, and Column tags inherit from Table tags unless overridden.
3. Grant Permissions on LF-Tag Expressions: Security policies grant privileges (SELECT, ALTER, DROP) to IAM principals based on logical tag expressions rather than explicit ARNs.

# Grant SELECT permission to DataScienceRole for any table tagged Confidentiality=Restricted
aws lakeformation grant-permissions \
+  --principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/DataScienceRole \
+  --permissions SELECT \
+  --resource '{"LFTagPolicy":{"CatalogId":"123456789012","ResourceType":"TABLE","Expression":[{"TagKey":"Confidentiality","TagValues":["Restricted","Public"]}]}}'

When a new table is cataloged and tagged Confidentiality=Restricted, DataScienceRole instantly gains query access without requiring an administrator to execute manual grant commands.

---

Secure Cross-Account Data Sharing

Enterprise data architectures frequently separate data producers (e.g., Central Data Platform Account) from data consumers (e.g., Business Unit Accounts). AWS Lake Formation enables cross-account data sharing natively without requiring S3 data replication or complex cross-account IAM role switching.

Cross-Account Sharing Workflow

1. AWS RAM Integration: Lake Formation uses AWS Resource Access Manager (RAM) under the hood to manage catalog resource shares across AWS accounts or within an AWS Organization.
2. Producer Account Grant: The Data Lake Admin in Account A grants SELECT permissions on a catalog table or LF-Tag to the AWS Account ID of Account B.
3. Consumer Account Acceptance & Resource Link: Account B receives the RAM invitation (automatically accepted within an Organization). The Data Lake Admin in Account B creates a Resource Link—a local catalog alias—in Account B's Glue Data Catalog pointing to the shared table in Account A.
4. Query Execution: Analysts in Account B query the Resource Link via Athena. Lake Formation vends short-lived S3 access credentials to Account B's query engine, fetching data directly from Account A's S3 bucket while enforcing Account A's row/column permissions.

---

Lake Formation vs. Traditional IAM & S3 Access Control