11.3 Data Discovery, Classification, & Microsoft Purview Integration

7.3 Data Discovery, Classification, & Microsoft Purview Integration

Enterprise security compliance requires complete visibility into where sensitive data resides across an organization's digital estate. As organizations scale across cloud environments, sensitive information—such as Personally Identifiable Information (PII), credit card numbers, national identification numbers, and confidential financial metrics—becomes distributed across thousands of database tables, data lakes, and storage accounts. Securing this data requires automated discovery, continuous classification, enterprise governance mapping, and automated Data Loss Prevention (DLP) enforcement.

---

SQL Data Discovery and Classification

SQL Data Discovery and Classification (built natively into Azure SQL Database, Azure SQL Managed Instance, and Azure Synapse Analytics) provides built-in capabilities for discovering, classifying, labeling, and reporting sensitive columns in database tables.

+-----------------------------------------------------------------------------------+
|                    SQL DATA DISCOVERY & CLASSIFICATION FLOW                       |
+-----------------------------------------------------------------------------------+
|  1. Recommendation Engine Scans Column Metadata (Names: SSN, Email, Balance)      |
|       |                                                                           |
|       v                                                                           |
|  2. Recommends Information Types (e.g., National ID) & Sensitivity Labels         |
|       |                                                                           |
|       v                                                                           |
|  3. Admin Accepts / Applies Labels -> Written to sys.extended_properties          |
|       |                                                                           |
|       v                                                                           |
|  4. Azure SQL Auditing Captures Sensitivity Labels in Audit Logs for Sentinel     |
+-----------------------------------------------------------------------------------+

Architecture & Core Components

1. Discovery & Recommendation Engine: Automated background scanners analyze database column names and data patterns against predefined sensitivity rules. For example, columns named SocialSecurityNumber, CreditCard, PassCode, or User_Email trigger automatic classification recommendations.
2. Sensitivity Labeling Taxonomy: Organizes classification into two primary dimensions:
   • Information Types: Defines the nature of the data (e.g., Networking, Contact Info, Credentials, Financial, National ID, SSN).
   • Sensitivity Labels: Defines the organizational confidentiality level (e.g., Public, General, Confidential, Confidential - Financial, Highly Confidential).

Persistent Metadata Storage

When classification labels are applied, Azure SQL stores the sensitivity metadata directly inside the database schema using column Extended Properties (sys.extended_properties). Because classification metadata is embedded inside the database schema, it persists through database backups, restores, and geo-replication.

Audit & SIEM Integration

SQL Data Discovery and Classification integrates directly with Azure SQL Auditing. When a user executes a query against a classified column, the audit log records the specific sensitivity label and information type accessed. Security Operations Center (SOC) analysts querying Log Analytics or Microsoft Sentinel can build alert rules tracking whenever Highly Confidential columns are queried by non-standard service principals or accessed during off-business hours.

---

Microsoft Purview Integration for Enterprise Data Governance

While SQL Data Discovery and Classification provides database-scoped labeling, enterprise environments require unified governance spanning hybrid and multi-cloud data estates. Microsoft Purview is a fully managed, enterprise-grade data governance solution that automates metadata discovery, data cataloging, sensitive data classification, and end-to-end lineage mapping.

+-----------------------------------------------------------------------------------+
|                    MICROSOFT PURVIEW ENTERPRISE DATA MAP                          |
+-----------------------------------------------------------------------------------+
|  Scanned Data Assets:                                                             |
|  [ Azure SQL ]  [ Azure Data Lake ]  [ Cosmos DB ]  [ AWS S3 ]  [ On-Prem SQL ]    |
|       |               |                  |             |              |           |
|       +---------------+------------------+-------------+--------------+           |
|                                       |                                           |
|                                       v Automated Scanning & Classification       |
|  +-----------------------------------------------------------------------------+  |
|  |                     MICROSOFT PURVIEW DATA MAP                              |  |
|  |   - Unified Data Catalog (Searchable asset discovery)                       |  |
|  |   - Automated Data Lineage (Trace data flow: SQL -> Pipeline -> PowerBI)    |  |
|  |   - Automated Labeling (Sensitivity & Information Protection Rules)         |  |
|  +-----------------------------------------------------------------------------+  |
+-----------------------------------------------------------------------------------+

Core Pillars of Microsoft Purview

• Purview Data Map: The foundational metadata engine. It connects to diverse data sources (Azure SQL, Azure Blob Storage, Azure Data Lake Storage Gen2, Amazon S3, SAP, Google Cloud Storage, and on-premises databases) via system-assigned managed identities or Key Vault credentials to perform automated scanning.
• Purview Data Catalog: A centralized, searchable portal enabling business users, data engineers, and compliance officers to locate data assets, inspect sensitivity labels, verify schema structures, and identify data owners.
• Automated Data Lineage: Graphically traces the end-to-end transformation and movement of data across the enterprise. For example, Purview illustrates how raw customer records flow from an Azure SQL Database through an Azure Data Factory ETL pipeline into an Azure Synapse data warehouse and finally into a Power BI dashboard.

---

Automated Data Labeling & Classification Engine

Microsoft Purview features an advanced automated scanning engine capable of classifying structured database tables and unstructured files (PDFs, Office documents, CSVs, Parquet files).

Classification Rulesets

• System Classification Rules: Over 200 built-in pattern matchers provided by Microsoft to detect standard sensitive data formats (e.g., EU Credit Card numbers, US Social Security Numbers, driver's license numbers, IP addresses).
• Custom Classification Rules: Custom-built classification rules designed by security teams to detect proprietary or industry-specific data formats. Rules combine:
  • Regular Expressions (RegEx): Pattern matching string structures (e.g., custom employee ID formats such as EMP-[0-9]{6}).
  • Dictionary Word Lists: Matches against customized lists of sensitive project codenames or confidential terms.
  • Match Thresholds: Requires a minimum number of pattern matches within a sample size (e.g., at least 5 matches per 100 rows) before applying the classification.

Integration with Microsoft Purview Information Protection (MPIP)

Purview syncs sensitivity labels with Microsoft Purview Information Protection (MPIP). When Purview detects sensitive files inside Azure Blob Storage or Data Lake stores, it can automatically apply official sensitivity labels, enforcing encryption and access controls defined in enterprise Information Protection policies.

---

Data Loss Prevention (DLP) for Azure Data Stores

Discovering and classifying data is ineffective without automated controls preventing unauthorized exfiltration. Data Loss Prevention (DLP) policies for Azure data stores extend Microsoft Purview protection directly to Azure storage and database services.

DLP Policy Engine Architecture

Purview DLP policies evaluate user and application access against configured sensitivity rules:

• Location Scopes: Targets specific Azure storage accounts, Azure Data Lake Gen2 repositories, or Azure SQL instances.
• Condition Rules: Defines conditions based on sensitivity labels (e.g., document tagged Confidential - Financial) or Sensitive Information Types (e.g., payload contains more than 10 Credit Card Numbers).
• Enforced Actions:
  • Access Restriction: Automatically blocks external or unauthorized user access to flagged storage blobs.
  • Alerting & Notification: Triggers high-priority security notifications to security administrators and records alerts in Microsoft Defender for Cloud.
  • User Remediation Guidance: Displays policy tip notifications explaining compliance policy violations to users.

Preventing Cloud Exfiltration

By combining SQL Data Discovery & Classification, Microsoft Purview Data Cataloging, and Purview Data Loss Prevention, security architects establish continuous end-to-end data protection: sensitive data is automatically discovered upon ingestion, classified with persistent metadata labels, monitored via SQL Auditing and Sentinel SIEM, and protected against unauthorized exfiltration by automated DLP policies.