2.2 Create and Manage Datastores

Create and Manage Datastores

Quick Answer: A datastore is a pointer to storage you already own. Use identity-based access (workspace or compute managed identity, Storage Blob Data Reader) for Blob and ADLS Gen2. Use keys, SAS, or a service principal only when identity-based access is unavailable—Azure Files has no identity-based option. Never paste account keys into notebooks.

A datastore is not a data asset and not a storage account. It is a named connection that Azure Machine Learning keeps in the workspace so jobs, data assets, and studio browsers can reach Azure storage through one API. The storage account already exists; az ml datastore create and MLClient.create_or_update only register the connection.

That distinction is the first exam filter. If the stem says “create a datastore,” the correct action is to link Blob, ADLS Gen2, or Azure Files. If the stem says “create a data asset,” you are versioning a URI (file, folder, or table) that usually sits on a datastore. Chapter 3 covers data assets; this section stays on the connection object.

What datastores buy you

• One API for Blob, Files, and ADLS Gen2 so training code can take a URI instead of a storage SDK maze
• Discoverable names (adls_claims_gold) so teammates do not copy long abfss:// strings
• For credential-based connections, secrets live in the workspace Key Vault instead of in Git

Datastores do not copy data into the workspace. They do not replace Azure RBAC on the storage account. A user who can see a datastore in studio still needs a data-plane role (or a cached credential) to read bytes.

Default datastores versus additional datastores

Every workspace already has datastores on its associated storage account. You do not create these; you need to know what they are for so you do not dump a 2 TB lake into the same containers that hold notebook files.

Additional datastores are the ones you create for project data: a Blob container of images, an ADLS Gen2 filesystem of parquet, an Azure Files share that a lift-and-shift app already writes to, or (preview) a Microsoft Fabric OneLake lakehouse. Extra datastores are also how you attach the account types that cannot be default workspace storage—premium Blob and hierarchical-namespace ADLS Gen2 from section 2.1.

Keep training data off workspaceworkingdirectory. That file share is mounted as the notebook working directory on every compute instance in the workspace. Small notebook writes are fine; large training corpora belong on Blob or ADLS Gen2 datastores and should be mounted into jobs, not into the notebook share.

Supported storage and authentication matrix

Current Azure Machine Learning datastore types you should design for in 2026:

• Azure Blob Storage — credential-based or identity-based
• Azure Data Lake Storage Gen2 — credential-based or identity-based
• Azure Files — credential-based only (account key or SAS). There is no identity-based Files datastore.
• Microsoft Fabric OneLake — preview; identity-based or service principal, targeting lakehouse Files
• Azure Data Lake Storage Gen1 — retired 29 February 2024. You cannot create new Gen1 accounts; existing Gen1 data is inaccessible. New work uses Gen2.

Credential-based authentication means Azure Machine Learning stores a secret:

• Storage account key
• Shared access signature (SAS) token
• Service principal (tenant ID, client ID, client secret)—common for ADLS Gen2 when a central app registration is the only identity the lake will trust

Those secrets land in the workspace Key Vault. Anyone whose workspace role includes Microsoft.MachineLearningServices/workspaces/datastores/listsecrets/action can retrieve them. Built-in Contributor, AzureML Data Scientist, and Azure AI Developer include that action. Reader can list and view datastore credentials in the workspace even though Reader cannot create assets. That is a security design issue, not a trivia footnote: if the exam stem cares about “analysts must not retrieve the storage key,” do not put an account key on the datastore. Use identity-based access instead.

Identity-based (often called credential-less) authentication stores only the account name and container or filesystem. At access time Azure Machine Learning uses:

• The interactive user’s Microsoft Entra ID token (studio, notebooks, jobs submitted with UserIdentityConfiguration)
• A managed identity on the workspace, compute cluster, compute instance, or serverless job

Grant Storage Blob Data Reader as the minimum data-plane role on Blob or ADLS Gen2. Writers need Storage Blob Data Contributor. ADLS Gen2 can go further with POSIX-like ACLs on folders. Control-plane Reader or Contributor on the storage account is not a substitute for the data-plane roles; a classic exam distractor is “the user is Contributor on the storage account, so the job can read blobs.” Contributor without a data-plane role often yields 403 on blob data.

Creating datastores with CLI v2 and SDK v2

Identity-based Blob datastore (preferred):

# blob-identity.yml
$schema: https://azuremlschemas.azureedge.net/latest/azureBlob.schema.json
name: blob_claims_raw
type: azure_blob
account_name: stclaimsprod
container_name: raw

az ml datastore create --file blob-identity.yml

The SDK v2 equivalent constructs AzureBlobDatastore(name=..., account_name=..., container_name=...) with no credentials= and calls ml_client.create_or_update(store).

ADLS Gen2 identity-based YAML uses type: azure_data_lake_gen2 plus filesystem: instead of container_name. If the lake still requires a service principal, add credentials with tenant_id, client_id, and client_secret. Azure Files YAML uses type: azure_file, file_share_name, and credentials—there is no identity-based Files path.

URI paths in jobs then look like:

azureml://datastores/blob_claims_raw/paths/2026/claims.parquet

Jobs can also take raw wasbs:// or abfss:// URIs. Datastore URIs are preferred in MLOps because the authentication mode is defined once on the datastore, not in every job YAML.

Which identity actually reads the data?

Walk this list when a job cannot see files:

1. If the datastore has cached credentials, those credentials are used. User and compute identities are ignored.
2. If the datastore is identity-based and the job sets identity: type: user_identity, the submitting user’s Entra token is used. Studio does not support this mode for all job types; CLI/SDK v2 do. Fine-grained “this scientist can read folder A, that scientist can read folder B” is the reason to choose it.
3. Otherwise the compute managed identity (cluster, instance, or the user-assigned identity on a serverless job) is used. Admins grant Storage Blob Data Reader to /workspace-name/computes/compute-name so data scientists never receive personal data-plane roles. That is the usual production pattern for confidential data.
4. Interactive studio previews can use the workspace managed identity when you enable “use workspace managed identity for data preview and profiling.”

Serverless compute supports user credential passthrough and user-assigned managed identity. It does not support a system-assigned identity on the serverless job itself.

You cannot mix user identity and compute managed identity in the same job. For pipelines, set identity on the step, not only on the root pipeline—especially when the pipeline contains nested pipeline components.

Cross-tenant storage access is not supported. If the lake lives in another tenant, that is a support exception, not a datastore checkbox.

When credential-based is still the right answer

Identity-based is the default you should argue for. Credential-based still appears on the exam when:

• The store is Azure Files (no identity-based option)
• A partner system can only mint a time-boxed SAS
• A central service principal is the only identity the storage firewall will allow, and compute identities cannot be added yet

Even then, keep the secret in the datastore (Key Vault) or in Azure Key Vault references used by jobs. Do not commit keys, and do not os.environ["AZURE_STORAGE_KEY"] = "..." in a notebook cell. Section 2.4 covers managed identities in more depth; the datastore rule is simple: if a managed identity can work, do not copy a key.

Storage firewalls change which identity is used for preview. If only listed subnets can reach the account, studio previews often use the workspace managed identity. If “allow Azure trusted services” is enabled, the workspace MSI is in play again. Identity-based datastores plus a locked storage firewall are compatible only when you also complete the network design (private endpoints or trusted services)—that pairing shows up again in Chapter 4.

Exam scenario

A bank keeps features in ADLS Gen2. The platform team refuses to put the storage account key in Azure Machine Learning because Reader and AzureML Data Scientist can list datastore secrets. Create an identity-based azure_data_lake_gen2 datastore. Grant the compute cluster’s managed identity Storage Blob Data Reader on the filesystem the training job needs. Data scientists keep AzureML Data Scientist on the workspace and never receive the account key.

Common trap

Two traps show up constantly. First, using workspaceblobstore as the project lake: snapshots and random uploads collide with production tables, and you inherit the default account’s SKU limits. Second, creating a credential-based Blob datastore “for convenience,” then assigning Reader to contractors—those contractors can view the cached key. Identity-based access plus data-plane RBAC avoids both.