Control: 3.1.8 Ensure that data at rest and in transit is encrypted in Azure Databricks using customer managed keys (CMK)

Description

Azure Databricks encrypts data in transit using TLS 1.2+ to secure API, workspace, and cluster communications. By default, data at rest is encrypted using Microsoft-managed keys.

Organizations with stricter needs for control of encryption keys should enable customer-managed keys (CMK) for greater control over data encryption, auditing, and regulatory compliance. Azure Key Vault should be used to store and manage CMKs.

Enforcing encryption at rest and in transit in Azure Databricks:

Protects sensitive data from unauthorized access.
Ensures regulatory compliance (ISO 27001, GDPR, HIPAA, SOC 2).
Allows key revocation and rotation control with customer-managed keys (CMK).
Mitigates insider threats by preventing unauthorized access to raw storage.

Remediation

NOTE: These remediations assume that an Azure KeyVault already exists in the subscription.

From Azure CLI

Create a dedicated key:

az keyvault key create --vault-name <keyvault-name> --name <key-name> --protection <"software" or "hsm">

Assign permissions to Databricks:

az keyvault set-policy --name <keyvault-name> --resource-group <resource-group-name> --spn <databricks-spn> --key-permissions get wrapKey unwrapKey

Enable encryption with CMK:

az databricks workspace update --name <databricks-workspace-name> --resource-group <resource-group-name> --key-source "Microsoft.KeyVault" --key-name <key-name> --keyvault-uri <keyvault-uri>

From PowerShell

$Key = Add-AzKeyVaultKey -VaultName <keyvault-name> -Name <key-name> -Destination <"software" or "hsm">
Set-AzDatabricksWorkspace -ResourceGroupName "<resource-group-name>" -WorkspaceName "<databricks-workspace-name>" -EncryptionKeySource "Microsoft.KeyVault" -KeyVaultUri $Key.Id

Default Value

By default, Azure Databricks uses Microsoft-managed keys for encryption. Data in transit is always encrypted using TLS 1.2+. Customer-Managed Keys (CMK) must be manually enabled.

Usage

Run the control in your terminal:

powerpipe control run azure_compliance.control.cis_v400_3_1_8

Snapshot and share results via Turbot Pipes:

powerpipe login
powerpipe control run azure_compliance.control.cis_v400_3_1_8 --share

SQL

This control uses a named query:

select
  a.id as resource,
  case
    when parameters -> 'customerManagedKeyId' is not null then 'ok'
    else 'alarm'
  end as status,
  case
    when parameters -> 'customerManagedKeyId' is not null then a.name || ' has a customer-managed key configured.'
    else a.name || ' does not have a customer-managed key configured.'
  end as reason
  
  , a.resource_group as resource_group
  , sub.display_name as subscription
from
  azure_databricks_workspace as a,
  azure_subscription as sub
where
  sub.subscription_id = a.subscription_id;