Control: 3.1.7 Ensure that diagnostic log delivery is configured for Azure Databricks

Description

Azure Databricks Diagnostic Logging provides insights into system operations, user activities, and security events within a Databricks workspace. Enabling diagnostic logs helps organizations:

Detect security threats by logging access, job executions, and cluster activities.
Ensure compliance with industry regulations such as SOC 2, HIPAA, and GDPR.
Monitor operational performance and troubleshoot issues proactively.

Diagnostic logging provides visibility into security and operational activities within Databricks workspaces while maintaining an audit trail for forensic investigations, and it supports compliance with regulatory standards that require logging and monitoring.

Remediation

From Azure Portal

Enable diagnostic logging for Azure Databricks:

Navigate to your Azure Databricks workspace.
In the left-hand menu, select Monitoring > Diagnostic settings.
Click + Add diagnostic setting.
Under Category details, select the log categories you wish to capture, such as AuditLogs, Clusters, Notebooks, and Jobs.
Choose a destination for the logs:

Log Analytics workspace: For advanced querying and monitoring.
Storage account: For long-term retention.
Event Hub: For integration with third-party systems.

Provide a Name for the diagnostic setting.
Click Save.

Implement log retention policies:

Navigate to your Log Analytics workspace.
Under General, select Usage and estimated costs.
Click Data Retention.
Adjust the retention period slider to the desired number of days (up to 730 days).
Click OK.

Monitor logs for anomalies:

Navigate to Azure Monitor.
Select Alerts > + New alert rule.
Under Scope, specify the Databricks resource.
Define Condition based on log queries that identify anomalies (e.g. unauthorized access attempts).
Configure Actions to notify stakeholders or trigger automated responses.
Provide an Alert rule name and description.
Click Create alert rule.

From Azure CLI

Enable diagnostic logging for Azure Databricks:

az monitor diagnostic-settings create --name "DatabricksLogging" --resource <databricks-resource-id> --logs '[{"category": "AuditLogs", "enabled": true}, {"category": "Clusters", "enabled": true}, {"category": "Notebooks", "enabled": true}, {"category": "Jobs", "enabled": true}]' --workspace <log-analytics-id>

Implement log retention policies:

az monitor log-analytics workspace update --resource-group <resource-group> --name <log-analytics-name> --retention-time 365

Monitor logs for anomalies:

az monitor activity-log alert create --name "DatabricksAnomalyAlert" --resource-group <resource-group> --scopes <databricks-resource-id> --condition "contains 'UnauthorizedAccess'"

Usage

Run the control in your terminal:

powerpipe control run azure_compliance.control.cis_v400_3_1_7

Snapshot and share results via Turbot Pipes:

powerpipe login
powerpipe control run azure_compliance.control.cis_v400_3_1_7 --share

SQL

This control uses a named query:

select
  id as resource,
  'info' as status,
  'Manual verification required.' as reason,
  display_name as subscription
from
  azure_subscription;