Control: 2.1.3 Ensure that traffic is encrypted between cluster worker nodes

Description

By default, data exchanged between worker nodes in an Azure Databricks cluster is not encrypted. To ensure that data is encrypted at all times, whether at rest or in transit, you can create an initialization script that configures your clusters to encrypt traffic between worker nodes using AES 256-bit encryption over a TLS 1.3 connection.

Protects sensitive data during transit between cluster nodes, mitigating risks of data interception or unauthorized access.
Aligns with organizational security policies and compliance requirements that mandate encryption of data in transit.
Enhances overall security posture by ensuring that all inter-node communications within the cluster are encrypted.

Remediation

Create a JKS keystore:

Generate a Java KeyStore (JKS) file that will be used for SSL/TLS encryption.
Upload the keystore file to a secure directory in DBFS (e.g./dbfs//jetty_ssl_driver_keystore.jks).

Develop an init script:

Create an init script that performs the following tasks:

Retrieves the JKS keystore file and password.
Derives a shared encryption secret from the keystore.
Configures Spark driver and executor settings to enable encryption.

    Example init script:
    #!/bin/bash
    set -euo pipefail
    keystore_dbfs_file="/dbfs/<keystoredirectory>/jetty_ssl_driver_keystore.jks"
    max_attempts=30
    while [ ! -f ${keystore_dbfs_file} ]; do
      if [ "$max_attempts" == 0 ]; then
        echo "ERROR: Unable to find the file : $keystore_dbfs_file. Failing the script."
      exit 1
      fi
      sleep 2s
      ((max_attempts--))
    done
    sasl_secret=$(sha256sum $keystore_dbfs_file | cut -d' ' -f1)
    if [ -z "${sasl_secret}" ]; then
      echo "ERROR: Unable to derive the secret. Failing the script."
      exit 1
    fi
    local_keystore_file="$DB_HOME/keys/jetty_ssl_driver_keystore.jks"
    local_keystore_password="gb1gQqZ9ZIHS"
    if [[ $DB_IS_DRIVER = "TRUE" ]]; then
      driver_conf=${DB_HOME}/driver/conf/spark-branch.conf
      echo "Configuring driver conf at $driver_conf"
      if [ ! -e $driver_conf ]; then
        echo "spark.authenticate true" >> $driver_conf
        echo "spark.authenticate.secret $sasl_secret" >> $driver_conf
        echo "spark.authenticate.enableSaslEncryption true" >>$driver_conf
        echo "spark.network.crypto.enabled true" >> $driver_conf
        echo "spark.network.crypto.keyLength 256" >> $driver_conf
        echo "spark.network.crypto.keyFactoryAlgorithm PBKDF2WithHmacSHA1" >> $driver_conf
        echo "spark.io.encryption.enabled true" >> $driver_conf
        echo "spark.ssl.enabled true" >> $driver_conf
        echo "spark.ssl.keyPassword $local_keystore_password" >> $driver_conf
        echo "spark.ssl.keyStore $local_keystore_file" >> $driver_conf
        echo "spark.ssl.keyStorePassword $local_keystore_password" >> $driver_conf
        echo "spark.ssl.protocol TLSv1.3" >> $driver_conf
      fi
    fi
    executor_conf=${DB_HOME}/conf/spark.executor.extraJavaOptions
    echo "Configuring executor conf at $executor_conf"
    if [ ! -e $executor_conf ]; then
      echo "-Dspark.authenticate=true" >> $executor_conf
      echo "-Dspark.authenticate.secret=$sasl_secret" >> $executor_conf
      echo "-Dspark.authenticate.enableSaslEncryption=true" >> $executor_conf
      echo "-Dspark.network.crypto.enabled=true" >> $executor_conf
      echo "-Dspark.network.crypto.keyLength=256" >> $executor_conf
      echo "-Dspark.network.crypto.keyFactoryAlgorithm=PBKDF2WithHmacSHA1" >> $executor_conf
      echo "-Dspark.io.encryption.enabled=true" >> $executor_conf
      echo "-Dspark.ssl.enabled=true" >> $executor_conf
      echo "-Dspark.ssl.keyPassword=$local_keystore_password" >> $executor_conf
      echo "-Dspark.ssl.keyStore=$local_keystore_file" >> $executor_conf
      echo "-Dspark.ssl.keyStorePassword=$local_keystore_password" >> $executor_conf
      echo "-Dspark.ssl.protocol=TLSv1.3" >> $executor_conf
    fi

Save.

Default Value:

By default, traffic is not encrypted between cluster worker nodes.

Usage

Run the control in your terminal:

powerpipe control run azure_compliance.control.cis_v500_2_1_3

Snapshot and share results via Turbot Pipes:

powerpipe login
powerpipe control run azure_compliance.control.cis_v500_2_1_3 --share

SQL

This control uses a named query:

select
  id as resource,
  'info' as status,
  'Manual verification required.' as reason,
  display_name as subscription
from
  azure_subscription;

Control: 2.1.3 Ensure that traffic is encrypted between cluster worker nodes

Description

Remediation

Default Value:

Usage

SQL

Tags