Logo

菜单

LarryDpk
发布于 2024-08-14 / 142 阅读
0

Secure Kafka with SSL(Shell Script Automation)

Secure Kafka with SSL(Shell Script Automation)

Securing Kafka with SSL (using Java KeyStore, JKS) involves several steps both on the server (Kafka broker) side and the client (producer/consumer) side. Here’s a detailed guide:

1. Generate SSL Certificates and Keystores

Step 1: Create a Certificate Authority (CA)

The CA will sign the certificates for your Kafka brokers and clients.

# Generate CA private key
openssl genpkey -algorithm RSA -out ca-key.pem

# Generate CA certificate
openssl req -new -x509 -key ca-key.pem -out ca-cert.pem -days 365 \
  -subj "/CN=MyKafkaCA"

Step 2: Create Keystores and Certificates for Kafka Brokers

You need to create a keystore for each Kafka broker.

  1. Generate a keystore:
keytool -genkeypair -alias kafka-broker -keyalg RSA -keystore kafka.broker.keystore.jks \
  -storepass broker-keystore-password -keypass broker-key-password \
  -dname "CN=kafka-broker, OU=IT, O=MyOrg, L=City, ST=State, C=US"
  1. Create a Certificate Signing Request (CSR):
keytool -certreq -alias kafka-broker -keystore kafka.broker.keystore.jks \
  -file kafka-broker.csr -storepass broker-keystore-password
  1. Sign the CSR with the CA:
openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem -in kafka-broker.csr \
  -out kafka-broker-cert.pem -days 365 -CAcreateserial -passin pass:ca-key-password
  1. Import the CA certificate and the signed certificate into the keystore:
# Import CA certificate
keytool -importcert -alias ca-cert -file ca-cert.pem -keystore kafka.broker.keystore.jks \
  -storepass broker-keystore-password -noprompt

# Import broker certificate
keytool -importcert -alias kafka-broker -file kafka-broker-cert.pem \
  -keystore kafka.broker.keystore.jks -storepass broker-keystore-password -noprompt

Step 3: Create Truststore for Kafka Brokers

The truststore contains the CA certificate.

keytool -importcert -alias ca-cert -file ca-cert.pem -keystore kafka.broker.truststore.jks \
  -storepass broker-truststore-password -noprompt

Step 4: Repeat for Each Kafka Broker

If you have multiple brokers, repeat the steps to generate unique keystores and certificates for each one.

2. Configure Kafka Brokers

Add the following configurations to each broker’s server.properties file:

# Enable SSL for inter-broker communication and client connections
listeners=SSL://kafka-broker-hostname:9093

# Specify keystore and truststore locations and passwords
ssl.keystore.location=/path/to/kafka.broker.keystore.jks
ssl.keystore.password=broker-keystore-password
ssl.key.password=broker-key-password

ssl.truststore.location=/path/to/kafka.broker.truststore.jks
ssl.truststore.password=broker-truststore-password

ssl.client.auth=required

# Enable SSL encryption
security.inter.broker.protocol=SSL

3. Create Keystore and Truststore for Kafka Clients

Step 1: Generate a Keystore for the Client

keytool -genkeypair -alias kafka-client -keyalg RSA -keystore kafka.client.keystore.jks \
  -storepass client-keystore-password -keypass client-key-password \
  -dname "CN=kafka-client, OU=IT, O=MyOrg, L=City, ST=State, C=US"

Step 2: Create a CSR and Sign it with the CA

keytool -certreq -alias kafka-client -keystore kafka.client.keystore.jks \
  -file kafka-client.csr -storepass client-keystore-password

openssl x509 -req -CA ca-cert.pem -CAkey ca-key.pem -in kafka-client.csr \
  -out kafka-client-cert.pem -days 365 -CAcreateserial -passin pass:ca-key-password

Step 3: Import the CA Certificate and Signed Certificate into the Client Keystore

keytool -importcert -alias ca-cert -file ca-cert.pem -keystore kafka.client.keystore.jks \
  -storepass client-keystore-password -noprompt

keytool -importcert -alias kafka-client -file kafka-client-cert.pem \
  -keystore kafka.client.keystore.jks -storepass client-keystore-password -noprompt

Step 4: Create a Truststore for the Client

keytool -importcert -alias ca-cert -file ca-cert.pem -keystore kafka.client.truststore.jks \
  -storepass client-truststore-password -noprompt

4. Configure Kafka Clients

For both producers and consumers, you need to configure SSL settings in your client code.

const { Kafka } = require('kafkajs');
const fs = require('fs');

const kafka = new Kafka({
  clientId: 'your-client-id',
  brokers: ['kafka-broker-hostname:9093'],
  ssl: {
    rejectUnauthorized: true, // Optionally, set to false for self-signed certs
    ca: [fs.readFileSync('/path/to/ca-cert.pem')],
    key: fs.readFileSync('/path/to/kafka.client.keystore.jks'),
    cert: fs.readFileSync('/path/to/kafka-client-cert.pem'),
  },
});

5. Test the Setup

  1. Start the Kafka Broker: Make sure the broker is configured correctly with SSL and start it.

  2. Run the Kafka Client: Ensure your producer/consumer is configured with SSL and attempt to connect to the Kafka broker. If everything is configured correctly, the client should be able to produce and consume messages securely over SSL.

6. Additional Security Configurations

  • SASL Authentication: You can combine SSL with SASL for additional security (e.g., SASL_SSL).
  • Authorization: Implement Access Control Lists (ACLs) to restrict access to specific topics or consumer groups.

By following these steps, you will have a Kafka cluster secured with SSL, ensuring encrypted communication between Kafka brokers and clients.

Shell Script for cert generation

#!/bin/bash

# Variables
DAYS_VALID=365
KEYSTORE_PASSWORD="keystore-password"
TRUSTSTORE_PASSWORD="truststore-password"
CA_CERT="ca-cert.pem"
CA_KEY="ca-key.pem"
CA_SUBJECT="/CN=MyKafkaCA"
BROKER_ALIAS="kafka-broker"
CLIENT_ALIAS="kafka-client"
BROKER_SUBJECT="CN=kafka-broker"
CLIENT_SUBJECT="CN=kafka-client"

# Paths
BROKER_KEYSTORE="kafka.broker.keystore.jks"
BROKER_TRUSTSTORE="kafka.broker.truststore.jks"
CLIENT_KEYSTORE="kafka.client.keystore.jks"
CLIENT_TRUSTSTORE="kafka.client.truststore.jks"
BROKER_CSR="kafka-broker.csr"
CLIENT_CSR="kafka-client.csr"
BROKER_CERT="kafka-broker-cert.pem"
CLIENT_CERT="kafka-client-cert.pem"

# Step 1: Create a Certificate Authority (CA)
echo "Creating Certificate Authority (CA)..."
openssl genpkey -algorithm RSA -out $CA_KEY
openssl req -new -x509 -key $CA_KEY -out $CA_CERT -days $DAYS_VALID -subj $CA_SUBJECT

# Step 2: Create Keystore and CSR for Kafka Broker
echo "Creating keystore and CSR for Kafka Broker..."
keytool -genkeypair -alias $BROKER_ALIAS -keyalg RSA -keystore $BROKER_KEYSTORE \
  -storepass $KEYSTORE_PASSWORD -keypass $KEYSTORE_PASSWORD -dname $BROKER_SUBJECT

keytool -certreq -alias $BROKER_ALIAS -keystore $BROKER_KEYSTORE \
  -file $BROKER_CSR -storepass $KEYSTORE_PASSWORD

# Step 3: Sign Broker CSR with CA
echo "Signing Broker CSR with CA..."
openssl x509 -req -CA $CA_CERT -CAkey $CA_KEY -in $BROKER_CSR \
  -out $BROKER_CERT -days $DAYS_VALID -CAcreateserial

# Step 4: Import CA Cert and Broker Cert into Broker Keystore
echo "Importing CA and Broker certificates into Broker keystore..."
keytool -importcert -alias ca-cert -file $CA_CERT -keystore $BROKER_KEYSTORE \
  -storepass $KEYSTORE_PASSWORD -noprompt

keytool -importcert -alias $BROKER_ALIAS -file $BROKER_CERT \
  -keystore $BROKER_KEYSTORE -storepass $KEYSTORE_PASSWORD -noprompt

# Step 5: Create Truststore for Kafka Broker
echo "Creating Truststore for Kafka Broker..."
keytool -importcert -alias ca-cert -file $CA_CERT -keystore $BROKER_TRUSTSTORE \
  -storepass $TRUSTSTORE_PASSWORD -noprompt

# Step 6: Create Keystore and CSR for Kafka Client
echo "Creating keystore and CSR for Kafka Client..."
keytool -genkeypair -alias $CLIENT_ALIAS -keyalg RSA -keystore $CLIENT_KEYSTORE \
  -storepass $KEYSTORE_PASSWORD -keypass $KEYSTORE_PASSWORD -dname $CLIENT_SUBJECT

keytool -certreq -alias $CLIENT_ALIAS -keystore $CLIENT_KEYSTORE \
  -file $CLIENT_CSR -storepass $KEYSTORE_PASSWORD

# Step 7: Sign Client CSR with CA
echo "Signing Client CSR with CA..."
openssl x509 -req -CA $CA_CERT -CAkey $CA_KEY -in $CLIENT_CSR \
  -out $CLIENT_CERT -days $DAYS_VALID -CAcreateserial

# Step 8: Import CA Cert and Client Cert into Client Keystore
echo "Importing CA and Client certificates into Client keystore..."
keytool -importcert -alias ca-cert -file $CA_CERT -keystore $CLIENT_KEYSTORE \
  -storepass $KEYSTORE_PASSWORD -noprompt

keytool -importcert -alias $CLIENT_ALIAS -file $CLIENT_CERT \
  -keystore $CLIENT_KEYSTORE -storepass $KEYSTORE_PASSWORD -noprompt

# Step 9: Create Truststore for Kafka Client
echo "Creating Truststore for Kafka Client..."
keytool -importcert -alias ca-cert -file $CA_CERT -keystore $CLIENT_TRUSTSTORE \
  -storepass $TRUSTSTORE_PASSWORD -noprompt

echo "SSL certificates generation completed!"

Test:

# bin/zookeeper-server-start.sh config/zookeeper.properties

# bin/kafka-server-start.sh /Users/larry/IdeaProjects/pkslow-samples/other/kafka-ssl-security/server.properties

Code

Please check the code in GitHub.

Automation UI Tes...
Node Kafka Produc...