Home

Kafka security

You can improve the Kafka cluster security by having Kafka authenticate connections to brokers from clients using either SSL or SASL.

SSL + Kerberos for Kafka clients

Prerequisite: Kafka brokers are configured with SSL and Kerberos. Refer to your Hadoop providers documentation for configuring SSL and Kerberos for Kafka brokers.

SSL+Kerberos is supported by new Kafka consumers and producers. The configuration is the same for consumers and producers. Replace items in red with values specific/relevant to your environment.

For single Kafka clients

  1. Create a file named consumerConfig.properties. Add the following properties. You can locate your SSL + Kerberos configuration.

    ssl.protocol = TLSv1
sasl.mechanism = GSSAPI security.protocol = SASL_SSL
sasl.kerberos.service.name = kafka
ssl.truststore.location = /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts1 
ssl.truststore.password = changeit ssl.truststore.type = JKS
ssl.keystore.location = /opt/cloudera/security/jks/server.keystore.jks 
ssl.keystore.password = password
ssl.keystore.type = JKS
ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1
sasl.jaas.config = \
com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \
keyTab="/etc/keytabs/kafka.keytab" \
principal="kafka/edge-1.uddev.unraveldata.com@UDDEV.UNRAVELDATA.COM"
    sasl.mechanism = GSSAPI
security.protocol = SASL_PLAINTEXT 
sasl.kerberos.service.name = kafka
sasl.jaas.config = com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \
keyTab="/etc/keytabs/kafka.keytab" \
principal="kafka/edge-1.uddev.unraveldata.com@UDDEV.UNRAVELDATA.COM";

  2. Copy/move consumerConfig.properties file to <Unravel_installation_directory/unravel/data/conf.

  3. Check the Kafka cluster name that you had added against com.unraveldata.ext.kafka.cluster.list property while configuring Kafka connection.

  4. Use the manager command from the Unravel installation directory to set the com.unraveldata.ext.kafka.<ClusterName>.consumer.config property. 

    <Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.<ClusterName>.consumer.config <Unravel_installation_directory>/unravel/data/conf

    Replace <clustername> with the name provided in the com.unraveldata.ext.kafka.cluster.list property.

    <ClusterName> is the user-defined name of the Kafka cluster, which is used to determine the Kafka cluster on the Unravel UI.

  5. Apply changes. 

    <Unravel installation directory>/unravel/manager config apply

  6. Restart Unravel.

    <Unravel installation directory>/unravel/manager restart kafka_monitor
For multiple Kafka clients

Warning

Each cluster must have a separate consumerConfig.properties files.

  1. Check the Kafka cluster names that you had added against com.unraveldata.ext.kafka.cluster.list property while configuring Kafka connection. The property is defined with a comma-separated list. If there is only one cluster name see above.

  2. Create a file named consumerConfig<ClusterName>.properties for each cluster. Replace <clustername> with the names in the com.unraveldata.ext.kafka.cluster.list property. 

    ssl.protocol = TLSv1 sasl.mechanism = GSSAPI security.protocol = SASL_SSL 
sasl.kerberos.service.name = kafka 
ssl.truststore.location = /usr/java/jdk1.7.0_67-cloudera/jre/lib/security/jssecacerts1 
ssl.truststore.password = changeit 
ssl.truststore.type = JKS 
ssl.keystore.location = /opt/cloudera/security/jks/server.keystore.jks 
ssl.keystore.password = password ssl.keystore.type = JKS 
ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1 sasl.jaas.config = com.sun.security.auth.module.Krb5LoginModule required \
useKeyTab=true \ keyTab="/etc/keytabs/kafka.keytab" \
 principal="kafka/edge-1.uddev.unraveldata.com@UDDEV.UNRAVELDATA.COM"

  3. Copy/move each file to <Unravel_installation_directory/unravel/data/conf.

  4. Add the following property for each cluster. 

    <Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.ClusterName.consumer.config <Unravel_installation_directory>/unravel/data/conf/consumerConfigClusterName.properties

  5. Apply changes. 

    <Unravel installation directory>/unravel/manager config apply

  6. Restart the Kafka monitor daemon.

    <Unravel installation directory>/unravel/manager restart kafka_monitor
Kafka authorizations

Unravel consumes messages to topic __consumer_offsets using consumer group UnravelOffsetConsumer.

Sentry authorization

The following privilege must be granted using sentry:

HOST=*->CONSUMERGROUP=UnravelOffsetConsumer→action=read
HOST=*->CONSUMERGROUP=UnravelOffsetConsumer→action=write
HOST=*->CONSUMERGROUP=UnravelOffsetConsumer→action=describe
HOST=*->TOPIC=__consumer_offsets→action=read
HOST=*->TOPIC=__consumer_offsets→action=write
HOST=*->TOPIC=__consumer_offsets->action=describe

For further details see Using Kafka with Sentry Authorization in the Cloudera Distribution of Apache Kafka documentation.

Kafka with Ranger authorization

The following privilege must be granted using Ranger for the topic __consumer_offsets.

Publish
Consume
Describe

For further details, see Security - Create a Kafka Policy in the HDP Security Guide.

References

For further information see Apache Kafka documentation chapter # 7 Security.

In this section: