Skip to main content

Home

Connecting to a Kafka cluster

To connect Unravel server to a Kafka cluster, change the configuration of the Kafka cluster and add some properties to unravel.properties.

  1. Change the configuration of the Kafka cluster.

    1. Export an available port for JMX_PORT. You need to export one port for each Kafka server.

      export JMX_PORT=jmx-port

      Tip

      The default JMX port for Kafka in CDH is 9393.

      In HDP you would export this parameter in Ambari Web UI under Advanced kafka-env | kafka-env template.

    2. Enable remote access for JMX monitoring by appending the following lines to KAFKA_JMX_OPTS in kafka_run_class.sh:

      -Dcom.sun.management.jmxremote.rmi.port=$JMX_PORT
-Dcom.sun.management.jmxremote.port=$JMX_PORT
-Djava.rmi.server.hostname=public-hostname
-Djava.net.preferIPv4Stack=true

      Tip

      Not required for HDP.

    3. Verify the configuration changes on the Kafka cluster.

    4. Restart the Kafka broker.

  2. In /usr/local/unravel/etc/unravel.properties, add properties to monitor the Kafka cluster.

    The Unravel daemon, unravel_km, relies on a list of Kafka servers to query and discover the entire cluster. You need to add this list of Kafka servers to /usr/local/unravel/etc/unravel.properties.

    For complete descriptions of properties, see Kafka Monitoring Properties.

    com.unraveldata.ext.kafka.clusters=cluster-list
com.unraveldata.ext.kafka.cluster-id.bootstrap_servers=boot-servers
com.unraveldata.ext.kafka.cluster-id.jmx_servers=jmx-servers
com.unraveldata.ext.kafka.cluster-id.jmx.jmx-server-id.host=jmx-hosts
com.unraveldata.ext.kafka.cluster-id.jmx.jmx-server-id.port=jmx-port

    For example,

    com.unraveldata.ext.kafka.clusters=c1,c2 
com.unraveldata.ext.kafka.c1.bootstrap_servers=localhost:9092,localhost:9093 
com.unraveldata.ext.kafka.c2.bootstrap_servers=localhost:9192,localhost:9193 
com.unraveldata.ext.kafka.c1.jmx_servers=kafka-test1,kafka-test2 
com.unraveldata.ext.kafka.c2.jmx_servers=kafka-test1,kafka-test2 
com.unraveldata.ext.kafka.c1.jmx.kafka-test1.host=localhost 
com.unraveldata.ext.kafka.c1.jmx.kafka-test2.host=localhost 
com.unraveldata.ext.kafka.c2.jmx.kafka-test1.host=localhost 
com.unraveldata.ext.kafka.c2.jmx.kafka-test2.host=localhost 
com.unraveldata.ext.kafka.c1.jmx.kafka-test1.port=5005 
com.unraveldata.ext.kafka.c1.jmx.kafka-test2.port=5010 
com.unraveldata.ext.kafka.c2.jmx.kafka-test1.port=5105 
com.unraveldata.ext.kafka.c2.jmx.kafka-test2.port=5110
In this section: