Skip to main content

Home

Configurations

The essential Unravel configurations are set when you install Unravel. Following are additional Unravel configurations that can be set if required.

Setting Unravel configurations

You can update Unravel settings using any of the following methods. Most of the configurations are set by the auto-configuration method when you install Unravel. However, there are some configurations that must be manually configured. For each of the following configurations, instructions are provided based on whether to use the auto-configuration method or the manual configuration method.

Auto-configuration

Run the manager config auto command to pull in the Unravel configurations automatically. Most of the Unravel configurations can be set with this command. When you run this command, the cluster information is automatically read, and the Hadoop properties are configured.

This is the most recommended method to configure Unravel. Auto-configuration is usually run during Unravel installation. However, if you want to update any specific cluster-level changes, you can run auto-configuration as follows:

  1. Stop Unravel

    <Unravel installation directory>/unravel/manager stop

  2. Run auto-configuration

    <unravel_installation_directory>/unravel/manager config auto

  3. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel

    <Unravel installation directory>/unravel/manager start

Manual configuration

The manager service is used to perform manual configurations in Unravel.

Manual configuration with commands

The configurations are set with the manager config <setting> command. Refer to Quick reference - Manager tool commands for all such configurations that can be updated using the manager config <setting> command.

Perform the following steps when you manually set the configurations with manager commands:

  1. Stop Unravel

    <Unravel installation directory>/unravel/manager stop

  2. Change the setting.

    <Unravel installation directory>/unravel/manager config <setting> <subcommand>

For example:
<Unravel installation directory>/unravel/manager config kerberos enable

  3. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply

  4. Start Unravel

    <Unravel installation directory>/unravel/manager start
Manual configuration with properties

Properties can be used to set configurations. You can set the properties either by using the manager config properties set command or by importing a property file where the properties are defined.

  • Option 1: Set properties by importing the properties file

    1. Create a file containing the properties that you want to configure and their corresponding values as shown in the example below:

      com.unraveldata.ext.kafka.cluster.list=CLTR1,CLTR2
com.unraveldata.ext.kafka.CLTR1.bootstrap_servers=broker1-host:9092,broker2-host:9092,broker3-host:9092
com.unraveldata.ext.kafka.CLTR1.jmx_servers=kafka1,kafka2,kafka3
com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.host=broker1-host
com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.port=9393
com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.host=broker2-host
com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.port=9393
com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.host=broker3-host
com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.port=9393
com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.run_period_sec=60
com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.run_period_sec=60
com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.run_period_sec=60
com.unraveldata.ext.kafka.CLTR2.bootstrap_servers=broker1-host:9092,broker2-host:9092,broker3-host:9092
com.unraveldata.ext.kafka.CLTR2.jmx_servers=kafka1,kafka2,kafka3
com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.host=broker1-host
com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.port=9393
com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.host=broker2-host
com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.port=9393
com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.host=broker3-host
com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.port=9393
com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.run_period_sec=60
com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.run_period_sec=60
com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.run_period_sec=60
com.unraveldata.ext.kafka.insight.interval_min=5
com.unraveldata.ext.kafka.insight.lag_threshold=75
com.unraveldata.ext.kafka.insight.sw_size=10

    2. From the installation directory, where Unravel binaries are installed, run the following command and provide the path to the properties file.

      <Unravel installation directory>/unravel/manager config properties import <path to the properties file>
For example:
/opt/unravel-install/unravel/manager config properties import /opt/properties.txt

  • Option 2: Set the properties with manager config properties set

    <Unravel installation directory>/unravel/manager config properties set property key

    For example:

    <Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.cluster.list CLTR1,CLTR2
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.bootstrap_servers broker1-host:9092,broker2-host:9092,broker3-host:9092
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx_servers kafka1,kafka2,kafka3
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.host broker1-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.host broker2-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.host broker3-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka1.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka2.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR1.jmx.kafka3.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.bootstrap_servers broker1-host:9092,broker2-host:9092,broker3-host:9092
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx_servers kafka1,kafka2,kafka3
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.host broker1-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.host broker2-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.host broker3-host
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.port 9393
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka1.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka2.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.CLTR2.jmx.kafka3.run_period_sec 60
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.insight.interval_min 5
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.insight.lag_threshold 75
<Unravel installation directory>/unravel/manager config properties set com.unraveldata.ext.kafka.insight.sw_size 10

    You can also return an updated property to its default value using the manager config <setting> unsetcommand. 

    ##For example
<Unravel installation directory>/unravel/manager config ports unset

Returns a custom port value to it's default value.

Manager service

Unravel manager is a service that provides a unified command-line method to manage all the Unravel configurations and services. It is the central point for interacting with Unravel on the command line. After you have installed Unravel, the manager tool is available in the services and base directory as manager. You can use the manager tool for the following purposes:

  • Manage services

  • Upgrades and downgrades

  • Setting configurations and properties.

  • Integrate all command-line tools such as healthcheck, autoconfig, AppStore apps, installation, and so on.

Understanding manager tool commands

The commands in the manager tool follow the verb-noun pattern.

  • All: Affects all services. For example, the stop all stops all the services while keeping the monitor up.

  • Alias

    • Informal names: stop unravel_sersor_1 or stop us_1 or stop us are equivalent .

    • Group services: stop ma will stop ma_1, ma_2, ma_3. stop etl will stop all ETL-related services.

  • tier:n: To group services per dependency, ex: tier:2 depends on services from tier:1

  • CSL: A comma-separated list of services, aliases, etc. For example,  stop tier:4,ew,ngui stops all tier:4 services, all ew services, and the UI service.

You can also chain and run multiple commands with the manager tool. However, if any of the commands fail, the chain stops. For example, if you run the chained command: manager stop config auto refresh files start watch, it triggers the following actions:

  • Stops Unravel

  • Runs autoconfig in automatic mode.

  • Refreshes all scripts and configuration files

  • Starts Unravel

  • Displays the service watch.

Quick reference - manager commands

You can refer to this section for the manager tool commands. Any commands from this list can be run from the Manager tool as follows:

<Unravel installation directory>/manager <manager command>

For example:
/opt/unravel/manager start

  • Main actions

    The table shows the main actions that are displayed when you run the manager tool. You can run any of these actions for setting up services and configurations.

    Action

    Description

    start

    Start Unravel

    stop

    Stop Unravel

    restart

    Restart Unravel

    report

    Report Unravel services

    watch

    Report Unravel services. Refreshes every two seconds

    service

    Additional service management commands

    config

    Configure Unravel

    refresh

    Refresh scripts and configurations

    activate

    Activate an Unravel version

    version

    Unravel version information

    precheck

    Verify that system configuration supports the currently running Unravel

    healthcheck

    Verify the health of an Unravel installation

    run

    Run Unravel scripts

    api

    Unravel API tools

  • BigQuery

    Commands

    Description

    config bigquery enable/disable

    Enables/disables support for BigQuery.

    manager config bigquery show

    Displays the current configuration set up for BigQuery in Unravel.

    manager config bigquery add <project-id> <subscription-id> </path/to/customer-supplied-credentials-file>

    Adds a single BigQuery project for monitoring by using customer-supplied credentials file.

    manager config bigquery add <project-id> <subscription-id> --create-credentials

    Adds a single BigQuery project for monitoring by using Unravel-managed credentials.

    manager config bigquery add --batch </path/to/project-ids-file> [--subscription-id <subscription-id>] --credentials </path/to/customer-supplied-credentials-file>

    Adds a multiple BigQuery projects for monitoring by using customer-supplied credentials file.

    manager config bigquery add --batch </path/to/project-ids-file> [--subscription-id <subscription-id>] --create-credentials

    Adds a multiple BigQuery projects for monitoring by using Unravel-managed credentials.

    manager config bigquery set-lr-endpoint <host> <port>

    Sets the Log Receiver (LR) endpoint for BigQuery projects in Unravel.

    manager config bigquery integrate [--skip-authorization]

    Integrates the BigQuery projects in Unravel without Unravel's interactive gcloud authentication. You must handle the gcloud cloud authentication on your own.

    manager config bigquery integrate

    Integrates the BigQuery projects in Unravel.

    manager config bigquery enable-datapage --batch <path/to/project-ids-file>

    Add multiple projects to the Data page.

    manager config bigquery enable-datapage <project-id>

    Adds a project to the Data page.

    manager config bigquery delete-datapage <project-id>

    Removes project added to the Data page.

    manager config bigquery delete-datapage --batch <path/to/project-ids-file>

    Removes multiple projects added to the Data page.

    manager config bigquery add

    Adds project details (Project ID, Subscription ID, Credentials file path) for BigQuery monitoring in Unravel.

    manager config bigquery remove

    Remove a project from BigQuery monitoring.

  • Configuration

    Commands

    Description

    config auto

    Starts autoconfig, which is the automatic mode.

    URL, username, and password will be prompted, unless --URL, --username and --password are passed.

    config notebook

    Configures and enables Jupyter notebook service.

    config archive

    Archives the content of data/conf and services into a tar file and stored in data/conf_archives.

    refresh <entity>

    Refreshes the specified entity. Entity can be any of the following:

    • all

    • files

    • kafka

    • db

    • es

    Note

    Stop Unravel before you run refresh <entity>.

  • Custom ports

    Commands

    Description

    help config ports

    Open the config ports menu.

    config ports show

    Display currently defined port values.

    config ports unset

    Returns a custom port value to its default value.

    manager config ports remap begin<specify start range> end<specify end range>

    Set range for Unravel ports.

  • email

    Commands

    Description

    config email

    Configure email.

  • Install and upgrade

    Commands

    Description

    version

    Shows the Unravel version and components information.

    verify <sensor|properties|files>

    • Sensor: Checks the cluster configuration and ensures that the sensor parameters are correct.

    • properties: Validates unravel.properties, checking for invalid, unknown, and duplicate properties.

    • files: Verifies and reports on any unexpected changes to unravel components, scripts, and configuration.

    activate

    Displays a list of available versions of Unravel for activation. This applies to upgrades, downgrades, and patches. The manager tool verifies the compatibility before making any changes.

    activate <version>

    Activates a specific version of Unravel.

    uninstall

    Uninstalls Unravel.

  • Log Receiver (LR) endpoint (Databricks) with TLS

    The following commands are used to update the Log Receiver (LR) endpoint if the Unravel LR is not accessible with the default hostname and port from Databricks, based on various TLS scenarios.

    Commands

    Description

    config databricks set-lr-endpoint <hostname> 4043

    Set LR hostname and LR port number to default, when TLS is NOT enabled.

    config databricks set-lr-endpoint <hostname> 4443

    Set LR hostname and LR port number to default, when TLS is enabled.

    config databricks set-lr-endpoint <hostname> 4043 --no-tls

    Set LR hostname and LR port number to default, when TLS is enabled but you want to keep using HTTP.

  • Monitoring

    Commands

    Description

    report

    Shows the status of all unravel services.

    report <service>

    Shows the status of the specified service.

    watch

    Shows the status of all unravel services.

    watch <service>

    Shows the status of the specified service.

    service list

    Display detailed status information about the service.

    service is-enabled <service>

    Shows whether the service is enabled.

    service is-running <service>

    Shows whether the service is running.

    service is-alive <service>

    Shows whether the service is alive.

    healthcheck

    To run healthcheck.

  • License

    Commands

    Description

    license set <filename>

    Takes a filename as input.

    license show

    Displays the content of the current license and its validity.

  • Kerberos

    Commands

    Description

    config kerberos enable|disable

    Enable/disable the Kerberos support for Unravel. This requires regenerating files.

    config kerberos set

    Allows setting the Kerberos principal path to keytab and optional path to a custom krb5.conf

    Also, refer to Creating Active Directory Kerberos principals and keytabs for Unravel.

  • Service Management

    Commands

    Description

    start

    Starts unravel and all the enabled services.

    start <service>

    Only starts the specified service.

    stop

    Stops unravel and all the running services.

    stop <service>

    Stops only the specified service.

    restart

    Restart all services.

    restart <service>

    Restart only a specific service.

    service enable <service>

    This enables a service that was temporarily disabled.

    Run the service enable <service> --permanent command to update unravel.yaml

    service disable <service>

    Temporarily disables a service. Run the service disable <service> --permanent, which updates the unravel.yaml file and permanently disables a service.

  • TLS

    Command

    Description

    config tls trust enable|disable

    Enable or disable Unravel using a truststore to use when connecting to services secured with TLS. This requires regenerating files.

    config tls trust add <path>

    config tls trust --jks <path>

    config tls trust --pem <path>

    config tls trust --pkcs12 <path>

    Imports certificates to trust. For example, autodetect format is based on the extension.

    Supported <format> are:

    • PEM

    • PKCS12

    • JKS

    <path> is the path to the file to import.

    config tls enable|disable

    Enable/disable using a TLS for Unravel. This requires regenerating files.

    config tls set <path>

    config tls set --jks <path>

    config tls set --pem <path>

    config tls set --pkcs12 <path>

    Set the certificate and key to use for TLS. First example autodetect format based on the extension.

    Supported <format> are:

    • PEM

    • PKCS12

    • JKS

    For PKCS12 and JKS, the user will be prompted for the password. <path> is the path to the file to import.

  • Tagging

    Commands

    Description

    config tagging enable|disable

    Enable/disable tagging support for UnravelThis requires regenerating files.

    config tagging set

    Allows setting the script and the method for tagging.

  • Other configurations

    Commands

    Description

    run <script>

    Runs the Unravel scripts such as unravel_hdp_setup.sh.

    For example:

    manager run unravel_hdp_setup.sh

In this section: