Home

Configure Spark properties

Run the following steps to configure Spark properties

  1. Stop Unravel.

    <Unravel installation directory>/unravel/manager stop
    
  2. From the installation, set the properties using any of the following option:

    • Option 1: Set properties by importing the properties file

      1. Create a file containing the properties that you want to configure and their corresponding values. For example:

        com.unraveldata.spark.live.pipeline.maxStoredStages=3000
        com.unraveldata.spark.master=local
        com.unraveldata.spark.eventlog.maxSize=2000000000
        com.unraveldata.spark.eventlog.appDuration.mins=2880
        com.unraveldata.spark.hadoopFsMulti.useFilteredFiles=true
        com.unraveldata.spark.appLoading.maxAttempts=4
        com.unraveldata.spark.events.enableCaching=true
      2. From the installation directory, where Unravel binaries are installed, run the following command and provide the path to the properties file.

        <Unravel installation directory>/unravel/manager config properties import <path to the properties file>
        
        ##For example:
        /opt/unravel-install/unravel/manager config properties import /opt/properties.txt
    • Option 2: Set the properties with manager config properties set

      <Unravel installation directory>/unravel/manager config properties set property key
      

      For example:

      <Unravel installation directory>/unravel/manager config properties set com.unraveldata.spark.live.pipeline.maxStoredStages 3000
      <Unravel installation directory>/unravel/manager config properties set com.unraveldata.spark.master local
      <Unravel installation directory>/unravel/manager config properties set com.unraveldata.spark.eventlog.maxSize 2000000000
      <Unravel installation directory>/unravel/manager config properties set com.unraveldata.spark.eventlog.appDuration.mins 2880
      <Unravel installation directory>/unravel/manager config properties set com.unraveldata.spark.hadoopFsMulti.useFilteredFiles true
      <Unravel installation directory>/unravel/manager config properties set com.unraveldata.spark.appLoading.maxAttempts 4
      <Unravel installation directory>/unravel/manager config properties set com.unraveldata.spark.events.enableCaching true
      

    Refer to Spark properties for the list of properties that can be configured.

  3. Apply the changes.

    <Unravel installation directory>/unravel/manager config apply
    
  4. Start Unravel

    <Unravel installation directory>/unravel/manager start