Azure
The following properties are required if you are using an Azure Data Lake Storage Gen2 account. You can either use the combination of account name and access key properties or you can use the combination of account name, tenant ID, and client ID properties to point Unravel to your Azure storage accounts.
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.azure.storage.abfs.account-name. Name of the ABFS storage account that the HDInsight cluster uses. You must define this property for each ABFS storage account.X. X=1 for the first storage account and then is incremented by one for each new account, that is, account numbers must be consecutive. (See finding the storage name.) | Optional | string | Azure storage account name. |
com.unraveldata.azure.storage.abfs.access-key. The access key for the corresponding ABFS storage account. (See finding the secret (access key).) | Optional | string | Azure storage account name. |
You must define these two properties for each storage account name.X starts with 1 and for each new account is incremented by 1 for each new storage account, that is, the set of properties must be consecutively numbered, (1, 2, 3, ...). For example, if you have two storage accounts you would define two sets.
com.unraveldata.azure.storage.abfs.account-name.1=abfsAccountName1 com.unraveldata.azure.storage.abfs.access-key.1=abfsAccessKey1 com.unraveldata.azure.storage.abfs.account-name.2=abfsAccountName2 com.unraveldata.azure.storage.abfs.access-key.2=abfsAccessKey2
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.azure.storage.abfs.account-name. Name of the ABFS storage account that the HDInsight cluster uses. You must define this property for each ABFS storage account.X. X=1 for the first storage account and then is incremented by one for each new account, that is, account numbers must be consecutive. (See finding the storage name.) | Optional | string | Azure storage account name. |
com.unraveldata.azure.storage.abfs.tenant. Tenant ID. This corresponds to fs.azure.account.oauth2.msi.tenant for account Use the directory ID as the Tenant ID. (See finding the Directory ID) (See finding the Tenant ID) | Required | string | Tenant ID of the managed identity. |
com.unraveldata.azure.storage.abfs.client-id. Client ID This corresponds to fs.azure.account.oauth2.client.id for account (See finding the Client ID) | Required | string | Client ID of the managed identity |
These properties are required if you are using an Azure Data Lake Storage.
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.azure.storage.adl.account-name. The Azure Data Lake Gen1 storage account. The name does not need to be fully qualified. For instance, you can use mydatalake or mydatalake.azuredatalakestore.net. You must define this property for each storage account. | Optional | string | Azure storage account name. (See finding the storage name.) |
com.unraveldata.azure.storage.adl.client-id. An application ID. An application registration has to be created in the Azure Active Directory. | Optional | string | Azure application id. (See finding the application Id.) |
com.unraveldata.azure.storage.adl.client-key. An application's "secret" (key) described in the ADL Gen1 client-id field. | Optional | string | Azure storage secret. (See finding the secret (access key).) |
com.unraveldata.azure.storage.adl.access-token-endpoint. The OAUTH 2.0 Access Token Endpoint. It is obtained from the application registration tab on Azure portal. | Optional | string | Azure OAUTH 2.0 token endpoint (See finding the OAUTH endpoint.) |
You must define these four properties for each storage account name.X starts with 1 and is incremented by 1 for each new storage account, that is, the set of properties must be consecutively numbered, (1, 2, 3, ...). For example, if you have two storage accounts you would define two sets.
com.unraveldata.azure.storage.adl.account-name.1=adlAccountName1 com.unraveldata.azure.storage.adl.client-id.1=adlClientId1 com.unraveldata.azure.storage.adl.client-key.1=adlClientKey1 com.unraveldata.azure.storage.adl.access-token-endpoint.1=adlTokenEndpoint1 com.unraveldata.azure.storage.adl.account-name.2=adlAccountName2 com.unraveldata.azure.storage.adl.client-id.2=adlClientId2 com.unraveldata.azure.storage.adl.client-key.2=adlClientKey2 com.unraveldata.azure.storage.adl.access-token-endpoint.2=adlTokenEndpoint2
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.databricks.HTTP.conn.timeout Databricks HTTP connection timeout in seconds. | s | 1000 | |
com.unraveldata.databricks.http.read.timeout Databricks http read timeout in seconds. | s | 6000 | |
com.unraveldata.databricks.http.poll.parallelism Databricks poll parallelism; expressed as a fraction of the total cores. | percent | .75 | |
com.unraveldata.databricks.workspaces Databricks workspace to monitor. | CSL | - |
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.login.mode=aad Set the login mode to Azure Active Directory (AAD) | Required | string | - |
com.unraveldata.login.aad.config Specify the path to the | Required | string | /usr/local/unravel/etc/aad.json |
These properties are required if you are using a WASB storage account.
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.azure.storage.wasb.account-name. Name of the WASB storage account that the HDInsight cluster uses. You must define this property for each WASB storage account. | Optional | string | Azure storage account name. (See finding the storage name.) |
com.unraveldata.azure.storage.wasb.access-key. WASB storage account key. For each storage account defined you must define the storage access key. If you have two keys, pick one to use here. | Optional | string | Azure storage account access key. (See finding the access key.) |
You must define these two properties for each storage account name. X starts with 1 and is incremented by 1 for each new storage account, that is, the set of properties must be consecutively numbered (1, 2, 3, ...). For example, if you have three storage accounts you would define three sets.
com.unraveldata.azure.storage.wasb.account-name.1=Storage1 com.unraveldata.azure.storage.wasb.access-key.1=Storage1AccessKey com.unraveldata.azure.storage.wasb.account-name.2=Storage2 com.unraveldata.azure.storage.wasb.access-key.2=Storage2AccessKey com.unraveldata.azure.storage.wasb.account-name.3=Storage3 com.unraveldata.azure.storage.wasb.access-key.3=Storage3AccessKey
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.adl.accountFQDN The data lake's fully qualified domain name, for example, mydatalake.azuredatalakestore.net. | Optional | string | Azure storage account name. (See finding the storage name.) |
com.unraveldata.adl.clientId An application ID. An application registration has to be created in the Azure Active Directory. | Optional | string | Azure application id. (See finding the application Id.) |
com.unraveldata.adl.clientKey An application access key which can be created after registering an application. | Optional | string | Azure storage access key. (See finding the storage access key.) |
com.unraveldata.adl.accessTokenEndpoint The OAUTH 2.0 Access Token Endpoint. It is obtained from the application registration tab on Azure portal. | Optional | string | Azure OAUTH 2.0 token endpoint (See finding the OAUTH endpoint.) |
com.unraveldata.adl.clientRootPath The path in the Data lake store where the target cluster has been given access. | Optional | string URL | Azure CONTAINER/DIRECTORY path for storage account name. (See finding the container path.) |
Property/Description | Set by user | Unit | Default |
|---|---|---|---|
com.unraveldata.hdinsight.storage-account. Storage account name that a HDInsight cluster uses. You must define this property for each storage account. | Optional | string | Azure storage account name. (See finding the storage name.) |
com.unraveldata.hdinsight.access-key. Storage account key. For each storage-account. | Optional | string | Azure storage account key. (See finding the access key.) |
You must define these two properties for each storage account name. X starts with 1 and is incremented by 1 for each new storage account, that is, the set of properties must be consecutively numbered (1, 2, 3, ...). For example, if you have three storage accounts you would define three sets.
com.unraveldata.hdinsight.storage-account.1=Storage1 com.unraveldata.hdinsight.access-key.1=Storage1AccessKey com.unraveldata.hdinsight.storage-account.2=Storage2 com.unraveldata.hdinsight.access-key.2=Storage2AccessKey com.unraveldata.hdinsight.storage-account.3=Storage3 com.unraveldata.hdinsight.access-key.3=Storage3AccessKey