Finding Unravel Properties in Azure

Home

Finding Unravel Properties in Azure

HDInsight Block Storage Properties

For a storage account named storage-name the corresponding storage account name is fs.azure.account.key.storage-name.blob.core.windows.net. A storage account name has two access keys. Both access keys are required.

Property/Description	Set By User	Unit	Default
com.unraveldata.hdinsight.storage-account. `X` Storage account name that a HDInsight cluster uses. You must define this property for each storage account. X starts with 1 and then is incremented by 1 for each additional account. The account numbers must be consecutive.	Optional	string	-
com.unraveldata.hdinsight.access-key. `X` Storage account key. For each storage account storage-account.`X` you must define access-key.`X` If you have two (2) access keys, pick one to use here.	Optional	string	-

Property/Description

Set By User

Unit

Default

com.unraveldata.hdinsight.storage-account. X

Storage account name that a HDInsight cluster uses.

You must define this property for each storage account. X starts with 1 and then is incremented by 1 for each additional account. The account numbers must be consecutive.

Optional

string

com.unraveldata.hdinsight.access-key. X

Storage account key.

For each storage account storage-account.X you must define access-key.X If you have two (2) access keys, pick one to use here.

Optional

string

For example, for three (3) storage accounts you would define three (3) pairs of <storage name, storage access key>.

com.unraveldata.hdinsight.storage-account.1=Storage1
com.unraveldata.hdinsight.access-key.1=Storage1AccessKey
com.unraveldata.hdinsight.storage-account.2=Storage2
com.unraveldata.hdinsight.access-key.2=Storage2AccessKey
com.unraveldata.hdinsight.storage-account.3=Storage3
com.unraveldata.hdinsight.access-key.3=Storage3AccessKey

Navigate to https://portal.azure.com
Find the name of the storage account.
1. In the Azure dashboard, click HDInsight clusters, select the cluster you want, and click Storage accounts.
2. Copy the storage name (storage-name).
  Tip
  The complete name of your storage is fs.azure.accountkey.storage-name.blob.core.windows.net.
3. Set both com.unraveldata.hdinsight.storage-account.1 and com.unraveldata.hdinsight.storage-account.2 to this name.
Find the storage account's access keys.
1. From the Azure portal, select Storage Accounts | storage-name | Access Keys.
2. Set com.unraveldata.hdinsight.access-key.1 to key1.
3. Set com.unraveldata.hdinsight.access-key.2 to key2.

Azure Data Lake Storage (ADLS) Properties

The following properties are set with information obtained from the ADL properties in Microsoft Azure. If you are using Azure Blob storage (WASB), you can skip these properties.

Property	Definition	Default
com.unraveldata.adl.accountFQDN	The data lake fully qualified domain name. For example, `mydatalake.azuredatalakestore.net`	Retrieve from Microsoft Azure
com.unraveldata.adl.clientId	Also known as the application ID. This is the application registration you created in the Azure Active Directory.	Retrieve from Microsoft Azure
com.unraveldata.adl.clientKey	Also known as the application access key. You can create this key after registering the application.	Retrieve from Microsoft Azure
com.unraveldata.adl.accessTokenEndpoint	The OAUTH 2.0 token endpoint as shown in the application registration tab.	Retrieve from Microsoft Azure
com.unraveldata.adl.clientRootPath	The path in the data lake store where the target cluster has been given access. For example, on your deployment cluster `spk21utj02` has been given access to `/clusters/spk21utj02` on the data lake store.	Retrieve from Microsoft Azure

Locate your account's fully qualified domain name (FQDN).
1. Select Data Lake Storage | Data Explorer, and select the folder in the Data Explorer panel.
  In the screenshot below, the folder name is datalake0001.
2. Click Folder properties.
3. Copy your account's FQDN in the Properties panel under PATH, and set com.unraveldata.adl.accountFQDN to this value.
  For example,
```
com.unraveldata.adl.accountFQDN=datalake0001.azuredatalakestore.net
```
  Note
  Don't include the leading adl:// in com.unraveldata.adl.accountFQDN.
Locate the OAUTH 2.0 token endpoint, application ID, and application key.
1. From the Azure portal, select Azure Active Directory | App Registrations | Endpoints.
2. Set com.unraveldata.adl.accessTokenEndpoint to the value of OAUTH 2.0 TOKEN ENDPOINT.
3. Click View all applications.
  - If your application is listed, select it.
  - If your application isn't listed, you need to create a new application:
    select + New application registration, fill in the application information, and select Create.
    Note
    You need to grant access to the new application. For steps, see Grant Access to a Newly Created App.
4. Set com.unraveldata.adl.clientID to your application ID.
  Whether you selected an existing application or created a new one, find the the app in the registered app panel. This panel contains your application ID.
5. Select Settings | Keys.
6. Create a new key.
7. Fill in the Description and select Duration.
8. To generate the key, click Save.
  Warning
  Copy and save the key value immediately. If you don't save it, when you leave this page you can't retrieve the key and must create a new one.
9. Set com.unraveldata.adl.clientKey to the key value.

Grant Access to a Newly Created Application

Select Data Lake Store | Data Explorer.
Select Access | +Add.
Click OK, select your app (in the screenshot, it's testTest) and click Select.
Grant the permissions to your app and click OK.
Your app is now listed under Assigned Permissions with the permissions noted.