Spark S3 specific properties

Home

Spark S3 specific properties

Property/Description	Set by user	Unit	Default
com.unraveldata.s3.profile.config.file.path The path to the s3 profile file, e.g., `/usr/local/unravel/etc/s3ro.properties`.		string	-
com.unraveldata.spark.s3.profilesToBuckets Comma separated list of profile to bucket mappings in the following format: <s3_profile>:<s3_bucket>, for example, com.unraveldata.spark.s3.profileToBuckets=profile-prod:com.unraveldata.dev,profile-dev:com.unraveldata.dev. Note Important Ensure that the profiles defined in the property above are actually present in the s3 properties file and that each profile has associated a corresponding pair of credentials `aws_access_key` and `aws_secret_access_key`. The old format: `access_key/secretKey` is no longer supported.)		CSL	-

Property/Description

Set by user

Unit

Default

com.unraveldata.s3.profile.config.file.path

The path to the s3 profile file, e.g., /usr/local/unravel/etc/s3ro.properties.

string

com.unraveldata.spark.s3.profilesToBuckets

Comma separated list of profile to bucket mappings in the following format: <s3_profile>:<s3_bucket>, for example, com.unraveldata.spark.s3.profileToBuckets=profile-prod:com.unraveldata.dev,profile-dev:com.unraveldata.dev.

Note

Important

Ensure that the profiles defined in the property above are actually present in the s3 properties file and that each profile has associated a corresponding pair of credentials aws_access_key and aws_secret_access_key. The old format: access_key/secretKey is no longer supported.)

CSL

Was this helpful?

Would you like to provide feedback? Just click here to suggest edits.

Home

Spark S3 specific properties

Important

Search results