- Snowflake Documentation
- Unravel Documentation
- Reference Guide
- Unravel Properties
- Snowflake
- Configure Polling (Snowflake)
Configure Polling (Snowflake)
The following properties can be configured for Polling in Snowflake.
Property/Description | Set by User | Unit | Default |
|---|---|---|---|
com.unraveldata.sf.account.usage.poll.latency.secs Latency to poll data from Snowflake by ACCOUNT_USAGE related pipelines. When set to 4 hours (4 * 60 * 60), Unravel will retain data up to the previous 4-hour period. This is used for both Secured data sharing and Realtime modes. | Mandatory | Seconds | 14400 |
com.unraveldata.sf.data.replication.latency.secs Snowflake job execution interval between two jobs for all ACCOUNT_USAGE tables to replicate the Snowflake data from the Source account to the Destination account. Until the job is executed, data is not available in Snowflake to poll. This is required for Secured data sharing. | Mandatory | Seconds | 3600 |
com.unraveldata.sf.data.replication.latency.rt.secs Snowflake job execution interval between two jobs for INFORMATION_SCHEMA (RT data) tables to replicate the Snowflake data from the Source account to the Destination account. Until the job is executed, data is not available in Snowflake to poll. This is required for Secured data sharing. | Mandatory | Seconds | 1800 |
com.unraveldata.sf.warehouse Specifies the name of the warehouse allocated to Unravel. | Mandatory | String | |
com.unraveldata.sf.database Specifies the name of the database from which data is polled. | Optional | String | SNOWFLAKE |
com.unraveldata.sf.compute.base.poll.secs Sets the polling interval, in seconds, for warehouses and tables. | Optional | Seconds | 1800 |
com.unraveldata.sf.compute.events.poll.secs Specifies the polling interval, in seconds, for warehouse events. | Optional | Seconds | 900 |
com.unraveldata.sf.compute.load.poll.secs Sets the polling interval, in seconds, for querying warehouse load tables. | Optional | Seconds | 300 |
com.unraveldata.sf.compute.charge.poll.secs Specifies the polling interval, in seconds, for warehouse charge and metering tables. | Optional | Seconds | 3600 |
com.unraveldata.sf.query.rt.poll.secs Sets the polling interval, in seconds, to poll the real-time query_history (INFORMATION_SCHEMA) table. NoteIn case the number of executed queries is greater than 10K in the set polling interval, then those excess records do not get polled in the real-time pipeline. To avoid this issue, you must adjust the poll interval for the real-time query pipeline and, thereby, configure the poll interval to a lesser value. You may consider setting it to 120-300 seconds if the count of queries being executed in default 15 mins is exceeding 10K records. If any records are missed during real-time polling, all the queries get polled in the query_history (ACCOUNT_USAGE) pipeline, configured using the com.unraveldata.sf.query.history.poll.secs property. | Optional | Seconds | 900 |
com.unraveldata.sf.query.history.poll.secs Specifies the polling interval, in seconds, for query_history (ACCOUNT_USAGE) table. | Optional | Seconds | 3600 |
com.unraveldata.sf.look.back.days Specifies the maximum number of days to look back when initially polling Snowflake for data. | Optional | Days | 60 |
com.unraveldata.sf.overwrite.record By default, Unravel loads the data from the last loaded timestamp for each category (R - Realtime, H - History, L - Warehouse Load, and so on). If you want to re-load the data for the configured lookback period, then this property can be set to true. Then, the loaded data is ignored, and the data is polled again for the configured lookback period. | Optional | boolean | False |
com.unraveldata.sf.login.account.user.query.enable Enables or disables Snowflake account user-related queries. By default, all user account-level queries are disabled and not shown on the user interface. | Optional | Boolean | False |