Skip to main content

Snowflake Documentation

Secured data sharing

Unravel leverages the Snowflake data sharing with secure data sharing option for data transfer over shared connections outside the Virtual Private Cloud (VPC) or through private links. This involves employing a serverless/warehouse task to independently replicate metadata from the schema in the customer Snowflake account, allowing you to share metadata directly with the Unravel Snowflake account in the same region. The data is polled and shared every 1 hour by default. The polling can be configured.

For secure data share from Snowflake, complete the prerequisites and configure the Secure data share option.

The solution provides the following benefits:

  • No requirement for a direct connection to Snowflake

  • Rapid setup and reduced complexity for security permissions

  • Availability of all Unravel features with configurable latency

Architecture

Unravel Snowflake Architecture - Secure Share

Snowflake-architecture.png

The Unravel Snowflake Architecture - Secure share is a comprehensive framework designed to integrate with the Snowflake data warehouse, providing users with robust data analytics and management capabilities. The architecture is divided into three main layers: User Layer, Core Layer, and Data Source Layer, each serving distinct purposes and functionalities.

The architecture emphasizes secure communication pathways, including external connections, HTTP/HTTPS traffic, and internal communications, ensuring data integrity and confidentiality. It provides a scalable and efficient platform for data analytics, capable of integrating with various external systems and notifying users through multiple channels.

  • User Layer: This layer interfaces directly with users and clients, offering web application access, REST API interactions, and external notifications through platforms such as Email, Slack, and Microsoft Teams. It ensures secure and authenticated access using username/password for web users and revocable tokens for API clients.

  • Core Layer: Deployed on virtual machines or cloud services like AWS, Azure, and SaaS, this layer includes critical components such as NGINX for load balancing, a notification service, various analytical and service modules, and a datastore. The Core Layer facilitates seamless data management and insight generation through components like the Intelligent Insight Generator (IIG) and Snowflake Worker. It supports development environments like Python and Java and runs on CentOS.

  • Data Source Layer: This layer securely connects to the Snowflake data warehouse using a Snowflake-JDBC secure link with local authentication. It handles both shared and customer-specific data, ensuring that data operations are performed within the same region (us-east-1) to optimize performance and reduce latency.

Sharing Data with Secure Data Sharing

Snowflake-Sharingdata.png

This diagram outlines the process of securely sharing metadata between a customer's Snowflake account and Unravel (or a partner) using the secure data sharing feature. This setup ensures that customer metadata remains within the customer's control while allowing Unravel to access and analyze the metadata as needed.

The workflow involves three main steps:

This process ensures secure, efficient, and on-demand access to customer metadata, enabling Unravel to perform analysis and monitoring

  • Export Metadata: Customers run scripts to create databases, schemas, and tables, exporting the metadata to transient tables.

  • Share Metadata via Snowshare: The customer uses Snowflake's secure share feature to share the exported metadata with Unravel.

  • Poll Metadata: Unravel polls the shared metadata directly from the customer's storage through Unravel's Snowflake warehouse, without storing it on their end.

Prerequisites for secured Data sharing

The following prerequisites must be fulfilled before you use the secured data-sharing option:

Permissions and Roles

  • Assign the ACCOUNTADMIN role to the user.

  • Grant SELECT permission on the schema SNOWFLAKE.ACCOUNT_USAGE.

Unravel Account ID

Contact Unravel Support for Unravel account ID corresponding to the platform, region, and edition.

Configuring the Secure data sharing option

Unravel provides two methods for secure data sharing between the customer and Unravel:

  • Health Check: One-time data share from the customer to Unravel.

  • Snow-Share Live: Continuous data sharing from the customer to Unravel.

Health Check: One-Time Data Share

Use the Health Check option to perform a one-time secure data transfer from your Snowflake environment to Unravel. This method is typically used for a snapshot of your data at a particular point in time for analysis and recommendations.

To configure the Health Check option:

  1. Run the SQL statement from this link to create procedures and necessary functions.

  2. You must replace it with the correct <Unravel Account ID> based on your Snowflake region and account type. Contact Unravel Support to obtain the appropriate Account ID for your region.

    /**
Step-1 (One time execution for health check for 35 days)
*/
CALL CREATE_TABLES('SECURE_SHARE','SCHEMA_4825');
CALL REPLICATE_ACCOUNT_USAGE('SECURE_SHARE','SCHEMA_4825', 35);
CALL REPLICATE_HISTORY_QUERY('SECURE_SHARE','SCHEMA_4825', 35);
CALL WAREHOUSE_PROC('SECURE_SHARE','SCHEMA_4825');
CALL CREATE_QUERY_PROFILE(dbname => 'SECURE_SHARE', schemaname => 'SCHEMA_4825', credit =>
'1', days => '15');
/**
Select one procedure from REPLICATE_REALTIME_QUERY or
REPLICATE_REALTIME_QUERY_BY_WAREHOUSE based on requirement.
Select and run REPLICATE_REALTIME_QUERY procedure if you wish to get real-time queries for all
warehouses. It will select a maximum of 10,000 real-time queries across all warehouses at intervals of 48
hours.
*/
CALL REPLICATE_REALTIME_QUERY('SECURE_SHARE','SCHEMA_4825', 48);
/**
SHARE tables to given accountId
*/
CALL SHARE_TO_ACCOUNT('<Unravel Account ID>');
Snow-Share Live: Continuous Data Share

Use the Snow-Share Live option to set up continuous, secure data sharing between your Snowflake environment and Unravel.

To configure the snow-share live option:

  1. Keep the following information ready:

    • Unravel account ID (provided by Unravel based on your Snowflake region). You must replace it with the correct <Unravel Account ID> based on your Snowflake region and account type. Contact Unravel Support to obtain the appropriate Account ID for your region.

    • Warehouse name (created in your Snowflake account). You must replace SET WAREHOUSE_NAME = '<WAREHOUSE_NAME>'; with the created warehouse name.

  2. Run the SQL statement from this link to create procedures and necessary functions.

  3. You can also update the following optional parameters if required. The default values will be applicable if these are not updated.

    • DATABASE_TO_SHARE: Name of the database to share.

    • SCHEMA_TO_SHARE: Name of the schema to share.

    • SHARE_NAME: Name of the secure share to create.

    • PROFILE_QUERY_CREDIT: Credits allocated for query profiling.

    • ACCOUNT_ID: Your Snowflake account ID.

    • R_DAYS: Number of days of real-time queries to poll.

    • H_DAYS: Number of days of historical queries to poll.

    • DAYS_TO_KEEP: Number of days to retain query and access history table data.

    • Task schedules for:

      • REPLICATE_METADATA

      • REPLICATE_STORAGE_METADATA

      • REPLICATE_HISTORY_QUERY

      • CREATE_PROFILE_TABLE

      • REPLICATE_WAREHOUSE_AND_REALTIME_QUERY

      • CLEANUP_DATA_TASK

In this section: