Daanalytics

Five days inside Snowflake Data Governance – Data Governance Accelerated

In a previous blog I mentioned that Snowflake’s Data Governance Framework focuses on three key areas:

  • Know your Data
  • Protect your data
  • Connect your Ecosystem

In this blog we will focus on the key area; ‘Connect your Ecosystem’. Specifically on the Data Governance Accelerated program within Snowflake. Recently Snowflake released a Governance dashboard in Public Preview. On this dashboard you can see some information about Tags and Policies on Tables and Columns.

Snowflake Governance Dashboard in Snowsight

This is a relatively static representation of the Data Governance setup in Snowflake. One of the selected partners in this program is Alation. With Alation you can natively connect to Snowflake, ingest metadata, view en manage applied Data Governance setup in Snowflake.

Snowflake - Alation Data Governance

Data Governance Accelerated

As part of Data Governance Accelerated, Snowflake integrates with Data Governance tools like Alation to enable seamless metadata management, data cataloging, and governance workflows within the Snowflake environment. This integration helps organizations establish a unified data governance framework that spans both the Snowflake platform and their chosen data governance tool.

Alation

How Alation powers Data Intelligence on Snowflake can be experienced by following this QuickStart. Setting things up can be done either via Snowflake Partner Connect or manually. In this blog we will perform a few manual steps to show the Alation-Snowflake integration.

Adding a Snowflake Data Source

First we add a Snowflake Data Source into Alation. We will continue on the Tasty Bytes Quistart dataset. Alation we perform the following functions in Snowflake:

  • Metadata Extraction (MDE),
    • “Job fetches the database’s metadata (including schemas, tables, columns, and procedures/functions) to keep the representation of the database up to date”
  • Profiling/Sampling,
    • “Job periodically selects a limited number of rows from each table in the database to give users of this database a quick preview of the data content. “
  • Query Log Ingestion (QLI)
    • “Alation can generate a wide variety of valuable information on database objects by examining the queries executed on them from a query history log. Some of the generated analysis includes lineage, popularity of the database objects, frequent users of database objects, and commonly used filters and expressions.”

Therefore we need to create a Alation-user in Snowflake which must be assigned a role with enough privileges for the above jobs.

Alation Apps Drop-down

If the Service Account is in place, we can add the mentioned FROSTBYTE_TASTY_BYTES-datasource to Alation. Creating the Data Source is a matter of filling in the connection details. After that the three above mentioned jobs need to be executed; ‘Run Metadata Extraction’, ‘Run Data Sampling’ and ‘Run Query Log Ingestion’.

First you run the ‘Metadata Extraction’. In the Snowflake Query History you can follow and monitor this process. You can see exactly which statements are executed and whether they were successful.

Second you run ‘Data Sampling’. You can choose to either do all the tables or a subset. In this case there are not so many tables, sow we can go for; ‘Sample all tables’. Again the Query History can be followed and monitored to see exactly which statements are executed and whether they were successful.

Third and final is running the ‘Query Log Ingestion’. First you can do a preview. In this case you can see the query’s of the first two jobs. After the Preview, you can import the query logs.

When all three processes have executed well, the Data Source is ready to be used.

Metadata Extraction

Alation Catalog ingested tags from Snowflake

While running Metadata Extraction, also (next to the schema’s, table, etc.) existing Tags will be imported from Snowflake. Make sure that the METADATA_DB-objects are created and available. Then you will be able to see and use the Tags

Via the Policy Center you can see the imported policies we created earlier. Drilling down to the Data Source, Table and Column you can see where the policy is applied.

Alation Catalog Policy Center

Compose Query’s

Alation provides the feature to query Snowflake directly and if desired save these query’s for later use. Querying Snowflake tables includes the Data Governance features, which we applied earlier according to the role one uses in Alation. In the example below you still see the asterisks inn the email-column and the phone_number-column.

Alation Compose query

If you have a query you want to store in the Alation Catalog; e.g. ‘Total Sales for Single Women per City’, you can write the query in compose and publish it to the Alation Catalog

When publishing the query, you can find the query on the associated table (in this case the CUSTOMER_LOYALTY_METRICS_V-view) in the Alation Catalog.

Alation Catalog published query

Because of this stored query, also a join is added to the Alation Catalog.

Alation Catalog Table joins

Data Lineage

Based on the imported Metadata and the Query Log Ingestion, Alation is capable of providing Data Lineage information. This information is very helpful when implementing changes to existing objects.

Alation Data Lineage

Alation Data lineage enables Impact Analysis (show objects that are impacted downstream from an object) and Upstream Audit (show upstream objects that impact an object).

Wrapping up – ‘Five days inside Snowflake Data Governance’

This blogpost was a brief introduction into Alation for Snowflake. As we have seen, Alation provides users with a centralized location to discover, understand, and collaborate on data assets stored in Snowflake. With a few simple steps Snowflake assets can be integrated into the the Alation platform and managed from there.

In following blogpost I’ll probably go deeper into the Alation functionalities for Snowflake.

With this last post there comes an end to five days inside Snowflake Data Governance. In five different blogpost we have covered various features of Data Governance and Snowflake. Using the Tasty Bytes Quickstart dataset we could create various examples.

We started to ‘Know your Data’. This aspect of Snowflake’s Data Governance Framework is all about understanding your data and its quality, lineage, and usage.

In our third blogpost we showed how to ‘Protect your Data’. Data protection is a critical component of Snowflake’s Data Governance Framework. These features enable you to define and enforce data access policies, monitor data access and usage, and ensure compliance with industry and government regulations.

The last key area in Snowflake’s Data Governance Framework is; ‘Connect your Ecosystem’. Collaboration is one of the important workloads in the Data Cloud. Snowflake Data Governance provides the necessary controls and mechanisms to ensure secure collaboration within the Snowflake Data Cloud.

An example of how Snowflake connects with its ecosystem is the Data Governance Accelerated program, where Snowflake integrates with Data Governance tools like Alation to enable seamless metadata management, data cataloging, and governance workflows within the Snowflake environment. We used some examples to show the integration between Alation and Snowflake. In later blogposts we will dive deeper into this integration.

Till next time.

Director Data & AI at Pong and Snowflake Data Superhero. Online better known as; DaAnalytics.

Daan Bakboord

Bekijk ook:

Snowflake Data Cloud Summit - Wrap Up

Snowflake Snowflake Data Cloud Summit — Wrap Up

Snowflake Data Cloud Summit proved that after all this years the core idea remains the same and is still strong. Technology should serve and Snowflake makes things simple. One Single Unified Platform, one product and one engine. Ease of use and Govenance. Maximum efficiency and maximum simplicity.

Bring the processing of data to the data instead of the other way around. Snowflake as a Platform where you build and share your Data, Apps and AI Products. Your data never has to leave the Platform and Snowflake takes care of this Platform.

Lees verder »
Snowflake Data Cloud Summit - Day II

Snowflake Data Cloud Summit – Day II

The second day of Snowflake Summit in the Moscone Center in San Francisco started with Platform Keynote packed with announcements and demo’s. The announcements were not necessarily completely new, but a continuation of things Snowflake was already working on. Lot’s of Developments and Previews have made it to GA status and are now Generally Available to the public. A few announcements were made as wel.

This blogpost a summarization of my notes with were possible a link to Snowflake publications or documentation.

Lees verder »
Snowflake Data Cloud Summit - Day I

Snowflake Data Cloud Summit – Day I

The first day of Snowflake Summit in the Moscone Center in San Francisco was a day with the first two announcements made during the Opening Keynote. These two announcements built upon the theme for this years Summit; “The Era of Enterprise AI”

Lees verder »