Five days inside Snowflake Summit 2023 – Data Governance & Privacy

I have stated earlier in a blogpost that, next to being a Snowflake Data Cloud enthusiast and part of the Snowflake ❄️ Data Superheroes 2023, I am also Director Data & AI at Pong. We provide Professional Services where we help organizations solve data related challenges. We believe building data products with e.g. Snowflake, can only succeed when the fundament is in place. For us this fundament can be achieved by applying Data Governance, which is all about people, processes, methods and technology.

Snowflake Unified Data Governance - Know, Protect and Share

At Snowflake Summit 2023, I was very curious which innovations are in the pipeline. Here a short overview of the new Data Governance capabilities in Snowflake.

Know your Data

  • Data Quality Monitoring (in Private Preview soon)

Poor Data Quality will cost an organization time, money and reputation. Data owners need a consistent, secure and cost-effective solution to proactively monitor Data Quality issues. Data Quality monitoring in Snowflake will enable organizations to define, measure and monitor Data Quality natively in the Data Cloud.

A Data Metric function allows to define own custom Data Quality metrics with a new schema-level object. There will also be some ready to use system-generated Data Quality metrics like; Freshness, Volume, Accuracy and Statistics.

Snowflake takes care of the measurement in the background, incrementally. No need to specify tasks, just apply a specific metric to a column. All measurement will be automatically recorded in a centralized table for alerting, visualization, and troubleshooting. Snowflake manages this process, but does not have access to this metrics data.

Data Quality metrics can be defined once and reused consistently across multiple objects. There is no management overhead, so no tasks, stored procedures or jobs to manage. The Snowflake solution can be cost effective because it evaluates incremental data only.

The Snowflake Partner eco-system is very important. The idea is that Snowflake will be delivering building blocks for Data Quality Monitoring that the Snowflake Partner eco-system can further leverage and extend.

Governance and Privacy UI experience in Snowsight

The next two developments are an improvement of the Snowsight UI experience.

  • Data Governance UI (in GA soon)

Summary Dashboard in Snowsight of tagged and protected assets with workflows to take action.

  • Classification UI (in Private Preview)

This provides a intuitive interactive (accept or reject Snowflake recommended semantic category tags) workflow to classify data at scale in a desired schema.

  • UK, Australia, Canada-based PII Classification (in Private Preview)

Snowflake has expanded the PII Classification (additional categories and grouping) to support UK, Australia, Canada-based data.

Advanced Data Access Audit.

The following two developments provide granular details about object access and modifications which are recorded for audibility.

  • Tags & Policies History (in Public Preview)

‘Masking Policy assigned to Queried Data’ audits whether an accessed sensitive column in a table or a view had masking policy at the time it was queried.

‘Tag and Policy modification history’ tracks modifications (who and when) to tag and policy associations – tag value, policy body and assignment

  • Table Schema Change History (in Public Preview)

Tracks new tables in sensitive schemas or new columns in monitored tables. This enables immediate action; e.g. trigger classification.

Protect your Data

  • Schema-wide Tag-based Masking (in Public Preview)

This a continuation of Tag-based Masking where you can assign a tag to a masking policy and apply this to a column. With this new Schema-wide Tag-based Masking you can enable the same for all (future) objects in a schema.

  • Query Constraints (in Private Preview)

Query Constraint policies enable organizations to enhance their Data Privacy with policies that constrain query types on protected data. This can be especially useful to be used in Data Clean rooms.

The Aggregation Constraint policy preserves individual privacy by allowing queries about groups, but not individuals.

The Projection Constraint policy, where you can use a column in a where-clause or a join-operation. At the same time this column cannot be exposed in the query-result.

  • Data Access Policies – What’s new
Snowflake Data Access Policies - What's new

Connect your Ecosystem

  • Policies on Shared Objects (in Private Preview)

This allows to create a policy on the provider side ànd at the same time make it relevant at the consumer side. Database Roles will be necessary to make this happen.

Alation Data Governance Partner of the Year

Normally Snowflake and Databricks not always agree. This time they were unanimous. Both techgiants award Alation Data Governance Partner of the Year 2023! For Alation it is the third time in a row. As Alation and Pong are partnering in The Netherlands I thought I should mention that here.

Alation Data Governance Partner of the Year 2023 for the third time in a row.

Check the Snowflake Announcement here.

I have briefly covered the integration between Alation and Snowflake in a previous blogpost. During Snowflake Summit 2023 Alation also announced new innovations.

  • Alation Connected Sheets

This self-service capability enables users to pull in data directly from up-to-date, governed and trusted Snowflake data while working natively in a spreadsheet (Microsoft Excel, Google Sheets).

Find a demo on the Alation website.

  • Alation Open Data Quality Framework

This enables Data Quality information form Snowflake visible in Alation via:

  • Data Health Tab
  • Trust Flags
  • Data Profiling 
  • Data Quality Policies

Read more in this press release.

Till next time.

Director Data & AI at Pong and Snowflake Data Superhero. Online better known as; DaAnalytics.

Daan Bakboord

Bekijk ook:

Snowflake Data Cloud Summit - Wrap Up

Snowflake Snowflake Data Cloud Summit — Wrap Up

Snowflake Data Cloud Summit proved that after all this years the core idea remains the same and is still strong. Technology should serve and Snowflake makes things simple. One Single Unified Platform, one product and one engine. Ease of use and Govenance. Maximum efficiency and maximum simplicity.

Bring the processing of data to the data instead of the other way around. Snowflake as a Platform where you build and share your Data, Apps and AI Products. Your data never has to leave the Platform and Snowflake takes care of this Platform.

Lees verder »
Snowflake Data Cloud Summit - Day II

Snowflake Data Cloud Summit – Day II

The second day of Snowflake Summit in the Moscone Center in San Francisco started with Platform Keynote packed with announcements and demo’s. The announcements were not necessarily completely new, but a continuation of things Snowflake was already working on. Lot’s of Developments and Previews have made it to GA status and are now Generally Available to the public. A few announcements were made as wel.

This blogpost a summarization of my notes with were possible a link to Snowflake publications or documentation.

Lees verder »
Snowflake Data Cloud Summit - Day I

Snowflake Data Cloud Summit – Day I

The first day of Snowflake Summit in the Moscone Center in San Francisco was a day with the first two announcements made during the Opening Keynote. These two announcements built upon the theme for this years Summit; “The Era of Enterprise AI”

Lees verder »