Loading Kaggle Data into Snowflake with the Kaggle API

If you want to load Kaggle data into Snowflake without manually downloading datasets, the Kaggle API can automate the first part of the workflow.

In this example, I use the Kaggle API to authenticate, download and unzip a dataset before continuing with the process of loading the CSV data into Snowflake. This example is different than the example in a previous blogpost I showed how to load .csv-files into Snowflake

How does the Kaggle-to-Snowflake workflow work?

The workflow in this example consists of two parts. First, the Kaggle API is used to authenticate and download the dataset. The downloaded files are then unzipped and prepared for the existing CSV-to-Snowflake loading process.

In short:

1. Authenticate with the Kaggle API
2. Download the Kaggle dataset
3. Unzip the downloaded files
4. Read the CSV data
5. Create the required Snowflake objects
6. Load the data into Snowflake

Install Kaggle

I have used Kaggle in a Anaconda environment. Therefore I have a separate environment in which I installed Kaggle.

Kaggle API key

First we have to create a Kaggle API key which is necessary to connect to Kaggle. If you have a Kaggle account, you can create new API Token from you account settings (https://www.kaggle.com/<user_name>/account).

Standard implementation

Clicking the button above generates a kaggle.json-file. This file needs to be stored in a folder called .kaggle in your home directory. The kaggle.json-file has the following structure:

Bekijk code op GitHub Gist

In the Python-script you can use the OS environment variables directly to authenticate, like presented below:

Bekijk code op GitHub Gist

Customised example

For this example, I was curious whether I could include the kaggle.json content to the Credentials-file I used in my previous example.

Bekijk code op GitHub Gist

Authentication in this customised example goes hand in hand with the authentication to Snowflake. The same Credentials-file is referenced for both Snowflake as well as Kagggle:

Bekijk code op GitHub Gist

Download from Kaggle

Next step is downloading files from Kaggle. For this we reference the Kaggle API, specifically; the dataset_download_files() method

Bekijk code op GitHub Gist

Unzip records

Data from Kaggle is downloaded in .zip-format. You can unzip the files from within the Kaggle API; 'unzip=True'.

An alternative is to unzip the files via the statement below:

Bekijk code op GitHub Gist

Continuing

The remainder of is similar to the previous post; From .csv to Snowflake.

• Reading .csv Data
• Creating Snowflake objects
• Loading Data into Snowflake

Find the code for this blogpost on Github.

Thanks for reading and till next time.

Daan Bakboord – DaAnalytics