udf

Documentation

TODO

Environment variables

• PROJECT_ID (required) Google Cloud Project ID.
• CHERRE_GOOGLE_CREDENTIALS (required) Base64 encoded Google Cloud JSON key file
• UDF_BUNDLE_GCS_URI (required) Google Cloud Storage URI to Javascript UDF function bundle

Development

Requirements

• NodeJS v10 or later

Run

• Install packages

#!bash

npm install

• Setup environment variables Option 1 - Directly change environment variable

#!bash
export PROJECT_ID=cherre-sandbox
export UDF_BUNDLE_GCS_URI=gs://cherre-sandbox/bigquery_udf/my_dev_udf/bundle.js

Option 2 - Create .env file in the same folder within package.json

#.env
PROJECT_ID=cherre-sandbox
UDF_BUNDLE_GCS_URI=gs://cherre-sandbox/bigquery_udf/my_dev_udf/bundle.js

• Run unit test in watch mode

Use npm ci instead of npm install in Dockerfiles (see https://docs.npmjs.com/cli/ci.html)

Adding new parsing function

Parsing functions are used in our ELT format by having all data loaded into big query as a single column. The parsing function then takes the single column and parses it into individual columns for further transformation.

Start with a test file

Add a test.js file under src/functitons/tests with the below format:

test('<SAMPLE ROW>', () => {
  expect(parseAvroll('<SAMPLE ROW>')).toMatchSnapshot()
})

You will provide your test with a sample row and it will apply your parsing function. The parsing function should create the output that matches the output in the snapshot in the /src/functions/test/__snapshots__ folder

Add snapshot test

the snapshot is an example of of how you will feed a row that needs to be parsed and you are expecting back a JSON object with key value pairs for column name and column data. Big Query will use this function on each row and return.

It is advised to copy a single from the raw ingested table and put it in to the exports. This example has commas as a seperator but your data may be seperated by tabs or pipes.

exports['Column 1 data,Column 2 data,Column 3 data'] = `
Object {
  "Column 1 Name": "Column 1 data",
  "Column 2 Name": "Column 2 data",
  "Column 3 Name": "Column 3 data"

 };`

Then fill in the names that correlate to the appropriate columns to show how you want the data to look.

Create snapshot function

Now that the tests are setup, you can create your javascript parsing function. Create a file with a function in it. The format should take in a single variable (convention is unparsed), apply the split function with whichever delimiter you use. (Attom uses pipe |) and return a const which will have each column named

Then run unit test in watch mode

#!bash

npm run test:unit:watch

If your tests pass you should be ready to deploy.

Deploy function

Run the following command to ensure that your javascript compiles correctly.

npm run udf

Once it compiles, add your function into the /src/bundle.js file as an import such as:

export { <your function> } from './functions/<your filename>'

Then run npm run udf:deploy to send the bundle.js into the bucket from UDF_BUNDLE_GCS_URI environment variable.

npm run udf:deploy