BigQuery

Prerequisites

You’ll need to create a custom role and service account in GCP. This role and service account must be added to each Project you want Sidecar to ingest metadata for.

1. Setting up a role and service account

GCP Console
Bash Script

Create a custom role in GCP

Navigate to Roles in the IAM & Admin section.
Click the + Create Role button.
Give the new role a name such as Sidecar Data BigQuery.
Click + Add Permissions, and add the following permissions:

#	Permission	Reason
1	`bigquery.datasets.get`	Metadata Ingestion
2	`bigquery.tables.get`	Metadata Ingestion
3	`bigquery.tables.getData`	Metadata Ingestion
4	`bigquery.tables.list`	Metadata Ingestion
5	`resourcemanager.projects.get`	Metadata Ingestion
6	`bigquery.jobs.create`	Metadata Ingestion
7	`bigquery.jobs.listAll`	Metadata Ingestion
8	`bigquery.routines.get`	Stored Procedure
9	`bigquery.routines.list`	Stored Procedure
10	`bigquery.readsessions.create`	Usage & Lineage Workflow
11	`bigquery.readsessions.getData`	Usage & Lineage Workflow
12	`logging.operations.list`	Incremental Metadata Ingestion

Click Create to create the role.
Repeat the role creation process for each Project you want Sidecar to ingest metadata for.

Create a service account

Navigate to Service accounts in the IAM & Admin section.
Click + Create Service Account.
Give the new service account a name like sidecar-data.
Under “Grant this service account access to project”, select the custom role you created above.
Click Done.
For all other Projects you want Sidecar to ingest metadata for:
1. Navigate to IAM in the IAM & Admin section.
2. Click + Grant Access.
3. Under “Add principals”, enter the email address generated for the service account.
4. Under “Assign roles”, select the custom role you created.
5. Click Save.

Replace the PROJECTS, ROLE_ID, ROLE_TITLE, and ROLE_DESCRIPTION with your own values:

#!/bin/bash

PROJECTS=("project-1-id" "project-2-id" "project-3-id")

ROLE_ID="sidecarDataBigQuery"
ROLE_TITLE="SidecarData BigQuery"
ROLE_DESCRIPTION="Custom IAM role for Sidecar's service account to access BigQuery"

PERMISSIONS=(
    "bigquery.datasets.create",
    "bigquery.datasets.get",
    "bigquery.tables.get",
    "bigquery.tables.getData",
    "bigquery.tables.list",
    "resourcemanager.projects.get",
    "bigquery.jobs.create",
    "bigquery.jobs.listAll",
    "bigquery.routines.get",
    "bigquery.routines.list",
    "bigquery.readsessions.create",
    "bigquery.readsessions.getData",
    "logging.operations.list",
)

PERMISSIONS_CSV=$(IFS=,; echo "${PERMISSIONS[*]}")

for PROJECT in "${PROJECTS[@]}"; do
    echo "Creating IAM role in project: $PROJECT"

    gcloud iam roles create $ROLE_ID \
        --project="$PROJECT" \
        --title="$ROLE_TITLE" \
        --description="$ROLE_DESCRIPTION" \
        --permissions="$PERMISSIONS_CSV" \
        --stage="GA"

    echo "IAM role created successfully in project: $PROJECT"
done

echo "IAM role creation process completed!"

2. Create and download key credentials

For your primary Project, navigate to Service accounts in the IAM & Credentials section.
Find the service account you created for Sidecar, click the 3 dots and click Manage keys.
Click Add key, then Create new key.
Select JSON as the Key type.
Click Create. This will download a JSON file to your machine.

3. Upload your service account key to Sidecar

In the Onboarding page (or Account Settings), navigate to “Connect your data warehouse” and select “BigQuery”.
Select the downloaded JSON key file from Step 2.
Click Test then Save.

4. Additional Permissions

Cloud Resource Manager API and Cloud Asset API must be enabled.
Add the Cloud Asset Viewer role to the Sidecar service account.

Data Warehouses

Transformation

BI & Analytics

Extract & Load

Communication

Project Management

Other

Prerequisites

1. Setting up a role and service account

Create a custom role in GCP

Create a service account

2. Create and download key credentials

3. Upload your service account key to Sidecar

4. Additional Permissions

Data Warehouses

Transformation

BI & Analytics

Extract & Load

Communication

Project Management

Other

Documentation Index

​Prerequisites

​1. Setting up a role and service account

​Create a custom role in GCP

​Create a service account

​2. Create and download key credentials

​3. Upload your service account key to Sidecar

​4. Additional Permissions

Prerequisites

1. Setting up a role and service account

Create a custom role in GCP

Create a service account

2. Create and download key credentials

3. Upload your service account key to Sidecar

4. Additional Permissions