BigQuery

Prerequisites

You’ll need to create a custom role and service account in GCP. This role and service account must be added to each Project you want Sidecar to ingest metadata for.

1. Setting up a role and service account

GCP Console
Bash Script

Create a custom role in GCP

Navigate to Roles in the IAM & Admin section.
Click the + Create Role button.
Give the new role a name such as Sidecar Data BigQuery.
Click + Add Permissions, and add the following permissions:

#	Permission	Reason
1	`bigquery.datasets.create`	Sidecar dataset setup
2	`bigquery.datasets.get`	Metadata Ingestion
3	`bigquery.tables.get`	Metadata Ingestion
4	`bigquery.tables.getData`	Metadata Ingestion
5	`bigquery.tables.list`	Metadata Ingestion
6	`resourcemanager.projects.get`	Metadata Ingestion
7	`bigquery.jobs.create`	Metadata Ingestion
8	`bigquery.jobs.listAll`	Metadata Ingestion
9	`bigquery.routines.get`	Stored Procedure
10	`bigquery.routines.list`	Stored Procedure
11	`bigquery.readsessions.create`	Usage & Lineage Workflow
12	`bigquery.readsessions.getData`	Usage & Lineage Workflow
13	`logging.operations.list`	Incremental Metadata Ingestion
14	`serviceusage.services.use`	Use the service account project for API quota and billing
15	`mcp.tools.call`	MCP tool calls for Agent Console jobs

serviceusage.services.use lets Sidecar use the service account project for API quota and billing. Keep it in this custom role; no additional IAM role is required for this access.

Click Create to create the role.
Repeat the role creation process for each Project you want Sidecar to ingest metadata for.

Create a service account

Navigate to Service accounts in the IAM & Admin section.
Click + Create Service Account.
Give the new service account a name like sidecar-data.
Under “Grant this service account access to project”, select the custom role you created above.
Click Done.
For all other Projects you want Sidecar to ingest metadata for:
1. Navigate to IAM in the IAM & Admin section.
2. Click + Grant Access.
3. Under “Add principals”, enter the email address generated for the service account.
4. Under “Assign roles”, select the custom role you created.
5. Click Save.

Replace the PROJECTS, ROLE_ID, ROLE_TITLE, and ROLE_DESCRIPTION with your own values:

#!/bin/bash

PROJECTS=("project-1-id" "project-2-id" "project-3-id")

ROLE_ID="sidecarDataBigQuery"
ROLE_TITLE="SidecarData BigQuery"
ROLE_DESCRIPTION="Custom IAM role for Sidecar's service account to access BigQuery"

PERMISSIONS=(
    "bigquery.datasets.create",
    "bigquery.datasets.get",
    "bigquery.tables.get",
    "bigquery.tables.getData",
    "bigquery.tables.list",
    "resourcemanager.projects.get",
    "bigquery.jobs.create",
    "bigquery.jobs.listAll",
    "bigquery.routines.get",
    "bigquery.routines.list",
    "bigquery.readsessions.create",
    "bigquery.readsessions.getData",
    "logging.operations.list",
    "serviceusage.services.use",
    "mcp.tools.call",
)

PERMISSIONS_CSV=$(IFS=,; echo "${PERMISSIONS[*]}")

for PROJECT in "${PROJECTS[@]}"; do
    echo "Creating IAM role in project: $PROJECT"

    gcloud iam roles create $ROLE_ID \
        --project="$PROJECT" \
        --title="$ROLE_TITLE" \
        --description="$ROLE_DESCRIPTION" \
        --permissions="$PERMISSIONS_CSV" \
        --stage="GA"

    echo "IAM role created successfully in project: $PROJECT"
done

echo "IAM role creation process completed!"

2. Create and download key credentials

For your primary Project, navigate to Service accounts in the IAM & Credentials section.
Find the service account you created for Sidecar, click the 3 dots and click Manage keys.
Click Add key, then Create new key.
Select JSON as the Key type.
Click Create. This will download a JSON file to your machine.

3. Upload your service account key to Sidecar

In the Onboarding page (or Account Settings), navigate to “Connect your data warehouse” and select “BigQuery”.
Select the downloaded JSON key file from Step 2.
Click Test then Save.

4. Additional permissions

Cloud Resource Manager API and Cloud Asset API must be enabled.
Add the Cloud Asset Viewer role to the Sidecar service account.

5. Optional Storage Read API activity

Capture Storage Read API activity for deprecation recommendations

Spark, Dataproc, and other clients can read BigQuery tables through the Storage Read API without creating a query job. Complete this setup if you want Sidecar to treat those direct reads as activity when evaluating query activity.Sidecar reads from one dedicated Cloud Logging view for each BigQuery integration. The view must receive matching events from every project that owns BigQuery tables selected in Sidecar.

Reuse the service account and JSON key from steps 1–3. You don’t need a new service account or key. The existing logging.operations.list permission does not grant access to Data Access logs.

Logging sinks route only new log entries. Creating a sink with 90-day retention does not backfill the previous 90 days. Existing recommendations might remain until the affected table is read again.

Before you begin

Gather the following values:

Sidecar service account email: the client_email in the JSON key you uploaded to Sidecar.
BigQuery project IDs: every project that owns tables selected in the Sidecar integration.
Logging project ID: the project that will own the dedicated log bucket and view. For one BigQuery project, use that same project.
Retention period: use at least 90 days. If your Sidecar deprecation inactivity threshold is longer than 90 days, use at least that threshold plus one day.

Enable logging.googleapis.com in the logging project and in the service account’s project (project_id in the JSON key). These are often the same project. Ensure the Sidecar custom role is granted to the service account in that project and includes serviceusage.services.use. If the custom role was created only in other projects, create or update the same project-level custom role here and grant it to the Sidecar service account. No other Service Usage permissions are required for this setup.

GCP Console
Terraform

Create the log bucket

Select the logging project in the Google Cloud Console.
Open APIs & Services → Library → Cloud Logging API and click Enable if the API isn’t already enabled.
Open Logging → Logs Storage and click Create log bucket.
Enter sidecar-bigquery-storage-reads as the bucket name.
Choose a location and record it. The location can’t be changed later.
Set the retention period determined above.
Leave Upgrade to use Observability Analytics unchecked. Sidecar doesn’t require it.
Click Create bucket.

Google Cloud charges additional retention costs for logs kept longer than 30 days.

Route Storage API reads into the bucket

Repeat these steps in every BigQuery project selected in Sidecar:

Select the BigQuery project in the Google Cloud Console.
Open Logging → Log Router and click Create sink.
Enter sidecar-bigquery-storage-reads as the sink name.
Choose Cloud Logging bucket as the sink service.

Select the dedicated bucket. If the bucket is in another project, use its full destination:

logging.googleapis.com/projects/LOGGING_PROJECT_ID/locations/LOCATION/buckets/sidecar-bigquery-storage-reads

In Build inclusion filter, enter:

log_id("cloudaudit.googleapis.com/data_access")
resource.type="bigquery_table"
protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryRead.ReadRows"

Add no exclusion filters and click Create sink.
If the bucket is in a different project, open the sink details and copy its writer identity. In the logging project, grant that identity Logs Bucket Writer (roles/logging.bucketWriter). A same-project destination doesn’t need this grant.

A project-level sink sees only logs that originate in that project. The Spark or Dataproc execution project isn’t a substitute for a project that owns selected BigQuery tables.

Create the restricted log view

Return to Logging → Logs Storage in the logging project.
Open the sidecar-bigquery-storage-reads bucket.
Under Log views, click Create log view.
Enter sidecar-bigquery-storage-reads as the view name.

Enter this view filter:

log_id("cloudaudit.googleapis.com/data_access") AND
resource.type="bigquery_table"

Save the view.
From the view’s Actions menu, select Adjust permissions.
Add the Sidecar service account email and assign Logs View Accessor (roles/logging.viewAccessor).

Grant the role on the view itself. Don’t grant Sidecar roles/logging.privateLogViewer, project-level roles/logging.viewAccessor, roles/logging.bucketWriter, or a Logging administration role.

The following example supports one or more BigQuery projects. Every source project routes matching events into one bucket and view.

variable "logging_project_id" {
  type = string
}

variable "logging_location" {
  type    = string
  default = "global"
}

variable "bigquery_project_ids" {
  type = set(string)
}

variable "sidecar_service_account_email" {
  type = string
}

variable "retention_days" {
  type    = number
  default = 90
}

locals {
  logging_api_projects = setunion(
    var.bigquery_project_ids,
    toset([var.logging_project_id]),
  )
}

resource "google_project_service" "logging" {
  for_each = local.logging_api_projects

  project            = each.value
  service            = "logging.googleapis.com"
  disable_on_destroy = false
}

resource "google_logging_project_bucket_config" "sidecar_storage_reads" {
  project        = var.logging_project_id
  location       = var.logging_location
  bucket_id      = "sidecar-bigquery-storage-reads"
  retention_days = var.retention_days

  depends_on = [google_project_service.logging]
}

resource "google_logging_project_sink" "sidecar_storage_reads" {
  for_each = var.bigquery_project_ids

  project     = each.value
  name        = "sidecar-bigquery-storage-reads"
  destination = "logging.googleapis.com/${google_logging_project_bucket_config.sidecar_storage_reads.id}"
  filter      = <<-EOT
    log_id("cloudaudit.googleapis.com/data_access")
    resource.type="bigquery_table"
    protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryRead.ReadRows"
  EOT

  unique_writer_identity = each.value != var.logging_project_id
}

locals {
  cross_project_sink_writers = {
    for project_id, sink in google_logging_project_sink.sidecar_storage_reads :
    project_id => sink.writer_identity
    if project_id != var.logging_project_id
  }
}

resource "google_project_iam_member" "cross_project_sink_writer" {
  for_each = local.cross_project_sink_writers

  project = var.logging_project_id
  role    = "roles/logging.bucketWriter"
  member  = each.value
}

resource "google_logging_log_view" "sidecar_storage_reads" {
  name   = "sidecar-bigquery-storage-reads"
  bucket = google_logging_project_bucket_config.sidecar_storage_reads.id
  filter = <<-EOT
    log_id("cloudaudit.googleapis.com/data_access") AND
    resource.type="bigquery_table"
  EOT
}

resource "google_logging_log_view_iam_member" "sidecar" {
  parent   = "projects/${var.logging_project_id}"
  location = var.logging_location
  bucket   = google_logging_project_bucket_config.sidecar_storage_reads.bucket_id
  name     = google_logging_log_view.sidecar_storage_reads.name
  role     = "roles/logging.viewAccessor"
  member   = "serviceAccount:${var.sidecar_service_account_email}"
}

output "sidecar_storage_read_log_view_resource_name" {
  value = google_logging_log_view.sidecar_storage_reads.id
}

If the service account’s project (project_id in the JSON key) isn’t included above, enable logging.googleapis.com there separately. Create or update the Sidecar project-level custom role in that project so it includes serviceusage.services.use, then grant that custom role to the Sidecar service account. No other Service Usage permissions are required for this setup.

Add the view to Sidecar

Copy the complete view resource name. It has this format:

projects/LOGGING_PROJECT_ID/locations/LOCATION/buckets/sidecar-bigquery-storage-reads/views/sidecar-bigquery-storage-reads

In Sidecar, open Integrations → BigQuery, enter this value in Storage Read LogView resource name, and click Save. Enter the full resource name, not a Google Cloud Console URL or a project ID.

Verify the view

After a new Spark or direct Storage API read completes, run this command in Cloud Shell:

gcloud logging read \
  'protoPayload.methodName="google.cloud.bigquery.storage.v1.BigQueryRead.ReadRows"' \
  --project=LOGGING_PROJECT_ID \
  --bucket=sidecar-bigquery-storage-reads \
  --location=LOCATION \
  --view=sidecar-bigquery-storage-reads \
  --freshness=1d \
  --limit=10

Sink and IAM changes can take several minutes to propagate. If your organization uses intercepting sinks, confirm that a fresh test read reaches this dedicated view.Learn more in Google’s guides to route logs, manage log buckets, and configure log views.

Add BigQuery MCP to Agent Console

The warehouse connection above does not automatically provide query tools to agents. The mcp.tools.call permission in the custom role enables Google’s managed MCP servers. Follow Set up BigQuery MCP to confirm the connection.

Source control

Data Warehouses

Transformation

BI & Analytics

Extract & Load

Communication

MCP

Project Management

Prerequisites

1. Setting up a role and service account

Create a custom role in GCP

Create a service account

2. Create and download key credentials

3. Upload your service account key to Sidecar

4. Additional permissions

5. Optional Storage Read API activity

Before you begin

Create the log bucket

Route Storage API reads into the bucket

Create the restricted log view

Add the view to Sidecar

Verify the view

Add BigQuery MCP to Agent Console

​Prerequisites

​1. Setting up a role and service account

​Create a custom role in GCP

​Create a service account

​2. Create and download key credentials

​3. Upload your service account key to Sidecar

​4. Additional permissions

​5. Optional Storage Read API activity

​Before you begin

​Create the log bucket

​Route Storage API reads into the bucket

​Create the restricted log view

​Add the view to Sidecar

​Verify the view

​Add BigQuery MCP to Agent Console

Prerequisites

1. Setting up a role and service account

Create a custom role in GCP

Create a service account

2. Create and download key credentials

3. Upload your service account key to Sidecar

4. Additional permissions

5. Optional Storage Read API activity

Before you begin

Create the log bucket

Route Storage API reads into the bucket

Create the restricted log view

Add the view to Sidecar

Verify the view

Add BigQuery MCP to Agent Console