GCP Certification Testing Infrastructure (#2714)

Signed-off-by: Roberto J Rojas <robertojrojas@gmail.com>
Signed-off-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>
Signed-off-by: Roberto Rojas <robertojrojas@gmail.com>
Co-authored-by: Alessandro (Ale) Segala <43508+ItalyPaleAle@users.noreply.github.com>

.github/infrastructure/terraform/infra-setup/gcp/dapr-gh-gcp-wif/README.md

@@ -0,0 +1,78 @@
+# Dapr Components Contrib Certification Tests GitHub Actions Workflow
+
+## Overview
+
+This sets up the [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) in the Components Contrib Gith Actions Workflow
+for the Conformance and Certification Tests.
+
+The Authn/Authz is handled by 2 resources already deployed in the GCP Project:
+
+- Workload Identity Pool & Workload Identity Provider Configured **specifically** for this GitHub repository -  **(Authentication)**
+- A GCP IAM Service Account (SA) used to impersonate this GitHub Actions workflow within the GCP Project **(Authorization)**. This SA has been assigned the roles **`roles/pubsub.admin`** and **`roles/datastore.owner`** which will be used for the `GCP PusbSub` and `GCP Firestore` Certification Tests.
+
+ **Note:** Changes to the roles for the SA should be made in the `roles` local variable in the file `service_account.tf`
+
+The Terraform scripts follow steps similar to the suggested in the [Google GitHub Actions Auth](https://github.com/google-github-actions/auth#setting-up-workload-identity-federation)
+
+The Terraform state is stored in [dapr-compoments-contrib-cert-tests](https://console.cloud.google.com/storage/browser/dapr-compoments-contrib-cert-tests?project=dapr-tests) Bucket
+of the GCP GCS within the [dapr-tests](https://console.cloud.google.com/home/dashboard?project=dapr-tests) GCP Project.
+
+
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|------|---------|:--------:|
+| `project_id` | The project id that hosts the WIF pool and Dapr OSS SA | `string` | n/a | yes |
+| `gh_repo`    | The GitHub Repo (username/repo_name) to associate with the WIF pool and Dapr SA | `string` | n/a | yes |
+| `service_account` | The Dapr OSS SA used for GitHub WIF OIDC | `string` | n/a | yes |
+| `wif_pool_name` | The Dapr OSS Workload Identity Pool Name | `string` | n/a | yes |
+
+## Requirements
+
+Before this module can be used on a project, you must ensure that the following pre-requisites are fulfilled:
+
+1. Required APIs are activated
+
+    ```
+    "iam.googleapis.com",
+    "cloudresourcemanager.googleapis.com",
+    "iamcredentials.googleapis.com",
+    "sts.googleapis.com",
+    ```
+
+1. The GCP Account or Service Account used to deploy this module has the following roles
+
+    ```
+    roles/iam.workloadIdentityPoolAdmin
+    roles/iam.serviceAccountAdmin
+    roles/storage.admin
+    ```
+
+## Run Terraform
+
+```
+$ terraform init
+
+$ terraform refresh -var="gh_repo=dapr/components-contrib" \
+                 -var="project_id=dapr-tests" -var="service_account=comp-contrib-wif" \
+                 -var="wif_pool_name=contrib-cert-tests"
+
+$ terraform plan -var="gh_repo=dapr/components-contrib" \
+                 -var="project_id=dapr-tests" -var="service_account=comp-contrib-wif" \
+                 -var="wif_pool_name=contrib-cert-tests"
+
+$ terraform apply --auto-approve -var="gh_repo=dapr/components-contrib" \
+                 -var="project_id=dapr-tests" -var="service_account=comp-contrib-wif" \
+                 -var="wif_pool_name=contrib-cert-tests"
+```
+
+
+## Outputs
+
+```
+$ terraform output                                                   
+    
+pool_name = "projects/***/locations/global/workloadIdentityPools/contrib-cert-tests-gh-pool"
+provider_name = "projects/***/locations/global/workloadIdentityPools/contrib-cert-tests-gh-pool/providers/contrib-cert-tests-gh-provider"
+sa_email = "***"
+```

.github/infrastructure/terraform/infra-setup/gcp/dapr-gh-gcp-wif/backend.tf

@@ -0,0 +1,21 @@
+/**
+# ------------------------------------------------------------
+# Copyright 2023 The Dapr Authors
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ------------------------------------------------------------
+*/
+
+terraform {
+  backend "gcs" {
+    bucket = "dapr-compoments-contrib-cert-tests"
+    prefix = "terraform-state"
+  }
+}

.github/infrastructure/terraform/infra-setup/gcp/dapr-gh-gcp-wif/main.tf

@@ -0,0 +1,28 @@
+/**
+# ------------------------------------------------------------
+# Copyright 2023 The Dapr Authors
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ------------------------------------------------------------
+*/
+
+module "oidc" {
+  source      = "terraform-google-modules/github-actions-runners/google//modules/gh-oidc"
+  version     = "~> 3.1.1"
+  project_id  = var.project_id
+  pool_id     = "${var.wif_pool_name}-gh-pool"
+  provider_id = "${var.wif_pool_name}-gh-provider"
+  sa_mapping = {
+    (google_service_account.sa.account_id) = {
+      sa_name   = google_service_account.sa.name
+      attribute = "attribute.repository/${var.gh_repo}"
+    }
+  }
+}

.github/infrastructure/terraform/infra-setup/gcp/dapr-gh-gcp-wif/outputs.tf

@@ -0,0 +1,29 @@
+/**
+# ------------------------------------------------------------
+# Copyright 2023 The Dapr Authors
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ------------------------------------------------------------
+ */
+
+output "pool_name" {
+  description = "Pool name"
+  value       = module.oidc.pool_name
+}
+
+output "provider_name" {
+  description = "Provider name"
+  value       = module.oidc.provider_name
+}
+
+output "sa_email" {
+  description = "WIF SA email"
+  value       = google_service_account.sa.email
+}

.github/infrastructure/terraform/infra-setup/gcp/dapr-gh-gcp-wif/pubsub_roles.tf

@@ -0,0 +1,36 @@
+/**
+# ------------------------------------------------------------
+# Copyright 2023 The Dapr Authors
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ------------------------------------------------------------
+*/
+
+locals {
+  pubsub_roles = toset([
+    "roles/pubsub.subscriber",
+    "roles/pubsub.publisher",
+  ])
+}
+
+data "google_project" "project" {
+    project_id  = var.project_id
+}
+
+resource "google_project_iam_binding" "project" {
+  for_each = local.pubsub_roles 
+  project  = var.project_id
+  role     =  each.key 
+
+  members = [
+    "serviceAccount:service-${data.google_project.project.number}@gcp-sa-pubsub.iam.gserviceaccount.com",
+  ]
+}
+

.github/infrastructure/terraform/infra-setup/gcp/dapr-gh-gcp-wif/service_account.tf

@@ -0,0 +1,33 @@
+/**
+# ------------------------------------------------------------
+# Copyright 2023 The Dapr Authors
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ------------------------------------------------------------
+*/
+
+locals {
+  roles = toset([
+    "roles/pubsub.admin",
+    "roles/datastore.owner",
+  ])
+}
+
+resource "google_service_account" "sa" {
+  project    = var.project_id
+  account_id = var.service_account
+}
+
+resource "google_project_iam_member" "project" {
+  for_each = local.roles
+  project  = var.project_id
+  role     =  each.key 
+  member   = "serviceAccount:${google_service_account.sa.email}"
+}

.github/infrastructure/terraform/infra-setup/gcp/dapr-gh-gcp-wif/variables.tf

@@ -0,0 +1,33 @@
+/**
+# ------------------------------------------------------------
+# Copyright 2023 The Dapr Authors
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ------------------------------------------------------------
+*/
+variable "project_id" {
+  type        = string
+  description = "The project id that hosts the WIF pool and Dapr OSS SA"
+}
+
+variable "gh_repo" {
+  type        = string
+  description = "The Github Repo (username/repo_name) to associate with the WIF pool and Dapr SA"
+}
+
+variable "service_account" {
+  type        = string
+  description = "The Dapr OSS SA used for Github WIF OIDC"
+}
+
+variable "wif_pool_name" {
+  type        = string
+  description = "The Dapr OSS Workload Identity Pool Name"
+}

.github/infrastructure/terraform/infra-setup/gcp/dapr-gh-gcp-wif/versions.tf

@@ -0,0 +1,28 @@
+/**
+# ------------------------------------------------------------
+# Copyright 2023 The Dapr Authors
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#     http://www.apache.org/licenses/LICENSE-2.0
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ------------------------------------------------------------
+*/
+
+terraform {
+  required_providers {
+    google = {
+      source  = "hashicorp/google"
+      version = "~> 4.0"
+    }
+    google-beta = {
+      source  = "hashicorp/google-beta"
+      version = "~> 4.0"
+    }
+  }
+  required_version = ">= 0.13"
+}

.github/scripts/test-info.mjs

@@ -582,6 +582,7 @@ const components = {
  * @property {string[]?} requiredSecrets Required secrets (if not empty, test becomes "cloud-only")
  * @property {string[]?} requiredCerts Required certs (if not empty, test becomes "cloud-only")
  * @property {boolean?} requireAWSCredentials If true, requires AWS credentials and makes the test "cloud-only"
+ * @property {boolean?} requireGCPCredentials If true, requires GCP credentials and makes the test "cloud-only"
  * @property {boolean?} requireCloudflareCredentials If true, requires Cloudflare credentials and makes the test "cloud-only"
  * @property {boolean?} requireTerraform If true, requires Terraform
  * @property {boolean?} requireKind If true, requires KinD
@@ -601,6 +602,7 @@ const components = {
  * @property {string?} required-secrets Required secrets
  * @property {string?} required-certs Required certs
  * @property {boolean?} require-aws-credentials Requires AWS credentials
+ * @property {boolean?} require-gcp-credentials Requires GCP credentials
  * @property {boolean?} require-cloudflare-credentials Requires Cloudflare credentials
  * @property {boolean?} require-terraform Requires Terraform
  * @property {boolean?} require-kind Requires KinD
@@ -632,6 +634,7 @@ function GenerateMatrix(testKind, enableCloudTests) {
                 comp.requiredSecrets?.length ||
                 comp.requiredCerts?.length ||
                 comp.requireAWSCredentials ||
+                comp.requireGCPCredentials ||
                 comp.requireCloudflareCredentials
             ) {
                 continue
@@ -660,6 +663,9 @@ function GenerateMatrix(testKind, enableCloudTests) {
             'require-aws-credentials': comp.requireAWSCredentials
                 ? 'true'
                 : undefined,
+            'require-gcp-credentials': comp.requireGCPCredentials
+                ? 'true'
+                : undefined,
             'require-cloudflare-credentials': comp.requireCloudflareCredentials
                 ? 'true'
                 : undefined,

.github/workflows/certification.yml

@@ -82,6 +82,13 @@ jobs:
 
   certification:
     name: ${{ matrix.component }} certification
+
+    # Add "id-token" with the intended permissions.
+    # Needed by the 'Authenticate to Google Cloud' step.
+    permissions:
+      contents: 'read'
+      id-token: 'write'
+
     runs-on: ubuntu-22.04
     env:
       UNIQUE_ID: ${{github.run_id}}-${{github.run_attempt}}
@@ -161,6 +168,22 @@ jobs:
           echo "$secretName=$value" >> $GITHUB_ENV
         done
 
+    # Authenticate with GCP Workload Identity Pool
+    # Exports GCP ENV Vars:
+    #    - GCP_PROJECT
+    #    - GOOGLE_APPLICATION_CREDENTIALS
+    - id: 'auth'
+      if: matrix.require-gcp-credentials == 'true'
+      name: 'Authenticate to Google Cloud'
+      uses: 'google-github-actions/auth@v1'
+      with:
+        token_format: 'access_token'
+        workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER_NAME }}
+        service_account: ${{ secrets.GCP_WIF_SA_EMAIL }}
+        create_credentials_file: true
+        export_environment_variables: true
+        cleanup_credentials: true
+
     # Download the required certificates into files, and set env var pointing to their names
     - name: Setup certs
       if: matrix.required-certs != ''

.github/workflows/conformance.yml

@@ -84,6 +84,13 @@ jobs:
 
   conformance:
     name: ${{ matrix.component }} conformance
+
+    # Add "id-token" with the intended permissions.
+    # Needed by the 'Authenticate to Google Cloud' step.
+    permissions:
+      contents: 'read'
+      id-token: 'write'
+
     runs-on: ubuntu-22.04
     env:
       UNIQUE_ID: ${{github.run_id}}-${{github.run_attempt}}
@@ -168,6 +175,22 @@ jobs:
           echo "$secretName=$value" >> $GITHUB_ENV
         done
 
+    # Authenticate with GCP Workload Identity Pool
+    # Exports GCP ENV Vars:
+    #    - GCP_PROJECT
+    #    - GOOGLE_APPLICATION_CREDENTIALS
+    - id: 'auth'
+      if: matrix.require-gcp-credentials == 'true'
+      name: 'Authenticate to Google Cloud'
+      uses: 'google-github-actions/auth@v1'
+      with:
+        token_format: 'access_token'
+        workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER_NAME }}
+        service_account: ${{ secrets.GCP_WIF_SA_EMAIL }}
+        create_credentials_file: true
+        export_environment_variables: true
+        cleanup_credentials: true
+
     # Download the required certificates into files, and set env var pointing to their names
     - name: Setup certs
       if: matrix.required-certs != ''