copy old content of registry.k8s.io from k/k8s.io repo

2023-02-20 19:05:41 +03:00 · 2023-02-20 19:05:41 +03:00 · 1ccbb1c651
parent 8d1d604a30
commit 1ccbb1c651
7 changed files with 428 additions and 0 deletions
--- a/experiment/MANUALLY_SYNCING_BUCKETS.md
+++ b/experiment/MANUALLY_SYNCING_BUCKETS.md
@ -0,0 +1,86 @@
+# Manually syncing buckets
+
+**NOTE**: we are no longer manually syncing buckets, however this may be a useful
+fallback reference.  Syncs are now done by the [gcs-to-s3-sync](https://testgrid.k8s.io/sig-k8s-infra-k8sio#gcs-to-s3-sync) prow job.
+
+## Background
+
+In the CNCF AWS accounts, there are two accounts of concern:
+
+- _cncf/kubernetes/k8s-infra-accounts_
+- _cncf/kubernetes/registry.k8s.io/registry.k8s.io_admin_
+
+Using an IAM user inside of the k8s-infra-accounts account, it is possible to write to the registry.k8s.io mirror buckets.
+
+## Logging in
+
+Log in as an IAM user, which has the ability to assume the _registry.k8s.io_s3writer_ role:
+
+```bash
+aws configure
+```
+
+## Assuming roles
+
+In order to gain permissions to write to the buckets, it is required to call STS to assume the _registry.k8s.io_s3writer_ role:
+
+```bash
+unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
+JSON=$(aws sts assume-role \
+  --role-arn "arn:aws:iam::513428760722:role/registry.k8s.io_s3writer"  \
+  --role-session-name registry.k8s.io_s3writer \
+  --duration-seconds 3600 \
+  --output json || exit 1)
+
+export \
+  AWS_ACCESS_KEY_ID=$(echo "${JSON}" | jq --raw-output ".Credentials[\"AccessKeyId\"]") \
+  AWS_SECRET_ACCESS_KEY=$(echo "${JSON}" | jq --raw-output ".Credentials[\"SecretAccessKey\"]") \
+  AWS_SESSION_TOKEN=$(echo "${JSON}" | jq --raw-output ".Credentials[\"SessionToken\"]")
+```
+
+## Set up rclone
+
+Configure rclone to auth both with GCS and S3
+
+```bash
+cat << EOF > ~/.rclone.conf
+[gcs]
+type = google cloud storage
+bucket_acl = private
+
+[s3]
+type = s3
+provider = AWS
+access_key_id = $AWS_ACCESS_KEY_ID
+secret_access_key = $AWS_SECRET_ACCESS_KEY
+session_token = $AWS_SESSION_TOKEN
+region = us-east-2
+EOF
+```
+
+## Performing the sync
+
+The following set of commands will perform a sync, first between GCS US region and S3 us-east-2, then between S3 us-east-2 and the remaining regions
+
+```bash
+REGIONS=(
+    prod-registry-k8s-io-ap-southeast-1
+    prod-registry-k8s-io-ap-southeast-2
+    prod-registry-k8s-io-ap-south-1
+
+    prod-registry-k8s-io-us-west-1
+    prod-registry-k8s-io-us-west-2
+    prod-registry-k8s-io-us-east-1
+
+    prod-registry-k8s-io-eu-central-1
+    prod-registry-k8s-io-eu-west-1
+)
+
+# initial sync (gcs us -> s3 us)
+rclone sync -P gcs:us.artifacts.k8s-artifacts-prod.appspot.com s3:prod-registry-k8s-io-us-east-2
+
+for REGION in "${REGIONS[@]}"; do
+  rclone config update s3 region "${REGION}"
+  rclone sync -P s3:prod-registry-k8s-io-us-east-2 "s3:${REGION}"
+done
+```
--- a/experiment/OWNERS
+++ b/experiment/OWNERS
@ -0,0 +1,10 @@
+# See the OWNERS docs at https://go.k8s.io/owners
+
+reviewers:
+- BobyMCbobs
+- hh
+- Riaankl
+
+labels:
+- area/artifacts
+- sig/release
--- a/experiment/README.md
+++ b/experiment/README.md
@ -0,0 +1,27 @@
+# registry.k8s.io
+
+Kubernetes's multi-cloud registry.
+
+*This folder is a snapshot of the contents of https://git.k8s.io/k8s.io/registry.k8s.io before k8s.gcr.io was frozen.*
+
+For more details on the design and implementation see the source repo: https://github.com/kubernetes/registry.k8s.io
+
+Most of the deployment configuration is in [infra/gcp/terraform/k8s-infra-oci-proxy](./../infra/gcp/terraform/k8s-infra-oci-proxy) (staging instance for development only) and [infra/gcp/terraform/k8s-infra-oci-proxy-prod](./../infra/gcp/terraform/k8s-infra-oci-proxy) (production / end users).
+
+## Accounts
+
+To host the /registry.k8s.io/ redirector, several resources are located in accounts across cloud providers.
+
+
+## GCP
+
+The GCP projects are under the standard kubernetes.io GCP org along with the other
+community infrastructure managed in this repo.
+
+### AWS
+
+In AWS the account to provide registry.k8s.io will be structured as follows
+
+![Account structure](./registry-k8s-io-account-structure.svg)
+
+Terraform management infra on AWS to support registry.k8s.io is available [here](../infra/aws/terraform/registry.k8s.io/README.md)
--- a/experiment/hack/initial-copy-between-s3.sh
+++ b/experiment/hack/initial-copy-between-s3.sh
@ -0,0 +1,110 @@
+#!/bin/bash
+
+# Copyright 2022 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dependencies
+# - rclone
+# - awscli
+
+# Usage
+#   sync all:       ./initial-copy-between-s3.sh
+#   specify region: ./initial-copy-between-s3.sh <REGION>
+
+# Extra usage
+#   launch parallel sync using tmate
+#     for REGION in $(./hack/initial-copy-between-s3.sh regions | yq e '.regions[]' -P - | xargs); do tmate -F -v -S "${TMATE_SOCKET:-/tmp/tmate.socket}" new-window -d -c "$PWD" -n sync-to-"${REGION:-}" "bash -x ./hack/initial-copy-between-s3.sh \"${REGION:-}\""; done
+
+function sync-a-region {
+    REGION="${1:-}"
+    DESTINATION="s3dest:prod-registry-k8s-io-${REGION:-}"
+    if [ ! -f /var/run/secrets/aws-iam-token/serviceaccount/token ]; then
+      unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
+      JSON=$(aws sts assume-role \
+          --role-arn "arn:aws:iam::513428760722:role/registry.k8s.io_s3writer"  \
+          --role-session-name "${CALLER_ID:-}-registry.k8s.io_s3writer" \
+          --duration-seconds 43200 \
+          --output json || exit 1)
+
+      AWS_ACCESS_KEY_ID=$(echo "${JSON}" | jq --raw-output ".Credentials[\"AccessKeyId\"]")
+      AWS_SECRET_ACCESS_KEY=$(echo "${JSON}" | jq --raw-output ".Credentials[\"SecretAccessKey\"]")
+      AWS_SESSION_TOKEN=$(echo "${JSON}" | jq --raw-output ".Credentials[\"SessionToken\"]")
+      export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
+    fi
+
+    RCLONE_CONFIG="$(mktemp -t rclone-"${REGION:-}"-XXXXX.conf)"
+    echo "Wrote rclone config to '${RCLONE_CONFIG:-}'"
+
+    cat << EOF > "${RCLONE_CONFIG:-}"
+[gcs]
+type = google cloud storage
+bucket_acl = private
+
+[s3]
+type = s3
+provider = AWS
+env_auth = true
+region = us-east-2
+
+[s3dest]
+type = s3
+provider = AWS
+env_auth = true
+region = ${REGION}
+EOF
+    echo "Running sync between '${SOURCE:-}' and '${DESTINATION:-}'"
+    rclone sync --config "${RCLONE_CONFIG:-}" -P "${SOURCE:-}" "${DESTINATION:-}"
+}
+
+REGIONS=(
+    ap-northeast-1
+    ap-south-1
+    ap-southeast-1
+
+    eu-central-1
+    eu-west-1
+
+    us-east-1
+    us-east-2
+    us-west-1
+    us-west-2
+)
+if [ "${1}" = "regions" ]; then
+    echo "regions:"
+    for REGION in "${REGIONS[@]}"; do
+        echo "- ${REGION:-}"
+    done
+    exit 0
+fi
+SOURCE=s3:prod-registry-k8s-io-us-east-2
+
+CALLER_ID="$(aws sts get-caller-identity --output json | jq -r .UserId)"
+
+SELECTED_REGION="${1:-}"
+FOUND_REGION=false
+for REGION in "${REGIONS[@]}"; do
+    if [ "${REGION:-}" = "${SELECTED_REGION:-}" ]; then
+        FOUND_REGION=true
+    fi
+done
+if [ ! "${FOUND_REGION:-}" = true ]; then
+    echo "No region specified of: ${REGIONS[*]}"
+    echo "Will sync all."
+    for REGION in "${REGIONS[@]}"; do
+        sync-a-region "${REGION:-}"
+    done
+    exit 0
+fi
+
+sync-a-region "${SELECTED_REGION:-}"
--- a/experiment/hack/initial-copy-to-s3.sh
+++ b/experiment/hack/initial-copy-to-s3.sh
@ -0,0 +1,62 @@
+#!/bin/bash
+
+# Copyright 2022 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Dependencies
+# - rclone
+# - awscli
+
+# Usage
+#   ./hack/initial-copy-to-s3.sh
+
+SOURCE=gcs:us.artifacts.k8s-artifacts-prod.appspot.com
+DESTINATION=s3:prod-registry-k8s-io-us-east-2
+
+CALLER_ID="$(aws sts get-caller-identity --output json | jq -r .UserId)"
+
+while true; do
+  if [ ! -f /var/run/secrets/aws-iam-token/serviceaccount/token ]; then
+    unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
+    JSON=$(aws sts assume-role \
+      --role-arn "arn:aws:iam::513428760722:role/registry.k8s.io_s3writer"  \
+      --role-session-name "${CALLER_ID:-}-registry.k8s.io_s3writer" \
+      --duration-seconds 43200 \
+      --output json || exit 1)
+
+    AWS_ACCESS_KEY_ID=$(echo "${JSON}" | jq --raw-output ".Credentials[\"AccessKeyId\"]")
+    AWS_SECRET_ACCESS_KEY=$(echo "${JSON}" | jq --raw-output ".Credentials[\"SecretAccessKey\"]")
+    AWS_SESSION_TOKEN=$(echo "${JSON}" | jq --raw-output ".Credentials[\"SessionToken\"]")
+    export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
+  fi
+
+  RCLONE_CONFIG="$(mktemp)"
+  echo "Wrote rclone config to '${RCLONE_CONFIG:-}'"
+
+  cat << EOF > "${RCLONE_CONFIG:-}"
+[gcs]
+type = google cloud storage
+bucket_acl = private
+
+[s3]
+type = s3
+provider = AWS
+env_auth = true
+region = us-east-2
+EOF
+  echo "Running sync between '${SOURCE:-}' and '${DESTINATION:-}'"
+  if rclone sync --config "${RCLONE_CONFIG:-}" -P "${SOURCE:-}" "${DESTINATION:-}"; then
+    exit 0;
+  fi
+done
--- a/experiment/registry-k8s-io-account-structure.dot
+++ b/experiment/registry-k8s-io-account-structure.dot
@ -0,0 +1,47 @@
+// Build the diagraph :noexport:
+
+digraph AccountStructure {
+    label="CNCF AWS project account structure"
+    labelloc="t"
+    graph[compound=true]
+
+    subgraph cluster0 {
+        label="CNCF root Org"
+
+        subgraph cluster0_0 {
+            label="Kubernetes Org Unit"
+
+            subgraph cluster0_0_0 {
+                label="registry.k8s.io Org Unit"
+
+                subgraph cluster0_0_0_0 {
+                    label="registry.k8s.io_admin Account"
+
+                    subgraph cluster0_0_0_0_0 {
+                        label="S3 resources"
+
+                        Buckets [label="Bucketⁿ" style=dashed]
+                    }
+
+                    subgraph cluster0_0_0_0_1 {
+                        label="IAM roles"
+
+                        s3writer [label="S3 writer role"]
+                    }
+                }
+            }
+            subgraph cluster0_0_1 {
+                label="k8s-infra-accounts Account"
+
+                subgraph cluster0_0_1_0 {
+                    label="IAM users"
+
+                    k8s_infra_accounts_IAMUsers [label="k8s-infra-accounts IAM Users"]
+                }
+            }
+        }
+    }
+
+    k8s_infra_accounts_IAMUsers -> s3writer [label="Switch role"]
+    s3writer -> Buckets [label="Writes to buckets"]
+}
--- a/experiment/registry-k8s-io-account-structure.svg
+++ b/experiment/registry-k8s-io-account-structure.svg
@ -0,0 +1,86 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
+ "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<!-- Generated by graphviz version 2.43.0 (0)
+ -->
+<!-- Title: AccountStructure Pages: 1 -->
+<svg width="384pt" height="531pt"
+ viewBox="0.00 0.00 384.00 531.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
+<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 527)">
+<title>AccountStructure</title>
+<polygon fill="white" stroke="transparent" points="-4,4 -4,-527 380,-527 380,4 -4,4"/>
+<text text-anchor="middle" x="188" y="-507.8" font-family="Times,serif" font-size="14.00">CNCF AWS project account structure</text>
+<g id="clust1" class="cluster">
+<title>cluster0</title>
+<polygon fill="none" stroke="black" points="8,-8 8,-492 368,-492 368,-8 8,-8"/>
+<text text-anchor="middle" x="188" y="-476.8" font-family="Times,serif" font-size="14.00">CNCF root Org</text>
+</g>
+<g id="clust2" class="cluster">
+<title>cluster0_0</title>
+<polygon fill="none" stroke="black" points="16,-16 16,-461 360,-461 360,-16 16,-16"/>
+<text text-anchor="middle" x="188" y="-445.8" font-family="Times,serif" font-size="14.00">Kubernetes Org Unit</text>
+</g>
+<g id="clust3" class="cluster">
+<title>cluster0_0_0</title>
+<polygon fill="none" stroke="black" points="87,-24 87,-285 333,-285 333,-24 87,-24"/>
+<text text-anchor="middle" x="210" y="-269.8" font-family="Times,serif" font-size="14.00">registry.k8s.io Org Unit</text>
+</g>
+<g id="clust4" class="cluster">
+<title>cluster0_0_0_0</title>
+<polygon fill="none" stroke="black" points="95,-32 95,-254 325,-254 325,-32 95,-32"/>
+<text text-anchor="middle" x="210" y="-238.8" font-family="Times,serif" font-size="14.00">registry.k8s.io_admin Account</text>
+</g>
+<g id="clust5" class="cluster">
+<title>cluster0_0_0_0_0</title>
+<polygon fill="none" stroke="black" points="133,-40 133,-115 243,-115 243,-40 133,-40"/>
+<text text-anchor="middle" x="188" y="-99.8" font-family="Times,serif" font-size="14.00">S3 resources</text>
+</g>
+<g id="clust6" class="cluster">
+<title>cluster0_0_0_0_1</title>
+<polygon fill="none" stroke="black" points="105,-148 105,-223 271,-223 271,-148 105,-148"/>
+<text text-anchor="middle" x="188" y="-207.8" font-family="Times,serif" font-size="14.00">IAM roles</text>
+</g>
+<g id="clust7" class="cluster">
+<title>cluster0_0_1</title>
+<polygon fill="none" stroke="black" points="24,-316 24,-430 352,-430 352,-316 24,-316"/>
+<text text-anchor="middle" x="188" y="-414.8" font-family="Times,serif" font-size="14.00">k8s&#45;infra&#45;accounts Account</text>
+</g>
+<g id="clust8" class="cluster">
+<title>cluster0_0_1_0</title>
+<polygon fill="none" stroke="black" points="32,-324 32,-399 344,-399 344,-324 32,-324"/>
+<text text-anchor="middle" x="188" y="-383.8" font-family="Times,serif" font-size="14.00">IAM users</text>
+</g>
+<!-- Buckets -->
+<g id="node1" class="node">
+<title>Buckets</title>
+<ellipse fill="none" stroke="black" stroke-dasharray="5,2" cx="188" cy="-66" rx="46.59" ry="18"/>
+<text text-anchor="middle" x="188" y="-62.3" font-family="Times,serif" font-size="14.00">Bucketⁿ</text>
+</g>
+<!-- s3writer -->
+<g id="node2" class="node">
+<title>s3writer</title>
+<ellipse fill="none" stroke="black" cx="188" cy="-174" rx="75.29" ry="18"/>
+<text text-anchor="middle" x="188" y="-170.3" font-family="Times,serif" font-size="14.00">S3 writer role</text>
+</g>
+<!-- s3writer&#45;&gt;Buckets -->
+<g id="edge2" class="edge">
+<title>s3writer&#45;&gt;Buckets</title>
+<path fill="none" stroke="black" d="M188,-155.97C188,-139.38 188,-113.88 188,-94.43"/>
+<polygon fill="black" stroke="black" points="191.5,-94.34 188,-84.34 184.5,-94.34 191.5,-94.34"/>
+<text text-anchor="middle" x="251" y="-126.8" font-family="Times,serif" font-size="14.00">Writes to buckets</text>
+</g>
+<!-- k8s_infra_accounts_IAMUsers -->
+<g id="node3" class="node">
+<title>k8s_infra_accounts_IAMUsers</title>
+<ellipse fill="none" stroke="black" cx="188" cy="-350" rx="147.57" ry="18"/>
+<text text-anchor="middle" x="188" y="-346.3" font-family="Times,serif" font-size="14.00">k8s&#45;infra&#45;accounts IAM Users</text>
+</g>
+<!-- k8s_infra_accounts_IAMUsers&#45;&gt;s3writer -->
+<g id="edge1" class="edge">
+<title>k8s_infra_accounts_IAMUsers&#45;&gt;s3writer</title>
+<path fill="none" stroke="black" d="M188,-332C188,-301.85 188,-238.79 188,-202.43"/>
+<polygon fill="black" stroke="black" points="191.5,-202.07 188,-192.07 184.5,-202.07 191.5,-202.07"/>
+<text text-anchor="middle" x="229" y="-296.8" font-family="Times,serif" font-size="14.00">Switch role</text>
+</g>
+</g>
+</svg>