# 2020 WG K8s Infra Annual Report

## You and Your Role

**When did you become a chair and do you enjoy the role?**

- **bartsmykla**: February 2020 and I enjoy the role
- **dims**: along with spiffxp, been there right from the beginning. Lately
  having some conflicts on the meeting time, but definitely enjoying the process
- **spiffxp**: Was a chair (organizer?) at group’s formation. I enjoy the role
  when I have time to dedicate to it.

**What do you find challenging?**

- **bartsmykla**: As our working group’s efforts are related to multiple SIGs,
  and there is multiple places, tools, repositories which are needed to move
  some things forward I sometimes feel overwhelmed and anxiety about not
  understanding some of the tools (Prow for example), what is also hard is I
  don’t feel I have enough access and knowledge to speed up and move things
  fasters in relation to Prow migration.
- **dims**: takes too long :) finding/building coalition is hard. Trying hard to
  avoid doing everything by a small set of folks, but not doing too good on that
  front.
- **spiffxp**: Prioritizing this group’s work, and the work necessary to tend to
  this group’s garden (by that I mean weeding/planting workstreams,
  building/smoothing onramps). Work that usually takes precedence is related to
  company-internal priorities, SIG Testing and kubernetes/kubernetes fires.  I
  often find myself unprepared for meetings unless I have been actively pushing
  a specific item in the interim.  Very rarely am I sufficiently aware of the
  group’s activity as a whole to effectively drive.

**Do you have goals for the group?**

- **bartsmykla**: My goal would be to improve documentation for people to be
  easier to understand what and where is happening and which tools and resources
  are being used for which efforts
- **dims**: breaking things up into small chunks that can be easily farmed out
- **spiffxp**: The thing I care most about is community ownership of
  prow.k8s.io, including on-call. If possible, I would like to see the group’s
  mission through to completion, ensuring that all project infrastructure is
  community-owned and maintained.

**Do you want to continue or find a replacement? If you feel that you aren’t
ready to pass the baton, what would you like to accomplish before you do?**

- **bartsmykla**: I would like to continue
- **dims**: Happy to if folks show up who can take over. Always on the look out.
- **spiffxp**:  I personally want to continue, but sometimes wonder if my
  best-effort availability is doing the group a disservice. If there’s a
  replacement and I’m the impediment, I’m happy to step down.  The ideal
  replacement or a dedicated TL would have: 
  - ability to craft and build consensus on operational policies, lead implementation
  - ability to identify cost hotspots, lead or implement cost-reduction solutions
  - ability to identify security vulnerabilities or operational sharp edges
    (e.g. no backups, easy accidents), lead or implement mitigations
  - familiarity with GCP and Kubernetes
  - ability to document/understand how existing project infra is wired together
    (e.g. could fix https://github.com/kubernetes/test-infra/issues/13063 )

**Is there something we can provide that would better support you?**

- **bartsmykla**: I can’t think about anything right now
- **dims**: what spiffxp said!
- **spiffxp**: TBH I feel like a lot of what we need to make this group as
  active/healthy as I would like needs to come from us. For example I don’t
  think a dedicated PM would help without a dedicated TL. I’m not sure how to
  more effectively motivate our contributing companies to prioritize this work.
  I have pined in the past for dedicated contractors paid by the CNCF for this,
  but I think that could just as easily be fulfilled by contributing companies
  agreeing to staff this.

**Do you have feedback for Steering? Suggestions for what we should work on?**

- **bartsmykla**: I can’t think about anything right now
- **dims**: yep, talking to CNCF proactively and formulating a plan. 
- **spiffxp**:  I think there are three things Steering could help with:
  - Policy guidance from Steering on what is in-scope / out-of-scope for
    Kubernetes’ project-infrastructure budget (e.g. mirroring
    dependency/ecosystem projects like cert-manager [1], ci jobs).  It might
    better drive billing requirements, and make it easier/quicker to decide what
    is appropriate to pursue.  At the moment we’re using our best judgement, and
    I trust it, but I sometimes feel like we’re flying blind or making stuff up.
    As far as existing spend and budgeting, we don’t have
    quotas/forecasts/alerts; we’re mostly hoping everyone is on their best
    behavior until something seems outsized, at which point it’s case-by-case on
    what to do.
  - I think it would be helpful to get spend on platforms other than Google
    above-the-table, and driven through this group. I know how much money Google
    has provided, and I know where it’s being spent (though not to the
    granularity of per-SIG).  I lack the equivalent for other companies “helping
    out” (e.g. AWS, Microsoft, DigitalOcean)
  - This is not a concrete request that can be acted upon now, but I anticipate
    we will want to reduce costs by ensuring that other clouds or large entities
    participate in mirroring Kubernetes artifacts.

## Working Group

**What was the initial mission of the group and if it's changed, how?**

Initial mission was to migrate Kubernetes project infrastructure to the CNCF,
creation of teams and processes to support ongoing maintenance.

There has been a slight growth in scope in that new infrastructure that
previously didn't exist is proposed and managed under this group. Examples
include:
- binary-artifact-promotion (project only had image-promotion internally, now
  externally, now attempting to expand to binary artifacts)
- [running triage-party for SIG Release](https://github.com/kubernetes/k8s.io/issues/906)
  (didn't exist until this year)
- [build infrastructure for windows-based images](https://docs.google.com/document/d/16VBfsFMynA7tObzuZGPpw-sKDKfFc_T5W_E4IeEIaOQ/edit#bookmark=id.3w0g7fo9cp7m)
- [image vulnerability dashboard](https://docs.google.com/document/d/16VBfsFMynA7tObzuZGPpw-sKDKfFc_T5W_E4IeEIaOQ/edit#bookmark=id.s3by3vki8jer)
  (it's not clear to me whether even google had this internally before)
- [sharding out / scaling up gitops-based Google Group management](https://docs.google.com/document/d/16VBfsFMynA7tObzuZGPpw-sKDKfFc_T5W_E4IeEIaOQ/edit#bookmark=id.ou5hk544r70m)

**What’s the current roadmap until completion?**

What has been migrated:
- DNS for kubernetes.io, k8s.io
- Container images hosted on k8s.gcr.io
- node-perf-dash.k8s.io
- perf-dash.k8s.io
- publishing-bot
- slack-infra
- 288 / 1780 prow jobs
- GCB projects used to create kubernetes/kubernetes releases
  (exception .deb/.rpm packages)

What remains (TODO: we need to update our issues to reflect this)
- migrate .deb/.rpm package building/hosting to community
  (this would be owned by SIG Release)
  - stop using google-internal tool "rapture"
  - come up with signing keys community agrees to host/trust
  - migrate apt.kubernetes.io to community
- stop using google-containers GCP project (this would be owned by SIG Release)
  - gs://kubernetes-release, dl.k8s.io
  - [gs://kubernetes-release-dev](https://github.com/kubernetes/k8s.io/issues/846)
- stop using k8s-prow GCP project (this would be owned by SIG Testing)
  - Prow.k8s.io
  - Ensure community-staffed on-call can support
- stop using k8s-prow-build GCP project (this would be owned by SIG Testing)
  - 288/1780 jobs migrated out thus far
  - Ensure community-staffed on-call can support
- [stop using k8s-gubernator GCP project](https://github.com/kubernetes/k8s.io/issues/1308)
  (this would be owned by SIG Testing)
  - migrate/replace gubernator.k8s.io/pr (triage-party?), drop gubernator.k8s.io
  - [migrate kette](https://github.com/kubernetes/k8s.io/issues/787)
  - [migrate k8s-gubernator:builds dataset](https://github.com/kubernetes/k8s.io/issues/1307)
  - [migrate triage.k8s.io](https://github.com/kubernetes/k8s.io/issues/1305)
  - [migrate gs://k8s-metrics](https://github.com/kubernetes/k8s.io/issues/1306)
- stop using kubernetes-jenkins GCP project (this would be owned by SIG Testing)
  - gs://kubernetes-jenkins (all CI artifacts/logs for prow.k8s.io jobs)
  - sundry other GCS buckets (gs://k8s-kops-gce, gs://kubernetes-staging*)
- [stop using k8s-federated-conformance GCP project](https://github.com/kubernetes/k8s.io/issues/1311)
  (this would be owned by SIG Testing)
  - Migrate to CNCF-owned k8s-conform (rename/copy sundry GCS buckets, distribute new service account keys)
-   [stop using k8s-testimages GCP project](https://github.com/kubernetes/k8s.io/issues/1312)
    (this could be owned either by SIG Testing or SIG Release)
  - Migrate images used by CI jobs (kubekins, bazel-krte, gcloud, etc.)
  - Migrate test-infra components (kettle, greenhouse, etc.)
  - (This may push us toward [limited/lifecycle-based retention of images, which
    GCR does not natively have](https://github.com/kubernetes/k8s.io/issues/525)?)
- stop using kubernetes-site GCP project (unsure, maybe SIG ContribEx or SIG Docs depending)
  - ???
- Ensure SIG ownership of all infra and services
  - Must be supportable by non-google community members
  - Ensure critical contributor user journeys are well documented for each service

**Have you produced any artifacts, reports, white papers to date?**

We provide a [publicly viewable billing report](https://datastudio.google.com/u/0/reporting/14UWSuqD5ef9E4LnsCD9uJWTPv8MHOA3e)
accessible to members of kubernetes-sig-k8s-infra@googlegroups.com.
The project was given $3M/yr for 3 years, and our third year started ~August 2020.
Our spend over the past 28 days has been ~$109K, which works out to ~$1.42M/yr.
A very rough breakdown of the $109k:
- $74k - k8s-artifacts-prod* (~ k8s.gcr.io)
- $34k - k8s-infra-prow*, k8s-infra-e2e*, k8s-staging* (~ project CI thus far, follows kubernetes/kubernetes traffic)
- $0.7k - kubernetes-public (~ everything else)

**Is everything in your readme accurate? posting meetings on youtube?**

Our community
[readme](https://github.com/kubernetes/community/tree/master/sig-k8s-infra) is
accurate if sparse. The
[readme](https://github.com/kubernetes/k8s.io/blob/main/README.md) in k8s.io,
which houses most of the actual infrastructure, is terse and slightly out of
date (missing triage party)

[We are having problems with our zoom automation](https://github.com/kubernetes/community/issues/5199),
causing [our youtube playlist](https://www.youtube.com/playlist?list=PL69nYSiGNLP2Ghq7VW8rFbMFoHwvORuDL)
to fall out of date; I noticed while writing this report and have gotten help
backfilling. We're currently missing 2020-10-14.

**Do you have regular check-ins with your sponsoring SIGs?**

No formal reporting in either direction. Meetings/slack/issues see active
participation from @spiffxp (SIG Testing chair), and occasional participation
from @justaugustus (SIG Release) and @nikhita (SIG Contributor Experience). We
also see participation on slack/issues/PRs from @dims (SIG Architecture) who has
a schedule conflict.
