kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
community/sig-release/meeting-notes-archive/2022.md

159 KiB
Raw Blame History

13 Dec, 2022 (recording)

Host (pronouns):

Attendees (pronouns):

  • Marko Mudrinić (he/him)
  • James Laverack (he/him)
  • Sascha Grunert (he/him)
  • Xander Grzywinski (he/him)
  • Leonard Pahlke (he/him)
  • Arnaud Meukam (he/him)
  • Angelos Kolaitis (he/him)
  • Grace Nguyen (she/her)
  • Rey Lejano (he/him)
  • Joseph Sandoval (he/him/él)
  • Nabarun Pal (he/him)
  • Drew Hagen (he/him)
  • Adolfo García Veytia (he/him)
  • Stephen Augustus (he/him)
  • Rudraksh Karpe(he/him)

Note Taker (pronouns):

  • Jeremy Rickard (he/him)

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • [Marko] Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • 1.26.0 released, was delayed from thur -> friday
      • Hit a conflict on fast forward job that was difficult to deal
      • See https://groups.google.com/a/kubernetes.io/g/release-managers/c/elijadh1vdE/m/FQZn2w6NAgAJ for more info and https://github.com/kubernetes/release/issues/2807
      • Patch releases hit some flakes, have tracking issues for those to address before next release
      • Included the Go updates as well
      • Go 1.20rc.1 is available and is looking at incorporating
      • OBS work - Marko is going to work on the KEP
      • [Stephen] please keep work life balance in mind, recognize we are a global team
      • [Stephen] if we have issues with releases, identify if it’s going to stop release and then communicate with community (first and foremost). Once we communicate we can go back to planning
      • [Stephen] release dates are optimistic, we will try our best to meet deadlines but it’s not a failure if we don’t
      • [Joseph] what is the criteria for deciding if we need to pick up CVE / go change?
      • [Stephen] AI to Joseph to open an issue to discuss in more depth, we have a set of criteria for what is release blocking, includes incomplete features, security issues in coordination w/ SRC. We need to align w/ the Go release cadence
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • Released v1.26 on Friday (Dec 9th)
      • We had some issues with getting the documentation published. Shoutout to Nate W. Tim B. Taylor D.
      • First retro is scheduled for today
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • [James L] 1.27 Shadow Survey
  • [Arnaud] Consensus on https://github.com/kubernetes/sig-release/discussions/2102
    • Background: we deprecated in 1.25
    • Ideally no new tags for 1.25 and later would not go to GCR
    • [Stephen] we need a communication plan
    • [Arnaud] communicate now, stop pushing in 1.28
    • [Adolfo] unrealistic to do it for 1.27, work on it during 1.27. When 1.28 is out, 1.24 is EOL. 1.25, 1.26, 1.27, 1.28 will all have the updated registry
  • [Jeremy] Cancel any of the meetings for the rest of the month?
    • Cancel 27th and January 3rd
  • [Arnaud] I would like to shadow the branch Manager of 1.27
  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)

(Optional) Walk the Board:

6 Dec, 2022 (recording)

Host (pronouns):

Attendees (pronouns):

  • Marko Mudrinić (he/him)
  • James Laverack (he/him)
  • Leonard Pahlke (he/him)
  • Meha Bhalodiya (she/her)
  • Nabarun Pal (he/him)
  • Jim Angel (he/him)
  • Cailyn Edwards (she/her)

Note Taker (pronouns):

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
    • GO updates - Patch releases 1.22, 1.23, 1.24,1.25 out this Thursday. Final 1.22 release with registry change. ETCD updates will be included as well. 1.26.0 is scheduled for Thursday and will Include the GO updates.
    • Image promotion - summary can be found here https://github.com/kubernetes/release/issues/2789#issuecomment-1339221532
    • OBS - meeting last Wed. Demo what we are doing. Scalability concerns. The OBS teams thinks it will be fine. The KEP will have all the updates.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • (Leo) Everything looks good. Change the release date to 12/8. Discussion related to change was in #release-management. Go/NoGo for the release is in #SIG-Release channel. No blockers at this point. Yesterday red status with Docs but this is now complete. In the release retro we should discuss Ready-to-review. Comms is a go with all the blogs. All blogs will be released before the webinar 1/17/23. Next release team is being assembled.
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • [Leo, Moved from 29 Nov.] How to proceed with the enhancements tracking project board, 1.27 and beyond (slack thread)

    • option1: reuse the enhancements project board for next cycle
      • (-) Using multiple views might be messy
      • (?) Unsure about limits of a single project board (limit of views, total items in a project board)
      • (-) issues can only be entered once in a project board, this makes is more difficult to keep record if KEP issues get postponed to the next release cycle
    • option2: create a new project board (Enhancements Tracking 1.27)
      • (-) We cannot duplicate the enhancements board, manual job atm copying the project board structure
      • (+) clean cut each release
    • (Nabarun) There are multiple reasons to bite the bullet and use a new board. The tracking spreadsheet was used previously. Lose archival functionality if we continue to use the existing board. Evolution of a KEP has its alpha, beta, stable data. If we tried to archive the history in the same project board. We will have different kinds of views and it will get cluttered. Github PM’s are in touch and have been asked what is in the roadmap and our feature needs have been expressed. This will not be resolved in the short term.
    • (James) Does GH API support a move?
    • (Nabarun) There is no API endpoint to do this. GH graphQL down the line will be able to support this.
    • (Jeremy) Sounds like a new board is the consensus. Action going forward is to make this change. GH issue will be created to track and tag with 1.27 and assign to the team. will create the issue and assign team members.

  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)

(Optional) Walk the Board:

Host (pronouns): Sascha Grunert (he/him)

Attendees (pronouns):

  • Xander Grzywinski (he/him)
  • Rey Lejano (he/him)
  • James Laverack (he/him)
  • Leonard Pahlke (he/him)
  • Jeremy Rickard (he/him)
  • Arnaud Meukam (he/him)
  • Nabarun Pal (he/him)
  • Marko Mudrinić (he/him)
  • Jim Angel (he/him)
  • Adolfo García Veytia (he/him)
  • Rajib Mitra (he/him)

Note Taker (pronouns):

  • Rey Lejano

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • [Marko] RC.0 was release last week and went well. RC.1 is scheduled for today. Patch release upcoming next week. Cherry-pick deadline is this Friday, Dec 2, an announcement will go out. Shoutout to Carlos and Sascha for signing for binary artifacts - merged today, and will be kicked off for rc.1, it’s a new subcommand in krel and integrated with the release step. Also OBS poc is going well and successfully completed, all packages are successfully created and published. Kubepkg generating spec files by OBS and all binaries for packages and will discuss later. Also for registry cherry-picks, doing well, all should be merged by now.
      • [Adolfo] Started to work on provenance generation. With Carlos started working on a new GitHub actions repository so others can use these tools in other projects.
      • [Jeremy] Should we hold off with rc.1 until it’s (Carlos’ PR) merged – yes hold off until it’s merged. Xander will cut rc.1
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • On track
      • [Leo] On track, waiting for a few Doc PRs to merge. Blogs some folks are ooo. There are several retro topics so will likely have 2 post-release retro sessions.
      • [Sascha] If we assume, we cut final release as schedule. Should we open the shadow application. Last year, we had the shadow application process over the holidays.
      • [Leo] May need to adjust the shadow app form and will need a few days.
      • [Nabarun] Looking for EA, hopefully decide on 1.27 to hand over EA tasks. And will send end of release shadow surveys this week. Leads are in the process of selecting the next lead.
      • [Leo] Have been working on the release team shadow app tool, no change in the last 2 months but there are a few small things that need adjustment. Have a session with incoming EA on how to use the tool to generate the markdown file so team leads can have an easier time selecting shadows

**
Open Discussion:**

  • Arnaud: Moved from 22 Nov] Stop push images to k8s.gcr.io
    • [jeremy] noticed someone asking specific questions about this in sig-k8s-infra (i.e. what patch releases for 1.25 or later won’t be available in k8s.gcr.io)
    • [Arnaud] Followup with blog post, suggest to stop publishing to k8s.gcr.io in the future starting with 1.27. Reason is that will reduce the cost related with container storage next year and we already made k8s.registry.io GA. Suggest to push to 1 registry, currently push to both in each image process.
    • [Adolfo] Stopping publishing of new 1.25 tags to old registry and with 1.26 looming, we should really do an effort to not publish releases there. Some details of infra has to be built, need private GCR registry.
    • [Arnaud] Still need to define how we’re going to do this. Can say all the existing tags can be pulled but starting in Jan 2023 have to use the new registry. You can still use old tag with new registry since we copied to new registry. For all patch releases with 1.25 going forward will be pulled from new registry
    • [Jeremy] There’s a thread in sig-k8s-infra on when can I not pull from k8s.gcr.io. People will use k8s.gcr.io until when it’s no longer available. In favor of trying to accelerate this
    • [Arnaud] Would like to issue communication in first week of Jan. Make sure we can stop pushing to old registry in Jan
    • [Adolfo] Communicate early and a lot more. We can iron out the implementation details. We’ve had extensive discussions already. We can draft a plan in next k8s-infra meeting and iron out details that’s needed on SIG Release side. We should set expectation that 1.26 will not be published in k8s.gcr.io
    • [Arnaud] The real impact is patch releases because people have the assumption to still pull from deprecated registry. Real concern is how to handle patch releases in the future. Discussions around patch releases for 1.24 and 1.23 are that patches will still be pushed to k8s.gcr.io.
      Need to decide on a plan and implement e.g. change configuration of prow
    • [Mahamad] Be nice to do technical work first and get approval
    • [Arnaud] We decide the timeline and first look at technical detail before we make communications. We should cover everything before we do communications.
    • [Adolfo] If anyone has any objections to not sunset the registry for 1.26, in next k8s infra meeting start working on implementation. Anyone with any concerns? There’s good will from SIG Release with k8s-infra to work on this.
    • [Arnaud] Let’s create a slack thread/github issue and have a consensus and by the next k8s-infra meeting then have a conversation and possibly a decision. Should we do a slack thread or github issue
    • [Jeremy] We’ve been preferring github discussion
  • [Marko] Future of the kubepkg tool
    • [Marko] Trying to accomplish to build spec files and to generate them and to publish them on OBS so they can be picked and generate debs and rpms, we need tooling for this to generate spec files. Before when we had rapture and google build to take care of packages, we were pulling like kubectl binary from bucket. For OBS, not going to be work, OBS builds are air gapped and have to provide binaries before build starts. For multi arch builds, Marko create a tarball and pushes to OBS service. For kubepkg, expecte to generate spec files and tarball that can be pushed to OBS service. Now, kubepkg is experimental and never really took off. Wondering, how much do we want to keep backwards compatibility. Do we maintain what’s there right now?
    • [Arnaud] Are we modifying specification of system packages?
    • [Marko] For each of these, need to generate a dedicated spec file like for 1.25.x
    • [Arnaud] We have the template and we generate spec file from template – yes. If we want to leverage kubepkg then it will be delicate to change it.
    • [Marko] Do we have any idea if people are using kubepkg and have to maintain kubepkg to work as-is.
    • [Sascha] Would like to have spec files as an intermediate solution, expect krel to modify spec files and OBS to build from them.
    • [Marko] Will include this in the KEP, would like to meet with OBS folks. So far best timeslot is tomorrow or Monday next week. Will try to push it tomorrow. For kubepkg, we have some logic to build, do we still need this. Do we have any objection to change kubepkg, please reach out to Marko. Will create the PR to update the KEP.
    • [Sascha] Updating the KEP is good way to keep discussion going
  • [Mahamed from 15th of Nov] https://github.com/kubernetes/release/issues/2734 Notarising macOS binaries
    • [Mahamed] Would like to do something similar for Kubernetes for macOS but not familiar with build process. Need to work out where in the build process after binaries are built to add a shell script to upload the binaries to Apple and supply an API key and certificates then it gets notarized and continue with build process
    • [Adolfo] We’ve been trying to get rid of bash, any chance this can be written in go
    • [Marko] We can wrap it in go
    • [Adolfo] Happy to work with Mahamed
    • [Marko] Do we need an Apple developer account, is there a fee
    • [Mahamed] Yes, we do. Can use Mahamed’s signing keys.
    • [Arnaud] Need to talk to CNCF/steering to get an Apple developer account. Asked steering about 3-4 months ago
    • [Marko] May be a way to integrate this in krel
    • [Marko] One issue is that it changes the checksums.
    • [Adolfo] Will have to be done after the build
    • [Mahamed] yes, done after gobuild
  • [Leo] How to proceed with the enhancements tracking project board, 1.27 and beyond (slack thread) Move to next week
    • option1: reuse the enhancements project board for next cycle
      • (-) Using multiple views might be messy
      • (?) Unsure about limits of a single project board (limit of views, total items in a project board)
      • (-) issues can only be entered once in a project board, this makes is more difficult to keep record if KEP issues get postponed to the next release cycle
    • option2: create a new project board (Enhancements Tracking 1.27)
      • (-) We cannot duplicate the enhancements board, manual job atm copying the project board structure
      • (+) clean cut each release

(Optional) Walk the Board:

Canceled because of light agenda

Host (pronouns): Adolfo García Veytia (he/him)

Attendees (pronouns):

  • Marko Mudrinić (he/him)
  • Xander Grzywinsk (he/him)
  • Angelos Kolaitis (he/him)
  • Leonard Pahlke (he/him)
  • James Laverack (he/him)
  • Jeremy Rickard (he/him)
  • Frederico Muñoz (he/him)tt
  • Sascha Grunert (he/him)
  • Aniruddha Basak (he/him)
  • Shivanshu Raj Shrivastava (he/him)
  • Stephen Augustus (he/him)
  • Drew Hagen (he/him)

Note Taker (pronouns):

  • Jeremy Rickard (he/him)

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • Shoutout to for improvements to image promotion
      • Moved Dec Patch releases up to Dec 7
      • Deciding to add one more 1.22 release with backport of registry change
      • In progress: v1.26.0-rc.0 (some failing jobs on master informing)
      • Will open a github discussion for stopping build of rc.0 for next patch, would apply to no earlier than January
        GH Discussion: https://github.com/kubernetes/sig-release/discussions/2090
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • We have a couple of failings on master-informing for a few days. Focus on resolving these..
      • Flake detection session in todays sig-testing meeting (presented by Antonio Ojea)
      • All Code freeze exception requests completed or postponed to next release :) (project board view)
      • Retro discussion item added, which likely lead to small process adjustments for the release team: “Improve the release process to better incorporate large PRs into releases” (see slack msg)
        • General consensus - not really a sig release issue. Reviewer bandwidth is really the issue
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • [Mahamed] https://github.com/kubernetes/release/issues/2734 Notarising macOS binaries
    • [puerco] Mahamed doing notarising for knative binaries, will defer to next meeting since Mahamed had a conflict
  • [Puerco] Announce of backport of registry.k8s.io to older branches
    • Background: working to shed serving load for artifacts from google cloud to better leverage budget from other cloud providers. KEP open for a long time, things are more pressing now. Short on budget this year by several hundred thousand USD. limited donation from Google to alleviate the burden. Working with sig-k8s-infra to figure out how to shift the burden. AWS taking some load now, but need to accelerate the process.
    • Many users still using older versions, change didn’t get introduced until 1.25. Want to back port these to releases under support still. Would also be good to take it back to 1.22
  • [Leo] Tracking image updates and making sure new images are used (done in two PRs, it happened that the second one that allowed using the container image was forgotten…) (slack thread, mentioned by dims)
    • Release Managers need to follow along with these and ensure last mile is completed
    • When someone, not from rel manager, opens image bumps, we should always assign a release manager
  • [Jeremy] https://github.com/kubernetes/sig-release/issues/2083 Use prow to approve cherry picks
  • [Jeremy] Couple of changes to be aware of in k/release:
  • [stephen] like to propose changes to leadership roster
    • Will open an issue
    • Propose Jeremy from TL -> Chair
    • Propose Verónica as new TL
  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)

No meeting, see subproject updates below:

**Host (pronouns): **Sascha Grunert (he/him)

Attendees (pronouns):

  • Leonard Pahlke (he/him)
  • Xander Grzywinski (he/him)
  • Angelos Kolaitis (he/him, Bug Triage Shadow)
  • Jeremy Rickard (he/him)
  • Rishit Dagli (he/ him, Docs Shadow)
  • Carlos Panato
  • Rey Lejano (he/him)
  • Marko Mudrinić (he/him)
  • Benjamin Elder (he/him)
  • Cici Huang (she/her)
  • Ashwin Kumar Uppala (He/Him)
  • Adolfo García Veytia

Note Taker (pronouns):

  • Jeremy Rickard (he/him) / Jim Angel

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
    • Deckers
    • Drew Hagen (went to KubeCon last week and met the SIGs and interested in sig release)
    • Ashwin (from the chat)
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • We found a bug in bom due to google’s license classifier. This broke release process. Fixed over the weekend, Carlos ran a test.
      • OBS effort going on (demo below)
      • Kpromo changes in place, just files that were touched by merging PR.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • All good so far!
      • Mid-cycle RT status email send out
      • First Retro Meeting scheduled for tomorrow :)
      • Code Freeze next week!
      • (rey) sig instrumentation has a PR open to add a new doc and would like to discuss having release team docs team handle part of that generation
      • https://github.com/kubernetes/website/pull/37522
      • PR not merged yet.
      • (sascha) discuss in the retro. It’s a process change, seems like a small adaptation. If it’s a huge process change probably worth a kep.

**
Open Discussion:**

  • [Sascha] Building deb/rpm packages via rapture in krel: https://github.com/kubernetes/release/issues/2737
    • We can integrate the package build script into krel since https://github.com/kubernetes/release/pull/2708 got merged
    • The code would be required in any case, because we can exchange the rapture implementation with something different (like calls to OBS or similar)
    • Jeremy will work on this with Xander and Cici.
    • Open question: where should we put the binaries.
    • [ben] put in an unadvertised gcs bucket (so people don’t rely on them). gsutils cp to a new bucket, build admins can download to the place they need to go.
  • [Marko] OBS demo (10-15 minutes)
    • See video for demo (it’s cool!)
    • QnA
      • Jim: ISV, is it in the repo name
        • Yes
        • Jason: problem moved to proxy in front
      • Ben: How does each release look?
        • Access to OBS and folder(s)? Per release
      • Puerco: How does auth work?
        • Marko: token - needs discussion
          • Could use OSB itself (vs. golang specifics). Invoked from krel to start. No OBS library for go
      • Ben: Builds are failing for certain archs (arm, powerpc), do we know what the blockers are?
        • Marko: Can be solved, but we need to determine priority and if this is blocking or not. How challenging it would be to work around it.
        • Ben: Why is it only those platforms not building?
          • Can build for each platform but those have specific errors.
      • Puerco: Limitations on free hosting provided?
        • Marko: They are sponsoring us. If we run into limitations we can self-host if needed. From what has been seen, it shouldn’t be an issue. Storage is a minor concern or potentially an area to pay attention to. No issues expected.
      • Sascha: Can rename a generic build add-on name to use as a wrapper around the API? We have to check if we want to support pulling in deps and which ones.
        • Marko: Need to discuss more on the side.
      • Ben: Aside from Windows, the rest of the build is possible to run locally and produce the outputs yourself. On the otherhand, we need ref systemd specs but maybe those packages are not the only way to install them. If we can host it on AWS, we are tight on budget and need more specifics.
        • Marko: Hosting the packages?
          • Ben: Yes, hosting GPG keys and downloads for end users is huge and we have no clue how expensive the current packages are. They are not a product.
        • Marko: OBS folks say they can do it and don’t expect issues. But shares the concern of the size of k8s.
        • Ben: opposed to us hosting it and also having this problem. Hope to get more folks involved.
      • Jason: End goal is to go all on OBS. Since we have span across cloud providers; we want to have a discussion if we introduce a proxy / cache / layer in front to provide flexibility to migrate in the future or deal with URL shuffling x2.
        • Migrating to selfhosted perk: OBS can provde the GPG keys.
        • Ben: I think we have one today that we can point with (apt.k8s.io?)
      • Marko: How do we want to proceed? Try to get all packages working and then work on krel integration. But we should have some green light that we are going to go with OBS.
        • Jason: Unless objections, start putting together a roadmap detailing the current problems, gaps to implement, and action plan (building) and add it to the KEP (what is alpha/beta/ga).
        • Jim: Need to define low-bar RED/GREEN since OBS seems like a go, let’s figure out what that means.
          • Jason: KEP would help provide that structure
        • Ben: I need something to point to Googlers about the path that the project is taking over. They are getting pulled into a larger team and going away. If we have a doc, we can use it to point.
          • Jason: Agree. We can point to the POC and to Marko’s demo. KEP is the next steps for “official doc” instead of “up in the air” we’re working on it
          • Ben: Will bring KEP to breakout and to their managers that we are making progress and thanks!

- KubeCon Contributor Summit Special

**Where: **412A (Huntington Place Convention Center) 1 Washington

Boulevard: https://sched.co/1CXNY

When: 10 am - 12 pm

Hosts:

  • Sascha Grunert
  • Jeremy Rickard
  • Carlos Panato
  • Adolfo García Veytia

Attendees (pronouns):

  • Joseph Sandoval
  • (Orlix)
  • Rey Lejano (he/him)
  • Leonard Pahlke (he/him)
  • Grace Nguyen (she/her)
  • Tim Pepper (they/them)
  • Josh Ferrell (he/him)
  • Nabarun Pal (he/him)
  • Rishit Dagli (he/him)
  • Nate W. (he/him)

Note Taker (pronouns):

**
Agenda:**

  • SIG Release evening hangout planned for tomorrow
    • [Jeremy] There Few places with outdoor options, otherwise go to the alleyway called the Belt (where the contributor summit social was), also a brewery nearby, and Detroit Shipping Company is an option. Coordinate over Slack.
  • SIG Release talk tomorrow 11am: https://sched.co/182MK (142 ABC)
    • [Sascha] All are invited to join. Hoping to have Stephen join us on-stage.
    • [Carlos]Who are the speakers?
    • Sascha, Adolfo, Carlos
  • State of the binary artifact signing KEP (issue) (KEP)
    • Umbrella tracker: https://github.com/kubernetes/release/issues/2586
    • New Step: https://github.com/kubernetes/release/issues/2618
    • [Sascha] tl;dr: KEP is about artifact signing and first step is container image signing
      • Plan to go from alpha to beta by completing signing effort, implement new steps and move signing logic to dedicate gce job. Will need to start implementation.
    • [Adolfo] https://github.com/kubernetes/release/issues/2618: Discussing ways to sign artifacts, we weren’t doing a few things correctly like signing within the release process and not the final release process. So we had two signatures, one from within the release process and one after the image promoter. We did the same with provenance attestation, we were creating inside the release process so we will move to outside. Work on moving provenance attestation is done, Arnaud approved the moving of the repository to Kubernetes org, now we can use new attestor for the builds. Missing is adding signing files/images to krel to be signed outside of the build process. Who is interested in working on this issue? One of the missing tasks in signing KEPs is the SBOMs. Its tricky. The idea is to rework how we are generating the SBOM structure. Leader SBOMs. Attach the SBOMs to the image. SBOM code is in a branch. Waiting to merge.
    • [Dims] What will the user experience be with Debian or Ubuntu when installing K8s or Kubectl?
    • [Jeremy] do we have existing issues with the Signing KEP?
    • [Adolfo] Referring to umbrella issue 2586. File signing
    • [Nabarun] Nabarun worked on file signing, person who worked on initial tests for file signing modified it alot, there were some API changes. When we run the new step, what are we going to do? Promotion/building happens twice, what do you sign, when do you sign. We should write this in the KEP before diving into the implementation. We should spend a few minutes defining what is signed, Go through SBOMs and list through them and what we generate then sign it. Another choice is to go to GCS bucket, then sign from there.
    • [Carlos] Still working on the issue
    • [Adolfo] Discussions on if we should build signing code in krel. We need to implement it ourselves and will take everything and resync the bucket
    • [Sascha] File signing will look different from container image signing since we don’t have artifact promotion for binary artifacts. Signing container images with cosign, need to have binary artifact promotion in place and will do it with k-promo. Consider changing how cosign works and work in our parallel use-case.
    • [Adolfo] Hit some issues with cosign, issues with how we interact with sigstore, fixed a race condition. Also hit with a rate limit which is now fixed.
    • [Sascha] We will split up the issue, Marko is interested in working with the files
  • Package Building POC
    • Options to review: https://github.com/kubernetes/enhancements/issues/1731#issuecomment-1286919467
    • [Jason] We’re looking out how to do deb/rpm packages, unlike the binaries and container images we can’t leverage cosign and they would work in various distros that consume them. Now we use gpg keys that live for the life of repo, right now that process is controlled by Googlers. Looked at a few ways like build similar to Google and run in Cloud Run. Decided to look at third-party systems that would host them. Looking at SUSE Open Build Service (OBS) we don’t interact with gpg keys, it’s controlled by OBS and leverage APIs to kick off the builds and will host the packages as well. Since we don’t build deb/rpms in a normal way, normally we would build from the source, we pre-build binary images and want to use binary images in the release process. We can also run OBS in our own infra if we want community owned infra. Marko tried to POC this out. If we decide to build own process in Cloud Run then we have to manage our own gpg keys. If this poc goes well, then only do a single migration and we can put a proxy or redirector on our own infra to point to obs packages behind in case we want to go self-hosted in the future. We can write a KEP after deciding if we want to move to OBS or have own OBS instance
    • [Adolfo] I’m a bit concerned with the state of that one. From I’m hearing from Google we need to make this happen now.
    • [Jeremy] Chatted with Ben. We need to migrate now. We run Rapture or whatever in the interim. It should be big priority and move quickly. Get a firm commitment if we need to do this before the next release.
    • [Sascha] Prefer on current path and get more time to do the migration. Instead of changing twice.
    • [Jason] Think about impact to users. It will be a few releases before we stop publishing from Google. We need time for a good rollout and limit impact to our users and ClusterAPI and other types of K8s installers.
    • [Jim] Can we run both paths in parallel? Have both channels running then make a switch? Or in case we lose Google support.
    • [Adolfo] One thing to buy us some time is if we build RPM’s. Alignment is taking as much time from them to schedule easily. Google people who help us are doing it on their own time.We can move the build inside KREL.
    • [Tim] What would be the consequences of deprecating RPMs and debs signing?
    • [Arnaud] Upgrades. Issue new GPG key. The system needs to verify the packages. From a package perspective. We have sub-projects that use this.
    • [Jason] A couple of aspects. No assurance that the package is from K8s community. Optics. We used to have signed packages and now we stopped. It could raise concerns with production readiness.
    • [Tim] We don't provide that assurance. Google does. That is different from the project.We should focus on what we provide to the community. Change your code or stop verifying? I like the optics of not verifying. In the short term we are an open source project and the assurances we provide.
    • [Jason] I don’t disagree. Not everyone has free access. By publishing and signing they can verify.
    • [Tim] how would Google verify if community built rpm/deb are intact ahead of signing (eg above that’s mentioned as an optimization to save them time). How do you download and verify trust?
    • [Sascha] Could cosign rpms/debs, so Google could gpg sign them. I’m doubtful companies are ready for this.
    • [Jason] All companies were using upstream
    • [Dims] I’m with Jason, we continue signing. If we want to swing to the right or left. Let’s look at policies across the board: eg: Apache Foundation has a source only releases policy for example. Don’t assume they are going to be there. It might break. We have to extend this outside of SIG-Release. Tell everyone this is the way we are going.
    • [Adolfo] push back a little bit on not signing, working on north start vision document that aligns priorities we want to do, two of those makes project more consumable and other one is to make it more secure. Not signing is a regression, understand that folks are trusting Google and not us. Worried if we take the build of packages within the current release process, isn’t that lead to us handling it ourselves instead of shelving it to OBS.
    • [Tim] Agrees, in abstract think less we build we can focus on high value artifacts. Would focus on containers over rpms/debs, could we also do fewer OS’s and hardware architectures? We always struggle on human resources.
    • [Sascha] Could also revive discussion on supported platforms and have to evaluate which platforms and architectures are necessary. For example, also talk about don’t want to ship binaries for PowerPC which means we don’t build packages for PowerPC
    • [Jason] Remove support for a platform/arch then makes it harder for folks
    • [Dims] If we do something, we should do the same for all artifacts. If we sign one set of artifacts then sign all. If we don’t have enough funding then that is another question. How do we reduce the sizes, the count, etc and do the policies for them and implement them
    • [Sam] Here are two options, we outsource building packages to distributions like deb, ubuntu, and another option is other vendors can produce the builds.
    • [Jason] Outsourcing to distribution vendors - the problem is distribution vendors have tight policies with releases on what they can update. Like release Fedora 37 then only release patches after. Nice about OBS, can build packages for all in one system. More burden if we have to interact with multiple vendors. In terms of vendors supporting this, building packages or hosting
    • [Jeremy] We want to get to the point that everything is community owned. If vendors are willing to do this today, are they willing to do it a year later. Want the project to have more control and don’t want to repeat this process
    • [Dims] Usually we get benchmarked with Docker. Lets try to replicate the same thing
    • [Jeremy] What should be the conclusion
    • [Arnaud] We should try to address one major issue. How do we shift the responsibility of shipping from Google to the community? Hosting and signing is a different conversation on policy.
    • [Jeremy] Immediate need is to shift responsibility, when do we want to do that?
    • [Adolfo] Google wants to see what happens for this release. Would have to check how much we need to rebuild.
    • [Jason] 1.26 is too soon. 1.27 parallel with Google. 1.28 with Google packages existing but default to ours.
    • [Jeremy]
    • [dims] We should do alpha this time.
    • [Nabarun] Can we run the build script ourselves and Google do the signing.
    • [Jeremy] Do we run that in Google cloud build job or on the CI?
    • [Adolfo] Everything new should be new and automated.
    • [Arnaud] Just a matter of days - we can try to build and push to something community owned.
    • [Sascha] What are the reasons, personas?
    • [Arnaud] Remove the needs for a human doing it
    • [Adolfo] We also have SLSA compliance and that breaks.
    • [Nabarun] The reason we can split is it makes it easier for Google. 2 hours from us telling them the release is ready until they finish it. Humans are still in the process.
    • [Jason] Will Google handle publishing?
    • [many] yes
    • [Arnaud] Google will need less time for publishing which is better
    • [Sasha] We’re at the point with OBS POC that we can move that step to now with krel
    • [Jason] OBS is strictly a POC right now. Let’s not be presumptive and see if it meets our needs
    • [Sascha] This is also possible with other implementations. Rapture script runs Docker so won’t have issues running in Google Cloud and push to the same location.
    • [Arnaud] Pushing to Google right now and we need to work on this
    • [Sascha] That’s the interesting part - we need to change more than the script.
    • [Arnaud] We can introduce:
      • Change build script to push to GCS
    • [Nabarun] If the script builds the packages and creates them in the environment where the script runs, if we run script in krel release then do a gcs sync. Have to investigate engineering efforts that work for Google build admins since release is near. AI to investigate and create solid issues and make progress
    • [Jeremy] Create an issue to investigate and how to put in current release step and investigate if its feasible for 1.26/1.27 in parallel to package building poc.
    • [Adolfo] In two weeks from now, anyone who has ideas, lets jump on a call and split up the work. The POC will keep going. https://github.com/kubernetes/release/pull/2708/files#diff-4b4aa17612cae24c2c02cff5d89180524381036f4b77b470a7137bde5c76ddce
  • State of Image Promotion?
    • [Adolfo] We are currently undergoing change with the infrastructure. TLDR. We are not publishing the images we promote. Process jumped from 30-1:00 to 6:00 hours. When we do image promotion. We send a ton of requests to registries. There is the signing process. When we start to promote to new regions it adds to the time to the process. We have a bug that is doing too many operations during promotion pre-stages.
    • [Sascha] We now do some adding to caching. The idea is digests and signatures should not change. We could serialize. This makes it insecure. We need a huge amount of connections to registries. Pushing images takes time. We can reuse connections. We have to see if it speeds things up. We could also split up the jobs by image name.
    • [Mahamed] I suspect we are hitting quotas on AR but no metrics because it is public(gcp product limitation). Also logs files are insanely large 180MB+
      • +1 to sharding the jobs
      • +1 to using git diff to calculate the actual changes we need to check
    • [Adolfo] We’ve been doing that manually with release managers cutting the releases
    • [Arnaud] In the past we used Google Registry this service got deprecated. To optimize cost, we create registry in each region where we have a proxy. The quota requests not empty but we’re at 90%. We don’t have support to raise it. Mitigation is multi-region and that might bring us back and stabilize. Release process should not take too long, don’t want RMs to finish at 3am
    • [Adolfo] We’re starting to see image promoter logs, rate limit errors
    • [Sascha] We changed concurrency from 10 to 100 so we should reduce that
    • [Arnaud] There was no impact, we don’t see utilization if we tweak the number of threads.
    • [Adolfo] The thing with resource usage, the promoter gets stopped waiting for traffic, increasing the concurrency may help with time but then we hit rate limit error. All of this started happening a few weeks ago so we’re trying to understand it
  • [Grace] Concerns with KEP authors about extending code freeze due to KubeCon
    • [Nabaran] We should be ok to ship less, for sustainability. People shouldn’t push hard. About code freeze extension, since this release is short, need to make CI stable after code freeze.
    • [Sascha] We outlined the schedule before, was there anything we could do better. Suggest we take action on this item and send communication to k-dev with the decision
    • [Ben] This is an expected interruption. If it is not expected, then we should communicate explicitly.
    • [Veronica] This is a repeating experience that people are surprised.
    • [Rey] Historically, we add an extra week for the last release of the year. Because of the delay from 1.25, the release is shorter by a week. Just a reminder, we used to do 4 releases a year.
    • [Nabarun] A formal request can be made to the mailing list.
  • [Arnaud] Backport registry.k8s.io to maintained versions.
    • Starting v1.25, k/k repo use registry.k8s.io
    • Proposal to to backport to v1.23, v1.24, v1.25
      • Infrastructure impact on the users/community
    • [Arnaud] Backport change to 1.23, 1.24, 1.25, and will talk to SIG Arch about impact and will also talk to SIG Node and SIG Cluster Lifecycle but from SIG Release perspective, this is changing endpoint in code base not about impacting users. So impact is more at the infra level. Can have a changelog in patch release. Patch release will introduce change to check your infra policy
    • [Adolfo] Doesn’t feel comfortable to backport change, may break users. Want to get a full understanding of all the points where we change it. Can’t keep promise of not backporting change.
    • [Ben] True but also have to understand container runtime have to permit this traffic, if we don’t have consumers to use it as possible then we don’t reduce spend on google cloud account. We see a lot of traffic on very old Kubernetes releases
    • [Sascha] Can backport back to 1.22
    • [Arnaud] If we want to cut a 1.22 release then need to do it this week, don’t want to push it just asking it
    • [Adolfo] Need to have further discussion
    • [Ben] Where we need to make changes, kubeadm and kubelet (for pause) so need to talk to cluster lifecycle and node.
  • [Ben] Stop promoting all 3 GCR registries. Only one.
    • Blockade to stop touch manifests for k8s.gcr.io
    • [Ben] Trying to get the GCR team to redirect the existing registry to a new endpoint. Users just see gcr is breaking them. Gcr team is having problems. There is 25 times more traffic hitting old versus new because users are using old versions. Do similar to vanity domain flip.
    • [Adolfo] Sounds like a more reasonable deprecation path. The question about patch releases is interesting.
    • [Ben] We have to not be afraid about breaking some users. The same users broken by this change will be broken by that change. We dont have transparency into the resources we have. We have to break this pattern. We have to seriously consider it to get the traffic shifting. In the meantime if we get resources from other providers we are stuck. Its hard to get Google to commit more resources. We can’t get traffic onto other providers if we don’t move users over. Users will need to mirror images or provide new endpoints. We can’t support users who expect free registries.
    • [Adolfo] We are sensitive and we know.
    • [Ben] We should stop publishing tags for 1.25. We could consider changing the default. They can opt out and use old registries. Its an OCI compliant registry.
    • [Adolfo] There is some doomsday scenarios for some companies.
    • [Ben] Users should consider mirroring. We don’t have SRE. If you need uptime you should mirror. Backport default to past images. For 1.25 forward we should stop publishing tags. We have to SIG-clusterlifecycle and SIG-Node.
    • [Jim] Comms is the action item. Blog posts. Social media messages for awareness.
    • [Ben] We were pushing this last year. We knew we are over budget. So much latency taking effect. Project is out of credits starting in Nov. Next year's budget is going to be tight. This is by far the largest cost. Taking over ⅔ of the budget. Binary downloads are billed directly to Google. Traffic shifting is the best thing we can do. My POV: we have to get costs sustainable. Going forward start to shift traffic. Letting end users download is a heavy cost. We are exceeding the budget for the community. If we can sort out the download costs. CI costs are more reasonable. We could fit CI and ancillary costs without the massive download bill.

**Host: **Sascha Grunert (he/him)

Attendees (pronouns):

  • Marko Mudrinić (he/him)
  • Rey Lejano (he/him)
  • James Laverack (he/him)
  • Stephen Augustus (he/him)
  • Jason DeTiberus (he/him)
  • Jeremy Rickard (he/him)
  • Joseph Sandoval (he/him)

Note Taker (pronouns):

  • Rey Lejano (he/him)

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • [Adolfo] 2 topics:
        • In process of moving slsa attestor, repo is being donated from Adolfo’s private repo to Kubernetes org.
        • State of the promotion process, image promotion is running very, very slow. Because we are now promoting to a ton of artifact registry regions, before staged images moved from staging registries to prod registries and used to copy to 3 locations now its to over 20. Needs a significant re-thinking of whole process. Everyone is coming up with solutions. Adolfo thanks release managers for releases that are taking more than a day to complete. We are trying to remediate the situation and also trying to rethink how promotion looks like in the future
          • [Sascha] Do we have a tracking issue
          • [Adolfo] No issue, there is an open PR to speed up process with auth but no overall issue
          • [Arnaud from Zoom chat] https://github.com/kubernetes-sigs/promo-tools/pull/632
          • [Sascha] Also affects presubmits
          • [Mahamad from Zoom chat] https://github.com/kubernetes/k8s.io/issues/4374
          • [Adolfo] Rely on the state we know from git, promote without looking at everything in the image. Will present changes at the Contributor Summit next week
          • [Mahamad] Need to rethink assumptions, get presubmits sped up by looking at diff vs head and branch. There are other ideas going around but nothing else in writing. We need to promote to an additional 15 locations on but will add 2-3 hours, don’t need to do this yet but sometime next year.
          • [Adolfo] There is also a plan to mirror images in AWS
          • [Mahamd] That is automated, mirror from bucket in GCR. We copy bucket from Google Cloud to S3. Also need to add more regions in AWS.
          • [Arnaud from Zoom chat] This one should really help us speed up things: https://github.com/kubernetes-sigs/promo-tools/issues/633.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)

**
Open Discussion:**

  • [Sascha] Sent a mail that we moved the Contributor Summit SIG Release meeting to Tuesday:
    https://groups.google.com/g/kubernetes-sig-release/c/yt55iBoxmag
    • [jeremy] related: we will have our social event on Wednesday, will create an invite when we lock down a venue
    • [Sascha] We assume people are traveling on Monday so moved the Contributor Summit SIG Release meeting to Tuesday at 10am EDT (local time) for 2 hours
    • [Jeremy] RE: Social, 2 venues are high on the list a rooftop and an outdoor sitting place. Jeremy will call today and try to reserve a place
  • [Marko] Discuss next steps for debs/rpms building and OpenBuildService proof-of-concept
    • KEP-1731: https://github.com/kubernetes/enhancements/pull/3434
    • We started working on a proof-of-concept for integrating with OpenBuildService from OpenSUSE
      • (+) The initial phase of POC is successful. We managed to build both debs and rpms for amd64 for all packages that we already publish (cri-tools, kubernetes-cni, kubelet, kubeadm, kubectl)
      • (+) Installing published packages works as expected. Migrating from Google to OBS packages also works as expected. Migration are steps are:
        • Remove old repo and GPG key from your system
        • Add a new repo and GPG key
        • Run update/try to upgrade packages
      • (*) Build times were variable -- building in isv:kubernetes was fairly quick, but building in home project was very slow. Considering that we’ll building in isv:kubernetes, I don’t consider this as a problem.
      • (*) Spec files and building process are slightly different than what we got used to
        • We were able to reuse the existing RPM spec file. There were some minor changes needed to get the build working, but nothing significant.
        • We had to modify Debian spec files to make building work. I decided to opt-in for debtransform instead of using dpkg-buildpackage (one less dependency). One problem that I encountered is that there’s no internet connection in Debian builders, so we had to download the binaries beforehand (previously we were doing that as part of the build process)
        • There are few problems on how to organize those RPM and Debian spec files:
          • Both Debian and RPM spec files are living in the same package and same directory (in OBS). If we push some change to Debian spec, that’s going to trigger building RPMs as well
          • Debian packages seem to require having a dedicated repo in OBS for each package. On the other side, RPM spec file can define multiple packages, so only one package in OBS is needed.
          • Relevant to the previous point, do we want to have one RPM spec that defines all packages or have a dedicated RPM spec file for each package?
      • (-) It’s still unclear how are we going to handle multi-arch builds
        • We would have to create a tarball that has binaries for all architectures and then push that tarball to OBS. Potential problem is storage requirements (binaries for all archs can be 1GB+)
        • Alternative approach is to build binaries inside OBS, but then debs/rpms are installing different binaries from what we release. There’s also a potential problem with signing those binaries
      • (-) It doesn’t seem possible to reuse packages built by OBS
        • We can’t build packages in OBS and then send those packages to Google folks to publish them
      • [Marko] Initial phase is successful, was able to built deb/rpms for packages we publish. Used 2 projects for this, isv:kubernetes and Marko’s home project. Isv:kubernetes is most complete and can take a look at how it looks like. How its organized, this is fairly ok. Migration process is fairly simple. Build times are ok, initial experience in home project some stuff was slow but in isv:kubernetes is higher priority and everything was finishing in a matter of minutes. Building packages shouldn’t take more than 30 min. What is different and need to pay attention, spec files need to be changed.
        • For deb packages, we have a package with dedicated kubelet, kubectl etc
        • For rpm, everything in same spec file
        • OBS wants 1 spec file but Marko prefers multiple spec files
        • Not so good, not clear on multi-arch builds, would have to create a tarball for all architectures and push tarball to OBS. But this will take a lot of storage
        • Will also have problem with signing, won’t be able to use keyless in OBS
        • Next steps, meet with OBS folks and use hosted OBS so we don’t have to run our own OBS
        • [Jason] OpenSuse already said they are willing to host on hosted OBS infra and if we want to migrate then we have flexibility to migrate. Recommend to put proxy in place, so if we need to change where we host images again (hosted/own OBS), users don’t need to change URLs again.
        • [Sascha from Zoom chat] OBS supports service files, for example to download files from builders: https://github.com/openSUSE/obs-service-download_files
        • [Adolfo from Zoom chat] A redirector like with the images
        • [Marko] Next step is to get multi-arch stuff working
        • [Sascha] What is your overall impression with integration with our current process and how OBS fits?
          • [Marko] This is still something we have to see, we have to create our own API layer to use. We don’t need too much, like need to push files
          • [Stephen] With regards to services, a value in OBS and having a spec file per version then people can build themselves. I would continue to try on that offline path assume their build environments will be air gapped. In the tool, we shell out so many things in general, this will be a trigger somewhere like a separate GCB job or another step in the GCB process.
          • [Stephen] this is great work, do you need anything from SIG leadership side?
          • [Marko] need to decide on how we use service files/specs per package/rpm, need help to make that decision
          • [Stephen] we should really have separate spec per package not the omnibus that we have now. We’ll need to think about how this will change how users will consume packages. Need to review supporting docs (install docs), etc and adjust expectations. As much as possible, we should be clear in communications and break things if/when necessary. Backward compatibility a nice goal, but shouldn’t sacrifice overall goal by being hampered by BC.
          • [Marko] good feedback, thank you. Next step will look into multiarch and meet with OBS folks to review and get feedback. Will continue working on this and report status. Should get KEP merged.
          • [Sascha] anything that will compromise the community owned aspect of this, can release managers debug everything?
          • [Marko] not truly community owned because running on OBS infra. We can give access though, and solves problems like GPG keys because only public keys will be exposed to those accesses. Running OBS could be done on our infra but its not ideal. Should satisfy the needs and spirit

**Host: **Jeremy Rickard

Attendees (pronouns):

  • Rey Lejano (he/him)
  • Jason DeTiberus (he/him)
  • Marko Mudrinić (he/him)
  • Xander Grzywinski (he/him)
  • James Laverack (he/him)
  • Arnaud Meukma (he/him)
  • Cailyn Edwards (she/her)
  • Leonard Pahlke (he/him)
  • Mahamed Ali
  • Joseph Sandoval (he/him)
  • Carol Valencia (she/her)

Note Taker (pronouns):

  • Rey Lejano (he/him)

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • [Marko] Migration from GCR is completed, thank you to Mahamed Ali for creating all the PRs. No other major updates other than a set of patch releases tomorrow for 1.23, 1.24, 1.25, and another alpha for 1.26 is coming out today. Also discussing on Slack the golang update
      • [Jeremy] 1.26 alpha cut is in stage and kicked off with Cici, first release done with new image promoter changes
      • [Jeremy] The package publishing stuff has some urgency, any updates or blockers?
      • [Marko] We started discussion with Jason and Rey to start testing, Sascha also updated KEP on publishing. We’ll try to do poc and see how it all works
      • [Adolfo] What’s the problem on the Google side? Why the urgency
      • [Jeremy] Less availability on the folks, Ben suggested to build packages and Google folks can sign and publish them. That team has less bandwidth and not much more bandwidth in the future. The initial proposal was that we build packages and set them up for them to sign. And now OBS can possibly take care of everything
      • [Jason] We take pre-signed pre built binaries. We also discussed doing a similar poc in cloud build
      • [Adolfo] From Adolfo’s understanding was that poc in cloud build was set aside and to do poc with OBS
      • [Jason] Waiting on access to project for the poc, we will work on it in personal project for now and then migrate. Marko will start in a personal project so we’re not working on it
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • Collecting major themes and deprecations for the mid-cycle blog
      • Doodle out to find a time slot for the APAC friday meeting
      • (enhancements freeze in effect) Exception requests:
        • (🟢) CCM Webhook Framework: request
        • (🟢/🔴) Subsecond Probes: request
        • (🟢) AppArmor Support GA: request
        • (🟢) Dynamic Kubelet Config : request
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • [Added by Rey, from Slack thread] Proposed KEP, KEP-3344 Release Team Roster
    • [James] Process KEP to change how release team is built/assembled, doesn’t change roles/jobs. Motivation is the experience for new contributors is laid out, experience for leads is laid out but experience for returning contributors in the next release is not great because you go through the same pathway and we see people falling out of being on the release team. This KEP gives a way for folks to take breaks and return to the release team. After shadowing for a few cycles, then the release team will be built from both stable roster and new shadows.
    • [Veronica] Stephen also wants to empower people and not want to be a deterrent for people not to join based on their time commitment.
    • [James] The intent for the KEP addresses the concerns mentioned. Should we change focus as Sascha’s suggestion, returning path to release team
    • [Jeremy] Maybe the term roster, may not be the approach. Maybe refining documentation to include how folks can come back. Maybe focusing on Sascha’s comments is a good first step
    • [Veronica] Similar to now, people tend to do the same tasks. Also there is a sense of trust earned with experience.
    • [James] People have been ambivalent or -1 so this may not have the support. Taking this as a signal that this needs to change. Do we think there is still value. Do people agree with me that process for people returning is an issue – consensus is yes
    • [Leo] Teams differ from complexity as well so some teams might benefit with more returning RT members. Might be good for the retro and brainstorm about the teams
    • [James] Will redraft KEP to be about returning contributor path
  • https://forms.gle/3qzSmYgYkdbrJkzS7 ← vote on when the sig release social should be
    • [Jeremy] May get a room on Tuesday to meet along with the sig release social (link above). Please take a look at the survey for the sig release social
  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)

(recording)

**Host: **Sascha Grunert (he/him)

Attendees (pronouns):

  • Jeremy Rickard (he/him)
  • Marko Mudrinić (he/him)
  • Jim Angel (he/him)
  • Mahamed Ali

Note Taker (pronouns):

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • (Marko) No major updates.
      • (Mahamed) GCR to AR migration is coming along well and is close to the end. Flood of pull requests so thanks for the support.
      • (Jeremy) Released a new version of promo-tools if you see issues let us know
      • (Sascha) Release artifact KEP is tracked for this release. Docs are required and will assess if we can achieve beta this release cycle.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • [mahamed] GCR to AR Migration. I need someone to cut a new patch release for k-sigs/promo-tools so I can carry on with the migration
  • [Joseph] Kubecon SIG-Release events and planning. Is there any support needed?
    • (Jeremy) Submitted a contributor summit session for an in person SIG-Release. Having an in-person meeting would be interesting. Topic potentials could include release signing. Identify which things we want to identify in person.
    • (Sascha) If you have attended past on-site events do you have suggestions on how to improve.
    • (Joseph) After attending Kubecon Valencia a formal agenda will help make it even more meaningful.
    • (Jeremy) We can try and record it. Best effort with limited A/V support from onsite staff.
    • (Sascha) Evening get together is still up in the air. We require a location. Spin up a doodle poll to vote on potential locations.
    • (Jeremy) Pulling together a list of spots. We can then vote on the options.
    • (Sascha) What day?
    • (Jeremy) Tuesday as a potential option.
    • (Sascha) Also maybe Thursday. We can get the list of locations. Present the day option with the poll.

Walk the Board:

Canceled because of light agenda

Canceled because of missing agenda

**Host: **Jeremy Rickard

Attendees (pronouns):

  • Rey Lejano (he/him)
  • Cici Huang (she/her)
  • Frederico Muñoz (he/him)
  • Shivanshu Raj Shrivastava (he/him)
  • Xander Grzywinski (he/him)
  • Tanisha Banik (she/hers)
  • Nabarun Pal (he/him)
  • Nate W. (he/him)
  • Rishit Dagli (he/him)
  • Arnaud Meukam (he/him)xx
  • David Espejo (he/him)
  • Verónica López (she/her)

Note Taker (pronouns):

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
    • David Espejo - Works with VMware and supports OpenSource projects. 1.26 Comms shadow.
    • Tanisha Banik - Part of DevOps Continuous Delivery team at Cisco
    • Ruheena Ansari - Part of the 1.26 shadow Enhancements.
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • Go 1.19.1 and Go 1.18.6 were released and we updated the relevant k/k branches.
      • This week patch releases for (thanks Arnaud and Jim for handling releases)
        • 1.22.14
        • 1.23.11
        • 1.24.5
        • 1.25.1
      • Will do the 1.26 alpha release soon.
      • (Rey) Branch renaming - what was decided? 1.26 or 2023?
        • (Jeremy) Postponed the branch renaming until 1.26.
        • (Cici) How do we follow the updates outside of email?
        • (Jeremy) Take advantage of community twitter, Kubecon etc. Be as communicative as possible. Someone will miss the message but we will do our best effort to reach all.. #2853
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • (Nabarun) Release cycle started.
      • Enhancements collection has started.
      • Shadow selection is almost complete. Email will be sent out to applicants not selected in the next hour.
      • Tomorrow will be introducing the new shadows at the release meeting.
      • Comms team is looking to use GitHub project boards. See https://kubernetes.slack.com/archives/CNT9Y603D/p1662994419135199 for discussion
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • Planning a SIG Release get-together for KubeCon NA on Oct 25
    • Meeting at the contributor summit on Tuesday in one of the available rooms
    • Hanging out in the evening in a chosen restaurant or bar
    • (Jeremy) CFP has been submitted for SIG-Release. Feel free to submit other topics like enhancement collection, branch rename and other SIG-Release themes. Social hangout in the evening would be cool. Are there things we would like to do?
    • (Joseph) Any support needed?
    • (Jeremy) Not yet but will open a slack thread when we do. We’ll try to do it on zoom if possible. Next week the meeting will be at an alternate time.
  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)

\

**Host: **Jeremy Rickard

Attendees (pronouns):

  • Rey Lejano (he/him)
  • Mark Rossetti (he/him)
  • Leonard Pahlke (he/him)
  • Nabarun Pal (he/him)
  • Nate W. (He/him)
  • Frederico Muñoz (he/him)
  • Jason DeTiberus (he/him)
  • Rishit Dagli (he/ him)

Note Taker (pronouns):

  • Rey Lejano (he/him)

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
    • Mark R (new to this meeting)
    • Fred (new to this meeting)
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • [Nabarun] 9th of Sept is deadline for patch releases, doing patch releases next Wed
      • [Veronica] Will take care of cherry-picks, it will be done in next 3 days
      • [Adolfo] Also have been doing cherry-pick reviews, as a reminder to Release Managers to review cherry-picks and better to review earlier to get signal. We are mostly going to cut a 1.22 patch release so if you see any back ports to 1.22
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • [Leo] In the first week of 1.26. The scheduled is finalized as of last week. Team is selecting shadows. Next week, expect to have shadows selected. Last week, there has been traffic on enhancement project board. We had a meeting last Thursday and sent out notification emails to the community. Some process to automate the project board is a WIP, changes that are relevant for community and SIG leads are set. Ryler (Enhancements lead) is currently OOO and trying to push things forward.
      • [Nabarun] Since work is in project board and not sheets, should be less work in the start

**
Open Discussion:**

  • [Leo] Release Team Shadow Analysis Report for 1.26; diagrams are ready for review if we like to publish the report (issue: https://github.com/kubernetes-sigs/release-team-shadow-stats/issues/7#issuecomment-1238387791)
    • [Leo] This release we used the analysis tool that generated markdown files to ease selection process and diagrams. Issue open to publish to the community
    • [Adolfo] Question on structure on that repo, missed the open PR last week and now added the repo to Adolfo’s daily workflow. Need anything in terms of automation
    • [Leo] Need to add documentation on what we need for the release. The automation part of things is a little tricky.
  • [Puerco] Started the migration process to donate Tejolote, the SLSA attester to implement our new provenance flow! https://github.com/kubernetes/org/issues/3656.
    • Currently running test builds in my fork
    • We need to follow-up with infra changes
    • We should set it upin our baby repos (ie bom, zeitgeist) to start more serious testing
    • [Adolfo] Note that Adolfo got the SLSA attester to a working state. Started to run tests on building Kubernetes releases, haven’t tested e2e yet because of some required infra changes. Started the donation process. If anyone wants to be an early tester. Need project to build on GCB or GitHub actions.
    • [Veronica] Proposing vitess to test
    • [Adolfo] Added new features like collecting native artifacts from buckets, etc and hopes to do a demo
  • [Jeremy] Current meeting alternation will conflict with sig testing. Propose swapping (so this meeting at the same time next week, earlier meeting in 2 weeks)
    • This new time conflicts w/ sig-node too
    • [Rey] Also conflicts with SIG Docs also
    • [Jeremy] Since this overlaps with SIG Testing, propose to swap alternations. Do next meeting at this time and we’ll do the right cadence. Will send another message to the mailing list and update the calendar
  • [Puerco] Let’s this discuss about signing mac binaries, continuing from this thread. Some pointers to previous issues:
    • CNCF MacOS Developer Account for Signing Release Binaries
    • Kcp: bug: Mac binaries don't run out of the box
    • Sign minikube binaries for macOS
    • [Adolfo] Raised about an hour ago, about signing mac binaries. Brought up by the KCP folks. If you want to run binaries on a Mac, has to be signed by authorized CA by Apple. Has also been raised in other projects.
    • [Mark] Also noticed an issue with Windows binaries being signed, has it been discussed before. Might be a good time to sign all the different platforms.
    • [Adolfo] Planning to move artifact builds and also run through promotion via file promoter like we do with container images. If you have folks that are interested in the project – it would be good to bring them to meetings and if you have experienced people to help that would be good.
    • [Mark] Can find some Windows people to help
    • [Arnaud] Requested funding for an Apple developer account to sign. This was brought up to steering in May. There was a discussion on how CNCF will reimburse developers for accounts for signing. Can bring this up to steering in the next meeting which is next Monday
    • [Jeremy] Can join the steering meeting with Arnaud to talk about funding for accounts for signing. Is there something different for Windows
    • [Mark] Need MSDN account first then ca

Walk the Board:

August 30, 2022

Canceled because of having no agenda.

August 23, 2022 (recording)

**Host: **Jeremy Rickard

Attendees (pronouns):

  • Nabarun Pal (he/him)
  • Sascha Grunert (he/him)
  • Cici Huang (she/her)
  • Arnaud Meukam (he/him)
  • Stephen Augustus (he/him)
  • Marko Mudrinić (he/him)
  • Rey Lejano (he/him)
  • Adolfo García Veytia (he/him)
  • Shivanshu Raj Shrivastava (he/him)
  • Mahamed Ali
  • Tim Bannister (he/him/they/them)

Note Taker (pronouns):

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees *
  • Subproject updates
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • 🎉🎉🎉 HAPPY RELEASE DAY 🎉🎉🎉
      • (Cici) No concerns. Looking forward to the release and thanks to everyone.
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • (@puerco) Work to resume on securing the k8s supply chain:
        • SLSA3
        • File signing
        • System Packages update?
        • (Adolfo) We’ll be resuming things we paused for the release. Artifact work with SIG-K8s-Infra will be a focus.
        • (Mahamed) We passed a key milestone. We got all images serving from the AR registries. OCI proxy project we will be ready to start telling everyone to start using it.
        • (Jason) System packages POC. The team met with OBS last week and explained our build and signing process. Meeting again this week (will send an invite out) and Suse will work with us to understand how to work on their platform. Storage optimization on their platform is one of the topics. They will require storage on the most expensive tier. OBS team has some ideas on how to alleviate this requirement. After this we will be able to kick off the POC. Arnaud was investigating a Google POC option. Notes: https://docs.google.com/document/d/1BktYecXMky4RFnMm3CPMmyKye_TA8YUah4igs18o3FE/edit
        • (Stephen) We should record meetings since they are community related. If there is no recording of the meeting then make sure to capture notes for the community.
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • Provenance builder demo
    • Repo: https://github.com/puerco/tejolote
    • (Adolfo) _Demo: _Observe the build as it runs. It checks the number of artifact storage locations. Supports directories and OCI. If it runs well it will give provenance attestation. The project supports G Cloud Build and GH Actions. Next up sync with Mahamed to extend what the tool supports. Opening a donation request to the Kubernetes community for comment. If you are interested and want to join let me know.
  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)

Walk the Board:

August 16, 2022 (recording)

**Host: **Jeremy Rickard

Attendees (pronouns):

  • Leonard Pahlke (he/him)
  • Xander Grzywinski (he/him)
  • Rey Lejano (he/him)
  • James Laverack (he/him)
  • Jason DeTiberus (he/him)
  • Arnaud Meukam(he/him)
  • Jim Angel (he/him)
  • Mahamed Ali

**Note Taker (pronouns): **James Laverack

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
    • No new attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • [Adolfo] Two things going on. First is SLSA attestor for the Kubernetes project. Moving slow but steady. Hope by the next meeting to have the request to donate the repository, maybe to a demo next Tuesday. Related to the SLSA effort in the overall project.
      • [Adolfo] The second thing is the effort by SIG K8s Infra to mirror the images. Mahamed has been doing a lot of work on this front. Jason has been doing some opinion polling about the packages.
      • [Jason] Less opinion polling, trying to get feedback from downstream projects and products that are using the DEBs and RPMs directly. So far we’ve gotten three hits on that. Need to follow up on that. We’re also following up work on the PoC to use OBS to build and publish the packages. We’re going to have a further follow up to walk though their tooling. Continuing the initial stages of that PoC with OBS. Kubeone and Lima are the only downstream projects that have responded as using the official k8s deb/rpms so far.
      • [Rey] I will create a namespace for Kubenretes on OBS.
      • [Jeremy] Jump into #sig-release if you have further comments.
      • [Mahamed] I’ll come back to images later
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • [Rey] Last full week of the release. 1.25.0-rc.1 is being cut by Verónica. Still on target for release August 21st.
      • [Xander] Smooth sailing.

Open Discussion:

  • [Jeremy] Meeting time change update
    • [Jeremy] Sent out an email today. Extended the lazy consensus, and set a target date.
  • [Mahamed] GCR to AR Migration Updates
    • [Mahamed] Coming along rather well. Since last week we managed to merge two things. One is a test PR to promote things. Question: If you have a bad promotion does that affect future ones?
    • [Adolfo] Depends. There are times when the promotion could fail and get stuck in a weird state. But usually you should just re-run it.
    • [Mahamed] It’s complaining about promotions, but they’ve been granted.
    • [Adolfo] As it didn’t copy anything, we can just re-trigger it. It could be trickier in the future, like if it failed half way through copying an image. It might make a mess if it doesn’t detect this in future runs.
    • [Mahamed] We need to get the permissions straightened out. The other PR is backfilling the repos, merged this morning. Running into some problems with images. Need to take another look. Should be all ready by the end of the month.
    • [Adolfo] you can see failed promos here: https://prow.k8s.io/?job=post-k8sio-image-promo
    • [Adolfo] If you want to share details as you go along, then let us know.
    • [Mahamed] I can’t debug as I don’t have permissions.
    • [Adolfo] I’ll check in a little bit to see if I do.
  • [Adolfo] New Tool Donation (discussed during the subproject update)

Walk the Board:

August 9, 2022 (recording)

**Host: **Jeremy Rickard

Attendees (pronouns):

  • Jason DeTiberus (he/him)
  • Rey Lejano (he/him)
  • Nabarun Pal (he/him)
  • Cici Huang (she/her)

Note Taker (pronouns):

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees *
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • (mahamed) GCR to AR Migration: Infra changes has been approved by sig-k8s-infra. I need lgtm from sig release leads. We will be provisioning 21 new registries to promote images to. Registry manifest changes also need lgtm from sig release leads.
      • (Adolfo) https://github.com/kubernetes/k8s.io/pull/3956 The change is to start promoting to new registries. Is it a change just to manifest?
      • (Arnaud) Can the image promoter handle this change in one process?
      • (Adolfo) We have take into account the time it takes to promote. We are promoting around 50 images to three target registries which takes around 20 minutes. It involves signing and copying images to different registries. Once this project is in place. We should test the promoters ability to scale.
      • (Arnaud) We are in 7 regions at the moment. There is some regions where we would not be deploying OCI-proxy.
      • (Mahamed) We might be limited by the amount of registries to promote to.
      • (Adolfo) Latency to CDMX to SA is high.
      • (Mahamed) Lets go with 21 regions for now.
      • (Adolfo) We need to take into account the amount of time it takes to replicate.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • (Cici) Past code freeze last week with 11 exceptions received(6 of them have been approved and all tracked)
      • 40 enhancements tracked; 16 feature blogs registered
      • Today: Test freeze; rc.0 cut; doc ready to review; start final draft of release notes
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)

Walk the Board:

August 2, 2022 (recording)

**Host: **Adolfo García Veytia

Attendees (pronouns):

  • Jason DeTiberus (he/him)
  • Sascha Grunert (he/him)
  • Marko Mudrinić (he/him)
  • James Laverack (he/him)
  • Rey Lejano (he/him)
  • Mahamed Ali
  • Cici Huang (she/her)
  • Arnaud Meukam (he/him)
  • Adolfo García Veytia (he/him)

Note Taker:

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

**
Open Discussion:**

  • Copied from the previous meeting: [Priyanka] GitHub Project Beta automation:
    • Ref: https://github.com/kubernetes/org/issues/3558
      • Could we use the Github token stored as secret k8s-triage-robot-github-token in the GKE cluster k8s-infra-prow-build-trusted for Prow tasks running on the GitHub Project Beta board? Or should we request a new one?
      • (Arnaud) Priyanka is looking for a +1. Get feedback from Christoph. Please share feedback on the issue.
      • Automation script for Bug Triage board. Need a review - https://github.com/kubernetes/sig-release/pull/1969
        • Action item - Jeremy will review
      • Need discussion on Enhancement automation script: https://github.com/kubernetes/sig-release/pull/1968
        • Jeremy will review
      • Feedback from the recent Enhancements sub-project meeting discussion: Two ways to move ahead:
        • Keeping the KEP opt-in process as it is & manually moving the opted-in KEP to the board
        • OR, come up with a new label on k/enhancements for release opting-in having access to limited folks & use that to filter issues & fill in the board.
  • Copied from the previous meeting: [Carlos Panato] Kubernetes/Kubernetes branch rename, planned/proposed to 8th August 2022, after the code freeze
    • Delayed until after 1.25.
    • (Rey) Any comms for this change?
    • (Sascha) We’ll review and provide comms with SIG- Contribex out to the community.
  • [jeremy] Meeting time Google Form! (not doodle anymore) https://docs.google.com/forms/d/e/1FAIpQLSerNaCLFK9gBAa4FPXCcHWp7QgCpy6jOc4hoQgJSZDcl8nZrQ/viewform?usp=sf_link
    • 10-11 PST for west coast. 4-5 CEST for EMEA times.
    • (Joseph) When will the time change be implemented?
    • (Jeremy) Send notice out for a few weeks before changes.
  • [arnaud] Publishing Kubernetes packages on community infrastructure
    • KEP PR: https://github.com/kubernetes/enhancements/pull/3434
    • GPG keys: https://github.com/kubernetes/release/issues/2627
      • (Jason) Avoid having to deal with GPG key owned by the community and the challenges that it entails. Reviewing public build services who can manage this. OpenSuse is the most promising option. A meeting is being arranged to field questions from SIG-Release. Looking for Suse employee to potentially see if they will host the part of the project. Scheduling the meeting later this week. It could simplify the process.
      • (Rey) I work with Suse and can help facilitate this.
      • (Arnaud) What Jason is working on is one part of the KEP. Change the way KREL works.
      • (Sascha) We can easily merge this KEP. We’re in favor of building on our own.
      • (Jason) GPG signing is needed before uploading packages with other vendors. Unless we want to take on GPG key management and access with members who rotate. We do need to delegate this to a third party. If we had Co-Sign it would be alot easier to invalidate previously signed packages. If we had to invalidate a GPG key we have to resign package and metadata.
      • (Marko) This is something that should be CI or in our case G Cloud Build. Release managers would be able to use a key.
      • (Arnaud) Jason is proposing using a third party vendor to manage a web of trust. We should explore multiple possibilities before making a decision.
      • (Sascha) In the Signing KEP we would move this out of KREL stage.
      • (Arnaud) We could start with a POC. Jason continue to investigate the third party vendors.
      • (Sascha) Do we have infrastructure to host pre-build?
      • (Arnaud) We could host in GCS bucket
      • (Jason) Willing to lead a PcC for an OBS based pipeline.
  • [JamesL] Release Team Selection redesign (KEP 3344) — Push to 1.27? * (James) Its best not to rush this into current release. Does this go through the KEP selection process? * (Jeremy) I don’t think it needs to go through the KEP process. * (James) It might confuse things if we did.

July 26, 2022 (https://youtu.be/-LinisivLhk)

Host: Sascha Grunert (he/him)

Attendees (pronouns):

  • Rey Lejano (he/him)
  • Cici Huang (she/her)
  • James Laverack (he/him)
  • Priyanka Saggu (she/her)
  • Carlos Panato (he/him)
  • Parul Sahoo (she/her)
  • Matthias Glastra (he/him)
  • Adolfo García Veytia (he/him)
  • Mahamed Ali
  • Joseph Sandoval (he/him/él)
  • Nabarun Pal (he/him)

Note Taker (pronouns):

  • Rey Lejano

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Note: we have combined the biweekly meeting with the releng meeting. Please see the releng archive http://bit.ly/k8s-releng-meeting for prior releng meetings!
    • [Sascha] Doodle out on SIG Release mailing list to add preferred times. Jeremy will reveal results soon and adjust appointments and hopefully it’ll satisfy many timezones
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • [Adolfo] Have met with Lauri in the last 2 weeks. Kicking off work on 3 compliance efforts.
      • [Sascha] Anything on Golang updates
      • [Arnaud] Dims is working on that and waiting for GA next week
      • [Carlos] Carlos was working on updates for Go 1.19 but went on vacation and saw Dims took over. May run into the same issue as Go 1.18 with memory leaks. Not sure if we want to move to Go 1.19 since we’re close to code freeze. Possibly push to Go 1.19 in 1.26
      • [Arnaud] Dims and Jordan working on moving to Go 1.19. May move to Go 1.19 after code freeze.
      • [Carlos] Will sync with Dims
      • [Carlos] With Go 1.17, 1.18 there are patches but didn’t see any issues on our side to update, will work on updates
      • [James] Do we have a plan in case there’s a bug with Go 1.19
      • [Carlos] Will work with Dims
      • [Veronica] Have seen people propose an additional RC that we can do. In Veronica’s opinion, may delay a week or 2. An RC would be a good way to gauge the situation and if things break bad then…
      • [Cici] Was going to ask the same discussion, saw someone propose if we can revert Go back to 1.18 so there’s no delay
      • [Carlos] In Carlos’ opinion, if we upgrade and it doesn’t work then we can go back but there is lots of steps to do (to go back)
      • [Arnaud] There are many factors to consider, if we identify a bug in Go 1.19 RC and fix can be in GA version of Go 1.19. No problem in having Go 1.18 for 1.25
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • 1.25 mid-cycle retro is tomorrow
      • Code freeze is next week

**
Open Discussion:**

  • [James Laverack] Release Team Selection redesign (KEP 3344)
    • Issue: https://github.com/kubernetes/enhancements/issues/3344
    • Proposed KEP: https://github.com/kubernetes/enhancements/pull/3347
    • [James] Intent or desire is to change how shadows are selected to make it easier for contributors to be part of the team. Suboptimal that folks have to apply a few times for the release team. If we did this, this would be a straight to GA announcement. No real way to trial this, the back-out operation is to try it for a release and go back to former process in following week. Not sure if there’s time to do this for 1.26. Communication would be for existing and former release team members as they will be part of a “roster”
    • [Sascha] How about sending an email to k-dev and get additional input from the community. In general, in favor for this enhancement. There would be some documentation changes and building the roster (e.g. gathering GH users etc)
  • [Adolfo García Veytia] Overview of proposed file signing flow
    • https://github.com/kubernetes/release/issues/2618
    • [Adolfo] There have been plans and ideas to build this into krel but Adolfo things this has to be another way. We currently sign images twice, once in staging and once in promotion. Problem with this approach, we expose credential of signing account to any code that generates the build. Krel is not a security scanner, it just runs make and anything in the environment is made available. Propose to move signing out of staging, so first build and stage images and add a step to sign the files and eventually the images, then since we know what we build from staging with sboms then we store signature in files in staging bucket then we push signatures to registries then carry on as usual with image promotion and release. Want to call out, building-out signing images outside of krel. Might as well use co-sign to generate signature for artifacts. We can add a step to use co-sign to sign, this applies to both files and images. Busy at work, building library to making sigstore tools easier. Still can use our code to move signing ability to kpromo.
    • [Matthias] Is signing in cloud promotion or is it in another step
    • [Adolfo] Adding a step to sign files in google cloud build runs. But only applies to kubernetes release binaries and files in release bucket. If we want to help whole community to sign files then help folks use cloud promoter
    • [Sascha] New workflow is way better than what was proposed in a KEP and we need to update the KEP. We may need to use some bash although we wanted to get rid of any bash like anago and we created krel and libraries and we have intermediate layers of testing. Prefer for an implementation release SDK.
    • [Adolfo] We need to add logic to loan the bucket and release the sbom.
  • [Adolfo García Veytia] Overview of proposed attestation process
    • https://github.com/kubernetes/release/issues/2611
    • [Adolfo] What is provenance attestation. Has 2 parts, a subject and predicate. I “did this” (predicate) to “that” (subject). What are we running to make this happen. Same story with attestation with signing the images, currently generating attestation to describe staging run and we do the same when we release. One of the requirements for SLSA level 2 is that we need to sign those, we want to sign these outside to not reveal credentials. The idea would be, before we run staging and after clone repo, we build partial attestation that captures build point, parameters to execute krel and other things we capture before the run and pass it to krel so the idea is to add a 2nd (post-staging) step to retrieve the attestation in a bucket or volume and use the previously partial attestation and subjects in sbom to combine those to form full attestation and sign it outside of krel and we store attestation in the bucket and do the same thing to describe the release step — do partial then run release then combine. Need to get everything from source to finish project so we generate attestation for stage and for release and info from image promoter like what we did to promote the images so we need an attestation for promotion.
    • [Sascha] Will this be part of the SLSA KEP?
    • [Adolfo] Right now, just proposing change to the flow to see if anyone has any objections. For next week, can get the builder ready and do an execution run in Adolfo’s branch to verify it works. And will build another tool to make this effort usable by other projects.
  • [Lauri Apple] roadmapping update: Publishing Kubernetes packages on community infrastructure issue updated with workplan; user personas WIP; signing release artifacts + user personas WIP
    • [Lauri] Work is in progress, go to links above and you can see work done in community infra issue and see milestones and tasks that are identified so far. There is an effort to create user personas. We did a user persona exercise yesterday
    • [Arnaud] Request for milestones on the issue, is it possible to merge milestone 1 and milestone 2 because aspects of both are directly related.
    • [Lauri] It is to show we have code needs and doc needs. \

Walk the Board:

July 12, 2022 ()

**Host: Jeremy Rickard **

Attendees (pronouns):

  • Cici Huang (she/her)
  • Leonard Pahlke (he/him)
  • Stephen Augustus (he/him)
  • Verónica López (she/her)
  • Mahamed Ali
  • Carlos Santana (he/him)
  • Jason DeTiberus (he/him)
  • Nabarun Pal (he/him)
  • Joseph Sandoval (he/him/él)

Note Taker (pronouns):

Recurring Topics [timebox to 20 min]:

**
Open Discussion:**

  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)
  • [Lauri] SIG Release roadmap progress: roadmapping session this Thursday, input sought on the doc, looking for contributor and user insights
    • Trying to take stated priorities and build them out so we know what is needed / scope things / figure out how to know what done looks like
    • Roadmapping session thursday 4-5PM CEST see #sig-release slack for more info
    • Need to send a gcal invite AI: sig release lead help lauri schedule
    • Will focus on first item in doc (moving deb/rpm builds to community infra), first session a few weeks ago, we need to go deeper
    • Reached out to puerco to work on next item on list
    • Few items w/o KEP/specific data and need to clarify and create documentation around goals (especially: cluster api first class signal)
    • [stephen] we have a bunch of tests that are generated for k/k and tests are regenerated but various states of decay, primary tests use cluster dir in k/k and that targets GCP. This isn’t actually a sig release item but many things in k/k have no true owners and the idea is if we can get CAPI provider tests running across multiple environments, can we get better signal for releases. Depends on who wants to own this / do this. Which providers willing to provide compute, etc etc.
    • [Nabarun] is there an issue for this?
    • [Stephen] probably? If there is an issue it should be linked to roadmap
    • [jeremy] we should figure if there is an issue and create on if not (lauri and stephen and nabarun looking)
    • [lauri] also not one for simplify version markers
    • [stephen] that does have issues and i’ll find
  • [Mahamed] Carryover topic from last week’s releng meeting(copied notes over):
    • _[Arnaud] Can we talk about migrating from GCR to Artifact Registry. I spoke to Ben and we might need to use the Artifact registry. _An item from last week’s meeting.
      • [Mahamed] Let's look at this gist briefly before we talk about this topic.
      • [Arnaud] Maybe not enough time and can carry over to the next meeting.
      • **[Mahamed] Make sure to read the gist before the next meeting. **
        • [arnaud] Need to figure out how we push blobs for images to s3
        • [mahamed] this is focused on GAR not s3
        • [arnaud] we should finish the path where we push blob images to s3 still need to finish image process to push to s3
        • **[mahamad] we need to do GAR at the same time? **
        • [arnaud] we can use GCR until we have the image promoter work done
        • [stephen] veering into k8s infra territory. S3 more important, but that’s k8s infra
        • [puerco] last update was offline sync to s3 and OCI proxy would keep track and handle replication async
        • [stephen] let’s come back to this as we’re heading into a different topic
        • [mahamed] two parallel pieces as work, i’m specifically interested in getting things into GAR, i need a few things approved to make progress
        • [puerco] will take a look and we can discuss offline
        • [stephen] can you link or thread off the image so we can discuss there?
        • **[mahamed] parent issue: https://github.com/kubernetes/k8s.io/issues/1343 **
  • [Jeremy] Doodle Update:
    • Not a lot of votes, do we have enough votes to make a change or extend the poll?
    • US Friendly time - 09:00 AM PDT
    • EU - 12:00 PM CEST (UTC+2)
    • [stephen] Keep open for another week from today and see if we can get more votes if we don’t get more we’ll just combine meetings and keep the tine.
    • [nabarun] we need to push the dates out and reopen
  • [stephen] we were discussing s3 blob replication, discussing if it needed to be sync or async
    • First thing we were discussing is we wanted it to be part of image promo process treating s3 as first class
    • Since then, proposal has changed a few times, curious what current state is.
    • [puerco] overview from my perspective:
      • TL;DR - when ben started working on OCI proxy first idea was image promo should bake in promo to buckets and started working. My objection in mega thread whatever we do shouldn’t increase the complexity of release promo process or the time it takes to not put burdon on RM
      • After the thread, one of the things that came up was that possible but not desired to do replication aysnc, handling of replication should fall on sig k8s infra
      • First plan: do replication in image promoter (sig release)
      • Now plan: async and image promoter won’t do this (something from sig k8s infra) and OCI proxy has capability to check if blobs exist and direct traffic as needed
      • [arnaud] that’s basically it. Currently we have production bucket and sync with GCS bucket and OCI proxy uses those production buckets. Caleb working on sync across regions. Next part, checking for each release that we have all the blob replicated. If we can do that async great and we can start to expose through oci proxy.
      • [puerco] caleb and jay doing tests to see how long it takes and if it’s dependable
      • https://github.com/jaypipes/k8s-sigs-promo-tools/commit/cb686e96e81f74c66d08c7142d97ca2fe7b2df56
      • No PR yet but this commit is what Jay is working on
      • [stephen] can we ask Jay to open aPR?
      • [arnaud] already asked
      • [arnaud] now the artifact registry is involved because Ben/Tim talked to GCR and GCR team doesn’t want to add features for OCI proxy and want folks to use GAR instead so move things to GAR now
      • [mahamed] one thing to add is there is a cost implication in october and it would be really nice if all the images come from GAR by then (driven through OCI proxy)
      • AI: we need to see that PR open (ping Jay again), important change and having it as PR is a better form for discussion
      • [arnaud] also in meantime we should help Mahamed work on moving staging repo to GAR and see how it impacts promo process. There are two direct impacts: we lose GCS bucket and we need to make sure we do replication between three new artifact registry
      • [puerco] we dont need the buckets for container images?
      • [stephen] in current state we need them. GCR backed by GCS. eventually we want to be able to delete original GCS buckets that back that but can’t happen till on GAR.
      • [puerco] not important for staging, promo doesnt need access to buckets it just uses registry apis. Some testing stephen did confirms tools work on GAR and we should be find just shifting and promoting from staging GAR to prod GCS for now. We can start testing on a project basis and setup one project and start testing from there.
      • https://github.com/kubernetes-sigs/promo-tools/pull/478
      • [arnaud from chat] please use k8s-staging-experimental
      • [mahamed] PR that does some of that https://github.com/kubernetes-sigs/promo-tools/pull/478
      • [arnaud] i think we first make sure we can migrate staging, then we can do all staging and look at k8s-artifacts-prod. It’s safe to go that way so we don’t have to deal with any production incidents.
      • [stephen] i’m fine cross publishing images but we should not use any of the releng-staging projects for this.
        • Arnaud: use _k8s-staging-experimental _GCP project.

Walk the Board:

June 28, 2022

()

**Host: **Jeremy Rickard (he/him)

Attendees (pronouns):

  • Priyanka Saggu (she/her)
  • Cici Huang(she/her)
  • Raj Gaurav Maurya(he/him)
  • Joseph Sandoval (he/him/él)
  • Cat Chu (she/her)
  • Frederico Muñoz (he/him)
  • Matthias Glastra
  • Mahamed Ali
  • Adolfo García Veytia (he/him/él)
  • Riaan Kleinhans
  • Sascha Grunert (he/him)
  • Heba Elayoty (she/her)

Note Taker:

  • James Laverack (he/him)

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
    • Welcome to Arya
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • [Adolfo] Couple of interesting things. Were going to cut the new release of krel, but it had a nasty regression. Planning to do that soon now it’s been fixed upstream. Will cut v0.4.0, including JSON output and a querying language. Also had a chance to meet with some SPDX maintainers in the open source summit. Adolfo becoming a maintainer of the Go libraries for the SPDX project. Looking for help on this if anyone wants to get involved.
      • [Veronica] Jeremy is cutting the v1.25.0-alpha.2 release today.
      • [RobK] Wanted to sync up with Adolfo offline about the CI Signal tooling.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • [Cici] Enhancements freeze last Thursday. One exception request, which we have approved. It’s already merged. 56 tracked enhancements, out of 76 opted-in. Looking forward to v1.25.0-alpha.2, release notes team is ready.
  • (feel free to add any topics you’d like to discuss as part of the agenda)

**
Open Discussion:**

  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)
  • [Joseph] https://github.com/kubernetes/sig-release/issues/1912
    • [Joseph] Bringing this topic back up. Presently we have both SIG Release and Release Engineering meetings. We end up conflating them. Proposal is to align them and also modify times to be more inclusive of non-US timezones. It’s been sat for about a month, so looking to bring it back up.
    • [Jeremy] We didn’t get a lot of feedback on GitHub. Need to broadcast it more widely. Started drafting an email on HackMD, and also two Doodle polls to try to select new times. One for APAC/EU and one for the US. Of EU people here, are there any times that would be better?
    • [RobK] Remember people south of the equator too.
    • [Riaan] Current time works for South Africa, not so well for New Zealand.
    • [Joseph] I was thinking about how this would work for the release leads on the release team, in particular for timezone diversity.
    • [Sascha] The times that are proposed would work for me. I’d be happy to have more meetings in EU friendly times.
    • [Jeremy] I’ll send out this email and Doodles later. The present slot conflicts with other meetings, such as the OpenSSF one.
    • [Adolfo] The times where Europe and the US overlap are very competitive for meeting times.
    • [Jeremy] Okay, I’ll send my email to SIG Release and dev@kubernetes.io for broad reach. We need to figure out how we’ll determine if this is successful. CALL TO ACTION: If people have ideas, add them to the GitHub issue please.
  • [psaggu] I'm interested in shadowing the Release Branch Manager associate role in the upcoming release cycles, and I'd like to stay for a longer period of time. Is there a formal process for expressing interest, opting in, and moving forward? Thank you so much!
    • I’m currently referring to this document for roles/responsibilities & commitments.
    • And have gone through the release-engineering role-handbooks + have shadowed Arnaud on the recent 1.21.14, 1.22.11, 1.23.8, and 1.24.2 patch release cuts
    • [Priyanka] I wanted to understand if there is a process to show interest in becoming a RBM Associate. What do I need to do?
    • [Verónica] https://kubernetes.io/releases/_print/#release-manager-associates
    • [Adolfo] Right now we’re thinking about better codeifying this process. Until now the process is discressional, in the sense that some of the leads would hear from people who have interest and try to offer those people and opportunity to become part of the release. The problem is that the process hasn’t been established, and and hasn’t been written anywhere. This has lead to some problems, for example the RMAs on the site are not active. This is a trust problem, we’d expected people to offboard themselves if they couldn’t continue to do it. Right now we’re going though some discussions to try to define the process. Looking to provide a soft onramp for contributors (not necessarily) a release team member. The problem is that becoming a RM means sensitive access, so vetting for trust with no gatekeeping and no kingmaking (i.e., doing selection in a non-transparent way). When the time came to pick a release manager for 1.25, the existing RMs are overworked (this is Verónica’s third, I’ve done two, Nabarun has done two). This doesn’t match with the expectation that there are enough of them to do this without overworking. I hope that we can start working on a draft for these processes in the coming weeks, we’ll share them as soon as we start.
    • [Priyanka] Thank you for that answer.
    • [Joseph] We’ve talked about this in the past. How do we do this in a way that’s not a cognitive burden? Verónica had to step up for various releases. Will there be thought to how to onboard people?
    • [Adolfo] Most likely training will be improving the branch manager shadow program. Was discussing with Stephen. For example do we permit non-release team contributors? Do they need extra training? We need to recognise that there is a lot to learn before we can let someone run this. We need to establish a process where the SIG Release community can approve of someone. This needs to be well thought out and put into some process document.
    • [Verónica] Aside from formal process, I’ve noted for the past 6 months that being a branch manager for a specific release (and being an RM/RMA) that sometimes people in different timezones need to collaborate. This is normal in tech. It’s a volunteer task. Sometimes you’re waiting for a person in another timezone. You need to communicate that you can or cannot follow up at specific times. Less of a technical requirement for a RM/RMA, but it’s a difficulty of the role. Requires availability in weird times, which can be a complication of the role. Can you even communicate that in a manual? What would you say? It’s not that you have to be available, but you have to coordinate.
    • [Adolfo] We need more help to avoid overworking people. We need options to hand over instead of always the same person doing a thing.
    • [James] Also trying to do something similar for the Release Team. They don’t need to be the same process, but should be aware of each other.
  • [matthias] Moving File signing forward. https://github.com/kubernetes-sigs/release-sdk/pull/68 for issue https://github.com/kubernetes-sigs/release-sdk/issues/30
    • [Matthias] I noticed that file signing is on the roadmap. There’s a PR on the Release SDK. Can we get some eyes on this for use in krel? Input would be appreciated. Sacha already reviewed it which I’m happy with.
    • [Sascha] More eyes would be good.
    • [Adolfo] I’ll take a look. One thing to note about file signing, and why I’ve left it until last in the signing effort, is that there is still a disconnect between how container images are signed keyless (and the verification) and the way that files are signed. There is an effort going on with a project inside sigstore. It’s the equivalent of curl but it’ll verify that the thing that is pulled has been signed. I think they’ve made progress recently, so maybe we can use that. That aside, the signing API is needed.
    • [Matthias] My current implementation is based on a known public and private key. Is container signing already keyless?
    • [Adolfo] Yes.
    • [Matthias] I understand the disconnect. Is there a key we can use?
    • [Adolfo] No. Same problem with RPMs and DEBs. Keyless would be preferred. Lets sync up.
  • [arnaud] GPG: The great debate
    • [Arnaud] Can I talk about this in RelEng next week July 5th? Signing for system packages.
    • [Jeremy] Sure.
    • [Sascha] +1
    • [Adolfo] Just put the subject on the agenda and try to elaborate as much as you can.
    • [Jeremy] Especially any pre-reading.
  • [Arnaud] Can we talk about migrating from GCR to our own registry. I spoke to Ben and we might need to use Artifact registry. Heads up.
    • [Jeremy] We need to follow up
    • [Mahamed] Who owns the new registry?
    • [Arnaud] K8s Infra. Migration still in early phases. We need to have a full discussion in release engineering.

Walk the Board:

June 14, 2022 ()

**Host: **Jeremy Rickard (he/him)

Attendees (pronouns):

  • Sascha Grunert (he/him)
  • Frederico Muñoz (he/him)
  • Leonard Pahlke (he/him)
  • Lauri Apple (she/her)
  • Cici Huang (she/her)
  • Adolfo García Veytia (he/him/él)
  • Rishit Dagli (he/ him)
  • Stephen Augustus (he/him)
  • Shubham Sah (he/him)
  • Rob Kielty (he/him)

Note Taker:

  • Sascha Grunert

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
    • Welcome to Shubham!
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • Work to parallelize the image promoter is done, waiting for this commit to trickle down to cosign → k/release-sdk → k/promo-tools
        • Should we wait for this PR before cutting the patches?
          • No need to wait for it, but prefer to start the releases until this has been resolved
          • Proposal: Promote each tag individual to not risk a timeout
        • If we fix something in sigstore, do we have to bump our golang deps?
      • Issue with the last patch releases, where for some reason the image promotion timed-out after multiple hours. PR open to run signing in parallel.
      • Please reserve time from the Google build admins for the upcoming patch releases
        • This should be in RM docs if it is not. We should confirm. [it should be]
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • Discussion last week about extending the enhancement freeze
        • Got extended by 1 week
      • PRR deadline moved to this Thursday as well
      • Release team completely onboarded
      • Still expecting the alpha 1 to be cut

**
Open Discussion:**

  • [Lauri] SIG Release Roadmap session results (notes)
    • Need some help with the other roadmap items, shaping the plan
    • To confirm: if we should send out two-question survey asking community to rank currently published SIG Release roadmap items, and why their #1 is their #1 (answer: yes)
      • Getting input from SIG Release first, then consider input from the whole community
    • Consider closing the checklist mentioned in https://github.com/kubernetes/release/issues/913#issue-513253044
    • Stephen started rewriting package building in golang, we need to revisit the story, take ownership and decide on the chosen architecture
    • Additional tasks should be added to the spreadsheet to collect them in one place
    • Arnaud: What are we using for signatures for the system packages?
    • Lauri: Reworking the artifact management EPIC?
      • Keep it as umbrella, consider creating a dedicated EPIC for debs/rpms
      • Stephen and Lauri will work on that in a dedicated session
  • [Lauri] Rel Eng items review (doc) — need help breaking items down and moving things forward
    • Lauri: How to make progress on the doc?
      • Revisit the older issues in the context of the new enhancements
      • Start working on the newest items first
      • Items which are unassigned should find an owner
      • Consider removing frozen items from the project board?
        • Take some time to re-do the project boards
  • [jeremy] Let’s talk about SBOMs
    • Will move to next time
  • [Arnaud] Deb/RPM packages on k8s-infra: https://github.com/kubernetes/release/issues/913
    • We had an agreement about this before but I’m sorry, I can’t remember what we decided about this. Breaking down in 2 practical questions:
      • Who owns the new GPG keys ?
      • How and where to store the GPG keys ?
      • [Stephen] we need to step back and determine if we are doing GPG or if we are owning the packages, should have a follow up
      • AI: Arnaud to schedule a meeting to specifically discuss

Walk the Board:

May 31, 2022 (https://www.youtube.com/watch?v=fU7OsugvqPY)

**Host: **Stephen Augustus (he/him)

Attendees (pronouns):

  • Rob Kielty (he/him)
  • Sascha Grunert (he/him)
  • Leonard Pahlke (he/him)
  • Rey Lejano (he/him)
  • Mike Lieberman (he/him)
  • Hossein Salahi (he/him)
  • Mahamed Ali (he/him)
  • Eddie Zaneski (he/him)
  • Atharva Shinde (he/him)
  • Devan Goodwin (he/him)
  • Verónica López (she/her)
  • Rishit Dagli (he/ him)

Note Taker:

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

**
Open Discussion:**

  • [Stephen] Timezone-skewed meetings: https://github.com/kubernetes/sig-release/issues/1912
    • Issue is open to comment to the community for the next week. Updates will be sent after the comment period closes. Expect updated calendar invites after decision is made.
  • [Rob] Update and check in to automate the running of Leo’s report that lists out the active flakes that are being tracked by the CI Signal Team over the course of a release cycle.
    • We have automation choices we can make. I would welcome review and input on my last update on the issue where we are tracking this work. ci-signal-reporter, automatically generate weekly ci signal report #2443
    • Feedback being solicited from SIG-Release regarding Prowjob vs Github Action in the above issue. Also looking for additional feedback on the CI Signal automated reports.
    • [Stephen] Completely fine with GH Actions. What permissions does this token require?
    • [Rob] Read only. (For this iteration of just running the report)
    • [Leo] Nothing special is needed with permissions.
    • [Stephen] Default action for the automated report should be writing to a repo vs Slack.
    • [Rob] The analytical steps after we have this report rolled out will need some design thinking to determine if implementation effort on this is redundant. (Added context post meeting by RK: lots of data is captured automatically; kettle from sig-testing collects data on the CI job level and stores that data in a BigTable db. Leo’s report captures CI Signal Team Activity. The aspiration is that data on the CI Signal Team activity will be useful for both the Reliability WG and the project at large. When we get to the stage where we can track trends we can discuss next steps within sig-release and share that out with the wider community)
    • [Stephen] The GH action should be identical to how you run locally. Tool should be packaged as a container image.
    • [Muhamed] “you can get GitHub Actions stuff to interact with Google Cloud to persist stuff there”
  • [Leo] Followup from 1.24 retros: add Inclusive Speaker Orientation (LFC101) course to every RT sub-team lead preparation
    • [Stephen] Agrees with this. It lays the foundation for future leads.
    • [Leo] Action item to update handbook.
  • [Mike Lieberman] Representing CNCF Security TAG - Supply chain security for kubernetes
    • CNCF Proposal - https://github.com/cncf/tag-security/issues/890
    • [Stephen] We need to decide who is doing what with the investigation? SIG-Release has a roadmap of Software Supply Chain Security roadmap. Moving to distro-less images was the first step. There are several images that still use Debian base. The current KEPs in motion with SSCP and map them to the CNCF Tag Security proposal would be ideal. SIG-Release are early adopters. From external perspective where do people see the gaps.
    • [Mike] All these efforts provide a better picture of Supply Chain Security picture. If there is implementation challenges the feedback from SIG-Release would be helpful.
    • [Stephen] When approaching SLSA we realized that level 4 was off the table.
    • [Mike] SLSA 3 is going to change a bit as we move to 1.0. How should I collaborate?
    • [Stephen] Come to the meetings and Slack. Release-management channel is where to find the team.
  • [Verónica] Branch Manager for 1.25. Cut 1st alpha. Improvements on release process times (promotion timeouts).
    • [Stephen] Working on branch manager recruiting. There is a schedule up already for 1.25 release.
    • [Verónica] Leo wants to shadow.

Walk the Board:

May 17, 2022: Canceled for KubeCon

May 3, 2022: Canceled for v1.24 release day

April 19, 2022 ()

**Host: **Sascha Grunert

Attendees (pronouns):

  • Sascha Grunert (he/him)
  • Rey Lejano (he/him)
  • Marko Mudrinić (he/him)
  • Leonard Pahlke (he/him)
  • Nabarun Pal (he/him)
  • Adolfo García Veytia (he/him/él)

Note Taker:

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • New Image Promoter are signed already with the last release cuts. Some work needed to improve speed. There is a need to rethink the copying of the layers of container images. This will affect the order of the steps. A new release has images staged the the promoter comes in and signs then propagates out. New approach would be to mirror before promoting them so signature needs to be in place. (Stephen) Define what done is for signing. After that we can consider the mirroring. What happens with signatures left behind or redone? Promoter is consistently running. (Adolfo) Promotion process takes all the blobs from the layers and into S3. We would have to add the signature to the blobs. The proxy that sends the traffic to AWS has no idea whats in the mirror just route traffic to an IP address. (Arnaud) The proxy is not a cache. Any request coming from docker client will be identified by location. Anyone pulling from AWS can get the images before they hit GCS. (Stephen) AWS will be a first class citizen for signed images. Whats the expectations for 1.25. (Arnaud) registry.k8s.io will be embedded in the k/k repo. (Stephen) K/promo images points to registry.k8s.io. Whats the backout plan if things don’t work as expected? (Adolfo) The plan to modify this is being guided by SIG-K8s-Infra. (Stephen) Functional process might not change but the UX might change. How do we revert without changing the references? (Arnaud) Revert is redeploy OCI process which can take 3-5 minutes. (Sascha) Do we want to recreate something like the mirroring project in the KEP? (Stephen) We should close out the KEP and the goals are separate. Do we have a merge KEP for this redirect? (Arnaud) No KEP but a proposal (Stephen) This needs to be a KEP for the community. It touches everyone. (Stephen) It’s infra and it needs to be signed by a few different SIGS that are affected. This proposal is KEP ready. SIG-K8s-Infra will be the owner of creating the KEP.
      • Patch releases (Nabarun) Cutting release candidates today for 1.24.0. Expect some hiccups. Run the rotator once and if an issue reach out to Stephen. Verónica is available to push the release along with Jim Angel. Google build admin support will be needed.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
    • (James) Release delayed by two weeks. Plan is to cut RC.1 next week. Go 1.18.1 fix is in progress. KREL has some issues they are working through. 1.25 team is in place. (Arnaud) Some of the KREL jobs are successful but with move to community infra there is issues with SSH. (James) 1.25 release outline

**
Open Discussion [timebox to 5 min]**

  • (Joseph) Kubecon Valencia Contributor Summit - Is SIG-Release planning to participate? (Stephen) I’ll be in the house. Big room and bunch of tables in the past. (Arnaud) No structure to the initial program. The rest is hallway track. (Arnaud) There is meeting later today to discuss this.

Walk the Board [timebox to 20 min]:

April 5, 2022 (recording)

**Host: **Jeremy Rickard

Attendees (pronouns):

  • Joseph Sandoval (he/him)
  • Leonard Pahlke (he/him)
  • Nate Waddington (he/him)
  • Heba Elayoty (she/her)
  • Lucas Dwyer (he/him)
  • Chris Negus (he/him)
  • Arnaud Meukam(he/him)
  • Eddie Zaneski (he/him)
  • Rey Lejano (he/him)
  • Verónica López (she/her)

**Note Taker: **Joseph Sandoval

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • Confirmation new image signing code is running as expected. RC candidate depending on timing is that the identity might not be the service account expected. Need a new release of co-sign.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • Code freeze passed. Test freeze today.
      • Big news is 1.24.0-rc.0 delayed until Monday because of a Go bug. If that fix doesn’t land in Go 1.18.1 (due Thursday) then we’ll likely have to delay 1.24. We can’t release without the fix and we can’t downgrade to Go 1.17. The fix is already in Go’s main branch and a cherry-pick is open to Go’s 1.18 release branch.
      • (James) Code freeze has passed. RC.0 cut is delayed. Blocking issue with Golang 1.18. A few exception requests are still pending. 6 PR’s are pending and once those are merged we can focus on CI.(Adolfo) Go 1.18 has been rejecting CSR with sha1. https://go-review.googlesource.com/c/go/+/394294 is the fix. Next step is delay or no delay.

Open Discussion [timebox to 5 min]

  • [jeremy] Followup from 1.24 Mid Cycle Retro
    • Add clarity on when is the best time for an enhancement to be a major theme and guidelines around what should be a major theme
  • [leo] CI Signal project Board or view for managing team reportings (see slack discussion). . Can we remove this magic file from which is handed down from lead to lead? Propose moving this to Github.
  • [leo] CI Signal team kind of misses power to have the things they found being addresses (for example this flake 107530, from KEP 3139) (Leo) The team continually pings regarding status. Is there something we can do more than just ask for to get flakes fixed. (Eddie) Have you dm the leads of the SIG’s? Leads have alot and lose the issues easily. DM direct is the recommendation. (James) The release team feels like they have no power. (Adolfo) We do have the power to delay a release if a test is flaking. (Leo) Perhaps using a priority label or escalation path to raise awareness. (Veronica) This is not a problem that can be solved with more processes. I’ve been on the other side of this issue. I don’t get offended when issues are escalated. (Adolfo) **AI - Raise this at the community call. **(Eddie) Update the leads contact sheet as this was useful.
  • [leo] If reliability monitoring is added to the CI signal responsibilities (see KEP 3139 discussion), something more is needed than just changing the description in the handbook. I think the team will not deliver “right out of the box” without making some adjustments to the team**. **Maybe two leads / emeritus or involvement of sig-release leads or similar measures to build a stronger team to handle the additional responsibility.
    • (Adolfo) What sort of effort is involved with monitoring? AI - Keep it under discussion.
  • (Arnaud/Adolfo) K8s images are pushed to GCE which copies to AWS. https://github.com/kubernetes-sigs/promo-tools/issues/533 Issue opened by Jay Pipes. (Arnaud) I can act as the liaison between K8s-engineering and AWS.

Walk the Board [timebox to 20 min]:

Kubernetes issues** **

Mar 22, 2022 (recording)

**Host: **Adolfo García Veytia

Attendees (pronouns):

  • Rey Lejano (he/him)
  • James Laverack (he/him)
  • Taylor Dolezal (he/him)
  • Leonard Pahlke (he/him)
  • Eddie Zaneski (he/him)
  • Batuhan Apaydın (he/him)
  • Wojtek Tyczynski (he/him)
  • Veronica Lopez (she/her)

Note Taker:

  • Rey Lejano

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • [Adolfo] Two main lines of work last week: work on bumping images to Go 1.18 and PR to sign images in image promoter is open but did not have enough contributors to get new version of image promoter promoted. The last patch releases could not be signed, hopefully we have tests for the first 1.24 beta to verify signatures are working
    • [James] Release Team (https://github.com/orgs/kubernetes/projects/29)
      • 28 Days until release. Exactly four weeks.
      • Code Freeze coming in one week. 01:00 UTC Wednesday 30th March 2022 / 02:00 BST Wednesday 30th March 2022 / 18:00 PDT Tuesday 29th March 2022
      • We’re going to ship the “code freeze is coming” email today
      • Mid-cycle retro tomorrow 23rd March at 17:00 UTC. https://bit.ly/k8s124-retro
      • Nabarun cut 1.24.0-alpha.4 today.
      • We’re starting to think about succession for 1.25.
      • [James] In week 11, Code Freeze is in 1 week. Email to k-dev, Code Freeze is coming email. Mid-cycle retro is tomorrow. A release-blocker was fixed just in time - shoutout to Jordan, Eddie, Jim on fixing the release-blocker issue
  • (feel free to add any topics you’d like to discuss as part of the agenda)

Open Discussion [timebox to 5 min]

  • [leonardpahlke] Add a CI Signal onboarding handbook to the handbook folder (see issue #1857)
    • It's hard to get a clear picture of what CI Signal is doing right now without reading more than 20 pages of.
      • More videos like the flake finding videos
      • [Eddie] Leads and certain people have lots of tacit knowledge that should be transferred
      • [Adolfo] The Flake Finders video was meant to be a recurring show. Adolfo’s old videos (even though they are no longer current) are still be being watched by new Release Team members. Shows how people like to consume information.
      • [Taylor] Folks have pitched similar options like a slide deck in release retros
      • [Rey] There are slow weeks for each RT sub team, it might be a good time for new RT members to point out the gaps from the handbook to onboard and experience RT members can create an onboarding slide deck
  • [leonardpahlke / wojtek-t / david / devan] KEP-3138 Increase the Reliability Bar proposal
    • Discussion pointed to sig-release to track reliability, report issues, and to poke the responsible sig's. This should be done by the CI signal team - thoughts?
      • CI Signal Team likely needs support / a different team structure to handle the additional responsibility
      • CI Signal tasks needs to get automated -> tools currently under development
    • Sippy to track reliability (hosted on openshift.org)
  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)

Walk the Board [timebox to 20 min]:

Feb 22, 2022 (canceled)

Feb 8, 2022 (recording)

Host: Sascha Grunert

Attendees (pronouns):

  • Sascha Grunert (he/him)
  • James Laverack (he/him)
  • Joseph Sandoval (he/him/él)
  • Marko Mudrinić (he/him)
  • Nate Waddington (he/him)
  • Rey Lejano (he/him)
  • Leonard Pahlke (he/him)
  • Lucas Dwyer (he/him)
  • Debabrata (he/him)
  • Dipto Chakrabarty (he/him)
  • Eddie Zaneski (he/him)
  • Verónica López (she/her)
  • Dishant Sethi (he/him)
  • Adolfo García Veytia (he/him/él)

Note Taker:

  • Joseph Sandoval

Recurring Topics [timebox to 20 min]:

  • Welcome any new members or attendees
    • Jake Plimack/Tesla.
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30
      • KEP-3031 - (Sascha) how to build releases and how to use keyless signing. Needs to be documented on how to verify signatures. It’s partially done.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • (James) Week five! 70 days left. One third of the way through.
      • Enhancements freeze completed, after two exceptions we are tracking 66 enhancements for v1.24.
      • Done shadow onboarding.
      • Gonna finalise time later this week for the mid-cycle retro on 23rd March.

**
Open Discussion [timebox to 15 min]**

  • [leonardpahlke] Release Team Shadow applicant analysis tool (see slack thread in contribex, applicant analysis repository)
    • I got overall positive feedback for the initiative. From the discussion I get that we aim for 1.25 to share some infos about the applicants. How to proceed from here? (previous release-team meeting discussion thread). Next steps on how to move forward?
      • Cleanup and donate the repo or
      • use it just as a reference and build something new.
      • (Sascha) do we have a process in mind on how to adopt? (Leonard) Create form for K/K branch rename each release cycle with the data analyzed. (Sascha) We could move forward with using the tool.
      • (Adolfo) Image Promoter: Refactoring the code to provide a reworking of the API service. The reason for this is the community is running low on funds so mirroring the images to major cloud providers. TBD is how to test locally. It should be functionally and if anyone has some cycles to review and can provide some comments on how to improve. (Sascha) This can be complete before the end of release cycle. GCP permission discussion could be potential blocker. How much effort will be needed? (Adolfo) WE should only need to provide the copying logic. (Eddie) AWS registry is stood up as POC. Stress testing is needed. (Docker pull ​​registry-sandbox.k8s.io/pause:3.1)
    • (Dishant) CLA signing as individual is not enabled but corporate works. Advised to reach out to SIG-Contribex. The migration to EasyCLA was recently introduced.

Walk the Board [timebox to 10 min]:

January 25, 2022 (recording)

Host: Stephen Augustus

Attendees (pronouns):

  • Stephen Augustus (he/him)
  • James Laverack (he/him)
  • Rey Lejano (he/him)
  • Leonard Pahlke (he/him)
  • Joseph Sandoval (he/him/é)l
  • Nate Waddington (he/him)
  • Sascha Grunert (he/him)
  • Eddie Zaneski (he/him)

Note Taker:

  • Joseph Sandoval

Topics [timebox to 20 min]:

  • Welcome any new members or attendees *
  • Subproject updates
  • Emergency Release 1.23.3 to be cut today
  • Discuss out-of-band releases: https://github.com/kubernetes/sig-release/issues/1832
  • SPDX DocFest coming up this Thursday (hit @puerco if you want to attend) K8s SBOM tool is starting to be used by other teams. The event will have teams compare the outputs. Check if this is going in the right direction. (Stephen) We should add these to the calendar in the future.
  • (James) 1.24 Release is going well with about 70 enhancements. How do we feel about the open communication in #SIG-Release channel. (Stephen) It’s great having all the communication in the open.

Walk the Board [timebox to 20 min]:

**
Open Discussion [timebox to 5 min]**

  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)
  • (Leo) https://github.com/kubernetes/sig-release/pull/1825 Update to the handbook waiting for comment. https://github.com/kubernetes/k8s.io/issues/3071 (Stephen) This would be managed by release managers. File a donation request.
  • (Eddie) CDN egress spend to other clouds. (Stephen) Is there anything we can do in the interim to ease the spend? (Eddie) Correction container registry proxy is what Ben the Elder is working on.
  • (Sascha) Implementation phase of container signing has help wanted issues. (Stephen) There could be a security concern. There are two tokens in the GCE run. Walkthrough of GCP jobs and ownership.

January 11, 2022 (recording)

**Host: **Jeremy Rickard (he/him)

Attendees (pronouns):

  • Leonard Pahlke (he/him)
  • James Laverack
  • Omar Elbanby
  • Jesse Butler (he/him)
  • Joseph Sandoval (he/him)
  • Nivedita Prasad (she/her)
  • Purneswar Prasad (he/him)
  • Eddie Zaneski (he/him)
  • Nate Waddington (he/him)
  • Debabrata (he/him)
  • Meha Bhalodiya (she/her)
  • Adolfo García Veytia (he/him)

Note Taker:

  • Joseph Sandoval

Topics [timebox to 20 min]:

  • Welcome any new members or attendees: Omar - works in DevOps. Nivedita - CI Signal Shadow Nate W - 1.24 Docs lead - Ahsan Khan looking to contribute - Purneswar - Student - Kep Reading Club - Akinpelu D - DevOps engineer.
  • Subproject updates
    • Release Engineering (https://github.com/orgs/kubernetes/projects/30)
      • (Adolfo) Merging KEPs for signing artifacts for release and SLSA 3 compliance. Projects will be updated at the next release meeting. (Jeremy) Next week is the sub-project meeting. (Purneswar) What does the release engineering team do and what is an artifact? (Adolfo) Provides overview of Release team.
    • Release Team (https://github.com/orgs/kubernetes/projects/29)
      • 1.24 release cycle begins! Full release team and onboarding tasks in progress. Meeting tomorrow 10am PST. Enhancements are now open. First deadline is the Enhancements freeze in the first week of Feb. Very limited spots available.
  • (feel free to add any topics you’d like to discuss as part of the agenda)

Walk the Board [timebox to 20 min]:

**
Open Discussion [timebox to 5 min]**

  • (feel free to add any topics you’d like to discuss, even when they came up during the meeting)
  • [arnaud] Migrate away from gs://kubernetes-release: https://github.com/kubernetes/k8s.io/issues/2396
    • Problem is network traffic and it's huge. Costs are a problem. New bucket with the CDN provider is the request. The bucket contains all the binaries from all the releases. Not sure if we have the budget for this request? Ask Steering to request a plan with CDN. (Jeremy) We need a +1 from SIG-Release. When do you plan to open this? (Arnaud) This week. Discuss this in release engineering channel.
    • Request an open source plan for a CDN Provider.
  • [leonardpahlke] CI Signal Monitoring platform - ownership. See github issue and diagram / overview
    • (Leonard) Who is the owner of the platform and project? How do we decide this? (Jeremy) CI Signal has a rotation so no ongoing owner. This would involve running a platform? Do we want to request infra? (Arnaud) We can provide infra. Who would we reach out to if this goes down? (Adolfo) Maybe the CI-Signal team to provide support. Take discussion to SIG-release channel.
  • [leonardpahlke] New tool, summarize and analyze release team shadow applicants (repo)
      1. Analyze all applicants by creating charts
      • Charts: applicants by team, pronouns, company / affiliation, timezone…
      • Charts have been generated but are currently not shared as it might be confidential (needs to be clarified first) Could we use these reports and charts going forward? Maybe advance this beyond these initial reports to help drive the selection process. (James) Concern about the data we collect from survey but do we have permission. Not sure if we have asked permission. Might need SIG-Contribex input or possibly CNCF. (James) Can we provide an anonymized version of the data and make sure we have the permission. We need to move this up to Steering or SIG-Contribex. (Omar) Applied for release shadow. It was challenging to answer some of the survey questions. (Eddie) Paris was asking for ideas. Put some thought into KEP tracking but maybe we can have this be funded.
      1. Summarize applicants by generating markdown files for sig-release sub-teams
    • Goals:
      • support the release team in the shadow selection (2.)
      • current state of the release team shadow program (1.)