diff --git a/sig-release/meeting-notes-archive/2023.md b/sig-release/meeting-notes-archive/2023.md new file mode 100644 index 000000000..8d707dcb0 --- /dev/null +++ b/sig-release/meeting-notes-archive/2023.md @@ -0,0 +1,6068 @@ +# SIG Release — Meeting Archive — 2023 + +## Dec 19, 2023 + +**Host (pronouns):** Marko Mudrinić (he/him) + +**Attendees (pronouns):** + +- Rudraksh Karpe (he/him) + +- Josh Berkus (he) + +- Megan Wolf (she/her) + +- Sreeram Venkitesh (he/him) + +- Siva (he/him) + +- Oluebube Princess Egbuna(She/her) + +- Joseph Sandoval (he/him) + +- Daniel Akinpelu(he/him) + +**Note Taker (pronouns):** + +- Joseph Sandoval + +**Recurring Topics \[timebox to 20 min\]:** + +**NO RECORDING THIS MEETING (no host key)** + +- Welcome any new members or attendees + + - + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: + +(Marko)Scheduled to be out today. + +- Action Items/Help Wanted: + + + +- [OBS + Packages](https://github.com/orgs/kubernetes/projects/137/views/1): + + - General Update: No major updates + + - Action Items/Help Wanted: + + - Approval needed for the obscli repository request: + [https://github.com/kubernetes/org/issues/4550](https://github.com/kubernetes/org/issues/4550) + +- Supply Chain Security: + + - General Update: + +> (Marko) Nothing new. + +- Action Items/Help Wanted: + + + +- [Artifact + Validation](https://github.com/orgs/kubernetes/projects/171): + + - General Update: + +> (Marko) Lauri Apple is running some great sessions. There are issues +> on the board if you are interested in picking something up. + +- Action Items/Help Wanted: + + + +- Release Team + +- (Marko) Release cycle hasn’t started yet. + + + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- \[Sreeram\] How are the [kepctl and + kepify](https://github.com/kubernetes/enhancements/tree/master/cmd) + tools in k/enhancements currently being used by SIG Release? + Possibility to merge code from the + [kept](https://github.com/salaxander/kept) CLI written by + Xander and folks from previous release teams. + + - (Marko) I don’t have any ideas about this. + + - (Joseph) What is the benefit if we do. + + - (Sreeram) There might be an opportunity to improve and get benefits + from merging the tools. It will make it easier for the enhancement + team. + + - (Joseph) The release team has generally drive improvements to the + their tooling. Under the governance of the Sig-release team managers + and leads. + + - (Sreeram) Go to all the three different tools and outline the + benefits we get from doing this. + + - (Joseph) Bring this discussion to the SIG-Release channel when you + have pulled the Enhancements tooling improvements. + +## Dec 12, 2023 + +**Host (pronouns):** + +- Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Megan Wolf (she/her) + +- Chris Hanson (he/him) + +- Rudraksh Karpe (he/him) + +- Jim Angel (he/him) + +- Marvin Beckers (he/him) + +- Lauri Apple (she) + +- Sreeram Venkitesh (he/him) + +- Sascha Grunert (he/him) + +**Note Taker (pronouns):** + +- Marko Mudrinić (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Marvin Beckers - software engineer/team lead @ Kubermatic, want to + learn more about upstream release engineering + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: + + - Patch releases are scheduled for the next week, Tuesday, + December 19 + + - Cherry-pick deadline is Friday, December 15 + + - Go updates are almost done, we only need to update Go version + in the publishing-bot rules, Marko will take care of that + today + + - This patch releases series will not include v1.29.1 + + - v1.29.1 is scheduled for January + + - OBS Packages: + + - General Update: + + - Still focused on legacy packages, no major updates for OBS + + - Nitish will do a presentation on the LFX project in January + + - Supply Chain Security: + + - General Update + + - [Cloud Native Security + Slam](https://community.cncf.io/events/details/cncf-cloud-native-security-slam-presents-2023-security-slam-kubernetes-lightning-round/) + coming this Friday + + - Planning contributor tasks + + - CLOMonitor: + [https://clomonitor.io/](https://clomonitor.io/) + + - We want to get the Kubernetes project there and improve + the score as much as possible + + - This is one of tasks that we want to tackle + + - There’ll be a webinar where you can ask questions and where + we’re going to discuss bigger tasks + + - Still low signs from the Kubernetes side, we only heard + about one or two SIGs + + - Even if the project is small and has one image or binary, + there’s work for sure + + - Sign up for individual contributors is closed, reach to + Adolfo if you want to be added to the list + + - Action Items/Help Wanted: + + - Help wanted guiding new contributors + + - Release Team + + - Priyanka: + + - Mock stage & release runs for 1.29.0 completed (thanks to Sascha + & Release Engineering) + + - Nomocks: when do we plan to start the process tomorrow? + + - We can start tomorrow in the morning European time. Last + time we had problems with packages taking 12+ hours to + build, so we want to decrease the risk of having to delay + the release if something like that happens again + + - Do we have to worry about embargo? No, we generally inform + the CNCF of the release date, we didn’t mention any concrete + release time + + - When do we plan to start the no-mock stage & release runs + tomorrow (Dec 13) PT time? I’ll plan the email comms + accordingly. Thanks! + + - \[OT - more of a prod-readiness question, question for one of + the media interviews\] + + - Did we make any key improvements this cycle w.r.t. Reliability + & security (prod-readiness)? + + - Efforts to expand the team to make the process more robust + with more capacity + + - Jeremy to give number of shadows to Priyanka + + - 1.29 Retro 2 meeting planned for 15 min after this call (8:30 AM + PT) + + - Need to ready the PR for lifting code freeze + + - RelEng folks to help with that + + - 1.30 shadow applications are open, the form is already out + + - [https://forms.gle/psoz3EaYDFt2CBjS6](https://forms.gle/psoz3EaYDFt2CBjS6) + + - [https://groups.google.com/g/kubernetes-sig-release/c/aHtIDQoLoDk/m/z20TnLAQAwAJ](https://groups.google.com/g/kubernetes-sig-release/c/aHtIDQoLoDk/m/z20TnLAQAwAJ) + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- \[mahamed\] kubekins-e2e v2 image + [https://github.com/kubernetes/test-infra/pull/31421](https://github.com/kubernetes/test-infra/pull/31421) + + - Main image that we use for e2e testing in Prow, it’s kind of bloated + and has a lot of dependencies that we don’t need + + - Main reason for this image is that etcd is looking for tests on ARM, + the old image doesn’t support arm, so we want to take this + opportunity to build a new image that supports arm and that doesn’t + have as much as cruft as the old one + + - Adolfo to take a look after the meeting + +- \[Marko\] Should we cancel meetings for the week of Dec 25 and Jan 1? + + - Jeremy will cancel these + +- \[Lauri\] Next conversation about artifact validation work being + scheduled: + [https://kubernetes.slack.com/archives/C2C40FMNF/p1702310201513499](https://kubernetes.slack.com/archives/C2C40FMNF/p1702310201513499) + + - Goal: Break down project board items to help people get started + + - Issues and project board for "Make Artifact Validation More Robust" + now here: + [https://github.com/orgs/kubernetes/projects/171/views/1](https://github.com/orgs/kubernetes/projects/171/views/1) + + - Issues organized in order based on dependencies. + + - Relevant notes now transferred from the Miro board. + + - Seeking collaborators on the end user item – need someone with a bit + more insight into CNCF operations, what useful resources they might + have+lend + + - \[Lauri\] The next session is being planned, we’ll review the action + items that have been created, we’ll go through the board and clean + goals, see if anyone want to pick any item + + - \[Lauri\] Milestones and tasks will be identified, as well as + contact points for those + + - \[Lauri\] Trying to figure out what’s expected by Kubernetes users, + which groups might be the best to hit for surveys to reach as many + Kubernetes users as possible + + - \[Jeremy\] WG LTS has a survey ongoing, there are not many ways to + reach out to users, the best is to share on social medias and + internally in your groups (can help if you work for large + employer) + + - \[Arnaud\] CNCF End users group is an option as well: + [https://www.cncf.io/enduser/](https://www.cncf.io/enduser/) + +- \[Jeremy\] Danny Brito from my team at MSFT will be coming to SIG + Release in the future and is currently working on + [https://github.com/kubernetes/test-infra/issues/29390](https://github.com/kubernetes/test-infra/issues/29390) + (working on generate_tests.py ⇒ go right now) + +- \[Jeremy\] Project Copacetic demo in the New Year + +- \[Adolfo\] Cross-SIG effort to improve the security feed, Adolfo + preparing a demo for the next SIG Security Tooling meeting (Wednesday, + December 20) + +- \[Rudraksh\] Question about applying for CI Signal + + - \[Priyanka\] There’s a new team called Release Signal as of this + release cycle (merge of Bug Triage and CI Signal), there’ll be no + shadow selection for this team this release cycle, but as of the + next one + + - \[Priyanka\] You need to be familiar with Prow, there are some great + resources and talks about it, you can use that as the starting point + + - \[Priyanka\] All other roles are taking applicants this release + cycle + +## Dec 5, 2023 + +**Host (pronouns):** Marko Mudrinić (he/him) + +**Attendees (pronouns):** + +- @sftim (he/him/they/them) + +- Sreeram Venkitesh (he/him) + +- Rudraksh Karpe (he/him) + +- Adolfo García Veytia (he/him) + +- Jim Angel (he/him) + +- Faeka Ansari (she/her) + +- Nitish Kumar (he/him) + +- Megan Wolf (she/her) + +- Mario Fahlandt (he/him) + +- Sandipan Panda + +- Mahamed Ali + +- Benjamin Elder + +- Josh Berkus (he) + +- Lauri Apple (she) + +- Divya Soundararajan (She/Her) + +**Note Taker (pronouns):** + +- Joseph Sandoval + anyone else who wants to edit + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Megan Wolf - here to learn and maybe get involved + + - Divya Soundararajan - has visited various sigs. Freelancer. + + - Rudraksh Karpe - just submitted his first PR. Here to learn. + + - Mario Fahlandt - involved with K8s SIG-Infra + + - Sandipan Panda - Involved in SIG Contribex, here to learn and + contribute + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: + + - Go security updates has delayed the patch releases. We moved + the schedule back by a week. 12/19 is the updated release + date. + + - Action Items/Help Wanted: + + - Keep an eye on incoming PRs. No help needed atm but keep an + eye on the release channel. + + - OBS Packages: + + - General Update: + + - No updates. + + - Action Items/Help Wanted: + + - (also see infrastructure topic later in meeting) + + - Supply Chain Security: + + - General Update: + + - CNCF [Cloud Native Security + Slam](https://community.cncf.io/cloud-native-security-slam/) + postponed to Fri Dec 15th + + - Help still wanted + + - Event page will be updated today with the new date + + - [Original announcement + k/dev](https://groups.google.com/a/kubernetes.io/g/dev/c/SbSrJckE1iI) + + - (Adolfo) Proposal for new SBOM Design Starts today + + - Action Items/Help Wanted: + + - LFX project: + + - General Update: + + - (Marko) We’ll have a demo from our SIG-Release intern. + + - Release Team + + - Post Dec 5, moving all the 1.29 Release Burndown updates to async + slack thread in \#sig-release + + - (Jim Angel) + + - (Already discussed in [slack + thread](https://kubernetes.slack.com/archives/C2C40FMNF/p1701372168588369)) + + - Release date postponed to Dec 13, 2023 + + - 1.29.0-rc.2 scheduled for Dec 7, 2023 + + - Retro 2 will be moved to next week (Xander to send out updated + invite) + +- \[Lauri\] update on release roadmapping + + - We’ve made progress: + [https://miro.com/app/board/uXjVMaKPi9c=/](https://miro.com/app/board/uXjVMaKPi9c=/) + + - Recordings: + + - Dec 5: + [https://www.youtube.com/watch?v=kif-gN8f5Qw](https://www.youtube.com/watch?v=kif-gN8f5Qw) + + - Nov 29: + [https://www.youtube.com/watch?v=kgXsKsYQXfE](https://www.youtube.com/watch?v=kgXsKsYQXfE) + + - We need to self-organize to tackle the spikes and evaluations first, + then be in a better spot for decision-making. + + - Will be making some issues and a project board + + - User research / outreach opportunities for folks interested + + - We still need to connect work being discussed (largely + image-promoter reworking) to SLSA themes. The workflow was laid out + with identifying pain points. Discussion around making it more + modular components. We have a lot to learn before we can start the + work. This week we are collecting data on how long a release is + taking. On the right side of the Miro board you can review the + decisions that are needed to be made. + + - Want to focus on “robust artifact validation” before pursuing other + efforts + + - To help newcomers we’ll match a SIG Release vet to co-work on a + spike. + + - We didn’t get to the SLSA requirements. + + - + +- Legacy / current Linux package repositories & infrastructure to + support them (@sftim) + + - (Marko) We have a new official package repository. This hosts K8s + releases starting with 1.24.0. We only publish packages to this + repository. We announced deprecation of legacy repository. (per the + August announcement ) Legacy packages will be removed sometime in + the future without announcement. Do we want to make snapshots + publicly available? + + - (Tim B) I would like to hear if there are any questions. + + - (Mario) What would the release team from a user perspective want to + see? If we removed packages from the registry. Users traffic will + eventually migrate. + + - (Adolfo) Anytime we make a change it usually breaks someone. So + being transparent is better for the users. I would prefer to not + have any breaking changes to users. + + - (Marko) We can’t serve packages forever. Put in a concrete deadline. + + - (Tim) In the past it was common to get packages from various + sources. We don’t have huge revenue. How do we feel about getting + users to download from mirrors? + + - (Marko) That was one of the reasons to use OBS. The bandwidth given + is limited. + + - (Ben) The former team inherited this. We have been negotiating to + keep this offering in place. It was a mistake to publish this but we + have to get off of this. This is not a surprise. The packages are + going away in the new year. There is not enough bandwidth. We have + not had a host for these packages which is now going away. We as a + project setup OBS. Older packages are what is in play. Our budget is + tight. We are burning about 3mil. We negotiated with Fastly. We + don’t have the resources available and need to discuss how we host + the OBS packages. We are not prepared to redirect the old ones. We + need to message that this is going to break. K8s docs are pointing + to Google repositories. + + - (Mahamed) + + - (Ben) Burning money at the beginning of the year is not a good way + to start with our budget. We need to get below with our run rate + + - If there is time we could tapo new alliances as Jeefy suggested. + + - Marko, can we come up with a number? A threshold where we could pull + the plug if we hit it + + - Ben: We need to know bandwidth, we have other resources such as + equinix. We have fastly but we need to talk to them with bandwidth + numbers. How much bandwidth are we reasonably talking about. We need + to observe data over time. + + - Tim: How does sig release if we break old packages and put a + friendly webpage instead + + - Marko, this is the only choice if we cannot serve, and put a how to + upgrade your cluster instead. + + - Puerco: Do we have time to observe data? + + - Marko: we can set it up quickly + + - Mario: What if we lower costs by running some expensive jobs less + often and use that to bridge until we get a deal with Fastly, etc + + - Marko Id like to see that + + - Ben: We don’t have full transparency to the full AWS bill yet + + - \[missed marios comment\] + + - Mahamed: We need to still allocate funds for the things we want to + pay. + + - Ben: We have been very reactive and we hope to be more proactive + next year, but we cannor make any guarantees on how effective that’s + going to be. + +- (Rudraksh) Looking for information on SBOMs. + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- ~~\[Nitish\] LFX project presentation (15 minutes)~~ (postponed for + next week) + +- + +## 28 Nov 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Sascha Grunert (he/him) + +- Leonard Pahlke (he/him) + +- Mah + +- Sreeram Venkitesh (he/him) + +- Stephen Augustus (he/him) + +- Drew Hagen (he/him) + +- Adolfo García Veytia (he/him) + +- Sujay Dey (he/him) + +- Arnaud Meukam (he/him) + +**Note Taker (pronouns):** + +- Sascha Grunert (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Drew works as shadow on the docs subteam. Welcome! + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: + + - Patches will be delayed by one week: + [https://github.com/kubernetes/website/pull/44107](https://github.com/kubernetes/website/pull/44107) + + - Jobs updates for release-1.29 were clunky, working on + improving the tooling around that + + - Publishing bot is broken since 1.29.0-rc.0, fixing everything + before the final release next week + + - Action Items/Help Wanted: + + - If anyone is interested in reworking the job generator for new + release branches, feel free to reach out to \#sig-release on + slack. + + - Follow up on publishing bot after, technically owned by + release engineering but not good ownership/maintainership + today + + - OBS Packages: + + - General Update: + + - Working on a full copy of the legacy package infra which + should get exposed to the public. + + - Action Items/Help Wanted: + + - Supply Chain Security: + + - General Update: + + - Continuing to meet with SIG Security and outlining a doc: + +> [\[WIP\] False Positives, VEX, and the Security +> Feed](https://docs.google.com/document/d/1a_3aMKexfhZwnbzmZZy0dhbQX7L_c8Cr2HSDKchLU7o/edit#heading=h.3xduq4wgcoc5) + +- Action Items/Help Wanted: + + + +- LFX project: + + - General Update: no updates right now + + + +- Release Team + + - \[Priyanka - will miss the call, meeting conflict\] + + - Final week in 1.29 Release Cycle! + + - 1.29.0-rc.1 is live (Thanks to Sascha!) + + - Nearing final Release Milestones for Docs, Release Notes, Release + Blogs completion (All are green, except Docs - 4 pending PRs, but + comms with KEP owners, is in progress) + + - Release Signal Handbook merged last week, Xander/Grace sent out + the email to [dev@kubernetes.io](mailto:dev@kubernetes.io) + ML to inform of the changes in 1.30 & onwards + + - Others - CNCF Release Webinar, Pre-release PR interviews, all + discussions are in-progress too. + + - Open question - should we hold release if publishing bot is still + broken + +** +Open Discussion:** + +- *Make artifact validation more robust' roadmapping* - tomorrow, Nov 29 + @ 1pm CET + [https://www.google.com/calendar/event?eid=NW9rZnJyaTByYnR0dXZjdWJvbWpubXEyamkgY2FsZW5kYXJAa3ViZXJuZXRlcy5pbw](https://www.google.com/calendar/event?eid=NW9rZnJyaTByYnR0dXZjdWJvbWpubXEyamkgY2FsZW5kYXJAa3ViZXJuZXRlcy5pbw) + +## Nov 21, 2023 + +**Host (pronouns):** + +**Attendees (pronouns):** + +- Xander Grzywinski (he/him) + +- Meha Bhalodiya (she/her) + +- [Pushkar Joglekar](mailto:pushkarj.at.work@gmail.com) (he/him) + +- Marko Mudrinić (he/him) + +- Leonard Pahlke (he/him) + +- Nishok Ganesan (he/him) + +- [Adolfo García Veytia](mailto:puercozon@gmail.com) (he/him) + +- Sujay Dey (he/him) + +- Jim Angel (he/him) + +- Friedrich Wilken (he/him) + +- Christopher Hanson (he/him) + +**Note Taker (pronouns):** + +- [Grace Nguyen](mailto:nnggrace@gmail.com)(she/her) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: + + - In process: 1.29.0-rc.0 + + - Action Items/Help Wanted: + + - OBS Packages: + + - General Update: + + - Public-facing docs for pkgs.k8s.io are complete and backported + up to v1.24 + + - We’re missing packages for v1.25.16, pending response from the + OBS team + + - Marko will ping them again + + - Built, on OBS infra but not published on our S3 package + + - Evaluating next steps and priorities + + - Action Items/Help Wanted: N/A + + - Project board: + [https://github.com/orgs/kubernetes/projects/137/views/1](https://github.com/orgs/kubernetes/projects/137/views/1) + + - Supply Chain Security: + + - General Update: + + - A bunch of us attended the SIG Security meeting on Nov 16th to + discuss an upcoming revamp of the kubernetes CVE feed. We’ll + continue our convo today, specifically to start thinking about + the work required from SIG Release / RelEng. + + - Vex issue: + [https://github.com/kubernetes/kubernetes/issues/121454](https://github.com/kubernetes/kubernetes/issues/121454) + + - How are we building the vuln feed today? How do we handle + security info from SRC? + + - Before, Google had a feed, now the community maintains a + feed. Merged alpha version of CVE in 1.25, beta in 1.27. Now + on k8s.io, JSON, RSS, markdown + + - Going into GA: getting feedbacks from SRC about how scanners + ingest, how to make feed more consumable + + - \[Jeremy\] Should we start a google doc about where we were, + and serve as a basis for roadmap and KEPs + + - [https://docs.google.com/document/d/1a_3aMKexfhZwnbzmZZy0dhbQX7L_c8Cr2HSDKchLU7o/edit](https://docs.google.com/document/d/1a_3aMKexfhZwnbzmZZy0dhbQX7L_c8Cr2HSDKchLU7o/edit) + + - Continue conversation tomorrow as well @ SIG-Security + Toolings meeting + + - Info about the feed starts with SRC created issues, internally + they share a YAML. There’s an opportunity to streamline this + + - \[PJ\] To clarify, we deal with 2 types of CVEs: + +1. Published by SRC with high confidence of impact, about Kubernetes + + 1. OK where we are but reduce repeat work for SRC + +2. Noisier, either from container images that the components published + as or Go dependencies + + 1. This is a good use case for VEX + + 2. This is what we’re discussing today + + - We have scanner runs 4 times a day with Snyk. + + - Recently merge go-vuln check on branch + + - There is an issue open where the go-vuln check run on release + branches + + - Noise from vulns in Go that we don’t use + + - No cherry-pick to silence scanner + + - \[PJ\] Idea: Additional labels where K8s can say this is not + affecting K8s. Create a trail of trust + + - \[puerco\] it is how VEX works. Initial model is last mile + of trust where the document can be sign and vouch for the + statements. + + - \[PJ\] Limit labeling to owners to improve trust + + - \[puerco\] create a system where more than one experts can + sign off on it + + - Nikhita has a thumbs-up page voting for TOC we can do + something similar + + - 2 VEX flow: + + - VEX generated from K8s itself + + - Assessment on 3rd parties modules/images + + - We can merge this into one flow. Big question is + implementation . + + - PJ needs drop continue conversation in sig-security tooling + + - What are we making? Specifically to VEX, a VEX document that + can be consumed by CVEs scanners + + - How does it work for SRC? SRC triage, identify fixes, publish + to embargoed list + + - Few source of truths + + - Release notes, issues that go into the CVE feed, data that + goes into external feed + + - What we want: maybe create a centralized feed internally + ([OS feed)](https://osv.dev/) that can feed to other + feeds + + - Pertaining to SIG Release: use our existing tooling to help + the effort and consume that data in our release process and + SBOM change to support VEX document + + + + - Action Items/Help Wanted: + + + + - LFX project: + + - General Update: The LFX project is coming to the end. Next + Thursday is the last day. Work on CLI is getting started and + Nitish is interested to stay around and help us with the CLI + after the internship! + + - Release Team + + - Review subteam merger comms -\> [Release Team Change Comms - + Google + Docs](https://docs.google.com/document/d/1QwdxigTCXYCX222tDCmdkFBLioLZAshJCZuLSW46enY/edit) + + - \[Review request\] Removal process + [documentation](https://github.com/kubernetes/sig-release/pull/2322) + + - Have a look at new project board -\> [Board · Release Team + (github.com)](https://github.com/orgs/kubernetes/projects/170/views/1) + + - Move over action items from retro doc + + - Not bound to release cycle, anyone can contribute + +- \[puerco\] Cloud Native Security Slam - Kubernetes Lightning Round + + - Please review Comms + [Doc calling for project + participation](https://docs.google.com/document/d/1nXaWs9ONjAVrM5FvMovHh6IU7a4AGWAuXqAOI7Hv5xs/edit) + \| [Event + Website](https://community.cncf.io/cloud-native-security-slam/) + +- \[jerickar\] revisit cherry pick criteria? Follow up to regression + data from [liggitt@google.com](mailto:liggitt@google.com) + + - Backporting leads to other regression + + - Too liberal cherry-pick right now. Don’t have strict criteria, + deferring to SIG-Leads and reviewers. We don’t have a checklist + + - Would benefit if we do LTS later on and potentially less regression + + - Is this something we want to discuss? We own this because we own and + ship cherry-pick + + - \[puerco\] our guideline is about what is allowed to be + cherry-picked. How do we restrict without requiring release managers + to verify + + - Suggestion from liggit is to change the issue template so folks + can attest that it falls under what we allow (we also need to + decide what we allow) + +## 14 Nov 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Meha Bhalodiya (she/her) + +- Sascha Grunert (he/him) + +- Jim Angel (he/him) + +- Sreeram Venkitesh (he/him) + +- Joseph Sandoval (he/him/él) + +- Adolfo García Veytia (he/him/él) + +- Priyanka Saggu (she/her) + +- Grace Nguyen (she/her) + +- Sujay (he/him) + +- Oluebube Princess Egbuna(she/her) + +**Note Taker (pronouns):** + +- Everyone + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: \[Jeremey\]No patch releases this month. Cherry + picks will fall into next month’s release. + + - Action Items/Help Wanted: + + - OBS Packages: + + - General Update: + + - Action Items/Help Wanted: + + - Supply Chain Security: + + - General Update: + + - \[Adolfo\] Discussed at Kubecon Chicago. What would it take + for us to issue Vex documents? Vex is a format that helps to + better inform if you are affected by a vulnerability. We tried + to improve SBOMs. The verification and security feed was an + area we were focusing on. Some improvements are needed to make + them useful. The other topic is about having a unified + security feed. Prototype that is running monitors GH and they + provide information after. Public disclosures limit access to + the fields. Requires more discussion with SIG-Security. + + - \[Jeremy\] Rita Zhang raised an + [issue](https://github.com/kubernetes/kubernetes/issues/121454) + with Vex documents. She has started to write the docs. It will + be discussed at SIG-Security (see: + [https://github.com/kubernetes/community/tree/master/sig-security](https://github.com/kubernetes/community/tree/master/sig-security)). + + - \[Joseph\] Have you received any user feedback? + + - \[Adolfo\] Limited. One other thing is fixing the Image + Promoter. \#1 focus for myself. + + - \[Jeremy\] Marko and I were discussing and will share our + notes. + + - Action Items/Help Wanted: + + - LFX project: + + - General Update: + + - Release Team + + - Beta.0 cut Thursday 11/16 (Veronica to start -\> handoff -\> Jim / + Mark) + + - 1.29 Test Freeze today - + [PSA](https://groups.google.com/a/kubernetes.io/g/dev/c/ukMtzsOHZDU/m/pF5HlappCAAJ) + + - Other 1.29 milestones for Nov 14, 2023: + + - Docs Deadline + + - Major Themes Deadline + + - Start final draft of Release Notes + + - [Discussion/AI](https://kubernetes.slack.com/archives/C2C40FMNF/p1699889130916229) + from KubeCon NA 23 SIG Release in-person meeting + + - Will schedule dedicated follow-up discussion for EOR retro + meeting & upcoming SIG Release weekly meeting. + +- + +** +Open Discussion:** + +- \[Lauri / Sascha\] Roadmap and Vision prioritization for 2024: + [https://github.com/kubernetes/sig-release/blob/master/roadmap.md](https://github.com/kubernetes/sig-release/blob/master/roadmap.md) + + - \[Sascha\] Put rough ideas in general board. Sync with Lauri about a + potential dedicated session. Potentially enabled downstream + consumers of K8s releases. + + - \[Adolfo\] SLSA spec is still being finished. 1) Move the SLSA + tester 2) + + - \[Sascha\] Would this tie into the signing release artifacts? + + - \[Adolfo\] Yes. We should go back and modernize what we have built + over the last two years and make sure they are all working + interoperably. + + - \[Lauri\] I would need some time to review and discuss what has been + done over the last year. Also understanding our users who they are? + + - \[Sascha\] It’s a mix. Do we have the capacity to work on next year? + + - \[Lauri\] How is the user participation? + + - \[Sascha\] Started strong and then we tapered off a bit with + contributors. + + - \[Lauri\] Dedicated conversations about sessions to identify the + biggest needs. We can use that data to help organize the work. We + could also discuss onboard new contributors. + + - \[Jeremy\] Data to see who is using our tools? Maybe a survey. + + - \[Lauri\] Could we use CNCF to help get user feedback? + + - \[Jeremy\] Do we wait until we receive the user data before issue + planning? + + - \[Lauri\] Ideally yes but do the must-do work you know is + non-negotiable and that’ll buy time while the data situation gets + sorted. + + - \[Jeremy\] Image promoter needs some attention and could be a focus. + + - \[Lauri\] We could walk through and identify the work. Focus on the + goal. + + - \[Adolfo\] There’s a few barrier of entry with respect to the + codebase and how fragile it is + + - \[Jeremy\] We might need to start over with docs and bring the + current version over + + - \[Lauri\] We might need a retro with the current design, what’s the + current problem and how to overcome it in the new version. What is + it setting out to do? How do we do that core function really well? + + - Then break it down into pieces that folks can pick up + + - \[puerco\] Building a wider base conversation looks completely + different whether or not we keep the existing system + + - \[Lauri\] Is it a monolith? (Yes) + + - \[Sascha\] we can have an issue where one or two people can start + working on it + + - \[Lauri\] keep this out of the roadmap, since its a technical + piece + +- \[puerco\] Cloud Native Security Slam: [Kubernetes Lightning + Round](https://community.cncf.io/cloud-native-security-slam/). + + - Maintainers for smaller projects can express interest and SIG + release can provide clear guidance on how to adopt tools and if we + have enough projects, we can guide them through their pull requests. + CNCF would produce some special edition swag for helping out with + these. + + - Plan to have a webinar on dec 6th if approved + +- \[jeremy\] SRC/SIG Security - VEX discussion @ next sig security + meeting - we covered this above + +- \[jim\] oauth tooling for sending gmail via \`krel announce\` + ([issue](https://github.com/kubernetes/release/issues/3360) / + [demo](https://github.com/jimangel/oauth-gmail-test)): + + - Arnaud: we should try to do this after 1.29 ? Happy to provide any + kind of help to do this so we can test it early 1.30 (all the alpha + releases ?). + +## Oct 31, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Priyanka Saggu (she/her) + +- Meha Bhalodiya (she/her) + +- Vyom Yadav (he/him) + +- Marko Mudrinić (he/him) + +- Christopher Hanson (he/him) + +- [Mansi Kulkarni](mailto:mankulka@redhat.com)(she/her) + +- Nishok Ganesan (he/him) + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Vyom from Release Team (CI Signal team), student + + - + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: No updates + + - OBS Packages: + + - General Update: No updates + + - Supply Chain Security: + + - General Update: No updates + + - LFX project: + + - General Update: Major work on library wrapped, Nitish started + working on CLI + + - Release Team + + - \[Priyanka\] + + - 1.29 Code Freeze begins on [01:00 UTC Wednesday 1st November + 2023 / 18:00 PDT Tuesday 31st October + 2023](https://everytimezone.com/s/24b27b03) + + - 1.29.0-alpha.3 scheduled for Nov 2, 2023: + + - Current CI Signal is No-Go (Red), 4 Failing Release Informing + Jobs (Fix PRs in progress, (capa) job marked as non + release-blocking) + + - Ongoing discussion – + [https://kubernetes.slack.com/archives/CJH2GBF7Y/p1698702103104369](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1698702103104369) + + - Question – If needed to delay the alpha.3 cut, thoughts/advice + on when to reschedule considering KubeCon next week? + + - Given that only informing is failing, we could go forward + with alpha.3 this week + + - etcd job can be ignored, it’s not relevant for this release + + - \[Priyanka\] Need help from Release Engineering group to create + the k/test-infra PR for 1.29 Code Freeze Enforcement. + + - Jeremy to sync with Mark about this + + - \[Priyanka\] Jim is away at this time, Mark will be the only + person responsible for release cuts + + - \[Marko\] Feel free to ping @release-managers group on Slack, + Priyanka to inform release managers day prior to release cut + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- \[Marko\] Feedback on repo for OBS CLI + +- \[Marko\] Kubernetes Contributor Summit @ KubeCon NA + + - SIG-Release in Person Meetup + ([https://sched.co/1TY7O](https://sched.co/1TY7O)) + + - A Practical Guide to Publishing System Packages for Kubernetes + Subprojects + ([https://sched.co/1SpA6](https://sched.co/1SpA6)) + + - Jeremy request an unconference session to discuss supply chain + security + +## Oct 24, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Rudraksh Karpe (he/him) + +- Josh Berkus + +- Nishok Ganesan(he/him) + +- Sreeram Venkitesh (he/him) + +- Mark Rossetti (he/him) + +- Anhelina Zelyk(she/her) + +- Christopher Hanson (he/him) + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: + + - \[Marko\] Patch releases for October went out successfully + last week. No major issues besides two known issues with image + promotion (replicating signatures) and sending announcements + to k-dev + + - \[Marko\] Reminder that we don’t have releases planned for + November because of Release Managers availability due to + KubeCon/vacations/Thanksgiving + + - \[Marko\] Patch release date for beginning of 2024 are live on + [website](https://kubernetes.io/releases/patch-releases/#upcoming-monthly-releases) + + - \[Marko\] Reminder that 1.25 is going EOL by end of this month + + - Action Items/Help Wanted : + + - \[Marko\] We need to remove 1.25 jobs once 1.25 reaches EOL. + If anyone wants to help with this, reach out to us on + \#release-management and we can pair on this + + - We’re still lacking docs and proper tooling for this and + we’ll need to do it manually again. These PRs can give some + idea what’s needed: + + - [https://github.com/kubernetes/test-infra/pull/30563](https://github.com/kubernetes/test-infra/pull/30563) + + - [https://github.com/kubernetes/test-infra/pull/29387](https://github.com/kubernetes/test-infra/pull/29387) + + - Mark R (marosset) can help + + - OBS Packages: + + - General Update: + + - \[Marko\] Documentation changes are backported to v1.27-v1.24 + docs. These docs are now advising users to use pkgs.k8s.io + + - \[Marko\] There are some documentation changes requested by + SIG Docs and I’m working on implementing these changes + + - Action Items/Help Wanted: + + - \[Marko\] The [project + board](https://github.com/orgs/kubernetes/projects/137/views/1) + has been recently tidied up. If you see anything interesting + there, you can reach out to me + + - Supply Chain Security: + + - General Update: + + - Action Items/Help Wanted: + + - LFX project: + + - General Update: + + - \[Marko\] The library part has been finished enough to start + working on the CLI. Nitish will start working on the CLI this + week + + - Release Team + + - \[Mark, Sreeram\] No major updates. Code freeze scheduled for next + week. + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +## 17 Oct 2023 + +Meeting got canceled. + +## Oct 10, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Anhelina Zelyk (she/her) + +- Oluebube Princess Egbuna(she/her) + +- Mark Rossetti (he/him) + +- Joseph Sandoval (he/him) + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - Patch Releases: + + - General Update: + + - Cherry-pick deadline at October 13 (this Friday) + + - Releases scheduled for Wednesday, October 18 + + - Reminder: we don’t have releases scheduled for November! + + - (Marko) This is due to Kubecon and release manager + availability. + + - Action Items/Help Wanted: + + - Send a reminder to k-dev for cherry-picks along with reminder + for packages + + - \[Jeremy\] Go releases should be updated before next patch + releases. + + - \[Mark\] The Go patch releases will be included. + [net/http2 + issues](https://groups.google.com/g/golang-announce/c/iNNxDTCjZvo/m/UDd7VKQuAAAJ) + being discussed. + + - OBS Packages: + + - General Update: + + - Updating docs to remove references to the legacy package + repositories across supported release branches + + - Action Items/Help Wanted: + [https://github.com/orgs/kubernetes/projects/137/views/1](https://github.com/orgs/kubernetes/projects/137/views/1) + + - Review needed for + [https://github.com/kubernetes/website/pull/43407](https://github.com/kubernetes/website/pull/43407) + + - Supply Chain Security: + + - General Update: + + - Action Items/Help Wanted: + + - LFX project: + + - General Update: + + - Two major tasks left for the library: + + - CRUD operations for OBS packages + + - Implementing client interface/structs + + - After these two major tasks are done, the plan is to start + working on the CLI tool. + + - Release Team + +(Priyanka - won’t be able to join the meeting, please find the updates +below. Thanks!) + +- 1.29 [Enhancements Freeze in + effect](https://groups.google.com/a/kubernetes.io/g/dev/c/AttpmuN6Hfo/m/Wn3M67FjAwAJ), + 58 tracked, 2 Exception Requests received – both approved + +- 1.29.0-alpha.2 planned for today, CI Signal is green for now + + - \[Mark\] Release in progress atm. + +- Removed “capg-conformance-main-ci-artifacts” job from + “sig-release-master-informing” board - + [https://github.com/kubernetes/test-infra/pull/30988](https://github.com/kubernetes/test-infra/pull/30988) + +- Ongoing discussions – 1.29 mid-cycle Deprecations/Removal and Major + changes blog + + + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- Arnaud: *Launching registry.k8s.io’s new regions*: + + - [https://github.com/kubernetes/registry.k8s.io/issues/236](https://github.com/kubernetes/registry.k8s.io/issues/236) + + - \[Arnaud\] Add more s3 buckets which will help reduce costs. + Launching new regions has impact on promotion process. Possible + timeout and potential failures during this rollout. + + - \[Marko\] Speaking of S3 buckets IIUC. This is a dedicated job and + not connected to promotion. Why do we need to add additional + artifact registries? + + - \[Arnaud\] We try to match a artifact registry with a S3 bucket. + Configuration of image and the blob. We pick one image and + replicate. Put the blob with one image in S3. + + - \[Joseph\] Is it a infrastructure strategy to align artifacts with + region? + + - \[Arnaud\] Goal is to opimize costs with this approach. Focus + where the costs are high. Join SIG-K8s-Infra meeting to discuss + costs. + + - \[Jeremy\] Is there good docs to reference about this? + + - \[Arnaud\] I’ll add some references into the SIG-release meeting + notes. + + - [https://github.com/kubernetes/registry.k8s.io/tree/main/cmd/archeio](https://github.com/kubernetes/registry.k8s.io/tree/main/cmd/archeio) + + - [https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io)](https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io)) + + - \[Jeremy\] Any actions needed? + + - \[Arnaud\] Not at this time. This is for awareness. + +- \[jeremy\]: Contributor Summit + + - Didn’t request a SIG Release meeting, should we ask for an + unconference session? + + - Jeremy submitted an “lts” unconference and a “security artifacts” + discussion topic: + + - [https://github.com/kubernetes/community/issues/7531#issuecomment-1755709900](https://github.com/kubernetes/community/issues/7531#issuecomment-1755709900) + + - \[Joseph\] What is our goal for meeting at Kubecon? + + - \[Jeremy\] Image promoter would benefit from discussion. Its been + awhile since we did SBOM’s. Maybe there is more artifacts we would + like to add. + + - \[Verónica\] SLSA has changed alot. + + - \[jeremy\] meet the contributors link: + [https://github.com/kubernetes/community/issues/7541](https://github.com/kubernetes/community/issues/7541) + +## Oct 3, 2023 + +**Host (pronouns):** Stephen Augustus (he/him) + +**Attendees (pronouns):** + +- Jordan Liggitt + +- Leonard Pahlke (he/him) + +- Grace Nguyen (she/her) + +- Marko Mudrinić (he/him) + +- Sreeram Venkitesh (he/him) + +- Amit Dsouza (he/him) + +- Priyanka Saggu (she/her) + +- Oluebube Princess Egbuna(she/her) + +- Jayson Du (he/him) + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Jayson Du from Amazon EKS team + + - Abhijeet Gaurav - contributing to CNCF projects + +- Release Engineering + + - Patch Releases: *please mention any issues that may affect upcoming + or recent patch releases.* + + - General Update: + + - \[Jim Angel\] + + - Patch releases went out last week. No major issues other than + image signing and sending announcements to k-dev + + - Action Items/Help Wanted: *(please include links to + issues/context)* + + - \[Marko\] Any updates on replacing SendGrid with something else? + + - \[Stephen\] Not sure that there’s an update at the moment + + - \[Veronica\] No solution at the moment, Jim has some solution + that might work for us + + - OBS Packages: + + - General Update: + + - \[Marko\] No major updates, we’re seeing more and more traffic + going over pkgs.k8s.io, daily average is about 200-300 GB + + - \[Marko\] Work on this to pick up again soon + + - Action Items/Help Wanted: + [https://github.com/orgs/kubernetes/projects/137/views/1](https://github.com/orgs/kubernetes/projects/137/views/1) + + - \[Stephen\] Can we send a reminder for freezing legacy repos + this week along with a reminder for backports? + + - \[Marko\] Yes, will take care of that + + - Supply Chain Security: + + - General Update: + + - Action Items/Help Wanted: + + - LFX project: + + - General Update: + + - \[Marko\] Nitish finished with CRUD operations for Projects, + next step is to write tests + +- Release Team + + - \[Priyanka\] + + - 1.29 Enhancements Freeze this week – Oct 6, 2023 + + - Team finished reaching out to KEP owners, reminder for chairs + and leads + + - 1.29.0-alpha.2 planned for Oct 10, 2023 (next week) + + - Delayed last week, new target date next week + + - v1.29 Mid-Cycle Release Retro scheduled for Oct 19, 2023 + + - Question to Marko/Jim – + [https://github.com/kubernetes/sig-release/issues/2291](https://github.com/kubernetes/sig-release/issues/2291) + is this information something that we need documented before our + next 1.29 release cuts? + If yes, could we please prioritize this? Thanks in advance! + + - \[Marko\] We definitely need to document this, but this is not a + blocker for 1.29, all OBS projects are already in place that we + need for 1.29. As part of our LFX project, we’re making + improvements to the process, and we plan to document everything + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- \[liggitt, 5 minutes\] review of go "major"-version bumps on release + branches + + - Context: + [https://github.com/kubernetes/enhancements/issues/3744](https://github.com/kubernetes/enhancements/issues/3744) + + - Aspects: + + - Timeline/checklist for bumping release branches + + - Tracking backports related to new go versions + + - [go1.20 + example](https://github.com/kubernetes/release/issues/2815#issuecomment-1373891562) + + - [go1.21 + example](https://github.com/kubernetes/release/issues/3076#issuecomment-1556053267) + + - Q: What is the best way for multiple release team members to + collaborate tracking go minor-version related backports and tasks? + + - Something on github seems ideal to track merged state of + individual prereq backports + + - Description / comment only editable by author or folks with repo + edit access + + - This seems like a good place to start + + - Github project might be a possibility + + - Q: What issue templates or playbooks should be updated to make this + a regular part of go minor version bumps? + + - AI(liggitt): update + [https://github.com/kubernetes/sig-release/blob/master/release-engineering/handbooks/go.md](https://github.com/kubernetes/sig-release/blob/master/release-engineering/handbooks/go.md) + + - [https://github.com/kubernetes/sig-release/pull/2364](https://github.com/kubernetes/sig-release/pull/2364) + + - AI(?): update + [https://github.com/kubernetes/release/blob/master/.github/ISSUE_TEMPLATE/dep-golang.md](https://github.com/kubernetes/release/blob/master/.github/ISSUE_TEMPLATE/dep-golang.md) + + - + + - Question: How did you come up with the list of changes? + + - Changes that we discovered were required to fix CI failures before + master could successfully update to new go version + + - Rarely - follow-ups post-merge after updating master + + - Question: where are post-submit tests? + + - [https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + + - [https://testgrid.k8s.io/sig-release-master-informing](https://testgrid.k8s.io/sig-release-master-informing) + + - [https://testgrid.k8s.io/sig-release-master-blocking](https://testgrid.k8s.io/sig-release-master-blocking) + + - Stephen: risk is not having a single person with a view to the whole + lifecycle of picking up a new major go version (easy for handoff to + have things fall through the cracks) + +- \[liggitt, 20 minutes\] Review of regression data from 1.19 to present + + - [Kubernetes patch release regression/bugfix + rate](https://docs.google.com/spreadsheets/d/1LbGKBC4D2sLkcmzY9qDx9u-1D9TKC_ZrM8iA1eHW4Hs/edit#gid=1283859152) + + - [Analysis of Kubernetes regression rates, patterns, + examples](https://docs.google.com/document/d/1Vr-m-KL7P2KYDPsvfpJyFVW1Q6XPS5d8xz9xCq_Azpw/edit?pli=1#bookmark=id.h4sqtkkivzm0) + + - Interesting data points + + - ~50% more regressions come from bugfix/cleanup work than from + feature work + + - 90% of minor versions \>= 1.19 have regressed in a patch release + (only 1.28 hasn't) + + - 28% of patch releases \>= 1.19 contained a regression relative to + the .0 release + + - Q: Should the "backport to all branches simultaneously" policy + change for some types of backports (non-security, non-data-loss, + non-regression) in light of several backports that caused + regressions in patch releases? + + - Veronica: + + - current process doesn't give branch managers much decision input + to backport / merge decisions + + - thinks backport call should be on approvers for whether + backports should go to all branches + + - find ways to make backport decisions for branch managers + feasible without all context + + - Stephen: + + - mechanical part of the backport process (traffic direct in / + branch manager approval) + + - double check backport meets cherrypick guidelines + + - Jordan + + - opportunity to augment cherry pick guidelines to guide area + approvers? + + - highlight categories of backports that regressed patch releases: + big changes, entangled changes, changes in undertested areas, + changes impacting out-of-tree library users (scheduler plugins, + aggregated apiservers) + + - + + - Veronica + + - branch managers do already dig into details and trying to + understand context / severity already + + - if reviewing / approving cherry picks requires more detail, do + we need more time between deadline and patch cut? + + - Stephen: backport authors / area owners need to feel ownership + for their areas and meeting schedules; for normal operations, + those schedules should hold; still need to accommodate + unforeseen or urgent fixes + + - Q: What playbooks should be updated to make review of the regression + labeling / release note for backports part of branch manager + approval? + + - Release note includes the specific minor / patch where the + regression was introduced (good for users and helpful for us) + + - Ensure kind/regression label is only set on backport PRs fixing a + minor where the regression was introduced + + - Ensure kind/feature label is only set on backport PRs fixing a + regression related to feature work in that minor + + - + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +## Sept 26, 2023 + +**Host (pronouns):** + +- + +**Attendees (pronouns):** + +- + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - Release Team + + - (Priyanka - won’t be able to join the meeting, please find the + notes/updates below. Thanks!) + + - Question to Marko/Jim – + [https://github.com/kubernetes/sig-release/issues/2291](https://github.com/kubernetes/sig-release/issues/2291) + is this information something that we need documented before our + next 1.29 release cuts? + If yes, could we please prioritize this? Thanks in advance! + + - Updates: + + - 1.29 PRR Freeze on Sept 28th + + - 1.29 Enhancements Freeze approaching next week - Oct 6th + + - 1.29 Alpha.2 delayed (was tentatively planned for this week). + Will discuss with Jim/Mark for the timeline. + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +## Sept 19, 2023 + +**Host (pronouns):** Carlos Panato + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Sascha Grunert (he/him) + +- Priyanka Saggu (she/her) + +- Nitish Kumar (he/him) + +- Fyka Ansari (she/her) + +- Paco Xu(he/him) + +- Anhelina Zelyk (she/her) + +- Sreeram Venkitesh (he/him) + +- Joseph Sandoval (he/him) + +- Leonard Pahlke (he/him) + +- Jim Angel (he/him) + +- Mengjiao Liu (she/her) + +- Jeremy Rickard (he/him) + +- Ricky Sadowski (he/him) + +- Rey Lejano (he/him) + +- Amit Dsouza (he/him) + +- Michael Singh (he/him) + +**Note Taker (pronouns):** + +- Joseph + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Amit Dsouza from Australia. Contributor to Argo. Trying to become + more active. + + - Mengjiao Liu - release notes shadow. Learn about release process. + + - Michael S - Looking for things that he can be a candidate to + contribute. Part of 1.12/1.13. + +- Subproject updates + + - Release Engineering + + - Patch Releases: *please mention any issues that may affect + upcoming or recent patch releases .* + + - General Update: + + - \[Jim Angel\] + + - Patch releases went out last week. No major issues other than + image signing and sending announcements to k-dev + + - Action Items/Help Wanted: *(please include links to + issues/context)* + + - Patch releases calendar needs to be updated to include the + latest patch releases: + [https://kubernetes.io/releases/patch-releases/](https://kubernetes.io/releases/patch-releases/) + + - Example PR: + [https://github.com/kubernetes/website/pull/40100](https://github.com/kubernetes/website/pull/40100) + + - \[Marko\] Staging job flaked two times for 1.26. Something + about not enough memory. Not sure why this is happening. + Action items: patch release calendar needs to be updated. + This is open to anyone to do it quickly and open to try. + Above is the sample PR. If not done by tomorrow Marko will + take care of it. + + - \[Amit Dsouza\] I can take it. + + - \[Michael S\] We can all pair up on it. + + - Alpha cut 1.29 is planned for today. Still blocked. + + - \[Priyanka\] Flaky job is blocking. [Stateful + set](https://github.com/kubernetes/kubernetes/issues/120700) + is blocking. CI Signal shadow is following up to determine if + this is still blocking. + + - \[Jim Angel\] Will see if alpha blocking or not and follow up + with the team on Slack. + + - \[Priyanka\] Dims suggested keep it on the board and try and + fix it. Its release informing. + + - \[Carlos P\] CAP provider GCE is deprecated. In touch with the + maintainers. The work to resolve the issue is almost done. The + job will be green again later in the week. The deploy needs to + change to resolve this issue. This doesn’t impact Alpha 1. + + - OBS Packages: + + - General Update: + + - Issue with kubeadm packages + + - [https://github.com/kubernetes/release/issues/3276](https://github.com/kubernetes/release/issues/3276) + + - LFX project: merged API types for managing OBS Projects + + - [https://github.com/kubernetes-sigs/release-sdk/pull/246](https://github.com/kubernetes-sigs/release-sdk/pull/246) + + - \[Marko\] There is pr to fix. This issue only applies to patch + releases. Can’t edit existing packages. Find way to + communicate this issue. I’ll look at this issue tomorrow. + + - \[Priyanka\] Today’s alpha cut will have the same issue? + + - \[Marko\] Alpha cut will not have any issues. But we can + verify with this cut. Status of LFX projects. This is the 3rd + week starting on this. Updates coming next week. AI - anything + on the board you would like to work on just reach out to + Marko. + + - Action Items/Help Wanted: *(please include links to + issues/context)* + + - Project board: + [https://github.com/orgs/kubernetes/projects/137/views/1](https://github.com/orgs/kubernetes/projects/137/views/1) + + - + + - Supply Chain Security: + + - General Update: + + - \[Sascha\] No updates at the moment. + + - Action Items/Help Wanted: *(please include links to + issues/context)* + + - Release Team + + - \[Priyanka\] Discuss: whether/not to demote the + [capg-conformance-main-ci-artifacts](https://testgrid.k8s.io/sig-release-master-informing#capg-conformance-main-ci-artifacts) + job from release-informing dashboard + +> (ref: [slack +> discussion](https://kubernetes.slack.com/archives/C2C40FMNF/p1694672335486569)) + +- 1.29.0-alpha.1 planned for today, current CI Signal is NO-Go, waiting + for follow-up on a flaky job + [https://github.com/kubernetes/kubernetes/issues/120700](https://github.com/kubernetes/kubernetes/issues/120700) + +- Others: + + - Shadow orientation meetings planned for this week! + + - \[Priyanka\] Invites have been sent. + + - \[Carlos P\] When are they happening? + + - \[Priyanka\] Both are on Thursday + + + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +## Sep 12, 2023 + +**Host (pronouns):** + +**Attendees (pronouns):** + +- Marko Mudrinic + +- Nitish Kumar + +- Rudraksh Karpe + +- Jeremy Rickard + +- Veronica + +- Sreeram + +- Gracenguyen + +- Mengjiao Liu + +- Arnaud + +- Jim Angel + +- Nishok + +- Mahamed Ali + +- Mark Rossetti + +**Note Taker (pronouns):** + +- Marko Mudrinić + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Nitish - SIG Contribex Comms and LFX project for SIG Release + + - Sreeram - part of the v1.29 Release Team + + - Rudraksh - exploring the landscape + +- Subproject updates + + - \[Veronica\] New proposed structure for Release Engineering updates + + - Action Item: Veronica will update the meeting agenda to reflect + new structure for updates + + - We’re missing a lot of context on various topics, like OBS, image + signing, SBOMs… + + - It would be nice to have structured updates so that we make sure + we provide proper updates and that folks are aware how are + different areas doing + + - A lot of folks want to join and help, but they’re not sure how and + where + + - We’ll use current format today and figure out new format going + forward + + - Have a link to project boards so folks can check if there’s + anything to pick up + + - We encourage people to come to the meeting, but that might not + always be possible due to conflicts/time zones. The template + should be covering enough stuff so that folks not attending can + figure out what’s going on + + - Release Engineering + + - \[Marko\] + + - Go updated to 1.21.1/1.20.8, thanks to Carlos! + + - Go update to 1.21 still ongoing, thanks to Carlos and Sascha for + dealing with this and base image updates! + + - kubepkg and rapture are considered deprecated and will be + removed after October patch releases: + [https://groups.google.com/a/kubernetes.io/g/release-managers/c/4Cl479abBaE/m/EFNfChPrBAAJ](https://groups.google.com/a/kubernetes.io/g/release-managers/c/4Cl479abBaE/m/EFNfChPrBAAJ) + + - OpenBuildService projects for v1.29 are now in place, there’s a + Slack thread describing the process step-by-step + + - Patch releases are scheduled for tomorrow, thanks to Veronica + for volunteering to help with these releases! + + - Reminder that we’re freezing the legacy repositories tomorrow! + [https://k8s.io/linuxrepos](https://k8s.io/linuxrepos) + + - We had OpenBuildService and pkgs.k8s.io introduction last week, + slides and recording are posted on \#release-mangement + + - Release Team + + - \[Priyanka\] + + - 1.29 Lead Shadow on-boarding complete - + [https://github.com/kubernetes/sig-release/issues/2329](https://github.com/kubernetes/sig-release/issues/2329) + + - 1.29 Shadow Selection done, all applicants are informed of their + selection status. Shadow onboarding in progress: + + - [\[1.29\] add release team shadow information + \#2339](https://github.com/kubernetes/sig-release/pull/2339) + + - [\[1.29\] update google groups/GCP IAM membership for + release team shadows + \#5831](https://github.com/kubernetes/k8s.io/pull/5831) + + - [\[1.29\] add release sub-teams role shadow to + release-team-\* gh team + \#4452](https://github.com/kubernetes/org/pull/4452) + + - \`1.29.0-alpha.1\` is scheduled for **Tuesday + 19th September 2023** (confirmed for Branch Manager + availability - + [discussion](https://kubernetes.slack.com/archives/C2C40FMNF/p1694181829355399?thread_ts=1694163871.580529&cid=C2C40FMNF)) + + - Updated [Kubernetes Release + Calendar](https://bit.ly/k8s-release-cal) for 1.29 release + cycle milestones (pending release cut dates timeline - waiting + for input from Branch Manager) + + - Scheduled 1.29 Weekly Release Team Meetings + (first meeting happened on Sept 6, 2023) + + - Scheduled optional 1.29 Release Leads sync + call (first meeting happened on Sept 11, 2023) + + - \[Jim\] We need to decide on if we need 4th + alpha cut, it was added last cycle to test the OBS integration, + so we can remove it + + - \[Jeremy\] +1 to removing it + + - \[Priyanka\] Thanks Jim for adding the + context, and +1 to removing it. + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +- \[Grace\] Looking for additional reviews and comments on the + [release team member removal + process](https://github.com/kubernetes/sig-release/issues/2332) + before I draft up the PR + + - Call to action to review it, discussion going on in the issue + +** +Open Discussion:** + +- \[Arnaud\] Deprecate gs://kubernetes-release + + - With Fastly shielding the bucket we can switch to a community-owned + bucket. Ideally, the new bucket will contain objects for each + release and will be private. + + - [https://github.com/kubernetes/k8s.io/issues/2396](https://github.com/kubernetes/k8s.io/issues/2396) + + - We serve all the binaries that we build from this bucket. It’s owned + by Google. We don’t have a full access to that bucket + + - With the rollout of Fastly, we’re shielding the bucket and can + switch to a community-owned bucket. How to establish the lifecycle + of objects through the release, and improve cache efficiency. We + have full control and can do more things + + - Start discussion on how we can move + + - Do we want to do it in v1.29? + + - New bucket should be private so we exclusively serve object through + Fastly + + - We can backfill, GCS protocol supports that, we are just deprecating + the old bucket, we’re not going to remove it, at least not at this + time + + - We can backfill it from the beginning with all releases + + - \[Mahamed\] How long is dl.k8s.io GA for? + + - More than 5 years + + - \[Mahamed\] Should we make the old bucket private? + + - No, we don’t want to do it. There are also CI artifacts in that + bucket + + - We can also move CI artifacts, but focus is on anything supported + for community, CI artifacts are to follow up + + - We definitely need to communicate this, but we need to figure out + how + + - \[Marko\] Can we push both to old and new buckets? + + - Could be doable + + - \[Marko\] Two milestones for deprecation and freezing + + - +1 from Mahamed, also to figure out CI stuff + +- \[Arnaud\] Migrate away for project *kubernetes-release-test* + + - Security exceptions were filled to keep use this project. We should + start look on how we migrate away from this project. + + - Short window for the transit. Try to do this before rc1 of 1.29 + + - kubernetes-release-test is a Google-owned project. This is tricky, + we have to do it at once, we have to just switch-over + + - We need to do inventory, some things to take care of: + + - Google Cloud Build + + - GCS buckets + + - KMS + + - ServiceAccounts + +- \[Arnaud\] Shutdown + [https://release.triage.sigs.k8s.io](https://release.triage.sigs.k8s.io) + + - Not currently maintained. + + - \[Jeremy\] I think we can shutdown this + + - \[Marko\] +1 to shutting it down + + - \[Anhelina\] Release Team is mostly using private chats for triaging + + - \[Arnaud\] If anyone wants it, we need to identify a team that’s + going to maintain it + + - \[Jeremy\] Let’s call out on Slack and see if anyone is using it + + - \[Marko\] Is it even up? + + - \[Arnaud\] It has been broken for some time + +## Sep 5, 2023 + +Meeting got canceled. + +## Aug 29, 2023 + +**Host (pronouns):** + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Mark Rossetti (hi/him) + +- Mickey Boxell (he/him) + +- Kat Cosgrove (she/her) + +- Natali Vlatko (she/her) + +- Sreeram Venkitesh (he/him) + +**Note Taker (pronouns):** + +- Grace Nguyen (she/her) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - \[Veronica\] Release Manager check-in survey + + - \[Jeremy\] we will be halting publishing packages to legacy repos + + - Comms for sep/oct freeze + + - \[Marko\] Please fill the Doodle for OBS meeting planned for next + week: + [https://doodle.com/meeting/participate/id/dLJygKWa](https://doodle.com/meeting/participate/id/dLJygKWa) + + - Release Team + (Priyanka - won’t be able to join the meeting, please find the + updates below. Thanks!) + + - \[Priyanka\] All Release Lead Shadows Selection done (& + [announced](https://github.com/kubernetes/sig-release/issues/2307#issuecomment-1693370438)) + + - \[Priyanka\] [Captured feedback from 1.28 release cycle + retro](https://github.com/kubernetes/sig-release/issues/2313#issue-1859099061) + and planned to work on them during 1.29 + + - \[Priyanka\] Discussed async with SIG Release Leads: + + - Decided to keep CI Signal & Bug Triage separate for 1.29 due to + no documentation for the new combined role (Release Signal). + We'll gradually merge the two existing handbooks to form the + "Release Signal" role handbook during the 1.29 cycle. ([wider + group + update](https://github.com/kubernetes/sig-release/issues/2307#issuecomment-1697341504)) + + - Also, for the 1.29 "Bug Triage" Role, I've + discussed with Furkat, and we have a potential candidate with a + +1 from Grace. + I'll confirm their interest and availability. + + - [Announced](https://github.com/kubernetes/sig-release/issues/2307#issuecomment-1697343071) + Bug Triage Role Lead (Yigit Demirbas) + + - \[Priyanka\] Reminder: The 1.29 Shadow + Selection application closes in 3 days on Sept 1, 2023. + + - \[Priyanka\] 1.29 Release Cycle begins on Sept + 4, 2023! 🎉 + + - Timeline Planning in progress - + [https://github.com/kubernetes/sig-release/issues/2313](https://github.com/kubernetes/sig-release/issues/2313) + + - **Request -** Need lgtm/approval to merge the + timeline PR if no objections! + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- \[RT Retro Item added by Leo\] + + - \[Grace\]: If a shadow is unresponsive, have a discussion and remove + them + + - What is the actual process for this? We’ve never done this + before - not true + + - \[Josh\] Not at all accurate. When we introduced the shadow system + in 1.10-1.15, we removed shadows all the time when they dropped + out. Probably 50% of shadows dropped out, which is why we went + from 1 to 2 shadows per lead. + + - \[Jeremy\] We have done this in the past just no docs. Something + we should do. You are signing up to do work. + + - \[Stephen\] Signaling to folks, this is a resume builder as it’s + not easy to get into + + - It is work to onboard, offboard and stay on the team and not + fair to people doing the work + + - Leads should be looked at the same way + + - \[Mickey\] +1 to removing inactive folks from the team + + - \[Natali\] Unresponsive lead, rank of shadows + + - \[Grace\] we often have 2 return shadows + + - \[Kat\] [PR to + document](https://github.com/kubernetes/sig-release/issues/2127#event-8791241644) + for a couple releases + + - \[Grace\] [PR in progress to capture + this](https://github.com/kubernetes/sig-release/pull/2322/files) + + - \[Josh\] when we start a shadow program, it's pretty normal to + drop out. We expect 50% dropout + + - Shadows communicate if they can’t continue + + - EA introduced and help dismiss shadows + + - \[Grace\] I think 50% drop will put some teams in a tough + positions like enhancements + + - \[Stephen\] Hard limit 4-5ish, as long as you can manage. Docs to + remove is not documented. What does backup and graceful removal + look like? + + - Stable roster is an option. Presumption of stability which is + not true and won’t work + + - \[Josh\] Still, if we need backfill, having a call list of folks + who have done the position before but aren't exactly shadows would + be worthwhile. + + - \[Mickey\] Warm pool of existing shadows. Reach out and ask about + time availability, for backups + + - \[Grace\] return shadows step up for lead, and pool is for newer + shadows + + - \[Stephen\] Check out the comments on the Stable Roster KEP: + [https://github.com/kubernetes/enhancements/pull/3347](https://github.com/kubernetes/enhancements/pull/3347) + + - \[Jeremy\] Few ways to back few: no release lead, look through + list of folks in past release (comms lead, EA) + + - \[Jeremy\] EA checking the pulse more. If they don’t attend + orientation, that’s a flag + + - \[Stephen\] Document backfill. Top role for EA is to make sure + that retro items don’t drop + + - \[Natali\] Do we have how available we are as a removal criteria + + - \[Kat\] when they sign up as a shadow, they are agreeing to show + up meeting. Don’t miss 2 meetings without comm to lead + + - \[Stephen\] multiple routes of escalation. + + - Shadow not supported by lead - go to lead or EA + + - Lead not supported by shadows - lead or EA + + - Bidirectional - lead to lead + + - Removing shadows: + + - + + - Removing lead: + + - SIG flags to release lead that role lead is not responsive + + - Docs - branch sync is a signal + + - Attend meeting or send proxies + + - Missing deadlines + + - \[Stephen and Jeremy\] resolve this in an issue. Escalation path + + - \[Priyanka\]: It is now possible to make a copy\* of an existing + GitHub Beta project + ([details](https://docs.github.com/en/issues/planning-and-tracking-with-projects/creating-projects/copying-an-existing-project)) + + - \[Grace\] what is this in relation to? + + - \[Grace\]: Comms and Docs lead attendance to sig-docs meeting + mandatory + + - \[Grace\]: Perhaps creating template for docs and comms in sig-docs + notes that caters to their specific tasks like updating branch, PRs? + + - \[Stephen\] SIG-Docs lead if they are not showing up at meeting, + let us know + + - Captured on Priyanka’s retro todo list + + - https://github.com/kubernetes/sig-release/issues/2313#issue-1859099061 + + - \[Grace\]: Review release timeline at the end of meeting + + - \[Stephen\] Moving towards a deliverables. + + - \[Leo\]: Reintroduce the RT Lead sync meeting to keep in touch + between leads, get to know each other and slowly surface any + problems they may have + + - \[Jeremy\] this was useful when we did it. + + - \[Jeremy\] would also suggest a kick off + + - \[Leo\]: Make RT Lead handovers more a thing, right now they are not + part of the playbook + + - \[Mickey\] not enough continuity. Current lead doesnt know what + they don’t know. Coordinate directly instead of over slack + + - \[Natali\] add returning shadows to that handover if possible + + - \[Kat\] continuity after the release. Feel confident and know who + to lean on + + - \[Stephen\] take intent with the role. Part of succession is to + support shadows + + - **Grace checkin with handbook about handover** + + - \[Leo\]: Include sig feedback upon lead succession (like sig-docs, + sig-test-infra) + + - \[Natali\] earn shadow role again before leading + + - \[Atharva\] We, the enhancements team, are + working on adding the Release Team update statuses (Red, Yellow, + Green) in our role handbook. Would be helpful if other teams include + it as well. + [Issue](https://github.com/kubernetes/sig-release/issues/2321) + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- + +## Aug 22, 2023 + +**Host (pronouns):** Sascha Grunert (he/him) + +**Attendees (pronouns):** + +- Joseph Sandoval (he/him) + +- Rey Lejano (he/him) + +- Anhelina Zelyk (she/her) + +- Jeremy Rickard (he/him) + +- Leonard Pahlke (he/him) + +- [Adolfo García Veytia](mailto:puercozon@gmail.com) (he/him/él) + +- Rakshit Gondwal (he/him) + +- Aashish Nehete + +- Mahamed Ali + +- Rajib Mitra (he/him) + +- Verónica López (she/her) + +- Rudraksh Karpe (he/him) + +**Note Taker (pronouns):** + +- Grace Nguyen (she/her) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Welcome Rakshit Gondwal! + +- Subproject updates + + - Release Engineering + + - Last week, Jeremy brought up failure on signing. It is now common + + - On infra/registry change, noone has had time to update the + process + + - Jeremy rallying to get together to fix issue and remediate the + signature + +1. Fix the bug + +2. Go back in the registry and update previous signatures + + - Patch releases scheduled for tomorrow, Jeremy will drive them. + + - Morning US time, limited avail with Google admin + + - \[Veronica\] Happy to kick off in her timezone. + + - Organized for that, will sync with Veronica offline + + - Preparing release manager check-in form, incoming to mailing list + + - See who is still interested and opening up opportunities + + + + - Release Team: + + - Priyanka and Xander as Lead and EA for 1.29 🎉 + + - 1.29 [shadow + survey](https://docs.google.com/forms/d/e/1FAIpQLSfo4EXVqjlHzz2QPeNqar8dprZV06ETm740VQASEQLHRCX1tw/viewform?usp=sf_link) + is out + + - PR for 1.29 release timeline is up - + [https://github.com/kubernetes/sig-release/pull/2314](https://github.com/kubernetes/sig-release/pull/2314) + + - As per liggit’s feedback - have moved “Code Freeze '' to week + 9 (before KubeCon) - need more feedback/eyes on this change! + + - Retro tomorrow - lots of discussion docs and comms + + - There’s interest for an **Intro to release eng** session + + - What goes into cutting a release, cherry picks, etc + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- The talk proposal for Kubecon - maintainer track has not been + announced yet + +- Next steps for fixing signing step: + + - Context: when we cut a release, we build things and move them to + buckets and registry. Part of that process is image promotion, build + in one place, sign and copy to the registry. Noone can modify the + images. Sign in build, carry signature to production registry and + then image promoter add final signature, so two signatures. This was + designed for 3 mirrors, now it is more complex and resilience at ~20 + mirrors + + - Process of copy and replicate signatures is failing. We run out of + quote on GCP because we’re hitting the registry too much + + - Adolfo’s proposal: sign one image , replicate async + + - Result: inconsistent signatures in the past couple cuts. Registry + might not have a copy of signature to verify. + + - New subcommand ([kpromo + sigcheck](https://github.com/kubernetes-sigs/promo-tools/pull/767)) + that checks which images in a registry - code is written. This will + allow us to proceed even if the signing process fails, the cron job + will come in async and fix up the signatures + + - Jeremy - set up a sync meeting + + - Adolfo: +1 + + - Sascha: Do we plan to outline this in an enhancements? + + - [Issue](https://github.com/kubernetes/release/issues/2962) + + - We have signing artifact enhancements to be GA + +- Mahamed: update the whole image promoter as part of this? + + - Adolfo: it’s a dream but it’s a big project + + - Redesign how we replicate signatures to be more resilience, less + real-time + + - Sascha - an idea discussed at Kubecon Amsterdam, sign one image and + have it as root of trust + +- We are close to quota limit and we could raise it. However, the + process is not good and we should improve it + +- Next steps: next sig-release meeting or another dedicated time + + - Doodle coming out today to sig-release by Jeremy + +- Aashish: auto-fetch release notes, deprecate subcommand that creates + PR + + - Sascha - Deprecate would be good, docs about this in handbook so we + should remove those part + + - We moved away from Google Analytics but maybe on Netlify + + - Rey - Netlify shows that site analytics is not enabled for the + release notes site + +- Adolfo: Dependabot opens PR with \`@\` that’s being rejected by prow + ([example](https://github.com/kubernetes-sigs/release-notes/pull/560)) + + - Is it fixed? + + - Sascha: Dependabot now supports grouping updates. Update + dependencies is a future goal + +## Aug 15, 2023 + +**Host (pronouns):** + +Jeremy Rickard + +**Attendees (pronouns):** + +- Grace Nguyen (she/her) + +- Rey Lejano (he/him) + +- Kartikey Rawat + +- Rishit Dagli (he/him) + +- Nishok + +- Chris Hanson + +- Jim Angel + +- Burpt + +- + +**Note Taker (pronouns):** + +- Rey Lejano + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - \[Jeremy\] Had to delay patch releases, problem with Go updates, + bumped release branches to latest version of 1.20 Go. Delayed + patch releases will happen on Oct 23. Release is happening now, + last part of the jobs are happening, then Google build admins will + take care of package signing, we have issue open to lift code + freeze and Grace will take care of the issue to lift code freeze. + + - Release Team + + - \[Grace\] Release management is working on creating the build, has + eyes on docs to get that ready. Release blog is ready to merge. + + - \[Jeremy\] Issue is open for next release team, 1.29 + [https://github.com/kubernetes/sig-release/issues/2307](https://github.com/kubernetes/sig-release/issues/2307) + Jim will be the branch manager for 1.29 + + - A/I: Follow up and identify EA and shadow survey for 1.29 + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- Signing and image promotion + + - \[Jeremy\] Will also discuss signing and image promotion next week. + + + +## Aug 8, 2023 + +Meeting got canceled. + +## Aug 1, 2023 + +**Host (pronouns):** + +Jeremy Rickard + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - Patch releases coming next week, cherry-pick deadline this Friday + + - Go 1.20.7 coming out this week with a security fix + + - A/I: Identify RM for cutting patch releases + + - Release Team + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- OpenBuildService pre-launch: pkgs.k8s.io is live + + - We plan to have official launch around 1.28.0 release + + - Marko is on vacation starting next week for two weeks, Marko to + prepare everything for Release Managers to do before the final + release + + - General overview of remaining important tasks + + - Finish documentation and blog post + + - Backfill repositories + + - Docs and blog post are looking for review: + + - Docs: + [https://github.com/kubernetes/website/pull/42022](https://github.com/kubernetes/website/pull/42022) + + - Blog post: + [https://hackmd.io/@xmudrii/ryoT2PIo2](https://hackmd.io/@xmudrii/ryoT2PIo2) + +## Jul 25, 2023 + +**Host (pronouns):** Carlos Panato + +**Attendees (pronouns):** + +- Leonard Pahlke (he/him) + +- Marko Mudrinić (he/him) + +- Anhelina Zelyk (she/her) + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - Release Team + +**Open Discussion:** + +- \[Leo\] Merge all Slack release subteam channels into one channel + beside ci-signal; essentially \#release-team, \#release-ci-signal + + - Leo will create an issue to track this proposal + + + +- \[Marko\] OBS status update and next steps + + - Walk the board: + [https://github.com/orgs/kubernetes/projects/137/views/1](https://github.com/orgs/kubernetes/projects/137/views/1) + + - Handling cri-tools and kubernetes-cni packages: + [https://github.com/kubernetes/release/issues/3169](https://github.com/kubernetes/release/issues/3169) + + - Initial versions for cri-tools and kubernetes-cni + + - Graduation criteria and phases for KEP-1731 + + - Documentation and migration comms: + [https://github.com/kubernetes/release/issues/3056](https://github.com/kubernetes/release/issues/3056) + + - + +- + +## Jul 18, 2023 + +Meeting got canceled. + +## Jul 11, 2023 + +**Host (pronouns):** Verónica López (she/her) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Xander Grzywinski (he/him) + +- Grace Nguyen (she/her) + +- Joseph Sandoval (he/him) + +- Mahamed Ali + +- Sascha Grunert (he/him) + +- Shivanshu Raj Shrivastava (he/him) + +- + +**Note Taker (pronouns):** + +- Joseph Sandoval + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Amit Dsouza - Based out of Australia. DevOps specialist. Recent + speaker at Argocon EU. + +- Subproject updates + + - Release Engineering + + - \[Marko\] July patch releases delayed to next Wednesday: + [https://groups.google.com/a/kubernetes.io/g/dev/c/A_hz52m4GUA/m/pWo7tJ0_BAAJ](https://groups.google.com/a/kubernetes.io/g/dev/c/A_hz52m4GUA/m/pWo7tJ0_BAAJ) + + - \[Marko\] The delay allows for the Go security updates to be + merged. + + - \[Marko\] Sascha and Carlos are working on updating the base image + to Debian 12 (bookworm) + + - \[Marko\] The mailing list has some feedback on image size. + + - \[Sascha\] Kubecross will be updated to bookworm with 1.28. + + - \[Marko\] We had v1.28.0-alpha.4 with some minor hiccups, but + overall went well + + - \[Marko\] Generating SBOM had a null pointer exception. Fixed + by @puerco. Also ran into rate limiting issues. This might + happen next week but be aware. Some release managers are having + issues with Sendgrid. + + - \[Marko\] 1.26-1.24 release branches are updated to Go 1.20 + (thanks Carlos!) + + - \[Marko\] Final PR’s need to be merged. Publishing bot is broken + and needs to be fixed. + + - Release Team: + + - One week from code freeze + + - \[Grace\] First retro will be held next week. + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- \[Marko\] OBS updates + + - \[Marko\] Outstanding item was How do we connect OBS with the + release pipeline and publish packages? This is now complete. + Additional tests are needed for 1.28. Next up is sorting out + infrastructure. The OBS platform has limited bandwidth. We don’t + know how much bandwidth we need. We don’t have any stats from + Google. OBS will be a single repo but point to our own mirrors. S3 + bucket with CDN in front of it. CloudFront will be used. If we get + additional bandwidth we will use Fastly. We can’t use Fastly atm. If + we get bumped to 20 PB’s of bandwidth we will then use Fastly. + [https://github.com/kubernetes/k8s.io/pull/5536](https://github.com/kubernetes/k8s.io/pull/5536) + is under review and if ok’ed will merge this week. Docs and feature + blog post are being drafted by @marko. CNCF will help to market this + change when we are ready. + +- \[Mahamed\] How do we determine which Prow jobs are blocking and + master blocking? + + - \[Sascha\] Sig-Release has + [docs](https://github.com/kubernetes/sig-release/blob/252777c7f9bec2297aaafeb12bf5984c4bc80705/release-blocking-jobs.md) + on this question. + + - \[Verónica\] Not everything is set in stone. Update docs if you find + something not up to date. + +## Jun 27, 2023 + +**Host (pronouns):** Sascha Grunert (he/him) + +**Attendees (pronouns):** + +- Stephen Augustus (he/him) + +- Mauren Berti (she/her) + +- Adolfo García Veytia (he/him) + +- Verónica López + +- Ronit Banerjee (he/him) + +- Joseph Sandoval (he/him) + +- Shivanshu Raj Shrivastava (he/him) + +**Note Taker (pronouns):** + +- Adolfo/Stephen + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Ronit Banerjee + + - Mauren Berti + +- Subproject updates + + - Release Engineering + + - Issues with container images signatures: working on a new + tooling + prowjob to audit the broken signatures + + - Image references used in signatures now use registry.k8s.io + + - Release Team + + - \[Grace in absentia\] Enhancements Freeze stats: + + - 59 (!!) tracked out of 68 enhancements + + - Highest rate of tracked to enhancements I’ve ever seen 🎉 + + - Starting 1.29, the bug triage team will be merged into the CI + Signal team. + + - The + [votes](https://kubernetes.slack.com/archives/C2C40FMNF/p1686765443006049) + are in and the new team will be called **Release Signal Team** + + - Currently working on docs for the merge team + +** +Open Discussion:** + +- KubeCon Maintainer track opened. Are you interested to speak about + anything SIG Release related? Feel free to reach out to the SIG leads! + :) + +## Jun 13, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Rey Lejano (he/him) + +- Grace Nguyen (she/her) + +- Matt Trachier (he/him) + +**Note Taker (pronouns):** + +- Rey Lejano (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - Go updates done - + [https://github.com/kubernetes/release/issues/3099](https://github.com/kubernetes/release/issues/3099) + + - \[Jeremy\] Release branch and master branch are updated, last of + cherry-picks merged last night. Weird issue were we had to + update go sum files which required an api approver to approve + every cherry-pick. Does it make sense to document in template? + + - \[Marko\] Yes it makes sense to update and provide an example + with link to PR + + - Patch Releases + + - Who wants to manage them? + + - \[Jim and Jeremy\] Patch releases target date is tomorrow, Jim + can help out tomorrow and can pair up with someone. Jeremy can + kick it off in the morning and tag team with Jim + + - \[Marko\] OBS updates for alpha cut + + - Had some flaky tests but it was fixed, OBS stuff went well + with 1.28 alpha 2 cut and worked. Still need to work on how to + handle cri tools and Kubernetes cni + + - Release Team + + - 66 enhancements! + + - Enhancements Freeze is on Thursday + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- \[Grace\] Speaking at CNCF Toronto meetup about Release Team and + SIG-Release + +## Jun 6, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Rey Lejano (he/him) + +- Grace Nguyen (she/her) + +- Meha Bhalodiya (she/her) + +- Dipesh Rawat (he/him) + +- Amir Monfared + +- Arnaud Meukam (he/him) + +- Christopher Hanson (he/him) + +**Note Taker (pronouns):** + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + + - \[Marko\] Cosign V2 + + - Carlos is mostly working on this, seems to be going well. + + - \[Marko\] OBS + + - merged core implementation last week, obs stage and obs + krel-release + + - Can publish all core packages now + + - Will do a an alpha2 release to test things this week + + - Cherry Pick Deadline Friday - get them in + + - \[Jeremy\] Go Updates: PRs open in draft form and non draft form, + will retest first PR that will kick off things like image + promotion then will open last wave of things, then will hear about + this soon and images will be available soon and be available + before next alpha release + + - \[Marko\] still have some issues to figure out with OBS like with + cri-tools and Kubernetes CNI + + - Release Team: + + - Enhancements Freeze next week on the 16th + + - Currently at 32 enhancements + + - Grace has reached out to all the SIG channels and will reach out + on the chairs-and-techleads channels + + - Alpha 2 scheduled for June 8th to account for Go release + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- **\[Jeremy\]** can we back port distroless-iptables images to other + release branches? + + - There is a CVE related to OpenSSL versions that is fixed in later + releases (1.27) + + - \[Jeremy\] How do people feel about cherry-picking change to have a + lower CVE footprint + + - [https://kubernetes.slack.com/archives/CJH2GBF7Y/p1685561727920539](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1685561727920539) + +- \[Rey\] Are we switching k/k master to main + + - \[Jeremy\] AI to ping Carlos about this enhancement + + - \[Arnaud\] AI to ping Contribex since there was concern from a SIG + about the switch + +## May 30, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Sascha Grunert (he/him) + +- Grace Nguyen (she/her) + +- Carlos Panato (he/him) + +- Rohan Sasne (he/him) + +- Jim Angel (he/him) + +- Jeremy Rickard (he/him) + +- Verónica López (she/her) + +**Note Taker (pronouns):** + +- Joseph Sandoval + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Rohan Sasne - new to the community. Looking to get involved with the + k\community. + +- Subproject updates + +- - Release Engineering + + - \[Carlos\] OBS update. Upgrade for signing. Using Co-Sign v2. Krel + is updated to sign with v2. Working on Promo tools. Failure with + E2E testing. Prow cluster is not happy. + + - \[Marko\] Tried migrating jobs to EKS. None of the jobs are + passing on GKE. + https://github.com/kubernetes/test-infra/issues/29622 + + - Release Team + + - \[Grace\] + [Enhancements](https://github.com/orgs/kubernetes/projects/140/views/1) + number is low. + + - Only 20 enhancements at the moment. Outreach to individual SIG + channels. + + - \[Jeremy\] All you can do is remind folks about enhancement + deadlines. + + - \[Grace\] waiting on sig-windows to clear a [failed + test](https://kubernetes.slack.com/archives/CV3BA6X8D/p1685330960844329) + for alpha cut today + + - Arnaud will cut the release today. + + - \[Grace\] Bug-triage and CI-Signal will merge for 1.29. Looking + for name suggestions! + + - + + - \[Grace\] Looking for host key for new zoom link :D + + - \[jeremy\] will update today + + - \[Marko\] Release schedule. Can we adjust the release schedule for + OBS? Marko is willing to drive the release. It will be the first + release using OBS. + [https://kubernetes.slack.com/archives/CV3BA6X8D/p1685330960844329](https://kubernetes.slack.com/archives/CV3BA6X8D/p1685330960844329) + + - \[Grace\] Do we have branch manager shadows? + + - \[Jeremy\] Mark Rosetti + +** +Open Discussion:** + +- \[Marko\] OBS updates + + - KREL OBS specs are done. Still in progress KREL OBS stage and + release. + [https://console.cloud.google.com/cloud-build/builds/4fc2cb3b-81ee-4336-8834-1e8414aa43c0?project=kubernetes-release-test](https://console.cloud.google.com/cloud-build/builds/4fc2cb3b-81ee-4336-8834-1e8414aa43c0?project=kubernetes-release-test) + for a view of what the logs will look like. (Only release managers + can view). Testing and tooling are next. Marko has started working + on tooling. Next week we can do another alpha. Feedback is needed on + how testing should look like. The current testing is using a tool + from Kubermatic. Is it ok to use a tool that is OpenSource but not + part of SIG-release? + + - \[Jeremy\] Ask Sig-testing if that is ok? + + - \[Adolfo\] I don’t see this being an issue. We are using OBS. An + external tool. + + - \[Carlos\] We can fork the tool if we think it's a concern. + +- \[Marko\] Do we want to proceed with [Remove "Kubernetes Source + Code" from being published on GitHub + Release](https://github.com/kubernetes/release/pull/2780)? + + - \[Marko\] It’s been a few months. I would consider merging this PR. + + - \[Adolfo\] Is this the tarball on GH releases? What do we want to + put on the GH release page? Started thinking on how to replicate our + releases to the GH page. The community expects the releases to be + there. + + - \[Marko\] If we can’t move artifacts. I don’t think this is a + blocker for the PR. Dropping a tarball should be fine. + + - \[Adolfo\] I was asking about a year ago to see if anyone is + dependent on the tarball. The PR has been open for a while. + + - \[Jeremy\] We can bring it back if we get comments when we remove + it. + +- \[Jeremy\] will be regenerating zoom links today + + - \[Jeremy\] Grace I’ll send you host codes. Please don’t share links + with passwords. + +- \[Jeremy\] please review + [https://github.com/kubernetes/community/pull/7287](https://github.com/kubernetes/community/pull/7287) + + - \[Jeremy\] SIGS have responded and approved the reformation of + WG-LTS. Tech leads please review. + +- \[Rohan\] Archive meeting notes for Kubernetes SIG Release meeting + [https://github.com/kubernetes/sig-release/issues/2255](https://github.com/kubernetes/sig-release/issues/2255) + + - \[Rohan\] Can I help and create a PR for this issue? + + - \[Jim\] I can pair up with you on this issue. + +## May 23, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Ricky Sadowski (he/him) + +- Rishit Dagli (he/ him) + +- Jim Angel (he/him) + +- Patryk Przekwas + +- Michael Levan + +**Note Taker (pronouns):** + +- Jim Angel (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Patryk - k8s infra, interested in obs + + - Michael - docs release team shadow + +- Subproject updates + + - Release Engineering + + - \[Marko\] Patch releases went out last week + + - Thanks Veronica, Jim, and Jeremy for taking care of those + releases! + + - The only minor hiccup we had is sending announcement because + SendGrid blocked Jim for some reason + + - Thanks Jeremy for taking care of announcements and sending an + apologize letter + + - \[Marko\] OBS implementation going very well so far, presentation + will be held later today + + - \[Marko\] Work in progress by Carlos to migrate to cosign v2 + + - Changes in flight for krel / release tooling + + - Release Team + + - \[Grace in absentia\] last meeting I proposed potential release + timeline changes, that will not be happening as the enhancements + and implementation periods are similar in length already. + + - \[Grace in absentia\] Shadows all confirmed and onboarding. + + - \[Jeremy\] Schedule is posted, if questions, ping Grace / slack to + have async convos. + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- \[Marko (15 min)\] [OBS + presentation](https://docs.google.com/presentation/d/1nYvvEGvylbTIFWnR5ecjDalAJCUlHUwEjkx8HosvhLo/edit?usp=sharing) + + - Overview (moving Google -\> OBS/Suse WIP for implementation) + + - Significant changes / recap today’s state: + + - Package specs to build - complete + + - OBS projects / repos - complete + + - cmd to generate specs and archives for OBS - complete + + - OBS 1:1 mapped to repos (see recording for detail) + + - Limitation for building packages requires us to build packages in + one project and publish in another (:\ vs :\:build) + + - If we used the same repo, each version would increment / build a + new source + + - CNI and cri-tools are not versioned like k8s, which requires + publishing each time for a new project. + + - Mission statement: “We want to make managing and publishing packages + possible for all kubernetes subprokects in hte same way as for core + packages” + + - Universal platform / tooling that works with all projects. + + - Future state: krel obs wrapper (specs / stage / release) + + - Demo of templates for specs (for reuse - in recording) + + - OBS operational modes (similar to how we build today) + + - **Integrated**: via krel stage / release + + - **Standalone**: via cloud build job to build and publish backages + + - (watch the recording for more info…..) + +## 16 May 2023 + +**Host (pronouns):** Sascha Grunert (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Leonard Pahlke (he/him) + +- Grace Nguyen (she/her) + +- Jim Angel (he/him) + +- Matt Trachier (he/him) + +- Shivam Singh (he/him) + +- Ricky Sadowski (he/him) + +- Scott Dodson (he/him) + +- Amir Monfared(he/him) + +- Stephen Augustus (he/him) + +- Atharva Shinde (he/him) + +**Note Taker (pronouns):** + +- Marko Mudrinić (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Mofi from 1.28 Release Team + + - Anhelina from 1.28 Release Team + + - Scott + + - Shivam + +- Subproject updates + + - Release Engineering + + - Kubernetes patch releases for May are scheduled for Wednesday + + - Veronica and Jim will take care of those releases + + - OBS implementation is back in progress (we’ll walk the board + later) + + - Marko is working on this again + + - Release Engineering and Release Team project boards that we had in + the past are now closed in favor of per-enhancement board (e.g. + for packaging and SLSA) + + - Release Team + + - Kicked off this week - first meeting + [tomorrow](https://kubernetes.slack.com/archives/C2C40FMNF/p1684245126254109)! + + - First meeting in less + + - Reducing Enhancements Freeze by a week to allow more time for Code + Freeze + + - Enhancements period was 6 weeks, but will be 5 weeks + + - 5 weeks until code freeze after that + + - Calendar to be finalized soon + + - Community feedback is that it’s better to have more time for + development + +** +Open Discussion:** + +- \[Sascha\] Possible image signing enhancements: + + - Supporting of recursive signing of multi-arch images in kpromo: + [https://github.com/kubernetes-sigs/release-sdk/pull/193](https://github.com/kubernetes-sigs/release-sdk/pull/193) + [https://github.com/kubernetes-sigs/release-sdk/releases/tag/v0.10.1](https://github.com/kubernetes-sigs/release-sdk/releases/tag/v0.10.1) + [https://github.com/kubernetes-sigs/promo-tools/pull/868](https://github.com/kubernetes-sigs/promo-tools/pull/868) + + - RFC: Being able to set the docker-reference to registry.k8s.io in + image signatures: + [https://github.com/sigstore/cosign/pull/2984](https://github.com/sigstore/cosign/pull/2984) + Right now we still point them to the actual mirror, which makes + signature validation hard: + \> cosign verify \\ + --certificate-identity + krel-trust@k8s-releng-prod.iam.gserviceaccount.com \\ + --certificate-oidc-issuer https://accounts.google.com \\ + registry.k8s.io/kube-apiserver:v1.27.1 \| jq + .\[0\].critical.identity + … + {"docker-reference":"europe-southwest1-docker.pkg.dev/k8s-artifacts-prod/images/kubernetes/kube-apiserver"} + + + + - \[Sascha\] We don’t use recursive signing, so in case of multi-arch + images, we sign the manifest, but don’t sign concrete images. Added + support for recursive signing to fix this, will be introduced in + promo-tools soon. Hopefully it’s not going to increase the promotion + time, but we’ll properly sign and validate images + + - \[Sascha\] We want to be able to replace docker-reference with + registry.k8s.io, but this requires changing cosign, but it’ll be + better for users. We’ll see how it turns out, depends on if the + change lands in cosign + + - \[Sascha\] Those are considered small changes, we don’t need to + change our current enhancement + + - \[Jim\] What do we need to move with this? + + - \[Sascha\] Proposed change as a RFC, we need to wait for feedback. + Without it, we don’t know how to make it better for users. If you + make it really strict, it’s wrong, because the reference is not + matching the image that's being verified + + + +- \[Marko\] Walking the [Packaging + board](https://github.com/orgs/kubernetes/projects/137/views/1) + + - Looking for feedback on board look and organization, and issues + +- \[Marko\] Status of + [https://github.com/kubernetes-sigs/promo-tools/pull/792](https://github.com/kubernetes-sigs/promo-tools/pull/792) + + - What do we need to merge this? + + - What’s the current overall status of promo-tools? + + - \[Adolfo\] We’re in a pretty good state, instability times are gone, + but we might look into improving resiliency. We should definitely + discuss retries, but we don’t have any issue for that. There was + some discussion with SIG K8s Infra, but that requires rewriting + promotion from scratch. We’re still considering where to take it + from here, but feeling is that it’s relatively stable + +- \[Marko\] Should we consider archiving the SIG Release meeting + minutes? + + - There are some severe performance issues with this document + especially when on Zoom + + - SIG Contribex is doing something similar every year: + [https://github.com/kubernetes/community/tree/master/sig-contributor-experience/meeting-notes-archive](https://github.com/kubernetes/community/tree/master/sig-contributor-experience/meeting-notes-archive) + + - \[Stephen and Jim\] There are docs in the README in SIG Contribex + directory, we can follow that link + + - General consensus is yes + + - Jim to take a look at this + + - [https://github.com/kubernetes/community/tree/master/sig-docs/meeting-notes-archive](https://github.com/kubernetes/community/tree/master/sig-docs/meeting-notes-archive) + +- \[Marko\] Subscription to debian-security mailing list + + - We’re currently subscribed to debian-security mailing list via our + release-managers mailing list + + - We’re receiving (in my opinion) a lot of irrelevant emails from them + + - Should we consider unsubscribing from that mailing list or finding + another way to collect those emails/alerts? + + - \[Stephen\] We’re concerned about Debian image updates. Do we have a + process to regularly update Debian images? What should be a signal + to update our images? + + - If it’s not providing value, it’s fine, but we should have a + process to build new images + + - Come up with a process to update images every month after patch + releases + + - \[Sascha\] Does it have to align with Go updates? + + - \[Stephen\] No, there might be out of bound updates for Go, and + the cadence in general might be different. As a baseline, doing + it once a month is fine. The idea behind after patch releases is + to reduce risks in patch releases, e.g. we would have a month to + catch potential regressions + + - AI(Marko): Create issue for this and find someone to work on + this + + - Jim and Marko to pair on this, Jim to file an issue + + - \[Matt Trachier\] We have a lot of concerns about CVEs that are + theoretically possible, but not practically. Compliance people are + often concerned if CVEs exist and not if they are exploitable or + not, so updating often can be helpful + +- \[grace\] can a SIG chair approve this typo PR ASAP pls: + [https://github.com/kubernetes/k8s.io/pull/5285](https://github.com/kubernetes/k8s.io/pull/5285) + +## May 9, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Nitish + +- Marko Mudrinić (he/him) + +- Leonard Pahlke (he/him) + +- Siming Weng (he/him) + +- Haitao Chen + +- Dipesh Rawat (he/him) + +- Jim Angel (he/him) + +- Christopher Hanson (he/him) + +- Josh Berkus (he) + +- Heba Elayoty (she/her) + +- Amir Monfared (he/him) + +- Arnaud Meukam (he/him) + +**Note Taker (pronouns):** + +- + +- Jim Angel (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - A few folks attending that are new + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - \[Marko\] Go 1.20.4 / 1.19.9 updates are done + + - \[Marko\] Patch releases are scheduled for the next week + + - Cherry-pick deadline: 2023-05-12 (this Friday) + + - Releases: 2023-05-17 + + - TODO: Need release managers/leads to be determined for cut + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - (Leo) Shadow form closed last week to be finalized with today + (5/9) being the deadline to send out the emails. + + - Grace is working on docs + + - To follow up, there’s an issue with checkboxes required for a + release cycle: + [https://github.com/kubernetes/sig-release/issues/2223](https://github.com/kubernetes/sig-release/issues/2223) + + - We are on track to begin next week for 1.28 + [🎉](https://emojipedia.org/party-popper/) + + - Arnaud will be the branch manager for 1.28 (thank you Arnaud) + [🎉](https://emojipedia.org/party-popper/) + + - Any update on the shadow selection process. It was mentioned + notifications would be send out latest until: 9th May 2023 (Today) + + - \[Leo\] we will confirm the team by the end of the day + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- \[Marko\] Migration of SIG Release and Release Blocking jobs to EKS + Prow build cluster + + - Thanks to Carlos for taking care of many of our projects! + + - Some of release blocking jobs will get migrated as part of + [https://github.com/kubernetes/test-infra/pull/29410](https://github.com/kubernetes/test-infra/pull/29410) + + - Merged May/9/2023 + + - Please let SIG K8s infra folks know if you notice anything strange + +- \[jeremy\] annual report + [https://github.com/kubernetes/community/pull/7214](https://github.com/kubernetes/community/pull/7214) + + - Ask: review ASAP and need to merge ASAP + +- \[jeremy\] Wg-lts + [https://github.com/kubernetes/community/pull/7287](https://github.com/kubernetes/community/pull/7287) + + - Ask: please review/comment + + - Jim: Have we identified what has changed since old WG-LTS (no + opinion, just asking): + + - [https://github.com/kubernetes/community/issues/7259](https://github.com/kubernetes/community/issues/7259) + + - Contains discussion around changes / differences + + - General consensus was: There are changes worth exploring but + with other leaders driving the WG. + +- \[Nitish\] Discussions about SIG-Spotlight blog on Sig-Release + + + +- \[Nitish\] We approach chairs of sigs to ask questions about + contributing / getting involved and noticed SIG-Release has not had a + spotlight. + + - Open questions? + + - Who can we reach out to for working with (vs. the ephemeral + release team)? + + - Should the blog be divided into two parts? And what is the process + to follow before publishing a blog. + + - \[Jeremy\] Depends on what you’re trying to cover in the blog + + - [https://github.com/kubernetes/community/tree/master/sig-release](https://github.com/kubernetes/community/tree/master/sig-release) + + - Two subprojects + + - RelEng + + - Cutting branches / tooling + + - Release Team + + - Rotates every cycle and those are folks that are staffed and + running a release + + - It makes sense to cover both (wide open paths) + + - \[Leo\] Over the past few conferences, we talk about releases / + teams / engineering and putting into a blog would be good. + + - Would this document require updates or be a blog that doesn’t + require updates? + + - No deadline for updating, but have updated in the past. Then + events driven for changes. + + - \[Jeremy\] Start with tech leads and maybe someone from the current + release team to work with. + +- Slack discussion: + [https://kubernetes.slack.com/archives/C2C40FMNF/p1683306707135349](https://kubernetes.slack.com/archives/C2C40FMNF/p1683306707135349) + + + +- There has been no Spotlight blog on Sig-Release yet. + + + +- Example: + [https://kubernetes.io/blog/2021/07/15/sig-usability-spotlight-2021/](https://kubernetes.io/blog/2021/07/15/sig-usability-spotlight-2021/) + + + +- \[jeremy\] new project boards, should we review them? + + - \[marko\] we should first schedule a meeting to create items + + - \[jeremy\] should we schedule those now? + + - \[marko\] carlos/adolfo are away till next week, marko will start + creating issues for packaging + + - \[veronica\] xander was interested in PMing + +- \[heba\] good issues for people starting? + + - \[jeremy\] we have good first issues in some repos. Most things will + be in tooling repos. + + - \[jim\] repos documented here: + [https://github.com/kubernetes/community/tree/master/sig-release#release-engineering](https://github.com/kubernetes/community/tree/master/sig-release#release-engineering) + + - + +**(Optional) Walk the Board:** + +- Project board review: + [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +- Incoming issue and PR triage + + - [SIG Release issues + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [SIG Release PRs + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [Kubernetes + issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +## May 2, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Sascha Grunert (he/him) + +- Leonard Pahlke (he/him) + +- Joseph Sandoval (he/him) + +- Rishit Dagli (he/him) + +- Xander Grzywinski (he/him) + +- Marko Mudrinić (he/him) + +- Matt Trachier (he/him) + +- Mickey Boxell (he/him) + +- Mahamed Ali + +- Amir Monfared + +- Christopher Hanson (he/him) + +- Rayan Das (he/him) + +- Carlos Panato + +- Ashwin Kumar Uppala (He/Him) + +**Note Taker (pronouns):** + +- Joseph + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Mickey Boxell + + - Chris - RXM - trying to get more involved in the Kubernetes project. + + - Amir - Second time joining the meeting. + + - Tobo - first time attending. Looking to get more involved in the + Kubernetes community. + + - Rayan - new to the meeting. Infracloud. Trying to get involved with + SIG-Release. + + - Ashwin - Coming back to learn more getting involved with + SIG-Release. + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - Go 1.20.4 / 1.19.4 update: + [https://github.com/kubernetes/release/issues/3025](https://github.com/kubernetes/release/issues/3025) + + - (Marko) Releases going out today. + + - K8s.gcr.io freeze is live 🎉 + + - (Mahamed) PR was merged last week after Kubecon. No images being + published to the old registry. + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - (Leo) 5/2 is the last day to apply to the release team. Shadow app + closes in 6 hours. A week to select the shadows and then start the + release cycle. (application form: + https://bit.ly/k8s-shadow-application-1-28) + + - Release Team Shadow Application Opens: 13th April 2023 + + - Release Team Shadow Application Closes: Tuesday 2nd May 2023 + (23:59 UTC) + + - Release Team Shadow Notifications send out latest until: 9th May + 2023 + + - Release Cycle Start: 15th May 2023 + + - Release Cycle End: 15th August 2023 (approximate!) + + - (Jeremy) Branch manager needs to be selected. + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- \[Marko\] Update on generating release branch jobs + + - PR with more details: + [https://github.com/kubernetes/test-infra/pull/29387](https://github.com/kubernetes/test-infra/pull/29387) + + - Issue to track fixing annotations: + [https://github.com/kubernetes/test-infra/issues/29388](https://github.com/kubernetes/test-infra/issues/29388) + + - Issue to track updating handbook: + [https://github.com/kubernetes/sig-release/issues/2227](https://github.com/kubernetes/sig-release/issues/2227) + + - Proposal to rewrite Python scripts in Go: + [https://github.com/kubernetes/test-infra/issues/29390](https://github.com/kubernetes/test-infra/issues/29390) + + - (Marko) There were some efforts to get rid of the 1.23 jobs. + Issues created that need feedback which are listed above. The 1.28 + release team will need to update the handbook. We should rewrite + the python scripts. They are wrappers around Go tools. We rewrite + in Go in a single library. Take a look at the issues and provide + feedback. + + - (Jeremy) Any reservations about this effort? Most in agreement. + + - (Matthias) In Krel or another binary? + + - (Marko) Still being determined. + +> Slack thread: +> [https://kubernetes.slack.com/archives/C2C40FMNF/p1682754798900429](https://kubernetes.slack.com/archives/C2C40FMNF/p1682754798900429) + +- \[Mickey\] Update to release handbooks + + - Breaking out the release team role handbooks + ([example](https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/docs)) + into two docs: a lead handbook and a shadow handbook or retitling + the current handbooks + + - If I were a current or prospective shadow, I wouldn’t think to look + for my responsibilities in a doc entitled “Kubernetes \[Insert + Team\] Lead Handbook”. + + - (Jeremy) It seems like a reasonable change. Retitling makes the + most sense. + + - (Leo) I like the idea of the change. The title might make it + confusing. Making it clear to new folks will be helpful. Have + everything in one document. + + - (Verónica) Important to show our shadows all aspects of the roles. + Yes to changing the title. + + - (Matt) It is difficult to get more involved without knowing what + I”m getting into. + + - (Mickey) What about a separate section for a shadow? + + - (Matt) This would be helpful + + - (Verónica) After a release cycle having shadows provide their + perspective on time commitments on the roles. + + - (Jim) Newcomers have invaluable perspectives. There might be an + area to improve information discovery. Creating a K8s.dev to help + make the information more discoverable. + +**(Optional) Walk the Board:** + +- Project board review: + [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +- Incoming issue and PR triage + + - [SIG Release issues + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [SIG Release PRs + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [Kubernetes + issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +## 18 Apr 2023 - KubeCon Contributor Summit Special + +**Where:** E108 - 1st Floor - Congress Complex RAI Amsterdam +([https://sched.co/1LXGs](https://sched.co/1LXGs)) + +**When:** 3:40 pm - 4:35 pm + +**Hosts (pronouns)** + +- Sascha Grunert (he/him) + +- + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Jeremy Rickard + +- Angelos Kolaitis + +- Brad McCoy (he/him) + +- Adolfo García Veytia + +- Carlos Panato + +- Jeremy Rickard (he/him) + +**Note Taker (pronouns):** + +- Joseph Sandoval + +** +Agenda:** + +- SIG Release Introduction meeting from Veronica and Marko tomorrow + (Wednesday) at 2:30 pm + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +- \[Marko\] OBS - Need to enable OBS in the release pipeline. If we can + get some help with the packages. Get some additional help from outside + resources in 1.28. No blockers. + + - \[Adolfo\] - What type of help? + + - \[Marko\] - Help is needed with implementing the API. We need to do + some planning and prioritization and create the issues. + + - \[Sascha\] - Do we have an updated KEP? + + - KEP: + [https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1731-publishing-packages](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1731-publishing-packages) + +- LTS Working Group will probably restart: + [https://docs.google.com/document/d/1C7utWMjGpDcBc5TB_ooddERsid59sFLC7gqLzyS5aKY/edit?usp=share_link](https://docs.google.com/document/d/1C7utWMjGpDcBc5TB_ooddERsid59sFLC7gqLzyS5aKY/edit?usp=share_link) + + - \[Jeremy\] There was a previous LTS effort. This effort will be + re-kicked off. An unconference session regarding this was done + during the Contributor Summit. The takeaway will come up with a + north star to what they are working for. 2 year LTS and skip level + upgrades. It will be a WG as its cross cutting across SIGs. This + will be a long process. If interested you can add your name to the + link above. Previous LTS information is contained in the link above. + + - What does LTS mean? + + - \[Jeremy\] It can mean various things. Longer support. Version being + in support for a longer time. + + - \[CiCi\] Will that involve extension of support? + + - \[Jeremy\] Is it just security fixes? Bugs? The community will work + on defining this. Does a version get LTS and extend the patching + cycle? Nothing is decided at the moment. There was no opposition to + the concept from the working session. Consensus on restarting the + effort. + + - \[Sascha\] How did we come back to this working group? + + - \[Jeremy\] Microsoft announcement on LTS started the discussion + again. Several providers were interested as well. + + - \[Adolfo\] There are several efforts from different vendors on LTS. + It's going to involve a lot of work. + + - \[Jeremy\] It's important to think about the problems to be solved. + Jordan Liggitt raised some problems that couldn’t be overcomed in + the previous LTS efforts. We support a forked version of Kubernetes. + Most cloud providers run forks. + + - \[Adolfo\] I was referring to release eng tooling that will be + affected. + + - + +- SLSA + + - \[Adolfo\] PR is ready for the Prow work. How can we ensure that all + projects can benefit from this? Find a way to enable this for the + different teams. When we promote images all projects received signed + images for free. No easy way to connect the Prow jobs with the + artifacts. Also help people with attestations. It needs some work + with the Image Promoter. Carlos and Adolfo will speak about this on + Thursday at Kubecon. + + - \[Marko\] There is a new version of SLSA. Will this affect the work + we have done? + + - \[Adolfo\] We have to update a couple of tools. Need to update the + SBOM tool as well. New version of SLSA is open for comments at the + moment. No urgent need to update. + + - \[Sascha\] The changes with the image promoter? + + - \[Adolfo\] One year ago we were promoting to three mirrors now we + are promoting to over 20+. We were running into rate limits. We did + slice the time for the image promotion. We leave the promoter as is. + + - \[Veronica\] Have you talked to SIG-K8s-infra regarding these + changes? This was discussed last week. Whoever cuts the release has + a difficult time. + + - \[Adolfo\] This will need to go under a KEP since this changes how + we promote images. We are cross collaborating with SIG-K8s-Infra. + + - \[Sascha\] We should update the KEP for 1.28. + + - \[Adolfo\] We would like to do it during this cycle. + +- \[Marko\] Should we consider planning sessions? + + - We would have sessions with Lauri. Our project board is a bit of a + disaster. We discussed OBS. SLSA is in discussion. Perhaps every + month try to keep our board up to date. We could do a dedicated + session on a topic. Make sure the issues make sense. + + - \[Xander\] I can help with this. If there is desire I can help with + this. + + - \[Veronica\] The sessions we used to have were very useful. Those + sessions were improvements to the release process and tooling. Now + our tooling has evolved alot. We don’t have enough hands to keep + this up to date. My proposal is if we have sessions then we need to + have ownership. Alot of people were interested in the pillars but + had to drop off for various reasons. For example SLSA we would have + someone assigned to that pillar. Like Adolfo. They would keep the + roadmap for that pillar up to date. + + - \[Sascha\] + + - \[Marko\] The person doesn’t have to be a planning expert. The + person should have context but don’t need to be technical. + + - \[Adolfo\] If we feel someone is a sole owner of a pillar or effort + like SLSA. We need to make sure we expand that to others. + + - \[Marko\] At times there is no way to find how to work on an item. + + - \[Sascha\] Can we create project boards? What about removing or + getting rid of existing project boards? + + - \[Marko\] Action item: I can help run some dedicated sessions. We’ll + start after Kubecon. + + - \[Jeremy\] Should we update the roadmap? It would help with the + annual report. + + - \[Xander\] GH boards make life easier. + + - \[James\] GH boards were celebrated in the 1.27 retro. + + - Project boards + + - Packaging: + [https://github.com/orgs/kubernetes/projects/137](https://github.com/orgs/kubernetes/projects/137) + + - SLSA: + [https://github.com/orgs/kubernetes/projects/138/views/1](https://github.com/orgs/kubernetes/projects/138/views/1) + +- \[Marko\] Version markers issue status + + - We stop cutting RC.0 releases. + + - After v1.26.1 it would look like v1.26.2-rc.0.9-hash + + - v1.26.1-9-hash + + - This change broke the tooling. Tests are invalid. We might have the + same issue with cutting 1.28.1. We should take a look at this. Not + having RC.0 was a huge improvement. But we are still having issues. + + - \[Adolfo\] Tag a repository but don’t build it. + + - \[Cici\] It results in the risk that having tooling for package + fetching by tags + + - \[Marko\] We should try once again with the logic. + + - \[Cici\] Are we only cutting RC.0? + + - \[Marko\] For minor releases we should cut with RC.1. + + - \[Sascha} There is an open issue on K/K. + +## + +## Apr 11, 2023 + +**Host (pronouns):** Verónica López (she/her) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Xander Grzywinski (he/him) + +- James Laverack (he/him) + +- Jim Angel (he/him) + +- Ashwin Kumar Uppala (he/him) + +- Cici Huang (she/her) + +- Matt Trachier (he/him) + +- Joseph Sandoval (he/him) + +**Note Taker (pronouns):** + +- Marko Mudrinić (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - April patch releases + + - Testgrid is having a party + + - 1.25 and 1.24 blocking dashboards are red + + - 1.25: + [https://testgrid.k8s.io/sig-release-1.25-blocking](https://testgrid.k8s.io/sig-release-1.25-blocking) + + - 1.24: + [https://testgrid.k8s.io/sig-release-1.24-blocking](https://testgrid.k8s.io/sig-release-1.24-blocking) + + - \[Veronica\] We should have a better way to track this so that + we don’t figure this out a day before releases + + - We might have to delay patch releases because we shouldn’t + release with jobs being red on blocking + + - \[Jim\] Ben talked about adding arm64 support a few weeks ago, + maybe it’s related to 1.24 failures + + - [https://kubernetes.slack.com/archives/CN0K3TE2C/p1679504185336529?thread_ts=1679463031.954529&cid=CN0K3TE2C](https://kubernetes.slack.com/archives/CN0K3TE2C/p1679504185336529?thread_ts=1679463031.954529&cid=CN0K3TE2C) + + - \[Cici\] We had issue with rc.0 tag, will be shared, but rc.0 + was much before failures started + + - Go updates are done + + - \[Puerco\] New version of promoter + + - We had a successful rc.1 release! + + - Thanks to everyone who helped! + + - We’ll see today how it performs for 1.27.0 + + - \[Veronica\] What are the first/next steps? + + - \[Puerco\] Fixing signatures. We’ll see about this after + KubeCon. Next steps will involve sitting together and seeing + how we can do it more efficiently + + - \[Veronica\] Is there any work that’s pending from Arnaud and + Mahamed? + + - Some work might be needed related to freeze after KubeCon + + - Some conversation is needed how we’ll proceed with the freeze + + - Update from Adolfo doesn’t affect this + + - \[Puerco\] For real real freeze we need to fix the signatures + + - \[Arnaud\] The redirect is done, so this is not necessarily + needed + + - Related PR: + [https://github.com/kubernetes-sigs/promo-tools/pull/792](https://github.com/kubernetes-sigs/promo-tools/pull/792) + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - 1.27.0 release is today! + + - \[Cici\] The release is going great! + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- SIG Release session at the Kubernetes Contributor Summit: + [https://sched.co/1LXGs](https://sched.co/1LXGs) + +- 1.27 Retro session at the Kubernetes Contributor Summit + \[[link](https://kcseu2023.sched.com/event/1LXJH/127-release-team-retrospective)\] + +- All sig-release talks/meetings for KubeCon: + + - [https://kubernetes.slack.com/archives/C2C40FMNF/p1681227034588669](https://kubernetes.slack.com/archives/C2C40FMNF/p1681227034588669) + +**(Optional) Walk the Board:** + +- Project board review: + [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +- Incoming issue and PR triage + + - [SIG Release issues + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [SIG Release PRs + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [Kubernetes + issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +## Apr 4, 2023 + +**Host (pronouns):** Verónica López + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Xander Grzywinski (he/him) + +- Amir Monfared(he/him) + +- Jim Angel (he/him) + +- Sascha Grunert (he/him) + +- Adolfo García Veytia (he/him) + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Welcome Amir! + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - Go updates + + - + + - Patch releases next week + + - Veronica and Marko will take care of cherry-pick reviews + + - Jim: can I help with cherry-pick reviews? + + - Veronica: we had idea to have a spreadsheet to collect + cherry-picks and provide some context there + + - Marko: Release Manager Associates cannot approve cherry-picks, + but we can still go through them together. We have an issue to + track allowing Associates to approve cherry-picks: + [https://github.com/kubernetes/sig-release/issues/2083](https://github.com/kubernetes/sig-release/issues/2083) + + - RC.1 on Thursday and final on Tuesday + + - There’s one PR in promoter that added limiter to one more place, + but sigstore doesn’t have limiter so we still expect issues + + - We’ll probably end up in the same situation + + - We’ll start testing the job to retroactively sign images, but + we cannot do that because of the registry rollout + + - We could try new promo-tools release to include the latest + changes and hope we get luckily + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - Rc1 cut moved to Thursday + + - Releases next week + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +- Shadow application survey + + - We’re in process of nominating leads + + - The applications will open a week or two after KubeCon + + - Jim: recommend to attend SIG Release and Release Team meetings to + stay in touch and up to date with all the news, deadlines… + + - Veronica: also make sure to follow our Slack channels, and feel free + to ping Veronica if you have questions + +- Produce structured test output for repo's tests + + - k/release: + [https://github.com/kubernetes/release/issues/2993](https://github.com/kubernetes/release/issues/2993) + + - k-sigs/promo-tools: + [https://github.com/kubernetes-sigs/promo-tools/issues/804](https://github.com/kubernetes-sigs/promo-tools/issues/804) + +**(Optional) Walk the Board:** + +- Project board review: + [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +- Incoming issue and PR triage + + - [SIG Release issues + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [SIG Release PRs + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [Kubernetes + issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +## Mar 28, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- [Adolfo García Veytia](mailto:puercozon@gmail.com)(he/him) + +- [Grace Nguyen](mailto:nng.grace@gmail.com) (she/her) + +- Laura Lorenz (she/her) + +- Benjamin Elder (he/him) + +- [Ashwin Kumar Uppala](mailto:kumarashwin2603@gmail.com)(He/Him) + +**Note Taker (pronouns):** + +- Joseph Sandoval + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Ashwin is checking out how things are working. Joining to follow + along. Observe the meeting with the time he has. + + - Josh Berkus / Sig-Contribex co-chair. + + - Adolfo/Puerco - technical lead + + - Jeremy - does release team stuff + + - Marco - Release Mgr + + - Cici - 1.27 Branch Mgr + + - Mahamed - Contributor to Sig-release and Sig-K8s infra + + - Joseph - Release Mgr Assoc + + - Grace - Works with the release team. Branch Mgr assoc + + - Jim Angel - Release Mgr Assoc + + - Laura Lorenz - 1.27 release team CI Signal lead. Updates on + CI-Signal stuff + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - \[Marko\] Things are going mostly fine. Cherry pick deadline is + April 7. April 12 is the patch cut. Some issues with signatures. + Rel-Manager channel on K8s slack has the details. Adolfo and Ben + have been helping on it. Migration from registry changes are the + reason why we have not fixed the missing signatures. + + - \[Adolfo\] Missing signatures is due to the promoter and rate + limiting.Create some jobs to do dry runs to make sure we have + proper credentials. Issue to track is + [here](https://github.com/kubernetes/release/issues/2962). + + - \[Marko\] We will wait for the Google team to give us the green + light. + + - \[mahamed\] PR to complete the image freeze. + [https://github.com/kubernetes/k8s.io/pull/5035](https://github.com/kubernetes/k8s.io/pull/5035) + + - \[Mahemed\] There is a snag related to inconsistencies with + signatures. Ben suggested we delay freezing. The freeze has less + value. PR is ready. The work is complete. + + - \[puerco\] Update on Bad Signatures - + [https://github.com/kubernetes/release/issues/2962](https://github.com/kubernetes/release/issues/2962) + + - \[cici37\] Did RC0 cut last week. Quick update on post branch + creation tasks + + - Mos of the work has + completed([kubernetes/kubernetes#116919](https://github.com/kubernetes/kubernetes/pull/116919), + [kubernetes/test-infra#29124](https://github.com/kubernetes/test-infra/pull/29124), + [kubernetes/test-infra#29125](https://github.com/kubernetes/test-infra/pull/29125)) + + - Pending for review: + [https://github.com/kubernetes/test-infra/pull/29137](https://github.com/kubernetes/test-infra/pull/29137) + + - Still have one PR under 1.27 milestone for fix: + [https://github.com/kubernetes/kubernetes/pull/116575](https://github.com/kubernetes/kubernetes/pull/116575) + + - \[Cici\] Ran into the rate limiting issue again. PR to fix the + tagging issue. We should have separate labels to merge the PR. + Current milestone and release blocker. PR for review on tagging + branch jobs. Expect 1.28 job to prepare the release branch. + Create an issue to investigate further. Branch creation for + 1.27. Two of the previous releases 1.22/1.23 are EOL. The + releases are past the EOL. Should we remove the jobs for those + releases? + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + +- \[lauralorenz/jberkus\] follow up on CI Signal report feedback process + + - [Discussed 3/7](#t5hvq1vi90e9), follow up was [this + issue](https://github.com/kubernetes/community/issues/7176) for + help from comms to discuss [this survey + draft](https://hackmd.io/@on2gXqykTESI4N6quhc9pw/rkKylZuy3) for + feedback on + + - \[Laura\] Weekly email report from CI-Signal. Template with some + duplication and extra information. It can vary over releases. I + wanted to find a way to streamline this communication. Above is + the issue and survey draft. + + - At the time we didn’t have much background on this but wanted to + follow up especially after connecting with Josh + + - \[Josh\] I created the original signal report for 1.11. Why do we + need to do a survey? If the team feels like effort to results + ratio is not worthwhile, is to just not do it? It was created at a + different time. A Lot has changed which has lowered its value. + Testgrid was not as good at that time. 1.13 CI-Signal reports were + spent in the red. It was a call to action to create this report. + Testgrid is in much better shape now. Faster response to failures. + Current template doesn’t have that call to action. It was very + prescriptive on actions needed by individuals. We probably don’t + need to do a survey. + + - \[Laura\] How does SIG-Release feel about the email report? If we + don’t have the same surgical details about fixing tests in the + red. We chase down issues very quickly now. We could stop doing it + and go dark. Looking for feedback on how we go forward with this. + + - \[Adolfo\] Let people know in advance if we decide to not do it. + If someone expresses that they need it when we announce we can + keep it. + + - \[Laura\] Everyone seems to feel like we don’t use it or get the + information from Slack. Some on the survey like the email report. + The sample size of the survey is small. + + - \[Adolfo\] Branch managers have enough information and don’t need + the report. We could automate the information. Maybe remove it for + one cycle? + + - \[Jeremy\] Drop from 1.28 and see if we have any issues? + + - \[Laura\] I’ll bring this information back to the release team + retro. + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- \[jeremy\] - Please review annual report + [https://github.com/kubernetes/community/pull/7214](https://github.com/kubernetes/community/pull/7214) + + - \[Jeremy\] The report has information on what we did and stats + regarding PR’s. + +- \[jeremy\] - What if we disabled prow’s merge capability for k/k when + cutting releases (rc.0 maybe when branch cut occurs)? + + - [https://kubernetes.slack.com/archives/CJH2GBF7Y/p1679613751918579?thread_ts=1679588340.415139&cid=CJH2GBF7Y](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1679613751918579?thread_ts=1679588340.415139&cid=CJH2GBF7Y) + + - [https://github.com/kubernetes/kubernetes/pull/116866](https://github.com/kubernetes/kubernetes/pull/116866) + + - \[Jeremy\] The branch FFwdr kicked in and created an issue. We + have seen this in the past. What if we disabled this or added + protection to enforce policy automagically? + + - \[Marko\] This should be fairly easy with a Prow plugin. Patch + releases are also affected by this issue. Patch releases take a + long time. + + - \[Mahamed\] Why do we squash and merge? + + - \[Marko\] Merge keeps the history. + + - \[Mahamed\] Squash and merge is very convenient. + + - \[Adolfo\] The clean history of commits. + + - \[Ben\] If you mention a username they get pings. The blocking + merge commits come from this. If there is any bad behavior. We can + address. Robots are trying to prevent you from doing bad things. + Dependabot includes reference to NPM packages. We may be blocking + more than we need to. For example K8s website might be different. + Have users opt into their PR’s. + + - \[Jeremy\] We had a PR merge during RC.0 cut. What if we disabled + merging during branch cuts? + + - \[Ben\] We only allow certain people to do this? + + - \[Jeremy\] Milestone maintainers are not a small group.. + + - \[Adolfo\] Some may not know and some who do that do this. + + - \[Ben\] Milestone maintainers need to ack. We may then have merges + at different phases. + + - \[Marko\] I don’t think we need milestone maintainers. Milestone + applier will do this for us. We have milestones in the PR’s. We + should evaluate if we need this thing? + + - \[Ben\] Intent is for the SIG to note if this is a significant bug + fix. We should trust our leaders but communicate what is expected. + Maybe this didn’t happen with RC.s. Maybe we need training on this + before giving the ability. + + - \[Marko\] K/K Sig-Release has the final say on what gets in. + + - \[Ben\] That is not the charter today. We would need a change. + This is a pretty big change. Cherry picks don’t work that way. + During code freeze we delegate this behavior. + + - \[Adolfo\] We could fix the issue that Marko + [linked](https://github.com/kubernetes/release/issues/2337) + in the chat. Releases are the strongest part of what we do. We + could write up a proposal. + + - \[Jeremy\] We can write something up and send to k/dev. + +**(Optional) Walk the Board:** + +- Project board review: + [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +- Incoming issue and PR triage + + - [SIG Release issues + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [SIG Release PRs + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [Kubernetes + issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +## Mar 21, 2023 + +**Host (pronouns):** Verónica López + +**Attendees (pronouns):** + +- Stephen Augustus (he/him) + +- Marko Mudrinić (he/him) + +- Leonard Pahlke (he/him) + +- Cici Huang (she/her) + +**Recurring Topics \[timebox to 20 min\]:** + +- \[cici37\] [current release + blockers](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Arelease-blocker) + + - Cici will follow up on with the team, we’re not sure if we need to + delay? + + - We’re not sure how long it takes to fix release blockers + + - We still need to check what it takes to fix it and merge the PR + +- \[cici37\] CI regex for missing rc.0 tag: [fix in + k/k](https://github.com/kubernetes/kubernetes/pull/116809) + + - GitHub issue: + [https://github.com/kubernetes/release/issues/2972](https://github.com/kubernetes/release/issues/2972) + + - krel got fixed with + [https://github.com/kubernetes/release/pull/2976](https://github.com/kubernetes/release/pull/2976) + + - We need to fix it in k/k before proceeding with the rc.0 release + +- \[Marko\] PR to fix release branch jobs generation: + [https://github.com/kubernetes/test-infra/pull/29098](https://github.com/kubernetes/test-infra/pull/29098) + + - Recommended to wait for this PR to get merged before generating jobs + + - Hopefully to be merged today + +- \[Veronica\] Proposal to start cutting releases a day earlier + + - Google Build Admins proposed that we cut release a day earlier so + that we can finish on time + + - We very often have to delay a release for a day or two, which is + making it hard to plan + + - We need to make sure that everything is okay before starting the + release: krel, promo-tools, and everything else that we might need + + - \[Stephen\] promo-tools are bumped manually, so that’s usually not + a problem + + - \[Marko/Stephen\] We might need to refactor promo-tools + + - promo-tools were made for a single registry, now we have many + registries + + - Kubernetes releases shouldn’t take two days, we’re complex + project, but not that complex + + - Publishing to many registries at the promo time is problematic, we + should see about doing it in other way + + - We need to reduce the needed to cut/publish releases + + - We’ll take to the next meeting to discuss with more people + +## Mar 14, 2023 + +**Host (pronouns):** Adolfo García Veytia + +**Attendees (pronouns):** + +- Xander Grzywinski (he/him) + +- Nabarun Pal (he/him) + +- Rodolfo Martínez (he/him) + +- Grace Nguyen (she/her) + +**Note Taker (pronouns):** + +- Jeremy Rickard (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - \[Nabarun\] Patch releases for March to go out tomorrow + + - Build Admins have confirmed availability + + - Nabarun will start in the AM / US folks can take over later + + - \[mahamed\] image freeze stuff going well, what is status of 3.5 + kpromo? + + - \[puerco\] change for signature validation, that’s done now. + + - \[mahamed\] need this PR + [https://github.com/kubernetes-sigs/promo-tools/pull/669](https://github.com/kubernetes-sigs/promo-tools/pull/669) + + - \[puerco\] promoter changes: + + - Changes related to fixing signatures that broke during last + patch release + + - Tracking issue + [https://github.com/kubernetes/release/issues/2962](https://github.com/kubernetes/release/issues/2962) + + - \[jeremy\] updated branch cut tasks to reflect kube-cross image + task and updated publishing bot (sorry for the delay) + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - Code freeze today + +- **Open Discussion:** + + - + +## Mar 7, 2023 + +**Host (pronouns):** Sascha Grunert (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Xander Grzywinski (he/him) + +- Laura Lorenz (she/her) + +- Jeremy Rickard (he/him) + +- Jim Angel + +- Mahamed Ali + +- Adolfo Garcia Veytia + +- Joseph Sandoval + +- Arnaud Meukam + +**Note Taker (pronouns):** + +- Marko Mudrinić (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Laura from CI Signal + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - Patch releases for February went out last Tuesday + + - We disabled signature verification and proceeded with the + release; some images are unsigned as a result + + - We still didn’t retroactively sign affected images + + - It’s still a mystery why this happened at all, we just know + that all traffic hit the same pod + + - Sigstore increased the rate limit a bit and we hope this + shouldn't hope again + + - We should also add some resiliency to the process + + - We thought that we have a solution for this, but last minute + realized that not all cases are covered + + - Images might be detected as signed, but they might be signed + with the staging identity instead of the production identity + + - We hope we’ll able to sign affected images this week + + - This tool that we’re working on is going to give us some + additional options, like moving signing to async + + - The first version is an emergency one, additional + improvements in the future + + - We cannot sign everything, for example if registry is + compromised, we need to make sure that image is coming from + the community + + - There are some upcoming changes to promotion, we’ll think + about smarter ways to sign + + - \[Sascha\] Can we summerize this in a GitHub issue? + + - Adolfo will take care of that + + - KEP-1731 implementation is work in progress + + - The first part is ready for review: + [https://github.com/kubernetes/release/issues/2947](https://github.com/kubernetes/release/issues/2947) + + - \[mahamed\] + [https://github.com/kubernetes/release/issues/2947](https://github.com/kubernetes/release/issues/2947) + image freeze implementation details + + - Two ways to this + + - Easiest: just remove all registries in the manifests and later + (after KubeCon) fix tools + + - Harder: changes in kpromo to ensure everything is done + properly but help from Adolfo and Carlos is needed to get + everything approved in short time + + - \[Adolfo\] What kind changes would you like to do? + + - Replace all k8s.gcr.io references with registry.k8s.io in + promo-tools + + - \[Arnaud\] Do we have a way to dry run this? + + - We should be able to early identify issues or blockers + + - There are some details in KEP-3720 about scope of + planned changes + + - \[Sascha\] How much time do we have if anything goes wrong? + + - Start merging pull request at week of 20th March and see + how thing look like + + - If anything goes wrong, we have ~5 days to rollback and go + to the first idea + + - Mahamed to prepare everything for merge this week and we can + start merging next week + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - \[Xander\] Things are still on autopilot, no major updates as of + now + + - \[lauralorenz\] Follow ups from first [1.27 + retro](https://docs.google.com/document/d/1VcabRbSlEd9qqUm1ENWVXVHQYjS2mxLgqydlQGeC19o/edit#bookmark=id.yvpu3ha4p6hz) + for CI Signal subteam + + - We want to get clarification on some aspect of the role should + we continue doing it or not + + - Confirm utility and scope of CI Signal report + + - CI Signal report is a weekly email blast that goes to k-dev + mailing list + + - It includes generated report + ([link](https://gist.github.com/lauralorenz/3ca69bb88eda5979490392d8f3a82e3b#file-2023-02-22-report-md)), + TestGrid status + + - We want to make sure that it’s still useful to attach this + in email + + - \[Laura\] There’s a lot of duplication in the content there, + wants to get feedback from SIG Release do we still need and + what’s the historical reason for including it + + - \[Sascha\] Email are easy to miss, we have the same issue + with both general and automated TestGrid emails + + - The right way to do it is to check with folks who are + actually responsible for this job. We should get feedback + from folks on k-dev to see if it’s useful. If it’s too + much work, we can deprecate sending that email + + - \[Laura\] CI Signal is spending a lot of energy finding + people who is responsible for job, we want to find out how + useful this general email is + + - \[Laura\] Maintaining tool for this takes time and CI + Signal team has a lot of different tasks and deadlines + that they have to take care of + + - specifically the attached generated report [like + this](https://gist.github.com/lauralorenz/3ca69bb88eda5979490392d8f3a82e3b#file-2023-02-22-report-md) + and the [per job details suggested in the + template](https://github.com/kubernetes/sig-release/blame/master/release-team/role-handbooks/ci-signal/template-weekly-ci-signal-report.md#L17). + It so happened the generated report tool needed some + unexpected maintenance this cycle and before putting updating + docs/training on it I wanted to make sure its output is still + useful. + + - Who is the audience? + + - Branch release management or sig release leads? + + - If so, from the experiences among those present it is + not used by those audiences + + - General dev + + - How can well tell if its being used or what parts are + useful? + + - Laura will figure out how to get general feedback and + we’ll keep us updated + + - \[Laura\] Who should we ask about if this is useful? + + - \[Adolfo\] Motivation is to alert release managers + before the release is out + + - \[Laura and Jeremy\] The tool was there for some time, + this was initially done manually. We started doing this + a long time ago + + - \[Adolfo\] There’s some duplication between TestGrid and + email + + - \[Sascha\] Release Managers don’t need this because we + pull data from TestGrid and we have GO/NO GO signal from + the CI Signal team before cutting release + + - \[Laura\] This takes a lot of effort to curate + + - sig-release-releg-\* dashboards and presubmits dashboards + + - \[Laura\] It’s documented that watching those dashboards is + the responsibility of CI Signal, but that’s not really being + done. Should we watch those dashboards or remove that from the + handbook? + + - \[Adolfo\] It’s responsibility of Release Managers to ensure + those dashboards are green + + - \[Sascha/Laura\] Only interesting dashboards are + master-blocking and master-informing for CI Signal, we want + to make clear should we also watch sig-release-releng-\* + + - We can drop that from handbooks to reduce effort on CI + Signal + + - We can also try to add why is that added in the first + place + + - Does it handle the release process if there are failures? + + - There are important tools and it can affect ability to + release. + + - \[Marko\] We can add it to the release cut template to + check if those dashboards are green + + - \[Adolfo\] We should already be doing that + + - \[Sascha\] We want to deduplicate work for CI Signal and + Release Managers, the current release branch is the most + important for CI Signal + + - Little to no docs/training on these dashboards re: CI Signal + team + + - What is the overlap and priority between these and + master-blocking/master-informing + + - Is this still important for the CI Signal team to be watching? + + - Laura will make updates to the CI Signal handbook and ping + someone from SIG Release to check + +** +Open Discussion:** + +- \[Joseph\] [KubeCon EU 2023 - SIG Meet and + Greet](https://github.com/kubernetes/community/issues/7161) the + issue is open to reserve a table during Kubecon on Friday, April 21 + 12:30-2:30 PM. Are we interested in staffing a table? [K8s Con(feel + free to add any topics you’d like to discuss as part of the + agenda)tributor Summit + CFP](https://docs.google.com/forms/d/e/1FAIpQLSei6q-WqSx0JVUbgFoGRfMvldhCa5ZaND4wRjs6inNEigbLsQ/viewform) + is now open as well. + + - The issue for reserving a table for SIG Meet and Greet is open + + - The CFP for Contributor Summit is open + + - Do we want to participate in CFP and have a table and Meet and + Greet? + + - We’ll have a dedicated meeting on Tuesday + + - Meeting on Tuesday is mostly for SIG, while the KubeCon meeting is + for the general community + + - We also have SIG introduction talk from Veronica and Marko on + KubeCon + +**(Optional) Walk the Board:** + +- Project board review: + [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +- Incoming issue and PR triage + + - [SIG Release issues + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [SIG Release PRs + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [Kubernetes + issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +## 28 Feb 2023 + +**Host (pronouns):** + +- Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Cici Huang(she/her) + +**Note Taker (pronouns):** + +- Marko Mudrinić (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - February patch releases - signing issues + + - The promoter got rate limited when signing images. We don’t know + why it got rate limited; sigstore folks checked logs and + everything, and it turns that all requested went to the same pod + + - The rate limit didn’t change for months, it only got increased + now + + - This impacted only patch releases, everything else signed + afterwards is okay + + - There are 4 different states + + - No signatures at all + + - Partial signatures (e.g. one image has signature but not + replicated to other mirrors) + + - Signature copied from staging, but promoter didn’t copy + properly + + - @puerco wrote another command to check and fix images + + - Check existing of tag, if you find signature, assume it’s + signed + + - But we don’t check if it’s staging or promoter signature + + - Options to continue + + - Update command to check for the promoter/production signature + + - We might need to delay patch releases for a week or even + longer + + - Make a patch to allow completing release and then “throw” that + release away, followed by new set of patch releases + + - We release without signatures, and eventually sign later if + possible + + - Might be problem if someone is already checking signatures, + but that’s going to happened with burnt release as well + + - Do we want to skip patch releases because next patch releases + are in March? + + - We need to finish this release because images are already + there + + - The only artifact we have right now are images + + - It seems like least risky is to just continue, let community + know that releases are not signed, and sign later + + - \[Cici\] Will lack of signatures cause reliability issues? + + - \[puerco\] Those who check for those signatures will have + issues at the time of releases; other than that we don’t + expect any other issues + + - \[Jeremy\] TL;DR. - Continue release without signatures. We will + send a notice to the community to communicate that signatures + will come later and you should skip this release at this point + if you need them + + - \[puerco\] We should clearly communicate that releases were cut + last week; releases will have changes only up until then + + - We must make sure to not merge anything on release branches + until we don’t get those releases out + + - We’ll resume releases tomorrow and alpha on Thursday. Carlos and + Adolfo will create a PR to get rid of the check in krel release + to unblock releases + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - \[Cici\] v1.27.0-alpha3 is planned on Thursday now + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- \[matttrach\] Is it possible to separate tags and artifacts? + + - Have tags before artifacts. There are folks (e.g. at Rancher) that + depend on tag but not on artifacts + + - There are some problems with that and there are processes; the most + complex part is in staging + + - If failures happen, we catch them before publishing tags + + - If we got tag early, we don’t have much options if we need to fix + anything, we would need to scrap release entirely + +- \[Mahamed\] Web banner on kubernetes.io + + - Legacy k8s.gcr.io container image registry will be frozen in early + April 2023 + + - It’ll be here for a little while before getting the KubeCon message + back + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +## 21 Feb 2023 + +**Host (pronouns): Sascha Grunert** + +**Attendees (pronouns):** + +- Xander Grzywinski (he/him) + +- Adolfo García Veytia (he/him) + +**Note Taker (pronouns):** + +- Joseph Sandoval (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - \[Sascha\] + [https://github.com/kubernetes/enhancements/pull/3750](https://github.com/kubernetes/enhancements/pull/3750) + needs final review. Out of tree enhancement. Marko has asked for + review with additional feedback. Optimistic we can merge it. + + - \[Adolfo\] I’ll take a look. Update on a project I want to kick + off in the 1st quarter. **Generate attestations**. There are other + projects that could use our tools. It feels like its time to kick + this off. Merge code that Mahammed has developed with Knative. + Integrate those tools. In parallel use the tester to generate the + attestations. Start with a smaller project and work all the way up + to Kubernetes. If anyone is interested, reach out to Adolfo. + + - \[Sascha\] Security profile operator is looking to reach similar + goals. + + - \[Adolfo\] Coming back from PTO and catching up on the KEP to + freeze the registry. + + - \[Sascha\] For example this + [PR](https://github.com/kubernetes/k8s.io/pull/4808). There + is alot of suggestions on phasing out old images. + + - \[Adolfo\] What's the state? + + - \[Mahammed\] Latest on the SLSA work. SLSA 1 and 2 are done. + + - \[Adolfo\] Attestation was what you showed me? + + - \[Mahammed\] We generate the attestation from the Prow job. + + - \[Adolfo\] I’ll borrow the code and provide an update. + + - \[Mahammed\] Image freezing is almost complete. Development is on + track. + + - \[Adolfo\] I’ll start attending the K8s-infra meetings + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - \[Xander\] It's quiet now we are past the Enhancement freeze. No + exceptions requested. Only lost 9 enhancements. Total tracked + enhancements = 80!!! + +- + +** +Open Discussion:** + +- Do we plan to do a KubeCon EU Contributor Summit SIG Release meetup? + + - \[Sascha\] Perhaps we could do a similar meeting like in Detroit. + Good discussions that lead to some actions. + + - \[Adolfo\] Having a similar meeting like the one in Detroit would be + great. Do some outreach for people who are not familiar with + SIG-Release. + + - \[Joseph\] + 1 to doing a follow up in Amsterdam + + - \[Sascha\] We could cover projects and also do an intro to + Sig-Release + + - \[Joseph\] Content track for Contributor summit should open this + week. + + - \[Sascha\] Marko and Veronica will be presenting for Sig-Release. + + + +- \[Arnadu\] What is the status of OBS? + + - \[Sascha\] Risk that the project might not make 1.27. Building + packages will be similar to what we are doing already. + +## 14 Feb 2023 + +**Host (pronouns):** + +- Verónica López + +**Attendees (pronouns):** + +- Rey Lejano (he/him) + +- Marko Mudrinić (he/him) + +- Joseph Sandoval (he/him) + +- Cici Huang (she/her) + +- Xander Grzywinski (he/him) + +- Leonard Pahlke (he/him) + +**Note Taker (pronouns):** + +- Joseph Sandoval and Rey Lejano + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - \[Cici\] Access clarifications , + + - \[Veronica\] Granting access makes sense, open a ticket/issue + for access + + - \[Cici\] To raise a separate PR to trigger + + - \[Veronica\] Suggest to err on the safe side, a PR may mess with + someone’s work so it’s safe to let everyone know on the Slack + channel. + + - \[Cici\] Will make PR and will tag release managers + + - \[Marko\] Do we want to update date for Go update + + - \[Cici\] On Go version bump, it’s a patch update so it should + not affect alpha cut since master should have latest Go version. + + - \[Marko\] Marko thinks 1.20 is also affected + + - \[Veronica\] Saw some work during the week, and on the mailing + list notes it says 1.20 is also affected but does not have the + details + + - \[Cici\] Not sure if it will delay the alpha cut for + 1.27.0-alpha2 since no one is using it + + - \[Veronica\] Alpha cuts in the past is to flush out things, so + not used by many. Request Cici to post this in the Slack channel + + - \[Cici\] Will send Slack message + + - \[Marko\] Please tag Dims and Liggit since they are triaging + some Go issues + + - \[mahamed\] The registry freeze kep has been merged and the work + has started + + - \[Mahamed\] Working on freezing old registry, working on + infrastructure change this week and next week. And can cut over + at end of the month. Need to find Adolfo and others to have a + chat. So far, we’re on track + + - \[Veronica\] Do you need any help + + - \[Mahamed\] Not from SIG Release right now + + - \[Marko\] Thank you Mahamed for working on this as its an + important cost saving measure + + - \[Mahamed\] Will work on a banner + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - \[Xander\] Enhancements freeze last week + + - \[Xander\] Only 9 KEPs did not meet requirements and no + exceptions so far. In the 70s for enhancement that made + Enhancements Freeze + + - \[Cici\] scheduled 1.27.0-alpha2 cut today. Delayed due to + [issue](https://github.com/kubernetes/kubernetes/issues/115767). + Working on verifying. + +- \[Veronica\] Discussed on Slack and will send comms, supposed to have + patch releases for Wednesday and they will be delayed by a day to make + room for the Go updates + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +## 7 Feb, 2023 + +Canceled due to no agenda + +## 31 Jan, 2023 + +Canceled due to no agenda + +## 24 Jan, 2023 + +**Host (pronouns):** Jeremy Rickard (he/him) + +**Attendees (pronouns):** + +- Marko Mudrinić (he/him) + +- Xander Grzywinski (he/him) + +- James Laverack (he/him) + +- Leonard Pahlke (he/him) + +- Ana Margarita Medina (she/her) + +- Stephen Augustus (he/him) + +- Rodolfo Martínez (he/him) + +- Rajesh Gunasekaran(he/him) + +- Yash Raj Singh (he/him) + +- Sascha Grunert (he/him) + +**Note Taker (pronouns):** + +- + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + + - Welcome Micah + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - \[mahamed\] Freezing k8s.gcr.io image registry. + [KEP](https://github.com/kubernetes/enhancements/pull/3723) + + - \[Marko\] January patch releases (1.26.1, 1.25.6, 1.24.10, + 1.23.16) are available! + + - Everything went mostly fine. We had a hiccup in the promotion + process requiring us delay 1.24.10 for one day + + - We are (sometimes very) often hitting signing flakes + [https://github.com/kubernetes/release/issues/2809](https://github.com/kubernetes/release/issues/2809) + + - We had to repeat some releases for 3 or 4 times + + - I increased the priority of this ticket to critical, it would + be nice to fix it in time for February releases + + - \[Marko\] OBS POC in progress again, added an agenda point to + discuss the + [KEP](https://github.com/kubernetes/enhancements/pull/3750) + + - \[Marko\] Release branches are updated to Go 1.19.5, master is + getting updated to Go 1.20rc3 ([WIP + PR](https://github.com/kubernetes/kubernetes/pull/114502), + tests are mostly green) + + - Thanks a lot to Carlos, Jordan, and Madhav for driving this + forward! + + - \[Jordan\] e2e and all canaries are passing on new go version + + - \[Cici\] Planned alpha2 cut for 1.27 + today([issue](https://github.com/kubernetes/sig-release/issues/2157)) + + - Blocker issue identified yesterday, but resolved. + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - 47 enhancements opted in so far + + - No major concerns + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- \[Marko\] Update and next steps for the OBS POC + + - KEP: + [https://github.com/kubernetes/enhancements/pull/3750](https://github.com/kubernetes/enhancements/pull/3750) + + - Please Review + + - Notes: some changes to OS builds, no more deb spec, RPM converted to + debspec by tooling + + - Highlights: will remove everything from our OBS project this week, + will start fresh + + - KEP needs review, need to setup account in GCP for implementation + + - Need to decide on domain + + - \[Stephen\] for names, we should make a clean break + + - \[Jordan\] the testing gaps for ppc, s390x are pre-existing yes? We + shouldn’t focus too much on existing orthogonal issues. Some + historical issues with supporting platforms we can't test well: + + - [https://github.com/kubernetes/kubernetes/issues/38067](https://github.com/kubernetes/kubernetes/issues/38067) + + - [https://github.com/kubernetes/kubernetes/pull/76974](https://github.com/kubernetes/kubernetes/pull/76974) + + - [https://github.com/kubernetes/kubernetes/issues/93621](https://github.com/kubernetes/kubernetes/issues/93621) + + - [https://github.com/kubernetes/kubernetes/issues/93620](https://github.com/kubernetes/kubernetes/issues/93620) + + - [https://github.com/kubernetes/sig-release/blob/master/release-engineering/platforms/guide.md](https://github.com/kubernetes/sig-release/blob/master/release-engineering/platforms/guide.md) + + - \[Stephen\] yes, there is also a supported arch KEP that has been + around for a while + + - \[Marko\] We could get rid of those archs + + - \[Stephen\] further down the line we should reassess what we build + + - \[Stephen\] Thanks for all the work on this Marko. + +- \[Marko\] PRs looking for feedback: + + - Minor releases in the patch releases calendar: + [https://github.com/kubernetes/website/pull/39086](https://github.com/kubernetes/website/pull/39086) + + - Note about incomplete cherry-pick series: + [https://github.com/kubernetes/test-infra/pull/28490](https://github.com/kubernetes/test-infra/pull/28490) + +- \[liggitt\] discussion of go update KEP: + [https://github.com/kubernetes/enhancements/pull/3749](https://github.com/kubernetes/enhancements/pull/3749) + + - \[Stephen\] for clarity, go doesn’t \*really\* follow semver so + every minor bump is really like a major bump + + - \[Jordan\] there were some go minor versions we could have adopted + without issues in the past, but some instances have been a problem + so we couldn't consistently do it, so we generally did not + + - \[Jordan\] as we’ve extended release support, older k8s patches are + on unsupported go versions and there have been CVEs. + + - \[Jordan\] KEP goal/proposal overview + + - \[stephen\] be concrete about developer actions to mitigate behavior + changes (envvar changes) + + - \[marco\] + + - consider updating sooner than 3 months if tests look good + + - not sure the 1-month post-kubernetes release buys us much + + - jordan: RC and qualification of .0 releases gets a decent amount + of reports from the field from people starting to qualify new + versions, even if they're not widely adopted in prod yet. not + perfect, but does build some confidence we didn't miss + Kubernetes-specific regressions + + - want to be sure we don't break release-1.x libraries with original + go versions + + - jordan: unit/integration tests on original go version *should* + ensure libraries remain usable without bumping go minors + + - stephen: need to look at tooling for creating release branches, + test jobs, etc + + - + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +**(Optional) Walk the Board:** + +- Project board review: + [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +- Incoming issue and PR triage + + - [SIG Release issues + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [SIG Release PRs + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [Kubernetes + issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +## 17 Jan, 2023 Canceled because of light agenda + +## 10 Jan, 2023 (recording) + +**Host (pronouns): Sascha Grunert (he/him)** + +**Attendees (pronouns):** + +- Xander Grzywinski (he/him) + +- Cailyn Edwards (she/her) + +- James Laverack (he/him) + +- Marko Mudrinić (he/him) + +- Leonard Pahlke (he/him) + +- Rey Lejano (he/him) + +- Jim Angel (he/him) + +- Mahamed Ali + +- Abhilipsa Sahoo (she/her) + +- Rudraksh Karpe(he/him) + +- Angelos Kolaitis (he/him) + +- Drew Hagen (he/him) + +- Ruheena Ansari (she/her) + +- Stephen Augustus (he/him) + +- Subhrodip Mohanta (he/him) + +- Adolfo García Veytia (he/him) + +- Joseph Sandoval (he/him) + +- Harsha Narayana (he/him) + +**Note Taker (pronouns):** + +- Rey Lejano (he/him) + +**Recurring Topics \[timebox to 20 min\]:** + +- Welcome any new members or attendees + +- Subproject updates + + - Release Engineering + ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + + - \[Marko\] OBS: no major updates -- will try to come up with a KEP + next week + + - \[Marko\] Will make a PR for a KEP next week and we can continue + discussion on the PR + + - \[Marko\] Go updates: + + - master to Go 1.20: + [https://github.com/kubernetes/kubernetes/pull/114502](https://github.com/kubernetes/kubernetes/pull/114502) + + - Release branches to Go 1.19 + + - 1.24: + [https://github.com/kubernetes/kubernetes/pull/113956](https://github.com/kubernetes/kubernetes/pull/113956) + + - 1.23: + [https://github.com/kubernetes/kubernetes/pull/113983](https://github.com/kubernetes/kubernetes/pull/113983) + + - To be in discussed in-depth later (added an agenda point in Open + Discussion) + + - \[Marko\] January patch releases (1.26.1, 1.25.6, 1.24.10, + 1.23.16) scheduled for January 18 (next week), with the + cherry-pick deadline on January 13 (this Friday) + + - 1.23 is in the maintenance mode, but we should cut at least one + patch release to ship the Go update + + - \[Sascha\] We can start working on the enhancements update like + choose OBS and build our own infra and have the goal for 1.27 + + - \[Marko\] List out solution (OBS), positives/negatives, the + process and details about the packages + + - \[Jim\] Did a pairing session with Marko and can take off some + load + + - \[Veronica\] Veronica has cherry-picks on Veronica’s agenda and + will do them in the last week + + - Release Team + ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + + - \[Xander\] Call for enhancements went out yesterday + + - \[Xander\] 9 enhancements so far + + - \[James\] Shadow selection ongoing + + - \[Xander\] Plan to finish shadow selection plans to end this + week + + - \[James\] Shadow notification is Tuesday next week and we are on + track + + - \[James\] Issue came up, missed two md files for shadow + selection and need to add to the EA handbook. + + - \[Stephen\] Also send email about shadow survey statistics to + k-dev + +- (feel free to add any topics you’d like to discuss as part of the + agenda) + +** +Open Discussion:** + +- \[Marko\] Policy for Go updates on release branches + + - \[Marko\] Had 2 PRs for 1.23 and 1.24 release branches to Go 1.19. + PRs merged, test grid is green, noticed no problems. But it’s + against the policy, Marko worries the effect on the community since + community has to update Go version also. Also send notice to the + community to update their Go when using these versions + + - \[Stephen\] Let’s discuss the reasoning since we haven’t done that. + With regards to policy changes, we should announce policy changes + before they start happening and we should have discussion around + policy changes before they happen. Let’s discuss why as next step + + - \[Marko\] Reason for that, for 1.23 was using Go 1.17 and it was + eol. 1.24 was on Go 1.18 and it is eol + + - \[Stephen\] Having a Go version to go out of support is a forcing + function for folks to upgrade Kubernetes. Was it timing with lagging + branch and Go 1.17 support? + + - \[Marko\] For 1.23 we could have skipped Go update since its in + maintenance mode. For 1.24 and it is going into maintenance mode in + end of May, so have 3-4 months of maintenance mode and Go 1.18 eol + (3 months before 1.24 reaches maintenance mode). + + - \[Stephen\] Makes sense for 1.24 + + - \[Marko\] Not sure about going back (reverting?) since may have + issues with publishing bot. + + - \[Stephen\] We should update our policy. Asking for feedback from + release managers on how we feel about updating Go versions on + already released branches + + - \[Puerco\] Need to review the policy and understand a little more + + - \[Veronica\] In general, we have done things to break boundaries of + our policies and its normal as things happen, it’s worth revisiting + policies like how we treated 1.22 – the policies are not serving + edge cases. If we have clear policies, because we are distributed + and work async, people make decisions when others are not around, + policies help with decision-making + + - \[Stephen\] We should block on discussion with having a policy in + place or something that dictates what happened in an edge case. + Sooner block for discussion because its harder to revert then just + do. + + - \[Arnaud from chat\] So no RC Golang for release branches ? + + - \[Stephen\] No. Goal is to not have a GA version of Go is release + blocking + + - \[Puerco\] Was there a CVE as a rational for the Go updates + + - \[Marko\] There were some CVEs. Agrees with Stephen and Veronica on + this. Marko jumped a little quickly on this, folks wanted it updated + to get signal. We should update the policy. A reason why, Go was + introducing some breaking changes + + - \[Stephen\] Request for Marko to go to the backlog on why the + changes happened. Have to make sure we have a good reason like a + massive security vulnerability or a super mega regression like 2-3 + releases ago. If it impacts people’s operation of Kubernetes then we + should document it. Because where we are seated, we are the group + that releases Kubernetes, we are in the role of in-service to other + groups so that causes us to be more laxed of policies in-service to + other groups. So we should question requests. + + - \[Puerco\] By bumping minor version we can alter how Kubernetes + behaves. When our branches go out of support and when Go goes out of + support – we should be conservative with changes. Also agree with + what Stephen said if there is something very critical then we can + bump. It’s not our job to make quick snap decisions – we should be + on the conservative side + + - \[Marko\] On the PR, on performance improvements, release branches + won’t receive certain performance changes with the use of env + variables + + - \[Marko\] To recap this, prepare an announcement with what happened + and state no negative impact on users, document policy change + + - \[Stephen\] The second we send an announcement and lock in another + policy, we start beholding ourselves to that. Expect updates on + previous branches as a result of that and expect people to point to + our policy + +- \[Marko\] Do we have a policy for announcing changes on k-announce? + +- \[Marko\] Green light for proceeding with + [https://github.com/kubernetes/release/pull/2780](https://github.com/kubernetes/release/pull/2780) + +- \[Marko\] What’s the status of the registry changes? + + - [https://kubernetes.slack.com/archives/CCK68P2Q2/p1672858680698639](https://kubernetes.slack.com/archives/CCK68P2Q2/p1672858680698639) + + - \[mahamed\] Can we start working on freezing k8s.gcr.io registry? It + is a major change and it would be nice to announce our intent early + in the 1.27 release cycle. + +- \[Marko\] Should we update the cherry-pick message (e.g. + [https://github.com/kubernetes/test-infra/blob/1958f266cee0b34ffc8a50e019911921f37884b3/config/prow/plugins.yaml#L993-L1002](https://github.com/kubernetes/test-infra/blob/1958f266cee0b34ffc8a50e019911921f37884b3/config/prow/plugins.yaml#L993-L1002)) + to note that you should mention if cherry-pick to other branches is + needed? + +- (feel free to add any topics you’d like to discuss, even when they + came up during the meeting) + +**(Optional) Walk the Board:** + +- Project board review: + [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +- Incoming issue and PR triage + + - [SIG Release issues + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [SIG Release PRs + needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + + - [Kubernetes + issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +BEYOND THIS YEAR MOVED (After [this +PR](https://github.com/kubernetes/community/pull/7329) merges, +replace with link)