# SIG Release — Meeting Archive — 2023

## Dec 19, 2023

**Host (pronouns):** Marko Mudrinić (he/him)

**Attendees (pronouns):**

- Rudraksh Karpe (he/him)

- Josh Berkus (he)

- Megan Wolf (she/her)

- Sreeram Venkitesh (he/him)

- Siva (he/him)

- Oluebube Princess Egbuna(She/her)

- Joseph Sandoval (he/him)

- Daniel Akinpelu(he/him)

**Note Taker (pronouns):**

- Joseph Sandoval

**Recurring Topics \[timebox to 20 min\]:**

**NO RECORDING THIS MEETING (no host key)**

- Welcome any new members or attendees

  - 

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update:

(Marko)Scheduled to be out today.

- Action Items/Help Wanted:

<!-- -->

- [<u>OBS
  Packages</u>](https://github.com/orgs/kubernetes/projects/137/views/1):

  - General Update: No major updates

  - Action Items/Help Wanted:

    - Approval needed for the obscli repository request:
      [<u>https://github.com/kubernetes/org/issues/4550</u>](https://github.com/kubernetes/org/issues/4550)

- Supply Chain Security:

  - General Update:

> (Marko) Nothing new.

- Action Items/Help Wanted:

<!-- -->

- [<u>Artifact
  Validation</u>](https://github.com/orgs/kubernetes/projects/171):

  - General Update:

> (Marko) Lauri Apple is running some great sessions. There are issues
> on the board if you are interested in picking something up.

- Action Items/Help Wanted:

<!-- -->

- Release Team

- (Marko) Release cycle hasn’t started yet.

<!-- -->

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- \[Sreeram\] How are the [<u>kepctl and
  kepify</u>](https://github.com/kubernetes/enhancements/tree/master/cmd)
  tools in k/enhancements currently being used by SIG Release?
  Possibility to merge code from the
  [<u>kept</u>](https://github.com/salaxander/kept) CLI written by
  Xander and folks from previous release teams.

  - (Marko) I don’t have any ideas about this.

  - (Joseph) What is the benefit if we do.

  - (Sreeram) There might be an opportunity to improve and get benefits
    from merging the tools. It will make it easier for the enhancement
    team.

  - (Joseph) The release team has generally drive improvements to the
    their tooling. Under the governance of the Sig-release team managers
    and leads.

  - (Sreeram) Go to all the three different tools and outline the
    benefits we get from doing this.

  - (Joseph) Bring this discussion to the SIG-Release channel when you
    have pulled the Enhancements tooling improvements.

## Dec 12, 2023

**Host (pronouns):**

- Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Megan Wolf (she/her)

- Chris Hanson (he/him)

- Rudraksh Karpe (he/him)

- Jim Angel (he/him)

- Marvin Beckers (he/him)

- Lauri Apple (she)

- Sreeram Venkitesh (he/him)

- Sascha Grunert (he/him)

**Note Taker (pronouns):**

- Marko Mudrinić (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Marvin Beckers - software engineer/team lead @ Kubermatic, want to
    learn more about upstream release engineering

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update:

        - Patch releases are scheduled for the next week, Tuesday,
          December 19

          - Cherry-pick deadline is Friday, December 15

        - Go updates are almost done, we only need to update Go version
          in the publishing-bot rules, Marko will take care of that
          today

        - This patch releases series will not include v1.29.1

          - v1.29.1 is scheduled for January

    - OBS Packages:

      - General Update:

        - Still focused on legacy packages, no major updates for OBS

        - Nitish will do a presentation on the LFX project in January

    - Supply Chain Security:

      - General Update

        - [<u>Cloud Native Security
          Slam</u>](https://community.cncf.io/events/details/cncf-cloud-native-security-slam-presents-2023-security-slam-kubernetes-lightning-round/)
          coming this Friday

          - Planning contributor tasks

          - CLOMonitor:
            [<u>https://clomonitor.io/</u>](https://clomonitor.io/)

            - We want to get the Kubernetes project there and improve
              the score as much as possible

            - This is one of tasks that we want to tackle

          - There’ll be a webinar where you can ask questions and where
            we’re going to discuss bigger tasks

          - Still low signs from the Kubernetes side, we only heard
            about one or two SIGs

            - Even if the project is small and has one image or binary,
              there’s work for sure

          - Sign up for individual contributors is closed, reach to
            Adolfo if you want to be added to the list

      - Action Items/Help Wanted:

        - Help wanted guiding new contributors

  - Release Team

    - Priyanka:

      - Mock stage & release runs for 1.29.0 completed (thanks to Sascha
        & Release Engineering)

        - Nomocks: when do we plan to start the process tomorrow?

          - We can start tomorrow in the morning European time. Last
            time we had problems with packages taking 12+ hours to
            build, so we want to decrease the risk of having to delay
            the release if something like that happens again

          - Do we have to worry about embargo? No, we generally inform
            the CNCF of the release date, we didn’t mention any concrete
            release time

      - When do we plan to start the no-mock stage & release runs
        tomorrow (Dec 13) PT time? I’ll plan the email comms
        accordingly. Thanks!

      - \[OT - more of a prod-readiness question, question for one of
        the media interviews\]

        - Did we make any key improvements this cycle w.r.t. Reliability
          & security (prod-readiness)?

          - Efforts to expand the team to make the process more robust
            with more capacity

          - Jeremy to give number of shadows to Priyanka

      - 1.29 Retro 2 meeting planned for 15 min after this call (8:30 AM
        PT)

      - Need to ready the PR for lifting code freeze

        - RelEng folks to help with that

      - 1.30 shadow applications are open, the form is already out

        - [<u>https://forms.gle/psoz3EaYDFt2CBjS6</u>](https://forms.gle/psoz3EaYDFt2CBjS6)

        - [<u>https://groups.google.com/g/kubernetes-sig-release/c/aHtIDQoLoDk/m/z20TnLAQAwAJ</u>](https://groups.google.com/g/kubernetes-sig-release/c/aHtIDQoLoDk/m/z20TnLAQAwAJ)

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- \[mahamed\] kubekins-e2e v2 image
  [<u>https://github.com/kubernetes/test-infra/pull/31421</u>](https://github.com/kubernetes/test-infra/pull/31421)

  - Main image that we use for e2e testing in Prow, it’s kind of bloated
    and has a lot of dependencies that we don’t need

  - Main reason for this image is that etcd is looking for tests on ARM,
    the old image doesn’t support arm, so we want to take this
    opportunity to build a new image that supports arm and that doesn’t
    have as much as cruft as the old one

  - Adolfo to take a look after the meeting

- \[Marko\] Should we cancel meetings for the week of Dec 25 and Jan 1?

  - Jeremy will cancel these

- \[Lauri\] Next conversation about artifact validation work being
  scheduled:
  [<u>https://kubernetes.slack.com/archives/C2C40FMNF/p1702310201513499</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1702310201513499)

  - Goal: Break down project board items to help people get started

  - Issues and project board for "Make Artifact Validation More Robust"
    now here:
    [<u>https://github.com/orgs/kubernetes/projects/171/views/1</u>](https://github.com/orgs/kubernetes/projects/171/views/1)

  - Issues organized in order based on dependencies.

  - Relevant notes now transferred from the Miro board.

  - Seeking collaborators on the end user item – need someone with a bit
    more insight into CNCF operations, what useful resources they might
    have+lend

  - \[Lauri\] The next session is being planned, we’ll review the action
    items that have been created, we’ll go through the board and clean
    goals, see if anyone want to pick any item

  - \[Lauri\] Milestones and tasks will be identified, as well as
    contact points for those

  - \[Lauri\] Trying to figure out what’s expected by Kubernetes users,
    which groups might be the best to hit for surveys to reach as many
    Kubernetes users as possible

    - \[Jeremy\] WG LTS has a survey ongoing, there are not many ways to
      reach out to users, the best is to share on social medias and
      internally in your groups (can help if you work for large
      employer)

    - \[Arnaud\] CNCF End users group is an option as well:
      [<u>https://www.cncf.io/enduser/</u>](https://www.cncf.io/enduser/)

- \[Jeremy\] Danny Brito from my team at MSFT will be coming to SIG
  Release in the future and is currently working on
  [<u>https://github.com/kubernetes/test-infra/issues/29390</u>](https://github.com/kubernetes/test-infra/issues/29390)
  (working on generate_tests.py ⇒ go right now)

- \[Jeremy\] Project Copacetic demo in the New Year

- \[Adolfo\] Cross-SIG effort to improve the security feed, Adolfo
  preparing a demo for the next SIG Security Tooling meeting (Wednesday,
  December 20)

- \[Rudraksh\] Question about applying for CI Signal

  - \[Priyanka\] There’s a new team called Release Signal as of this
    release cycle (merge of Bug Triage and CI Signal), there’ll be no
    shadow selection for this team this release cycle, but as of the
    next one

  - \[Priyanka\] You need to be familiar with Prow, there are some great
    resources and talks about it, you can use that as the starting point

  - \[Priyanka\] All other roles are taking applicants this release
    cycle

## Dec 5, 2023

**Host (pronouns):** Marko Mudrinić (he/him)

**Attendees (pronouns):**

- @sftim (he/him/they/them)

- Sreeram Venkitesh (he/him)

- Rudraksh Karpe (he/him)

- Adolfo García Veytia (he/him)

- Jim Angel (he/him)

- Faeka Ansari (she/her)

- Nitish Kumar (he/him)

- Megan Wolf (she/her)

- Mario Fahlandt (he/him)

- Sandipan Panda

- Mahamed Ali

- Benjamin Elder

- Josh Berkus (he)

- Lauri Apple (she)

- Divya Soundararajan (She/Her)

**Note Taker (pronouns):**

- Joseph Sandoval + anyone else who wants to edit

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Megan Wolf - here to learn and maybe get involved

  - Divya Soundararajan - has visited various sigs. Freelancer.

  - Rudraksh Karpe - just submitted his first PR. Here to learn.

  - Mario Fahlandt - involved with K8s SIG-Infra

  - Sandipan Panda - Involved in SIG Contribex, here to learn and
    contribute

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update:

        - Go security updates has delayed the patch releases. We moved
          the schedule back by a week. 12/19 is the updated release
          date.

      - Action Items/Help Wanted:

        - Keep an eye on incoming PRs. No help needed atm but keep an
          eye on the release channel.

    - OBS Packages:

      - General Update:

        - No updates.

      - Action Items/Help Wanted:

      - (also see infrastructure topic later in meeting)

    - Supply Chain Security:

      - General Update:

        - CNCF [<u>Cloud Native Security
          Slam</u>](https://community.cncf.io/cloud-native-security-slam/)
          postponed to Fri Dec 15th

          - Help still wanted

          - Event page will be updated today with the new date

          - [<u>Original announcement
            k/dev</u>](https://groups.google.com/a/kubernetes.io/g/dev/c/SbSrJckE1iI)

        - (Adolfo) Proposal for new SBOM Design Starts today

      - Action Items/Help Wanted:

    - LFX project:

      - General Update:

        - (Marko) We’ll have a demo from our SIG-Release intern.

  - Release Team

    - Post Dec 5, moving all the 1.29 Release Burndown updates to async
      slack thread in \#sig-release

      - (Jim Angel)

    - (Already discussed in [<u>slack
      thread</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1701372168588369))

      - Release date postponed to Dec 13, 2023

      - 1.29.0-rc.2 scheduled for Dec 7, 2023

      - Retro 2 will be moved to next week (Xander to send out updated
        invite)

- \[Lauri\] update on release roadmapping

  - We’ve made progress:
    [<u>https://miro.com/app/board/uXjVMaKPi9c=/</u>](https://miro.com/app/board/uXjVMaKPi9c=/)

    - Recordings:

    - Dec 5:
      [<u>https://www.youtube.com/watch?v=kif-gN8f5Qw</u>](https://www.youtube.com/watch?v=kif-gN8f5Qw)

    - Nov 29:
      [<u>https://www.youtube.com/watch?v=kgXsKsYQXfE</u>](https://www.youtube.com/watch?v=kgXsKsYQXfE)

  - We need to self-organize to tackle the spikes and evaluations first,
    then be in a better spot for decision-making.

  - Will be making some issues and a project board

  - User research / outreach opportunities for folks interested

  - We still need to connect work being discussed (largely
    image-promoter reworking) to SLSA themes. The workflow was laid out
    with identifying pain points. Discussion around making it more
    modular components. We have a lot to learn before we can start the
    work. This week we are collecting data on how long a release is
    taking. On the right side of the Miro board you can review the
    decisions that are needed to be made.

  - Want to focus on “robust artifact validation” before pursuing other
    efforts

  - To help newcomers we’ll match a SIG Release vet to co-work on a
    spike.

  - We didn’t get to the SLSA requirements.

  - 

- Legacy / current Linux package repositories & infrastructure to
  support them (@sftim)

  - (Marko) We have a new official package repository. This hosts K8s
    releases starting with 1.24.0. We only publish packages to this
    repository. We announced deprecation of legacy repository. (per the
    August announcement ) Legacy packages will be removed sometime in
    the future without announcement. Do we want to make snapshots
    publicly available?

  - (Tim B) I would like to hear if there are any questions.

  - (Mario) What would the release team from a user perspective want to
    see? If we removed packages from the registry. Users traffic will
    eventually migrate.

  - (Adolfo) Anytime we make a change it usually breaks someone. So
    being transparent is better for the users. I would prefer to not
    have any breaking changes to users.

  - (Marko) We can’t serve packages forever. Put in a concrete deadline.

  - (Tim) In the past it was common to get packages from various
    sources. We don’t have huge revenue. How do we feel about getting
    users to download from mirrors?

  - (Marko) That was one of the reasons to use OBS. The bandwidth given
    is limited.

  - (Ben) The former team inherited this. We have been negotiating to
    keep this offering in place. It was a mistake to publish this but we
    have to get off of this. This is not a surprise. The packages are
    going away in the new year. There is not enough bandwidth. We have
    not had a host for these packages which is now going away. We as a
    project setup OBS. Older packages are what is in play. Our budget is
    tight. We are burning about 3mil. We negotiated with Fastly. We
    don’t have the resources available and need to discuss how we host
    the OBS packages. We are not prepared to redirect the old ones. We
    need to message that this is going to break. K8s docs are pointing
    to Google repositories.

  - (Mahamed)

  - (Ben) Burning money at the beginning of the year is not a good way
    to start with our budget. We need to get below with our run rate

  - If there is time we could tapo new alliances as Jeefy suggested.

  - Marko, can we come up with a number? A threshold where we could pull
    the plug if we hit it

  - Ben: We need to know bandwidth, we have other resources such as
    equinix. We have fastly but we need to talk to them with bandwidth
    numbers. How much bandwidth are we reasonably talking about. We need
    to observe data over time.

  - Tim: How does sig release if we break old packages and put a
    friendly webpage instead

  - Marko, this is the only choice if we cannot serve, and put a how to
    upgrade your cluster instead.

  - Puerco: Do we have time to observe data?

  - Marko: we can set it up quickly

  - Mario: What if we lower costs by running some expensive jobs less
    often and use that to bridge until we get a deal with Fastly, etc

  - Marko Id like to see that

  - Ben: We don’t have full transparency to the full AWS bill yet

  - \[missed marios comment\]

  - Mahamed: We need to still allocate funds for the things we want to
    pay.

  - Ben: We have been very reactive and we hope to be more proactive
    next year, but we cannor make any guarantees on how effective that’s
    going to be.

- (Rudraksh) Looking for information on SBOMs.

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- ~~\[Nitish\] LFX project presentation (15 minutes)~~ (postponed for
  next week)

- 

## 28 Nov 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Sascha Grunert (he/him)

- Leonard Pahlke (he/him)

- Mah

- Sreeram Venkitesh (he/him)

- Stephen Augustus (he/him)

- Drew Hagen (he/him)

- Adolfo García Veytia (he/him)

- Sujay Dey (he/him)

- Arnaud Meukam (he/him)

**Note Taker (pronouns):**

- Sascha Grunert (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Drew works as shadow on the docs subteam. Welcome!

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update:

        - Patches will be delayed by one week:
          [<u>https://github.com/kubernetes/website/pull/44107</u>](https://github.com/kubernetes/website/pull/44107)

        - Jobs updates for release-1.29 were clunky, working on
          improving the tooling around that

        - Publishing bot is broken since 1.29.0-rc.0, fixing everything
          before the final release next week

      - Action Items/Help Wanted:

        - If anyone is interested in reworking the job generator for new
          release branches, feel free to reach out to \#sig-release on
          slack.

        - Follow up on publishing bot after, technically owned by
          release engineering but not good ownership/maintainership
          today

    - OBS Packages:

      - General Update:

        - Working on a full copy of the legacy package infra which
          should get exposed to the public.

      - Action Items/Help Wanted:

    - Supply Chain Security:

      - General Update:

        - Continuing to meet with SIG Security and outlining a doc:

> [<u>\[WIP\] False Positives, VEX, and the Security
> Feed</u>](https://docs.google.com/document/d/1a_3aMKexfhZwnbzmZZy0dhbQX7L_c8Cr2HSDKchLU7o/edit#heading=h.3xduq4wgcoc5)

- Action Items/Help Wanted:

<!-- -->

- LFX project:

  - General Update: no updates right now

<!-- -->

- Release Team

  - \[Priyanka - will miss the call, meeting conflict\]

    - Final week in 1.29 Release Cycle!

    - 1.29.0-rc.1 is live (Thanks to Sascha!)

    - Nearing final Release Milestones for Docs, Release Notes, Release
      Blogs completion (All are green, except Docs - 4 pending PRs, but
      comms with KEP owners, is in progress)

    - Release Signal Handbook merged last week, Xander/Grace sent out
      the email to [<u>dev@kubernetes.io</u>](mailto:dev@kubernetes.io)
      ML to inform of the changes in 1.30 & onwards

    - Others - CNCF Release Webinar, Pre-release PR interviews, all
      discussions are in-progress too.

    - Open question - should we hold release if publishing bot is still
      broken

**  
Open Discussion:**

- *Make artifact validation more robust' roadmapping* - tomorrow, Nov 29
  @ 1pm CET  
  [<u>https://www.google.com/calendar/event?eid=NW9rZnJyaTByYnR0dXZjdWJvbWpubXEyamkgY2FsZW5kYXJAa3ViZXJuZXRlcy5pbw</u>](https://www.google.com/calendar/event?eid=NW9rZnJyaTByYnR0dXZjdWJvbWpubXEyamkgY2FsZW5kYXJAa3ViZXJuZXRlcy5pbw)

## Nov 21, 2023

**Host (pronouns):**

**Attendees (pronouns):**

- Xander Grzywinski (he/him)

- Meha Bhalodiya (she/her)

- [<u>Pushkar Joglekar</u>](mailto:pushkarj.at.work@gmail.com) (he/him)

- Marko Mudrinić (he/him)

- Leonard Pahlke (he/him)

- Nishok Ganesan (he/him)

- [<u>Adolfo García Veytia</u>](mailto:puercozon@gmail.com) (he/him)

- Sujay Dey (he/him)

- Jim Angel (he/him)

- Friedrich Wilken (he/him)

- Christopher Hanson (he/him)

**Note Taker (pronouns):**

- [<u>Grace Nguyen</u>](mailto:nnggrace@gmail.com)(she/her)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update:

        - In process: 1.29.0-rc.0

      - Action Items/Help Wanted:

    - OBS Packages:

      - General Update:

        - Public-facing docs for pkgs.k8s.io are complete and backported
          up to v1.24

        - We’re missing packages for v1.25.16, pending response from the
          OBS team

          - Marko will ping them again

          - Built, on OBS infra but not published on our S3 package

        - Evaluating next steps and priorities

      - Action Items/Help Wanted: N/A

        - Project board:
          [<u>https://github.com/orgs/kubernetes/projects/137/views/1</u>](https://github.com/orgs/kubernetes/projects/137/views/1)

    - Supply Chain Security:

      - General Update:

        - A bunch of us attended the SIG Security meeting on Nov 16th to
          discuss an upcoming revamp of the kubernetes CVE feed. We’ll
          continue our convo today, specifically to start thinking about
          the work required from SIG Release / RelEng.

        - Vex issue:
          [<u>https://github.com/kubernetes/kubernetes/issues/121454</u>](https://github.com/kubernetes/kubernetes/issues/121454)

        - How are we building the vuln feed today? How do we handle
          security info from SRC?

          - Before, Google had a feed, now the community maintains a
            feed. Merged alpha version of CVE in 1.25, beta in 1.27. Now
            on k8s.io, JSON, RSS, markdown

          - Going into GA: getting feedbacks from SRC about how scanners
            ingest, how to make feed more consumable

          - \[Jeremy\] Should we start a google doc about where we were,
            and serve as a basis for roadmap and KEPs

            - [<u>https://docs.google.com/document/d/1a_3aMKexfhZwnbzmZZy0dhbQX7L_c8Cr2HSDKchLU7o/edit</u>](https://docs.google.com/document/d/1a_3aMKexfhZwnbzmZZy0dhbQX7L_c8Cr2HSDKchLU7o/edit)

          - Continue conversation tomorrow as well @ SIG-Security
            Toolings meeting

        - Info about the feed starts with SRC created issues, internally
          they share a YAML. There’s an opportunity to streamline this

        - \[PJ\] To clarify, we deal with 2 types of CVEs:

1.  Published by SRC with high confidence of impact, about Kubernetes

    1.  OK where we are but reduce repeat work for SRC

2.  Noisier, either from container images that the components published
    as or Go dependencies

    1.  This is a good use case for VEX

    2.  This is what we’re discussing today

        - We have scanner runs 4 times a day with Snyk.

        - Recently merge go-vuln check on branch

        - There is an issue open where the go-vuln check run on release
          branches

        - Noise from vulns in Go that we don’t use

        - No cherry-pick to silence scanner

        - \[PJ\] Idea: Additional labels where K8s can say this is not
          affecting K8s. Create a trail of trust

          - \[puerco\] it is how VEX works. Initial model is last mile
            of trust where the document can be sign and vouch for the
            statements.

          - \[PJ\] Limit labeling to owners to improve trust

          - \[puerco\] create a system where more than one experts can
            sign off on it

          - Nikhita has a thumbs-up page voting for TOC we can do
            something similar

          - 2 VEX flow:

            - VEX generated from K8s itself

            - Assessment on 3rd parties modules/images

          - We can merge this into one flow. Big question is
            implementation .

          - PJ needs drop continue conversation in sig-security tooling

        - What are we making? Specifically to VEX, a VEX document that
          can be consumed by CVEs scanners

        - How does it work for SRC? SRC triage, identify fixes, publish
          to embargoed list

          - Few source of truths

          - Release notes, issues that go into the CVE feed, data that
            goes into external feed

          - What we want: maybe create a centralized feed internally
            ([<u>OS feed)</u>](https://osv.dev/) that can feed to other
            feeds

          - Pertaining to SIG Release: use our existing tooling to help
            the effort and consume that data in our release process and
            SBOM change to support VEX document

        <!-- -->

        - Action Items/Help Wanted:

        <!-- -->

        - LFX project:

          - General Update: The LFX project is coming to the end. Next
            Thursday is the last day. Work on CLI is getting started and
            Nitish is interested to stay around and help us with the CLI
            after the internship!

    - Release Team

      - Review subteam merger comms -\> [<u>Release Team Change Comms -
        Google
        Docs</u>](https://docs.google.com/document/d/1QwdxigTCXYCX222tDCmdkFBLioLZAshJCZuLSW46enY/edit)

      - \[Review request\] Removal process
        [<u>documentation</u>](https://github.com/kubernetes/sig-release/pull/2322)

      - Have a look at new project board -\> [<u>Board · Release Team
        (github.com)</u>](https://github.com/orgs/kubernetes/projects/170/views/1)

        - Move over action items from retro doc

        - Not bound to release cycle, anyone can contribute

- \[puerco\] Cloud Native Security Slam - Kubernetes Lightning Round

  - Please review Comms  
    [<u>Doc calling for project
    participation</u>](https://docs.google.com/document/d/1nXaWs9ONjAVrM5FvMovHh6IU7a4AGWAuXqAOI7Hv5xs/edit)
    \| [<u>Event
    Website</u>](https://community.cncf.io/cloud-native-security-slam/)

- \[jerickar\] revisit cherry pick criteria? Follow up to regression
  data from [<u>liggitt@google.com</u>](mailto:liggitt@google.com)

  - Backporting leads to other regression

  - Too liberal cherry-pick right now. Don’t have strict criteria,
    deferring to SIG-Leads and reviewers. We don’t have a checklist

  - Would benefit if we do LTS later on and potentially less regression

  - Is this something we want to discuss? We own this because we own and
    ship cherry-pick

  - \[puerco\] our guideline is about what is allowed to be
    cherry-picked. How do we restrict without requiring release managers
    to verify

    - Suggestion from liggit is to change the issue template so folks
      can attest that it falls under what we allow (we also need to
      decide what we allow)

## 14 Nov 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Meha Bhalodiya (she/her)

- Sascha Grunert (he/him)

- Jim Angel (he/him)

- Sreeram Venkitesh (he/him)

- Joseph Sandoval (he/him/él)

- Adolfo García Veytia (he/him/él)

- Priyanka Saggu (she/her)

- Grace Nguyen (she/her)

- Sujay (he/him)

- Oluebube Princess Egbuna(she/her)

**Note Taker (pronouns):**

- Everyone

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - 

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update: \[Jeremey\]No patch releases this month. Cherry
        picks will fall into next month’s release.

      - Action Items/Help Wanted:

    - OBS Packages:

      - General Update:

      - Action Items/Help Wanted:

    - Supply Chain Security:

      - General Update:

        - \[Adolfo\] Discussed at Kubecon Chicago. What would it take
          for us to issue Vex documents? Vex is a format that helps to
          better inform if you are affected by a vulnerability. We tried
          to improve SBOMs. The verification and security feed was an
          area we were focusing on. Some improvements are needed to make
          them useful. The other topic is about having a unified
          security feed. Prototype that is running monitors GH and they
          provide information after. Public disclosures limit access to
          the fields. Requires more discussion with SIG-Security.

        - \[Jeremy\] Rita Zhang raised an
          [<u>issue</u>](https://github.com/kubernetes/kubernetes/issues/121454)
          with Vex documents. She has started to write the docs. It will
          be discussed at SIG-Security (see:
          [<u>https://github.com/kubernetes/community/tree/master/sig-security</u>](https://github.com/kubernetes/community/tree/master/sig-security)).

        - \[Joseph\] Have you received any user feedback?

        - \[Adolfo\] Limited. One other thing is fixing the Image
          Promoter. \#1 focus for myself.

        - \[Jeremy\] Marko and I were discussing and will share our
          notes.

      - Action Items/Help Wanted:

    - LFX project:

      - General Update:

  - Release Team

    - Beta.0 cut Thursday 11/16 (Veronica to start -\> handoff -\> Jim /
      Mark)

    - 1.29 Test Freeze today -
      [<u>PSA</u>](https://groups.google.com/a/kubernetes.io/g/dev/c/ukMtzsOHZDU/m/pF5HlappCAAJ)

    - Other 1.29 milestones for Nov 14, 2023:

      - Docs Deadline

      - Major Themes Deadline

      - Start final draft of Release Notes

    - [<u>Discussion/AI</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1699889130916229)
      from KubeCon NA 23 SIG Release in-person meeting

      - Will schedule dedicated follow-up discussion for EOR retro
        meeting & upcoming SIG Release weekly meeting.

- 

**  
Open Discussion:**

- \[Lauri / Sascha\] Roadmap and Vision prioritization for 2024:
  [<u>https://github.com/kubernetes/sig-release/blob/master/roadmap.md</u>](https://github.com/kubernetes/sig-release/blob/master/roadmap.md)

  - \[Sascha\] Put rough ideas in general board. Sync with Lauri about a
    potential dedicated session. Potentially enabled downstream
    consumers of K8s releases.

  - \[Adolfo\] SLSA spec is still being finished. 1) Move the SLSA
    tester 2)

  - \[Sascha\] Would this tie into the signing release artifacts?

  - \[Adolfo\] Yes. We should go back and modernize what we have built
    over the last two years and make sure they are all working
    interoperably.

  - \[Lauri\] I would need some time to review and discuss what has been
    done over the last year. Also understanding our users who they are?

  - \[Sascha\] It’s a mix. Do we have the capacity to work on next year?

  - \[Lauri\] How is the user participation?

  - \[Sascha\] Started strong and then we tapered off a bit with
    contributors.

  - \[Lauri\] Dedicated conversations about sessions to identify the
    biggest needs. We can use that data to help organize the work. We
    could also discuss onboard new contributors.

  - \[Jeremy\] Data to see who is using our tools? Maybe a survey.

  - \[Lauri\] Could we use CNCF to help get user feedback?

  - \[Jeremy\] Do we wait until we receive the user data before issue
    planning?

  - \[Lauri\] Ideally yes but do the must-do work you know is
    non-negotiable and that’ll buy time while the data situation gets
    sorted.

  - \[Jeremy\] Image promoter needs some attention and could be a focus.

  - \[Lauri\] We could walk through and identify the work. Focus on the
    goal.

  - \[Adolfo\] There’s a few barrier of entry with respect to the
    codebase and how fragile it is

  - \[Jeremy\] We might need to start over with docs and bring the
    current version over

  - \[Lauri\] We might need a retro with the current design, what’s the
    current problem and how to overcome it in the new version. What is
    it setting out to do? How do we do that core function really well?

    - Then break it down into pieces that folks can pick up

  - \[puerco\] Building a wider base conversation looks completely
    different whether or not we keep the existing system

  - \[Lauri\] Is it a monolith? (Yes)

    - \[Sascha\] we can have an issue where one or two people can start
      working on it

    - \[Lauri\] keep this out of the roadmap, since its a technical
      piece

- \[puerco\] Cloud Native Security Slam: [<u>Kubernetes Lightning
  Round</u>](https://community.cncf.io/cloud-native-security-slam/).

  - Maintainers for smaller projects can express interest and SIG
    release can provide clear guidance on how to adopt tools and if we
    have enough projects, we can guide them through their pull requests.
    CNCF would produce some special edition swag for helping out with
    these.

  - Plan to have a webinar on dec 6th if approved

- \[jeremy\] SRC/SIG Security - VEX discussion @ next sig security
  meeting - we covered this above

- \[jim\] oauth tooling for sending gmail via \`krel announce\`
  ([<u>issue</u>](https://github.com/kubernetes/release/issues/3360) /
  [<u>demo</u>](https://github.com/jimangel/oauth-gmail-test)):

  - Arnaud: we should try to do this after 1.29 ? Happy to provide any
    kind of help to do this so we can test it early 1.30 (all the alpha
    releases ?).

## Oct 31, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Priyanka Saggu (she/her)

- Meha Bhalodiya (she/her)

- Vyom Yadav (he/him)

- Marko Mudrinić (he/him)

- Christopher Hanson (he/him)

- [<u>Mansi Kulkarni</u>](mailto:mankulka@redhat.com)(she/her)

- Nishok Ganesan (he/him)

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Vyom from Release Team (CI Signal team), student

  - 

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update: No updates

    - OBS Packages:

      - General Update: No updates

    - Supply Chain Security:

      - General Update: No updates

    - LFX project:

      - General Update: Major work on library wrapped, Nitish started
        working on CLI

  - Release Team

    - \[Priyanka\]

      - 1.29 Code Freeze begins on [<u>01:00 UTC Wednesday 1st November
        2023 / 18:00 PDT Tuesday 31st October
        2023</u>](https://everytimezone.com/s/24b27b03)

      - 1.29.0-alpha.3 scheduled for Nov 2, 2023:

        - Current CI Signal is No-Go (Red), 4 Failing Release Informing
          Jobs (Fix PRs in progress, (capa) job marked as non
          release-blocking)

        - Ongoing discussion –
          [<u>https://kubernetes.slack.com/archives/CJH2GBF7Y/p1698702103104369</u>](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1698702103104369)

        - Question – If needed to delay the alpha.3 cut, thoughts/advice
          on when to reschedule considering KubeCon next week?

          - Given that only informing is failing, we could go forward
            with alpha.3 this week

          - etcd job can be ignored, it’s not relevant for this release

    - \[Priyanka\] Need help from Release Engineering group to create
      the k/test-infra PR for 1.29 Code Freeze Enforcement.

      - Jeremy to sync with Mark about this

    - \[Priyanka\] Jim is away at this time, Mark will be the only
      person responsible for release cuts

      - \[Marko\] Feel free to ping @release-managers group on Slack,
        Priyanka to inform release managers day prior to release cut

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- \[Marko\] Feedback on repo for OBS CLI

- \[Marko\] Kubernetes Contributor Summit @ KubeCon NA

  - SIG-Release in Person Meetup
    ([<u>https://sched.co/1TY7O</u>](https://sched.co/1TY7O))

  - A Practical Guide to Publishing System Packages for Kubernetes
    Subprojects
    ([<u>https://sched.co/1SpA6</u>](https://sched.co/1SpA6))

  - Jeremy request an unconference session to discuss supply chain
    security

## Oct 24, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Rudraksh Karpe (he/him)

- Josh Berkus

- Nishok Ganesan(he/him)

- Sreeram Venkitesh (he/him)

- Mark Rossetti (he/him)

- Anhelina Zelyk(she/her)

- Christopher Hanson (he/him)

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update:

        - \[Marko\] Patch releases for October went out successfully
          last week. No major issues besides two known issues with image
          promotion (replicating signatures) and sending announcements
          to k-dev

        - \[Marko\] Reminder that we don’t have releases planned for
          November because of Release Managers availability due to
          KubeCon/vacations/Thanksgiving

        - \[Marko\] Patch release date for beginning of 2024 are live on
          [<u>website</u>](https://kubernetes.io/releases/patch-releases/#upcoming-monthly-releases)

        - \[Marko\] Reminder that 1.25 is going EOL by end of this month

      - Action Items/Help Wanted :

        - \[Marko\] We need to remove 1.25 jobs once 1.25 reaches EOL.
          If anyone wants to help with this, reach out to us on
          \#release-management and we can pair on this

          - We’re still lacking docs and proper tooling for this and
            we’ll need to do it manually again. These PRs can give some
            idea what’s needed:

            - [<u>https://github.com/kubernetes/test-infra/pull/30563</u>](https://github.com/kubernetes/test-infra/pull/30563)

            - [<u>https://github.com/kubernetes/test-infra/pull/29387</u>](https://github.com/kubernetes/test-infra/pull/29387)

          - Mark R (marosset) can help

    - OBS Packages:

      - General Update:

        - \[Marko\] Documentation changes are backported to v1.27-v1.24
          docs. These docs are now advising users to use pkgs.k8s.io

        - \[Marko\] There are some documentation changes requested by
          SIG Docs and I’m working on implementing these changes

      - Action Items/Help Wanted:

        - \[Marko\] The [<u>project
          board</u>](https://github.com/orgs/kubernetes/projects/137/views/1)
          has been recently tidied up. If you see anything interesting
          there, you can reach out to me

    - Supply Chain Security:

      - General Update:

      - Action Items/Help Wanted:

    - LFX project:

      - General Update:

        - \[Marko\] The library part has been finished enough to start
          working on the CLI. Nitish will start working on the CLI this
          week

  - Release Team

    - \[Mark, Sreeram\] No major updates. Code freeze scheduled for next
      week.

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

## 17 Oct 2023

Meeting got canceled.

## Oct 10, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Anhelina Zelyk (she/her)

- Oluebube Princess Egbuna(she/her)

- Mark Rossetti (he/him)

- Joseph Sandoval (he/him)

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

    - Patch Releases:

      - General Update:

        - Cherry-pick deadline at October 13 (this Friday)

        - Releases scheduled for Wednesday, October 18

        - Reminder: we don’t have releases scheduled for November!

          - (Marko) This is due to Kubecon and release manager
            availability.

      - Action Items/Help Wanted:

        - Send a reminder to k-dev for cherry-picks along with reminder
          for packages

          - \[Jeremy\] Go releases should be updated before next patch
            releases.

          - \[Mark\] The Go patch releases will be included.
            [<u>net/http2
            issues</u>](https://groups.google.com/g/golang-announce/c/iNNxDTCjZvo/m/UDd7VKQuAAAJ)
            being discussed.

    - OBS Packages:

      - General Update:

        - Updating docs to remove references to the legacy package
          repositories across supported release branches

      - Action Items/Help Wanted:
        [<u>https://github.com/orgs/kubernetes/projects/137/views/1</u>](https://github.com/orgs/kubernetes/projects/137/views/1)

        - Review needed for
          [<u>https://github.com/kubernetes/website/pull/43407</u>](https://github.com/kubernetes/website/pull/43407)

    - Supply Chain Security:

      - General Update:

      - Action Items/Help Wanted:

    - LFX project:

      - General Update:

        - Two major tasks left for the library:

          - CRUD operations for OBS packages

          - Implementing client interface/structs

        - After these two major tasks are done, the plan is to start
          working on the CLI tool.

  - Release Team

(Priyanka - won’t be able to join the meeting, please find the updates
below. Thanks!)

- 1.29 [<u>Enhancements Freeze in
  effect</u>](https://groups.google.com/a/kubernetes.io/g/dev/c/AttpmuN6Hfo/m/Wn3M67FjAwAJ),
  58 tracked, 2 Exception Requests received – both approved

- 1.29.0-alpha.2 planned for today, CI Signal is green for now

  - \[Mark\] Release in progress atm.

- Removed “capg-conformance-main-ci-artifacts” job from  
  “sig-release-master-informing” board -
  [<u>https://github.com/kubernetes/test-infra/pull/30988</u>](https://github.com/kubernetes/test-infra/pull/30988)

- Ongoing discussions – 1.29 mid-cycle Deprecations/Removal and Major
  changes blog

<!-- -->

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- Arnaud: *Launching registry.k8s.io’s new regions*:

  - [<u>https://github.com/kubernetes/registry.k8s.io/issues/236</u>](https://github.com/kubernetes/registry.k8s.io/issues/236)

    - \[Arnaud\] Add more s3 buckets which will help reduce costs.
      Launching new regions has impact on promotion process. Possible
      timeout and potential failures during this rollout.

    - \[Marko\] Speaking of S3 buckets IIUC. This is a dedicated job and
      not connected to promotion. Why do we need to add additional
      artifact registries?

    - \[Arnaud\] We try to match a artifact registry with a S3 bucket.
      Configuration of image and the blob. We pick one image and
      replicate. Put the blob with one image in S3.

    - \[Joseph\] Is it a infrastructure strategy to align artifacts with
      region?

    - \[Arnaud\] Goal is to opimize costs with this approach. Focus
      where the costs are high. Join SIG-K8s-Infra meeting to discuss
      costs.

    - \[Jeremy\] Is there good docs to reference about this?

    - \[Arnaud\] I’ll add some references into the SIG-release meeting
      notes.

      - [<u>https://github.com/kubernetes/registry.k8s.io/tree/main/cmd/archeio</u>](https://github.com/kubernetes/registry.k8s.io/tree/main/cmd/archeio)

      - [<u>https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io)</u>](https://github.com/kubernetes/k8s.io/wiki/New-Registry-url-for-Kubernetes-(registry.k8s.io))

    - \[Jeremy\] Any actions needed?

    - \[Arnaud\] Not at this time. This is for awareness.

- \[jeremy\]: Contributor Summit

  - Didn’t request a SIG Release meeting, should we ask for an
    unconference session?

  - Jeremy submitted an “lts” unconference and a “security artifacts”
    discussion topic:

    - [<u>https://github.com/kubernetes/community/issues/7531#issuecomment-1755709900</u>](https://github.com/kubernetes/community/issues/7531#issuecomment-1755709900)

    - \[Joseph\] What is our goal for meeting at Kubecon?

    - \[Jeremy\] Image promoter would benefit from discussion. Its been
      awhile since we did SBOM’s. Maybe there is more artifacts we would
      like to add.

    - \[Verónica\] SLSA has changed alot.

    - \[jeremy\] meet the contributors link:
      [<u>https://github.com/kubernetes/community/issues/7541</u>](https://github.com/kubernetes/community/issues/7541)

## Oct 3, 2023

**Host (pronouns):** Stephen Augustus (he/him)

**Attendees (pronouns):**

- Jordan Liggitt

- Leonard Pahlke (he/him)

- Grace Nguyen (she/her)

- Marko Mudrinić (he/him)

- Sreeram Venkitesh (he/him)

- Amit Dsouza (he/him)

- Priyanka Saggu (she/her)

- Oluebube Princess Egbuna(she/her)

- Jayson Du (he/him)

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Jayson Du from Amazon EKS team

  - Abhijeet Gaurav - contributing to CNCF projects

- Release Engineering

  - Patch Releases: *please mention any issues that may affect upcoming
    or recent patch releases.*

    - General Update:

      - \[Jim Angel\]

      - Patch releases went out last week. No major issues other than
        image signing and sending announcements to k-dev

    - Action Items/Help Wanted: *(please include links to
      issues/context)*

      - \[Marko\] Any updates on replacing SendGrid with something else?

        - \[Stephen\] Not sure that there’s an update at the moment

        - \[Veronica\] No solution at the moment, Jim has some solution
          that might work for us

  - OBS Packages:

    - General Update:

      - \[Marko\] No major updates, we’re seeing more and more traffic
        going over pkgs.k8s.io, daily average is about 200-300 GB

      - \[Marko\] Work on this to pick up again soon

    - Action Items/Help Wanted:
      [<u>https://github.com/orgs/kubernetes/projects/137/views/1</u>](https://github.com/orgs/kubernetes/projects/137/views/1)

      - \[Stephen\] Can we send a reminder for freezing legacy repos
        this week along with a reminder for backports?

        - \[Marko\] Yes, will take care of that

  - Supply Chain Security:

    - General Update:

    - Action Items/Help Wanted:

  - LFX project:

    - General Update:

      - \[Marko\] Nitish finished with CRUD operations for Projects,
        next step is to write tests

- Release Team

  - \[Priyanka\]

    - 1.29 Enhancements Freeze this week – Oct 6, 2023

      - Team finished reaching out to KEP owners, reminder for chairs
        and leads

    - 1.29.0-alpha.2 planned for Oct 10, 2023 (next week)

      - Delayed last week, new target date next week

    - v1.29 Mid-Cycle Release Retro scheduled for Oct 19, 2023

    - Question to Marko/Jim –
      [<u>https://github.com/kubernetes/sig-release/issues/2291</u>](https://github.com/kubernetes/sig-release/issues/2291)
      is this information something that we need documented before our
      next 1.29 release cuts?  
      If yes, could we please prioritize this? Thanks in advance!

      - \[Marko\] We definitely need to document this, but this is not a
        blocker for 1.29, all OBS projects are already in place that we
        need for 1.29. As part of our LFX project, we’re making
        improvements to the process, and we plan to document everything

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- \[liggitt, 5 minutes\] review of go "major"-version bumps on release
  branches

  - Context:
    [<u>https://github.com/kubernetes/enhancements/issues/3744</u>](https://github.com/kubernetes/enhancements/issues/3744)

  - Aspects:

    - Timeline/checklist for bumping release branches

    - Tracking backports related to new go versions

    - [<u>go1.20
      example</u>](https://github.com/kubernetes/release/issues/2815#issuecomment-1373891562)

    - [<u>go1.21
      example</u>](https://github.com/kubernetes/release/issues/3076#issuecomment-1556053267)

  - Q: What is the best way for multiple release team members to
    collaborate tracking go minor-version related backports and tasks?

    - Something on github seems ideal to track merged state of
      individual prereq backports

    - Description / comment only editable by author or folks with repo
      edit access

      - This seems like a good place to start

    - Github project might be a possibility

  - Q: What issue templates or playbooks should be updated to make this
    a regular part of go minor version bumps?

    - AI(liggitt): update
      [<u>https://github.com/kubernetes/sig-release/blob/master/release-engineering/handbooks/go.md</u>](https://github.com/kubernetes/sig-release/blob/master/release-engineering/handbooks/go.md)

      - [<u>https://github.com/kubernetes/sig-release/pull/2364</u>](https://github.com/kubernetes/sig-release/pull/2364)

    - AI(?): update
      [<u>https://github.com/kubernetes/release/blob/master/.github/ISSUE_TEMPLATE/dep-golang.md</u>](https://github.com/kubernetes/release/blob/master/.github/ISSUE_TEMPLATE/dep-golang.md)

      - 

  - Question: How did you come up with the list of changes?

    - Changes that we discovered were required to fix CI failures before
      master could successfully update to new go version

    - Rarely - follow-ups post-merge after updating master

  - Question: where are post-submit tests?

    - [<u>https://testgrid.k8s.io/sig-release</u>](https://testgrid.k8s.io/sig-release)

    - [<u>https://testgrid.k8s.io/sig-release-master-informing</u>](https://testgrid.k8s.io/sig-release-master-informing)

    - [<u>https://testgrid.k8s.io/sig-release-master-blocking</u>](https://testgrid.k8s.io/sig-release-master-blocking)

  - Stephen: risk is not having a single person with a view to the whole
    lifecycle of picking up a new major go version (easy for handoff to
    have things fall through the cracks)

- \[liggitt, 20 minutes\] Review of regression data from 1.19 to present

  - [<u>Kubernetes patch release regression/bugfix
    rate</u>](https://docs.google.com/spreadsheets/d/1LbGKBC4D2sLkcmzY9qDx9u-1D9TKC_ZrM8iA1eHW4Hs/edit#gid=1283859152)

  - [<u>Analysis of Kubernetes regression rates, patterns,
    examples</u>](https://docs.google.com/document/d/1Vr-m-KL7P2KYDPsvfpJyFVW1Q6XPS5d8xz9xCq_Azpw/edit?pli=1#bookmark=id.h4sqtkkivzm0)

  - Interesting data points

    - ~50% more regressions come from bugfix/cleanup work than from
      feature work

    - 90% of minor versions \>= 1.19 have regressed in a patch release
      (only 1.28 hasn't)

    - 28% of patch releases \>= 1.19 contained a regression relative to
      the .0 release

  - Q: Should the "backport to all branches simultaneously" policy
    change for some types of backports (non-security, non-data-loss,
    non-regression) in light of several backports that caused
    regressions in patch releases?

    - Veronica:

      - current process doesn't give branch managers much decision input
        to backport / merge decisions

      - thinks backport call should be on approvers for whether
        backports should go to all branches

      - find ways to make backport decisions for branch managers
        feasible without all context

    - Stephen:

      - mechanical part of the backport process (traffic direct in /
        branch manager approval)

      - double check backport meets cherrypick guidelines

    - Jordan

      - opportunity to augment cherry pick guidelines to guide area
        approvers?

      - highlight categories of backports that regressed patch releases:
        big changes, entangled changes, changes in undertested areas,
        changes impacting out-of-tree library users (scheduler plugins,
        aggregated apiservers)

      - 

    - Veronica

      - branch managers do already dig into details and trying to
        understand context / severity already

      - if reviewing / approving cherry picks requires more detail, do
        we need more time between deadline and patch cut?

        - Stephen: backport authors / area owners need to feel ownership
          for their areas and meeting schedules; for normal operations,
          those schedules should hold; still need to accommodate
          unforeseen or urgent fixes

  - Q: What playbooks should be updated to make review of the regression
    labeling / release note for backports part of branch manager
    approval?

    - Release note includes the specific minor / patch where the
      regression was introduced (good for users and helpful for us)

    - Ensure kind/regression label is only set on backport PRs fixing a
      minor where the regression was introduced

    - Ensure kind/feature label is only set on backport PRs fixing a
      regression related to feature work in that minor

    - 

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

## Sept 26, 2023

**Host (pronouns):**

- 

**Attendees (pronouns):**

- 

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

  - Release Team

    - (Priyanka - won’t be able to join the meeting, please find the
      notes/updates below. Thanks!)

      - Question to Marko/Jim –
        [<u>https://github.com/kubernetes/sig-release/issues/2291</u>](https://github.com/kubernetes/sig-release/issues/2291)
        is this information something that we need documented before our
        next 1.29 release cuts?  
        If yes, could we please prioritize this? Thanks in advance!

      - Updates:

        - 1.29 PRR Freeze on Sept 28th

        - 1.29 Enhancements Freeze approaching next week - Oct 6th

        - 1.29 Alpha.2 delayed (was tentatively planned for this week).
          Will discuss with Jim/Mark for the timeline.

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

## Sept 19, 2023

**Host (pronouns):** Carlos Panato

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Sascha Grunert (he/him)

- Priyanka Saggu (she/her)

- Nitish Kumar (he/him)

- Fyka Ansari (she/her)

- Paco Xu(he/him)

- Anhelina Zelyk (she/her)

- Sreeram Venkitesh (he/him)

- Joseph Sandoval (he/him)

- Leonard Pahlke (he/him)

- Jim Angel (he/him)

- Mengjiao Liu (she/her)

- Jeremy Rickard (he/him)

- Ricky Sadowski (he/him)

- Rey Lejano (he/him)

- Amit Dsouza (he/him)

- Michael Singh (he/him)

**Note Taker (pronouns):**

- Joseph

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Amit Dsouza from Australia. Contributor to Argo. Trying to become
    more active.

  - Mengjiao Liu - release notes shadow. Learn about release process.

  - Michael S - Looking for things that he can be a candidate to
    contribute. Part of 1.12/1.13.

- Subproject updates

  - Release Engineering

    - Patch Releases: *please mention any issues that may affect
      upcoming or recent patch releases .*

      - General Update:

        - \[Jim Angel\]

        - Patch releases went out last week. No major issues other than
          image signing and sending announcements to k-dev

      - Action Items/Help Wanted: *(please include links to
        issues/context)*

        - Patch releases calendar needs to be updated to include the
          latest patch releases:
          [<u>https://kubernetes.io/releases/patch-releases/</u>](https://kubernetes.io/releases/patch-releases/)

          - Example PR:
            [<u>https://github.com/kubernetes/website/pull/40100</u>](https://github.com/kubernetes/website/pull/40100)

          - \[Marko\] Staging job flaked two times for 1.26. Something
            about not enough memory. Not sure why this is happening.
            Action items: patch release calendar needs to be updated.
            This is open to anyone to do it quickly and open to try.
            Above is the sample PR. If not done by tomorrow Marko will
            take care of it.

          - \[Amit Dsouza\] I can take it.

          - \[Michael S\] We can all pair up on it.

      - Alpha cut 1.29 is planned for today. Still blocked.

        - \[Priyanka\] Flaky job is blocking. [<u>Stateful
          set</u>](https://github.com/kubernetes/kubernetes/issues/120700)
          is blocking. CI Signal shadow is following up to determine if
          this is still blocking.

        - \[Jim Angel\] Will see if alpha blocking or not and follow up
          with the team on Slack.

        - \[Priyanka\] Dims suggested keep it on the board and try and
          fix it. Its release informing.

        - \[Carlos P\] CAP provider GCE is deprecated. In touch with the
          maintainers. The work to resolve the issue is almost done. The
          job will be green again later in the week. The deploy needs to
          change to resolve this issue. This doesn’t impact Alpha 1.

    - OBS Packages:

      - General Update:

        - Issue with kubeadm packages

          - [<u>https://github.com/kubernetes/release/issues/3276</u>](https://github.com/kubernetes/release/issues/3276)

        - LFX project: merged API types for managing OBS Projects

          - [<u>https://github.com/kubernetes-sigs/release-sdk/pull/246</u>](https://github.com/kubernetes-sigs/release-sdk/pull/246)

        - \[Marko\] There is pr to fix. This issue only applies to patch
          releases. Can’t edit existing packages. Find way to
          communicate this issue. I’ll look at this issue tomorrow.

        - \[Priyanka\] Today’s alpha cut will have the same issue?

        - \[Marko\] Alpha cut will not have any issues. But we can
          verify with this cut. Status of LFX projects. This is the 3rd
          week starting on this. Updates coming next week. AI - anything
          on the board you would like to work on just reach out to
          Marko.

      - Action Items/Help Wanted: *(please include links to
        issues/context)*

        - Project board:
          [<u>https://github.com/orgs/kubernetes/projects/137/views/1</u>](https://github.com/orgs/kubernetes/projects/137/views/1)

        - 

    - Supply Chain Security:

      - General Update:

        - \[Sascha\] No updates at the moment.

      - Action Items/Help Wanted: *(please include links to
        issues/context)*

  - Release Team

    - \[Priyanka\] Discuss: whether/not to demote the
      [<u>capg-conformance-main-ci-artifacts</u>](https://testgrid.k8s.io/sig-release-master-informing#capg-conformance-main-ci-artifacts)
      job from release-informing dashboard

> (ref: [<u>slack
> discussion</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1694672335486569))

- 1.29.0-alpha.1 planned for today, current CI Signal is NO-Go, waiting
  for follow-up on a flaky job  
  [<u>https://github.com/kubernetes/kubernetes/issues/120700</u>](https://github.com/kubernetes/kubernetes/issues/120700)

- Others:

  - Shadow orientation meetings planned for this week!

    - \[Priyanka\] Invites have been sent.

    - \[Carlos P\] When are they happening?

    - \[Priyanka\] Both are on Thursday

<!-- -->

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

## Sep 12, 2023

**Host (pronouns):**

**Attendees (pronouns):**

- Marko Mudrinic

- Nitish Kumar

- Rudraksh Karpe

- Jeremy Rickard

- Veronica

- Sreeram

- Gracenguyen

- Mengjiao Liu

- Arnaud

- Jim Angel

- Nishok

- Mahamed Ali

- Mark Rossetti

**Note Taker (pronouns):**

- Marko Mudrinić

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Nitish - SIG Contribex Comms and LFX project for SIG Release

  - Sreeram - part of the v1.29 Release Team

  - Rudraksh - exploring the landscape

- Subproject updates

  - \[Veronica\] New proposed structure for Release Engineering updates

    - Action Item: Veronica will update the meeting agenda to reflect
      new structure for updates

    - We’re missing a lot of context on various topics, like OBS, image
      signing, SBOMs…

    - It would be nice to have structured updates so that we make sure
      we provide proper updates and that folks are aware how are
      different areas doing

    - A lot of folks want to join and help, but they’re not sure how and
      where

    - We’ll use current format today and figure out new format going
      forward

    - Have a link to project boards so folks can check if there’s
      anything to pick up

    - We encourage people to come to the meeting, but that might not
      always be possible due to conflicts/time zones. The template
      should be covering enough stuff so that folks not attending can
      figure out what’s going on

  - Release Engineering

    - \[Marko\]

      - Go updated to 1.21.1/1.20.8, thanks to Carlos!

      - Go update to 1.21 still ongoing, thanks to Carlos and Sascha for
        dealing with this and base image updates!

      - kubepkg and rapture are considered deprecated and will be
        removed after October patch releases:
        [<u>https://groups.google.com/a/kubernetes.io/g/release-managers/c/4Cl479abBaE/m/EFNfChPrBAAJ</u>](https://groups.google.com/a/kubernetes.io/g/release-managers/c/4Cl479abBaE/m/EFNfChPrBAAJ)

      - OpenBuildService projects for v1.29 are now in place, there’s a
        Slack thread describing the process step-by-step

      - Patch releases are scheduled for tomorrow, thanks to Veronica
        for volunteering to help with these releases!

      - Reminder that we’re freezing the legacy repositories tomorrow!
        [<u>https://k8s.io/linuxrepos</u>](https://k8s.io/linuxrepos)

      - We had OpenBuildService and pkgs.k8s.io introduction last week,
        slides and recording are posted on \#release-mangement

  - Release Team

    - \[Priyanka\]

      - 1.29 Lead Shadow on-boarding complete -
        [<u>https://github.com/kubernetes/sig-release/issues/2329</u>](https://github.com/kubernetes/sig-release/issues/2329)

      - 1.29 Shadow Selection done, all applicants are informed of their
        selection status. Shadow onboarding in progress:

        - [<u>\[1.29\] add release team shadow information
          \#2339</u>](https://github.com/kubernetes/sig-release/pull/2339)

        - [<u>\[1.29\] update google groups/GCP IAM membership for
          release team shadows
          \#5831</u>](https://github.com/kubernetes/k8s.io/pull/5831)

        - [<u>\[1.29\] add release sub-teams role shadow to
          release-team-\* gh team
          \#4452</u>](https://github.com/kubernetes/org/pull/4452)

      - <span class="mark">\`1.29.0-alpha.1\` is scheduled for **Tuesday
        19th September 2023** (confirmed for Branch Manager
        availability -
        [<u>discussion</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1694181829355399?thread_ts=1694163871.580529&cid=C2C40FMNF))</span>

      - <span class="mark">Updated [<u>Kubernetes Release
        Calendar</u>](https://bit.ly/k8s-release-cal) for 1.29 release
        cycle milestones (pending release cut dates timeline - waiting
        for input from Branch Manager)</span>

      - <span class="mark">Scheduled 1.29 Weekly Release Team Meetings
        (first meeting happened on Sept 6, 2023)</span>

      - <span class="mark">Scheduled optional 1.29 Release Leads sync
        call (first meeting happened on Sept 11, 2023)</span>

      - <span class="mark">\[Jim\] We need to decide on if we need 4th
        alpha cut, it was added last cycle to test the OBS integration,
        so we can remove it</span>

        - <span class="mark">\[Jeremy\] +1 to removing it</span>

        - <span class="mark">\[Priyanka\] Thanks Jim for adding the
          context, and +1 to removing it.</span>

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

- \[Grace\] Looking for additional reviews and comments on the
  [<u>release team member removal
  process</u>](https://github.com/kubernetes/sig-release/issues/2332)
  before I draft up the PR

  - Call to action to review it, discussion going on in the issue

**  
Open Discussion:**

- \[Arnaud\] Deprecate gs://kubernetes-release

  - With Fastly shielding the bucket we can switch to a community-owned
    bucket. Ideally, the new bucket will contain objects for each
    release and will be private.

    - [<u>https://github.com/kubernetes/k8s.io/issues/2396</u>](https://github.com/kubernetes/k8s.io/issues/2396)

  - We serve all the binaries that we build from this bucket. It’s owned
    by Google. We don’t have a full access to that bucket

  - With the rollout of Fastly, we’re shielding the bucket and can
    switch to a community-owned bucket. How to establish the lifecycle
    of objects through the release, and improve cache efficiency. We
    have full control and can do more things

  - Start discussion on how we can move

  - Do we want to do it in v1.29?

  - New bucket should be private so we exclusively serve object through
    Fastly

  - We can backfill, GCS protocol supports that, we are just deprecating
    the old bucket, we’re not going to remove it, at least not at this
    time

    - We can backfill it from the beginning with all releases

  - \[Mahamed\] How long is dl.k8s.io GA for?

    - More than 5 years

  - \[Mahamed\] Should we make the old bucket private?

    - No, we don’t want to do it. There are also CI artifacts in that
      bucket

    - We can also move CI artifacts, but focus is on anything supported
      for community, CI artifacts are to follow up

  - We definitely need to communicate this, but we need to figure out
    how

  - \[Marko\] Can we push both to old and new buckets?

    - Could be doable

    - \[Marko\] Two milestones for deprecation and freezing

      - +1 from Mahamed, also to figure out CI stuff

- \[Arnaud\] Migrate away for project *kubernetes-release-test*

  - Security exceptions were filled to keep use this project. We should
    start look on how we migrate away from this project.

  - Short window for the transit. Try to do this before rc1 of 1.29

  - kubernetes-release-test is a Google-owned project. This is tricky,
    we have to do it at once, we have to just switch-over

  - We need to do inventory, some things to take care of:

    - Google Cloud Build

    - GCS buckets

    - KMS

    - ServiceAccounts

- \[Arnaud\] Shutdown
  [<u>https://release.triage.sigs.k8s.io</u>](https://release.triage.sigs.k8s.io)

  - Not currently maintained.

  - \[Jeremy\] I think we can shutdown this

  - \[Marko\] +1 to shutting it down

  - \[Anhelina\] Release Team is mostly using private chats for triaging

  - \[Arnaud\] If anyone wants it, we need to identify a team that’s
    going to maintain it

    - \[Jeremy\] Let’s call out on Slack and see if anyone is using it

  - \[Marko\] Is it even up?

    - \[Arnaud\] It has been broken for some time

## Sep 5, 2023

Meeting got canceled.

## Aug 29, 2023

**Host (pronouns):**

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Mark Rossetti (hi/him)

- Mickey Boxell (he/him)

- Kat Cosgrove (she/her)

- Natali Vlatko (she/her)

- Sreeram Venkitesh (he/him)

**Note Taker (pronouns):**

- Grace Nguyen (she/her)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

    - \[Veronica\] Release Manager check-in survey

    - \[Jeremy\] we will be halting publishing packages to legacy repos

      - Comms for sep/oct freeze

    - \[Marko\] Please fill the Doodle for OBS meeting planned for next
      week:
      [<u>https://doodle.com/meeting/participate/id/dLJygKWa</u>](https://doodle.com/meeting/participate/id/dLJygKWa)

  - Release Team  
    (Priyanka - won’t be able to join the meeting, please find the
    updates below. Thanks!)

    - \[Priyanka\] All Release Lead Shadows Selection done (&
      [<u>announced</u>](https://github.com/kubernetes/sig-release/issues/2307#issuecomment-1693370438))

    - \[Priyanka\] [<u>Captured feedback from 1.28 release cycle
      retro</u>](https://github.com/kubernetes/sig-release/issues/2313#issue-1859099061)
      and planned to work on them during 1.29

    - \[Priyanka\] Discussed async with SIG Release Leads:

      - Decided to keep CI Signal & Bug Triage separate for 1.29 due to
        no documentation for the new combined role (Release Signal).  
        We'll gradually merge the two existing handbooks to form the
        "Release Signal" role handbook during the 1.29 cycle. ([<u>wider
        group
        update</u>](https://github.com/kubernetes/sig-release/issues/2307#issuecomment-1697341504))

      - <span class="mark">Also, for the 1.29 "Bug Triage" Role, I've
        discussed with Furkat, and we have a potential candidate with a
        +1 from Grace.  
        I'll confirm their interest and availability.</span>

        - <span class="mark">[<u>Announced</u>](https://github.com/kubernetes/sig-release/issues/2307#issuecomment-1697343071)
          Bug Triage Role Lead (Yigit Demirbas)</span>

    - <span class="mark">\[Priyanka\] Reminder: The 1.29 Shadow
      Selection application closes in 3 days on Sept 1, 2023.</span>

    - <span class="mark">\[Priyanka\] 1.29 Release Cycle begins on Sept
      4, 2023! 🎉</span>

      - <span class="mark">Timeline Planning in progress -
        [<u>https://github.com/kubernetes/sig-release/issues/2313</u>](https://github.com/kubernetes/sig-release/issues/2313)</span>

      - <span class="mark">**Request -** Need lgtm/approval to merge the
        timeline PR if no objections!</span>

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- \[RT Retro Item added by Leo\]

  - \[Grace\]: If a shadow is unresponsive, have a discussion and remove
    them

    - What is the actual process for this? We’ve never done this
      before - not true

    - \[Josh\] Not at all accurate. When we introduced the shadow system
      in 1.10-1.15, we removed shadows all the time when they dropped
      out. Probably 50% of shadows dropped out, which is why we went
      from 1 to 2 shadows per lead.

    - \[Jeremy\] We have done this in the past just no docs. Something
      we should do. You are signing up to do work.

    - \[Stephen\] Signaling to folks, this is a resume builder as it’s
      not easy to get into

      - It is work to onboard, offboard and stay on the team and not
        fair to people doing the work

      - Leads should be looked at the same way

    - \[Mickey\] +1 to removing inactive folks from the team

    - \[Natali\] Unresponsive lead, rank of shadows

      - \[Grace\] we often have 2 return shadows

    - \[Kat\] [<u>PR to
      document</u>](https://github.com/kubernetes/sig-release/issues/2127#event-8791241644)
      for a couple releases

    - \[Grace\] [<u>PR in progress to capture
      this</u>](https://github.com/kubernetes/sig-release/pull/2322/files)

    - \[Josh\] when we start a shadow program, it's pretty normal to
      drop out. We expect 50% dropout

      - Shadows communicate if they can’t continue

      - EA introduced and help dismiss shadows

      - \[Grace\] I think 50% drop will put some teams in a tough
        positions like enhancements

    - \[Stephen\] Hard limit 4-5ish, as long as you can manage. Docs to
      remove is not documented. What does backup and graceful removal
      look like?

      - Stable roster is an option. Presumption of stability which is
        not true and won’t work

    - \[Josh\] Still, if we need backfill, having a call list of folks
      who have done the position before but aren't exactly shadows would
      be worthwhile.

    - \[Mickey\] Warm pool of existing shadows. Reach out and ask about
      time availability, for backups

      - \[Grace\] return shadows step up for lead, and pool is for newer
        shadows

      - \[Stephen\] Check out the comments on the Stable Roster KEP:
        [<u>https://github.com/kubernetes/enhancements/pull/3347</u>](https://github.com/kubernetes/enhancements/pull/3347)

      - \[Jeremy\] Few ways to back few: no release lead, look through
        list of folks in past release (comms lead, EA)

      - \[Jeremy\] EA checking the pulse more. If they don’t attend
        orientation, that’s a flag

    - \[Stephen\] Document backfill. Top role for EA is to make sure
      that retro items don’t drop

    - \[Natali\] Do we have how available we are as a removal criteria

    - \[Kat\] when they sign up as a shadow, they are agreeing to show
      up meeting. Don’t miss 2 meetings without comm to lead

    - \[Stephen\] multiple routes of escalation.

      - Shadow not supported by lead - go to lead or EA

      - Lead not supported by shadows - lead or EA

      - Bidirectional - lead to lead

    - Removing shadows:

      - 

    - Removing lead:

      - SIG flags to release lead that role lead is not responsive

      - Docs - branch sync is a signal

      - Attend meeting or send proxies

      - Missing deadlines

    - \[Stephen and Jeremy\] resolve this in an issue. Escalation path

  - \[Priyanka\]: It is now possible to make a copy\* of an existing
    GitHub Beta project
    ([<u>details</u>](https://docs.github.com/en/issues/planning-and-tracking-with-projects/creating-projects/copying-an-existing-project))

    - \[Grace\] what is this in relation to?

  - \[Grace\]: Comms and Docs lead attendance to sig-docs meeting
    mandatory

  - \[Grace\]: Perhaps creating template for docs and comms in sig-docs
    notes that caters to their specific tasks like updating branch, PRs?

    - \[Stephen\] SIG-Docs lead if they are not showing up at meeting,
      let us know

    - Captured on Priyanka’s retro todo list

    - https://github.com/kubernetes/sig-release/issues/2313#issue-1859099061

  - \[Grace\]: Review release timeline at the end of meeting

    - \[Stephen\] Moving towards a deliverables.

  - \[Leo\]: Reintroduce the RT Lead sync meeting to keep in touch
    between leads, get to know each other and slowly surface any
    problems they may have

    - \[Jeremy\] this was useful when we did it.

    - \[Jeremy\] would also suggest a kick off

  - \[Leo\]: Make RT Lead handovers more a thing, right now they are not
    part of the playbook

    - \[Mickey\] not enough continuity. Current lead doesnt know what
      they don’t know. Coordinate directly instead of over slack

    - \[Natali\] add returning shadows to that handover if possible

    - \[Kat\] continuity after the release. Feel confident and know who
      to lean on

    - \[Stephen\] take intent with the role. Part of succession is to
      support shadows

    - **Grace checkin with handbook about handover**

  - \[Leo\]: Include sig feedback upon lead succession (like sig-docs,
    sig-test-infra)

    - \[Natali\] earn shadow role again before leading

  - <span class="mark">\[Atharva\] We, the enhancements team, are
    working on adding the Release Team update statuses (Red, Yellow,
    Green) in our role handbook. Would be helpful if other teams include
    it as well.
    [<u>Issue</u>](https://github.com/kubernetes/sig-release/issues/2321)</span>

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- 

## Aug 22, 2023

**Host (pronouns):** Sascha Grunert (he/him)

**Attendees (pronouns):**

- Joseph Sandoval (he/him)

- Rey Lejano (he/him)

- Anhelina Zelyk (she/her)

- Jeremy Rickard (he/him)

- Leonard Pahlke (he/him)

- [<u>Adolfo García Veytia</u>](mailto:puercozon@gmail.com) (he/him/él)

- Rakshit Gondwal (he/him)

- Aashish Nehete

- Mahamed Ali

- Rajib Mitra (he/him)

- Verónica López (she/her)

- Rudraksh Karpe (he/him)

**Note Taker (pronouns):**

- Grace Nguyen (she/her)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Welcome Rakshit Gondwal!

- Subproject updates

  - Release Engineering

    - Last week, Jeremy brought up failure on signing. It is now common

      - On infra/registry change, noone has had time to update the
        process

      - Jeremy rallying to get together to fix issue and remediate the
        signature

1.  Fix the bug

2.  Go back in the registry and update previous signatures

    - Patch releases scheduled for tomorrow, Jeremy will drive them.

      - Morning US time, limited avail with Google admin

      - \[Veronica\] Happy to kick off in her timezone.

      - Organized for that, will sync with Veronica offline

    - Preparing release manager check-in form, incoming to mailing list

      - See who is still interested and opening up opportunities

    <!-- -->

    - Release Team:

      - Priyanka and Xander as Lead and EA for 1.29 🎉

      - 1.29 [<u>shadow
        survey</u>](https://docs.google.com/forms/d/e/1FAIpQLSfo4EXVqjlHzz2QPeNqar8dprZV06ETm740VQASEQLHRCX1tw/viewform?usp=sf_link)
        is out

      - PR for 1.29 release timeline is up -
        [<u>https://github.com/kubernetes/sig-release/pull/2314</u>](https://github.com/kubernetes/sig-release/pull/2314)

        - As per liggit’s feedback - have moved “Code Freeze '' to week
          9 (before KubeCon) - need more feedback/eyes on this change!

      - Retro tomorrow - lots of discussion docs and comms

      - There’s interest for an **Intro to release eng** session

        - What goes into cutting a release, cherry picks, etc

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- The talk proposal for Kubecon - maintainer track has not been
  announced yet

- Next steps for fixing signing step:

  - Context: when we cut a release, we build things and move them to
    buckets and registry. Part of that process is image promotion, build
    in one place, sign and copy to the registry. Noone can modify the
    images. Sign in build, carry signature to production registry and
    then image promoter add final signature, so two signatures. This was
    designed for 3 mirrors, now it is more complex and resilience at ~20
    mirrors

  - Process of copy and replicate signatures is failing. We run out of
    quote on GCP because we’re hitting the registry too much

  - Adolfo’s proposal: sign one image , replicate async

  - Result: inconsistent signatures in the past couple cuts. Registry
    might not have a copy of signature to verify.

  - New subcommand ([<u>kpromo
    sigcheck</u>](https://github.com/kubernetes-sigs/promo-tools/pull/767))
    that checks which images in a registry - code is written. This will
    allow us to proceed even if the signing process fails, the cron job
    will come in async and fix up the signatures

  - Jeremy - set up a sync meeting

    - Adolfo: +1

  - Sascha: Do we plan to outline this in an enhancements?

    - [<u>Issue</u>](https://github.com/kubernetes/release/issues/2962)

    - We have signing artifact enhancements to be GA

- Mahamed: update the whole image promoter as part of this?

  - Adolfo: it’s a dream but it’s a big project

  - Redesign how we replicate signatures to be more resilience, less
    real-time

  - Sascha - an idea discussed at Kubecon Amsterdam, sign one image and
    have it as root of trust

- We are close to quota limit and we could raise it. However, the
  process is not good and we should improve it

- Next steps: next sig-release meeting or another dedicated time

  - Doodle coming out today to sig-release by Jeremy

- Aashish: auto-fetch release notes, deprecate subcommand that creates
  PR

  - Sascha - Deprecate would be good, docs about this in handbook so we
    should remove those part

  - We moved away from Google Analytics but maybe on Netlify

  - Rey - Netlify shows that site analytics is not enabled for the
    release notes site

- Adolfo: Dependabot opens PR with \`@\` that’s being rejected by prow
  ([<u>example</u>](https://github.com/kubernetes-sigs/release-notes/pull/560))

  - Is it fixed?

  - Sascha: Dependabot now supports grouping updates. Update
    dependencies is a future goal

## Aug 15, 2023

**Host (pronouns):**

Jeremy Rickard

**Attendees (pronouns):**

- Grace Nguyen (she/her)

- Rey Lejano (he/him)

- Kartikey Rawat

- Rishit Dagli (he/him)

- Nishok

- Chris Hanson

- Jim Angel

- Burpt

- 

**Note Taker (pronouns):**

- Rey Lejano

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

    - \[Jeremy\] Had to delay patch releases, problem with Go updates,
      bumped release branches to latest version of 1.20 Go. Delayed
      patch releases will happen on Oct 23. Release is happening now,
      last part of the jobs are happening, then Google build admins will
      take care of package signing, we have issue open to lift code
      freeze and Grace will take care of the issue to lift code freeze.

  - Release Team

    - \[Grace\] Release management is working on creating the build, has
      eyes on docs to get that ready. Release blog is ready to merge.

    - \[Jeremy\] Issue is open for next release team, 1.29  
      [<u>https://github.com/kubernetes/sig-release/issues/2307</u>](https://github.com/kubernetes/sig-release/issues/2307)  
      Jim will be the branch manager for 1.29

    - A/I: Follow up and identify EA and shadow survey for 1.29

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- Signing and image promotion

  - \[Jeremy\] Will also discuss signing and image promotion next week.

<img src="media/image1.png" style="width:4.00521in;height:2.93528in" />

## Aug 8, 2023

Meeting got canceled.

## Aug 1, 2023

**Host (pronouns):**

Jeremy Rickard

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

    - Patch releases coming next week, cherry-pick deadline this Friday

    - Go 1.20.7 coming out this week with a security fix

    - A/I: Identify RM for cutting patch releases

  - Release Team

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- OpenBuildService pre-launch: pkgs.k8s.io is live

  - We plan to have official launch around 1.28.0 release

  - Marko is on vacation starting next week for two weeks, Marko to
    prepare everything for Release Managers to do before the final
    release

  - General overview of remaining important tasks

    - Finish documentation and blog post

    - Backfill repositories

  - Docs and blog post are looking for review:

    - Docs:
      [<u>https://github.com/kubernetes/website/pull/42022</u>](https://github.com/kubernetes/website/pull/42022)

    - Blog post:
      [<u>https://hackmd.io/@xmudrii/ryoT2PIo2</u>](https://hackmd.io/@xmudrii/ryoT2PIo2)

## Jul 25, 2023

**Host (pronouns):** Carlos Panato

**Attendees (pronouns):**

- Leonard Pahlke (he/him)

- Marko Mudrinić (he/him)

- Anhelina Zelyk (she/her)

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

  - Release Team

**Open Discussion:**

- \[Leo\] Merge all Slack release subteam channels into one channel
  beside ci-signal; essentially \#release-team, \#release-ci-signal

  - Leo will create an issue to track this proposal

<!-- -->

- \[Marko\] OBS status update and next steps

  - Walk the board:
    [<u>https://github.com/orgs/kubernetes/projects/137/views/1</u>](https://github.com/orgs/kubernetes/projects/137/views/1)

  - Handling cri-tools and kubernetes-cni packages:
    [<u>https://github.com/kubernetes/release/issues/3169</u>](https://github.com/kubernetes/release/issues/3169)

    - Initial versions for cri-tools and kubernetes-cni

  - Graduation criteria and phases for KEP-1731

  - Documentation and migration comms:
    [<u>https://github.com/kubernetes/release/issues/3056</u>](https://github.com/kubernetes/release/issues/3056)

  - 

- 

## Jul 18, 2023

Meeting got canceled.

## Jul 11, 2023

**Host (pronouns):** Verónica López (she/her)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Xander Grzywinski (he/him)

- Grace Nguyen (she/her)

- Joseph Sandoval (he/him)

- Mahamed Ali

- Sascha Grunert (he/him)

- Shivanshu Raj Shrivastava (he/him)

- 

**Note Taker (pronouns):**

- Joseph Sandoval

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Amit Dsouza - Based out of Australia. DevOps specialist. Recent
    speaker at Argocon EU.

- Subproject updates

  - Release Engineering

    - \[Marko\] July patch releases delayed to next Wednesday:
      [<u>https://groups.google.com/a/kubernetes.io/g/dev/c/A_hz52m4GUA/m/pWo7tJ0_BAAJ</u>](https://groups.google.com/a/kubernetes.io/g/dev/c/A_hz52m4GUA/m/pWo7tJ0_BAAJ)

      - \[Marko\] The delay allows for the Go security updates to be
        merged.

    - \[Marko\] Sascha and Carlos are working on updating the base image
      to Debian 12 (bookworm)

      - \[Marko\] The mailing list has some feedback on image size.

      - \[Sascha\] Kubecross will be updated to bookworm with 1.28.

    - \[Marko\] We had v1.28.0-alpha.4 with some minor hiccups, but
      overall went well

      - \[Marko\] Generating SBOM had a null pointer exception. Fixed
        by @puerco. Also ran into rate limiting issues. This might
        happen next week but be aware. Some release managers are having
        issues with Sendgrid.

    - \[Marko\] 1.26-1.24 release branches are updated to Go 1.20
      (thanks Carlos!)

      - \[Marko\] Final PR’s need to be merged. Publishing bot is broken
        and needs to be fixed.

  - Release Team:

    - One week from code freeze

      - \[Grace\] First retro will be held next week.

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- \[Marko\] OBS updates

  - \[Marko\] Outstanding item was How do we connect OBS with the
    release pipeline and publish packages? This is now complete.
    Additional tests are needed for 1.28. Next up is sorting out
    infrastructure. The OBS platform has limited bandwidth. We don’t
    know how much bandwidth we need. We don’t have any stats from
    Google. OBS will be a single repo but point to our own mirrors. S3
    bucket with CDN in front of it. CloudFront will be used. If we get
    additional bandwidth we will use Fastly. We can’t use Fastly atm. If
    we get bumped to 20 PB’s of bandwidth we will then use Fastly.
    [<u>https://github.com/kubernetes/k8s.io/pull/5536</u>](https://github.com/kubernetes/k8s.io/pull/5536)
    is under review and if ok’ed will merge this week. Docs and feature
    blog post are being drafted by @marko. CNCF will help to market this
    change when we are ready.

- \[Mahamed\] How do we determine which Prow jobs are blocking and
  master blocking?

  - \[Sascha\] Sig-Release has
    [<u>docs</u>](https://github.com/kubernetes/sig-release/blob/252777c7f9bec2297aaafeb12bf5984c4bc80705/release-blocking-jobs.md)
    on this question.

  - \[Verónica\] Not everything is set in stone. Update docs if you find
    something not up to date.

## Jun 27, 2023

**Host (pronouns):** Sascha Grunert (he/him)

**Attendees (pronouns):**

- Stephen Augustus (he/him)

- Mauren Berti (she/her)

- Adolfo García Veytia (he/him)

- Verónica López

- Ronit Banerjee (he/him)

- Joseph Sandoval (he/him)

- Shivanshu Raj Shrivastava (he/him)

**Note Taker (pronouns):**

- Adolfo/Stephen

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Ronit Banerjee

  - Mauren Berti

- Subproject updates

  - Release Engineering

    - Issues with container images signatures: working on a new
      tooling + prowjob to audit the broken signatures

    - Image references used in signatures now use registry.k8s.io

  - Release Team

    - \[Grace in absentia\] Enhancements Freeze stats:

      - 59 (!!) tracked out of 68 enhancements

      - Highest rate of tracked to enhancements I’ve ever seen 🎉

    - Starting 1.29, the bug triage team will be merged into the CI
      Signal team.

      - The
        [<u>votes</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1686765443006049)
        are in and the new team will be called **Release Signal Team**

      - Currently working on docs for the merge team

**  
Open Discussion:**

- KubeCon Maintainer track opened. Are you interested to speak about
  anything SIG Release related? Feel free to reach out to the SIG leads!
  :)

## Jun 13, 2023 

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Rey Lejano (he/him)

- Grace Nguyen (she/her)

- Matt Trachier (he/him)

**Note Taker (pronouns):**

- Rey Lejano (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

    - Go updates done -
      [<u>https://github.com/kubernetes/release/issues/3099</u>](https://github.com/kubernetes/release/issues/3099)

      - \[Jeremy\] Release branch and master branch are updated, last of
        cherry-picks merged last night. Weird issue were we had to
        update go sum files which required an api approver to approve
        every cherry-pick. Does it make sense to document in template?

        - \[Marko\] Yes it makes sense to update and provide an example
          with link to PR

    - Patch Releases

      - Who wants to manage them?

        - \[Jim and Jeremy\] Patch releases target date is tomorrow, Jim
          can help out tomorrow and can pair up with someone. Jeremy can
          kick it off in the morning and tag team with Jim

      - \[Marko\] OBS updates for alpha cut

        - Had some flaky tests but it was fixed, OBS stuff went well
          with 1.28 alpha 2 cut and worked. Still need to work on how to
          handle cri tools and Kubernetes cni

  - Release Team

    - 66 enhancements!

    - Enhancements Freeze is on Thursday

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- \[Grace\] Speaking at CNCF Toronto meetup about Release Team and
  SIG-Release

## Jun 6, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Rey Lejano (he/him)

- Grace Nguyen (she/her)

- Meha Bhalodiya (she/her)

- Dipesh Rawat (he/him)

- Amir Monfared

- Arnaud Meukam (he/him)

- Christopher Hanson (he/him)

**Note Taker (pronouns):**

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering

    - \[Marko\] Cosign V2

      - Carlos is mostly working on this, seems to be going well.

    - \[Marko\] OBS

      - merged core implementation last week, obs stage and obs
        krel-release

      - Can publish all core packages now

      - Will do a an alpha2 release to test things this week

    - Cherry Pick Deadline Friday - get them in

    - \[Jeremy\] Go Updates: PRs open in draft form and non draft form,
      will retest first PR that will kick off things like image
      promotion then will open last wave of things, then will hear about
      this soon and images will be available soon and be available
      before next alpha release

    - \[Marko\] still have some issues to figure out with OBS like with
      cri-tools and Kubernetes CNI

  - Release Team:

    - Enhancements Freeze next week on the 16th

    - Currently at 32 enhancements

      - Grace has reached out to all the SIG channels and will reach out
        on the chairs-and-techleads channels

    - Alpha 2 scheduled for June 8th to account for Go release

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- **\[Jeremy\]** can we back port distroless-iptables images to other
  release branches?

  - There is a CVE related to OpenSSL versions that is fixed in later
    releases (1.27)

  - \[Jeremy\] How do people feel about cherry-picking change to have a
    lower CVE footprint

  - [<u>https://kubernetes.slack.com/archives/CJH2GBF7Y/p1685561727920539</u>](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1685561727920539)

- \[Rey\] Are we switching k/k master to main

  - \[Jeremy\] AI to ping Carlos about this enhancement

  - \[Arnaud\] AI to ping Contribex since there was concern from a SIG
    about the switch

## May 30, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Sascha Grunert (he/him)

- Grace Nguyen (she/her)

- Carlos Panato (he/him)

- Rohan Sasne (he/him)

- Jim Angel (he/him)

- Jeremy Rickard (he/him)

- Verónica López (she/her)

**Note Taker (pronouns):**

- Joseph Sandoval

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Rohan Sasne - new to the community. Looking to get involved with the
    k\community.

- Subproject updates

- - Release Engineering

    - \[Carlos\] OBS update. Upgrade for signing. Using Co-Sign v2. Krel
      is updated to sign with v2. Working on Promo tools. Failure with
      E2E testing. Prow cluster is not happy.

    - \[Marko\] Tried migrating jobs to EKS. None of the jobs are
      passing on GKE.
      https://github.com/kubernetes/test-infra/issues/29622

  - Release Team

    - \[Grace\]
      [<u>Enhancements</u>](https://github.com/orgs/kubernetes/projects/140/views/1)
      number is low.

      - Only 20 enhancements at the moment. Outreach to individual SIG
        channels.

      - \[Jeremy\] All you can do is remind folks about enhancement
        deadlines.

    - \[Grace\] waiting on sig-windows to clear a [<u>failed
      test</u>](https://kubernetes.slack.com/archives/CV3BA6X8D/p1685330960844329)
      for alpha cut today

      - Arnaud will cut the release today.

    - \[Grace\] Bug-triage and CI-Signal will merge for 1.29. Looking
      for name suggestions!

      - 

    - \[Grace\] Looking for host key for new zoom link :D

      - \[jeremy\] will update today

    - \[Marko\] Release schedule. Can we adjust the release schedule for
      OBS? Marko is willing to drive the release. It will be the first
      release using OBS.
      [<u>https://kubernetes.slack.com/archives/CV3BA6X8D/p1685330960844329</u>](https://kubernetes.slack.com/archives/CV3BA6X8D/p1685330960844329)

    - \[Grace\] Do we have branch manager shadows?

      - \[Jeremy\] Mark Rosetti

**  
Open Discussion:**

- \[Marko\] OBS updates

  - KREL OBS specs are done. Still in progress KREL OBS stage and
    release.
    [<u>https://console.cloud.google.com/cloud-build/builds/4fc2cb3b-81ee-4336-8834-1e8414aa43c0?project=kubernetes-release-test</u>](https://console.cloud.google.com/cloud-build/builds/4fc2cb3b-81ee-4336-8834-1e8414aa43c0?project=kubernetes-release-test)
    for a view of what the logs will look like. (Only release managers
    can view). Testing and tooling are next. Marko has started working
    on tooling. Next week we can do another alpha. Feedback is needed on
    how testing should look like. The current testing is using a tool
    from Kubermatic. Is it ok to use a tool that is OpenSource but not
    part of SIG-release?

  - \[Jeremy\] Ask Sig-testing if that is ok?

  - \[Adolfo\] I don’t see this being an issue. We are using OBS. An
    external tool.

  - \[Carlos\] We can fork the tool if we think it's a concern.

- \[Marko\] Do we want to proceed with [<u>Remove "Kubernetes Source
  Code" from being published on GitHub
  Release</u>](https://github.com/kubernetes/release/pull/2780)?

  - \[Marko\] It’s been a few months. I would consider merging this PR.

  - \[Adolfo\] Is this the tarball on GH releases? What do we want to
    put on the GH release page? Started thinking on how to replicate our
    releases to the GH page. The community expects the releases to be
    there.

  - \[Marko\] If we can’t move artifacts. I don’t think this is a
    blocker for the PR. Dropping a tarball should be fine.

  - \[Adolfo\] I was asking about a year ago to see if anyone is
    dependent on the tarball. The PR has been open for a while.

  - \[Jeremy\] We can bring it back if we get comments when we remove
    it.

- \[Jeremy\] will be regenerating zoom links today

  - \[Jeremy\] Grace I’ll send you host codes. Please don’t share links
    with passwords.

- \[Jeremy\] please review
  [<u>https://github.com/kubernetes/community/pull/7287</u>](https://github.com/kubernetes/community/pull/7287)

  - \[Jeremy\] SIGS have responded and approved the reformation of
    WG-LTS. Tech leads please review.

- \[Rohan\] Archive meeting notes for Kubernetes SIG Release meeting
  [<u>https://github.com/kubernetes/sig-release/issues/2255</u>](https://github.com/kubernetes/sig-release/issues/2255)

  - \[Rohan\] Can I help and create a PR for this issue?

  - \[Jim\] I can pair up with you on this issue.

## May 23, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Ricky Sadowski (he/him)

- Rishit Dagli (he/ him)

- Jim Angel (he/him)

- Patryk Przekwas

- Michael Levan

**Note Taker (pronouns):**

- Jim Angel (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Patryk - k8s infra, interested in obs

  - Michael - docs release team shadow

- Subproject updates

  - Release Engineering

    - \[Marko\] Patch releases went out last week

      - Thanks Veronica, Jim, and Jeremy for taking care of those
        releases!

      - The only minor hiccup we had is sending announcement because
        SendGrid blocked Jim for some reason

        - Thanks Jeremy for taking care of announcements and sending an
          apologize letter

    - \[Marko\] OBS implementation going very well so far, presentation
      will be held later today

    - \[Marko\] Work in progress by Carlos to migrate to cosign v2

      - Changes in flight for krel / release tooling

  - Release Team

    - \[Grace in absentia\] last meeting I proposed potential release
      timeline changes, that will not be happening as the enhancements
      and implementation periods are similar in length already.

    - \[Grace in absentia\] Shadows all confirmed and onboarding.

    - \[Jeremy\] Schedule is posted, if questions, ping Grace / slack to
      have async convos.

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- \[Marko (15 min)\] [<u>OBS
  presentation</u>](https://docs.google.com/presentation/d/1nYvvEGvylbTIFWnR5ecjDalAJCUlHUwEjkx8HosvhLo/edit?usp=sharing)

  - Overview (moving Google -\> OBS/Suse WIP for implementation)

  - Significant changes / recap today’s state:

    - Package specs to build - complete

    - OBS projects / repos - complete

    - cmd to generate specs and archives for OBS - complete

  - OBS 1:1 mapped to repos (see recording for detail)

  - Limitation for building packages requires us to build packages in
    one project and publish in another (:\<MINOR\> vs :\<MINOR\>:build)

    - If we used the same repo, each version would increment / build a
      new source

  - CNI and cri-tools are not versioned like k8s, which requires
    publishing each time for a new project.

  - Mission statement: “We want to make managing and publishing packages
    possible for all kubernetes subprokects in hte same way as for core
    packages”

    - Universal platform / tooling that works with all projects.

  - Future state: krel obs wrapper (specs / stage / release)

  - Demo of templates for specs (for reuse - in recording)

  - OBS operational modes (similar to how we build today)

    - **Integrated**: via krel stage / release

    - **Standalone**: via cloud build job to build and publish backages

  - (watch the recording for more info…..)

## 16 May 2023

**Host (pronouns):** Sascha Grunert (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Leonard Pahlke (he/him)

- Grace Nguyen (she/her)

- Jim Angel (he/him)

- Matt Trachier (he/him)

- Shivam Singh (he/him)

- Ricky Sadowski (he/him)

- Scott Dodson (he/him)

- Amir Monfared(he/him)

- Stephen Augustus (he/him)

- Atharva Shinde (he/him)

**Note Taker (pronouns):**

- Marko Mudrinić (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Mofi from 1.28 Release Team

  - Anhelina from 1.28 Release Team

  - Scott

  - Shivam

- Subproject updates

  - Release Engineering

    - Kubernetes patch releases for May are scheduled for Wednesday

      - Veronica and Jim will take care of those releases

    - OBS implementation is back in progress (we’ll walk the board
      later)

      - Marko is working on this again

    - Release Engineering and Release Team project boards that we had in
      the past are now closed in favor of per-enhancement board (e.g.
      for packaging and SLSA)

  - Release Team

    - Kicked off this week - first meeting
      [<u>tomorrow</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1684245126254109)!

      - First meeting in less

    - Reducing Enhancements Freeze by a week to allow more time for Code
      Freeze

      - Enhancements period was 6 weeks, but will be 5 weeks

        - 5 weeks until code freeze after that

        - Calendar to be finalized soon

        - Community feedback is that it’s better to have more time for
          development

**  
Open Discussion:**

- \[Sascha\] Possible image signing enhancements:

  - Supporting of recursive signing of multi-arch images in kpromo:  
    [<u>https://github.com/kubernetes-sigs/release-sdk/pull/193</u>](https://github.com/kubernetes-sigs/release-sdk/pull/193)  
    [<u>https://github.com/kubernetes-sigs/release-sdk/releases/tag/v0.10.1</u>](https://github.com/kubernetes-sigs/release-sdk/releases/tag/v0.10.1)  
    [<u>https://github.com/kubernetes-sigs/promo-tools/pull/868</u>](https://github.com/kubernetes-sigs/promo-tools/pull/868)

  - RFC: Being able to set the docker-reference to registry.k8s.io in
    image signatures:  
    [<u>https://github.com/sigstore/cosign/pull/2984</u>](https://github.com/sigstore/cosign/pull/2984)  
    Right now we still point them to the actual mirror, which makes
    signature validation hard:  
    \> cosign verify \\  
    --certificate-identity
    krel-trust@k8s-releng-prod.iam.gserviceaccount.com \\  
    --certificate-oidc-issuer https://accounts.google.com \\  
    registry.k8s.io/kube-apiserver:v1.27.1 \| jq
    .\[0\].critical.identity  
    …  
    {"docker-reference":"europe-southwest1-docker.pkg.dev/k8s-artifacts-prod/images/kubernetes/kube-apiserver"}

  <!-- -->

  - \[Sascha\] We don’t use recursive signing, so in case of multi-arch
    images, we sign the manifest, but don’t sign concrete images. Added
    support for recursive signing to fix this, will be introduced in
    promo-tools soon. Hopefully it’s not going to increase the promotion
    time, but we’ll properly sign and validate images

  - \[Sascha\] We want to be able to replace docker-reference with
    registry.k8s.io, but this requires changing cosign, but it’ll be
    better for users. We’ll see how it turns out, depends on if the
    change lands in cosign

  - \[Sascha\] Those are considered small changes, we don’t need to
    change our current enhancement

  - \[Jim\] What do we need to move with this?

    - \[Sascha\] Proposed change as a RFC, we need to wait for feedback.
      Without it, we don’t know how to make it better for users. If you
      make it really strict, it’s wrong, because the reference is not
      matching the image that's being verified

<!-- -->

- \[Marko\] Walking the [<u>Packaging
  board</u>](https://github.com/orgs/kubernetes/projects/137/views/1)

  - Looking for feedback on board look and organization, and issues

- \[Marko\] Status of
  [<u>https://github.com/kubernetes-sigs/promo-tools/pull/792</u>](https://github.com/kubernetes-sigs/promo-tools/pull/792)

  - What do we need to merge this?

  - What’s the current overall status of promo-tools?

  - \[Adolfo\] We’re in a pretty good state, instability times are gone,
    but we might look into improving resiliency. We should definitely
    discuss retries, but we don’t have any issue for that. There was
    some discussion with SIG K8s Infra, but that requires rewriting
    promotion from scratch. We’re still considering where to take it
    from here, but feeling is that it’s relatively stable

- \[Marko\] Should we consider archiving the SIG Release meeting
  minutes?

  - There are some severe performance issues with this document
    especially when on Zoom

  - SIG Contribex is doing something similar every year:
    [<u>https://github.com/kubernetes/community/tree/master/sig-contributor-experience/meeting-notes-archive</u>](https://github.com/kubernetes/community/tree/master/sig-contributor-experience/meeting-notes-archive)

  - \[Stephen and Jim\] There are docs in the README in SIG Contribex
    directory, we can follow that link

    - General consensus is yes

    - Jim to take a look at this

    - [<u>https://github.com/kubernetes/community/tree/master/sig-docs/meeting-notes-archive</u>](https://github.com/kubernetes/community/tree/master/sig-docs/meeting-notes-archive)

- \[Marko\] Subscription to debian-security mailing list

  - We’re currently subscribed to debian-security mailing list via our
    release-managers mailing list

  - We’re receiving (in my opinion) a lot of irrelevant emails from them

  - Should we consider unsubscribing from that mailing list or finding
    another way to collect those emails/alerts?

  - \[Stephen\] We’re concerned about Debian image updates. Do we have a
    process to regularly update Debian images? What should be a signal
    to update our images?

    - If it’s not providing value, it’s fine, but we should have a
      process to build new images

    - Come up with a process to update images every month after patch
      releases

    - \[Sascha\] Does it have to align with Go updates?

      - \[Stephen\] No, there might be out of bound updates for Go, and
        the cadence in general might be different. As a baseline, doing
        it once a month is fine. The idea behind after patch releases is
        to reduce risks in patch releases, e.g. we would have a month to
        catch potential regressions

      - AI(Marko): Create issue for this and find someone to work on
        this

        - Jim and Marko to pair on this, Jim to file an issue

    - \[Matt Trachier\] We have a lot of concerns about CVEs that are
      theoretically possible, but not practically. Compliance people are
      often concerned if CVEs exist and not if they are exploitable or
      not, so updating often can be helpful

- \[grace\] can a SIG chair approve this typo PR ASAP pls:
  [<u>https://github.com/kubernetes/k8s.io/pull/5285</u>](https://github.com/kubernetes/k8s.io/pull/5285)

## May 9, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Nitish

- Marko Mudrinić (he/him)

- Leonard Pahlke (he/him)

- Siming Weng (he/him)

- Haitao Chen

- Dipesh Rawat (he/him)

- Jim Angel (he/him)

- Christopher Hanson (he/him)

- Josh Berkus (he)

- Heba Elayoty (she/her)

- Amir Monfared (he/him)

- Arnaud Meukam (he/him)

**Note Taker (pronouns):**

- 

- Jim Angel (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - A few folks attending that are new

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - \[Marko\] Go 1.20.4 / 1.19.9 updates are done

    - \[Marko\] Patch releases are scheduled for the next week

      - Cherry-pick deadline: 2023-05-12 (this Friday)

      - Releases: 2023-05-17

        - TODO: Need release managers/leads to be determined for cut

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - (Leo) Shadow form closed last week to be finalized with today
      (5/9) being the deadline to send out the emails.

      - Grace is working on docs

      - To follow up, there’s an issue with checkboxes required for a
        release cycle:
        [<u>https://github.com/kubernetes/sig-release/issues/2223</u>](https://github.com/kubernetes/sig-release/issues/2223)

      - We are on track to begin next week for 1.28
        [<span class="mark">🎉</span>](https://emojipedia.org/party-popper/)

    - Arnaud will be the branch manager for 1.28 (thank you Arnaud)
      [<span class="mark">🎉</span>](https://emojipedia.org/party-popper/)

    - Any update on the shadow selection process. It was mentioned
      notifications would be send out latest until: 9th May 2023 (Today)

      - \[Leo\] we will confirm the team by the end of the day

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- \[Marko\] Migration of SIG Release and Release Blocking jobs to EKS
  Prow build cluster

  - Thanks to Carlos for taking care of many of our projects!

  - Some of release blocking jobs will get migrated as part of
    [<u>https://github.com/kubernetes/test-infra/pull/29410</u>](https://github.com/kubernetes/test-infra/pull/29410)

    - Merged May/9/2023

  - Please let SIG K8s infra folks know if you notice anything strange

- \[jeremy\] annual report
  [<u>https://github.com/kubernetes/community/pull/7214</u>](https://github.com/kubernetes/community/pull/7214)

  - Ask: review ASAP and need to merge ASAP

- \[jeremy\] Wg-lts
  [<u>https://github.com/kubernetes/community/pull/7287</u>](https://github.com/kubernetes/community/pull/7287)

  - Ask: please review/comment

  - Jim: Have we identified what has changed since old WG-LTS (no
    opinion, just asking):

    - [<u>https://github.com/kubernetes/community/issues/7259</u>](https://github.com/kubernetes/community/issues/7259)

      - Contains discussion around changes / differences

      - General consensus was: There are changes worth exploring but
        with other leaders driving the WG.

- \[Nitish\] Discussions about SIG-Spotlight blog on Sig-Release

<!-- -->

- \[Nitish\] We approach chairs of sigs to ask questions about
  contributing / getting involved and noticed SIG-Release has not had a
  spotlight.

  - Open questions?

    - Who can we reach out to for working with (vs. the ephemeral
      release team)?

    - Should the blog be divided into two parts? And what is the process
      to follow before publishing a blog.

  - \[Jeremy\] Depends on what you’re trying to cover in the blog

    - [<u>https://github.com/kubernetes/community/tree/master/sig-release</u>](https://github.com/kubernetes/community/tree/master/sig-release)

    - Two subprojects

      - RelEng

        - Cutting branches / tooling

      - Release Team

        - Rotates every cycle and those are folks that are staffed and
          running a release

    - It makes sense to cover both (wide open paths)

  - \[Leo\] Over the past few conferences, we talk about releases /
    teams / engineering and putting into a blog would be good.

    - Would this document require updates or be a blog that doesn’t
      require updates?

    - No deadline for updating, but have updated in the past. Then
      events driven for changes.

  - \[Jeremy\] Start with tech leads and maybe someone from the current
    release team to work with.

- Slack discussion:
  [<u>https://kubernetes.slack.com/archives/C2C40FMNF/p1683306707135349</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1683306707135349)

<!-- -->

- There has been no Spotlight blog on Sig-Release yet.

<!-- -->

- Example:
  [<u>https://kubernetes.io/blog/2021/07/15/sig-usability-spotlight-2021/</u>](https://kubernetes.io/blog/2021/07/15/sig-usability-spotlight-2021/)

<!-- -->

- \[jeremy\] new project boards, should we review them?

  - \[marko\] we should first schedule a meeting to create items

  - \[jeremy\] should we schedule those now?

  - \[marko\] carlos/adolfo are away till next week, marko will start
    creating issues for packaging

  - \[veronica\] xander was interested in PMing

- \[heba\] good issues for people starting?

  - \[jeremy\] we have good first issues in some repos. Most things will
    be in tooling repos.

  - \[jim\] repos documented here:
    [<u>https://github.com/kubernetes/community/tree/master/sig-release#release-engineering</u>](https://github.com/kubernetes/community/tree/master/sig-release#release-engineering)

  - 

**(Optional) Walk the Board:**

- Project board review:
  [<u>https://github.com/orgs/kubernetes/projects/23</u>](https://github.com/orgs/kubernetes/projects/23)

- Incoming issue and PR triage

  - [<u>SIG Release issues
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>SIG Release PRs
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>Kubernetes
    issues</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)

## May 2, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Sascha Grunert (he/him)

- Leonard Pahlke (he/him)

- Joseph Sandoval (he/him)

- Rishit Dagli (he/him)

- Xander Grzywinski (he/him)

- Marko Mudrinić (he/him)

- Matt Trachier (he/him)

- Mickey Boxell (he/him)

- Mahamed Ali

- Amir Monfared

- Christopher Hanson (he/him)

- Rayan Das (he/him)

- Carlos Panato

- Ashwin Kumar Uppala (He/Him)

**Note Taker (pronouns):**

- Joseph

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Mickey Boxell

  - Chris - RXM - trying to get more involved in the Kubernetes project.

  - Amir - Second time joining the meeting.

  - Tobo - first time attending. Looking to get more involved in the
    Kubernetes community.

  - Rayan - new to the meeting. Infracloud. Trying to get involved with
    SIG-Release.

  - Ashwin - Coming back to learn more getting involved with
    SIG-Release.

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - Go 1.20.4 / 1.19.4 update:
      [<u>https://github.com/kubernetes/release/issues/3025</u>](https://github.com/kubernetes/release/issues/3025)

      - (Marko) Releases going out today.

    - K8s.gcr.io freeze is live 🎉

      - (Mahamed) PR was merged last week after Kubecon. No images being
        published to the old registry.

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - (Leo) 5/2 is the last day to apply to the release team. Shadow app
      closes in 6 hours. A week to select the shadows and then start the
      release cycle. (application form:
      https://bit.ly/k8s-shadow-application-1-28)

      - Release Team Shadow Application Opens: 13th April 2023

      - Release Team Shadow Application Closes: Tuesday 2nd May 2023
        (23:59 UTC)

      - Release Team Shadow Notifications send out latest until: 9th May
        2023

      - Release Cycle Start: 15th May 2023

      - Release Cycle End: 15th August 2023 (approximate!)

    - (Jeremy) Branch manager needs to be selected.

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- \[Marko\] Update on generating release branch jobs

  - PR with more details:
    [<u>https://github.com/kubernetes/test-infra/pull/29387</u>](https://github.com/kubernetes/test-infra/pull/29387)

  - Issue to track fixing annotations:
    [<u>https://github.com/kubernetes/test-infra/issues/29388</u>](https://github.com/kubernetes/test-infra/issues/29388)

  - Issue to track updating handbook:
    [<u>https://github.com/kubernetes/sig-release/issues/2227</u>](https://github.com/kubernetes/sig-release/issues/2227)

  - Proposal to rewrite Python scripts in Go:
    [<u>https://github.com/kubernetes/test-infra/issues/29390</u>](https://github.com/kubernetes/test-infra/issues/29390)

    - (Marko) There were some efforts to get rid of the 1.23 jobs.
      Issues created that need feedback which are listed above. The 1.28
      release team will need to update the handbook. We should rewrite
      the python scripts. They are wrappers around Go tools. We rewrite
      in Go in a single library. Take a look at the issues and provide
      feedback.

    - (Jeremy) Any reservations about this effort? Most in agreement.

    - (Matthias) In Krel or another binary?

    - (Marko) Still being determined.

> Slack thread:
> [<u>https://kubernetes.slack.com/archives/C2C40FMNF/p1682754798900429</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1682754798900429)

- \[Mickey\] Update to release handbooks

  - Breaking out the release team role handbooks
    ([<u>example</u>](https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/docs))
    into two docs: a lead handbook and a shadow handbook or retitling
    the current handbooks

  - If I were a current or prospective shadow, I wouldn’t think to look
    for my responsibilities in a doc entitled “Kubernetes \[Insert
    Team\] Lead Handbook”.

    - (Jeremy) It seems like a reasonable change. Retitling makes the
      most sense.

    - (Leo) I like the idea of the change. The title might make it
      confusing. Making it clear to new folks will be helpful. Have
      everything in one document.

    - (Verónica) Important to show our shadows all aspects of the roles.
      Yes to changing the title.

    - (Matt) It is difficult to get more involved without knowing what
      I”m getting into.

    - (Mickey) What about a separate section for a shadow?

    - (Matt) This would be helpful

    - (Verónica) After a release cycle having shadows provide their
      perspective on time commitments on the roles.

    - (Jim) Newcomers have invaluable perspectives. There might be an
      area to improve information discovery. Creating a K8s.dev to help
      make the information more discoverable.

**(Optional) Walk the Board:**

- Project board review:
  [<u>https://github.com/orgs/kubernetes/projects/23</u>](https://github.com/orgs/kubernetes/projects/23)

- Incoming issue and PR triage

  - [<u>SIG Release issues
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>SIG Release PRs
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>Kubernetes
    issues</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)

## 18 Apr 2023 - KubeCon Contributor Summit Special

**Where:** E108 - 1st Floor - Congress Complex RAI Amsterdam
([<u>https://sched.co/1LXGs</u>](https://sched.co/1LXGs))

**When:** 3:40 pm - 4:35 pm

**Hosts (pronouns)**

- Sascha Grunert (he/him)

- 

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Jeremy Rickard

- Angelos Kolaitis

- Brad McCoy (he/him)

- Adolfo García Veytia

- Carlos Panato

- Jeremy Rickard (he/him)

**Note Taker (pronouns):**

- Joseph Sandoval

**  
Agenda:**

- SIG Release Introduction meeting from Veronica and Marko tomorrow
  (Wednesday) at 2:30 pm

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

- \[Marko\] OBS - Need to enable OBS in the release pipeline. If we can
  get some help with the packages. Get some additional help from outside
  resources in 1.28. No blockers.

  - \[Adolfo\] - What type of help?

  - \[Marko\] - Help is needed with implementing the API. We need to do
    some planning and prioritization and create the issues.

  - \[Sascha\] - Do we have an updated KEP?

    - KEP:
      [<u>https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1731-publishing-packages</u>](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/1731-publishing-packages)

- LTS Working Group will probably restart:
  [<u>https://docs.google.com/document/d/1C7utWMjGpDcBc5TB_ooddERsid59sFLC7gqLzyS5aKY/edit?usp=share_link</u>](https://docs.google.com/document/d/1C7utWMjGpDcBc5TB_ooddERsid59sFLC7gqLzyS5aKY/edit?usp=share_link)

  - \[Jeremy\] There was a previous LTS effort. This effort will be
    re-kicked off. An unconference session regarding this was done
    during the Contributor Summit. The takeaway will come up with a
    north star to what they are working for. 2 year LTS and skip level
    upgrades. It will be a WG as its cross cutting across SIGs. This
    will be a long process. If interested you can add your name to the
    link above. Previous LTS information is contained in the link above.

  - What does LTS mean?

  - \[Jeremy\] It can mean various things. Longer support. Version being
    in support for a longer time.

  - \[CiCi\] Will that involve extension of support?

  - \[Jeremy\] Is it just security fixes? Bugs? The community will work
    on defining this. Does a version get LTS and extend the patching
    cycle? Nothing is decided at the moment. There was no opposition to
    the concept from the working session. Consensus on restarting the
    effort.

  - \[Sascha\] How did we come back to this working group?

  - \[Jeremy\] Microsoft announcement on LTS started the discussion
    again. Several providers were interested as well.

  - \[Adolfo\] There are several efforts from different vendors on LTS.
    It's going to involve a lot of work.

  - \[Jeremy\] It's important to think about the problems to be solved.
    Jordan Liggitt raised some problems that couldn’t be overcomed in
    the previous LTS efforts. We support a forked version of Kubernetes.
    Most cloud providers run forks.

  - \[Adolfo\] I was referring to release eng tooling that will be
    affected.

  - 

- SLSA

  - \[Adolfo\] PR is ready for the Prow work. How can we ensure that all
    projects can benefit from this? Find a way to enable this for the
    different teams. When we promote images all projects received signed
    images for free. No easy way to connect the Prow jobs with the
    artifacts. Also help people with attestations. It needs some work
    with the Image Promoter. Carlos and Adolfo will speak about this on
    Thursday at Kubecon.

  - \[Marko\] There is a new version of SLSA. Will this affect the work
    we have done?

  - \[Adolfo\] We have to update a couple of tools. Need to update the
    SBOM tool as well. New version of SLSA is open for comments at the
    moment. No urgent need to update.

  - \[Sascha\] The changes with the image promoter?

  - \[Adolfo\] One year ago we were promoting to three mirrors now we
    are promoting to over 20+. We were running into rate limits. We did
    slice the time for the image promotion. We leave the promoter as is.

  - \[Veronica\] Have you talked to SIG-K8s-infra regarding these
    changes? This was discussed last week. Whoever cuts the release has
    a difficult time.

  - \[Adolfo\] This will need to go under a KEP since this changes how
    we promote images. We are cross collaborating with SIG-K8s-Infra.

  - \[Sascha\] We should update the KEP for 1.28.

  - \[Adolfo\] We would like to do it during this cycle.

- \[Marko\] Should we consider planning sessions?

  - We would have sessions with Lauri. Our project board is a bit of a
    disaster. We discussed OBS. SLSA is in discussion. Perhaps every
    month try to keep our board up to date. We could do a dedicated
    session on a topic. Make sure the issues make sense.

  - \[Xander\] I can help with this. If there is desire I can help with
    this.

  - \[Veronica\] The sessions we used to have were very useful. Those
    sessions were improvements to the release process and tooling. Now
    our tooling has evolved alot. We don’t have enough hands to keep
    this up to date. My proposal is if we have sessions then we need to
    have ownership. Alot of people were interested in the pillars but
    had to drop off for various reasons. For example SLSA we would have
    someone assigned to that pillar. Like Adolfo. They would keep the
    roadmap for that pillar up to date.

  - \[Sascha\]

  - \[Marko\] The person doesn’t have to be a planning expert. The
    person should have context but don’t need to be technical.

  - \[Adolfo\] If we feel someone is a sole owner of a pillar or effort
    like SLSA. We need to make sure we expand that to others.

  - \[Marko\] At times there is no way to find how to work on an item.

  - \[Sascha\] Can we create project boards? What about removing or
    getting rid of existing project boards?

  - \[Marko\] Action item: I can help run some dedicated sessions. We’ll
    start after Kubecon.

  - \[Jeremy\] Should we update the roadmap? It would help with the
    annual report.

  - \[Xander\] GH boards make life easier.

  - \[James\] GH boards were celebrated in the 1.27 retro.

  - Project boards

    - Packaging:
      [<u>https://github.com/orgs/kubernetes/projects/137</u>](https://github.com/orgs/kubernetes/projects/137)

    - SLSA:
      [<u>https://github.com/orgs/kubernetes/projects/138/views/1</u>](https://github.com/orgs/kubernetes/projects/138/views/1)

- \[Marko\] Version markers issue status

  - We stop cutting RC.0 releases.

  - After v1.26.1 it would look like v1.26.2-rc.0.9-hash

    - v1.26.1-9-hash

  - This change broke the tooling. Tests are invalid. We might have the
    same issue with cutting 1.28.1. We should take a look at this. Not
    having RC.0 was a huge improvement. But we are still having issues.

  - \[Adolfo\] Tag a repository but don’t build it.

  - \[Cici\] It results in the risk that having tooling for package
    fetching by tags

  - \[Marko\] We should try once again with the logic.

  - \[Cici\] Are we only cutting RC.0?

  - \[Marko\] For minor releases we should cut with RC.1.

  - \[Sascha} There is an open issue on K/K.

## 

## Apr 11, 2023

**Host (pronouns):** Verónica López (she/her)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Xander Grzywinski (he/him)

- James Laverack (he/him)

- Jim Angel (he/him)

- Ashwin Kumar Uppala (he/him)

- Cici Huang (she/her)

- Matt Trachier (he/him)

- Joseph Sandoval (he/him)

**Note Taker (pronouns):**

- Marko Mudrinić (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - April patch releases

      - Testgrid is having a party

        - 1.25 and 1.24 blocking dashboards are red

        - 1.25:
          [<u>https://testgrid.k8s.io/sig-release-1.25-blocking</u>](https://testgrid.k8s.io/sig-release-1.25-blocking)

        - 1.24:
          [<u>https://testgrid.k8s.io/sig-release-1.24-blocking</u>](https://testgrid.k8s.io/sig-release-1.24-blocking)

        - \[Veronica\] We should have a better way to track this so that
          we don’t figure this out a day before releases

        - We might have to delay patch releases because we shouldn’t
          release with jobs being red on blocking

        - \[Jim\] Ben talked about adding arm64 support a few weeks ago,
          maybe it’s related to 1.24 failures

          - [<u>https://kubernetes.slack.com/archives/CN0K3TE2C/p1679504185336529?thread_ts=1679463031.954529&cid=CN0K3TE2C</u>](https://kubernetes.slack.com/archives/CN0K3TE2C/p1679504185336529?thread_ts=1679463031.954529&cid=CN0K3TE2C)

        - \[Cici\] We had issue with rc.0 tag, will be shared, but rc.0
          was much before failures started

    - Go updates are done

    - \[Puerco\] New version of promoter

      - We had a successful rc.1 release!

      - Thanks to everyone who helped!

      - We’ll see today how it performs for 1.27.0

      - \[Veronica\] What are the first/next steps?

        - \[Puerco\] Fixing signatures. We’ll see about this after
          KubeCon. Next steps will involve sitting together and seeing
          how we can do it more efficiently

      - \[Veronica\] Is there any work that’s pending from Arnaud and
        Mahamed?

        - Some work might be needed related to freeze after KubeCon

        - Some conversation is needed how we’ll proceed with the freeze

        - Update from Adolfo doesn’t affect this

        - \[Puerco\] For real real freeze we need to fix the signatures

          - \[Arnaud\] The redirect is done, so this is not necessarily
            needed

        - Related PR:
          [<u>https://github.com/kubernetes-sigs/promo-tools/pull/792</u>](https://github.com/kubernetes-sigs/promo-tools/pull/792)

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - 1.27.0 release is today!

    - \[Cici\] The release is going great!

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- SIG Release session at the Kubernetes Contributor Summit:
  [<u>https://sched.co/1LXGs</u>](https://sched.co/1LXGs)

- 1.27 Retro session at the Kubernetes Contributor Summit
  \[[<u>link</u>](https://kcseu2023.sched.com/event/1LXJH/127-release-team-retrospective)\]

- All sig-release talks/meetings for KubeCon:

  - [<u>https://kubernetes.slack.com/archives/C2C40FMNF/p1681227034588669</u>](https://kubernetes.slack.com/archives/C2C40FMNF/p1681227034588669)

**(Optional) Walk the Board:**

- Project board review:
  [<u>https://github.com/orgs/kubernetes/projects/23</u>](https://github.com/orgs/kubernetes/projects/23)

- Incoming issue and PR triage

  - [<u>SIG Release issues
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>SIG Release PRs
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>Kubernetes
    issues</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)

## Apr 4, 2023

**Host (pronouns):** Verónica López

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Xander Grzywinski (he/him)

- Amir Monfared(he/him)

- Jim Angel (he/him)

- Sascha Grunert (he/him)

- Adolfo García Veytia (he/him)

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Welcome Amir!

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - Go updates

      - 

    - Patch releases next week

      - Veronica and Marko will take care of cherry-pick reviews

      - Jim: can I help with cherry-pick reviews?

        - Veronica: we had idea to have a spreadsheet to collect
          cherry-picks and provide some context there

        - Marko: Release Manager Associates cannot approve cherry-picks,
          but we can still go through them together. We have an issue to
          track allowing Associates to approve cherry-picks:
          [<u>https://github.com/kubernetes/sig-release/issues/2083</u>](https://github.com/kubernetes/sig-release/issues/2083)

    - RC.1 on Thursday and final on Tuesday

      - There’s one PR in promoter that added limiter to one more place,
        but sigstore doesn’t have limiter so we still expect issues

      - We’ll probably end up in the same situation

        - We’ll start testing the job to retroactively sign images, but
          we cannot do that because of the registry rollout

      - We could try new promo-tools release to include the latest
        changes and hope we get luckily

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - Rc1 cut moved to Thursday

    - Releases next week

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

- Shadow application survey

  - We’re in process of nominating leads

  - The applications will open a week or two after KubeCon

  - Jim: recommend to attend SIG Release and Release Team meetings to
    stay in touch and up to date with all the news, deadlines…

  - Veronica: also make sure to follow our Slack channels, and feel free
    to ping Veronica if you have questions

- Produce structured test output for repo's tests

  - k/release:
    [<u>https://github.com/kubernetes/release/issues/2993</u>](https://github.com/kubernetes/release/issues/2993)

  - k-sigs/promo-tools:
    [<u>https://github.com/kubernetes-sigs/promo-tools/issues/804</u>](https://github.com/kubernetes-sigs/promo-tools/issues/804)

**(Optional) Walk the Board:**

- Project board review:
  [<u>https://github.com/orgs/kubernetes/projects/23</u>](https://github.com/orgs/kubernetes/projects/23)

- Incoming issue and PR triage

  - [<u>SIG Release issues
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>SIG Release PRs
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>Kubernetes
    issues</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)

## Mar 28, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- [<u>Adolfo García Veytia</u>](mailto:puercozon@gmail.com)(he/him)

- [<u>Grace Nguyen</u>](mailto:nng.grace@gmail.com) (she/her)

- Laura Lorenz (she/her)

- Benjamin Elder (he/him)

- [<u>Ashwin Kumar Uppala</u>](mailto:kumarashwin2603@gmail.com)(He/Him)

**Note Taker (pronouns):**

- Joseph Sandoval

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Ashwin is checking out how things are working. Joining to follow
    along. Observe the meeting with the time he has.

  - Josh Berkus / Sig-Contribex co-chair.

  - Adolfo/Puerco - technical lead

  - Jeremy - does release team stuff

  - Marco - Release Mgr

  - Cici - 1.27 Branch Mgr

  - Mahamed - Contributor to Sig-release and Sig-K8s infra

  - Joseph - Release Mgr Assoc

  - Grace - Works with the release team. Branch Mgr assoc

  - Jim Angel - Release Mgr Assoc

  - Laura Lorenz - 1.27 release team CI Signal lead. Updates on
    CI-Signal stuff

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - \[Marko\] Things are going mostly fine. Cherry pick deadline is
      April 7. April 12 is the patch cut. Some issues with signatures.
      Rel-Manager channel on K8s slack has the details. Adolfo and Ben
      have been helping on it. Migration from registry changes are the
      reason why we have not fixed the missing signatures.

    - \[Adolfo\] Missing signatures is due to the promoter and rate
      limiting.Create some jobs to do dry runs to make sure we have
      proper credentials. Issue to track is
      [<u>here</u>](https://github.com/kubernetes/release/issues/2962).

    - \[Marko\] We will wait for the Google team to give us the green
      light.

    - \[mahamed\] PR to complete the image freeze.
      [<u>https://github.com/kubernetes/k8s.io/pull/5035</u>](https://github.com/kubernetes/k8s.io/pull/5035)

      - \[Mahemed\] There is a snag related to inconsistencies with
        signatures. Ben suggested we delay freezing. The freeze has less
        value. PR is ready. The work is complete.

    - \[puerco\] Update on Bad Signatures -
      [<u>https://github.com/kubernetes/release/issues/2962</u>](https://github.com/kubernetes/release/issues/2962)

    - \[cici37\] Did RC0 cut last week. Quick update on post branch
      creation tasks

      - Mos of the work has
        completed([<span class="mark">kubernetes/kubernetes#116919</span>](https://github.com/kubernetes/kubernetes/pull/116919),
        [<span class="mark"><u>kubernetes/test-infra#29124</u></span>](https://github.com/kubernetes/test-infra/pull/29124),
        [<span class="mark">kubernetes/test-infra#29125</span>](https://github.com/kubernetes/test-infra/pull/29125))

      - Pending for review:
        [<u>https://github.com/kubernetes/test-infra/pull/29137</u>](https://github.com/kubernetes/test-infra/pull/29137)

      - Still have one PR under 1.27 milestone for fix:
        [<u>https://github.com/kubernetes/kubernetes/pull/116575</u>](https://github.com/kubernetes/kubernetes/pull/116575)

      - \[Cici\] Ran into the rate limiting issue again. PR to fix the
        tagging issue. We should have separate labels to merge the PR.
        Current milestone and release blocker. PR for review on tagging
        branch jobs. Expect 1.28 job to prepare the release branch.
        Create an issue to investigate further. Branch creation for
        1.27. Two of the previous releases 1.22/1.23 are EOL. The
        releases are past the EOL. Should we remove the jobs for those
        releases?

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

- \[lauralorenz/jberkus\] follow up on CI Signal report feedback process

  - [<u>Discussed 3/7</u>](#t5hvq1vi90e9), follow up was [<u>this
    issue</u>](https://github.com/kubernetes/community/issues/7176) for
    help from comms to discuss [<u>this survey
    draft</u>](https://hackmd.io/@on2gXqykTESI4N6quhc9pw/rkKylZuy3) for
    feedback on

    - \[Laura\] Weekly email report from CI-Signal. Template with some
      duplication and extra information. It can vary over releases. I
      wanted to find a way to streamline this communication. Above is
      the issue and survey draft.

  - At the time we didn’t have much background on this but wanted to
    follow up especially after connecting with Josh

    - \[Josh\] I created the original signal report for 1.11. Why do we
      need to do a survey? If the team feels like effort to results
      ratio is not worthwhile, is to just not do it? It was created at a
      different time. A Lot has changed which has lowered its value.
      Testgrid was not as good at that time. 1.13 CI-Signal reports were
      spent in the red. It was a call to action to create this report.
      Testgrid is in much better shape now. Faster response to failures.
      Current template doesn’t have that call to action. It was very
      prescriptive on actions needed by individuals. We probably don’t
      need to do a survey.

    - \[Laura\] How does SIG-Release feel about the email report? If we
      don’t have the same surgical details about fixing tests in the
      red. We chase down issues very quickly now. We could stop doing it
      and go dark. Looking for feedback on how we go forward with this.

    - \[Adolfo\] Let people know in advance if we decide to not do it.
      If someone expresses that they need it when we announce we can
      keep it.

    - \[Laura\] Everyone seems to feel like we don’t use it or get the
      information from Slack. Some on the survey like the email report.
      The sample size of the survey is small.

    - \[Adolfo\] Branch managers have enough information and don’t need
      the report. We could automate the information. Maybe remove it for
      one cycle?

    - \[Jeremy\] Drop from 1.28 and see if we have any issues?

    - \[Laura\] I’ll bring this information back to the release team
      retro.

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- \[jeremy\] - Please review annual report
  [<u>https://github.com/kubernetes/community/pull/7214</u>](https://github.com/kubernetes/community/pull/7214)

  - \[Jeremy\] The report has information on what we did and stats
    regarding PR’s.

- \[jeremy\] - What if we disabled prow’s merge capability for k/k when
  cutting releases (rc.0 maybe when branch cut occurs)?

  - [<u>https://kubernetes.slack.com/archives/CJH2GBF7Y/p1679613751918579?thread_ts=1679588340.415139&cid=CJH2GBF7Y</u>](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1679613751918579?thread_ts=1679588340.415139&cid=CJH2GBF7Y)

  - [<u>https://github.com/kubernetes/kubernetes/pull/116866</u>](https://github.com/kubernetes/kubernetes/pull/116866)

    - \[Jeremy\] The branch FFwdr kicked in and created an issue. We
      have seen this in the past. What if we disabled this or added
      protection to enforce policy automagically?

    - \[Marko\] This should be fairly easy with a Prow plugin. Patch
      releases are also affected by this issue. Patch releases take a
      long time.

    - \[Mahamed\] Why do we squash and merge?

    - \[Marko\] Merge keeps the history.

    - \[Mahamed\] Squash and merge is very convenient.

    - \[Adolfo\] The clean history of commits.

    - \[Ben\] If you mention a username they get pings. The blocking
      merge commits come from this. If there is any bad behavior. We can
      address. Robots are trying to prevent you from doing bad things.
      Dependabot includes reference to NPM packages. We may be blocking
      more than we need to. For example K8s website might be different.
      Have users opt into their PR’s.

    - \[Jeremy\] We had a PR merge during RC.0 cut. What if we disabled
      merging during branch cuts?

    - \[Ben\] We only allow certain people to do this?

    - \[Jeremy\] Milestone maintainers are not a small group..

    - \[Adolfo\] Some may not know and some who do that do this.

    - \[Ben\] Milestone maintainers need to ack. We may then have merges
      at different phases.

    - \[Marko\] I don’t think we need milestone maintainers. Milestone
      applier will do this for us. We have milestones in the PR’s. We
      should evaluate if we need this thing?

    - \[Ben\] Intent is for the SIG to note if this is a significant bug
      fix. We should trust our leaders but communicate what is expected.
      Maybe this didn’t happen with RC.s. Maybe we need training on this
      before giving the ability.

    - \[Marko\] K/K Sig-Release has the final say on what gets in.

    - \[Ben\] That is not the charter today. We would need a change.
      This is a pretty big change. Cherry picks don’t work that way.
      During code freeze we delegate this behavior.

    - \[Adolfo\] We could fix the issue that Marko
      [<u>linked</u>](https://github.com/kubernetes/release/issues/2337)
      in the chat. Releases are the strongest part of what we do. We
      could write up a proposal.

    - \[Jeremy\] We can write something up and send to k/dev.

**(Optional) Walk the Board:**

- Project board review:
  [<u>https://github.com/orgs/kubernetes/projects/23</u>](https://github.com/orgs/kubernetes/projects/23)

- Incoming issue and PR triage

  - [<u>SIG Release issues
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>SIG Release PRs
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>Kubernetes
    issues</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)

## Mar 21, 2023

**Host (pronouns):** Verónica López

**Attendees (pronouns):**

- Stephen Augustus (he/him)

- Marko Mudrinić (he/him)

- Leonard Pahlke (he/him)

- Cici Huang (she/her)

**Recurring Topics \[timebox to 20 min\]:**

- \[cici37\] [<u>current release
  blockers</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Arelease-blocker)

  - Cici will follow up on with the team, we’re not sure if we need to
    delay?

    - We’re not sure how long it takes to fix release blockers

    - We still need to check what it takes to fix it and merge the PR

- \[cici37\] CI regex for missing rc.0 tag: [<u>fix in
  k/k</u>](https://github.com/kubernetes/kubernetes/pull/116809)

  - GitHub issue:
    [<u>https://github.com/kubernetes/release/issues/2972</u>](https://github.com/kubernetes/release/issues/2972)

  - krel got fixed with
    [<u>https://github.com/kubernetes/release/pull/2976</u>](https://github.com/kubernetes/release/pull/2976)

  - We need to fix it in k/k before proceeding with the rc.0 release

- \[Marko\] PR to fix release branch jobs generation:
  [<u>https://github.com/kubernetes/test-infra/pull/29098</u>](https://github.com/kubernetes/test-infra/pull/29098)

  - Recommended to wait for this PR to get merged before generating jobs

  - Hopefully to be merged today

- \[Veronica\] Proposal to start cutting releases a day earlier

  - Google Build Admins proposed that we cut release a day earlier so
    that we can finish on time

    - We very often have to delay a release for a day or two, which is
      making it hard to plan

  - We need to make sure that everything is okay before starting the
    release: krel, promo-tools, and everything else that we might need

    - \[Stephen\] promo-tools are bumped manually, so that’s usually not
      a problem

  - \[Marko/Stephen\] We might need to refactor promo-tools

    - promo-tools were made for a single registry, now we have many
      registries

    - Kubernetes releases shouldn’t take two days, we’re complex
      project, but not that complex

    - Publishing to many registries at the promo time is problematic, we
      should see about doing it in other way

    - We need to reduce the needed to cut/publish releases

  - We’ll take to the next meeting to discuss with more people

## Mar 14, 2023

**Host (pronouns):** Adolfo García Veytia

**Attendees (pronouns):**

- Xander Grzywinski (he/him)

- Nabarun Pal (he/him)

- Rodolfo Martínez (he/him)

- Grace Nguyen (she/her)

**Note Taker (pronouns):**

- Jeremy Rickard (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - \[Nabarun\] Patch releases for March to go out tomorrow

      - Build Admins have confirmed availability

      - Nabarun will start in the AM / US folks can take over later

    - \[mahamed\] image freeze stuff going well, what is status of 3.5
      kpromo?

    - \[puerco\] change for signature validation, that’s done now.

    - \[mahamed\] need this PR
      [<u>https://github.com/kubernetes-sigs/promo-tools/pull/669</u>](https://github.com/kubernetes-sigs/promo-tools/pull/669)

    - \[puerco\] promoter changes:

      - Changes related to fixing signatures that broke during last
        patch release

      - Tracking issue
        [<u>https://github.com/kubernetes/release/issues/2962</u>](https://github.com/kubernetes/release/issues/2962)

    - \[jeremy\] updated branch cut tasks to reflect kube-cross image
      task and updated publishing bot (sorry for the delay)

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - Code freeze today

- **Open Discussion:**

  - 

## Mar 7, 2023

**Host (pronouns):** Sascha Grunert (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Xander Grzywinski (he/him)

- Laura Lorenz (she/her)

- Jeremy Rickard (he/him)

- Jim Angel

- Mahamed Ali

- Adolfo Garcia Veytia

- Joseph Sandoval

- Arnaud Meukam

**Note Taker (pronouns):**

- Marko Mudrinić (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Laura from CI Signal

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - Patch releases for February went out last Tuesday

      - We disabled signature verification and proceeded with the
        release; some images are unsigned as a result

      - We still didn’t retroactively sign affected images

        - It’s still a mystery why this happened at all, we just know
          that all traffic hit the same pod

        - Sigstore increased the rate limit a bit and we hope this
          shouldn't hope again

        - We should also add some resiliency to the process

        - We thought that we have a solution for this, but last minute
          realized that not all cases are covered

          - Images might be detected as signed, but they might be signed
            with the staging identity instead of the production identity

        - We hope we’ll able to sign affected images this week

        - This tool that we’re working on is going to give us some
          additional options, like moving signing to async

          - The first version is an emergency one, additional
            improvements in the future

          - We cannot sign everything, for example if registry is
            compromised, we need to make sure that image is coming from
            the community

        - There are some upcoming changes to promotion, we’ll think
          about smarter ways to sign

        - \[Sascha\] Can we summerize this in a GitHub issue?

          - Adolfo will take care of that

    - KEP-1731 implementation is work in progress

      - The first part is ready for review:
        [<u>https://github.com/kubernetes/release/issues/2947</u>](https://github.com/kubernetes/release/issues/2947)

    - \[mahamed\]
      [<u>https://github.com/kubernetes/release/issues/2947</u>](https://github.com/kubernetes/release/issues/2947)
      image freeze implementation details

      - Two ways to this

        - Easiest: just remove all registries in the manifests and later
          (after KubeCon) fix tools

        - Harder: changes in kpromo to ensure everything is done
          properly but help from Adolfo and Carlos is needed to get
          everything approved in short time

          - \[Adolfo\] What kind changes would you like to do?

            - Replace all k8s.gcr.io references with registry.k8s.io in
              promo-tools

          - \[Arnaud\] Do we have a way to dry run this?

            - We should be able to early identify issues or blockers

              - There are some details in KEP-3720 about scope of
                planned changes

          - \[Sascha\] How much time do we have if anything goes wrong?

            - Start merging pull request at week of 20th March and see
              how thing look like

            - If anything goes wrong, we have ~5 days to rollback and go
              to the first idea

      - Mahamed to prepare everything for merge this week and we can
        start merging next week

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - \[Xander\] Things are still on autopilot, no major updates as of
      now

    - \[lauralorenz\] Follow ups from first [<u>1.27
      retro</u>](https://docs.google.com/document/d/1VcabRbSlEd9qqUm1ENWVXVHQYjS2mxLgqydlQGeC19o/edit#bookmark=id.yvpu3ha4p6hz)
      for CI Signal subteam

      - We want to get clarification on some aspect of the role should
        we continue doing it or not

      - Confirm utility and scope of CI Signal report

        - CI Signal report is a weekly email blast that goes to k-dev
          mailing list

          - It includes generated report
            ([<u>link</u>](https://gist.github.com/lauralorenz/3ca69bb88eda5979490392d8f3a82e3b#file-2023-02-22-report-md)),
            TestGrid status

          - We want to make sure that it’s still useful to attach this
            in email

          - \[Laura\] There’s a lot of duplication in the content there,
            wants to get feedback from SIG Release do we still need and
            what’s the historical reason for including it

          - \[Sascha\] Email are easy to miss, we have the same issue
            with both general and automated TestGrid emails

            - The right way to do it is to check with folks who are
              actually responsible for this job. We should get feedback
              from folks on k-dev to see if it’s useful. If it’s too
              much work, we can deprecate sending that email

            - \[Laura\] CI Signal is spending a lot of energy finding
              people who is responsible for job, we want to find out how
              useful this general email is

            - \[Laura\] Maintaining tool for this takes time and CI
              Signal team has a lot of different tasks and deadlines
              that they have to take care of

        - specifically the attached generated report [<u>like
          this</u>](https://gist.github.com/lauralorenz/3ca69bb88eda5979490392d8f3a82e3b#file-2023-02-22-report-md)
          and the [<u>per job details suggested in the
          template</u>](https://github.com/kubernetes/sig-release/blame/master/release-team/role-handbooks/ci-signal/template-weekly-ci-signal-report.md#L17).
          It so happened the generated report tool needed some
          unexpected maintenance this cycle and before putting updating
          docs/training on it I wanted to make sure its output is still
          useful.

          - Who is the audience?

            - Branch release management or sig release leads?

              - If so, from the experiences among those present it is
                not used by those audiences

            - General dev

              - How can well tell if its being used or what parts are
                useful?

              - Laura will figure out how to get general feedback and
                we’ll keep us updated

            - \[Laura\] Who should we ask about if this is useful?

              - \[Adolfo\] Motivation is to alert release managers
                before the release is out

              - \[Laura and Jeremy\] The tool was there for some time,
                this was initially done manually. We started doing this
                a long time ago

              - \[Adolfo\] There’s some duplication between TestGrid and
                email

              - \[Sascha\] Release Managers don’t need this because we
                pull data from TestGrid and we have GO/NO GO signal from
                the CI Signal team before cutting release

              - \[Laura\] This takes a lot of effort to curate

      - sig-release-releg-\* dashboards and presubmits dashboards

        - \[Laura\] It’s documented that watching those dashboards is
          the responsibility of CI Signal, but that’s not really being
          done. Should we watch those dashboards or remove that from the
          handbook?

          - \[Adolfo\] It’s responsibility of Release Managers to ensure
            those dashboards are green

          - \[Sascha/Laura\] Only interesting dashboards are
            master-blocking and master-informing for CI Signal, we want
            to make clear should we also watch sig-release-releng-\*

            - We can drop that from handbooks to reduce effort on CI
              Signal

            - We can also try to add why is that added in the first
              place

            - Does it handle the release process if there are failures?

              - There are important tools and it can affect ability to
                release.

              - \[Marko\] We can add it to the release cut template to
                check if those dashboards are green

                - \[Adolfo\] We should already be doing that

            - \[Sascha\] We want to deduplicate work for CI Signal and
              Release Managers, the current release branch is the most
              important for CI Signal

        - Little to no docs/training on these dashboards re: CI Signal
          team

        - What is the overlap and priority between these and
          master-blocking/master-informing

        - Is this still important for the CI Signal team to be watching?

        - Laura will make updates to the CI Signal handbook and ping
          someone from SIG Release to check

**  
Open Discussion:**

- \[Joseph\] [<u>KubeCon EU 2023 - SIG Meet and
  Greet</u>](https://github.com/kubernetes/community/issues/7161) the
  issue is open to reserve a table during Kubecon on Friday, April 21
  12:30-2:30 PM. Are we interested in staffing a table? [<u>K8s Con(feel
  free to add any topics you’d like to discuss as part of the
  agenda)tributor Summit
  CFP</u>](https://docs.google.com/forms/d/e/1FAIpQLSei6q-WqSx0JVUbgFoGRfMvldhCa5ZaND4wRjs6inNEigbLsQ/viewform)
  is now open as well.

  - The issue for reserving a table for SIG Meet and Greet is open

  - The CFP for Contributor Summit is open

  - Do we want to participate in CFP and have a table and Meet and
    Greet?

  - We’ll have a dedicated meeting on Tuesday

  - Meeting on Tuesday is mostly for SIG, while the KubeCon meeting is
    for the general community

  - We also have SIG introduction talk from Veronica and Marko on
    KubeCon

**(Optional) Walk the Board:**

- Project board review:
  [<u>https://github.com/orgs/kubernetes/projects/23</u>](https://github.com/orgs/kubernetes/projects/23)

- Incoming issue and PR triage

  - [<u>SIG Release issues
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>SIG Release PRs
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>Kubernetes
    issues</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)

## 28 Feb 2023

**Host (pronouns):**

- Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Cici Huang(she/her)

**Note Taker (pronouns):**

- Marko Mudrinić (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - February patch releases - signing issues

      - The promoter got rate limited when signing images. We don’t know
        why it got rate limited; sigstore folks checked logs and
        everything, and it turns that all requested went to the same pod

        - The rate limit didn’t change for months, it only got increased
          now

      - This impacted only patch releases, everything else signed
        afterwards is okay

      - There are 4 different states

        - No signatures at all

        - Partial signatures (e.g. one image has signature but not
          replicated to other mirrors)

        - Signature copied from staging, but promoter didn’t copy
          properly

      - @puerco wrote another command to check and fix images

        - Check existing of tag, if you find signature, assume it’s
          signed

        - But we don’t check if it’s staging or promoter signature

      - Options to continue

        - Update command to check for the promoter/production signature

          - We might need to delay patch releases for a week or even
            longer

        - Make a patch to allow completing release and then “throw” that
          release away, followed by new set of patch releases

        - We release without signatures, and eventually sign later if
          possible

          - Might be problem if someone is already checking signatures,
            but that’s going to happened with burnt release as well

        - Do we want to skip patch releases because next patch releases
          are in March?

          - We need to finish this release because images are already
            there

          - The only artifact we have right now are images

        - It seems like least risky is to just continue, let community
          know that releases are not signed, and sign later

          - \[Cici\] Will lack of signatures cause reliability issues?

            - \[puerco\] Those who check for those signatures will have
              issues at the time of releases; other than that we don’t
              expect any other issues

      - \[Jeremy\] TL;DR. - Continue release without signatures. We will
        send a notice to the community to communicate that signatures
        will come later and you should skip this release at this point
        if you need them

      - \[puerco\] We should clearly communicate that releases were cut
        last week; releases will have changes only up until then

        - We must make sure to not merge anything on release branches
          until we don’t get those releases out

      - We’ll resume releases tomorrow and alpha on Thursday. Carlos and
        Adolfo will create a PR to get rid of the check in krel release
        to unblock releases

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - \[Cici\] v1.27.0-alpha3 is planned on Thursday now

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- \[matttrach\] Is it possible to separate tags and artifacts?

  - Have tags before artifacts. There are folks (e.g. at Rancher) that
    depend on tag but not on artifacts

  - There are some problems with that and there are processes; the most
    complex part is in staging

    - If failures happen, we catch them before publishing tags

    - If we got tag early, we don’t have much options if we need to fix
      anything, we would need to scrap release entirely

- \[Mahamed\] Web banner on kubernetes.io

  - Legacy k8s.gcr.io container image registry will be frozen in early
    April 2023

  - It’ll be here for a little while before getting the KubeCon message
    back

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

## 21 Feb 2023

**Host (pronouns): Sascha Grunert**

**Attendees (pronouns):**

- Xander Grzywinski (he/him)

- Adolfo García Veytia (he/him)

**Note Taker (pronouns):**

- Joseph Sandoval (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - \[Sascha\]
      [<u>https://github.com/kubernetes/enhancements/pull/3750</u>](https://github.com/kubernetes/enhancements/pull/3750)
      needs final review. Out of tree enhancement. Marko has asked for
      review with additional feedback. Optimistic we can merge it.

    - \[Adolfo\] I’ll take a look. Update on a project I want to kick
      off in the 1st quarter. **Generate attestations**. There are other
      projects that could use our tools. It feels like its time to kick
      this off. Merge code that Mahammed has developed with Knative.
      Integrate those tools. In parallel use the tester to generate the
      attestations. Start with a smaller project and work all the way up
      to Kubernetes. If anyone is interested, reach out to Adolfo.

    - \[Sascha\] Security profile operator is looking to reach similar
      goals.

    - \[Adolfo\] Coming back from PTO and catching up on the KEP to
      freeze the registry.

    - \[Sascha\] For example this
      [<u>PR</u>](https://github.com/kubernetes/k8s.io/pull/4808). There
      is alot of suggestions on phasing out old images.

    - \[Adolfo\] What's the state?

    - \[Mahammed\] Latest on the SLSA work. SLSA 1 and 2 are done.

    - \[Adolfo\] Attestation was what you showed me?

    - \[Mahammed\] We generate the attestation from the Prow job.

    - \[Adolfo\] I’ll borrow the code and provide an update.

    - \[Mahammed\] Image freezing is almost complete. Development is on
      track.

    - \[Adolfo\] I’ll start attending the K8s-infra meetings

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - \[Xander\] It's quiet now we are past the Enhancement freeze. No
      exceptions requested. Only lost 9 enhancements. Total tracked
      enhancements = 80!!!

- 

**  
Open Discussion:**

- Do we plan to do a KubeCon EU Contributor Summit SIG Release meetup?

  - \[Sascha\] Perhaps we could do a similar meeting like in Detroit.
    Good discussions that lead to some actions.

  - \[Adolfo\] Having a similar meeting like the one in Detroit would be
    great. Do some outreach for people who are not familiar with
    SIG-Release.

  - \[Joseph\] + 1 to doing a follow up in Amsterdam

  - \[Sascha\] We could cover projects and also do an intro to
    Sig-Release

  - \[Joseph\] Content track for Contributor summit should open this
    week.

  - \[Sascha\] Marko and Veronica will be presenting for Sig-Release.

<!-- -->

- \[Arnadu\] What is the status of OBS?

  - \[Sascha\] Risk that the project might not make 1.27. Building
    packages will be similar to what we are doing already.

## 14 Feb 2023

**Host (pronouns):**

- Verónica López

**Attendees (pronouns):**

- Rey Lejano (he/him)

- Marko Mudrinić (he/him)

- Joseph Sandoval (he/him)

- Cici Huang (she/her)

- Xander Grzywinski (he/him)

- Leonard Pahlke (he/him)

**Note Taker (pronouns):**

- Joseph Sandoval and Rey Lejano

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - \[Cici\] Access <u>clarifications ,</u>

      - \[Veronica\] Granting access makes sense, open a ticket/issue
        for access

      - \[Cici\] To raise a separate PR to trigger

      - \[Veronica\] Suggest to err on the safe side, a PR may mess with
        someone’s work so it’s safe to let everyone know on the Slack
        channel.

      - \[Cici\] Will make PR and will tag release managers

      - \[Marko\] Do we want to update date for Go update

      - \[Cici\] On Go version bump, it’s a patch update so it should
        not affect alpha cut since master should have latest Go version.

      - \[Marko\] Marko thinks 1.20 is also affected

      - \[Veronica\] Saw some work during the week, and on the mailing
        list notes it says 1.20 is also affected but does not have the
        details

      - \[Cici\] Not sure if it will delay the alpha cut for
        1.27.0-alpha2 since no one is using it

      - \[Veronica\] Alpha cuts in the past is to flush out things, so
        not used by many. Request Cici to post this in the Slack channel

      - \[Cici\] Will send Slack message

      - \[Marko\] Please tag Dims and Liggit since they are triaging
        some Go issues

    - \[mahamed\] The registry freeze kep has been merged and the work
      has started

      - \[Mahamed\] Working on freezing old registry, working on
        infrastructure change this week and next week. And can cut over
        at end of the month. Need to find Adolfo and others to have a
        chat. So far, we’re on track

      - \[Veronica\] Do you need any help

      - \[Mahamed\] Not from SIG Release right now

      - \[Marko\] Thank you Mahamed for working on this as its an
        important cost saving measure

      - \[Mahamed\] Will work on a banner

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - \[Xander\] Enhancements freeze last week

      - \[Xander\] Only 9 KEPs did not meet requirements and no
        exceptions so far. In the 70s for enhancement that made
        Enhancements Freeze

    - \[Cici\] scheduled 1.27.0-alpha2 cut today. Delayed due to
      [<u>issue</u>](https://github.com/kubernetes/kubernetes/issues/115767).
      Working on verifying.

- \[Veronica\] Discussed on Slack and will send comms, supposed to have
  patch releases for Wednesday and they will be delayed by a day to make
  room for the Go updates

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

## 7 Feb, 2023

Canceled due to no agenda

## 31 Jan, 2023

Canceled due to no agenda

## 24 Jan, 2023

**Host (pronouns):** Jeremy Rickard (he/him)

**Attendees (pronouns):**

- Marko Mudrinić (he/him)

- Xander Grzywinski (he/him)

- James Laverack (he/him)

- Leonard Pahlke (he/him)

- Ana Margarita Medina (she/her)

- Stephen Augustus (he/him)

- Rodolfo Martínez (he/him)

- Rajesh Gunasekaran(he/him)

- Yash Raj Singh (he/him)

- Sascha Grunert (he/him)

**Note Taker (pronouns):**

- 

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

  - Welcome Micah

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - \[mahamed\] Freezing k8s.gcr.io image registry.
      [<u>KEP</u>](https://github.com/kubernetes/enhancements/pull/3723)

    - \[Marko\] January patch releases (1.26.1, 1.25.6, 1.24.10,
      1.23.16) are available!

      - Everything went mostly fine. We had a hiccup in the promotion
        process requiring us delay 1.24.10 for one day

      - We are (sometimes very) often hitting signing flakes
        [<u>https://github.com/kubernetes/release/issues/2809</u>](https://github.com/kubernetes/release/issues/2809)

        - We had to repeat some releases for 3 or 4 times

        - I increased the priority of this ticket to critical, it would
          be nice to fix it in time for February releases

    - \[Marko\] OBS POC in progress again, added an agenda point to
      discuss the
      [<u>KEP</u>](https://github.com/kubernetes/enhancements/pull/3750)

    - \[Marko\] Release branches are updated to Go 1.19.5, master is
      getting updated to Go 1.20rc3 ([<u>WIP
      PR</u>](https://github.com/kubernetes/kubernetes/pull/114502),
      tests are mostly green)

      - Thanks a lot to Carlos, Jordan, and Madhav for driving this
        forward!

      - \[Jordan\] e2e and all canaries are passing on new go version

    - \[Cici\] Planned alpha2 cut for 1.27
      today([<u>issue</u>](https://github.com/kubernetes/sig-release/issues/2157))

      - Blocker issue identified yesterday, but resolved.

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - 47 enhancements opted in so far

    - No major concerns

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- \[Marko\] Update and next steps for the OBS POC

  - KEP:
    [<u>https://github.com/kubernetes/enhancements/pull/3750</u>](https://github.com/kubernetes/enhancements/pull/3750)

  - Please Review

  - Notes: some changes to OS builds, no more deb spec, RPM converted to
    debspec by tooling

  - Highlights: will remove everything from our OBS project this week,
    will start fresh

  - KEP needs review, need to setup account in GCP for implementation

  - Need to decide on domain

  - \[Stephen\] for names, we should make a clean break

  - \[Jordan\] the testing gaps for ppc, s390x are pre-existing yes? We
    shouldn’t focus too much on existing orthogonal issues. Some
    historical issues with supporting platforms we can't test well:

    - [<u>https://github.com/kubernetes/kubernetes/issues/38067</u>](https://github.com/kubernetes/kubernetes/issues/38067)

    - [<u>https://github.com/kubernetes/kubernetes/pull/76974</u>](https://github.com/kubernetes/kubernetes/pull/76974)

    - [<u>https://github.com/kubernetes/kubernetes/issues/93621</u>](https://github.com/kubernetes/kubernetes/issues/93621)

    - [<u>https://github.com/kubernetes/kubernetes/issues/93620</u>](https://github.com/kubernetes/kubernetes/issues/93620)

    - [<u>https://github.com/kubernetes/sig-release/blob/master/release-engineering/platforms/guide.md</u>](https://github.com/kubernetes/sig-release/blob/master/release-engineering/platforms/guide.md)

  - \[Stephen\] yes, there is also a supported arch KEP that has been
    around for a while

  - \[Marko\] We could get rid of those archs

  - \[Stephen\] further down the line we should reassess what we build

  - \[Stephen\] Thanks for all the work on this Marko.

- \[Marko\] PRs looking for feedback:

  - Minor releases in the patch releases calendar:
    [<u>https://github.com/kubernetes/website/pull/39086</u>](https://github.com/kubernetes/website/pull/39086)

  - Note about incomplete cherry-pick series:
    [<u>https://github.com/kubernetes/test-infra/pull/28490</u>](https://github.com/kubernetes/test-infra/pull/28490)

- \[liggitt\] discussion of go update KEP:
  [<u>https://github.com/kubernetes/enhancements/pull/3749</u>](https://github.com/kubernetes/enhancements/pull/3749)

  - \[Stephen\] for clarity, go doesn’t \*really\* follow semver so
    every minor bump is really like a major bump

  - \[Jordan\] there were some go minor versions we could have adopted
    without issues in the past, but some instances have been a problem
    so we couldn't consistently do it, so we generally did not

  - \[Jordan\] as we’ve extended release support, older k8s patches are
    on unsupported go versions and there have been CVEs.

  - \[Jordan\] KEP goal/proposal overview

  - \[stephen\] be concrete about developer actions to mitigate behavior
    changes (envvar changes)

  - \[marco\]

    - consider updating sooner than 3 months if tests look good

    - not sure the 1-month post-kubernetes release buys us much

      - jordan: RC and qualification of .0 releases gets a decent amount
        of reports from the field from people starting to qualify new
        versions, even if they're not widely adopted in prod yet. not
        perfect, but does build some confidence we didn't miss
        Kubernetes-specific regressions

    - want to be sure we don't break release-1.x libraries with original
      go versions

      - jordan: unit/integration tests on original go version *should*
        ensure libraries remain usable without bumping go minors

      - stephen: need to look at tooling for creating release branches,
        test jobs, etc

  - 

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

**(Optional) Walk the Board:**

- Project board review:
  [<u>https://github.com/orgs/kubernetes/projects/23</u>](https://github.com/orgs/kubernetes/projects/23)

- Incoming issue and PR triage

  - [<u>SIG Release issues
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>SIG Release PRs
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>Kubernetes
    issues</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)

## 17 Jan, 2023 Canceled because of light agenda

## 10 Jan, 2023 (recording)

**Host (pronouns): Sascha Grunert (he/him)**

**Attendees (pronouns):**

- Xander Grzywinski (he/him)

- Cailyn Edwards (she/her)

- James Laverack (he/him)

- Marko Mudrinić (he/him)

- Leonard Pahlke (he/him)

- Rey Lejano (he/him)

- Jim Angel (he/him)

- Mahamed Ali

- Abhilipsa Sahoo (she/her)

- Rudraksh Karpe(he/him)

- Angelos Kolaitis (he/him)

- Drew Hagen (he/him)

- Ruheena Ansari (she/her)

- Stephen Augustus (he/him)

- Subhrodip Mohanta (he/him)

- Adolfo García Veytia (he/him)

- Joseph Sandoval (he/him)

- Harsha Narayana (he/him)

**Note Taker (pronouns):**

- Rey Lejano (he/him)

**Recurring Topics \[timebox to 20 min\]:**

- Welcome any new members or attendees

- Subproject updates

  - Release Engineering
    ([<u>https://github.com/orgs/kubernetes/projects/30</u>](https://github.com/orgs/kubernetes/projects/30))

    - \[Marko\] OBS: no major updates -- will try to come up with a KEP
      next week

      - \[Marko\] Will make a PR for a KEP next week and we can continue
        discussion on the PR

    - \[Marko\] Go updates:

      - master to Go 1.20:
        [<u>https://github.com/kubernetes/kubernetes/pull/114502</u>](https://github.com/kubernetes/kubernetes/pull/114502)

      - Release branches to Go 1.19

        - 1.24:
          [<u>https://github.com/kubernetes/kubernetes/pull/113956</u>](https://github.com/kubernetes/kubernetes/pull/113956)

        - 1.23:
          [<u>https://github.com/kubernetes/kubernetes/pull/113983</u>](https://github.com/kubernetes/kubernetes/pull/113983)

      - To be in discussed in-depth later (added an agenda point in Open
        Discussion)

    - \[Marko\] January patch releases (1.26.1, 1.25.6, 1.24.10,
      1.23.16) scheduled for January 18 (next week), with the
      cherry-pick deadline on January 13 (this Friday)

      - 1.23 is in the maintenance mode, but we should cut at least one
        patch release to ship the Go update

    - \[Sascha\] We can start working on the enhancements update like
      choose OBS and build our own infra and have the goal for 1.27

    - \[Marko\] List out solution (OBS), positives/negatives, the
      process and details about the packages

    - \[Jim\] Did a pairing session with Marko and can take off some
      load

    - \[Veronica\] Veronica has cherry-picks on Veronica’s agenda and
      will do them in the last week

  - Release Team
    ([<u>https://github.com/orgs/kubernetes/projects/29</u>](https://github.com/orgs/kubernetes/projects/29))

    - \[Xander\] Call for enhancements went out yesterday

      - \[Xander\] 9 enhancements so far

    - \[James\] Shadow selection ongoing

      - \[Xander\] Plan to finish shadow selection plans to end this
        week

      - \[James\] Shadow notification is Tuesday next week and we are on
        track

      - \[James\] Issue came up, missed two md files for shadow
        selection and need to add to the EA handbook.

      - \[Stephen\] Also send email about shadow survey statistics to
        k-dev

- (feel free to add any topics you’d like to discuss as part of the
  agenda)

**  
Open Discussion:**

- \[Marko\] Policy for Go updates on release branches

  - \[Marko\] Had 2 PRs for 1.23 and 1.24 release branches to Go 1.19.
    PRs merged, test grid is green, noticed no problems. But it’s
    against the policy, Marko worries the effect on the community since
    community has to update Go version also. Also send notice to the
    community to update their Go when using these versions

  - \[Stephen\] Let’s discuss the reasoning since we haven’t done that.
    With regards to policy changes, we should announce policy changes
    before they start happening and we should have discussion around
    policy changes before they happen. Let’s discuss why as next step

  - \[Marko\] Reason for that, for 1.23 was using Go 1.17 and it was
    eol. 1.24 was on Go 1.18 and it is eol

  - \[Stephen\] Having a Go version to go out of support is a forcing
    function for folks to upgrade Kubernetes. Was it timing with lagging
    branch and Go 1.17 support?

  - \[Marko\] For 1.23 we could have skipped Go update since its in
    maintenance mode. For 1.24 and it is going into maintenance mode in
    end of May, so have 3-4 months of maintenance mode and Go 1.18 eol
    (3 months before 1.24 reaches maintenance mode).

  - \[Stephen\] Makes sense for 1.24

  - \[Marko\] Not sure about going back (reverting?) since may have
    issues with publishing bot.

  - \[Stephen\] We should update our policy. Asking for feedback from
    release managers on how we feel about updating Go versions on
    already released branches

  - \[Puerco\] Need to review the policy and understand a little more

  - \[Veronica\] In general, we have done things to break boundaries of
    our policies and its normal as things happen, it’s worth revisiting
    policies like how we treated 1.22 – the policies are not serving
    edge cases. If we have clear policies, because we are distributed
    and work async, people make decisions when others are not around,
    policies help with decision-making

  - \[Stephen\] We should block on discussion with having a policy in
    place or something that dictates what happened in an edge case.
    Sooner block for discussion because its harder to revert then just
    do.

  - \[Arnaud from chat\] So no RC Golang for release branches ?

    - \[Stephen\] No. Goal is to not have a GA version of Go is release
      blocking

  - \[Puerco\] Was there a CVE as a rational for the Go updates

  - \[Marko\] There were some CVEs. Agrees with Stephen and Veronica on
    this. Marko jumped a little quickly on this, folks wanted it updated
    to get signal. We should update the policy. A reason why, Go was
    introducing some breaking changes

  - \[Stephen\] Request for Marko to go to the backlog on why the
    changes happened. Have to make sure we have a good reason like a
    massive security vulnerability or a super mega regression like 2-3
    releases ago. If it impacts people’s operation of Kubernetes then we
    should document it. Because where we are seated, we are the group
    that releases Kubernetes, we are in the role of in-service to other
    groups so that causes us to be more laxed of policies in-service to
    other groups. So we should question requests.

  - \[Puerco\] By bumping minor version we can alter how Kubernetes
    behaves. When our branches go out of support and when Go goes out of
    support – we should be conservative with changes. Also agree with
    what Stephen said if there is something very critical then we can
    bump. It’s not our job to make quick snap decisions – we should be
    on the conservative side

  - \[Marko\] On the PR, on performance improvements, release branches
    won’t receive certain performance changes with the use of env
    variables

  - \[Marko\] To recap this, prepare an announcement with what happened
    and state no negative impact on users, document policy change

  - \[Stephen\] The second we send an announcement and lock in another
    policy, we start beholding ourselves to that. Expect updates on
    previous branches as a result of that and expect people to point to
    our policy

- \[Marko\] Do we have a policy for announcing changes on k-announce?

- \[Marko\] Green light for proceeding with
  [<u>https://github.com/kubernetes/release/pull/2780</u>](https://github.com/kubernetes/release/pull/2780)

- \[Marko\] What’s the status of the registry changes?

  - [<u>https://kubernetes.slack.com/archives/CCK68P2Q2/p1672858680698639</u>](https://kubernetes.slack.com/archives/CCK68P2Q2/p1672858680698639)

  - \[mahamed\] Can we start working on freezing k8s.gcr.io registry? It
    is a major change and it would be nice to announce our intent early
    in the 1.27 release cycle.

- \[Marko\] Should we update the cherry-pick message (e.g.
  [<u>https://github.com/kubernetes/test-infra/blob/1958f266cee0b34ffc8a50e019911921f37884b3/config/prow/plugins.yaml#L993-L1002</u>](https://github.com/kubernetes/test-infra/blob/1958f266cee0b34ffc8a50e019911921f37884b3/config/prow/plugins.yaml#L993-L1002))
  to note that you should mention if cherry-pick to other branches is
  needed?

- (feel free to add any topics you’d like to discuss, even when they
  came up during the meeting)

**(Optional) Walk the Board:**

- Project board review:
  [<u>https://github.com/orgs/kubernetes/projects/23</u>](https://github.com/orgs/kubernetes/projects/23)

- Incoming issue and PR triage

  - [<u>SIG Release issues
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>SIG Release PRs
    needs-triage</u>](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage)

  - [<u>Kubernetes
    issues</u>](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)

<span class="mark">BEYOND THIS YEAR MOVED (After [<u>this
PR</u>](https://github.com/kubernetes/community/pull/7329) merges,
replace with link)</span>
