diff --git a/sig-release/meeting-notes-archive/2017.md b/sig-release/meeting-notes-archive/2017.md new file mode 100644 index 000000000..cc67446f5 --- /dev/null +++ b/sig-release/meeting-notes-archive/2017.md @@ -0,0 +1,358 @@ +## 19 December 2017 + + + +* [Volunteer for 1.10 release team!](https://github.com/kubernetes/sig-release/pull/49) + + +## 7 November 2017 + +Agenda + + + +* Revisit release process automation discussion + * Federation wants to release for 1.9 but is now in a separate repo + * @david-mcmahon: I can’t make this time slot unfortunately, but please send me any docs or information on what exactly you're trying to achieve here. + * My team is working on a few solutions to assist in releasing non-core Kubernetes components. Most don’t need the full orchestration that[ anago](https://github.com/kubernetes/release#kubernetes-release-process) provides for k/k. + * See some related work[ here for cluster-registry](https://github.com/kubernetes/cluster-registry/compare/master...javier-b-perez:cloud-builder) building using Google Cloud Builder + * Also the recent related[ update to push-build.sh](https://github.com/kubernetes/release/pull/444) to support federation pushes. +* Can we apply[ milestone automation](https://github.com/kubernetes/community/blob/master/contributors/devel/release/issues.md) to[ PRs too](https://github.com/kubernetes/sig-release/issues/26)? [enisoc] + + +## 24 October 2017 + +Agenda + + + +* [Code freeze enforcement](https://github.com/kubernetes/features/pull/493) + + +## 10 October 2017 + +Agenda + + + +* Kops team would like to partner with sig-release to have kops release process automated. Draft outline:[ https://docs.google.com/a/cnmconsulting.net/document/ d/1i29ntaLjRVmQuUkUXfnkL3_jLby38At98kaE5VEJvg8/edit?usp=sharing](https://docs.google.com/a/cnmconsulting.net/document/d/1i29ntaLjRVmQuUkUXfnkL3_jLby38At98kaE5VEJvg8/edit?usp=sharing) + * Want to partner with SIG Release + * Focus on release mechanics + * Where to put trusted keys? + * Want to find common denominators around projects + * General guidelines are important, and perhaps an end-state where artifacts get published + * Define how we handle release artifacts and build backwards from there + * Need to put something on the end of the test infra that does the push + * Q: Who can make a call on where the bits in the registry live? \ +A: + * Notes +* Reminder about[ release roles for 1.9](https://docs.google.com/spreadsheets/d/1NGEtufpSfYlJwsB_0D9gF80Y24JQEreut9gT_QIN6Kk/edit#gid=0) + * solidify the roles +* When/How do release roles get finalized? + * Jaice will send announcement to begin timer on lazy consensus +* Product Management reboot proposal[ draft](https://docs.google.com/document/d/113gnr3tKXv79J0IYHyg6VbQwLfb-sCYGphW4D9dLZ_E/edit#) +* Updated milestone munger has been deployed + * Next step is migrating to a prow plugin + * Better maintainability + * Enables slack notifications + + +## 29 August 2017 + +Agenda - Recording + + + +* Attending + * Jaice Singer DuMars + * Radhika + * Jennifer Rondeau + * Mehdy Bohlool + * Caleb Miles + * Aaron Crickenberger + +Notes: + + + +* 1.8 Release update + * [Issues](http://bit.ly/18issues) + * [Release notes](http://bit.ly/18relnotes) + * [Feature tracking](http://bit.ly/18features) + * [Tests](http://k8s-testgrid.appspot.com/release-master-blocking#Summary) + * Anything else we need to be tracking + * Upgrade tests are failing - Eric and Mehdy will look at these + * Serial tests are looking mostly ok + * SIGs are not checking off their release note additions, so we need to closely monitor this + * Issue automation for milestone labeling is merged, needs an announcement + * Getting a Prow plugin to limit who can apply “approved-for-milestone” label, akin to LGTM controls + * Use of[ OWNERS_ALIASES](https://github.com/kubernetes/kubernetes/blob/master/OWNERS_ALIASES) might be fraught with peril + * Short term, we may not need auditability as a primary concern, and will defer to teams + * Aaron: SIG Release creates a GitHub team for this purpose, specifically + * Having a limited entry point into the team may increase security around the implementation + * Notify: kubernetes-dev, kubernetes-sig-leads, kubernetes-sig-release, Community meeting + * Caleb will help Maru create the team + * What are the true release-blocking tests? How do we actually get a 100% passing version + * A release-blocking job that only runs X days may not be useful + * More frequent runs may be required, and ergo removed from required if it can’t be + * Currently many failing GKE/GCE jobs + * Need to establish accountability + * Guidelines around “how long can something block the submit queue”? + * Triage + plan to resolution / project-wide SLA = policy + * Who actually owns these tests? + * Jaice will draft _something_ here + * +* Announcements? + * + + +## 15 August 2017 + +Agenda - Recording + + + +* Attending + * Aaron Crickenberger + * Jaice Singer DuMars + * Phillip Wittrock + * Mehdy Bohlool + * Steve Perry + * Jennifer Rondeau + * Radhika + * Caleb Miles + * Garrett Rodrigues + * Brian Tardell + * Ryan Kubiak + * David McMahon + +Notes + + + +* 1.8 Release update + * Feature freeze exceptions + * [https://github.com/kubernetes/features/issues/382](https://github.com/kubernetes/features/issues/382) + * [https://github.com/kubernetes/features/issues/384](https://github.com/kubernetes/features/issues/384) + * [https://github.com/kubernetes/features/issues/383](https://github.com/kubernetes/features/issues/383) + * [https://github.com/kubernetes/features/issues/386](https://github.com/kubernetes/features/issues/386) + * [https://github.com/kubernetes/features/issues/387](https://github.com/kubernetes/features/issues/387) + * Jaice would be deferent to features which are already in progress + * Jaice to take ownership of the PRs with Ihor and see what their status is ~ _Done as of 8/16 JSD_ + * **Question**: how does the feature freeze impact the release cadence? **<-** **Jaice **Idea is to have some idea what work is planned before coding starts. Are these features “funded” with development efforts. Ideally we would have a release planning session where SIGs present what work is scheduled, what cross SIG dependencies exist, and SIGs would be accountable to their commitments + * [https://github.com/kubernetes/features/blob/master/release-1.8/release-1.8.md](https://github.com/kubernetes/features/blob/master/release-1.8/release-1.8.md) + * Still no alphas cut? + * Should be three already. Is this a risk? + * Jaice: it is a risk, we have not have met the criteria for releasing with regards to CI + * Caleb: I believe this is + * Grod: next (last) alpha is scheduled for 23rd August. We should aim to have all tests passing by then. Suggest using 23 August as hard deadline for all tests passing + * Thur 17 plan to cut an alpha - even if tests aren’t passing + * Meant to validate release cutting infra is correct + * Need to find a branch manager for weeks Adam is out +* [Merged PRs in the 1.8 milestone](https://docs.google.com/spreadsheets/d/18V_8YvqemC96W17O_3WZ6H_kS7r77NAI9FqUHAM2MS0/edit?usp=sharing) with release-note=true + * [GitHub query](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20label%3Arelease-note%20milestone%3Av1.8) +* Dawn’s proposal lives! Jago hiring 2 people to work on the coding + * Strive to link issues + * How do we get more support for all the back end processes/bots/CI? + * +* [Docs PRs](https://github.com/kubernetes/features/pulls) in flight + * Radhika sending an email to each group, then collect, edit and apply tech writer magic sprinkles + * re: action required, need to untangle release note PRs + * Better release note template might help + * 2 classes of release notes: + * Items in the k/features + * items in PRs against k/k ~ not organized … +* Planning for roll-out of milestone-maintaining munger + * Proposal:[ https://github.com/kubernetes/community/blob/master/contributors/devel/release/issues.md](https://github.com/kubernetes/community/blob/master/contributors/devel/release/issues.md) + * PR:[ https://github.com/kubernetes/test-infra/pull/4052](https://github.com/kubernetes/test-infra/pull/4052) + * If you don’t label it sufficient for triage, it warns, then moves out of the milestone + * priority, type, SIG + * 3 day grace period + pings to GH UIDs (no email currently) + * Need to send email to all the SIGs / present at community meeting + * For critical/urgent they warn without kicking out + * labels will allow us to track issues that get ejected + * Initial grace period could be 1 weeks - Enough to give SIGs a chance to take action + * This is step one, but the proposal has additional work items + * Is there a notion of SIGs acknowledging ownership? + * Maru needs a cool hat + + +## 1 August 2017 + +Agenda - Recording + + + +* Attending + * Jaice Singer DuMars + * Erick Fejta + * David McMahon + * Aaron Crickenberger + * Phillip Wittrock + * Dan Gillespie + * Joe Betz + * Steve Perry + * Maru Newby + * Caleb Miles + +Notes: JSD + + + +* 1.8 Release update + * Feature freeze status check + * Any issues with no/wrong SIG + * This is a “stability” release, so are there things that we see in the feature queue that should be deferred? + * Does anyone know of notable missing features? + * Release team roll call + * [https://github.com/kubernetes/features/blob/master/release-1.8/release_team.md](https://github.com/kubernetes/features/blob/master/release-1.8/release_team.md) + * Release risks + * Upgrade testing + * Issue inclusion process ~_ need a way to decide what a substantial PR is for the contrib ladder_, also ensure that PRs are properly documented for release notes + * Is Dawn’s proposal being staffed? + * Work to extract release notes from the issue has not been completed + * This part happens in[ relnotes](https://github.com/kubernetes/release/blob/master/relnotes) and I can do this [@david-mcmahon] + * Need to ensure there’s a clear value prop for the requirement + * Phil’s proposal needs some dry run +* kops test failures and escalation of[ this](https://github.com/kubernetes/kubernetes/issues/49981) + * escalate to Justin SB / Kris Nova / Luis + * kops is unique in that it’s fully owned + * Test fails because of X and is owned by SIG Y, Z, B + * Ownership gets dropped so it’s difficult to get traction + * Release team has to find someone to fix it, which can be circular + * When low-level or high-impact tests fail, hard to know where to send this + * etcd testing specifically is tricky to understand + * umbrella issue for putting [sig-foo] in e2e test names ~ on Aaron’s backlog + Phillip Wittrock + * [https://github.com/kubernetes/kubernetes/issues/49161](https://github.com/kubernetes/kubernetes/issues/49161) + * + * Escalations for SIGs: mailing list / Slack / Individuals + * Put ACTION REQUIRED in the subject or IMPORTANT + * SIG rep for release if you have a SIG-owned issue or feature in the release ( if ! sig-lead ; then someone else) + * +* upgrade and serial tests - massive failures [pwittrock] + * Upgrade - Mehdy Bohlool? + * These have been highly problematic in past releases + * Serial - ??? + * Must pass before a release is cut currently, but not blocking + * Could SIG merge be blocked by failing serial test? + * SIG-specific tests are the only way to validate a SIG’s contribution, so if it’s not fixed, then merges for code needing its validation should be blocked + * Break up serial jobs and apply them to a cluster that is not subject to prior changes + * Write this up so we can get feedback ASAP [ Maru & Phillip ] + * Execution frequency would be lowered +* etcd3 test flakes [pwittrock] +* [Hooking in deb/rpm publishing in anago](https://github.com/kubernetes/release/issues/365) [calebamiles] + * A/B/RC/Final + * Needs someone from Google because of the bucket (Mike Danese has done this up to now) + * Permissions model may be interfering + * Anago is OSS so someone could update it, but there may be some tripwire around Google accounts vs. not + * Caleb will connect with mikedanese@ (@mikedanese) to get a walkthrough + * Also reach out to djmm@ (@david-mcmahon) for any details on the included google layer that does google-specifics (push) + + +## 18 July 2017 + +Agenda -[ Recording](https://youtu.be/I0KbWz8MTMk) + + + +* Attending + * Jaice Singer DuMars + * Phillip WIttrock + * Caleb Miles + * Dan Gillespie + * Aaron Crickenberger + * +* Notes: JSD +* 1.8 Release Team [DG] + * Empty + * Aaron willing to help but cannot commit to a role full-time + * Assisted with issue triage and pushing PR’s forward last time + * Trying to highlight escalation paths + * Companies should be funding a role, especially in terms of commitment to the project + * Going to advertise for more role involvement + * At community meeting we will announce that we have a tentative lead/second & ask (final) if there are any other nominations + * Also posting to SIG release list + * Alpha release stability needs to be understood - no more releases without green tests + * Phil is going to check on Google-specific roles +* Establishing release criteria / standards / due dates [DG] + * **Set[ release due dates](https://github.com/kubernetes/features/pull/305) for all of these items**: (merge[ https://github.com/kubernetes/features/pull/305](https://github.com/kubernetes/features/pull/305) ) + * CI test signal + * Which suites/jobs are required? + * As of 1.7 anago polls testgrid’s config.yaml for release blocking jobs;[ issue for context](https://github.com/kubernetes/release/issues/340) + * Anago polling testgrid’s config.yaml[ https://github.com/kubernetes/release/blob/master/lib/releaselib.sh#L115](https://github.com/kubernetes/release/blob/master/lib/releaselib.sh#L115) + * The specific set of tesjobsts from config.yaml[ https://github.com/kubernetes/test-infra/blob/master/testgrid/config/config.yaml#L2036-L2119](https://github.com/kubernetes/test-infra/blob/master/testgrid/config/config.yaml#L2036-L2119) + * Of note: upgrade jobs are not included in this list + * Of velocity note: slow jobs are + * What tolerance for flakes do we expect? + * ie. 5 passes in a row on a commit we are releasing + * In the past, we as humans have used “3 passes in a row” + * Anago appears to look at 2 passes in a row if[ this comment in find_green_build](https://github.com/kubernetes/release/blob/2338077e3911ad2ede6473b2777ead38e9edf5b0/find_green_build#L32) is to be believed + * When in the release cycle should this be stabilized? + * Prior to cutting a build/release + * Need to determine how long we support version-specific test cases, e.g. etcd2 upgrade path, beta upgrades ~ what SIG owns upgrade lifecycles? (SIG Cluster Lifecycle?) + * API Policy + * Component upgrades + * Management of failing tests + * Need to raise this in community and get test-owner reps in the release meetings + * Upgrade test signal + * Which upgrade conditions do we expect to validate? + * Multi-master? + * etcd version? + * When in the release cycle should this be stabilized? + * Release Notes + * What format are they expected to be in? + * When in the release should this be complete? + * By the time feature freeze happens, we should have enough to create the PM-oriented “theme release notes” + * Need to enact the ‘marketing’ and ‘release notes’ roles + * A draft of the notes by code freeze will be required + * Call out features with no docs + * Call out features with no release notes + * Cherry picks are hard, and porting info across PRs is difficult + * Maybe this is not in the PR, and lives somewhere else + * Documentation + * When in the release should this be complete? + * Bugs +* **Action items summary from this meeting:** + * ~~Raise awareness (again) about release roles~~ (@ community and @ SIG-Release Google Group) ~ **Jaice & Caleb** + * **~~PWittrock to seek out Googlers for G-specific release roles~~** [ ~~branch manager~~ | patch release manager ] + * ~~Give attention to Community PR 305 and ratify on Friday~~ ~ **SIG Release team** + * ~~Schedule follow-up meeting for Friday at 2PM PT / 5PM ET / 9PM UTC~~ ~ **Caleb or Dan** + * Determine if/how an Alpha is possible ~ **SIG Release team** + * ~~Use Anago test list as the source of truth for release-blocking tests~~ ~ **Aaron** will provide this info + + +## 6 June 2017 + +Agenda - Recording + + + +* Attending: + * Jaice Singer DuMars + * Anthony Yeh + * Ihor Dvoretskyi + * Maru Newby + * Caleb Miles + * Doug Claar + * Javier Perez + * David McMahon + * Andrew Chen + * Phil Wittrock +* Notes: JSD +* [ 0:01 ] What are we hoping to achieve from this meeting? + * How do we make long-term improvements to the release process? + * How do we empower teams to do more of the release tasking themselves? + * Different than the work cutting releases + * Iterating and improving on every release as a group + * Release team forms/disbands around the release so not a durable group + * Tools we need in order to improve the process, vs. release team uses current tool set +* [ 0:09 ] Updates + * SIG CLI was working on a page in the test grid, filtered for easy understanding + * [1.6 Retro Action Items](https://docs.google.com/document/d/1JjQW14R9FCaKceZV2hE9Wx41LIUoZST8gp8FF5EqMmk/edit) + * We need a measurable goal + * Do a mock release weekly to verify the process and testing [ Phil leading this ] + * What are the tools needed to make this happen? + * Start with a PR for discussing how to enact this work + * Release team retro [ Jaice ] + * ID bugs and issues we already know about [ ID targets for continuous improvement ] + * Schedule meeting \ No newline at end of file diff --git a/sig-release/meeting-notes-archive/2018.md b/sig-release/meeting-notes-archive/2018.md new file mode 100644 index 000000000..7e563c1aa --- /dev/null +++ b/sig-release/meeting-notes-archive/2018.md @@ -0,0 +1,764 @@ + + + + + + + + + + + + + + + +
Date: + Dec 18, 2018 +
Recording + https://youtu.be/WRJw27_QkiI +
Attending: + +
    + +
  • \ +Tim Pepper + +
  • Aaron Crickenberger + +
  • Kendrick Coleman + +
  • Josh Berkus + +
  • Caleb Miles + +
  • Aish Sundar + +
  • Benjamin Elder + +
  • Davanum Srinivas + +
  • Stephen Augustus +
    • +
+
+ + + + +* \ +Part #2 of the 1.13 release retrospective:[ https://docs.google.com/document/d/1mKrJm3N4dC4rfOwa5WcvOptmjpteMcNOg1ZSoACKU1g/edit#](https://docs.google.com/document/d/1mKrJm3N4dC4rfOwa5WcvOptmjpteMcNOg1ZSoACKU1g/edit#) +* Part #1 was[ https://youtu.be/e5oHnmUOz3k?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ](https://youtu.be/e5oHnmUOz3k?list=PL69nYSiGNLP1pkHsbPjzAewvMgGUpkCnJ) + + + + + + + + + + + + + + +
+Date: + Dec 4, 2018 +
Recording + https://youtu.be/7gnQw8Jd2NA +
Attending: + +
    + +
  • \ +Tim Pepper + +
  • Aaron Crickenber (@spiffxp) + +
  • Kendrick Coleman + +
  • Hart Hoover + +
  • Josh Berkus + +
  • Claire Laurence + +
  • Nicholas Lane + +
  • Caleb Miles +
    • +
+
+ + + + +* \ +1.13 anything else to do? +* 1.13 retrospective: + * Thursday at community meeting. + * [1.13 Retro doc](https://docs.google.com/document/d/1mKrJm3N4dC4rfOwa5WcvOptmjpteMcNOg1ZSoACKU1g/edit#heading=h.tw06ll716grh) (and[ 1.12 retro doc](https://docs.google.com/document/d/1OgylAYqU0YoJz-PTd8uzyHtMcxYSewSq06AGeh1F-A8/edit#) for reference) + * Moderator? Don’t have Jaice confirmed yet. + * Jaice can do it, but probably should pass the baton so more people get experience doing this +* 1.13 Changelog, or what is a good changelog in general: + * [Major Themes](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.13.md#major-themes) section: reads very high level and broad for some SIGs. Content is delegated to the SIGs, but the overarching doc then needs coherent editorial voice. The release team relnotes lead needs help (SIG Architecture) help on the “reduce” part following the “map” + * Should it refer to commercial offerings in the SIG ${cloud provider name} sections? + * Steering committee has two bullets in their charter: + * Decide how and when official releases of Kubernetes artifacts are made and what they include + * Declare a release, so that the committee can ensure quality/feature/other requirements are met. \ +If it's steering who makes these choices (and associated support stance declarations?), some might feel they have insufficient voice, even though Steering is elected by all of us. + * Aaron will bring up question on changelog content in steering + * We as a group can discuss in the retro on Thursday +* Test flakes (spiffxp, jberkus): + * can we do some targeted work in December? + * Shifting issues from 1.13 milestone to master + * Umbrella: this is a massive amount of work, plus complex interdepencies and runtime context specific interactions increasing the flake likelihood + * Specific areas: + * Upgrade / Downgrade: needs critical mass of the right stakeholders to declare a better test architecture + * Event passing anti-pattern:[ https://github.com/kubernetes/kubernetes/issues/71646](https://github.com/kubernetes/kubernetes/issues/71646) + * 5k node Scale test + * Numretries: is this hiding flakes? + * Dashboard: community view into what is flaking, how much, feel the scope of problem. Partly in[ velodrome/bigquery](http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1), also[ triage dashboard](https://storage.googleapis.com/k8s-gubernator/triage/index.html), and daily pre-submits. + * Release blocking criteria: need to improve flakes to make the testgrid board truly reliably release blocking, starting with measurement. + * Flakiest job, flakiest test are two separate things. + * Josh & Aaron want a dashboard of flakiness to assess where to prioritize fixing versus replacing + * Release-master-informing dashboard refining: 5k node? Serial jobs? + * [https://github.com/kubernetes/sig-release/issues/347](https://github.com/kubernetes/sig-release/issues/347) + * Also[ https://github.com/kubernetes/sig-release/issues/405](https://github.com/kubernetes/sig-release/issues/405) Why “extra”? + * Scale testing: things aren’t progressed far along enough to CNCF to know what tests will be there versus dedicated hardware. Needs dollars and staffers. WG K8s Infra is working to move this along, but is at the level of determining things like access controls, moving tooling like gcbmgr, etc. Not quite at the level of “who’s got a spare 5k node” on which to run tests, but if those are laying around, serial runs on such a cluster could be plumbed in. See #k8s-infra-team +* Release teams’ shadow selection process (justaugustus, jberkus, tpepper) + * Status check: drawing up a lightweight process and sample questionnaire. Finalize to ungate 1.14 shadow selection. + * Josh: we should go ahead. Remaining one incompleteness is: too hard to set up forms per role. Could do a generic questionnaire form, if the role handbooks all have a requirements section. ToDo during 1.14: add requirements to the role handbooks. In the meantime, 1.14’s leads have an informal conversation about requirements with prospective shadows. Send out the survey. + * Aaron: a good goal is to have a list of easily/readily delegatable tasks +* SIG charter cleanups (chairs ongoing TODO):[ https://github.com/kubernetes/sig-release/issues/378](https://github.com/kubernetes/sig-release/issues/378) +* Formalization of alpha/beta/stable graduations: + * Move enhancements tracking to PR based flow instead of issues + * KEP-like? + * SIG Architecture oversight sign off? Something that’s beyond the SIG(s) in question on a SIG(s) focused KEP. Also need to not be a bottleneck, or inject very late surprises. + * Release lead should have a simple checklist set of criteria including tests, coverage, stability, key contact individuals, reviewers/approvers. Too much for the release lead (or enhancements lead) to be able to assess from a KEP that it is genuinely on track. + * Is 1.14 the release in which to require tracked enhancements have a KEP? No, not yet. Need to finalized that more broadly across the org/project, establish criteria. Implement for 1.15. + + + + + + + + + + + + + + +
+Date: + Nov 20, 2018 +
Recording + https://youtu.be/jmJOczWHFik +
Attending: + +
    + +
  • \ +Tim Pepper + +
  • Aaron Crickenberger + +
  • Arnaud Meukam + +
  • Hannes Hoerl + +
  • Hart Hoover + +
  • Kendrick Coleman + +
  • Cole Wagner + +
  • Niko Penteridis + +
  • Aish Sundar + +
  • Nicholas Lane +
    • +
+
+ + + + +* (justaugustus) Updates to SIG leadership + * [https://github.com/kubernetes/community/pull/2778](https://github.com/kubernetes/community/pull/2778) +* (justaugustus) Review SIG Release charter + * [https://github.com/kubernetes/sig-release/pull/348](https://github.com/kubernetes/sig-release/pull/348) + * [https://github.com/kubernetes/community/pull/2919](https://github.com/kubernetes/community/pull/2919) +* (justaugustus) Subproject / WG formation updates + * RT (release team) + * Release Engineering + * PST (project security team) + * LTS (long term support) +* (justaugustus) Enforce milestone for merge + * [https://github.com/kubernetes/sig-release/issues/321](https://github.com/kubernetes/sig-release/issues/321) (help wanted) + * Discussion noted in the issue + * Two sides: + * Is this for a gate of what might go in? + * Is this for post-tracking of what went in? +* (justaugustus) What qualifies for a milestone? + * [https://github.com/kubernetes/sig-release/issues/320](https://github.com/kubernetes/sig-release/issues/320) (help wanted) + * We have a loose process, which covers test cases, test health, docs. + * Declaration of “critical-urgent” during code freeze is sort of intentionally meant to be a discussion between the release team and stakeholder SIGs. + * Feature branches with demonstrable test can help. + * Green CI signal is one criteria. + * Do we have a problem that needs fixing? Is code missing a milestone due to vague acceptance criteria? We’re not seeing it as release leads this year. + * 1.10, 1.11 Josh notes there’s been “a team” that would abuse code freeze. The team specifically varied each time, so not a clear pattern. One was a SIG/developers who weren’t paying attention and just straight missed code freeze. + * 1.12 Tim each SIG was responsible and was on track, and then admitting they were behind, and then all who were behind did make a no-go call + * 1.13 Aish similar to 1.12. Most of friction now at code freeze is bug fixes so the question is how critical of a bug fix is it. Leads are erroring on the side of caution. +* (tpepper) What is supported? PR clarifying supported component version skews and upgrade:[ https://github.com/kubernetes/website/pull/11060](https://github.com/kubernetes/website/pull/11060) +* (nikopen) Decision making, critical thinking, “rule” malleability and definition of “urgency” + * How strictly are the role handbooks meant to be followed? Does following handbooks in a word-by-word manner create issues in more nuanced scenarios? + * Do labels always have a strict definition? Example: the priority label, and its usage across separate cycle stages (precode slush, code freeze etc) + * Does the enforcement of an ‘important-soon’ label on code slush offer any benefit? The priority filtering already occurs in code freeze where the merge conditions tighten. + * How tight are merge conditions in code slush and code freeze? How strict are rules meant to be followed? An example[ being this](https://github.com/kubernetes/kubernetes/pull/71208): fix to a CI flake. Details were not given but intent was clear. + * To avoid being overly strict in the future, can it be written down that handbooks are meant to be guidelines in order to solve actual problems (i.e. getting new code in a new release) and to not lose track of the actual goals over guidelines. + * Iterating on role handbooks == GOOD! + * Do it before and after and during release cycle + * Good to capture some specific examples: in release 1.x we worked through issue foo, weighed x/y/z evidence from SIG’s A/BC, decided such and such based on that. + * “Critical” and “Urgent” are clear and strong words. Same for “Important” and “Soon”, especially relevant to the other two. The “Kind” labels also are meant to be clarifying. + * It does remain subjective often and require conversation. + * Human conversation 1:1 continues to be our most productive means of getting something done urgently from a responsible SIG. + * +* (justaugutus, jberkus and aishs) Discuss 1.14 RT nominations + * Fill in lead roles[ https://github.com/kubernetes/sig-release/issues/372](https://github.com/kubernetes/sig-release/issues/372) + * Need especially a candidate + * Discuss shadow selection proposal -[ https://github.com/kubernetes/sig-release/issues/368](https://github.com/kubernetes/sig-release/issues/368) + * This isn’t about having a job interview, rather a conversation with prospective mentees/shadows..have they read the role handbook, do they have questions, is the time commitment something they can afford, are they ready and willing to learn & contribute. This is necessary to insure a successful mentor/mentee relationship. + * If we go this route SIG Release wouldn’t be involved in the shadow selection, rather just role lead and the release team lead + * Need to make sure time commitments across the release cycle are clearly in each role handbook + * Discuss roles and processes for Patch release team -[ https://github.com/kubernetes/sig-release/issues/369](https://github.com/kubernetes/sig-release/issues/369) +* Backlog: (tpepper and maybe jberkus) Upgrade test improvement plan: + * Upgrade issues and test case ambiguity causing continual release friction + * Forward plan is roughly to drop kube up and cluster/*, move to kubeadm only, assess coverage gaps, improve test coverage, insure skew testing works, version test cases. + * But this spans SIGs Release, Cluster Lifecycle, and Testing and that means it lacks a clear owner and staffing. + * Does this merit a WG or subproject so somebody’s name is owning creating a plan and coordinating execution of the plan? Is it a first task for the proposed (above) Release Engineering subproject/WG? + * What about downgrade? + * See also “what is supported” PR:[ https://github.com/kubernetes/website/pull/11060](https://github.com/kubernetes/website/pull/11060) + + + + + + + + + + +
+Date: + Nov. 6, 2018 (Cancelled. Too many people with conflicts +

+Oct 23, 2018 (Cancelled. No agenda and meeting invite has gone missing :/ ) +

Attending: + N/A +
+ + + + + + + + + + + + +
Date: + Oct 9, 2018 +
Attending: + +
    + +
  • \ +Jaice Singer DuMars + +
  • Dhawal Yogesh Bhanushali + +
  • Kendrick Coleman + +
  • Aaron Crickenberger + +
  • Aish Sundar + +
  • Benjamin Elder + +
  • Caleb Miles + +
  • Arnaud Meukam +
    • +
+
+ + + + +* \ +**POSTPONE**: 2nd portion of 1.12 release retrospective +* Charter draft:[ https://github.com/kubernetes/community/pull/2778](https://github.com/kubernetes/community/pull/2778) + * Needs issues addressed ~ See PR, perhaps move to collab doc for edits +* Subprojects + * release team + * Owning the document of the process + * OWNERS = release process + * Each team has latitude to run the release + * SIG Release as owners of the handbooks (e.g. process), + * release engineering and infrastructure/automation improvements + * + * licensing/compliance + * + * Security / Threat Model / Provenance + * + * LTS (or[ WG](https://docs.google.com/document/d/1PWhUhm_a27BoWCGKZ5Bg6bV-DW-N06gIQIbrpwoscck/edit#heading=h.46dch1el9d8c)?): + * [https://github.com/kubernetes/community/issues/2720](https://github.com/kubernetes/community/issues/2720) + * [Doc](https://docs.google.com/document/d/1PWhUhm_a27BoWCGKZ5Bg6bV-DW-N06gIQIbrpwoscck/edit#heading=h.46dch1el9d8c) /[ ppt](https://docs.google.com/presentation/d/1-Z-mUNIs3mUi7AdP1KwoAVNviwKrCoo3lxMb5wzCWbk/edit#slide=id.g338ac0a8b6_0_36) + * Proceed with discovery on this +* Release cadence of patch releases -[ https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/patch-release-manager#release-timing](https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/patch-release-manager#release-timing) + * Predictability would be helpful here + * The patch release team….? +* Feature graduation and release guidance for out-of-tree Cloud Provider projects + * Skip to stable? + * SIG Cloud Provider / Architecture owns this +* Release-blocking job criteria[ PROPOSAL](https://docs.google.com/document/d/1kCDdmlpTnHPQt5z8JzODdFCc3T2D4MKR53twsDZu20c/edit) + * + + + + + + + + + + +
+Date: + Sept 25, 2018 +
Attending: + +
    + +
  • \ +Arnaud Meukam + +
  • Tim Pepper + +
  • Jaice Singer DuMars + +
  • Kevin Kelani + +
  • Aish Sundar + +
  • Jim Angel + +
  • Kendrick Coleman + +
  • Lubomir I. Ivanov + +
  • Silvia T. Sandy-Martinez + +
  • Chuck Ha + +
  • Josh Berkus + +
  • Jorge Castro + +
  • Dhawal Yogesh Bhanushali + +
  • Stephen Augustus + +
  • Sebastien Vas +
    • +
+
+ + + + +* RPM(s) and DEB(s) for Release Candidates - dims + * Next meeting +* 1.13 Updates - AishSundar + * [1.13 timeline ](https://github.com/kubernetes/sig-release/pull/294) + * 1.12 is wrapping up + * 1.13 should start 10/1 - fully staffed release team + * Will adjust the timeline as needed + * Minding feature fallthrough from 1.12 because of a short timeline + * 1.13 team - you should have emails/invites + * +* SIG sessions for Seattle, Shanghai + * Shanghai + * [Intro](https://kccncchina2018english.sched.com/event/FuLM/intro-sig-release-tim-pepper-vmware): Tim Pepper - intro and how to get involved + * [Deep-dive](https://kccncchina2018english.sched.com/event/FuLi/deep-dive-sig-release-tim-pepper-vmware): Tim Pepper - open discussion of China ecosystem vendors’ needs + * Seattle + * [Intro](https://kccna18.sched.com/event/Grbb/intro-release-sig-tim-pepper-vmware-aishwarya-sundar-google) - Tim Pepper & Aish Sundar - intro, how to get involved, panel + * [Deep-dive](https://kccna18.sched.com/event/GrdX/deep-dive-release-sig-josh-berkus-red-hat-chuck-ha-heptio) - Chuck Ha on build tools. +* SIG Leadership + * (Stephen / Tim) Revisit Charter + * Formally define subprojects + * Release Team + * Release Engineering + * Patch Release Team - rotating slot on RT + * Meta KEP to define tooling + * Licensing & Compliance + * Product Security Team + * Time to rotate leads? + + + + + + + + + + +
+Date: + Sept 11, 2018 +
Attending: + +
    + +
  • \ +Aaron Crickenberger (@spiffxp) + +
  • Aish Sundar + +
  • Tim Fogarty + +
  • Justin Santa Barbara + +
  • Lubomir I. Ivanov + +
  • Arnaud Meukam + +
  • Morten + +
  • Mike Arpaia + +
  • Caleb Miles + +
  • Kendrick Coleman +
    • +
+
+ + + + +* Discuss 1.13 release team nominations ![ https://github.com/kubernetes/sig-release/issues/280#issuecomment-420059544](https://github.com/kubernetes/sig-release/issues/280#issuecomment-420059544) + * Possibly cap # of shadows to 4 per role. Specifically for Branch manager role, guidance is to not have >2 shadows. +* [https://github.com/kubernetes/test-infra/issues/9336](https://github.com/kubernetes/test-infra/issues/9336) - turning down misc-mungers and cherrypick-queue instances of mungegithub [spiffxp] + * Have reached out to existing patch release managers to verify I’ve surveyed their usage of cherry pick related labels and merging + * We need someone to adopt improving the cherrypick process, IIRC @nikhita and @guineversaenger have expressed interest +* [https://github.com/kubernetes/test-infra/pull/9342](https://github.com/kubernetes/test-infra/pull/9342) - moving all kubernetes repos over to tide [spiffxp] + + + + + + + + + + +
+Date: + Aug 28, 2018 +
Attending: + +
    + +
  • \ +name1 +
    • +
+
+ + + + +* Adding sig labels to PRs automatically based on OWNERS files is a thing, but too many OWNERS files don’t have “labels:” section. Could be good first issue for somebody looking to learn codebase layout and ownership:[ https://github.com/kubernetes/community/issues/1808#issuecomment-410309860](https://github.com/kubernetes/community/issues/1808#issuecomment-410309860) +* [TimP] Build-and-Publish-Infra-and-Automation: + * Is this a sub-part of test-infra? + * How do we work to pull this out more into community managed workflows and code maintenance? Eg:[ https://github.com/kubernetes/release/issues/591](https://github.com/kubernetes/release/issues/591) backs onto something building from a master branch instead of a release specific branch, but most of us don’t understand what/where this is or how to change it. + * Could we get to a workflow of build-validate-publish-validate, where we know all the expected artifacts, and have scripts to validate they’re created correctly, and were published correctly? +* + + + + + + + + + + +
+Date: + Aug 14, 2018 +
Attending: + +
    + +
  • \ +No meeting +
    • +
+
+ + + + + + + + + + + + +
Date: + July 31, 2018 +
Attending: + +
    + +
  • \ +Jaice Singer DuMars + +
  • Kendrick Coleman (1.12 Features Shadow) + +
  • Stephen Augustus + +
  • Aaron Crickenberger +
    • +
+
+ + + + +* SIG Release Charter - Needs to be revamped with new template + * Link to[ template](https://github.com/kubernetes/community/blob/master/committee-steering/governance/sig-charter-template.md) + * Volunteers needed: + * Stephen Augustus + * Tim Pepper + * Jaice Singer DuMars + * Aish Sundar +* Using discuss.kubernetes.io for release announcements - Jorge Castro + * Currently working on autoingesting k8s releases and changelogs, then autogenerating a post on discuss.k8s.io + * We can then post to Slack/twitter automatically + * Submission is currently human gated while we sort out the exact workflow and test usage cases + * We could remove the burden of handing off -announce list access to each new release lead + * Contribex (jorge) owes you a fully documented process +* Features process reboots ~ PTAL + * Why do we need to change this? + * Current process is too manual and error-prone + * There is no feature planning process framework + * Release artifacts outside of GitHub are not traceable or transparent + * KEPs do not tie cleanly to work delivered + * Jaice’s[ proposal](https://docs.google.com/document/d/113gnr3tKXv79J0IYHyg6VbQwLfb-sCYGphW4D9dLZ_E/edit#heading=h.wzakwvll8dsp) + * Stephen’s[ proposal](https://docs.google.com/document/d/169x1nuvBTOvZyU8tx7-aZycsF6HGzluqm41rCXd4Rl4/edit#heading=h.ajp15jfaoupa) +* Revisiting milestone-munger - spiffxp + * Yes, I’m proposing we stop using it entirely, I can’t tell how we’re measuring its success + * Or at least identify what parts it’s doing that are actually useful vs. noise + * [https://github.com/kubernetes/sig-release/issues/243](https://github.com/kubernetes/sig-release/issues/243) +* EOL policy for older release-1.y jobs - spiffxp + * [https://github.com/kubernetes/sig-release/issues/244](https://github.com/kubernetes/sig-release/issues/244) +* Docs and policy for Milestone-Maintainers membership: + * [https://github.com/kubernetes/sig-release/pull/239](https://github.com/kubernetes/sig-release/pull/239) + + +## Next meeting - TBD + +Potential topics (moved here from older[ SIG-Release minutes doc](https://docs.google.com/document/d/1Q6Ux-vsiRuiXhXSkVqGr9FUZifqVGRdCWkzVI1UT4l0)) + + + +* SIG-Release intro and deep dives at KubeCon China and North America +* Release process improvements: + * artifact provenance & signing + * release process threat model + * Ability for non-Google employees to be release branch maintainers + * More documentation details on what metadata feeds what machinery to generate what artifacts + * add test cases on artifact creation/publication so SIG-Cluster-Lifecycle's not pointing out that some part of the artifacts didn't build/publish correctly +* LTS releases +* Promotion of e2e test to conformance: when should this happen for features that are (or should be) going stable in the current release? Don’t want it late, because it could cause signal instability or other late surprises. Don’t want it too early in case the feature slips, but in that case could relegate from conformance back to e2e? + + +--- + + + + + + + + + + + +
Date: + June 5, 2018 +
Attending: + +
    + +
  • \ +Jaice Singer DuMars + +
  • Stephen Augustus + +
  • Paris Walters + +
  • Zach Arnold + +
  • Tim Pepper + +
  • Kendrick Coleman + +
  • Benjamin Elder + +
  • Aish Sundar +
    • +
+
+ + +**AGENDA**: + + + +* SIG Release Charter - Need volunteers + * Need to create a draft and get it ratified + * Volunteers: Stephen Augustus, Tim Pepper + * Open / hold PR in k/community for RFC + * Move Release Team to a subproject + * Overlap with SIG-PM and the new features[ planning](https://docs.google.com/document/d/1qi4LKV3W9B5JJ5JLjmAY33jESYAqWMWFyO5bWoYEBDo/edit)/[tracking](https://docs.google.com/document/d/113gnr3tKXv79J0IYHyg6VbQwLfb-sCYGphW4D9dLZ_E/edit#heading=h.wzakwvll8dsp) process + * Need everything in GH somehow + * SIGs SHOULD be planning release work more than one release ahead + * KEPs are great, but not fully implemented yet + * SIG PM Charter draft PR:[ https://github.com/kubernetes/community/pull/2068](https://github.com/kubernetes/community/pull/2068) +* Release artifact signing and validation + * Tim Pepper was going to look at this? + * Threat model: + * Not an existing model, but would like one + * Create and collaborate on a draft + * How do signed artifacts support cluster operator requirements and use cases? + * How do we connect with Google peeps working on this? At KubeCon EU Caleb mentioned work underway to remove the requirement that release branch managers be Googlers. Whatever automation improvements are underway there tie into the overall release threat model and any signing story. + * Notary /[ TUF](https://theupdateframework.github.io/) could be very helpful? + * +* Scalability pre-submits + * Stabilized from add a month ago + * Every PR is being scale tested + * Can these be made blocking? + * 500 node kubemark would become the slowest test at 1:20:00 + * Catch ~60% of regressions ahead of time + * +* Release cadence discussion + * 3x? 20x? LTS? + * 3 + 2 release idea? + * Need to figure out how to gate release content + * +* What release policies are missing? + * Documentation as a blocker + * To whom is this controversial? + * Alpha/Beta features should not be an “out” for docs + * if in_features == true ; docs_required + * label for this? + * Reverts (in greater detail) + * SIG vs. Release Team + * Unmaintained/able vs. policy-violating + * + * Revisit[ requirements](https://github.com/kubernetes/sig-release/blob/master/release-process-documentation/release-team-guides/release-lead.md#skills-and-experience-required) for the Release Lead + * + * Better define requirements for other Release Team roles + * [CI Signal lead role / tasks outlined a bit here](https://docs.google.com/document/d/1uDVrlzh9gZZukz-GYHWSyOnFW_2lplZ4JgR3D4tV4EA/edit) + * How are leads actually elected/selected? + * Nomination -> ratified by the subproject + * Tim Pepper! + * ========= Continue Next Meeting =========== + * Big features (k/features) vs others (k/k) + * Is there / should there be a “maximum term” for incumbents? + * Do leads need to be from a different company each time? + * How do we replace a release lead (or other position) if: + * They leave/quit/expire + * The SIG receives complaints, or the person is not performing the required duties + * Conformance test discussion / coverage as a blocker for features going to Stable each release + * PR conformance related tasks into a release role + * Identify few areas / features to increase conformance coverage per release [Aish] + + +--- + +Template: + + + + + + + + + + + +
Date: + +
Attending: + +
    + +
  • +
    • +
+
+ + +**AGENDA**: + + + +* Item 1 + + +--- + +**RECOVERED NOTES** + + +--- + + +## 27 February 2018 + +Agenda + + +## 16 January 2018 + +Agenda + + + +* @jdumars working on some process improvements + * sending out a weekly update to kubernetes-dev + * documenting all release roles + * would like to see more investment in strategic improvements beyond the release team +* Master upgrades on GCE/GKE need attention +* @calebamiles working to onboard as 1.10 branch manager +* Possibly looking for a replacement SIG colead for @pwittrock + * No objections from participants to replace @pwittrock with @jdumars +* Need to find earlier time to accommodate europe + * Possibly move to asynchronous meeting to replace biweekly Zoom meeting \ No newline at end of file diff --git a/sig-release/meeting-notes-archive/2019.md b/sig-release/meeting-notes-archive/2019.md new file mode 100644 index 000000000..d316425b1 --- /dev/null +++ b/sig-release/meeting-notes-archive/2019.md @@ -0,0 +1,1796 @@ +## December 16, 2019 (recording) + +**Host:** Chris Short? + +**Note Taker**: TBD + +**Attendees:** + +**Agenda:** + + + +* **FINISH 1.17 RETROSPECTIVE[ https://docs.google.com/document/d/1AtZ_81F3E4y_04Gx31mnG5w6AG3E0AXDOuLU7RcUIso/edit#bookmark=id.w6sgvtx220ar](https://docs.google.com/document/d/1AtZ_81F3E4y_04Gx31mnG5w6AG3E0AXDOuLU7RcUIso/edit#bookmark=id.w6sgvtx220ar)** + +## + December 2, 2019 (recording) + + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Stephen Augustus +* Sascha Grunert +* Taylor Dolezal +* Marko Mudrinić +* Daniel Mangum +* Ace Eldeib +* Hannes Hoerl +* Jorge Alarcon +* Kenny Coleman +* Tim Pepper + +Regrets: + + Josh Berkus + + Bob Killen + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Managers + * hyperkube + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Krel Integration Testing + * **Action Item**: setup repo to use for integration tests +* Krel Unit Testing Framework + * Use frameworks that are used across Kubernetes so that common patterns can be copied / implemented + * ginkgo/gomega is used in other Kubernetes places + * Circle back to this at Release Engineering meeting next week +* Whitebox vs Blackbox Testing:[ https://github.com/kubernetes/release/pull/942#issuecomment-558707516](https://github.com/kubernetes/release/pull/942#issuecomment-558707516) +* [https://docs.google.com/document/d/1M4670sP4PxDi1tRf1AkYGBDvahnKGZekrERneaBDVs0/edit#heading=h.ppwls1rczyo9](https://docs.google.com/document/d/1M4670sP4PxDi1tRf1AkYGBDvahnKGZekrERneaBDVs0/edit#heading=h.ppwls1rczyo9) +* Add your suggested topic here and move this to the line below [firstname lastname, email or @slackname] + +## + November 4, 2019 (recording) + + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Stephen Augustus +* Jorge Alarcon +* Taylor Dolezal +* Bart Smykla +* Josh Berkus +* Sascha Grunert +* Nikolaos Moraitis +* Daniel Mangum +* Savitha Raghunathan +* Tim Pepper +* Todd Sedano + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Managers + * hyperkube + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Add your suggested topic here and move this to the line below [firstname lastname, email or @slackname] +* WG LTS discussions update [tpepper, jberkus] + * [12-month KEP](https://docs.google.com/document/d/1uOQkN_B4SDepvPGEJ0FX89Q589zPqSItpWVAvdvELMU/edit?ts=5db7af27) + * Today we support a release for 9 months, but actually it’s 9mo’s plus 6-7weeks + * Discussion is 12 months, with formalized trailing period of 2 months with probably only CVEs during that period. + * Today one might upgrade in April, January, and October in order to keep aligned with supported patch releases. The proposal would allow one to pick a time period (eg: April) and always plan an annual major migration/upgrade at that time. + * What is the cost? Human and infra. + * What is the benefit? Survey showed people do want longer support. Would an additional 3 months help a significant portion of users? Survey data can be read multiple ways. + * How much are our community supported artifacts actually used? + * What portion of Kubernetes users get their bits from community versus a vendor? + * Vendor/distributors: + * Is there any support they expect from the community? + * What value do they see in our community releases and patch releases? + * Should the community make patch releases at all? + * Cherry-picking vs. accepting numbered patch release (e.g. Kernel) + * Would a vendor share this information with us? Is their code visible (eg: RedHat’s are, although Apache license does not require it) to see if they use community patch releases or manage their own cherry pick support branches. + * RedHat: likely no value in 3-month extension, because overall support is 4+ years. + * Pivotal: their customers would like this longer cycle + * CoreOS/Tectonic in the past would have benefited from this incremental breathing room + * Upgrade scenarios: + * Today are already fraught when doing in-place upgrades of cluster nodes across a large version skew. + * The KEP would make that worse. + * Community/industry seems to be moving toward preferring deploying new clusters at new versions, migrating the workload from an older cluster to a newer one. That is also non-trivial today. + * The KEP does not address any of this, but expects future work should be a focus so cluster operators can more easily deploy and roll over to newer versions. +* Release Notes quality [tpepper]: + * Does[ https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md](https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md) sufficiently describe when to add a release note and how to format it for maximum information sharing. + * SIG Release owns collating a changelog. Are we responsible for the content too and if so how do we push 2-3000 people towards creating better release-note stanzas in their PRs. + * Josh: it describes formatting, but not WHEN to add a release note. + * Issue open about that + * [https://github.com/kubernetes/community/issues/484](https://github.com/kubernetes/community/issues/484) + * Also need to describe what goes in the release note + * Suggestions: + * [https://chris.beams.io/posts/git-commit/](https://chris.beams.io/posts/git-commit/) in particular the sub-set of how to write a good one-line description short log summary + * “User-visible” changes: who is the user persona? + * Actionable, concise vs. detailed, thorough + * Take out the individuals’ feelings regarding what the submitter/reviewer feel is good, give concrete, objective criteria details on types of things that need release note and how the associated release note for that case should look + * Needs socialized everywhere...k/dev, contrib ex, community meeting, SIG leads, etc. + * Action: Sascha (and others on release notes team) to start looking at what could be done around[ https://github.com/kubernetes/community/issues/484](https://github.com/kubernetes/community/issues/484) docs updates +* Add kind/deprecation label [Stephen] + * PR:[ https://github.com/kubernetes/test-infra/pull/15040](https://github.com/kubernetes/test-infra/pull/15040) + * How to keep required labels easy for contributors? + * Need reviewer/approvers enforcing expected quality + * Feels like we need a simplified review checklist for reviewer/approvers with all the required things. + * Action: Tim Pepper will look through devel guide and discuss with SIG Contrib Ex if this is a gap they’d be interested in driving. + * Bot should include a link to review checklist when tagging reviewers automatically + * How to socialize these expectations and build common culture around them? There’s no global “reviewer/approver” mailing list. For some reason people don’t seem to read k-dev list for announcements. +* PR Template suggestion for “primary SIG”:[ https://github.com/kubernetes/kubernetes/pull/82449](https://github.com/kubernetes/kubernetes/pull/82449) [Stephen] + * Proposes adding a label + * Issues/PRs today can have multiple SIGs listed. But who is the primary contact point? + * Original motivation was to clarify things in the release notes, where today many major changes will be marked with multiple SIGs, so which SIG’s summary stanza would the release note go in? A reader interested in networking topics today might need to read the SIG Networking section and a bunch of others. Solving this would need some additional level of categorization. + * Example: the submariner PRs are about multicluster networking. Should those be limited to SIG Multi-Clusterr or SIG Network? + * Group feels the benefit here is minor, and also wouldn’t make more clear the situation of multiple SIGs who are actually involved on a PR/issue. Effort/benefit tradeoff is small. Today there is a lot of manual work the team does to decide in which category to best put a release note or if to duplicate it and that would still be needed. + * Are people finding the markdown file or the release notes web site more useful? + * Survey results split. + * If the interactive site is useful, people can pick their individual preferred categorizations on the fly + * If we did not split on SIG in the markdown file, what would the best categorization be? + * Is splitting by SIG actually useful to people who aren’t closely familiar with project internals? “What is a SIG” remains our number one question. SIG Networking may be obvious, but what’s the difference between say SIG UI and SIG UX? + * Would splitting by Kind make sense? Or Priority (questionable value today, few use this label)? Or Area? + * Action: Sascha Grunert to demo what it might look like split on Kind. +* [justaugustus] Help me close some PRs! + * [https://github.com/kubernetes/kubernetes/pull/84662](https://github.com/kubernetes/kubernetes/pull/84662) + * [https://github.com/kubernetes/sig-release/pull/839](https://github.com/kubernetes/sig-release/pull/839) + * [https://github.com/kubernetes/test-infra/pull/15040](https://github.com/kubernetes/test-infra/pull/15040) + * [https://github.com/kubernetes/test-infra/pull/15083](https://github.com/kubernetes/test-infra/pull/15083) + * [https://github.com/kubernetes/test-infra/pull/15068](https://github.com/kubernetes/test-infra/pull/15068) + * [https://github.com/kubernetes/release/pull/921](https://github.com/kubernetes/release/pull/921) + * [https://github.com/kubernetes/sig-release/pull/847](https://github.com/kubernetes/sig-release/pull/847) + +## + Oct. 21, 2019 (recording) + + +**Host:** Tim Pepper + +**Note Taker**: TBD + +**Attendees:** + + + +* Tim Pepper +* Bart Smykla +* Sascha Grunert +* Jorge Alarcon +* Kenny Coleman +* Marko Mudrinić +* Daniel Mangum +* Stephen Augustus +* Todd Sedano +* Maria Ntalla +* Add your names here + +**Recurring Topics [timebox to N min]:** + + + +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Managers + * hyperkube + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * Oct. 18/19 networking failure, possible fix in[ https://github.com/kubernetes/kubernetes/pull/84159](https://github.com/kubernetes/kubernetes/pull/84159) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * Test-infra freeze: + * idea disliked by SIG Testing, they don’t want to pause Prow development and it has other users + * We’re not looking for Prow to stop, simply the variation in Prow as used by just Kubernetes to slow down at times. + * [https://github.com/kubernetes/test-infra/commits/master/config/prow/config.yaml](https://github.com/kubernetes/test-infra/commits/master/config/prow/config.yaml) bot does a daily update of the prow commit in usage by k8s project. What is the quality gate ahead of that bot updating? If there’s good initial gating, good test coverage, signal triage and fix, perhaps a freeze is not necessary. + * We’re asking for folks to be more deliberate about when and where new things are rolled out so we’re not solely testing in production and reacting to issues. + * + +**Open Discussion [timebox to N min]:** + + + +* Add your suggested topic here and move this to the line below [firstname lastname, email or @slackname] + +## + Oct. 7, 2019 (recording) + + +**Host:** TBD + +**Note Taker**: TBD + +**Attendees:** + + + +* Tim Pepper +* Carlos Panato +* Daniel Mangum +* Dims +* Ihor Dvoretskyi +* Jeremy Rickard +* Max Körbächer +* Lubomir Ivanov +* Marko Mudrinic +* Nikolaos Moraitis +* Savitha Raghunathan +* Seth McCombs +* Stephen Augustus +* Jorge Alarcon +* Eddie Villalba +* Nabarun Pal +* Marky Jackson +* Bob Killen +* Jordan Liggitt +* Linus Arver (@listx, Google) +* Adam Kaplan (@adambkaplan, Red Hat) +* Bart Smykla +* + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Managers + * 1.13.12, 1.14.8, 1.15.5, and 1.16.3[ scheduled](https://github.com/kubernetes/sig-release/blob/master/releases/patch-releases.md) for Tues. Oct. 15. + * hyperkube + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Should artifacts published by publishing-bot be release-blocking? [Aaron, Nikhita] + * Came from 1.16 retrospective + * Reach out to Aaron/Nikhita asynchronously regarding this in[ https://github.com/kubernetes/sig-release/issues/806](https://github.com/kubernetes/sig-release/issues/806) +* Publish instructions on how to consume alpha, beta, rc artifacts _today_. We could write up a TL;DR. + * Stephen did an informal poll on Twitter. Lots of people didn’t know we make alpha/beta/rc releases today. + * Need to define the artifact list. + * There are things we can’t do today without release channels, but it will get better once we have release channels for daily, testing, prod builds. + * Doc on kubeadm w/ pre-releases:[ https://github.com/kubernetes/kubeadm/blob/master/docs/testing-pre-releases.md](https://github.com/kubernetes/kubeadm/blob/master/docs/testing-pre-releases.md) + * Buckets + * Release:[ https://gcsweb.k8s.io/gcs/kubernetes-release/release/](https://gcsweb.k8s.io/gcs/kubernetes-release/release/) + * CI:[ https://gcsweb.k8s.io/gcs/kubernetes-release-dev/ci/](https://gcsweb.k8s.io/gcs/kubernetes-release-dev/ci/) + * Carlos to work with Marky on one-pager summarizing basic workflow of getting/running alpha/beta/rc, aim for PR’ing into contributor or developer guide. +* Definitions of “priority/**”: + * Multiple places with a definition today: + * k/* on GitHub have labels defined at:[ https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.yaml#L362](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.yaml#L362) + * priority/awaiting-more-evidence: not used much at all + * priority/backlog: we thought it was used minimally, but… + * [1.16 PRs merged marked backlog](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Amerged+is%3Apr+label%3Apriority%2Fbacklog+milestone%3Av1.16+): 120 + * priority/critical-urgent: not too hard for issue triage on release team to manage this volume of issues: + * [1.16 PRs merged marked critical-urgent](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Amerged+is%3Apr+label%3Apriority%2Fcritical-urgent+milestone%3Av1.16+): 54 + * priority/important-longterm + * [1.16 PRs merged marked important longterm](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Amerged+is%3Apr+label%3Apriority%2Fimportant-longterm+milestone%3Av1.16+): 107 + * priority/important-soon + * [1.16 PRs merged marked important soon](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Amerged+is%3Apr+label%3Apriority%2Fimportant-soon+milestone%3Av1.16+): 411 + * needs-priority: frequently used... + * [1.16 PRs merged w/o priority](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Amerged+is%3Apr+label%3Aneeds-priority+milestone%3Av1.16) label: 259 + * [1.16 PRs merged w/ priority](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Amerged+is%3Apr+label%3Aneeds-priority+milestone%3Av1.16) label: 692 + * k/community: + * [contributors/devel/sig-release/release.md](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/release.md) + * [contributors/devel/sig-release/cherry-picks.md](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md) + * [contributors/guide/issue-triage.md](https://github.com/kubernetes/community/blob/master/contributors/guide/issue-triage.md) + * k/test-infra: + * [label_sync/labels.yaml](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.yaml#L368) leads to mousehover tooltip on GitHub + * [https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md](https://github.com/kubernetes/test-infra/blob/master/label_sync/labels.md) generated from the yaml + * k/sig-release: + * [release-team/role-handbooks/ci-signal/README.md](https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/ci-signal#priority-labels) + * [release-team/role-handbooks/bug-triage/README.md](https://github.com/kubernetes/sig-release/blob/master/release-team/role-handbooks/bug-triage/README.md) + * Example search on hound:[ https://cs.k8s.io/?q=priority%2Fcritical-urgent&i=nope&files=&repos=](https://cs.k8s.io/?q=priority%2Fcritical-urgent&i=nope&files=&repos=) + * We’ve conflated severity of issue and urgency in fixing issue and also indicating activity on resolving issue (lifecycle/* labels) + * Multiple use cases: + * Release team uses them for priorities + * Patch release team uses them for assessing urgency, but only minimally and practically ignores them. They’re a very low quality issue. + * Individual SIGs use them for triaging and stack ranking today’s work relative to a milestone. + * Implementation varies from SIG to SIG and contributor to contributor...do we need to drive out of SIG Release a cross-project KEP for better contributor experience? + * Release team struggles to answer is a given item “priority/important-soon” really important? Or backlog (which should be covered by “priority/important longterm” + * May mean something different in master branch versus patch release branches + * Is priority varying based on time-to-release? + * how are items promoted or demoted in priority in general? + * how are items promoted or demoted from current milestone to no milestone or to next milestone? [Marko Mudrinić, Niko Pen, Stephen Augustus, Maria Ntalla] + * What are we trying to solve if we change these? + * Maybe nothing specifically and maybe not actually change anything...just focus on making the multiple places of documentation consistent. + * Volunteer to clean the existing documentation to be consistent? Make one document describing what we have today. Link it from the other places that currently describe things in different ways. @alejandrox1 will look into this. + * [https://cs.k8s.io/?q=priority%2Fcritical-urgent&i=nope&files=&repos=](https://cs.k8s.io/?q=priority%2Fcritical-urgent&i=nope&files=&repos=) +* Discuss stability releases [Stephen] + * Tracking issue:[ https://github.com/kubernetes/sig-release/issues/809](https://github.com/kubernetes/sig-release/issues/809) + * What is stability? + * Need to improve the CI signal + * KEPs are supposed to come with test cases. If they include a couple links to the testgrid place that show their green-ness or red-ness, that would be hugely valuable to the release team. Most KEPs start out with just an intent noted for adding tests. Could pick a tag name for the e2e test and pre-compose a (initially broken) link to the test grid that will show their healthy test eventually. + * Update KEP template. Add feedback here:[ https://github.com/kubernetes/enhancements/issues/822](https://github.com/kubernetes/enhancements/issues/822) +* GitHub issue for Retro AIs (action items):[ https://github.com/kubernetes/sig-release/issues/806](https://github.com/kubernetes/sig-release/issues/806) +* Add your suggested topic here and move this to the line below [firstname lastname, email or @slackname] +* + +--- + + +## + disSeptember 23, 2019 ([recording](https://youtu.be/K2CZAxfeVIU)) + + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Add your names here +* Todd Sedano +* Savitha Raghunathan +* Bart Smykla +* Marko Mudrinić +* Jorge Alarcon +* Bob Killen +* Craig Peters +* Jeffrey Sica +* Nikolaos Moraitis +* Tim Pepper + +**New topic proposals:** + + + +* (Craig Peters) 1.16 Release Retro, part two: + * See retro doc for full details:[ https://bit.ly/k8s116-retro](https://bit.ly/k8s116-retro) + +## + September 4, 2019 ([recording](https://youtu.be/uztvn3NJu5U)) + + +**Host:** Stephen Augustus + +**Note Taker**: Guinevere Saenger + +**Attendees:** + + + +* Add your names here +* Josh Berkus +* Hannes Hörl +* Jorge Alarcon +* Marko Mudrinić +* Nicholas Lane +* Lachlan Evenson +* Guinevere Saenger +* Yang Li +* Jim Angel +* Josiah Bjorgaard +* Tim Pepper + +**Open Discussion [timebox to N min]:** + + + +* [Blocking/Informing Jobs discussion and documentatio](https://github.com/kubernetes/sig-release/pull/752)n + * How do we set criteria for flakiness? + * Tracking issue:[ https://github.com/kubernetes/sig-release/issues/773](https://github.com/kubernetes/sig-release/issues/773) + * “Does the test job reliably pass when it ought to pass” => this is not something that we can reliably check. We need to come up with a new criterion that can be used by our infra so we can instantly see. + * Tie this to existing test-infra UI: e.g. must not be on the flaky test board. TODO: everyone go comment on the above-linked issue + * What's the procedure for adding new jobs to -informing and documenting them? What about Blocking? + * Tracking issue:[ https://github.com/kubernetes/sig-release/issues/774](https://github.com/kubernetes/sig-release/issues/774) + * we do not have a good process to determine which jobs should get added. This is an issue in the 1.16 release with the new windows jobs. We have neither process nor documentation; past process has been rather ad-hoc + * one suggestion is to pass job responsibility to the SIGs to their individual testgrid boards; this will need cooperation from SIGs but SIGs may feel more responsible and hence faster to react. + * concern: would this mean more boards for CI Signal to watch? + * Jorge’s comment: nothing should change for CI signal. SIGs will communicate with release team / SIG Release about what goes on their dashboards. + * we want SIGs to be able to say “these jobs are absolutely necessary to pass” vs “these jobs are under construction” + * we have release-informing and release blocking jobs already + * we should focus on reducing our burden but not create process for other SIGs + * Jorge’s comment: We propose to ask SIGs to look at their dashboards in a similar manner as CI sigal does for better communication. + * we do want a way to be able to communicate back and forth between CI Signal and the SIGs’ jobs + * CI Signal has been reacting to Jobs that _do not_ alert SIGs, which is a misalignment/miscommunication. + * TODO: Please send these as comments on the above-linked issue. + * recently a SIG-cli job was merged but did not have a SIG-cli alert on it: make sure the correct people are responsible and aware + * master-informing, after discussion with SIG-Release, can be done fairly easily. master-blocking should be much more strict and definitely be approved by SIG-Release; perhaps a specific issue template that outlines criteria and check boxes. That way it’s under SIG-Release OWNERS file. + * We also need test-infra to have some enforcement so a test can’t just be added via a kubernetes annotation on the Job. + * Prerequisites: + * CI Signal Lead should need to sign off (coordinate with rel team lead) + * SIG-Release chairs also needs to sign off + * [How should the Release Team decide that Informing failures are tolerated/ignorable?](https://github.com/kubernetes/sig-release/issues/775) + * this is more of a release team process issue; we should document this. + * THANK YOU to the CI-Signal team and Josh for putting all this work into it so we can work on it! +* What would need to be true so that we are confident to remove[ the blockage](https://github.com/kubernetes/test-infra/blob/824921ffbcdda2b8fbfd88002dd305f98c2da190/prow/plugins.yaml#L165-L169) on k/release? [Hannes Hoerl, hhorl@pivotal.io @hoegaarden] + * Issue:[ https://github.com/kubernetes/release/issues/816](https://github.com/kubernetes/release/issues/816) + * this surfaced during cleanup of the push-build script + * there’s a few ci-signal build jobs that push k8s up to gcp and they got broken, bringing down kubernetes + * blockade exists to prevent this from happening + * what does it take to prevent this from happening: we need to feel comfortable making safe changes; one idea is to configure jobs to a good tag of kubernetes + * we should sync k/release with this + * for discussion in next meeting: what can we do with tags and branches? +* WG LTS requesting feedback on[ draft KEP](https://docs.google.com/document/d/1uOQkN_B4SDepvPGEJ0FX89Q589zPqSItpWVAvdvELMU/edit#): + * Staffing impact of supporting an additional release branch (ie: 3-4 additional months) + * Patch release inputs: Minor costs for 4 instead of 3 branches for cherry picks since most picks land on all branches in practice today (have data on hundreds of patches showing this) + * Patch release outputs: building is straight forward, mostly automated, minor additional cost. Packaging is currently a long pole (depends on small set of Googlers still), so becomes a longer pole even...could consume about a full day of one person’s time to get out a set of 4 branches’ patch release packages. + * Incremental cost of CI on an additional release branch (also asking SIG Testing) for a cycle (ie: 3-4 additional months) + * Unknown, would need numbers from Google or guestimate / mock + * Formalization of process for removing “oldest” branch’s CI (eg: a “Remove-CIPresubmit-jobs” section like[ https://github.com/kubernetes/sig-release/blob/master/release-engineering/role-handbooks/branch-manager.md#Create-CIPresubmit-jobs](https://github.com/kubernetes/sig-release/blob/master/release-engineering/role-handbooks/branch-manager.md#Create-CIPresubmit-jobs)). + * Has moved out of release team into branch manager role, effectively an arbitrary date today (when beta is cut and new branch is created) but could be a more reasoned date that is further out in time (ie: +2 months) + * Better definition of: + * minimum period after new release N where the oldest release can still accept critical fixes. Practically this is ~1 month today, this proposal suggests 2 months. + * policy for patch merges during this special 2 month window, eg: critical security fixes (ie: CVE assigned by Product Security Committee and cherry pick of fixes initiated to release branch by Product Security Committee). + * Formalize 3rd party dependency update policy (eg: golang, but also others too) + * Patch release updates: Need to track upstream golang patch releases continually. + * Major release updates: Can we also formalize when to move to a new Golang major release. This is needed as two of our annual releases today (and all of them if we move to a yearly lifecycle) run beyond the end of Golang’s patch release lifecycle. + * Suggestion from Stephen Augustus: maybe go for 15 months + * Concerns on additional burdens: + * staffing impact for additional release branch (patch release team) + * vast majority of cherry picks go onto all branches + * we need to chip away at removing the Google curtain: updates to apt and yum + * Stability concerns for APIs and migration issues + * This would be a policy change with a large group of stakeholders + * Discussion to continue at kubecon / contributor summit (WG LTS has proposed a session for the contrib summit; if not accepted on the fixed schedule, it will also be proposed there at the un-conference) +* KubeCon NA 2019 SIG Release Intro session proposed: + * [https://docs.google.com/document/d/1kcKRA18qQ1MPJD0jtir1hzRASvctATuIrS1lClxXz10/edit?usp=sharing](https://docs.google.com/document/d/1kcKRA18qQ1MPJD0jtir1hzRASvctATuIrS1lClxXz10/edit?usp=sharing) +* KubeCon NA 2019 SIG Release Deep Dive session proposed: + * [https://docs.google.com/document/d/1Bv8mc_zzNWPNXrziQfQFCkA6I6z6Rux08SDnKE5KUNM/edit?usp=sharing](https://docs.google.com/document/d/1Bv8mc_zzNWPNXrziQfQFCkA6I6z6Rux08SDnKE5KUNM/edit?usp=sharing) +* Contributor Summit (day before KubeCon) session[ proposals](https://docs.google.com/forms/d/e/1FAIpQLSek9mwSra2AMaoPPCMlkyTOkVFocqBz_6u3BmGv7jqJcX2gYA/viewform): + * Due by Sept. 9 - verify deadline(?) + * ...what do we have needing f2f discussion? + * Alternatively can propose un-conference on the spot there +* + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Managers + * hyperkube + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +## + August 12, 2019 ([recording](https://www.youtube.com/watch?v=HycpkheG7hU&list=PL69nYSiGNLP3QKkOsDsO6A0Y1rhgP84iZ&index=12&t=0s)) + + +**Host:** Tim Pepper + +**Note Taker**: Josh Berkus + +**Attendees:** + + + +* Bob Killen (Enhancements Shadow) +* Hannes Hoerl (patchReleaseTeam) +* Nikhita Raghunath +* Kenny Coleman (enhancement lead) +* Jeffrey Sica (release team) +* Sascha Grunert (release team) +* Josh Berkus (release team, etc.) +* Tommy Chong +* Claire Laurence +* Savitha Raghunathan +* Bart Smykla +* Angela Lewis (New attendee,yay!) +* Carlos Panato (Release Manager Associate) +* Imran Pochi (Communications Team shadow) +* Ace Eldeib (Release Manager Associate) + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Hyperkube (probably won't be doing status in the future) + * Licensing + * really need to have meetings; having trouble finding a time + * Josh votes to have licensing status updates in release meetings + * Publishing-bot (after today will report status through release engineering status bucket) + * Currently removing 1.12 stuff, and adding 1.16:[ https://github.com/kubernetes/kubernetes/pull/81287](https://github.com/kubernetes/kubernetes/pull/81287) + * [Moved to CNCF infrastructure](https://github.com/kubernetes/publishing-bot/pull/198) + * Bot is happy + * Release Engineering + * First subproject meeting was last week: agenda/minutes doc is:[ https://docs.google.com/document/d/16GqCjnEh86w8yADcrUylNoE1y1sqjIMYNC_gdk5WPSQ/edit#](https://docs.google.com/document/d/16GqCjnEh86w8yADcrUylNoE1y1sqjIMYNC_gdk5WPSQ/edit#) + * Been working on some high-level ideas + * Now working on implementing a lot of those; testing them in the 1.16 cycle. + * First -- continuing to move stuff to places where non-Googlers can access them + * Second -- publish to CNCF infra + * Just publishing one Deb repo and one Yum repo, with all k8s versions in one spot. Need to switch to one repo per k8s version. + * Then can also split each of those into nightly, testing (eg: alpha/beta/RC), stable with a promotion flow for artifacts. + * Do the CICD pipeline thing + * Release Team + * 1.12 jobs leaving test-infra + * 1.16 jobs to start with branch creation this week + * A few enhancement exceptions are coming through + * 1.16 branch cut this week + * Next week burndown begins + * Code freeze at end of month + * CI is reddish but we should be cleaning it up +* Testgrid dashboard review ([https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release)) + * Went over test boards + * sig-release-misc: green + * sig-release-orphaned + * currently all upgrade/downgrade tests are orphaned, except Kind + * Kind test is release informing (see[ https://github.com/kubernetes/kubeadm/issues/1599](https://github.com/kubernetes/kubeadm/issues/1599) regarding moving to blocking) + * need to have upgrade/downgrade tests that we trust + * some SIG-Cluster-Lifecycle tests might be OK to promote to at least informing for 1.16 (and blocking for 1.17?) -- Josh to follow-up + * SIG-Cluster-Lifecycle will not do downgrade coverage + * Hopefully cluster API and kubeadm work enable us to also get signal on providers beyond GCE (and Kind) +* [Project board review](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* n/a + +## + July 30, 2019 ([https://youtu.be/I38zmgbpdwE](https://youtu.be/I38zmgbpdwE)) + + +**Host:** Stephen Augustus + +**Note Taker**: Tim Pepper + +**Attendees:** + + + +* Stephen Augustus +* Lachlan Evenson +* Mike Crute +* Josh Berkus +* Tim Pepper +* Caleb Miles +* Aaron Crickenberger +* Ben Elder +* Dhawal Yogesh Bhanushali +* Kendrick Coleman +* Khosrow Moossavi +* Michael Singh +* Savitha Raghunathan +* Khosrow Moossavi + +**New agenda items** + + + +* Dhawal Yogesh Bhanushali: KEP draft has started in WG LTS to propose changes to K8s release support lifecycle and upgrade path. + * [https://docs.google.com/document/d/1uOQkN_B4SDepvPGEJ0FX89Q589zPqSItpWVAvdvELMU/edit#heading=h.ul9lgso7jzs6](https://docs.google.com/document/d/1uOQkN_B4SDepvPGEJ0FX89Q589zPqSItpWVAvdvELMU/edit#heading=h.ul9lgso7jzs6) + * Proposes move from 9-ish months support on each of 3 releases to 12-ish months support on each of 4 releases + * Policy has been specific around 9 months. But “-ish” on 9 and 12 months because today the patch release team actually is giving more like 10-11 months on the trailing most release instead of just 9. In aiming to give a year of support it could similarly be one year plus a bit of overlap. + * There’s been discussion of more explicitly allowing critical CVE fixes after the 9mo’s. Don’t have data to show that’s ever actually happened though. + * Today technically we could support an older release branch relatively easily and with some confidence so long as CI is still available on the branch. Once CI goes away it’s much riskier. + * This is not a large delta on what is done today, but is surely more palatable if more diverse companies are more active in the patch release team showing that the trailing support is sustainable + * N-2 upgrades and deprecation cycles: + * Multiple folks express more skepticism here in that this is a more radical shift in what is done today and would require much more buy-in broadly in the community (Steering, Architecture, etc.) + * Why is this a blocking pre-requisite on the other aspect of expanding the support timeline a few months? Dhawal suggests this is intended to be complementary, because an end user who is lagging behind struggles to upgrade. + * [spiffxp] Upgrade being hard upstream means it’s also hard downstream. + * [Caleb] But downstream productization teams, if they are having notable friction on upgrade, should be staffing reducing that friction. If they’re looking for easier qualification, why don’t we see them contributing to easier qualification in the community? + * [Ben] it’s not just people costs where companies need to show up and contribute, but also just on moving to a longer lifecycle does incur real infra dollars costs + * [jberkus] we don’t have confidence in our N-1 tests today and that’ a pressing need to resolve before expanding it to N-2. +* Stephen Augustus:[ Doodle](https://doodle.com/poll/z5i65er4hymr3s4h) for changing SIG Release meeting time to be more internationally friendly, also switching off weeks with Release Engineering Subproject. + * Maybe 10am Pacific: SIG Release one week and then the other week Release Engineering Subproject. + * SIG Testing has **_just_** decided to move to that slot. + * A doodle should be in folks inboxes later today. + * Notice sent to MLs:[ https://groups.google.com/d/topic/kubernetes-sig-release/jw2UV-fKjEY/discussion](https://groups.google.com/d/topic/kubernetes-sig-release/jw2UV-fKjEY/discussion) + * Tracking issue:[ https://github.com/kubernetes/sig-release/issues/411](https://github.com/kubernetes/sig-release/issues/411) +* (if time) Aaron Crickenberger:[ moving sigs.k8s.io/kind CI into release-blocking](https://github.com/kubernetes/sig-release/issues/724) + * Kind is cool + * Kind is quick + * Kind is reliable on conformance tests + * Especially for IPv6 gives first ability for A/B network test on k8s versus underlying network infra + * Kinder’s doing some upgrade testing in release-informing board. + * Kind could come into pre-submits in the near term maybe. + * It is finding real bugs today. + * Group’s liking the proposal...make it so. + +**Standing agenda items** + + + +* Subproject status + * Hyperkube: should this code just be clumped under release-engineering? + * Do we need to be tracking its status? It’s owned by SIG Release. Connected to rel eng work would more people have awareness of its state and health and grow to be contributors on it when needed? + * Hyperkube is used for Azure and Talos and a couple of things, and doesn't need much/any actual maintenance + * Licensing + * Nikhita’s taking point on this with help from Dims and Steve Winslow from LF + * It is 3am her time...need a meeting she can reasonably attend + * Publishing-bot: should this code just be clumped under release-engineering? + * This will go away eventually once the monolith splits, but this is big and complicated. + * This subproject is staffed reasonably sustainably. Need a meeting Nikhita can attend. + * Is it really part of the release process, versus an artifact of code organization? + * Release-engineering + * OWNERS for k/k:[ https://github.com/kubernetes/kubernetes/blob/master/OWNERS_ALIASES#L125-L140](https://github.com/kubernetes/kubernetes/blob/master/OWNERS_ALIASES#L125-L140) + * beta/stable1/stable2/stable3: confusion remains. Do we need a more visible location for this info vs[ https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md](https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md) (recently moved out of the top level test-infra/README.md) + * Release Team + +**Issue backlog scrub** ([https://github.com/kubernetes/sig-release/issues](https://github.com/kubernetes/sig-release/issues)) + + + +* SIG Release project board:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Release Engineering project board:[ https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30) +* Release Team project board:[ https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29) +* Licensing project board:[ https://github.com/orgs/kubernetes/projects/31 \ +](https://github.com/orgs/kubernetes/projects/31) + + + + + + + + + + + + + + +
+Date: + July 16, 2019 +
Attending: + +
    + +
  • \ +Jeffrey Sica + +
  • Bob Killen + +
  • Amy Chen + +
  • Paul Bouwer + +
  • Tim Pepper + +
  • Josh Berkus + +
  • Lubomir I. Ivanov + +
  • Dhawal Yogesh Bhanushali + +
  • Stephen Augustus + +
  • Arnaud Meukam +
    • +
+
Note taker: + +
    + +
  • +
    • +
+
+ + + + +* \ +[BenTheElder][ kind](https://testgrid.k8s.io/conformance-kind) as release blocking + * Particularly IPv6 + * Be will open issue in k/sig-release + * (see also below in notes discussion of release blocking criteria) +* [Stephen Augustus] Migrate orphaned release jobs + * [https://github.com/kubernetes/test-infra/pull/13468](https://github.com/kubernetes/test-infra/pull/13468) + * No owner of test means not a release blocking test. +* [Stephen Augustus] Blockade, release related PR merge status + * A bunch of small improvement patches merged and it broke stuff + * Turns out small changes to k/release magically can break blocking CI!!! + * Patches reverted. + * Issue:[ https://github.com/kubernetes/release/issues/816](https://github.com/kubernetes/release/issues/816) + * Prow has path-based regex for blocking patches. Until we get better on test coverage, we need to have clear SIG Release / releng subproject review of patches to k/release that might cause an issue. Prow will add “do-not-merge/blocked-paths” and block PR merge until that label is removed. + * k/release repo needs tagging so it is clear which code was used to build which k8s releases, or what are known good points in the repo + * CI can be pointed at these tags, instead of possibly unsafe intermediate states + * [Josh Berkus] should we have a blockade on k/release during code freeze? + * The more of k/release that is teased into k/k/build, the more that is implicitly versioned and frozen/thawed in sync with k/k. + * More on that release engineering cleanup brainstorm in[ https://docs.google.com/document/d/1Js_36K51Q6AjEsVUjRBMISTA4D7cnjZmoSkn43_Jmxo/edit#heading=h.y81m7zlfl31g](https://docs.google.com/document/d/1Js_36K51Q6AjEsVUjRBMISTA4D7cnjZmoSkn43_Jmxo/edit#heading=h.y81m7zlfl31g) + * [Ben Elder] push-build is a key tool that comes from k/release +* Release Engineering Subproject: + * What are we building/releasing? Bart Smykla did a deep dive into making a manifest document of all our artifacts: + * [https://github.com/kubernetes/sig-release/blob/master/release-engineering/artifacts.md](https://github.com/kubernetes/sig-release/blob/master/release-engineering/artifacts.md) + * The better we have this documented, the easier it is to make a change and show the change still outputs the same outputs as the prior tooling (and write validators/tests for each output) + * WG K8s Infra gave us GCP projects for PoC test building and “releasing” + * “Omni” KEP and GH issues in the works to spell out tasks to improve the build/release pipe and tools +* [Josh Berkus] Implementing[ blocking criteria](https://github.com/kubernetes/sig-release/issues/347) + * This has gone from proposal to effectively accepted policy, but...is informal + * Time to accept, etc? What more needs to be done? + * Aside from CI-Signal handbook, where would docs on this go? + * [https://github.com/kubernetes/community/tree/master/contributors/devel/sig-release](https://github.com/kubernetes/community/tree/master/contributors/devel/sig-release) + * Josh will write a document and PR it +* Misc: + * Paul Bouwer: is new 1.16 release team shadow member helping with release notes + * Release Team happenings: + * “Branch manager” role is a part of “release managers” group now. + * The idea is we build a peer team relationship between the next major release’s branch management and the patch release streams’ branch management. The part release team is embargoed and deals with security patches. + * We need a pipeline of volunteers who have demonstrated their ability and trustworthiness, eg: branch management during a release. These are “associates”:[ https://github.com/kubernetes/sig-release/blob/master/release-managers.md#associates](https://github.com/kubernetes/sig-release/blob/master/release-managers.md#associates) + * Need to do more formalization on what are the time bounds in apprenticing, what the path / ladder looks like in detail, what is the next step a person grows up into + * + + + + + + + + + + + + + + +
+Date: + July 2, 2019 +
Attending: + +
    + +
  • \ +Stephen Augustus (SIG Release Chair) + +
  • Christine P. + +
  • Jeffrey Sica + +
  • Claire Laurence + +
  • Lachlan Evenson + +
  • Peter Swica + +
  • Seth McCombs + +
  • Taylor Dolezal +
    • +
+
Note taker: + +
    + +
  • +
    • +
+
+ + + + 1.15 Retro: + + + +* Circle back to making sure discussion points are documented in issues/PRs with owners +* Pick up with remainder of retro:[ https://docs.google.com/document/d/1Re80f4qICEKLEhOEIFuzr0IZTp2es82urMi45PrScLM/edit#bookmark=kix.occneu8jg1e5](https://docs.google.com/document/d/1Re80f4qICEKLEhOEIFuzr0IZTp2es82urMi45PrScLM/edit#bookmark=kix.occneu8jg1e5) + + SIG Scalability and Release interactions: + +* Informing tests that are going to block need to actually be on the blocking board. This means those tests need to meet the criteria for the blocking board. +* Scale tests need to be more easy to spin up on an ephemeral context to validate some suspicion. +* Major/large scale tests likely need to happen not just on master +* SIG Scalability[ wants revert priv’s](https://groups.google.com/d/topic/kubernetes-sig-release/Qz7sVbMYu2c/discussion) ; SIG Release needs a stable release with scale quality. Some other SIG(s) want a feature or fix or enhancement in the code base. A simple revert doesn’t satisfy more than 2 of these 3 stakeholders. Who owns driving things to a place where all 3 are satisfied? +* SIG Scalability has indicated there are changes coming to help, but no record of what this is can be found in their meeting agendas (one in 2019?) or youtube recordings (none since 2017). + + + + + + + + + + + + + + +
+Date: + June 18, 2019 +
Attending: + +
    + +
  • \ +Josh Berkus + +
  • Savitha Raghunathan + +
  • Christian Jantz + +
  • Tim pepper + +
  • Josh Hoak + +
  • Stephen Augustus + +
  • Jeff Sica + +
  • Rael Garcia + +
  • Lachlan Evenson + +
  • Christine Pukropski + +
  • Arnaud Meukam + +
  • Nii Nai + +
  • Amy Chen + +
  • Peter Swica + +
  • Benjamin Elder + +
  • Aaron Crickenberger + +
  • Nikhil Manchanda +
    • +
+
Note taker: + +
    + +
  • \ +Everyone, right? :) +
    • +
+
+ + + + +* \ +[Stephen] 1.15 Release Delay + * [https://github.com/kubernetes/kubernetes/issues/79096](https://github.com/kubernetes/kubernetes/issues/79096) + * Waiting on SIG Scalability, should hear back tomorrow morning +* Proposal to remove test infra role from release team + * See[ https://github.com/kubernetes/sig-release/issues/631](https://github.com/kubernetes/sig-release/issues/631) + * Current role description:[ https://git.k8s.io/sig-release/release-team/role-handbooks/test-infra](https://git.k8s.io/sig-release/release-team/role-handbooks/test-infra) + * Item to be discussed in 1.15 retrospective[ https://docs.google.com/document/d/1Re80f4qICEKLEhOEIFuzr0IZTp2es82urMi45PrScLM/edit](https://docs.google.com/document/d/1Re80f4qICEKLEhOEIFuzr0IZTp2es82urMi45PrScLM/edit) + * Any concerns from SIG Release? + * Stephen is +1 + * Tim Pepper worried that we might be transferring responsibility to a smaller set of people + * Aaron will do everything +* 1.15 retrospective ([https://docs.google.com/document/d/1Re80f4qICEKLEhOEIFuzr0IZTp2es82urMi45PrScLM/edit](https://docs.google.com/document/d/1Re80f4qICEKLEhOEIFuzr0IZTp2es82urMi45PrScLM/edit)) tentatively scheduled for June 20 in Community Meeting[ http://bit.ly/k8scommunity](http://bit.ly/k8scommunity). + * Broken over two meetings + * first the community meeting (this thursday) + * Second in sig-release + * Tim: brought up discussion of whether to have it not + * Delay of 1.15 as a retro item + * sig-scalability gating to the release discussion as a separate discussion (is happening in thread here:[ https://groups.google.com/d/topic/kubernetes-sig-release/Qz7sVbMYu2c/discussion](https://groups.google.com/d/topic/kubernetes-sig-release/Qz7sVbMYu2c/discussion) ) +* SIG Scalability issues in general + * Tests split to release informing: their tests don’t formally meet the criteria for release blocking. + * But….if SIG Scalability can block on some context, how do we get that represented in a blocking test case? + * Where / how (beyond the 1.15 retro) can we SIG Release have an active discussion with SIG Scalability. They don’t seem to have met but once this year:[ https://docs.google.com/document/d/1hEpf25qifVWztaeZPFmjNiJvPo-5JX1z0LSvvVY5G2g/edit](https://docs.google.com/document/d/1hEpf25qifVWztaeZPFmjNiJvPo-5JX1z0LSvvVY5G2g/edit) +* [jhoak]:[ https://github.com/GoogleCloudPlatform/k8s-cluster-bundle](https://github.com/GoogleCloudPlatform/k8s-cluster-bundle) as a method for packaging Kubernetes manifests/yamls and CRDs. + * KubeCon EU 2019 presentation: h[ttps://www.youtube.com/watch?v=6D5JMFqlov4](https://www.youtube.com/watch?v=6D5JMFqlov4) + * Similar to, inspired by kustomize. Based on custom resources. Includes a cli. + * Intriguing patterns around: + * decoupling build from packaging and following that with qualification and promotion/release + * Machine readable output of total dependency set + * Output to channels (rapid, regular, stable) +* [jsica] relnotes.k8s.io -- dot-release data population? + * 1.15.1: just run the normal patch release process with the existing rel notes generator + * 1.15.x: we can iteratively prove out an alternative rel notes generator and shift if/when we’re happy +* Container Image Promoter + * Via Slack Linus Arver says [14:05] “doh, i can’t make today’s meeting. but, i do have an update: i will be working on the e2e testing for the container image promoter for the rest of the month” (in conjunction with Amy Chen) + * #wg-k8s-infra slack channel is where to contribute if you’re interested + * Primary current gap on e2e test is how do we prove and insure that if somehow this tool went wrong could it in-advertantly delete everything we’ve ever published? (one option is to not delete things) + * Is it possible for this to become a generate tool that can pull and promote arbitrary artifacts? +* Release Engineering subproject meeting and regular backlog burndown to start once 1.15 is done + * Project board:[ https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30) +* Licensing subproject same as ^^^ + * Project board:[ https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31) +* [Draft Exit Questionnaire for Release Team Mentoring](https://docs.google.com/document/d/11Bw-QwM547u9X0L2g_QI0PNXzcisofK25Big_ycZMGg/edit?usp=sharing) [jberkus] +* [Lachie] 1.16 Things + * Proposed schedule:[ https://github.com/kubernetes/sig-release/pull/678](https://github.com/kubernetes/sig-release/pull/678) + * Shadow application closing out THIS Friday. Let’s plug it! -[ Survey](https://docs.google.com/forms/d/e/1FAIpQLSe5voYfOk1pDyjBwotC0go8u0iFtiUpN8iN0LZ8K15dezW6oQ/viewform) + + + + + + + + + + + + + + +
+Date: + June 4, 2019 +
Attending: + +
    + +
  • \ +Tim Pepper + +
  • Jeffrey Sica + +
  • Josh Berkus + +
  • Yassine Tijani + +
  • Claire Laurence + +
  • Lachlan Evenson + +
  • Rael Garcia + +
  • Savitha Raghunathan +
    • +
+
Note taker: + +
    + +
  • \ +Tim Pepper +
    • +
+
+ + + + +* Release Section Lead policy on doing a position a 2nd time [Jberkus] + * Should we have a policy of encouraging turnover? + * A couple current release leads are interested in staying in their role into the next cycle. + * All roles currently have shadows and it appears all of them _could_ step up into the lead role. + * Multiple voices expressing that turnover is good, growing the pool of skills. + * Not letting a shadow step up is a negative. + * Option still for current lead to contribute on “emeritus” work or spin out onto sub-projects with their experience from being lead. Don’t need a title to do good things. + * Possibly limit lead to two terms. + * Shadow could also desire to stay as shadow for an additional cycle. + * Josh Berkus will PR some sample text into the selection process document...something light weight, pragmatic, balancing leading/shadowing, continuity/turnover. +* 1.16 Release Team [Claire] + * Assembling next team is underway + * Lachlan Evenson nominated/accepted to lead 1.16 release team + * [https://github.com/kubernetes/sig-release/issues/648#issuecomment-498467764](https://github.com/kubernetes/sig-release/issues/648#issuecomment-498467764) + * Expect activity in coming week. +* Shadow Questionnaire [jberkus] + * [1.16 questionnaire revisions](https://github.com/kubernetes/sig-release/issues/645#issuecomment-498339020) + * Exit questionnaire for leads & shadows? Goal of shadow process and emeritus lead’s involvement there is to _actually_ grow contributors and contributions. How are we measuring that at the end of a cycle? Folks agree a feedback loop is important. +* Emeritus Advisor -- continue to 1.16? [jberkus] + * Josh feels this needs done for at least one more cycle. He’s happy to do it, but also happy to share the work tightening the feedback loop. + * Beyond 1.16 should consider whether this needs to be a permanent role. +* Release Notes Website [jeefy] + * [relnotes.k8s.io](https://relnotes.k8s.io/) (Hooray) it’s alive, with 1.14 content as sample + * Draft 1.15 notes should be in by end of week + * Should we rip off the band-aid and prefer that site over the markdown? + * How much work to maintain both? + * Releasenotes drafting of verbage would need duplicated. + * Otherwise not too hard. + * Jeff willing to do both in parallel. + * [dims] would there still be a single file somewhere that can be used and viewed and presented offline? [jeff] The design intent was to take the huge list of changes out of the changelog and put them in a tool that makes them searchable on dynamic criteria. + * Human readable summary in static changelog is still desirable on an ongoing basis. + * Relnotes.k8s.io currently shows the latest release. Needs a tweak for permalinking so that the 1.15 changelog links to relnotes.k8s.io 1.15 content. This also gives permalinks to queries and results in the tool. People do want to share a view they’ve found in the data. + * Let’s dooooo it! (Pending Claire’s approval) community +* Publishing version requirements (etcd, cni, coreDNS..) for kubernetes releases [yastij] + * [https://storage.googleapis.com/kubernetes-release/release/stable-1.14.txt](https://storage.googleapis.com/kubernetes-release/release/stable-1.14.txt) has _our_ current version and in a machine readable form + * [https://github.com/kubernetes/kubernetes/blob/release-1.14/CHANGELOG-1.14.md#external-dependencies](https://github.com/kubernetes/kubernetes/blob/release-1.14/CHANGELOG-1.14.md#external-dependencies) has dependencies in a human generated form. + * Umbrella issue:[ https://github.com/kubernetes/sig-release/issues/601](https://github.com/kubernetes/sig-release/issues/601) + * The set of things published, MUST be the set of things tested. Need the tests to depend on a machine readable form, then that can also be published. + * What automation can track bumps of dep’s? Watch the many known places we know dep’s are stated, eg: “verify_external_deps.sh” that does regexp parsing to create a current components yaml file. Mismatch can cause this verify to fail and warn them to also update some other central file. Over time shift preferring the central file as the single source of truth. +* Triage Workflow improvements [nikopen] + * Eventual removal of Triage role after the following Epic is implemented: + * [https://github.com/kubernetes/community/issues/3456](https://github.com/kubernetes/community/issues/3456) + * [https://github.com/kubernetes/test-infra/pull/11818](https://github.com/kubernetes/test-infra/pull/11818) +* Git workflows on our repos [tpepper] + * issue is that repo admins can edit files directly on the repo + * educate everyone to NOT do this + + + + + + + + + + + + + + +
+Date: + May 7, 2019 +
Recording + +
Attending: + +
    + +
  • \ +Stephen Augustus + +
  • Claire Laurence + +
  • Savitha Raghunathan + +
  • Arnaud Meukam + +
  • Tim Pepper + +
  • Craig Peters + +
  • Kenny Coleman + +
  • Tim St. Clair + +
  • Travis Rhoden +
    • +
+
+ + + + +* [justaugustus] SIG Release project boards + * SIG Release:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * Release Team:[ https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29) + * RelEng:[ https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30) + * Licensing:[ https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31) +* [Niko Pen] Triage Workflow improvements + * [https://github.com/kubernetes/community/issues/3456](https://github.com/kubernetes/community/issues/3456) + * [https://github.com/kubernetes/test-infra/pull/11818](https://github.com/kubernetes/test-infra/pull/11818) + + + + + + + + + + + + + + +
+Date: + April 23, 2019 +
Recording + https://youtu.be/s24kIs1RF_4 +
Attending: + +
    + +
  • \ +Stephen Augustus + +
  • Claire Laurence + +
  • Tim Pepper + +
  • Christian Hernandez + +
  • Craig Peters + +
  • Arnaud Meukam + +
  • Dims + +
  • Kenny Coleman + +
  • Lubomir Ivanov + +
  • Savitha Raghunathan +
    • +
+
+ + + + +* [Claire Laurence] if needed, insure release team group has had a time/place to discuss the changes proposed from 1.14 retrospective: + * [https://docs.google.com/document/d/1he2axf3adOIk3gA3vxFAewejtE2tm3Wl1NA1p-ooXpo/edit#bookmark=id.131v7vi6pe2n](https://docs.google.com/document/d/1he2axf3adOIk3gA3vxFAewejtE2tm3Wl1NA1p-ooXpo/edit#bookmark=id.131v7vi6pe2n) +* [claurence] Release Ecosystem Projects that have dependencies on the Release Cycle + * Earlier today SIG PM had some discussion on this + * Release lead handbook indicates lead should sync up with other projects / dependencies to coordinate ahead of release. But aside from kubeadm there’s not much specified. + * In-tree cloud providers could be another place coordination might be required. + * What does “ecosystem” mean? + * cAdvisor? + * etcd? + * Kubernetes-cni (and the others? all the CSI and CRI providers?) + * Kubeadm: was bound to pull the latest existing stable label, but it does not exist yet. That was subseque + * AI: SIG Release leads to investigate tracking dependencies + * Draft a KEP: we need a machine readable, structured, single source of truth, broad OWNERS set. Code that needs to get “etcd” should get the version specified in this file. Release notes should use this file. A PR changing a dependency in this file should get a special label and release notation and special review. +* [justaugustus] Tracking out-of-tree enhancements + * [https://docs.google.com/presentation/d/1L0yM5t1e_61tUi1bu_8swnStznocjqka-aQvMIp1OpM/edit#slide=id.g4a3f70faaf_0_107](https://docs.google.com/presentation/d/1L0yM5t1e_61tUi1bu_8swnStznocjqka-aQvMIp1OpM/edit#slide=id.g4a3f70faaf_0_107) + * Could use a “one sheet” howto on doing KEP work. A final step in that would include a reminder to PR against the KEP a line referencing the PR implementing history section of the KEP. + * Release team tracking enhancements needs a steel thread to understand that intended work is on track, see the PRs merged, see they have test code, see the test is green. + * SIG PM’s had some change in leads and folks unblocked to do more, expect some more roadmap on improvement plans for process (and possible also tools). +* [justaugustus] How should we handle stale enhancements? + * 1.15 enhancement collection found 130 open issues in initially. It’s down to about 36 tracked issues. What are the other hundred issues? Kenny’s commented on those asking if the enhancement would target the current milestone. A small percentage responded no, and about 80% non-response rate. + * Understanding graduation (and ejection?): if an enhancement is proposed, comes in at alpha and sits there with no meaningful progress, what is the next action? + * There is a deprecation policy, but not exactly ejection/deprecation criteria for languished alpha’s. Should enhancement tracking process in Release Team include a nudge to owning SIGs to consider removing code? Also SIG Arch / PM / Testing / Docs. + * All alpha’s should have feature gates (features.go), this is a place we can look at whether there are ones stalled out. + * Unified graduation criteria does not exist. + * It’s easy for us to focus on attacking the things that are moving, but we neglect these things that are languishing. Important for overall code health. +* Release Engineering needs, planning + * see also:[ https://docs.google.com/presentation/d/1usEVcXJirUXCYr7DIBnS7QFZwxzniOEXlm_ek-yy85M/edit#slide=id.g4dff9d87c8_0_643](https://docs.google.com/presentation/d/1usEVcXJirUXCYr7DIBnS7QFZwxzniOEXlm_ek-yy85M/edit#slide=id.g4dff9d87c8_0_643) + * WG K8s Infra pre-requisites: + * Current focus is container images: + * All the cloud providers and C[RNS?]I providers have their own disjoint repositories. + * A tool has been built to allow a limited set of people to “promote” their image into a gcr.io repository. + * An owning subproject would do their dev, do their build, do their test, publish their image to their repository and ping the core project. The core (as represented today in this “alpha” phase by wg-k8s-infra) can then review and accept the image into the k8s release image repos. There will be a directory structure hierarchy and OWNERs files to insure this becomes an ungated process. + * See[ https://github.com/kubernetes/kubernetes/issues/76738](https://github.com/kubernetes/kubernetes/issues/76738) + * A second work area: package repos (RPMs / Debs). + * Needs a clear owner or two. Hannes Horl and Tim Pepper are interested. + * Can follow the same process as image promotion. The next needed step is getting packages built nightly. + * Do we need a consolidated KEP? Currently they’re split on package build and package publish. These are distinct, independent steps. But the vision of the pipeline in which they sit also needs to be a shared artifact. [ https://docs.google.com/document/d/1d4eYH5_RlJ5kuJ0fWjVBFtanBezOavSOjQ6h2Yr8IGU/edit?usp=sharing](https://docs.google.com/document/d/1d4eYH5_RlJ5kuJ0fWjVBFtanBezOavSOjQ6h2Yr8IGU/edit?usp=sharing) + + + + + + + + + + + + + + +
+Date: + April 9, 2019 +
Recording + https://youtu.be/jWNLF6Iotyo +
Attending: + +
    + +
  • \ +Stephen Augustus + +
  • Savitha Raghunathan + +
  • Jorge Alarcon + +
  • Tim Pepper + +
  • Kenny Coleman + +
  • Dhawal Yogesh Bhanushali (slack:dbhanushali, github:imkin) + +
  • Hui Luo + +
  • Abubakr-Sadik Nii Nai Davis +
    • +
+
+ + + + +* \ +[tpepper] rpms/debs support policy + * See:[ https://groups.google.com/d/topic/kubernetes-sig-release/zkYkzzMGjic/discussion](https://groups.google.com/d/topic/kubernetes-sig-release/zkYkzzMGjic/discussion) + * [https://github.com/kubernetes/release/pull/693](https://github.com/kubernetes/release/pull/693) + * [https://github.com/kubernetes/release/pull/688](https://github.com/kubernetes/release/pull/688) + * Should get a document together for consideration of competing stances and then create a KEP +* [tpepper] Patch Release Management: + * GitHub: @kubernetes/patch-release-team + * Email: kubernetes-patch-release-team@googlegroups.com + * Schedule: + * [https://git.k8s.io/sig-release/releases/patch-releases.md](https://git.k8s.io/sig-release/releases/patch-releases.md) + * Should we establish a more regular cadence? Eg: specific day / week of each month? + * [Abubakr Sadik] Like Microsoft Patch Tuesday, could the set of k8s related projects put out the patch releases at the same time make it easier for consumers to know when to look at security issues and applying updates. + * [jberkus] on other projects having a regular (2 months?) cadence has been successful. Should cadence be less often than 2-3 weeks? + * [Stephen Augustus] monthly feels reasonable, beyond + * Action: Tim Pepper to send email to SIG Release, Patch Release Team, and Security Team proposing w/ lazy consensus committing to a monthly release cadence. If agreed, post specific proposal to k/dev for lazy consensus. +* [jimangel] Shadow Selection Survey timeline + * Context:[ https://kubernetes.slack.com/archives/C2C40FMNF/p1554819581205900](https://kubernetes.slack.com/archives/C2C40FMNF/p1554819581205900) + * Trying to outline the survey timeline / wrap up loose ends from 1.14 cycle +* [josh] Need to[ update release role handbooks](https://github.com/kubernetes/sig-release/issues/574) with volunteering information + + + + + + + + + + + + + + +
+Date: + March 26, 2019 +
Recording + https://youtu.be/u9gK3zWDO9M +
Attending: + +
    + +
  • \ +Fabio Rapposelli + +
  • Tim Pepper + +
  • Maria Ntalla + +
  • Caleb Miles + +
  • Claire Laurence + +
  • Hamid Rasool + +
  • Josh Berkus + +
  • Jeffrey Sica + +
  • Jorge Alarcon + +
  • Lubomir Ivanov + +
  • Savitha Raghunathan + +
  • Aaron Crickenberger + +
  • Josh Berkus + +
  • Dhawal yogesh Bhanushali +
    • +
+
+ + + + +* \ +[mariantalla] sig-release testgrid dashboards + * Will go through a thing that I would like to start, hastily described in[ http://bit.ly/sig-release-ci](http://bit.ly/sig-release-ci) + * Desire is to bring (most?) of master’s dashboard config over to 1.14, and possibly prior + * The test boards don’t quite all match, some prior branches’ boards don’t have equivalent tests in master + * Aaron: + * suggests simplify master first, then work back. Consider whether to move upgrade to master blocking (incredibly flakey, take a long time, not run often), otherwise move to informing? + * Can we enforce with automation / linting / tooling that the boards are the same. Ultimately might have boards generated so they’re consistent. + * New jobs expected in a board so they’re observed. + * Can testgrid config be split into multiple files to ungate OWNERS out to the corresponding SIGs + * Now is the time...1.14 is done + * Claire: +1 + * Jberkus: +1 + * Jorge: +1 + * Upgrade: + * This is a worry area for coverage. + * Today’s coverage is via cluster/* bash kube up scripting, poorly maintained by the community (one person now Justin Santa Barbara), fragile. It isn’t actually intended to be Google specific, it’s fully open source, all the runtime debug is available. Does anybody actually use this to do production upgrades in the real world? + * Kubeadm upgrade is coming-ish. There is a doc on how to use it for upgrade, but it’s also not well maintained. [Lubomir] SIG Cluster Lifecycle is planning their 1.15 cycle, which may include kubeadm upgrade. + * cluster/* scripting does fancy testing to insure an app keeps running across an upgrade, but still questions of what is the preferred ordering for upgrading nodes vs control plane and component ordering in nodes. If we just use kubeadm to quickly test that KinD cluster was upgraded to KinD cluster, this doesn’t show that simplistic cluster was actually functional. + * What is the open source canonical preferred cluster configuration and upgrade? Cluster API is also too early to depend on and is unlikely to have upgrade tests in 1.15 cycle. + * Can we get more people on supporting the bash scripting or agree that it’s not used/supported and will just be a release informing test case. + * When (today?) is a kubeadm open source upgrade path ready to be called supported, preferred and release blocking? It has an upgrade option today. + * Proposal agreed: move upgrade tests to informing, continue discussions in community on what has potential to be a release blocking upgrade suite. +* [Josh] Shadow applications for 1.15 RT. + * Need to get started on shadow selection process + * Our apprenticeship shadow mentoring is not completely working. For 1.15 we’re going to do things slightly different. Josh is going to follow the set of shadows more closely and work to insure we are sufficiently building them up. In future cycles we may formalize this “head mentor” role somewhere (under the release lead shadows?) + * Should we continue with the form/questionnaire for prospective shadows? Folks seem to agree, possibly an issue open on improving it…[https://github.com/kubernetes/sig-release/issues/454](https://github.com/kubernetes/sig-release/issues/454) + * Form should be posted tomorrow or Thursday as new folks are commenting on[ https://github.com/kubernetes/sig-release/pull/548](https://github.com/kubernetes/sig-release/pull/548) wanting to participate +* [Claire] Anything else for 1.15 release team? + * Team leads are set:[ https://github.com/kubernetes/sig-release/pull/548/files](https://github.com/kubernetes/sig-release/pull/548/files) + * Prelim schedule: + * Begin April 8 + * Similar to prior spans for enhancements and code freezes + * Release target June 24 + * Awaiting retro feedback + * KEP merged in implementable state will continue to be a requirement, and not a net new one so it should be clear to people (no slight grace period) +* 1.14 retrospective reminder: [ https://bit.ly/k8s114-retro](https://bit.ly/k8s114-retro) Thursday at the community meeting + * There was a loooot proposed after 1.13 and some notable things went quite well in 1.14 + * On time, not crazy last minute drama + * Quite clean CI signal across most of the cycle and especially at the end + * KEPs! +* Rel Eng: + * We still have 3+ build options: + * k/release via Google (current in use) + * k/release via community (no rpm signing, no publishing to official repos) + * k/k/build (needs love) + * New / from scratch artifact generator[ https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/20190227-artifact-generation.md](https://github.com/kubernetes/enhancements/blob/master/keps/sig-cluster-lifecycle/20190227-artifact-generation.md) + * Also #WG-K8s-Infra working in this area + * CNI update package breakage + * Showed up today in[ https://k8s-testgrid.appspot.com/sig-release-misc#periodic-packages-install-deb](https://k8s-testgrid.appspot.com/sig-release-misc#periodic-packages-install-deb) (24hrs late) + * Folks also reported in slack and on GitHub + * Options: + * Issue new packages (iterate on package building scripting): 1.11.6-0 -> 1.11.6-1. For all the “supported” older name-version-release’s (1.11.*, 1.12.*, 1.13*, kubectl, kubeadm, kubelet, cri-tools, kubernetes-cni). + * General consensus to not do this. + * From Jeffrey Sica to Everyone: (02:46 PM): I feel like we try to cover too many edge cases. The vast majority of people are going to want a cleaner upgrade path to 1.11.7, not cherry-pick upgrade dependencies for 1.11.6 + * Tell users how to work around on the command line with flags to yum/dnf/apt/rpm/dpkg and manual downloading. + * Current packages do work. + * This info has flowed in the GitHub issues and on slack. + * Remove older versions from repos, these are not supported and could have security issues. Publish just the head instance of component build. + * What if somebody wants to use them, for reasons...do they need to build their own, maintain their own mirrors...probably are already? + * Act like a distro, maintain the Kubernetes project preferred integration set. + * Can we give meaningful support for the install/runtime variations of all the dependency packaging, testing, integrations? + * Jordan Liggitt has posted a relevant discussion: + * [https://groups.google.com/forum/#!topic/kubernetes-sig-release/LtRKMcwfBBE](https://groups.google.com/forum/#!topic/kubernetes-sig-release/LtRKMcwfBBE) + * [https://docs.google.com/document/d/1WA8N7C48nkJmme9a96DU0o9jBpeycPhht8WF-Eam9QQ/edit#](https://docs.google.com/document/d/1WA8N7C48nkJmme9a96DU0o9jBpeycPhht8WF-Eam9QQ/edit#) + * This doc will help us more clearly define external dependencies and their management + * Do we really need the packages? Who uses them? There are actually people using them. Why aren’t they using their OS’s packages? WG LTS survey hopefully will give us some info on this. + * TBD: Does removing older packages need broader buy in from the community? + * Update docs: Packages today are best effort, and that best effort is at the head of the release branches. + + + + + + + + + + + + + + +
+Date: + Feb 26, 2019 +
Recording + https://youtu.be/Rx3n1Pg2HLw +
Attending: + +
    + +
  • \ +Stephen Augustus + +
  • Jeffrey Sica + +
  • Hannes Hörl + +
  • Mike Arpaia + +
  • Linus Arver + +
  • Jorge Alarcon + +
  • Aaron Crickenberger + +
  • Lubomir I. Ivanov + +
  • Jim Angel + +
  • Claire Laurence + +
  • Arnaud Meukam + +
  • Kendrick Coleman +
    • +
+
+ + + + +* \ +[marpaia] Release Team update + * Enhancements + * 34 slated, 1 w/o KEP + * Out-of-tree handling: marketing, relnotes, comms + * 15 at-risk (no test plan) + * CI Signal + * Follow on repeated test failures for specific SIGs + * Making sure things are appropriately labeled + * Bug Triage + * Not enough issues in milestone + * Gaining better signal prior to freeze + * Test Infra + * Looking good + * Docs + * Looking good + * Release Notes + * Holding for Jeefy to discuss + * Communications + * Out-of-tree discussions: how best to handle + * Blog posts? + * Branch Mgmt + * Daily fast-forward +* [] Go 1.12 + * [https://github.com/kubernetes/kubernetes/issues/73286](https://github.com/kubernetes/kubernetes/issues/73286) + * Now that go 1.12 is out, kubernetes 1.10 and 1.11 are built using unsupported versions of go + * If someone is willing to attempt to bump master to 1.12, aaron is supportive of it (if it lands before Friday), has no opinion on the earlier branches + * Minimal bandwidth to take this on + * SIG Scalability jobs as the signal + * Hesitant to introduce major change this close to Code Freeze + * [justaugustus] Seems we don’t have a policy for this and we should + * [calebmiles] We should figure out how to implement this. Maybe working with Hannes on a pipeline to make this happen. + * Bandwidth concerns + * Let’s punt to 1.15 + * go1.12 brings underlying go versions for k8 +* [tpepper] Patch Release Team comms + * Issue:[ https://github.com/kubernetes/sig-release/issues/516](https://github.com/kubernetes/sig-release/issues/516) + * [justaugustus] Already created a Google Group for this + * Need to update the list and issue + * [caleb] Could we maybe choose something like discuss, that might be more inclusive +* [jeefy] Release notes + * The purpose of release notes and the reality of Themes + * Better release notes formatting + * Comments on out-of-tree release notes:[ https://github.com/kubernetes/sig-release/issues/486](https://github.com/kubernetes/sig-release/issues/486) + * Comments on release notes in general:[ https://github.com/kubernetes/community/issues/484](https://github.com/kubernetes/community/issues/484) + * Move Major Themes to Kubernetes Blog? + * [spiffxp] Relnotes should have the authority to help define the theme + * [caleb] As a project, we should be able to understand/define what Kubernetes is (SIG Arch) and allow that to help guide themes +* [augustus] 1.15 Planning +* [augustus] 1.15 Enhancements + Comms + * Submitting Enhancements + * Staffing + + + + + + + + + + + + + + +
+Date: + Feb 12, 2019 +
Recording + https://youtu.be/Nb7SxI6dJrU +
Attending: + +
    + +
  • \ +Stephen Augustus + +
  • Tim Pepper + +
  • Savitha Raghunathan + +
  • Claire Laurence + +
  • Mike Arpaia + +
  • Linus Arver + +
  • Daisy Tsang + +
  • Timmy Carr + +
  • Abubakr-Sadik Nii Nai Davis +
    • +
+
+ + + + +* \ +[tpepper] Jan 31 gave SIG Release update at the community meeting ([Slides](https://docs.google.com/presentation/d/1B3FVf8B21qBMD1FCKvKx2uS1dNPTG9J9BPRynBpYir8/edit?usp=sharing)); next due April 18 +* New subproject definition work: + * [tpepper] Release-engineering: + * [https://github.com/kubernetes/sig-release/tree/master/release-engineering](https://github.com/kubernetes/sig-release/tree/master/release-engineering) + * Vision:[ https://docs.google.com/presentation/d/1usEVcXJirUXCYr7DIBnS7QFZwxzniOEXlm_ek-yy85M/edit?usp=sharing](https://docs.google.com/presentation/d/1usEVcXJirUXCYr7DIBnS7QFZwxzniOEXlm_ek-yy85M/edit?usp=sharing) + * [Linus Arver] question of [ https://github.com/kubernetes/sig-release/issues/471#issuecomment-457886296](https://github.com/kubernetes/sig-release/issues/471#issuecomment-457886296) ...what is the difference between k/release and k/sig-release/release-engineering? + * [justaugustus] Licensing: + * [https://github.com/kubernetes/sig-release/tree/master/licensing](https://github.com/kubernetes/sig-release/tree/master/licensing) +* [justaugustus] KEPs: + * these have been mostly optional, required in 1.14 + * Retrospective for 1.14’s enhancements freeze and KEPs handling to be scheduled soon via SIG PM (ping Stephen Augustus to get invited) + * KEP template feedback. Please provide thoughts: + * [https://github.com/kubernetes/enhancements/issues/822](https://github.com/kubernetes/enhancements/issues/822) +* [jberkus] Cleaning up sig-release CI boards. + * Testgrid’s SIG Release board split into “[blocking](https://testgrid.k8s.io/sig-release-master-blocking)” and “[informing](https://testgrid.k8s.io/sig-release-master-informing)” + * Also some tests moved over to other SIG’s boards + * Owners of tests: There is an open issue to define an owner per test. Ones that don’t have an owner will be removed from release blocking. + * Should we carry over the updated layout of tests to the other versioned boards for the prior (1.13, 1.12, 1.11) release branches or leave them as-is, and starting with 1.14 the layout would be different? + * Decision made to leave the old branches as-is. 1.14’s branch will be the first to receive the changed layout from the master branch. +* [tpepper for Nishi Davidson] Out-of-tree features:[ https://github.com/kubernetes/sig-release/issues/486](https://github.com/kubernetes/sig-release/issues/486) + * Tracking requirements are not well documented + * Base assumption that out of tree features would be held to a similar or better standard relative to k/k’s practices and processes + * How to coalesce information on status and changes from the out-of-tree work? + * What is the output of the release team? A designation that k8s works abstractly? Works in conjunction with specific tested dependencies as integrated in testing? ...Is Kubernetes a kernel or distribution? + * Should SIGs own their things and done? + * SIG AWS had feature they wanted to go straight to GA, skipping alpha/beta + * SIG Windows felt they were ready to declare an initial minimal viable state as GA for windows containers, rest of the community didn’t agree. Felt to SIG as if the bar for acceptance was constantly getting higher across 9 mo’s + * We need to establish criteria, document it + * Vendors / distributions would need to validate their specific distro reference integration + * Core validation today is against GCE and AWS (in a funkily coupled way), need to improve that to be both broader (more coverage), and narrow validation (mocked?) of the interface + * Changelog historically has been about more than just the core Kubernetes, implying reference integration. Sections give info to downstream users and vendors to help understand changes of the point interfaces and components. + * If cluster/* is removed and replaced with declarative component list that feeds test automation for release...the situation is more clear. + * Do KEPs capture out of tree information? There should be (per Stephen / SIG PM) a KEP for things even outside k/k. The release team likely would not track those KEPs. + * We need a communication mechanism to drive change information out to the ecosystem. + * [dims] How do we handle features released/integrated later? Especially something that is a community project (not a vendor/distribution). + * [Caleb] Subprojects sponsored by SIGs would clearly be “part of Kubernetes” and included in the vendor flow. If there are no SIG-${Vendor provider}, rather they’re subprojects of SIG Cloud Provider, the Cloud Provider abstraction is the primary statement in the comm’s flow. + * [dims] what about things like KinD or Minikube that follow? Trust but verify? As opposed to gating release on their validation data. + * [Caleb] deferred commitment for inclusion of any comm’s from across all of k-org. If things are ready by some cut-off date (~code freeze?) their comm’s are included in that of the k/k release, otherwise they missed the train. + * [Stephen] does this mean release team has to track everything in all of k-org? + * [Claire] if being tracked, what are the criteria? Implementable KEP and issue in k/features repo? Ie: release team tracks k-org (highlight?) features. Consensus: Yes. + * [Caleb] release team controls a valuable resource. Projects have no right to it. It is a privilege. If a SIG wants their work on that 1.XX release train, they need to engage in the 1.XX release process. + * [dims] How to convey this requirement and get buy in broadly? + * [Jim Angel] is there similarity to SIG Docs criteria for deciding what is in the official docs and website? + * [Dhawal / Lubomir] how to create common declarative yaml indicating interdependencies (and replace cluster/*). + * Lubomir’s thinking of a KEP maybe later in 1.15? + * Caleb has some ideas on possible shape of implementation. + * “All the information is out there”, except there are multiple places and sometimes they declare information differently (one part integrates with verX and another with verY). + * [dims] see also: + * [https://docs.google.com/document/d/1P2Eiy7L-TJqG1pU9So-yrf8SftLCQwlMbWq4qfulyuA/edit#](https://docs.google.com/document/d/1P2Eiy7L-TJqG1pU9So-yrf8SftLCQwlMbWq4qfulyuA/edit#) + * [https://github.com/kubernetes/enhancements/pull/746](https://github.com/kubernetes/enhancements/pull/746) + * [Dhawal] when would a KEP k-org be required in implementable state to get sufficient core review ahead of a core release cycle? + * [tpepper] there is general awareness starting in 1.14 cycle of the need for KEPs, probably even beyond just k/k’s contributors. As we move more toward having split the monolith (today we are starting to split) .. ie: the split is mostly done, contributors probably understand the need to get on the train. Wouldn’t need to have their KEP ready months in advance. + + + + + + + + + + + + + + +
+Date: + Jan 29, 2019 +
Recording + https://youtu.be/ZiZiuaN5pGE +
Attending: + +
    + +
  • \ +Tim Pepper + +
  • Sumi Raghunathan + +
  • Hannes Hoerl + +
  • Mike Arpaia + +
  • Claire Laurence + +
  • Raghunathan Savitha + +
  • Craig Peters + +
  • Lachie Evenson + +
  • Arnaud Meukam + +
  • Kendrick Coleman + +
  • Benjamin Elder + +
  • Josh Berkus + +
  • Linus Arver + +
  • Caleb Miles + +
  • Lindsey Tulloch +
    • +
+
+ + + + +* \ +Subprojects: need definition work + * [https://github.com/kubernetes/sig-release/pull/453/files#diff-322d9f3e61a4edcf0b1f9a8c0802b107](https://github.com/kubernetes/sig-release/pull/453/files#diff-322d9f3e61a4edcf0b1f9a8c0802b107) is just place holders + * Releng: + * Packaging (deb/rpm)[ discussion](https://www.youtube.com/watch?v=UxJR5zUdXWQ&index=2&t=0s) + * Multiple KEPs: + * [https://github.com/kubernetes/enhancements/pull/721](https://github.com/kubernetes/enhancements/pull/721) + * [https://github.com/kubernetes/enhancements/pull/707](https://github.com/kubernetes/enhancements/pull/707) + * SIG Cluster Lifecycle is driving actions on unifying the build side + * SIG Release needs to drive on the official builds’ publication, hosting (in conjunction with #wg-k8s-infra), and signing. +* Interns: do we have specific items for which we could use an intern and can commit mentoring resource(s)? + * Google Summer of Code (GSoC): these are supposed to be defined in specificity, but if we write it up in a GitHub issue, we need it held for the applicant to actually do versus somebody just doing it in the meantime. (Deadline 6th Feb.) + * Josh Berkus is submitting for a spring or summer Outreachy intern, for more intensive projects either for SIG-Release or Test-Infra. Outreachy can be a 2-3x large scope project versus GSoC + * Visualization related to test results might be something for which a larger pool of applicants have skills already. + * Ben Elder’s interested in mentoring, having benefited from past GSoC participation + * Finding committed mentors is a key starting point. + * Caleb: automated go/no-go reports daily, visualization of current project heath, deltas day to day in the health metric. Can help with scoping an idea on this. + * Airtable work could fit with GSoC. + * Machine readable dependencies declared across org, consumable by a tool for change management (release notes) + * Update the bash scripting in k/release to something more modern and tied to CI/CD automation. This (and k/k/hack) are hard to maintain, and hard for newcomers to approach, impacts long term sustainability in project. Doing this well also hinges on a solid definition of “what is Kubernetes” and “what is releasing Kubernetes” +* KubeCon: intro and deep dive sessions… + * EU / Barcelona: + * Intro: Tim willing to do our “standard” intro, who’d like to help update the deck and co-present? + * Deep Dive: topic? volunteer(s)? + * Rel eng subproject status update + * Patch release management / team update + * BoF w/ #wg-k8s-infra + * Container image promoter tool and associated build/release workflow changes? + * Packaging improvements + * China / Shanghai + * Intro: volunteer(s) for the standard intro? + * Josh Berkus could do, but a new person would be better. An Asian member of SIG-release would be ideal. Josh to make sure it gets submitted. + * Deep Dive: topic? volunteer(s)? + + + + + + + + + + + + + + +
+Date: + Jan 15, 2019 +
Recording + https://www.youtube.com/watch?v=Al1ClqqXyHk +
Attending: + +
    + +
  • \ +Stephen Augustus + +
  • Tim Pepper + +
  • Arnaud Meukam + +
  • Craig Peters + +
  • Jordan Liggitt + +
  • Maria Ntalla + +
  • Mike Arpaia + +
  • Raghunathan S + +
  • Steven Greenberg + +
  • Julian Modesto + +
  • Aaron Crickenberger + +
  • Linus Arver + +
  • Brian Tardell + +
  • Jim Angel + +
  • Benjamin Elder + +
  • Claire Laurence + +
  • Gianluca Arbezzano +
    • +
+
+ + + + +* \ +Proposed v1.10.13 release [@liggitt] + * Proposal:[ single commit](https://github.com/kubernetes/kubernetes/pull/72860/commits/577157a8734b61d1e471194cc7918eb3e122b94b) to fix[ destabilizing regression](https://github.com/kubernetes/kubernetes/issues/71696) introduced in v1.10.12 + * CI jobs for release-1.10 are still intact and running ([https://testgrid.k8s.io/sig-release-1.10-blocking](https://testgrid.k8s.io/sig-release-1.10-blocking)) with this regression being the red line of FAILING. N-3’s CI jobs are removed around week 5 of the current release’s cycle. + * Discussion[ in slack](https://kubernetes.slack.com/archives/C2C40FMNF/p1547490781256800) and[ in the PR](https://github.com/kubernetes/kubernetes/pull/72860#issuecomment-454072746) on precedent of cutting an n-4 release + * Need to be cognizant of this perhaps introducing a slippery slope, and of the past month’s worth of PRs having been declined because 1.10 was “done”. Documentation is sufficiently clear today in most people’s opinions, for this PR in its GitHub history note the justification. + * AI: @liggitt to summarize discussion[ in PR](https://github.com/kubernetes/kubernetes/pull/72860#issuecomment-454579674) + * AI: document interaction between spinning down CI for an n-4 release branch and inability to cut new release + * AI: document/highlight (where) the caution required on the last patch release of a release branch (when no future patch releases to fix a regression can be expected), definitely more is needed in[ patch release manager handbook](https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/patch-release-manager), but also in other more visible places + * Discussed criteria for acceptance of this PR: + * Supporting CI jobs are still in place + * The issue being fixed was a regression in a patch release + * The issue being fixed is of sufficient severity + * The fix is well understood and contained (doesn’t introduce risk of additional regressions) + * ...Jim Angel opening an issue in SIG-Release to track formalizing/documenting these criteria + * [https://github.com/kubernetes/sig-release/issues/450](https://github.com/kubernetes/sig-release/issues/450) + * Decision: move forward with PR +* 1.14 release team shadows questionnaire is out for folks to submit their desire to volunteer and get reviewed asap (Friday?) + * +* Propose staffing subprojects, create area/foo labels for them [spiffxp] + * hyperkube + * Currently[ k/cluster/images/hyperkube/OWNERS](https://raw.githubusercontent.com/kubernetes/kubernetes/master/cluster/images/hyperkube/OWNERS) has approvers ixdy, luxas, mikedanese + * release-team + * Currently[ k/release/OWNERS](https://raw.githubusercontent.com/kubernetes/release/master/OWNERS) +[ k/sig-release/release-team/OWNERS](https://raw.githubusercontent.com/kubernetes/sig-release/master/release-team/OWNERS) + * Propose splitting into 2 subprojects: release-engineering = k/release/OWNERS (ref: Oct 9 meeting), release-team = k/sig-release/release-team + * Currently k/sig-release/release-team approvers = SIG Release Chairs + 4 Release Team Leads + * Propose Current Release Team Lead + people, volunteers? + * Currently k/release/OWNERS approvers = ixdy, listx, mikedanese + SIG Release Chairs (out of date) + Patch Release Managers (back to 1.8) + Release Leads (back to 1.8) + * Propose Current Release Team Lead + people, volunteers? + * publishing-bot + * Currently k/publishing-bot/OWNERS is caesarxuchao, sttts + * Propose adding nikhita to approvers + * Sig-release (not actually a subproject) + * As-is + * Licensing + * Suggestions for licensing? justaugustus is usually point person in fielding responses, is this a subproject or just a long standing sig-release issue? + * Leads: justaugustus, nikhita, Steve Winslow (LF) + * Release Engineering + * Have OWNERS of this subproject distinct from “release-team” subproject, eg: start with Branch Managers + PRMT (Patch Release Mgmt Team) + * Process and procedures around release tooling and artifacts + * Overlaps with #k8s-infra-team and #sig-testing perhaps, and especially #sig-cluster-lifecycle + * The members of this subproject would be OWNERS of[ http://git.k8s.io/release](http://git.k8s.io/release) repo + * Others: + * PST (Product Security Team)? Who really owns this? Their docs live in k/sig-release, but they are a bit of an outlier on where they fit in the governance model. Not our SIG Release issue to resolve right now, versus steering committee. TBD...table for now. + * Security? There is wg-security-audit, distinct from incident response (Product Security Team). But there’s artifact signing, release thread model, provenance, etc. Could fall under PST, or under release engineering subproject. TBD..table for now. + * “Project support stance / release cadence” aka WG LTS Were it not to get WG approval is it a SIG Release subproject? TBD..table for now. +* FYI[ Jan 31 SIG Release update](https://docs.google.com/presentation/d/1B3FVf8B21qBMD1FCKvKx2uS1dNPTG9J9BPRynBpYir8/edit#slide=id.g401c104a3c_0_0) at community meeting...need to start drafting +* [KEP for Image Promotion Process](https://github.com/kubernetes/enhancements/blob/master/keps/sig-release/k8s-image-promoter.md) and efforts behind the Container Image Promoter ~~(to be open sourced)~~[ open sourced here](https://github.com/GoogleCloudPlatform/k8s-container-image-promoter) -- Linus Arver (listx) + * Q: Do[ test images](https://github.com/kubernetes/kubernetes/tree/master/test/images) follow this process too? (@patricklang - can’t stay, will check notes or Slack if not covered today) + * A: Yes if those images land in k8s.gcr.io (aka gcr.io/google-containers). + * Q from #k8s-infra-team: Is the goal for everybody to use a blessed GCR image? Or a model where the blessed repo is configurable? + * A: The proof-of-concept tool (Container Image Promoter) is still in an early stage; for now the goal is to just demonstrate how the image promotion process for some arbitrarily small set of images. It may be the case that there will be multiple instances of CIP running at the same time to handle different repositories/environments. But AFAIK the current model is that we have blessed GCR images in k8s.gcr.io which everyone uses. + * Q: Which images are in scope versus out of scope for this? + * A: Initially the core images. The goal is to demonstrate how this would work for a small set of images and then expand later as necessary. +* [Cleaning up blocking test boards](https://github.com/kubernetes/test-infra/pull/10505) jberkus + * spiffxp notes sig-release-misc board could be good to keep around for example for any tests that are useful for the various SIG Release subprojects (eg: rel eng subproject hopefully makes a lot of tests for the release mechanisms) + * Owners: some things span SIGs or are handled mostly be folks who are perhaps mostly in SIG Testing. By declaring a SIG Release owner for the job, that owner would responsible for chasing resolution from whoever would be the right person. Alternatively put them under SIG Testing? We primarily need a responsible party who will see the notification of test breaking and start driving to resolution. We’re still hoping that group membership remains curated and group based email notifications lead to folks filtering and highlighting to themselves these in-bound notifications during their tenure as owning. It is also possible to then start measuring notification to response lags. + * Part #3 of the 1.13[ release retrospective](https://docs.google.com/document/d/1mKrJm3N4dC4rfOwa5WcvOptmjpteMcNOg1ZSoACKU1g/edit#heading=h.tw06ll716grh)...discuss the final undiscussed couple bullets and the next release actions table \ No newline at end of file diff --git a/sig-release/meeting-notes-archive/2020.md b/sig-release/meeting-notes-archive/2020.md new file mode 100644 index 000000000..7d221ed48 --- /dev/null +++ b/sig-release/meeting-notes-archive/2020.md @@ -0,0 +1,1822 @@ + + +## Dec 15, 2020 (recording) + +**Host:** Stephen Augustus + +**Note Taker**: Carlos + +**Attendees:** + + + +* Marky Jackson +* Rob Kielty +* Rey Lejano +* Jeremy Rickard +* Max Körbächer +* Carlos Panato +* Marko Mudrinić +* Nabarun Pal +* Daniel Mangum +* Adolfo García Veytia +* Kirsten Schumy +* Lauri Apple +* Joseph Sandoval + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [org:kubernetes SIG Release issue needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [org:kubernetes SIG Release PR needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + +**Open Discussion [timebox to N min]:** + + + +* [Jeremy] Can we make “freeze” dates more enforceable with merge blocking? + * 1.20 we had several things merge after freeze dates w/o coordination w/ release team + * Some thoughts:[ https://hackmd.io/2zJll3I2SLq8pdVNEAX0rQ](https://hackmd.io/2zJll3I2SLq8pdVNEAX0rQ) + * [Stephen] Merge blocking plugin would be useful here, but: + * currently too restrictive, applies to entire repo + * based currently on an issue, if issue closes blockade drops + * Would need to be extended to be team based, selective for branch (main only?) + * Would need a break-glass escape like /override + * Dovetails with what wg-reliability has proposed about blocking for quality + * AI: Jeremy to investigate / research plugin more, craft a proposal and discuss with wg-reliability + sig-release + sig-arch +* [Max] Observing the latest discussion e.g. about “critical” or at least market shaking changes (dockershim deprecation) as well as having long run topic within the SIG release; I got the idea if it wouldn’t make sense to extend the Emeritus Adviser to a Emeritus Adviser Team, where each team member is assigned for 1y to Comms/Enhancements/Triage and so on. This way we would have per team someone who can take on long running topics, and really ensure that the teams are improving. How it help with the latest topic of Docker? I personally believe it would make sense if Enhancements & Comms identify such topics, leverage them to the release team lead & SIG lead, and address this changes right in time (with first release note) but before the release blog posts) + * [Stephen] Issue opened on k/community:[ https://github.com/kubernetes/community/issues/5344](https://github.com/kubernetes/community/issues/5344) + * [Stephen] Having a release note with kind deprecation merged in the repo is too late to notify. + * [Stephen] (Ideas) Deprecation Page that lives outside the release notes documentation + * [Lauri] Idea to have a visual Path to Release with all steps and things related. + * Onboarding for RT members + * Onboarding for general new contributors + * Re-onboarding of veterans who might assume what the processes are now but don’t have new info and context + * Should we create an issue to track this and make a mini project [jeremy]? +* [Stephen] Release Cadence update + * [Adolfo] Aren’t end users most impacted? + * Yes, we should get feedback out as early, often and loudly as possible + * Only so far we can go + + receipts process: aggregate a manifest of the release + + + [jeremy] who can own the idea of a visual roadmap to release: anyone interested can take it + +* AI: Jeremy/Nabarun to chat about notional schedules and provide 3 vs 4 schedule for the year + * (should we notionally plan out the entire year for the purpose of the informational summary that will go out?) +* Delay release maybe a few weeks into Jan and allow decisions to be made and not make date changes that make people upset. +* Can shift a bit after if needed to accommodate 4 releases +* Stephen to aggregate all thoughts around 3 vs 4 releases per year and send to k-dev +* [Nabarun] 1.21 Updates + * In Progress + * Scaffolding the release-1.21 directory + * Decision on schedule + * Pushing Enhancements collection to Week 2 + * Enhancements Team has only one staff until shadows are selected + * Risk: Less time for Enhancements collection if rest of schedule is not changed + * 1.21 Shadow survey should be going out by the end of the week. +* [Lauri] Meetings in 2021 + * Walking the project board + * Sidebar conversations on topics + * Stephen: we need to be more precise, skip recurring topics. Not putting the right amount of focus on important stuff like walking the board. + * We should strive to unblock things, difficult to do as boards are very large. Some effort has been done to trim them. + * More effort will go into walking the boards: SIG Release mtg the bigger things. Giving status updates during releng meeting + * Make sure meetings are more focused by requiring topics to be sent to the mailing list first. +* Meetings for December are canceled we will return on Jan/2021 + + +## Dec 1, 2020 (recording) + +**Host:** TBD + +**Note Taker**: Marko Mudrinić + +**Attendees:** + + + +* Marko Mudrinić +* Nabarun Pal +* Daniel Mangum +* Gianluca Arbezzano +* Jeremy Rickard +* Joyce Kung +* Lauri Apple +* Max Körbächer +* Carlos Panato +* Adolfo García Veytia +* Rob Kielty +* Sascha Grunert +* Rin Oliver + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [org:kubernetes SIG Release issue needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [org:kubernetes SIG Release PR needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + +**Open Discussion [timebox to N min]:** + + + +* **New meeting times?** + * We have new contributors, should we consider new meeting times that would work better for everyone? + * [Jeremy] Earlier would be hard for the west coast, maybe 30 minutes earlier. + * [Marko] Later could be harder for folks in Europe. + * People are generally okay with the current meeting time and day. +* **2021 Wishlist** + * There is still no doc for this, but if anyone has any wish or have an idea of something that should be on our roadmap for 2021, leave it in the notes and we will turn it into something more concrete. +* **[Jeremy] 1.20/1.21 RT updates** + * We’re wrapping up 1.20, the release is scheduled for next Tuesday. + * SIG-Scalability has no issues so far with tests. + * Nabarun is nominated as a Release Lead for 1.21. + * Lead Nomination issue + * [https://github.com/kubernetes/sig-release/issues/1363](https://github.com/kubernetes/sig-release/issues/1363) + * Joyce will be CI Signal lead for 1.21. + * [Stephen] Did we select lead shadows? + * People are reaching out to Nabarun. Nabarun will decide probably next week. + * [Stephen] Congrats to everyone on the release team for the great job! + * [Stephen] Branch management activities during 1.20 + * Generating release branch jobs is in progress - there are some bugs with config rotater and forker that Stephen and Dan are working on. + * We hope that those bugs will be solved soon. We will provide an updated once jobs/dashboards are ready. + * What about release cadence for the next year? + * [Stephen] A lot of feedback since the keynote, but still not ready to make the decision now. + * **AI: **Leads to come together to go through the feedback. Stephen to start aggregating the feedback and ping the lists again (k-dev, CNCF TLs and SIG-Contributor Strategy) to get feedback from the wider community. + * Seems like 3 is not a bad idea. 3 is easiest and next lowest, but we need to make sure about API graduation, and tests. +* **[Dan/Sascha] Arch support updates** + * [https://github.com/kubernetes/sig-release/issues/1337](https://github.com/kubernetes/sig-release/issues/1337) + * Support Tiers:[ https://github.com/kubernetes/sig-release/pull/1360](https://github.com/kubernetes/sig-release/pull/1360) + * Artifact docs cleanup: + * [https://github.com/kubernetes/sig-release/pull/1359](https://github.com/kubernetes/sig-release/pull/1359) + * [https://github.com/kubernetes/sig-release/pull/1358](https://github.com/kubernetes/sig-release/pull/1358) + * [https://github.com/kubernetes/sig-release/pull/1357](https://github.com/kubernetes/sig-release/pull/1357) + * Next steps: building some tooling around tests. + * This work might intersect with work Stephen is working on artifacts, might collaborate on this. + * PR reviews would be appreciated. + * BOM (Bill of Materials) can be put as a roadmap item for the next year. +* **[Sascha] krel updates** + * krel can now be considered as feature complete + * krel stage/release + * We have used it today and it works. There are some follows-up on announcement generation that are in-flight. + * We had a walkthrough yesterday, the recording will be available: \ +[https://zoom.us/rec/share/bZ-tccQ--G_a20fw3n-tYDQ4VfNzR-oSlKjcd92Kmmp6YLBStRxxmYtKDYLIBK57.0ETFZZ4nIHjaPThN](https://zoom.us/rec/share/bZ-tccQ--G_a20fw3n-tYDQ4VfNzR-oSlKjcd92Kmmp6YLBStRxxmYtKDYLIBK57.0ETFZZ4nIHjaPThN) + * We are not completely done. Initially, we mimicked the anago behavior, now we can optimize. + * Improvements around branches/tags and changelog generation. + * We can do image promotion directly from the GCB. + * JSON Release Notes/Changelog. + * [Lauri] What are metrics for what to go with first? + * We don’t have metrics currently, but we can create them. + * [Stephen] Dig into individual packages and see what we can improve. Prioritize the feature requests. + * [Stephen] We should make clear what’s happening inside Krel. We can create a mindmap of what’s happening and boundaries, for example what happens on a local laptop, what happens on Google Cloud. That would allow more people to look into krel. + * Sascha, Carlos, Adolfo to do a session with Lauri about creating the mindmap. + * [Stephen] People are startings holidays soon, so if there’s anything that should be done this year, this is a good time to reach out. You can also DM us if you’d like a private discussion. + * [Stephen] We got a lot of feedback from the Release Managers check-in. + * Updated k/release README:[ https://github.com/kubernetes/release/blob/master/README.md](https://github.com/kubernetes/release/blob/master/README.md) ([https://github.com/kubernetes/release/pull/1803](https://github.com/kubernetes/release/pull/1803)) +* **[Stephen] Artifact management updates** + * **[https://github.com/kubernetes-sigs/k8s-container-image-promoter/releases/tag/v3.0.0](https://github.com/kubernetes-sigs/k8s-container-image-promoter/releases/tag/v3.0.0)** + * **k/artifacts:[ https://github.com/kubernetes/org/issues/2342](https://github.com/kubernetes/org/issues/2342)** + * **CIP repo improvements:[ https://github.com/kubernetes-sigs/k8s-container-image-promoter/issues/266](https://github.com/kubernetes-sigs/k8s-container-image-promoter/issues/266)** + * Bunch of different points where we interact with the community + * push-build script - when we push a build, we push an API reference/version marker which are available in the bucket and that can be used by the community + * dl.k8s.io - reference point to gs://kubernetes-release/release bucket + * Anything that happens in release process primarily happens on Google Cloud + * Stephen plans to work on manifest generation and files promotion + * Some tools are specific (such as krel that works only with Kubernetes), some are generalized (such as release-notes). + * General purposes store interface - for example, what if we want gc2gcs to work with S3 or some other interfaces? The interface should help if someone wants to implement something like that. + * For example, this would help if GCR goes away in favor of Google Artifacts Registry. + * Artifacts repo will be in the Kubernetes organization and controlled by Release Engineering. Transition to having access to administer artifacts on the global scope. + * Stephen hopes to build on the deb/rpm story this year. + * [Rob] Suggestion based on CI - Spyglass has Artifacts links, but it only kinda points to Artifacts, doesn’t point to any build artifacts. We should build formality/language about what are artifacts and build artifacts. + * [Stephen] Balance between the test process (happening in SIG-Testing). Jobs must be configured to dump into the bucket. Jobs using podutilites have this configured by default. + * Those artifacts would be local to build node, instead of global in the bucket. + * kubetest2 work in progress ([https://github.com/kubernetes-sigs/kubetest2](https://github.com/kubernetes-sigs/kubetest2)) + * [Jim] Is the release page in scope of this work? + * Stephen would call it in scope. + * We have an idea of the images we are promoting, but we don’t have idea about files. BOM work is relevant to this. JSON with information about what’s consumable. +* **[Stephen] WG Reliability proposal on blocking contributions:[ https://groups.google.com/g/kubernetes-sig-release/c/P5gFtnjXDqI](https://groups.google.com/g/kubernetes-sig-release/c/P5gFtnjXDqI)** + * Please take a look at the proposal and give feedback + + +## October 20, 2020 (recording) + +**Host:** Stephen Augustus + +**Note Taker**: Joseph Sandoval + +**Attendees:** + + + +* Rob Kielty +* Marko Mudrinić +* Nabarun Pal +* Jeremy Rickard +* Arnaud Meukam +* Eddie Zaneski +* Carlos Panato +* Seth McCombs +* Joyce Kung +* Max Körbächer +* Gianluca Arbezzano +* Adolfo García Veytia + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [org:kubernetes SIG Release issue needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [org:kubernetes SIG Release PR needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + +**Open Discussion [timebox to N min]:** + + + +* CVE Information on changelogs (@puerco) \ +Ref:[ https://github.com/kubernetes/release/pull/1441](https://github.com/kubernetes/release/pull/1441) + + Sig-security meeting discussion with Adolfo about old PR about CVE information. RFC was needed on the old PR. Regenerate the change log up until the last minute. They would like the vulnerability released at the last minute. + + + Stephen shouts out release manager associates. Jumping through all the hurdles that included patches and CVE’. One thing missed was patching release notes. What we need to do is talk about the workflow as we land the above PR. The process needs a bit more definition. If you are doing the patch note you may not be privy to the CVE up until the last minute. Product security committee might need to be consulted. + + + Do we lift the mapfile to GCS? Do we need something private to do the generation of release notes? + + + Release-engineering tagged in PR to review. + +* What do we do alerts? “To make CI-Signal less magic and more sciency.” Lauri has authored a doc and solicited comments from the release teams. First goal is what we do with CI-Signal alerts. The aim is to document the very first step in the process. The new PR workflow follows the K/K template. The overall goal is to provide documentation for CI-Signal. Looking for flakes will start with CI-Signal board. Issues being found on the board and the follow through with developers is receiving no response. Being able to deliver flake reports that are easy to read for the devs. Simplifying the job and test will help make it easier for the users. Linting flake issues so that they are conforming to policy. + * PR:[ https://github.com/kubernetes/community/pull/5244](https://github.com/kubernetes/community/pull/5244) +* Questions: Jobs related to master release blocking. There is not a good gate from putting something on master release blocking. There is a doc:[ https://github.com/kubernetes/sig-release/blob/master/release-blocking-jobs.md](https://github.com/kubernetes/sig-release/blob/master/release-blocking-jobs.md) You have to be informing before blocking. The job should also fork. Forking annotations. Make sure to be informed when something is added to the board. Jobs that land on the board informing or blocking it needs to be forked. +* [https://github.com/kubernetes-sigs/k8s-container-image-promoter](https://github.com/kubernetes-sigs/k8s-container-image-promoter) Stephen working on improvements. CIP Repo Improvements:[ https://github.com/kubernetes-sigs/k8s-container-image-promoter/issues/266](https://github.com/kubernetes-sigs/k8s-container-image-promoter/issues/266) Next thing is to decompose the code flow. CIP is not in an importable state. Promobot files are being used by KOPs. Getting CIP into an importable state is the first step. Moving all tools into K/release is the final goal. More discussion is needed.[ https://github.com/kubernetes/test-infra/pull/19626](https://github.com/kubernetes/test-infra/pull/19626) submitted by Carlos. +* [CI Signal draft roadmap](https://docs.google.com/document/d/1Ath8r0Q0KggfAuQe0L_0_HXn_Pg7pwO9HliA6QVl4Qo/edit?userstoinvite=rob.kielty@gmail.com&ts=5f7b5754&actionButton=1#) +* Also review Dan’s PR:[ https://github.com/kubernetes/sig-release/pull/1163](https://github.com/kubernetes/sig-release/pull/1163) +* [Walk the project board](https://github.com/orgs/kubernetes/projects/23) + + +## October 6, 2020 ([recording](https://youtu.be/FNwPuRMnJJQ)) + +**Host:** Stephen Augustus + +**Note Taker**: Tim Pepper + +**Attendees:** + + + +* Rob Kielty +* Marko Mudrinić +* Jim Angel +* Rey Lejano +* James Laverack +* Savitha Raghunathan +* Lauri Apple +* Tim Pepper +* Gianluca Arbezzano +* Nabarun Pal +* Daniel Mangum +* Carlos Panato +* Jorge Alarcon +* Sascha Grunert +* Verónica López +* Eddie Zaneski + +**Recurring Topics [timebox to N min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * CI Signal Report dev in progress sample output + * [Flaking Jobs as at Mon Oct 5 16:44:32 Irish Standard Time 2020](https://docs.google.com/spreadsheets/d/1E5pFHC7kiDTv6QGUKVxWfA-lSJdrs71JJk2nQ9eNjV8/edit?usp=sharing) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Update about krel/anago effort—big chunks left in releaselib.sh: + * Whole build process, building the artifacts (maybe won’t be much) + * Testing done for Alpha but not for the other release types + * Dropping md5/sha1: these were communicated as deprecated, hopefully things are clean now, but odds are somebody will be broken and need to fix their code still + * Action Item: to communicate to community the previous message +* [CI Signal draft roadmap](https://docs.google.com/document/d/1Ath8r0Q0KggfAuQe0L_0_HXn_Pg7pwO9HliA6QVl4Qo/edit?userstoinvite=rob.kielty@gmail.com&ts=5f7b5754&actionButton=1#) + * braindumping which docs must we create to happy-path "what to do when I get a testgrid alert”? +* [Enhance and evangelise milestone-maintainers documentation and do training/retraining ](https://github.com/kubernetes/sig-release/issues/1257) +* [Walk the project board](https://github.com/orgs/kubernetes/projects/23) + + +## September 22, 2020 ([recording](https://youtu.be/DL6P5_QY8iQ)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Vlad Gorodetsky +* Lauri Apple +* Sascha Grunert +* James Laverack +* Marky Jackson +* Marko Mudrinić +* Tim Pepper +* Jordan Liggitt +* Daniel Mangum +* Jeremy Rickard +* Nabarun Pal +* Savitha Raghunathan +* Rob Kielty +* Nick Young +* Rey Lejano +* Max Körbächer +* Eddie Zaneski +* Arnaud Meukam +* Gianluca Arbezzano +* Adolfo García Veytia +* Hosam Kamel + +**Agenda:** + + + +* WG LTS status read out and decision on continuation or completion + * 1.19 and newer will have 1 year patch release support + * Does 1.18 and 1.17 have N-3 release patch release support term or 9 months patch release support, unless somebody pushes for consensus on another choice + * Jordan: support policy always stated in terms of N-3 release + * 1.19, 1.20 slow down has pushed preceding three get ~1 year support + * If cadence stays slow, 3 releases and 1 year align + * Stephen: We plan on having a discussion at upcoming meeting to discuss cadence + * “[Scientific Twitter Poll](https://twitter.com/stephenaugustus/status/1305902993095774210)”: Asked 3 or 4 releases, >700 votes. + * Lots of conversations will come into play, ext deps will box a lot of this. + * N-3 (this is the[ stated support policy](https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions)) + * [1.20 schedule](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.20#timeline) targets release on 2020-12-08, meaning 1.17 will have support at least through then. 1.17.0 was released 2019-12-05, meaning 1.17 will have ~1 year of support per[ 3-release support policy](https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions). + * 1.21 schedule is TBD but is unlikely to be before the end of March 2021, meaning 1.18 will have support at least through then. 1.18.0 was released 2020-03-25, meaning 1.18 will have ~1 year of support per[ 3-release support policy](https://kubernetes.io/docs/setup/release/version-skew-policy/#supported-versions). + * NOTE: 1.16 is now out of support, but also got ~1 year + * 9mo’s (**_not_** the stated support policy) + * 1.17 and 1.18 would go out of support based on the calendar, relative to their initial release date + * 1.17 released 2020-01-10 so would go out of support in October 2020 + * 1.18 released 2020-04-06 so would go out of support in December 2020 + * The topic of release cadence (currently 3 months and 4 per year) was not addressed with a KEP. No consensus among competing proposals. Consensus could be driven out of in a continuing WG LTS if there is a passionate leader for the task. Otherwise it could fall back to SIG Release to push if it has the resources. Either way if a change is community agreed, SIG Release can operationalize it. + * **Decision**: consensus to end WG LTS. It drove good discussion and concrete outcomes and next topics have logical owners. + * **AI**: report for steering + * final readout, decisions made, things accomplished, capture remaining questions or workstreams to shift back into sig-level tracking + * **AI**: Nick can sleep instead of be in meetings at 1-2am +* 1.20 reopen for kubernetes/kubernetes + * Background: + * From July into August the long code freeze for 1.19 left lots of PRs to merge for 1.20. Added milestone restriction and controlled merges. Residual effect: restriction still in place + * Original reopening plan:[ https://docs.google.com/document/d/1yI7o7R2bOB7QtkHUFMHMb5XObiJmzIAnJPuxhdAX-CA/edit](https://docs.google.com/document/d/1yI7o7R2bOB7QtkHUFMHMb5XObiJmzIAnJPuxhdAX-CA/edit) + * Prior discussions: + * [https://groups.google.com/g/kubernetes-dev/c/YXGBa6pxLzo/m/SPb6yhNBBAAJ](https://groups.google.com/g/kubernetes-dev/c/YXGBa6pxLzo/m/SPb6yhNBBAAJ) + * [https://groups.google.com/a/kubernetes.io/g/leads/c/BZZk0tG2cJU](https://groups.google.com/a/kubernetes.io/g/leads/c/BZZk0tG2cJU) + * Question: Lift it or Leave In Place? + * Leaving in place increases friction + * Stephen: don’t have strong opinions at this point. Some value in building up reviewers/milestone maintainers and treating as an active triage activity. Not sure increased friction is worth it. Opened PR to remove the restriction. Planned to merge yesterday but held for discussion. + * Jordan: what is the goal of the milestone requirement: if it is to make sure someone with project health is making decisions about bug/feature merge, some SIGs do a good job triaging, some do not. Leaving the restriction in place will push more people to MM, adding more people to MM that don’t pay attention to correct signals won’t help. WG Reliability adding automated feedback seems like a better idea than human gate + * Stephen: points to a triage problem, some work release team does is triage. Eventually the goal is to get to the point where RT goes away because we don’t need cat herders and having some RT functions we mask some triage issues. + * Current membership of team is made up of: + * release team leads + * individual sig designated milestone maintainers (typically sig leads) + * Responsibilities/standard for milestoning an issue/PR is not currently clear + * [https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/release.md](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/release.md) + * [https://github.com/kubernetes/sig-release/blob/master/release-team/README.md](https://github.com/kubernetes/sig-release/blob/master/release-team/README.md) + * How to communicate these docs’ content to the broader Kubernetes org group? + * Has been unclear and variable for many years, eg: required all along for tracked features/enhancements, required middle/late in release cycle (eg: after freeze), not required for other items or at other times, manually applied to most things and auto applied to some things. + * SIG Release’s CI Signal folks can go out to other SIG and advertise themselves and the point of their workstream + * SIGs have gotten info about all of this stuff in the chairs&leads forum, but some need to do more onboarding new people, doing triage/reviews/approves, handling milestone maintaining + * Deflaking requires first a deep knowledge of the test itself (SIG is logical place for that) but ALSO a solid understanding of the Kubernetes CI system and this is a notable hurdle. Education/training is the answer. + * Lauri: data about processes SIG to SIG (triage, feature health check):[ https://docs.google.com/spreadsheets/d/1V6ixS-8OH2opd5wxeF5FOCTv6Qw9Fs9vjIj6zmuFKYw/edit#gid=243279358](https://docs.google.com/spreadsheets/d/1V6ixS-8OH2opd5wxeF5FOCTv6Qw9Fs9vjIj6zmuFKYw/edit#gid=243279358) + * Eddie: testgrid is a veeeery intimidating tool, the user experience needs improved. + * Testgrid: It is open source, owned & maintained by non-SIG-Release, suggestions are acted on, presumably contributors welcome) + * PR:[ https://github.com/kubernetes/test-infra/pull/19277](https://github.com/kubernetes/test-infra/pull/19277) + * **Decisions**: + * Need an enhanced education/communication plan developed and operationalized to help SIGs with their CI signal improvements. + * The PR removing the milestone restriction from k/k should merge, going back to only requiring the milestone during freeze. + * Avoid dramatic changes to milestone maintainers list until education and standards around milestone is communicated better + * Re-evaluate a better quality gating process (like the ones being proposed in WG Reliability) +* Triage Party and progress by Arnaud ([related prototype doc](https://docs.google.com/document/d/1Yen26GS7VBX5CydwT8lk2j1m6yyllZaqd-oXjyaEGYE/edit#heading=h.e87k3ltggrs8)) + * SIG Release:[ http://release.triage.k8s.io/s/bug-triage](http://release.triage.k8s.io/s/bug-triage) + * eddiezane: SIG-CLI implementation[ https://tp.kuberneddies.dev/s/daily](https://tp.kuberneddies.dev/s/daily) +* Quick updates (one minute each): + * Clearing out old items: Release Team retro items (GitHub issues and from the docs), RT other-items, Release Engineering board + * Release Engineering has a[ Ways of Working agreement](https://docs.google.com/document/d/11IWywLmG2sPNt7ttAX6I9Pmq3ewxD-9l6vKgLg-FCQE/edit) in progress + * [Project board](https://github.com/orgs/kubernetes/projects/23): is walking the board in this meeting still a goal(?) + * Walking the board is good for delegation, status, reevaluating priorities, transparency + * Release Team is walking their board now + * Posting items in Slack as threads and making headway closing out old items + + +## September 8, 2020 ([recording](https://youtu.be/aBJkf4e5iNU)) + +**Host:** Jeffrey Sica (Retro Moderator) + +**Note Taker**: TBD + +**Attendees:** + + + +* Carlos Panato +* Lauri Apple +* Savitha Raghunathan +* Vlad Gorodetsky +* Adolfo García Veytia + +**Kubernetes 1.19 Release Retro, Part Two:** + + + +* Please head to the retro doc:[ https://bit.ly/k8s119-retro](https://bit.ly/k8s119-retro) + +**Open Discussion [timebox to N min]:** + + + +* + + +## August 25, 2020 ([recording](https://www.youtube.com/watch?v=fGvVElSoBG8)) + +**Host:** Stephen Augustus + +**Note Taker**: Jorge Alarcon + +**Attendees:** + + + +* Sascha Grunert +* Max Körbächer +* Savitha Raghunathan +* Zachary Estrella +* Lauri Apple +* Daniel Mangum +* Rey Lejano +* Taylor Dolezal +* Vlad Gorodetsky +* Adolfo García Veytia +* Robert Kielty +* Carlos Panato +* Marko Mudrinić +* Marky Jackson + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 1.19.0 delayed until Wednesday (tomorrow) + * giving some soak time to tests + * Community meeting, planning to move to next week pending feedback from the team. + * CI Signal + * integration and verify were flaking due to infra change. Infra change was reverted and no flakes since. + * Few critical PRs merged late and we want to allow scalability jobs run a bit + * Flakes appearing seem to be infra-related flakes but pinging SIGs to make sure these are not critical + * We are a GO! + * 1.20 updates + * Jeremy Rickard selected as 1.20 lead + * All role leads have been selected and PR-ed in + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Final 1.16 patch (1.16.15) scheduled for 9/2. Cherry pick deadline this Friday (8/28) + * Rolled new image tag format for Debian base images: os_codename-vx.y.z, example: buster-v1.0.0 + * we need to surface what things have changed in between images - work for 1.20 + * Migrated go-runner image building to k/release + * few things we built into distroless + * starting 1.19 this will be used for core images. We have been observing during 1.19 and they seem happy. + * core images are not the only things that depend on debian base. + * there is an overarching work to move images to distroless. + * When building images, we usually start go debian-base -> promote it -> debian-iptables -> promote (previous release we have to also build a hyperkube image). FInally we PR into kubernetes to update images. + * distroless images are built on debian. + * There are 2 streams of distroless static images : static-debian9 and static-debian10. We want to make sure this info is in the name. + * go runner build has 2 stages: build binary and copy into status debian image. + * k/release has a merge blocker applied until 1.19.0 is outCI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Thank you to Sascha, Adolfo, Carlos, and few others. + * we will get rid of anago - hopefully by end of year - but we want to minimize the variance so this will go until after 1.19. + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review + * Interesting issues to look at + * Docker Hub's new policy for limiting image pulls[ https://github.com/kubernetes/kubernetes/issues/94018](https://github.com/kubernetes/kubernetes/issues/94018) + * gcr.io/google-containers and gcr.io/k8s-artifacts-prod don't work anymore[ https://github.com/kubernetes/kubernetes/issues/94013](https://github.com/kubernetes/kubernetes/issues/94013) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Proposal approved for reopening k/k for 1.20 development + + +## August 11, 2020 ([recording](https://www.youtube.com/watch?v=LIBXW2P4Rsw)) + +**Host:** Stephen Augustus + +**Note Taker**: Lauri Apple + +**Attendees:** + + + +* Stephen Augustus +* Taylor Dolezal +* Marko Mudrinić +* James Laverack +* Sascha Grunert +* Jim Angel +* Carlos Panato +* Josh Berkus +* Mikael Johansson (Docs Shadow) +* Nabarun Pal +* Lauri Apple +* Jeremy Rickard +* Rey Lejano +* Rob Kielty +* Adolfo García Veytia +* Jorge Alarcon +* Daniel Mangum +* Verónica López +* Joseph Sandoval +* Vlad Gorodetsky + +**Recurring Topics [timebox to N min]:** + + + +* We’re following the agenda! +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * No updates + * Moving into 1.20, we’ll want to look at some of those issues. Looking for volunteers. + * Don’t probably need a meeting on this subproject, but if you want to get involved let us know. + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * k/release @ v0.4.0 is out (IT CHONKY):[ https://github.com/kubernetes/release/releases/tag/v0.4.0](https://github.com/kubernetes/release/releases/tag/v0.4.0) + * Lots of new image builds + * improvements to krel (thanks Adolfo and Sascha and everyone else on krel) + * Nice updates on port from bash libraries over to Go -- moving on nicely. Project to see Anago replacement by end of 2020. Fingers crossed. + * [Stephen] Debian base images + * Some new ones. We’re moving to distroless images for lots of components living in k/k. New image called Go Runner -- some sugar on top of the distroless static image. Makes some images more lightweight -- should see images built with this from 1.19. We believe all images for core components are now updated. + * Currently on Debian Buster, and we’ve pulled in security updates (CVE for Perl, for ex.) -- patched over last weekend+half. + * Still rolling PRs for updates to previous branches (hopefully done by end of the week) + * Want system for automated Debian base images + * Automated everything -- Stephen has ideas, wants to turn them into something cohesive before opening to discussion but essentially leveraging same mechanism as test-infra to update their images. Would involve making the auto-bumper and gce builder across multiple repos. Pattern could be useful to the entire community. Images triggering their own builds or on their own schedule. Could alleviate lots of CVE-related issues. 1.20, not trailing-1.19. + * [Stephen] Golang updates + * Master and release branches were already on Go rc.1 as of Sunday. PR merged for rc.2. then. Tracking well. Low friction outside of mechanics of doing a Go update. + * Release is essentially gated on Go 1.15 going live. So far so good -- by early September? + * AI: Stephen to ping the Go team to check in, get an update. They have a release milestone and set of issues to aggregate (blocking, etc.). They don’t directly dictate timing. + * [Lauri] updates on process + * New columns in the project board + * Want to go through the “In Progress” work slowly + * Sig-testing will talk about CI Policy Improvements tonight at their regular meeting -- great work done, thanks Rob Kielty and others for your contributions + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Stephen] Timing / Code Thaw / Comms to k-dev + * Still on target + * KubeCon blackout, give folks time to decompress, take a break, prepare for their talks + * After this period we’ll release if everything looks good + * CI Policy Improvements going well + * Often falls to wayside as we discuss features instead but stability’s key + * Every release should be stable, so if we must pump brakes during a release we should: sig-testing, Tim, Stephen support this + * Code Freeze: We’ve extended it and pushed Code Thaw to some unknown date. Initially discussed in the first draft of the updated release schedule. + * Stephen talking to RT -- aim to have no thaw until end of the cycle. + * AI: RT Leads to discuss today and decide. + * AI: Stephen to write a note about this. + * Josh Berkus: for rest of dev community that’s not attending regular RT meetings, what’s going on with Code Freeze and Code Thaw has been a bit mysterious. Josh talked to Taylor about adding some things to Release Lead handbook about providing more frequent updates when the schedule changes. + * But the schedule hasn’t changed? Outside of RCs and initial Freeze not much change, but seems like it. + * Yes it has: k-dev@ been silent. + * Taylor: more clarity is better, weekly updates (here’s what happened) makes a lot of sense. + * Meeting updates or a template around internal comms. + * What are the most effective communication vehicles for big updates? + * Some SIGs now using GitHub + * What about Hack.md? + * These notes should also probably go out to the SIG leads list. Expectation: if you’re a lead, you’re paying attention to that list. + * [Taylor] Team updates + * We have a 1.20 Release Lead: Jeremy Rickard YAYYYY + * Talk to Jeremy about 1.20 topics + * 7 open PRs for 1.19 / 1 critical/urgent + * [https://github.com/kubernetes/kubernetes/pulls?q=is%3Aopen+is%3Apr+milestone%3Av1.19](https://github.com/kubernetes/kubernetes/pulls?q=is%3Aopen+is%3Apr+milestone%3Av1.19) + * 15 open Issues for 1.19 / 6 critical/urgent + * [https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+milestone%3Av1.19](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+milestone%3Av1.19) + * KubeCon/CloudNativeCon break! + * Resuming meetings on Monday, August 24th + * Targeting August 25th for 1.19 release + * [Kubernetes 1.19 Release Retro](https://docs.google.com/document/d/1vJW2AyTS4RTTnnlmVA76vZFzdQt9XDBR80qOi5gvxl4/edit) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * [Stephen] Formation this week, Jorge is point person and execution lead; Dan M., subproject co-owner (YAY) + * Mission: Do exactly what the CI Signal team of the RT does today, but for all release branches + * A lot of our sub-teams for the RT are extensions of other groups and CI Signal has a close relationship with sig-testing (yay collaboration for CI Improvements and stability tasks) + * Would be responsible for defining what release blocking and informing looks like + * Sascha (Rel Eng) and Jorge (CI Signal) to align with the focus they’ve had over multiple cycles + * [Dan/Jorge] SIG Testing collab updates + * Adding limits and resource requests has increased flakes but this number is going down because we know how things are scheduled now + * Presubmits have been outside scope of CI Signal, but continuing the push + * have seen some improvements (seeing more out of memory errors, but that’s good because we can now address it, why it’s happening, and create specific jobs) + * On all release-blocking jobs we should have limits and resource requests now + * On 1.19, cleaning up flakes, SIGs super-responsive, so keep Code Freeze around longer to let them fix things + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * WG LTS… is basically sleeping until 1.19 finishes support + * Length of patch support for 1.17 and 1.18 is up to SIG-Release + * Movement on retroactive support: it’s not part of the LTS KEP. 1.17 and 1.18 are the retroactive support. + * If you’re supporting for three releases, it’s nearly a year anyway, so if release engineering wants to follow three releases then it becomes a sort of de facto policy. But LTS won’t decide that. + * AI: Back to SIG Release to decide. + * AI: Josh can circle back with Lauri, go through the KEP +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + +## July 28, 2020 ([recording](https://www.youtube.com/watch?v=OZB7DM2SHEo)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Stephen Augustus +* Tim Pepper +* Taylor Dolezal +* Till Wegmüller +* Daniel Mangum +* Nabarun Pal +* Robert Kielty +* Adolfo García Veytia +* Joseph Sandoval +* Lauri Apple +* Jorge Alarcon +* Carlos Panato + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* **Go1.15** + * 1.19 is being built with 1.15 rc1 (a prerequisite for rc1 is Google being willing to run all production on it). + * Scalability also runs some tests on the latest. +* Kubernetes 1.19 + * trimming down critical issues +* Kubernetes 1.20 + * Starting to discuss timelines around conferences, holidays, and TBD 1.19. Broader conversation will start next week. + * If interested in the release team please be on the lookout for the call for volunteers +* We have an action items Issue for 1.19:[ https://github.com/kubernetes/kubernetes/issues/93430](https://github.com/kubernetes/kubernetes/issues/93430) + * CI Signal subproject ([https://github.com/kubernetes/sig-release/issues/966](https://github.com/kubernetes/sig-release/issues/966)) could be helping with a lot of these (see also “CI Signal” below) +* We’re still moving forward with our RT prioritization session results + * 1:1’s with Leads complete + * Drafting plans based on our priorities to organize how we can deliver them + * Setting up our backlog to start Triage Partying + * If you’ve served on a past RT, get in touch +* CI Signal + * SIG Testing meeting later today + * Jordan’s proposal “Policies to improve Kubernetes CI”:[ https://docs.google.com/document/d/1uuP9kS28u1LFyx83Yl-Sodu7-2N_exPRrEoiwbAe0wc/edit?ts=5f1f3819#heading=h.xgjl2srtytjt](https://docs.google.com/document/d/1uuP9kS28u1LFyx83Yl-Sodu7-2N_exPRrEoiwbAe0wc/edit?ts=5f1f3819#heading=h.xgjl2srtytjt) + + +## July 14, 2020 ([recording](https://youtu.be/xcLl9AE_ULI)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Marko Mudrinić +* Jeremy Rickard +* Nabarun Pal +* Adolfo García Veytia +* Sascha Grunert +* Jorge Alarcon +* Arnaud Meukam +* Tim Pepper +* Taylor Dolezal +* Wilson Husin +* Tina Tsou +* Veronica Lopez +* Marko Mudrinic +* Robert Kielty +* Carlos Panato +* Lauri Apple +* Jim Angel + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Trending this week + * 1.19 rc.1 July 14: happening right now, looking on track + * Golang security bump: + * [https://groups.google.com/u/1/g/golang-announce/c/f2c5bqrGH_g](https://groups.google.com/u/1/g/golang-announce/c/f2c5bqrGH_g): + * No info, but coming July 14 + * If high severity, options: + * Delay k8s patch release (July 15 target), or + * Do k8s patch release, incorporate without release again next week? + * K8s patch releases target July 15 + * See also golang update...maybe need to delay this if PSC read of golang issue is high sev which we need to fix immediately + * Branch ff & 1.19 & cherry picks: + * will end at 1.19.0-rc.2 July 21, + * cherry picks required after that + * Prior releases usually had a few days of final cherry picks ahead of the .0 release, this time it will span ~2 weeks. + * Branch management, CI Signal, Release Team, and Release Team lead will need to have conversations and evaluate what has merged on master, is it truly critical critical urgent for .0, is it ready, should it be pulled it. With extra time, more people may try to push more content into .0 last minute, increasing risks. + * VDF (Vanity domain flip) : next week, misc. PRs to tidy up urls, also svc acct kubernetes-release-test needs access to write staging +* Longer term + * Patch release bug or feature relative to cloud providers: + * Common issue of cloud provider wanting to add support for new, previously unknown scenario. These may be complex (big feature, unlikely to backport), or fairly simple (append the specs of a new VM type to a list of available VM types, could we going ahead with these) + * Need heuristics across the spectrum of issue types. (current criteria here:[ https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md#what-kind-of-prs-are-good-for-cherry-picks](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-release/cherry-picks.md#what-kind-of-prs-are-good-for-cherry-picks)) + * Golang 1.15 trending early (maybe later?) August maybe?[ https://groups.google.com/d/msg/golang-dev/0XatS61k9j4/zIhil_L1AQAJ](https://groups.google.com/d/msg/golang-dev/0XatS61k9j4/zIhil_L1AQAJ) + * Bazel: ongoing but on-hold discussions around removing Bazel from Kubernetes. We have 3+ ways of building, Bazel is complicated, but useful in test-infra. Many would like to see it removed and a simpler mechanism for the caching used by test-infra. Most recent discussion at[ https://github.com/kubernetes/kubernetes/issues/88553](https://github.com/kubernetes/kubernetes/issues/88553). Would like to see this happen, but it needs dedicated staffing from people with the interest and know-how spanning k/k, release, and test + * Kubernetes Golang image building: consuming straight from upstream golang’s community run semi-officialimages means we have more ability to build variations (and faster, instead of waiting for their volunteers). Doing this would enable us to get earlier signal on golang integration issues (eg: perf/scale) and speed up our regular patch version updates too. +* Triage party: having a health check issue versus ingress + + +## June 30, 2020 ([recording](https://youtu.be/7M4RIwKmzNk)) + +**Host:** Tim Pepper + +**Note Taker**: TBD + +**Attendees:** + + + +* Marky Jackson +* Sascha Grunert +* Adolfo García Veytia +* Taylor Dolezal +* Jim Angel +* Tina Tsou +* Tim Pepper +* Marko Mudrinić +* Arnaud Meukam +* Jorge Alarcon +* Gianluca Arbezzano +* Rey Lejano +* Rob Kielty +* Carlos Panato +* Vlad Gorodetsky +* Lauri Apple + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Mapping Release Notes to fix-up: + * Can iterate on the notes early in the cycle and be able to add information \ +[https://github.com/kubernetes/release/pull/1373](https://github.com/kubernetes/release/pull/1373) + * Can add special CVE details mentioned once they’re being made public[ https://github.com/kubernetes/release/issues/1354](https://github.com/kubernetes/release/issues/1354)) + * Yaml mapping files to override the release-note (and other) fields automatically pulled in for a given PR + * Thanks Adolfo, Sascha! Makes it easier to spread work across time on large .0 releases and eases last minute handling of security issues. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Eight (8) weeks away from release week! + * FF until RC.2, everything else is Cherry Picks (one week of FF, three weeks of CPs) + * Merge restrictions for roughly a month (from Code Freeze/Branch Cut to CP deadline) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * Have a first conformance run green after (a week?) red...hopefully fixed on 3rd attempt +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Migrating kubelet from Hyperkube [ Marko Mudrinic] + * Do we have any recommendations how kubelet should be run, especially for container Linux distros (Flatcar, Fedora)? Check with SIG Node if they have preferences. Kubelet in container has been considered deprecated. + * Should we consider making a kubelet image? + * If not, can we provide some recommendations on what image should be used or how the image should be built? + * For reference: deprecate “containerized kubelet”[ https://github.com/kubernetes/kubernetes/issues/74148](https://github.com/kubernetes/kubernetes/issues/74148) + * [https://github.com/kubernetes/kubernetes/pull/74267](https://github.com/kubernetes/kubernetes/pull/74267) “Containerized kubelet was originally part of a level of self-hosting, but the idea ran into multiple issues and folks are no longer using it. There are no owners for this feature and the CI jobs are flaky as well.” + * kind still runs kubelet in a container, it just does not use the flag (--containerized) or volume mounts. +* ARM64 initial support [ Tina Tsou, Robin Lu, Howard Zhang et al from ARM ] + * Walk through adding the CI jobs to the release informing board and how to move the test reports into release-blocking. + * Discussion thread:[ https://github.com/kubernetes/kubernetes/issues/86295](https://github.com/kubernetes/kubernetes/issues/86295) + * Currently, we can use kubeadm/kops to deploy an arm64 conformance test cluster. + * All known tests related to arm64: + * [https://testgrid.k8s.io/conformance-arm](https://testgrid.k8s.io/conformance-arm) + * [https://testgrid.k8s.io/sig-node-arm64](https://testgrid.k8s.io/sig-node-arm64) + * [https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kops#kops-aws-arm64](https://k8s-testgrid.appspot.com/sig-cluster-lifecycle-kops#kops-aws-arm64) + * What else can we do? + * Have discussed Dim & Ben + * Have discussed with SIG Node and are on next agenda + * Should discuss with SIG Arch (along with Illumos)[ https://docs.google.com/document/d/1BlmHq5uPyBUDlppYqAAzslVbAO8hilgjqZUTaNXUhKM/edit#](https://docs.google.com/document/d/1BlmHq5uPyBUDlppYqAAzslVbAO8hilgjqZUTaNXUhKM/edit#) Thursday July 2 + * Potential gaps: + * SIG Node evaluation of readiness? + * SIG Arch thoughts on process for adding globally “support” for arch/distro? + * Federated tests: + * are running in a VM and had some host networking issues. Test is stable (periodic set to daily), but weren’t getting + * Lubomir: aware in SIG Cluster Lifecycle that ARM works based on user reports, but haven’t had the test signal so very much appreciate the effort to formalize it. The project needs a succinct list of arch/OS variations and their current status. + * Tina: many companies internally have it running and are doing CI, so now is time for community CI + * Build artifacts: need to understand if/which additional build artifacts are needed. We have golang ones and containers. Do we also need RPMs/Debs + * Phrasing: this isn’t “initial support”. We’ve had that. This is “Additional test signal” and quality assurance, which gets at a more formal concept of support +* Triage-Party Update + * Marky: Almost there on tool, some DNS tweeking + * Tim: when will we do our parties? + * Marky: Doodle coming soon to the mailing list + * Lauri: let know if help on moderating the party +* KubeCon EU: will need to record our SIG sessions soon (presenters watch for email on it) + + +## June 16, 2020 ([recording](https://youtu.be/ypnXMZcKJxI)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Sascha Grunert +* Daniel Mangum +* Tim Pepper +* Nikhita Raghunath +* Carlos Panato +* Seth McCombs +* Marky Jackson +* Adolfo Garcia Veytia +* Rey Lejano +* Arnaud Meukam +* Jorge Alarcon (the people ↑ are cool) + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* SIG Technical Leads discussion + * Proposing Jorge and Sascha as TLs:[ https://groups.google.com/d/topic/kubernetes-sig-release/2VZndEcjPTo/discussion](https://groups.google.com/d/topic/kubernetes-sig-release/2VZndEcjPTo/discussion) + * Lazy consensus timeout: Thursday, June 18th at 4pm US Eastern +* Deploy triage-party:[ https://github.com/kubernetes/k8s.io/pull/967](https://github.com/kubernetes/k8s.io/pull/967) + * Awaiting github token in cluster and dns + * Could start against k/sig-release and k/release (and k/enhancements?) for simpler SIG Release start than the volume of k/k +* WG Naming proposal[ https://groups.google.com/g/kubernetes-dev/c/kry8QbIpxRs/m/xaWKZYUGBgAJ](https://groups.google.com/g/kubernetes-dev/c/kry8QbIpxRs/m/xaWKZYUGBgAJ) + * Basic goal to remove racist or non-inclusive language from project + * Some docs PRs already moving away from whitelist/blacklist + * Control plane terminology (master/slave) + * The way we discuss control plane members mailing list thread[ https://groups.google.com/forum/#!msg/kubernetes-sig-cluster-lifecycle/IjN_9n0KEUg/h4i93a_EAgAJ](https://groups.google.com/forum/#!msg/kubernetes-sig-cluster-lifecycle/IjN_9n0KEUg/h4i93a_EAgAJ) +* 1.19 release schedule change[ https://groups.google.com/g/kubernetes-dev/c/TVXhcNO3SPU/m/-Uj-xJP2BQAJ](https://groups.google.com/g/kubernetes-dev/c/TVXhcNO3SPU/m/-Uj-xJP2BQAJ) + * How do we communicate more effectively? + * Document how we do external communications +* Illumos OS support? [ https://github.com/kubernetes/kubernetes/issues/91570](https://github.com/kubernetes/kubernetes/issues/91570) + * First recorded Hi from the[ illumos](http://illumos.org/) community: continuation of OpenSolaris + * Till Wegmueller is going to work up a KEP for Kubernetes support of the illumos zones implementation (eg: add Illumos container support next to Linux and Windows in k8s) + * OS has stable interfaces and CRI/CNI implementations + * Implications for k8s dev/test/release/conformance processes across the board + * SIG Architecture meeting agenda/time/links:[ http://bit.ly/sig-architecture](http://bit.ly/sig-architecture) + * ML discussion:[ https://groups.google.com/d/topic/kubernetes-sig-architecture/eHBFgfd6Qxg/discussion](https://groups.google.com/d/topic/kubernetes-sig-architecture/eHBFgfd6Qxg/discussion) +* Kubelet.service crashloop service change:[ https://github.com/kubernetes/release/pull/1352](https://github.com/kubernetes/release/pull/1352) + * Want to change the systemd unit file to not be default enabled/started + * Shouldn’t change that in patch releases + * The file lives in k/release, but is not forked (or easily forkable?) + * Where should the spec files for packages live (eg: k/release or k/k)? Preferably in the component repo. + * Where should the systemd unit files for project components like kubelet exist (eg: k/release or the corresponding component’s repo...k/k, k/kubeadm, k/kubelet)? Preferably in the component repo. + * Long term need to allow the package content (and spec?) to be forked and versioned + * Today SIG CL / kubeadm sees the systemd unit file issue as a minor annoyance. People understand it and workaround it if needed. Bigger issue is the packages’ current drop in config file which includes config elements which are going away as a potentially breaking change, and fixing that would require the packaging be forked and versioned +* Golang updates + * Still in-flight: + * go1.14.4 + * go1.13.12 + * [Stephen] Pending a repo-infra bump for bazel rules_go + + +## June 2, 2020 -[ Meeting canceled](https://groups.google.com/d/topic/kubernetes-sig-release/ZElQ4flH3so/discussion) + + +## May 19, 2020 ([recording](https://youtu.be/GwFnwmCkOGI)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Daniel Mangum +* Vlad Gorodetsky +* Marko Mudrinić +* Rey Lejano +* Jorge Alarcon +* Tim Pepper +* Sascha Grunert +* Verónica López +* Nabarun Pal +* John T Skarbek +* Taylor Dolezal +* Harsha Narayana +* Carlos Panato +* Ricky Lindenhovius +* Gianluca Arbezzano +* Rob Kielty +* Adolfo García Veytia + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Base images + * debian-base / debian-iptables: first update in quite a while. These images have had (and will always) a smattering of CVEs, so important to keep up to date. + * go-runner + * Updated base image exception list:[ https://github.com/kubernetes/sig-release/blob/master/release-engineering/baseimage-exception-list.md](https://github.com/kubernetes/sig-release/blob/master/release-engineering/baseimage-exception-list.md) + * Etcd + * Do we need our own? It is just basic etcd on our baseimage, plus a migration shell script. Do we have upgrade[ tests depending on its migrate-if-needed.sh](https://github.com/kubernetes/kubernetes/tree/master/cluster/images/etcd) script? + * Follow-up w/ Dims on status for this + * [https://github.com/kubernetes/kubernetes/tree/d63d77dc4cae044702dac1fc5a97fafebebcbb0f/cluster/images/etcd/migrate](https://github.com/kubernetes/kubernetes/tree/d63d77dc4cae044702dac1fc5a97fafebebcbb0f/cluster/images/etcd/migrate) includes a go based etcd migration and a commit from 2 years ago “Reimplement migrate-if-needed.sh in go” +* 1.19.0-beta.0 released + * [https://groups.google.com/forum/#!topic/kubernetes-announce/E7C1to9mrSA](https://groups.google.com/forum/#!topic/kubernetes-announce/E7C1to9mrSA) + * Includes updated base images (as above) + * Go dependency report is now autogenerated (thx Sascha!). See the release note link on kubernetes-announce: Dependencies {Added, Changed, Removed} +* Patch releases tomorrow +* Updates from Marky: + * Triage-Party: @veronica, @xmudrii and I met yesterday to start lining up the tasks to drive to completion. + * Smooth experience trialing w/ minikube. + * Looking at options for infra hosting. Not this week, but next week WG K8s Infra:[ https://bit.ly/wg-k8s-infra-notes](https://bit.ly/wg-k8s-infra-notes) …this initial triage party hosting is quite possibly something they could get up quickly. + * [Stephen] Could there be a top level navigator page (like testgrid.k8s.io) to drop into the triage party for any SIG using it? + * Continue working on go version bump + * [Stephen] go 1.14.[0,1,2] update attempt on master has seen some unexpected scalability issue + * [Stephen] Needs go1.13.11 (since it just came out for a 390x nil dereference in the compiler) +* Annual support update:[ https://github.com/kubernetes/enhancements/pull/1782](https://github.com/kubernetes/enhancements/pull/1782) +* Cherrypick approval request - “Add back anti-affinity to kube-dns pods” [prameshj] + * 1.18[ https://github.com/kubernetes/kubernetes/pull/91061](https://github.com/kubernetes/kubernetes/pull/91061) + * 1.17[ https://github.com/kubernetes/kubernetes/pull/91060](https://github.com/kubernetes/kubernetes/pull/91060) + * Re-re-fixing past issue (1.15 timeframe?), with possible scalability implications, but as of yesterday has SIG Scalability (wojtek-t) has ACK’d + * Kube-dns not default on these newer versions so less test coverage. + * Past cherry pick deadline for tomorrow’s release. Will merge Thursday for June’s patch release, leaving time to possibly see if there’s any additional test feedback on 1.19/master or 1.18/1.17 ahead of next patch release. +* Re-re-fixing past issue with complex possible scalability implications, but as of yesterday has SIG Scalability (@wojtek-t) ACK +* KEPS + * [https://git.k8s.io/enhancements/keps/sig-release/1729-rebase-images-to-distroless](https://git.k8s.io/enhancements/keps/sig-release/1729-rebase-images-to-distroless) + * [https://git.k8s.io/enhancements/keps/sig-release/1731-publishing-packages](https://git.k8s.io/enhancements/keps/sig-release/1731-publishing-packages) + * [https://git.k8s.io/enhancements/keps/sig-release/1732-artifact-management](https://git.k8s.io/enhancements/keps/sig-release/1732-artifact-management) + * [https://git.k8s.io/enhancements/keps/sig-release/1733-release-notes](https://git.k8s.io/enhancements/keps/sig-release/1733-release-notes) + * [https://git.k8s.io/enhancements/keps/sig-release/1734-k8s-image-promoter](https://git.k8s.io/enhancements/keps/sig-release/1734-k8s-image-promoter) +* + + +## May 5, 2020 ([recording](https://youtu.be/Vq0KLO5dm1s)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Sascha Grunert +* Vlad Gorodetsky (Bug Triage Shadow) +* Marko Mudrinić +* Mikael Johansson (Docs Shadow) +* Daniel Mangum +* Rey Lejano +* Jim Angel +* Jeremy Rickard +* Taylor Dolezal +* Nabarun Pal +* Wilson Husin (Release Notes Shadow) +* Gianluca Arbezzano (Bug Triage Lead) +* Fei Yang +* Adolfo García Veytia +* Carlos Panato + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Kubernetes 1.19 release schedule update:[ https://github.com/kubernetes/sig-release/blob/master/releases/release-1.19/README.md](https://github.com/kubernetes/sig-release/blob/master/releases/release-1.19/README.md) + * **Monday, April 13**: Week 1 - Release cycle begins + * **Tuesday, May 19**: Week 6 -[ Enhancements Freeze](https://github.com/kubernetes/sig-release/blob/master/releases/release_phases.md#enhancements-freeze) + * **Thursday, June 25**: Week 11 -[ Code Freeze](https://github.com/kubernetes/sig-release/blob/master/releases/release_phases.md#code-freeze) + * **Thursday, July 9**: Week 14 - Docs must be completed and reviewed + * **Tuesday, August 4**: Week 17 - Kubernetes v1.19.0 released + * **Thursday, August 20**: Week 19 - Release Retrospective +* Kubernetes 1.15 release this week:[ https://github.com/kubernetes/sig-release/issues/1051](https://github.com/kubernetes/sig-release/issues/1051) +* kube-cross variant builds:[ https://github.com/kubernetes/release/pull/1258](https://github.com/kubernetes/release/pull/1258) +* Release Team meetings moving to bi-weekly until June 1st. + + +## April 21, 2020 ([recording](https://youtu.be/6I3tetpOlK8)) + +**Host:** Stephen Augustus + +**Note Taker**: Tim Pepper + +**Attendees:** + + + +* Marky Jackson +* Gianluca Arbezzano +* Jim Angel +* Jorge Alarcon +* Jordan Liggitt +* Vlad Gorodetsky +* Sascha Grunert +* John Belamaric +* Wilson Husin +* Bart Smykla +* Rey Lejano +* Mo Khan +* Daniel Mangum +* Bob Killen +* Taylor Dolezal +* Adolfo García Veytia +* Nikhita Raghunath +* Jeremy Rickard +* Tim Pepper +* Verónica López + +**Open Discussion [timebox to N min]:** + + + +* **Discuss changes to the release timeline** + * Community announcement:[ https://groups.google.com/d/topic/kubernetes-dev/IVpiIOZ4WcM/discussion](https://groups.google.com/d/topic/kubernetes-dev/IVpiIOZ4WcM/discussion) + * PR:[ https://github.com/kubernetes/sig-release/pull/1058](https://github.com/kubernetes/sig-release/pull/1058) + * Could we make a declaration of no removal of deprecated features, or even no feature alpha/beta/ga promotions? + * How much risk do we have unmanaged today in a release cycle? How much risk accumulates during an elongated release cycle? + * Two dimensions of activity and required resources need consideration: + * Developer activity and inability/ability to make forward progress + * Consumer complexity in a new release (especially the changelog section list “must do before upgrade”...how can we drive it to be empty this cycle) + * Desires are: + * less human toil in our project’s developers + * less new feature quality risk and at least maintained quality: + * Can we get SIGs to decrease/defer any major feature work? + * Can we get SIGs to shift some feature focus to test enhancing focus? + * Action Required label (“release-note-action-required”) is a sign post we could key off (potentially requiring late reverts) + * less end-user/consumer toil in adopting releases right now: + * Adding a supported release has generally been accepted as doable and gives some breathing room to consumers (although they may need more now) + * WG LTS has been looking at this problem already, but consumers are likely now in a worse position than they’d been in during the WG’s discussions + * KEP[ https://github.com/kubernetes/enhancements/pull/1497](https://github.com/kubernetes/enhancements/pull/1497) likely to be proposed this week as “implementable” with lazy consensus deadline to merge + * Contributor experience impact: + * Need to be cognizant of new barriers to entry now are piled on top of an already difficult contributor experience for first time contribution in the project. + * Even some existing contributors are seeing notable increased activity and developer efficiency because crisis drives focus for them. + * What about contributors whose sole job is to deliver some content in k8s and they are focused on it and it brings them joy and now they’re not allowed by the community to do it? + * How much has a longer than three weeks elongation been discussed? + * 4, 5, 6 months for 1.19? + * If we push 1.19 a bit more to +6weeks-ish, then 1.20 would be the last release for 2020 and perhaps then go back to normal in 2021 + * Or if we keep 1.19 only slightly elongated, then a normal 1.20, and then a tiny December 1.21 “stability release”. [ What is a “stability release”?](https://github.com/kubernetes/sig-release/issues/809) + * Schedule options + * + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
+Event + +Date (normal?) + +Date (+3 weeks) + +Date (+6 weeks) + +Date (+8 weeks) +
+

+Enhancements freeze

+ + 		+May 5 + +May 12 + +May 19 + +May 19 +
+

+Code freeze

+ + 		+June 11 + +June 9 + +June 18 + +June 18 +
+

+Cherry pick deadline

+ + 		+June 25 + +June 25 + +July 31 + +July 30 +
+

+Release target

+ + 		+June 30 + +July 21 + +August 18 + +August 25 +
+Consequences + +Human pressure + +
    + +
  • \ +Still has enhancements really soon + +
  • Very short gap from enhancements to code freeze +
    • +
+ 		+
    + +
  • \ +Get 1.20 out in December + +
  • Encourage/require feature branch development + +
  • Shortness between enhancements freeze and code freeze means only smaller, done-er things make this release, others continue dev for 1.20 + +
  • [Stephen] Post-meeting update +
      + +
    • This happens immediately after KubeCon +
      • +
    +
    • +
+ 		+
    + +
  • \ +Get 1.20 out in DecemberEncourage/require feature branch development + +
  • Shortness between enhancements freeze and code freeze means only smaller, done-er things make this release, others continue dev for 1.20 + +
  • [Stephen] Post-meeting update +
      + +
    • Longer Code Freeze + +
    • Adds a release blackout period of two weeks to compensate for KubeCon +
      • +
    +
    • +
+
+ + + + +* +* + + +## April 7, 2020 ([recording](https://youtu.be/DEa9_2suWUI)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Daniel Mangum +* Jorge +* Marky Jackson +* Sascha Grunert +* Taylor Dolezal +* Joseph Sandoval +* Rey Lejano +* Bob Killen + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New[ kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Kubernetes 1.19 discussion + * Kicking off the 1.19 release: this is how we do it 😄 + * [https://github.com/kubernetes/sig-release/pull/1043](https://github.com/kubernetes/sig-release/pull/1043) + * Should we delay? + * Doesn’t make sense due to holiday conflicts with timelines + * Release to start officially 4/13 + * [Stephen] Focus doc for the quarter + * kubeadm OOT KEP + * [https://github.com/kubernetes/enhancements/pull/1425](https://github.com/kubernetes/enhancements/pull/1425) + * LTS support KEP + * [https://github.com/kubernetes/enhancements/pull/1497](https://github.com/kubernetes/enhancements/pull/1497) + * KEPs are now in a directory structure:[ https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/617-improve-kep-implementation](https://github.com/kubernetes/enhancements/tree/master/keps/sig-architecture/617-improve-kep-implementation) + * Removing fast-forwards in the 1.19 release + * Do we need them? + * Want to decrease the amount of alphas, do more betas, add one more RC to the cycle + * Issue Triage Workflow and Automation + * [https://github.com/kubernetes/enhancements/pull/1554](https://github.com/kubernetes/enhancements/pull/1554) + + +## March 23, 2020 ([recording](https://youtu.be/utNRJXDOMz0)) + +**Host:** Tim Pepper + +**Note Taker**: TBD + +**Attendees:** + + + +* Bob Killen +* Jeremy Rickard +* Josh Berkus +* Daniel Mangum +* Bart Smykla +* Jorge Alarcon +* Taylor Dolezal +* Joseph Sandoval +* Sascha Grunert +* Adolfo Garcia Veytia +* Rey Lejano + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees + * Joseph Sandoval from Adobe +* 1.19 Release Lead Intro (Taylor) + * Release team lead selection is open and happening, see GitHub: + * [https://github.com/kubernetes/sig-release/issues/1031](https://github.com/kubernetes/sig-release/issues/1031) (Assemble 1.19 Release Team) + * Full list of 1.19 Milestone Issues:[ https://github.com/kubernetes/sig-release/milestone/14](https://github.com/kubernetes/sig-release/milestone/14) +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)): stalled + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Branch management: in the past we’ve created the release branch very early with mostly manual git-fast-forward. + * Reasons: + * Gives extra CI signal (redundant test of same commits on different branches) + * extra eyes on code coming from master to release branch (ie: look at all merged commits and affirm they should technically go to the release branch) + * practice operationally the tooling and processes around shifting content from master to release branch + * Upgrade test variations + * Can we create the branch later? Yes. + * How much later? Code thaw is too late. Code freeze might be sufficient + * Issue to discuss further:[ https://github.com/kubernetes/release/issues/1207](https://github.com/kubernetes/release/issues/1207) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 1.18: last week was potentially looking to delay a few days, but looking good as the week starts + * 1.19: team forming aiming for around first week of April + * WG-LTS + * KEP still in discussion at[ https://github.com/kubernetes/enhancements/pull/1497](https://github.com/kubernetes/enhancements/pull/1497) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * Promote KinD tests to blocking[ https://github.com/kubernetes/kubeadm/issues/1599](https://github.com/kubernetes/kubeadm/issues/1599): + * Is there general consensus to move this from informing to blocking? Informing board tests are already intended as just as important as blocking, but in informing as recognition of the test not being perfectly stable. + * Useful signal + * Flakes exist occasionally, usually around bringing up the test env (docker copy fails) + * SIG Cluster Lifecycle is willing to try to support as blocking, but also limited hands and the flake debug is complex. Lubomir’s preference is to get to bottom of the actual failure and keep it in informing until then. + * Image management (eg:[ https://github.com/kubernetes/kubernetes/issues/70249](https://github.com/kubernetes/kubernetes/issues/70249),[ https://github.com/kubernetes/kubernetes/issues/75114](https://github.com/kubernetes/kubernetes/issues/75114) and others) → next rel eng meeting topic + * CI Signal subproject starting up needs update, relates to + * [https://github.com/kubernetes/sig-release/issues/966](https://github.com/kubernetes/sig-release/issues/966) + * [https://github.com/kubernetes/community/issues/3455](https://github.com/kubernetes/community/issues/3455) + * [https://github.com/kubernetes/enhancements/pull/1554](https://github.com/kubernetes/enhancements/pull/1554) + * [https://github.com/kubernetes/enhancements/issues/1553](https://github.com/kubernetes/enhancements/issues/1553) + * [https://github.com/kubernetes/community/issues/3456](https://github.com/kubernetes/community/issues/3456) + * Golang related dep management topics: + * toolchain version:[ https://github.com/kubernetes/release/issues/1139](https://github.com/kubernetes/release/issues/1139) + * Module deps:[ https://github.com/kubernetes/kubernetes/issues/82531](https://github.com/kubernetes/kubernetes/issues/82531) + +**Open Discussion [timebox to N min]:** + + + +* New Doodle for SIG Release / Releng meeting times:[ https://doodle.com/poll/yp3wrghih6r4td65](https://doodle.com/poll/yp3wrghih6r4td65) +* What qualifies for a milestone, cherry-picking (Taylor) -[ https://github.com/kubernetes/community/pull/4628](https://github.com/kubernetes/community/pull/4628) + + +## March 9, 2020 ([recording](https://youtu.be/jcZJPvO4AH4)) + +**Host:** Stephen Augustus + +**Note Taker**: Tim Pepper + +**Attendees:** + + + +* Taylor Dolezal +* Sascha Grunert +* Rey Lejano +* Marko Mudrinić +* Bart Smykla +* Daniel Mangum +* Tim Pepper +* Jorge Alarcon +* Lubomir Ivanov +* + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Establish Kubernetes build/stage/release process on k8s-infra (@bsmykla): + * [https://github.com/kubernetes/release/issues/911](https://github.com/kubernetes/release/issues/911) +* Annual support cycle KEP (tpepper):[ https://github.com/kubernetes/enhancements/pull/1497](https://github.com/kubernetes/enhancements/pull/1497) + * SIG Release has been involved in discussion, but lacking Contrib Summit session at KubeCon EU 2020, WG LTS wants to explicitly reach out again regarding: + * Are there specific costs/efforts for SIG Release not captured in the proposal? + * Readiness? Could this be implementable from SIG Release’s perspective? + * In Tim St. Clair’s alternate suggestion comment on the KEP, if doing three by four month releases with three releases supported at a time still yielding 12 months of patch support, can we get release team volunteers for 16 weeks instead of the current 12 week commitment? + * Tools supportability/maintainability complexities: + * Order of operations + * Tagging, branching of k/release and related k/* repos +* Removing debian-* images?[ https://github.com/kubernetes/kubernetes/pull/80909](https://github.com/kubernetes/kubernetes/pull/80909) + * Still looking at this; will probably be a 1.19 endeavor +* Updates on [RFC] CI Signal subproject[ https://github.com/kubernetes/sig-release/issues/966](https://github.com/kubernetes/sig-release/issues/966). What can we do to move forward? (alejandrox1) + * [Stephen] I need to strip contentious parts of the proposal around scope and have defining scope be the first task of this group + + +## February 24, 2020 ([recording](https://youtu.be/HsCtsPlQ2Pk)) + +**Host:** Tim Pepper + +**Note Taker**: Bob Killen + +**Attendees:** + + + +* Sascha Grunert +* Bob Killen +* Daniel Mangum +* Marko Mudrinić +* Stephen Augustus +* Tim Pepper +* Bart Smykla +* Gianluca Arbezzano +* Jim Angel +* Augustas Verbickas +* Adolfo García Veytia +* Joseph Sandoval +* Marky Jackson + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Potentially looking for new owners / contributors + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Krel](https://github.com/kubernetes/release/tree/master/cmd/krel) + * Lots of effort this past cycle + * Successfully using krel ff. And also a PR to remove branchff. Either way, for discussion of why there is an ff, see last week:[ https://kubernetes.slack.com/archives/CJH2GBF7Y/p1582216315166800](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1582216315166800) + * Incrementally switching over to krel for more components + * Right now replicating functionality (bash -> golang) versus enhancing, simplifying, rearchitecting + * Potentially rename gcbmanager command, it was named gcbmanager to mirror the current command. + * Gcb components are difficult to mock and test locally. + * Might render gcbtemplate locally in the future for local cloud build + * Some of the thoughts on the development of krel and future big picture changes -[ https://docs.google.com/document/d/1Js_36K51Q6AjEsVUjRBMISTA4D7cnjZmoSkn43_Jmxo/edit?usp=sharing](https://docs.google.com/document/d/1Js_36K51Q6AjEsVUjRBMISTA4D7cnjZmoSkn43_Jmxo/edit?usp=sharing) + * Need to determine “end goal” for krel / tooling + * Initial goal: + * non-googlers being able to publish a release (publishing images, publishing/signing debs/rpms) + * More maintainable implementation (more people involved, ability to do local dev/test w/o priv’s, smaller units, units with tests in CI) + * WG K8s Infra: What are plans from Release which need new/different infra? When are they needed? What are the artifacts to be delivered? Let’s create an issue describing this with a checklist of the 2020 deliverables. Also these two exist: + * “Can we do release on the new infra?” This is the TOP priority.[ https://github.com/kubernetes/release/issues/911](https://github.com/kubernetes/release/issues/911) ← needs input from WG K8s Infra + * [https://github.com/kubernetes/release/issues/964](https://github.com/kubernetes/release/issues/964) + * Release Managers + * hyperkube + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* KEP Updates for Graduated Enhancements (Jeremy Rickard) +* k/enhancement issues per “stage” (Jeremy Rickard) + * 1 KEP for the entire lifecycle of the Enhancement + * Should be addressed by:[ https://github.com/kubernetes/enhancements/pull/1545](https://github.com/kubernetes/enhancements/pull/1545) +* SIG Release related KEPs:[ https://github.com/kubernetes/enhancements/pulls?q=is%3Aopen+is%3Apr+label%3Asig%2Frelease](https://github.com/kubernetes/enhancements/pulls?q=is%3Aopen+is%3Apr+label%3Asig%2Frelease) + * Plus revisions to the KEP process:[ https://github.com/kubernetes/enhancements/pull/1545](https://github.com/kubernetes/enhancements/pull/1545) + * Conformance documentation for k8s releases: + * [https://github.com/kubernetes/enhancements/pull/717](https://github.com/kubernetes/enhancements/pull/717) + * Stale (1yr+), sig-testing owned + * Kubeadm split out of k/k: + * [https://github.com/kubernetes/enhancements/pull/1425](https://github.com/kubernetes/enhancements/pull/1425) + * Quiet since a month ago, not everybody in sync, but deferred from 1.18 to 1.19 + * What tools changes needed? Lubomir’s looking at changing the proposal in a way that would not require changes in anago. Experimenting with tools around branching and tag management. Need to discuss more with Release to see how this can blend with where krel is going. + * Dims: are these changes just to make it easier relative to release tooling today? Or does it represent that way we want things to be for all things that split out of k/k? We should think broadly about how to support not just core + kubeadm, but core + any of the things we expect to split out. Need a shared concept of what should be the release tooling architecture. +* + + +## February 10, 2020 ([recording](https://youtu.be/yZM4NOsuAJU)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Sascha Grunert +* Carlos Panato +* Bob Killen +* Jim Angel +* James Laverack +* Tim Pepper +* Joseph Sandoval +* Taylor Dolezal +* Daniel Mangum +* Bart Smykla +* Marko Mudrinić +* Adolfo García Veytia + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Managers + * hyperkube + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* New repo plugins + * [https://github.com/kubernetes/sig-release/pull/980](https://github.com/kubernetes/sig-release/pull/980) + * [https://github.com/kubernetes/release/pull/1063](https://github.com/kubernetes/release/pull/1063) + * [https://github.com/kubernetes/test-infra/pull/16161](https://github.com/kubernetes/test-infra/pull/16161) +* Note on commit messages +* Change to generate updates to[ https://relnotes.k8s.io/](https://relnotes.k8s.io/) as part of the krel tool.[ https://github.com/kubernetes/release/issues/1061](https://github.com/kubernetes/release/issues/1061) @james.laverack, @puerco +* Progress report on krel (Sascha, Daniel) \ +[https://github.com/kubernetes/release/issues/918](https://github.com/kubernetes/release/issues/918) +* krel gcbmgr in progress:[ https://github.com/kubernetes/release/pull/1081](https://github.com/kubernetes/release/pull/1081) +* k/k image building (Hannes):[ https://github.com/kubernetes/kubernetes/pull/80909#issuecomment-574230358](https://github.com/kubernetes/kubernetes/pull/80909#issuecomment-574230358) +* Add a 'kind/test' category for release milestone PRs -[ https://github.com/kubernetes/kubernetes/issues/65180](https://github.com/kubernetes/kubernetes/issues/65180) (Bob Killen) +* Rename release-note-* and cherrypick- labels to conform to category/sub-category norm -[ https://github.com/kubernetes/kubernetes/issues/24057](https://github.com/kubernetes/kubernetes/issues/24057) (Bob Killen) +* Add your suggested topic here and move this to the line below [firstname lastname, email or @slackname] + + +--- + + +## January 27, 2020 ([recording](https://www.youtube.com/watch?v=ChpG7h-nhnA&list=PL69nYSiGNLP3QKkOsDsO6A0Y1rhgP84iZ&index=6&t=0s)) + +**Host:** Tim Pepper + +**Note Taker**: Tim Pepper + +**Attendees:** + + + +* Sascha Grunert +* Daniel Mangum +* Hannes Hoerl +* Marko Mudrinić +* Taylor Dolezal +* Jorge Alarcon +* Adolfo García Veytia +* Savitha Raghunathan + +**Recurring Topics [timebox to N min]:** + + + +* Subproject updates + * Licensing: No representative + * Release Engineering + * Anago issues Dec./Jan.: + * Associative array traversed incorrectly, minimally worked around for now. + * Could simplify the top level control logic in Anago. Might want to / need to for Kubeadm KEP. Somewhat prefer not to given complexity to debug, and intent to replace anago ([https://github.com/kubernetes/release/issues/918](https://github.com/kubernetes/release/issues/918)). + * Kubeadm KEP: + * [https://github.com/kubernetes/enhancements/pull/1425](https://github.com/kubernetes/enhancements/pull/1425) Please read/comment + * Could modify Anago to support also branching, tagging, building k/kubeadm + * Could modify k/k’s build to pull in k/kubeadm, build, output to the expected places; have a job to + * Package building: + * When do we start using it? + * Is getting close to ready + * Most tools today have presumed github.com/kubernetes, specific repos, and pre-defined taxonomy of branches (master, release-1.YY) + * How to transfer off anago: + * Rewrite from scratch? Easier and cleaner for us. Hard to know the replacement is truly ready. This path has much easier testing, but does require having shadow repositories + * Modify anago piecemeal, eg: remove old changelog generator, call new one. + * Tagging: today we tag, then build changelog, then commit changelog. The changelog is not “a part of the release” if we build from the release tag. Build from HEAD, or some other commit(s) just after the tag? Change the version tagging scheme? + * K/kubeadm repo is looking (see KEP above) to add a prow post-submit job to mirror k/k repo branches and tags, without consideration for build reproducibility. + * Lubomir proposes not having changelogs in subsidiary repos (eg: kubeadm). SIG Release’s tool acts as an aggregator, puts changelogs from a list of repos in a single place such as k/k. This would also give common style to “the Kubernetes project changelog”. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* Reminder [RFC] CI Signal subproject[ https://github.com/kubernetes/sig-release/issues/966](https://github.com/kubernetes/sig-release/issues/966) + + +## January 13, 2020 ([recording](https://youtu.be/fCrxWgkZGls)) + +**Host:** Stephen Augustus + +**Note Taker**: TBD + +**Attendees:** + + + +* Jorge Alarcon +* Taylor Dolezal +* Marky Jackson +* Bob Killen +* Daniel Mangum +* Hannes Hoerl +* Marko Mudrinić +* Tim Pepper +* Gianluca Arbezzano +* Rey Lejano +* Thomas Wodarek +* Carlos Panato +* Arnaud Meukam +* Josh Berkus +* John Belamaric +* Maria Ntalla + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Managers + * hyperkube + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Testgrid dashboard review:[ https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* Project board review:[ https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + +**Open Discussion [timebox to N min]:** + + + +* New meeting times. Stephen to send Doodle this week. +* 1.18 async grooming. Happening over the next two weeks. +* ContribEx survey:[ https://www.surveymonkey.com/r/VYRJZ5G](https://www.surveymonkey.com/r/VYRJZ5G) +* kubeadm out-of-tree:[ https://github.com/kubernetes/enhancements/pull/1425](https://github.com/kubernetes/enhancements/pull/1425) +* kubepkg (Deb/rpm builder) in alpha:[ https://github.com/kubernetes/release/blob/master/cmd/kubepkg/README.md](https://github.com/kubernetes/release/blob/master/cmd/kubepkg/README.md) + * Image is already live, but image building PR needs merge:[ https://github.com/kubernetes/test-infra/pull/15871](https://github.com/kubernetes/test-infra/pull/15871) + * Initial PRs: + * [https://github.com/kubernetes/release/pull/935](https://github.com/kubernetes/release/pull/935) + * [https://github.com/kubernetes/release/pull/998](https://github.com/kubernetes/release/pull/998) +* Patch Releases tomorrow: + * Stephen and Tim to handle to be watching for errors from[ https://github.com/kubernetes/kubernetes/issues/86182](https://github.com/kubernetes/kubernetes/issues/86182) and[ https://github.com/kubernetes/release/issues/968#issuecomment-564792325](https://github.com/kubernetes/release/issues/968#issuecomment-564792325). + * The build/release code is supposed to do: + * tag v1.16.3 + * commit openapi-spec versioned files + * tag v1.16.4-beta.0 + * But it is instead doing: + * commit openapi-spec versioned files + * tag v1.16.5-beta.0 + * tag v1.16.4 + * By way of[ https://github.com/kubernetes/kubernetes/pull/84654](https://github.com/kubernetes/kubernetes/pull/84654) (and cherry picks onto each of the release branches), the openapi-spec commit is no longer needed. At that point it doesn’t matter if the tagging is out of order. + * We need to remove the build code bash that makes a commit on the openapi-spec. + * We still need to understand how a sequential, non-parallel series of bash commands is happening in a different order than shown in the code in the repo we think we’re building from. Is GCB doing something weird? Are we misunderstanding this bash? + * Patch releases will be: + * 1.17.1:[ https://github.com/kubernetes/sig-release/issues/948](https://github.com/kubernetes/sig-release/issues/948) + * 1.16.5:[ https://github.com/kubernetes/sig-release/issues/949](https://github.com/kubernetes/sig-release/issues/949) + * 1.15.8:[ https://github.com/kubernetes/sig-release/issues/950](https://github.com/kubernetes/sig-release/issues/950) + * 1.14.11?? May need to because 1.14.10 was one of the builds that had out of order tagging as[ https://github.com/kubernetes/kubernetes/commit/575467a0eaf3ca1f20eb86215b3bde40a5ae617a](https://github.com/kubernetes/kubernetes/commit/575467a0eaf3ca1f20eb86215b3bde40a5ae617a) was included in the 1.14.10 build marking it as 1.14.11 in the openapi-spec/swagger.json. + * Issues remain with the “marker files” used by test-infra. Test configs are relatively static, using a set of “channels” as described in[ https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md](https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md): + * channel branch + * dev master + * beta release-1.15 + * stable1 release-1.14 + * stable2 release-1.13 + * stable3 release-1.12 + * When we drop 1.12, stable3 is marked to refer not to release-1.12, but rather release-1.13. The release versions rotate relative to the channel marker names. The tests always use the same channel marker name. This leads to weird skews in test results where the results of stable3 in testgrid are non-continuous, but testgrid’s heuristic for “green” or “red” does not understand this. There is some caching and cache invalidation and it does not function correctly relative to how the release versions are rotated, and this very much confuses the task of CI signal analysis +* Creating CI Signal subproject + * It will soon be time to remove the 1.14 CI, ie:[ https://github.com/kubernetes/test-infra/pull/15875](https://github.com/kubernetes/test-infra/pull/15875) + * If CI Signal is a subproject we gain more continuity across the year, instead of rotating teams fairly aggressively in each release team. We can grow the set of folks watching issues to be more than just the smaller set of a lead and shadows in the release team + * Release team role is still needed as they report status into the release team in order to enable the go / no-go decisions on releasing. +* Tagging/branching (request for comments):[ https://github.com/kubernetes/release/issues/857](https://github.com/kubernetes/release/issues/857) + * Anago prototype:[ https://github.com/kubernetes/release/pull/957](https://github.com/kubernetes/release/pull/957) +* KubeCon EU SIG Release maintainers track Intro & Deep Dive + * Attendening Kubecon EU: + * Marky Jackson + * Gianluca Arbezzano + * Josh Berkus + * Marko Mudrinić + * Carlos Panato + * Tim Pepper + * Stephen Augustus + * Daniel Mangum + * SIG Intro: Sascha (and somebody additional?) do details of release tooling and process[ https://docs.google.com/document/d/1M4670sP4PxDi1tRf1AkYGBDvahnKGZekrERneaBDVs0/edit#](https://docs.google.com/document/d/1M4670sP4PxDi1tRf1AkYGBDvahnKGZekrERneaBDVs0/edit#) + * SIG Deep Dive: Do a pair of deep dives into two of our subprojects (eg: releng, CI signal), each 15-20 minutes, plus Q&A. + * Abstracts for these are due by Jan. 20. Caleb/Tim/Stephen as SIG chairs will collate info from above into specific abstract and speakers proposal then enter them into the conference tool ahead of the deadline. \ No newline at end of file diff --git a/sig-release/meeting-notes-archive/2021.md b/sig-release/meeting-notes-archive/2021.md new file mode 100644 index 000000000..c2338eba5 --- /dev/null +++ b/sig-release/meeting-notes-archive/2021.md @@ -0,0 +1,1759 @@ + +## December 14, 2021 - [Async meeting via Slack](https://kubernetes.slack.com/archives/C2C40FMNF/p1639493266140100) + +**Topics [timebox to 20 min]:** + + + +* [Rey] [Release team selection process](https://kubernetes.slack.com/archives/C2C40FMNF/p1639493281140400) - [GH discussion](https://github.com/kubernetes/sig-release/discussions/1714) + * Common themes as highlighted by James: + * Reduce the volume of shadows by attaching additional criteria, e.g., "need at least one contribution already". + * Implement a long-term application tracking process, and reach out to repeat applicants in some way. + * Maintain a stable release team "roster" from which release teams are built. Select fewer shadows to compliment "roster" folk each cycle, and give shadows the chance to join the "roster" after one or two cycles. + * Proactively increase diversity in applications, e.g., outreach, refine role handbooks, publicize success stories. + * Add video call interviews for shadow applicants. + + +## November 30, 2021 (recording) + +**Host:** Sascha Grunert + +**Attendees (pronouns):** + + + +* Nabarun Pal (he/him) +* Rey Lejano (he/him) +* James Laverack (he/him) +* Meha Bhalodiya (she/her) +* Jim Angel (he/him) +* Adolfo García Veytia (he/him/él) +* Verónica López (she/her) + +**Note Taker:** + + + +* Rey Lejano + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Not many changes recently, last big one is SPDX licensing and moving BOM to its own repo. We ran first release pulling from that repo, things are going well. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Rey] Release 1.23 status + * Currently we have a failing job, gce-cos-k8s-beta-default in sig-release-1.23-blocking. An [issue](https://github.com/kubernetes/kubernetes/issues/106694) is open and an open [PR in k/test-infra](https://github.com/kubernetes/test-infra/pull/24490) +* Roadmap and Vision update: \ +[https://groups.google.com/g/kubernetes-dev/c/0vagxffQ6j4/m/s65AfYNFBgAJ](https://groups.google.com/g/kubernetes-dev/c/0vagxffQ6j4/m/s65AfYNFBgAJ) + * SLSA compliance: [https://github.com/kubernetes/enhancements/pull/3051](https://github.com/kubernetes/enhancements/pull/3051) + * Signing release artifacts: [https://github.com/kubernetes/enhancements/pull/3061](https://github.com/kubernetes/enhancements/pull/3061) + * [Sascha] Goal is to merge both KEPs before Christmas, encourage reviews and need support from the whole SIG. Next goal is to integrate signing of release artifacts, not completely clear how we will do that, we will experiment with container image signing + * [Adolfo] SLSA KEP received comments from Tim and will look and address. Lagging a little behind with signing artifacts since the initial technologies that were initially talked about are no longer in use + * [Veronica] Familiar with SLSA compliance framework, not sure where this is at, at this point, does the main effort on SIG Release? + * [Adolfo] After a few discussions and trials in code, suggesting as a SIG to use cosign and implement SLSA framework. Have had discussions and see consensus see most momentum with cosign and SLSA. The KEPs are open for comments and suggestions. This is a decision which technologies to use and framework we’re adopting. As a SIG we’ll own the implementation. + * [Sascha] SIG Release is main owner, maybe SIG Testing with technical details + * [Adolfo] Asked feedback from SIG Security but SIG Release will be driving the KEPs + * [Sascha] We can carve out follow-ups to the KEPs + * [James] Asking for a call to action + * [Sascha] First thing is to merge KEPs +* [Marko, Nabarun, Adolfo] 1.23 merges + cherry picks + * [Adolfo] Background, before 1.19 to make branches even, a release manager would do a ff. In 1.19 a decision was made to stop doing ff when PRs didn’t have proper checks and milestones. So in 1.19 we require PR authors to file a cherry-pick besides their feature PR. Today when submitting a PR, submit a cherry-pick to the release branch e.g. release-1.23. The decision was not well communicated. For 1.21 and 1.22, people did not know this process existed. We expect PR authors to follow with cherry-picks, even now in 1.23 we see some confusion. + * [Veronica] Saw that 1.21 and 1.22, people were unaware of the change. This topic was set for a bigger discussion. We would like some input from the release team + * [Nabarun] Just checked the master branch, there were 3 PRs merged, we can do the cherry-picks ourselves for 1.23 as we did this for 1.22 and 1.21. + * [James] What’s the history + * [Sascha] We want to reduce tooling effort, minimize time between Code Freeze and Thaw (down to 2 weeks), and create branch with RC. Benefits is a short period where k/k master is frozen. Same process as cherry-picking to a release-branch for a patch release. With FFs we have to maintain tools and documentation. For now, we can watch the master branch merges and we can ask PR authors to make cherry-pick PRs or we make them manually. + * [James] Are we trying to reducing Code Freeze more? + * [Sascha] Idea is not to block the master branch, PR authors would have to cherry-pick + * [Adolfo] In 1.22, when some objections were raised about the cherry-pick requirements, Adolfo looked at the history of the decision and sent an email to the community \ +[https://groups.google.com/g/kubernetes-dev/c/NiKByLbWZ1g/m/GYhi_Ip3BAAJ](https://groups.google.com/g/kubernetes-dev/c/NiKByLbWZ1g/m/GYhi_Ip3BAAJ) + * [Adolfo] For 1.24, we should establish a roadmap. It’s not currently in the Branch Manager handbook + * [Veronica] Should we continue with cherry-picking? + * [Marko] Discussion with Jordan, cherry-picking is a manual step and can be forgotten. + * [Sascha] Doing cherry-picks is more manual process + * [Sacha] For now, we can’t change the process, we can write another summary to k-dev and sig-release and the Branch Managers are watching what is merged to master branch and PR authors should make a corresponding cherry-pick PR +* Publishing bot issue: [https://github.com/kubernetes/release/issues/2337](https://github.com/kubernetes/release/issues/2337) + * [Marko] Tags are missing for 1.23.0-beta.0 and 1.23.0-rc.0. Publishing bot failed to add new tag. Reason is that the master branch for staging repos like kubectl/client-go is missing a commit. Publishing bot does not know how to handle the tree when we build when we push the release. In June, we changed the process and don’t rebase instead we do merge commits. Others (Dims, Nikhita, et. al.) are trying to fix this and trying to push the missing commits. They suggest to us that we should look at the process with merge commits. + * [Sascha] Don’t understand what caused the missing commit. Any issues with the release? Did we have to retry? + * [Marko] Not sure and have to check + * [Adolfo] Why did it happen on the beta release? + * [Marko] We probably had many commits and it triggered this behavior, we can investigate more + * [Nabarun] When the publishing bot runs, it found commit from where 2 different branches forked. We’re not using the parent master as the parent. When we push our changes, instead of pushing to master, we can raise a PR so it goes through usual work flow. So it has the right parent commit and the publishing bot has the right parent commit. + * [Sascha] Agree that the publishing bot doesn’t see the right parent commit, creating a PR is a good process. The release tag should land on master after the release is done + * [Marko] We shouldn’t have a problem with release branches, master is an issue since we do multiple releases with multiple patch releases. The PR idea is a probably a good one + * [Sascha] Let’s follow up on the issue + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## November 02, 2021 (recording) + +**Host: Jeremy Rickard** + +**Attendees (pronouns):** + + + +* Sascha Grunert (he/him) +* Rey Lejano (he/him) +* Verónica López (she/her) +* Nabarun Pal (he/him) +* Priyanka Saggu (she/her) +* Eddie Zaneski (he/him) + +**Note Taker:** + + + +* Rey Lejano + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Cut Today + * Jim will be doing it + * Adolfo did a PR for access + * Will start soonish + * CI Signal looks _ok_ could be better but OK for cutting + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Reminder: 1.23 Mid Cycle Retro Tomorrow During 1.23 RT Meeting + * Rey will send a k-dev reminder + * Alpha.4 today, feature blog freeze today + * Nov 16th - code freeze + * +* Release team selection process [GitHub Discussion 1714](https://github.com/kubernetes/sig-release/discussions/1714) + * Should have a few actionable items + * Summarize high level points in the discussion and send to the SIG Release mailing list +* Mid-cycle shadow surveys + * About 50% shadows completed the survey + * Will add several relevant topics to the release retro +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## October 19, 2021 (recording) + +**Host: **Adolfo García Veytia (@puerco) + +**Attendees (pronouns):** + + + +* Rey Lejano (he/him) +* Shivanshu Raj Shrivastava (he/him) +* Marko Mudrinić (he/him) +* Arsh Sharma (he/him) +* Nabarun Pal (he/him) +* Pravar Agrawal (he/him) +* Eddie Zaneski (he/him) +* Joseph Sandoval (he/him/él) +* Sascha Grunert (he/him) +* Meha Bhalodiya (she/her) + +**Note Taker:** + + + +* + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Pushing release process to be SLSA compliant to make supply chain more secure. Currently under testing to be SLSA level 0. Can build proper technical documentation, no changes or features. Kicking off this week new KEPs, one KEP to discuss implement changes for SLSA + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Rey] Upcoming milestones + * First retro is coming up on Nov 3 + * Feature blog freeze Nov 2 + * 1.23.0-alpha.4 scheduled to be cut on Nov 2 + * Code Freeze starts Nov 16 + * 1.23.0-beta.0 scheduled to be cut on Nov 17 + * Docs Placeholder PR deadline on Nov 18 +* CI Signal Report tool repo migration [@arsh] + * Slack thread - [https://kubernetes.slack.com/archives/C2C40FMNF/p1634138505387200?thread_ts=1622718810.193000&cid=C2C40FMNF](https://kubernetes.slack.com/archives/C2C40FMNF/p1634138505387200?thread_ts=1622718810.193000&cid=C2C40FMNF) + * [https://github.com/kubernetes/org/issues/3044](https://github.com/kubernetes/org/issues/3044) + * Having functionality built into krel (possibility) +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* Discussion around improving the shadow selection process: \ +[https://github.com/kubernetes/sig-release/discussions/1714](https://github.com/kubernetes/sig-release/discussions/1714) + * Need to dig into surveys + * Define a clear path and expectation for release team members + * Attract more diversity + * Make space inclusivity explicitly inclusive +* Future possibilities around the SBOM tool (Adolfo) + * [Adolfo] Approached by Open SSF and proposed to host tool with them + * [Adolfo] LF has SBOM generator that overlaps with our tool, talk to merge with LF tool and SPDX libraries. But also talks that LF should not have a tool and should have a standard + * [Adolfo] Presenting the tool in early Nov to Open SSF + * [Adolfo] Our tool only analyzes Go, LF’s tool can analyze 17 languages +* Discussion on optimal start of enhancements freeze [https://github.com/kubernetes/sig-release/discussions/1680](https://github.com/kubernetes/sig-release/discussions/1680) +* Discussion on optimal length of code freeze [https://github.com/kubernetes/sig-release/discussions/1674](https://github.com/kubernetes/sig-release/discussions/1674) +* KEP stages and transitions [https://github.com/kubernetes/enhancements/issues/2960](https://github.com/kubernetes/enhancements/issues/2960) + + +## October 5th, 2021 + +**Host:** Jeremy Rickard + +**Attendees (pronouns):** + + + +* Rey Lejano (he/him) +* Arsh Sharma (he/him) +* Joseph Sandoval (he/him) +* Sascha Grunert (he/him) +* Nabarun Pal (he/him) +* Appu Goundan (he/him) +* Stephen Augustus (he/him) +* Chun Wu (he/him) +* Pravar (he/him) + +**Note Taker: **Joseph Sandoval + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees: + * Appu Goundan (loosebazooka)/Google software supply security. + * Works on building audit trails with build and releases. +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)): (Adolfo) Push 1st stage of SLSA compliance. The goal is to get the release process compliant. Switch from staging to release there is provenance. For reference: [slsa-framework](https://github.com/slsa-framework/slsa). Investigation into ways to add the metadata into the container releases. + * PR for stage SLSA provenance metadata [https://github.com/kubernetes/release/pull/2273](https://github.com/kubernetes/release/pull/2273) + * SLSA Level 1 umbrella issue: + + [https://github.com/kubernetes/release/issues/2267](https://github.com/kubernetes/release/issues/2267) + + * (Stephen) Image promotion might be broken. Will investigate after the meeting. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)): (Rey)Week 7 of the release cycle. 1.23 has a 12/7 target. Holiday season is upon us and we have milestones to work around those dates. Open PR’s for handbooks and Rey will be tagging for reviews. (Stephen) GH discussion around the release team altogether and the selection process. [https://github.com/kubernetes/sig-release/discussions/1714](https://github.com/kubernetes/sig-release/discussions/1714) + * Upcoming mid-cycle release team retro coming up shortly. +* Follow up to Releasing and versioning kubectl independently + * Discuss what coordination is needed + * Discuss where things will go + * Who will have access to repos + * Draw inspiration from Kops? + * Kubeadm issue: [https://github.com/kubernetes/enhancements/issues/1424](https://github.com/kubernetes/enhancements/issues/1424) (Nabarun) How do we build Kubeadm after it's out of tree? How do we know how to build the binary? Small KEP to move out code generator ([https://github.com/kubernetes/enhancements/pull/1252/files](https://github.com/kubernetes/enhancements/pull/1252/files)) . It raised the issue “what is versioning schema?”. Now if they move to a new tagging schema. ([https://docs.google.com/presentation/d/1Utf8NgLZTrS8FmVoU6dZN3siewjGOwvTmFcsJ6vgQLI/edit#slide=id.p](https://docs.google.com/presentation/d/1Utf8NgLZTrS8FmVoU6dZN3siewjGOwvTmFcsJ6vgQLI/edit#slide=id.p)) Currently we have our own shell scripts and localized knowledge of the process. Move out the business logic but leave the entry point. Will that be ok? We move all traces of source code. What version, how do want to build? After Kubectl then expect Kubeadm to be next. How do we ensure the bump we do passes all the tests? How do we run the tests? + * [Stephen] Probably time to make decisions about path we go. One state could be a build system where we input shas and that is your release. Should think about the value of only partially removing (i.e. leaving cmd/ in k/k?). + * [Stephen] What do we do for last mile of the release - binaries we are producing don’t require container images, but require debs/RPMs (kubectl and kubeadm), which means they need to be in the same spot-ish or play nicely with packaging tools. Not difficult, just pointing to multiple repos. + * [Stephen] versioning - probably easiest to break away from k/k versioning. Look at cluster autoscaler azure? + * [Stephen] then the question: how do we build it. Each repo could have own build scripts, but things should get to same place via artifact promo or into same staging bucket. Who is in control - is it shared? We don’t want to own repos we don’t need to. Where build happens should be toward portion we control. Cri tools/etcd/cni plugins => examples where repo external to project but we consume staged artifacts and control to make sure build is promoted, maintaining that pattern seems reasonable. Prow has support for skew testing already, we can work matrix + * [Nabarun] Q: SIG owns code, do we own build scripts or do we own a contract they follow? + * [Stephen] We own contracts - produce artifacts, outside of pre-submits should have CI jobs running that represent the expected state (e.g. expected state is to be in a bucket and test should see if it’s in a bucket) + * [Nabarun] Eventually we won’t have the code (for kubectl) + * [Stephen] We ensure the artifact ends up somewhere (e.g. in a bucket) and a subsequent part of release that picks it up from a staging bucket to a production bucket. We remove the need to vendor it in the core code. Only care that the artifacts get to where they need to. Today there is no explicit requirement for kubectl to be released when core components are released. + * [Stephen] Better that out-of-tree components don’t follow release cadence + * [Nabarun] Will SIG Release help them or how do they get help from us + * [Stephen] Artifact promotion - [https://github.com/kubernetes/k8s.io/tree/main/artifacts](https://github.com/kubernetes/k8s.io/tree/main/artifacts). This link should be enough for people to start on artifact promotion - used by kops + * [Jeremy] Good opportunity to help SIG CLI and volunteer to help + * [Stephen] Give cleanup of file promotion bits around validation to Jeremy from Adolfo + * [Stephen] Would like to get some baseline validation in, start with repos that we co-own. Would like to prove the process with cri tools and cni plugins (had issues defining their release process) and pull releases and move to staging buckets and issue promotion PRs like how we do for krel promote-images. As we prove this out for repos that we need for the release, then roll out to others + * [Stephen] As we’re heading to KubeCon, next release engineering meeting is cancelled. Nabarun is sort-of point for kubectl out-of-tree and suggest to take notes to Q & A, and we can build a timeline. + * [Nabarun] We will start a kep on this and formalize thoughts. Lubomir also had thoughts because kubeadm is also up to be moved out-of-tree. + * [Stephen] We should do one first (kubectl) and apply practices to the next one (kubeadm) + * [Stephen] Suggest to publish Q & A before the kep and use it for the kep +* [Adolfo] Release cherry-pick deadline falls in KubeCon week, should we send advance notice or move it +* [Jeremy] Bump it for a week and send notice + + +## September 21st, 2021 [Recording](https://www.youtube.com/watch?v=iFImVu-QvUc&list=PL69nYSiGNLP3QKkOsDsO6A0Y1rhgP84iZ&index=2) + +**Host: **Jeremy Rickard + +**Attendees (pronouns):** + + + +* Rey Lejano (he/him) +* James Laverack (he/him) +* Maciej Szulik (he/him) +* Taylor Dolezal (he/him) +* Max Körbächer (he/him) +* Joseph Sandoval (he/him/él) +* Chun Wu (he/him) +* Mostafa Elmenbawy (he/him) +* Bhargav Krishna (he/him) +* Pravar Agrawal (he/him) + +**Note Taker:** + + + +* Rey Lejano + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Bhargav Krishna, joined yesterday + * Chun from Apple, applied to be a shadow but was not selected + * Maciej, SIG Apps and SIG CLI lead, works at Red Hat works in CLI and controllers + * Pravar works at IBM in India started to make contributions to SIG Scheduling and scheduling code +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Rey] Week 5 of the release, 57 enhancements with 2 approved exceptions + * Call for feature blogs and state of release sent + * No milestones in Sept/Oct, quiet period + * Many milestones close to holidays in November +* [eddiezane, soltysh] Releasing and versioning kubectl independently + * [Maciej] Short idea, we’ve had staging for a long time, kubectl and is one of the projects/subprojects interested in being part of a separate repo and they are already syncing the code and the last bit is figuring out how to release kubectl outside of k8s, the goal is to release kubectl faster than k8s. The first goal is to align them and make the kubectl release go faster. + * [Jeremy] We haven’t done anything outside of k/k in terms of the core artifacts. This is an area to collaborate. + * [Maciej] For a period of time, may have to release kubectl artifacts in k/k and in their own repo + * [Maciej] Already have a KEP to move kubectl related code to staging, either update this KEP or create a follow-up KEP. Is releasing in a separate repo an issue? + * [Jeremy] We need to follow-up on this, is there a GH issue + * [https://github.com/kubernetes/enhancements/issues/1020](https://github.com/kubernetes/enhancements/issues/1020) (old KEP) + * [https://github.com/kubernetes/enhancements/tree/master/keps/sig-cli/1020-kubectl-staging](https://github.com/kubernetes/enhancements/tree/master/keps/sig-cli/1020-kubectl-staging) (KEP) + * [Maciej] Since KEP only talks about a move to staging, have to create a follow-up on moving to another repo. Will sync with Eddie to create the enhancement issue and start to sketch out the follow-up KEP +* [James Laverack] Release team selection improvements for future releases. + * [James] Start a conversation before the start of 1.24, we’re outgrowing the existing shadow mentoring process. It’s a victim of its own success, had ~185 shadow applications, and have ~24-30 spots. Need to provide more consistency to have people come back more consistent, and have a better process to apply. Max has a diagram of an idea + * [Jeremy] Worth re-visiting this and make improvements. It’s difficult for applicants who applied multiple times. We talked about this in the 1.22 retro part 2, first start a GH discussion in the SIG Release repo and start collecting thoughts and diagrams like the one Max made. We can start there, start with the known state and work in the open. Don’t have good suggestions on what it should look like. Applications will continue to grow. Once concern, is we don’t get the long term succession planning out of it like we had in the past. For 1.23 we had challenges to get role leads staffed. We need a better way to back-fill. Like we got Karen Chu who was a previous lead, we need to identify better reach-back policy. + * [Taylor] There’s some merit, in the rejection letters it gives people opportunities to jump into things in the projects so we may be able to reach out to SIG leads if they’re looking for someone + * [Joseph] It’s a good problem we had, maybe SIG Contribex can help with this, is there a way to channel energy from the release team to other places. Some people need more than bread crumbs. + * [Taylor] There’s an issue from onboarding and something comes up like burnout or something else comes up. + * [Veronica] One of the reasons is lack of processes or lack of robust documentation, can see new people come in, it’s hard to start working in this project who is new. In all areas of SIG Release, documentation can be better. Do an iterative process of people who are shadows can leave documentation in a better state, document own process on how they learned things + * [Taylor] Maybe have documentation in easy-to-ingest format like Google slides + * [Jeremy] Couple of parallel issues: selection process and onboarding/training/documentation available. Worth starting a GH issue and make some concrete action items / selection process + * [AI - James] To start with a GH issue about selection process (EDIT: GitHub discussion opened [https://github.com/kubernetes/sig-release/discussions/1714](https://github.com/kubernetes/sig-release/discussions/1714)) + * [AI - Jeremy] To start with a GH issue about documentation/onboarding/training and work with Rey and Veronica +* [Taylor Dolezal] Zoom/Splain/YouTube integration is back on track! + * Meetings will now upload to YouTube, but do require setting live and adding to the SIG Release playlist (which I or the YouTube admins can assist with). + * Currently working through the SIG Release backlog of uploads. + * [Taylor] Issue with recorded meetings weren’t not pushed to Youtube, the integration used is called Splain. Integration setup was fine, what changed is when we changed to GSuite email address, so needed to remove integration and re-create integration. Does not associate the video to a playlist and integration sets the video to private. After meetings Taylor can set the video to public and put the video in the right playlist manually + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## September 7th, 2021 + +**Host: **Jeremy Rickard (he/him) + +**Attendees (pronouns)** + + + +* Sascha Grunert (he/him) +* Rey Lejano (he/him) +* Marko Mudrinić (he/him) +* Anubhav Joshi (he/him) +* Subhrodip Mohanta (he/him/his) +* Debabrata(he/him) +* Mostafa Elmenbawy (he/him) + +**Note Taker:** + + + +* Rey Lejano (he/him) + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Sascha] A branch manager for 1.23 will be selected this week + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Rey] Enhancements Freeze is Thursday + * 66 enhancements opt-in + * Few days left, final reminder out today + * [Rey] Shadow onboarding almost done for group membership + * [Jeremy] Shadow orientation will be scheduled this week +* Release Retro Part 3 [Guin] + * [https://docs.google.com/document/d/1deFXjK07TyGFds1wk2LjkRanqpjoGVmTFqmwTUgsSEM/edit#heading=h.ukbaidczvy3r](https://docs.google.com/document/d/1deFXjK07TyGFds1wk2LjkRanqpjoGVmTFqmwTUgsSEM/edit#heading=h.ukbaidczvy3r) + * +* [James Laverack] Release team selection improvements for future releases. +* Discussion Topics: + * Add topics +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) +* + + +## August 24th, 2021 + +**Host: **Jeremy Rickard (he/him) + +**Attendees (pronouns)** + + + +* Sascha Grunert** **(he/him) +* James Laverack (he/him) +* Nabarun Pal (he/him) +* Marko Mudrinić (he/him) +* Adolfo García Veytia (he/him) +* Savitha Raghunathan (she/her) +* Rey Lejano (he/him) +* Kunal Verma (he/him) +* Arnaud Meukam (he/him) +* Mritunjay Sharma(he/him) +* Jesse Butler (he/him) +* Joseph Sandoval(he/him/él) + +**Note Taker:** + + + +* Rey Lejano + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Start to plan around 1.23, discussions with LF and SPDX about the future of the tools. Some folks from LF and SPDX team will get together this week to find common ground + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Retro Part 2 [Guin] + * Retro document: [https://docs.google.com/document/d/1deFXjK07TyGFds1wk2LjkRanqpjoGVmTFqmwTUgsSEM/edit](https://docs.google.com/document/d/1deFXjK07TyGFds1wk2LjkRanqpjoGVmTFqmwTUgsSEM/edit) + * + * +* [James Laverack] Release team selection improvements for future releases. +* [Rey] Release 1.23 has started, call for enhancements email went out. Upcoming dates: + * PRR questionnaire deadline is on Sept 2 + * Enhancements freeze starts on Sept 9 at 23:59 PDT + * Call for exceptions and code freeze warning goes out on Nov 7 + * Code freeze starts on Nov 16 at 18:00 PST (*DST ends on Nov 7) + * Started a [GH discussion on when is the optimal start of Enhancements Freeze](https://github.com/kubernetes/sig-release/discussions/1680) +* Discussion Topics: + * Optimal Code Freeze Length: [https://github.com/kubernetes/sig-release/discussions/1674](https://github.com/kubernetes/sig-release/discussions/1674) + * Optimal Enhancement Freeze Start: [https://github.com/kubernetes/sig-release/discussions/1680](https://github.com/kubernetes/sig-release/discussions/1680) +* Roadmap and Vision + * [https://github.com/kubernetes/sig-release/blob/master/roadmap.md](https://github.com/kubernetes/sig-release/blob/master/roadmap.md) + * What do we want / need to accomplish in 1.23 +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) +* + + +## July 27, 2021 + +**Host:** Jeremy Rickard + +**Attendees:** + + + +* Rey Lejano +* Nabarun Pal +* Taylor Dolezal +* James Laverack +* Sascha Grunert + +**Note Taker: **Rey Lejano + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Welcome to Rishabh Jain! Worked on local up cluster script to have it work on MacOS +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Cherry-picks: master branch & 1.22 branch divergent, in 1.19 stopped fast-forwards and now have any changes in master to cherry-pick in the release branch. There was some confusion and some PR authors are not aware that they should raise to cherry-pick after the PR merged so the master & 1.22 branches diverged. In 1.21, Nabarun filed a chunk of cherry-picks to make sync the branches. Plan to communicate when adding PRs to the 1.22 milestone then file a cherry-pick + * [Nabarun] For the current set of diffs, should we file cherry-picks on our end since cherry-pick approvers are the release managers themselves as a one-time thing to get the cherry-picks done. + * [AI] Nabarun volunteers to file cherry-picks + * [Adolfo] Yesterday, Nabarun created a diff of commits. Adolfo will link the Slack thread to this agenda. There is a problem re-generating the test jobs when we create a new branch, the test generation scripts are outdated and may have a bug. Aaron C. has cleaned up the script but the script is still buggy. Request for others to take a look. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 1.23 Release Formation + * Congrats and thank you to Rey for picking up the 1.23 lead role + * Jeremy will be EA + * Looking to fill comms still + * Reached out to a few people + * Shadow application is open + * 21 Responses so far + * [James] There is a PR in flight to update hours for Enhancements shadows. The 1.23 Enhancments lead, Xander, is aware of this PR \ +https://github.com/kubernetes/sig-release/pull/1581 +* Recording publish automation + * Jeremy has been trying to upload videos but it’s not great/efficient + * We need to reach out to YouTube admins/ContribEx (?) to see if we can get splain configured + * Probably need zoom creds (Stephen?) + * [Taylor] Taylor is a YouTube admin, log into YouTube with the creds from Stephen. Splain does break often like have to re-run the webhook, haven’t found an alternative to Splain. + * [Jeremy] Jeremy has been downloading/uploading manually + * [AI] Taylor to connect with Stephen + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * [Sascha] Roadmap and Vision needs to be reviewed [https://github.com/kubernetes/sig-release/pull/1529/](https://github.com/kubernetes/sig-release/pull/1529/) + * [AI] Adolfo and Jeremy to review [PR 1529](https://github.com/kubernetes/sig-release/pull/1529/) + * [Nabarun] Working on [moving kubeadm out of k/k](https://github.com/kubernetes/enhancements/issues/1424) first working on moving kubectl out of k/k as the pilot project, that is the umbrella issue, there is not separate issue for kubectl out of k/k -- move to 1.23 + * [Jeremy] [Soliciting feedback for release cadence out-of-tree features](https://github.com/kubernetes/sig-release/issues/486), split between Enhancements subproject and release team. Feel free to look into if there is an immediate action needed +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + +## July 13, 2021 (recording) + +**Host:** Carlos Panato + +**Attendees:** + + + +* Rey Lejano +* Sascha Grunert +* Stephen Augustus +* Marko Mudrinić +* Adolfo García Veytia +* Jeremy Rickard +* Chris Negus +* James Laverack +* Joseph Sandoval +* Pushkar Joglekar (PJ) + +**Note Taker:** + + + +* Rey Lejano + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * PJ is from SIG Security +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Main issue going on, seen updates in testing-infra repo but haven’t been able to follow closely + * [Stephen] Currently in process to move from Google infra to community-owned infra, next thing working on is way we interact with artifacts like version markers (text-based API) that shows where the artifacts are based on the version. In the background, moving from Kubernetes-release-dev bucket (Google owned bucket) to K8s-release-dev bucket which is a community-owned bucket. There are changes from doing the build jobs. Previously had krel CI build jobs and they are running in parallel with new bootstrap jobs. Making sure everything is pointed to new community-owned infra. Cluster API Azure jobs are cleaned up, kops jobs need to be cleaned up + * [Stephen] go1.16.6 / go1.15.14 security updates + * Marko on point, Veronica + Mengjia shadowing + * Slack thread: [https://kubernetes.slack.com/archives/CJH2GBF7Y/p1626164119119300](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1626164119119300) + * [Stephen] Will cut a release, in the tracking issue Marko to add get people access to repo infra. For 1.19, need to bump repo definitions to point to go sdk versions. We’ll continue to bump repo definitions until it’s phased out + * [Stephen] Tyler / Linus + * [Stephen] Tyler is an intern at Google, working on quality of life on image promotion. Tyler removed bazel. Since bazel is gone, Stephen will bring back any promotion based things backed into the repo. Proposed repo name is sigs.k8s.io/kpromo. + * [Adolfo] Saw an issue from Linus about improving tools for image promoter. Stephen brought old tools to a legacy folder to check which tools should still be used. + * [Adolfo] Yesterday, last building PR for bom tool merge, posted in Twitter about artifacts we are producing. Request to review build of materials. \ +[https://twitter.com/puerco/status/1414784210800422929](https://twitter.com/puerco/status/1414784210800422929) \ +Please comment on the epic if you have any thoughts: \ + + * [Stephen] Compare results of tools, find a new name for bom + * [Stephen] [https://github.blog/2021-06-21-github-packages-container-registry-generally-available/](https://github.blog/2021-06-21-github-packages-container-registry-generally-available/) + * Potential discussion on where we put artifacts like where we put boms + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Tracking 56 Enhancements post Code Freeze + * Enhancements: Status Yellow + * CI Signal: Status Yellow + * Bug Triage: Status Yellow + * Docs: Status Red (based on two missing placeholder PRs) + * Waiting on couple of more placeholder PRs to be opened - [https://kubernetes.slack.com/archives/C2C40FMNF/p1626176987077700](https://kubernetes.slack.com/archives/C2C40FMNF/p1626176987077700) + * Tracking ~50 docs enhancements: Complete (Merged): 20; Ready For Review 12; No PR (Needs docs) 2; Draft (PR) 16 + * Release Notes: Status Green + * Comms: Status Green + * 1.22.0-beta.2 release cut is happening today! + * Test Freeze - July 15th (This thursday) + * Release Engineering - patch releases this week - [https://kubernetes.io/releases/patch-releases/#1-21](https://kubernetes.io/releases/patch-releases/#1-21) +* [PJ] Identify contributor(s) to give a deep dive / code walkthrough about container images in k/k during sig-security-tooling[ subgroup meeting](https://docs.google.com/document/d/1_Rn7S8UPMtXdu6EDl4_L2Ogb433z7hlxPs7El8TfNNg/edit) on 07/20 (8:30-9:15 AM PST) + * [Stephen] For security related release things, release managers need to be added to the private Slack channel. + * [https://github.com/kubernetes/community/pull/5853/files](https://github.com/kubernetes/community/pull/5853/files) +* [Jeremy] Release Cadence Change blog PR: [https://github.com/kubernetes/website/pull/28912](https://github.com/kubernetes/website/pull/28912) +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + +## (feel free to add any topics you’d like to discuss, even when they came up during the meeting) \ +June 29, 2021 (recording) + +**Host:** + +**Attendees:** + + + +* Sascha Grunert +* Marko Mudrinić +* +* Carlos Panato +* Chris Negus +* Derrik Campau +* Supriya Premkumar +* Adolfo García Veytia + +**Note Taker:** + + + +* + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * No new members +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Carlos] No particular update. + * [Sascha] No issues encountered during recent releases + * [Adolfo] Finished writing out all of the bugs that surfaced during the 1.22-beta.0 wrt the SBOM work. Biggest was a problem with versioning with Go packages inside Google Cloud Build. During this week and next week, plan to present the final proposal for the architecture of the SBOM work. Contact Adolfo if you want to help. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Monday-Wednesday-Thursday burndown meetings have started. + * Tracking 71 enhancements. + * Alpha: 28 + * Beta: 26 + * Stable: 14 + * Deprecation: 3 + * At risk: 43 (code not complete) + * Call for Exceptions email was sent yesterday + * Code freeze is on July 8th + * Tracking [12 feature blogs](http://bit.ly/k8s122-feature-blog). Deprecation announcement discussion is happening here - [https://github.com/kubernetes/sig-release/discussions/1606](https://github.com/kubernetes/sig-release/discussions/1606). Please feel free to add your thoughts. + * Release docs status: GREEN + * Integration branch is healthy again, PR that fixes the conflicts and syncs the branch has been merged. + * There are 42 enhancements that need docs, 11 Drafts, 10 ready for review and 5 merged. +* [Kirsten/Anna/Savitha/Divya]: We’d like to kick off a discussion and gather thoughts: re: exception deadlines. Exceptions can come in at any point after the freeze(specifically enhancements), the release might benefit from adding some more formalized guardrails to the process taking into the consideration of timing, priority and impact of enhancements exceptions that come after the freeze. + * [Derrik] Concerns around consistency between release teams. This discussion was triggered by a very late exception request in the 1.22 cycle. + * [Adolfo] Is the problem that things are not well defined? + * [Derrik] It has knock on effects on teams like enhancements and docs. + * [James] For enhancements we do a bunch of checks running up to code freeze. A late enhancement misses that. Additionally Is time since the deadline a factor in deciding on an exception request? Would we entertain a request for an enhancements freeze exception the day before code freeze? + * [Derrik] Community concerns that with a longer release if you miss it, then your work is pushed out quite far (4 months!). Maybe just a perception concern. + * [Sascha] All exceptions are case-by-case. Will more processes improve things? How long does the release team have to respond to exception requests, should there be an SLA on that? + * [Adolfo] Where is best to discuss this? Slack? Release retro? GitHub? KEP? + * [Sascha] Avoid a KEP for now, a lightweight discussion on GitHub is preferable. +* (feel free to add any topics you’d like to discuss as part of the agenda) +* [Chris] We’re pinging enhancements to open docs PRs. Not everyone has done it yet. + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * k/k 102822 — assign to Carlos + * k/sig-release 1595 — assign to Adolfo + * k/sig-release 486 — + * [Sascha] Maybe too broad in scope? + * [Adolfo] Needs to be developed further before we can take a concrete action + * k/k 72871 — Move to release engineering + * k/k 72638 — + * [Adolfo] We should keep an eye on this, but it’s up to each project to work this out. Should we take this on? Can we add a “parked but keep an eye on” column to the SIG Release board? + * [Sascha] We can put it into the backlog. We’ve triaged it multiple times. + * k/k 45506 — Probably not relevant for us, assign to Adolfo to take a look later. + * k/k 49313 — + * [Adolfo] We’re more a consumer of those tests and beneficiary of their output. This is more of a SIG testing thing. + * [Sascha] Agree. Why is it on our (SIG Release) board? Looks like we’re searching for a technical solution, so more of a SIG Testing problem. + * [Adolfo] Another one to watch and keep track of, but not do anything with. Backlog it for now. + * k/k 91570 — The issue is closed, so remove it from our board. + * k/sig-release 1389 — + * [Adolfo] This is related to branch protection. No effort around that right now to my knowledge. Already stale too. Removing stale so we can work on this in the future. +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) +* Next meeting - [Savitha]: Collaboration between upstream marketing & RT communications. + + +## June 15, 2021 (recording) + +**Host: **Jeremy Rickard + +**Attendees:** + + + +* Stephen Augustus +* James Laverack +* Supriya Premkumar +* Divya mohan +* Carlos Panato +* Joseph Sandoval +* John Gardiner Myers +* Sascha Grunert +* Rajas Kakodkar +* Rey Lejano +* Marko Mudrinić +* Luke Philips +* Adolfo García Veytia +* Savitha Raghunathan + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Go updates last week (go1.15.13 / go1.16.5) - [https://github.com/kubernetes/release/issues/2107](https://github.com/kubernetes/release/issues/2107) + * Patch releases for 1.19, 1.20. 1.21 + * [https://github.com/kubernetes/sig-release/issues/1599](https://github.com/kubernetes/sig-release/issues/1599) + * [https://github.com/kubernetes/sig-release/issues/1598](https://github.com/kubernetes/sig-release/issues/1598) + * [https://github.com/kubernetes/sig-release/issues/1597](https://github.com/kubernetes/sig-release/issues/1597) + * [https://github.com/kubernetes/sig-release/issues/1596](https://github.com/kubernetes/sig-release/issues/1596) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Tracking 70 enhancements so far + * 1.22.0-alpha.3 released on June 8th + * Tracking 6 feature blog posts + * Code freeze [reminder mail](https://groups.google.com/g/kubernetes-sig-release/c/PgHIKcrxpSg/m/p_OQ2n7tCgAJ) sent out + * Most recent CI Signal report (6/7): [https://groups.google.com/g/kubernetes-dev/c/zPySWW09j9I/m/FUXtjwC3AQAJ](https://groups.google.com/g/kubernetes-dev/c/zPySWW09j9I/m/FUXtjwC3AQAJ) + * +* Planning the release schedule in advance ([https://github.com/kubernetes/sig-release/discussions/1566](https://github.com/kubernetes/sig-release/discussions/1566)) + * Schedule pretty deterministic per [https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/2572-release-cadence#schedule-policy](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/2572-release-cadence#schedule-policy) + * Based on policy, some **planned dates**: + * 1.23 Begin - Aug 23rd (2 weeks after 1.22) + * Explicit Break for KubeCon North America (Week 8) + * 1.23 End - Dec 7 (Week 16) + * [Stephen] Consider some wiggle room around KubeCon + * [savitha] some concerns around length of code freeze, etc. + * [Action Item] Savitha to tighten up code-freeze definition + * [Action Item] Jeremy and James to draft blog of release changes, this blog will be released outside the release blog process + * [Sascha] Survey for the release is in progress: \ +[https://github.com/kubernetes/sig-release/issues/1526](https://github.com/kubernetes/sig-release/issues/1526) \ +Already in surveymonkey, ready for review and feedback. + * + * Present this at community meeting Thursday to get feedback, spread awareness + * Will also send to sig-release mailing list + * Possibly open a PR w/ rough begin/end dates + * RT Lead can fill in the rest of the schedule (freeze dates, etc) +* (feel free to add any topics you’d like to discuss as part of the agenda) + * [johngmyers] [k8s.io#2046](https://github.com/kubernetes/k8s.io/issues/2046) How do I review an image promotion PR? + * [John] Been working on kops, need to know how to verify the SHA of the image + * [Stephen] Tool called CIPMM (container image promoter merge manifest). Need access to staging repositories for kops. First step is to create PR for image build. Triggers Prow job and triggers GCB run and can get SHAs from the logs. Start with the staging repository. Every subproject has a different way to promote images. Release managers can help John out. + * [stephen] here are the jobs [https://github.com/kubernetes/test-infra/tree/master/config/jobs/image-pushing/releng](https://github.com/kubernetes/test-infra/tree/master/config/jobs/image-pushing/releng) + * [https://github.com/kubernetes/test-infra/tree/master/images/builder](https://github.com/kubernetes/test-infra/tree/master/images/builder) + * + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * In-review: + * Rewrite milestone maintainer (Max) - Carlos to review + * Add roadmap and vision - Stephen to work on this + * In-progress: + * Add Carlos and Adolfo to Tech Leads - May have a PR missing for k/org and needs top level approval for change to owners alias. Need to change the Slack config + * Developer guide audit - assigned to Savitha + * pull-kubernetes-e2e-gce is nearing its timeout - On GH, Jeremy asked reviewer to revisit this + * [Stephen] If we are releasing with blocking jobs consistently failing then we need to demote the jobs to informing. + * [Savitha] Max is working on pruning the jobs + * [Action Item] Marko to get in touch with Max on status of pruning jobs + * [Stephen] We need to establish a pattern for jobs. +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## June 1, 2021 (recording) + +**Host: **Stephen Augustus + +**Attendees:** + + + +* Daman Arora (Release Notes - 1.22 Shadow) +* James Laverack (1.22 Enhancements Lead) +* Derrik Campau (Release Lead - 1.22 Shadow) +* Supriya Premkumar(1.22 Enhancements Shadow) +* Trent Albright +* Chris Negus (Docs - 1.22 Shadow) +* Adolfo García Veytia +* Nabarun Pal +* Christoph Voigt (Bug Triage - 1.22 Shadow) +* Jeremy Rickard +* Joyce Kung +* Sascha Grunert +* Joseph Sandoval + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * docs team is tracking 67 enhancements with 8 drafts, 6 ready for review and 3 PRs merged, integration branch healthy + * EU/APAC meeting has spun up + * CI Signal has some red blocking jobs, but PRs have been opened to address most of them + * Comms yellow, but early still + * CI Signal report on k/dev, latest here: [https://groups.google.com/g/kubernetes-dev/c/H03JF6v3nu4/m/wOMbXci0BwAJ](https://groups.google.com/g/kubernetes-dev/c/H03JF6v3nu4/m/wOMbXci0BwAJ) +* Adolfo and Carlos as SIG Release TLs: [https://github.com/kubernetes/community/pull/5807](https://github.com/kubernetes/community/pull/5807) +* Nabarun and Joyce are joining as Release Manager Associates! + * Joyce and Nabarun joined as RMA last week-ish + * Nabarun was previously 1.21 lead + * Joyce has been around release team and is super awesome at reviews +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) [Move Kubeadm out of K/K #1424](https://github.com/kubernetes/enhancements/issues/1424) @nabarun will try and pick it up before the next meeting. [Feedback on release cadence for out of tree #486](https://github.com/orgs/kubernetes/projects/23) There are 2 states in-tree and out-of-tree. Release team does not track out-of-tree enhancements. How do we programmatically track scope? How do we bring it into the KEP creation process? Release impacting yes or no? Type of KEP - is it code or policy? Does scope make sense? +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## May 18, 2021 (recording) + +**Host:** Stephen Augustus + +**Attendees:** + + + +* Arsh Sharma +* Divya Mohan (RT Lead Shadow) +* Nabarun Pal +* Subhrodip Mohanta +* James Laverack (1.22 Enhancements Lead) +* Sascha Grunert +* Marko Mudrinić +* Joseph Sandoval +* Grace Nguyen (1.22 Enhancements Shadow) +* Max Körbächer (1.22 CI Signal Lead) +* Joyce Kung +* Chris Negus +* Kunal Kushwaha (1.22 release comms shadow) +* Rajas Kakodkar +* (1.22 release docs shadow) +* Luke Philips + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * [Stephen] [https://kubernetes.io/releases/](https://kubernetes.io/releases/) Great work by Jim Angel! FAQ’s about SIG-release as future content. This work gets us closer to that vision of community able to get answers to their questions. [https://github.com/kubernetes/website/pull/27997](https://github.com/kubernetes/website/pull/27997) First release with the patch schedule. (Action) Release managers review the patch schedule. Merge should happen this week. + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Marko] Progress has been made on [SPDX feature ](https://spdx.dev/)([https://github.com/kubernetes/release/pull/2064](https://github.com/kubernetes/release/pull/2064)). Thanks to Adolfo. Check out a demo from last week's SIG-release meeting. Patch releases happened last week. 1.18 is EOL. Veronica will cut 1.22.0 release. [Stephen] Hyperkube is officially deprecated. If you notice documentation referencing it let the release team know. Do we have a KREL BOM? Not yet. There is a standalone BOM utility. [Stephen] Let's make sure we have clarity with our tools and are easily “Googalable”. There is an [executive order ](https://www.whitehouse.gov/briefing-room/presidential-actions/2021/05/12/executive-order-on-improving-the-nations-cybersecurity/)around SBOM. Is there a way to package all the tools we have? We have requests for binaries. Hesitant until we clean up the repo. Image building is not at a satisfactory state. [https://github.com/google/ko](https://github.com/google/ko) KO is a utility to build container images. It's similar to Build Packs. We have many tools across many files. Looking at how this is shaped. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) [Divya] + * Tracking 67 enhancements so far + * We have gotten 5 exceptions in total + * 4 approved and tracked + * 1 under review (Issue: https://github.com/kubernetes/enhancements/issues/2033) + * Important deadlines: + * 1.22.0-alpha.2 release cut - May 18th (today) + * Docs team has started reaching out to enhancements owners + * CI Signal - L7 GCE Ingress failing causes multiple test to fail: [https://github.com/kubernetes/kubernetes/issues/102006](https://github.com/kubernetes/kubernetes/issues/102006) +* Roadmap and Vision: [https://github.com/kubernetes/sig-release/pull/1529](https://github.com/kubernetes/sig-release/pull/1529) Feel free to do another round of review [Stephen] This is a culmination from all the feedback to SIG-Release to provide a vision. Goal is to provide a secure release supply chain. How supportable and who is supporting our builds? Dan and Sascha have kicked this work off. There is an image promoter and a staging repository. Part of making artifacts consumable is putting them in a place that is consumable. There is a wider need to make CI more useful to the people testing Kubernetes. Define and collect metrics about the consumables. We need to have data to make assessments to improve. Release cadence survey. Make sure the community is satisfied with what is provided. Simplify the CVE process which depends on the CVE. If it's critical then it needs to happen asap vs low risk. How does the communication happen? Kubetest2 is a replacement for Kubetest. Enhance and simplify version markers. File based api’s to determine where a build is located. Making our work more consumable similar to the release page. Distroless images are now signed by co-sign. Next step is signing our own artifacts. You will continue to hear about software supply chain security. Please add comments to the PR. +* Triage Party: [Arnaud] Will sync with Carlos on the issues. Application issue or missing secret. [Carlos] Will investigate tomorrow morning. Triage Party to be used over WtB. It's important we get this back up and running. All have permissions to work on it. + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) [Stephen] Prioritize reviewing the board. +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## May 4, 2021 (recording) + +_Canceled due to KubeCon Europe_ + + +## April 20, 2021 (recording) + +**Host:** Sascha Grunert + +**Attendees:** + + + +* Sascha Grunert +* Jeremy Rickard +* Nabarun Pal +* Marko Mudrinić +* Savitha Raghunathan +* Joseph Sandoval +* James Laverack +* Derrik Campau +* Dan Mangum +* Luke Philips +* Joyce Kung +* Adolfo García Veytia (2nd half) + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * [No new attendees] +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Daniel Mangum] + * Question from Jeremy: Another 1.18 patch? + * [https://github.com/kubernetes/sig-release/pull/1523#issuecomment-820719600](https://github.com/kubernetes/sig-release/pull/1523#issuecomment-820719600) We will do another patch. [Sascha] We decided to use the mailing list. [Daniel] Release manager and associate call for interest in the role will be announced. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Ok + * [Savitha] Shadow responses have been received. PR’s have been setup. + * [Jeremey] Pt 2 of the 1.21 retrospective will be happening today 11am PST. + * [Josh Berkus] When does the schedule go up? WIP +* [Release Cadence KEP](https://github.com/kubernetes/enhancements/pull/2567) + + [Jeremy] Will transition to 3 release schedules similar to 2020. Lightweight structure to build schedule out. 15 weeks. Finish by mid-December. Timelines have been proposed for 2021 and 2022. Josh and Jordan have provided input with the last release being 14 weeks. The first release will be in Aprili. 2nd will be August. Last release in mid December. Kubecon has been noted and paused during the event. KEP will be in Alpha during 1.22. The new community meetings will have this KEP presented. Lean heavily on mailing list communication with formal write up. Collect feedback in 1.22-1.25 to gauge how this is going. Didn’t want to make a change during 1.22. + + + [Josh] When do we do the evaluation in the springtime? [Sascha] We will create metrics around the feedback. Josh will follow up on adding to the survey to help capture sentiment. Reach out to CNCF about how to handle the survey. [Josh] The way to reach end users is have CNCF make to market the survey. Also reach out to vendors to reach their users to provide feedback for the survey. Conflicting surveys need to be taken into considering. So plan in advance. Potentially August 2022 but Kubecon will block out all comms for the month. + + + [Jeremy] Collecting end user feedback up to Aug 2022. Then plan for August 2023. [Lauri] When does Kubecon take over the CNCF comms? [Josh] Usually a month out ahead of the event. + + + [Sascha] Approvers please provide your input. + +* [Policy needs](https://docs.google.com/document/d/16BZ1c23u21_L7wbY77vcTilQnYjNLloxLQkGlhDFhQ8/edit#) + * First up for discussion, delivery: [Policy regarding backports of dependency updates](https://docs.google.com/document/d/1npm6aXRyXIgrgL1SRbFn16GkNGkTH7zWEYy4sO5JccU/edit) + * [Lauri] We learned alot from the cadence KEP. This list is policy needs that have been needed. Tackling highest impact/needs from this list. Release cadence KEP can be crossed out. How to build artifacts in k/k. Providing an outline to build artifacts. Is that still the goal? What is the core related issue today? There hasn’t been any activity on this question. The policy would be general. SIG-release needs to provide the guidance when the source code is not in k/k. + * [Sascha] This could be a starting point on how artifacts get promoted to the release bucket. + * [Lauri] We can surface these topics. Can’t prioritize. + * [Lauri] [https://github.com/kubernetes/release/issues/1139](https://github.com/kubernetes/release/issues/1139). [Marko] Is there something we can do about this issue? Documentation is needed. [Lauri] Who is needed in this discussion? SIG-testing along with SIG-release. SIG-architecture? What would be the message for them? Are there any restrictions with GO versions? What about SIG-security? [Marko] yes. Also should we ask the GO team? [Sascha] Don’t need to add the GO team. [Lauri] What is the first step? + + [Marko] Draft document and share it. Lauri can kick that off. If we don’t do it the risk is we have no documentation and potentially miss a GO release. + + + [Adolfo] What was missing was the documentation. Following the template ended up needing some help from Carlos. + + + [https://github.com/kubernetes/sig-release/issues/486](https://github.com/kubernetes/sig-release/issues/486) [Marko] Once we solve the Kubeadm issue we can resolve the other issues. [Lauri] What is a good step to take? Review and next meeting discuss what is being asked of SIG-release. Define lightweight policies will help drive or open more questions with out of tree issues. + + + + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## April 6, 2021 (recording) + +**Host:** Sascha Grunert + +**Attendees:** + + + +* Sascha Grunert +* Nabarun Pal +* Carlos Panato +* James Laverack +* Joyce Kung +* Daniel Mangum +* Marko Mudrinić +* Jeremy Rickard +* Wilson Husin +* Lauri Apple +* Luke Philips +* Rey Lejano +* Taylor Dolezal + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * [No new attendees] +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Not much to report. + * [Carlos] Working to update Go. We're using 1.15.11 in release-branch 1.19. PRs in place. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 1.21 Release in 2 days!!! + * Release Signal was NO-GO in Monday’s burndown + * Today’s signal analysis is [async](https://kubernetes.slack.com/archives/C2C40FMNF/p1617710535238900) + * Backup Plan + * Proactive pushing of release date + * Options are next week Monday or Tuesday + * Decision to be taken during today’s Release Signal Analysis + * [Jeremy/Marko/Taylor] No release Mondays, prefer Tuesday. + * [CI Signal](https://testgrid.k8s.io/sig-release) + * Tests are very flaky + * master-blocking is RED + * Kind-ipv6-master-parallel is failing [Latest Run](https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-kubernetes-kind-ipv6-e2e-parallel/1379415968295424000) + * master-informing is RED + * Gce-master-scale-correctness failed yesterday. Waiting for today’s run. + * kubeadm-kinder-* jobs failing since a [PR](https://github.com/kubernetes/kubeadm/pull/2412) was merged on k/kubeadm yesterday. Author on it for a fix. Mostly it’s a configuration issue. + * Windows jobs are in better form after [some fixes](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1617647622287100) went in recently + * 1.21-blocking is VIOLET + * Mostly jobs are flaking due to timeouts + * 1.21-informing is RED + * Windows jobs are failing + * [Daniel] kind-ipv6 is notoriously flaky. Do we think it's an actual issue or just a bad run? [Joyce] We think it's a bad run, hard to get a consistent pattern. [Daniel] Most of the errors look like timeouts, maybe we should evaluate if these are a blocking test in the future. + * Action Item: Daniel to look at kind-ipv6's future inclusion as a blocking job. + * Bug Triage + * [Pod timeout flake issue](https://github.com/kubernetes/kubernetes/issues/100047) is open + * [Discussions](https://kubernetes.slack.com/archives/CN0K3TE2C/p1617296486010200) happening to resolve the release-blocker part of the issue + * A [bug fix](https://github.com/kubernetes/kubernetes/pull/100694) was filed to resolve an issue due to [a feature merged in 1.](https://github.com/kubernetes/kubernetes/pull/94991)20 + * Open question about it’s severity and whether it can be merged for the .1 patch release + * [Nabarun] I'll open a Slack thread in #sig-release to discuss async + * [Daniel] First-look is that it's cloud-provider specific, which changes how we treat it. It feels like something that should be in a patch release, so that a vendor-specific issue doesn't impact the community as a whole (in delaying the release). + * Release Notes + * [Major themes](https://github.com/kubernetes/sig-release/pull/1508) to be merged before EOD Tuesday + * Docs + * [Integration branch](https://github.com/kubernetes/website/pull/26153) is not healthy + * [PR 27432](https://github.com/kubernetes/website/pull/27432) is open to fix the integration branch + * Reference docs generation is in progress + * [Rey] Waiting for a k/website tag, some generation will happen on the day. + * Comms + * All okay on this front + * [PSP Deprecation blog](https://github.com/kubernetes/website/pull/27369) is ready to be shipped before the release blog to prevent surprises + * [Nabarun] Waiting for some CNCF approval for embargoes wrt the deprecation blog. + * Release blog and feature blogs are ready + * [Daniel] Is it realistic to release on Thursday? [Jeremy] That is optimistic given this update. We're better off delaying now until Tuesday because of the number of downstream consumers. [Nabarun] +1 for delay now until Tuesday, especially wrt the CNCF comms about the embargo. Looking to make a decision EoD. + * [Nabarun] If the pod timeout flake is a big issue, maybe email out to the community (dev mailing list) asking for help? Only useful if we have a lot of flakes. [Taylor] It's a good idea, especially giving k-dev a technical breakdown of the issues leading to release delay. [Nabarun] I was thinking of two emails. One to k-dev with a list of flakes, with request for assistance. Second email EoD PT, about the decision to delay or not. +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * Skipping #2992, #908 and #2234 because they’ve been triaged in a dedicated session: \ +[https://docs.google.com/document/d/1nINF7EEznlhoXDtjHqaMoQUQ71N0Ata2rZD4FDLvKb4/edit#heading=h.fwj4ypzcapo8](https://docs.google.com/document/d/1nINF7EEznlhoXDtjHqaMoQUQ71N0Ata2rZD4FDLvKb4/edit#heading=h.fwj4ypzcapo8) + * k/k 99427 Adolfo tagged sig-release 28 days ago, he's not fully sure and will ping later about it. + * k/enhancements 2567 [Sascha] In review [Lauri] what's our timeline? [Sascha] My plan is to have it merged before the 1.21 release. [Lauri] Who's driving that? [Sascha] Lets re-ping Stephen. I should have some time this week otherwise. + * k/k 96692 Assigned Wilson and Sascha. [Sascha] Wilson has a PR open. Lets move the ticket to "In review" + * k/release 1139 assigned to Stephen [Lauri] Carlos was going to work on it. [Carlos] Waiting on Stephen to publish the first draft. [Lauri] How long have you been waiting? [Carlos] Early Feb. + * k/sig-release 734 Assigned Jeremy and Stephen. Neither on the call so leaving for now. + * k/enhancements 1424 Discussed on another call. [Lauri] Something that needs discussion after 1.21 ships. + * k/k 74375 [Sascha] Aaron is working on it. + * k/sig-release 1257 [Marko] Max is working on it. [Sascha] Move it into in-progress + * k/kubeadm 1599 [Sascha] Question is if we want a test to be release blocking. Some concerns about doing that. Unsure on the decision process and how to move that discussion forward. [Lauri] Are we awaiting evidence? Do we need more research? [Marko] We'll add that label. + * [Marko] We have eight untriaged issues. [Lauri] We have in-depth triage sessions on Fridays if anyone would like to join for that. + * k/sig-release 486 [Marko] Last update in 2020. [Lauri] This is more of an 'idea' post, we need to figure out what the 'ask' is and what the action would be. Still important. + * k/k 74965 [Marko] Assigned to Daniel, from 2019 so quite an old issue. + * Action item: Marko to ping Daniel about this. + * k/k 72638 [Marko] Also from 2019, currently frozen. Is there anything we can do? [Sascha] Related to moving kubeadm out of tree, I don't think we can do anything right now. + * k/k 72638 [Marko] Also from 2019, Stephen asked if it's still relevant. + * Action item: Marko to follow up about this issue to find out if it's still relevant. +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## March 23, 2021 (recording) + +**Host:** Sascha Grunert + +**Attendees:** + + + +* Sascha Grunert +* Nabarun Pal +* Adolfo García Veytia +* Rey Lejano +* Carlos Panato +* Lauri Apple +* Jeremy Rickard +* Pavel Malinov +* Stephen Augustus +* Wilson Husin +* Luke Philips + +** \ +Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Slight agenda update: + * CI Signal report is now part of the “Release Team” update + * “Walk the Board” section reordered +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Cut last patch releases. Wrote some code to automate the bug publication which wasn’t done in 1.20 release and hit some snags in the patch releases with Go 1.16 automation. Investigated and fixed the building of deb/rpm packages. + * [Sascha] Link to relevant issue [https://github.com/kubernetes/release/issues/1960](https://github.com/kubernetes/release/issues/1960) + * [Stephen] More conversations with folks about securing supply chain. One of the groups talked to who worked on SIG Store. Chat with Luke who is a maintainer in the group and who is a PSC member. Look into the tool [Cosign](https://github.com/sigstore/cosign) for container image signing and looking into the signing workflow. Talk to Cosign about importable packages and may build around in krel like a `krel sign` command.(Article for cosign: [https://dlorenc.medium.com/cosign-signed-container-images-c1016862618](https://dlorenc.medium.com/cosign-signed-container-images-c1016862618)) + * [Sascha] Lauri started threads on the release engineering channel [https://kubernetes.slack.com/archives/CJH2GBF7Y/p1616145933069400](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1616145933069400) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * **Timeline Review** + * **March 24th - Docs deadline - PRs Ready for Review** + * **March 24th - Test Freeze** + * **March 25th - End of collection for Major Release Themes** + * March 31st - Docs Complete + * March 31st - Release Notes Ready for Review + * **Enhancements** + * Tracked: 50 + * Alpha: 19 + * Beta: 15 + * Stable: 15 + * Deprecation: 1 + * **CI Signal **([tracker](https://bit.ly/k8s121-ci-signal)) + * [master-blocking](https://testgrid.k8s.io/sig-release-master-blocking) is still flaky + * [master-informing](https://testgrid.k8s.io/sig-release-master-informing) is still failing + * Failing Jobs + * [Conformance - Open Stack](https://testgrid.k8s.io/sig-release-master-informing#Conformance%20-%20OpenStack) + * [capg-conformance-v1alpha3-k8s-master](https://testgrid.k8s.io/sig-release-master-informing#capg-conformance-v1alpha3-k8s-master) + * All windows jobs + * [gce-cos-master-serial](https://testgrid.k8s.io/sig-release-master-informing#gce-cos-master-serial) ([fix in flight](https://github.com/kubernetes/kubernetes/pull/100465)) + * **Bug Triage** + * **22 open issues **in the v1.21 milestone + * **3 critical-urgent** + * **7 open PRs **in the v1.21 milestone + * **4 critical-urgent** + * **Docs** + * Ready for review deadline is **Tomorrow** + * 18 merged already + * 19 ready for review + * 2 still in draft + * 1 is late + * **Release Notes** + * Major themes collection in progress. Due by March 25th. +* (feel free to add any topics you’d like to discuss as part of the agenda) + * [Stephen] Possibly push Docs Placeholder PR Deadline before Code Freeze to give SIG Docs an understanding of the workload + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +**Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## March 9, 2021 (recording) + +**Host:** Sascha Grunert + +**Attendees:** + + + +* Vlad Gorodetsky (RT Lead Shadow) +* Taylor Dolezal +* Marko Mudrinić +* Dan Mangum (SIG Release TL) +* Jeremy Rickard +* Sascha Grunert +* Kunal Kushwaha +* Verónica López +* Joseph Sandoval (RT Enhancement Shadow) +* Max Körbächer +* Wilson Husin +* Adolfo García Veytia +* Nabarun Pal (1.21 Release Lead) +* Arnaud Meukam +* Luke Philips + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Reorganization of repositories + * [https://github.com/kubernetes-sigs/release-sdk](https://github.com/kubernetes-sigs/release-sdk) + * Interfaces and implementations for building Kubernetes releases. + * A more formal package where we are driving towards stable APIs. + * [https://github.com/kubernetes-sigs/release-utils](https://github.com/kubernetes-sigs/release-utils) + * Kind of a playground for functionality that we use across repos + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 1.21 Updates + * Code Freeze is today!!!! + * [Milestone requirement PR in place](https://github.com/kubernetes/test-infra/pull/21227) (thanks Adolfo!) - LGTM’ed and Approved. Hold will be removed at 6PM PST + * The email will be sent at 6PM PST right after the milestone requirement PR is merged. + * [Nabarun] will send the email out to k-dev tonight. Will detail how things gets merged during code freeze + * 1.21.0-beta.1 is released today. (thanks Max!) + * [Nabarun] release went smoothly, thanks Max! + * [Enhancements](https://bit.ly/k8s121-enhancements) (thanks Anna and all the Enhancements team shadows!) + * 62 being tracked + * 38 code complete or very near to completion + * 24 are At RIsk of being removed from milestone + * [Nabarun] PRs are either not open, not close to completion, or haven’t had API review yet + * Exceptions + * 3 exceptions till now + * [Reject Invalid PVC DataSource](https://groups.google.com/g/kubernetes-sig-release/c/6humzS-PU84/m/HeVrAGA6AwAJ) + * [Structured Logging](https://groups.google.com/g/kubernetes-sig-release/c/shnk5YW5Yag/m/ICHY1R9BAwAJ) + * [Nabarun] major overhaul, lots of PRs in flights + * [MaxUnavailable for Stateful Sets](https://groups.google.com/g/kubernetes-sig-release/c/Sht6SZxWf6s/m/dPIZVyxgAwAJ) + * [Nabarun] The first and third above did not ever opt into the release, so they may be denied -- decision by end of day + * Bug Triage (thanks to Derrik and his bug triage team!) + * 32 open issues in v1.21 milestone + * Around 9 are related to flaking/failing tests + * [Nabarun] in pretty good shape here + * 37 open PRs in the milestone + * CI Signal (thanks to Joyce and their team!) + * To be covered in the next section + * Other verticals like Release Notes, Comms and Docs are green. + * [Nabarun] Docs have a lot of incomplete but will expect it to drop as enhancement number drops + * [Wilson] Fun fact: there are 108 PRs between 1.21.0-beta.0 and 1.21.0-beta.1 \o/ + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review: [https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * Master-blocking looks good these days + * Master-informing still flakes often on the *-windows-* + * Overall you see the progress in deflaking master-blocking in the project board, most of the flakes are picked up or resolved + * In addition not so many new flakes opened the last days + * [Max] sig-release-master-blocking has gotten much better over the past few weeks + * [Max] still a lot of red on sig-release-master-informing, especially windows jobs (per usual) -- active discussion in Slack to get these addressed + * [Max] some of the issues need to be advanced on the board +* [hasheddan] Flake Finder Fridays #001 this Friday! + * [https://youtu.be/9muoWaXZK8I](https://youtu.be/9muoWaXZK8I) +* [Wilson] Self-promotion of my pet project, looking to see if there are interest in collaborating / suggestions moving forward to get this migrated under kubernetes / kubernetes-sigs org + * Currently lives here: [https://github.com/wilsonehusin/k8s-release-notes-data](https://github.com/wilsonehusin/k8s-release-notes-data) + * This approach separates the "gather", "edit", "render" section of the release notes + * The ultimate goal is that folks can submit overrides to release notes without having to do local operations of krel setup +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* New [kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * [Adolfo] k/k 99427 seems to have stalled for a while, any takers? + * [Sascha] one of the release managers? + * [Veronica] k/release 1139 and k/community 5235 can fall under Marko's project for documentation write up + * [Sascha + Adolfo] k/k 74375 collaborate with SIG Testing + * [Artifact Management board](https://github.com/orgs/kubernetes/projects/48) +* Incoming issue and PR triage + * [org:kubernetes SIG Release issue needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * Ben is looking on [#99818](https://github.com/kubernetes/kubernetes/issues) + * [Sascha] 99656 -- we didn't change it recently, unclear who actually owns it + * [Adolfo] we fixed the tagging problems, so it should be correct, but will take a look + * [Adolfo] 99562 -- will follow up with Stephen on progress + * [org:kubernetes SIG Release PR needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + +**Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## Feb 23, 2021 (recording) + +**Host:** Jeremy Rickard + +**Note Taker**: Rey Lejano + +**Attendees:** + + + +* Wilson Husin +* Rey Lejano +* Adolfo García Veytia +* Max Körbächer +* Savitha Raghunathan +* Pavel Malinov +* Taylor Dolezal +* Tim Pepper +* Nabarun Pal +* Stephen Augustus +* Lauri Apple +* Luke Philips +* Verónica López + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Luke Philips - lurk and learn and looking to see where he can help out +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [hasheddan] Krel Config API ([https://github.com/kubernetes/release/pull/1926](https://github.com/kubernetes/release/pull/1926)) + * [Dan] WIP PR for API to configure krel, would like to be more declarative + * [hasheddan] Using Crossplane to manage k8s community infra ([demo](https://youtu.be/eSjp9XnCIiM)) + * [Dan] Demo video, look at Crossplane to manage infrastructure, scoping out a POC for a mvp, will open an issue on k8s.io + * [hasheddan] Documenting release-blocking tests + * [Dan] Part of [supported platform](https://github.com/kubernetes/sig-release/issues/1337), would like to have better documentation around release-blocking tests, the strategy over time has been informal. So people are aware of the implications of job failures. Will try to get a template for people to build on. Asking for contributions to the effort + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Nabarun] 14 days from code freeze + * [Nabarun] Feature blog freeze on March 1. Only have 1 blog opted in so far, would like to get 4. + * [Jeremy] Lots of flaky jobs on testgrid, is the team looking into it or need help + * [Nabarun] Joyce is on top of it and the team is on top triaging flakes + * [Stephen] Aaron is working on lots of changes on master blocking jobs, [one is around using now using k8s release dev bucket](https://github.com/kubernetes/test-infra/pull/20964) to pull artifacts (k8s community bucket) versus using a Google infrastructure bucket. + * 65 Enhancements being tracked + * MWF burndown meetings to begin next week + * 1.21-beta.0 release today + * 1 feature blog opted-in. Team will be pinging SIG Leads on respective SIG slack channels. + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review: [https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * +* (feel free to add any topics you’d like to discuss as part of the agenda) + * Any guidelines for requesting security reviews? [Savitha] + * Discussing how to include PRR reviews, don’t want to add more burden on KEP owners. Takeaways: time to request a PRR is when feature is proposed to SIGs, have an opt-in process and it will be up to the SIG leads to request for a security review. Also attached the WIP hardening guide + * [Stephen] Around enhancements tracking repo is around scope. Trying to determine the scope of an enhancement. Two types of KEPs: KEP that the release team cares about and everything else. Suggest to look at the miro board + * [Lauri] Link to miro board [https://miro.com/app/board/o9J_lbvQp4Y=/](https://miro.com/app/board/o9J_lbvQp4Y=/) + + Lauri will add security reviews + +* [Stephen] Link to scope issue [https://github.com/kubernetes/enhancements/issues/2311](https://github.com/kubernetes/enhancements/issues/2311) +* [Stephen] Meetings around KEPs (need more context) on Thursdays at US 1pm EST +* [Stephen] Looking at recent releases, we have many more enhancements in the releases and this will lead into the project planning/roadmap and tools that we’re writing now + +**Walk the Board [timebox to 20 min]:** + + + +* New [kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * [Artifact Management board](https://github.com/orgs/kubernetes/projects/48) + * [Veronica] Triage Party: url was down, Arnaud fixed the url. Arnaud also sent a PR with a bump for 1.21 and Carlos approved it. It’s been painful behind the scenes for all the downtime root-causes. Arnaud and Veronica discussed with this bump that it should be good-to-go but not 100% sure if this is what you would consider “done”. Call for people to collaborate or if people want to understand Triage Party. Link to Triage Party: [https://release.triage.k8s.io](https://release.triage.k8s.io/s/bug-triage) + * [Stephen] Triage Party is currently pointing at 1.20. We need to create success criteria, ask what people want to see in dashboards, want boards for each sub project and ask what can we do to “close the books” on this. Need to look at automating triage party, can it be a Deployment? + * [Lauri] Look at prototype plans and look at feature needs + * [Veronica] Open to what the “complete criteria” should be + * [Veronica] PR to bump triage party to 1.4.0-beta.1 [https://github.com/kubernetes/k8s.io/pull/1700](https://github.com/kubernetes/k8s.io/pull/1700) + * [Stephen] Start using triage party for SIG Release and subprojects and get a “wish” list going + * [Veronica] Send us your wish list. + * [Stephen] Need to have SIG release and subprojects using Triage Party and request Veronica and Arnaud to document gaps + * Use docker buildx for the build-image k/k PR #99080 + * [Stephen] Have more clarity on what image it's referring to. Needs a review. Add reviewers to the PR + * Automate some work for the release-docs/enhancement team + * Unassigned but in-progress + * Should be on the release-team project + * Rewrite milestone maintainer - Max (not on the call) + * [Stephen] Let’s get reviewers, add Jeremy + * Chair/Technical Lead access for Jeremy + * Done but Jeremy will make the template more detailed and Stephen will fix Jeremy’s announce privileges + * Create some more CI Signal documentation + * [Lauri] Two weeks ago, they were nearly done + * [Stephen] Dan and Joyce are on point for review + * [Joyce] Rob pushed a second set of changes, will take a look soon + * [Stephen] [Action-item] Put this on the release team board + * Draft a policy for updating Go versions across the Kubernetes code base/infra + * [Stephen] Policy Doc will hand off to Carlos and Arnaud + * CVEs for Dependencies + * From 2018 + * [Stephen] We are doing things differently, we should sync with SIG Security and Product Security committee. Adolfo is also adding CVEs to release notes template but that is a presentation template for disclosed CVEs not so much as the implementation path for CVEs. Need to refine the goal of this issue since the issue is 3 years old. Assign to Dan and Stephen + * GitHub PR reviews as /lgtm + /approve for SIG Release repos +* Incoming issue and PR triage + * [org:kubernetes SIG Release issue needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [org:kubernetes SIG Release PR needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + +**Open Discussion [timebox to 5 min]** + + +## Feb 9, 2021 (recording) + +**Host:** Lauri Apple / Sascha Grunert + +**Note Taker**: TBD + +**Attendees:** + + + +* Sascha Grunert +* Vlad Gorodetsky +* Marko Mudrinić +* Dan Mangum +* Arnaud Meukam +* Jim Angel +* Rey Lejano +* Adolfo García Veytia +* Lauri Apple +* Pavel Malinov +* Victor Palade +* Wilson Husin +* Joseph Sandoval +* Luke Philips + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* [Sascha] Refurbished meeting agenda: + * “Recurring Topics” has been renamed to “Topics” and can now be used by everyone to add their own content. Topics are now timeboxed to 20minutes. + * Walking the board is a main part of the meeting, which is timeboxed to 20minutes, too. + * Open discussion should be left open and will only take 5minutes at the end of the meeting from now on. +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * [Sascha] Closed and remove from future agendas + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Dan] Working on API for krelfile + * [Adolfo] Working on SPDX inspection, having to workaround issue with distroless base images + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Rey] Enhancement freeze is today, many at risk because of Production Readiness Review + * [Wilson] PodSecurityPolicy announcement in progress + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review: [https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) + * [Dan] Quite a few flaky tests on master blocking most are very, very infrequent. We worked on canary work last week and using krel now. Fixed version marker issues from k8s infra + * gce-cos-master-serial issue: [https://github.com/kubernetes/kubernetes/issues/98880](https://github.com/kubernetes/kubernetes/issues/98880) +* FYI: SIG Testing KEPs relevant to release engineering being presented at SIG Testing meeting today [@spiffxp] + * [Reducing Kubernetes Build Maintenance](https://github.com/kubernetes/enhancements/pull/2469) + * [Kubetest2 CI migration](https://github.com/kubernetes/enhancements/pull/2465) +* Move gs://k8s-release bucket to k8s-artifacts-prod project? [@spiffxp] + * context: [redirect dl.k8s.io traffic to community-owned GCS bucket instead of kubernetes-release](https://github.com/kubernetes/k8s.io/issues/1569) + * gs://k8s-release currently lives in k8s-release project + * tl;dr much easier re:billing to lump all artifact hosting costs under one project +* Idea / Proposal: [collaborative release notes review](https://github.com/kubernetes/release/issues/1889) [@wilsonehusin] + * make the workflow more collaborative and less dependent on krel (i.e. "release team does not need to run krel locally") + * please comment or reach out if you have thoughts! + +**Walk the Board [timebox to 20 min]:** + + + +* New [kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * [Artifact Management board](https://github.com/orgs/kubernetes/projects/48) + * [Lauri] Action item: ping sig-release channel about[ Go versions](https://github.com/kubernetes/release/issues/1139) + * [Lauri] Action item: ping sig-release channel about [Developer Guide Audit Release](https://github.com/kubernetes/community/issues/5235) + * [Lauri] Action item: ping in sig-release channel about [Kubernetes packages returning 404](https://github.com/kubernetes/kubernetes/issues/97264) +* Incoming issue and PR triage + * [org:kubernetes SIG Release issue needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [org:kubernetes SIG Release PR needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + +**Open Discussion [timebox to 5 min]:** + + + +* Ask around for the subproject updates before the meeting that we can ensure we get an update for each one + + +## Jan 26, 2021 (recording) + +**Host:** Stephen Augustus + +**Note Taker**: Sascha Grunert + +**Attendees:** + + + +* Sascha Grunert +* Vlad Gorodetsky +* Joyce Kung +* Adolfo García Veytia +* Evelyn Cupil-Garcia +* Rey Lejano +* Dan Mangum +* Jim Angel +* Lauri Apple +* Wilson Husin +* Arun Krishnakumar +* Pavel Malinov +* Rin Oliver +* Seth McCombs +* Kirsten Schumy +* Somtochi Onyekwere + +**Recurring Topics [timebox to N min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * Subproject has closed + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review: [https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New [kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * [Artifact Management board](https://github.com/orgs/kubernetes/projects/48) +* Incoming issue and PR triage + * [org:kubernetes SIG Release issue needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [org:kubernetes SIG Release PR needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) +* Currently ongoing: prioritization of topics we want to work on in this cycle + * Let us know if there are any topics which should get a higher priority +* Email about SIG Release 2021 teaser: + * Feedback from retro was that community did not get enough updates from SIG Release + * Community meetings should be utilized to provide monthly SIG Release updates + * Covers subproject work, too + * February edition of the “SIG Release Digest” should be done until the end of the week \ +Feel free to add your suggestions to the HackMD document: [https://hackmd.io/Q1B4GcU3TLamyssrjKpeNg](https://hackmd.io/Q1B4GcU3TLamyssrjKpeNg) + +**Open Discussion [timebox to N min]:** + + + +* [Dan] Other hardware arch & OS support? Discussion in SIG Arch + * [Number of different platforms are supported](https://github.com/kubernetes/sig-release/blob/master/release-engineering/artifacts.md) from SIG Release perspective + * Applies to binaries and container images we release + * Most tests are right now running for the linux/amd64 platform, improved testing coverage is one goal of the overall effort + * Finding current set of supported “Tiers” should categorize support and test coverage: [https://github.com/kubernetes/sig-release/pull/1360](https://github.com/kubernetes/sig-release/pull/1360) + * SIG Architecture meeting recording: <link> + * “Bill of Materials” epic is one part of the support strategy, too: [https://github.com/kubernetes/release/issues/1837](https://github.com/kubernetes/release/issues/1837) +* [Stephen] SIG annual reports for Steering + * Steering committee started reporting for working groups (goals, needs, achievements) + * Now applies to SIGs, too + * Reach out to Stephen to find out metrics regarding our goals +* [Jim] Stakeholders discussion on a release page + * [https://kubernetes.slack.com/archives/C2C40FMNF/p1603814400441200](https://kubernetes.slack.com/archives/C2C40FMNF/p1603814400441200) + * Standardized Kubernetes release page is the overall target + * Parts of k/website could be used to move ownership to k/release + * Stephen will draft a PR by the end of the week +* [Lauri/Derrik] Bug Triage wishes to run a grafana dashboard to track Issues/PR’s for milestones. Where should this be hosted (or should we piggyback off of [CNCF Devstats](https://k8s.devstats.cncf.io/d/22/open-issues-prs-by-milestone-and-repository?orgId=1&var-sig_name=All&var-milestone_name=v1.21&var-repo_name=kubernetes%2Fkubernetes)?) + * User experience improvements planned for the CNCF Devstats + * Can we track our work with the help of the stats? + * Reporting template could be used for other workflow needs: [https://docs.google.com/document/d/1JOXKBDgXmQzz8YQSYa7XYcfVteM79iMtvId1aQXC1e8/edit](https://docs.google.com/document/d/1JOXKBDgXmQzz8YQSYa7XYcfVteM79iMtvId1aQXC1e8/edit) + * When are we going to use triage party? What are the required bits to start using it? \ +Lauri will follow-up in the sig-release slack channel after the meeting. \ +[https://kubernetes.slack.com/archives/C2C40FMNF/p1610469778354000](https://kubernetes.slack.com/archives/C2C40FMNF/p1610469778354000) + * Possible alternative Orbit: [https://orbit.love/solutions/for-maintainers](https://orbit.love/solutions/for-maintainers) \ +Following-up with SIG Contribex to check if it’s applicable as well. (Rin/Charles) +* Release Cadence decision: + * Follow-up email containing the feedback we’ve got will come soon + * The next necessary steps have to be defined + * Likely to write-up a KEP for applying the new cadence \ +(targeting enhancements deadline at Feb 9th) + * Community survey to gather data is also an option right now +* **Back to future topics:** + * [Jeremy] - Merge Blocking after Milestones + * [Kirsten/Jeremy] Clarify and memorialize criteria for removing unmerged items at Code Freeze; + * [Taylor] For the release team shadow application responses, since the results contain PII, what is the best way to share out that privacy concern with applicants, and how do we gate access around that point of collection? + * [Taylor] For people that do not get selected to be a part of the release team in a specific cycle, can we engage them in any other capacity or role(s)? + + +## Jan 12, 2021 (recording) + +**Host:** Sascha Grunert + +**Note Taker**: Verónica López + +**Attendees:** + + + +* Daniel Mangum +* Sascha Grunert +* Nabarun Pal +* Rey Lejano +* Verónica López +* Vlad Gorodetsky +* Taylor Dolezal +* Carlos Panato +* Joseph Sandoval +* Stephen Augustus +* Kirsten Schumy +* Adolfo García Veytia + +**Recurring Topics [timebox to N min]:** + + + +* (if you have any… here are some suggestions) +* Welcome any new members or attendees +* Subproject updates + * Licensing ([https://github.com/orgs/kubernetes/projects/31](https://github.com/orgs/kubernetes/projects/31)) + * [Stephen] No one actively working on the Licensing subproject + * [Stephen] May drop in 2021 + * [Lauri] Any steps to shutdown the subproject? + * [Stephen] Don’t have to do much (to shut down the subproject) + * [Dims] For some background, a bunch of dependencies that we were using, cleaned up, removed. The work has wrapped up and the subproject has achieved its goals and is not needed. + * Links from Dims: Wrapping up Licensing subproject \ +https://github.com/kubernetes/sig-release/pull/1412 \ +https://github.com/kubernetes/test-infra/pull/20450 + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Miro board ](https://miro.com/app/board/o9J_ld0yX48=/?utm_source=notification&utm_medium=email&utm_campaign=daily-updates&utm_content=go-to-board): if anyone wants to adopt an issue/task on the board, please add your name under the relevant “team”, and let us know on #release-management on Slack + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 1.21 Release Cycle kicked off yesterday Monday January 11 + * [Adolfo] : considerations for proposed release schedule? (holidays, dates, deadlines, etc.) + * [Stephen] : Alpha releases should take place immediately after the previous official release. + * enhancements subproject: discussions to change the process (motivation: customer satisfaction). Allow release managers to opt in into the release process. Goal: have the workflow ready for 1.22 + * Schedule + * Major milestones have been decided + * **Monday, January 11th**: Week 1 - Release cycle begins + * **Tuesday, February 9th**: Week 5 -[ Enhancements Freeze](https://github.com/kubernetes/sig-release/blob/master/releases/release_phases.md#enhancements-freeze) + * **Tuesday, March 9th**: Week 9 -[ Code Freeze](https://github.com/kubernetes/sig-release/blob/master/releases/release_phases.md#code-freeze) + * **Tuesday, March 16**: Week 10 - Docs deadline - Open placeholder PRs + * **Wednesday, March 24**: Week 11 - Docs deadline - PRs ready for review + * **Wednesday, March 31**: Week 12 - Docs must be completed and reviewed + * **Thursday, April 8th**: Week 13 - Kubernetes v1.21.0 released + * [Branch management milestones are being discussed](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1610382888260400) \ +A WIP Calendar -> [here](https://docs.google.com/spreadsheets/d/1q7UvVvZQKtrZaXCWejlUZCJYrAYckHPaTsnZ0lEJ9f4/edit#gid=0) [SIG Release Leads and Release Managers have access] + * Shadow selection in progress + * Important Milestones this week: + * Enhancements Collections to begin Today, Tuesday January 12 + * Proposed [changes to process](https://github.com/kubernetes/sig-release/pull/1406) have been merged + * Communication to go out after the All SIG Leads meeting today + * Possible alpha.1 release Today, Tuesday January 12 + * Schedule to be finalized by Thursday, January 14 + * Shadows to be selected by Friday, January 15 + * CI Signal ([https://github.com/orgs/kubernetes/projects/11](https://github.com/orgs/kubernetes/projects/11)) + * Testgrid dashboard review: [https://testgrid.k8s.io/sig-release](https://testgrid.k8s.io/sig-release) +* New [kubernetes/kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) review +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * [Artifact Management board](https://github.com/orgs/kubernetes/projects/48) +* Incoming issue and PR triage + * [org:kubernetes SIG Release issue needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [org:kubernetes SIG Release PR needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + +**Open Discussion [timebox to N min]:** diff --git a/sig-release/meeting-notes-archive/2022.md b/sig-release/meeting-notes-archive/2022.md new file mode 100644 index 000000000..315967c5e --- /dev/null +++ b/sig-release/meeting-notes-archive/2022.md @@ -0,0 +1,2199 @@ + + +## 13 Dec, 2022 (recording) + +**Host (pronouns):** + +**Attendees (pronouns):** + + + +* Marko Mudrinić (he/him) +* James Laverack (he/him) +* Sascha Grunert (he/him) +* Xander Grzywinski (he/him) +* Leonard Pahlke (he/him) +* Arnaud Meukam (he/him) +* Angelos Kolaitis (he/him) +* Grace Nguyen (she/her) +* Rey Lejano (he/him) +* Joseph Sandoval (he/him/él) +* Nabarun Pal (he/him) +* Drew Hagen (he/him) +* Adolfo García Veytia (he/him) +* Stephen Augustus (he/him) +* Rudraksh Karpe(he/him) + +**Note Taker (pronouns):** + + + +* Jeremy Rickard (he/him) + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * [Marko] Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * 1.26.0 released, was delayed from thur -> friday + * Hit a conflict on fast forward job that was difficult to deal + * See [https://groups.google.com/a/kubernetes.io/g/release-managers/c/elijadh1vdE/m/FQZn2w6NAgAJ](https://groups.google.com/a/kubernetes.io/g/release-managers/c/elijadh1vdE/m/FQZn2w6NAgAJ) for more info and [https://github.com/kubernetes/release/issues/2807](https://github.com/kubernetes/release/issues/2807) + * Patch releases hit some flakes, have tracking issues for those to address before next release + * Included the Go updates as well + * Go 1.20rc.1 is available and is looking at incorporating + * OBS work - Marko is going to work on the KEP + * [Stephen] please keep work life balance in mind, recognize we are a global team + * [Stephen] if we have issues with releases, identify if it’s going to stop release and then communicate with community (first and foremost). Once we communicate we can go back to planning + * [Stephen] release dates are optimistic, we will try our best to meet deadlines but it’s not a failure if we don’t + * [Joseph] what is the criteria for deciding if we need to pick up CVE / go change? + * [Stephen] AI to Joseph to open an issue to discuss in more depth, we have a set of criteria for what is release blocking, includes incomplete features, security issues in coordination w/ SRC. We need to align w/ the Go release cadence + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Released v1.26 on Friday (Dec 9th) + * We had some issues with getting the documentation published. Shoutout to Nate W. Tim B. Taylor D. + * First retro is scheduled for today +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* [James L] 1.27 Shadow Survey + * Intent to open survey this week + * [Stephen] we need to rework RMA a little and identify branch manager + * [https://github.com/kubernetes/sig-release/pull/1958](https://github.com/kubernetes/sig-release/pull/1958) + * [Xander] https://github.com/kubernetes/sig-release/pull/2108 +* [Arnaud] Consensus on [https://github.com/kubernetes/sig-release/discussions/2102](https://github.com/kubernetes/sig-release/discussions/2102) + * Background: we deprecated in 1.25 + * Ideally no new tags for 1.25 and later would not go to GCR + * [Stephen] we need a communication plan + * [Arnaud] communicate now, stop pushing in 1.28 + * [Adolfo] unrealistic to do it for 1.27, work on it during 1.27. When 1.28 is out, 1.24 is EOL. 1.25, 1.26, 1.27, 1.28 will all have the updated registry +* [Jeremy] Cancel any of the meetings for the rest of the month? + * Cancel 27th and January 3rd +* [Arnaud] I would like to shadow the branch Manager of 1.27 +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + +**(Optional) Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## 6 Dec, 2022 (recording) + +**Host (pronouns):** + +**Attendees (pronouns):** + + + +* Marko Mudrinić (he/him) +* James Laverack (he/him) +* Leonard Pahlke (he/him) +* Meha Bhalodiya (she/her) +* Nabarun Pal (he/him) +* Jim Angel (he/him) +* Cailyn Edwards (she/her) + +**Note Taker (pronouns):** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * GO updates - Patch releases 1.22, 1.23, 1.24,1.25 out this Thursday. Final 1.22 release with registry change. ETCD updates will be included as well. 1.26.0 is scheduled for Thursday and will Include the GO updates. + * Image promotion - summary can be found here [https://github.com/kubernetes/release/issues/2789#issuecomment-1339221532](https://github.com/kubernetes/release/issues/2789#issuecomment-1339221532) + * OBS - meeting last Wed. Demo what we are doing. Scalability concerns. The OBS teams thinks it will be fine. The KEP will have all the updates. + * [https://github.com/kubernetes/release/pull/2798](https://github.com/kubernetes/release/pull/2798) (add 1.26 variant for kube cross) + * Pending branch manager handbook updates in the handbook. Publishing bot rules changes will also be added as well. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * (Leo) Everything looks good. Change the release date to 12/8. Discussion related to change was in #release-management. Go/NoGo for the release is in #SIG-Release channel. No blockers at this point. Yesterday red status with Docs but this is now complete. In the release retro we should discuss Ready-to-review. Comms is a go with all the blogs. All blogs will be released before the webinar 1/17/23. Next release team is being assembled. +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* [Leo, Moved from 29 Nov.] How to proceed with the enhancements tracking project board, 1.27 and beyond ([slack thread](https://kubernetes.slack.com/archives/C02BY55KV7E/p1669057356018669)) + * option1: reuse the enhancements project board for next cycle + * (-) Using multiple views might be messy + * (?) Unsure about limits of a single project board (limit of views, total items in a project board) + * (-) issues can only be entered once in a project board, this makes is more difficult to keep record if KEP issues get postponed to the next release cycle + * option2: create a new project board (Enhancements Tracking 1.27) + * (-) We cannot duplicate the enhancements board, manual job atm copying the project board structure + * (+) clean cut each release + * (Nabarun) There are multiple reasons to bite the bullet and use a new board. The tracking spreadsheet was used previously. Lose archival functionality if we continue to use the existing board. Evolution of a KEP has its alpha, beta, stable data. If we tried to archive the history in the same project board. We will have different kinds of views and it will get cluttered. Github PM’s are in touch and have been asked what is in the roadmap and our feature needs have been expressed. This will not be resolved in the short term. + * (James) Does GH API support a move? + * (Nabarun) There is no API endpoint to do this. GH graphQL down the line will be able to support this. + * (Jeremy) Sounds like a new board is the consensus. Action going forward is to make this change. GH issue will be created to track and tag with 1.27 and assign to the team. will create the issue and assign team members. + + + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + +**(Optional) Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## + +**Host (pronouns): Sascha Grunert (he/him)** + +**Attendees (pronouns):** + + + +* Xander Grzywinski (he/him) +* Rey Lejano (he/him) +* James Laverack (he/him) +* Leonard Pahlke (he/him) +* Jeremy Rickard (he/him) +* Arnaud Meukam (he/him) +* Nabarun Pal (he/him) +* Marko Mudrinić (he/him) +* Jim Angel (he/him) +* Adolfo García Veytia (he/him) +* Rajib Mitra (he/him) + +**Note Taker (pronouns):** + + + +* Rey Lejano + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Marko] RC.0 was release last week and went well. RC.1 is scheduled for today. Patch release upcoming next week. Cherry-pick deadline is this Friday, Dec 2, an announcement will go out. Shoutout to Carlos and Sascha for signing for binary artifacts - merged today, and will be kicked off for rc.1, it’s a new subcommand in krel and integrated with the release step. Also OBS poc is going well and successfully completed, all packages are successfully created and published. Kubepkg generating spec files by OBS and all binaries for packages and will discuss later. Also for registry cherry-picks, doing well, all should be merged by now. + * [Adolfo] Started to work on provenance generation. With Carlos started working on a new GitHub actions repository so others can use these tools in other projects. + * Released blog post with k8s sig-infra about backporting the default registry name: [https://kubernetes.io/blog/2022/11/28/registry-k8s-io-faster-cheaper-ga/](https://kubernetes.io/blog/2022/11/28/registry-k8s-io-faster-cheaper-ga/). + * [Jeremy] Should we hold off with rc.1 until it’s (Carlos’ PR) merged – yes hold off until it’s merged. Xander will cut rc.1 + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * On track + * [Leo] On track, waiting for a few Doc PRs to merge. Blogs some folks are ooo. There are several retro topics so will likely have 2 post-release retro sessions. + * [Sascha] If we assume, we cut final release as schedule. Should we open the shadow application. Last year, we had the shadow application process over the holidays. + * [Leo] May need to adjust the shadow app form and will need a few days. + * [Nabarun] Looking for EA, hopefully decide on 1.27 to hand over EA tasks. And will send end of release shadow surveys this week. Leads are in the process of selecting the next lead. + * [Leo] Have been working on the release team shadow app tool, no change in the last 2 months but there are a few small things that need adjustment. Have a session with incoming EA on how to use the tool to generate the markdown file so team leads can have an easier time selecting shadows + +** \ +Open Discussion:** + + + +* Arnaud: Moved from 22 Nov] Stop push images to k8s.gcr.io + * [jeremy] noticed someone asking specific questions about this in[ sig-k8s-infra](https://kubernetes.slack.com/archives/CCK68P2Q2/p1669678777556329) (i.e. what patch releases for 1.25 or later won’t be available in k8s.gcr.io) + * [Arnaud] Followup with blog post, suggest to stop publishing to k8s.gcr.io in the future starting with 1.27. Reason is that will reduce the cost related with container storage next year and we already made k8s.registry.io GA. Suggest to push to 1 registry, currently push to both in each image process. + * [Adolfo] Stopping publishing of new 1.25 tags to old registry and with 1.26 looming, we should really do an effort to not publish releases there. Some details of infra has to be built, need private GCR registry. + * [Arnaud] Still need to define how we’re going to do this. Can say all the existing tags can be pulled but starting in Jan 2023 have to use the new registry. You can still use old tag with new registry since we copied to new registry. For all patch releases with 1.25 going forward will be pulled from new registry + * [Jeremy] There’s a thread in sig-k8s-infra on when can I not pull from k8s.gcr.io. People will use k8s.gcr.io until when it’s no longer available. In favor of trying to accelerate this + * [Arnaud] Would like to issue communication in first week of Jan. Make sure we can stop pushing to old registry in Jan + * [Adolfo] Communicate early and a lot more. We can iron out the implementation details. We’ve had extensive discussions already. We can draft a plan in next k8s-infra meeting and iron out details that’s needed on SIG Release side. We should set expectation that 1.26 will not be published in k8s.gcr.io + * [Arnaud] The real impact is patch releases because people have the assumption to still pull from deprecated registry. Real concern is how to handle patch releases in the future. Discussions around patch releases for 1.24 and 1.23 are that patches will still be pushed to k8s.gcr.io. \ +Need to decide on a plan and implement e.g. change configuration of prow + * [Mahamad] Be nice to do technical work first and get approval + * [Arnaud] We decide the timeline and first look at technical detail before we make communications. We should cover everything before we do communications. + * [Adolfo] If anyone has any objections to not sunset the registry for 1.26, in next k8s infra meeting start working on implementation. Anyone with any concerns? There’s good will from SIG Release with k8s-infra to work on this. + * [Arnaud] Let’s create a slack thread/github issue and have a consensus and by the next k8s-infra meeting then have a conversation and possibly a decision. Should we do a slack thread or github issue + * [Jeremy] We’ve been preferring github discussion +* [Marko] Future of the kubepkg tool + * [Marko] Trying to accomplish to build spec files and to generate them and to publish them on OBS so they can be picked and generate debs and rpms, we need tooling for this to generate spec files. Before when we had rapture and google build to take care of packages, we were pulling like kubectl binary from bucket. For OBS, not going to be work, OBS builds are air gapped and have to provide binaries before build starts. For multi arch builds, Marko create a tarball and pushes to OBS service. For kubepkg, expecte to generate spec files and tarball that can be pushed to OBS service. Now, kubepkg is experimental and never really took off. Wondering, how much do we want to keep backwards compatibility. Do we maintain what’s there right now? + * [Arnaud] Are we modifying specification of system packages? + * [Marko] For each of these, need to generate a dedicated spec file like for 1.25.x + * [Arnaud] We have the template and we generate spec file from template – yes. If we want to leverage kubepkg then it will be delicate to change it. + * [Marko] Do we have any idea if people are using kubepkg and have to maintain kubepkg to work as-is. + * [Sascha] Would like to have spec files as an intermediate solution, expect krel to modify spec files and OBS to build from them. + * [Marko] Will include this in the KEP, would like to meet with OBS folks. So far best timeslot is tomorrow or Monday next week. Will try to push it tomorrow. For kubepkg, we have some logic to build, do we still need this. Do we have any objection to change kubepkg, please reach out to Marko. Will create the PR to update the KEP. + * [Sascha] Updating the KEP is good way to keep discussion going +* [Mahamed from 15th of Nov] https://github.com/kubernetes/release/issues/2734 Notarising macOS binaries + * [Mahamed] Would like to do something similar for Kubernetes for macOS but not familiar with build process. Need to work out where in the build process after binaries are built to add a shell script to upload the binaries to Apple and supply an API key and certificates then it gets notarized and continue with build process + * [Adolfo] We’ve been trying to get rid of bash, any chance this can be written in go + * [Marko] We can wrap it in go + * [Adolfo] Happy to work with Mahamed + * [Marko] Do we need an Apple developer account, is there a fee + * [Mahamed] Yes, we do. Can use Mahamed’s signing keys. + * [Arnaud] Need to talk to CNCF/steering to get an Apple developer account. Asked steering about 3-4 months ago + * [Marko] May be a way to integrate this in krel + * [Marko] One issue is that it changes the checksums. + * [Adolfo] Will have to be done after the build + * [Mahamed] yes, done after gobuild +* [Leo] How to proceed with the enhancements tracking project board, 1.27 and beyond ([slack thread](https://kubernetes.slack.com/archives/C02BY55KV7E/p1669057356018669)) **Move to next week** + * option1: reuse the enhancements project board for next cycle + * (-) Using multiple views might be messy + * (?) Unsure about limits of a single project board (limit of views, total items in a project board) + * (-) issues can only be entered once in a project board, this makes is more difficult to keep record if KEP issues get postponed to the next release cycle + * option2: create a new project board (Enhancements Tracking 1.27) + * (-) We cannot duplicate the enhancements board, manual job atm copying the project board structure + * (+) clean cut each release + +**(Optional) Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## Canceled because of light agenda + + +## + +**Host (pronouns): Adolfo García Veytia (he/him)** + +**Attendees (pronouns):** + + + +* Marko Mudrinić (he/him) +* Xander Grzywinsk (he/him) +* Angelos Kolaitis (he/him) +* Leonard Pahlke (he/him) +* James Laverack (he/him) +* Jeremy Rickard (he/him) +* Frederico Muñoz (he/him)tt +* Sascha Grunert (he/him) +* Aniruddha Basak (he/him) +* Shivanshu Raj Shrivastava (he/him) +* Stephen Augustus (he/him) +* Drew Hagen (he/him) + +**Note Taker (pronouns):** + + + +* Jeremy Rickard (he/him) + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Shoutout to for improvements to image promotion + * Moved Dec Patch releases up to Dec 7 + * Deciding to add one more 1.22 release with backport of registry change + * In progress: v1.26.0-rc.0 (some failing jobs on master informing) + * Will open a github discussion for stopping build of rc.0 for next patch, would apply to no earlier than January \ +GH Discussion: [https://github.com/kubernetes/sig-release/discussions/2090](https://github.com/kubernetes/sig-release/discussions/2090) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * We have a couple of failings on master-informing for a few days. Focus on resolving these.. + * Flake detection session in todays [sig-testing meeting](https://docs.google.com/document/d/1z8MQpr_jTwhmjLMUaqQyBk1EYG_Y_3D4y4YdMJ7V1Kk/edit) (presented by Antonio Ojea) + * All Code freeze exception requests completed or postponed to next release :) ([project board view](https://github.com/orgs/kubernetes/projects/98/views/22)) + * Retro discussion item added, which likely lead to small process adjustments for the release team: “_Improve the release process to better incorporate large PRs into releases”_ (see [slack msg](https://kubernetes.slack.com/archives/C2C40FMNF/p1668210928263979)) + * General consensus - not really a sig release issue. Reviewer bandwidth is really the issue +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* [Mahamed] [https://github.com/kubernetes/release/issues/2734](https://github.com/kubernetes/release/issues/2734) Notarising macOS binaries + * [puerco] Mahamed doing notarising for knative binaries, will defer to next meeting since Mahamed had a conflict +* [Puerco] Announce of backport of registry.k8s.io to older branches + * Background: working to shed serving load for artifacts from google cloud to better leverage budget from other cloud providers. KEP open for a long time, things are more pressing now. Short on budget this year by several hundred thousand USD. limited donation from Google to alleviate the burden. Working with sig-k8s-infra to figure out how to shift the burden. AWS taking some load now, but need to accelerate the process. + * Many users still using older versions, change didn’t get introduced until 1.25. Want to back port these to releases under support still. Would also be good to take it back to 1.22 +* [Leo] Tracking image updates and making sure new images are used (done in two PRs, it happened that the second one that allowed using the container image was forgotten…) (slack [thread](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1668465718963579?thread_ts=1668461996.684349&cid=CJH2GBF7Y), mentioned by dims) + * Release Managers need to follow along with these and ensure last mile is completed + * When someone, not from rel manager, opens image bumps, we should always assign a release manager +* [Jeremy] [https://github.com/kubernetes/sig-release/issues/2083](https://github.com/kubernetes/sig-release/issues/2083) Use prow to approve cherry picks +* [Jeremy] Couple of changes to be aware of in k/release: + * [https://github.com/kubernetes/release/issues/2702](https://github.com/kubernetes/release/issues/2702) (you can override distroless and builder for gorunner now) + * [https://github.com/kubernetes/release/issues/2759](https://github.com/kubernetes/release/issues/2759) (removed darwin/386 from kube-cross) + * Investigating removing the pre-compile of std lib from kube-cross +* [stephen] like to propose changes to leadership roster + * Will open an issue + * Propose Jeremy from TL -> Chair + * Propose Verónica as new TL +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## + +No meeting, see subproject updates below: + + + +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Code freeze [today](https://everytimezone.com/s/d3187dcc) 🥶 + * 3 Exception requests received, one accepted, two in review + * Weekly Monday meeting starts next week + * A couple of failiXanderngs on [master-informing](https://testgrid.k8s.io/sig-release-master-informing) but nothing concerning now, the CI signal team chases people + * Deprecations & Removals blog should be finalized soon + + +## + +**Host (pronouns): **Sascha Grunert (he/him) + +**Attendees (pronouns):** + + + +* Leonard Pahlke (he/him) +* Xander Grzywinski (he/him) +* Angelos Kolaitis (he/him, Bug Triage Shadow) +* Jeremy Rickard (he/him) +* Rishit Dagli (he/ him, Docs Shadow) +* Carlos Panato +* Rey Lejano (he/him) +* Marko Mudrinić (he/him) +* Benjamin Elder (he/him) +* Cici Huang (she/her) +* Ashwin Kumar Uppala (He/Him) +* Adolfo García Veytia + +**Note Taker (pronouns):** + + + +* Jeremy Rickard (he/him) / Jim Angel + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Deckers + * Drew Hagen (went to KubeCon last week and met the SIGs and interested in sig release) + * Ashwin (from the chat) +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * We found a bug in bom due to google’s license classifier. This broke release process. Fixed over the weekend, Carlos ran a test. + * OBS effort going on (demo below) + * Kpromo changes in place, just files that were touched by merging PR. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * All good so far! + * Mid-cycle RT status [email send out](https://groups.google.com/a/kubernetes.io/g/dev/c/_nToVaHVN1Q) + * First Retro Meeting scheduled for tomorrow :) + * Code Freeze next week! + * (rey) sig instrumentation has a PR open to add a new doc and would like to discuss having release team docs team handle part of that generation + * [https://github.com/kubernetes/website/pull/37522](https://github.com/kubernetes/website/pull/37522) + * PR not merged yet. + * (sascha) discuss in the retro. It’s a process change, seems like a small adaptation. If it’s a huge process change probably worth a kep. + +** \ +Open Discussion:** + + + +* [Sascha] Building deb/rpm packages via rapture in krel: [https://github.com/kubernetes/release/issues/2737](https://github.com/kubernetes/release/issues/2737) + * We can integrate the package build script into krel since [https://github.com/kubernetes/release/pull/2708](https://github.com/kubernetes/release/pull/2708) got merged + * The code would be required in any case, because we can exchange the rapture implementation with something different (like calls to OBS or similar) + * Jeremy will work on this with Xander and Cici. + * Open question: where should we put the binaries. + * [ben] put in an unadvertised gcs bucket (so people don’t rely on them). gsutils cp to a new bucket, build admins can download to the place they need to go. +* [Marko] OBS demo (10-15 minutes) + * See video for demo (it’s cool!) + * QnA + * Jim: ISV, is it in the repo name + * Yes + * Jason: problem moved to proxy in front + * Ben: How does each release look? + * Access to OBS and folder(s)? Per release + * Puerco: How does auth work? + * Marko: token - needs discussion + * Could use OSB itself (vs. golang specifics). Invoked from krel to start. No OBS library for go + * Ben: Builds are failing for certain archs (arm, powerpc), do we know what the blockers are? + * Marko: Can be solved, but we need to determine priority and if this is blocking or not. How challenging it would be to work around it. + * Ben: Why is it only those platforms not building? + * Can build for each platform but those have specific errors. + * Puerco: Limitations on free hosting provided? + * Marko: They are sponsoring us. If we run into limitations we can self-host if needed. From what has been seen, it shouldn’t be an issue. Storage is a minor concern or potentially an area to pay attention to. No issues expected. + * Sascha: Can rename a generic build add-on name to use as a wrapper around the API? We have to check if we want to support pulling in deps and which ones. + * Marko: Need to discuss more on the side. + * Ben: Aside from Windows, the rest of the build is possible to run locally and produce the outputs yourself. On the otherhand, we need ref systemd specs but maybe those packages are not the _only_ way to install them. If we can host it on AWS, we are tight on budget and need more specifics. + * Marko: Hosting the packages? + * Ben: Yes, hosting GPG keys and downloads for end users is huge and we have no clue how expensive the current packages are. They are not a product. + * Marko: OBS folks say they can do it and don’t expect issues. But shares the concern of the size of k8s. + * Ben: opposed to us hosting it and _also_ having this problem. Hope to get more folks involved. + * Jason: End goal is to go all on OBS. Since we have span across cloud providers; we want to have a discussion if we introduce a proxy / cache / layer in front to provide flexibility to migrate in the future or deal with URL shuffling x2. + * Migrating to selfhosted perk: OBS can provde the GPG keys. + * Ben: I think we have one today that we can point with (apt.k8s.io?) + * Marko: How do we want to proceed? Try to get all packages working and then work on krel integration. But we should have some green light that we are going to go with OBS. + * Jason: Unless objections, start putting together a roadmap detailing the current problems, gaps to implement, and action plan (building) and add it to the KEP (what is alpha/beta/ga). + * Jim: Need to define low-bar RED/GREEN since OBS seems like a go, let’s figure out what that means. + * Jason: KEP would help provide that structure + * Ben: I need something to point to Googlers about the path that the project is taking over. They are getting pulled into a larger team and going away. If we have a doc, we can use it to point. + * Jason: Agree. We can point to the POC and to Marko’s demo. KEP is the next steps for “official doc” instead of “up in the air” we’re working on it + * Ben: Will bring KEP to breakout and to their managers that we are making progress and thanks! + + +## - KubeCon Contributor Summit Special + +**Where: **412A (Huntington Place Convention Center) 1 Washington + +Boulevard: [https://sched.co/1CXNY](https://sched.co/1CXNY) + +**When:** 10 am - 12 pm + +**Hosts:** + + + +* Sascha Grunert +* Jeremy Rickard +* Carlos Panato +* Adolfo García Veytia + +**Attendees (pronouns):** + + + +* Joseph Sandoval +* (Orlix) +* Rey Lejano (he/him) +* Leonard Pahlke (he/him) +* Grace Nguyen (she/her) +* Tim Pepper (they/them) +* Josh Ferrell (he/him) +* Nabarun Pal (he/him) +* Rishit Dagli (he/him) +* Nate W. (he/him) + +**Note Taker (pronouns):** + + + +* + +** \ +Agenda:** + + + +* SIG Release evening hangout planned for tomorrow + * [Jeremy] There Few places with outdoor options, otherwise go to the alleyway called the Belt (where the contributor summit social was), also a brewery nearby, and Detroit Shipping Company is an option. [Coordinate over Slack](https://kubernetes.slack.com/archives/C2C40FMNF/p1666707534307859). + * +* SIG Release talk tomorrow 11am: [https://sched.co/182MK](https://sched.co/182MK) (142 ABC) + * [Sascha] All are invited to join. Hoping to have Stephen join us on-stage. + * [Carlos]Who are the speakers? + * Sascha, Adolfo, Carlos +* State of the binary artifact signing KEP ([issue](https://github.com/kubernetes/enhancements/issues/3031)) ([KEP](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/3031-signing-release-artifacts)) + * Umbrella tracker: [https://github.com/kubernetes/release/issues/2586](https://github.com/kubernetes/release/issues/2586) + * New Step: [https://github.com/kubernetes/release/issues/2618](https://github.com/kubernetes/release/issues/2618) + * [Sascha] tl;dr: KEP is about artifact signing and first step is container image signing + * Plan to go from alpha to beta by completing signing effort, implement new steps and move signing logic to dedicate gce job. Will need to start implementation. + * [Adolfo] [https://github.com/kubernetes/release/issues/2618](https://github.com/kubernetes/release/issues/2618): Discussing ways to sign artifacts, we weren’t doing a few things correctly like signing within the release process and not the final release process. So we had two signatures, one from within the release process and one after the image promoter. We did the same with provenance attestation, we were creating inside the release process so we will move to outside. Work on moving provenance attestation is done, Arnaud approved the moving of the repository to Kubernetes org, now we can use new attestor for the builds. Missing is adding signing files/images to krel to be signed outside of the build process. Who is interested in working on this issue? One of the missing tasks in signing KEPs is the SBOMs. Its tricky. The idea is to rework how we are generating the SBOM structure. Leader SBOMs. Attach the SBOMs to the image. SBOM code is in a branch. Waiting to merge. + * [Dims] What will the user experience be with Debian or Ubuntu when installing K8s or Kubectl? + * [Jeremy] do we have existing issues with the Signing KEP? + * [Adolfo] Referring to umbrella issue 2586. File signing + * [Nabarun] Nabarun worked on file signing, person who worked on initial tests for file signing modified it alot, there were some API changes. When we run the new step, what are we going to do? Promotion/building happens twice, what do you sign, when do you sign. We should write this in the KEP before diving into the implementation. We should spend a few minutes defining what is signed, Go through SBOMs and list through them and what we generate then sign it. Another choice is to go to GCS bucket, then sign from there. + * [Carlos] Still working on the issue + * [Adolfo] Discussions on if we should build signing code in krel. We need to implement it ourselves and will take everything and resync the bucket + * [Sascha] File signing will look different from container image signing since we don’t have artifact promotion for binary artifacts. Signing container images with cosign, need to have binary artifact promotion in place and will do it with k-promo. Consider changing how cosign works and work in our parallel use-case. + * [Adolfo] Hit some issues with cosign, issues with how we interact with sigstore, fixed a race condition. Also hit with a rate limit which is now fixed. + * [Sascha] We will split up the issue, Marko is interested in working with the files +* Package Building POC + * Options to review: [https://github.com/kubernetes/enhancements/issues/1731#issuecomment-1286919467](https://github.com/kubernetes/enhancements/issues/1731#issuecomment-1286919467) + * [Jason] We’re looking out how to do deb/rpm packages, unlike the binaries and container images we can’t leverage cosign and they would work in various distros that consume them. Now we use gpg keys that live for the life of repo, right now that process is controlled by Googlers. Looked at a few ways like build similar to Google and run in Cloud Run. Decided to look at third-party systems that would host them. Looking at SUSE Open Build Service (OBS) we don’t interact with gpg keys, it’s controlled by OBS and leverage APIs to kick off the builds and will host the packages as well. Since we don’t build deb/rpms in a normal way, normally we would build from the source, we pre-build binary images and want to use binary images in the release process. We can also run OBS in our own infra if we want community owned infra. Marko tried to POC this out. If we decide to build own process in Cloud Run then we have to manage our own gpg keys. If this poc goes well, then only do a single migration and we can put a proxy or redirector on our own infra to point to obs packages behind in case we want to go self-hosted in the future. We can write a KEP after deciding if we want to move to OBS or have own OBS instance + * [Adolfo] I’m a bit concerned with the state of that one. From I’m hearing from Google we need to make this happen now. + * [Jeremy] Chatted with Ben. We need to migrate now. We run Rapture or whatever in the interim. It should be big priority and move quickly. Get a firm commitment if we need to do this before the next release. + * [Sascha] Prefer on current path and get more time to do the migration. Instead of changing twice. + * [Jason] Think about impact to users. It will be a few releases before we stop publishing from Google. We need time for a good rollout and limit impact to our users and ClusterAPI and other types of K8s installers. + * [Jim] Can we run both paths in parallel? Have both channels running then make a switch? Or in case we lose Google support. + * [Adolfo] One thing to buy us some time is if we build RPM’s. Alignment is taking as much time from them to schedule easily. Google people who help us are doing it on their own time.We can move the build inside KREL. + * [Tim] What would be the consequences of deprecating RPMs and debs **_signing_**? + * [Arnaud] Upgrades. Issue new GPG key. The system needs to verify the packages. From a package perspective. We have sub-projects that use this. + * [Jason] A couple of aspects. No assurance that the package is from K8s community. Optics. We used to have signed packages and now we stopped. It could raise concerns with production readiness. + * [Tim] We don't provide that assurance. Google does. That is different from the project.We should focus on what we provide to the community. Change your code or stop verifying? I like the optics of not verifying. In the short term we are an open source project and the assurances we provide. + * [Jason] I don’t disagree. Not everyone has free access. By publishing and signing they can verify. + * [Tim] how would Google verify if community built rpm/deb are intact ahead of signing (eg above that’s mentioned as an optimization to save them time). How do you download and verify trust? + * [Sascha] Could cosign rpms/debs, so Google could gpg sign them. I’m doubtful companies are ready for this. + * [Jason] All companies were using upstream + * [Dims] I’m with Jason, we continue signing. If we want to swing to the right or left. Let’s look at policies across the board: eg: Apache Foundation has a source only releases policy for example. Don’t assume they are going to be there. It might break. We have to extend this outside of SIG-Release. Tell everyone this is the way we are going. + * [Adolfo] push back a little bit on not signing, working on north start vision document that aligns priorities we want to do, two of those makes project more consumable and other one is to make it more secure. Not signing is a regression, understand that folks are trusting Google and not us. Worried if we take the build of packages within the current release process, isn’t that lead to us handling it ourselves instead of shelving it to OBS. + * [Tim] Agrees, in abstract think less we build we can focus on high value artifacts. Would focus on containers over rpms/debs, could we also do fewer OS’s and hardware architectures? We always struggle on human resources. + * [Sascha] Could also revive discussion on supported platforms and have to evaluate which platforms and architectures are necessary. For example, also talk about don’t want to ship binaries for PowerPC which means we don’t build packages for PowerPC + * [Jason] Remove support for a platform/arch then makes it harder for folks + * [Dims] If we do something, we should do the same for all artifacts. If we sign one set of artifacts then sign all. If we don’t have enough funding then that is another question. How do we reduce the sizes, the count, etc and do the policies for them and implement them + * [Sam] Here are two options, we outsource building packages to distributions like deb, ubuntu, and another option is other vendors can produce the builds. + * [Jason] Outsourcing to distribution vendors - the problem is distribution vendors have tight policies with releases on what they can update. Like release Fedora 37 then only release patches after. Nice about OBS, can build packages for all in one system. More burden if we have to interact with multiple vendors. In terms of vendors supporting this, building packages or hosting + * [Jeremy] We want to get to the point that everything is community owned. If vendors are willing to do this today, are they willing to do it a year later. Want the project to have more control and don’t want to repeat this process + * [Dims] Usually we get benchmarked with Docker. Lets try to replicate the same thing + * [Jeremy] What should be the conclusion + * [Arnaud] We should try to address one major issue. How do we shift the responsibility of shipping from Google to the community? Hosting and signing is a different conversation on policy. + * [Jeremy] Immediate need is to shift responsibility, when do we want to do that? + * [Adolfo] Google wants to see what happens for this release. Would have to check how much we need to rebuild. + * [Jason] 1.26 is too soon. 1.27 parallel with Google. 1.28 with Google packages existing but default to ours. + * [Jeremy] + * [dims] We should do alpha this time. + * [Nabarun] Can we run the build script ourselves and Google do the signing. + * [Jeremy] Do we run that in Google cloud build job or on the CI? + * [Adolfo] Everything new should be new and automated. + * [Arnaud] Just a matter of days - we can try to build and push to something community owned. + * [Sascha] What are the reasons, personas? + * [Arnaud] Remove the needs for a human doing it + * [Adolfo] We also have SLSA compliance and that breaks. + * [Nabarun] The reason we can split is it makes it easier for Google. 2 hours from us telling them the release is ready until they finish it. Humans are still in the process. + * [Jason] Will Google handle publishing? + * [many] yes + * [Arnaud] Google will need less time for publishing which is better + * [Sasha] We’re at the point with OBS POC that we can move that step to now with krel + * [Jason] OBS is strictly a POC right now. Let’s not be presumptive and see if it meets our needs + * [Sascha] This is also possible with other implementations. Rapture script runs Docker so won’t have issues running in Google Cloud and push to the same location. + * [Arnaud] Pushing to Google right now and we need to work on this + * [Sascha] That’s the interesting part - we need to change more than the script. + * [Arnaud] We can introduce: + * Change build script to push to GCS + * [Nabarun] If the script builds the packages and creates them in the environment where the script runs, if we run script in krel release then do a gcs sync. Have to investigate engineering efforts that work for Google build admins since release is near. AI to investigate and create solid issues and make progress + * [Jeremy] Create an issue to investigate and how to put in current release step and investigate if its feasible for 1.26/1.27 in parallel to package building poc. + * [Adolfo] In two weeks from now, anyone who has ideas, lets jump on a call and split up the work. The POC will keep going. https://github.com/kubernetes/release/pull/2708/files#diff-4b4aa17612cae24c2c02cff5d89180524381036f4b77b470a7137bde5c76ddce +* State of Image Promotion? + * [Adolfo] We are currently undergoing change with the infrastructure. TLDR. We are not publishing the images we promote. Process jumped from 30-1:00 to 6:00 hours. When we do image promotion. We send a ton of requests to registries. There is the signing process. When we start to promote to new regions it adds to the time to the process. We have a bug that is doing too many operations during promotion pre-stages. + * [Sascha] We now do some adding to caching. The idea is digests and signatures should not change. We could serialize. This makes it insecure. We need a huge amount of connections to registries. Pushing images takes time. We can reuse connections. We have to see if it speeds things up. We could also split up the jobs by image name. + * [Mahamed] I suspect we are hitting quotas on AR but no metrics because it is public(gcp product limitation). Also logs files are insanely large 180MB+ + * +1 to sharding the jobs + * +1 to using git diff to calculate the actual changes we need to check + * [Adolfo] We’ve been doing that manually with release managers cutting the releases + * [Arnaud] In the past we used Google Registry this service got deprecated. To optimize cost, we create registry in each region where we have a proxy. The quota requests not empty but we’re at 90%. We don’t have support to raise it. Mitigation is multi-region and that might bring us back and stabilize. Release process should not take too long, don’t want RMs to finish at 3am + * [Adolfo] We’re starting to see image promoter logs, rate limit errors + * [Sascha] We changed concurrency from 10 to 100 so we should reduce that + * [Arnaud] There was no impact, we don’t see utilization if we tweak the number of threads. + * [Adolfo] The thing with resource usage, the promoter gets stopped waiting for traffic, increasing the concurrency may help with time but then we hit rate limit error. All of this started happening a few weeks ago so we’re trying to understand it +* [Grace] Concerns with KEP authors about extending code freeze due to KubeCon + * [Nabaran] We should be ok to ship less, for sustainability. People shouldn’t push hard. About code freeze extension, since this release is short, need to make CI stable after code freeze. + * [Sascha] We outlined the schedule before, was there anything we could do better. Suggest we take action on this item and send communication to k-dev with the decision + * [Ben] This is an expected interruption. If it is not expected, then we should communicate explicitly. + * [Veronica] This is a repeating experience that people are surprised. + * [Rey] Historically, we add an extra week for the last release of the year. Because of the delay from 1.25, the release is shorter by a week. Just a reminder, we used to do 4 releases a year. + * [Nabarun] A formal request can be made to the mailing list. +* [Arnaud] Backport registry.k8s.io to maintained versions. + * Starting v1.25, k/k repo use registry.k8s.io + * Proposal to to backport to v1.23, v1.24, v1.25 + * Infrastructure impact on the users/community + * [Arnaud] Backport change to 1.23, 1.24, 1.25, and will talk to SIG Arch about impact and will also talk to SIG Node and SIG Cluster Lifecycle but from SIG Release perspective, this is changing endpoint in code base not about impacting users. So impact is more at the infra level. Can have a changelog in patch release. Patch release will introduce change to check your infra policy + * [Adolfo] Doesn’t feel comfortable to backport change, may break users. Want to get a full understanding of all the points where we change it. Can’t keep promise of not backporting change. + * [Ben] True but also have to understand container runtime have to permit this traffic, if we don’t have consumers to use it as possible then we don’t reduce spend on google cloud account. We see a lot of traffic on very old Kubernetes releases + * [Sascha] Can backport back to 1.22 + * [Arnaud] If we want to cut a 1.22 release then need to do it this week, don’t want to push it just asking it + * [Adolfo] Need to have further discussion + * [Ben] Where we need to make changes, kubeadm and kubelet (for pause) so need to talk to cluster lifecycle and node. +* [Ben] Stop promoting all 3 GCR registries. Only one. + * Blockade to stop touch manifests for k8s.gcr.io + * [Ben] Trying to get the GCR team to redirect the existing registry to a new endpoint. Users just see gcr is breaking them. Gcr team is having problems. There is 25 times more traffic hitting old versus new because users are using old versions. Do similar to vanity domain flip. + * [Adolfo] Sounds like a more reasonable deprecation path. The question about patch releases is interesting. + * [Ben] We have to not be afraid about breaking some users. The same users broken by this change will be broken by that change. We dont have transparency into the resources we have. We have to break this pattern. We have to seriously consider it to get the traffic shifting. In the meantime if we get resources from other providers we are stuck. Its hard to get Google to commit more resources. We can’t get traffic onto other providers if we don’t move users over. Users will need to mirror images or provide new endpoints. We can’t support users who expect free registries. + * [Adolfo] We are sensitive and we know. + * [Ben] We should stop publishing tags for 1.25. We could consider changing the default. They can opt out and use old registries. Its an OCI compliant registry. + * [Adolfo] There is some doomsday scenarios for some companies. + * [Ben] Users should consider mirroring. We don’t have SRE. If you need uptime you should mirror. Backport default to past images. For 1.25 forward we should stop publishing tags. We have to SIG-clusterlifecycle and SIG-Node. + * [Jim] Comms is the action item. Blog posts. Social media messages for awareness. + * [Ben] We were pushing this last year. We knew we are over budget. So much latency taking effect. Project is out of credits starting in Nov. Next year's budget is going to be tight. This is by far the largest cost. Taking over ⅔ of the budget. Binary downloads are billed directly to Google. Traffic shifting is the best thing we can do. My POV: we have to get costs sustainable. Going forward start to shift traffic. Letting end users download is a heavy cost. We are exceeding the budget for the community. If we can sort out the download costs. CI costs are more reasonable. We could fit CI and ancillary costs without the massive download bill. + + +## + +**Host: **Sascha Grunert (he/him) + +**Attendees (pronouns):** + + + +* Marko Mudrinić (he/him) +* Rey Lejano (he/him) +* James Laverack (he/him) +* Stephen Augustus (he/him) +* Jason DeTiberus (he/him) +* Jeremy Rickard (he/him) +* Joseph Sandoval (he/him) + +**Note Taker (pronouns):** + + + +* Rey Lejano (he/him) + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] 2 topics: + * In process of moving slsa attestor, repo is being donated from Adolfo’s private repo to Kubernetes org. + * State of the promotion process, image promotion is running very, very slow. Because we are now promoting to a ton of artifact registry regions, before staged images moved from staging registries to prod registries and used to copy to 3 locations now its to over 20. Needs a significant re-thinking of whole process. Everyone is coming up with solutions. Adolfo thanks release managers for releases that are taking more than a day to complete. We are trying to remediate the situation and also trying to rethink how promotion looks like in the future + * [Sascha] Do we have a tracking issue + * [Adolfo] No issue, there is an open PR to speed up process with auth but no overall issue + * [Arnaud from Zoom chat] [https://github.com/kubernetes-sigs/promo-tools/pull/632](https://github.com/kubernetes-sigs/promo-tools/pull/632) + * [Sascha] Also affects presubmits + * [Mahamad from Zoom chat] [https://github.com/kubernetes/k8s.io/issues/4374](https://github.com/kubernetes/k8s.io/issues/4374) + * [Adolfo] Rely on the state we know from git, promote without looking at everything in the image. Will present changes at the Contributor Summit next week + * [Mahamad] Need to rethink assumptions, get presubmits sped up by looking at diff vs head and branch. There are other ideas going around but nothing else in writing. We need to promote to an additional 15 locations on but will add 2-3 hours, don’t need to do this yet but sometime next year. + * [Adolfo] There is also a plan to mirror images in AWS + * [Mahamd] That is automated, mirror from bucket in GCR. We copy bucket from Google Cloud to S3. Also need to add more regions in AWS. + * [Arnaud from Zoom chat] This one should really help us speed up things: [https://github.com/kubernetes-sigs/promo-tools/issues/633](https://github.com/kubernetes-sigs/promo-tools/issues/633). + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + +** \ +Open Discussion:** + + + +* [Sascha] Sent a mail that we moved the Contributor Summit SIG Release meeting to Tuesday: \ +[https://groups.google.com/g/kubernetes-sig-release/c/yt55iBoxmag](https://groups.google.com/g/kubernetes-sig-release/c/yt55iBoxmag) + * [jeremy] related: we will have our social event on Wednesday, will create an invite when we lock down a venue + * [Sascha] We assume people are traveling on Monday so moved the Contributor Summit SIG Release meeting to Tuesday at 10am EDT (local time) for 2 hours + * [Jeremy] RE: Social, 2 venues are high on the list a rooftop and an outdoor sitting place. Jeremy will call today and try to reserve a place +* [Marko] Discuss next steps for debs/rpms building and OpenBuildService proof-of-concept + * KEP-1731: [https://github.com/kubernetes/enhancements/pull/3434](https://github.com/kubernetes/enhancements/pull/3434) + * We started working on a proof-of-concept for integrating with [OpenBuildService](https://build.opensuse.org/) from OpenSUSE + * (+) The initial phase of POC is successful. We managed to build both debs and rpms for amd64 for all packages that we already publish (cri-tools, kubernetes-cni, kubelet, kubeadm, kubectl) + * isv:kubernetes project: [https://build.opensuse.org/project/show/isv:kubernetes](https://build.opensuse.org/project/show/isv:kubernetes) + * home:xmudrii:k8s-poc project: [https://build.opensuse.org/project/show/home:xmudrii:k8s-poc](https://build.opensuse.org/project/show/home:xmudrii:k8s-poc) + * (+) Installing published packages works as expected. Migrating from Google to OBS packages also works as expected. Migration are steps are: + * Remove old repo and GPG key from your system + * Add a new repo and GPG key + * Run update/try to upgrade packages + * (*) Build times were variable -- building in isv:kubernetes was fairly quick, but building in home project was very slow. Considering that we’ll building in isv:kubernetes, I don’t consider this as a problem. + * (*) Spec files and building process are slightly different than what we got used to + * We were able to reuse the existing RPM spec file. There were some minor changes needed to get the build working, but nothing significant. + * We had to modify Debian spec files to make building work. I decided to opt-in for debtransform instead of using dpkg-buildpackage (one less dependency). One problem that I encountered is that there’s no internet connection in Debian builders, so we had to download the binaries beforehand (previously we were doing that as part of the build process) + * There are few problems on how to organize those RPM and Debian spec files: + * Both Debian and RPM spec files are living in the same package and same directory (in OBS). If we push some change to Debian spec, that’s going to trigger building RPMs as well + * Debian packages seem to require having a dedicated repo in OBS for each package. On the other side, RPM spec file can define multiple packages, so only one package in OBS is needed. + * Relevant to the previous point, do we want to have one RPM spec that defines all packages or have a dedicated RPM spec file for each package? + * (-) It’s still unclear how are we going to handle multi-arch builds + * We would have to create a tarball that has binaries for all architectures and then push that tarball to OBS. Potential problem is storage requirements (binaries for all archs can be 1GB+) + * Alternative approach is to build binaries inside OBS, but then debs/rpms are installing different binaries from what we release. There’s also a potential problem with signing those binaries + * (-) It doesn’t seem possible to reuse packages built by OBS + * We can’t build packages in OBS and then send those packages to Google folks to publish them + * [Marko] Initial phase is successful, was able to built deb/rpms for packages we publish. Used 2 projects for this, isv:kubernetes and Marko’s home project. Isv:kubernetes is most complete and can take a look at how it looks like. How its organized, this is fairly ok. Migration process is fairly simple. Build times are ok, initial experience in home project some stuff was slow but in isv:kubernetes is higher priority and everything was finishing in a matter of minutes. Building packages shouldn’t take more than 30 min. What is different and need to pay attention, spec files need to be changed. + * For deb packages, we have a package with dedicated kubelet, kubectl etc + * For rpm, everything in same spec file + * OBS wants 1 spec file but Marko prefers multiple spec files + * Not so good, not clear on multi-arch builds, would have to create a tarball for all architectures and push tarball to OBS. But this will take a lot of storage + * Will also have problem with signing, won’t be able to use keyless in OBS + * Next steps, meet with OBS folks and use hosted OBS so we don’t have to run our own OBS + * [Jason] OpenSuse already said they are willing to host on hosted OBS infra and if we want to migrate then we have flexibility to migrate. Recommend to put proxy in place, so if we need to change where we host images again (hosted/own OBS), users don’t need to change URLs again. + * [Sascha from Zoom chat] OBS supports service files, for example to download files from builders: https://github.com/openSUSE/obs-service-download_files + * [Adolfo from Zoom chat] A redirector like with the images + * [Marko] Next step is to get multi-arch stuff working + * [Sascha] What is your overall impression with integration with our current process and how OBS fits? + * [Marko] This is still something we have to see, we have to create our own API layer to use. We don’t need too much, like need to push files + * [Stephen] With regards to services, a value in OBS and having a spec file per version then people can build themselves. I would continue to try on that offline path assume their build environments will be air gapped. In the tool, we shell out so many things in general, this will be a trigger somewhere like a separate GCB job or another step in the GCB process. + * [Stephen] this is great work, do you need anything from SIG leadership side? + * [Marko] need to decide on how we use service files/specs per package/rpm, need help to make that decision + * [Stephen] we should really have separate spec per package not the omnibus that we have now. We’ll need to think about how this will change how users will consume packages. Need to review supporting docs (install docs), etc and adjust expectations. As much as possible, we should be clear in communications and break things if/when necessary. Backward compatibility a nice goal, but shouldn’t sacrifice overall goal by being hampered by BC. + * [Marko] good feedback, thank you. Next step will look into multiarch and meet with OBS folks to review and get feedback. Will continue working on this and report status. Should get KEP merged. + * [Sascha] anything that will compromise the community owned aspect of this, can release managers debug everything? + * [Marko] not truly community owned because running on OBS infra. We can give access though, and solves problems like GPG keys because only public keys will be exposed to those accesses. Running OBS could be done on our infra but its not ideal. Should satisfy the needs and spirit + + +## + +**Host: **Jeremy Rickard + +**Attendees (pronouns):** + + + +* Rey Lejano (he/him) +* Jason DeTiberus (he/him) +* Marko Mudrinić (he/him) +* Xander Grzywinski (he/him) +* James Laverack (he/him) +* Arnaud Meukma (he/him) +* Cailyn Edwards (she/her) +* Leonard Pahlke (he/him) +* Mahamed Ali +* Joseph Sandoval (he/him) +* Carol Valencia (she/her) + +**Note Taker (pronouns):** + + + +* Rey Lejano (he/him) + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Marko] Migration from GCR is completed, thank you to Mahamed Ali for creating all the PRs. No other major updates other than a set of patch releases tomorrow for 1.23, 1.24, 1.25, and another alpha for 1.26 is coming out today. Also discussing on Slack the golang update + * [Jeremy] 1.26 alpha cut is in stage and kicked off with Cici, first release done with new image promoter changes + * [Jeremy] The package publishing stuff has some urgency, any updates or blockers? + * [Marko] We started discussion with Jason and Rey to start testing, Sascha also updated KEP on publishing. We’ll try to do poc and see how it all works + * [Adolfo] What’s the problem on the Google side? Why the urgency + * [Jeremy] Less availability on the folks, Ben suggested to build packages and Google folks can sign and publish them. That team has less bandwidth and not much more bandwidth in the future. The initial proposal was that we build packages and set them up for them to sign. And now OBS can possibly take care of everything + * [Jason] We take pre-signed pre built binaries. We also discussed doing a similar poc in cloud build + * [Adolfo] From Adolfo’s understanding was that poc in cloud build was set aside and to do poc with OBS + * [Jason] Waiting on access to project for the poc, we will work on it in personal project for now and then migrate. Marko will start in a personal project so we’re not working on it + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Collecting major themes and [deprecations](https://kubernetes.slack.com/archives/C2C40FMNF/p1664813648155459) for the mid-cycle blog + * [Doodle](https://doodle.com/meeting/participate/id/bW7DWyXb) out to find a time slot for the APAC friday meeting + * (enhancements freeze in effect) Exception requests: + * (🟢) CCM Webhook Framework: [request](https://groups.google.com/a/kubernetes.io/g/release-team/c/ZODxaU5lKeA) + * (🟢/🔴) Subsecond Probes: [request](https://groups.google.com/a/kubernetes.io/g/release-team/c/nmS7WkOR8o0) + * (🟢) AppArmor Support GA: [request](https://groups.google.com/a/kubernetes.io/g/release-team/c/Z219Bv-BT6U) + * (🟢) Dynamic Kubelet Config : [request](https://groups.google.com/a/kubernetes.io/g/release-team/c/WztK1I66o_A) +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* [Added by Rey, from [Slack thread](https://kubernetes.slack.com/archives/C2C40FMNF/p1664914638248429?thread_ts=1664874197.295239&cid=C2C40FMNF)] Proposed KEP, [KEP-3344 Release Team Roster](https://github.com/kubernetes/enhancements/pull/3347) + * [James] Process KEP to change how release team is built/assembled, doesn’t change roles/jobs. Motivation is the experience for new contributors is laid out, experience for leads is laid out but experience for returning contributors in the next release is not great because you go through the same pathway and we see people falling out of being on the release team. This KEP gives a way for folks to take breaks and return to the release team. After shadowing for a few cycles, then the release team will be built from both stable roster and new shadows. + * [Veronica] Stephen also wants to empower people and not want to be a deterrent for people not to join based on their time commitment. + * [James] The intent for the KEP addresses the concerns mentioned. Should we change focus as Sascha’s suggestion, returning path to release team + * [Jeremy] Maybe the term roster, may not be the approach. Maybe refining documentation to include how folks can come back. Maybe focusing on Sascha’s comments is a good first step + * [Veronica] Similar to now, people tend to do the same tasks. Also there is a sense of trust earned with experience. + * [James] People have been ambivalent or -1 so this may not have the support. Taking this as a signal that this needs to change. Do we think there is still value. Do people agree with me that process for people returning is an issue – consensus is yes + * [Leo] Teams differ from complexity as well so some teams might benefit with more returning RT members. Might be good for the retro and brainstorm about the teams + * [James] Will redraft KEP to be about returning contributor path +* [https://forms.gle/3qzSmYgYkdbrJkzS7](https://forms.gle/3qzSmYgYkdbrJkzS7) ← vote on when the sig release social should be + * [Jeremy] May get a room on Tuesday to meet along with the sig release social (link above). Please take a look at the survey for the sig release social +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## ([recording](https://zoom.us/rec/share/tLzMoDN-JUtBG7cevT0YhVUVqvz0KD2_lLmnJvqtug8M5EKKiMu4vWNWXeO2wOVh.x21FMalqE-slLdun)) + +**Host: **Sascha Grunert (he/him) + +**Attendees (pronouns):** + + + +* Jeremy Rickard (he/him) +* Marko Mudrinić (he/him) +* Jim Angel (he/him) +* Mahamed Ali + +**Note Taker (pronouns):** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * (Marko) No major updates. + * (Mahamed) GCR to AR migration is coming along well and is close to the end. Flood of pull requests so thanks for the support. + * (Jeremy) Released a new version of promo-tools if you see issues let us know + * (Sascha) Release artifact KEP is tracked for this release. Docs are required and will assess if we can achieve beta this release cycle. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* [mahamed] GCR to AR Migration. I need someone to cut a new patch release for k-sigs/promo-tools so I can carry on with the migration + * This was done: [https://github.com/kubernetes-sigs/promo-tools/issues/625](https://github.com/kubernetes-sigs/promo-tools/issues/625) + * Will try and keep a better cadence for this as we get any other changes in +* [Joseph] Kubecon SIG-Release events and planning. Is there any support needed? + * (Jeremy) Submitted a contributor summit session for an in person SIG-Release. Having an in-person meeting would be interesting. Topic potentials could include release signing. Identify which things we want to identify in person. + * (Sascha) If you have attended past on-site events do you have suggestions on how to improve. + * (Joseph) After attending Kubecon Valencia a formal agenda will help make it even more meaningful. + * (Jeremy) We can try and record it. Best effort with limited A/V support from onsite staff. + * (Sascha) Evening get together is still up in the air. We require a location. Spin up a doodle poll to vote on potential locations. + * (Jeremy) Pulling together a list of spots. We can then vote on the options. + * (Sascha) What day? + * (Jeremy) Tuesday as a potential option. + * (Sascha) Also maybe Thursday. We can get the list of locations. Present the day option with the poll. + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## Canceled because of light agenda + + +## Canceled because of missing agenda + + +## + +**Host: **Jeremy Rickard + +**Attendees (pronouns):** + + + +* Rey Lejano (he/him) +* Cici Huang (she/her) +* Frederico Muñoz (he/him) +* Shivanshu Raj Shrivastava (he/him) +* Xander Grzywinski (he/him) +* Tanisha Banik (she/hers) +* Nabarun Pal (he/him) +* Nate W. (he/him) +* Rishit Dagli (he/him) +* Arnaud Meukam (he/him)xx +* David Espejo (he/him) +* Verónica López (she/her) + +**Note Taker (pronouns):** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * David Espejo - Works with VMware and supports OpenSource projects. 1.26 Comms shadow. + * Tanisha Banik - Part of DevOps Continuous Delivery team at Cisco + * Ruheena Ansari - Part of the 1.26 shadow Enhancements. +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Go 1.19.1 and Go 1.18.6 were released and we updated the relevant k/k branches. + * This week patch releases for (thanks Arnaud and Jim for handling releases) + * 1.22.14 + * 1.23.11 + * 1.24.5 + * 1.25.1 + * Will do the 1.26 alpha release soon. + * (Rey) Branch renaming - what was decided? 1.26 or 2023? + * (Jeremy) Postponed the branch renaming until 1.26. + * (Cici) How do we follow the updates outside of email? + * (Jeremy) Take advantage of community twitter, Kubecon etc. Be as communicative as possible. Someone will miss the message but we will do our best effort to reach all.. [#2853](https://github.com/kubernetes/enhancements/tree/master/keps/sig-release/2853-k-core-branch-rename) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * (Nabarun) Release cycle started. + * Enhancements collection has started. + * Shadow selection is almost complete. Email will be sent out to applicants not selected in the next hour. + * Tomorrow will be introducing the new shadows at the release meeting. + * Comms team is looking to use GitHub project boards. See [https://kubernetes.slack.com/archives/CNT9Y603D/p1662994419135199](https://kubernetes.slack.com/archives/CNT9Y603D/p1662994419135199) for discussion +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* Planning a SIG Release get-together for KubeCon NA on Oct 25 + * Meeting at the contributor summit on Tuesday in one of the available rooms + * Hanging out in the evening in a chosen restaurant or bar + * (Jeremy) CFP has been submitted for SIG-Release. Feel free to submit other topics like enhancement collection, branch rename and other SIG-Release themes. Social hangout in the evening would be cool. Are there things we would like to do? + * (Joseph) Any support needed? + * (Jeremy) Not yet but will open a slack thread when we do. We’ll try to do it on zoom if possible. Next week the meeting will be at an alternate time. +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + + +## \ + + +**Host: **Jeremy Rickard + +**Attendees (pronouns):** + + + +* Rey Lejano (he/him) +* Mark Rossetti (he/him) +* Leonard Pahlke (he/him) +* Nabarun Pal (he/him) +* Nate W. (He/him) +* Frederico Muñoz (he/him) +* Jason DeTiberus (he/him) +* Rishit Dagli (he/ him) + +**Note Taker (pronouns):** + + + +* Rey Lejano (he/him) + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Mark R (new to this meeting) + * Fred (new to this meeting) +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Nabarun] 9th of Sept is deadline for patch releases, doing patch releases next Wed + * [Veronica] Will take care of cherry-picks, it will be done in next 3 days + * [Adolfo] Also have been doing cherry-pick reviews, as a reminder to Release Managers to review cherry-picks and better to review earlier to get signal. We are mostly going to cut a 1.22 patch release so if you see any back ports to 1.22 + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Leo] In the first week of 1.26. The scheduled is finalized as of last week. Team is selecting shadows. Next week, expect to have shadows selected. Last week, there has been traffic on enhancement project board. We had a meeting last Thursday and sent out notification emails to the community. Some process to automate the project board is a WIP, changes that are relevant for community and SIG leads are set. Ryler (Enhancements lead) is currently OOO and trying to push things forward. + * [Nabarun] Since work is in project board and not sheets, should be less work in the start + +** \ +Open Discussion:** + + + +* [Leo] Release Team Shadow Analysis Report for 1.26; diagrams are ready for review if we like to publish the report (issue: [https://github.com/kubernetes-sigs/release-team-shadow-stats/issues/7#issuecomment-1238387791](https://github.com/kubernetes-sigs/release-team-shadow-stats/issues/7#issuecomment-1238387791)) + * [Leo] This release we used the analysis tool that generated markdown files to ease selection process and diagrams. Issue open to publish to the community + * [Adolfo] Question on structure on that repo, missed the open PR last week and now added the repo to Adolfo’s daily workflow. Need anything in terms of automation + * [Leo] Need to add documentation on what we need for the release. The automation part of things is a little tricky. +* [Puerco] Started the migration process to donate [Tejolote](https://github.com/puerco/tejolote), the SLSA attester to implement [our new provenance flow](https://github.com/kubernetes/release/issues/2611)! [https://github.com/kubernetes/org/issues/3656](https://github.com/kubernetes/org/issues/3656). + * Currently running test builds in my fork + * We need to follow-up with infra changes + * We should set it upin our baby repos (ie bom, zeitgeist) to start more serious testing + * [Adolfo] Note that Adolfo got the SLSA attester to a working state. Started to run tests on building Kubernetes releases, haven’t tested e2e yet because of some required infra changes. Started the donation process. If anyone wants to be an early tester. Need project to build on GCB or GitHub actions. + * [Veronica] Proposing vitess to test + * [Adolfo] Added new features like collecting native artifacts from buckets, etc and hopes to do a demo +* [Jeremy] Current meeting alternation will conflict with sig testing. Propose swapping (so this meeting at the same time next week, earlier meeting in 2 weeks) + * This new time conflicts w/ sig-node too + * [Rey] Also conflicts with SIG Docs also + * [Jeremy] Since this overlaps with SIG Testing, propose to swap alternations. Do next meeting at this time and we’ll do the right cadence. Will send another message to the mailing list and update the calendar +* [Puerco] Let’s this discuss about signing mac binaries, [continuing from this thread](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1662475217130869). Some pointers to previous issues: + * [CNCF MacOS Developer Account for Signing Release Binaries](https://github.com/kubernetes/funding/issues/30) + * [Kcp: bug: Mac binaries don't run out of the box](https://github.com/kcp-dev/kcp/issues/1899) + * [Sign minikube binaries for macOS](https://github.com/kubernetes/minikube/issues/5792) + * [Adolfo] Raised about an hour ago, about signing mac binaries. Brought up by the KCP folks. If you want to run binaries on a Mac, has to be signed by authorized CA by Apple. Has also been raised in other projects. + * [Mark] Also noticed an issue with Windows binaries being signed, has it been discussed before. Might be a good time to sign all the different platforms. + * [Adolfo] Planning to move artifact builds and also run through promotion via file promoter like we do with container images. If you have folks that are interested in the project – it would be good to bring them to meetings and if you have experienced people to help that would be good. + * [Mark] Can find some Windows people to help + * [Arnaud] Requested funding for an Apple developer account to sign. This was brought up to steering in May. There was a discussion on how CNCF will reimburse developers for accounts for signing. Can bring this up to steering in the next meeting which is next Monday + * [Jeremy] Can join the steering meeting with Arnaud to talk about funding for accounts for signing. Is there something different for Windows + * [Mark] Need MSDN account first then ca + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## August 30, 2022 + +Canceled because of having no agenda. + + +## August 23, 2022 (recording) + +**Host: **Jeremy Rickard + +**Attendees (pronouns):** + + + +* Nabarun Pal (he/him) +* Sascha Grunert (he/him) +* Cici Huang (she/her) +* Arnaud Meukam (he/him) +* Stephen Augustus (he/him) +* Marko Mudrinić (he/him) +* Rey Lejano (he/him) +* Adolfo García Veytia (he/him) +* Shivanshu Raj Shrivastava (he/him) +* Mahamed Ali +* Tim Bannister (he/him/they/them) + +**Note Taker (pronouns):** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * +* Subproject updates + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 🎉🎉🎉 **HAPPY RELEASE DAY** 🎉🎉🎉 + * (Cici) No concerns. Looking forward to the release and thanks to everyone. + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * (@puerco) Work to resume on securing the k8s supply chain: + * SLSA3 + * File signing + * System Packages update? + * (Adolfo) We’ll be resuming things we paused for the release. Artifact work with SIG-K8s-Infra will be a focus. + * (Mahamed) We passed a key milestone. We got all images serving from the AR registries. OCI proxy project we will be ready to start telling everyone to start using it. + * (Jason) System packages POC. The team met with OBS last week and explained our build and signing process. Meeting again this week (will send an invite out) and Suse will work with us to understand how to work on their platform. Storage optimization on their platform is one of the topics. They will require storage on the most expensive tier. OBS team has some ideas on how to alleviate this requirement. After this we will be able to kick off the POC. Arnaud was investigating a Google POC option. Notes: [https://docs.google.com/document/d/1BktYecXMky4RFnMm3CPMmyKye_TA8YUah4igs18o3FE/edit](https://docs.google.com/document/d/1BktYecXMky4RFnMm3CPMmyKye_TA8YUah4igs18o3FE/edit) + * (Stephen) We should record meetings since they are community related. If there is no recording of the meeting then make sure to capture notes for the community. +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* Provenance builder demo + * Repo: [https://github.com/puerco/tejolote](https://github.com/puerco/tejolote) + * (Adolfo) _Demo: _Observe the build as it runs. It checks the number of artifact storage locations. Supports directories and OCI. If it runs well it will give provenance attestation. The project supports G Cloud Build and GH Actions. Next up sync with Mahamed to extend what the tool supports. Opening a donation request to the Kubernetes community for comment. If you are interested and want to join let me know. +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## August 16, 2022 (recording) + +**Host: **Jeremy Rickard + +**Attendees (pronouns):** + + + +* Leonard Pahlke (he/him) +* Xander Grzywinski (he/him) +* Rey Lejano (he/him) +* James Laverack (he/him) +* Jason DeTiberus (he/him) +* Arnaud Meukam(he/him) +* Jim Angel (he/him) +* Mahamed Ali + +**Note Taker (pronouns): **James Laverack + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * No new attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Two things going on. First is SLSA attestor for the Kubernetes project. Moving slow but steady. Hope by the next meeting to have the request to donate the repository, maybe to a demo next Tuesday. Related to the SLSA effort in the overall project. + * [Arnaud] For donation, check: [https://github.com/kubernetes/community/blob/9d39e8172019c95a927cf1aef7062c26e4e56336/github-management/kubernetes-repositories.md#rules-for-donated-repositories](https://github.com/kubernetes/community/blob/9d39e8172019c95a927cf1aef7062c26e4e56336/github-management/kubernetes-repositories.md#rules-for-donated-repositories) + * [Adolfo] The second thing is the effort by SIG K8s Infra to mirror the images. Mahamed has been doing a lot of work on this front. Jason has been doing some opinion polling about the packages. + * [Jason] Less opinion polling, trying to get feedback from downstream projects and products that are using the DEBs and RPMs directly. So far we’ve gotten three hits on that. Need to follow up on that. We’re also following up work on the PoC to use OBS to build and publish the packages. We’re going to have a further follow up to walk though their tooling. Continuing the initial stages of that PoC with OBS. Kubeone and Lima are the only downstream projects that have responded as using the official k8s deb/rpms so far. + * [Rey] I will create a namespace for Kubenretes on OBS. + * [Jeremy] Jump into #sig-release if you have further comments. + * [Mahamed] I’ll come back to images later + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Rey] Last full week of the release. 1.25.0-rc.1 is being cut by Verónica. Still on target for release August 21st. + * [Xander] Smooth sailing. + +**Open Discussion:** + + + +* [Jeremy] Meeting time change update + * [Jeremy] Sent out an email today. Extended the lazy consensus, and set a target date. +* [Mahamed] GCR to AR Migration Updates + * [Mahamed] Coming along rather well. Since last week we managed to merge two things. One is a test PR to promote things. Question: If you have a bad promotion does that affect future ones? + * [Adolfo] Depends. There are times when the promotion could fail and get stuck in a weird state. But usually you should just re-run it. + * [Mahamed] It’s complaining about promotions, but they’ve been granted. + * [Adolfo] As it didn’t copy anything, we can just re-trigger it. It could be trickier in the future, like if it failed half way through copying an image. It might make a mess if it doesn’t detect this in future runs. + * [Mahamed] We need to get the permissions straightened out. The other PR is backfilling the repos, merged this morning. Running into some problems with images. Need to take another look. Should be all ready by the end of the month. + * [Adolfo] you can see failed promos here: https://prow.k8s.io/?job=post-k8sio-image-promo + * [Adolfo] If you want to share details as you go along, then let us know. + * [Mahamed] I can’t debug as I don’t have permissions. + * [Adolfo] I’ll check in a little bit to see if I do. +* ~~[Adolfo] New Tool Donation (discussed during the subproject update)~~ + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## August 9, 2022 (recording) + +**Host: **Jeremy Rickard + +**Attendees (pronouns):** + + + +* Jason DeTiberus (he/him) +* Rey Lejano (he/him) +* Nabarun Pal (he/him) +* Cici Huang (she/her) +* + +**Note Taker (pronouns):** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * (mahamed) [GCR to AR Migration](https://github.com/kubernetes/k8s.io/issues/3976): [Infra changes](https://github.com/kubernetes/k8s.io/pull/3968) has been approved by sig-k8s-infra. I need lgtm from sig release leads. We will be provisioning 21 new registries to promote images to. [Registry manifest changes](https://github.com/kubernetes/k8s.io/pull/3956) also need lgtm from sig release leads. + * (Adolfo) [https://github.com/kubernetes/k8s.io/pull/3956](https://github.com/kubernetes/k8s.io/pull/3956) The change is to start promoting to new registries. Is it a change just to manifest? + * (Arnaud) Can the image promoter handle this change in one process? + * (Adolfo) We have take into account the time it takes to promote. We are promoting around 50 images to three target registries which takes around 20 minutes. It involves signing and copying images to different registries. Once this project is in place. We should test the promoters ability to scale. + * (Arnaud) We are in 7 regions at the moment. There is some regions where we would not be deploying OCI-proxy. + * (Mahamed) We might be limited by the amount of registries to promote to. + * (Adolfo) Latency to CDMX to SA is high. + * (Mahamed) Lets go with 21 regions for now. + * (Adolfo) We need to take into account the amount of time it takes to replicate. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * (Cici) Past code freeze last week with 11 exceptions received(6 of them have been approved and all tracked) + * 40 enhancements tracked; 16 feature blogs registered + * Today: Test freeze; rc.0 cut; doc ready to review; start final draft of release notes +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## August 2, 2022 (recording) + +**Host: **Adolfo García Veytia + +**Attendees (pronouns):** + + + +* Jason DeTiberus (he/him) +* Sascha Grunert (he/him) +* Marko Mudrinić (he/him) +* James Laverack (he/him) +* Rey Lejano (he/him) +* Mahamed Ali +* Cici Huang (she/her) +* Arnaud Meukam (he/him) +* Adolfo García Veytia (he/him) + +**Note Taker:** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Marko is back!!! Yes! Welcome back. +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * (Sascha) Updates to Signing KEP. Moving the KEP to 1.26 release. + * [https://github.com/kubernetes/enhancements/pull/3451](https://github.com/kubernetes/enhancements/pull/3451) + * SLSA KEP will also be updated and moved to 1.26 + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * (Cici) Code freeze is today. Four exception requests are being reviewing or approved. Daily burndown meetings have started. Jeremy will be handling the code freeze pr. +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* Copied from the previous meeting: [[Priyanka](https://github.com/Priyankasaggu11929)] GitHub Project Beta automation: + * Ref: [https://github.com/kubernetes/org/issues/3558](https://github.com/kubernetes/org/issues/3558) + * Could we use the Github token stored as secret `k8s-triage-robot-github-token` in the GKE cluster `k8s-infra-prow-build-trusted` for Prow tasks running on the GitHub Project Beta board? Or should we request a new one? + * (Arnaud) Priyanka is looking for a +1. Get feedback from Christoph. Please share feedback on the issue. + * Automation script for Bug Triage board. Need a review - [https://github.com/kubernetes/sig-release/pull/1969](https://github.com/kubernetes/sig-release/pull/1969) + * Action item - Jeremy will review + * Need discussion on Enhancement automation script: [https://github.com/kubernetes/sig-release/pull/1968](https://github.com/kubernetes/sig-release/pull/1968) + * Jeremy will review + * Feedback from the recent Enhancements sub-project meeting discussion: Two ways to move ahead: + * Keeping the KEP opt-in process as it is & manually moving the opted-in KEP to the board + * OR, come up with a new label on k/enhancements for release opting-in having access to limited folks & use that to filter issues & fill in the board. +* Copied from the previous meeting: [Carlos Panato] Kubernetes/Kubernetes branch rename, planned/proposed to 8th August 2022, after the code freeze + * Delayed until after 1.25. + * (Rey) Any comms for this change? + * (Sascha) We’ll review and provide comms with SIG- Contribex out to the community. +* [jeremy] Meeting time Google Form! (not doodle anymore) [https://docs.google.com/forms/d/e/1FAIpQLSerNaCLFK9gBAa4FPXCcHWp7QgCpy6jOc4hoQgJSZDcl8nZrQ/viewform?usp=sf_link](https://docs.google.com/forms/d/e/1FAIpQLSerNaCLFK9gBAa4FPXCcHWp7QgCpy6jOc4hoQgJSZDcl8nZrQ/viewform?usp=sf_link) + * 10-11 PST for west coast. 4-5 CEST for EMEA times. + * (Joseph) When will the time change be implemented? + * (Jeremy) Send notice out for a few weeks before changes. +* [arnaud] Publishing Kubernetes packages on community infrastructure + * KEP PR: [https://github.com/kubernetes/enhancements/pull/3434](https://github.com/kubernetes/enhancements/pull/3434) + * GPG keys: [https://github.com/kubernetes/release/issues/2627](https://github.com/kubernetes/release/issues/2627) + * (Jason) Avoid having to deal with GPG key owned by the community and the challenges that it entails. Reviewing public build services who can manage this. OpenSuse is the most promising option. A meeting is being arranged to field questions from SIG-Release. Looking for Suse employee to potentially see if they will host the part of the project. Scheduling the meeting later this week. It could simplify the process. + * (Rey) I work with Suse and can help facilitate this. + * (Arnaud) What Jason is working on is one part of the KEP. Change the way KREL works. + * (Sascha) We can easily merge this KEP. We’re in favor of building on our own. + * (Jason) GPG signing is needed before uploading packages with other vendors. Unless we want to take on GPG key management and access with members who rotate. We do need to delegate this to a third party. If we had Co-Sign it would be alot easier to invalidate previously signed packages. If we had to invalidate a GPG key we have to resign package and metadata. + * (Marko) This is something that should be CI or in our case G Cloud Build. Release managers would be able to use a key. + * (Arnaud) Jason is proposing using a third party vendor to manage a web of trust. We should explore multiple possibilities before making a decision. + * (Sascha) In the Signing KEP we would move this out of KREL stage. + * (Arnaud) We could start with a POC. Jason continue to investigate the third party vendors. + * (Sascha) Do we have infrastructure to host pre-build? + * (Arnaud) We could host in GCS bucket + * (Jason) Willing to lead a PcC for an OBS based pipeline. +* [JamesL] Release Team Selection redesign ([KEP 3344](https://github.com/kubernetes/enhancements/issues/3344)) — Push to 1.27? + * (James) Its best not to rush this into current release. Does this go through the KEP selection process? + * (Jeremy) I don’t think it needs to go through the KEP process. + * (James) It might confuse things if we did. + + +## July 26, 2022 (https://youtu.be/-LinisivLhk) + +**Host: Sascha Grunert (he/him)** + +**Attendees (pronouns):** + + + +* Rey Lejano (he/him) +* Cici Huang (she/her) +* James Laverack (he/him) +* Priyanka Saggu (she/her) +* Carlos Panato (he/him) +* Parul Sahoo (she/her) +* Matthias Glastra (he/him) +* Adolfo García Veytia (he/him) +* Mahamed Ali +* Joseph Sandoval (he/him/él) +* Nabarun Pal (he/him) + +**Note Taker (pronouns):** + + + +* Rey Lejano + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Note: we have combined the biweekly meeting with the releng meeting. Please see the releng archive [http://bit.ly/k8s-releng-meeting](http://bit.ly/k8s-releng-meeting) for prior releng meetings! + * [Sascha] Doodle out on SIG Release mailing list to add preferred times. Jeremy will reveal results soon and adjust appointments and hopefully it’ll satisfy many timezones +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Have met with Lauri in the last 2 weeks. Kicking off work on 3 compliance efforts. + * [Sascha] Anything on Golang updates + * [Arnaud] Dims is working on that and waiting for GA next week + * [Carlos] Carlos was working on updates for Go 1.19 but went on vacation and saw Dims took over. May run into the same issue as Go 1.18 with memory leaks. Not sure if we want to move to Go 1.19 since we’re close to code freeze. Possibly push to Go 1.19 in 1.26 + * [Arnaud] Dims and Jordan working on moving to Go 1.19. May move to Go 1.19 after code freeze. + * [Carlos] Will sync with Dims + * [Carlos] With Go 1.17, 1.18 there are patches but didn’t see any issues on our side to update, will work on updates + * [James] Do we have a plan in case there’s a bug with Go 1.19 + * [Carlos] Will work with Dims + * [Veronica] Have seen people propose an additional RC that we can do. In Veronica’s opinion, may delay a week or 2. An RC would be a good way to gauge the situation and if things break bad then… + * [Cici] Was going to ask the same discussion, saw someone propose if we can revert Go back to 1.18 so there’s no delay + * [Carlos] In Carlos’ opinion, if we upgrade and it doesn’t work then we can go back but there is lots of steps to do (to go back) + * [Arnaud] There are many factors to consider, if we identify a bug in Go 1.19 RC and fix can be in GA version of Go 1.19. No problem in having Go 1.18 for 1.25 + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 1.25 mid-cycle retro is tomorrow + * Code freeze is next week + +** \ +Open Discussion:** + + + +* [James Laverack] Release Team Selection redesign (KEP 3344) + * Issue: [https://github.com/kubernetes/enhancements/issues/3344](https://github.com/kubernetes/enhancements/issues/3344) + * Proposed KEP: [https://github.com/kubernetes/enhancements/pull/3347](https://github.com/kubernetes/enhancements/pull/3347) + * [James] Intent or desire is to change how shadows are selected to make it easier for contributors to be part of the team. Suboptimal that folks have to apply a few times for the release team. If we did this, this would be a straight to GA announcement. No real way to trial this, the back-out operation is to try it for a release and go back to former process in following week. Not sure if there’s time to do this for 1.26. Communication would be for existing and former release team members as they will be part of a “roster” + * [Sascha] How about sending an email to k-dev and get additional input from the community. In general, in favor for this enhancement. There would be some documentation changes and building the roster (e.g. gathering GH users etc) +* [Adolfo García Veytia] Overview of proposed file signing flow + * [https://github.com/kubernetes/release/issues/2618](https://github.com/kubernetes/release/issues/2618) + * [Adolfo] There have been plans and ideas to build this into krel but Adolfo things this has to be another way. We currently sign images twice, once in staging and once in promotion. Problem with this approach, we expose credential of signing account to any code that generates the build. Krel is not a security scanner, it just runs make and anything in the environment is made available. Propose to move signing out of staging, so first build and stage images and add a step to sign the files and eventually the images, then since we know what we build from staging with sboms then we store signature in files in staging bucket then we push signatures to registries then carry on as usual with image promotion and release. Want to call out, building-out signing images outside of krel. Might as well use co-sign to generate signature for artifacts. We can add a step to use co-sign to sign, this applies to both files and images. Busy at work, building library to making sigstore tools easier. Still can use our code to move signing ability to kpromo. + * [Matthias] Is signing in cloud promotion or is it in another step + * [Adolfo] Adding a step to sign files in google cloud build runs. But only applies to kubernetes release binaries and files in release bucket. If we want to help whole community to sign files then help folks use cloud promoter + * [Sascha] New workflow is way better than what was proposed in a KEP and we need to update the KEP. We may need to use some bash although we wanted to get rid of any bash like anago and we created krel and libraries and we have intermediate layers of testing. Prefer for an implementation release SDK. + * [Adolfo] We need to add logic to loan the bucket and release the sbom. +* [Adolfo García Veytia] Overview of proposed attestation process + * [https://github.com/kubernetes/release/issues/2611](https://github.com/kubernetes/release/issues/2611) + * [Adolfo] What is provenance attestation. Has 2 parts, a subject and predicate. I “did this” (predicate) to “that” (subject). What are we running to make this happen. Same story with attestation with signing the images, currently generating attestation to describe staging run and we do the same when we release. One of the requirements for SLSA level 2 is that we need to sign those, we want to sign these outside to not reveal credentials. The idea would be, before we run staging and after clone repo, we build partial attestation that captures build point, parameters to execute krel and other things we capture before the run and pass it to krel so the idea is to add a 2nd (post-staging) step to retrieve the attestation in a bucket or volume and use the previously partial attestation and subjects in sbom to combine those to form full attestation and sign it outside of krel and we store attestation in the bucket and do the same thing to describe the release step — do partial then run release then combine. Need to get everything from source to finish project so we generate attestation for stage and for release and info from image promoter like what we did to promote the images so we need an attestation for promotion. + * [Sascha] Will this be part of the SLSA KEP? + * [Adolfo] Right now, just proposing change to the flow to see if anyone has any objections. For next week, can get the builder ready and do an execution run in Adolfo’s branch to verify it works. And will build another tool to make this effort usable by other projects. +* [Lauri Apple] roadmapping update: [Publishing Kubernetes packages on community infrastructure](https://github.com/kubernetes/enhancements/issues/1731) issue updated with workplan; [user personas](https://miro.com/app/board/o9J_lLjpyJs=/) WIP; [signing release artifacts](https://miro.com/app/board/o9J_lLjpyJs=/) + user personas WIP + * [Lauri] Work is in progress, go to links above and you can see work done in community infra issue and see milestones and tasks that are identified so far. There is an effort to create user personas. We did a user persona exercise yesterday + * [Arnaud] Request for milestones on the issue, is it possible to merge milestone 1 and milestone 2 because aspects of both are directly related. + * [Lauri] It is to show we have code needs and doc needs. \ + + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## July 12, 2022 () + +**Host: Jeremy Rickard ** + +**Attendees (pronouns):** + + + +* Cici Huang (she/her) +* Leonard Pahlke (he/him) +* Stephen Augustus (he/him) +* Verónica López (she/her) +* Mahamed Ali +* Carlos Santana (he/him) +* Jason DeTiberus (he/him) +* Nabarun Pal (he/him) +* Joseph Sandoval (he/him/él) + +**Note Taker (pronouns):** + + + +* + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Nabarun] Patch releases tomorrow + * AI: Need Google Build Admins to specify their availability ([slack thread](https://kubernetes.slack.com/archives/CJH2GBF7Y/p1657636643019119)) + * v1.22.12: [https://github.com/kubernetes/sig-release/issues/1961](https://github.com/kubernetes/sig-release/issues/1961) + * v1.23.9: [https://github.com/kubernetes/sig-release/issues/1962](https://github.com/kubernetes/sig-release/issues/1962) + * v1.24.3: [https://github.com/kubernetes/sig-release/issues/1963](https://github.com/kubernetes/sig-release/issues/1963) + * 1.21 is OOS now. Nabarun will create an action to remove 1.21 from testgrid. + * [Veronica] reminder a typo in alpha3 release date + + [Veronica] cherry picks approved + + * [Veronica] shout out to puerco for updated promo tools and we can use them for alpha 3 + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Started APAC meeting last week but low attendance might check for new time + * +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) +* [Lauri] [SIG Release roadmap](https://docs.google.com/document/d/1n1Znf-SY85Bjo_B_ReKsqAvEbU4Z8Vw-AvgeiSGVXBA/edit#heading=h.s27b3e38ccgy) progress: roadmapping session this Thursday, input sought on the doc, looking for contributor and user insights + * Trying to take stated priorities and build them out so we know what is needed / scope things / figure out how to know what done looks like + * Roadmapping session thursday 4-5PM CEST [see #sig-release slack for more info](https://kubernetes.slack.com/archives/C2C40FMNF/p1657301068090529) + * Need to send a gcal invite AI: sig release lead help lauri schedule + * Will focus on first item in doc (moving deb/rpm builds to community infra), first session a few weeks ago, we need to go deeper + * Reached out to puerco to work on next item on list + * Few items w/o KEP/specific data and need to clarify and create documentation around goals (especially: cluster api first class signal) + * [stephen] we have a bunch of tests that are generated for k/k and tests are regenerated but various states of decay, primary tests use cluster dir in k/k and that targets GCP. This isn’t actually a sig release item but many things in k/k have no true owners and the idea is if we can get CAPI provider tests running across multiple environments, can we get better signal for releases. Depends on who wants to own this / do this. Which providers willing to provide compute, etc etc. + * [Nabarun] is there an issue for this? + * [Stephen] probably? If there is an issue it should be linked to roadmap + * [jeremy] we should figure if there is an issue and create on if not (lauri and stephen and nabarun looking) + * [lauri] also not one for simplify version markers + * [stephen] that does have issues and i’ll find + * +* [Mahamed] Carryover topic from last week’s releng meeting(copied notes over): + * _[Arnaud] Can we talk about migrating from GCR to Artifact Registry. I spoke to Ben and we might need to use the Artifact registry. _An item from last week’s meeting. + * [Mahamed] Let's look at this [gist](https://gist.github.com/upodroid/a33723a7e1abc5e9c6fabc6b07e7aac0) briefly before we talk about this topic. + * [Arnaud] Maybe not enough time and can carry over to the next meeting. + * **[Mahamed] Make sure to read the gist before the next meeting. ** + * **[arnaud] Need to figure out how we push blobs for images to s3** + * **[mahamed] this is focused on GAR not s3** + * **[arnaud] we should finish the path where we push blob images to s3 still need to finish image process to push to s3** + * **[mahamad] we need to do GAR at the same time? ** + * **[arnaud] we can use GCR until we have the image promoter work done** + * **[stephen] veering into k8s infra territory. S3 more important, but that’s k8s infra** + * **[puerco] last update was offline sync to s3 and OCI proxy would keep track and handle replication async** + * **[stephen] let’s come back to this as we’re heading into a different topic** + * **[mahamed] two parallel pieces as work, i’m specifically interested in getting things into GAR, i need a few things approved to make progress** + * **[puerco] will take a look and we can discuss offline** + * **[stephen] can you link or thread off the image so we can discuss there?** + * **[mahamed] parent issue: [https://github.com/kubernetes/k8s.io/issues/1343](https://github.com/kubernetes/k8s.io/issues/1343) ** +* [Jeremy] Doodle Update: + * Not a lot of votes, do we have enough votes to make a change or extend the poll? + * US Friendly time - 09:00 AM PDT + * EU - 12:00 PM CEST (UTC+2) + * [stephen] Keep open for another week from today and see if we can get more votes if we don’t get more we’ll just combine meetings and keep the tine. + * [nabarun] we need to push the dates out and reopen +* [stephen] we were discussing s3 blob replication, discussing if it needed to be sync or async + * First thing we were discussing is we wanted it to be part of image promo process treating s3 as first class + * Since then, proposal has changed a few times, curious what current state is. + * [puerco] overview from my perspective: + * TL;DR - when ben started working on OCI proxy first idea was image promo should bake in promo to buckets and started working. My objection in mega thread whatever we do shouldn’t increase the complexity of release promo process or the time it takes to not put burdon on RM + * After the thread, one of the things that came up was that possible but not desired to do replication aysnc, handling of replication should fall on sig k8s infra + * First plan: do replication in image promoter (sig release) + * Now plan: async and image promoter won’t do this (something from sig k8s infra) and OCI proxy has capability to check if blobs exist and direct traffic as needed + * [arnaud] that’s basically it. Currently we have production bucket and sync with GCS bucket and OCI proxy uses those production buckets. Caleb working on sync across regions. Next part, checking for each release that we have all the blob replicated. If we can do that async great and we can start to expose through oci proxy. + * [puerco] caleb and jay doing tests to see how long it takes and if it’s dependable + * [https://github.com/jaypipes/k8s-sigs-promo-tools/commit/cb686e96e81f74c66d08c7142d97ca2fe7b2df56](https://github.com/jaypipes/k8s-sigs-promo-tools/commit/cb686e96e81f74c66d08c7142d97ca2fe7b2df56) + * No PR yet but this commit is what Jay is working on + * [stephen] can we ask Jay to open aPR? + * [arnaud] already asked + * [arnaud] now the artifact registry is involved because Ben/Tim talked to GCR and GCR team doesn’t want to add features for OCI proxy and want folks to use GAR instead so move things to GAR now + * [mahamed] one thing to add is there is a cost implication in october and it would be really nice if all the images come from GAR by then (driven through OCI proxy) + * AI: we need to see that PR open (ping Jay again), important change and having it as PR is a better form for discussion + * [arnaud] also in meantime we should help Mahamed work on moving staging repo to GAR and see how it impacts promo process. There are two direct impacts: we lose GCS bucket and we need to make sure we do replication between three new artifact registry + * [puerco] we dont need the buckets for container images? + * [stephen] in current state we need them. GCR backed by GCS. eventually we want to be able to delete original GCS buckets that back that but can’t happen till on GAR. + * [puerco] not important for staging, promo doesnt need access to buckets it just uses registry apis. Some testing stephen did confirms tools work on GAR and we should be find just shifting and promoting from staging GAR to prod GCS for now. We can start testing on a project basis and setup one project and start testing from there. + * [https://github.com/kubernetes-sigs/promo-tools/pull/478](https://github.com/kubernetes-sigs/promo-tools/pull/478) + * [arnaud from chat] please use k8s-staging-experimental + * [mahamed] PR that does some of that [https://github.com/kubernetes-sigs/promo-tools/pull/478](https://github.com/kubernetes-sigs/promo-tools/pull/478) + * [arnaud] i think we first make sure we can migrate staging, then we can do all staging and look at _k8s-artifacts-prod_. It’s safe to go that way so we don’t have to deal with any production incidents. + * [stephen] i’m fine cross publishing images but we should not use any of the releng-staging projects for this. + * Arnaud: use _k8s-staging-experimental _GCP project. + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## June 28, 2022 + + +## () + +**Host: **Jeremy Rickard (he/him) + +**Attendees (pronouns):** + + + +* Priyanka Saggu (she/her) +* Cici Huang(she/her) +* Raj Gaurav Maurya(he/him) +* Joseph Sandoval (he/him/él) +* Cat Chu (she/her) +* Frederico Muñoz (he/him) +* Matthias Glastra +* Mahamed Ali +* Adolfo García Veytia (he/him/él) +* Riaan Kleinhans +* Sascha Grunert (he/him) +* Heba Elayoty (she/her) + +**Note Taker:** + + + +* James Laverack (he/him) + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Welcome to Arya +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Couple of interesting things. Were going to cut the new release of krel, but it had a nasty regression. Planning to do that soon now it’s been fixed upstream. Will cut v0.4.0, including JSON output and a querying language. Also had a chance to meet with some SPDX maintainers in the open source summit. Adolfo becoming a maintainer of the Go libraries for the SPDX project. Looking for help on this if anyone wants to get involved. + * [Veronica] Jeremy is cutting the v1.25.0-alpha.2 release today. + * [RobK] Wanted to sync up with Adolfo offline about the CI Signal tooling. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * [Cici] Enhancements freeze last Thursday. One exception request, which we have approved. It’s already merged. 56 tracked enhancements, out of 76 opted-in. Looking forward to v1.25.0-alpha.2, release notes team is ready. +* (feel free to add any topics you’d like to discuss as part of the agenda) + +** \ +Open Discussion:** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) +* [Joseph] [https://github.com/kubernetes/sig-release/issues/1912](https://github.com/kubernetes/sig-release/issues/1912) + * [Joseph] Bringing this topic back up. Presently we have both SIG Release and Release Engineering meetings. We end up conflating them. Proposal is to align them and also modify times to be more inclusive of non-US timezones. It’s been sat for about a month, so looking to bring it back up. + * [Jeremy] We didn’t get a lot of feedback on GitHub. Need to broadcast it more widely. Started drafting an email on HackMD, and also two Doodle polls to try to select new times. One for APAC/EU and one for the US. Of EU people here, are there any times that would be better? + * [RobK] Remember people south of the equator too. + * [Riaan] Current time works for South Africa, not so well for New Zealand. + * [Joseph] I was thinking about how this would work for the release leads on the release team, in particular for timezone diversity. + * [Sascha] The times that are proposed would work for me. I’d be happy to have more meetings in EU friendly times. + * [Jeremy] I’ll send out this email and Doodles later. The present slot conflicts with other meetings, such as the OpenSSF one. + * [Adolfo] The times where Europe and the US overlap are very competitive for meeting times. + * [Jeremy] Okay, I’ll send my email to SIG Release and [dev@kubernetes.io](mailto:dev@kubernetes.io) for broad reach. We need to figure out how we’ll determine if this is successful. **CALL TO ACTION:** If people have ideas, add them to the GitHub issue please. +* [psaggu] I'm interested in shadowing the Release Branch Manager associate role in the upcoming release cycles, and I'd like to stay for a longer period of time. Is there a formal process for expressing interest, opting in, and moving forward? Thank you so much! + * I’m currently referring to this [document](https://github.com/kubernetes/website/blob/main/content/en/releases/release-managers.md#release-manager-associates) for roles/responsibilities & commitments. + * And have gone through the release-engineering [role-handbooks](https://github.com/kubernetes/sig-release/tree/master/release-engineering/role-handbooks) + have shadowed Arnaud on the recent 1.21.14, 1.22.11, 1.23.8, and 1.24.2 patch release cuts + * [Priyanka] I wanted to understand if there is a process to show interest in becoming a RBM Associate. What do I need to do? + * [Verónica] https://kubernetes.io/releases/_print/#release-manager-associates + * [Adolfo] Right now we’re thinking about better codeifying this process. Until now the process is discressional, in the sense that some of the leads would hear from people who have interest and try to offer those people and opportunity to become part of the release. The problem is that the process hasn’t been established, and and hasn’t been written anywhere. This has lead to some problems, for example the RMAs on the site are not active. This is a trust problem, we’d expected people to offboard themselves if they couldn’t continue to do it. Right now we’re going though some discussions to try to define the process. Looking to provide a soft onramp for contributors (not necessarily) a release team member. The problem is that becoming a RM means sensitive access, so vetting for trust with no gatekeeping and no kingmaking (i.e., doing selection in a non-transparent way). When the time came to pick a release manager for 1.25, the existing RMs are overworked (this is Verónica’s third, I’ve done two, Nabarun has done two). This doesn’t match with the expectation that there are enough of them to do this without overworking. I hope that we can start working on a draft for these processes in the coming weeks, we’ll share them as soon as we start. + * [Priyanka] Thank you for that answer. + * [Joseph] We’ve talked about this in the past. How do we do this in a way that’s not a cognitive burden? Verónica had to step up for various releases. Will there be thought to how to onboard people? + * [Adolfo] Most likely training will be improving the branch manager shadow program. Was discussing with Stephen. For example do we permit non-release team contributors? Do they need extra training? We need to recognise that there is a lot to learn before we can let someone run this. We need to establish a process where the SIG Release community can approve of someone. This needs to be well thought out and put into some process document. + * [Verónica] Aside from formal process, I’ve noted for the past 6 months that being a branch manager for a specific release (and being an RM/RMA) that sometimes people in different timezones need to collaborate. This is normal in tech. It’s a volunteer task. Sometimes you’re waiting for a person in another timezone. You need to communicate that you can or cannot follow up at specific times. Less of a technical requirement for a RM/RMA, but it’s a difficulty of the role. Requires availability in weird times, which can be a complication of the role. Can you even communicate that in a manual? What would you say? It’s not that you have to be available, but you have to coordinate. + * [Adolfo] We need more help to avoid overworking people. We need options to hand over instead of always the same person doing a thing. + * [James] Also trying to do something similar for the Release Team. They don’t need to be the same process, but should be aware of each other. +* [matthias] Moving File signing forward. [https://github.com/kubernetes-sigs/release-sdk/pull/68](https://github.com/kubernetes-sigs/release-sdk/pull/68) for issue [https://github.com/kubernetes-sigs/release-sdk/issues/30](https://github.com/kubernetes-sigs/release-sdk/issues/30) + * [Matthias] I noticed that file signing is on the roadmap. There’s a PR on the Release SDK. Can we get some eyes on this for use in krel? Input would be appreciated. Sacha already reviewed it which I’m happy with. + * [Sascha] More eyes would be good. + * [Adolfo] I’ll take a look. One thing to note about file signing, and why I’ve left it until last in the signing effort, is that there is still a disconnect between how container images are signed keyless (and the verification) and the way that files are signed. There is an effort going on with a project inside sigstore. It’s the equivalent of curl but it’ll verify that the thing that is pulled has been signed. I think they’ve made progress recently, so maybe we can use that. That aside, the signing API is needed. + * [Matthias] My current implementation is based on a known public and private key. Is container signing already keyless? + * [Adolfo] Yes. + * [Matthias] I understand the disconnect. Is there a key we can use? + * [Adolfo] No. Same problem with RPMs and DEBs. Keyless would be preferred. Lets sync up. +* [arnaud] GPG: The great debate + * [Arnaud] Can I talk about this in RelEng next week July 5th? Signing for system packages. + * [Jeremy] Sure. + * [Sascha] +1 + * [Adolfo] Just put the subject on the agenda and try to elaborate as much as you can. + * [Jeremy] Especially any pre-reading. +* [Arnaud] Can we talk about migrating from GCR to our own registry. I spoke to Ben and we might need to use Artifact registry. Heads up. + * [Jeremy] We need to follow up + * [Mahamed] Who owns the new registry? + * [Arnaud] K8s Infra. Migration still in early phases. We need to have a full discussion in release engineering. + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## June 14, 2022 () + +**Host: **Jeremy Rickard (he/him) + +**Attendees (pronouns):** + + + +* Sascha Grunert (he/him) +* Frederico Muñoz (he/him) +* Leonard Pahlke (he/him) +* Lauri Apple (she/her) +* Cici Huang (she/her) +* Adolfo García Veytia (he/him/él) +* Rishit Dagli (he/ him) +* Stephen Augustus (he/him) +* Shubham Sah (he/him) +* Rob Kielty (he/him) + +**Note Taker:** + + + +* Sascha Grunert + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Welcome to Shubham! +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Work to parallelize the image promoter is done, waiting for [this commit](https://github.com/sigstore/sigstore/pull/503) to trickle down to cosign → k/release-sdk → k/promo-tools + * Should we wait for this PR before cutting the patches? + * No need to wait for it, but prefer to start the releases until this has been resolved + * Proposal: Promote each tag individual to not risk a timeout + * If we fix something in sigstore, do we have to bump our golang deps? + * Yes, we import sigstore/cosign + * A bunch of libraries are moved from cosign to sigstore/sigstore + * [https://github.com/sigstore/sigstore/pull/503](https://github.com/sigstore/sigstore/pull/503) + * [https://github.com/sigstore/cosign/pull/1866](https://github.com/sigstore/cosign/pull/1866) + * + * Issue with the last patch releases, where for some reason the image promotion timed-out after multiple hours. PR open to run signing in parallel. + * Please reserve time from the Google build admins for the upcoming patch releases + * This should be in RM docs if it is not. We should confirm. [it should be] + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Discussion last week about extending the enhancement freeze + * Got extended by 1 week + * PRR deadline moved to this Thursday as well + * Release team completely onboarded + * Still expecting the alpha 1 to be cut + +** \ +Open Discussion:** + + + +* [Lauri] SIG Release Roadmap session results ([notes](https://docs.google.com/spreadsheets/d/1Vgpb7HPg8WgoFtDuddaRD6j7jQLoqNK6vUu6Lb0ea28/edit#gid=1353314046)) + * Need some help with the other roadmap items, shaping the plan + * To confirm: if we should send out two-question survey asking community to rank currently published SIG Release roadmap items, and why their #1 is their #1 (answer: yes) + * Getting input from SIG Release first, then consider input from the whole community + * Consider closing the checklist mentioned in [https://github.com/kubernetes/release/issues/913#issue-513253044](https://github.com/kubernetes/release/issues/913#issue-513253044) + * Stephen started rewriting package building in golang, we need to revisit the story, take ownership and decide on the chosen architecture + * Additional tasks should be added to the spreadsheet to collect them in one place + * Arnaud: What are we using for signatures for the system packages? + * Right now we use Google owned GPG keys, but we have no community access to this process + * Main goal: Ship the debs/rpms in community infra and then we can consider using our own GPG keys + * Cost is not really an issue when providing community packages + * [Arnaud] + * Not as big an issue now we will shield usage + * For cdn usage: [https://github.com/kubernetes/k8s.io/issues/3556](https://github.com/kubernetes/k8s.io/issues/3556) + * issue open on steering side : [https://github.com/kubernetes/steering/issues/239](https://github.com/kubernetes/steering/issues/239) + * Lauri: Reworking the artifact management EPIC? + * Keep it as umbrella, consider creating a dedicated EPIC for debs/rpms + * Stephen and Lauri will work on that in a dedicated session +* [Lauri] Rel Eng items review ([doc](https://docs.google.com/spreadsheets/d/1Vgpb7HPg8WgoFtDuddaRD6j7jQLoqNK6vUu6Lb0ea28/edit#gid=732483138)) — need help breaking items down and moving things forward + * Lauri: How to make progress on the doc? + * Revisit the older issues in the context of the new enhancements + * Start working on the newest items first + * Items which are unassigned should find an owner + * Consider removing frozen items from the project board? + * Take some time to re-do the project boards +* [jeremy] Let’s talk about SBOMs + * Will move to next time +* [Arnaud] Deb/RPM packages on k8s-infra: [https://github.com/kubernetes/release/issues/913](https://github.com/kubernetes/release/issues/913) + * We had an agreement about this before but I’m sorry, I can’t remember what we decided about this. Breaking down in 2 practical questions: + * Who owns the new GPG keys ? + * How and where to store the GPG keys ? + * [Stephen] we need to step back and determine if we are doing GPG or if we are owning the packages, should have a follow up + * AI: Arnaud to schedule a meeting to specifically discuss + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## May 31, 2022 (https://www.youtube.com/watch?v=fU7OsugvqPY) + +**Host: **Stephen Augustus (he/him) + +**Attendees (pronouns):** + + + +* Rob Kielty (he/him) +* Sascha Grunert (he/him) +* Leonard Pahlke (he/him) +* Rey Lejano (he/him) +* Mike Lieberman (he/him) +* Hossein Salahi (he/him) +* Mahamed Ali (he/him) +* Eddie Zaneski (he/him) +* Atharva Shinde (he/him) +* Devan Goodwin (he/him) +* Verónica López (she/her) +* Rishit Dagli (he/ him) + +**Note Taker:** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Mike Liebermen: Part of CNCF TAG Security. + * Rishit Dagli: Shadow applicant for 1.25. Looking to contribute to Kubernetes + * Mahamed Ali: Part of SIG-K8s-Infra. Already picked up some work and being guided by Sascha. + * Atharva Shinde: !.25 Enhancement Shadow +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + +** \ +Open Discussion:** + + + +* [Stephen] Timezone-skewed meetings: [https://github.com/kubernetes/sig-release/issues/1912](https://github.com/kubernetes/sig-release/issues/1912) + * Issue is open to comment to the community for the next week. Updates will be sent after the comment period closes. Expect updated calendar invites after decision is made. +* [Rob] Update and check in to automate the running of Leo’s report that lists out the active flakes that are being tracked by the CI Signal Team over the course of a release cycle. + * We have automation choices we can make. I would welcome review and input on my last update on the issue where we are tracking this work. [ci-signal-reporter, automatically generate weekly ci signal report #2443](https://github.com/kubernetes/release/issues/2443#issuecomment-1138376284) + * Feedback being solicited from SIG-Release regarding Prowjob vs Github Action in the above issue. Also looking for additional feedback on the CI Signal automated reports. + * [Stephen] Completely fine with GH Actions. What permissions does this token require? + * [Rob] Read only. (For this iteration of just running the report) + * [Leo] Nothing special is needed with permissions. + * [Stephen] Default action for the automated report should be writing to a repo vs Slack. + * Issue, store report in a repo: [https://github.com/kubernetes/release/issues/2477](https://github.com/kubernetes/release/issues/2477) + * [Rob] The analytical steps after we have this report rolled out will need some design thinking to determine if implementation effort on this is redundant. (Added context post meeting by RK: lots of data is captured automatically; kettle from sig-testing collects data on the CI job level and stores that data in a BigTable db. Leo’s report captures CI Signal Team Activity. The aspiration is that data on the CI Signal Team activity will be useful for both the Reliability WG and the project at large. When we get to the stage where we can track trends we can discuss next steps within sig-release and share that out with the wider community) + * [Stephen] The GH action should be identical to how you run locally. Tool should be packaged as a container image. + * [Muhamed] “you can get GitHub Actions stuff to interact with Google Cloud to persist stuff there” +* [Leo] Followup from 1.24 retros: add Inclusive Speaker Orientation ([LFC101](https://training.linuxfoundation.org/training/inclusive-speaker-orientation/)) course to every RT sub-team lead preparation + * [Stephen] Agrees with this. It lays the foundation for future leads. + * [Leo] Action item to update handbook. +* [Mike Lieberman] Representing CNCF Security TAG - Supply chain security for kubernetes + * CNCF Proposal - [https://github.com/cncf/tag-security/issues/890](https://github.com/cncf/tag-security/issues/890) + * [Stephen] We need to decide who is doing what with the investigation? SIG-Release has a roadmap of Software Supply Chain Security roadmap. Moving to distro-less images was the first step. There are several images that still use Debian base. The current KEPs in motion with SSCP and map them to the CNCF Tag Security proposal would be ideal. SIG-Release are early adopters. From external perspective where do people see the gaps. + * [Mike] All these efforts provide a better picture of Supply Chain Security picture. If there is implementation challenges the feedback from SIG-Release would be helpful. + * [Stephen] When approaching SLSA we realized that level 4 was off the table. + * [Mike] SLSA 3 is going to change a bit as we move to 1.0. How should I collaborate? + * [Stephen] Come to the meetings and Slack. Release-management channel is where to find the team. +* [Verónica] Branch Manager for 1.25. Cut 1st alpha. Improvements on release process times (promotion timeouts). + * [Stephen] Working on branch manager recruiting. There is a schedule up already for 1.25 release. + * [Verónica] Leo wants to shadow. + +**Walk the Board:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## May 17, 2022: Canceled for KubeCon + + +## May 3, 2022: Canceled for v1.24 release day + + +## April 19, 2022 () + +**Host: **Sascha Grunert + +**Attendees (pronouns):** + + + +* Sascha Grunert (he/him) +* Rey Lejano (he/him) +* Marko Mudrinić (he/him) +* Leonard Pahlke (he/him) +* Nabarun Pal (he/him) +* Adolfo García Veytia (he/him/él) + +**Note Taker:** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * New Image Promoter are signed already with the last release cuts. Some work needed to improve speed. There is a need to rethink the copying of the layers of container images. This will affect the order of the steps. A new release has images staged the the promoter comes in and signs then propagates out. New approach would be to mirror before promoting them so signature needs to be in place. (Stephen) Define what done is for signing. After that we can consider the mirroring. What happens with signatures left behind or redone? Promoter is consistently running. (Adolfo) Promotion process takes all the blobs from the layers and into S3. We would have to add the signature to the blobs. The proxy that sends the traffic to AWS has no idea whats in the mirror just route traffic to an IP address. (Arnaud) The proxy is not a cache. Any request coming from docker client will be identified by location. Anyone pulling from AWS can get the images before they hit GCS. (Stephen) AWS will be a first class citizen for signed images. Whats the expectations for 1.25. (Arnaud) registry.k8s.io will be embedded in the k/k repo. (Stephen) K/promo images points to registry.k8s.io. Whats the backout plan if things don’t work as expected? (Adolfo) The plan to modify this is being guided by SIG-K8s-Infra. (Stephen) Functional process might not change but the UX might change. How do we revert without changing the references? (Arnaud) Revert is redeploy OCI process which can take 3-5 minutes. (Sascha) Do we want to recreate something like the mirroring project in the KEP? (Stephen) We should close out the KEP and the goals are separate. Do we have a merge KEP for this redirect? (Arnaud) No KEP but a [proposal](https://docs.google.com/document/d/1yNQ7DaDE5LbDJf9ku82YtlKZK0tcg5Wpk9L72-x2S2k/edit#heading=h.x9snb54sjlu9) (Stephen) This needs to be a KEP for the community. It touches everyone. (Stephen) It’s infra and it needs to be signed by a few different SIGS that are affected. This proposal is KEP ready. SIG-K8s-Infra will be the owner of creating the KEP. + * Patch releases (Nabarun) Cutting release candidates today for 1.24.0. Expect some hiccups. Run the rotator once and if an issue reach out to Stephen. Verónica is available to push the release along with Jim Angel. Google build admin support will be needed. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * (James) Release delayed by two weeks. Plan is to cut RC.1 next week. Go 1.18.1 fix is in progress. KREL has some issues they are working through. 1.25 team is in place. (Arnaud) Some of the KREL jobs are successful but with move to community infra there is issues with SSH. (James) [1.25 release outline](https://github.com/kubernetes/sig-release/pull/1885) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (Joseph) [Kubecon Valencia Contributor Summit](https://github.com/kubernetes/community/issues/6554) - Is SIG-Release planning to participate? (Stephen) I’ll be in the house. Big room and bunch of tables in the past. (Arnaud) No structure to the initial program. The rest is hallway track. (Arnaud) There is meeting later today to discuss this. + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## April 5, 2022 (recording) + +**Host: **Jeremy Rickard + +**Attendees (pronouns):** + + + +* Joseph Sandoval (he/him) +* Leonard Pahlke (he/him) +* Nate Waddington (he/him) +* Heba Elayoty (she/her) +* Lucas Dwyer (he/him) +* Chris Negus (he/him) +* Arnaud Meukam(he/him) +* Eddie Zaneski (he/him) +* Rey Lejano (he/him) +* Verónica López (she/her) + +**Note Taker: **Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Confirmation new image signing code is running as expected. RC candidate depending on timing is that the identity might not be the service account expected. Need a new release of co-sign. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * Code freeze passed. Test freeze today. + * Big news is 1.24.0-rc.0 delayed until Monday because of a Go bug. If that fix doesn’t land in Go 1.18.1 (due Thursday) then we’ll likely have to delay 1.24. We can’t release without the fix and we can’t downgrade to Go 1.17. The fix is already in Go’s main branch and a cherry-pick is open to Go’s 1.18 release branch. + * (James) Code freeze has passed. RC.0 cut is delayed. Blocking issue with Golang 1.18. A few exception requests are still pending. 6 PR’s are pending and once those are merged we can focus on CI.(Adolfo) Go 1.18 has been rejecting CSR with sha1. [https://go-review.googlesource.com/c/go/+/394294](https://go-review.googlesource.com/c/go/+/394294) is the fix. Next step is delay or no delay. + * + +**Open Discussion [timebox to 5 min]** + + + +* [jeremy] Followup from 1.24 Mid Cycle Retro + * Add clarity on when is the best time for an enhancement to be a major theme and guidelines around what should be a major theme +* [leo] CI Signal project Board or view for managing team reportings (see [slack discussion](https://kubernetes.slack.com/archives/CN0K3TE2C/p1648500977041789)). . Can we remove this magic file from which is handed down from lead to lead? Propose moving this to Github. +* [leo] CI Signal team kind of misses power to have the things they found being addresses (for example this [flake 107530](https://github.com/kubernetes/kubernetes/issues/107530), from [KEP 3139](https://github.com/kubernetes/enhancements/pull/3139)) (Leo) The team continually pings regarding status. Is there something we can do more than just ask for to get flakes fixed. (Eddie) Have you dm the leads of the SIG’s? Leads have alot and lose the issues easily. DM direct is the recommendation. (James) The release team feels like they have no power. (Adolfo) We do have the power to delay a release if a test is flaking. (Leo) Perhaps using a priority label or escalation path to raise awareness. (Veronica) This is not a problem that can be solved with more processes. I’ve been on the other side of this issue. I don’t get offended when issues are escalated. (Adolfo) **AI - Raise this at the community call. **(Eddie) Update the leads contact sheet as this was useful. +* [leo] If reliability monitoring is added to the CI signal responsibilities (see [KEP 3139](https://github.com/kubernetes/enhancements/pull/3139) discussion), something more is needed than just changing the description in the handbook. I think the team will not deliver “right out of the box” without making some adjustments to the team**. **Maybe two leads / emeritus or involvement of sig-release leads or similar measures to build a stronger team to handle the additional responsibility. + * (Adolfo) What sort of effort is involved with monitoring? **AI - Keep it under discussion.** +* (Arnaud/Adolfo) K8s images are pushed to GCE which copies to AWS. [https://github.com/kubernetes-sigs/promo-tools/issues/533](https://github.com/kubernetes-sigs/promo-tools/issues/533) Issue opened by Jay Pipes. (Arnaud) I can act as the liaison between K8s-engineering and AWS. + + + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + +[Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease)** ** + + +## Mar 22, 2022 (recording) + +**Host: **Adolfo García Veytia + +**Attendees (pronouns):** + + + +* Rey Lejano (he/him) +* James Laverack (he/him) +* Taylor Dolezal (he/him) +* Leonard Pahlke (he/him) +* Eddie Zaneski (he/him) +* Batuhan Apaydın (he/him) +* Wojtek Tyczynski (he/him) +* Veronica Lopez (she/her) + +**Note Taker:** + + + +* **Rey Lejano** + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * [Adolfo] Two main lines of work last week: work on bumping images to Go 1.18 and PR to sign images in image promoter is open but did not have enough contributors to get new version of image promoter promoted. The last patch releases could not be signed, hopefully we have tests for the first 1.24 beta to verify signatures are working + * [James] Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 28 Days until release. Exactly **four weeks**. + * Code Freeze coming in one week. 01:00 UTC Wednesday 30th March 2022 / 02:00 BST Wednesday 30th March 2022 / 18:00 PDT Tuesday 29th March 2022 + * We’re going to ship the “code freeze is coming” email today + * Mid-cycle retro **tomorrow** 23rd March at 17:00 UTC. [https://bit.ly/k8s124-retro](https://bit.ly/k8s124-retro) + * Nabarun cut 1.24.0-alpha.4 today. + * We’re starting to think about succession for 1.25. + * [James] In week 11, Code Freeze is in 1 week. Email to k-dev, Code Freeze is coming email. Mid-cycle retro is tomorrow. A release-blocker was fixed just in time - shoutout to Jordan, Eddie, Jim on fixing the release-blocker issue +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Open Discussion [timebox to 5 min]** + + + +* [leonardpahlke] Add a CI Signal onboarding handbook to the [handbook folder](https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/ci-signal) (see issue [#1857](https://github.com/kubernetes/sig-release/issues/1857)) + * It's hard to get a clear picture of what CI Signal is doing right now without reading more than 20 pages of. + * More videos like the flake finding videos + * [Eddie] Leads and certain people have lots of tacit knowledge that should be transferred + * [Adolfo] The Flake Finders video was meant to be a recurring show. Adolfo’s old videos (even though they are no longer current) are still be being watched by new Release Team members. Shows how people like to consume information. + * [Taylor] Folks have pitched similar options like a slide deck in release retros + * [Rey] There are slow weeks for each RT sub team, it might be a good time for new RT members to point out the gaps from the handbook to onboard and experience RT members can create an onboarding slide deck +* [leonardpahlke / wojtek-t / david / devan] [KEP-3138](https://github.com/kubernetes/enhancements/pull/3139/) Increase the Reliability Bar proposal + * Discussion pointed to sig-release to track reliability, report issues, and to poke the responsible sig's. This should be done by the CI signal team - thoughts? + * CI Signal Team likely needs support / a different team structure to handle the additional responsibility + * CI Signal tasks needs to get automated -> tools currently under development + * [Sippy](https://kube-sippy.dptools.openshift.org/sippy-ng/) to track reliability (hosted on openshift.org) +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + + +## Feb 22, 2022 (canceled) + + +## Feb 8, 2022 (recording) + +**Host: Sascha Grunert** + +**Attendees (pronouns):** + + + +* Sascha Grunert (he/him) +* James Laverack (he/him) +* Joseph Sandoval (he/him/él) +* Marko Mudrinić (he/him) +* Nate Waddington (he/him) +* Rey Lejano (he/him) +* Leonard Pahlke (he/him) +* Lucas Dwyer (he/him) +* Debabrata (he/him) +* Dipto Chakrabarty (he/him) +* Eddie Zaneski (he/him) +* Verónica López (she/her) +* Dishant Sethi (he/him) +* Adolfo García Veytia (he/him/él) + +**Note Taker:** + + + +* Joseph Sandoval + +**Recurring Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * Jake Plimack/Tesla. +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30) + * KEP-3031 - (Sascha) how to build releases and how to use keyless signing. Needs to be documented on how to verify signatures. It’s partially done. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * (James) Week five! 70 days left. **One third** of the way through. + * Enhancements freeze completed, after two exceptions we are tracking **66** enhancements for v1.24. + * Done shadow onboarding. + * Gonna finalise time later this week for the mid-cycle retro on 23rd March. + +** \ +Open Discussion [timebox to 15 min]** + + + +* [leonardpahlke] Release Team Shadow applicant analysis tool (see [slack thread in contribex](https://kubernetes.slack.com/archives/C1TU9EB9S/p1641941787049000), [applicant analysis repository](https://github.com/leonardpahlke/application-summary)) + * I got overall positive feedback for the initiative. From the discussion I get that we aim for 1.25 to share some infos about the applicants. How to proceed from here? (previous [release-team meeting discussion thread](https://kubernetes.slack.com/archives/C2C40FMNF/p1641917880062200)). Next steps on how to move forward? + * Cleanup and donate the [repo](https://github.com/leonardpahlke/application-summary) or + * use it just as a reference and build something new. + * (Sascha) do we have a process in mind on how to adopt? (Leonard) Create form for [K/K branch rename](https://github.com/orgs/kubernetes/projects/23?card_filter_query=2853) each release cycle with the data analyzed. (Sascha) We could move forward with using the tool. + * (Adolfo) [Image Promoter](https://github.com/kubernetes-sigs/promo-tools/pull/494): Refactoring the code to provide a reworking of the API service. The reason for this is the community is running low on funds so mirroring the images to major cloud providers. TBD is how to test locally. It should be functionally and if anyone has some cycles to review and can provide some comments on how to improve. (Sascha) This can be complete before the end of release cycle. GCP permission discussion could be potential blocker. How much effort will be needed? (Adolfo) WE should only need to provide the copying logic. (Eddie) AWS registry is stood up as POC. Stress testing is needed. (_Docker pull ​​registry-sandbox.k8s.io/pause:3.1_) + * (Dishant) CLA signing as individual is not enabled but corporate works. Advised to reach out to SIG-Contribex. The migration to EasyCLA was recently introduced. + +**Walk the Board [timebox to 10 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * (Carlos) [K/K branch rename](https://github.com/orgs/kubernetes/projects/23?card_filter_query=2853) + * [Rewrite Milestone Maintainer](https://github.com/kubernetes/sig-release/issues/1257) Max is assigned. WIP. + * [Yubikey](https://github.com/kubernetes/sig-release/issues/1790) still in progress + * + + +## January 25, 2022 (recording) + +**Host: Stephen Augustus** + +**Attendees (pronouns):** + + + +* Stephen Augustus (he/him) +* James Laverack (he/him) +* Rey Lejano (he/him) +* Leonard Pahlke (he/him) +* Joseph Sandoval (he/him/é)l +* Nate Waddington (he/him) +* Sascha Grunert (he/him) +* Eddie Zaneski (he/him) + +**Note Taker:** + + + +* Joseph Sandoval + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees + * +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) +* Emergency Release 1.23.3 to be cut today + * A [regression was found](https://github.com/kubernetes/kubernetes/issues/107690) where CRDs may drop fields + * [Patched the release schedule](https://github.com/kubernetes/website/pull/31494) and processed all outstanding CPs in the 1.23 queue. + * Sent a [notice to dev@k8s](https://groups.google.com/u/1/a/kubernetes.io/g/dev/c/Xl1sm-CItaY) with the details provided to us by Kevin Delgado + * Jim Angel will perform the out-of-band release cut today. + * The cherry pick queue was drained last night. +* Discuss out-of-band releases: [https://github.com/kubernetes/sig-release/issues/1832](https://github.com/kubernetes/sig-release/issues/1832) +* SPDX DocFest coming up this Thursday (hit @puerco if you want to attend) K8s SBOM tool is starting to be used by other teams. The event will have teams compare the outputs. Check if this is going in the right direction. (Stephen) We should add these to the calendar in the future. +* (James) 1.24 Release is going well with about 70 enhancements. How do we feel about the open communication in #SIG-Release channel. (Stephen) It’s great having all the communication in the open. + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) +* (Leo) [https://github.com/kubernetes/sig-release/pull/1825](https://github.com/kubernetes/sig-release/pull/1825) Update to the handbook waiting for comment. [https://github.com/kubernetes/k8s.io/issues/3071](https://github.com/kubernetes/k8s.io/issues/3071) (Stephen) This would be managed by release managers. File a [donation request](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-repo&template=repo-create.yml&title=REQUEST%3A+%3CCreate+or+Migrate%3E+%3Cgithub+repo%3E). +* (Eddie) CDN egress spend to other clouds. (Stephen) Is there anything we can do in the interim to ease the spend? (Eddie) Correction container registry proxy is what Ben the Elder is working on. +* (Sascha) Implementation phase of [container signing](https://github.com/kubernetes/release/issues/2383) has help wanted issues. (Stephen) There could be a security concern. There are two tokens in the GCE run. Walkthrough of [GCP jobs](https://github.com/kubernetes/sig-release/blob/master/release-engineering/gcp.md) and ownership. + + +## January 11, 2022 (recording) + +**Host: **Jeremy Rickard (he/him) + +** ** + +**Attendees (pronouns):** + + + +* Leonard Pahlke (he/him) +* James Laverack +* Omar Elbanby +* Jesse Butler (he/him) +* Joseph Sandoval (he/him) +* Nivedita Prasad (she/her) +* Purneswar Prasad (he/him) +* Eddie Zaneski (he/him) +* Nate Waddington (he/him) +* Debabrata (he/him) +* Meha Bhalodiya (she/her) +* Adolfo García Veytia (he/him) + +**Note Taker:** + + + +* Joseph Sandoval + +**Topics [timebox to 20 min]:** + + + +* Welcome any new members or attendees: Omar - works in DevOps. Nivedita - CI Signal Shadow Nate W - 1.24 Docs lead - Ahsan Khan looking to contribute - Purneswar - Student - Kep Reading Club - Akinpelu D - DevOps engineer. +* Subproject updates + * Release Engineering ([https://github.com/orgs/kubernetes/projects/30](https://github.com/orgs/kubernetes/projects/30)) + * (Adolfo) Merging KEPs for signing artifacts for release and SLSA 3 compliance. Projects will be updated at the next release meeting. (Jeremy) Next week is the sub-project meeting. (Purneswar) What does the release engineering team do and what is an artifact? (Adolfo) Provides overview of Release team. + * Release Team ([https://github.com/orgs/kubernetes/projects/29](https://github.com/orgs/kubernetes/projects/29)) + * 1.24 release cycle begins! Full release team and onboarding tasks in progress. Meeting tomorrow 10am PST. Enhancements are now open. First deadline is the Enhancements freeze in the first week of Feb. Very limited spots available. +* (feel free to add any topics you’d like to discuss as part of the agenda) + +**Walk the Board [timebox to 20 min]:** + + + +* Project board review: [https://github.com/orgs/kubernetes/projects/23](https://github.com/orgs/kubernetes/projects/23) + * Kubernetes keys will be distributed to the release team. +* Incoming issue and PR triage + * [SIG Release issues needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Aissue+label%3Asig%2Frelease+label%3Aneeds-triage) + * [SIG Release PRs needs-triage](https://github.com/search?q=org%3Akubernetes+is%3Aopen+is%3Apr+label%3Asig%2Frelease+label%3Aneeds-triage) + * [Kubernetes issues](https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+is%3Aissue+label%3Asig%2Frelease) + +** \ +Open Discussion [timebox to 5 min]** + + + +* (feel free to add any topics you’d like to discuss, even when they came up during the meeting) +* [arnaud] Migrate away from gs://kubernetes-release: [https://github.com/kubernetes/k8s.io/issues/2396](https://github.com/kubernetes/k8s.io/issues/2396) + * Problem is network traffic and it's huge. Costs are a problem. New bucket with the CDN provider is the request. The bucket contains all the binaries from all the releases. Not sure if we have the budget for this request? Ask Steering to request a plan with CDN. (Jeremy) We need a +1 from SIG-Release. When do you plan to open this? (Arnaud) This week. Discuss this in release engineering channel. + * Request an open source plan for a CDN Provider. +* [leonardpahlke] CI Signal Monitoring platform - ownership. See [github issue](https://github.com/kubernetes/k8s.io/issues/3071) and [diagram / overview](https://github.com/kubernetes/sig-release/tree/master/release-team/role-handbooks/ci-signal#ci-signal-reporting-tool) + * (Leonard) Who is the owner of the platform and project? How do we decide this? (Jeremy) CI Signal has a rotation so no ongoing owner. This would involve running a platform? Do we want to request infra? (Arnaud) We can provide infra. Who would we reach out to if this goes down? (Adolfo) Maybe the CI-Signal team to provide support. Take discussion to SIG-release channel. +* [leonardpahlke] New tool, summarize and analyze release team shadow applicants ([repo](https://github.com/leonardpahlke/application-summary)) + * 1. Analyze all applicants by creating charts + * Charts: applicants by team, pronouns, company / affiliation, timezone… + * Charts have been generated but are currently not shared as it might be confidential (needs to be clarified first) Could we use these reports and charts going forward? Maybe advance this beyond these initial reports to help drive the selection process. (James) Concern about the data we collect from survey but do we have permission. Not sure if we have asked permission. Might need SIG-Contribex input or possibly CNCF. (James) Can we provide an anonymized version of the data and make sure we have the permission. We need to move this up to Steering or SIG-Contribex. (Omar) Applied for release shadow. It was challenging to answer some of the survey questions. (Eddie) Paris was asking for ideas. Put some thought into KEP tracking but maybe we can have this be funded. + * 2. Summarize applicants by generating markdown files for sig-release sub-teams + * Goals: + * support the release team in the shadow selection (2.) + * current state of the release team shadow program (1.)