## 19 December 2017



* [Volunteer for 1.10 release team!](https://github.com/kubernetes/sig-release/pull/49)


## 7 November 2017

Agenda



* Revisit release process automation discussion
    * Federation wants to release for 1.9 but is now in a separate repo
    * @david-mcmahon: I can’t make this time slot unfortunately, but please send me any docs or information on what exactly you're trying to achieve here.  
        * My team is working on a few solutions to assist in releasing non-core Kubernetes components.  Most don’t need the full orchestration that[ anago](https://github.com/kubernetes/release#kubernetes-release-process) provides for k/k.
        * See some related work[ here for cluster-registry](https://github.com/kubernetes/cluster-registry/compare/master...javier-b-perez:cloud-builder) building using Google Cloud Builder
        * Also the recent related[ update to push-build.sh](https://github.com/kubernetes/release/pull/444) to support federation pushes.
* Can we apply[ milestone automation](https://github.com/kubernetes/community/blob/master/contributors/devel/release/issues.md) to[ PRs too](https://github.com/kubernetes/sig-release/issues/26)? [enisoc]


## 24 October 2017

Agenda



* [Code freeze enforcement](https://github.com/kubernetes/features/pull/493)


## 10 October 2017

Agenda



* Kops team would like to partner with sig-release to have kops release process automated.  Draft outline:[ https://docs.google.com/a/cnmconsulting.net/document/ d/1i29ntaLjRVmQuUkUXfnkL3_jLby38At98kaE5VEJvg8/edit?usp=sharing](https://docs.google.com/a/cnmconsulting.net/document/d/1i29ntaLjRVmQuUkUXfnkL3_jLby38At98kaE5VEJvg8/edit?usp=sharing) 
    * Want to partner with SIG Release
    * Focus on release mechanics
    * Where to put trusted keys?
    * Want to find common denominators around projects
    * General guidelines are important, and perhaps an end-state where artifacts get published
    * Define how we handle release artifacts and build backwards from there
    * Need to put something on the end of the test infra that does the push
    * Q: Who can make a call on where the bits in the registry live? \
A: 
    * Notes
* Reminder about[ release roles for 1.9](https://docs.google.com/spreadsheets/d/1NGEtufpSfYlJwsB_0D9gF80Y24JQEreut9gT_QIN6Kk/edit#gid=0)
    * solidify the roles
* When/How do release roles get finalized?
    * Jaice will send announcement to begin timer on lazy consensus
* Product Management reboot proposal[ draft](https://docs.google.com/document/d/113gnr3tKXv79J0IYHyg6VbQwLfb-sCYGphW4D9dLZ_E/edit#)
* Updated milestone munger has been deployed
    * Next step is migrating to a prow plugin
        * Better maintainability
        * Enables slack notifications


## 29 August 2017

Agenda - Recording



* Attending
    * Jaice Singer DuMars
    * Radhika
    * Jennifer Rondeau
    * Mehdy Bohlool
    * Caleb Miles
    * Aaron Crickenberger

Notes:



* 1.8 Release update
    * [Issues](http://bit.ly/18issues)
    * [Release notes](http://bit.ly/18relnotes)
    * [Feature tracking](http://bit.ly/18features)
    * [Tests](http://k8s-testgrid.appspot.com/release-master-blocking#Summary)
    * Anything else we need to be tracking
        * Upgrade tests are failing - Eric and Mehdy will look at these
        * Serial tests are looking mostly ok
        * SIGs are not checking off their release note additions, so we need to closely monitor this
    * Issue automation for milestone labeling is merged, needs an announcement
        * Getting a Prow plugin to limit who can apply “approved-for-milestone” label, akin to LGTM controls
        * Use of[ OWNERS_ALIASES](https://github.com/kubernetes/kubernetes/blob/master/OWNERS_ALIASES) might be fraught with peril
        * Short term, we may not need auditability as a primary concern, and will defer to teams
        * Aaron: SIG Release creates a GitHub team for this purpose, specifically
        * Having a limited entry point into the team may increase security around the implementation
        * Notify:  kubernetes-dev, kubernetes-sig-leads, kubernetes-sig-release, Community meeting 
        * Caleb will help Maru create the team
    * What are the true release-blocking tests?  How do we actually get a 100% passing version
        * A release-blocking job that only runs X days may not be useful
            * More frequent runs may be required, and ergo removed from required if it can’t be
        * Currently many failing GKE/GCE jobs
        * Need to establish accountability
            * Guidelines around “how long can something block the submit queue”?
            * Triage + plan to resolution / project-wide SLA = policy
        * Who actually owns these tests?
        * Jaice will draft _something_ here
    * 
* Announcements?
    * 


## 15 August 2017

Agenda - Recording



* Attending
    * Aaron Crickenberger
    * Jaice Singer DuMars
    * Phillip Wittrock
    * Mehdy Bohlool
    * Steve Perry
    * Jennifer Rondeau
    * Radhika
    * Caleb Miles
    * Garrett Rodrigues
    * Brian Tardell
    * Ryan Kubiak
    * David McMahon

Notes



* 1.8 Release update
    * Feature freeze exceptions
        * [https://github.com/kubernetes/features/issues/382](https://github.com/kubernetes/features/issues/382)
        * [https://github.com/kubernetes/features/issues/384](https://github.com/kubernetes/features/issues/384) 
        * [https://github.com/kubernetes/features/issues/383](https://github.com/kubernetes/features/issues/383)
        * [https://github.com/kubernetes/features/issues/386](https://github.com/kubernetes/features/issues/386)
        * [https://github.com/kubernetes/features/issues/387](https://github.com/kubernetes/features/issues/387)
            * Jaice would be deferent to features which are already in progress
            * Jaice to take ownership of the PRs with Ihor and see what their status is ~ _Done as of 8/16 JSD_
            * **Question**: how does the feature freeze impact the release cadence? **&lt;-** **Jaice **Idea is to have some idea what work is planned before coding starts. Are these features “funded” with development efforts. Ideally we would have a release planning session where SIGs present what work is scheduled, what cross SIG dependencies exist, and SIGs would be accountable to their commitments
    * [https://github.com/kubernetes/features/blob/master/release-1.8/release-1.8.md](https://github.com/kubernetes/features/blob/master/release-1.8/release-1.8.md)
        * Still no alphas cut?
            * Should be three already. Is this a risk?
                * Jaice: it is a risk, we have not have met the criteria for releasing with regards to CI
                * Caleb: I believe this is 
                * Grod: next (last) alpha is scheduled for 23rd August. We should aim to have all tests passing by then. Suggest using 23 August as hard deadline for all tests passing
            * Thur 17 plan to cut an alpha - even if tests aren’t passing
                * Meant to validate release cutting infra is correct
            * Need to find a branch manager for weeks Adam is out
* [Merged PRs in the 1.8 milestone](https://docs.google.com/spreadsheets/d/18V_8YvqemC96W17O_3WZ6H_kS7r77NAI9FqUHAM2MS0/edit?usp=sharing) with release-note=true 
    * [GitHub query](https://github.com/kubernetes/kubernetes/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Amerged%20label%3Arelease-note%20milestone%3Av1.8)
* Dawn’s proposal lives!  Jago hiring 2 people to work on the coding
    * Strive to link issues
    * How do we get more support for all the back end processes/bots/CI?
    * 
* [Docs PRs](https://github.com/kubernetes/features/pulls) in flight
    * Radhika sending an email to each group, then collect, edit and apply tech writer magic sprinkles 
    * re: action required, need to untangle release note PRs
    * Better release note template might help
    * 2 classes of release notes:
        * Items in the k/features
        * items in PRs against k/k ~ not organized … 
* Planning for roll-out of milestone-maintaining munger
    * Proposal:[ https://github.com/kubernetes/community/blob/master/contributors/devel/release/issues.md](https://github.com/kubernetes/community/blob/master/contributors/devel/release/issues.md)
    * PR:[ https://github.com/kubernetes/test-infra/pull/4052](https://github.com/kubernetes/test-infra/pull/4052)
    * If you don’t label it sufficient for triage, it warns, then moves out of the milestone
        * priority, type, SIG
        * 3 day grace period + pings to GH UIDs (no email currently)
        * Need to send email to all the SIGs / present at community meeting
        * For critical/urgent they warn without kicking out
        * labels will allow us to track issues that get ejected
        * Initial grace period could be 1 weeks -  Enough to give SIGs a chance to take action
        * This is step one, but the proposal has additional work items
        * Is there a notion of SIGs acknowledging ownership?
        * Maru needs a cool hat


## 1 August 2017

Agenda - Recording



* Attending
    * Jaice Singer DuMars
    * Erick Fejta
    * David McMahon
    * Aaron Crickenberger
    * Phillip Wittrock
    * Dan Gillespie
    * Joe Betz
    * Steve Perry
    * Maru Newby
    * Caleb Miles

Notes: JSD



* 1.8 Release update
    * Feature freeze status check
        * Any issues with no/wrong SIG
        * This is a “stability” release, so are there things that we see in the feature queue that should be deferred?
        * Does anyone know of notable missing features?
    * Release team roll call
        * [https://github.com/kubernetes/features/blob/master/release-1.8/release_team.md](https://github.com/kubernetes/features/blob/master/release-1.8/release_team.md)
    * Release risks
        * Upgrade testing
        * Issue inclusion process ~_ need a way to decide what a substantial PR is for the contrib ladder_, also ensure that PRs are properly documented for release notes
            * Is Dawn’s proposal being staffed?
            * Work to extract release notes from the issue has not been completed
                * This part happens in[ relnotes](https://github.com/kubernetes/release/blob/master/relnotes) and I can do this [@david-mcmahon]
            * Need to ensure there’s a clear value prop for the requirement
        * Phil’s proposal needs some dry run
* kops test failures and escalation of[ this](https://github.com/kubernetes/kubernetes/issues/49981)
    * escalate to Justin SB / Kris Nova / Luis 
    * kops is unique in that it’s fully owned
    * Test fails because of X and is owned by SIG Y, Z, B
        * Ownership gets dropped so it’s difficult to get traction
        * Release team has to find someone to fix it, which can be circular
    * When low-level or high-impact tests fail, hard to know where to send this
    * etcd testing specifically is tricky to understand
    * umbrella issue for putting [sig-foo] in e2e test names ~ on Aaron’s backlog + Phillip Wittrock
        * [https://github.com/kubernetes/kubernetes/issues/49161](https://github.com/kubernetes/kubernetes/issues/49161)
        * 
    *  Escalations for SIGs:  mailing list / Slack / Individuals
        * Put ACTION REQUIRED in the subject or IMPORTANT
        * SIG rep for release if you have a SIG-owned issue or feature in the release ( if ! sig-lead ; then someone else)
    * 
* upgrade and serial tests - massive failures [pwittrock]
    * Upgrade - Mehdy Bohlool?
        * These have been highly problematic in past releases
    * Serial - ???
        * Must pass before a release is cut currently, but not blocking
        * Could SIG merge be blocked by failing serial test?
        * SIG-specific tests are the only way to validate a SIG’s contribution, so if it’s not fixed, then merges for code needing its validation should be blocked
        * Break up serial jobs and apply them to a cluster that is not subject to prior changes
        * Write this up so we can get feedback ASAP [ Maru & Phillip ]
        * Execution frequency would be lowered
* etcd3 test flakes [pwittrock]
* [Hooking in deb/rpm publishing in anago](https://github.com/kubernetes/release/issues/365) [calebamiles]
    * A/B/RC/Final 
    * Needs someone from Google because of the bucket (Mike Danese has done this up to now)
    * Permissions model may be interfering
    * Anago is OSS so someone could update it, but there may be some tripwire around Google accounts vs. not
    * Caleb will connect with mikedanese@ (@mikedanese) to get a walkthrough
    * Also reach out to djmm@ (@david-mcmahon) for any details on the included google layer that does google-specifics (push)


## 18 July 2017

Agenda -[ Recording](https://youtu.be/I0KbWz8MTMk)



* Attending
    * Jaice Singer DuMars
    * Phillip WIttrock
    * Caleb Miles
    * Dan Gillespie
    * Aaron Crickenberger
    * 
* Notes: JSD
* 1.8 Release Team [DG]
    * Empty
    * Aaron willing to help but cannot commit to a role full-time
        * Assisted with issue triage and pushing PR’s forward last time
        * Trying to highlight escalation paths
    * Companies should be funding a role, especially in terms of commitment to the project
    * Going to advertise for more role involvement
        * At community meeting we will announce that we have a tentative lead/second & ask (final) if there are any other nominations
        * Also posting to SIG release list 
        * Alpha release stability needs to be understood - no more releases without green tests
    * Phil is going to check on Google-specific roles
* Establishing release criteria / standards / due dates [DG]
    * **Set[ release due dates](https://github.com/kubernetes/features/pull/305) for all of these items**: (merge[ https://github.com/kubernetes/features/pull/305](https://github.com/kubernetes/features/pull/305) )
    * CI test signal
        * Which suites/jobs are required?
            * As of 1.7 anago polls testgrid’s config.yaml for release blocking jobs;[ issue for context](https://github.com/kubernetes/release/issues/340)
            * Anago polling testgrid’s config.yaml[ https://github.com/kubernetes/release/blob/master/lib/releaselib.sh#L115](https://github.com/kubernetes/release/blob/master/lib/releaselib.sh#L115)
            * The specific set of tesjobsts from config.yaml[ https://github.com/kubernetes/test-infra/blob/master/testgrid/config/config.yaml#L2036-L2119](https://github.com/kubernetes/test-infra/blob/master/testgrid/config/config.yaml#L2036-L2119)
            * Of note: upgrade jobs are not included in this list
            * Of velocity note: slow jobs are
        * What tolerance for flakes do we expect?
            * ie. 5 passes in a row on a commit we are releasing
            * In the past, we as humans have used “3 passes in a row”
            * Anago appears to look at 2 passes in a row if[ this comment in find_green_build](https://github.com/kubernetes/release/blob/2338077e3911ad2ede6473b2777ead38e9edf5b0/find_green_build#L32) is to be believed
        * When in the release cycle should this be stabilized?
            * Prior to cutting a build/release
        * Need to determine how long we support version-specific test cases, e.g. etcd2 upgrade path, beta upgrades ~  what SIG owns upgrade lifecycles?  (SIG Cluster Lifecycle?)
            * API Policy
            * Component upgrades
            * Management of failing tests
            * Need to raise this in community and get test-owner reps in the release meetings
    * Upgrade test signal
        * Which upgrade conditions do we expect to validate?
            * Multi-master?
            * etcd version?
        * When in the release cycle should this be stabilized?
    * Release Notes
        * What format are they expected to be in?
        * When in the release should this be complete?
        * By the time feature freeze happens, we should have enough to create the PM-oriented “theme release notes”
        * Need to enact the ‘marketing’ and ‘release notes’ roles
        * A draft of the notes by code freeze will be required
            * Call out features with no docs
            * Call out features with no release notes
        * Cherry picks are hard, and porting info across PRs is difficult
        * Maybe this is not in the PR, and lives somewhere else
    * Documentation
        * When in the release should this be complete?
    * Bugs
* **Action items summary from this meeting:**
    * ~~Raise awareness (again) about release roles~~ (@ community and @ SIG-Release Google Group) ~ **Jaice & Caleb**
    * **~~PWittrock to seek out Googlers for G-specific release roles~~** [ ~~branch manager~~ | patch release manager ]
    * ~~Give attention to Community PR 305 and ratify on Friday~~ ~ **SIG Release team**
    * ~~Schedule follow-up meeting for Friday at 2PM PT / 5PM ET / 9PM UTC~~ ~ **Caleb or Dan**
    * Determine if/how an Alpha is possible ~ **SIG Release team**
    * ~~Use Anago test list as the source of truth for release-blocking tests~~ ~ **Aaron** will provide this info


## 6 June 2017

Agenda - Recording



* Attending:
    * Jaice Singer DuMars
    * Anthony Yeh
    * Ihor Dvoretskyi
    * Maru Newby
    * Caleb Miles
    * Doug Claar
    * Javier Perez
    * David McMahon
    * Andrew Chen
    * Phil Wittrock
* Notes: JSD
* [ 0:01 ] What are we hoping to achieve from this meeting?
    * How do we make long-term improvements to the release process?
    * How do we empower teams to do more of the release tasking themselves?
    * Different than the work cutting releases
    * Iterating and improving on every release as a group
    * Release team forms/disbands around the release so not a durable group
    * Tools we need in order to improve the process, vs. release team uses current tool set
* [ 0:09 ] Updates
    * SIG CLI was working on a page in the test grid, filtered for easy understanding 
    * [1.6 Retro Action Items](https://docs.google.com/document/d/1JjQW14R9FCaKceZV2hE9Wx41LIUoZST8gp8FF5EqMmk/edit)
    * We need a measurable goal
        * Do a mock release weekly to verify the process and testing [ Phil leading this ]
            * What are the tools needed to make this happen?
            * Start with a PR for discussing how to enact this work 
            * Release team retro [ Jaice ]
                * ID bugs and issues we already know about [ ID targets for continuous improvement ] 
                * Schedule meeting