kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
community/sig-node/archive/ci-subgroup-notes-2021.md

60 KiB
Raw Blame History

Kubernetes SIG-Node CI subgroup notes

12/29/2021

Cancelled - year-end holiday break

12/22/2021

12/15/2021

12/09/2021

  • ehashman
    • PR: https://github.com/kubernetes/test-infra/pull/24595
    • There are some tests I don’t want to migrate as part of this PR with special configs: e.g. CPU manager, hugepages, etc. but are failing so no point in running them
      • Ideally, since these are serial, would like to see all tests using the same config moved under a single job
    • Presubmits need to be done separately
    • Will file issues for split work (presumbits)
  • ruiwen-zhao
    • Summary API test flaky on kubelet-gce-e2e-swap-ubuntu since the beginning of test job history
    • Same test passes on -fedora testgrid
  • Agreed: cancelling Thursday alternate time meeting due to lack of attendance.

12/01/2021

11/24/2021

Perf (memory chart) - let’s see if trend continues next week
![][image13]

11/17/2021

11/11/2021

  • https://github.com/kubernetes/kubernetes/issues/106204
    • Release blocker?
  • danielle
    • Lets drop the dependency on GPUs and reduce some maintenance for ourselves. Also apparently the device tests were broken in the same way GPU tests originally were (but we missed it bc of [Flaky])
  • aditi
    • Thoughts on adding credential provider to node e2e?
  • Status of node serial
    • Let’s make it green by removing flakes out

10/26/2021

10/20/2021

  • fromani
    • Memory manager tests have implicit dep on memory fragmentation (or lack thereof)
    • ContainerRuntimeRestart - fails for timeout, but we WANT to saturate the node with pods (that’s the whole point of the test!)
    • How do we keep these tests while keeping it reliable (no false negatives)?
    • Separate lanes seem a bit excessive, any other idea?
    • Wait for the PR to reduce amount of allocated hugepages under the test
      • If the PR will fix flakes, remove the separated lane, otherwise remove the test from serial lane with notes why it needs a separate lane
  • imran
    Updated the test-infra PR to include a skip value with `--skip="\[Flaky\]|\[Serial\]"`
    Just need to merge and proceed with the next set of steps.
  • mmiranda96
  • jlebon
    • Danielle will take a look
      • Fromani will have a look as well
    • What about cloud-init?
    • Cloud ignition is an alternative - needs to be adapted
    • Have a way to prepare all the binaries upfront?
  • mmiranda96
  • ehashman
    • Let’s hold another one in november and then if poor attendance again, cancel

10/14/2021

10/06/2021

- all these tools are deprecated and in maintenance mode. Tools evolved from bash script and became unmaintainable
- kubetest2 is designed to be extensible and will replace it.
- PLuggable on where to test (GCE, AWS, Kind, etc.)
- Pluggable on what to test

Thousands of jobs using old tools. The process of switching all the jobs will be slow. Some of the jobs will be moved to kubetest2. Presubmits and release blocking jobs are the first target.

Mainly: awareness of the project. Feature requests must go to kubetest2 now.

Most significant impacting change - node tester. kubetest2 will use a makefile as a source of truth. make test_e2e lets you run tests, but kubetest is not using it. kubetest2 will change this and will only use makefile. Dealing with test infra will be mostly when bugs are encountered, no need to deal with it any longer.

Danielle: some tests needs more features
Amwat: yes, can be added in node tester

	Timeline?

Amwat: scoped to presubmits and release blocking - 1.24 is a target version. At least jobs will start be running. No timeline for other jobs.

  • SergeyKanzhelev
    • File an issue for the future
    • Revert to image family for now
    • Find somebody to investigate
  • SergeyKanzhelev
  • mmiranda96
    • Issue on kops for last year event: https://github.com/kubernetes/kops/issues/9920
    • No - Kubernetes has explicitly opted out of Hacktoberfest project wide. The quality of contributions we’ve historically gotten have been very low and created a lot of extra cleanup work for maintainers.

09/29/2021

09/22/2021

  • mmiranda96
    • Update list of features manually to make progress on the alpha job cleanup.
    • Work on updating tags for jobs
  • ehashman
    • GCP accounts for node contributors - need a list of use cases
    • Dani to drive when she returns from PTO?
  • Sergey

09/15/2021

  • alukiano
    • The feature should be deprecated under 1.23
    • The feature tests took the most time of the serial lane, for comparison with DynamicKubeletConfig serial lane takes ~3h, without ~1h
    • We can create a separate lane for deprecated features(I know we do not like an idea about the separate lane)
    • Let’s prioritize removing the feature
    • Need unified approach for kubelet configuration
    • DynamicKubeletConfig flakes are often due to the feature being unreliable/kubelet not restarting
    • Action: Elana to file an issue to detail work that needs to be done to move tests off DynamicKubeletConfig, assign Danielle and cc Sergey 
  • arnaud
  • ehashman
    • Danielle: there are some optimizations we can do, including eviction tests, GPU tests, etc. to be less wasteful
      • Fromani: agreed. Only few tests actually benefit from availability of GPU devices (and they need to be changed accordingly)
    • Action: Sergey to follow up on combining node pool with the rest of the project
  • mmiranda96
  • Bugs triage: https://github.com/orgs/kubernetes/projects/59

09/09/2021

09/01/2021

08/25/2021 Cancelled due to hosts unavailability

08/18/2021

  • mmiranda96
    • AI: create issue to review other NPD tests.
  • mmiranda96
    • Mike: investigate and find the test grid. + create an issue.
  • SergeyKanzhelev
    • 18 is the number of projects used.
    • The best way to improve utilization is less test jobs or faster tests inside the job
  • alukiano

08/12/2021

08/04/2021

07/28/2021

07/21/2021

  • haircommander
  • fromani
    • The k8s e2e test suite has a fair amount of tests which need device plugin, because they exercise device manager -directly or indirectly.
      We mostly use SRIOV devices, because SRIOV devices are just the cheapest and easiest supported device to get, so this is why we wrote the tests in k8s to consume them.
      But we don't have device plugin support on CI. We do have gpus-enabled machine, but it's a subset and should be used sparingly (e.g. not every PR should use them. Or can we just use gpus every time? I expect no for cost reasons, but worth mentioning).
      So today a large amount of tests just skip on CI. This is especially evident in the serial lane and in the resource management area
      In RH we actually have machines with SRIOV devices which run the e2e testsuite, but this is of course suboptimal for a number of reasons; a much better state for everyone would be to actually have some device plugins in u/s CI.
      There are some options we can discuss as community:
      • 1. use sample plugin
      • 2. fake sriov devices (I can elaborate on this if there is interest
      • 3. just use GPUs?
      • 4. Just bump the spec of the CI machines to have SRIOV devices?
  • mkmir
  • ehashman
  • SergeyKanzhelev

07/14/2021

  • Sergey
    • NodeConformance historically tried to be two things:
      • A set of e2e tests that you just needed a single node to run
      • Conformance-like test for nodes
    • ehashman
      • For CRI conformance-like tests, label them CRIValidation -- NodeAgnostic?
      • For the set of e2e tests you just need a single node to run, let’s come up with a new name -- needs a name (anything in test/e2e_node) (SingleNodeTest or KubeletLocal)
      • Note: some tests may overlap between both sets
    • Action: add a plan to cover splitting the use cases for current tests
    • Action: send out NodeConformance plan, soliciting feedback
  • ehashman
  • fromani

07/08/2021

06/30/2021

06/23/2021

Attendees:
![][image14]

06/16/2021

Attendees:
![][image15]

06/10/2021

Attendees:

  • Sergey

Agenda:

06/02/2021

Agenda:

05/25/2021

Agenda:

05/19/2021

Attendees:
![][image16]

Agenda:

05/12/2021

Agenda:

05/04/2021 Cancel for KubeCon

04/28/2021 Short sync up

Attendees:
![][image17]

Agenda:

  • Follow ups - need to move to the next week: 
  • fromani
    • Shared global state between tests
    • Any objections to move away from it?
    • Storing state of the kubelet
    • Ideally each e2e test has it’s own state
  • ehashman
    • Currently don’t have any perf/scalability tests for upstream kubelet resource utilization (CPU/memory)

04/21/2021

Attendees:

  • Sergey
  • Elana
  • Artyom

Agenda:

  • triage

04/14/2021

Attendees:

  • fromani
  • ehashman
  • Sergey Kanzhelev
  • David Porter
  • Amim Knabben
  • Qiutong Song
  • Madhav Jivrajani
  • Jiaming Xu

Agenda:

04/07/2021

Attendees:

  • Elana Hashman
  • Sergey Kanzhelev
  • Alukiano
  • Tao
  • Qiutong

Agrenda:

  • Artem will start looking at fake NUMA flag.

03/30/2021 Cancelled

03/24/2021

Agenda

https://github.com/kubernetes/kubernetes/issues?q=is%3Aopen+milestone%3Av1.21+label%3Asig%2Fnode

Discussing https://github.com/kubernetes/kubernetes/pull/99336:

  • Overall feeling is that it’s too late for unknown unknowns introduced by this PR.
  • 1.18 cherry-picking is not Node team problem, more release team problem. Maybe release team will need an exception.

03/17/2021

Attendees

Agenda

Direction long term is not to test the whole matrix on presubmits, but have a good signal with failures easy to investigate by contributors. Maybe PR needs to be replaced with a single job with both runtimes.

Action: determine a long-term plan to merge all node presubmits into one job, using a name that doesn’t reveal the underlying runtimes. (e.g. cleanup ubuntu-containerd* tests)
Timeframe: dependent on a presubmit policy, maybe will happen next release cycle (1.22)

03/10/2021

Attendees

  • ehashman
  • fromanirh
  • swsehgal

Agenda

03/03/2021

Attendees

Agenda

  • Containerd tests. TODO: insert links
  • Triage
  • alukiano

02/24/2021.

Attendees (7 on call):
![][image18]

  1. Containerd plan https://github.com/kubernetes/test-infra/issues/18570
  2. Questions about https://github.com/kubernetes/test-infra/pull/20937 and node-kubelet-serial tests (https://testgrid.k8s.io/sig-node-kubelet#node-kubelet-serial)
  3. Announcement: node n-2 version skew tests to be discussed at SIG Arch tomorrow: https://groups.google.com/g/kubernetes-sig-architecture/c/QX-4qq2krm0/m/998T3cJUBQAJ

Product triage: https://github.com/orgs/kubernetes/projects/49

02/17/2021

Product triage: https://github.com/orgs/kubernetes/projects/49

02/08/2021

Sergey

https://doodle.com/poll/ii5vyde6wpp3migm?utm_campaign=poll_update_participant_admin&utm_medium=email&utm_source=poll_transactional&utm_content=gotopoll-cta#table

![][image19]

Triage: https://github.com/orgs/kubernetes/projects/43

  • Suggest MHBauer to approver. Elana to reach out

02/01/2021 [skipping]

01/25/2021

Agenda:

Aditi
Sergey

https://github.com/kubernetes/k8s.io/issues/956#issuecomment-764847132?

  • Need for deflaking tests. “Pause” job and SSH access
  • Sometime hard to understand what failed and why PR validation failed
knabben
ehashman
  • Non-CI PR triage: Node board https://github.com/orgs/kubernetes/projects/49
  • Action (ehashman): will add note cards to columns of project board to explain how it works
  • Action (ehashman): will draft doc on HOW-TO node review for community repo, tag attendees, and leave open for lazy consensus/feedback
Meeting time
  • Action: Sergey to send doodle for maybe moving this meeting? Mondays frequently conflict.

01/11/2021

Attendees (7 on the call):

Agenda:

  • triage

01/04/2021

Attendees:

Agenda:

knabben
Sergey

Volunteers to help with this effort

Victor Pickard (Red Hat), nick=vpickard, vpickard@redhat.com
Jay Pipes (AWS), nick=jaypipes, gh=jaypipes, jaypipes@gmail.com
Balaji (AWS), nick=srisaranbalaji, gh=SaranBalaji90, srisaranbalaji@gmail.com
Morgan Bauer (IBM), slack=mhb, gh=mhbauer, mail=mbauer@us.ibm.com
Ning Liao (Google), nick=nliao, mail=ningliao@google.com
David Porter (Google), nick=davidporter; mail=porterdavid@google.com
Hanfei Lin (Google), nick=hanfeil; mail=hanfeil@google.com
Hugo Huang (Google), nick=tangent; mail=tangent@google.com
Roy Yang(Google), nick=roy; mail=royyang@google.com
Aaron Crickenberger (Google), nick=spiffxp, spiffxp@gmail.com
nick=Archer
Ed Bartosh (Intel), slack=Ed, github=bart0sh eduard.bartosh@intel.com
Daniel Mangum (upbound.io), nick=hasheddan
Chirag Tayal (PayPal) nick=ctayal, chiragtayal@gmail.com
Zhi Feng(Airbnb), nick=Zhi, helenfengzhi@gmail.com
Dims, nick=dims, davanum@gmail.com
Jacob Blain Christen (Rancher Labs), nick=dweomer; mail=dweomer5@gmail.com
Artyom Lukianov(Red Hat), nick(github)=cynepco3hahue,nick(slack)=alukiano,mail=alukiano@redhat.com
Swati Sehgal (Red Hat), slack=swsehgal; mail swsehgal@redhat.com
Jorge Alarcon, nick=alejandrox1, alarcj137@gmail.com
Sascha Grunert (SUSE), nick=sascha, sgrunert@suse.com
Srini Brahmaroutu(IBM), slack=srbrahma, gh=brahmaroutu, mail=srbrahma@us.ibm.com
Tim Pepper (VMware), slack=tpepper, gh=tpepper, mail=tpepper@vmware.com
John Taylor (IBM), mail=jtaylor1@uk.ibm.com
Francesco Romani (Red Hat), nick=fromani; mail=fromani@redhat.com
Karan Goel (Google), nick=karan; mail=karangoel@google.com
Sergey Kanzhelev (Google), nick=SergeyKanzhelev; mail=skanzhelev@google.com
Mike Carlise (Salesforce), nick=micarlise, mail=micarlise@gmail.com
Matt Merkes (AWS), nick=merkes, mail=matt.merkes@gmail.com
Amim Knabben (Loadsmart), nick=knabben, mail=amim.knabben@gmail.com
Swati Sehgal (Red Hat), nick(slack)=swsehgal, nick(github)= swatisehgal, mail = swsehgal@redhat.com
Harshal Patil (Red Hat), slack=Harshal, gh=harche, mail=harpatil@redhat.com
Elana Hashman (Red Hat), nick=ehashman, mail=ehashman@redhat.com
Paco Xu(DaoCloud), nick=paco,mail=paco.xu@daocloud.io