kubernetes
/
community
mirror of https://github.com/kubernetes/community.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Add sig-instrumentation meeting note archive

This commit is contained in:
Elana Hashman 2020-11-12 14:10:51 -08:00
parent 839dafd0a5
commit 9cf380a9c2
No known key found for this signature in database
GPG Key ID: D37F7B2A20B48FA0
4 changed files with 1495 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

380
sig-instrumentation/archive/meeting-notes-2016.md Normal file
View File

@ -0,0 +1,380 @@
## 2016-12-15
Agenda:
* Demo by Datadog (rescheduled)
* Kubernetes Metric Conventions: [https://docs.google.com/document/d/1YVs02Li6QFCg8Th2Wa4z1u2NBlQHDp2dj3EdAt6uskE/edit#](https://docs.google.com/document/d/1YVs02Li6QFCg8Th2Wa4z1u2NBlQHDp2dj3EdAt6uskE/edit#)
* Resource metrics API: looking towards beta
* [https://docs.google.com/document/d/1t0G7OS6OP9qPndkkNROCu0pF3-vkDmzonmT-6gEWcx0/edit?ts=5852bda8](https://docs.google.com/document/d/1t0G7OS6OP9qPndkkNROCu0pF3-vkDmzonmT-6gEWcx0/edit?ts=5852bda8)
Notes:
* Put metric convention document somewhere visible for reference
* [https://github.com/kubernetes/community/tree/master/contributors/devel](https://github.com/kubernetes/community/tree/master/contributors/devel)
* Resource metrics API should be moved towards beta
* To be finalized after holiday break
* Working towards beta in 1.7
* Custom metrics API:
* [https://github.com/kubernetes/community/pull/152/files](https://github.com/kubernetes/community/pull/152/files)
## 2016-12-08
**Warning: This meeting will be about logging. If you are not interested please skip.**
Agenda
* Restart LogDir proposal ([https://github.com/kubernetes/kubernetes/pull/13010](https://github.com/kubernetes/kubernetes/pull/13010))
* Alternative [https://github.com/kubernetes/kubernetes/pull/33111](https://github.com/kubernetes/kubernetes/pull/33111)
Meeting notes: [https://gist.github.com/leahnp/463501f6dfe39f6f21ea5d3ebcb787d7](https://gist.github.com/leahnp/463501f6dfe39f6f21ea5d3ebcb787d7)
## 2016-12-01
### Agenda
* Heapster needs your help
* [sross] Need to come up with map of sinks to maintainers
* Maybe consider dropping sinks without mainters
* [sross] need statement of plans for Heapster
* [sross] putting into maintenance mode, what does maintenance mode entail, should we continue accepting sinks?
* [piosz] to write something up and send out
* [mwringe] what is plan for timeline for monitoring pipeline work
* [piosz] plan is starting work Q2 2017, unless anyone else can help
* [piosz] major missing component is discovery summarizer
* [sross] we (Red Hat) are willing to help out in this area
## [Cancelled] 2016-11-24: Thanksgiving in US
## [Cancelled] 2016-11-17: no meeting week
## [Cancelled] 2016-11-10: Kubecon
## [Cancelled] 2016-11-03
## 2016-10-27
### Agenda
* F2f meeting about monitoring in Seattle during KubeCon (on Monday Nov 7th)
## 2016-10-20
**Warning: This meeting will be about logging. If you are not interested please skip.**
### Agenda
* f2f meeting about logging in Seattle during KubeCon (probably on Monday Nov 7th)
* There is going to be a kubernetes dev summit (Nov 10th) meeting for logging
* Group administrivia: frequency? Length? Topics?
* Current state of logging in Kubernetes
* What’s going on with logging?
Notes
Developers Summit - 45 minute unconference topic on the future of logging
- moderated by Vishnu and Patrick
- open to anyone who is attending the Kubernetes Developers Conference
Discussion of Face to Face meeting - Piotr and Patrick to sync up offline
Frequency: every three weeks, going to skip next week/push back one week next meeting is during KubeCon Developers Summit.
- There will be an announcement for exactly when the next meeting is
Logging Discussion Topics:
- logging volumes (proposal started by David Cowden -[ https://docs.google.com/document/d/1K2hh7nQ9glYzGE-5J7oKBB7oK3S_MKqwCISXZK-sB2Q/edit#](https://docs.google.com/document/d/1K2hh7nQ9glYzGE-5J7oKBB7oK3S_MKqwCISXZK-sB2Q/edit#))
- hot loop logging and verbosity for scalability issues.
- how to detect spammy instances
- how to not let this wreck the cluster
- general dissatisfaction with the logging facility
- structured logging kubernetes wide for consistent consumption
- application log type detection
- what metadata do we need to carry through a logging pipeline to id a source system (e.g. mysql, user application)
- what do logging vendors need supplied to aid in this
Current logging pipelines
- fluentd direct to GCP or ES
- fluentd to kafka to fluentd to ES
Action Items
- Piotr & Patrick to determine f2f details
- Try and get logging vendors to join the SIG
## [Cancelled] 2016-10-13
## 2016-10-06
### Agenda
* No response from sig api machinery (moving to next meeting)
* Continue discussion on monitoring architecture
* Agreed to versioned, well-defined API
* Rest API vs. Query Language
* A webhook model was suggested for the APIs (like Auth in Kube today)
* [sross] has concerns over discoverability of webhooks
* Webhook vs API server is largely an implementation question
* will decide on discovery vs webhook for consumption once we get the API design in place
* [sross] will propose an API design for the custom metrics API and historical metrics API
* Discuss [roadmap](https://docs.google.com/document/d/1j6uHkU8m6GvElNKCJdBN8KrejkUzVbp2l0zTyeSxrl8/edit)
* Discussed briefly, please go read afterwards
* [sross] to lead push on custom metrics design/implementation for 1.5
* 1.5 API features will be mainly implemented in terms of Heapster
* looking forward for one-click install of 3rd party monitoring (possibly Prometheus, but as an out of the box, one command setup; possible choices for deployment: helm, kpm)
* Logging discussion feasibility conversation (ie: is this a reasonable location for having discussions about logging)
* This may be a reasonable place for logging discussions, if we explicitly note which meetings will discuss logging (and/or when logging will be discussed)
* May also just want to create a separate SIG
* [decarr] mentioned CRI discussion on logging and metrics
* Outcome was that we should sync with SIG node on that, but it should probably stay more in SIG node
## 2016-09-29
### Agenda
* Discuss [Kubernetes monitoring architecture proposal ](https://docs.google.com/document/d/1z7R44MUz_5gRLwsVH0S9rOy8W5naM9XE5NrbeGIqO2k/edit#)
*
### Notes
* Main metrics pipeline used by Kubernetes components
* Separate operator-defined monitoring pipeline for user-exposed monitoring
* Generally collects core metrics redundantly/independently
* Should it be possible to implement the core metrics pipeline on top of the custom monitoring system
* As long as one implements the core metrics API, one could swap it out for scheduler etc.
* Upstream Kubernetes would test against the stable core pipeline
* Replaceable != Pluggable – the entire thing gets replaced in a custom scenario
* Master Metrics API part of main Kubernetes API
* Should further APIs like for historic metrics also be in that group?
* Discussion for sig-apimachinery
* Should Infrastore be part of core Kubernetes
* Provides historic time series data about the system
* Would require implementing a subset of a TSDB
* Not an implemented component, just an API
* What are core metrics exactly?
* CPU, memory, disk
* What about network and ingress?
* Resource estimator would not read from master metrics API but collect information itself (e.g. from kubelet)
## 2016-09-22
### Agenda
* Mission statement: [https://docs.google.com/document/d/15Q47xbYTGHEZ-wVULGSgOSD5Kq-OehJj-MEChVH1kqk/edit?usp=sharing](https://docs.google.com/document/d/15Q47xbYTGHEZ-wVULGSgOSD5Kq-OehJj-MEChVH1kqk/edit?usp=sharing)
* Kubesnap demo
### Notes
* Kubesnap demo by Andrzej Kuriata, Intel ([slides](https://docs.google.com/presentation/d/1fgGik1nq-yEN7Y2dRIQWTjb7r5HEWaG9paDCdvzE_IA/edit?usp=sharing)):
* Daemon set in k8s
* Integration with Heapster
* Mission Statement:
* Enough people to coordinate, but small enough to be focused
* List of people actually doing development/design in the scope of this sig
* Scratchpad before a meeting to set discussions of features before meeting
* Sig autoscaling discussed and committed to features/metrics in previous meetings
* A plan for an api for 1.5?
## 2016-09-15
### Agenda
* Presentation by Eric Lemoine (Mirantis): monitoring Kubernetes with [Snap](http://snap-telemetry.io/) and [Hindsight](https://github.com/trink/hindsight). [Slides](https://docs.google.com/presentation/d/1XWM0UmuYdcP_VsbKg6yiSDb6TR1JmouHdZAnLelBWXg/edit?usp=sharing)
* Meeting frequency
* Ownership SIG instrumentation vs SIG autoscaling
* [Discuss how to export pod labels for cAdvisor metrics (see kubernetes/kubernetes#32326)](https://github.com/trink/hindsight)
### Notes
* Meeting frequency - defer until ownership clarified
* Ownership SIG autoscaling vs instrumentation
* Triggering issue: [https://github.com/kubernetes/kubernetes/issues/31784](https://github.com/kubernetes/kubernetes/issues/31784)
* HPA is consumer of Master Metrics API (also kubectl top, scheduler, UI)
* Could potentially be relevant to monitoring as well
* Make distinction between metrics used by the cluster and metrics about the cluster
* One SIG lead cares about system level metrics, one about the external/monitoring side. Good setup for the SIG to handle both areas?
* Follow up with mission statement on the mailing list taking these things into account
* Kube-state-metrics v0.2.0 was released with many more metrics:
* [https://github.com/kubernetes/kube-state-metrics#metrics](https://github.com/kubernetes/kube-state-metrics#metrics)
## 2016-09-08
### Agenda
* Sylvain Boily showing their monitoring solution
### Notes
* Demo by Sylvain on their monitoring setup using InfluxDB+Grafana+Kapacitor
* Scraping metrics from Heapster, Eventer, and apiserver
* Separation apiserver vs kube-state-metrics
* The apiserver exposes metrics on /metrics about the running state of the apiserver process
* How man requests came in from clients? What was their latency?
* Outbound latency to the etcd cluster?
* Kube-state-metrics aims to provide metrics on logical state of the entire Kubernetes cluster
* How many deployments exist?
* How many restarts did pod X have?
* How many available/desired pods does a deployment have?
* How much capacity does node X have?
* Separation Heapster vs [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics/commits/master)
* Heapster holds metrics about characteristics about things running on Kubernetes, used by other system components.
* Currently Heapster asks the Kubelet for cAdvisor metrics vs. kube-state-metrics collecting information from the apiserver
* Should eventer information be consolidated with kube-state-metrics?
* Should we look into the creation of a monitoring namespace / service for all other namespace to use?
* Should monitoring be available out of the box with a k8s installation when done in a private datacenter ?
## 2016-09-01
### Agenda
* State of [Kubernetes monitoring at Soundcloud](https://drive.google.com/file/d/0B_br6xk3Iws3aGZ5NkFMMDRqRjhvM1p1RWZXbVF2aVhiWGZz/view?usp=sharing) (Matthias Rampke)
* Future of [kube-state-metrics](https://github.com/kubernetes/kube-state-metrics)
* Application metric separation in cAdvisor ([https://github.com/google/cadvisor/issues/1420](https://github.com/google/cadvisor/issues/1420))
* ...
### Notes
* Matthias Rampke giving an intro to their Kubernetes monitoring setup
* Currently running Prometheus generally outside of Kubernetes
* Easy migration path from previous infrastructure
* Still using DNS as service discovery instead of Kubernetes API
* Sharded Prometheus servers by team for application monitoring
* Severe lack of metrics around Kubernetes cluster state itself
* Long-term vision (1yr): all services and their dependencies running inside of Kubernetes
* Prometheus part of that via a standard configuration
* Easy to spin up monitoring new components
* People using Heapster as it gives them all metrics in one component
* Something as easy to deploy as Heapster would be useful
* Three sets of metrics
* Those useful only for monitoring (e.g. number of pods)
* Metrics for auto-scaling (CPU, custom app metrics)
* Those that fit both
* Make Prometheus a first-class citizen/best practice for exposing custom auto-scaling metrics?
* Overlap between auto-scaling and monitoring metrics seems generally fine
* storing them twice is okay, auto-scaling metrics are way fewer
* Kube-state-metrics
* Keep it as a playground or fold it into controller manager?
*
## 2016-08-25
### Notes
* CoreOS would like to see
* more instrumentation as insight into cluster
* Remove orthogonal features in for example cadvisor
* RedHat
* Good out-of-the-box solution for cluster observability, component interaction
* Collaboration with sig-autoscaling
* SoundCloud:
* Prometheus originated at SoundCloud
* Bare metal kubernetes setup: separation of monitoring
* Separation of heapster and overall kubernetes architecture
* How are people instrumenting around kubernetes
* Mirantis:
* Scalability of monitoring solutions
* More metadata from kubelet “stats” API: labels are missing for example
* Also interested in “Separation of heapster and overall kubernetes architecture” (from SoundCloud)
* Extended insight into OpenStack & Kubernetes
* During our scalability tests we want to measure k8s behaviour in some set of defined metrics
* Intel:
* Integration of snap into kubernetes
* Help deliver monitoring goals
Where should guides for flavors of monitoring live?
→ ad hoc currently, not all the same
→ best practices in the community
Where are we and where do we want to do? → Google doc will be setup
Next meeting: Discuss google doc & Matthias from SoundCloud will give an insight of how they are using Prometheus to monitor Kubernetes and its pain points.
Next time will use Zoom as hangout limit is 10 participants.
Kubernetes monitoring architecture (~~requires joining [https://groups.google.com/forum/#!forum/kubernetes-sig-node](https://groups.google.com/forum/#!forum/kubernetes-sig-node)~~): [https://docs.google.com/document/d/1HMvhhtV3Xow85iZdowJ7GMsryU6pvjOzruqcJYY9MMI/edit?ts=57b0eec1#heading=h.gav7ymlujqys](https://docs.google.com/document/d/1HMvhhtV3Xow85iZdowJ7GMsryU6pvjOzruqcJYY9MMI/edit?ts=57b0eec1#heading=h.gav7ymlujqys)

485
sig-instrumentation/archive/meeting-notes-2017.md Normal file
View File

@ -0,0 +1,485 @@
## 2017-12-28 (Cancelled - Christmas week)
## 2017-12-14
* Kubernetes Contributors Summit report by Solly
* 1.9 release notes: due EOD today
## 2017-11-30
* Ways of exporting Counter metrics from the subprocess in Prometheus. \
https://github.com/prometheus/snmp_exporter
## 2017-11-16
* Update for community meeting
* Core and Custom Metrics APIs promoted to beta
* Multiple kube-state-metrics releases (many new metrics, stability, features)
* Prometheus backed metrics API gateway: [https://github.com/DirectXMan12/k8s-prometheus-adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter)
* Current work: removing heapster dependencies
## 2017-10-05
Agenda:
* Clayton would like to talk about securing instrumentation endpoints
Notes:
* How to expose etcd metrics if etcd is generally locked down
* Proxy metrics?
* Read status/alert endpoints and transform those metrics?
## 2017-09-21
Agenda:
* Sematext demo (20 minutes)
* https://github.com/sematext/sematext-agent-docker
* Clayton would like to talk about securing instrumentation endpoints
* More components adding instrumentation
* Some are sensitive (raised in sig-auth)
* Would like to identify how we can endorse / suggest instrumentation best practices going forward
## 2017-09-07
Notes:
* Summary of what has been delivered for 1.8
* Metrics api graduation
* Metrics server as recommended for serving metrics for cluster
* As replacement for heapster
* Where to host kube-state-metrics containers?
* No way to give permissions to maintainers on gcr.io
* Keep it in quay.io/coreos for now and provide gcr.io in a best-effort manner
*
## 2017-08-24
Notes:
* Want to to enable metric server by default in 1.8
## 2017-08-10
Agenda:
* Kube-state-metrics in GA
* Custom Metrics API adapters
* Master metrics API going beta
* Kubecon CFP
* Prometheus output is growing
Notes:
* Kube-state-metrics 1.0 was released
* Compatibility aligned with the client-go version it uses
* Load testing was performed and it scales really well
* Well below 200MB even for 1000 nodes clusters with 30 pods/node
* Default deployment manifest comes with addon resizer configuration
* Custom metrics API adapters
* Solly just tagged first release of Prometheus adapter: [https://github.com/DirectXMan12/k8s-prometheus-adapter](https://github.com/DirectXMan12/k8s-prometheus-adapter)
* Stackdriver work in progress
* Potentially move to beta for 1.8
* Historic metrics API not target for 1.8 as it’s a stabilization release
* Master Metrics API going beta [#50148](https://github.com/kubernetes/kubernetes/issues/50148)
* Kubecon coming up
* CFP closing soon
* Metrics exposition with Prometheus client library
* Google ran into problems where they are exposing metrics about different k8s objects over time. This linearly grows the metric registry and the number of metrics linearly increase
* Solution: we have to distinguish between metrics about the running application itself and logical objects like k8s resources. Using custom collectors allows to determine exposed metrics at collection time: [http://godoc.org/github.com/prometheus/client_golang/prometheus#hdr-Custom_Collectors_and_constant_Metrics](http://godoc.org/github.com/prometheus/client_golang/prometheus#hdr-Custom_Collectors_and_constant_Metrics)
* This is how exporters like kube-state-metrics usually handle this. Example of the methods that need to be implemented for a custom collector: [https://github.com/kubernetes/kube-state-metrics/blob/master/collectors/service.go#L79-L94](https://github.com/kubernetes/kube-state-metrics/blob/master/collectors/service.go#L79-L94)
* It’s advisable to serve metrics on two different ports in this case. One port with metrics about the process itself (e.g. requests it received, open FDs), and another one for the objects its extracting metrics for. That’s because one generally may want to apply different rules for extending the metrics’ label set with external information at ingestion time. (Not doing this has been a long running problem with the kubelet which mixed metrics about itself with cAdvisor metrics in pre-1.7)
## 2017-06-29
Agenda:
* Limited-scope API for retrieval of historical metrics (i.e. can we build an API that suits VPA and things like idling w/o re-inventing PrometheusQL)
* Public API for retrieving events from long-term storage.
Notes:
* Proposal: add API to retrieve historical metrics, e.g. for basic dashboarding data that’s currently collected from Heapster, VPA, idling, …
* Can this be folded into the custom metrics API?
* Piotr: let’s first start defining idling as a feature before proposing an API required to implement it
* SIG autoscaling needs historical API for VPA
* Idea: Drop-in replacement for event data that can merge local events and events from long-term storage
* Solly: should be reasonably doable with existing features, similar to metrics API
* Have replacement with higher priority in API aggregator than default handler
* Might need to extend existing events API to deal with lack of time bounds in current API
* Kube State Metrics
* Goal in have release for the 1.7 release
* Haven’t gotten bugs in a while, seems stable
* 1 more metric that needs to be reviewed for consistency, but otherwise consistent
* Piotr: need scalability test, Google has capacity to test, should have some free cycles next week
* Might need sharding, might not (Heapster needs 20GiB for 5k node cluster, which is doable on a cluster that size)
*
## 2017-06-22 (cancelled - lack of agenda)
## 2017-06-15
## 2017-06-08 (cancelled - GKE Summit)
## 2017-06-01 (cancelled - OSS Leadership summit)
## 2017-05-25
Agenda:
* One week to code freeze - checkpoint
* Metrics server [design doc](https://docs.google.com/a/google.com/document/d/1w6-ZfnA18aKYLJ8DCLBFKlv_1umm74x0I2X4V188kvU/edit?usp=sharing) + implementation [kubernetes-incubator/metrics-server](https://github.com/kubernetes-incubator/metrics-server)
Notes:
* Status:
* [WILL NOT BE DONE] Move master metrics API to beta
* Kube-state-metrics stable release [kube-state-metrics#124](https://github.com/kubernetes/kube-state-metrics/issues/124)
* [IN PROGRESS] Metrics-server
* Evaluate custom metrics API state
* Hawkular
* [IN PROGRESS] Prometheus
* [WILL BE DELAYED] Stackdriver
* [POSTPONED TO 1.8] discuss/propose historical metrics API
## 2017-05-18
Agenda:
* Metrics server + [master metrics API](https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1alpha1/types.go) to beta ([original proposal](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md))
Notes:
* Master Metrics API
* Lives in `k8s.io/metrics` (lives in staging now, will get sycned to github.com/kubernetes/metrics), as well as Heapster
* Provides basic resource usage metrics for Pods and Nodes, only one data point, no sophisticated query language
* Uses:
* HPA for the `Resource` source type,
* `kubectl top`
* scheduler (in future)
* Dashboard
* Need to decide which approach to use when graduating to beta:
* Currently, there is a direct mapping between structure of pod resource requests and resource metrics API
* Alternative is to have arbitrary names like “cpu_usage_average_5m”.
* [piosz] to create issue
* [sross] comment: probably should replace `kind: Pod, metadata.name: <podname>` with `kind: MetricValue, target: <object reference to pod>`.
* Metrics Server
* Minimal server (similar to Heapster), but no storage, limited aggregation, no history, all data in memory
* Scrapes from summary API on Kubelets
* (summary API may be revamped and moved to beta)
* Will be available through aggregator
* [piosz] Design doc coming soon (early next week, hopefully)
* Custom Metrics Prometheus Adapter
* Initial rough form at https://github.com/directxman12/k8s-prometheus-adapter
## 2017-05-11 (Logging)
Agenda:
* Samsung CNCT presenting (non-standard log collection)
* Slides: https://docs.google.com/presentation/d/13LUq6TyaWSZTmYKQKPCxVffX-v3_nRJSKYrbas5Jt4M/edit?usp=sharing
* Owner for ES setup
* Logging vision (@piosz)
* High level vision
* Sources
* Format
## 2017-05-04
Agenda:
* Historical API design for VPA, etc
Notes:
* Kube-state-metrics 0.5 released with new metrics for more resource kinds [https://github.com/kubernetes/kube-state-metrics/releases/tag/v0.5.0](https://github.com/kubernetes/kube-state-metrics/releases/tag/v0.5.0)
*
## 2017-04-27
Agenda:
* ~~Metrics server + master metrics API to beta~~
* [tstclair] Discuss auditing, history, and even offloading…
Notes:
* Plumbing to get auditing and event data out of Kubernetes
* Some of that could be part of infrastore proposal
* System external to Kubernetes
* Need to define an idea of what auditing means
* Needs standard set of APIs that define how that data can be collected
* cAdvisor
* Breaking changes to metrics
* Metrics following wrong format
* Need to discuss with sig-node to overhaul the metrics and move cAdvisor out of /metrics of the kubelet
* [https://github.com/heptio/eventrouter](https://github.com/heptio/eventrouter)
*
## [Cancelled] 2017-04-20 (Logging)
## 2017-04-13
Agenda
* 1.7 Feature Planning Update
* [sross] Update on Custom Metrics API boilerplate
* Kubecon report
* Prometheus Operator Code
Meeting notes:
* 1.7 planning (copied from 2017-03-02)
* Move master metrics API to beta
* Kube-state-metrics stable release [kube-state-metrics#124](https://github.com/kubernetes/kube-state-metrics/issues/124)
* Metrics-server
* Evaluate custom metrics API state
* Implementations for testing server, Hawkular, Prometheus, Stackdriver
* probably won’t move to beta until 1.8 (we want a release where at least two implementations exist)
* discuss/propose historical metrics API
* Updates on Custom Metrics API boilerplate
* PR in progress to switch away from custom patched version of apiserver repository (should make it easier to consume)
* Prometheus Operator code
* Interest in moving towards incubator
* Probably also move towards aggregated API server from TPR
* [sross] can be pinged for some “getting started tips” on making aggregated API servers
## [Cancelled] 2017-04-06 no meetings week
## [Cancelled] 2017-03-30 Kubecon
## 2017-03-23
Agenda
* [Solly] Custom metrics API server building
* Boilerplate repository: [https://github.com/directxman12/custom-metrics-boilerplate](https://github.com/directxman12/custom-metrics-boilerplate)
* Need to implement “pkg/provider”.CustomMetricsProvider and wrap command as in “sample-main.go” and “pkg/sample-cmd”
* You can either vendor or fork the repository
* Due to some issues, currently there’s no vendor directory in the repo (I’m going to try and fix this soon). You’ll need most of the same vendor directories as kubernetes, but with the kubernetes from [https://github.com/directxman12/kubernetes/tree/feature/dynamic-resource-routes](https://github.com/directxman12/kubernetes/tree/feature/dynamic-resource-routes) for the mean time (specifically, the k8s.io/apiserver code there)
* Feel free to ping @directxman12 on Slack with questions
## 2017-03-16 (Logging)
Agenda
* Integration with logging on the node ([kubernetes/kubernetes#42718](https://github.com/kubernetes/kubernetes/issues/42718))
## [Cancelled] 2017-03-09
## 2017-03-02
Agenda:
* Heapster 1.3 release
* Code freeze at Friday, March 10th, 6pm PST
* [sross] to send out email
* kube-state-metrics status
* Has most of important features
* Need to find good balance between adding new metrics and not caching entirety of the API server
* Not rushed to release anything in 1.6
* Plan to release stable version for 1.7
* Need performance test on huge cluster first
* 1.7 planning
* Move master metrics API to beta
* Kube-state-metrics stable release
* Metrics-server
* Evaluate custom metrics API state (start moving towards beta?)
* Implementations for testing server, Hawkular, Prometheus, Stackdriver
* discuss/propose historical metrics API
## 2017-02-23 (Logging)
Agenda:
* Fluent Bit: Intro & status update by Eduardo Silva ([eduardo@treasure-data.com](mailto:eduardo@treasure-data.com)) \
[https://docs.google.com/presentation/d/1Ovbvk5TsOzVy7wLcyJiBonv6EmET36XSba9zvm_l7tM/edit?usp=sharing](https://docs.google.com/presentation/d/1Ovbvk5TsOzVy7wLcyJiBonv6EmET36XSba9zvm_l7tM/edit?usp=sharing)
* releases planed for march 1st and may 17th
questions:
- what happens when to many buffers?
- no limit. feature requested
- buffers can be check pointed to disk for reliability
- can lose messages if log in parse/filter during a bad restart
- metric integration?
- metrics already being written to a file and a webservice for core systems
- going to expand this mechanism to be available for all plugins
- journald as input not supported yet
- feature requested
- memory consumption numbers
- do not have comparison to old fluentd tracking
- approximate number show improvement over fluentd
- only talks to kubernetes API right now
Log rotation issues (by vmik@)
* logrotate acts independently of log aggregators. this can cause log loss. change for 1.6 is to move from logrotate to native docker mechanism for rotating. currently only enabled for GCE. will be a flag and will be shared widely for how others can enable it. Releated PR [#40634](https://github.com/kubernetes/kubernetes/pull/40634)
## 2017-02-16
Agenda:
* [Monasca](https://wiki.openstack.org/wiki/Monasca) demo: Quick architecture overview, demo of running in kubernetes environment by Michael Hoppal ([hoppalm@gmail.com](mailto:hoppalm@gmail.com))
* Announcement: k8s.io/metrics repository (for metrics API type definitions)
* Graduating master metrics API to beta soon (hopefully in Q2) [https://github.com/kubernetes/heapster/blob/master/metrics/apis/metrics/v1alpha1/types.go](https://github.com/kubernetes/heapster/blob/master/metrics/apis/metrics/v1alpha1/types.go) \
[https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metAnnouncement: k8s.io/metrics repository (for metrics API type definitions)rics-api.md](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md)
## [Cancelled] 2017-02-09
## 2017-02-02 (Logging)
Agenda:
* Log rotation problem [#40634](https://github.com/kubernetes/kubernetes/pull/40634), [#38495](https://github.com/kubernetes/kubernetes/issues/38495)
## 2017-01-26
No agenda, meeting was shortened
## 2017-01-19
No agenda, meeting was shortened
## 2017-01-12 (Logging)
Topics:
* Logging to files inside containers, possible solution
* [Proposed solution](https://goo.gl/IgCSjI)
* [PR with the proposed solution](https://github.com/kubernetes/kubernetes.github.io/pull/2141)
Meeting Notes:
* [https://docs.google.com/document/d/1kDnQphHJogwGR6U5oX-tpHM6k3mC49TNnFgQV_uBxSk/edit?usp=sharing](https://docs.google.com/document/d/1kDnQphHJogwGR6U5oX-tpHM6k3mC49TNnFgQV_uBxSk/edit?usp=sharing)
## 2017-01-05 (Monitoring)
Notes:
* Maintenance model of Heapster:
* Rotation for maintainers for responding to issues; exact people TBD
* Sink owners to discuss pull requests in meetings, give ok-to-merge
* Rieman sink potentially deprecated
* Full document specifying to be submitted to Heapster, will be under docs/ directory
* Custom metrics API proposal: [https://github.com/kubernetes/community/pull/152](https://github.com/kubernetes/community/pull/152)
* Instrumentation guidelines: [https://github.com/kubernetes/community/pull/195](https://github.com/kubernetes/community/pull/195)
* What are deprecation rules? Potentially carry new and old metric for one release
* Deprecating builtin-cAdvisor from kubelet
* Kubelet exposes a lot of (all) cAdvisor metrics but only needs a small subset itself
* Moving towards a slimmer API only exposing data cAdvisor needs
* Users will have to run their own cAdvisor for monitoring purposes
* [https://github.com/kubernetes/community/pull/252](https://github.com/kubernetes/community/pull/252)

344
sig-instrumentation/archive/meeting-notes-2018.md Normal file
View File

@ -0,0 +1,344 @@
## Agenda (2018-12-13)
* Metrics overhaul KEP discussion - in person in Seattle at KubeCon
* Discussed what needs to be done, priority and what is already in-flight
* Decided to keep any non-conformant metric labels for v1.14 but clearly state they are deprecated and will be removed in v1.15 (or v1.16 if we get any pushback)
* Add histograms wherever there are summaries
* Make summary metrics opt-in with a kubelet flag
* Not a breaking change, can be done after v1.14 target
* Update KEP status to implementable
* Thanks @ehashman
* Create plan to add dev, operator and user docs to metrics
* I don’t remember all of the context on this, @directmanx12 this was something you brought up, can you fill it in a bit?
* Discussed how to change a single global metrics registry to something that gets passed in and can be replaced with a no-op registry if desired
* This pattern has been implemented in client-go as part of the controller runtime implementation with the logger object
## Agenda (2018-11-29)
* Demo on tracing Sam Naser
* KEP here: [https://github.com/kubernetes/enhancements/pull/650](https://github.com/kubernetes/enhancements/pull/650)
* Next steps:
* create tracing feature proposal
* house mutating webhook for adding trace to an object in kubernetes-sigs
* use annotations for not to not go through an immediate API review
## Agenda 2018-11-15
* [https://github.com/kubernetes/community/pull/2909/](https://github.com/kubernetes/community/pull/2909/)
* Current state of tracing in Kubernetes
* [https://docs.google.com/document/d/1cqdw7JfHSovl1E-FoH4rTpI32Xt0saZvdKv6q6-v4uc/edit?usp=sharing](https://docs.google.com/document/d/1cqdw7JfHSovl1E-FoH4rTpI32Xt0saZvdKv6q6-v4uc/edit?usp=sharing) <- link to public design document
* [https://github.com/Monkeyanator/kubernetes/pulls](https://github.com/Monkeyanator/kubernetes/pulls)
## Agenda 2018-11-1
* Elasticsearch logging addon - @coffeepac
* Additional OWNER
* New image repo
* Metrics overhaul KEP opened and targeted for 1.14
## Agenda 2018-10-18
* Review initial KEP draft: [https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/TMUTDP4cLQw](https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/TMUTDP4cLQw)
* Introduce promtool in order to check for metric best practices
* Open pull request to add KEP to repository
* Bug [https://github.com/kubernetes/kubernetes/issues/68918](https://github.com/kubernetes/kubernetes/issues/68918)
* Introduce heuristic for detecting cardinality explosions in releases
* Community demo: Filebeat hints based autodiscover (exekias / [carlos@elastic.co](mailto:carlos@elastic.co))
* Kube-state-metrics performance optimization update
## Agenda 2018-10-04
* Canceled due to having no agenda points to discuss.
## Agenda 2018-09-06
* Charter merged
* We need to write a KEP (Kubernetes Enhancement Proposal) for metrics overhaul, because it affects lots of users
* Will there be a draft and feedback? - Yes, just like design proposals
* Follow up: setup google doc to flesh out initial proposal for this KEP and start collaborating on it and review it together in the next meeting
* Done: [https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/TMUTDP4cLQw](https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/TMUTDP4cLQw)
* SIG Instrumentation has to use the Kubernetes organizations for now
* Kube-state-metrics performance optimization
* Second PR up for early feedback, refactoring collectors logic to cache metrics instead of Kubernetes objects
[https://github.com/kubernetes/kube-state-metrics/pull/534](https://github.com/kubernetes/kube-state-metrics/pull/534)
* Can there be a docker image be provided with these changes? - Yes, mxinden will provide a personal one
## Agenda 2018-08-23:
* Charter document [https://github.com/kubernetes/community/pull/2266](https://github.com/kubernetes/community/pull/2266)
* Kube-state-metrics performance optimization
* [https://github.com/kubernetes/kube-state-metrics/issues/498](https://github.com/kubernetes/kube-state-metrics/issues/498)
* Kubernetes metrics overhaul
* [https://github.com/kubernetes/kubernetes/pull/67476#issuecomment-413785762](https://github.com/kubernetes/kubernetes/pull/67476#issuecomment-413785762)
* Consider renaming cAdvisor labels [https://github.com/kubernetes/kubernetes/issues/66790](https://github.com/kubernetes/kubernetes/issues/66790)
* General consensus is: yes we should do this at once, probably aiming for 1.13
* We need to figure out whether we need a KEP or feature.
* Researched answer: Asked a couple of people and unanimously was told a KEP would be more appropriate and give this the appropriate visibility.
* [sross] metrics-server status/release prep
* Preparing a new release of a rather major cleanup of metrics-server
* Soon alpha version
* Probably a stable version soon afterwards
* [sross] Moving stuff to kubernetes-sigs
* Can we have our own org?
* Researched answer: Orgs per sig is currently not managable so currently everything goes into kubernetes-sigs.
## Agenda 2018-07-26:
* [Proposed] - Review of [feature idea](https://docs.google.com/document/d/1PjbaImDrSs3qj1oqu46lSChGgJ6ka_N5AuQv0HVkBbI/edit#heading=h.te3fbxigdo0t) - CRD for “Draining” namespaces to a `syslog:// `endpoint
* Charter: [https://github.com/kubernetes/community/pull/2266](https://github.com/kubernetes/community/pull/2266)
* Needs more review
* Sig update in community meeting
* Heapster deprecated
* Deprecation timeline ([https://github.com/kubernetes/heapster/blob/master/docs/deprecation.md](https://github.com/kubernetes/heapster/blob/master/docs/deprecation.md)) -- next step is setup removal in 1.12, completely deprecated as of 1.13
* Node metrics reworking
* Metrics-server refactoring (not yet merged, calling for feedback) - [https://github.com/kubernetes-incubator/metrics-server/pull/65](https://github.com/kubernetes-incubator/metrics-server/pull/65)
* k8s-prometheus-adapter advanced config merged
* A number of third party service involving e2e tests have been put behind a feature flag in the test infrastructure (to improve flaking tests from sig-instrumentation)
## Agenda 2018-06-28:
* Charter: [https://github.com/kubernetes/community/pull/2266](https://github.com/kubernetes/community/pull/2266)
* Needs more review
* Non googlers to push images to gcr.io
* Third party e2e test results: [https://github.com/kubernetes/test-infra/blob/master/docs/contributing-test-results.md](https://github.com/kubernetes/test-infra/blob/master/docs/contributing-test-results.md)
* This is how we will recommend that third party tools submit their test results for inclusion in testgrid
## 2018-06-14:
* Charter: [https://github.com/kubernetes/community/pull/2266](https://github.com/kubernetes/community/pull/2266)
* Needs more review
* How to enforce instrumentation guidelines, when there are existing violations? [https://github.com/kubernetes/kubernetes/pull/64481#discussion_r192527282](https://github.com/kubernetes/kubernetes/pull/64481#discussion_r192527282)
* Do a review of all metrics in a certain release, make public in release notes
* Then introduce stricter workflow for introducing metrics
* No metric stability currently, but we also shouldn’t frustrate users by breaking often
*
* Testing PRs, need review from @piosz
* [https://github.com/kubernetes/test-infra/pull/8451](https://github.com/kubernetes/test-infra/pull/8451)
* [https://github.com/kubernetes/kubernetes/pull/64564](https://github.com/kubernetes/kubernetes/pull/64564)
* None needed for log interface, [already exists](https://github.com/kubernetes/kubernetes/blob/master/test/e2e_node/log_path_test.go).
## 2018-05-31:
* Sig-instrumentation charter
* Testing notes
* Sig-instrumentation breaking e2e owned tests
* [https://docs.google.com/spreadsheets/d/1OirZorG4bbwlEkxAW-2qdp0dXDZrVKtDBDFC0Nq226s/edit?usp=sharing](https://docs.google.com/spreadsheets/d/1OirZorG4bbwlEkxAW-2qdp0dXDZrVKtDBDFC0Nq226s/edit?usp=sharing)
* Check if SIg-node has any logging interface tests, if not write one
* @piosz move the top level testgrid google-gke-stackdriver somewhere else
## 2018-06-14
* How to submit test results as a third party
* Prefer to find sig-testing doc, will try and prepare a minimal sig-inst doc if needed
## 2018-05-31
* Charter PR or doc should be coming tomorrow (6/1)
* Charter defaults align with what we already do
## 2018-05-17
* KubeCon recap
* Medium well attended and lots of good questions
* Very good audience
* Lengthen one session to include a compressed intro and the entire deep dive and not one shorter topic on each
* Energetic custom metric adapter interest from vendors (at least 3 new)
* Public link for videos forthcoming
* Heapster is now deprecated
* Thanks @directxman12
* This is official, feature requests closed
* Make sure this makes it to the v1.11 release notes
* What are the next steps to graduate kube-state-metrics out of alpha
* Action item: @piosz to find current dashboard maintainers and determine what the current state of the dashboard is,
* Historical API, does dashboard want to access data directly
* Sig-instrumentation-kubernetes group
* What is the policy for allowing projects
* Need a charter
* Includes official processes for a sig, structure of sig, etc.
* @brancz to fill out template prior to next meeting ~~@coffeepac to add template to this~~
* [README](https://github.com/kubernetes/community/blob/9565401b5702a3deffb0e5d9f2999e8d12bbc9a2/committee-steering/governance/README.md) for what the process is, includes link to template
* 3rd party/vendor test comments
* What should be marked as ‘e2e’
* @coffeepac to generate list of e2e tests we own, if a reasonable number share a spreadsheet to #sig-instrumentation slack
* How to label 3rd party/vendor tests for viewing
* @coffeepac to write up how to do this
## 2018-04-19
* “Ignoring flakes: sig-instrumentation” [https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/cbbzkMXSMaw](https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/cbbzkMXSMaw)
* If it is not kube code, then we should not have tests on them - Solly
* Given we have one kind of e2e tests we are not fixing in time, we shouldn’t add more (Regarding last meetings discussion) - Frederic
* What is the Kubernetes code being tested here (it looks like “can Stackdriver scrape Kube logs”)? If it’s “can thing X connect to Kubernetes”, then it probably shouldn’t be in Kubernetes e2e tests - Solly
* Can we have a way for external projects to test integrations with Kube? Might want to reach out to SIG testing - Frederic
* @coffeepac to ask sig-instrumentation about what is the desired way to handle 3rd party/vendor integrations for e2e testing
* Prometheus cluster-monitoring addon [https://github.com/kubernetes/kubernetes/pull/62195#issuecomment-382778622](https://github.com/kubernetes/kubernetes/pull/62195#issuecomment-382778622)
* Addons should not belong in the Kubernetes repository - Frederic/Solly
* Cluster-monitoring seems like a lot larger scope than discussed e2e setup from last meeting - Frederic
* Should have gone into a sig-instrumentation specific repo - @coffeepac
* Contrib repo recommends Prometheus Operator - Frederic
* Kubernetes Node Monitoring - Solly
* Draft: [https://docs.google.com/document/d/1_CdNWIjPBqVDMvu82aJICQsSCbh2BR-y9a8uXjQm4TI/edit?usp=sharing](https://docs.google.com/document/d/1_CdNWIjPBqVDMvu82aJICQsSCbh2BR-y9a8uXjQm4TI/edit?usp=sharing)
* Kube-pod-exporter POC demo
## 2018-04-05
* [piosz] kube-up is in a bit of shaky position
* Deprecate InfluxDB kube-up in 1.11, remove in 1.12
* [sross] deprecate Influx e2e tests as well
* [piosz] deploy Prometheus as well
* [sross] it’s not needed for e2e tests, so I’d lean against
* [piosz] want a “real” test for custom metrics, with an actual monitoring solution, Prometheus would be good for that, non-blocking
* [sross] just need to be careful to avoid maintenance issues with Influx in the future
* [brancz] have PoC for pod exporter, blocked on getting crio up with supports for stats endpoint, share it hopefully next meeting
## 2018-03-22
* Aligning cAdvisor labels with official Kubernetes instrumentation guidelines (possibly related to [https://github.com/kubernetes/kubernetes/issues/45043](https://github.com/kubernetes/kubernetes/issues/45043))
* TODO(brancz): Share POC of pod-exporter once CRI implementation with stats endpoints is available
* Further: brancz and directxman12 will take lead on stable metrics for pods in Kubernetes
* Need to figure out pod-level cgroups, other data endpoints (device metrics, etc)
* Road to heapster deprecation/phase out? Should we put a deprecation note at the top of the heapster readme?
* Mark Heapster as being in maintenance mode
* No new features
* No new sinks
* Only bugfixes
* Come up with timeline for deprecation
* No support
* No new bugfixes
* Need better docs on metrics-server setup
* Docs missing?
* Metrics Server Cleanup
* Backport fixes from Heapster (IPV6, etc)
* Remove unneeded code
* Abstract out serving interface to serve resource metrics API from other sources (e.g. directly from monitoring pipeline), implement testing tools, etc
* [directxman12] to publish a bunch of the refactor code
* Proxying counter metrics in Prometheus client
* Pain point of prometheus client library when writing exporters, where counter semantics cannot necessarily applied with available abstractions by the golang Prometheus library
* Interim solution: Implement necessary semantics with “lower level” Prometheus “const” metrics
* Long term: Learn from the interim solution in order to provide re-usable abstraction to Prometheus client-library
## 2018-02-22
* Kubecon sig-instrumentation deep dives sessions
* Best practices for exposing kubelet health checks?
* Probably health checks has to be exposed on different endpoint (not a _/metrics_).
* AI(Solly): Include details in issue [https://github.com/kubernetes/kubernetes/issues/58235](https://github.com/kubernetes/kubernetes/issues/58235)
* Commented on [https://github.com/kubernetes/kubernetes/pull/58827](https://github.com/kubernetes/kubernetes/pull/58827)
* We will need to write our own exporter of metrics
* External Metrics API/HPA changes
* [https://github.com/kubernetes/community/pull/1801](https://github.com/kubernetes/community/pull/1801)
* [https://github.com/kubernetes/community/pull/1802](https://github.com/kubernetes/community/pull/1802)
## 2018-02-08
* Metrics-server cleanup continued - needs to be taken care of
* [https://github.com/kubernetes-incubator/metrics-server/issues/37](https://github.com/kubernetes-incubator/metrics-server/issues/37)
* External Metrics API - a proposal will be written up
* cAdvisor, core/resource metrics and CRI? What’s our stand, everything consumed via CRI? (RE: [https://github.com/kubernetes/kubernetes/issues/55905](https://github.com/kubernetes/kubernetes/issues/55905)) - Solly will revise his proposal and then share
* Log file separation? [https://github.com/kubernetes/kubernetes/issues/58638#issuecomment-359979485](https://github.com/kubernetes/kubernetes/issues/58638#issuecomment-359979485)
* Kubernetes workload benchmarker
* [https://docs.google.com/document/d/1hYOzX8jBHceuXgDVzlasveMqetpKtnq433aNMj1_x0o/edit](https://docs.google.com/document/d/1hYOzX8jBHceuXgDVzlasveMqetpKtnq433aNMj1_x0o/edit)
* [https://github.com/ZJU-SEL/capstan/tree/prometheus](https://github.com/ZJU-SEL/capstan/tree/prometheus)
- Failing e2e test: https://github.com/kubernetes/kubernetes/issues/58837
## 2018-01-25
* Intro and Deep Dive Sessions in Copenhagen
* The road to heapster deprecation?
* State of metrics-server
* Are we intending to keep sinks?
* Cleanups necessary (many heapster things still lurking around)
* PVC stats? [https://github.com/kubernetes/features/issues/497](https://github.com/kubernetes/features/issues/497)
* Prometheus-k8s-adapter
Notes:
* brancz@ is interested in making Intro for KubeCon (and DeepDive as well). Piotr can also prepare something for Intro.
* Heapster deprecation:
* kubectl top switched to metric-server in 1.10.
* Google is need heapster for exporting metrics to Stackdriver. Their team is going to support it.
* We can remove Metrics API from the Heapster. Dashboard may still rely on Model API of heapster.
* Metric-server:
* We don’t want to keep sinks in the codebase
* Need well defined interface between metric-server and kubelet. Summary API is not ideal right now.
* It’s not clear if PVC should be represented as separate entity or as a part of Pod stats.
## 2018-01-11
* 2018 Vision
* Move all sig-instrumentation projects to new home (cluster addons, contrib, standlone apps, etc) - @coffeepac to start planning
* Make build/release of projects be publically viewable/triggerable
* Find out where kubernetes/kubernetes is and start moving sig-inst work to mainline process - @coffeepac to find starting issue
* Historical metrics API - @brancz follow up on VPA design doc to find out involvement needed from sig-instrumentation
* Kubernetes Pod exporter - @brancz share prototype and figure out what the plan of CRI stats is going to be going forward
* kube-state-metrics release

286
sig-instrumentation/archive/meeting-notes-2019.md Normal file
View File

@ -0,0 +1,286 @@
## Agenda (2019-12-12)
* [hase1128]Discuss how to proceed KEP(#1348) \
[https://github.com/kubernetes/enhancements/pull/1348](https://github.com/kubernetes/enhancements/pull/1348) \
And I would like to consult about the following slack comment \
[https://kubernetes.slack.com/archives/C20HH14P7/p1574840290078600](https://kubernetes.slack.com/archives/C20HH14P7/p1574840290078600)
* [serathius] Discuss high level design of Structured logging and collect feedback [https://github.com/kubernetes/enhancements/pull/1367](https://github.com/kubernetes/enhancements/pull/1367)
* [serathius] Ask for contributions to Metrics Server [https://github.com/kubernetes-sigs/metrics-server/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22](https://github.com/kubernetes-sigs/metrics-server/issues?q=is%3Aopen+is%3Aissue+label%3A%22help+wanted%22)
* [Tom Kerkhove] Improve Kubernetes events to be CloudEvents 1.0 compatible \
[https://github.com/kubernetes/kubernetes/issues/85544](https://github.com/kubernetes/kubernetes/issues/85544) \
## Agenda (2019-11-28)
* Cancelled.
## Agenda (2019-11-18) [KubeCon NA]
@Contributor Summit / Technical Discussion Room
Strategy Session Notes
* Goals
* Review Open Keps
* Discuss and set next of goals
* Road map with releases
*
* Open Keps
* [https://github.com/kubernetes/enhancements/pull/650](https://github.com/kubernetes/enhancements/pull/650)
* [https://github.com/kubernetes/enhancements/pull/1343](https://github.com/kubernetes/enhancements/pull/1343) [Metric Stability - Beta]
* [https://github.com/kubernetes/enhancements/pull/1367](https://github.com/kubernetes/enhancements/pull/1367) [Structured Logging]
* Metric Stability Framework
* Provides API for specifying stable metrics
* Beta: must use stability metric framework
* GA: allow for end users to selectively turn off individual metrics
* Future: rules for allowing metrics to graduate
* Allows for continuous cleanup of metrics
* @logicalhan: to file HelpWanted/GoodFirstIssue for this KEP
* Structured Logging
* Original decision was to use glog because it was simple
* Migrate all logging to an API
* Json as format
* Standard metadata enforced via schema
* Migration is a massive job
* Double writing but only one will be used (other is a NoOp), user flag decides which to use
* Reviews wanted
* Timeline:
* KEP reviewed/approved by 1.18 (mid-january 2020)
* Initial implementation in 1.18
* Alpha auto migration framework by end of 2020
* Beta milestone by end of 2020
* GA 2021 at some point
* Using Tracing for Kubernetes Object Lifecycle
* Intend to use OpenTelemetry when it's ready, currently PoC’d in OpenCensus
* [https://github.com/open-telemetry/opentelemetry-go](https://github.com/open-telemetry/opentelemetry-go)
* Changes needed (high level)
* All component boundaries modified to include context for propagation
* This should happen in client-go no matter what
* Requires discussion with how it will interact with structured logging
* Timeline:
* KEP almost approved
* Initial implementation in 1.18
* API graduation:
* Metrics, beta -> GA
* Custom Metrics, beta -> GA
* External Metrics, beta -> GA
* All three require coordination with sig-autoscaling
* Must implement `watch`, this is not supported by any known current monitoring implementation
* Metrics Server graduation
* Default implementation for Metrics API
* Could use developers
* Kube State Metrics
* v2.0
* Could use developers
* Unused tests
* Remove unused e2e tests, these are mostly behind feature flags already
* Moving anything we own in /cluster to bit bucket or kubernetes-sigs org
* @piotr and @brancz to create document as a 2 year planning document with clear goals
* Expected by end meeting on Dec 12th
## Agenda (2019-11-14) [Cancelled]
* Skipping in preparation for KubeCon
## Agenda (2019-10-31)
* [RainbowMango] 1.17 plans again
* I think following tasks should be in 1.17, list them with dependency
* I think we should hide deprecated metrics with stability framework in 1.17
* [https://github.com/kubernetes/kubernetes/pull/83836](https://github.com/kubernetes/kubernetes/pull/83836)
* [https://github.com/kubernetes/kubernetes/pull/83837](https://github.com/kubernetes/kubernetes/pull/83837)
* [https://github.com/kubernetes/kubernetes/pull/83838](https://github.com/kubernetes/kubernetes/pull/83838)
* [https://github.com/kubernetes/kubernetes/pull/83839](https://github.com/kubernetes/kubernetes/pull/83839)
* [https://github.com/kubernetes/kubernetes/pull/83841](https://github.com/kubernetes/kubernetes/pull/83841)
* But they rely on [https://github.com/kubernetes/kubernetes/pull/84135](https://github.com/kubernetes/kubernetes/pull/84135)
* Migration task(as well as remove prometheus reference) almost done except custom collector. And this rely on [https://github.com/kubernetes/kubernetes/pull/83062](https://github.com/kubernetes/kubernetes/pull/83062)
* The last one I think it should be in 1.17 is the flag for kube-binaries
* The first one is: [https://github.com/kubernetes/kubernetes/pull/84292](https://github.com/kubernetes/kubernetes/pull/84292)
* KubeCon
## Agenda (2019-10-17)
* [alejandrox1] quick hello from release team
* [https://github.com/kubernetes/sig-release/tree/master/releases/release-1.17#enhancements-freeze](https://github.com/kubernetes/sig-release/tree/master/releases/release-1.17#enhancements-freeze)
* All enhancements wishing to be included in 1.17 must have
* A KEP in an implementable state
* Including Testing Plans
* Including Graduation Criteria
* An open issue in the 1.17 Milestone
* [piosz] 1.17 plans
* Finish metrics kep - metrics stability implementation (especially wrt hidden metrics) deferred to 1.18
* OOM kill metrics
* AI(brancz, piosz): keep the stuff trackable
* Long-term plans (2020) - discuss at Contributor Summit during KubeCon?
* Han/Elana’s intro to instrumentation talk scheduled at the same time as sig-inst intro session
* Elana to email about fixing the schedule conflict
* Done: SIG Instrumentation intro moved forward to 4:25pm
* Deep dive session - metrics stability discussion
* Tariq to get back to sig-inst on format of Contributor Summit SIG Meet and Greet
## Agenda (2019-10-03)
* Cancelled
## Agenda (2019-09-05)
* Status of 1.16 SIG Instrumentation feature implementation (Han, Frederic)
* Roadmap for 1.17 release
* Remove direct dependency on Prometheus (did I get that right?)
* Improving OOMKill metrics (gauge -> counter?)
## Agenda (2019-07-25)
* [Discuss metric validation/verification KEP](https://github.com/kubernetes/enhancements/pull/1169)
* PR for removing deprecated cadvisor labels is live: [https://github.com/kubernetes/kubernetes/pull/80376](https://github.com/kubernetes/kubernetes/pull/80376)
* Need SIG Node + SIG Testing approval (due to test case update)
## Agenda (2019-07-11)
* Update on feedback from sig-cloud provider (logicalhan) and sig-node (ehashman)
* Sig cloud provider
* They’re refactoring cloud providers out of tree, into separate binaries
* Sig-node
* Informed them of the *_name label removal on cadvisor metrics, hoping to get this feature in for 1.16 (see [https://github.com/kubernetes/kubernetes/pull/69099](https://github.com/kubernetes/kubernetes/pull/69099) for label duplication, landed in 1.14)
* They said they would get back to us at the sig-node meeting next week with approval
* Once we get sig-node’s blessing, we should probably mention this at the next community meeting
## Agenda (2019-06-27)
* [Discuss metrics migration (control-plane stability) KEP](https://github.com/kubernetes/enhancements/pull/1093/)
## Agenda (2019-06-13) ~~(2019-05-30 _deferred until next meeting_~~)
* [https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/XbElxDtww0Y](https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/XbElxDtww0Y)
* [https://github.com/kubernetes/kubernetes/pull/76496](https://github.com/kubernetes/kubernetes/pull/76496)
* Consensus was that we should include component owners as reviewers on KEPs which will affect their binaries and verify that they read and are aware of upcoming changes.
* Initial discussion for metrics migration (control-plane stability)
* Issue for discussion (migration of shared metrics, i.e. client-go), how can we do component based migration (i.e. per metrics endpoint) if we have metrics which are shared between migrated and non-migrated components?
* [Link to draft KEP](https://github.com/kubernetes/enhancements/pull/1093/)
* Initial discussion for metrics conformance (control-plane stability)
* [https://github.com/kubernetes/enhancements/pull/1089](https://github.com/kubernetes/enhancements/pull/1089)
## Agenda (2019-05-02)
* Continued OpenCensus/OpenTracing discussion from last meeting
* Structured Logging ([https://github.com/kubernetes/kubernetes/issues/69825](https://github.com/kubernetes/kubernetes/issues/69825)) & ([https://groups.google.com/forum/#!topic/kubernetes-sig-architecture/wCWiWf3Juzs](https://groups.google.com/forum/#!topic/kubernetes-sig-architecture/wCWiWf3Juzs))
* [https://github.com/go-commons/commons/issues/1](https://github.com/go-commons/commons/issues/1) (standard logger interface discussion after dotGo 2017)
## Agenda (2019-04-18)
* Watch API
* Review [control-plane metric stability KEP](https://github.com/kubernetes/enhancements/pull/946).
* Start discussion around OpenCensus/OpenTracing (especially since [we are introducing some OpenCensus stuff in container-runtime](https://github.com/kubernetes-sigs/controller-runtime/pull/368), [corresponding groups discussion](https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/n0Fq2Dg5Ixs))
## Agenda (2019-04-04)
* Should we standardize success/failure label values? (coming out of: [https://github.com/kubernetes/kubernetes/issues/75839](https://github.com/kubernetes/kubernetes/issues/75839))
* Discuss metrics stability proposal ([https://docs.google.com/document/d/1CcbfC-M8CHDfq1rMAOtW0-LKHvermyUiV6BMXXYiqoM/edit#heading=h.r5x1ipcsw2c8](https://docs.google.com/document/d/1CcbfC-M8CHDfq1rMAOtW0-LKHvermyUiV6BMXXYiqoM/edit#heading=h.r5x1ipcsw2c8))
* Watch API
## Agenda (2019-03-21)
* Mail thread: unbounded metric labels [https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/7wbr6eQ58b0 ](https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/7wbr6eQ58b0)
* Discuss possible modifications to existing kubelet probe metrics
* Metric stability KEP (Han)
* ehashman is giving a talk on Kubernetes monitoring at SREcon next week: [https://www.usenix.org/conference/srecon19americas/presentation/hashman](https://www.usenix.org/conference/srecon19americas/presentation/hashman)
## Agenda (2019-03-07)
* Mail thread: Metric deprecation: [https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/XbElxDtww0Y](https://groups.google.com/forum/#!topic/kubernetes-sig-instrumentation/XbElxDtww0Y)
* Plan to cut Kube-state-metrics v1.6.0
* PR to cut release candidate is out: [https://github.com/kubernetes/kube-state-metrics/pull/702](https://github.com/kubernetes/kube-state-metrics/pull/702)
* Shoutout to [https://github.com/tariq1890](https://github.com/tariq1890) for the awesome work on kube-state-metrics
## Agenda (2019-02-21)
* Cancelled due to no agenda
## Agenda (2019-02-07)
* [Kubelet Resource Metrics Endpoint KEP](https://github.com/kubernetes/enhancements/pull/726) review (dashpole@) [Slides](https://docs.google.com/presentation/d/14zM8S7Ftymo3OabGc208EIjLCXpDheA8yjVV7hWUr2M/edit?usp=sharing)
* Our dev docs need a review [https://github.com/kubernetes/community/issues/3097](https://github.com/kubernetes/community/issues/3097)
* Metrics Overhaul Review
* Outstanding PRs:
* [https://github.com/kubernetes/kubernetes/pull/69099](https://github.com/kubernetes/kubernetes/pull/69099)
* [https://github.com/kubernetes/kubernetes/pull/72470](https://github.com/kubernetes/kubernetes/pull/72470)
* [https://github.com/kubernetes/kubernetes/pull/73366](https://github.com/kubernetes/kubernetes/pull/73366) (this one might need a discussion)
* Fluentd-elasticsearch addon image repository move PR [https://github.com/kubernetes/kubernetes/pull/73819](https://github.com/kubernetes/kubernetes/pull/73819) (@coffeepac)
## Agenda (2019-01-24)
* Cancelled due to no agenda
## Agenda (2019-01-10)
* Demo on Prometheus Adapter replacing Metrics Server for resource metrics
* [Metrics overhaul KEP](https://github.com/kubernetes/enhancements/blob/master/keps/sig-instrumentation/0031-kubernetes-metrics-overhaul.md) review
* Only one that seems to be contentious is: [https://github.com/kubernetes/kubernetes/pull/67476](https://github.com/kubernetes/kubernetes/pull/67476)
* Status on kube-state-metrics
* V1.5.0 stable release PR is out, please review! [https://github.com/kubernetes/kube-state-metrics/pull/629](https://github.com/kubernetes/kube-state-metrics/pull/629)
* Update on Pod Termination Reason Counter discussion with sig-node (Brian)
* [https://github.com/kubernetes/kubernetes/issues/69676#issuecomment-442391695](https://github.com/kubernetes/kubernetes/issues/69676#issuecomment-442391695)