Commit Graph

1 Commits

Author SHA1 Message Date
apoger d2c76a0739 Implement a culling controller for Notebooks (kubeflow/kubeflow#6807)
* Implement a culling controller for notebooks

Changes:

 * Move the idleness/culling logic into a separate controller
   as part of the Notebooks Controller/Operator.

 * Introduce an "notebooks.kubeflow.org/last_activity_check_timestamp".
   annotation in each Notebook CR to keep the timestamp of the last
   performed idleness check

The controller can then compare this timestamp with the current time to
ensure that notebooks will get reconciled every IDLENESS_CHECK_PERIOD
minutes.

The culling-controller will:

* reconcile only notebooks CRs
* set/update culling annotations
  - 'notebooks.kubeflow.org/last_activity'
  - 'notebooks.kubeflow.org/last_activity_check_timestamp'
* perform idleness checks every 'IDLENESS_CHECK_PERIOD' minutes
  and set the 'kubeflow-resource-stopped' annotation, if a notebook
  needs to be culled.

Refs: kubeflow/kubeflow#6767

Signed-off-by: Apostolos Gerakaris <apoger@arrikto.com>

* review: Remove culling annotations when Pod is not found

Signed-off-by: Apostolos Gerakaris <apoger@arrikto.com>

* review: Improve logs

Add a log message at the beginning of the reconciliation loop
to make it clear that a Reconcile was called for a notebook.

Signed-off-by: Apostolos Gerakaris <apoger@arrikto.com>

* Run the controller locally

* Introduce make rule for running the controller locally with
  culling enabled

* Introduce a dev_culling_authorization_policy which must be
  applied when testing the culling-controller locally

Signed-off-by: Apostolos Gerakaris <apoger@arrikto.com>

* Update README instructions

Signed-off-by: Apostolos Gerakaris <apoger@arrikto.com>

Signed-off-by: Apostolos Gerakaris <apoger@arrikto.com>
2023-01-26 13:32:10 +00:00