Commit Graph

3 Commits

Author SHA1 Message Date
Athanasios Markou fbf5110f01 notebooks: Extend Notebook Controller to expose idleness for Jupyter (kubeflow/kubeflow#6297)
* notebooks: Update image's tag in make

Modify Makefile to update properly the TAG
based on the git TAG.

Signed-off-by: Athanasios Markou <athamark@arrikto.com>
Reviewed-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Expose last-activity

Extend the notebook-controller to:
* cull idle Notebook Servers based on their new `last-activity`
  annotation
* expose the last activity of each Notebook Server as an annotation
  on the metadata of the corresponding CR object

Modify notebook_controller.go to:
* update the Last Activity of each Notebook Server that has a
  Running pod
* delete the Last Activity Annotation for every Notebook Server
  that does not have a Running pod

Extend culler.go to:
* perform culling based on the new `last-activity` annotation and
  not based on the `/api/status` endpoint.
* update the last activity of a Notebook Server, based on the
  kernels' execution states.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Reviewed-by: Athanasios Markou <athamark@arrikto.com>

* notebooks: Introduce a DEV env var

We introduce a DEV ENV var to allow admins
develop and test on their local machine their
custom Notebook Controller.
We provide information and instructions inside
the components/notebook-controller/README.md.

Signed-off-by: Athanasios Markou <athamark@arrikto.com>
Reviewed-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* notebooks: Add unit tests for last-activity

* Introduce new tests for allKernelsAreIdle()
* Extend the tests for NotebookIsIdle() and for
  NotebookNeedsCulling().

Signed-off-by: Athanasios Markou <athamark@arrikto.com>
Reviewed-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* review: UpdateNotebookLastActivityAnnotation()

Ensure that UpdateNotebookLastActivityAnnotation() does not return
"true". This function should not return any value.

Signed-off-by: Athanasios Markou <athamark@arrikto.com>
2022-02-07 15:19:17 +00:00
Ben Ye 2e7dc7ec06 add culling metrics (kubeflow/kubeflow#4336)
Signed-off-by: yeya24 <yb532204897@gmail.com>
2019-10-17 21:37:57 -07:00
Kimonas Sotirchos 08f43598c2 Culling of Idle Jupyter Notebooks (kubeflow/kubeflow#3856)
* Create a culler as a package

Helper functions for culling resources. Takes for granted that ISTIO is
installed to the system and queries Prometheus to get metrics.
Specifically, requests/{configurable time}.

If the resource should be culled, then it should be done by setting an
annotation. This way the UIs can also show that the Resource is stopping
and also easily stop a resource by making a PATCH request.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Culling logic enhancements

Add necessary ENV Vars. Culling won't happen by default. To enable it
the user will need to set the ENABLE_CULLING=true

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Misc fixes in logging and comment cleanup

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Fix typo

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Add Notebooks specific culling

Query the /api/status endpoint of each Server

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Remove the generic culling logic

We need to discuss if it would make sense to have this logic as a go
library, or use knative.

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Add unit tests

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Remove unused code

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Review changes #1

* rename `getEnvDef` to `getEnvDefault`
* Add a comment to describe how the STOP_ANNOTATION gets used

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>

* Make cluster domain configurable

Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
2019-08-26 04:40:21 -07:00