939e1e22ca
* build: Update components makefiles for building
We'll create a top-level Makefile under components/ dir
that has the following rules:
* build-all:
To build all images locally
* push-all:
* We can use a specific REGISTRY and retag the images
* Push all the images
This top-level Makefile will run the sub-Makefiles that every
component has for building and pushing the images.
We modified every sub-Makefile as follows:
* We don't use a registry in images by default
* Removed unused rules and vars
* Use the --dirty flag of git describe in TAG
--dirty[=<mark>]
Describe the working tree. It means describe HEAD and
appends <mark> (-dirty by default) if the working tree
is dirty.
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
* build: Create makefiles for notebook servers
The common starting point of building the notebook-server
images are the following 4 *base* images:
- base
- codeserver
- jupyter
- rstudio
All other server images need to build on top of them. We'll
dynamically pass the base images in every Dockerfile by
using an ARG IMG. We can set the value of this ARG during
docker build with the --build-arg CLI argument.
This way we build both the base images with a tag locally,
and then we pass that image as arg via the Makefile and build the rest
So we modified our building procedure as follows:
1. Build the base image since everything starts from there
2. Pass the base image as an ARG in the Dockerfiles of
jupyter, codeserver, rstudio images and build on top
3. Pass the base images in all other server images and build
on top
For that we will:
1. Create a Makefile for each of the notebook servers, in each folder
a. Each makefile will be responsible for building the bases and use args for passing them on
2. Use the central Makefile to call each Makefile from above
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
* cherry-pick: Notebook server upstream fixes
Relevant upstream PR: https://github.com/kubeflow/kubeflow/pull/6466/files
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
* build: Fix Makefiles
- Remove build-gcp and build-gcr rules as we don't use them anywhere in
the project
- Fix code conficts
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
* build: Fix Dockerfiles for notbook-server images
We remove the previous logic of using already built images as bases.
The users must use only the Makefiles to pass the appropriate BASE_IMG
and build the images correctly.
Thus, we have Makefiles everywhere that:
- Can build any base image
- If an image requires another notebook base, then we first build that one using its makefile,
and then use it as docker ARG for building the next one
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
* build: Fix notebook-controller Makefile
Removed a misplaced "|" char that breaks the Makefile
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
* build: Update GH action workflows
* Update workflow for notebook-server images:
- Add a step for building all images by using the
central-Makefile under components/example-notebook-servers/ dir.
- Add a step for pushing all images by using the
central-Makefile under components/example-notebook-servers/ dir.
* Update workflow for all Kubeflow images:
- Add a step for building & pushing all images by using the
top-level Makefile under components/ dir.
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
* build: Remove completely ECR references from images
Replace everywhere the "public.ecr.aws/j1r0q0g6/notebooks/notebook-servers"
prefix with "kubeflownotebookswg"
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
* build: Fix GH actions for Kubeflow components
Fix GH actions to use the updated make rules
when building the Kubeflow component images.
Remove the "docker.io" prefix when building with
GH action workflows
Signed-off-by: Apotolos Gerakaris <apoger@arrikto.com>
|
||
|---|---|---|
| .. | ||
| base | ||
| codeserver | ||
| codeserver-python | ||
| jupyter | ||
| jupyter-pytorch | ||
| jupyter-pytorch-full | ||
| jupyter-scipy | ||
| jupyter-tensorflow | ||
| jupyter-tensorflow-full | ||
| rstudio | ||
| rstudio-tidyverse | ||
| Makefile | ||
| OWNERS | ||
| README.md | ||
| image-flow-chart.png | ||
README.md
Example Notebook Servers
🛑️️ Images are provided as examples and are supported on a best-effort basis. Contributions are greatly appreciated.
Images
Images // Base
These images provide a common starting point for Kubeflow Notebook containers. See the custom images guide to learn how to extend them with your own packages.
| Dockerfile | Registry | Notes |
|---|---|---|
| ./base | kubeflownotebookswg/base:{TAG} |
common base image |
| ./codeserver | kubeflownotebookswg/codeserver:{TAG} |
base code-server (Visual Studio Code) image |
| ./jupyter | kubeflownotebookswg/jupyter:{TAG} |
base JupyterLab image |
| ./rstudio | kubeflownotebookswg/rstudio:{TAG} |
base RStudio image |
Images // Full
These images extend the base images with common packages used in the real world.
Images // Relationship Chart
This chart shows how the images are related to each other.
Images // Important Information
- make your custom images by extending one of the base images
- images run as the
jovyanuser - images use the s6-overlay init system to manage processes
Custom Images
Packages installed by users after spawning a Kubeflow Notebook will only last the lifetime of the pod (unless installed into a PVC-backed directory). To ensure packages are preserved throughout Pod restarts users will need to either:
- Build custom images that include them, or
- Ensure they are installed in a PVC-backed directory
Custom Images // Python Packages
⚠️ a common cause of errors is users running
pip install --user ..., causing the home-directory (which is backed by a PVC) to contain a different or incompatible version of a package contained in/opt/conda/...
Extend one of the base images and install any pip or conda packages your Kubeflow Notebook users are likely to need.
As a guide, look at jupyter-pytorch-full.cpu for a pip install ... example, and the rstudio-tidyverse for conda install ....
Custom Images // Linux Packages
⚠️ ensure you swap to
rootin the Dockerfile before runningapt-get, and swap back tojovyanafter.
Extend one of the base images and install any apt-get packages your Kubeflow Notebook users are likely to need.
Custom Images // S6
Some use-cases might require custom scripts to run during the startup of the Notebook Server container, or advanced users might want to add additional services that run inside the container (for example, an Apache or NGINX web server). To make this easy, we use the s6-overlay.
The s6-overlay differs from other init systems like tini.
While tini was created to handle a single process running in a container as PID 1, the s6-overlay is built to manage multiple processes and allows the creator of the image to determine which process failures should silently restart, and which should cause the container to exit.
Custom Images // S6 // Scripts
Scripts that need to run during the startup of the container can be placed in /etc/cont-init.d/, and are executed in ascending alphanumeric order.
An example of a startup script can be found in ./rstudio/s6/cont-init.d/02-rstudio-env-fix.
This script uses the with-contenv helper so that environment variables (passed to container) are available in the script.
The purpose of this script is to snapshot any KUBERNETES_* environment variables into the Renviron.site at pod startup, as without these variables kubectl does not work.
Custom Images // S6 // Services
Extra services to be monitored by s6-overlay should be placed in their own folder under /etc/services.d/ containing a script called run and optionally a finishing script finish.
For more information about the run and finish scripts, please see the s6-overlay documentation.
An example of a service can be found in the run script of jupyter/s6/services.d/jupyterlab which is used to start JupyterLab itself.
Custom Images // S6 // Run As Root
⚠️ our example images run
s6-overlayas$NB_USER(notroot), meaning any files or scripts related tos6-overlaymust be owned by the$NB_USERuser to successfully run
There may be cases when you need to run a service as root, to do this, you can change the Dockerfile to have USER root at the end, and then use s6-setuidgid to run the user-facing services as $NB_USER.
Troubleshooting
Troubleshooting // Jupyter
Kernel stuck in connecting state:
This is a problem that occurs from time to time and is not a Kubeflow problem, but rather a browser. It can be identified by looking in the browser error console, which will show errors regarding the websocket not connecting. To solve the problem, please restart your browser or try using a different browser.