SDP support for JupyterHub - Phase I #191
Description
As a machine learning engineer, I want to test/asses the viability of using JupyterHub notebooks from within a kubernetes environment. This involves the following:
- how does JuypterHub integrate with Kubernetes?
- authentication/authorization
- resource management
- external resources
is it possible to deploy JupyterHub via an existing Helm chart, or is an operator required?Yes, we use it in demos- what are the possibilities for interacting with kubernetes-managed clusters directly from a notebook environment?
The result of this story will be:
- documentation outlining the findings
- a decision on how to proceed
- optionally (dependent on previous point): implementation steps
Links:
- Build jupyterhub images for use with e.g. spark, to co-ordinate library & package versions #653
- https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html#jupyter-pyspark-notebook
- https://quay.io/repository/jupyter/pyspark-notebook?tab=tags
From the a comment in the decision:
Remarks / Questions / Priorities:
| Demo | Tutorial | Stage 1 tasks | To demonstrate | Effort | Remarks | Priority |
|---|---|---|---|---|---|---|
| 🟢 | 🟢 | Custom images | images that co-ordinate jupyterhub, SDP spark images and python | M | Proposal: we document how to build custom images on top of official spark ones, which do not have jar/python compatibility issues with the notebook images | 1 |
| extend these images to offer a bundle of typical libraries (e.g. pandas, numpy, sklearn, keras) | L | We suggest to deliver one "Basic Data Science Image" (details tbd, effort size M?! ) and extend them only on customer request (as individual agreement) | 12 | |||
| Notebook integration with source control | directly with terminal | S | 6 | |||
| gitsync (required to be in the image) | L | 11 | ||||
| vscode integration | L | 7 | ||||
| 🟢 | Authentication | native authentication | S | (will document only, incl. changes between 3.0.0 and 4.0.0) | 4 | |
| 🟢 | 🟢 | keycloak authentication | M/L | (Only) Keycloak as implementation of GenericOAuthenticator | 5 | |
| 🟢 | 🟢 | Profiles | image selection | S | 9 | |
| 🟢 | storage selection | S | 8 | |||
| 🟢 | 🟢 | resource settings (requests and limits) | S | 10 | ||
| load external resources (via lifecycle hook) | M | 13 | ||||
| 🔴 | 🔴 | Interaction with storage components | HDFS | M/S | (removed: we have this in the jupyterhub-pyspark demo) | 3 |
| 🟢 | 🟢 | S3 | M | 2 |