Kubeflow "Unboxing" - Part 6 - Jupyter Notebooks for Real!

So, in Part 3 we made an unsuccessful attempt to get to the Jupyter Notebooks. We were thwarted by a bug in Kubeflow 0.2.x that didn't expose web sockets for the Ambassador pod that gives us access to the Jupyter server. Now we are running 0.3.4 everything should work - but going back and retrying the steps in #3 flagged up an important difference. Our pods are now in the "kubeflow" namespace, not default, so you need to start by proxying your connection exactly as shown at https://www.kubeflow.org/docs/guides/accessing-uis/

export NAMESPACE=kubeflow

kubectl port-forward -n ${NAMESPACE} `kubectl get pods -n ${NAMESPACE} --selector=service=ambassador -o jsonpath='{.items[0].metadata.name}'` 8080:80

After this, you can open a browser to localhost:8080 and proceed with the rest of the steps shown in Part 3 of this series - except, note that you won't be prompted to login until you click JupyterHub.

Also, after logging in, it now only prompts for the image - unless you click "Advanced". You have to type a character to get the drop down list of choices. Choose the second from the bottom - even though you'll find it is now TF 1.10.1 instead of 1.8.

Spawning a notebook now spins and spins and spins... why is this? Well, running kubectl describe pods jupyter-XXXXX where XXXXX is the name with which I logged in gives a clue. It says that the pod has inbound persistent volume claims that haven't been met. A little Googling on this indicates that this means that we don't have a default storage class set up. Running kubectl get sc --all-namespaces clarifies that, in fact, we have NO storage classes configured. :-( .

Creating a default storage class didn't resolve the issue. It appears to want a genuine Persistent Volume. But those appear to only be available for fancier systems that I have -- public cloud, NFS, etc. I need to learn more about this before I can proceed.

---

So, 16 hours later, I have made my first bit of progress. At lunch, it occurred to me that I had changed my login in Jupyter between .2 and .3, and my previous login was an actual linux username, so I thought *maybe* that was the key. No - that was a silly idea, and I really didn't give it much chance.

As soon as I got home, I tried kubectl get pvc --all-namespaces and, at last, a clue!

So, the "sc.yml" file above is what I ran last night to create a default storage class. Note, the name of that is now in the "claim-derek", which was created when I tried to spin up a Jupyter notebook using my actual Linux login. The name of my login doesn't matter, but it appears that having a default StorageClass influences the parameters of the required storage, as shown by the other columns in this table. My theory now is that if I can define a Persistent Volume that provides storage matching "local-storage" and - probably - try a new login (since my previous claim, which I deleted so can't show you, had a blank storageclass - so clearly changes don't take effect except between login attempts) - then it should work.

So, a little Googling lead me to this article https://kubernetes.io/blog/2018/04/13/local-persistent-volumes-beta/, which seems to suggest that I can reserve a directory on any of my nodes to be my persistent volume and that will be what gets used when a claim for local-storage is issued. I'll have to look to see where I have the most disk space.

And this approach finally yielded success! I simply went to a server called "mini" and created a directory called /k8sstorage. Then I sent a yml to setup a persistant volume there.

After this, I went back to the JupyterHub UI and logged out and back in as a new user called "mutant" :-) Running kubectl get pvc --all-namespaces then showed a row for my new user's claim bound to sample-local-pv (as named in the image above). I chose TensorFlow 10 and it took a long time to load. During the load, running kubectl describe pods jupyter-mutant now shows...

After about 15 minutes, it loaded into my browser window!!

Here it is with some sample code loaded and ready to run!