Installing Kubeflow .4

So, since last we spoke, Kubeflow has released 0.4.1 and I have swapped out the old "trader desktop" from 2008 in my k8s cluster for a new "gamer PC". The new PC has a GPU, so I am hoping for some big improvements in training times.

I was told that upgrading from .3 to .4 was an unsupported path, so I have gone "whole hog" on this and completely re-installed K8S, to allow for a clean install of Kubeflow, following the latest instructions available here.

So, first -- I changed things around in my cluster a bit since last time. I had been running my K8S master on my laptop last time, which was an odd choice, given that my laptop is one of the most powerful computers in my cluster AND frequently disconnected. This time, I switched it to a Mac Mini from 2011 running Ubuntu. It'll never be K8S' choice to perform useful work and is always sitting at my home turned on, so perfect choice to play master.

What this means, though, is that I have to start with pre-requisites again. I already set up my cluster, so have all the K8S tools, but I need ksonnet. So, I download the linux binary from here, unzip it, untar it and copy the binary from the tar ("ks") to /usr/sbin.

The instructions for downloading kfctl.sh, which is basically the Kubeflow installer, imply that you need to set an environment variable indicating where Kubeflow will live first. So, I do a simple "KUBEFLOW_src=/home/derek/kubeflow" and proceed with the instructions as written on the above setup page.

I now realize that it also wants me to assign a value for $KFAPP, which is going to be the Kubernetes namespace where all my Kubeflow processes will run. So, I do a simple "KFAPP=kubeflow41" and continue. I figure appending the version number will help me set up another side-by-side newer instance if needed later.

I get as far as "${KUBEFLOW_SRC}/scripts/kfctl.sh apply k8s" and it runs for several minutes before blowing up with ERROR handle object: patching object from cluster: merging object with existing state: unable to recognize "/tmp/ksonnet-mergepatch905621021": no matches for kind "Application" in version "app.k8s.io/v1beta1". Some Googling indicates that I'm seeing the error here, which is supposed to rectify itself if I reapply the ksonnet. I assume the simplest way to do this is to simply rerun the "kfctl.sh apply k8s" command - so I do that. And yes, it zooms right through to completion in less than a minute! :-)

---

So, let's go ahead and try getting into the Kubeflow UIs using the instructions here. It begins by saying that we need to set up a proxy and the namespace will be required as a part of this, so I run "kubectl get namespaces" and see several new ones created for Kubeflow.

Let's try "kubeflow" - it is the oldest and most basic, and is also the one shown in the example. But now I realize something I hadn't realized before I reorganized my cluster. The method for getting access is to do a port-forwarding using kubectl. This will expose a service's port locally on the box where I run the port forwarding operation. I'm used to running this on my master node, but my master node now has no UI desktop - so I need to run it from a node.

After about 30 minutes of futzing, I have forwarded the port and connected to the Kubeflow UI. The main complexity was that - somehow or another - when I created the node for my new "gaming PC" I didn't set up the "admin.conf" for kubectl as I had expected. So, I had to catch up with that by copying it from the computer where I *had* set this up. Now, I'm all set with access to the UI. Given my issues running Jupyter when I first installed Kubeflow .31, I will save that effort for my next Blog post. For now -- all seems well!