Working around Kubeflow .5.1 issue with Kubenetes 1.15
So, this morning I encountered a previously reported issue where the latest release of Kubeflow does not install onto the latest release of Kubernetes. :-( . In this article, I'm going to see what the process of reverting back to a previous version of Kubernetes is in order to make this work.
I'm hoping that we might be able to get by with just a downgrade of the client first. This article shows me how to get a list of all the previous packages. This article gives me the syntax for downgrading the packages to those versions. And this is the article I followed to install the packages in the first place, so I'll try downgrading all the same packages... basically anything that has a 1.15 version here I'll pull back to the latest 1.14.
Kubelet goes first and seems to work fine. Kubeadm goes next - works just as well. Kubectl, too. Kubernetes-cni does not have a version at 1.15, so let's try just this much on just the client, for starters.
It still shows the 1.15 libraries, so... thinking maybe a reboot is in order.
A reboot doesn't work, so I remove the entire "kubeflow" directory and regenerate it. That doesn't work, either. So, I look around for the libraries referenced in the error and find them in the kubeflow directory. This gives me an idea - there is an app.yaml file that gets generated fairly early on in the process... it has the Kubernetes version in it. All I can guess is that it is getting this from the cluster rather than the client. Now, I could downgrade my cluster but, I'm thinking - what if I went back to the start of the Kubeflow installation process, swapped that version back to 1.14.3 as soon as the file is generated, and then proceed.
rm -rf kubeflow
kfctl init ${KFAPP}
cd ${KFAPP}
Oooh -- there it is! I pop it open in vi and... oops -- the versions aren't set yet... I must have to go a little further. I go on to the next step and... it is 1.15 and the library directory has downloaded that version of libraries. It must be taking it from the server... so, I'll have to try downgrading that. :-(
Kubelet, kubeadm, kubectl and then a reboot. I can't imagine this working without a reboot. As soon as it restarts, I will rerun the "kubectl version" command on my node and hope that the server version has changed. If it has, I will go on to regenerate the kubeflow install scripts.
The kubectl command shows it on exactly the same server version. So, I will try removing and rejoining the node.
This article explains how to remove the node, so I follow it.
I think go to my node and uninstall kubeadm, kubelet, and kubectl using apt-get remove, and then reinstall the 1.14.3-00 versions of all these using apt-get again. I was hoping that "kubectl version" would be broken at this point, but it still claims to be connected and to see a 1.15 version server. So, I will just proceed with trying to re-join.
Re-joining didn't work. So, at long last, I wind up completely rebuilding my server and rebuilding my client. At this point, I discover something charming about "kubeadm join", which is that if you mistype a single character of the join token, it will simply hang -- no error... just hanging. :-(
Finally I get everything running at level 1.14.3 and retry the kfctl stuff. And, yes -- this works. :-/. So, sometimes you lose time trying to save time. At this point, I'm just happy to have the YML I need! :-)
Comments