Support Bundle Control for Troubleshoot.sh

Treva Williams
 | 
Apr 5, 2022

To learn more about Support Bundle Control for Troubleshoot.sh, and how to troubleshoot Kubernetes clusters, please sign up today for our webinar, Troubleshoot Your K8s Cluster, happening April 27th, 2022 at 10:00AM PST.

Sign up to attend here!

What it does

Regardless of your level of experience, whether you're a budding junior developer just starting out, a battle-worn DevOps Engineer who raises app uptime with their mere presence, or if you fall somewhere in between, reading raw JSON isn't what anyone would describe as a good time. Sure, you can just `grep` for a specific keyword or number, but that option is only helpful when you know exactly what you’re looking for. But, that may miss important supplementary information if said info happens to be a line down or up from the keyword you searched for. 

How it works

At its core, a troubleshoot support-bundle is basically like running a few dozen `kubectl` commands in one go, with all of the output saved to compressed files that can be searched instead of having to scroll through terminal output - a Godsend for busy support teams. Believe it or not, our engineers have made troubleshoot.sh even better by creating a supplementary plugin that converts `support-bundle` output from JSON to standard `kubectl` output, saving even more time & flowing seamlessly into standard workflows for Kubernetes admins.

As of right now, only a few commands are supported with more being added every day, which means there’s ample opportunity to become a foundational contributor to what is sure to be an extremely popular open source project. If you’d like to contribute to sbctl, please feel free to open an issue with a short outline of what you would like to add to the project & we’ll help you along. 

Installation

As a lover of anything that makes life easier, it is my pleasure to present a walkthrough on sbctl. Let’s get into it!

If you want to give sbctl a try, head over to the Releases page on GitHub & download the latest version for your operating system. As of now, 64-bit Linux & MacOS is supported but if you have suggestions or requests, feel  free to head on over to the issues page to request support for other operating systems. I just so happen to have a single-node Kubernetes cluster running on Ubuntu, so I’m going to demonstrate installation on that. Both Linux & Mac users should be able to follow along using these commands.

[.pre][email protected]:~$ wget https://github.com/replicatedhq/sbctl/releases/latest/download/sbctl_linux_amd64.tar.gz--2022-03-24 14:47:01-- …2022-03-24 14:47:02 (32.9 MB/s) - ‘sbctl_linux_amd64.tar.gz’ saved [11738600/11738600][.pre]

Once the download completes, unpack the archive using your favorite archiving tool. I’m going with tar because it’s included with Ubuntu:

[.pre][email protected]:~$ tar xvf sbctl_linux_amd64.tar.gz README.mdsbctl[.pre]

That should leave you with two files: the `sbctl` executable & a README with usage instructions. Keep the README handy for later reference, but we’ll need to move `sbctl` to the system `$PATH` for easiest use. If you need help finding it, just run the following command to print the system path to STDOUT:

[.pre][email protected]:~$ echo $PATH/home/trilliams/.krew/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin[.pre]

I’m going to add `sbctl` to `/usr/local/bin` because I don’t want to `sudo`, but you’re free to store in whatever location you prefer. That can be done with the `mv` command:

[.pre][email protected]:~$ mv sbctl /usr/local/bin/[email protected]:~$ which sbctl/usr/local/bin/sbctl[.pre]

Finally, let’s make sure that `sbctl` is running properly by executing the help flag:

[.pre][email protected]:~$ sbctl -hRun commands against a support bundleUsage:sbctl [command]Available Commands:completiongenerate the autocompletion script for the specified shellhelpHelp about any commandserve Start API serverFlags:-h, --help help for sbctlUse "sbctl [command] --help" for more information about a command.[.pre]

Installation is now complete! Now we’re ready to try it out.

Let's use it!

The first thing we’ll need to do is create a `support-bundle`. There are several example `support-bundle` templates available in troubleshoot docs. If you’d like use `sbctl` to do some serious investigating, feel free to create a new YAML definition with some custom Analyzers, Collectors &/or Redactors using the template below:

[.pre]apiVersion: troubleshoot.sh/v1beta2kind: SupportBundlemetadata:name: supportbundle-tutorialspec:collectors: []analyzers: [][.pre]

Or if you just want to play around, use the sample `support-bundle` hosted on GitHub:

[.pre][email protected]:~$ kubectl support-bundle \https://support-bundle.replicated.com[.pre]

Once you’ve generated a `support-bundle`, you’re ready to run the `sbctl` command, which will proxy a connection to the kubernetes API & also print out location of a new kubeconfig file for as long as the connection is open. If you’re unable to open a new terminal window to connect to the instance, you can shift the `sbctl serve` command to the background by appending an `&` to the end of the command, then hit the ENTER key to return to command line as shown below:

[.pre][email protected]:~$ sbctl serve --support-bundle-location=/home/trilliams/support-bundle-2022-03-24T15_12_24 &Server is runningexport KUBECONFIG=/tmp/local-kubeconfig-4178765460[.pre]

Now that `sbctl` proxy has connected, we are now able to pull `support-bundle` details directly from `kubectl`.

Sbctl

Pods.json w/o sbctl

[.pre][email protected]:~$ kubectl get po -n schemahero-system -o wideNAME READY STATUSRESTARTSAGEIP NODE NOMINATED NODE READINESS GATESschemahero-0 1/1 Running 4 (12h ago) 6d8h 10.32.0.25 replicated <none><none>[email protected]:~$ kubectl get po -n schemahero-system -o wideNAME READY STATUSRESTARTSAGEIP NODE NOMINATED NODE READINESS GATESschemahero-0 1/1 Running 4 (11h ago) 6d8h ***HIDDEN*** replicated <none><none>[.pre]

If you’d like to see more, have a feature request, or just want to connect with fellow Troublemakers, subscribe to the Replicated Community Calendar for invites to community meetings & events for troubleshoot.sh & SchemaHero.

TL;dr - support-bundle is the equivalent of running a couple dozen kubectl commands at once & saving to file. Support bundle control makes getting to that output way easier by reprinting the support bundle contents as kubectl output

Output bank

w/o sbctl

[.pre][email protected]:~$ kubectl get nsNAMESTATUS AGEdefault Active 5d22hkube-node-lease Active 5d22hkube-public Active 5d22hkube-system Active 5d22hkurlActive 5d22hlonghorn-system Active 5d22hminio Active 5d22hmonitoringActive 5d22hprojectcontourActive 5d22hschemahero-system Active [email protected]:~$ kubectl get pods -n kube-systemNAME READY STATUSRESTARTS AGEcoredns-64897985d-7qj481/1 Running 5 (39m ago)5d22hcoredns-64897985d-d4vjl1/1 Running 5 (39m ago)5d22hetcd-replicated1/1 Running 5 (39m ago)5d22hkube-apiserver-replicated1/1 Running 5 (39m ago)5d22hkube-controller-manager-replicated 1/1 Running 5 (39m ago)5d22hkube-proxy-p5t6n 1/1 Running 5 (39m ago)5d22hkube-scheduler-replicated1/1 Running 5 (39m ago)5d22hweave-net-6m7wf2/2 Running 16 (38m ago) [email protected]:~$ kubectl describe pod --namespace=schemahero-system schemahero-0Name: schemahero-0Namespace:schemahero-systemPriority: 0Node: replicated/192.168.1.127Start Time: Fri, 18 Mar 2022 18:00:33 +0000Labels: control-plane=schemaherocontroller-revision-hash=schemahero-77bcc67945statefulset.kubernetes.io/pod-name=schemahero-0Annotations:<none>Status: RunningIP: 10.32.0.25IPs:IP: 10.32.0.25Controlled By:StatefulSet/schemaheroContainers:manager:Container ID:containerd://3592b77cb960970ab6f70c51ce4274085aadf738be32e01731144dfe67a86c8fImage: schemahero/schemahero-manager:0.12.7Image ID:docker.io/schemahero/[email protected]:9516f0ee8b90ec53c9d61e5cef007b3bb49da60f771d5031052e83ea380a9165Port:9876/TCPHost Port: 0/TCPCommand:/managerrun--enable-database-controllerState:RunningStarted:Thu, 24 Mar 2022 14:37:34 +0000Last State: TerminatedReason: UnknownExit Code:255Started:Tue, 22 Mar 2022 13:29:30 +0000Finished: Thu, 24 Mar 2022 14:36:48 +0000Ready:TrueRestart Count:4Limits:cpu: 1memory:150MiRequests:cpu: 100mmemory:50MiEnvironment:POD_NAMESPACE:schemahero-system (v1:metadata.namespace)SECRET_NAME:webhook-server-secretMounts:/tmp/cert from cert (ro)/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b54q4 (ro)Conditions:TypeStatusInitialized True Ready True ContainersReady True PodScheduledTrue Volumes:cert:Type:Secret (a volume populated by a Secret)SecretName:webhook-server-secretOptional:falsekube-api-access-b54q4:Type:Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds:3607ConfigMapName: kube-root-ca.crtConfigMapOptional: <nil>DownwardAPI: trueQoS Class: BurstableNode-Selectors:<none>Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300sEvents:Type ReasonAgeFrom Message---- ---------- ---- -------WarningFailedMount 44m (x2 over 44m)kubeletMountVolume.SetUp failed for volume "cert" : failed to sync secret cache: timed out waiting for the conditionNormal SandboxChanged43m (x3 over 44m)kubeletPod sandbox changed, it will be killed and re-created.Normal Pulling 43mkubeletPulling image "schemahero/schemahero-manager:0.12.7"Normal Pulled43mkubeletSuccessfully pulled image "schemahero/schemahero-manager:0.12.7" in 638.523822msNormal Created 43mkubeletCreated container managerNormal Started 43mkubeletStarted container [email protected]:~$ [.pre]

With sbctl

[.pre][email protected]:~$ kubectl describe pod --namespace=schemahero-system schemahero-0Name: schemahero-0Namespace:schemahero-systemPriority: 0Node: replicated/***HIDDEN***Start Time: Fri, 18 Mar 2022 18:00:33 +0000Labels: control-plane=schemaherocontroller-revision-hash=schemahero-77bcc67945statefulset.kubernetes.io/pod-name=schemahero-0Annotations:<none>Status: RunningIP: ***HIDDEN***IPs:IP: ***HIDDEN***Controlled By:StatefulSet/schemaheroContainers:manager:Container ID:containerd://3592b77cb960970ab6f70c51ce4274085aadf738be32e01731144dfe67a86c8fImage: schemahero/schemahero-manager:0.12.7Image ID:docker.io/schemahero/[email protected]:9516f0ee8b90ec53c9d61e5cef007b3bb49da60f771d5031052e83ea380a9165Port:9876/TCPHost Port: 0/TCPCommand:/managerrun--enable-database-controllerState:RunningStarted:Thu, 24 Mar 2022 14:37:34 +0000Last State: TerminatedReason: UnknownExit Code:255Started:Tue, 22 Mar 2022 13:29:30 +0000Finished: Thu, 24 Mar 2022 14:36:48 +0000Ready:TrueRestart Count:4Limits:cpu: 1memory:150MiRequests:cpu: 100mmemory:50MiEnvironment:POD_NAMESPACE:schemahero-system (v1:metadata.namespace)SECRET_NAME:webhook-server-secretMounts:/tmp/cert from cert (ro)/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b54q4 (ro)Conditions:TypeStatusInitialized True Ready True ContainersReady True PodScheduledTrue Volumes:cert:Type:Secret (a volume populated by a Secret)SecretName:webhook-server-secretOptional:falsekube-api-access-b54q4:Type:Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds:3607ConfigMapName: kube-root-ca.crtConfigMapOptional: <nil>DownwardAPI: trueQoS Class: BurstableNode-Selectors:<none>Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300sEvents:Type ReasonAgeFrom Message---- ---------- ---- -------WarningFailedMount 56m (x2 over 56m)kubeletMountVolume.SetUp failed for volume "cert" : failed to sync secret cache: timed out waiting for the conditionNormal SandboxChanged55m (x3 over 55m)kubeletPod sandbox changed, it will be killed and re-created.Normal Pulling 55mkubeletPulling image "schemahero/schemahero-manager:0.12.7"Normal Pulled55mkubeletSuccessfully pulled image "schemahero/schemahero-manager:0.12.7" in 638.523822msNormal Created 55mkubeletCreated container managerNormal Started 55mkubeletStarted container manager[.pre]