Blockbridge provides a Container Storage Interface (CSI) driver to deliver persistent, secure, multi-tenant, cluster-accessible storage for Kubernetes. This guide describes how to deploy Blockbridge as the storage backend for Kubernetes containers.
If you’ve configured other Kubernetes storage drivers before, you may want to start with the Quickstart section. The rest of the guide has detailed information about features, driver configuration options and troubleshooting.
REQUIREMENTS & VERSIONS
Supported Versions
Blockbridge supports Kubernetes version 1.14+.
Component | Version |
---|---|
Blockbridge | 5.1 |
Blockbridge K8s Driver | 2.0.0 |
Kubernetes | 1.14+ |
CSI (container storage) Specification | 1.0.0 |
Supported Features
- Dynamic volume provisioning
- Automatic volume failover and mobility across the cluster
- Integration with RBAC for multi-namespace, multi-tenant deployments
- Quality of Service
- Encryption in-flight for control (always) and data (optional)
- Encryption at rest
- Multiple Storage Classes provide programmable, deterministic storage characteristics
Supported K8s Environments
- Rancher 2.4+
- Mirantis Kubernetes Engine 3.1+ (formerly Docker EE)
Requirements
The following minimum requirements must be met to use the Blockbridge driver in Kubernetes:
- Kubernetes 1.14+.
--allow-privileged
flag must be set to true for the Kubernetes API server.- MountPropagation must be enabled (default to true since version 1.10).
- If you use Docker, the Docker daemon of the cluster nodes must allow shared mounts.
-
Ensure the host running the kubelet has iSCSI client support installed and iscsid started on the host/node.
For CentOS/RHEL, install the iscsi-initiator-utils package:
yum install iscsi-initiator-utils systemctl systemctl enable --now iscsid
For Ubuntu, install the open-iscsi package:
apt install open-iscsi systemctl systemctl enable --now iscsid
See CSI Deploying for more information.
QUICKSTART
This is a brief guide on how to install and configure the Blockbridge Kubernetes driver. In this section, you will:
- Create a Blockbridge account for your Kubernetes storage.
- Create an authentication token for the Kubernetes driver.
- Define a secret in Kubernetes with the token and the Blockbridge API host.
- Deploy the Kubernetes driver.
Many of these topics have more information available by selecting the information ⓘ links next to items where they appear.
Blockbridge Configuration
These steps use the containerized Blockbridge CLI utility to create an account and an authorization token.
-
Set
BLOCKBRIDGE_API_HOST
to point to your Blockbridge API endpoint. ⓘ$ export BLOCKBRIDGE_API_HOST=blockbridge.mycompany.example
-
Use the containerized CLI to create the account. ⓘ
$ docker run --rm -it -e BLOCKBRIDGE_API_HOST docker.io/blockbridge/cli:latest-alpine bb --no-ssl-verify-peer account create --name kubernetes
-
When prompted, authenticate as the
system
user. ⓘAuthenticating to https://blockbridge.mycompany.example/api Enter user or access token: system Password for system: .... Authenticated; token expires in 3599 seconds.
-
Use the containerized CLI to create the auth token. ⓘ
$ export BLOCKBRIDGE_API_HOST=blockbridge.mycompany.example $ export BLOCKBRIDGE_API_SU=kubernetes $ docker run --rm -it -e BLOCKBRIDGE_API_HOST -e BLOCKBRIDGE_API_SU docker.io/blockbridge/cli:latest-alpine bb --no-ssl-verify-peer authorization create --notes 'csi-blockbridge driver access'
-
Again, authenticate as the
system
user. ⓘAuthenticating to https://blockbridge.mycompany.example/api Enter user or access token: system Password for system: Authenticated; token expires in 3599 seconds.
-
Set the
BLOCKBRIDGE_API_KEY
environment variable to the new token. ⓘ$ export BLOCKBRIDGE_API_KEY="1/Nr7qLedL/P0KXxbrB8+jpfrFPBrNi3X+8H9BBwyOYg/mvOot50v2vA"
Kubernetes Configuration
The following steps install and configure the Blockbridge Kubernetes driver on your cluster. Your session must already be authenticated with your Kubernetes cluster to proceed.
-
Create a file with the definition of a secret for the Blockbridge API. ⓘ
$ cat > secret.yml <<- EOF apiVersion: v1 kind: Secret metadata: name: blockbridge namespace: kube-system stringData: api-url: "https://${BLOCKBRIDGE_API_HOST}/api" access-token: "$BLOCKBRIDGE_API_KEY" ssl-verify-peer: "false" EOF
-
Create the secret in Kubernetes. ⓘ
$ kubectl create -f ./secret.yml
-
Check that the secret exists. ⓘ
$ kubectl -n kube-system get secrets blockbridge NAME TYPE DATA AGE blockbridge Opaque 3 2m
-
Deploy the Blockbridge driver. ⓘ
$ kubectl apply -f https://get.blockbridge.com/kubernetes/5.1/csi/csi-blockbridge-v2.0.0.yaml
-
Check that the driver is running. ⓘ
$ kubectl -n kube-system get pods -l role=csi-blockbridge NAME READY STATUS RESTARTS AGE csi-blockbridge-controller-0 3/3 Running 0 6s csi-blockbridge-node-4679b 2/2 Running 0 5s
CONFIGURATION & DEPLOYMENT
This section discusses how to configure the Blockbridge Kubernetes driver in detail.
Linked Blockbridge Account
The Blockbridge driver creates and maintains its storage under a tenant account on your Blockbridge installation.
The driver is configured with two pieces of information: the API endpoint and the authentication token.
configuration | description |
---|---|
BLOCKBRIDGE_API_URL | Blockbridge controlplane API endpoint URL specified as https://hostname.example/api |
BLOCKBRIDGE_API_KEY | Blockbridge controlplane access token |
The API endpoint is specified as a URL pointing to the Blockbridge controlplane’s API. The access token authenticates the driver with the Blockbridge controlplane, in the context of the specified account.
Account Creation
Use the containerized Blockbridge CLI to create the account.
$ export BLOCKBRIDGE_API_HOST=blockbridge.mycompany.example
$ docker run --rm -it -e BLOCKBRIDGE_API_HOST docker.io/blockbridge/cli:latest-alpine bb --no-ssl-verify-peer account create --name kubernetes
When prompted, authenticate to the Blockbridge controlplane as the system
user.
Authenticating to https://blockbridge.mycompany.example/api
Enter user or access token: system
Password for system: ....
Authenticated; token expires in 3599 seconds.
Validate that the account has been created.
== Created account: kubernetes (ACT0762194C40656F03)
== Account: kubernetes (ACT0762194C40656F03)
name kubernetes
label kubernetes
serial ACT0762194C40656F03
created 2018-11-19 16:15:15 +0000
disabled no
Authorization Token
Blockbridge supports revokable persistent authorization tokens. This section
demonstrates how to create a persistent authorization token in the freshly
created kubernetes
account suitable for use as authentication for the driver.
To create the token, first authenticate as the system
user. Then use the
containerized Blockbridge CLI tool with the BLOCKBRIDGE_API_SU
environment
variable to create a persistent authorization in the kubernetes
account.
$ export BLOCKBRIDGE_API_HOST=blockbridge.mycompany.example
$ export BLOCKBRIDGE_API_SU=kubernetes
$ docker run --rm -it -e BLOCKBRIDGE_API_HOST -e BLOCKBRIDGE_API_SU docker.io/blockbridge/cli:latest-alpine bb --no-ssl-verify-peer authorization create --notes 'csi-blockbridge driver access'
Once again, authenticate using the system
username and password.
Authenticating to https://blockbridge.mycompany.example/api
Enter user or access token: system
Password for system:
Authenticated; token expires in 3599 seconds.
This creates the authorization and displays the access token.
== Created authorization: ATH4762194C4062668E
== Authorization: ATH4762194C4062668E
serial ATH4762194C4062668E
account kubernetes (ACT0762194C40656F03)
user kubernetes (USR1B62194C40656FBD)
enabled yes
created at 2018-11-19 11:15:47 -0500
access type online
token suffix ot50v2vA
restrict auth
enforce 2-factor false
== Access Token
access token 1/Nr7qLedL/P0KXxbrB8+jpfrFPBrNi3X+8H9BBwyOYg/mvOot50v2vA
*** Remember to record your access token!
Make a note of the displayed access token somewhere safe. Set the environment
variable BLOCKBRIDGE_API_KEY
to use in the forthcoming steps to install
the driver.
$ export BLOCKBRIDGE_API_KEY="1/Nr7qLedL/P0KXxbrB8+jpfrFPBrNi3X+8H9BBwyOYg/mvOot50v2vA"
Driver Installation
Here’s how to install the Blockbridge driver in your Kubernetes cluster.
Authenticate with Kubernetes
First, ensure your session is authenticated to your Kubernetes cluster. Running
kubectl version
should show a version for both the client and server
successfully.
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.7", GitCommit:"1dd5338295409edcfff11505e7bb246f0d325d15", GitTreeState:"clean", BuildDate:"2021-01-13T13:23:52Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:03:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"}
kubectl
authentication is beyond the
scope of this guide. Please refer to the specific instructions for the
Kubernetes service or installation you are using.Create a Secret
Next, create a secret containing both the Blockbridge API endpoint URL and access token.
ssl-verify-peer
flag. This setting implicitly trusts the default controlplane
self-signed certificate. Configuring certificate verification, including
specifying custom-supplied CA certificates, is beyond the scope of this
guide. Please contact Blockbridge Support for
more information.Use BLOCKBRIDGE_API_HOST
and BLOCKBRIDGE_API_KEY
with the correct values
for the Blockbridge controlplane, and the access token you created earlier in
the kubernetes account. Here’s how to do it with a “here” document that
expands the variables:
$ cat > secret.yml <<- EOF
apiVersion: v1
kind: Secret
metadata:
name: blockbridge
namespace: kube-system
stringData:
api-url: "https://${BLOCKBRIDGE_API_HOST}/api"
access-token: "$BLOCKBRIDGE_API_KEY"
ssl-verify-peer: "false"
EOF
Verify that its contents are correct.
$ cat secret.yml
apiVersion: v1
kind: Secret
metadata:
name: blockbridge
namespace: kube-system
stringData:
api-url: "https://blockbridge.mycompany.example/api"
access-token: "1/Nr7qLedL/P0KXxbrB8+jpfrFPBrNi3X+8H9BBwyOYg/mvOot50v2vA"
ssl-verify-peer: "false"
Next, use secret.yml
to create the secret in Kubernetes.
$ kubectl create -f ./secret.yml
secret "blockbridge" created
Finally, ensure the secret exists in the kube-system
namespace.
$ kubectl -n kube-system get secrets blockbridge
NAME TYPE DATA AGE
blockbridge Opaque 3 2m
Deploy the Blockbridge Driver
Deploy the Blockbridge Driver as a DaemonSet / StatefulSet using kubectl
.
$ kubectl apply -f https://get.blockbridge.com/kubernetes/5.1/csi/csi-blockbridge-v2.0.0.yaml
If everything was created successfully, the command output should look like
this, with all lines ending in created
.
csidriver.storage.k8s.io/csi.blockbridge.com created
storageclass.storage.k8s.io/blockbridge-gp created
storageclass.storage.k8s.io/blockbridge-tls created
statefulset.apps/csi-blockbridge-controller created
serviceaccount/csi-blockbridge-controller-sa created
clusterrole.rbac.authorization.k8s.io/csi-blockbridge-provisioner-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-blockbridge-provisioner-binding created
clusterrole.rbac.authorization.k8s.io/csi-blockbridge-attacher-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-blockbridge-attacher-binding created
daemonset.apps/csi-blockbridge-node created
serviceaccount/csi-blockbridge-node-sa created
clusterrole.rbac.authorization.k8s.io/csi-blockbridge-node-driver-registrar-role created
clusterrolebinding.rbac.authorization.k8s.io/csi-blockbridge-node-driver-registrar-binding created
For reference, the Blockbridge CSI Driver is deployed using the recommended mechanism of deploying CSI drivers on Kubernetes.
Ensure the Driver is Operational
Finally, check that the driver is up and running.
$ kubectl -n kube-system get pods -l role=csi-blockbridge
NAME READY STATUS RESTARTS AGE
csi-blockbridge-controller-0 3/3 Running 0 6s
csi-blockbridge-node-4679b 2/2 Running 0 5s
Storage Classes
The Blockbridge driver comes with a default “general purpose” StorageClass
named blockbridge-gp
. This is the default StorageClass for dynamic
provisioning of storage volumes. It provisions using the default Blockbridge
storage template configured in the Blockbridge controlplane.
$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
blockbridge-gp (default) csi.blockbridge.com Delete Immediate true 65m
blockbridge-tls csi.blockbridge.com Delete Immediate false 65m
There are a variety of additional storage class configuration options available, including:
- Using transport encryption (tls).
- Using a custom tag-based query.
- Using a named service template.
- Using explicitly specified provisioned IOPS.
There are several additional example storage classes in
csi-storageclass.yaml
. You can download, edit, and apply these storage
classes as needed.
$ curl -OsSL https://get.blockbridge.com/kubernetes/5.1/csi/csi-storageclass.yaml
$ cat csi-storageclass.yaml
... [output trimmed] ...
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: blockbridge-gp
namespace: kube-system
annotations:
storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.blockbridge.com
allowVolumeExpansion: true
$ kubectl apply -f ./csi-storageclass.yaml
storageclass.storage.k8s.io "blockbridge-gp" configured
Testing
This section has a few basic tests you can use to validate that your Blockbridge driver is working properly.
Create a Volume
This verifies that Blockbridge storage volumes are now available via Kubernetes persistent volume claims (PVC).
To test this out, create a PersistentVolumeClaim. It will dynamically provision a volume in Blockbridge and make it accessible to applications.
$ kubectl apply -f https://get.blockbridge.com/kubernetes/5.1/examples/csi-pvc.yaml
persistentvolumeclaim "csi-pvc-blockbridge" created
Alternatively, download the example volume yaml, modify it as needed, and apply.
$ curl -OsSL https://get.blockbridge.com/kubernetes/5.1/examples/csi-pvc.yaml
$ cat csi-pvc.yaml
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: csi-pvc-blockbridge
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 5Gi
storageClassName: blockbridge-gp
$ kubectl apply -f ./csi-pvc.yaml
persistentvolumeclaim "csi-pvc-blockbridge" created
Use get pvc csi-pvc-blockbridge
to check that the PVC was created
successfully.
$ kubectl get pvc csi-pvc-blockbridge
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc-blockbridge Bound pvc-6cb93ab2-ec49-11e8-8b89-46facf8570bb 5Gi RWO blockbridge-gp 4s
Create a Pod
This test creates a Pod (application) that uses the PVC just created. When you create the Pod, it attaches the volume, then formats and mounts it, making it available to the specified application.
Create the application.
$ kubectl apply -f https://get.blockbridge.com/kubernetes/5.1/examples/csi-app.yaml
pod "blockbridge-demo" created
Alternatively, download the application yaml, modify as needed, and apply.
$ curl -OsSL https://get.blockbridge.com/kubernetes/5.1/examples/csi-app.yaml
$ cat csi-app.yaml
---
kind: Pod
apiVersion: v1
metadata:
name: blockbridge-demo
spec:
containers:
- name: my-frontend
image: busybox
volumeMounts:
- mountPath: "/data"
name: my-bb-volume
command: [ "sleep", "1000000" ]
- name: my-backend
image: busybox
volumeMounts:
- mountPath: "/data"
name: my-bb-volume
command: [ "sleep", "1000000" ]
volumes:
- name: my-bb-volume
persistentVolumeClaim:
claimName: csi-pvc-blockbridge
$ kubectl apply -f ./csi-app.yaml
pod "blockbridge-demo" created
Verify that the pod is running successfully.
$ kubectl get pod blockbridge-demo
NAME READY STATUS RESTARTS AGE
blockbridge-demo 2/2 Running 0 13s
Write Data From the Pod
Inside the app container, write data to the mounted volume.
$ kubectl exec -ti blockbridge-demo -c my-frontend /bin/sh
/ # df /data
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/blockbridge/2f93beb2-61eb-456b-809e-22e27e4f73cf
5232608 33184 5199424 1% /data
/ # touch /data/hello-world
/ # exit
$ kubectl exec -ti blockbridge-demo -c my-backend /bin/sh
/ # ls /data
hello-world
TROUBLESHOOTING
App Stuck in ContainerCreating
When the application is stuck in ContainerCreating, check to see if the mount has failed.
Symptom
Check the app status.
$ kubectl get pod/blockbridge-demo
NAME READY STATUS RESTARTS AGE
blockbridge-demo 0/2 ContainerCreating 0 20s
$ kubectl describe pod/blockbridge-demo
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10s default-scheduler Successfully assigned default/blockbridge-demo to kubelet.localnet
Normal SuccessfulAttachVolume 10s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-71c37e84-b302-11e9-a93f-0242ac110003"
Warning FailedMount 1s kubelet, kubelet.localnet MountVolume.MountDevice failed for volume "pvc-71c37e84-b302-11e9-a93f-0242ac110003" : rpc error: code = Unknown desc = runtime_error: /etc/iscsi/initiatorname.iscsi not found; ensure 'iscsi-initiator-utils' is installed.
Resolution
- Ensure the host running the kubelet has iSCSI client support installed on the host/node.
- For CentOS/RHEL, install the
iscsi-initiator-utils
package on the host running the kubelet.
yum install iscsi-initiator-utils
- For Ubuntu, install the
open-iscsi
package on the host running the kubelet.
apt install open-iscsi
- Delete/re-create the application pod to retry.
Symptom
Check the app status.
$ kubectl get pod/blockbridge-demo
NAME READY STATUS RESTARTS AGE
blockbridge-demo 0/2 ContainerCreating 0 20s
$ kubectl describe pod/blockbridge-demo
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 10m default-scheduler Successfully assigned default/blockbridge-demo to crc-l6qvn-master-0
Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-9b2e8116-62a6-4089-8b0d-fab0f839b7aa"
Warning FailedMount 9m56s kubelet MountVolume.MountDevice failed for volume "pvc-9b2e8116-62a6-4089-8b0d-fab0f839b7aa" : rpc error: code = Unknown desc = exec_error: Failed to connect to bus: No data available
iscsiadm: can not connect to iSCSI daemon (111)!
iscsiadm: Could not login to [iface: default, target: iqn.2009-12.com.blockbridge:t-pjwajzvdho-471c1b66-e24d-4377-a16b-71ac1d580061, portal: 172.16.100.129,3260].
iscsiadm: initiator reported error (20 - could not connect to iscsid)
iscsiadm: Could not log into all portals
Resolution
- Ensure the host running the kubelet has iSCSI daemon installed and started on the host/node.
- For CentOS/RHEL, install the
iscsi-initiator-utils
package on the host running the kubelet.
yum install iscsi-initiator-utils
systemctl enable iscsid
systemctl start iscsid
- For Ubuntu, install the
open-iscsi-utils
package on the host running the kubelet.
apt install open-iscsi-utils
systemctl enable iscsid
systemctl start iscsid
- Delete/re-create the application pod to retry.
Provisioning Unauthorized
In this failure mode, provisioning fails with an “unauthorized” message.
Symptom
Check the PVC describe output.
$ kubectl describe pvc csi-pvc-blockbridge
Provisioning failed due to “unauthorized” because the authorization access token is not valid. Ensure the correct access token is entered in the secret.
Warning ProvisioningFailed 6s (x2 over 19s) csi.blockbridge.com csi-provisioner-blockbridge-0 2caddb79-ec46-11e8-845d-465903922841 Failed to provision volume with StorageClass "blockbridge-gp": rpc error: code = Internal desc = unauthorized_error: unauthorized: unauthorized
Resolution
- Edit
secret.yml
and ensure correct access token and API URL are set. - delete secret:
kubectl delete -f secret.yml
- create secret:
kubectl create -f secret.yml
- remove old configuration:
kubectl delete -f https://get.blockbridge.com/kubernetes/5.1/examples/csi-pvc.yaml
kubectl delete -f https://get.blockbridge.com/kubernetes/5.1/csi/csi-blockbridge-v2.0.0.yaml
- re-apply configuration:
kubectl apply -f https://get.blockbridge.com/kubernetes/5.1/csi/csi-blockbridge-v2.0.0.yaml
kubectl delete -f https://get.blockbridge.com/kubernetes/5.1/examples/csi-pvc.yaml
Provisioning Storage Class Invalid
Provisioning fails with an “invalid storage class” error.
Symptom
Check the PVC describe output:
$ kubectl describe pvc csi-pvc-blockbridge
Provisioning failed because the storage class specified was invalid.
Warning ProvisioningFailed 7s (x3 over 10s) persistentvolume-controller storageclass.storage.k8s.io "blockbridge-gp" not found
Resolution
Ensure the StorageClass exists with the same name.
$ kubectl get storageclass blockbridge-gp
Error from server (NotFound): storageclasses.storage.k8s.io "blockbridge-gp" not found
- If it doesn’t exist, then create the storageclass.
$ kubectl apply -f https://get.blockbridge.com/kubernetes/5.1/csi/csi-storageclass.yaml
- Alternatively, download and edit the desired storageclass.
$ curl -OsSL https://get.blockbridge.com/kubernetes/5.1/csi/csi-storageclass.yaml
$ edit csi-storageclass.yaml
$ kubectl -f apply ./csi-storageclass.yaml
In the background, the PVC continually retries. Once the above changes are complete, it will pick up the storage class change.
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc-blockbridge Bound pvc-6cb93ab2-ec49-11e8-8b89-46facf8570bb 5Gi RWO blockbridge-gp 4s
App Stuck in Pending
One of the causes for an application stuck in pending is a missing Persistent Volume Claim (PVC).
Symptom
The output of get pod
show that the app is stuck in pending.
$ kubectl get pod blockbridge-demo
NAME READY STATUS RESTARTS AGE
blockbridge-demo 0/2 Pending 0 14s
Use describe pod
to reveal more information. In this case, the PVC is not
found.
$ kubectl describe pod blockbridge-demo
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 12s (x6 over 28s) default-scheduler persistentvolumeclaim "csi-pvc-blockbridge" not found
Resolution
Create the PVC if necessary and ensure that it’s valid. First, validate that it’s missing.
$ kubectl get pvc csi-pvc-blockbridge
Error from server (NotFound): persistentvolumeclaims "csi-pvc-blockbridge" not found
If it’s missing, create it.
$ kubectl apply -f https://get.blockbridge.com/kubernetes/5.1/examples/csi-pvc.yaml
persistentvolumeclaim "csi-pvc-blockbridge" created
In the background, the application retries automatically and succeeds in starting.
$ kubectl describe pod blockbridge-demo
Normal Scheduled 8s default-scheduler Successfully assigned blockbridge-demo to aks-nodepool1-56242131-0
Normal SuccessfulAttachVolume 8s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-5332e169-ec4f-11e8-8b89-46facf8570bb"
Normal SuccessfulMountVolume 8s kubelet, aks-nodepool1-56242131-0 MountVolume.SetUp succeeded for volume "default-token-bx8b9"