– Documentation

Docs: Install

Mon, 01 Jan 0001 00:00:00 +0000

To install the operator follow the installation page for your orchestrator.

Docs: Kubernetes

Mon, 01 Jan 0001 00:00:00 +0000

Make sure the prerequisites for StorageOS are satisfied before proceeding.

Any Kubernetes managed service such as EKS, AKS, GKE, DO or DockerEE platform can use the following Kubernetes guide to install StorageOS.

StorageOS supports the five most recent Kubernetes releases, at minimum.

Make sure to add a StorageOS licence after installing.

Install StorageOS on Kubernetes 1.21

The StorageOS Cluster Operator is a Kubernetes native application developed to deploy and configure StorageOS clusters, and assist with maintenance operations. We recommend its use for standard installations.

The operator is a Kubernetes controller that watches the StorageOSCluster CRD. Once the controller is ready, a StorageOS cluster definition can be created. The operator will deploy a StorageOS cluster based on the configuration specified in the cluster definition.

Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.

Steps to install StorageOS:

1. Install StorageOS operator

Install the StorageOS Cluster Operator using the following yaml manifest.

kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

Verify the Cluster Operator Pod Status

[root@master03]# kubectl -n storageos-operator get pod
NAME                                         READY     STATUS    RESTARTS   AGE
storageoscluster-operator-68678798ff-f28zw   1/1       Running   0          3m

The READY 1/1 indicates that storageoscluster resources can be created.

2. Create a Secret

Before deploying a StorageOS cluster, create a Secret defining the StorageOS API Username and Password in base64 encoding. The API username and password are used to create the default StorageOS admin account which can be used with the StorageOS CLI and to login to the StorageOS GUI. The CSI credentials are used to register the CSI accounts, so Kubernetes and StorageOS communicate over an authenticated API.

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

This example contains a default password, for production installations, use a unique, strong password.

You can define a base64 value by echo -n "mystring" | base64.

Make sure that the encoding of the credentials doesn’t have special characters such as ‘\n’. The echo -n ensures that a trailing new line is not appended to the string.

If you wish to change the default accounts details post-install please see Managing Users

3 Trigger a StorageOS installation

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "upstream"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "eks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "aks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "gke"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

Additional spec parameters are available on the Cluster Operator configuration page.

You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.

Verify StorageOS Installation

[root@master03]# kubectl -n kube-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
storageos-csi-helper-5cf59b5b4-f5nwr    2/2     Running   0          3m
storageos-daemonset-75f6c               3/3     Running   0          3m
storageos-daemonset-czbqx               3/3     Running   0          3m
storageos-daemonset-zv4tq               3/3     Running   0          3m
storageos-scheduler-6d67b46f67-5c46j    1/1     Running   0          3m

The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.

4. License cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.

To obtain a license, follow the instructions on our licensing operations page.

Install StorageOS on Kubernetes 1.20

Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.

Steps to install StorageOS:

1. Install StorageOS operator

Install the StorageOS Cluster Operator using the following yaml manifest.

kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

Verify the Cluster Operator Pod Status

[root@master03]# kubectl -n storageos-operator get pod
NAME                                         READY     STATUS    RESTARTS   AGE
storageoscluster-operator-68678798ff-f28zw   1/1       Running   0          3m

The READY 1/1 indicates that storageoscluster resources can be created.

2. Create a Secret

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

This example contains a default password, for production installations, use a unique, strong password.

You can define a base64 value by echo -n "mystring" | base64.

Make sure that the encoding of the credentials doesn’t have special characters such as ‘\n’. The echo -n ensures that a trailing new line is not appended to the string.

If you wish to change the default accounts details post-install please see Managing Users

3 Trigger a StorageOS installation

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "upstream"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "eks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "aks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "gke"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

Additional spec parameters are available on the Cluster Operator configuration page.

You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.

Verify StorageOS Installation

[root@master03]# kubectl -n kube-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
storageos-csi-helper-5cf59b5b4-f5nwr    2/2     Running   0          3m
storageos-daemonset-75f6c               3/3     Running   0          3m
storageos-daemonset-czbqx               3/3     Running   0          3m
storageos-daemonset-zv4tq               3/3     Running   0          3m
storageos-scheduler-6d67b46f67-5c46j    1/1     Running   0          3m

The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.

4. License cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.

To obtain a license, follow the instructions on our licensing operations page.

Install StorageOS on Kubernetes 1.19

Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.

Steps to install StorageOS:

1. Install StorageOS operator

Install the StorageOS Cluster Operator using the following yaml manifest.

kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

Verify the Cluster Operator Pod Status

[root@master03]# kubectl -n storageos-operator get pod
NAME                                         READY     STATUS    RESTARTS   AGE
storageoscluster-operator-68678798ff-f28zw   1/1       Running   0          3m

The READY 1/1 indicates that storageoscluster resources can be created.

2. Create a Secret

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

This example contains a default password, for production installations, use a unique, strong password.

You can define a base64 value by echo -n "mystring" | base64.

Make sure that the encoding of the credentials doesn’t have special characters such as ‘\n’. The echo -n ensures that a trailing new line is not appended to the string.

If you wish to change the default accounts details post-install please see Managing Users

3 Trigger a StorageOS installation

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "upstream"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "eks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "aks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "gke"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

Additional spec parameters are available on the Cluster Operator configuration page.

You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.

Verify StorageOS Installation

[root@master03]# kubectl -n kube-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
storageos-csi-helper-5cf59b5b4-f5nwr    2/2     Running   0          3m
storageos-daemonset-75f6c               3/3     Running   0          3m
storageos-daemonset-czbqx               3/3     Running   0          3m
storageos-daemonset-zv4tq               3/3     Running   0          3m
storageos-scheduler-6d67b46f67-5c46j    1/1     Running   0          3m

The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.

4. License cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.

To obtain a license, follow the instructions on our licensing operations page.

Install StorageOS on Kubernetes 1.18

Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.

Steps to install StorageOS:

1. Install StorageOS operator

Install the StorageOS Cluster Operator using the following yaml manifest.

kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

Verify the Cluster Operator Pod Status

[root@master03]# kubectl -n storageos-operator get pod
NAME                                         READY     STATUS    RESTARTS   AGE
storageoscluster-operator-68678798ff-f28zw   1/1       Running   0          3m

The READY 1/1 indicates that storageoscluster resources can be created.

2. Create a Secret

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

This example contains a default password, for production installations, use a unique, strong password.

You can define a base64 value by echo -n "mystring" | base64.

Make sure that the encoding of the credentials doesn’t have special characters such as ‘\n’. The echo -n ensures that a trailing new line is not appended to the string.

If you wish to change the default accounts details post-install please see Managing Users

3 Trigger a StorageOS installation

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "upstream"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "eks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "aks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "gke"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

Additional spec parameters are available on the Cluster Operator configuration page.

You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.

Verify StorageOS Installation

[root@master03]# kubectl -n kube-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
storageos-csi-helper-5cf59b5b4-f5nwr    2/2     Running   0          3m
storageos-daemonset-75f6c               3/3     Running   0          3m
storageos-daemonset-czbqx               3/3     Running   0          3m
storageos-daemonset-zv4tq               3/3     Running   0          3m
storageos-scheduler-6d67b46f67-5c46j    1/1     Running   0          3m

The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.

4. License cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.

To obtain a license, follow the instructions on our licensing operations page.

Install StorageOS on Kubernetes 1.17

Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.

Steps to install StorageOS:

1. Install StorageOS operator

Install the StorageOS Cluster Operator using the following yaml manifest.

kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

Verify the Cluster Operator Pod Status

[root@master03]# kubectl -n storageos-operator get pod
NAME                                         READY     STATUS    RESTARTS   AGE
storageoscluster-operator-68678798ff-f28zw   1/1       Running   0          3m

The READY 1/1 indicates that storageoscluster resources can be created.

2. Create a Secret

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

This example contains a default password, for production installations, use a unique, strong password.

You can define a base64 value by echo -n "mystring" | base64.

Make sure that the encoding of the credentials doesn’t have special characters such as ‘\n’. The echo -n ensures that a trailing new line is not appended to the string.

If you wish to change the default accounts details post-install please see Managing Users

3 Trigger a StorageOS installation

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "upstream"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "eks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "aks"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "gke"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

Additional spec parameters are available on the Cluster Operator configuration page.

You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.

Verify StorageOS Installation

[root@master03]# kubectl -n kube-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
storageos-csi-helper-5cf59b5b4-f5nwr    2/2     Running   0          3m
storageos-daemonset-75f6c               3/3     Running   0          3m
storageos-daemonset-czbqx               3/3     Running   0          3m
storageos-daemonset-zv4tq               3/3     Running   0          3m
storageos-scheduler-6d67b46f67-5c46j    1/1     Running   0          3m

The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.

4. License cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.

To obtain a license, follow the instructions on our licensing operations page.

First StorageOS volume

If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.

Docs: What is StorageOS V2

Mon, 01 Jan 0001 00:00:00 +0000

Over the past several months, we’ve been hard at work on StorageOS V2, which contains some significant enhancements over our v1 product. We’ve built V2 based on our observations of trends in the industry, as well as our own experience.

Many of our customers want to run big clusters - in the tens or hundreds of nodes. In these sorts of big environments, the challenges multiply. Not only do we need to scale well, but we also need to be more failure tolerant. Bigger environments typically suffer higher failure rates (more nodes = greater chance of something failing), but are also subject to all sorts of transient conditions such as network partitions.

The second trend we’ve seen become increasingly common is the desire to run multiple clusters, and consume storage between them in some way - sometimes to implement novel topologies such as a centralised storage cluster with satellites consuming the storage, and sometimes to replicate data between those clusters for HA or DR purposes.

We’ve built V2 with these architectures and design patterns in mind. Not only does it scale well, but it contains the foundations we need to implement a rich set of multi-cluster functionality.

Upgraded Control Plane

At the heart of the V2 release is an upgraded control plane. We’ve changed a lot here. Firstly, our usage of etcd is vastly improved. We’ve learnt a lot about the subtleties of distributed consensus in the last year, particularly in noisy or unpredictable environments. Not only is StorageOS V2 much lighter on your etcd cluster, but it’s a lot more tolerant of transient failure conditions that are often found in cloud environments, or clusters under heavy load.

We spent some time describing and testing our internal state machine using the TLA+ formal verification language. This allows us to have a much higher degree of confidence that our algorithms will behave correctly, particularly under hard-to-test edge cases and failure conditions.

Additionally, we’ve changed the way volumes behave with respect to centralised scheduling. Each volume group (consisting of a master and 0 or more replicas) now behaves as a strongly consistent unit allowing it to take action independent of the activities of the rest of the cluster. Other state can be distributed around the cluster using eventually consistent mechanisms. This approach inherently scales better and allows StorageOS V2 to effectively manage many more nodes and volumes than before.

We’ve implemented TLS on all endpoints. Not only does this give you encrypted traffic between nodes in your storage cluster, it also protects all endpoints with strong, public key based authentication. Today’s IT environments can’t rely on firewalls to keep bad actors out - they must implement security at all layers within the stack - defense in depth. While we recognise that this brings a welcome relief to many security conscious administrators, we also know that managing certificate authorities (CAs) can be an unwelcome source of complexity. For this reason, StorageOS V2 implements an internal CA by default, to manage this complexity for you. If you’d prefer to integrate your own CA, we support that too - it’s up to you.

Finally - our logging has undergone a complete transformation in this edition. We know that systems engineers and operators don’t just value headline features, but that observability and diagnostics are equally important. All logs are now decorated with rich context to help you understand what is happening within your cluster, and we’ll output in json by default, for easy ingestion into log aggregators such as Elasticsearch.

Upgraded Data Plane

Not to be outdone, our data plane contains some significant improvements.

Firstly, we’ve completely re-written our sync algorithm (see Delta Sync, used when seeding or catching up replicas that have been offline or partitioned. Our new algorithm uses a Hash List to sync only changed sections of a volume (similar in some ways to what rsync does). StorageOS maintains these hashes during normal operation, meaning that when resyncing a failed replica, for example after a node reboot, we can very quickly and efficiently catch this replica up, rather than needing to promote and build a new one from scratch. This improves resiliency within your cluster, and prevents using excessive network bandwidth during failover conditions - at a time when it might be needed the most.

Secondly, a new threading model, with dynamic pool sizing, means that StorageOS is faster, a lot faster. In our tests we observed improvements across the board, with improvements in throughput of up to 135% for some scenarios.

Docs: Admission Controller

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS implements a MutatingAdmissionWebhook Admission Controller to ensure that Pods using StorageOS Volumes use the storageos-scheduler. An admission controller intercepts requests to the Kubernetes API server prior to persistence of the object, but after the request is authenticated and authorized.

The Admission Controller is responsible for mutating the PodSpec at creation time to populate the PodSpec.schedulerName field with the name of the StorageOS Scheduler - storageos-scheduler.

During Pod creation, Kubernetes sends a web request to the StorageOS WebHook with the Pod specification. The PodSpec is only altered to use the StorageOS scheduler if the Pod uses a StorageOS volume.

Web Server

The Web Server hosting the web hook is executed in the StorageOS Cluster Operator. Since only HTTPS requests are allowed, the Operator generates a self-signed x509 certificate every time it starts. The Cluster Operator will also renew certificates upon expiry (certs are valid for one year).

There is no manual intervention required regarding the SSL configuration as the setup is completely transparent between StorageOS and Kubernetes.

Skipping Mutation

To avoid scheduler mutation, the storageos.com/scheduler=false annotation can be added to resources that use StorageOS volumes.

When using StatefulSets the annotation can be set on the spec.template.metadata.annotations field.

apiVersion: apps/v1
kind: StatefulSet
spec:
  ...
  template:
    metadata:
      annotations:
        storageos.com/scheduler: "false" # N.B. the value must be a string and not a boolean

When using Pods the annotation is set on the metadata.annotations field.

apiVersion: v1
kind: Pod
metadata:
    ...
    annotations:
        storageos.com/scheduler: "false" # N.B. the value must be a string and not a boolean
    ...

Explicit SchedulerName

It is not necessary to explicitly set the SchedulerName as the Admission Controller automatically populates the PodSpec field. Set the SchedulerName in your manifests, manually, only if you disable or can’t execute the StorageOS Admission Controller.

Kubernetes allows the use of different schedulers by setting the field .spec.schedulerName: storageos-scheduler.

For instance a Pod manifest utilising the StorageOS scheduler would appear as follows:

apiVersion: v1
kind: Pod
metadata:
  name: d1
spec:
  schedulerName: storageos-scheduler # --> StorageOS Scheduler
                                     # No need if using Admission Controller
                                     # (enabled by default)
  containers:
    - name: debian
      image: debian:9-slim
      command: ["/bin/sleep"]
      args: [ "3600" ]
      volumeMounts:
        - mountPath: /mnt
          name: v1
  volumes:
    - name: v1
      persistentVolumeClaim:
        claimName: persistent-volume # ----> StorageOS PVC

Compatibility

The Admission Controller doesn’t need to be enabled at Kubernetes cluster bootstrap time because it is a Dynamic Admission Controller. Hence, any cluster that has the MutatingAdmissionWebhook enabled is supported. Most Kubernetes cluster enable the Webhook admission controller by default.

The MutatingAdmissionWebhook is available from Kubernetes v1.13.

You can check your Kubernetes cluster compatibility by checking if the following object exists.

kubectl api-versions | grep admissionregistration.k8s.io

Docs: Configuration

Mon, 01 Jan 0001 00:00:00 +0000

StorageOSCluster Resource Configuration

The following table lists the configurable spec parameters of the StorageOSCluster custom resource and their default values.

Parameter	Description	Default
`csi.deploymentStrategy`	CSI helper deployment strategy (`statefulset` or `deployment`)	`statefulset`
`csi.enable`	Enable CSI setup	`false`
`csi.enableControllerPublishCreds`	Enable CSI controller publish credentials	`false`
`csi.enableNodePublishCreds`	Enable CSI node publish credentials	`false`
`csi.enableProvisionCreds`	Enable CSI provision credentials	`false`
`debug`	Enable debug mode for all the cluster nodes	`false`
`disableFencing`	Disable Pod fencing	`false`
`disableScheduler`	Disable StorageOS scheduler	`false`
`disableTCMU`	Disable TCMU to allow co-existence with other TCMU users. Disabling TCMU degrades performance	`false`
`disableTelemetry`	Disable telemetry reports	`false`
`forceTCMU`	Forces TCMU to be enabled or causes StorageOS to abort startup	`false`
`images.apiManagerContainer`	StorageOS API Manager container image	`storageos/api-manager:v1.0.0`
`images.csiClusterDriverRegistrarContainer`	CSI Cluster Driver Registrar Container image	`quay.io/k8scsi/csi-cluster-driver-registrar:v1.0.1`
`images.csiExternalAttacherContainer`	CSI External Attacher Container image	`quay.io/k8scsi/csi-attacher:v1.0.1`
`images.csiExternalProvisionerContainer`	CSI External Provisioner Container image	`storageos/csi-provisioner:v1.0.1`
`ìmages.csiLivenessProbeContainer`	CSI Liveness Probe Container Image	`quay.io/k8scsi/livenessprobe:v1.0.1`
`images.csiNodeDriverRegistrarContainer`	CSI Node Driver Registrar Container image	`quay.io/k8scsi/csi-node-driver-registrar:v1.0.1`
`images.hyperkubeContainer`	Deprecated field - HyperKube Container image	Default dependent on Scheduler version
`images.initContainer`	StorageOS init container image	`storageos/init:2.1.0`
`images.kubeSchedulerContainer`	Kube scheduler container image	Default dependent on Scheduler version
`images.nfsContainer`	StorageOS nfs container image	`storageos/nfs:1.0.0`
`images.nodeContainer`	StorageOS node container image	`storageos/node:v2.4.4`
`ingress.annotations`	Annotations of the ingress used by the cluster
`ingress.enable`	Enable ingress for the cluster	`false`
`ingress.hostname`	Hostname to be used in cluster ingress	`storageos.local`
`ingress.tls`	Enable TLS for the ingress	`false`
`k8sDistro`	The name of the Kubernetes distribution is use, e.g. `rancher` or `eks`
`kvBackend.address`	Comma-separated list of addresses of external key-value store. (`1.2.3.4:2379,2.3.4.5:2379`)
`kvBackend.backend` (v2 deprecated)	Name of the key-value store to use. Set to `etcd` for external key-value store.	`embedded`
`namespace`	Namespace where storageos cluster resources are created	`kube-system`
`nodeSelectorTerms`	Set node selector for storageos pod placement
`pause`	Pause the operator for cluster maintenance	`false`
`resources`	Set resource requirements for the containers
`secretRefName`	Reference name of storageos secret
`secretRefNamespace`	Namespace of storageos secret
`service.annotations`	Annotations of the Service used by the cluster
`service.externalPort`	External port of the Service used by the cluster	`5705`
`service.internalPort`	Internal port of the Service used by the cluster	`5705`
`service.name`	Name of the Service used by the cluster	`storageos`
`service.type`	Type of the Service used by the cluster	`ClusterIP`
`sharedDir`	Path to be shared with kubelet container when deployed as a pod	`/var/lib/kubelet/plugins/kubernetes.io~storageos`
`storageClassName`	The name of the default StorageClass created for StorageOS volumes	`fast`
`tlsEtcdSecretRefName`	Secret containing etcd client certificates
`tlsEtcdSecretRefNamespace`	Namespace of the tlsEtcdSecretRefName
`tolerations`	Set pod tolerations for storageos pod placement

Docs: OpenShift

Mon, 01 Jan 0001 00:00:00 +0000

Make sure the prerequisites for StorageOS are satisfied before proceeding. Including the deployment of an etcd cluster and configuration of CRI-O PID limits.

For OpenShift upgrades, refer to the OpenShift platform page.

If you have installed OpenShift in AWS ensure that the requisite ports are opened for the worker nodes’ security group.

Make sure to add a StorageOS licence after installing.

StorageOS v2 supports OpenShift v4. For more information check the OpenShift platform page.

OperatorHub

Select the OperatorHub from the Catalog sub menu and search for StorageOS

Choose between using the RedHat Market Place or the Community Operators installation.
Select StorageOS and click install
Select the install options

Make sure the Approval Strategy is set to Manual. So the StorageOS Operator doesn’t upgrade versions without explicit approval.
Start the approval procedure
Follow the approval link
Approve the installation
The StorageOS Cluster Operator is installed along the required CRDs
Create a Secret in the openshift-operators project

Use the YAML options to create a secret containing the apiUsername and an apiPassword key. The username and password defined in the secret will be used to authenticate when using the StorageOS CLI and GUI. For the communication between StorageOS and OpenShift, the CSI credentials csiProvisionUsername, csiProvisionPassword, csiControllerPublishUsername, csiControllerPublishPassword, csiNodePublishUsername, csiNodePublishPassword are needed. Take note of which project you created the secret in.

Input the Secret as YAML for simplicity.

apiVersion: v1
kind: Secret
metadata:
  name: storageos-api
  namespace: openshift-operators
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

Go to the “Installed Operators”

Verify that the StorageOS Cluster Operator is installed
Go to the “StorageOS Cluster” section
Create the StorageOS Cluster

A StorageOS Cluster is defined using a Custom Resource Definition

Create the Custom Resource

The StorageOS cluster resource describes the StorageOS cluster that will be created. Parameters such as the secretRefName, the secretRefNamespace and the kvBackend.address are mandatory.

Additional spec parameters are available on the Cluster Operator configuration page.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: storageos
  namespace: openshift-operators
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference the Secret created in the previous step
  secretRefNamespace: "openshift-operators"  # Namespace of the Secret created in the previous step
  k8sDistro: "openshift"
  kvBackend:
    address: 'storageos-etcd-client.etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
  # nodeSelectorTerms:
  #   - matchExpressions:
  #     - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
  #       operator: In
  #       values:
  #       - "true"

Verify that the StorageOS Cluster Resource enters a running state.

It can take up to a minute to report the StorageOS Pods ready
Check the StorageOS Pods in the kube-system project

A Status of 3/3 for the Daemonset Pods indicates that StorageOS is bootstrapped successfully.
License cluster

A newly installed StorageOS cluster does not include a licence. A cluster must be licensed within 24 hours of the installation. For more information, check the reference licence page.

You can apply a Free Developer licence following the operations licensing page, or purchase a licence contacting sales@storageos.com.

Red Hat Marketplace

The installation of StorageOS using the Red Hat Marketplace requires the Openshift cluster to be registered to the Marketplace Portal, including the roll out of the PullSecret in your cluster. Failure to do so will result in a image pull authentication failure with the Red Hat registry.

Select the OperatorHub from the Catalog sub menu and search for StorageOS.

Choose the RedHat Marketplace option.
Select StorageOS and click purchase. Note that Openshift needs to be registered with the Red Hat Marketplace portal.
Select the most suitable install option.

Project Edition is suitable for production workloads, Developer Edition for personal experimentation and evaluation.
Specify the product configuration to fit your needs.
Navigate to your software within Red Hat Marketplace and install the StorageOS software as specified in the image.
Install the Operator. Set the update approval strategy to Automatic to ensure that you always have the latest version of StorageOS installed.
The StorageOS Cluster Operator is installed into your specified cluster.
Create a Secret in the openshift-operators project

Input the Secret as YAML for simplicity.

apiVersion: v1
kind: Secret
metadata:
  name: storageos-api
  namespace: openshift-operators
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

Go to StorageOS in your “Installed Operators”

Verify that the StorageOS Cluster Operator is installed
Go to the “StorageOS Cluster” section
Create the StorageOS Cluster

A StorageOS Cluster is defined using a Custom Resource Definition

Create the Custom Resource

The StorageOS cluster resource describes the StorageOS cluster that will be created. Parameters such as the secretRefName, the secretRefNamespace and the kvBackend.address are mandatory.

Additional spec parameters are available on the Cluster Operator configuration page.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: storageos
  namespace: openshift-operators
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference the Secret created in the previous step
  secretRefNamespace: "openshift-operators"  # Namespace of the Secret created in the previous step
  k8sDistro: "openshift"
  kvBackend:
    address: 'storageos-etcd-client.etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
  # nodeSelectorTerms:
  #   - matchExpressions:
  #     - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
  #       operator: In
  #       values:
  #       - "true"

Verify that the StorageOS Cluster Resource enters a running state.

It can take up to a minute to report the StorageOS Pods ready
Check the StorageOS Pods in the kube-system project

A Status of 3/3 for the Daemonset Pods indicates that StorageOS is bootstrapped successfully.
License cluster

A newly installed StorageOS cluster does not include a licence. A cluster must be licensed within 24 hours of the installation. For more information, check the reference licence page.

You can apply a Free Developer licence following the operations licensing page, or purchase a licence contacting sales@storageos.com.

Manual install

Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.

Steps to install StorageOS:

1. Install StorageOS operator

Install the StorageOS Cluster Operator using the following yaml manifest.

oc create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

Verify the Cluster Operator Pod Status

[root@master03]# oc -n storageos-operator get pod
NAME                                         READY     STATUS    RESTARTS   AGE
storageoscluster-operator-68678798ff-f28zw   1/1       Running   0          3m

The READY 1/1 indicates that storageoscluster resources can be created.

2. Create a Secret

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

This example contains a default password, for production installations, use a unique, strong password.

You can define a base64 value by echo -n "mystring" | base64.

Make sure that the encoding of the credentials doesn’t have special characters such as ‘\n’. The echo -n ensures that a trailing new line is not appended to the string.

If you wish to change the default accounts details post-install please see Managing Users

3 Trigger a StorageOS installation

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "openshift"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

Additional spec parameters are available on the Cluster Operator configuration page.

You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.

Verify StorageOS Installation

[root@master03]# oc -n kube-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
storageos-csi-helper-5cf59b5b4-f5nwr    2/2     Running   0          3m
storageos-daemonset-75f6c               3/3     Running   0          3m
storageos-daemonset-czbqx               3/3     Running   0          3m
storageos-daemonset-zv4tq               3/3     Running   0          3m
storageos-scheduler-6d67b46f67-5c46j    1/1     Running   0          3m

The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.

4. License cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.

To obtain a license, follow the instructions on our licensing operations page.

First StorageOS volume

If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.

Docs: Quickstart

Mon, 01 Jan 0001 00:00:00 +0000

Start by understanding the architecture.
Follow our installation guides for Kubernetes, OpenShift or Rancher.
Dive into our Use Case documentation for Workloads deployed in Kubernetes.

Docs: Cluster Operator examples

Mon, 01 Jan 0001 00:00:00 +0000

Before deploying a StorageOS cluster, create a Secret to define the StorageOS API Username and Password in base64 encoding.

kubectl create -f - <<END
apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "default"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z

  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z
END

This example contains a default password, for production installations, use a unique, strong password.

Make sure that the encoding of the credentials doesn’t have special characters such as ‘\n’.

You can define a base64 value by echo -n "mystring" | base64.

Create a cluster-config.yaml according to your needs from the examples below.

kubectl create -f cluster-config.yaml

Note that StorageOS will be deployed in spec.namespace (storageos by default), irrespective of what NameSpace the CR is defined in.

Examples

You can checkout all the parameters configurable in the configuration page.

All examples must reference the storageos-api Secret.

spec:
  secretRefName: "storageos-api" # Reference to the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret

Check out Cluster Definition examples for full CR files.

Installing with an external etcd

spec:
  kvBackend:
    address: '10.43.93.95:2379' # IP of the SVC that exposes ETCD
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can specify individual IPs of the etcd servers

If using Etcd with mTLS, you need to specify the secret that hold the certificates with the following parameters:

spec:
  # External mTLS secured etcd cluster specific properties
  tlsEtcdSecretRefName: "etcd-client-tls"                          # Secret containing etcd client certificates
  tlsEtcdSecretRefNamespace: "storageos-etcd"                      # Namespace of the client certificates secret

Follow the etcd operations page to setup the secret with the Etcd client certificate, client key and CA.

Installing to a subset of nodes

In this case we select nodes that are workers. To make sure that StorageOS doesn’t start in Master nodes.

You can see the labels in the nodes by kubectl get node --show-labels.

spec:
  nodeSelectorTerms:
    - matchExpressions:
      - key: "node-role.kubernetes.io/worker"
        operator: In
        values:
        - "true"

# OpenShift uses "node-role.kubernetes.io/compute=true"
# Rancher uses "node-role.kubernetes.io/worker=true"
# Kops uses "node-role.kubernetes.io/node="

Different provisioners and Kubernetes distributions use node labels differently to specify master vs workers. Node Taints are not enough to make sure StorageOS doesn’t start in a node. The JOIN variable is defined by the operator by selecting all the nodes that match the nodeSelectorTerms.

Enabling CSI

spec:
  csi:
    enable: true
    deploymentStrategy: deployment
    enableProvisionCreds: true
    enableControllerPublishCreds: true
    enableNodePublishCreds: true
    enableControllerExpandCreds: true

The credentials must be defined in the storageos-api Secret

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z

   # Add base64 encoded creds below for CSI credentials.
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

Specifying a shared directory for use with kubelet as a container

spec:
  sharedDir: '/var/lib/kubelet/plugins/kubernetes.io~storageos'

Defining pod resource requests and reservations

spec:
  resources:
    requests:
      memory: "512Mi"
  #   cpu: "1"
  # limits:
  #   memory: "4Gi"

Limiting StorageOS can cause malfunction for IO to StorageOS volumes, therefore we do not currently recommend applying upper limits to resources for StorageOS pods.

Specifying custom Tolerations

spec:
  tolerations:
  - key: "key1"
    operator: "Equal"
    value: "value1"
    effect: "EffectToTolerate"
  - key: "key2"
    operator: "Exists"

Custom tolerations specified in the StorageOSCluster definition are added to all StorageOS components; the StorageOS daemonset, CSI helper and scheduler.

In the above example a toleration key1=value1:EffectToTolerate would be tolerated and key2 would be tolerated regardless of the value and effect. For more information about tolerations please see the Kubernetes documentation.

Docs: Rancher

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS is a certified Rancher application. We offer two installation methods:

Rancher Catalogue - this is the easiest and requires just a few clicks
Manual - allowing more control and visibility

Before proceeding, ensure that you have followed our prerequisites. On Rancher, pay particular attention to the OS version and image used - some platforms require extra mainline kernel modules to be enabled.

Catalog Install

StorageOS is a Certified application in the Rancher Catalog. You can install StorageOS using the Rancher application install.

Before completing the steps below, you will need an etcd cluster. For evaluations it is sufficient to use our simple test recipe. For production installations, please follow our production recipe. Make a note of the etcd endpoint URL in either case.

Select the System project of your cluster
Select the Apps tab and click Launch
Search for StorageOS and click on the App

This will install the StorageOS operator, which manages the StorageOS DaemonSet.
Check and ammend installation options

A generic configuration for StorageOS is preset using the default values in the form. Be sure to check the etcd address and ensure it matches the value you noted at the beginning of this guide.

The catalog form exposes several useful parameters - documented below.

For further customization, you can opt to set the option to ‘Install StorageOS Cluster’ to false and install a custom CR. See below for this.
Launch the StorageOS cluster
Verify the cluster bootstrap has successfully completed
License the newly installed cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our personal license is free, and supports up to 1TiB of provisioned storage.

You will need access to the StorageOS API on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):
```
$ kubectl port-forward -n kube-system svc/storageos 5705
```
Now follow the instructions on our licensing operations page to obtain and apply a license.

Installation of StorageOS is now complete.

Simple Customization - Modify Catalog Form

The following options are exposed by the catalog form to allow some simple customization of the StorageOS installation.

Cluster Operator namespace : The Kubernetes namespace where the StorageOS Cluster Operator and other resources will be created.
Container Images : By default images are pulled from DockerHub, you can
specify the image URLs when using private registries.
Install StorageOS cluster : Controls the automatic deployment of StorageOS after installing the Cluster Operator. If set to false, the Operator will be created, but a Custom Resource will not be applied to the cluster. Launch the operator and proceed to the section Advanced Customization below.
Namespace : The Kubernetes namespace where StorageOS will be installed. By default, StorageOS installs into the kube-system namespace, which will add a priority class to ensure high priority resource allocation. Installing StorageOS with the priority class prevents StorageOS from being evicted during periods of resource contention. It is inadvisable to modify this under normal circumstances.
Username/Password : Default Username and Password for the admin account to be created at StorageOS bootstrap. A random password will be generated by leaving the field empty or clicking the Generate button.
External etcd address(es) : Connection and configuration details for an external Etcd cluster.See our documentation here.
Node Selectors and Tolerations : Control placement of StorageOS DaemonSet Pods. StorageOS will only be installed on the selected nodes.
Tolerations : Define any tolerations you wish the DaemonSet to observe.

Advanced Customization - Apply Custom CR

If Install StorageOS Cluster was set to false, StorageOS will not be bootstrapped automatically. After the StorageOS Operator is installed, you can now create a Custom Resource that describes the StorageOS cluster.

Select the System Workloads and Import YAML

Create the Secret and CustomResource

This is an example.

---
apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z
---
apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "rancher"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  sharedDir: '/var/lib/kubelet/plugins/kubernetes.io~storageos' # Needed when Kubelet as a container
  resources:
    requests:
      memory: "512Mi"
  nodeSelectorTerms:
    - matchExpressions:
      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
        operator: In
        values:
        - "true"

Additional spec parameters are available on the Cluster Operator configuration page.

You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.

Manual Installation

Helm Note: If you want to use Helm to install StorageOS, follow the StorageOS Operator Helm Chart documentation.

Steps to install StorageOS:

1. Install StorageOS operator

Install the StorageOS Cluster Operator using the following yaml manifest.

kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

Verify the Cluster Operator Pod Status

[root@master03]# kubectl -n storageos-operator get pod
NAME                                         READY     STATUS    RESTARTS   AGE
storageoscluster-operator-68678798ff-f28zw   1/1       Running   0          3m

The READY 1/1 indicates that storageoscluster resources can be created.

2. Create a Secret

apiVersion: v1
kind: Secret
metadata:
  name: "storageos-api"
  namespace: "storageos-operator"
  labels:
    app: "storageos"
type: "kubernetes.io/storageos"
data:
  # echo -n '<secret>' | base64
  apiUsername: c3RvcmFnZW9z
  apiPassword: c3RvcmFnZW9z
  # CSI Credentials
  csiProvisionUsername: c3RvcmFnZW9z
  csiProvisionPassword: c3RvcmFnZW9z
  csiControllerPublishUsername: c3RvcmFnZW9z
  csiControllerPublishPassword: c3RvcmFnZW9z
  csiNodePublishUsername: c3RvcmFnZW9z
  csiNodePublishPassword: c3RvcmFnZW9z
  csiControllerExpandUsername: c3RvcmFnZW9z
  csiControllerExpandPassword: c3RvcmFnZW9z

This example contains a default password, for production installations, use a unique, strong password.

You can define a base64 value by echo -n "mystring" | base64.

Make sure that the encoding of the credentials doesn’t have special characters such as ‘\n’. The echo -n ensures that a trailing new line is not appended to the string.

If you wish to change the default accounts details post-install please see Managing Users

3 Trigger a StorageOS installation

This is a Cluster Definition example.

apiVersion: "storageos.com/v1"
kind: StorageOSCluster
metadata:
  name: "example-storageos"
  namespace: "storageos-operator"
spec:
  # StorageOS Pods are in kube-system by default
  secretRefName: "storageos-api" # Reference from the Secret created in the previous step
  secretRefNamespace: "storageos-operator"  # Namespace of the Secret
  k8sDistro: "rancher"
  images:
    nodeContainer: "storageos/node:v2.4.4" # StorageOS version
  kvBackend:
    address: 'storageos-etcd-client.storageos-etcd:2379' # Example address, change for your etcd endpoint
  # address: '10.42.15.23:2379,10.42.12.22:2379,10.42.13.16:2379' # You can set ETCD server ips
  resources:
    requests:
      memory: "512Mi"
      cpu: 1
#  nodeSelectorTerms:
#    - matchExpressions:
#      - key: "node-role.kubernetes.io/worker" # Compute node label will vary according to your installation
#        operator: In
#        values:
#        - "true"

Additional spec parameters are available on the Cluster Operator configuration page.

You can find more examples such as deployments referencing a external etcd kv store for StorageOS in the Cluster Operator examples page.

Verify StorageOS Installation

[root@master03]# kubectl -n kube-system get pods -w
NAME                                    READY   STATUS    RESTARTS   AGE
storageos-csi-helper-5cf59b5b4-f5nwr    2/2     Running   0          3m
storageos-daemonset-75f6c               3/3     Running   0          3m
storageos-daemonset-czbqx               3/3     Running   0          3m
storageos-daemonset-zv4tq               3/3     Running   0          3m
storageos-scheduler-6d67b46f67-5c46j    1/1     Running   0          3m

The above command watches the Pods created by the Cluster Definition example. Note that pods typically take approximately 65 seconds to enter the Running Phase.

4. License cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our developer license is free, and supports up to 5TiB of provisioned storage.

To obtain a license, follow the instructions on our licensing operations page.

First StorageOS volume

If this is your first installation you may wish to follow the StorageOS Volume guide for an example of how to mount a StorageOS volume in a Pod.

Docs: Cluster Operator Upgrade

Mon, 01 Jan 0001 00:00:00 +0000

Upgrade StorageOS operator from yaml manifest

Upgrade the StorageOS operator using the following yaml manifest.

kubectl apply -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

When you run the above command StorageOS Operator resources will be updated. Since, the Update Strategy of the StorageOS Operator Deployment is set to rolling update, a new StorageOS Operator Pod will be created. Only when the new Pod enters the Running Phase will the old Pod be deleted. Your StorageOS Cluster will not be affected while the StorageOS Operator is upgrading.

Upgrade StorageOS Operator using Helm

If you have installed the StorageOS Operator using the Helm Chart, then you can upgrade the operator using the following commands.

$ helm list

NAME            REVISION        STATUS          CHART                           APP VERSION     NAMESPACE   
storageos-v1   4               DEPLOYED        storageos-operator-0.2.11       1.3.0           storageos-operator

$ helm repo update
$ helm upgrade $NAME storageos/storageos-operator

When you run the above command StorageOS Operator resources will be updated. Since, the Update Strategy of the StorageOS Operator Deployment is set to rolling update, a new StorageOS Operator Pod will be created. Only when the new Pod enters the Running Phase will the old Pod be deleted. Your StorageOS Cluster will not be affected while the StorageOS Operator is upgrading.

Docs: Supported Platforms and Orchestrators

Mon, 01 Jan 0001 00:00:00 +0000

OS

Linux X86_64
Kernels satisfying our module prerequisites
3.x kernels have a limitation of 256 active volumes per node
4.x kernels have a limitation of 4096 active volumes per node
We are distribution agnostic as long as our prerequisites are met

Orchestrators

Kubernetes 1.17 to 1.21
OpenShift 4.0+

Docs: System Configuration

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS requires certain kernel modules to function. In particular it requires Linux-IO, an open-source implementation of the SCSI target, on all nodes that will execute StorageOS (usually the workers).

We require the following modules to be loaded:

target_core_mod
tcm_loop
target_core_file
configfs
target_core_user
uio

N.B. Other applications utilising TCMU cannot be run concurrently with StorageOS. Doing so may result in corruption of data. On startup, StorageOS will detect if other applications are using TCMU and fall back to FUSE. However if StorageOS is started first there is no mechanism for StorageOS to fallback to FUSE if another application begins to use TCMU. TCMU can be disabled using the DISABLE_TCMU StorageOSCluster spec parameter.

Depending on the distribution, the modules are shipped as part of the base kernel package or as part of a kernel extras package which needs to be installed.

Distribution Specifics

The following distributions are supported by default:

RHEL 7.5
CentOS 7
Debian 9
Ubuntu Azure
RancherOS - Note CSI is not supported on RancherOS

Ubuntu 16.04/18.04 requires the installation of additional packages.

N.B. Ubuntu 16.04/18.04 AWS and Ubuntu 18.04 GCE do not provide the necessary linux-image-extra package - see below for more information

Ubuntu Package Installation

Ubuntu 16.04/18.04 Generic and Ubuntu 16.04 GCE require extra packages:

Ubuntu 16.04:

sudo apt -y update
sudo apt -y install linux-image-extra-$(uname -r)

Ubuntu 18.04+:

sudo apt -y update
sudo apt -y install linux-modules-extra-$(uname -r)

Ubuntu With AWS Or GCE Kernels

Ubuntu 16.04/18.04 AWS and Ubuntu 18.04 GCE do not yet provide the linux-image-extra package. As such you should either use Debian, CentOS or RHEL, or install the non-cloud-provider optimised Ubuntu kernel.

Installing the non-cloud-provider optimised Ubuntu kernel is something that should only be done with full understanding of potential ramifications.

sudo apt -y update
sudo apt install -y linux-virtual linux-image-extra-virtual
sudo apt purge -y linux*aws

# Reboot the machine
sudo shutdown -r now

Automatic Configuration

Once required kernel modules are installed on the system, for convenience we provide a container which will ensure the appropriate modules are loaded and ready for use at runtime. On Docker installations, you will need to run the init container prior to starting StorageOS. Our installation guides for Kubernetes and OpenShift include this step.

# Load the required kernel modules. The Kubernetes and OpenShift installations include this step.
docker run --name enable_lio                  \
           --privileged                       \
           --rm                               \
           --cap-add=SYS_ADMIN                \
           -v /lib/modules:/lib/modules       \
           -v /sys:/sys:rshared               \
           storageos/init:0.2

Manual Configuration

For those wishing to manage their own kernel configuration, rather than using the init container, perform the following steps:

Ensure kernel modules are all loaded per list above
Ensure configfs is loaded and mounted at /sys/kernel/config

Docs: Availability of IPv6

Mon, 01 Jan 0001 00:00:00 +0000

Availability of IPv6 Address Family

Certain StorageOS components need to be able to listen on a standard dual-stack socket of type AF_INET6. The IPv6 address family must be supported on the server so that this socket can be allocated. StorageOS does not require IPv6 to be configured on the server - no addressing or routing needs to be in place, however StorageOS does need this functionality to be enabled in the kernel.

Docs: Firewalls

Mon, 01 Jan 0001 00:00:00 +0000

Port list

StorageOS daemons listen on specific ports, which we require to be accessible between all nodes in the cluster:

Port Number	TCP/UDP	Use
5701	TCP	gRPC
5703	TCP	DirectFS
5704	TCP	Dataplane Supervisor
5705	TCP	REST API
5711	TCP & UDP	Gossip service
25705-25960	TCP	RWX Volume Endpoints

StorageOS also uses ephemeral ports to dial-out to these ports on other StorageOS nodes. For this reason, outgoing traffic should be enabled.

Firewalls and VPS providers

Some VPS providers (such as Digital Ocean) ship default firewall rulesets which must be updated to allow StorageOS to run. Some example rules are shown below - modify to taste.

UFW

For distributions using UFW, such as RHEL and derivatives:

ufw default allow outgoing
ufw allow 5701:5711/tcp
ufw allow 5711/udp
ufw allow 25705:25960/tcp

Firewalld

For distributions that enable firewalld to control iptables such as some installations of OpenShift.

firewall-cmd --permanent  --new-service=storageos
firewall-cmd --permanent  --service=storageos --add-port=5700-5800/tcp --add-port=25705-25960/tcp
firewall-cmd --add-service=storageos  --zone=public --permanent
firewall-cmd --reload

Iptables

For those using plain iptables:

# Inbound traffic
iptables -I INPUT -i lo -m comment --comment 'Permit loopback traffic' -j ACCEPT
iptables -I INPUT -m state --state ESTABLISHED,RELATED -m comment --comment 'Permit established traffic' -j ACCEPT
iptables -I INPUT -p tcp --dport 5701:5711 -m comment --comment 'StorageOS' -j ACCEPT
iptables -I INPUT -p udp --dport 5711 -m comment --comment 'StorageOS' -j ACCEPT
iptables -I INPUT -p tcp --dport 25705:25960 -m comment --comment 'StorageOS' -j ACCEPT

# Outbound traffic
iptables -I OUTPUT -o lo -m comment --comment 'Permit loopback traffic' -j ACCEPT
iptables -I OUTPUT -d 0.0.0.0/0 -m comment --comment 'Permit outbound traffic' -j ACCEPT

Please ensure that the iptables rules you have added above come before any default DROP or REJECT rules.

Docs: Mount Propagation

Mon, 01 Jan 0001 00:00:00 +0000

Modern versions of Kubernetes, Docker or other Container Runtimes enable mount propagation by default.

StorageOS requires mount propagation enabled to present devices as volumes for containers (see linux kernel documentation here).

Certain versions of docker ship with a systemd manifest with MountFlags set to ‘slave’, thus preventing StorageOS from working. This can be removed or set to ‘shared’ with a systemd drop in:

mkdir -p /etc/systemd/system/docker.service.d/
cat <<EOF > /etc/systemd/system/docker.service.d/mount_propagation_flags.conf
[Service]
MountFlags=shared
EOF

# systemctl daemon-reload
# systemctl restart docker.service

To confirm behaviour, the following command should run without error.

docker run -it --rm -v /mnt:/mnt:shared busybox sh -c /bin/date

Orchestrators such as Kubernetes or OpenShift have their own ways of exposing this setting. Kubernetes 1.10 and OpenShift 3.10 have mount propagation enabled by default. Previous versions require that feature gates are enabled on the Kubernetes master’s controller-manager and apiserver services and in the kubelet service on each node.

Installations of orchestrators using Docker require that mount propagation is enabled for both.

Refer to our installation pages for the orchestrators to see details on how to check and enable mount propagation where appropriate.

Docs: PID Limits

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS recommends that a PID cgroup limit of 32768 be used for StorageOS pods.

Most environments fulfill this prerequisite by default. Check the StorageOS init container logs as shown below to ensure this is the case.

StorageOS pods running in Kubernetes are part of a PID cgroup that may limit the maximum number of PIDs that all containers in the PID cgroup slice can spawn. As the Linux kernel assigns a PID to processes and Light Weight Processes (LWP) a low limit can be easily reached under certain circumstances. The PID limit can be set by the Kubernetes distribution or by the container runtime. Generally the limit is set to the machine wide default limit of 32768 but some environments can set this as low as 1024. A low PID limit may prevent StorageOS from spawning the required threads.

The StorageOS init container runs a script that checks for the PID limit of the PID cGroup slice that the StorageOS pod runs in. If the script finds that the limit is less than 32768 it will log a warning. This warning can be viewed using kubectl to check the init container logs.

$ kubectl -n kube-system logs -l app=storageos,kind=daemonset -c storageos-init
WARNING: Effective max.pids limit (1024) less than RECOMMENDED_MAX_PIDS_LIMIT (32768)

Setting a Kubernetes PID limit

Kubernetes defaults to an unlimited PodPidsLimit, which results in the usage of the machine wide limit; typically 32768.

For information on how to configure the Kubernetes PID limit see the Kubernetes documentation here.

Setting a CRI-O PID limit

Certain orchestrators or setups use CRI-O as the container runtime. Openshift 4.x currently has CRI-O set a PID limit of 1024 by default. To configure the default CRI-O limit in Openshift 4.x see the RedHat documentation here. To configure CRI-O more generally see the CRI-O documentation here.

Docs: StorageOS Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

Use an external Etcd cluster

StorageOS uses the etcd distributed key-value store to store essential cluster metadata and manage distributed configuration state. For production environments and testing of production workloads, we recommend deploying an external etcd cluster. For more details about, and an example of, how to run etcd, see the External etcd Operations page.

It is highly recommended to use external etcd for cloud environments and place the etcd cluster on stable nodes. Placing the etcd on nodes that are recycled often might affect the normal operations of StorageOS.

Etcd low latency IO

It is recommended to run etcd on low-latency disks and keep other IO-intensive applications separate from the etcd nodes. Etcd is very sensitive to IO latency. Thus, the effect of disk contention can cause etcd downtime.

Batch jobs such as backups, builds or application bundling can easily cause a high usage of disks making etcd unstable. It is recommended to run such workloads apart from the etcd servers.

Setup of storage on the hosts

We recommend creating a separate filesystem for StorageOS to mitigate the risk of filling the root filesystem on nodes. This has to be done for each node in the cluster.

Follow the managing host storage best practices page for more details.

Resource reservations

StorageOS resource consumption depends on the workloads and the StorageOS features in use.

The recommended minimum memory reservation for the StorageOS Pods is 512MB for non-production environments. However it is recommended to prepare nodes so StorageOS can operate with at least with 1-2GB of memory. StorageOS frees memory when possible.

For production environments, we recommend 4GB of Memory and 1 CPU as a minimum and to test StorageOS using realistic workloads and tune resources accordingly.

StorageOS Pods resource allocation will impact directly on the availability of volumes in case of eviction or resource limit triggered restart. It is recommended to not limit StorageOS Pods.

StorageOS implements a storage engine, therefore limiting CPU consumption might affect the I/O throughput of your volumes.

Setting a Kubernetes PID limit

StorageOS recommends that a PID cgroup limit of 32768 be set. StorageOS is a multi-threaded application and while most Kubernetes distributions set the PID cgroup limit to 32768, some environments can set a limit as low as 1024. The StorageOS init container will print a log message warning if the PID cgroup limit is too low. See our prerequisites for more information.

Maintain a sufficient number of nodes for replicas to be created

To ensure that a new replica can always be created, an additional node should be available. To guarantee high availability, clusters using Volumes with 1 replica must have at least 3 storage nodes. When using Volumes with 2 replicas, at least 4 storage nodes, 3 replicas, 5 nodes, etc.

Minimum number of storage nodes = 1 (primary) + N (replicas) + 1

For more information, see the section on replication.

StorageOS API username/password

The API grants full access to StorageOS functionality, therefore we recommend that the default administrative password of ‘storageos’ is reset to something unique and strong.

You can change the default parameters by encoding the apiUsername and apiPassword values (in base64) into the storageos-api secret.

To generate a unique password, a technique such as the following, which generates a pseudo-random 24 character string, may be used:

# Generate strong password
PASSWORD=$(cat -e /dev/urandom | tr -dc 'a-zA-Z0-9-!@#$%^&*()_+~' | fold -w 24 | head -n 1)

# Convert password to base64 representation for embedding in a K8S secret
BASE64PASSWORD=$(echo -n $PASSWORD | base64)

Note that the Kubernetes secret containing a strong password must be created before bootstrapping the cluster. Multiple installation procedures use this Secret to create a StorageOS account when the cluster first starts.

StorageOS Pod placement

StorageOS must run on all nodes that will contribute storage capacity to the cluster or that will host Pods which use StorageOS volumes. For production environments, it is recommended to avoid placing StorageOS Pods on Master nodes.

StorageOS is deployed with a DaemonSet controller, and therefore tolerates the standard unschedulable (:NoSchedule) action. If that is the only taint placed on master or cordoned nodes StorageOS pods might start on them (see the Kubernetes docs for more details). To avoid scheduling StorageOS pods on master nodes, you can add an arbitrary taint to them for which the StorageOS DaemonSet won’t have a toleration.

Dedicated instance groups

Cloud environments give users the ability to quickly scale the number of nodes in a cluster in response to their needs. Because of the ephemeral nature of the cloud, StorageOS recommends setting conservative downscaling policies.

For production clusters, it recommended to use dedicated instance groups for Stateful applications that allow the user to set different scaling policies and define StorageOS pools based on node selectors to collocate volumes.

Losing a few nodes at the same time could cause the loss of data even when volume replicas are being used.

Port blocking

StorageOS exposes ports to operate. It is recommended that the ports are not accessible from outside the scope of your cluster.

StorageOS in Docker EE

StorageOS does not support running on Swarm nodes nor on mixed (Kubernetes and Swarm) nodes. StorageOS volumes have to be provisioned and used from Kubernetes nodes.

Docs: Max AIO

Mon, 01 Jan 0001 00:00:00 +0000

As part of the dataplane operation, StorageOS uses Linux AIO (Asynchronous Input Output) contexts to serve I/O requests without blocking. StorageOS requires 4 AIO contexts per deployment (i.e. a StorageOS volume deployment, whether master or replica).

Max AIO prerequisite.

By default there is a maximum number of AIO contexts that can be allocated at once.

The current and maximum number of AIO requests is visible in the virtual files /proc/sys/fs/aio-nr and /proc/sys/fs/aio-max-nr.

The default context limit has been set at 2^16 or 65536. This figure may vary so please check your /proc/sys/fs/aio-max-nr

When aio-nr reaches aio-max-nr the io_setup syscall will fail with EAGAIN. For more information please see the Linux kernel docs here.

Why is this relevant?

As StorageOS requires 4 AIO contexts per deployed volume, there is a limit to the number of volumes that can be deployed per node. Trying to provision additional deployments once the aio-max-nr has been reached will fail as the kernel will be unable to create enough new AIO contexts.

Increasing your AIO context cap.

If your nodes aio-max-nr is set too low you can either provision additional nodes to reduce the number of deployments per node, or increase the aio-max-nr kernel parameter.

You can do this by editing your /etc/sysctl.conf file with the following example line:

fs.aio-max-nr = 1048576

To activate the new settings, run the following command:

$ sysctl -p /etc/sysctl.conf

Docs: Etcd

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS requires an etcd cluster in order to function. For more information on why etcd is required please see our etcd concepts page.

We do not support using the Kubernetes etcd for StorageOS installations.

We provide two methods for installing etcd. For those looking for a quick route to evaluating StorageOS, our first method installs etcd into your Kubernetes cluster using the CoreOS operator. Due to limitations in the CoreOS operator, this installation is not backed by any persistent storage, and therefore is unsuitable for production installations.

For production installations, there is currently no satisfactory way of installing a production-grade etcd cluster inside Kubernetes (although the landscape is changing rapidly, watch this space), and our production guidance remains to install etcd on separate machines outside of your Kubernetes cluster. This method is the best way to ensure a stable StorageOS cluster. Please see our etcd operations page for additional information on deployment best practices and concerns.

Ephemeral pods within Kubernetes (Testing)
External Virtual Machines (Production)

Click the tabs below to select the installation method of your choice.

Testing - Installing Etcd Into Your Kubernetes Cluster

This fast and convenient method is useful for quickly creating an etcd cluster in order to evaluate StorageOS. Do not use it for production installations.

This method uses the CoreOS etcd-operator to install a 3 node etcd cluster within your Kubernetes cluster, in the storageos-etcd namespace. We then install a Kubernetes service in that same namespace.

The official etcd-operator repository also has a backup deployment operator that can help backup etcd data. A restore of the etcd keyspace from a backup might cause issues due to the disparity between the cluster state and its metadata in a different point in time. If you need to restore from a backup after a failure of etcd, contact the StorageOS support team.

Quick Install

For a one command install, the following script uses kubectl to create an etcd cluster in the storageos-etcd namespace. It requires kubectl in the system path, and the context set to the appropriate cluster.

curl -s https://docs.storageos.com/sh/deploy-etcd.sh | bash

Installation Step by Step

For those who would prefer to execute the steps by themselves, they are as follows:

Configure Namespace
```
export NAMESPACE=storageos-etcd
```
Create Namespace
```
kubectl create namespace $NAMESPACE
```

If running in Openshift, an SCC is needed to start Pods

oc adm policy add-scc-to-user anyuid system:serviceaccount:$NAMESPACE:default

Create ClusterRole and ClusterRoleBinding

 $ kubectl -n $NAMESPACE create -f-<<END
 apiVersion: rbac.authorization.k8s.io/v1beta1
 kind: ClusterRoleBinding
 metadata:
   name: etcd-operator
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: ClusterRole
   name: etcd-operator
 subjects:
   - kind: ServiceAccount
     name: default
     namespace: $NAMESPACE
END

$ kubectl -n $NAMESPACE create -f-<<END
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: etcd-operator
rules:
- apiGroups:
  - etcd.database.coreos.com
  resources:
  - etcdclusters
  - etcdbackups
  - etcdrestores
  verbs:
  - "*"
- apiGroups:
  - apiextensions.k8s.io
  resources:
   - customresourcedefinitions
  verbs:
  - "*"
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - endpoints
  - persistentvolumeclaims
  - events
  verbs:
  - "*"
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - "*"
# The following permissions can be removed if not using S3 backup and TLS
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get
END

Deploy Etcd Operator

$ kubectl -n $NAMESPACE create -f - <<END
apiVersion: apps/v1
kind: Deployment
metadata:
  name: etcd-operator
spec:
  selector:
    matchLabels:
      app: etcd-operator
  replicas: 1
  template:
    metadata:
      labels:
        app: etcd-operator
    spec:
      containers:
      - name: etcd-operator
        image: quay.io/coreos/etcd-operator:v0.9.4
        command:
        - etcd-operator
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
END

Wait for the Etcd Operator Pod to start

kubectl -n $NAMESPACE get pod -lapp=etcd-operator

Create the EtcdCluster resource

$ kubectl -n $NAMESPACE create -f - <<END
apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
  name: "storageos-etcd"
spec:
  size: 3
  version: "3.4.7"
  pod:
    etcdEnv:
    - name: ETCD_QUOTA_BACKEND_BYTES
      value: "2147483648"  # 2 GB
    - name: ETCD_AUTO_COMPACTION_RETENTION
      value: "1000" # Keep 1000 revisions (default)
    - name: ETCD_AUTO_COMPACTION_MODE
      value: "revision" # Set the revision mode
    resources:
      requests:
        cpu: 200m
        memory: 300Mi
    securityContext:
      runAsNonRoot: true
      runAsUser: 9000
      fsGroup: 9000
    tolerations:
    - operator: "Exists"
    affinity:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
              - key: etcd_cluster
                operator: In
                values:
                - storageos-etcd
            topologyKey: kubernetes.io/hostname
END

Installation Verification

$ kubectl -n storageos-etcd get pod,svc
NAME                                 READY   STATUS    RESTARTS   AGE
pod/etcd-operator-55978c4587-8kx7b   1/1     Running   0          2h
pod/storageos-etcd-qm9tmrpnlm        1/1     Running   0          2h
pod/storageos-etcd-rzhjdz74hp        1/1     Running   0          2h
pod/storageos-etcd-wvvv2d9g98        1/1     Running   0          2h

NAME                            TYPE        CLUSTER-IP       EXTERNAL-IP PORT(S)             AGE
service/storageos-etcd          ClusterIP   None             <none>      2379/TCP,2380/TCP   22h
service/storageos-etcd-client   ClusterIP   172.30.132.255   <none>      2379/TCP            22h

The URL from the Service storageos-etcd-client.storageos-etcd.svc:2379 will be used later in the StorageOS Cluster CustomResource the kvBackend.address.

Known etcd-operator issues

Etcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. If quorum is lost, etcd nodes stop and etcd marks its contents as read-only. This is because it cannot guarantee that new data will be valid. Quorum is fundamental for etcd operations. When running etcd using the CoreOS Operator, it is important to consider that a loss of quorum could arise from etcd pods being evicted from nodes.

Operations such as Kubernetes Upgrades with rolling node pools could cause a total failure of the etcd cluster as nodes are discarded in favor of new ones.

A 3 etcd node cluster can survive losing one node and recover, a 5 node cluster can survive the loss of two nodes. Loss of further nodes will result in quorum being lost.

The etcd-operator doesn’t support a full stop of the cluster. Stopping the etcd cluster causes the loss of all the etcd keystore and make StorageOS unable to perform metadata changes.

Production - Etcd on External Virtual Machines

For production installations, StorageOS strongly recommends running etcd outside of Kubernetes on a minimum of 3 dedicated virtual machines. This topology offers strong guarantees of resilience and uptime. We recommend this architecture in all environments, including those where Kubernetes is being deployed as a managed service.

StorageOS doesn’t require a high performance etcd cluster, as the throughput of metadata to the cluster is low. However, we recommend a careful assessment of IOPS capacity best practices to ensure that etcd operates normally.

Depending on the level of redundancy you feel comfortable with you can install etcd on the Kubernetes Master nodes. Take extreme care to avoid collisions of the StorageOS etcd installation with the Kubernetes etcd when using the Kubernetes Master nodes. Precautions such as changing the default configuration for the client and peer ports, and ensuring the etcd data directory is modified. The ansible playbook below will default the etcd installation directory to /var/lib/storageos-etcd.

You can choose between two installation options.

Installation - Manual

This section documents the steps required for manual installation of etcd using standard package management commands and systemd manifests.

Repeat the following steps on all the nodes that will run etcd as a systemd service.

Configure Etcd version and ports
```
export ETCD_VERSION="3.4.9"
export CLIENT_PORT="2379"
export PEERS_PORT="2380"
```
If targeting Kubernetes Master nodes, you must change CLIENT_PORT, PEERS_PORT

Download Etcd from CoreOS official site

curl -L https://github.com/coreos/etcd/releases/download/v${ETCD_VERSION}/etcd-v${ETCD_VERSION}-linux-amd64.tar.gz -o /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz
mkdir -p /tmp/etcd-v${ETCD_VERSION}-linux-amd64
tar -xzvf /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz -C /tmp/etcd-v${ETCD_VERSION}-linux-amd64 --strip-components=1
rm /tmp/etcd-${ETCD_VERSION}-linux-amd64.tar.gz

Install Etcd binaries

cd /tmp/etcd-v${ETCD_VERSION}-linux-amd64
mv etcd /usr/local/sbin/etcd3
mv etcdctl /usr/local/sbin/etcdctl
chmod 0755 /usr/local/sbin/etcd3 /usr/local/sbin/etcdctl

Set up persistent Etcd data directory
```
mkdir /var/lib/storageos-etcd
```

Create the systemd environment file

On all nodes that will run etcd create a systemd environemnt file /etc/etcd.conf which has the IPs of all the nodes. The NODE_IP will need to change to correspond to the node IP where the environment file resides. NODE1_IP, NODE2_IP and NODE3_IP will remain the same across all three files.

$ cat <<END > /etc/etcd.conf
# NODE_IP is the IP of the node where this file resides.
NODE_IP=10.64.10.228
# Node 1 IP
NODE1_IP=10.64.10.228
# Node 2 IP
NODE2_IP=10.64.14.233
# Node 3 IP  
NODE3_IP=10.64.12.111
CLIENT_PORT=${CLIENT_PORT}
PEERS_PORT=${PEERS_PORT}
END

# Verify that variables are expanded in the file
$ cat /etc/etcd.conf

Create the systemd unit file for etcd3 service

Create a systemd unit file /etc/systemd/system/etcd3.service with the following information:

[Unit]
Description=etcd3
Documentation=https://github.com/coreos/etcd
Conflicts=etcd2.service

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
EnvironmentFile=/etc/etcd.conf

ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \
    --heartbeat-interval 500 \
    --election-timeout 5000 \
    --max-snapshots 10 \
    --max-wals 10 \
    --data-dir /var/lib/storageos-etcd \
    --quota-backend-bytes 8589934592 \
    --snapshot-count 100000 \
    --auto-compaction-retention 20000 \
    --auto-compaction-mode revision \
    --initial-cluster-state new \
    --initial-cluster-token etcd-token \
    --listen-client-urls http://${NODE_IP}:${CLIENT_PORT},http://127.0.0.1:${CLIENT_PORT} \
    --advertise-client-urls http://${NODE_IP}:${CLIENT_PORT} \
    --listen-peer-urls http://${NODE_IP}:${PEERS_PORT} \
    --initial-advertise-peer-urls http://${NODE_IP}:${PEERS_PORT} \
    --initial-cluster etcd-${NODE1_IP}=http://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=http://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=http://${NODE3_IP}:${PEERS_PORT}


[Install]
WantedBy=multi-user.target

$NODE_IP is the IP address of the machine you are installing etcd on.`

Note that setting the advertise-client-urls incorrectly will cause any client connection to fail. StorageOS will fail to communicate to Etcd.

If enabling TLS, it is recomended to generate your own CA certificate and key. You will need to distribute the keys and certificates for the client auth on all etcd nodes. Moreover, the ExecStart value should look as below:

    ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \
    --heartbeat-interval 500 \
    --election-timeout 5000 \
    --max-snapshots 10 \
    --max-wals 10 \
    --data-dir /var/lib/storageos-etcd \
    --quota-backend-bytes 8589934592 \
    --snapshot-count 100000 \
    --auto-compaction-retention 20000 \
    --auto-compaction-mode revision \
    --peer-auto-tls \
    --client-cert-auth --trusted-ca-file=/path/to/client-cert.pem \
    --cert-file=/path/to/ca.pem \
    --key-file=/path/to/client-key.pem \
    --initial-cluster-state new \
    --initial-cluster-token etcd-token \
    --listen-client-urls https://${NODE_IP}:${CLIENT_PORT} \
    --advertise-client-urls https://${NODE_IP}:${CLIENT_PORT} \
    --listen-peer-urls https://${NODE_IP}:${PEERS_PORT} \
    --initial-advertise-peer-urls https://${NODE_IP}:${PEERS_PORT} \
    --initial-cluster etcd-${NODE1_IP}=https://${NODE1_IP}:${PEERS_PORT},etcd-${NODE2_IP}=https://${NODE2_IP}:${PEERS_PORT},etcd-${NODE3_IP}=https://${NODE3_IP}:${PEERS_PORT}

Reload and start the etc3 systemd service

$ systemctl daemon-reload
$ systemctl enable etcd3.service
$ systemctl start  etcd3.service

Installation Verification

The etcdctl binary is installed at /usr/local/bin on the nodes.

$ ssh $NODE # Any node running the new etcd
$ ETCDCTL_API=3 etcdctl --endpoints=http://127.0.0.1:${CLIENT_PORT} member list # $NODE_IP - the IP of the node
66946cff1224bb5, started, etcd-b94bqkb9rf,  http://172.28.0.1:2380, http://172.28.0.1:2379
17e7256953f9319b, started, etcd-gjr25s4sdr, http://172.28.0.2:2380, http://172.28.0.2:2379
8b698843a4658823, started, etcd-rqdf9thx5p, http://172.28.0.3:2380, http://172.28.0.3:2379

Read the etcd operations page for our etcd recommendations.

Installation - Ansible

For a repeatable and automated installation, use of a configuration management tool such as ansible is recommended. StorageOS provides an ansible playbook to help you deploy etcd on standalone virtual machines.

Clone StorageOS deployment repository

git clone https://github.com/storageos/deploy.git
cd k8s/deploy-storageos/etcd-helpers/etcd-ansible-systemd

Edit the inventory file

The inventory file targets the nodes that will run etcd. The file hosts is an example of such an inventory file.

$ cat hosts
[nodes]
centos-1 ip="10.64.10.228" fqdn="ip-10-64-10-228.eu-west-2.compute.internal"
centos-2 ip="10.64.14.233" fqdn="ip-10-64-14-233.eu-west-2.compute.internal"
centos-3 ip="10.64.12.111" fqdn="ip-10-64-12-111.eu-west-2.compute.internal"

# Edit the inventory file
$ vi hosts # Or your own inventory file

The ip or fqdn are used to expose the advertise-client-urls of Etcd. Failing to provide valid ip/fqdn will cause any client connection to fail. StorageOS will fail to communicate to Etcd.

Edit the etcd configuration

If targeting Kubernetes Master nodes, you must change etcd_port_client, etcd_port_peers

$ cat group_vars/all
etcd_version: "3.4.9"
etcd_port_client: "2379"
etcd_port_peers: "2380"
etcd_quota_bytes: 8589934592  # 8 GB
etcd_auto_compaction_mode: "revision"
etcd_auto_compaction_retention: "1000"
members: "{{ groups['nodes'] }}"
installation_dir: "/var/lib/storageos-etcd"
advertise_format: 'fqdn' # fqdn || ip
backup_file: "/tmp/backup.db"

tls:
  enabled: false
  ca_common_name: "eu-west-2.compute.internal"
  etcd_common_name: "*.eu-west-2.compute.internal"
  cert_dir: "/etc/etcdtls"
  ca_cert_file: "etcd-ca.pem"
  etcd_server_cert_file: "server.pem"
  etcd_server_key_file: "server-key.pem"
  etcd_client_cert_file: "etcd-client.crt"
  etcd_client_key_file: "etcd-client.key"

$ vi group_vars/all

Choose between using IP addressing or FQDN in the advertise_format parameter. It allows you to decide how Etcd advertises its address to clients. This is particularly relevant when using TLS.

If enabling TLS, it is recomended to generate your own CA certificate and key. You can do it by generating the CA from the machine running Ansible by: ansible-playbook create_ca.yaml.

Install
```
ansible-playbook -i hosts install.yaml
```

Installation Verification

The playbook installs the etcdctl binary on the nodes, at /usr/local/bin.

$ ssh $NODE # Any node running the new etcd
$ ETCDCTL_API=3 etcdctl --endpoints=127.0.0.1:2379 member list
66946cff1224bb5, started, etcd-b94bqkb9rf,  http://172.28.0.1:2380, http://172.28.0.1:2379
17e7256953f9319b, started, etcd-gjr25s4sdr, http://172.28.0.2:2380, http://172.28.0.2:2379
8b698843a4658823, started, etcd-rqdf9thx5p, http://172.28.0.3:2380, http://172.28.0.3:2379

Benefits of Running External to Kubernetes

Etcd is a distributed key-value store database focused on strong consistency. That means that etcd nodes perform operations across the cluster to ensure quorum. In the case that quorum is lost, an etcd node stops and marks its contents as read-only. Another peer might have a newer version that has not been committed to the database. Quorum is fundamental for etcd operations.

In a Kubernetes environment, applications are scheduled across and in some scenarios such as “DiskPressure” they may need to be evicted from a node, and be scheduled onto a different node. With an application such as etcd, the scenario described can result in quorum being lost, making the cluster unable to recover automatically. Usually a 3 node etcd cluster can survive losing one node and recover. However, losing a second node at the same time or even having a network partition between them will result in quorum lost.

Bind Etcd IPs to Kubernetes Service

Kubernetes external services use a DNS name to reference external endpoints, making them easy to reference from inside the cluster. You can use the example from the helper github repository to deploy the external Service. Using an external service can make monitoring of etcd from Prometheus easier.

Using Etcd with StorageOS

During installation of StorageOS the kvBackend.address parameter of the StorageOS operator is used to specify the address of the etcd cluster. See the StorageOS cluster operator configuration examples for more information.

Docs: Support

Mon, 01 Jan 0001 00:00:00 +0000

When you need support, please raise a ticket via our Help Desk portal. It is important to select the priority of your ticket in accordance with the severity:

Severity 1 - Urgent
Severity 2 - High
Severity 3 - Medium
Severity 4 - Low

This helps us route and prioritise the ticket accordingly. You can find definitions of our support severity levels on our website.

Responses to tickets will be cc’d via email.

For personal support and general enquiries, please join our public Slack channel.

Information to include in tickets

To help us provide effective support, we request that you provide as much information as possible when contacting us. The list below is a suggested starting point. Additionally, please include anything specific, such as log entries, that may help us debug your issue.

Platform

Cloud provider/Bare metal
OS distribution and version
Kernel version
docker version and installation procedure (distro packages or docker install)

StorageOS

Version of StorageOS
storageos get nodes
storageos get volumes
storageos describe volume VOL_ID # in case of issues with a specific volume

Version and installation method
Managed or self managed?
kubectl -n kube-system get pod
kubectl -n kube-system logs -lapp=storageos -c storageos
kubectl -n kube-system get storageclass
Specific for your namespaces: kubectl describe pvc PVC_NAME
Specific for your namespaces: kubectl describe pod POD_NAME

Environment Changes

Details of any recent changes to your environment such as planned maintenance, node reboots, network failures, etcd outage, etc.. This can help speed up ticket triage and resolution considerably

StorageOS Support Bundle

StorageOS provides the ability to generate a support bundle that aggregates cluster information. See Support Bundle for a list of what is included.

StorageOS engineers might ask for a support bundle to be generated during support cases.

The information in the bundle is used only for support purposes, and will be removed once it is no longer needed. If the information is sensitive and can’t be given to StorageOS, please make sure that the support engineers have as much information about your environment as possible.

Refer to the Support Bundle documentation page for details of how to generate a bundle.

Docs: Self Evaluation Guide

Mon, 01 Jan 0001 00:00:00 +0000

Our self-evaluation guide is a step by step recipe for installing and testing StorageOS. This guide is divided into three sections:

Installation - install StorageOS with a single command
Feature Testing - short walkthrough of some of our features
Benchmarking - a recipe to benchmark StorageOS on your infrastructure

For more comprehensive documentation including installation advice for complex setups, operational guides, and use-cases, please see our main documentation site.

Support for Self Evaluations

Should you have questions or require support, there are several ways to get in touch with us. The fastest way to get in touch is to join our public Slack channel. You can also get in touch via email to info@storageos.com.

Installation

In this document we detail a simple installation suitable for evaluation purposes. The etcd we install uses a 3 node cluster with ephemeral storage, and as such is not suitable for production workloads. However, for evaluation purposes it should be sufficient. For production deployments, please see our main documentation pages.

A standard StorageOS installation uses the StorageOS operator, which performs most platform-specific configuration for you. The StorageOS operator has been certified by Red Hat and is open source.

The basic installation steps are:

Check prerequisites
Install an etcd cluster
Install the StorageOS Operator
Install StorageOS by applying a CR for the Operator to action

Prerequisites

While we do not require custom kernel modules or additional userspace tooling, StorageOS does have a few basic prerequisites that are met by default by most modern distributions:

At least 1 CPU core, 2GB RAM free.
A Kubernetes release within the four most recent versions.
TCP ports 5701-5710 and TCP & UDP 5711 open between all nodes in the cluster.
A 64bit supported operating system - StorageOS can run without additional packages in Debian 9, RancherOS, RHEL7.5,8 and CentOS7,8 and need the package linux-image-extra for Ubuntu.
Mainline kernel modules target_core_mod, tcp_loop, target_core_file, target_core_user, configfs, and ui. These are present by default on most modern linux distributions, and can be installed with standard package managers. See our system configuration page for instructions.

Quick Installation

For those feeling brave and/or lucky, we have encapsulated the installation instructions in this guide into a short bash script. As with similar instructions elsewhere on the internet, caution is advised here. The script is liberally commented and we recommend that you inspect it in your text editor of choice before running.

Run the following command where kubectl is installed and with the context set for your Kubernetes cluster

curl -sL https://storageos.run | bash

Note that this script deploys a cluster with etcd on transient storage, suitable for evaluation purposes only. Do not use this cluster for production workloads.

Your new cluster can be used straight away. You can now jump to the Provision a Volume section below to provision your first volume and license the cluster.

Installing StorageOS Step By Step

This section describes the installation steps in some more detail.

Install etcd

Here we install a 3 node etcd cluster using the CoreOS operator to install on ephemeral storage within your Kubernetes cluster. This is suitable for evaluation purposes. For production deployment instructions, please see our production etcd page.

Download repo

$ git clone https://github.com/coreos/etcd-operator.git

Configure NS, Role and RoleBinding

$ export ROLE_NAME=etcd-operator
$ export ROLE_BINDING_NAME=etcd-operator
$ export NAMESPACE=etcd

Create Namespace
```
$ kubectl create namespace $NAMESPACE
```

Deploy Operator

$ ./etcd-operator/example/rbac/create_role.sh
$ kubectl -n $NAMESPACE create -f - <<END
apiVersion: apps/v1
kind: Deployment
metadata:
  name: etcd-operator
spec:
  selector:
    matchLabels:
      app: etcd-operator
  replicas: 1
  template:
    metadata:
      labels:
        app: etcd-operator
    spec:
      containers:
      - name: etcd-operator
        image: quay.io/coreos/etcd-operator:v0.9.4
        command:
        - etcd-operator
        env:
        - name: MY_POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: MY_POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
END

Create the EtcdCluster resource

$ kubectl -n $NAMESPACE create -f - <<YAML
apiVersion: "etcd.database.coreos.com/v1beta2"
kind: "EtcdCluster"
metadata:
 name: "storageos-etcd"
spec:
 size: 3 # Set to 1 if insufficient nodes
 version: "3.4.7"
 pod:
   etcdEnv:
   - name: ETCD_QUOTA_BACKEND_BYTES
     value: "2147483648"  # 2 GB 
   - name: ETCD_AUTO_COMPACTION_RETENTION
     value: "1000" # Keep 1000 revisions
   - name: ETCD_AUTO_COMPACTION_MODE
     value: "revision" # Set the revision mode
   resources:
     requests:
       cpu: 200m
       memory: 300Mi
   securityContext:
     runAsNonRoot: true
     runAsUser: 9000
     fsGroup: 9000
   tolerations:
   - operator: "Exists" # add tolerations to match all taints on the master nodes where etcd will be deployed
   affinity:
     podAntiAffinity:
       preferredDuringSchedulingIgnoredDuringExecution:
       - weight: 100
         podAffinityTerm:
           labelSelector:
             matchExpressions:
             - key: etcd_cluster
               operator: In
               values:
               - storageos-etcd
           topologyKey: kubernetes.io/hostname
YAML

This creates a 3 node etcd cluster and associated service in Kubernetes. We can use the service endpoint when creating the StorageOS cluster.

Install StorageOS Operator

In order to install the StorageOS operator download the requisite yaml manifests or apply them with kubectl.

$ kubectl create -f https://github.com/storageos/cluster-operator/releases/download/v2.4.4/storageos-operator.yaml

You can verify the operator is running using the following command

$ kubectl get pods -n storageos-operator

Install StorageOS

Once the StorageOS operator has been installed a StorageOS cluster can be generated by creating a StorageOSCluster resource.

A StorageOSCluster resource describes the state of the StorageOS cluster that is desired and the StorageOS operator will create the desired StorageOS cluster. For examples of StorageOSCluster resources please see our examples page here. For a full list of the configurable spec parameters of the StorageOSCluster resource please see here.

Create a secret defining the API username and password

$ kubectl create -f - <<END
apiVersion: v1
kind: Secret
metadata:
 name: "storageos-api"
 namespace: "storageos-operator"
 labels:
   app: "storageos"
type: "kubernetes.io/storageos"
data:
 # echo -n '<secret>' | base64
 apiUsername: c3RvcmFnZW9z
 apiPassword: c3RvcmFnZW9z
 # CSI Credentials
 csiProvisionUsername: c3RvcmFnZW9z
 csiProvisionPassword: c3RvcmFnZW9z
 csiControllerPublishUsername: c3RvcmFnZW9z
 csiControllerPublishPassword: c3RvcmFnZW9z
 csiNodePublishUsername: c3RvcmFnZW9z
 csiNodePublishPassword: c3RvcmFnZW9z
 csiControllerExpandUsername: c3RvcmFnZW9z
 csiControllerExpandPassword: c3RvcmFnZW9z
END

Create a StorageOSCluster resource

$ kubectl create -f - <<END
apiVersion: storageos.com/v1
kind: StorageOSCluster
metadata:
 name: example-storageoscluster
 namespace: "storageos-operator"
spec:
 # StorageOS Pods are in kube-system by default
 secretRefName: "storageos-api"
 secretRefNamespace: "storageos-operator"
 k8sDistro: "upstream"  # Set the Kubernetes distribution for your cluster (upstream, eks, aks, gke, rancher, dockeree)
 # storageClassName: fast # The storage class creates by the StorageOS operator is configurable
 kvBackend:
   address: "storageos-etcd-client.etcd.svc.cluster.local:2379"
END

Confirm that the cluster has been created and that StorageOS pods are running
```
$ kubectl -n kube-system get pods
```
StorageOS pods enter a ready state after a minimum of 45s has passed.

Deploy the StorageOS CLI as a container

$ kubectl -n kube-system run \
--image storageos/cli:v2.4.4 \
--restart=Never                          \
--env STORAGEOS_ENDPOINTS=storageos:5705 \
--env STORAGEOS_USERNAME=storageos       \
--env STORAGEOS_PASSWORD=storageos       \
--command cli                            \
-- /bin/sh -c "while true; do sleep 999999; done"

Provision a StorageOS Volume

Now that we have a working StorageOS cluster, we can provision a volume to test everything is working as expected.

Create a PVC

$ kubectl create -f - <<END
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-1
spec:
  storageClassName: "fast"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
END

Create 2 replicas by labeling your PVC

$ kubectl label pvc pvc-1 storageos.com/replicas=2

Verify that the volume and replicas were created with the CLI

pvc-1 should be listed in the CLI output
```
$ kubectl -n kube-system exec -it cli -- storageos get volumes
```

Create a pod that consumes the PVC

$ kubectl create -f - <<END
apiVersion: v1
kind: Pod
metadata:
 name: d1
spec:
 containers:
   - name: debian
     image: debian:9-slim
     command: ["/bin/sleep"]
     args: [ "3600" ]
     volumeMounts:
       - mountPath: /mnt
         name: v1
 volumes:
   - name: v1
     persistentVolumeClaim:
       claimName: pvc-1
END

Check that the pod starts successfully. If the pod starts successfully then the StorageOS cluster is working correctly
```
$ kubectl get pod d1 -w
```
The pod mounts a StorageOS volume under /mnt so any files written there will persist beyond the lifetime of the pod. This can be demonstrated using the following commands.

Execute a shell inside the pod and write some data to a file

$ kubectl exec -it d1 -- bash
root@d1:/# echo Hello World! > /mnt/hello
root@d1:/# cat /mnt/hello

Hello World! should be printed out.

Delete the pod
```
$ kubectl delete pod d1
```

Recreate the pod

$ kubectl create -f - <<END
apiVersion: v1
kind: Pod
metadata:
 name: d1
spec:
 containers:
   - name: debian
     image: debian:9-slim
     command: ["/bin/sleep"]
     args: [ "3600" ]
     volumeMounts:
       - mountPath: /mnt
         name: v1
 volumes:
   - name: v1
     persistentVolumeClaim:
       claimName: pvc-1
END

Open a shell inside the pod and check the contents of /mnt/hello
```
$ kubectl exec -it d1 -- cat /mnt/hello
```
Hello World! should be printed out.

License cluster

Newly installed StorageOS clusters must be licensed within 24 hours. Our personal license is free, and supports up to 1TiB of provisioned storage.

You will need access to the StorageOS API on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):

$ kubectl port-forward -n kube-system svc/storageos 5705

Now follow the instructions on our operations licensing page to obtain and apply a license.

Installation of StorageOS is now complete.

StorageOS Features

Now that you have a fully functional StorageOS cluster we will explain some of our features that may be of use to you as you complete application and synthetic benchmarks.

StorageOS features are all enabled/disabled by applying labels to volumes. These labels can be passed to StorageOS via persistent volume claims (PVCs) or can be applied to volumes using the StorageOS CLI or GUI.

The following is not an exhaustive feature list but outlines features which are commonly of use during a self-evaluation.

Volume Replication

StorageOS enables synchronous replication of volumes using the storageos.com/replicas label.

The volume that is active is referred to as the master volume. The master volume and its replicas are always placed on separate nodes. In fact if a replica cannot be placed on a node without a replica of the same volume, the volume will fail to be created. For example, in a three node StorageOS cluster a volume with 3 replicas cannot be created as the third replica cannot be placed on a node that doesn’t already contain a replica of the same volume.

See our replication documentation for more information on volume replication.

To test volume replication create the following PersistentVolumeClaim

$ kubectl create -f - <<END
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: pvc-replicated
 labels:
   storageos.com/replicas: "1"
spec:
 storageClassName: "fast"
 accessModes:
 - ReadWriteOnce
 resources:
   requests:
     storage: 5Gi
END

Note that volume replication is enabled by setting the storageos.com/replicas label on the volume.

Confirm that a replicated volume has been created by using the StorageOS CLI or UI
```
$ kubectl -n kube-system exec -it cli -- storageos get volumes
```

Create a pod that uses the PVC

$ kubectl create -f - <<END
apiVersion: v1
kind: Pod
metadata:
  name: replicated-pod
spec:
  containers:
   - name: debian
     image: debian:9-slim
     command: ["/bin/sleep"]
     args: [ "3600"  ]
     volumeMounts:
       - mountPath: /mnt
         name: v1
  volumes:
   - name: v1
     persistentVolumeClaim:
       claimName: pvc-replicated
END

Write data to the volume

$ kubectl exec -it replicated-pod -- bash
root@replicated-pod:/# echo Hello World! > /mnt/hello
root@replicated-pod:/# cat /mnt/hello

Hello World! should be printed out.

Find the location of the master volume and shutdown the node

Shutting down a node causes all volumes, with online replicas, on the node to be evicted. For replicated volumes this immediately promotes a replica to become the new master.

$ kubectl get pvc
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-replicated Bound    pvc-29e2ad6e-8c4e-11e9-8356-027bfbbece86   5Gi        RWO            fast           1m

$ kubectl exec -it -n kube-system cli -- storageos get volumes
NAMESPACE  NAME                                      SIZE     LOCATION              ATTACHED ON   REPLICAS  AGE
default    pvc-4e796a62-0271-45f9-9908-21d58789a3fe  5.0 GiB  kind-worker (online)  kind-worker2  1/1       26 seconds ago

Check the location of the master volume and notice that it is on a new node. If the pod that mounted the volume was located on the same node that was shutdown then the pod will need to be recreated.

$ kubectl exec -it -n kube-system cli -- storageos get volumes
NAMESPACE  NAME                                      SIZE     LOCATION               ATTACHED ON   REPLICAS  AGE
default    pvc-4e796a62-0271-45f9-9908-21d58789a3fe  5.0 GiB  kind-worker2 (online)  kind-worker2  1/1       46 seconds ago

Check that the data is still accessible to the pod
```
$ kubectl exec -it replicated-pod -- bash
root@replicated-pod:/# cat /mnt/hello
```
Hello World! should be printed out.

Benchmarking

Benchmarking storage is a complex topic. Considering the many books and papers that have been written about benchmarking, we could write many paragraphs here and only begin to scratch the surface.

Taking this complexity into account we present recipes for both synthetic benchmarks using FIO, and a sample application benchmark to test PostgreSQL using pgbench. The approaches are complementary - synthetic benchmarks allow us to strictly control the parameters of the IO we put through the system in order to stress various aspects of it. Application benchmarks allow us to get a sense of the performance of the system when running an actual representative workload - which of course is the ultimate arbiter of performance.

Despite the inherent complexity of benchmarking storage there are a few general considerations to keep in mind.

Considerations

Local vs. Remote Volumes

When a workload is placed on the same node as a volume, there is no network round trip for IO, and performance is consequently improved. When considering the performance of StorageOS it is important to evaluate both local and remote volumes; since for certain workloads we want the high performance of a local attachment, but we also desire the flexibility of knowing that our less performance-sensitive workloads can run from any node in the cluster.

The Effect of Synchronous Replication

Synchronous replication does not impact the read performance of a volume, but it can have a significant impact on the write performance of the volume, since all writes must be propagated to replicas before being acked to the application. The impact varies in proportion to the inter-node latency. For an inter-node latency of 1ms, we have a max ceiling of 1000 write IOPS, and in reality a little less than that to allow for processing time on the nodes. This is less concerning then it may first appear, since many applications will issue multiple writes in parallel (known as increasing the queue depth).

Synthetic Benchmarks

Prerequisites

Kubernetes cluster with a minimum of 3 nodes and a minimum of 30 Gb available capacity
StorageOS CLI running as a pod in the cluster

Synthetic benchmarks using tools such as FIO are a useful way to begin measuring StorageOS performance. While not fully representative of application performance, they allow us to reason about the performance of storage devices without the added complexity of simulating real world workloads, and provide results easily comparable across platforms.

FIO has a number of parameters that can be adjusted to simulate a variety of workloads and configurations. Parameters that are particularly important are:

Block Size - the size of the IO units performed.
Queue Depth - the amount of concurrent IOs in flight
IO Pattern - access can be random across the disk, or sequentially. IO subsystems typically perform better with sequential IO, because of the effect of read ahead caches, and other factors

For this self-evaluation we will run a set of tests based on the excellent DBench, which distills the numerous FIO options into a series of well-crafted scenarios:

Random Read/Write IOPS and BW - these tests measure the IOPS ceiling (with a 4k block size) and bandwidth ceiling (with a 128K block size) of the volume using a random IO pattern and high queue depth
Average Latency - these tests measure the IO latency of the volume under favourable conditions using a random access pattern, low queue depth and low block size
Sequential Read/Write - these tests measure the sequential read/write throughput of the volume with a high queue depth and high block size
Mixed Random Read/Write IOPS - these tests measure the performance of the volume under a 60/40 read/write workload using a random access pattern and low blocksize

For convenience we present a single script to run the scenarios using local and remote volumes both with and without a replica. Deploy the FIO tests for the four scenarios using the following command:

curl -sL https://docs.storageos.com/sh/deploy-synthetic-benchmarks.sh | bash

The script will take approximately 20 minutes to complete, and will print the results to STDOUT.

The exact results observed will depend on the particular platform and environment, but the following trends should be observable:

local volumes perform faster than remote volumes
read performance is independent of the number of replicas
write performance is dependent on the number of replicas

Application Benchmarks

While synthetic benchmarks are useful for examining the behaviour of StorageOS with very specific workloads, in order to get a realistic picture of StorageOS performance actual applications should be tested.

Many applications come with test suites which provide standard workloads. For best results, test using your application of choice with a representative configuration and real world data.

As an example of benchmarking an application the following steps lay out how to benchmark a Postgres database backed by a StorageOS volume.

Start by cloning the StorageOS use cases repository. Note this is the same repository that we cloned earlier so if you already have a copy just cd storageos-usecases/pgbench.
```
$ git clone https://github.com/storageos/use-cases.git storageos-usecases
```
Move into the Postgres examples folder
```
$ cd storageos-usecases/pgbench
```
Decide which node you want the pgbench pod and volume to be located on. The node needs to be labelled app=postgres
```
$ kubectl label node <NODE> app=postgres
```
Then set the storageos.com/hint.master label in 20-postgres-statefulset.yaml file to match the StorageOS nodeID for the node you have chosen before creating all the files. The StorageOS nodeID can be obtained using the cli and doing a describe node
```
$ kubectl create -f .
```
Confirm that Postgres is up and running
```
$ kubectl get pods -w -l app=postgres
```
Use the StorageOS CLI or the GUI to check the master volume location and the mount location. They should match
```
$ kubectl -n kube-system exec -it cli -- storageos get volumes
```

Exec into the pgbench container and run pgbench

$ kubectl exec -it pgbench -- bash -c '/opt/cpm/bin/start.sh'

Conclusion

After completing these steps you will have benchmark scores for StorageOS. Please keep in mind that benchmarks are only part of the story and that there is no replacement for testing actual production or production like workloads.

StorageOS invites you to provide feedback on your self-evaluation to the slack channel or by directly emailing us at info@storageos.com]

Docs: Apply

Mon, 01 Jan 0001 00:00:00 +0000

Apply changes to resources

$ storageos apply --help
Make changes to existing resources

Usage:
  storageos apply [command]

Available Commands:
  licence     Apply a product licence to the cluster

Flags:
  -h, --help   help for apply

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Use "storageos apply [command] --help" for more information about a command.

Apply License

$ storageos apply licence --help

Apply a product licence to the cluster

Usage:
  storageos apply licence [flags]

Aliases:
  licence, license

Examples:

$ storageos apply licence --from-file <path-to-licence-file>

$ echo "<licence file contents>" | storageos apply licence --from-stdin 


Flags:
      --cas string         make changes to a resource conditional upon matching the provided version
      --from-file string   reads a StorageOS product licence from a specified file path
      --from-stdin         reads a StorageOS product licence from the standard input
  -h, --help               help for licence

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Docs: Attach

Mon, 01 Jan 0001 00:00:00 +0000

$ storageos attach --help

Attach a volume to a node

Usage:
  storageos attach [flags]

Examples:

$ storageos attach --namespace my-namespace-name my-volume my-node


Flags:
  -h, --help   help for attach

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Docs: AWS EKS

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS is fully compatible with AWS EKS. To install StorageOS on EKS, please follow our Kubernetes installation instructions page.

An EKS deployment of Kubernetes uses AWS Linux by default with an optimized kernel. As the requisite kernel modules are not available for StorageOS to use TCMU, FUSE will be used as a fallback. Using FUSE instead of TCMU has performance implications.

For more details about the OS Distributions check the System Configuration page.

Docs: Azure AKS

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS is fully compatible with Azure AKS. To install StorageOS on AKS please follow our Kubernetes installation instructions page.

An AKS deployment of Kubernetes uses Ubuntu by default with an optimized kernel. All versions of Ubuntu with a kernel version later than4.15.0-1029-azure meet the StorageOS prerequisites.

Docs: Cassandra

Mon, 01 Jan 0001 00:00:00 +0000

Cassandra is a popular distributed NoSQL open source database.

Using StorageOS persistent volumes with Cassandra means that if a Cassandra pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up the pod data is immediately available. Should Kubernetes schedule the Cassandra pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not a StorageOS master is located on the same node.

As Cassandra has features to allow it to handle replication careful consideration of whether to allow StorageOS or Cassandra to handle replication is required.

Before you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.

Deploying Cassandra on Kubernetes

You can find the latest files in the StorageOS use cases repository

git clone https://github.com/storageos/use-cases.git storageos-usecases

StatefulSet defintion

apiVersion: apps/v1
kind: StatefulSet
metadata:
 name: cassandra
spec:
 selector:
   matchLabels:
     app: cassandra
 serviceName: cassandra
 replicas: 3
 ...
 spec:
      ...
      volumeMounts:
       - name: cassandra-data
         mountPath: /var/lib/cassandra
 ...
volumeClaimTemplates:
 - metadata:
     name: cassandra-data
   spec:
     accessModes: ["ReadWriteOnce"]
     storageClassName: "fast" # StorageOS storageClass
     resources:
       requests:
         storage: 5Gi

This excerpt is from the StatefulSet definition. This file contains the VolumeClaim template that will dynamically provision storage, using the StorageOS storage class. Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.

Move into the Cassandra examples folder and create the objects
```
cd storageos-usecases
kubectl create -f ./cassandra
```

Confirm Cassandra is up and running.

$ kubectl get pods -w -l app=cassandra
NAME          READY   STATUS    RESTARTS   AGE
cassandra-0   1/1     Running   0          8m32s
cassandra-1   1/1     Running   0          7m51s
cassandra-2   1/1     Running   0          6m36s

Connect to the Cassandra client pod and connect to the Cassandra server through the service

$ kubectl exec -it cassandra-0 -- cqlsh cassandra-0.cassandra
Connected to K8Demo at cassandra-0.cassandra:9042.
[cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> SELECT cluster_name, listen_address FROM system.local;

 cluster_name | listen_address
--------------+----------------
       K8Demo |   100.96.7.124

(1 rows)

Docs: Cluster Topologies

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS makes it possible for you to organise your cluster in multiple ways, depending on your priorities and needs. The approaches below are idealised representations of possible StorageOS clusters and can be mixed, modified and changed at execution time.

StorageOS performs file I/O over the network, which is how we ensure that your data is always available throughout your cluster. This also affords the user certain possibilities for organising your cluster, as we suggest below.

Hyperconverged Model

In this topology all nodes can store data and present data. This gives maximum flexibility to the StorageOS and Kubernetes schedulers, and maximum choice for pod placement. No matter how you deploy your workloads they will be able to store and access data on every node.

By default we place workloads locally where possible - this means that we default to a hybrid of the hyperconverged/high-performance models, maximising performance without extra effort.

Centralised Storage Model

In this cluster topology volume data is placed on a particular subset of nodes, while the remaining nodes in your cluster are set to computeonly, allowing access to data hosted on other nodes, while consuming no storage themselves. This model can be advantageous if, for example, you want to take advantage of the hardware characteristics of particular nodes. A centralised storage model can also help avoid problems with naive resource allocation, since storage nodes and compute workloads can be kept apart.

This mode is also very suitable for elastic fleets with burstable workloads. A fleet can be expanded with many new machines for computing, while maintaining a central data store not impacted by rapid and repeated cluster scaling.

High Performance Mode

In this mode we co-locate pods with the volumes they are using in order to take advantage of the performance gains from running on the same node, while retaining the utility of orchestrators for managing app lifecycle.

Docs: Find ClusterID

Mon, 01 Jan 0001 00:00:00 +0000

Every StorageOS cluster has a unique ClusterID generated at bootstrap. A StorageOS Licence is specific for a ClusterID.

How to obtain the ClusterID

You can obtain the ClusterID using either the CLI or GUI.

GUI

You will need access to the StorageOS GUI on port 5705 of any of your nodes. For convenience, it is often easiest to port forward the service using the following kubectl incantation (this will block, so a second terminal window may be advisable):

$ kubectl port-forward -n kube-system svc/storageos 5705

As an alternative, an Ingress controller may be preferred.

Once you have obtained access to the GUI, login using whatever credentials you used to create the cluster and go to the “Licence” section on the left navigation menu.

CLI

This CLI command can print the cluster ID:

$ storageos get cluster
ID:               704dd165-9580-4da4-a554-0acb96d328cb
Licence:
  expiration:     2021-03-25T13:48:46Z (1 year from now)
  capacity:       5.0 TiB
  kind:           professional
  customer name:  storageos
Created at:       2020-03-25T13:48:33Z (1 hour ago)
Updated at:       2020-03-25T13:48:46Z (1 hour ago)

Given the Cluster ID, the StorageOS team can generate a licence.

Docs: Clusters

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS clusters represent groups of nodes which run a common distributed control plane.

Typically, a StorageOS cluster maps one-to-one to a Kubernetes (or similar orchestrator) cluster, and we expect our daemonset to run on all worker nodes within the cluster that will consume or present storage.

Clusters use etcd to maintain state and manage distributed consensus between nodes.

Docs: Compression

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS compression is handled on a per volume basis and is disabled by default in v2.2+, as performance is generally increased when compression is disabled due to block alignment. This means that there is a trade off between volume performance and the space the volume occupies on the backend device.

Compression can be enabled by setting the label storageos.com/nocompress=false on a volume at volume creation time.

StorageOS utilises the lz4 compression algorithm when writing to the backend store and when compressing replication traffic before it is sent across the network.

StorageOS detects whether a block can be compressed or not by creating a heuristic that predicts the size of a compressed block. If the heuristic indicates that the compressed block is likely to be larger than the original block then the uncompressed block is stored. Block size increases post compression if the compression dictionary is added to a block that cannot be compressed. By verifying whether blocks can be compressed, disk efficiency is increased and CPU resources are not wasted on attempts to compress uncompressible blocks. StorageOS’ patented on disk format is used to tell whether individual blocks are compressed without overhead. As such volume compression can be dynamically enabled/disabled even while a volume is in use.

When compression and encryption are both enabled for a volume, blocks are compressed then encrypted.

Docs: Contributing to the docs

Mon, 01 Jan 0001 00:00:00 +0000

We are always looking to improve our documentation. If you like to help people and can write, read on for the process for submitting your contributions. If your guide is published, you’ll receive $250 per article by PayPal.

Content guidelines

A guide contains step by step instructions for how to accomplish a specific task using StorageOS. To be accepted, guides must be:

Written in English.
Relevant, accurate and complete.
Technically correct and thoroughly tested.
Follow the writing style guide.

Guides should avoid:

Duplicating an existing guide or other sources, such as blogs or forum posts.
Including irrelevant material.

Submission and review

You should submit your guide as a pull request to the GitHub repo.

Your submission will be left open for community review for two weeks. After this, your submission will be reviewed internally for about another week.

If accepted, your pull request will be approved and you will have 36 hours to send your submission title and PayPal account information. Non-response will be taken as a go-ahead to publish.

Legal

COPYRIGHT OWNERSHIP. The StorageOS Guides & Tutorials repository is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0) license.

CREDIT. Nothing contained in this Agreement shall be deeded to require StorageOS to use the Work, or any part thereof, in connection with StorageOS Guides & Tutorials or otherwise. Credit for the Work shall read, “Contributed by writer’s name.”

PAYMENT. Upon publication of a submission to the StorageOS Guides & Tutorials Repository, the writer will be paid the sum of USD $250.00 as an electronic payment.

Docs: Create

Mon, 01 Jan 0001 00:00:00 +0000

$ storageos create --help

Create new resources

Usage:
  storageos create [command]

Available Commands:
  namespace    Provision a new namespace
  policy-group Provision a new policy group
  user         Create a new user account
  volume       Provision a new volume

Flags:
  -h, --help   help for create

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Use "storageos create [command] --help" for more information about a command.

create volume

$ storageos create volume --help

Provision a new volume

Usage:
  storageos create volume [flags]

Examples:

$ storageos create volume --description "This volume contains the data for my app" --fs-type "ext4" --labels env=prod,rack=db-1 --size 10GiB --namespace my-namespace-name my-app

$ storageos create volume --replicas 1 --namespace my-namespace-name my-replicated-app


Flags:
      --async                perform the operation asynchronously, using the configured timeout duration
      --cache                caches volume data (default true)
      --compress             compress data stored by the volume at rest and during transit
  -d, --description string   a human-friendly description to give the volume
  -f, --fs-type string       the filesystem to format the new volume with once provisioned (default "ext4")
  -h, --help                 help for volume
  -l, --labels strings       an optional set of labels to assign to the new volume, provided as a comma-separated list of key=value pairs
  -r, --replicas uint        the number of replicated copies of the volume to maintain
  -s, --size string          the capacity to provision the volume with (default "5GiB")
      --throttle             deprioritises the volumes traffic by reducing the rate of disk I/O

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

create user

$ storageos create user --help

Create a new user account

Usage:
  storageos create user [flags]

Examples:

$ storageos create user --with-username=alice --with-admin=true 


Flags:
  -h, --help                      help for user
      --with-admin                control whether the user is given administrative privileges
      --with-groups stringArray   the list of policy groups to assign to the user
      --with-password string      the password to assign to the user. If not specified, this will be prompted for.
      --with-username string      the username to assign

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

create namespace

$ storageos create namespace --help

Provision a new namespace

Usage:
  storageos create namespace [flags]

Examples:

$ storageos create namespace --labels env=prod,rack=db-1 my-namespace-name


Flags:
  -h, --help             help for namespace
  -l, --labels strings   an optional set of labels to assign to the new namespace, provided as a comma-separated list of key=value pairs

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

create policy-group

$ storageos create policy-group --help

Provision a new policy group

Usage:
  storageos create policy-group [flags]

Examples:

$ storageos create policy-group -r 'namespace-name:*:r' -r 'namespace-name-2:volume:w'  my-policy-group-name
$ storageos create policy-group -r 'namespace-name:*:r,namespace-name-2:volume:w'  my-policy-group-name


Flags:
  -h, --help            help for policy-group
  -r, --rules strings   set of rules to assign to the new policy group, provided as a comma-separated list of namespace:resource:rw triples.

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Docs: Delete

Mon, 01 Jan 0001 00:00:00 +0000

$ storageos delete -h

Delete resources in the cluster

Usage:
  storageos delete [command]

Available Commands:
  namespace    Delete a namespace
  node         Delete a node
  policy-group Delete a policy group
  user         Delete a user
  volume       Delete a volume. By default the target volume must be online. If the volume is offline then the request must specify that an offline delete is desired.

Flags:
  -h, --help   help for delete

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Use "storageos delete [command] --help" for more information about a command.

delete volume

$ storageos delete volume --help

Delete a volume. By default the target volume must be online. If the volume is offline then the request must specify that an offline delete is desired.

Usage:
  storageos delete volume [volume name] [flags]

Examples:

$ storageos delete volume my-test-volume my-unneeded-volume

$ storageos delete volume --namespace my-namespace my-old-volume


Flags:
      --async            perform the operation asynchronously, using the configured timeout duration
      --cas string       make changes to a resource conditional upon matching the provided version
  -h, --help             help for volume
      --offline-delete   request deletion of an offline volume. Volume data is not removed until the node reboots

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

delete user

$ storageos delete user --help

Delete a user

Usage:
  storageos delete user [user name] [flags]

Examples:

$ storageos delete user my-unneeded-user
$ storageos delete user --use-ids my-user-id


Flags:
      --cas string   make changes to a resource conditional upon matching the provided version
  -h, --help         help for user

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

delete policy-group

$ storageos delete policy-group --help

Delete a policy group

Usage:
  storageos delete policy-group [policy group name] [flags]

Examples:

$ storageos delete policy-group my-unneeded-policy-group
$ storageos delete policy-group --use-ids my-policy-group-id


Flags:
      --cas string   make changes to a resource conditional upon matching the provided version
  -h, --help         help for policy-group

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

delete node

$ storageos delete node --help

Delete a node

Usage:
  storageos delete node [node name] [flags]

Examples:

$ storagoes delete node my-old-node


Flags:
      --async        perform the operation asynchronously, using the configured timeout duration
      --cas string   make changes to a resource conditional upon matching the provided version
  -h, --help         help for node

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

delete namespace

$ storageos delete namespace --help

Delete a namespace

Usage:
  storageos delete namespace [namespace name] [flags]

Examples:

$ storageos delete namespace my-unneeded-namespace
$ storageos delete namespace --use-ids my-namespace-id


Flags:
      --cas string   make changes to a resource conditional upon matching the provided version
  -h, --help         help for namespace

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Docs: Deleting StorageOS Objects

Mon, 01 Jan 0001 00:00:00 +0000

When a Kubernetes object is deleted, StorageOS controllers automatically sync this deletion to StorageOS. For example, when a Kubernets node is deleted, this will automatically be mirrored in StorageOS, likewise with Namespaces.

Here is a StorageOS node, running on Kubernetes node worker1. A StorageOS node is any machine that is running the StorageOS daemonset pod. The node is visible below in kubectl.

$ kubectl describe -n kube-system pod storageos-daemonset-6q4g8
Name:                 storageos-daemonset-6q4g8
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 worker1/192.168.152.238
Start Time:           Thu, 06 May 2021 15:53:34 +0100
Labels:               app=storageos
                      app.kubernetes.io/component=storageos-daemonset
                      app.kubernetes.io/instance=example-storageos
                      app.kubernetes.io/managed-by=storageos-operator
                      app.kubernetes.io/name=storageos
                      app.kubernetes.io/part-of=storageos
                      controller-revision-hash=f5dcf577d
                      kind=daemonset
                      pod-template-generation=1
                      storageos_cr=example-storageos
Annotations:          kubectl.kubernetes.io/default-logs-container: storageos
Status:               Running
IP:                   192.168.152.238
...

The nodes in your cluster can be seen with storageos get nodes.

$ storageos get nodes
NAME          HEALTH  AGE        LABELS                              
worker1       online  1 day ago  beta.kubernetes.io/arch=amd64,      
                                 beta.kubernetes.io/os=linux,        
                                 cattle.io/creator=norman,           
                                 kubernetes.io/arch=amd64,           
                                 kubernetes.io/hostname=worker1,
                                 kubernetes.io/os=linux,             
                                 node-role.kubernetes.io/worker=true
                                 storageos.com/computeonly=true 
worker2       online  1 day ago  beta.kubernetes.io/arch=amd64,      
                                 beta.kubernetes.io/os=linux,        
                                 cattle.io/creator=norman,           
                                 kubernetes.io/arch=amd64,           
                                 kubernetes.io/hostname=worker2,
                                 kubernetes.io/os=linux,             
                                 node-role.kubernetes.io/worker=true 
worker3       online  1 day ago  beta.kubernetes.io/arch=amd64,      
                                 beta.kubernetes.io/os=linux,        
                                 cattle.io/creator=norman,           
                                 kubernetes.io/arch=amd64,           
                                 kubernetes.io/hostname=worker3,
                                 kubernetes.io/os=linux,             
                                 node-role.kubernetes.io/worker=true
...

Delete the node.
```
$ kubectl delete node worker1
```

Verify that the node has been deleted with kubectl get nodes or storageos get nodes. The node has now disappeared from StorageOS.

$ storageos get nodes
NAME          HEALTH  AGE        LABELS                              
worker2       online  1 day ago  beta.kubernetes.io/arch=amd64,      
                                 beta.kubernetes.io/os=linux,        
                                 cattle.io/creator=norman,           
                                 kubernetes.io/arch=amd64,           
                                 kubernetes.io/hostname=worker2,
                                 kubernetes.io/os=linux,             
                                 node-role.kubernetes.io/worker=true 
worker3       online  1 day ago  beta.kubernetes.io/arch=amd64,      
                                 beta.kubernetes.io/os=linux,        
                                 cattle.io/creator=norman,           
                                 kubernetes.io/arch=amd64,           
                                 kubernetes.io/hostname=worker3,
                                 kubernetes.io/os=linux,             
                                 node-role.kubernetes.io/worker=true 
...

Docs: Describe

Mon, 01 Jan 0001 00:00:00 +0000

$ storageos describe --help

Fetch extended details for resources

Usage:
  storageos describe [command]

Available Commands:
  cluster      Retrieve detailed information for the current cluster
  licence      Fetch current licence configuration details
  namespace    Retrieve detailed information for one or many namespaces
  node         Retrieve detailed information for nodes in the cluster
  policy-group Show detailed information for policy groups
  user         Show detailed information for users
  volume       Show detailed information for volumes

Flags:
  -h, --help   help for describe

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Use "storageos describe [command] --help" for more information about a command.

describe cluster

$ storageos describe cluster --help

Retrieve detailed information for the current cluster

Usage:
  storageos describe cluster [flags]

Aliases:
  cluster, nodes

Examples:

$ storageos describe cluster


Flags:
  -h, --help   help for cluster

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

describe license

$ storageos describe license --help

Fetch current licence configuration details

Usage:
  storageos describe licence [flags]

Aliases:
  licence, license

Examples:

$ storageos describe licence


Flags:
  -h, --help   help for licence

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

describe namespace

$ storageos describe namespace --help

Retrieve detailed information for one or many namespaces

Usage:
  storageos describe namespace [flags]

Aliases:
  namespace, namespaces

Examples:

$ storageos describe namespace my-namespace-name
$ storageos describe namespace --use-ids my-namespace-id
$ storageos describe namespaces


Flags:
  -h, --help   help for namespace

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

describe node

$ storageos describe node --help

Retrieve detailed information for nodes in the cluster

Usage:
  storageos describe node [node names...] [flags]

Aliases:
  node, nodes

Examples:

$ storageos describe node my-node-name


Flags:
  -h, --help                   help for node
  -l, --selector stringArray   filter returned results by a set of comma-separated label selectors

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

describe policy-group

$ storageos describe user --help

Show detailed information for policy groups

Usage:
  storageos describe policy-group [policy group names...] [flags]

Aliases:
  policy-group, policy-groups

Examples:

$ storageos describe policy-groups
$ storageos describe policy-group my-policy-group-name


Flags:
  -h, --help   help for policy-group

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

describe user

$ storageos describe user --help

Show detailed information for users

Usage:
  storageos describe user [user names...] [flags]

Aliases:
  user, users

Examples:

$ storageos describe users
$ storageos describe user my-username
$ storageos describe user my-username-1 my-username-2
$ storageos describe user --use-ids my-userid
$ storageos describe user --use-ids my-userid-1 my-userid-2


Flags:
  -h, --help   help for user

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

describe volume

$ storageos describe volume --help

Show detailed information for volumes

Usage:
  storageos describe volume [volume names...] [flags]

Aliases:
  volume, volumes

Examples:

$ storageos describe volumes

$ storageos describe volume --namespace my-namespace-name my-volume-name


Flags:
  -A, --all-namespaces         retrieves volumes from all accessible namespaces. This option overrides the namespace configuration
  -h, --help                   help for volume
  -l, --selector stringArray   filter returned results by a set of comma-separated label selectors

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Docs: Detach Volume

Mon, 01 Jan 0001 00:00:00 +0000

$ storageos detach --help

Detach a volume from its current location

Usage:
  storageos detach [flags]

Examples:

$ storageos detach --namespace my-namespace-name my-volume


Flags:
      --async        perform the operation asynchronously, using the configured timeout duration
      --cas string   make changes to a resource conditional upon matching the provided version
  -h, --help         help for detach

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Docs: Diagnostic bundle

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS can generate a cluster diagnostic bundle from the GUI or CLI under user request. The bundle packages the information needed for the engineering team to understand the context of your cluster and begin troubleshooting any issues you may be experiencing.

Generate Bundle

CLI

The StorageOS CLI can generate the bundle.

$ storageos get diagnostics

The file generated is in the form of diagnostics-${TIME_STAMP}.gz

GUI

Or you can use the StoregeOS GUI.

Go to section “Cluster”
Press the button “DOWNLOAD DIAGNOSTICS”.

Data collected in the bundle

Most of the data collected in the bundle is regarding the state of the StorageOS cluster, however some other information regarding the infrastructure is also gathered. The information is used to have a clear view of the cluster where StorageOS is running.

The bundle incorporates for each node:

StorageOS Daemonset Pod logs
lshw
dmesg (kernel logs)
StorageOS metadata for the ControlPlane and DataPlane

StorageOS metadata collected

cluster metadata
namespaces metadata
nodes metadata
volumes metadata
capacity stats
environment variables
health

Privacy

StorageOS can only obtain the bundle if it is downloaded by the user and given to our engineering team, or uploaded for analysis. The data received by StorageOS is private and never leaves nor will leave StorageOS Inc.

The data contained in the cluster diagnostic bundle has the sole purpose of helping customers troubleshoot their issues.

Docs: Disk Full

Mon, 01 Jan 0001 00:00:00 +0000

When the physical disks in your cluster get full, or start using disk space too fast, Kubernetes will automatically apply the DiskPressure:NoExecute taint.

The StorageOS Daemonset has a toleration applied for this taint, so it will continue to run.

Detecting a full disk

You will see the following when viewing the StorageOS container logs on the affected node with kubectl logs storageos-daemonset-xxxx -c storageos:

"msg": "StartAsyncFallocate: insufficient free space on file system free_space=1048305664 required_free_space=1073741824",
"msg": "Write: write failed volid=155051 error=BlobStorage::PrepWrite encountered a previous IO error preventing future IO for safety",
"msg": "Write: write failed volid=155051 error=all blob files are full - can not complete write",
"msg": "Write: write failed volid=155051 error=BlobStorage::PrepWrite encountered a previous IO error preventing future IO for safety",
"msg": "SCSI command failed type=write error=FATAL retries=0 time_to_deadline_secs=89",

When trying to attach a volume to a node with a full disk you will see the following when running kubectl describe pod $POD_USING_VOLUME:

AttachVolume.Attach failed for volume "pvc-xxx" : rpc error: code = Internal desc = internal error: rpc error: code = Internal desc = rpc error: code = Internal desc = fs: STATUS_FORBIDDEN: create failed in Notify handler error=Failed to create LUN for FsConfigVolume{volume_id=

Consequences of a full disk

Volume Provisioning

When provisioning a new volume, the dataplane checks that there is at least 1GiB of space left on the nodes’ underlying filesystem for the blob files located at /var/lib/storageos/data/dev[0-9]+/vol.xxxxxx.0.blob and /var/lib/storageos/data/dev[0-9]+/vol.xxxxxx.1.blob.

If there is insufficient space for both of the blob files that StorageOS uses to store data, then the dataplane fails the volume create request.

Runtime Access

At runtime, if an attempted write to a blob file returns an ENOSPC exception, the dataplane marks the file as full. Once both StorageOS blob files in a volume are marked as full, the dataplane marks the deployment with an error flag, and all subsequent read/write operations will return an I/O error.

This flag is only stored in memory, therefore, to clear this flag, the StorageOS daemonset pod on the affected node must be restarted after remediating the disk space issue.

Replica Failover

A replica on a full disk can’t be failed over successfully.

Recovering from a full disk

To recover from a disk full error, you can either add new storage space into the affected node, or delete existing persistent volume claims`.

N.B. However you choose to resolve the issue, the StorageOS Daemonset Pod must be restarted.

This is because the blob files disallow operations at runtime via the previously discussed error flag stored in memory. This flag does not survive boot cycles, so after the pod restarts, volumes can operate normally once more.

Adding New Storage Space

If you choose to recover by expanding your capacity, you have two main options:

Add new storage devices at /var/lib/storageos/data/dev[0-9]+ as described here.
Expand the underlying filesystem that StorageOS is using as described here.

After expanding capacity remember to restart the StorageOS daemonset pod.

Deleting existing PVCs

If you choose to recover by deleting existing persistent volume claims simply:

kubectl delete pvc <PVCOnFullNodeName>

Then restart the StorageOS daemonset pod.

Docs: Docker Enterprise Edition

Mon, 01 Jan 0001 00:00:00 +0000

To install StorageOS on DockerEE, please follow our Kubernetes installation instructions page.

Docker EE and the Universal Control Plane can run on different Linux distributions. StorageOS supports RHEL, CentOS, Debian, and selected Ubuntu images. For more details, check out the supported OSs in the prerequisites page.

StorageOS only supports Kubernetes nodes managed by Docker Enterprise Edition, not those nodes running Swarm. Mixed nodes (those running Kubernetes and Swarm workloads) are not supported. As a consequence, StorageOS volumes can only be provisioned on Kubernetes nodes, and only these nodes should be used for stateful workloads.

Docs: Elasticsearch

Mon, 01 Jan 0001 00:00:00 +0000

Elasticsearch is a distributed, RESTful search and analytics engine, most popularly used to aggregate logs, but also to serve as a search backend to a number of different applications.

Using StorageOS persistent volumes with ElasticSearch (ES) means that if a pod fails, the cluster is only in a degraded state for as long as it takes Kubernetes to restart the pod. When the pod comes back up, the pod data is immediately available. Should Kubernetes schedule the Elasticsearch pod on a new node, StorageOS allows for the data to be available to the pod, irrespective of whether or not the original StorageOS master volume is located on the same node.

Elasticsearch has features to allow it to handle data replication, and as such careful consideration of whether to allow StorageOS or Elasticsearch to handle replication is required.

Before you start, ensure you have StorageOS installed and ready on a Kubernetes cluster. See our guide on how to install StorageOS on Kubernetes for more information.

Deploying Elasticsearch on Kubernetes

Prerequisites

Some OS tuning is required, which is done automatically when using our example from the use cases repository.

Elasticsearch requires vm.max_map_count to be increased to a minimum of 262144, which is a system wide setting. One way to achieve this is to run sysctl -w vm.max_map_count=262144 and update /etc/sysctl.conf to ensure it persists over a reboot. See ElasicSearch reference here.

Administrators should be aware that this impacts the behaviour of nodes and that there may be collisions with other application settings. Administrators are advised to centrally collate sysctl settings using the tooling of their choice.

Deployment of the application

StatefulSet defintion

---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: esdata

[...]

    spec:
      serviceAccountName: elasticsearch
      containers:
        - name: data
          image: elasticsearch:6.7.0
          imagePullPolicy: IfNotPresent

[...]

          volumeMounts:
            - name: data
              mountPath: /usr/share/elasticsearch/data/data

[...]

  volumeClaimTemplates:
    - metadata:
        name: "data"
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: "fast" # <--- default StorageOS storage class name
        resources:
          requests:
            storage: 10Gi   # <--- change this to the appropriate value

This excerpt is from the StatefulSet definition (/elasticsearch/10-es-data.yaml). The file contains the PersistentVolumeClaim template that will dynamically provision the necessary storage, using the StorageOS storage class.

Dynamic provisioning occurs as a volumeMount has been declared with the same name as a VolumeClaimTemplate.

Installation

Clone the use cases repo

You can find the latest files in the StorageOS use cases repostiory in /elasticsearch/

git clone https://github.com/storageos/use-cases.git storageos-usecases
cd storageos-usecases

Create the kubernetes objects

This will install an ES cluster with 3 master, 3 data and 3 coordinator nodes. Combined they will require ~ 14 GiB of available memory in your cluster, however, more may be used as the application is being used
```
kubectl apply -f ./elasticsearch/
```
Once completed, an internal service object will have been created making the cluster available as http://elasticsearch:9200/ which is the default Kibana (when installed via Helm) will be using.

Confirm Elasticsearch is up and running

kubectl get pods -l component=elasticsearch

NAME                                    READY   STATUS    RESTARTS   AGE
elasticsearch-exporter-d86ffd94-zw45l   1/1     Running   0          5m44s
es-coordinator-b7b984dd4-7wlz5          1/1     Running   0          5m44s
es-coordinator-b7b984dd4-89w26          1/1     Running   0          5m44s
es-coordinator-b7b984dd4-b4t6j          1/1     Running   0          5m44s
es-master-78dfd5b49f-9gf5c              1/1     Running   0          5m44s
es-master-78dfd5b49f-smsbw              1/1     Running   0          5m44s
es-master-78dfd5b49f-z4qpj              1/1     Running   0          5m44s
esdata-0                                1/1     Running   0          5m44s
esdata-1                                1/1     Running   0          4m34s
esdata-2                                1/1     Running   0          3m22s

Connect to ElasticSearch

To connect to ES directly, you can use the following port-forward command
```
kubectl port-forward svc/elasticsearch 9200
```
and then access it via http://localhost:9200

Kibana (optional)

One of the most popular uses of ES is to use it for log aggregation and indexing, Kibana helps us visualize the data in these indices and can be easily used when installed via its Helm chart

Install the helm chart.
```
helm install stable/kibana
```

Once installed, use a port-foward to Kibana instead of directly to ES

kubectl port-forward --namespace default $(kubectl get pods --namespace default -l "app=kibana" -o jsonpath="{.items[0].metadata.name}") 5601

and then access it via http://localhost:5601

Monitoring (optional)

As part of the example deployment, ES metrics are exposed and can be scraped by Prometheus on port 9108 (see 77-es-exporter.yaml). This is enabled by default, and should work with the default Prometheus install via Helm. If you’re using the Prometheus service monitors, you can monitor this installation by creating a monitor for the es-exporter service. For an example of how this is done to monitor StorageOS, please see prometheus-setup.

Docs: Encryption

Mon, 01 Jan 0001 00:00:00 +0000

For more detail on encryption at rest in StorageOS, please see the reference page.

Enabling encryption on a volume

Encrypting a volume is done by simply creating a volume with the storageos.com/encryption=true label. This can be set on the PVC or on the PVC’s StorageClass.

This label is all that is needed. If it is present, the mutating admission webhook that runs as part of the StorageOS API Manager will create the encryption key, link it to the PVC and store it in a secret.

Encryption is enabled when a volume is provisioned, and it can not be removed during during the volume’s lifetime.

An example encrypted volume

Option 1: Label the PVC

Add the label in the PVC definition, for instance:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: encrypted-vol
  labels:
    storageos.com/encryption: "true" # Label <-----
spec:
  storageClassName: "fast"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1G

The encryption label as set on a PVC takes precedence over the encryption label as set on the PVC’s StorageClass.

Option 2: Add a parameter to the StorageClass

Add a parameter to the StorageClass definition. This will cause the above label to be present on PVCs created using this StorageClass. For instance:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: storageos-encrypted
parameters:
  fsType: ext4
  storageos.com/encryption: "true" # Label   <--------
  # Change the NameSpace below if StorageOS doesn't run in kube-system
  csi.storage.k8s.io/controller-expand-secret-name: csi-controller-expand-secret
  csi.storage.k8s.io/controller-publish-secret-name: csi-controller-publish-secret
  csi.storage.k8s.io/node-publish-secret-name: csi-node-publish-secret
  csi.storage.k8s.io/provisioner-secret-name: csi-provisioner-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: kube-system   # NameSpace that runs StorageOS Daemonset
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system  # NameSpace that runs StorageOS Daemonset
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system        # NameSpace that runs StorageOS Daemonset
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system         # NameSpace that runs StorageOS Daemonset
provisioner: storageos # CSI driver (recommended)

Docs: Encryption

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS supports encryption for data-at-rest and data-in-transit.

Data-in-transit is data as it is travelling between nodes. It is encrypted by default with mTLS. Data-at-rest is the data stored in your volumes as blob files. Encryption of these blob files is optional and can be enabled by adding a label to your volume definitions before they’re provisioned.

For information on how to enable encryption on your volumes, please see our Encryption Operations page.

How volumes are encrypted

Volumes are encrypted using AES-256 in the XTS-AES mode with 512-bit keys, as specified by IEEE Standard 1619-2007. There is a non-zero performance impact of using encrypted volumes. A 10-25% cost in read/write throughput can be expected from XTS-AES, dependent on workload. Thin provisioning still applies to encrypted volumes.

Encryption Key Generation

On PVC creation, if encryption is enabled, StorageOS will automatically generate up to two keys as Kubernetes secrets. Both keys are stored in the same namespace as the PVC.

Firstly, if it doesn’t already exist, a namespace key is generated. It is always named storageos-namespace-key and only one exists per namespace.

Secondly a volume key is created for each encrypted volume. It has a name in the format storageos-volume-key-<random-id>, with no connection to the name of the volume. The volume it is associated with can be determined by looking at the storageos.com/pvc label on the secret. The storageos.com/encryption-secret-name and storageos.com/encryption-secret-namespace annotations are added to the PVC by an admission controller to map the PVC back to its secret.

The encryption key is passed to StorageOS as part of the CSI volume creation request and is used to encrypt the volume.

Encryption Key Use

The volume specific secret is needed whenever a volume is attached to a node for use by a pod. When this happens, the StorageOS node container’s Service Account reads the secret and passes it to the StorageOS controlplane.

A volume missing its key or with a malformed key will be unable to attach.

The key is stored in memory by StorageOS only on the node that the volume is being used on. As a result, encryption and decryption are performed where the data is consumed, rather than where it is stored.

Because of this, the use of encrypted volumes is transparent to the user. There is a complete integration between Kubernetes applications and StorageOS encryption.

Key Management Best Practices

StorageOS saves encryption keys in Kubernetes Secrets. Backups are therefore imperative in case the Kubernetes Etcd is lost. StorageOS has no ability to decrypt a volume whose encryption keys have been lost.

Secrets in Kubernetes are not encrypted by default, they are stored in the Kubernetes Etcd in simple Base64 encoding. As StorageOS encryption keys are stored as Kubernetes Secrets, this means that anyone with access to a Kubernetes Etcd installation can read encryption keys and decrypt volumes, unless the cluster has an external secrets store.

For better security check Kubernetes secret encryption.

Secrets are not garbage-collected by StorageOS. To clean up completely upon deletion of a volume it is necessary to also delete that volume’s secret. There is no benefit to doing this, however.

Key Management with Kubernetes KMS provider

StorageOS encryption keys are stored within Etcd as Kubernetes secrets. Whilst the Etcd and kubernetes secrets can also be encrypted, many organisations choose to use an external KMS provider.

To address this from a Kubernetes limitations perspective and provide an agnostic solution, our encryption design allows the user to benefit from any Kubernetes KMS provider plugin to envelop the secrets into the KMS provider encryption scheme.

StorageOS allows customers to transparently integrate any supported KMS plugin with StorageOS encryption key management using the standard Kubernetes API and Kubernetes KMS provider framework. The below figure provides an overview of the process.

The KMS plugin is deployed within the Kubernetes cluster.
The KMS plugin is configured to act as a broker between the Kubernetes API server and the KMS server API endpoint.
At volume creation, StorageOS will create a Kubernetes secret using Kubernetes API calls
The KMS plugin will handle the Kubernetes API Secret creation call and interface to the KMS server instance.
The KMS server will return the secret using its encryption envelop scheme.
The KMS plugin will store the encrypted secret within the Kubernetes Etcd.

Contact the StorageOS sales team for more information about the dedicated StorageOS Vault KMS plugin integration offering.

Docs: Etcd

Mon, 01 Jan 0001 00:00:00 +0000

Etcd is an open-source distributed, strongly consistent key value store that is used by StorageOS to durably persist the StorageOS cluster state. As the backing store for Kubernetes, StorageOS uses etcd for many of the same reasons.

StorageOS uses etcd as the single source of truth for all StorageOS objects. Whenever a request is made to create, update or delete an object the result is written to etcd before the request is completed. Using etcd as a configuration store allows nodes to retrieve the current cluster state after being offlined, allowing offlined nodes to rejoin the cluster.

N.B. StorageOS v2.0 does not provide an embedded etcd server as previous versions did. You will need to setup an etcd server for StorageOS to use prior to installation of StorageOS. Please see our etcd prerequisites page for more information on how to install and configure etcd.

Docs: Etcd node migration

Mon, 01 Jan 0001 00:00:00 +0000

This procedure explains how to add a new etcd member for your StorageOS etcd cluster while removing one of the current members. This is useful when the nodes hosting etcd must be recycled.

It is assumed that the StorageOS etcd cluster is installed following the production etcd installation page, where etcd nodes are installed on their own machines.

It is also assumed that etcd members are referenced from Kubernetes using a External Service. Example available in the etcd external Service example. This service should be referred to in the spec.kvbackend.address section of your StorageOS CustomResource. If that Service is not used, a full restart of the StorageOS cluster will be required. The StorageOS CustomResource would need to be removed, and amended to reflect the new etcd urls created.

Preparation

Prepare the installation of etcd on a new node, making sure that etcd is not starting on that new node.

The steps for preparing an etcd node can be found in the etcd prerequisites page.

Back up etcd

$ export ETCDCTL_API=3

$ # Set all your endpoints
$ export endpoints="192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379"

$ etcdctl --endpoints $endpoints snapshot save /var/tmp/etcd-snapshot.db

Verify etcd health

$ export ETCDCTL_API=3

$ # Set all your endpoints
$ export endpoints="192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379"
$ etcdctl member list --endpoints $endpoints -wtable
+------------------+---------+-----------------------+------------------------------+-----------------------------+------------+
|        ID        | STATUS  |         NAME          |         PEER ADDRS           |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+-----------------------+------------------------------+-----------------------------+------------+
| 7817aa073b059aab | started |  etcd-192.168.195.168 |  http://192.168.195.168:2380 | http://192.168.195.168:2379 |      false |
| e22cdd20a03e5e73 | started |  etcd-192.168.202.40  |  http://192.168.202.40:2380  | http://192.168.202.40:2379  |      false |
| e5d0f0e242014d3d | started |  etcd-192.168.174.117 |  http://192.168.174.117:2380 | http://192.168.174.117:2379 |      false |
+------------------+---------+-----------------------+------------------------------+-----------------------------+------------+

$ etcdctl endpoint health --endpoints $endpoints -wtable
+---------------------+--------+------------+-------+
|      ENDPOINT       | HEALTH |    TOOK    | ERROR |
+---------------------+--------+------------+-------+
|192.168.174.117:2379 |   true | 5.048177ms |       |
|192.168.195.168:2379 |   true | 5.926681ms |       |
|192.168.202.40:2379  |   true | 5.526928ms |       |
+---------------------+--------+------------+-------+

$ etcdctl endpoint status --endpoints $endpoints -wtable
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|192.168.174.117:2379 | e5d0f0e242014d3d |   3.4.9 |  311 kB |     false |      false |         2 |       4281 |               4281 |        |
|192.168.195.168:2379 | 7817aa073b059aab |   3.4.9 |  315 kB |     false |      false |         2 |       4281 |               4281 |        |
|192.168.202.40:2379  | e22cdd20a03e5e73 |   3.4.9 |  352 kB |      true |      false |         2 |       4281 |              4281  |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Migration

In the following procedure NODE4 is a new member to add to the cluster, while NODE1 ought to be removed.

Amend etcd configuration to reference the new node (NODE4)

Make the following change on all running etcd members and NODE4

$ # NODE4_IP is the NEW_NODE_ADDRESS
$ echo "NODE4_IP=192.168.152.142" >> /etc/etcd.conf

$ # Check the contents of /etc/etcd.conf
$ cat /etc/etcd.conf
CLIENT_PORT=2379
PEERS_PORT=2380

# NODE_IP is the IP of the node where this file resides.
NODE_IP=192.168.202.40

NODE1_IP=192.168.195.168
NODE2_IP=192.168.202.40
NODE3_IP=192.168.174.117
NODE4_IP=192.168.152.142

Amend SystemD service file on the new etcd node (NODE4)

The SystemD service file is expected in /etc/systemd/system/etcd3.service

Change the --initial-cluster-state to existing and add the reference to NODE4 in the --initial-cluster variable.

vi /etc/systemd/system/etcd3.service

The resulting changes would appear as follows:

...

ExecStart=/usr/local/sbin/etcd3 --name etcd-${NODE_IP} \
   ...
   --initial-cluster-state existing \
   ...
   --initial-cluster \
        etcd-${NODE1_IP}=http://${NODE1_IP}:${PEERS_PORT},\
        etcd-${NODE2_IP}=http://${NODE2_IP}:${PEERS_PORT},\
        etcd-${NODE3_IP}=http://${NODE3_IP}:${PEERS_PORT},\
        etcd-${NODE4_IP}=http://${NODE4_IP}:${PEERS_PORT}
...

Note the reference to NODE4 at the end of the --initial-cluster variable

Make sure etcd is not started on the new member NODE4

Add etcd member as a learner

# Set environment variable for the the new etcd member (NODE4)
$ NODE4_IP=192.168.152.142

$ ETCD_NEW_MEMBER="etcd-${NODE4_IP}"
$ ETCD_NEW_MEMBER_PEER="http://$NODE4_IP:2380"

# Add the new member to the cluster
$ export ETCDCTL_API=3

$ etcdctl member add \
     --learner $ETCD_NEW_MEMBER \
     --peer-urls="$ETCD_NEW_MEMBER_PEER"

 Member 52e5c9ac117b3df2 added to cluster b4f4ed717ea44b8d

 ETCD_NAME="etcd-192.168.152.142"
 ETCD_INITIAL_CLUSTER="etcd-192.168.152.142=http://192.168.152.142:2380,etcd-192.168.195.168=http://192.168.195.168:2380,etcd-192.168.202.40=http://192.168.202.40:2380,etcd-192.168.174.117=http://192.168.174.117:2380"
 ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.152.142:2380"
 ETCD_INITIAL_CLUSTER_STATE="existing"

Check the etcd members

$ export endpoints="192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379"

$ etcdctl member list --endpoints $endpoints -wtable
+------------------+-----------+----------------------+-----------------------------+-----------------------------+------------+
|        ID        |  STATUS   |         NAME         |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+-----------+----------------------+-----------------------------+-----------------------------+------------+
| 52e5c9ac117b3df2 | unstarted |                      | http://192.168.152.142:2380 |                             |       true |
| 7817aa073b059aab |   started | etcd-192.168.195.168 | http://192.168.195.168:2380 | http://192.168.195.168:2379 |      false |
| e22cdd20a03e5e73 |   started | etcd-192.168.202.40  | http://192.168.202.40:2380  | http://192.168.202.40:2379  |      false |
| e5d0f0e242014d3d |   started | etcd-192.168.174.117 | http://192.168.174.117:2380 | http://192.168.174.117:2379 |      false |
+------------------+-----------+----------------------+-----------------------------+-----------------------------+------------+

Note that the learner is not started yet

Start etcd on the new node (NODE4)

Make sure that /etc/systemd/system/etcd.service only have currently active nodes specified in the --initial-cluster flag.
```
# On the new node (NODE4)

systemctl daemon-reload
systemctl enable etcd3.service
systemctl start etcd3.service
```

Check the etcd members

$ export endpoints="192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379"

$ etcdctl member list --endpoints $endpoints -wtable
+------------------+---------+----------------------+-----------------------------+-----------------------------+------------+
|        ID        | STATUS  |         NAME         |         PEER ADDRS          |        CLIENT ADDRS         | IS LEARNER |
+------------------+---------+----------------------+-----------------------------+-----------------------------+------------+
| 52e5c9ac117b3df2 | started | etcd-192.168.152.142 | http://192.168.152.142:2380 | http://192.168.152.142:2379 |       true |
| 7817aa073b059aab | started | etcd-192.168.195.168 | http://192.168.195.168:2380 | http://192.168.195.168:2379 |      false |
| e22cdd20a03e5e73 | started | etcd-192.168.202.40  | http://192.168.202.40:2380  | http://192.168.202.40:2379  |      false |
| e5d0f0e242014d3d | started | etcd-192.168.174.117 | http://192.168.174.117:2380 | http://192.168.174.117:2379 |      false |
+------------------+---------+----------------------+-----------------------------+-----------------------------+------------+

Note that the learner is started

Check that the new learner has the same revision applied as the current members

$ export ETCDCTL_API=3
# Added NODE4 in the endpoints variable
$ export endpoints="192.168.174.117:2379,192.168.195.168:2379,192.168.174.117:2379,192.168.152.142:2379"

etcdctl endpoint status --endpoints $endpoints -wtable
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 192.168.174.117:2379| e5d0f0e242014d3d |   3.4.9 |  352 kB |     false |      false |         2 |      24570 |              24570 |        |
| 192.168.195.168:2379| 7817aa073b059aab |   3.4.9 |  352 kB |     false |      false |         2 |      24570 |              24570 |        |
| 192.168.202.40:2379 | e22cdd20a03e5e73 |   3.4.9 |  352 kB |     true  |      false |         2 |      24570 |              24570 |        |
| 192.168.152.142:2379| 52e5c9ac117b3df2 |   3.4.9 |  467 kB |     false |       true |         2 |      24570 |              24570 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Wait until the cluster has the learner ready, by ensuring that the RAFT TERM and RAFT INDEX of the learner node match the rest of the cluster.

Remove the node that needs to be evicted (NODE1)

Before promoting the learner to a full member, it is best to remove the node from the cluster that initially was selected to be decommissioned to avoid breaking quorum while having 4 nodes being part of the cluster. For more details, check the official etcd documentation regarding this topic.
```
$ export ETCDCTL_API=3

$ # Select member of id of the node to remove (NODE1)
$ NODE1_MEMBER_ID=e22cdd20a03e5e73

$ etcdctl member remove $NODE1_MEMBER_ID
Member e22cdd20a03e5e73 removed from cluster b4f4ed717ea44b8d
```

Promote the learner to a member

$ export ETCDCTL_API=3

$ # Select member of id of the node to remove (NODE1)
$ NODE4_MEMBER_ID=52e5c9ac117b3df2

$ etcdctl member promote $NODE4_MEMBER_ID
Member 52e5c9ac117b3df2 promoted in cluster b4f4ed717ea44b8d

The promotion will fail if the learner is not in sync with the leader member.

Check the etcd health

$ export endpoints=192.168.174.117:2379,192.168.195.168:2379,192.168.152.142:2379

$ etcdctl member list --endpoints $endpoints -wtable
+------------------+---------+---------------------+----------------------------+----------------------------+------------+
|        ID        | STATUS  |        NAME         |         PEER ADDRS         |        CLIENT ADDRS        | IS LEARNER |
+------------------+---------+---------------------+----------------------------+----------------------------+------------+
| 52e5c9ac117b3df2 | started |etcd-192.168.152.142 |http://192.168.152.142:2380 |http://192.168.152.142:2379 |      false |
| 7817aa073b059aab | started |etcd-192.168.195.168 |http://192.168.195.168:2380 |http://192.168.195.168:2379 |      false |
| e5d0f0e242014d3d | started |etcd-192.168.174.117 |http://192.168.174.117:2380 |http://192.168.174.117:2379 |      false |
+------------------+---------+---------------------+----------------------------+----------------------------+------------+

$ etcdctl endpoint status --endpoints $endpoints -wtable
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|      ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|192.168.174.117:2379 | e5d0f0e242014d3d |   3.4.9 |  352 kB |     false |      false |         3 |      35939 |              35939 |        |
|192.168.195.168:2379 | 7817aa073b059aab |   3.4.9 |  352 kB |      true |      false |         3 |      35939 |              35939 |        |
|192.168.152.142:2379 | 52e5c9ac117b3df2 |   3.4.9 |  467 kB |     false |      false |         3 |      35939 |              35939 |        |
+---------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Note that NODE4 is now a full quorum member, while NODE1 is no longer part of the cluster

Edit Endpoints referencing the Kubernetes Service

Remove the reference to NODE1 and add the IP for NODE4
```
$ kubectl edit -n storageos-etcd endpoints/storageos-etcd
```
Make amendments in the SystemD configuration files removing any reference to NODE1

It is not required to restart the etcd service, but to keep the service file up to date.

Docs: Failure Modes

Mon, 01 Jan 0001 00:00:00 +0000

For more information about replication and failure modes please see our Replication concepts page.

The failure mode for a specific volume can be set using a label on a PVC or it can be set as a parameter on a StorageClass. The PVC definition takes precedence over the StorageClass.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-vol-1
  labels:
      storageos.com/replicas: "2"
      storageos.com/failure-mode: "soft"
spec:
  storageClassName: "fast"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Failure Modes

StorageOS failure modes offer different guarantees with regards to a volume’s mode of operation in the face of replica failure. If the failure mode is not specified it defaults to Hard. Volume failure modes can be dynamically updated at run time.

Hard

Hard failure mode requires that the number of declared replicas matches the available number of replicas at all times. If a replica fails StorageOS will attempt creation of a new replica for 90 seconds. After 90s if the old replica is not available and a new replica cannot be provisioned, StorageOS cannot guarantee that the data is stored on the number of multiple nodes requested by the user. StorageOS will therefore set the volume to be read-only.

If a volume has gone read-only there are two stages to making it read-write again. Firstly, sufficient replicas must be provisioned to match the desired replica count. Depending on your environment, additional nodes and/or disk capacity may be required for this. Secondly, the volume must be remounted - necessitating pod deletion/recreation in Kubernetes.

storageos.com/failure-mode: hard

Number of nodes required for hard failure mode

When a node fails, a new replica is provisioned and synced as described above. To ensure that a new replica can always be created, an additional node should be available. To guarantee high availability using storageos.com/failure-mode: hard, clusters using volumes with 1 replica must have at least 3 storage nodes. When using volumes with 2 replicas, at least 4 storage nodes, 3 replicas, 5 nodes, etc.

Minimum number of storage nodes = 1 (primary) + N (replicas) + 1

Soft

Soft failure mode allows a volume to continue serving I/O even when a replica goes offline and a new replica fails to provision. So long as there are not less than max(1, n-1) available replicas where n is the number of replicas for the volume.

For example, if a volume with 2 replicas loses 1 replica, then I/O would continue to be served since 1 replica remaining >= max(1, 1). If a volume with 1 replica loses 1 replica, then I/O would halt after 90 seconds since 0 replicas remaining < max(1, 0).

storageos.com/failure-mode: soft

Number of nodes required for soft failure mode

To ensure that a storageos.com/failure-mode: soft volume is highly available, clusters using volumes with 1 replica must have at least 2 storage nodes. When using volumes with 2 replicas, at least 3 storage nodes, 3 replicas, 3 nodes, etc.

Minimum number of storage nodes = 1 (primary) + N (replicas)

Threshold

Threshold failure mode allows the user to set the minimum required number of online replicas for a volume. For example for a volume with 2 replicas, setting the threshold to 1 would allow a single replica to be offline, whereas setting threshold to 0 would allow 2 replicas to be offline.

storageos.com/failure-mode: (0-5)

Number of nodes required for threshold failure mode

The minimum number of nodes for a threshold volume is determined by the threshold that is set.

Minimum number of storage nodes = 1 (primary) + T (threshold)

AlwaysOn

AlwaysOn failure mode allows all replicas for a volume to be offline and keeps the volume writeable. A volume with failure mode AlwaysOn will continue to serve I/O regardless of how many replicas it currently has. This mode should be used with caution as it effectively allows for only a single copy of the data to be available.

storageos.com/failure-mode: alwayson

Number of nodes required for AlwaysOn failure mode

A storageos.com/failure-mode: alwayson volume is highly available albeit at the cost of reliability. The minimum node count here is 1 as the loss of all replicas will be tolerated.

Minimum number of storage nodes = 1 (primary)

Docs: Fencing

Mon, 01 Jan 0001 00:00:00 +0000

StatefulSet behaviour

In order to understand what StorageOS Fencing for Kubernetes is and when it is needed, it is required to first understand the behaviour of StatefulSets.

StatefulSets are the de facto Kubernetes controller to use for stateful applications. The StatefulSet controller offers guarantees around pod uniqueness, sticky identities and the persistence of PVCs beyond the lifetime of their pods. As such, StatefulSets have different characteristics and provide different guarantees than Deployments.

Deployments guarantee the amount of healthy replicas by reconciling towards the deployment desired state. Attempts to align the number of healthy pods with the deployment’s desired state happen as fast as possible by aggressively initializing and terminating pods. If one pod is terminating, another will be automatically scheduled to start even if the first pod is not yet completely terminated. Stateless applications benefit from this behaviour as one pod executes the same work as any other in the deployment.

StatefulSets, on the other hand, guarantee that every pod scheduled has a unique identity, which is to say that only a single copy of a pod is running in the cluster at any one time. Whenever scheduling decisions are made, the StatefulSet controller ensures that only one copy of this pod is running at any time. If a pod is deleted, a new pod will not be scheduled until the first pod is fully terminated. This is an important guarantee as FileSystems need to be unmounted before they can be remounted in a new pod. Any ReadWriteOnce PVC defining a device requires this behaviour to ensure the consistency of the data and thus the PVC.

To protect data integrity, Kubernetes guarantees that there will never be more than one instance of a StatefulSet Pod running at a time. It assumes that when a node is determined to be offline it may still be running the workload but partitioned from the network. Since Kubernetes is unable to verify that the Pod has been stopped it errs on the side of caution and does not allow a replacement to start on another node.

Kubernetes does reschedule pods from some controllers when nodes become unavailable. The default behaviour is that when a node becomes unavailable its status becomes “Unknown” and after the pod-eviction-timeout has passed pods are scheduled for deletion. By default, the pod-eviction-timeout is 300 seconds.

For this reason, Kubernetes requires manual intervention to initiate timely failover of a StatefulSet Pod. The StorageOS Fencing Controller gives the capability to enable fast failover of workloads when a node goes offline.

For more information on the rationale behind the design of StatefulSets please see the Kubernetes design proposal for Pod Safety.

StorageOS Fencing Controller

The StorageOS Fencing Controller is part of the StorageOS API Manager which is deployed in high availability when StorageOS is installed.

HA for StatefulSet applications can be achieved with the StorageOS Fencing feature.

Since StorageOS is able to determine when a node is no longer able to access a volume and has protections in place to ensure that a partitioned or formerly partitioned node can not continue to write data, it can work with Kubernetes to perform safe, fast failovers of Pods, including those running in StatefulSets.

When StorageOS detects that a node has gone offline or become partitioned, it marks the node offline and performs volume failover operations.

The StorageOS Fencing Controller watches for these node failures and determines if there are any pods assigned to the failed node with the label storageos.com/fenced=true, and if the pods have any PVCs backed by StorageOS volumes.

When a Pod has StorageOS volumes and if they are all healthy, the StorageOS fencing controller deletes the Pod to allow it to be rescheduled on another node. It also deletes the VolumeAtachments for the corresponding volumes so that they can be immediately attached to the new node.

No changes are made to Pods that have StorageOS volumes that are unhealthy. This is usually because a volume was configured to not have any replicas, and the node with the single copy of the data is offline. In this case it is better to wait for the node to recover.

Fencing works with both dynamically provisioned PVCs and PVCs referencing pre-provisioned volumes.

The fencing feature is opt-in and Pods must have the storageos.com/fenced=true label set, and be using at least one StorageOS volume, to enable fast failover.

For more information about how to enable pod fencing please see our Fencing Operations page.

Docs: Fencing

Mon, 01 Jan 0001 00:00:00 +0000

For information regarding the StorageOS Fencing feature please see our fencing concepts page.

Labeling a pod for fencing

When StorageOS detects that a node has gone offline or become partitioned, it marks the node offline and performs volume failover operations.

The StorageOS Fencing Controller watches for node failures and determines if there are any pods targeted for fencing. In order for a pod to be fenced, the following criteria is required.

The pod must have the label storageos.com/fenced=true
The pod to be fenced must claim a StorageOS volume
The StorageOS volume claimed by the pod needs to be online

If the node becomes offline and these criteria are met, the pod is deleted and rescheduled on another node.

No changes are made to Pods that have StorageOS volumes that are unhealthy. This is typically the case when a volume was configured to not have any replicas, and the node with the single copy of the data is offline. In this case it is better to wait for the Node to recover.

StatefulSet example with fencing enabled

The following example shows how a Statefulset can enable fencing to all its pods.

Note that the fenced label is applied only in the .spec.template.metadata.label section, as the label must only be present on the Pod, but not on the PVC.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: my-statefulset
spec:
  selector:
    matchLabels:
      app: prod
  serviceName: "default"
  replicas: 1
  template:
    metadata:
      labels:
        app: prod
        storageos.com/fenced: "true" # Fencing label
    spec:
      containers:
      - name: debian
        image: debian:10-slim
        command: ["/bin/sleep"]
        args: [ "3600" ]
        volumeMounts:
        - name: storageos-volume
          mountPath: /mnt
  volumeClaimTemplates:
  - metadata:
      name: storageos-volume
      labels:
        env: prod
        storageos.com/replicas: "1"
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: "fast"
      resources:
        requests:
          storage: 10Gi

Fencing trigger

The StorageOS Fencing Controller checks the StorageOS node health every 5 seconds. This is how quickly the fencing controller can react to node failures.

Pods assigned to unhealthy nodes will be evaluated immediately on state change, and then re-evaluated every hour, though this is configurable. This retry allows pods that had unhealthy volumes which have now recovered to eventually failover, or pods that were rescheduled on an unhealthy node to be re-evaluated for fencing.

Fencing example

The following example shows how the StorageOS API manager fences a Pod.

Ensure the storageos.com/fenced label is present

$ kubectl -n mysql get pod --show-labels -owide
NAME      READY   STATUS    RESTARTS   AGE     IP          NODE             NOMINATED NODE   READINESS GATES   LABELS
mysql-0   1/1     Running   0          6m33s   10.42.3.7   worker1   <none>           <none>            app=mysql,controller-revision-hash=mysql-799fd74b87,env=prod,statefulset.kubernetes.io/pod-name=mysql-0,storageos.com/fenced=true

$ kubectl -n mysql get pvc --show-labels -owide
NAME           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE   VOLUMEMODE   LABELS
data-mysql-0   Bound    pvc-5d7b23a6-e754-4998-98fd-318b3f9382bb   5Gi        RWO            fast           19m   Filesystem   app=mysql,env=prod,storageos.com/replicas=1

Note that the mysql-0 pod has the storageos.com/fenced=true label.

Stop node hosting the mysql-0 pod
```
root@worker1:~# shutdown -h now
```

Check logs from the StorageOS API Manager

$ kubectl -n kube-system logs storageos-api-manager-68759bbc78-7l5fw
...
{"level":"info","timestamp":"2021-04-28T13:06:49.413811357Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"nginx-ingress-controller-xbqjf","namespace":"ingress-nginx"}

{"level":"info","timestamp":"2021-04-28T13:06:49.417605039Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"storageos-api-manager-68759bbc78-7l5fw","namespace":"kube-system"}
{"level":"info","timestamp":"2021-04-28T13:06:49.417748651Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"coredns-7c5566588d-8g5xq","namespace":"kube-system"}
{"level":"info","timestamp":"2021-04-28T13:06:49.417792281Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"metrics-server-6b55c64f86-cnwtk","namespace":"kube-system"}
{"level":"info","timestamp":"2021-04-28T13:06:49.417883383Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"default-http-backend-67cf578fc4-w8sm4","namespace":"ingress-nginx"}
{"level":"info","timestamp":"2021-04-28T13:06:49.417975663Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"coredns-autoscaler-65bfc8d47d-ph6pn","namespace":"kube-system"}
{"level":"info","timestamp":"2021-04-28T13:06:49.418024204Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"canal-stzdv","namespace":"kube-system"}
{"level":"info","timestamp":"2021-04-28T13:06:49.418065315Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"storageos-etcd-2hkff82fq2","namespace":"storageos-etcd"}
{"level":"info","timestamp":"2021-04-28T13:06:49.418092165Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"storageos-daemonset-6zrkk","namespace":"kube-system"}
{"level":"info","timestamp":"2021-04-28T13:06:49.418182036Z","msg":"skipping pod without storageos.com/fenced=true label set","name":"cattle-node-agent-sjspk","namespace":"cattle-system"}
{"level":"info","timestamp":"2021-04-28T13:06:49.439513312Z","msg":"pod has fenced label set and volume(s) still healthy after node failure, proceeding with fencing","pod":"mysql-0","namespace":"mysql"}
{"level":"info","timestamp":"2021-04-28T13:06:49.495807296Z","msg":"pod deleted","pod":"mysql-0","namespace":"mysql"}
{"level":"info","timestamp":"2021-04-28T13:06:49.505411162Z","msg":"volume attachment deleted","pod":"mysql-0","namespace":"mysql","pvc":"data-mysql-0","va":"csi-c2b44cee5a647e20d77e0e217dfaec07afd592eae57bcccc09b3447de653ae8c","node":"worker1"}
{"level":"info","timestamp":"2021-04-28T13:06:49.505439792Z","msg":"fenced pod"}
{"level":"info","timestamp":"2021-04-28T13:06:49.573478266Z","msg":"set scheduler","scheduler":"storageos-scheduler","pod":"mysql/mysql-0"}
...

The StorageOS API Manager detects all the pods that are on the failed node, and selects only the ones that meet the fencing criteria as described above. In this case only mysql-0 is selected for fencing.

Check the pod’s new node

$ kubectl -n mysql get pod --show-labels -owide
NAME      READY   STATUS    RESTARTS   AGE     IP          NODE             NOMINATED NODE   READINESS GATES   LABELS
mysql-0   1/1     Running   0          6m33s   10.42.3.7   worker2   <none>           <none>            app=mysql,controller-revision-hash=mysql-799fd74b87,env=prod,statefulset.kubernetes.io/pod-name=mysql-0,storageos.com/fenced=true

The pod mysql-0 started on a different node successfully.

Docs: StorageOS Volume Guide

Mon, 01 Jan 0001 00:00:00 +0000

Follow the recipes on this page to create your first PVC (Persistent Volume Claim) using StorageOS. StorageOS implements dynamic provisioning, so the creation of a PVC will automatically provision a PV (PersistentVolume) that can be used to persist data written by a Pod.

Create the PersistentVolumeClaim

You can find the basic examples in the StorageOS use-cases repository, in the 00-basic directory.

git clone https://github.com/storageos/use-cases.git storageos-usecases
cd storageos-usecases/00-basic

PVC definition

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-vol-1
spec:
  storageClassName: "fast" # StorageOS StorageClass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

The above PVC will dynamically provision a 5GB volume using the fast StorageClass. This StorageClass was created during the StorageOS install and triggers creation of a PersistentVolume by StorageOS.

For installations with CSI, you can create multiple StorageClasses in order to specify default labels.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: storageos-replicated
provisioner: csi.storageos.com # Provisioner when using CSI
parameters:
  csi.storage.k8s.io/fstype: ext4
  storageos.com/replicas: "1" # Enforces 1 replica for the Volume

  # Change the Namespace below if StorageOS doesn't run in kube-system
  csi.storage.k8s.io/node-publish-secret-namespace: kube-system       # Namespace that runs StorageOS Daemonset
  csi.storage.k8s.io/provisioner-secret-namespace: kube-system        # Namespace that runs StorageOS Daemonset
  csi.storage.k8s.io/controller-publish-secret-namespace: kube-system # Namespace that runs StorageOS Daemonset
  csi.storage.k8s.io/node-publish-secret-name: csi-node-publish-secret
  csi.storage.k8s.io/provisioner-secret-name: csi-provisioner-secret
  csi.storage.k8s.io/controller-publish-secret-name: csi-controller-publish-secret

The above StorageClass has the storageos.com/replicas label set. This label tells StorageOS to create a volume with a replica. Adding StorageOS feature labels to the StorageClass ensures all volumes created with the StorageClass have the same labels. For simplicity’s sake this example will use unreplicated volumes.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-vol-1
spec:
  storageClassName: "storageos-replicated" # Reference to the StorageClass
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

You can also choose to add the label in the PVC definition rather than the StorageClass. The PVC definition takes precedence over the SC.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-vol-1
  labels:
      storageos.com/replicas: "1"
spec:
  storageClassName: "fast"
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

The above PVC has the storageos.com/replicas label set. This label tells StorageOS to add a replica for the volume that is created. For the sake of keeping this example simple an unreplicated volume will be used.

Move into the examples folder and create a PVC using the PVC definition above.

$ # from storageos-usecases/00-basic
$ kubectl create -f ./pvc-basic.yaml

You can view the PVC that you have created with the command below

$ kubectl get pvc
NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc-1     Bound    pvc-f8ffa027-e821-11e8-bc0b-0ac77ccc61fa   5Gi        RWO            fast           1m

Create a pod that mounts the PVC created in step 2.

$ kubectl create -f ./pod.yaml

The command above creates a Pod that uses the PVC that was created in step 1.

apiVersion: v1
kind: Pod
metadata:
  name: d1
spec:
  containers:
    - name: debian
      image: debian:9-slim
      command: ["/bin/sleep"]
      args: [ "3600" ]
      volumeMounts:
        - mountPath: /mnt
          name: v1
  volumes:
    - name: v1
      persistentVolumeClaim:
        claimName: pvc-1

In the Pod definition above volume v1 references the PVC created in step 2, and is mounted in the pod at /mnt. In this example a debian image is used for the container but any container image with a shell would work for this example.

Confirm that the pod is up and running

$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
d1        1/1     Running   0          1m

Execute a shell inside the container and write some contents to a file
```
$ kubectl exec -it d1 -- bash
root@d1:/# echo "Hello World!" > /mnt/helloworld
root@d1:/# cat /mnt/helloworld
Hello World!
```
By writing to /mnt inside the container, the StorageOS volume created by the PVC is being written to. If you were to kill the pod and start it again on a new node, the helloworld file would still be avaliable.

If you wish to see more use cases with actual applications please see our Use Cases documentation.

Docs: Get

Mon, 01 Jan 0001 00:00:00 +0000

$ storageos get --help
Fetch basic details for resources

Usage:
  storageos get [command]

Available Commands:
  cluster      Fetch cluster-wide configuration details
  diagnostics  Fetch a cluster diagnostic bundle
  licence      Fetch current licence configuration details
  namespace    Retrieve basic details of cluster namespaces
  node         Retrieve basic details of nodes in the cluster
  policy-group Retrieve basic details of policy groups
  user         Fetch user details
  volume       Retrieve basic details of volumes

Flags:
  -h, --help   help for get

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Use "storageos get [command] --help" for more information about a command.

get cluster

$ storageos get cluster --help

Fetch cluster-wide configuration details

Usage:
  storageos get cluster [flags]

Examples:

$ storageos get cluster


Flags:
  -h, --help   help for cluster

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

get diagnostics

$ storageos get diagnostics --help

Fetch a cluster diagnostic bundle from the target node. Due to the work involved this command will run with a minimum command timeout duration of 1h, although accepts longer durations

Usage:
  storageos get diagnostics [flags]

Examples:

$ storageos get diagnostics

$ storageos get diagnostics --output-file ~/my-diagnostics.gz


Flags:
  -h, --help                 help for diagnostics
      --output-file string   writes the generated diagnostic bundle to a specified file path

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

get license

$ storageos get license --help

Fetch current licence configuration details

Usage:
  storageos get licence [flags]

Aliases:
  licence, license

Examples:

$ storageos get licence


Flags:
  -h, --help   help for licence

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

get namespace

$ storageos get namespace --help

Retrieve basic details of cluster namespaces

Usage:
  storageos get namespace [namespace names...] [flags]

Aliases:
  namespace, namespaces

Examples:

$ storageos get namespaces

$ storageos get namespace my-namespace-name


Flags:
  -h, --help                   help for namespace
  -l, --selector stringArray   filter returned results by a set of comma-separated label selectors

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

get node

storageos get node --help
Retrieve basic details of nodes in the cluster

Usage:
  storageos get node [node names...] [flags]

Aliases:
  node, nodes

Examples:

$ storageos get node my-node-name


Flags:
  -h, --help                   help for node
  -l, --selector stringArray   filter returned results by a set of comma-separated label selectors

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

get policy-group

$ storageos get policy-group --help

Retrieve basic details of policy groups

Usage:
  storageos get policy-group [policy-group names...] [flags]

Aliases:
  policy-group, policy-groups

Examples:

$ storageos get policy-group
$ storageos get policy-group my-policy-group-name
$ storageos get policy-group --use-ids my-policy-group-id


Flags:
  -h, --help   help for policy-group

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

get user

$ storageos get user --help

Fetch user details

Usage:
  storageos get user [user names...] [flags]

Aliases:
  user, users

Examples:

$ storageos get user my-username
$ storageos get user my-username-1 my-username-2
$ storageos get user --use-ids my-userid
$ storageos get user --use-ids my-userid-1 my-userid-2


Flags:
  -h, --help   help for user

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

get volume

$ storageos get volume --help

Retrieve basic details of volumes

Usage:
  storageos get volume [volume names...] [flags]

Aliases:
  volume, volumes

Examples:

$ storageos get volumes --all-namespaces

$ storageos get volume --namespace my-namespace-name my-volume-name


Flags:
  -A, --all-namespaces         retrieves volumes from all accessible namespaces. This option overrides the namespace configuration
  -h, --help                   help for volume
  -l, --selector stringArray   filter returned results by a set of comma-separated label selectors

Global Flags:
      --cache-dir string        set the directory used by the StorageOS CLI to cache data that can be used for future commands (default "/root/.cache/storageos")
  -c, --config string           specifies the config file path (default "/root/.config/storageos/config.yaml")
      --endpoints stringArray   set the list of endpoints which are used when connecting to the StorageOS API (default [http://localhost:5705])
  -n, --namespace string        specifies the namespace to operate within for commands that require one (default "default")
      --no-auth-cache           disable the CLI's caching of authentication sessions
  -o, --output string           specifies the output format (one of [json yaml text]) (default "text")
      --password string         set the StorageOS account password to authenticate with (default "storageos")
      --timeout duration        set the timeout duration to use for execution of the command (default 15s)
      --use-ids                 specify existing StorageOS resources by their unique identifiers instead of by their names
      --username string         set the StorageOS account username to authenticate as (default "storageos")

Docs: Google GKE

Mon, 01 Jan 0001 00:00:00 +0000

StorageOS is fully compatible with GKE when using the Ubunutu images. To install StorageOS on GKE, please follow our Kubernetes installation instructions page.

For StorageOS to work normally it is required to use the Ubuntu images for the GKE node pools. The default container image does not fulfil the system requirements because of the lack of the TCMU kernel modules.

For more details about the OS Distributions check the System Configuration page.

Google Anthos

StorageOS is is compatible with Google Anthos. However it is required that the Linux image distribution used fulfils the System Configuration prerequisites.

Once a Kubernetes cluster is provisioned, StorageOS can be installed following the instructions page.