{:new_window: target="_blank"} {:shortdesc: .shortdesc} {:screen: .screen} {:pre: .pre} {:table: .aria-labeledby="caption"} {:codeblock: .codeblock} {:tip: .tip} {:download: .download}

{: #cs_security}

You can use built-in security features for risk analysis and security protection. These features help you to protect your cluster infrastructure and network communication, isolate your compute resources, and ensure security compliance across your infrastructure components and container deployments. {: shortdesc}

Built-in cluster security settings in {{site.data.keyword.containershort_notm}}
Kubernetes master	The Kubernetes master in each cluster is managed by IBM and is highly available. It includes {{site.data.keyword.containershort_notm}} security settings that ensure security compliance and secure communication to and from the worker nodes. Updates are performed by IBM as required. The dedicated Kubernetes master centrally controls and monitors all Kubernetes resources in the cluster. Based on the deployment requirements and capacity in the cluster, the Kubernetes master automatically schedules your containerized apps to deploy across available worker nodes. For more information, see [Kubernetes master security](#cs_security_master).
Worker node	Containers are deployed on worker nodes that are dedicated to a cluster and that ensure compute, network, and storage isolation for IBM customers. {{site.data.keyword.containershort_notm}} provides built-in security features to keep your worker nodes secure on the private and public network and to ensure worker node security compliance. For more information, see [Worker node security](#cs_security_worker).
Images	As the cluster admin, you can set up your own secure Docker image repository in {{site.data.keyword.registryshort_notm}} where you can store and share Docker images between your cluster users. To ensure safe container deployments, every image in your private registry is scanned by Vulnerability Advisor. Vulnerability Advisor is a component of {{site.data.keyword.registryshort_notm}} that scans for potential vulnerabilities, makes security recommendations, and provides instructions to resolve vulnerabilities. For more information, see [Image security in {{site.data.keyword.containershort_notm}}](#cs_security_deployment).

{: #cs_security_master}

Review the built-in Kubernetes master security features to protect the Kubernetes master and to secure the cluster network communication. {: shortdesc}

Fully managed and dedicated Kubernetes master

Every Kubernetes cluster in {{site.data.keyword.containershort_notm}} is controlled by a dedicated Kubernetes master that is managed by IBM in an IBM-owned IBM Bluemix Infrastructure (SoftLayer) account. The Kubernetes master is set up with the following dedicated components that are not shared with other IBM customers.

• etcd data store: Stores all Kubernetes resources of a cluster, such as Services, Deployments, and Pods. Kubernetes ConfigMaps and Secrets are app data that are stored as key value pairs so that they can be used by an app that runs in a pod. Data in etcd is stored on an encrypted disk that is managed by IBM and is encrypted via TLS when sent to a pod to assure data protection and integrity.
• kube-apiserver: Serves as the main entry point for all requests from the worker node to the Kubernetes master. The kube-apiserver validates and processes requests and can read from and write to the etcd data store.
• kube-scheduler: Decides where to deploy pods, taking into account capacity and performance needs, hardware and software policy constraints, anti-affinity specifications, and workload requirements. If no worker node can be found that matches the requirements, the pod is not deployed in the cluster.
• kube-controller-manager: Responsible for monitoring replica sets, and creating corresponding pods to achieve the desired state.
• OpenVPN: {{site.data.keyword.containershort_notm}}-specific component to provide secured network connectivity for all Kubernetes master to worker node communication.

TLS secured network connectivity for all worker node to Kubernetes master communication

To secure the network communication to the Kubernetes master, {{site.data.keyword.containershort_notm}} generates TLS certificates that encrypts the communication to and from the kube-apiserver and etcd data store components for every cluster. These certificates are never shared across clusters or across Kubernetes master components.

OpenVPN secured network connectivity for all Kubernetes master to worker node communication

Although Kubernetes secures the communication between the Kubernetes master and worker nodes by using the `https` protocol, no authentication is provided on the worker node by default. To secure this communication, {{site.data.keyword.containershort_notm}} automatically sets up an OpenVPN connection between the Kubernetes master and the worker node when the cluster is created.

Continuous Kubernetes master network monitoring

Every Kubernetes master is continuously monitored by IBM to control and remediate process level Denial-Of-Service (DOS) attacks.

Kubernetes master node security compliance

{{site.data.keyword.containershort_notm}} automatically scans every node where the Kubernetes master is deployed for vulnerabilities found in Kubernetes and OS-sepcific security fixes that need to be applied to assure master node protection. If vulnerabilities are found, {{site.data.keyword.containershort_notm}} automatically applies fixes and resolves vulnerabilities on behalf of the user.

{: #cs_security_worker}

Review the built-in worker node security features to protect the worker node environment and to assure resource, network and storage isolation. {: shortdesc}

Compute, network and storage infrastructure isolation

When you create a cluster, virtual machines are provisioned as worker nodes in the customer IBM Bluemix Infrastructure (SoftLayer) account or in the dedicated IBM Bluemix Infrastructure (SoftLayer) account by IBM. Worker nodes are dedicated to a cluster and do not host workloads of other clusters.
Every {{site.data.keyword.Bluemix_notm}} account is set up with IBM Bluemix Infrastructure (SoftLayer) VLANs to assure quality network performance and isolation on the worker nodes.
To persist data in your cluster, you can provision dedicated NFS based file storage from IBM Bluemix Infrastructure (SoftLayer) and leverage the built-in data security features of that platform.

Secured worker node set up

Every worker node is set up with an Ubuntu operating system that cannot be changed by the user. To protect the operating system of the worker nodes from potential attacks, every worker node is configured with expert firewall settings that are enforced by Linux iptable rules.
All containers that run on Kubernetes are protected by predefined Calico network policy settings that are configured on every worker node during cluster creation. This set up assures secure network communication between worker nodes and pods. To further restrict the actions that a container can perform on the worker node, users can choose to configure [AppArmor policies ](https://kubernetes.io/docs/tutorials/clusters/apparmor/) on the worker nodes.
By default, SSH access for the root user is disabled on the worker node. If you want to install additional features on your worker node, you can use [Kubernetes daemon sets ](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset) for everything that you want to run on every worker node, or [Kubernetes jobs ](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/) for any one-time action you must execute.

Kubernetes worker node security compliance

IBM works with internal and external security advisory teams to address potential security compliance vulnerabilities. IBM maintains SSH access to the worker nodes in order to deploy updates and security patches to the operating system.
Important: Reboot your worker nodes on a regular basis to ensure the installation of the updates and security patches that are automatically deployed to the operating system. IBM does not reboot your worker nodes.

Support for IBM Bluemix Infrastructure (SoftLayer) network firewalls

{{site.data.keyword.containershort_notm}} is compatible with all [IBM Bluemix Infrastructure (SoftLayer) firewall offerings ](https://www.ibm.com/cloud-computing/bluemix/network-security). On {{site.data.keyword.Bluemix_notm}} Public, you can set up a firewall with custom network policies to provide dedicated network security for your cluster and to detect and remediate network intrusion. For example, you might choose to set up a [Vyatta ](https://knowledgelayer.softlayer.com/topic/vyatta-1) to act as your firewall and block unwanted traffic. When you set up a firewall, [you must also open up the required ports and IP addresses](#opening_ports) for each region so that the master and the worker nodes can communicate. On {{site.data.keyword.Bluemix_notm}} Dedicated, firewalls, DataPower, Fortigate, and DNS are already configured as part of the standard dedicated environment deployment.

Keep services private or selectively expose services and apps to the public internet

You can choose to keep your services and apps private and leverage the built-in security features described in this topic to assure secured communication between worker nodes and pods. To expose services and apps to the public internet, you can leverage the Ingress and load balancer support to securely make your services publicly available.

Securely connect your worker nodes and apps to an on-premise data center

You can set up a Vyatta Gateway Appliance or Fortigate Appliance to configure an IPSec VPN endpoint that connects your Kubernetes cluster with an on-premise data center. Over an encrypted tunnel, all services that run in your Kubernetes cluster can communicate securely with on-premise apps, such as user directories, databases, or mainframes. For more information, see [Connecting a cluster to an on-premise data center ](https://www.ibm.com/blogs/bluemix/2017/07/kubernetes-and-bluemix-container-based-workloads-part4/).

Continuous monitoring and logging of cluster activity

For standard clusters, all cluster-related events, such as adding a worker node, rolling update progress, or capacity usage information are logged and monitored by {{site.data.keyword.containershort_notm}} and sent to the IBM Monitoring and Logging Service.

{: #opening_ports}

Review these situations in which you might need to open specific ports and IP addresses in your firewalls:

• To allow communication between the Kubernetes master and the worker nodes when either a firewall is set up for the worker nodes or the firewall settings are customized in your IBM Bluemix Infrastructure (SoftLayer) account

• To access the load balancer or Ingress controller from outside of the cluster

• To run kubectl commands from your local system when corporate network policies prevent access to public internet endpoints via proxies or firewalls

  1. Note the public IP address for all your worker nodes in the cluster.

     bx cs workers <cluster_name_or_id>
     

     {: pre}

  2. In your firewall for OUTBOUND connectivity from your worker nodes, allow outgoing network traffic from the source worker node to the destination TCP/UDP port range 20000-32767 and port 443 for <each_worker_node_publicIP>, and the following IP addresses and network groups.

     • Important: You must allow outgoing traffic to port 443 and all the locations within the region to each other, to balance the load during the bootstrapping process. For example, if your cluster is in US South, you must allow traffic from port 443 to dal10 and dal12, and from dal10 and dal12 to each other.

  Region	Location	IP address
  AP South	mel01 syd01	168.1.97.67 168.1.8.195
  EU Central	ams03 fra02	169.50.169.110 169.50.56.174
  UK South	lon02 lon04	159.122.242.78 158.175.65.170
  US East	wdc06 wdc07	169.60.73.142 169.61.83.62
  US South	dal10 dal12 dal13	169.46.7.238 169.47.70.10 169.60.128.2

3. Allow outgoing network traffic from the worker nodes to {{site.data.keyword.registrylong_notm}}:
   • TCP port 443 FROM <each_worker_node_publicIP> TO <registry_publicIP>
   • Replace <registry_publicIP> with all of the addresses for registry regions to which you want to allow traffic:

Registry IP addresses
registry.au-syd.bluemix.net	168.1.45.160/27 168.1.139.32/27
registry.eu-de.bluemix.net	169.50.56.144/28 159.8.73.80/28
registry.eu-gb.bluemix.net	159.8.188.160/27 169.50.153.64/27
registry.ng.bluemix.net	169.55.39.112/28 169.46.9.0/27 169.55.211.0/27

4. Optional: Allow outgoing network traffic from the worker nodes to {{site.data.keyword.monitoringlong_notm}} and {{site.data.keyword.loganalysislong_notm}} services:

   • TCP port 443, port 9095 FROM <each_worker_node_publicIP> TO <monitoring_publicIP>
   • Replace <monitoring_publicIP> with all of the addresses for the monitoring regions to which you want to allow traffic:

     Monitoring Public IP addresses
     metrics.eu-de.bluemix.net	159.122.78.136/29
     metrics.eu-gb.bluemix.net	169.50.196.136/29
     metrics.ng.bluemix.net	169.47.204.128/29

Logging Public IP addresses
ingest.logging.eu-de.bluemix.net	169.50.25.125
ingest.logging.eu-gb.bluemix.net	169.50.115.113
ingest.logging.ng.bluemix.net	169.48.79.236 169.46.186.113

5. For private firewalls, allow the appropriate IBM Bluemix Infrastructure (SoftLayer) private IP ranges. Consult this link beginning with the Backend (private) Network section.

   • Add all of the locations within the regions that you are using
   • Note that you must add the dal01 location (data center)
   • Open ports 80 and 443 to allow the cluster bootstrapping process

6. Optional: To access the load balancer from outside of the VLAN, open the port for incoming network traffic on the specific IP address of that load balancer.

7. Optional: To access the Ingress controller from outside of the VLAN, open either port 80 or 443 for incoming network traffic on the specific IP address of that Ingress controller, depending on which port you have configured.

{: #cs_security_network_policies}

Every Kubernetes cluster is set up with a network plug-in that is called Calico. Default network policies are set up to secure the public network interface of every worker node. You can use Calico and native Kubernetes capabilities to configure more network policies for a cluster when you have unique security requirements. These network policies specify the network traffic that you want to allow or block to and from a pod in a cluster. {: shortdesc}

You can choose between Calico and native Kubernetes capabilities to create network policies for your cluster. You might use Kubernetes network policies to get started, but for more robust capabilities, use the Calico network policies.

• [Kubernetes network policies ](https://kubernetes.io/docs/concepts/services-networking/network-policies/): Some basic options are provided, such as specifying which pods can communicate with each other. Incoming network traffic can be allowed or blocked for a protocol and port. This traffic can be filtered based on the labels and Kubernetes namespaces of the pod that is trying to connect to other pods.
  These policies can be applied by using `kubectl` commands or the Kubernetes APIs. When these policies are applied, they are converted into Calico network policies and Calico enforces these policies.
• [Calico network policies ](http://docs.projectcalico.org/v2.4/getting-started/kubernetes/tutorials/advanced-policy): These policies are a superset of the Kubernetes network policies and enhance the native Kubernetes capabilities with the following features.
• Allow or block network traffic on specific network interfaces, not only Kubernetes pod traffic.
• Allow or block incoming (ingress) and outgoing (egress) network traffic.
• [Block incoming (ingress) traffic to LoadBalancer or NodePort Kubernetes services](#cs_block_ingress).
• Allow or block traffic that is based on a source or destination IP address or CIDR.



These policies are applied by using calicoctl commands. Calico enforces these policies, including any Kubernetes network policies that are converted to Calico policies, by setting up Linux iptables rules on the Kubernetes worker nodes. Iptables rules serve as a firewall for the worker node to define the characteristics that the network traffic must meet to be forwarded to the targeted resource.

{: #concept_nq1_2rn_4z}

When a cluster is created, default network policies are automatically set up for the public network interface of each worker node to limit incoming traffic for a worker node from the public internet. These policies do not affect pod to pod traffic and are set up to allow access to the Kubernetes nodeport, load balancer, and Ingress services.

Default policies are not applied to pods directly; they are applied to the public network interface of a worker node by using a Calico host endpoint . When a host endpoint is created in Calico, all traffic to and from that worker node's network interface is blocked, unless that traffic is allowed by a policy.

Note that a policy to allow SSH does not exist, so SSH access by way of the public network interface is blocked, as are all other ports that do not have a policy to open them. SSH access, and other access, is available on the private network interface of each worker node.

Important: Do not remove policies that are applied to a host endpoint unless you fully understand the policy and know that you do not need the traffic that is being allowed by the policy.

Default policies for each cluster
allow-all-outbound	Allows all outbound traffic.
allow-icmp	Allows incoming icmp packets (pings).
allow-kubelet-port	Allows all incoming traffic to port 10250, which is the port that is used by the kubelet. This policy allows `kubectl logs` and `kubectl exec` to work properly in the Kubernetes cluster.
allow-node-port-dnat	Allows incoming nodeport, load balancer, and ingress service traffic to the pods that those services are exposing. Note the port that those services expose on the public interface does not have to be specified because Kubernetes uses destination network address translation (DNAT) to forward those service requests to the correct pods. That forwarding takes place before the host endpoint policies are applied in iptables.
allow-sys-mgmt	Allows incoming connections for specific IBM Bluemix Infrastructure (SoftLayer) systems that are used to manage the worker nodes.
allow-vrrp	Allow vrrp packets, which are used to monitor and move virtual IP addresses between worker nodes.

{: #adding_network_policies}

In most cases, the default policies do not need to be changed. Only advanced scenarios might require changes. If you find that you must make changes, install the Calico CLI and create your own network policies

Before you begin:

1. Install the {{site.data.keyword.containershort_notm}} and Kubernetes CLIs.
2. Create a lite or standard cluster.
3. Target the Kubernetes CLI to the cluster. Include the --admin option with the bx cs cluster-config command, which is used to download the certificates and permission files. This download also includes the keys for the Super User role, which you need to run Calico commands.

bx cs cluster-config <cluster_name> --admin

{: pre}

Note: Calico CLI version 1.4.0 is supported.

To add network policies:

1. Install the Calico CLI.

   1. Download the Calico CLI .

      Tip: If you are using Windows, install the Calico CLI in the same directory as the {{site.data.keyword.Bluemix_notm}} CLI. This setup saves you some filepath changes when you run commands later.

   2. For OSX and Linux users, complete the following steps.

      1. Move the executable file to the /usr/local/bin directory.

         • Linux:

         mv /<path_to_file>/calicoctl /usr/local/bin/calicoctl
         

         {: pre}

         • OS X:

         mv /<path_to_file>/calicoctl-darwin-amd64 /usr/local/bin/calicoctl
         

         {: pre}

      2. Make the file executable.

         chmod +x /usr/local/bin/calicoctl
         

         {: pre}

   3. Verify that the calico commands ran properly by checking the Calico CLI client version.

      calicoctl version
      

      {: pre}

2. Configure the Calico CLI.

   1. For Linux and OS X, create the /etc/calico directory. For Windows, any directory can be used.

   sudo mkdir -p /etc/calico/
   

   {: pre}

   2. Create a calicoctl.cfg file.

      • Linux and OS X:

      sudo vi /etc/calico/calicoctl.cfg
      

      {: pre}

      • Windows: Create the file with a text editor.

   3. Enter the following information in the calicoctl.cfg file.

      apiVersion: v1
      kind: calicoApiConfig
      metadata:
      spec:
          etcdEndpoints: <ETCD_URL>
          etcdKeyFile: <CERTS_DIR>/admin-key.pem
          etcdCertFile: <CERTS_DIR>/admin.pem
          etcdCACertFile: <CERTS_DIR>/<ca-*pem_file>
      

      {: codeblock}

      1. Retrieve the <ETCD_URL>. If this command fails with a calico-config not found error, then see this troubleshooting topic.

      • Linux and OS X:

        kubectl get cm -n kube-system calico-config -o yaml | grep "etcd_endpoints:" | awk '{ print $2 }'
        

        {: pre}

      • Output example:

        https://169.1.1.1:30001
        

        {: screen}

      • Windows:

        1. Get the calico configuration values from the config map.
           

           kubectl get cm -n kube-system calico-config -o yaml

           
           
        2. In the data section, locate the etcd_endpoints value. Example: https://169.1.1.1:30001

      2. Retrieve the <CERTS_DIR>, the directory that the Kubernetes certificates are downloaded in.

         • Linux and OS X:

         dirname $KUBECONFIG
         

         {: pre}

         Output example:
         

         /home/sysadmin/.bluemix/plugins/container-service/clusters/<cluster_name>-admin/
         

         {: screen}

         • Windows:

         ECHO %KUBECONFIG%
         

         {: pre}

         Output example:
         

         C:/Users/<user>/.bluemix/plugins/container-service/<cluster_name>-admin/kube-config-prod-<location>-<cluster_name>.yml
         

         {: screen}

         Note: To get the directory path, remove the file name kube-config-prod-<location>-<cluster_name>.yml from the end of the output.

      3. Retrieve the ca-*pem_file.

         • Linux and OS X:

         ls `dirname $KUBECONFIG` | grep "ca-"
         

         {: pre}

         • Windows:
         1. Open the directory you retrieved in the last step.
            

            C:\Users\\.bluemix\plugins\container-service\<cluster_name>-admin\

         2. Locate the ca-*pem_file file.

      4. Verify that the Calico configuration is working correctly.

         • Linux and OS X:

         calicoctl get nodes
         

         {: pre}

         • Windows:

         calicoctl get nodes --config=<path_to_>/calicoctl.cfg
         

         {: pre}

         Output:

         NAME
         kube-dal10-crc21191ee3997497ca90c8173bbdaf560-w1.cloud.ibm
         kube-dal10-crc21191ee3997497ca90c8173bbdaf560-w2.cloud.ibm
         kube-dal10-crc21191ee3997497ca90c8173bbdaf560-w3.cloud.ibm
         

         {: screen}

3. Examine the existing network policies.

   • View the Calico host endpoint.

   calicoctl get hostendpoint -o yaml
   

   {: pre}

   • View all of the Calico and Kubernetes network policies that were created for the cluster. This list includes policies that might not be applied to any pods or hosts yet. For a network policy to be enforced, it must find a Kubernetes resource that matches the selector that was defined in the Calico network policy.

   calicoctl get policy -o wide
   

   {: pre}

   • View details for a network policy.

   calicoctl get policy -o yaml <policy_name>
   

   {: pre}

   • View the details of all network policies for the cluster.

   calicoctl get policy -o yaml
   

   {: pre}

4. Create the Calico network policies to allow or block traffic.

   1. Define your Calico network policy by creating a configuration script (.yaml). These configuration files include the selectors that describe what pods, namespaces, or hosts that these policies apply to. Refer to these sample Calico policies to help you create your own.

   2. Apply the policies to the cluster.

      • Linux and OS X:

      calicoctl apply -f <policy_file_name.yaml>
      

      {: pre}

      • Windows:

      calicoctl apply -f <path_to_>/<policy_file_name.yaml> --config=<path_to_>/calicoctl.cfg
      

      {: pre}

apiVersion: v1
kind: policy
metadata:
  name: deny-kube-node-port-services
spec:
  preDNAT: true
  selector: ibm.role in { 'worker_public', 'master_public' }
  ingress:
  - action: deny
    protocol: tcp
    destination:
      ports:
      - 30000:32767
  - action: deny
    protocol: udp
    destination:
      ports:
      - 30000:32767

/opt/bin/calicoctl apply -f deny-kube-node-port-services.yaml