Solving everyday problems with code

Using Envoy Proxy for client side load balancing

2020-01-18T00:00:00+00:00

We are using Envoy for client side load balancing.

We use elasticsearch exporter for getting prometheus format metrics for monitoring our clusters. Our elasticsearch cluster is not configured behind a load balancer, and we don’t have dns pointing to the multiple So we needed a way to allow smart client side load balancing, and most application doesn’t have this.

So rather than making this an application layer concerns, we can move the concerns to the network layers.

By using envoy, which is a out-of-process proxy, we can easily implement client side load balancing, and also move tls concerns out ( TLS origination ),

Envoy can do health checks, and passive ejections of endpoints.

With just a simple envoy side car running beside elasticsearch exporter, we can pull metrics in a highly available manner from a list of elasticsearch nodes, because the metrics are all the same from whichever nodes we query.

Envoy is amazing, and seeing adoption in many large projects, e.g. istio, aws app mesh.

It is also an enabler for a lot of capabilities, such as fault injections, traffic mirroring and shifting for safe canaries deployment.

Highly recommended to check it out! http://envoyproxy.io

Elasticsearch + LVM Striping

2019-01-06T00:00:00+00:00

At work, while managing a fleet of elasticsearch clusters, we have several linux boxes that have inconsistent number of block devices attached. So to simplify the playbook, we don’t want each hosts to have different host_vars with path.data settings, it will lead to configuration mess.

So our solutions is to go with linux logical volume manager, or lvm in short, to create a single logically volume with striping enabled, out of multiple physical disks.

With this solution, for every data nodes in our cluster, we have elasticsearch writing to a common path. We used striping to increase performance. For data durability, elasticsearch sharding model already taken care of that so we don’t need raid 10 properties.

some reference here on how to setup lvm: https://www.digitalocean.com/community/tutorials/an-introduction-to-lvm-concepts-terminology-and-operations

and some discussion around raid10 vs elasticsearch multiple data path performance. https://discuss.elastic.co/t/to-multi-path-data-or-hardware-raid-or-not/40143/3

Mitigating DDOS attacks on AWS

2018-01-23T00:00:00+00:00

Recently, a friend asked about how to handle ddos attacks on AWS. We discussed and theses are some pointers.

Lets say there are 3 components; nodejs api, redis, mysql which we need to setup for our web stack in AWS.

Basic architectures setup:

register the domain with route53 name servers for highly available dns.
point route53 to elastic load balancer.
elb points to nodejs servers.
nodejs api servers should sit in private subnet routed to NAT gateway. ( for outbound http calls to other services or downloading of apt / yum packages.
each components should have its own security groups; nodejs, redis, mysql.

Mitigate ddos attacks:

ensure nodejs web apps is design to be horizontally scalable. ( follow 12factor app design https://12factor.net )
use auto scaling group to take traffic spike.
use durable message queues in the architecture to smoothen the spike requests to other backend services.
use aws sheid for ddos protection; to detect ddos traffic patterns.
authenticate caller, and rate limit the api per caller.
enable vpc flow log to detect anomaly
use monitoring tools to detect attacks and trigger off alerts. ( elastalert, prometheus/grafana )
only exposing route53 and elastic load balancer to be public facing allow aws to take care of infrastructure availability.

Accessing the servers:

using either vpn and bastion, they would sit in the public subnet allowing ssh. then ssh agent forward to access web, mysql, redis servers.

Running Twemproxy In Kubernetes

2018-01-20T00:00:00+00:00

In this post, i would like share on how to setup twemproxy for redis cluster sharding running in kubernetes.

Setup

These are the tasks required:

Running twemproxy as a single replica deployment in kubernetes.
Running redis replicas as statefulsets with with in kubernetes.
Allowing other pods in kubernetes to access twemproxy.
Allowing twemproxy to connect to the redis’s pods.

Twemproxy

We can run twemproxy as a single deployment in kubernetes. We can use the following community image jgoodall/twemproxy

twemproxy’s deployment & service spec:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: twemproxy
  name: twemproxy
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: twemproxy
    spec:
      containers:
      - image: jgoodall/twemproxy:latest
        name: twemproxy
        ports:
        - containerPort: 63791
        - containerPort: 63792
        - containerPort: 6222
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: twemproxy
  name: twemproxy
spec:
  selector:
    app: twemproxy
  ports:
  - name: "63791"
    port: 63791
    targetPort: 63791
  - name: "63792"
    port: 63792
    targetPort: 63792
  - name: "6222"
    port: 6222
    targetPort: 6222

The configuration above will deploy twemproxy’s deployment and service. service in kubernetes allow pods to network with each other, this service will allow other pods to reach twemproxy.

twemproxy.yml:

local:
  listen: 0.0.0.0:63791
  hash: fnv1a_64
  hash_tag: "{}"
  distribution: ketama
  timeout: 500
  redis: true
  servers:
    - redis-0:6379:1
    - redis-1:6379:1
    - redis-2:6379:1

session:
  listen: 0.0.0.0:63792
  hash: fnv1a_64
  hash_tag: "{}"
  distribution: ketama
  auto_eject_hosts: true
  server_retry_timeout: 30000
  server_failure_limit: 3
  timeout: 500
  redis: true
  servers:
    - redis-0:6379:1
    - redis-1:6379:1
    - redis-2:6379:1

Above is twemproxy’s configuration which need to be mount as configmap volume in twemproxy’s deployment.

Redis Statefulset

Next, we need to create redis’s statefulset. Statefulset with 3 replicas will create the following pods redis-0, redis-1 and redis-2. Pods created by statefulset will be labeled as statefulset.kubernetes.io/pod-name: , we can use this as selector for the service. Unfortunably, we will still need to create 3 services to allow each pod to be addressable by twemproxy,

redis statefulset:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  selector:
    matchLabels:
      app: redis
  serviceName: "redis"
  replicas: 3
  template:
    metadata:
      labels:
        app: redis
    spec:
      terminationGracePeriodSeconds: 3
      containers:
      - name: redis
        image: redis:3.0.7
        ports:
        - containerPort: 6379

redis services:

apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  ports:
  - port: 6379
    name: redis
  clusterIP: None
  selector:
    app: redis
---
apiVersion: v1
kind: Service
metadata:
  name: redis-1
  labels:
    app: redis-1
spec:
  ports:
  - port: 6379
    name: redis-1
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: redis-1
---
apiVersion: v1
kind: Service
metadata:
  name: redis-1
  labels:
    app: redis-1
spec:
  ports:
  - port: 6379
    name: redis-1
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: redis-1
---
apiVersion: v1
kind: Service
metadata:
  name: redis-2
  labels:
    app: redis-2
spec:
  ports:
  - port: 6379
    name: redis-2
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: redis-2

Easy Setup Of OpenVPN Bastion Server Using Docker

2018-01-08T00:00:00+00:00

OpenVPN is a full-featured open source Secure Socket Layer (SSL) VPN solution. While trying to secure amazon elasticsearch service, we decided that an OpenVPN bastion server would be the simplest way for us to securely access kibana and elasticsearch service from our home, office, or anywhere as long as we have proper client certificate.

Setup

We use a small t2.micro amazon linux ami with elastic ip attached and docker installed on the machine, the security group is set to open udp on port 1194 to 0.0.0.0/32 cidr range. The server sit in a public subnet of the vpc.

OpenVPN itself is ran using the docker image https://github.com/kylemanna/docker-openvpn, this is an amazing image that really ease the setup and creation of client certificate of the OpenVPN server.

Once we have OpenVPN server up and running in docker, and the client .ovpn files generated, the files can be distributed to the users that require access. We are ready to connect to the vpn.

I use TunnelBlick as my openvpn client, which is free. Simply install it and drag the .ovpn file to TunnelBlick and everything is setup.

Once connected, we could securely access amazon elasticsearch and kibana using its vpc endpoint.

Conclusion

With Docker, the installation of OpenVPN is easy and simplified. Using OpenVPN as a bastion server is a good way to allow secure access to our aws’s vpc from anywhere.

Troubleshooting Terraform And AWS RequestLimitExceeded

2017-12-27T00:00:00+00:00

Recently, when running terraform apply, my script has been failing intermittently when provisioning around 90 ec2 instances.

Luckily, terraform provide debugging output by passing TF_LOG=debug. It is interesting that terraform uses aws-sdk-go under the hood to issues API calls to AWS.

The root cause of my script failing was due to raising the parallelism count beyond the default ( 10 ). When tracing the debug logs, i noticed a lot of API requests to AWS that terraform made has failed with RequestLimitExceeded, after setting back to default, i am able to consistently provision 200 ec2 instances without failure.

There are more discussion about this issue on github here.

What Is Docker?

2017-12-26T00:00:00+00:00

Docker is the probably the hottest technology right now in the technology space. In this post, i am writing about the basic of docker and the challenges it solves.

What Is Docker?

Docker is a platform which can be used to package, distribute and run your applications.
Docker provides an easy and efficient way to encapsulate applications (e.g. a Java web application) and any required infrastructure to run that application (e.g. Red hat Linux OS, Apache web server, Tomcat application server, mySQL database etc.) as a single “Docker image”.
The image can then be used to launch a Docker container which makes the containerized application available from the host.

DevOps Matrix From Hell

DevOps Matrix From Hell is the challenge of packaging any application, regardless of language/frameworks/dependencies, so that it can run on any cloud, regardless of operating systems/hardware/infrastructure. This is due to the multipicity of many language/technology/framework stacks that need to installed on so different hardware and operating system environments.

Docker act as a system container for code. It encapsulate all the dependencies within a container. The container can then be run the same way in any environment. Docker solved for the matrix from hell by decoupling the application from the underlying operating system and hardware. It did this by packaging all dependencies inside Docker containers, including the OS.

This makes Docker containers “portable”, i.e. they can run on any cloud or machine without the dreaded “it works on this machine” problems. The manipulation and operations of the docker containers is the same for all supported hardware platforms.

Containers vs Virtual Machines

Docker container is a kind of light weight virtual machine with considerably smaller memory and disk space footprint than a full blown virtual machine.

Virtual machines take up a lot of system resources. Each VM runs not just a full copy of an operating system, but a virtual copy of all the hardware that the operating system needs to run. This quickly adds up to a lot of RAM and CPU cycles. In contrast, all that a container requires is enough of an operating system, supporting programs and libraries, and system resources to run a specific program.

What this means in practice is you can put two to three times as many as applications on a single server with containers than you can with a VM.

Seperation Of Concerns

Developers can focus on packaging applications and dependencies as containers, without worrying about underlying infrastructure.
Developers worries about what is inside the container. e.g. code, libraries, package manager, apps, data, etc
Operators can focus on managing containers, without worrying about the contents of those containers.
Operators worries about what is outside of the container. e.g. networking, logging, monitoring, dynamic configuration.

For Developers

Build once… (finally) run anywhere.
No worries about missing dependencies, packages, or mismatched dependencies versioning.
Reduce/eliminate concerns about compatibility on different platforms, either your own or your customers.
Consistent development environments for your entire team. All developers use the same OS, same system libraries, same language runtime, no matter what host OS they are using
Deployment is easy. If it runs in your container, it will run on your server just the same.

For Operators

Deployment is more efficient, consistent, and repeatable.
Eliminate inconsistencies between development, test, production, and customer environments.
Support segregation of duties.
Significantly improves the speed and reliability of continuous deployment and continuous.
containers are so lightweight, address significant performance, costs, deployment, and portability issues normally associated with virtual machines.

Deep Dive

Docker works by using linux containers and kernel facilities.

Linux kernel namespaces provides the isolation.
Linux kernel cgroups provide the resource limiting ( cpu, memory, I/O )
AUFS provide the union file system for docker. ( to avoid duplicating a complete set of files each time you run an image as a new container)

Tmux

2017-12-01T00:00:00+00:00

Recently, I discovered the tool; Tmux. This is an awesome tool that has improved my productivity and i feel i am just scratching the surface of what it can do.

What is TMUX?

Tmux is a “terminal multiplexer”, it enables a number of terminals (or windows) to be accessed and controlled from a single terminal. tmux is intended to be a simple, modern, BSD-licensed alternative to programs such as GNU screen.

At the very basic, it allows me to slice up my terminal screens into windows and panes so i can visualize multiple workflow in one screen. This is a lot of productivity improvement for me as i am using a large monitor. Using tmux, i was able to look at more stuffs without creating new bash terminal and switching around.

Advanced usage

Advanced usage of tmux is sessions, we can actually installed tmux on a remote machine and then let background tasks run in tmux sessions. Even after we close ssh connections, we can ssh back into the machine and reattached to the tmux session which is still running, this can be use as a replacement for nohup.

Kubernetes, Kops, Logging and Monitoring

2017-10-25T00:00:00+00:00

TL;DR

To quickly setup kubernetes and monitoring using kops, follow my example github repository.

Background

Recently, i am helping our client move their applications to containerized microservices.

Moving to docker containers provide many benefits. However, docker alone does not allow us to orchestrate our containers in productions. To solve that problem, we need a container platform/orchestrator technology. We adopted kubernetes as our containers orchestrator of choice. Kubernetes allow us to horizontally scale our microservices across nodes to provide highly availablility and scalability.

In this post, i would like to document how we are explored the various orchestrator options and how we are running kubernetes in production.

KOPS ( Kubernetes Operation)

There are various ways to operate a kubernetes clusters and it can range from being really complex to simple hosted cloud solutions.

These are some of the various tools and platforms:

Kops ( run on aws )
Openshift
Google container engine ( google cloud )
Jujucharms, canonical distribution of kubernetes ( baremetal, metal cloud ( Machine as a service ), virtual machines )
Kubeadm ( baremetal, virtual machines )
More..

After evaluating the various options, we decided with kops which will provision our cluster on aws. We found kops to be the most robust and it works seamlessly for us. Using kops we get multi-az nodes deployment and auto scaling group at the cluster level. On top of it, our client is already on aws platform. kops project has a strong community behind it.

Learn more about kops here

Kubernetes Dashboard

The first deployment to run in kubernetes is the kubernetes dashboard. Running kubernetes dashboard allow us to visualize what kubernetes resources are running or getting scheduled, destroyed. It also shows quickly let us know if there is any errors in the cluster.

Datastore

We run elasticsearch outside of kubernetes to store log data generated by fluentd and metricbeat daemonsets. Running ELK outside kubernetes is more robust, in case kubernetes clusters goes down, we can still see logs on what happened before kubernetes died.

Centralized logging

It is important that before going into production we setup centralized logging of pods/containers running in kubernetes. Pods/containers are running in different kubernetes nodes, scaling up and down, it is hard to understand what is going on without centralized logging. What we want is a single place to look logs from stdout and stderr streams of running containers to quickly detect and diagnose issues.

We run fluentd as daemonsets on every kubernetes nodes. In kubernetes, all containers write their stdout/stderr streams to the /var/log/containers/ folders. Fluentd can be configured to monitor this folder and then send logs into elasticsearch.

Monitoring

Monitoring is done using metricbeat and kube-state-metric. kube-state-metric is a simple service that listens to the Kubernetes API server and generates metrics about the state of the objects. Metricbeat grab data from kube-state-metric into elasticsearch.

Kubernetes dashboard in kibana:

Metricbeat dashboard in kibana:

Conclusion

Centralized logging and monitoring is important to setup before moving to production with kubernetes. I have created an example kubernetes github repo that contains the kubernetes yaml files to deploy everything we discussed.

Deploying Elastic Stack With Ansible Playbook

2017-10-15T00:00:00+00:00

As a devops consultant, I have worked with ansible quite a bit. Following the principles of infrastructure as code and with the motivation to automate as much infrastructure work as possible, many of our configuration are automated and manage as normal code, we commit them with git and even run yaml syntax check in our ci/cd pipeline.

One of the configuration scripts I did was for automating the configuration of the elastic stack.

This script is availabe at my github repo, https://github.com/Misterhex/ansible-elk

This script configure:

elasticsearch cluster with N number of nodes.
kibana
curator
metricbeats on all compute nodes.
import default dashboard for metricbeat

Below is the main playbook file site.yml

---
- hosts: all
  gather_facts: false
  become: true
  roles:
    - python

- hosts: elasticsearch
  become: true
  vars:
    cluster_name: openstack_test
    es_heap_size: 2g
  roles:
    - java8
    - elastic5x
    - elasticsearch

- hosts: kibana
  become: true
  vars:
    kibana_port: 5601
  roles:
    - elastic5x
    - kibana

- hosts: compute
  become: true
  roles:
    - elastic5x
    - metricbeat
    - filebeat

- hosts: curator
  become: true
  vars:
    delete_older_than_days: 1
  roles:
    - curator

- hosts: compute[0]
  tasks: 
  - name: import metricbeats dashboards to kibana
    shell: /usr/share/metricbeat/scripts/import_dashboards
  - name: import filebeat dashboards to kibana
    shell: /usr/share/filebeat/scripts/import_dashboards

Concepts

Playbook contains plays that will run when executed, plays maps to inventory. Plays contains Roles. Roles are basically recommended encapsulation of a series of steps that invoke ansible modules. Modules are built-in idempontent functions that configure the target servers. Servers are grouped and defined in inventories, they are defined in an inventory file.

Below is an example inventory file:

[all_servers:children]
compute
elasticsearch
kibana

[all_servers:vars]
ansible_user = ubuntu
ansible_ssh_private_key_file = xxx

[compute]
xxx.xxx.xxx.xxx

[elasticsearch]
xxx.xxx.xxx.xxx

[kibana]
xxx.xxx.xxx.xxx

Running the playbook

To run the playbook, it is as simple as:

ansible-playbook -i inventory-sample site.yml

Conclusion

In conclusion, ansible is a simple yet awesome tools for automating configuration of softwares/services. Get started with installation here.