Cloud-native application security is a cloud-first approach used to deploy applications securely at scale by embedding security into the software development lifecycle to detect vulnerabilities earlier. This article will walk through the critical challenges of cloud-native application security, demonstrate how to build security into the CI/CD pipeline, and introduce the core practices of cloud-native security.

Key Challenges with Cloud-Native Application Security 

Cloud-native architectures bring in challenges related to application and infrastructure security. Let us look at a few of the most prominent challenges organizations face related to cloud-native security.

Traditional Security vs. Cloud-Native Security: A Paradigm Shift

Traditional security tooling is built for static environments and is ineffective in the dynamic and rapidly changing cloud-native landscape. Furthermore, with the advent of microservices, containers, service meshes, and multi-cloud environments, it has become increasingly difficult for organizations to track software vulnerabilities. As a result, there is an increased dependency on automation and continuous monitoring throughout the application lifecycle.

Lack of Security Mindset

When development teams build products, their primary focus areas are functionality and usability. Faster release cycles make it difficult to inspect and resolve security vulnerabilities correctly. In addition, development teams do not always have the required skillset to identify security issues and, at the same time, do not want to be slowed down by unknown security concerns. As a result, security often takes a back seat. However, it would be best to consider security an integral part of the DevOps pipeline amidst the need to deliver high-quality software in a cloud-native landscape.

Software Dependency Problem

By adding reusable external dependencies in the codebase, developers can leverage complex functionalities without developing and maintaining them. However, open-source libraries are susceptible to being compromised, causing security issues in your application. Therefore, you must do your due diligence to ensure that software dependencies are inspected for malware and vulnerabilities.

Selecting the Right Observability Tools

Security in the cloud brings a new set of challenges that your organization might not be trained to handle. Hence it is imperative that you evaluate and finalize the right tools to secure your applications in a cloud-native world. 

With containers spinning up and down within seconds, you need tools to provide real-time visibility into your containerized environments. The attack surface in the cloud is rapidly increasing, and there are numerous cases of data breaches, compliance issues, and compromised APIs. From a security standpoint, having complete observability of your workloads by leveraging the right tools for logging, metrics, traces, and alerting is critical.

Enforcing Consistent Security Policies and Guardrails

Today, enterprises leverage third-party security tooling and managed services provided by their public cloud provider to build their cloud security posture. However, it is challenging to develop centralized policies and guardrails that apply across your cloud-native environments. This requires your development teams to work closely with the security team. As a best practice, you should have guardrails in place, which can disallow actions that lead to policy violations.

Injecting Security Into DevOps

Having DevOps processes in place results in improved efficiency, reduced failures, faster deployment cycles, enhanced application performance, and better customer experience. Taking a step further, DevSecOps can be defined as a practice to deliver secure software through a continuous delivery model. Therefore, security should be considered an integral part of your CI/CD pipeline, as seen in Figure 1, and you need to ensure that it is built into the application lifecycle phases in an iterative and automated manner.

Figure 1: Building security into the DevOps pipeline

Shift Left Security Strategy

The velocity and frequency of feature deployment had increased tremendously in a cloud-native world. A single security team in an organization cannot be entirely responsible for the security of all your applications in the cloud. Bridging the gap between development, operations, and security teams are critical to deploying secure applications. It would be best to build security controls into all your pipeline stages to shift security left. Fixing security issues in production is expensive, and hence incorporating security practices during the development phase is highly recommended. Shifting left requires collaboration and engagement between teams during the early stages of your development cycle.

Automated Testing

Hardening security requirements during the initial design and development phases is essential. It is best to encourage development teams to keep security in mind while writing unit, integration, and end-to-end tests. As a best practice, do not just focus on happy-path workflows but have effective coverage on negative workflows, boundary conditions, and edge cases. Always test the error handling scenarios’ authentication workflows and maintain extensive coverage for high-risk and frequently used code. Since testing is built into the CI/CD process, you cannot release code to production without passing tests.

Static Code Analysis

Static code analysis tools have many security-related rules that cover well-established security standards such as OWASP Top 10 and CWE. You can also add custom rules to identify security issues. Security injection rules like cross-site scripting, SQL injection, denial of service, and code injection indicate issues at the application level that needs to be addressed by developers who follow coding standards. 

Security hotspots are sensitive pieces of code to be reviewed during the code review process. However, when a security vulnerability is detected, it might have a broader impact on your application and need to be fixed immediately. As part of the CI/CD pipeline, every code change will get scanned by these security rules and flagged if there are outliers. You can fail your quality gates, as seen in Figure 2, when the security standards are not met.

Figure 2: SonarQube Quality Gate

Code Review

Peer code reviews are a common practice among development teams. You can implement mandatory code reviews to promote secure code writing by catching common mistakes and vulnerabilities before it gets committed to source control. When a pull request gets created for a particular functionality, ensure a security focus while reviewing the changes. Look out for secure practices like sanitizing outputs, proper secret management, no hardcoding of sensitive data, authentication workflows, session management, logging, and exception handling.

Cloud-Native Security Patterns and Anti-Patterns

The cloud-native architecture enables organizations to build and run scalable applications in a dynamic environment. However, it does come with several challenges — security, cost, governance, observability, and more. Let us look at some of the best practices every development team working in the cloud-native space needs to embrace to secure their applications.

Zero-Trust Architecture

Zero trust is a strategic approach to rebuild and modernize security by enforcing strict access controls to protect data, applications, and networks. By inspecting and monitoring network traffic to catch any malicious activity, the zero-trust architecture helps reduce the blast radius in case of a compromise. In a cloud-native architecture that uses a combination of microservices and containers, service mesh helps reduce the surface area of attack and implement the zero-trust security model.

Identity and Access Management 

IAM is a core component of the security management posture within an organization that enables the proper entities to access the right resources. IAM protects against compromised access, safeguards resources within the network, and provides comprehensive security against phishing and ransomware attacks. 

Principle of Least Privilege

The least-privilege policy grants permissions to only the resources required to perform the task; no other access gets assigned. Having overprivileged users and roles in an organization increases the risk factor. With an increasing number of security breaches caused by privileged credentials, it is best to always validate policies and adopt the least-privilege principle by default.

Secrets Management 

Cloud secrets management refers to tools and methods to securely manage secrets — passwords, certificates, SSH keys, encryption keys, and API tokens. You should have a strategy to rotate your passwords periodically. Public cloud providers provide managed services to handle secrets and their management.

Incident Response 

Building incident response and triaging strategies are challenging when you have microservices running in a Kubernetes cluster in a cloud-native landscape. When you treat your workloads running in containers as cattle and not pets, performing post-mortem analysis and gathering audit trail events become difficult. 

Containers spin up and down frequently, and so, responding to security threats in a transient environment requires a different strategy. Incident response is critical to resolving security issues efficiently and spreading awareness within your organization about operational duties. 

Data Protection

Cloud-native microservices support polyglot persistence, and therefore, development teams have flexibility in choosing the appropriate database technology, as seen in Figure 3, for developing their services. These datastores can store both structured and unstructured data to support a variety of functions like search, reporting, time-series, caching, transactional, etc. 

Figure 3: Polyglot persistence in cloud-native applications

Container Image Security

Many organizations are running containerized workloads in production. Containers make it easy to package, deploy and run your code, thereby increasing the speed and portability of your application. It is necessary to secure the container image to secure containers.

Threat Modeling

Developing services in the public cloud can trigger new security threats like malware and ransomware. You can leverage managed services provided by cloud providers or third-party vendors that use machine learning and artificial intelligence to identify security threats and vulnerabilities across your organization.  

Using Infrastructure as Code

Cloud-native architectures leverage the principle of immutability to manage infrastructure resources. If you need to make any configuration changes, you don’t modify the server; instead, build a new server with the updated configuration. IaC ensures consistency between environments and enables better DevOps practices by deploying infrastructure code in an automated and repeatable manner. 

Runtime Observability

With enterprises growing their workloads rapidly and adapting multi-cluster/multi-cloud environments, it becomes crucial to have a centralized view of your systems. Furthermore, to have a sound observability strategy, you need to continuously profile your applications and collect a considerable volume of data round the clock. 

Shared Responsibility Model for Security

Security is a shared responsibility between the cloud service provider and its customers in the public cloud. The shared model helps to reduce the operational burden on customers, as the cloud provider protects the entire infrastructure containing the service deployments. 

At the same time, customers are responsible for securing the application code, data, identity and access, containers, and workloads running in the cloud that contain business logic. Once you have clarity on these shared responsibilities, development teams can focus on building business features and not worry about the day-to-day operational issues in the infrastructure layer. 

To summarize, the cloud provider is responsible for the security “of” the cloud, whereas the customer is responsible for the security “in” the cloud. 

Figure 4 below illustrates Microsoft’s shared responsibility model in the cloud and the various responsibilities between Microsoft and its customers.

Figure 4: Microsoft’s shared responsibility model

Image Source: “Shared responsibility in the cloud”

OWASP Framework

OWASP Top 10 is a set of development techniques that helps developers improve their web applications’ security and enables teams to shift security earlier into the design and coding phases. It encourages guidelines like integrating security into the CI/CD pipeline, parameterizing queries, validating all inputs, implementing error handling, improving logging strategy, leveraging the benefits of security frameworks, protecting data at rest and encryption, reducing sensitive data exposure, implementing secure access controls, and more.

Figure 5: OWASP Top 10 web application security risks

Conclusion

Cloud-native architectures have seen rapid adoption in the recent past. However, there are numerous security challenges because of the complex and dynamic landscape. Users have faced multiple security risks like data breaches, data loss, denial of service, insecure APIs, account hijacking, vulnerabilities, and identity and access management. Enterprises need to continuously adapt security best practices to handle these issues, as explained in this article. 

The core security concepts cannot be isolated and must be consistently integrated into the development lifecycle. Enterprises have been able to find ways to balance security and the speed of delivery by embracing automation, continuous delivery, and most importantly, building a DevOps culture.

It is highly recommended that article readers also study the CNCF Cloud Native Security Whitepaper that focuses on key challenges cloud-native application security and provides guidance to architects and developers.

Editor’s Note: This is one of my articles that was originally published in the DZone’s Refcard for CloudNative Application Security

This article will describe Infrastructure as Code, its benefits, and the popular IaC tools. You will also learn how to model infrastructure as part of the CI/CD pipeline and incorporate it into the standard development lifecycle.

What Is Infrastructure as Code? 

IaC helps to manage and provision infrastructure resources through code and automation. In a traditional on-premises environment, operators log into the server and execute a series of commands via the command line or console to perform changes. However, these manual configurations are prone to drifts and create inconsistent environments. It’s also a time-consuming process to deploy similar changes across your infrastructure. There is no quick way to verify the correctness of these manual changes. If there are issues, it isn’t easy to recreate them. In comes IaC to simplify the management and provisioning of your infrastructure resources. With IaC, you don’t make manual changes to servers, instances, containers, or environments. Both the creation and modification of the resources are automated. IaC is the practice of using code to create, describe, and manage infrastructure resources.

Infrastructure as Code Benefits

Let’s now take a close look at how IaC can benefit your organization.

Increased Productivity Through Automation

Making infrastructure changes and deploying them is a repetitive process, and it is time-consuming if your developers/operations team needs to do these manually in a recurring interval. With IaC, you can focus on coding and be more productive by automating the infrastructure deployments. As a result, IaC enables faster time to market for your business features.

Repeatable Deployments with high Predictability

Having your infrastructure defined in source control lets you automate the deployment process and enable developers to follow the software development lifecycle for infrastructure changes. It also empowers developers to perform successful deployments backed by efficient practices like peer review, static code analysis, and automated testing.

Improved Consistency With Minimal Configuration Drift

Performing infrastructure changes manually can lead to inconsistency between the servers and is generally the primary reason for configuration drifts. Applying changes manually is error-prone, and so IaC saves the day by standardizing the infrastructure modification process. The deployment process is faster, repeatable, and more consistent with automation and version control.

Documented Process for Deploying Changes

It would be best not to completely rely upon your sysadmin or operations team to deploy infrastructure changes. This increased dependency on selected operators to make server changes can create blocker scenarios when they are not available. In the long run, it is not a scalable solution. Having the infrastructure code in source control provides visibility to everyone in your organization about the current state. Developers can deploy infrastructure changes if all the stages in the CI/CD pipeline pass. If there are any issues, it’s easy to troubleshoot by looking at the change history in the source control.

Infrastructure as Code Tools

Some open-source IaC tools can be categorized into various groups:

• Configuration management tools
  • Chef, Puppet, and Ansible are popular tools that allow you to install, update and manage resources on existing infrastructure.
• Server templating tools
  • Tools like Docker and Vagrant allow you to create an image from a configuration file, which is then used to provision infrastructure in a repeatable manner. They promote the idea of an immutable infrastructure, which we will explain later in this article.
• Container orchestration tools
  • Tools like Kubernetes and Docker Swarm allow you to orchestrate container workloads in a dynamic cloud-native landscape.
• Provisioning tools
  • Tools like Terraform, Azure Resource Manager, Google Cloud Deployment Manager, and AWS CloudFormation allow you to provision servers and other resources in the respective cloud environment.

Mutable vs. Immutable Infrastructure

With immutable infrastructure, you create a new server with the revised configuration if you want to modify an existing server. In a cloud-native environment, where containers are spinning up and down every minute, immutability is a required feature. You should package your application and its dependencies into a container image, and as you want to modify the configuration, you deploy a new container version. The pets vs. cattle analogy for infrastructure management is popular in the dynamic container space. You treat your infrastructure resources like cattle so that you can delete and build them from scratch when a configuration change is required. Pets are indispensable resources where you make configuration changes in place. Immutability helps simplify your operations, minimize drift, boost consistency between environments, and build a secure infrastructure.

Declarative vs. Procedural Approach

With a procedural approach, you write separate scripts to achieve the desired state. Over time, you tend to have many scripts that have been applied to your environment, and you can review the modifications via change history. The order of execution of the scripts matters, or else you end up with a different end state. Tools like Chef and Puppet follow the procedural style of infrastructure management. 

Please find below some of the frequently used Puppet commands:

With the declarative approach, you maintain the desired state in a configuration template. If you want to make any modifications, you update the same template to reflect the desired state and let the IaC tool generate the difference script and apply it to the environment. Tools like Terraform follow the declarative approach, where the code in source control always reflects the current state of the infrastructure. The declarative approach helps you to create reusable code since you are just focused on describing the desired state and offloading the complexity of syncing the current and desired state to the IaC tool.

Using Terraform to Manage Infrastructure

Terraform is a cloud-agnostic, open-source tool for infrastructure provisioning. Created by HashiCorp in 2014, Terraform uses HashiCorp Configuration Language (HCL) to describe infrastructure code. It supports many providers and can help manage resources in individual cloud providers, such as AWS, Azure, and Google Cloud. Terraform is backed by a large and growing community.

The primary function of Terraform is to help you create, modify, and destroy infrastructure resources. To provision resources using Terraform, you need to use the following commands:

Infrastructure Automation with GitOps

GitOps workflows apply DevOps best practices for application development (version control, code review, automated deployments, etc.) to infrastructure deployments. GitOps is based on the declarative model, where configuration files get stored in Git, and approved changes get automatically deployed to the environment. As shown in Figure 4, the GitOps operator ensures that the desired state stored in Git is in sync with the actual state of the deployed infrastructure. Enterprises are rapidly adopting GitOps to manage their Kubernetes cloud-native environments at scale. Applying GitOps practices enables continuous deployments with proper auditing and high reliability.

Conclusion

IT infrastructures are growing exponentially, and embedding automation into every stage of the delivery pipeline ensures faster and more consistent deployments. Automating your IaC processes is as important as the automation of your application deployment. You can leverage multiple IaC tools together to automate your infrastructure management. Each of the tools mentioned in this article has its strengths and weaknesses, so having a sound understanding of those and selecting the right tool that fits your use case is critical. IaC is the future, and organizations are readily embracing it to increase the reliability of their infrastructure.

Editor’s Note: This is one of my articles that was originally published in the DZone’s Trend Report for DevOps

Kubernetes is an open-source container orchestration system for automating the deployment and management of containerized applications. Kubernetes provides capabilities like service discovery, horizontal autoscaling, and load balancing, while ensuring that application configurations are declarative and that systems are self-healing.

In this article, I will explain how to monitor your Kubernetes cluster and implement automated health checks, and discuss the various monitoring tools available.

Monitoring Challenges with Kubernetes

Monitoring is one of the top challenges for organizations that embrace Kubernetes as their container orchestration platform. 

A Kubernetes ecosystem has significantly more moving parts than a traditional monolithic on-premise environment, which can make monitoring and troubleshooting more complex. In such a dynamic environment, where containers and pods are constantly being created and deleted, it’s a challenge to keep track of components and to have complete observability. 

In order to have confidence in your deployed applications, you’ll need your Kubernetes monitoring solution to provide full visibility into the health of your deployed applications, containers, and underlying infrastructure.

The Kubernetes Client

Kubectl is a command-line interface that you can use to interact with your Kubernetes clusters. You can download and install the kubectl client binary to run basic and advanced commands and get a quick insight into the Kubernetes cluster. 

In this section, I’ll show you how to use the kubectl tool to monitor your Kubernetes cluster.

You can learn more about managing logs in Kubernetes and different types of logging architectures here.

Kubernetes Health Checks

You can implement health checks in Kubernetes and apply automatic remedial measures to increase the application’s resiliency. The liveness and readiness probe are used to control the health of the application running inside a pod’s container. 

I would highly recommend that you add the liveness and readiness probes to your deployment so that Kubernetes can take action in an automated fashion if the application is not healthy. You can add both these probes to the “containers” section of the pod specification.

Liveness Probe

If the liveness probe fails, this indicates that the application is not responding or is in a failed state from which it cannot recover by itself. In such cases, Kubernetes will automatically restart the offending pod. 

You can add a basic HTTP probe to the pod specification, which the kubelet can use to restart the containers if the health check probe fails. 

In the configuration file below, I have configured the kubelet to perform a liveness probe every two seconds. I have also added an initial delay of three seconds in performing the first probe. If the request returns an error code, the kubelet will terminate the container and automatically restart it.

Readiness Probe

If the readiness probe fails, this indicates that the application is not ready to accept client requests. In such cases, Kubernetes stops sending traffic to the offending pod until the readiness probe passes.

You can add a basic HTTP probe that the kubelet can use to identify when a container is ready to accept traffic. The load balancers will evict the pod if it is not ready and won’t route any traffic. The readiness probe configuration is nearly identical to the configuration of the liveness probe, but you will need to use the “readinessProbe” field instead of the “livenessProbe” field.

Resource Utilizations of Pods

To ensure the smooth functioning of the pods in your Kubernetes cluster, it is essential to configure their resource requests and limits. If you create your pods without paying attention to the CPU and memory resources, the results might not be optimal. Please keep in mind that the pod’s resource requests and limits are the sum of all its containers’ requests and limits. The Kubernetes scheduler uses these computational resources to schedule the pods on a node.

Resource Requests

When creating your pod, you can specify the minimum amount of CPU and memory that an application container needs. Kubernetes uses these metrics to schedule pods across the available nodes in the cluster.

Resource Limits

When creating your pod, you can specify the hard limit of CPU and memory that an application container may consume. This helps Kubernetes ensure that pods act as “good neighbors” and do not affect other pods by consuming too many resources.

Here is the configuration file for a pod showing resource requests and resource limits for a container:

Figure 1: Grafana panel showing the container’s memory utilization (approx 0.4 GB), memory request ( 2 GB), and memory limit (set as 4 GB)

Kubernetes Monitoring with Prometheus

Prometheus is an open-source metrics-based monitoring system that helps you analyze how your application and infrastructure are functioning .Prometheus uses service discovery to scrape targets.The application metrics can be scraped via an exporter and stored in a time-series database. You can query this data using PromQL or view it in Grafana dashboards.

Prometheus is a pull-based monitoring system, and hence, your services don’t need to be aware of where the monitoring system is hosted. They just need to expose the metrics via an HTTP endpoint, and Prometheus can pull metrics from it. Prometheus also has a flexible, multidimensional data model that allows you to collect data and query it in a highly reliable way. This data can then be leveraged for monitoring and troubleshooting purposes. 

Prometheus has built-in integration with Alertmanager, which allows you to send notifications to teams when thresholds are not met. Alertmanager performs custom handling of alerts by grouping, deduplicating, and silencing alerts, and routing them to integrations like email, Opsgenie, and PageDuty.

You can leverage the Prometheus expression browser to run PromQL queries to view the data stored inside Prometheus.

Figure 2: Prometheus expression browser

With the increased adoption of containers and microservices architectures, the number of components to manage and monitor has also increased. Prometheus is the best tool available for collecting data from such dynamic environments, storing them in a time-series data store, and providing monitoring capabilities for your applications deployed in Kubernetes and for the underlying infrastructure.

Kubernetes Data Visualization with Grafana

Grafana is an open-source data visualization tool that provides you with in-depth visibility into the health of your Kubernetes cluster and the applications running in it. 

You can leverage Grafana dashboards to monitor both system-level and application-level metrics such as:

• Overview of nodes, pods, and containers.
• Actual CPU/memory usage of Kubernetes cluster nodes.
• Health status of Kubernetes master node components (API server, scheduler, controller, etcd).
• Node disk I/O usage.
• Node network in/out traffic.
• Pod health and availability.
• Resource allocation for pods.
• Requested usage vs. actual usage of resources. 
• Container restarts and percentage of throttled containers.

Grafana supports a number of data sources out of the box, like Prometheus, Elasticsearch, InfluxDB, AWS CloudWatch, and Azure Monitor. While building your custom dashboards in Grafana, you can combine data from multiple sources and consolidate them into a single dashboard. You can also associate a panel with a specific data source to provide visualizations.

 Figure 3: Data sources officially supported by Grafana

You can create a Grafana dashboard displaying summary metrics for monitoring the health of the Kubernetes cluster and identifying any workload bottlenecks. Below are screenshots of a few dashboards you can use to track your cluster resources. For custom monitoring, you can use one of the community-built Grafana dashboards. 

Figure 4: Grafana dashboard showing Kubernetes cluster health

Figure 5: Grafana dashboard showing overall health of pods in Kubernetes cluster

Figure 6: Grafana dashboard showing overall status of nodes, pods, and containers

Kubernetes Monitoring Simplified

Monitoring is critical for gaining an in-depth understanding of how your Kubernetes cluster is functioning. In this article, I explained different ways to monitor your Kubernetes cluster and pods. Using the Kubernetes command-line interface, you can dive deep into cluster resources. With liveness and readiness probes, you can implement automated health checks, and with Prometheus and Grafana, you get one of the most effective monitoring solutions for Kubernetes.

Editor’s Note: This is one of my articles that was originally published in the IOD Blogs

Organizations are rapidly embracing cloud-native design patterns to modernize their business operations and enable faster time to market. Cloud-native architecture combines technologies like microservices, containers, automated CI/CD pipelines, container orchestration, unified observability, and cloud infrastructure. However, modern-day cloud computing services face security risks like data breaches, application vulnerabilities, account hijacking, insecure APIs, malicious insiders, data loss, denial of service, and insufficient credential management. 

To protect against these threats, organizations should adopt a zero-trust model for their data and services, as well as embrace the DevSecOps movement to integrate security practices throughout their software development lifecycle (SDLC). Enterprises are using container technologies like Docker to simplify the packaging and deployment workflow of their cloud-native applications. Kubernetes is a popular container orchestration system for automating containerized application deployment, scaling, and management. 

Adopting DevOps principles enables teams to deploy and release features rapidly, but it also introduces challenges to the security of your cloud-native applications. Hence, introducing security practices into DevOps is required. 

This article will: 

• Explain the four Cs of cloud-native security 
• Overview cloud-native security challenges 
• Showcase security considerations for cloud-native applications 
• Explain how to integrate security into your CI/CD pipeline 

Four Cs of Cloud-Native Security 

The four Cs of cloud-native security are cloud, clusters, containers, and code. Each layer, as seen in Figure 1 on the next page, benefits from the outer security layers, with the code layer built on top of the cloud, cluster, and container layers. Let’s briefly go through the recommendations at each layer to secure your architecture: 

• Cloud – It is the foundation of all security layers, and every cloud provider (like AWS, Microsoft Azure, Google Cloud, IBM Cloud) provides infrastructure security for the running workloads. 
• Clusters – The primary security concerns in this layer are RBAC authorization, secrets management, pod security policies, and network policies, and Kubernetes is the standard operating tool. 
• Containers – The security postures recommended in this layer are container vulnerability scanning, image signing, and prohibiting privileged users. 
• Code – Organizations have the most control over this layer and can implement security recommendations like performing static code analyses, adopting DevSecOps practices, and making security part of the CI/CD pipeline. 

Cloud-Native Security Challenges 

With organizations embracing a remote work model due to the COVID-19 pandemic, the need to secure data and systems has become more critical than ever before. The “Zero Trust” security model can improve your organization’s security posture in general, especially if they plan to adopt a mobile workforce or hybrid workspace environment. The basic principle is to avoid blindly trusting everything within the security parameter (corporate network) and to always authenticate and authorize every user, application, and device that tries to access the network — whether internal or external to the organization. 

The key technologies and principles to build zero-trust-based architectures are: 

• Multi-factor authentication 
• Identity and access management 
• Visibility into device inventory 
• Firewall management 
• Data classification and encryption 
• Least privilege access 
• Continuous network traffic monitoring 

Cloud-Native Application Security 

Containers make it easy to package and deploy the runtime dependencies of your application and help resolve configuration management issues between your development and production environments. However, containers are transient and generally have a short life span, making container security challenging compared to traditional security mechanisms for threat detection and vulnerability scanning. 

CONTAINER SECURITY 

Out of the box, containers provide some level of isolation and security, but at the same time, they also introduce a set of security concerns like kernel exploits, denial of service attacks, poisoned images, container breakouts, and compromised secrets. Reducing the container attack surface is critical since issues in one container can potentially impact other instances running on the same host. Applying the principle of least privilege and having restrictive user access to containers are good practices. A secret management solution is required to store credentials securely and allow the containers to access sensitive data during operation. 

In addition, container images are immutable, so any vulnerability in the container image will persist for the life of the image. Therefore, you need to ensure that the images are regularly scanned for vulnerabilities as part of your CI/CD pipeline, updated with the latest patches, and pulled from trusted registries. When running your cloud-native applications in a containerized environment, you need complete visibility into your cluster configuration and to have monitoring in place. 

SHARED RESPONSIBILITY MODEL FOR SECURITY 

In the public cloud, security is a shared responsibility between the cloud service provider and its customers. The differentiation of responsibility can be viewed as security “of” the cloud vs. security “in” the cloud. The cloud provider protects the overall infrastructure where the services are running and takes care of the operational concerns in the physical and network layer. On the other hand, customers are responsible for their business logic, including the application code and data layer protection. 

Figure 2 illustrates Microsoft’s shared responsibility model, including areas of responsibility between the customer and Microsoft for workloads running on-premise and in the cloud. 

Automating Cloud-Native Security

The DevOps methodology focuses on increasing collaboration and transparency between development and operations processes. However, in the pursuit of quick time to market, security practices must not be neglected and pushed further downstream in the pipeline. This is where DevSecOps comes into the picture, incorporating operations and security measures earlier in the development cycle.

SHIFT-LEFT SECURITY STRATEGY 

Shifting security left in the development process is essential because you don’t want security to be an afterthought. Instead, you should design and build systems with security top of mind. It is expensive and time-consuming to identify and fix security vulnerabilities in production, so the goal of shifting security left is to implement security practices and perform security testing during the development process — and not just before deployment to production. Your DevOps pipeline should allow you to deploy software with quality and speed while also adhering to security best practices. 

Most vulnerabilities are introduced at the application level, and an attacker can compromise your system if your application contains vulnerabilities. Static application security testing (SAST) and SAST tools enable you to scan your entire codebase
and execute security-related rules to detect vulnerabilities like SQL injection, cross-site scripting, denial of service, and code injection issues. In addition, as part of the continuous integration workflow, you can perform static code analysis for every code commit and implement automated vulnerability detection and compliance reporting. 

INJECTING SECURITY INTO CI/CD PIPELINES 

Integrating security controls into your automated pipelines, as seen in Figure 3, is key to successfully delivering high-quality software. The DevOps pipeline has the required permissions to deploy changes into your environment, so you must have stringent security guardrails around it. Developers can leverage several open-source and commercial security tools for their CI/ CD pipelines. The goal is to identify security issues early, and low friction measures like following secure coding practices, the peer review process, and performing static code analysis are easier to implement. If you are using open-source software or third-party libraries, you should have tooling in place for vulnerability management and threat detection. Implementing a lightweight penetration test in your pipeline helps enhance the security posture — for example, a dynamic application security testing (DAST) tool like OWASP ZAP acts as a security gate before deploying code to production. Along with scanning your containers, consider scanning your infrastructure code before deploying to the cloud. 

Conclusion 

With the cultural shift of building security into DevOps, development teams have an additional responsibility to automate application security testing and integrate it into the release pipeline. Training development teams on security principles and best practices can help bridge the knowledge gap. In addition, development teams working closely in collaboration with IT security teams can help both mitigate security vulnerabilities earlier in the SDLC and achieve the shift-left concept in the context of application development and security. 

The Open Web Application Security Project (OWASP) is a non-profit global community that promotes application security mechanisms. It is highly recommended that developers working in the cloud-native space acclimate themselves with the OWASP Top 10 application security vulnerabilities, how hackers exploit them, and methods to eliminate these risks. Most of these vulnerabilities — like code injection, sensitive data exposure, broken authentication, security misconfigurations, cross-site scripting, and insufficient logging/monitoring — can be addressed with secure coding practices and following a stringent code review checklist. 

Editor’s Note: This is one of my articles that was originally published in the DZone’s Trend Report for Application Security.

If your answer to any of the above questions is “yes,” then this article is for you.

I’ll explain some of my experiences with troubleshooting and resolving Elasticsearch’s performance issues. By the end of this article, you should have a good understanding of the critical metrics to monitor when you bump into performance or operational problems with your Elasticsearch cluster. I’ll be citing examples and monitoring metrics from the Amazon Elasticsearch service, but the same metrics hold true if you’re hosting Elasticsearch on your own or using some other managed providers.

Top 5 Elasticsearch metrics to monitor –

• Cluster Configuration
• Master Node Metrics
• Data Node Metrics
• Indexing Performance
• Search Performance

Read my full article in the IOD Tech Content blog space below —
https://iamondemand.com/blog/top-5-elasticsearch-metrics-to-monitor/

However, there is no magical formula for successfully implementing DevOps. In this article, I will explain how DevOps can help you gain a competitive edge. I’ll also dive deep into the top challenges of DevOps adoption.

Driving Factors for DevOps Adoption

• Speed to market
• Less waste
• Improved collaboration between teams
• Happier teams and customers
• Cost optimization

Top 5 Challenges in DevOps Adoption

1. Resistance to Change
2. Adoption of New Tools
3. Technical Complexity
4. Migration from Monolithic Architectures to Microservices
5. Creation of CI/CD Pipelines for Legacy Components

Read the full article in the IOD Tech Content blog space below —

5 Reasons Your Company Is Struggling to Implement a DevOps Culture

In this article, I would like to summarize the key CNCF announcements made today and provide you with links having blog posts/press releases with additional information.

Links for more details on CNCF announcements from yesterday and today are listed below:

• Envoy Mobile Joins the CNCF

• Cloud Native Survey 2020: Containers in production jump 300% from our first survey

• Kubernetes Security Specialist Certification Now Available

• CNCF and FinOps Foundation Collaborate to Advance Cloud Financial Management

• Cloud Native Computing Foundation Welcomes Cox Communications as Gold Member

• Cloud Native Computing Foundation Welcomes Kasten by Veeam as Platinum Member

• Cloud Native Computing Foundation Adds 46 New Members

• Cloud Native Network Functions Conformance Launched by CNCF

• CNCF End User Technology Radar: Database Storage, November 2020

• Announcing Significant Training Benefits to CNCF’s End User Community

• TOC Approves Cloud Native Buildpacks from Sandbox to Incubation

• Announcing the Cloud Native Security White Paper

Please also check out the news that has come out from the CNCF/LF community, including:

• HAProxy Technologies: HAProxy Kubernetes Ingress Controller Surpasses 2.5 Million Downloads 

• Commvault: Commvault Unveils Latest Enhancements for Containers, Launches Metallic BaaS for Kubernetes 

• Red Hat: Red Hat Drives Hybrid Cloud Ubiquity with OpenShift Innovation Across Architectures, Applications and Infrastructure 

• Red Hat: Red Hat Extends Open Hybrid Cloud to the Edge with Red Hat Enterprise Linux and Red Hat OpenShift

• StackRox: StackRox Secures Cybrary’s Fast-growing Cybersecurity Education Platform Running on Google Kubernetes Engine 

• Trilio: Trilio Launches TrilioVault for Kubernetes v2.0 with New Management Console to Manage Data Protection and Migration Across Clouds 

• Diamanti: Diamanti Announces Support for AWS at KubeCon North America

• CodeFresh: Launching the Future of DevOps: GitOps 2.0

• MayaData: MayaData launches Kubera Propel and Kubera Chaos 

• Arrikto: MiniKF, the all in one Kubeflow, is now available on the AWS Marketplace

• Mirantis: Mirantis Lens Adds Extension API, Offering Seamless Integration with any Kubernetes Integrated Component, Toolkit, or Service 

• Elastisys: Elastisys Announces General Availability of Compliant Kubernetes 

• NeuVector: NeuVector Announces Promo for New Customers: Switch to NeuVector and Improve ROI with Full Lifecycle Container Security

• Sysdig: Sysdig and IBM Embed Sysdig Secure for Container Security 

• Sysdig: Sysdig Introduces Low-Touch Zero Trust Network Security for Kubernetes 

• Zettaset: New Research Confirms Enterprise Organizations Have Embraced Containers and Kubernetes but are Struggling to Properly Secure Data Stored in Cloud-Native Environments

• Oracle: Two Priorities for Oracle Cloud Infrastructure: Interoperability and Cloud Native

• CD Foundation: Welcome to the Tekton Hub! Public Invitation from CD Foundation

• Dynatrace: Dynatrace announces PurePath 4 with OpenTelemetry and advanced cloud support

• Reblaze: New Envoy Extension Provides WAF, Bot Protection, and More

• BroBridge: Brobridge launches micro-service data caching and provisioning platform “GRAVITY”, making the micro-service journey of enterprises suddenly clear

• Opsani: Optimizing Code Delivery, Performance, and Budget Too Complex for Humans

• Instana: Instana Allows Kubernetes-Based Deployment and Management of Self-Hosted On-Prem Version of Application Performance Management Solution

• Cloud 66: Skycap v2 makes Kubernetes usable

• Cloud 66: Maestro: a simple and flexible deployment platform for Web Development Agencies Cloud 66: Introducing Multiple Database Support for Kubernetes

• Accurics: Open Source Software Terrascan Extends Policy as Code Support to Helm, Kustomize 

• Styra: Styra Expands Declarative Authorization Service with Free and Mid-tier Offerings to Manage Open Policy Agent at Scale

• Veritas: Veritas Empowers Customers with Enterprise-Class Availability for Containers Managed by Kubernetes

• Garden: Garden Exits Stealth with €3.1M in Funding to Transform the Future of Cloud Native Development 

• Cloudical: VanillaCloud announced – the open source cloud based on Kubernetes

• Lenses.io: Simplified data streaming to AWS S3 with new free connector for Apache Kafka 

• Lenses.io: First data catalog for Apache Kafka built by Lenses.io

• Zerto: Zerto Continues Momentum with Data Protection and Disaster Recovery Capabilities for Containerized Applications 

• Kinvolk: Kinvolk expands Flatcar Container Linux offerings to meet enterprise Kubernetes requirements

• Hammerspace: Knock-out the Persistent Data Dilemma with Storageless Data from Hammerspace

• Puppet: Puppet Invests in Platform Approach, Extends Self-Service Automation to the Global 5000

• KIOXIA America, Inc.: KIOXIA’s KumoScale Software Brings Faster NVMe-oF Storage to Containers 

• Intuit: Running TurboTax on Kubernetes: Lessons Learned on Our Journey 

• Kubecost: Kubernetes spend: Getting the complete picture

• Humio: Humio accelerates cloud management and extends security with Humio Operator 

• Section: Section’s Adaptive Edge Engine Performs up to 7x Faster than Cloud

• Cloudify: Cloudify Rolls Out  v5.1 ft Extensive Kubernetes & Cluster Support

• Kublr: Kublr 1.19 Includes New User Interface, Full Support for External Clusters, AWS Configurability Improvements and Kubernetes Chaos Engineering

• KubeMQ: KubeMQ Control Center is Upgraded with Automatic Network Creation Functionalities 

• Kubermatic: Kubermatic Releases KubeOne 1.1: Optimized Resource Usage With New Autoscaler 

• RX-M: RX-M Announces First CKS Boot Camp Training Course at KubeCon + CloudNativeCon North America 2020

• Lablup Inc.: Backend.AI v20.09 RC will be released in NOV. 17, 2020

• Gremlin: Gremlin Soundproofs Kubernetes by Helping DevOps Teams Isolate Noisy Neighbors within a Cluster

• Ambassador Labs: Ambassador Labs Accelerates Developer Workflows on Kubernetes with $18M Investment from Insight Partners

• Kubernative GmbH: SINA – the new rising star of Kubernative

• Replicated: Replicated Announces Troubleshoot As a Stand-Alone Open Source Project

• CircleCI: CircleCI extends CI cloud with self-hosted solution, CircleCI runner

• Magalix Corporation: Magalix launches KubeAdvisor 2.0 with dozens of policies for SecOps & DevOps 

• HAProxy Technologies: HAProxy Kubernetes Ingress Controller Surpasses 2.5 Million Downloads 

• Equinix: Equinix Doubles Down on Investment in CNCF Community Infrastructure Lab to Fuel Open Source Innovation

I am looking forward to Day 2 of the event. See you there!

You can learn how to translate your existing SQL Server expertise to Azure SQL, including Azure SQL Database and Azure SQL Managed Instance through these all-new content on YouTube, GitHub, and Microsoft Learn. After completing the video series, learning path, or workshop, you will have a foundational knowledge of what to use when, as well as how to configure, secure, monitor, and troubleshoot Azure SQL. To support you as you learn, you can join Bob & Anna in four-day series of live sessions on August 17-20th.

This is an excellent initiative by the Microsoft Azure Data Team to help the community learn about Azure SQL and bring the SQL Server community to the cloud. I am listing down the resources available —

Landing Page for Azure SQL – https://aka.ms/learnazuresql

Azure SQL Fundamentals – a 6-course learning path on Microsoft Learn

aka.ms/azuresqlfundamentals

After completing this learning path, you will have a foundational knowledge of what to use when and how to configure, secure, monitor, and troubleshoot Azure SQL.

Azure SQL for Beginners – a 61-video series on YouTube that walks through all the content on the learning path 

aka.ms/azuresql4beginners

Microsoft insiders Bob Ward and Anna Hoffman walk you through configuring, monitoring, and troubleshooting database security, performance, and availability in this Azure SQL video series on more than 60 topics.

Azure SQL Workshop (aka.ms/asqlworkshop, then select Workshop: Azure SQL) – a 1-2 day workshop including hands-on labs and slides which are free/open source for customers and partners to leverage.

Azure SQL Bootcamp signup (aka.ms/azuresqlbootcamp) – 4-day (2 hours each day) live boot camp run by Bob and Anna on YouTube, with moderators from the Azure SQL PG, from August 17-20

Enjoy the free learning resources

In this article, I will explain how to upgrade your AWS Elasticsearch manually via AWS Console and in an automated fashion using CloudFormation.

Amazon ES provides in-place upgrades, and this is the most simple and straight forward process. Amazon ES starts the upgrade, which can take from 15 minutes to several hours to complete. During this time, all the cluster operations can be performed as normal if the upgrade has been performed on a healthy cluster. Also, this has been tested and observed that Kibana might be unavailable for a few minutes during the upgrade process. It is recommended to take a manual snapshot of the ES domain before initiating the upgrade and test the pre-upgrade checks. Please refer to the below links for detailed information on AWS in-place upgrades & note that the upgrade process is irreversible and cannot be passed nor canceled once initiated –

Upgrading Elasticsearch

Zero Downtime In-Place version upgrades for Elasticsearch

We can perform this upgrade manually via the AWS Console and also in an automated fashion using the CloudFormation template.

Manual Upgrade via AWS Console

When I navigate to AWS Console, select my domain, and click on ‘Upgrade Domain’ – I see options to upgrade in place to max 6.8 version.

In-place upgrade from 6.2 to 7.1 is a two-step process —

1. First, you will have to update the domain from 6.2 to 6.8.
2. Once that the domain is upgraded to 6.8 version, you can update it to desired and final version 7.1

It is recommended to take a manual snapshot before each upgrade.

You can upgrade the Elasticsearch version for your domain without creating a separate domain and migrating your data. The upgrade process first checks if the domain is eligible to upgrade. If this check succeeds, Amazon Elasticsearch Service takes a snapshot of the domain and initiates the upgrade. You can also perform an upgrade eligibility check without performing the actual upgrade.

After initiating an upgrade eligibility check, you can’t cancel it. You can continue to read and write data while the upgrade is in progress, but you can’t change your domain configuration.

It takes few minutes to do the upgrade eligibility check, and the results are stored in the ‘Upgrade history‘ tab in the ES Cluster overview screen —

If you decide to go ahead and upgrade your cluster following the 2 step approach, you will see the upgrade summary in the ‘Upgrade history’ tab.

Please note that the upgrade process can sometimes take hours to complete. In my case, it took close to 10 hours. Behind the scene, a blue-green deployment is initiated to handle this upgrade.

Blue/Green deployment of the cluster will be triggered whenever there is configuration change to the domain. During the upgrade, a new set of nodes is created, and data is migrated from old nodes to the new nodes. This practice minimizes downtime and maintains the original environment in the event that deployment to the new environment is unsuccessful. This deployment should not cause any issues with the cluster activities, but sometimes you might notice the performance degrade due to additional data migration and shard balancing activities.

In-place upgrades are recommended strategy, but there is also an alternate option of creating a new 7.1 ES domain and restore the data to the new domain using a snapshot. This process has low risk during the upgrade process as you are not making any changes to the existing/functioning domain, and if you see any issues in the newly upgraded cluster, you can always refer/go back to the old 6.2 cluster.

Automated Upgrade via CloudFormation

For performing any resource updates, I prefer to use AWS CloudFormation, so that the process is automated via coding and it ensures the same resource state across all environments.

The ES Upgrade from 6.2 to 6.8 works fine manually via AWS Console, but it did not via CloudFormation. I got an error message stating, ‘CloudFormation cannot update a stack when a custom-named resource requires replacing. Rename and update the stack again.’

Hence as the next step, I modified the domain name and deployed the changes. However, when you remove or change the DomainName value in the existing CF, it results in data loss. I noticed that the existing domain ‘titlesearch’ was removed, and a new domain ‘titlesearch-ke92iyvwu5re’ was created with no documents.

This is an unexpected behavior since we do not want either a new domain to be created or a data loss.

To fix the above problem, there is an easy solution. Go ahead and set the ‘EnableVersionUpgrade‘ UpdatePolicy to True. When EnableVersionUpgrade is set to false, or is not specified, updating Elasticsearch Version results in replacement. Once I made this change, my existing  Elasticsearch domain was upgraded without any issues. There was no need to modify the domain name, there was no data loss in this process and no interruption or downtime. I would still recommend performing this upgrade during a time window when there are minimal requests to your Elasticsearch instance.

To sum it up, you can perform zero downtime version upgrades for your Elasticsearch clusters. In this article, I explained how you could do this upgrade seamlessly via AWS Console and CloudFormation. In case you bump into any issues with your update, please comment below, and I will be happy to assist.

If you are interested to learn more about Elasticsearch performance, you can read my article about the Five critical Elasticsearch metrics to monitor

]]> https://samirbehara.com/2020/07/23/aws-elasticsearch-version-upgrade-from-6-2-to-7-1/feed/ 1 7092 steven-wright-mq8QogEBy00-unsplash samirbehara Domain+Upgrade+ES Upgrade Domain Upgrade Confirmation PreCheck Upgrade Success Successful ES Upgrade CloudFormation ES Error Enable Version Upgrade Elasticsearch https://samirbehara.com/2020/07/20/elasticsearch-result-window-is-too-large/ https://samirbehara.com/2020/07/20/elasticsearch-result-window-is-too-large/#respond Mon, 20 Jul 2020 13:33:40 +0000 http://atomic-temporary-98113587.wpcomstaging.com/?p=7063 Problem Statement

I have configured Error Logs for my Elasticsearch cluster, and I see a frequent error below in the logs —

org.elasticsearch.ElasticsearchException$1: Result window is too large, from + size must be less than or equal to: [10000] but was [15020]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level setting.

Explanation

My understanding is that when we are doing a search that is too wide, Elasticsearch is limiting it to 10,000 results by default for better memory usage & fast performance.

For example – The search query might return 15,000 results, but Elasticsearch displays only 10,000, and just because it truncated the result set, it logs this error to notify.

It is important to understand that the Index setting, which controls the behavior, is index.max_result_window. This index setting defines the maximum value of from + size for searches to this index. It defaults to 10000. Search requests take heap memory and time proportional to from + size, and this limits that memory. A search request in Elasticsearch generally spans across multiple shards. Each shard generates its sorted results, which need to be sorted centrally to ensure that the overall order is correct.

Search in ES contains two phases — the Query phase, and the Fetch phase.

Figure 1: An Elasticsearch three-node cluster with the index divided into six shards

During the QUERY phase, the search query is sent to all the shards in the Elasticsearch index. Each shard executes the search operation locally and builds a priority queue of matching documents. It then returns the result set containing the DocumentID and the score to the coordinating node.

During the FETCH phase, the coordinating node first decides which documents should be retrieved. For example — If the elasticsearch query specifies { “from”: 50, “size”: 10 }, then the first 50 results would be discarded, and only the next 10 results would need to be retrieved.

To summarize, the query phase identifies the documents that satisfy the search request, and the fetch phase retrieves the documents itself and passes it to the coordinating nodes.

Example —

This means the first 1000 results would be discarded, and only the next 100 results will be retrieved. Note that from + size can not be more than the index.max_result_window index setting, which defaults to 10,000.

How do you modify the ‘index.max_result_window’ to a higher value?

I ran the below update query in Kibana to increase the max result window to 20000 and got an ‘” acknowledged”: true’ message —

If you are interested to learn more about Elasticsearch performance, you can read my article about the Five critical Elasticsearch metrics to monitor here.

https://iamondemand.com/blog/top-5-elasticsearch-metrics-to-monitor/

I will be talking about the new capabilities of SQL Server 2019 that have rocked the industry including Intelligent Query Processing, Data Virtualization and enhanced Polybase capabilities, Apache Spark and HDFS in the box, Big Data Clusters, Security, High Availability and Performance Features, T-SQL with Java, and enhancements for SQL Server on Linux, Docker Containers, and Kubernetes. SQL Server 2019 is the new unified data platform allowing you to keep your data where it is located, build your own data lakes, and enable new container-based solutions never before possible.

The key learning objectives from my session will be an overview of  —

• Introduction to SQL Server 2019
• Intelligent Query Processing
• Data Classification and Auditing
• Data Virtualization and Polybase
• Big Data Clusters
• SQL Server on Linux and Containers
• SQL Server on Kubernetes
• Accelerated Database Recovery
• Modern Database Platform
• Migration to SQL Server 2019

Please RSVP to the meeting here if you are interested to attend —

https://www.meetup.com/SteelCitySQL/events/265624181/

SQL Server 2019 Resources

The Microsoft SQL Server Engineering Team has done an excellent job in creating learning content for getting up to speed on SQL Server 2019 – there are detailed blogs, videos, workshops, presentations slides all available to the community free of cost.

Below is a great list of resources which I collected from Bob Ward’s presentation —

Download and try it: http://aka.ms/ss19

Use free training: https://aka.ms/sqlworkshops

Use examples through our https://aka.ms/SQL2019Notebooks

Learn from videos: https://aka.ms/sql2019learning

Read what’s new for SQL 2019: documentation

Watch how to Modernize SQL Server: check out the video

Get the book: https://aka.ms/sql2019book

What: Four User Groups in Birmingham, AL meet at the same time and in the same general location every month.  That makes it a great melting pot to connect and network with people who have similar and tangential interests to your own.  It is a fantastic way to expand or begin your knowledge.

When: The third Tuesday of every month making the next event February 18.  The event starts at 6:00 PM but doesn’t get serious until around 6:15 and will run no later than 7.30PM

Where: 3rd Floor of Thomas Russell D Hall , 800 Lakeshore Drive, Samford University

Food: Is provided

Price: $0

RSVP: You can RSVP using one or more of the Meetup links listed below —

FEBRUARY 2020 EVENTS:

Steel City SQL

https://www.meetup.com/SteelCitySQL/events/265624181/

Birmingham .NET 

https://www.meetup.com/Birmingham-NET-Meetup/events/266917258/

Power BI

https://www.meetup.com/BhamQuants/events/vcfkdrybcdbxb/

Agile Birmingham

https://www.meetup.com/Agile-Birmingham/events/268275417/

Hope to see you there! 

What: Four User Groups in Birmingham, AL meet at the same time and in the same general location every month.  That makes it a great melting pot to connect and network with people who have similar and tangential interests to your own.  It is a fantastic way to expand or begin your knowledge.

When: The third Tuesday of every month making the next event Jan 21.  The event starts at 6:00 PM but doesn’t get serious until around 6:15 and will run no later than 7.30PM

Where: 3rd Floor of Thomas Russell D Hall , 800 Lakeshore Drive, Samford University

Food: Is provided

Price: $0

RSVP: You can RSVP using one or more of the Meetup links listed below —

JANUARY 2020 EVENTS:

Steel City SQL – Big data clusters for the absolute beginner:

Mohammad Darab will lead a session on the cutting-edge technology that SQL Server 2019 has to offer: Big Data Clusters.

https://www.meetup.com/SteelCitySQL/events/266402346/

Birmingham .NET – Open Q&A:

Robb Schiefer will host an Q&A session where any question is fair game and anyone can help answer the questions. We can discuss everything from soft skills, to career advice, to CS theory, to architectural discussions.

https://www.meetup.com/Birmingham-NET-Meetup/events/266917210/

Power BI – Compound visualizations:

Nick Sanford will continue the lecture series designed to prepare you for the 70-778 certification exam.  There are parts for beginners, experts, and everyone in between.  https://www.meetup.com/BhamQuants/events/vcfkdrybccbcc/

Agile Birmingham – What’s new is SAFe 5.0:

Brad Baker will discuss what is new in SAFe 5.0 and how SAFe is moving from Program to Business Agility. All aspects of an SAFe Agile transformation will be discussed from teams to scaling to business agility.

https://www.meetup.com/Agile-Birmingham/events/267620693/

Earlier in the day, I was working on one of my microservices hosted in the cloud. The service uses AWS Aurora as the database and I generally connect locally to the DB instance via MySQL Workbench. I logged into the database in MySQL Workbench with the ‘admin’ credentials, however was not able to run any DML statements. I received the below error —

Error Code: 1290. The MySQL server is running with the –read-only option so it cannot execute this statement

Resolution

After an hour of troubleshooting and researching, I finally figured out what the issue was. I thought of writing a quick blog post on this in case you bump into the same issue – you know what to look out for.

With AWS Aurora DB Cluster you have 2 types of endpoints which you can connect with – Writer and Reader. When you are setting up your MySQL Workbench make sure to add the ‘Writer’ endpoint and not the ‘Reader’ endpoint.

If you have the cluster endpoint, then you can connect to the primary instance for the database cluster. You can perform both read and write operations using the cluster endpoint. The cluster endpoint always points to the primary instance.

I realized that I had used the Reader endpoint while establishing the DB connection via MySQL Workbench. This was the reason behind the ‘read-only option’ error. Once I changed the endpoint to the Writer one, I was able to successfully execute DML scripts.

Conclusion

While connecting to the database, ensure that you are using either the Cluster endpoint or the Writer endpoint.

Check out my other recent blog posts around troubleshooting issues —

Lost connection to MySQL server during query

Troubleshooting Issues with AWS Database Migration Service

Flyway Error: Found non-empty schema(s) without schema history table

Tomcat Error – java.net.BindException: Address already in use

Resolving Gradle issues in IntelliJ IDEA

Atlanta Code Camp 2019 is your opportunity to join other Atlanta developer community members for a day of training unparalleled in the southeast. Come hear from some of the best national and local speakers, including Microsoft MVPs and open-source advocates!

Registration: https://atlantacodecamp2019.eventbrite.com
Speakers: https://atlantacodecamp.com/2019/Speakers
Sessions: https://atlantacodecamp.com/2019/Sessions
Schedule: https://atlantacodecamp.com/2019/Schedule

Event Venue –
Kennesaw State University – MARIETTA Campus
Engineering Technology Center
Building 840 (formerly known as Bldg. Q)
1100 S Marietta Pkwy SE
Marietta, GA 30060

Coming back to my presentation, I will be presenting on —

https://atlantacodecamp.com/2019/Speakers/Samir-Behara

Designing Data-Intensive Cloud-Native Applications

Cloud is rapidly changing the way modern-day applications are being designed. Data is at the center of multiple challenges while architecting solutions in the cloud. With technology changing rapidly, there are new possibilities for processing data efficiently. Come join me in this session to learn more about the tools and technologies that will help you to design data-intensive systems. We will take a structured approach towards architecting data-centric solutions, covering technologies like message queues, data partitioning, search index, data cache, event sourcing, and NoSQL solutions. If you are a Data professional, Software Engineer or Architect, come join me in this session to understand the high-level design principles that will help you build scalable, resilient and maintainable systems in the cloud.

I hope to see you there. Cheers!

In this blog post, I will show you how to troubleshoot and resolve connectivity issues with Aurora MySQL RDS Instance.

Problem Statement 

We were unable to establish connection with one of our Aurora MySQL RDS data store. I use MySQL Workbench locally to connect to the AWS databases and I was able to connect to every other database except this one. Please find below the error details —

Lost connection to MySQL server during query

Please:
1 Check that MySQL is running on address *****
2 Check that MySQL is reachable on port 3306 (note: 3306 is the default, but this can be changed)
3 Check the user admin has rights to connect to ***** from your address (MySQL rights define what clients can connect to the server and from which machines)
4 Make sure you are both providing a password if needed and using the correct password for ***** connecting from the host address you’re connecting from.

Reading through the above seems like there is a connection issue between the MySQL client and the Database server and the connection timed out before returning data.

Is the issue specific to MySQL Workbench?

• Open the MySQL Workbench Preferences.
• Check if the SSH Timeout and DBMS Timeout value is set to only a few seconds.
• Try to increase the default value of the connection timeouts.
• Save the settings, close the MySQL Workbench and reopen the connection to see if you are able to connect to the database.

Did you try rebooting the Database instance?

Rebooting a DB instance restarts the database engine service. It will result in a momentary outage though.

You can also reboot the database instance using AWS CLI —

aws rds reboot-db-instance –db-instance-identifier databasename –profile live –region us-east-1

Make sure to check the Database status and confirm it is ‘Available’

 

Did you check the Cloudwatch Metrics for the Database Cluster?

You can check the database Cloudwatch metrics – especially DatabaseConnections, EngineUptime and UpdateThroughput as a troubleshooting step.

In this case seems like after an engine restart on 8/11, there has been some connectivity issues.

Did you check the Database Error Logs?

You can view, download, and watch database logs using the Amazon RDS console.

This is where I was able to figure out the root cause of the connectivity failure. Evidently there was an error in the logs stating – ‘init_connect command failed’

Init_connect — This parameter defines a string for the server to run for each client that connects. There is no default value for this parameter. The string consists of one or more SQL statements, separated by semicolons.

I had an incorrect value mentioned  the ‘init_connect’ parameter which caused the issue. Once I corrected the value and applied the changes, I was able to connect to the database.

 

Conclusion 

Summarizing this blog post, there are a number of things you can do to troubleshoot connectivity issues with your AWS RDS database while trying to connect locally via MySQL Workbench —

1. Confirm the issue is not specific to MySQL Workbench
2. Reboot the Database instance
3. Check the Cloudwatch Metrics for the Database Cluster
4. Check the Database Error Logs

Problem Statement 

As part of our Cloud Migration strategy, we had to migrate data from On-Premise SQL Server tables to AWS Aurora MySQL tables. One of the tables we wanted to migrate data had about 700 million rows. But the AWS DMS task kept restarting by itself after 4-5 hours from being 60-70% complete. The other small tables we wanted to migrate data completed successfully without issues. So this issue was specific to scenarios where we had more than 500-600 million rows and DMS kept restarting by itself during the data migration.

The first step to troubleshoot this error was by enabling error logs in AWS DMS. Since the DMS task ran for a couple of hours, it was clear that there were no issues with the DMS task start event. Maybe it pointed to the fact that it was not able to move some specific data from the source table to destination.

2019-05-25T19:24:58 [TASK_MANAGER ]I: Task running full load and CDC in resume mode after recoverable error, retry #3 (replicationtask.c:1239)

 For troubleshooting any issues with AWS DMS, it is necessary to have logs enabled. The DMS logs would typically give a better picture and helps find errors or warnings that would indicate the root cause of the failure. If the logs are not available there is nothing much you can do from a detailed troubleshooting analysis perspective. So basically next step is to turn on DMS logs and kick the job again and validate if the errors are captured in the logs.

If logs are not enabled, you need to set up a new task with logging enabled so if and when it errors out, you can take a look and troubleshoot the same.

Please find below the list of all logger levels in a migration phase:

1. LOGGER_SEVERITY_DEFAULT – Default logging
2. LOGGER_SEVERITY_ERROR – Log only errors for the appropriate phase
3. LOGGER_SEVERITY_WARNING – Log only warnings
4. LOGGER_SEVERITY_INFO – Log only info lines
5. LOGGER_SEVERITY_DEBUG – Log in debug mode (more verbose than default)
6. LOGGER_SEVERITY_DETAILED_DEBUG – Most verbose logging

The default log level severity is ‘LOGGER_SEVERITY_DEFAULT‘

AWS DMS restarts table loading from the beginning when it has not finished the initial load of a table. When a task is restarted, AWS DMS does not reload tables that completed the initial load but will reload tables from the beginning when the initial load did not complete.

Here’s the troubleshooting document for DMS for reference: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Troubleshooting.html 

We reviewed the DMS logs but were not able to find anything concrete about the root cause. We wanted some additional information to troubleshoot this error and hence reached out to AWS Support to investigate why the DMS task errored out.

Resolution

 The AWS DMS team got back to us with some suggestions before another migration run. They recommended to set up a new task for migrating only the table which fails right now after making below two changes:

   1. Add the below Extra Connection Attributes to the source endpoint :

   cdcTimeout=1200 — by default the value is set to 600 seconds

   ignoreTxnCtxValidityCheck=false

Significance of ignoreTxnCtxValidityCheck=false –> This internal property tells to replicate and not give an error on such events which can avoid the task failure. 

   2. Enable detailed logging for the new task which would capture more detailed logs for this specific investigation. Enable detailed debugging for — SOURCE_CAPTURE, TARGET_APPLY, SOURCE_UNLOAD, TARGET_LOAD

Detailed logging is something you can try if the errors aren’t sometimes visible on the first level of logging. So that’s not something you should always have, only in scenarios where you might need more detailed logs to understand what’s happening in the migration when issues come up.

So whenever you set up new tasks, these configuration changes are not needed initially. You can stick with the original configuration. If you run into errors for which you need more data, you can enable it to debug the issue.

Detailed logging consumes more storage on the replication instance, so it is not suggested to be enabled always. It is typically recommended if the default logging already doesn’t have enough information, and we need more detailed information for a deep dive.

Conclusion

After adding the above configuration settings the DMS task for the large table completed successfully and we were able to migrate 620 million rows of data from OnPremise to AWS.

Check out some of my other recent blogs –

• Cloud-Native Application Security
• Infrastructure as Code for Cloud-Native Applications
• Monitoring Kubernetes in Production
• Securing Cloud-Native Applications
• Top 5 Elasticsearch Metrics to Monitor

If you have not attended a SQL Saturday event before or are not familiar with the concept – It is a free training event for Microsoft Data Platform professionals and those wanting to learn about SQL Server, Business Intelligence, Application Development, PowerShell  and Analytics. It is a great opportunity to listen to some of the best speakers around, interact with the brightest minds and network with the awesome SQL Family.

The event went very smooth and I would like to thank all the attendees, speakers, volunteers and Samford facilities for making this edition of SQL Saturday Birmingham a runaway success.

Big round of thanks and cheers to our event sponsors —

• Platinum Sponsor – Rubrik
• Gold Sponsor – Sparkhound and ITAC
• Bronze Sponsor – Accelebrate and COZYROC
• Partner Sponsor – Atlanta MDF
• Swag Sponsor – Redgate Software and O’Reilly
• Personal Sponsor – Dallas DBAs
• Global SQL Saturday Partner – Microsoft Corporation

We had 3 tracks and a total of 18 sessions this time. Based on the community feedback we selected presentations in the below categories during the SQL Saturday event —

• Application & Database Development
• BI Platform Architecture, Development & Administration
• Cloud Application Development & Deployment
• Enterprise Database Administration & Deployment
• Professional Development

The best part of the event was that all the speakers had packed rooms and there was lot of good discussions during the sessions. Thanks to all the folks who filled up the Event evaluations and it was encouraging to receive positive feedback about the event and the fact that you were able to learn something new.

The Event Presentation slides can be downloaded from —
https://www.sqlsaturday.com/899/Sessions/Schedule.aspx

O’Reilly has provided a free 30-day trial to the O’Reilly Online Learning Platform for all the attendees of SQL Saturday Birmingham.

Please use the code below –
https://learning.oreilly.com/get-learning/?code=SQLS19

Lot of folks have asked me for the Event pics. So here it goes —

SQL Saturday Birmingham 2017 — https://dotnetvibes.wordpress.com/2017/03/20/sql-saturday-birmingham-2017/

This year I presented on another new topic – ‘Designing Data-Intensive Cloud Native Applications‘. I talked about how the cloud is rapidly changing the way modern-day applications are being designed. Data is at the center of multiple challenges while architecting solutions in the cloud. With technology changing rapidly, there are new possibilities for processing data efficiently. I talked about the tools and technologies that will help you to design highly scalable data-intensive systems. We took a structured approach towards architecting data-centric solutions, covering technologies like message queues, data partitioning, search index, data cache, event sourcing, NoSQL solutions and Serverless databases.
Thanks to everyone who came to my session and provided feedback. I have uploaded my presentation slide deck here –

Designing Data Intensive Cloud Native Applications – SQL Saturday Pensacola 2019

https://conferences.oreilly.com/software-architecture/sa-ca/public/schedule/detail/75345

Description —
Microservices and containers have taken the software industry by storm. Transitioning from a monolith to microservices enables you to deploy your application more frequently, independently, and reliably. However, microservice architecture has its own challenges, and it has to deal with the same problems encountered while designing distributed systems.

Enter service mesh technology to the rescue. A service mesh reduces the complexity associated with microservices and provides functionality like load balancing, service discovery, traffic management, circuit breaking, telemetry, fault injection, and more. Istio is one of the best implementations of a service mesh at this point, while Kubernetes provides a platform for running microservices and automating deployment of containerized applications.

==============================================================

You can view my presentation slides here –

I have also written a number of blog posts detailing the content which I will cover during this presentation. You can refer it for further reading/research —

Monolith to Microservices using Strangler Pattern

Breaking the Monolithic Database in your Microservices Architecture

Making your Microservices Resilient and Fault Tolerant

Eight Fallacies of Distributed Computing

Smart Pipes and Smart Endpoints with Service Mesh

The Rise of Service Mesh Architecture

Sidecar Design Pattern in your Microservices Ecosystem

Simplifying Microservice Architecture with Envoy and Istio

Istio Service Mesh Control Plane

Istio Service Mesh Data Plane

CNCF Cloud Native Landscape and Trail Map

Microservices Journey from Netflix OSS to Istio Service Mesh

Resilient Microservices with Istio Circuit Breaker

Istio Circuit Breaker with Outlier Detection

Retry Design Pattern with Istio

Handling Service Timeouts using Istio

Chaos Testing your Microservices with Istio

Visualizing the Istio Service Mesh using Kiali

Cloud Native Monitoring with Prometheus

Please let me know in case you have any questions. Happy to discuss.