Category: DevOps

Manipulating JSON with jq

Recently I was working on a project using AWS CLI and happened to come across some cool jq techniques of manipulating JSON. I will describe what my use-case was, but similar techniques can be applied whenever JSON is involved.

jq is a lightweight JSON processor written in C. You can find more information about the tool and how to install it in their official documentation. It’s quite powerful and is capable of doing quite a lot – i.e parsing, manipulating and processing json files. Now let’s see how it was kinda easy to get my work done using jq.

The Use Case

We use AWS Image Builder to build EC2 AMI’s and this requirement was to explicitly update the ami_name field of an existing Image Builder Distribution Configuration whenever a pipeline is triggered. There are some options in the distribution configuration to specify what we want to name the output AMI, but we wanted it to be very custom (with the base ami version) and dynamic. I will not be talking about how Image Builder works as its a separate topic in itself, however you can read about it here.

The Problem

So to achieve this (using AWS CLI), we just needed to run 2 AWS CLI commands:

  1. Get the existing distribution configuration:
aws imagebuilder get-distribution-configuration --output json --distribution-configuration-arn $distribution_config_arn

The above will return a json response of the existing distribution configuration like below: 

{
    "requestId": "42b6bcf5-9505-4c42-ad38-7efd8177f2ac",
    "distributionConfiguration": {
        "arn": "arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/amazon-eks-node-latest-pipeline-distribution-config",
        "name": "amazon-eks-node-latest-pipeline-distribution-config",
        "description": "amazon-eks-node-latest image builder pipeline",
        "distributions": [
            {
                "region": "us-east-1",
                "amiDistributionConfiguration": {
                    "name": "amazon-eks-node-latest-golden-ami-{{ imagebuilder:buildDate }}",
                    "description": "amazon-eks-node-latest image builder pipeline",
                    "amiTags": {
                        "family": "amazon-eks-node-latest",
                        "Name": "amazon-eks-node-latest-golden-ami-{{ imagebuilder:buildDate }}"
                    },
                    "launchPermission": {}
                }
            }
        ],
        "dateCreated": "2024-03-06T14:59:37.286Z",
        "tags": {
            "owner": "platform",
            "project": "image-builder",
            "env": "prod",
            "family": "amazon-eks-node-latest",
            "managedby": "terraform"
        }
    }
}

2. Update only the amiDistributionConfiguration.name field of this distribution configuration.

Now it is not possible to just update one field of the distribution configuration. If you see distributions is a list in the response and to update it we need to get the full response, change the fields we want and then pass this json as an input to the update-distribution-configuration CLI command.

But wait, if you see the input json syntax that update-distribution-configuration expects (shown below), it does not exactly match the response got from the get-distribution-configuration command.

{
    "distributionConfigurationArn": "arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/amazon-eks-node-latest-pipeline-distribution-config",
    "description": "amazon-eks-node-latest image builder pipeline",
    "distributions": [
        {
            "region": "us-east-1",
            "amiDistributionConfiguration": {
                "name": "Name {{imagebuilder:buildDate}}",
                "description": "An example image name with parameter references"
            }
        },
        {
            "region": "eu-east-2",
            "amiDistributionConfiguration": {
                "name": "My {{imagebuilder:buildVersion}} image {{imagebuilder:buildDate}}"
            }
        }
    ]
}

This problem can be true for a variety of other scenarios (AWS or non AWS). Comes jq to the rescue! With jq, it’s just a one line command to process the json and transform it the way we want.

The Solution

The solution is, to transform the response from get-distribution-configuration to the required syntax using jq and then passing it to the update-distribution-configuration command, which can be effectively done with the below commands:

# 1. Get the existing distribution configuration
distribution_config=`aws imagebuilder get-distribution-configuration --output json --distribution-configuration-arn $arn`

# 2. Update the JSON syntax of the response to match the target commands requirements
updated_distribution_config=$(echo "$distribution_config" | jq '{ distributionConfigurationArn: .distributionConfiguration.arn, description: .distributionConfiguration.description, distributions: [.distributionConfiguration.distributions[] | .amiDistributionConfiguration.name = UPDATED_NAME]}')

aws imagebuilder update-distribution-configuration --cli-input-json $updated_distribution_config

The transformation happens in the 2nd command. I have split the command into new-lines to make it clearer. Let’s see how it works:

echo "$distribution_config" | 
jq '{ 
  distributionConfigurationArn: .distributionConfiguration.arn, 
  description: .distributionConfiguration.description, 
  distributions: [.distributionConfiguration.distributions[] |.amiDistributionConfiguration.name = UPDATED_NAME]
}'

The above filter tells jq to create a JSON object containing:

  • A distributionConfigurationArn attribute containing the value of .distributionConfiguration.arn
  • A description attribute containing the value of .distributionConfiguration.description
  • A distributions attribute containing a list of distributions from .distributionConfiguration.distributions[]
  • Also we update all .amiDistributionConfiguration.name fields from this list of distributions to our desired value (UPDATED_NAME here) for all occurrences.

As you see, the new json structure is created and existing json is parsed to get the required fields, manipulated to replace the fields items as per requirement in this new structure simultaneously. This ends up creating a json like below, which is what the update-distribution-configuration command expects:

{
  "distributionConfigurationArn": "arn:aws:imagebuilder:us-east-1:123456789012:distribution-configuration/amazon-eks-node-latest-pipeline-distribution-config",
  "description": "amazon-eks-node-latest image builder pipeline",
  "distributions": [
    {
      "region": "us-east-1",
      "amiDistributionConfiguration": {
        "name": "UPDATED_NAME" }}",
        "description": "amazon-eks-node-latest image builder pipeline",
        "amiTags": {
          "family": "amazon-eks-node-latest",
          "Name": "amazon-eks-node-latest-golden-ami-{{ imagebuilder:buildDate }}"
        },
        "launchPermission": {}
      }
    }
  ]
}

Though this is just a simple example, jq as you see is quite powerful. 🙂

DevOps Interview with Glovo

Recently I was interviewed by Glovo for the position of Senior DevOps Engineer which is one of Barcelona’s fastest growing Gen 2 startup. It was a great experience and I am dotting down the process, my learnings and some tips which should be helpful to people attending interviews with similar companies/roles.

There were 6 rounds and a final feedback session in total:

  1. Introductory HR + Basic technical – 45 mins
  2. Technical round – 1.5 hrs
  3. Codility – Online take home test (a 2 hrs timed test, to be completed within a week)
  4. System architecture & scalability – 1hr
  5. Pair Programming/Scripting – 1.5 hrs
  6. Behavioural round with manager – 1 hr
  7. Final result and feedback with HR – 30 mins

Each round was well planned and structured. Post every round, the next step in the process was explained and feedback was shared over email. Expectation and interview structure/interviewer details were shared with some basic tips before the interview.

Now lets discuss each round in detail.

  1. Introductory HR + Basic technical – 45 mins
    • This was a friendly HR round where I was asked basic stuff about myself, my interests, experience etc.
    • Also knowledge regarding the company was checked and the job profile was discussed in detail.
    • Some technical questions (basic screening questions) were thrown in towards the end.
    • Time was given to answer my questions as well.
    • Overall they checked my interest, communication skills, basic technical knowledge and if I fit the role being offered.
Tips
- Have your CV tailored according to the JD.
- Ensure to read about the company and know basic stats/it's core business etc.
  1. Technical round – 1.5 hrs
    • A document with the expectation and topics to come prepared was given well before the interview.
    • It was a good technical discussion with one of the engineers whom I would be probably working with.
    • Questions were mostly situation based and how I would approach the problem and solve it.
    • Although some direct technical questions were asked as well.
    • Basically my technical expertise and problem solving approach was checked.
Tips
- Answer all approaches you can think of for a given problem.
- Give real world examples to show your expertise on the topic.
- Ensure to brush up basics before the interview.
  1. Codility – Online take home test
    • There were 2 questions and Codility was the platform where the code was to be written and submitted on any language of my choice. (I chose Python3)
    • Some sample questions were shared and it’s very helpful to solve these sample questions to get used to with the Codility environment.
    • It was a 2 hr timed test, so timing was the key.
    • There were 2 questions – 1 simple (which had to be coded in shell script) and 1 medium to hard level difficulty (IMO)
    • Codility also runs performance test on the solution submitted and scores accordingly, so taking care of time and space complexity while writing the code was crucial.
Tips
- Get some competitive coding experience (Join Hackerrank, Leetcode, Codility, Codechef, Codeforce, Geeksforgeeks whichever you find good)
- Ensure to solve some sample questions in Codility to get a hang of it.
- You are also allowed to code in your IDE and copy the code in Codility (if that helps)
  1. System architecture & scalability – 1hr
    • Again the basic expectation and tools to be used (draw.io to draw HLD for eg.) was shared before hand.
    • A common application was asked to be designed end to end with some constraints/conditions pre-defined.
    • Any approach/design/tool/technology was allowed be used, however every selection had to be backed with proper justification. (e.g. monolith vs microservices)
    • It was a highly interactive session and I was allowed to ask as many questions and justify/explain my design decisions.
    • The architecture diagram (HLD) was to be designed in draw.io with screen sharing/video turned on.
    • From DevOps perspective: HA, scalability, DR, CDN, LB, use of DB (CAP theorem) had to be known and explained in detail.
    • Overall it was a very interesting session.
Tips
- Read and have a good grasp on best infra design patterns, HA-DR, CDN, MessageBrokers, Caching, DB's (when to use what), CAP theorem, common bottlenecks etc
- Subscribe to any wesbite/youtube channel that provides system designing courses/videos.(e.g GauravSen's channel)
- Ensure to go through the system design of common apps like - Youtube, Facebook, Swiggy, Whatsapp, Uber. Seldom an out of the box app is asked to be designed. It will mostly be one of the commonly used apps.
  1. Pair Programming/Scripting – 1.5 hrs
    • This was again on Codility, with screen sharing and video turned on.
    • It was pair programming, and the interviewer coded some lines and asked me to code next in some questions/or if stuck.
    • There were 2 interviewers, one active and one passive.
    • Again I felt the questions were given in order of their difficulty, starting with a program to be coded with bash script and the remaining 2 on any language of my choice.
    • It was again very interactive, and all constraints/edge cases were promptly answered when asked.
    • Overall my coding skills, problem solving approach, attention to details, good coding practices was tested.
Tips
- Ensure to ask as many questions as possible to narrow down all edge cases and constraints.
- Use comments, proper naming convention and test cases. (I missed writing test cases and was told in the feedback)
- Write the algo you intend to follow before starting to code.
- Think loud and convey your thought process.
- If you are unable to code the solution, at-least try to provide a high level algo.
  1. Behavioural round with manager – 1 hr
    • This round was taken by one of the engineering managers and focussed totally on my values/communication skills and thought process.
    • Questions were basically on how I would react on a given situation, what I think are good engineering practices, how to promote collaboration, how good I can be in a diverse work environment etc.
    • Also how I handle failure and criticism, my view on feedback coupled with some questions on high level engineering practices/value delivery/scrum processes/CI-CD techniques etc.
    • Being a senior role, questions related to coaching junior developers, conflict management etc. were asked as well.
    • Sufficient time was allotted to answer all questions asked my me in the end.
Tips
- Give examples from your experience wherever possible.
- Prepare well for answering questions like - Your strength, weakness; Your most difficult project; How you handle failure etc.
- Speak slowly and confidently.
- Ask questions you may want, to show your interest and also convey your working style.

The final HR discussion was to give detailed feedback of all the rounds and declare the final outcome. Also a detailed explanation of the company benefits was given.


Fortunately the outcome was positive in my case 🙂

Design a site like this with WordPress.com
Get started