Engineer taking Dev?Ops notes

About

Fri, 03 Jul 2020 00:00:00 +0000

My name is Deepak and I currently live in Bengaluru working as a systems engineer. In my own time and at work, I do some of the below:

Write code [C#, PowerShell, Python].
Work on Cloud [Azure].
Work on Application Build/Release/Deploy.

I am trying to learn how to code

Fri, 03 Jul 2020 00:00:00 +0000

Kubernetes API Terms

Fri, 15 Oct 2021 12:23:30 +0530

Overview

My notes on the Kubernetes API terminology.

API-Server

The K8s core component api-server is at the heart of the interactions made by Client and other K8s components. It is the only component that can interact with the the etcd (key-value) store.

Reponsibilities:

Serve the RESTful HTTP API
- Reading state - fetch single object, list of objects and stream changes (via events)
- Modify state - Create, Update & Delete objects
Proxy cluster components e.g. Dashboard, kubectl exec sessions etc.

API server HTTP interface handles HTTP requests to query and manipulate K8s resources using the standard HTTP verbs:

HTTP GET - retrieve data with a specific resource or collection or list of resources
HTTP POST - create a resource
HTTP PUT - Updating an existing resource
HTTP PATCH - Partial updates to resources
HTTP DELETE - Destroy a resource (non-recoverable manner)

Above means that when we use a client like Kubectl to interact with the API-Server it is actually issuing HTTP requests against a certain endpoint.

API terminology

Below are some of the most common terms thrown around when referring to the REST HTTP API.

Kind

To simply put this is the data type of the underlying object.

Broadly there are 3 categories of types:

Objects - this category represents individual object types like pod, deployment etc.
List - this data type is basically a collection of objects defined above.
Special kind of types - this collections refers to types which define action on objects or non-persistent entities.

Note - In K8s the kind directly corresponds with a Golang type.

API Group

When logically related kind(s) are grouped together it is referred to as API group. For example - Kind Job or ScheduledJob are placed under the Batch API group.

Version

Each API group can exist in multiple version e.g. an API group first appears as v1alpha1 and is then promoted to v1beta1 and finally graduates to v1.

K8s supports multiple API versions for making it easy to extend, these are served under different API paths e.g. /api/v1 or /apis/extensions/v1beta1.

Different API verions imply diff level of stability and support:

Alpha level (e.g. v1alpha1) - disabled by default, support for a feature may be dropped at any time, use only in short lived testing clusters.
Beta level (e.g. v2beta3) enabled by default, code well tested. However object semantics might get a breaking change in next release.
Stable (e.g. v1) will appear in software for many subsequent versions.

Resource

This is usually a plural word and uniquely identifies the HTTP endpoint exposing CRUD semantics for an object type e.g. /pods, /pods/busybox. Common path patterns are:

The root e.g. /pods which lists all the instances of that type
Individual named resources e.g /pods/busybox

Resources correspond to HTTP endpoint paths

Kinds are the type of objects returned and recieved by these endpoints, which are then persisted into the datastore.

Points to note

Resources are always part of an API group and version, also referred to as GroupVersionResource (GVR)
GVR basically uniquely defines an HTTP endpoint path
Each kind also is part of an API group and versio, referred to as GroupVersionKind (GVK)
Relation between GVK & GVR is - GVKs are served under the HTTP Path identified by GVR.
Above process of mapping GVK to GVR is also termed as REST Mapping in the K8s SDK.

Azure Vms Resource Graph Queries

Mon, 24 May 2021 10:47:38 +0530

Why use Resource Graph instead Az CLI/ PowerShell?

If we want to search for resources meeting certain criteria across all our subscriptions, we can’t use Az CLI or Az PowerShell to do this type of queries since it would require a lot of overhead to filter and switch between subscription contexts.

Pesudo-Code (not so performant)

Foreach subscription in subscriptions:
    set Az Context
    Get the VMs, Filter based on criteria

Resource Graph queries can help here, as they can be used to fetch metadata about the resource and then after their presence is validated, maybe perform some operation on it. With PowerShell we can even group these resources together based on SubscriptionID and then iterate over each subscription (set the right context) and perform actions.

Pesudo-Code (single query and performant)

Query Azure Graph for resources based on criteria across all the subscriptions

In this post, I share some of the Resource Graph Queries I have found useful while working with Virtual Machines.

Prerequisite

Prerequisite is the Az CLI (with graph extension) or Az.ResourceGraph PowerShell module which supports this.

# Install the Resource Graph module from PowerShell Gallery
Install-Module -Name Az.ResourceGraph

For Az CLI run the below to install the extension:

# Add the Resource Graph extension to the Azure CLI environment
az extension add --name resource-graph

Virtual Machine Queries

Below are list of queries for Virtual machines.

Find Virtual Machine with Name

If you want to find out if a virtual machine is present across all the subscriptions you have access to, you can use the below Resource graph query with Az CLI.

# Using Az.ResourceGraph Module
Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' and name == 'testvm01'"

Now for most of the queries in this post, you can do one to one translation to Az CLI commands by extracting the query and using it, I prefer to use PowerShell and that will be used in the rest of the examples.

# Using AZ CLI
az graph query -q "where type =~ 'Microsoft.Compute/VirtualMachines' and Name == 'testvm01'"

Find Virtual machines with name regex matching

Note - This regex matching can be applied to any resource or any property defined in the Resource schema as well.

Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' and name matches regex 'test-[0-9].*'"

Find Virtual machines with a specific tag

Gist is Kusto allows for filtering at all the levels, so we can filter based on a specific tag or can chain multiple tags to filter.

# Filter based on a tag called the department
# Note that tags are case-sensitive
Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' and tags.department =~ 'ITDepartment'"
 
 
# Filter based on the groupEmail name
Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' | where tags.group =~ 'SRE'"
 
# Combine both the department & group
 Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' | where tags.group =~ 'SRE' or tags.department =~ 'ITDepartment'"

Find Virtual machines deployed using MarketPlace images

This query utilizes the fact that the publisher field for a market place will not be empty.

Note - Remove the end ‘limit’ command expression at the end of query if you need the exhaustive list.

Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' and isnotempty(properties.storageProfile.imageReference.publisher)|  limit 1"

Find Virtual machines deployed using custom Images

This query utilizes the fact that the publisher field does not exist for a VM deployed using a generalized image.

Note - Remove the end ‘limit’ command expression at the end of query if you need the exhaustive list.

Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' and isempty(properties.storageProfile.imageReference.publisher)| limit 1"

Find Virtual machines which do not have a Tag populated

In this example the query checks for VMs which do not have a specific tag named Department created but not populated.

Note - Remove the end ‘limit’ command expression at the end of query if you need the exhaustive list.

Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' and isempty(tags.group)
| project name, location, resourceGroup, subscriptionId, Group=tags.group
| limit 5"

Find Virtual machine with an IPAddress

This can simply be done by doing a reverse lookup of the IPaddress but in some cases where the machines did not have these DNS records created it was a pain.

We start by looking for network interfaces which have this IP address

$nic = Search-AzGraph -Query "where type =~ 'Microsoft.Network/networkInterfaces' |
where properties.ipConfigurations[0].properties.privateIPAddress == '10.10.10.6'"
 
$nic.properties.virtualMachine.id # this is the resource ID of the VM
 
# Now we can fire off another query to get the VM info
Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' and id == `'$($nic.properties.virtualMachine.id)`'"

Gather extra Virtual Machine information

To gather the relevant information for different VM resources e.g. network, storage, subscription etc, one can use PowerShell to extend the object returned from the resource graph query.

Below I show usage of a crude way of using PowerShell to do this e.g. using a filter.

Filter GetVMInfo {
    $networkQuery = "where type =~ 'Microsoft.Network/networkInterfaces' and id == `'$($PSItem.properties.networkProfile.networkInterfaces[0].id)`'"
    $network = Search-AzGraph -Query $networkQuery
  
    [pscustomObject]@{
        Name = $PSItem.name
        Location = $PSItem.location
        Tags = $PSItem.Tags
        ResourceGroup = $PSItem.resourceGroup
        SubscriptionID = $PSItem.subscriptionId
        SubscriptionName = (Get-AzSubscription -SubscriptionId $PSItem.subscriptionId).Name
        ProvisioningState = $PSItem.properties.provisioningState
        VMSize = $PSItem.properties.hardwareProfile.vmSize
        BaseOSImage = $PSItem.properties.storageProfile.imageReference.id.Split('/')[-1]
        OSType = $PSItem.properties.storageProfile.osDisk.osType
        OSDiskSize = $PSItem.properties.storageProfile.osDisk.diskSizeGB
        DataDisks = foreach ($disk in $PSItem.properties.storageProfile.dataDisks) {
                        "Name = {0}, Size = {1}, IsManaged = $(if ($disk.managedDisk) { $true} else {$false})"    -f $disk.name, $disk.diskSizeGB
                    }
        IPAddress = $network.properties.ipConfigurations[0].properties.privateIPAddress
        SubnetName = $network.properties.ipConfigurations[0].properties.subnet.id.Split('/')[-1]
        DNSservers = @($network.properties.dnsSettings.dnsServers)
    }
}

Use the above filter like below:

# Search for the VM first using the Az Resurce Graph
$VM = Search-AzGraph -Query "where type =~ 'Microsoft.Compute/VirtualMachines' and name == 'testvm01'"

# Pipe the object(s) found to the filter we defined
$VM | GetVMInfo

Conclusion

Azure Resource Graph is a powerful CMDB solution for querying Azure resources, they can be used wisely to enhance existing scripts or create new robust and scalable automations.

PowerShell to C# & back - JSON Create, Beautify

Mon, 05 Oct 2020 09:07:45 +0530

Background

I have been following up the C# Tutorial and working my way through examples and converting them to PowerShell and notes.

JSON Create object

Interesting to see that System.Text.Json namespace offers Utf8JsonWriter type to write JSON (UTF-8 encoded) string from common .NET types e.g. String, Int32 and DateTime. What this means is we can write custom JSON strings in-memory (using MemoryStream) and later flush Utf8JsonWriter content to get the string back. I wanted to use this new writer to create a JSON string which will look like below

{
    "ComputerName": "TestVM01"
}

Let’s do this in C# first. Below is a C# script which I am using.

#!/usr/bin/env dotnet-script
using System;
using System.Text;
using System.Text.Json;
using System.Net;
using System.IO;

using (var memStream = new MemoryStream())
{
    using (var jsonWriter = new Utf8JsonWriter(memStream))
    {
        jsonWriter.WriteStartObject(); // this is the start of JSON Object
        jsonWriter.WriteString("ComputerName", "TestVM01");
        jsonWriter.WriteEndObject(); // this is the end of the JSON Object
        //jsonWriter.Flush();
    }

    string json = Encoding.UTF8.GetString(memStream.ToArray());
    Console.WriteLine(json);
}

This effectively translates to below in PowerShell, see inline comments.

#!/usr/bin/env pwsh
using namespace System.IO
using namespace System.Text
using namespace System.Text.Json

# create a memory stream to hold the data in-memory
# the memory stream is a performant way to hold data without locking the source
$memStream = [MemoryStream]::new()

# use the Utf8 Json Writer object
$jsonWriter = [Utf8JsonWriter]::new($memStream)

# indicate the JSON object is starting
$jsonWriter.WriteStartObject();

# now the JSON string is written
$jsonWriter.WriteString("ComputerName", "TestVM01");

# instruct the JSON object is ending
$jsonWriter.WriteEndObject();

# flush the json writer content to the stream
$jsonWriter.Flush();

# dispose of the json writer
$jsonWriter.Dispose();

# Get the string from the byte[] array from the memory stream
[Encoding]::UTF8.GetString($memStream.ToArray())

# dispose the memory stream once read the json string
$memStream.Dispose()

Below is the sample output that gets generated. Note it is a regular JSON string, let’s make it prettier in next section.

Beautify JSON

This is where tapping into the underlying .NET APIs to learn something really shines, there are so many tweaks that can be applied.

One such instance is to make out output JSON string look pretty, we can pass some options to our Utf8JsonWriter instance.

#!/usr/bin/env dotnet-script
using System;
using System.Text;
using System.Text.Json;
using System.Net;
using System.IO;

using (var memStream = new MemoryStream())
{
    using (var jsonWriter = new Utf8JsonWriter(memStream, new JsonWriterOptions {Indented = true}))
    {
        jsonWriter.WriteStartObject(); // this is the start of JSON Object
        jsonWriter.WriteString("ComputerName", "TestVM01");
        jsonWriter.WriteEndObject(); // this is the end of the JSON Object
        //jsonWriter.Flush();
    }

    string json = Encoding.UTF8.GetString(memStream.ToArray());
    Console.WriteLine(json);
}

This is pretty similar in PowerShell as well, create options for the Utf8JsonWriter and pass them.

#!/usr/bin/env pwsh
using namespace System.IO
using namespace System.Text
using namespace System.Text.Json

# create a memory stream to hold the data in-memory
# the memory stream is a performant way to hold data without locking the source
$memStream = [MemoryStream]::new()

<#
create Json writer options which is a structure (value-type) using PowerShell cast initialization technique
or you could do below:
$jsonWriterOptions = [JsonWriterOptions]::new(); $jsonWriterOptions.Iden
$jsonWriterOptions.Indented = $true
#>
$jsonWriterOptions = [JsonWriterOptions]@{"Indented" = $true}

# use the Utf8 Json Writer object
$jsonWriter = [Utf8JsonWriter]::new($memStream, $jsonWriterOptions)

# indicate the JSON object is starting
$jsonWriter.WriteStartObject();

# now the JSON string is written
$jsonWriter.WriteString("ComputerName", "TestVM01");

# instruct the JSON object is ending
$jsonWriter.WriteEndObject();

# flush the json writer content to the stream
$jsonWriter.Flush();

# dispose of the json writer
$jsonWriter.Dispose();

# Get the string from the byte[] array from the memory stream
[Encoding]::UTF8.GetString($memStream.ToArray())

# dispose the memory stream once read the json string
$memStream.Dispose()

Output is below:

Conclusion

As a PowerShell developer, I learned using Utf8JsonWriter to create JSON strings from scratch and beautify them.

There will be a follow up post on this soon…

Reference links 📖

PowerShell to C# & Back leanpub book

C# JSON Tutorial

PowerShell to C# & back - JSON Seriazlie & Deserialize

Fri, 25 Sep 2020 07:12:33 +0530

Background

In the below previous post

PowerShell to C# & back - JSON parse & enumerate

there were notes on how to enumerate & iterate over JSON documents. Let’s continue down the rabbit hole. In this post we will look at how to use System.Text.Json namespace to:

Serialize to JSON
Deserialize from JSON

Serialze to JSON

This means converting any type into a JSON string. We start off with a type named Computer and convert it into a JSON string. We call the JsonSerializer.Serialze() static method for this operation.

#!/usr/bin/env dotnet-script
using System;
using System.Text.Json;

class Computer
{
    public string hostName { get; set; }

    // In C# 6 and later, we can assign a default value to an auto property
    public string domainName { get; set; } = "contoso.com";
    public string IpAddress { get; set; }
}

var machine1 = new Computer()
{
    hostName = "testvm01",
    IpAddress = "10.10.10.101"
};

// serialize the object into JSON string
var json = JsonSerializer.Serialize(machine1);
Console.WriteLine(json);

Now doing the same in PowerShell 🤑

#!/usr/bin/env pwsh
using namespace System.Text.Json

class Computer {
    [string] $hostName
    [string] $domainName = "contoso.com" # this is the initial value
    [string] $IpAddress
}

$machine1 = [Computer] @{
    hostName = "testvm01"
    IpAddress = "10.10.10.101"
}

# Invoke the static method on the class, it requires 2 arguments
# the Value to convert and any JsonSerializer options which we have passed as $null
$json = [JsonSerializer]::Serialize($machine1, $null)
Write-Host -Object $json

Deserialize from JSON

This means constructing a typed object from JSON string. No surprise there exists a JsonSerializer.Deserialze() static method which can take a JSON string as input and can deserialized it into a typed object instance.

#!/usr/bin/env dotnet-script
using System;
using System.Text.Json;
using System.Net;

class Computer
{
    public string hostName { get; set; }
    public string domainName { get; set; } = "contoso.com";
    public string IpAddress { get; set; }
}

string jsonComputer = "{\"domainName\":\"contoso.com\",\"hostName\":\"testvm01\",\"IpAddress\":\"10.10.10.101\"}";

var computerObject = JsonSerializer.Deserialize<Computer>(jsonComputer);
Console.WriteLine(computerObject.domainName);

In PowerShell the deserialization works really well with classes. If we have a type defined in PowerShell class, we can use the Deserialze() static method and pass it the type to deserialize the JSON string into.

#!/usr/bin/env pwsh
using namespace System.Text.Json

class Computer {
    [string] $hostName
    [string] $domainName = "contoso.com" # this is the initial value
    [string] $IpAddress
}

$jsonComputer = '{"hostName":"testvm01","IpAddress":"10.10.10.101"}'

# Invoke the Deserialize static method on the class, it requires 3 arguments
# the Value to convert, and the type to convert it into and JsonSerializer options which we have passed as $null
$computer = [JsonSerializer]::Deserialize($jsonComputer, [Computer], $null)
# $computer | Get-Member # verify that the type got back is a Computer object type
Write-Host -Object $computer.domainName

Bonus

Above way to deserialize the JSON string into a typed object in PowerShell is pretty neat but we do mention another subtle way to achieve the same in our Book.

Cast-Initialization technique, which works by invoking an empty constrcutor and initializing values.

$jsonComputer = '{"hostName":"testvm01","IpAddress":"10.10.10.101"}'
[Computer]($jsonComputer | ConvertFrom-Json)

Conclusion 🗞

As a PowerShell developer, I learned using System.Text.Json namespace in PowerShell core to serialize/deserialize JSON in this post.

There will be a follow up post on this soon…

Reference links 📖

PowerShell to C# & Back leanpub book

C# JSON Tutorial

PowerShell to C# & back - JSON parse & enumerate

Wed, 16 Sep 2020 11:12:33 +0530

Background

Coming from PowerShell background, while learning ASP.NET Core based development I wanted to wrap my head around how to handle JSON, so ended up taking notes on how to do this in C# and as an exercise convert those into PowerShell code snippets for my reference.

In my explorations I like using the dotnet-script global tool for interactive user experience with C# and PowerShell core (which is already an interactive shell). So, you will see that I mostly use C# script files (.csx) and PowerShell script files (.ps1).

Giving the credit where it’s due, I followed this tutorial to learn JSON handling in C# and made these notes of my own.

In this post the topics covered are:

Parsing JSON
Enumerating JSON

System.Text.Json

This is the new namespace in .NET core which provides the API to work with JSON.

JSON parsing

JSON parsing is basically reading the JSON data and converting it into a JsonDocument typed object in C#.

Long story short:

Use the static method Parse on the JsonDocument class to consume a JSON string and get an object back.
RootElement on the instance of JsonDocument gives you the entire JSON object
Root element if it contains array can be indexed
To get the value of a property on the JSON object use GetProperty('propertyName')

Below are the contents of a test.csx file.

#!/usr/bin/env dotnet-script
using System;
using System.Text.Json;

string data = " [ {\"name\": \"John Doe\", \"occupation\": \"gardener\"}, " +
                "{\"name\": \"Peter Novak\", \"occupation\": \"driver\"} ]";

// using statement is used to dispose of the JsonDocument object later
// Parse static method parses the JSON string and returns a JsonDocument instance
using (JsonDocument doc = JsonDocument.Parse(data))
{
    var root = doc.RootElement; // RootElement is the real object we are after
    var user1 = root[0]; // since our object contains an array, we can index into it
    var user2 = root[2];

    // Use the method GetProperty() to get properties of the JsonElement object
    Console.WriteLine(user1.GetProperty("name")); // GetProperty()
    Console.WriteLine(user2.GetProperty("name"));
}

To do the same in PowerShell, below are the contents of a test.ps1 file.

#!/usr/bin/env pwsh
using namespace System.Text.Json

$data = '[{"name": "John Doe", "occupation": "gardener"},' +
                '{"name": "Peter Novak", "occupation": "driver"} ]';

$doc = [JsonDocument]::Parse($data)
$root = $doc.RootElement
$user1 = $root[0]
$user2 = $root[1]

Write-Host $User1.GetProperty("name")
Write-Host $User2.GetProperty("name")

# dispose the JsonDocument once done
# We don't do this in C# coz of the using statement automatically doing this
$doc.Dispose()

JSON Enumerate

In the above example we indexed into the elements of the array directly and then used GetProperty() this only works if you know the number of elements in advance along with property names, but there are better ways to iterate and enumerate.

Trick for someone new is to remember:

Enumerating arrays to fetch different items
Enumerating objects to fetch properties.

#!/usr/bin/env dotnet-script

using System;
using System.Text.Json;

string data = " [ {\"name\": \"John Doe\", \"occupation\": \"gardener\"}, " +
                "{\"name\": \"Peter Novak\", \"occupation\": \"driver\"} ]";

using (var doc = JsonDocument.Parse(data))
{
    JsonElement root = doc.RootElement;

    // call method EnumerateArray() to iterate over items in the array
    foreach (var user in root.EnumerateArray())
    {
        Console.WriteLine("====== User details ====");
        // EnumerateObject() method is used return an ObjectEnumerator
        // which is then used to get the Name & Value pair for each property
        foreach (var prop in user.EnumerateObject())
        {
            Console.WriteLine($"{prop.Name} -> {prop.Value}");
        }
    }
}

Doing the same in PowerShell makes it easier to grasp.

#!/usr/bin/env pwsh
using namespace System.Text.Json

$data = "[ {`"name`": `"John Doe`", `"occupation`": `"gardener`"}, " +
                "{`"name`": `"Peter Novak`", `"occupation`": `"driver`"} ]";

$doc = [JsonDocument]::Parse($data)
$root = $doc.RootElement


foreach ($user in $root.EnumerateArray()) {
    Write-Host "====== User details ===="
    foreach ($property in $user.EnumerateObject()) {
        Write-Host $("{0} -> {1}" -f $property.Name, $property.Value)
    }
}

Conclusion 🗞

In my continued efforts to learn C# language and take some of the learnings back to PowerShell, I realized that though the JSON cmdlets provide a lot of ease working with JSON, there are other ways to skin the cat.

There will be a follow up post on this soon…

Reference links 📖

PowerShell to C# & Back leanpub book

C# JSON Tutorial

PowerShell Classes - Validating ARM parameters

Mon, 29 Jun 2020 19:42:59 +0530

Origin❓

(shameless plug, alert!) 🙃

Recently, I was discussing with my colleague about this new book I am co-authoring (with Prateek) about why to learn .NET to be a better PowerShell programmer and upon further discussion we pondered some interesting ways to use PowerShell classes.

Brain-storming 🤔

All was lost, until we had another quick conversation about how to validate ARM templates. Well, I suggested to write Pester tests to check the input being passed and perform Test-AzDeployment for ARM templates.

Another idea that popped up in my mind was what if we write a PowerShell class to model the ARM parameters file and use that to validate the ARM template parameter inputs.

Solution 🚀

There exists the ConvertToClass module by Doug Finke, which comes to the rescue to automatically convert a JSON object to a PowerShell class. It even has VSCode integration, check it out.

Let’s start by taking a sample azuredeploy.parameters.json file.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountNamePrefix": {
      "value": "Azurearm6754"
    },
    "numberofStorageAccounts": {
      "value": 2
    },
    "storageAccountType": {
      "value": "Standard_LRS"
    },
    "location": {
      "value": "westus2"
    }
  }
}

Convert Json to Class ☯

Let’s run the ConverTo-Class function in the ConvertToClass module against this azuredeploy.parameters.json file.

ConvertTo-Class -Target (Get-Content ./azuredeploy.parameters.json -Raw)

Above command generates below content:

class RootObject1 {
    [string]$$schema
    [string]$contentVersion
    [parameters]$parameters
}

class parameters1 {
    [storageAccountNamePrefix]$storageAccountNamePrefix
    [numberofStorageAccounts]$numberofStorageAccounts
    [storageAccountType]$storageAccountType
    [location]$location
}

class storageAccountNamePrefix1 {
    [string]$value
}

class numberofStorageAccounts1 {
    [int]$value
}

class storageAccountType1 {
    [string]$value
}

class location1 {
    [string]$value
}

Let’s rename the above classes like below:

$$schema property on RootObject1 to ${$schema}, done to escape $ char in property name
RootObject1 to AzureParameters
parameters1 to Parameters
storageAccountNamePrefix1 to StorageAccountType
numberofStorageAccounts1 to StorageAccountType
storageAccountType1 to StorageAccountType
location1 to Location

Add validation attributes 🔨

We can add validation attributes to the property $value present inside the auto-generated class to add some quick validation rules to the properties present on the class.

Note the name $value is given to the property because this is how ARM parameters file take input.

class AzureParameters {
    [string]${$schema}
    [string]$contentVersion
    [Parameters]$parameters
}

class Parameters {
    [StorageAccountNamePrefix]$storageAccountNamePrefix
    [NumberofStorageAccounts]$numberofStorageAccounts
    [StorageAccountType]$storageAccountType
    [Location]$location
}

class StorageAccountNamePrefix {
    # validate the prefix length
    [ValidateLength(3,15)]
    [string]$value
}

class StorageAccountNamePrefix {
    # restricting the min=1 and max=10 storage accounts that one can request
    [ValidateRange(1,10)]
    [int]$value
}

class StorageAccountType {
    # Restricting the StorageAccount SKUs
    [ValidateSet('Standard_LRS', 'Standard_GRS', 'Standard_ZRS')]
    [string]$value
}

class Location {
    # Restricting the locations
    [ValidateSet('westus2', 'northcentralus')]
    [string]$value
}

Add logic inside Empty Constructor ⌛

We can add one more trick to the bag to add an empty constructor explicitly (this is present when no constructor exists) and put some more validation logic if the current validate attributes doesn' suit the needs.

class AzureParameters {
    [string]${$schema}
    [string]$contentVersion
    [Parameters]$parameters
}

class Parameters {
    [StorageAccountNamePrefix]$storageAccountNamePrefix
    [NumberofStorageAccounts]$numberofStorageAccounts
    [StorageAccountType]$storageAccountType
    [Location]$location
}

class StorageAccountNamePrefix {
    # validate the prefix length
    [ValidateLength(3,15)]
    [string]$value
}

class NumberofStorageAccounts {
    # restricting the min=1 and max=10 storage accounts that one can request
    [ValidateRange(1,10)]
    [int]$value
}

class StorageAccountType {
    # Restricting the StorageAccount SKUs
    [ValidateSet('Standard_LRS', 'Standard_GRS', 'Standard_ZRS')]
    [string]$value

    # Add some more validation logic inside the empty constructor
    storageAccountType() {
        # Perform some validation on the property
        if ($this.value -ne 'Standard_ZRS') {
            # you can perform more checks or logging, throwing a warning here
            Write-Warning -Message "No zone redundancy"
        }
    }
}

class Location {
    # Restricting the locations
    [ValidateSet('westus2', 'northcentralus')]
    [string]$value
}

Why does this has to be an empty constructor? The answer is that we will be a using a trick with how we create an instance of the class in next section

Secret-Sauce : Cast-Initialization 🍲

Credit goes to Bruce Payette’s Windows PowerShell in Action 3rd edition which talks a bit about this technique in brief.

In short, you can take a hashtable or PSObjects and cast them as strongly type object instance. This will become more clear when we perform this casting a bit later.

For cast-initalization technique to work an empty constructor needs to be present in the Class definition. It is present by default if there is no constructor on the class present or you can add one explicitly.

Modify some value in the azuredeploy.parameters.json file to not-follow some validation logic we added e.g. putting the value for numberofStorageAccount as 20, remember our [ValidateRange(1,10)] attribute on the property should fails. See the parameters file below:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountNamePrefix": {
      "value": "azurearm6754"
    },
    "numberofStorageAccounts": {
      "value": 20
    },
    "storageAccountType": {
      "value": "Standard_LRS"
    },
    "location": {
      "value": "westus2"
    }
  }
}

Now, see the result when casting this as our ArmParameters class.

[azureparameters] (Get-Content -Path $PSScriptRoot/azuredeploy.parameters.json | ConvertFrom-Json)

Throws below error:

InvalidArgument: Cannot convert value "@{$schema=https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#; contentVersion=1.0.0.0; parameters=}" to type "AzureParameters". Error: "Cannot convert value "@{storageAccountNamePrefix=; numberofStorageAccounts=; storageAccountType=; location=}" to type "Parameters". Error: "Cannot create object of type "NumberofStorageAccounts". The 20 argument is greater than the maximum allowed range of 10. Supply an argument that is less than or equal to 10 and then try the command again.""

Fix the value for the numberofStorageAccounts parameter.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountNamePrefix": {
      "value": "azurearm6754"
    },
    "numberofStorageAccounts": {
      "value": 2
    },
    "storageAccountType": {
      "value": "Standard_LRS"
    },
    "location": {
      "value": "westus2"
    }
  }
}

Run the below again:

[azureparameters] (Get-Content -Path $PSScriptRoot/azuredeploy.parameters.json | ConvertFrom-Json)

Conclusion ✅

This is maybe a very basic way of validating input data but I read about a topic and used it to solve a problem in a unique way, which is a win for me 😎.

I think slowly embracing more Object-Oriented programming using PowerShell classes (or C#) can open up some interesting ways to solve problems in my tooling.

Resource links 📚

PowerShell to C# and back - Disclaimer: co-author on this one.
PowerShell to C# and Back – Introduction to Classes
Windows PowerShell in Action, 3rd Edition
Doug Finke’s ConvertToClass module
PSClassUtils

.NET notes - create SHA256 hash

Fri, 05 Jun 2020 10:18:40 +0530

Background 🧐

Today I was looking to generate SHA256 hash for input string data. Below are my notes on how I used dotnet script (interactive scripting experience) in .NET to experiment with it.

P.S. - Writing these small .NET recipes helps me in absorbing more.

Walkthrough ⚡

Using dotnet script

To quickly test features in .NET core, I use the dotnet script global tool.

Begin with creating a dotnet script file (.csx extension).

mkdir hashing
cd hashing
dotnet script init sha256hash

Above dotnet command creates an executable file with below content.

#!/usr/bin/env dotnet-script

Console.WriteLine("Hello world!");

Since this is an executable script, we can run it like this as well.

./sha256hash.csx

Output:

Hello world!

Using statements in scripts

Now, let’s modify this script to explore creating a SHA256 hash.

First, thing is we need to place some using statements to bring in the Cryptography and Text namespace like below.

So the content now looks like.

#!/usr/bin/env dotnet-script

// Add the using statements
using System.Text;
using System.Security.Cryptography;

Reading the docs

Moving on, after browsing the documentation of the SHA256 class I noticed a Create() method which creates a default instance.

Also, noticed that the inheritance chain for this class is Object -> HashAlgorithm -> SHA256

HashAlgorithm is the base class with below signature, notice it in turn inherits from IDisposable interface this immediately reminded me to use the using statement syntax to conveniently dispose this object after re-use, rather than calling Dispose() method myself inside try/catch/finally statements.

public abstract class HashAlgorithm : IDisposable, System.Security.Cryptography.ICryptoTransform

#!/usr/bin/env dotnet-script

using System.Text;
using System.Security.Cryptography;

string text = "DexterPOSH"; // this is our text for which we will generate hash

using (SHA256 hashAlgorithm = SHA256.Create())
{
    // place holder
};

The docs also reveal the ComputeHash() method which takes byte[] array as argument and returns the byte array back as well. We need some way to convert our string input to byte array.

string to byte[] conversion

Quick search suggests to use Encoding.UTF8.GetBytes() static method for converting string to byte array. Using that in code now leads us to this point.

#!/usr/bin/env dotnet-script

using System.Text;
using System.Security.Cryptography;

string text = "DexterPOSH"; // this is our text for which we will generate hash

using (SHA256 hashAlgorithm = SHA256.Create())
{
    var hashedByteArray = hashAlgorithm.ComputeHash(Encoding.UTF8.GetBytes(input));
    Console.WriteLine(hashedByteArray);
}

Output:

System.Byte[]

Let’s see how we can convert the byte[] to string object.

byte[] to string conversion

Again search and got a hint of using the BitConverter.ToString() static method. Let’s add that logic in our script.

#!/usr/bin/env dotnet-script

using System.Text;
using System.Security.Cryptography;

string text = "DexterPOSH"; // this is our text for which we will generate hash

using (SHA256 hashAlgorithm = SHA256.Create())
{
    var hashedByteArray = hashAlgorithm.ComputeHash(Encoding.UTF8.GetBytes(input));
    Console.WriteLine(BitConverter.ToString(hashedByteArray));
};

Output:

1C-29-A2-30-0A-8A-99-6F-67-60-70-7E-21-0D-BD-61-B1-C9-3A-B7-4F-86-EE-13-7B-2E-DE-B6-01-6E-87-93

remove ‘-’ from string

As the last step let’s replace the char - from the output string. String class has replace method which invoke.

#!/usr/bin/env dotnet-script

using System.Text;
using System.Security.Cryptography;
using (SHA256 hashAlgorithm = SHA256.Create())
{
    string input = "DexterPOSH";

    byte[] data = hashAlgorithm.ComputeHash(
        Encoding.UTF8.GetBytes(input)
    );
    Console.WriteLine(BitConverter.ToString(data).Replace("-", String.Empty));
}

Output:

1C29A2300A8A996F6760707E210DBD61B1C93AB74F86EE137B2EDEB6016E8793

Solution 😎

Content of the sha256hash.csx file.

#!/usr/bin/env dotnet-script

using System.Text;
using System.Security.Cryptography;
using (SHA256 hashAlgorithm = SHA256.Create())
{
    string input = "DexterPOSH";

    byte[] data = hashAlgorithm.ComputeHash(
        Encoding.UTF8.GetBytes(input)
    );
    Console.WriteLine(BitConverter.ToString(data).Replace("-", String.Empty));
}

Run the above file

dotnet script run ./sha256hash.csx # or simply ./sha256hash.csx

Output:

1C29A2300A8A996F6760707E210DBD61B1C93AB74F86EE137B2EDEB6016E8793

Reference links 📖

System.Security.CryptoGraphy Class

HashAlgorithm Base Class

using statement reference in C#

C# String To Byte Array

Convert Byte Array To String In C#

ARM templates - iterate & deploy resource

Tue, 26 May 2020 15:18:35 +0530

Background 🧐

I like ARM templates, I use it a lot to deploy Azure cloud resources but as all things it has some pain points associated with it. In this post, let’s see how you can iterate over based on certain logic and deploy multiple resources using linked templates.

As it stands out this logic of iterating over and deploying multiple instances of a resource tripped me a lot in the beginning.

Walkthrough 🏃

Let’s work through the whole process of writing an ARM template which deploys multiple resources.

Github Repository - ArmTemplateLoopExample

This is a simple post demonstrating looping logic I often use, feel free to sprinkle your own best practices & modifications on top e.g. storing templates in a private Cloud blob container, adding more parameters, names etc.

Scenario

Let’s take a scenario of deploying many storage accounts based on the user input.

Ideally, if you’re in this situation you should write 2 templates and utilize ARM linked templates to deploy them because it becomes too cumbersome to maintain a single ARM template to deploy a resource and loop over user-input and deploy multiple iterations of that resource. Trust me this is coming from experience 😉

So, let’s start by creating 2 templates, I am going to use GitHub repository here for storing those but you can use a Cloud blob store account as well.

Below is how my project directory layout looks like.

.
├── azuredeploy.json
└── linkedTemplate
   └── storageaccount.json

Author linked template

First thing to do when you’re writing an ARM template is to make sure you understand that component properly, how it works, best practices while using that Azure component etc. Why? you might be wondering because ARM templates is how you deploy your Azure cloud infrastructure and it would be as good as you make your ARM templates, they’re called blueprints for your Azure resources for this reason.

But at the same time start small and head over to azure-quickstart-templates repository to get some samples.

I found out that the template stored here in this 101-storage-account-create example is good enough for me. So, let me copypasta ✍ this and place the content inside my linkedtemplate\storageaccount.json file.

So, we have a starting point which can deploy a single storage account for us, but you would notice on closer inspection that this storageaccount.json template doesn’t take storageAccountName as a parameter but generates it in the variables section.

Let’s quickly modify it. Changes made:

add parameter storageAccountName
remove variable storageAccountName
change [variables('storageAccountName')] references to [parameters('storageAccountName')]

{
   "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
   "contentVersion": "1.0.0.0",
   "parameters": {
     "storageAccountName": { // added the storageAccountName parameter
       "type": "string"
     },
     "storageAccountType": {
       "type": "string",
       "defaultValue": "Standard_LRS",
       "allowedValues": [
         "Standard_LRS",
         "Standard_GRS",
         "Standard_ZRS",
         "Premium_LRS"
       ],
       "metadata": {
         "description": "Storage Account type"
       }
     },
     "location": {
       "type": "string",
       "defaultValue": "[resourceGroup().location]",
       "metadata": {
         "description": "Location for all resources."
       }
     }
   },
   "variables": {
     // removed the storageAccountName variable from here
   },
   "resources": [
     {
       "type": "Microsoft.Storage/storageAccounts",
       "apiVersion": "2019-04-01",
       "name": "[parameters('storageAccountName')]",
       "location": "[parameters('location')]",
       "sku": {
         "name": "[parameters('storageAccountType')]"
       },
       "kind": "StorageV2",
       "properties": {}
     }
   ],
   "outputs": {
     "storageAccountName": {
       "type": "string",
       "value": "[parameters('storageAccountName')]"
     }
   }
 }

Author the stitching logic

Moving on to the logic of consolidating user input and then looping over and deploying a storage account multiple times.

The gist is that we have to do below:

Use variable iteration to create an array of objects based on our numberofStorageAccounts parameter value
Use resource iteration later with a linked template deployment and index into the array created above for parameter values.

Don’t worry if this is a bit daunting. It was for me the first time.

Adding barebone template

Let’s start by creating a blank ARM template. Open the azuredeploy.json in VSCode. Key in arm and it would give you a snippet dropdown, select the first one for targeting a Resource group deployment.

So, we get this.

{
 "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
 "contentVersion": "1.0.0.0",
 "parameters": {},
 "functions": [],
 "variables": {},
 "resources": [],
 "outputs": {}
}

Add parameters

Time to add in some parameters to our azuredeploy.json which is end-user facing. So you need to take input in this one from the user (which could be yourself as well) and then pass those over to the linkedtemplate.

storageAccountNamePrefix - prefix for the storage accounts to be deployed. Length 5-10.
numberofStorageAccounts - integer representing how many storage accounts to deploy. [Default - 1, Min 1, Max 10.
storageAccountType - Type of the storage accounts, predefined allowed values. Default - Standard_GRS.
location - location for the storage accounts. Default is RG location.

Below is how the parameters object looks now.

"parameters": {
  "storageAccountNamePrefix": {
    "type": "string",
    "minLength": "5",
    "maxLength": "10"
  },
  "numberofStorageAccounts": {
    "type": "int",
    "defaultValue": 1,
    "minValue": "1",
    "maxValue": "10"
  },
  "storageAccountType": {
    "type": "string",
    "defaultValue": "Standard_GRS",
    "allowedValues": [
      "Standard_LRS",
      "Standard_GRS",
      "Standard_ZRS",
      "Premium_LRS"
    ]
  },
  "location": {
    "type": "string",
    "defaultValue": "[resourceGroup().location]",
    "metadata": {
      "description": "Location for all resources."
    }
  }
}

Add the variables (iteration logic)

I typically like to use variables a lot for transforming the input parameters and then using these variables later in the resources because it makes it easier in future to just modify these variables at once place.

Use the concept of variable iteration in ARM templates.

We use the above concept to do the below

Create a variable named _deployMultipleStorageAccounts (sort of a convention I follow to name these variables used later in deployment to preceed with _deploy)
Use the parameter numberofStorageAccounts to loop over that many times
Use the input property in the copy loop object to generate an object containing properties which will be mapped one to one with the linked template storage.

I prefer one to one mapping between the properties inside the input in the copy loop to the parameters of the linked template. It makes it easier to index into them and specify them (you’ll see later).

Below is a gist of what I added in the variables property.

"variables": {
  "copy": [
    {
      "name": "_deployMultipleStorageAccounts", // variable name used later with resources
      "count": "[parameters('numberofStorageAccounts')]", // loop over numberofStorageAccounts time
      "input": { // input object con
        "storageAccountType": "[parameters('storageAccountType')]",
        "location": "[parameters('location')]",
        "storageAccountName": "[
          concat(
            parameters('storageAccountNamePrefix'),
            uniqueString(resourceGroup().id),
            copyIndex('_deployMultipleStorageAccounts')
          )
        ]"
      }
    }
  ]
}

In the outputs section we added a variables property which essentially displays the value for the variable _deployMultipleStorageAccounts. This can be used later with a trick to see what values go inside this variable.

"outputs": {
  "variables": {
    "type": "object",
    "value": {
      "_deployMultipleStorageAccounts": "[variables('_deployMultipleStorageAccounts')]"
    }
  }
}

Also, I added a variables property in the output which is used to display what goes inside this variable once it is run by ARM API.

From a data-view point above creates a variable named _deployMultipleStorageAccounts which is an array of Json objects.

If we assume the parameters('numberofStorageAccounts') is 2, parameters('storageAccountType') is Standard_GRS and parameters('location') is SouthEastAsia, then it creates an array like below:

[
  {
    "storageAccountName": "<generatedValuebyARM>",
    "storageAccountType": "Standard_GRS",
    "loation": "SouthEastAsia",
  },
  {
    "storageAccountName": "<generatedValuebyARM>",
    "storageAccountType": "Standard_GRS",
    "location": "SouthEastAsia",
  }
]

Add the deployment resource

Now, you already know we have the linked template to deploy a single storage account. So, we just need to invoke/call that template multiple times and pass in paramters.

This is done by an ARM template technique called as ARM template linked templates. Read about it more here

Follow a Tutorial to deploy a linked template, if this is the firs time you’re hearing about this.

In short, within our azuredeploy.json template we need to use a resource of type Microsoft.Resources/deployments to link to our storageaccount.json template and inside this resource we need to use another concept termed as Resource iteration

Let’s add it.

This is how my resources array property looks like.

"resources": [
  {
    "type": "Microsoft.Resources/deployments",
    "apiVersion": "2019-10-01",
    "condition": false,
    "name": "[concat('deploy-linkedStorageTemplate', copyIndex())]",
    "copy": {
      "count": "[parameters('numberofStorageAccounts')]",
      "name": "_loopToDeployStorageAccounts",
      "mode": "Parallel"
    },
    "properties": {
      "mode": "Incremental",
      "templateLink": {
        "contentVersion": "1.0.0.0",
        "uri": "https://raw.githubusercontent.com/DexterPOSH/ArmTemplateLoopExample/master/linkedtemplate/storageaccount.json"
      },
      "parameters": {
        "storageAccountName": {
          "value": "[variables('_deployMultipleStorageAccounts')[copyIndex()].storageAccountName]"
        },
        "storageAccountType": {
          "value": "[variables('_deployMultipleStorageAccounts')[copyIndex()].storageAccountType]"
        },
        "location": {
          "value": "[variables('_deployMultipleStorageAccounts')[copyIndex()].location]"
        }
      }
    }
  }
]

I know it’s a handful but below is a breakdown of major things it does.

Uses Microsoft.Resources/deployments resource type to deploy another template.

Note we generate a unique name for the deployment by concating the copyIndex() The condition property is set to false

{
  "type": "Microsoft.Resources/deployments",
  "apiVersion": "2019-10-01",
  "condition": false,
  "name": "[concat('deploy-linkedStorageTemplate', copyIndex())]"
  //<-- skipped below properties -->
}

Uses the uri of the raw template link for the storageaccount.json in the GitHub repository.

"templateLink": {
  "contentVersion": "1.0.0.0",
  "uri": "https://raw.githubusercontent.com/DexterPOSH/ArmTemplateLoopExample/master/linkedtemplate/storageaccount.json"
}

Uses resource iteration by using copy property and using the parameters('numberofStorageAccounts') as the value for count, which means it loops over this resource this many times. Also, gives this copy loop a friendly name _loopToDeployStorageAccounts.

"copy": {
  "count": "[parameters('numberofStorageAccounts')]",
  "name": "_loopToDeployStorageAccounts"
}

Passes on the parameters to this linked template by indexing into the variable _deployMultipleStorageAccounts

"storageAccountName": {
  "value": "[variables('_deployMultipleStorageAccounts')[copyIndex()].storageAccountName]"
}

Dry run - Verify Variable iteration logic

Based on my experience with this approach of looping over, we can most of the time validate what is inside the variable created for looping to verify it will work.

Remember the condition is set to false for our linked template deployment resource, which means when we submit this ARM template for deployment it won’t trigger it but however the azuredeploy.json will be processed and we will get the output back which contains the variables property.

Let’s create an ARM template parameters file, with the new release of the ARM tools VSCode extension, it is natively possible to generate these parameters file. Read more in the release notes here.

Click on Select Parameter File… (at the bottom) > New > All parameters > Save it. Open it and fill the values.

This is how it looks after adding values.

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "storageAccountNamePrefix": {
      "value": "azurearm6754"
    },
    "numberofStorageAccounts": {
      "value": 2
    },
    "storageAccountType": {
      "value": "Standard_GRS"
    },
    "location": {
      "value": "southindia"
    }
  }
}

Deploy it using Az PowerShell module cmdelt

$rg =  "test_arm_rg"
New-AzResourceGroupDeployment -TemplateFile ./azuredeploy.json -TemplateParameterFile ./azuredeploy.parameters.json -ResourceGroupName $rg

DeploymentName          : azuredeploy
ResourceGroupName       : test_arm_rg
ProvisioningState       : Succeeded
Timestamp               : 06/07/2020 14:24:21
Mode                    : Incremental
TemplateLink            :
Parameters              :
                          Name                        Type                       Value
                          ==========================  =========================  ==========
                          storageAccountNamePrefix    String                     azurearm6754
                          numberofStorageAccounts     Int                        2
                          storageAccountType          String                     Standard_GRS
                          location                    String                     southindia

Outputs                 :
                          Name             Type                       Value
                          ===============  =========================  ==========
                          variables        Object                     {
                            "_deployMultipleStorageAccounts": [
                              {
                                "storageAccountType": "Standard_GRS",
                                "location": "southindia",
                                "storageAccountName": "azurearm67543ub5zsu77klvq0"
                              },
                              {
                                "storageAccountType": "Standard_GRS",
                                "location": "southindia",
                                "storageAccountName": "azurearm67543ub5zsu77klvq1"
                              }
                            ]
                          }

DeploymentDebugLogLevel :

Look at the outputs section, it clearly lists out the variable we generated and upon which our whole logic of depolying multiple resources existed.

"_deployMultipleStorageAccounts": [
  {
    "storageAccountType": "Standard_GRS",
    "location": "southindia",
    "storageAccountName": "azurearm67543ub5zsu77klvq0"
  },
  {
    "storageAccountType": "Standard_GRS",
    "location": "southindia",
    "storageAccountName": "azurearm67543ub5zsu77klvq1"
  }
]

Test the solution

Once the variable iteration logic is verified, it is time to deploy the tempalte to see that it actually creates.

Wait! before you jump into testing it you need to make a minor change. Can you guess what? Set condition to true inside the linked template deployment resource.

Below is a snippet of where that change goes inside the template.

"resources": [
  {
    "type": "Microsoft.Resources/deployments",
    "apiVersion": "2019-10-01",
    "condition": true // set this to true to actually deploy the linked template,
    //<-- skipped below properties -->
  }
]

Deploy again, this time it should deploy multiple storage accounts.

$rg =  "test_arm_rg"
New-AzResourceGroupDeployment -TemplateFile ./azuredeploy.json -TemplateParameterFile ./azuredeploy.parameters.json -ResourceGroupName $rg

TLDR; Solution 🗞

Head over to this GitHub repository to see the ARM templates.

ArmTemplateLoopExample

References 📚

Azure Resource Manager (ARM) Tools VSCode extension

Azure QuickStart Templates

Linked Templates

Deploy a linked template tutorial

Resource-Ieration

AKS PowerShell Tip - Add Authorized Ip

Fri, 01 May 2020 12:21:28 +0530

Background 🐼

Recently, I found out that there is no sane way to perform adding a public IP address to the authorized IP address ranges using either the Az CLI or Az.Aks PowerShell (no cmdlets available yet) module.

From the official docs it says to use something like below format with Az CLI.

az aks update \
    --resource-group myResourceGroup \
    --name myAKSCluster \
    --api-server-authorized-ip-ranges 73.140.245.0/24

But it doesn’t tell you how to append the IP to the range, instead you need to supply a comma separated value of public IP addresses.

Challenge ☁️

Well, this is can be done by using Az CLI with PowerShell or Bash and parsing output then generating a comma separated string and passing it back to Az CLI 😞

Solution ⚡

Often, when I am hit with such limitations with cmdlets or Az CLI making life hard. I go back to using simply the 2 cmdlets provided by Az.Resources module.

Behold mighty!

Get-AzResource - Gets the Az resource
Set-AzResource - Modifies the Az resource

I ended up doing the below and creating a utility function out of it.

First, get the AKS Cluster resource. Make sure to specify the -ExpandProperties switch to get back full fledged resource otherwise it returns a shallow instance.

$ResourceGroup = "test-aks-rg"
$Name = "aksCluster001"
$IP = "110.91.234.43"
$AksCluster = Get-AzResource -ResourceGroupName $ResourceGroup -Name $Name -ExpandProperties -ErrorAction Stop

Once you have the resource, walk-through the properties and append the IP (+= operator in PowerShell) to the local copy of the resource.

$orgClusterInfo.Properties.apiServerAccessProfile.authorizedIpRanges += $Ip

Finally, perform a Set operation by piping the modified local resource copy to Set-AzResource cmdlet.

$orgClusterInfo | Set-AzResource -ErrorAction Stop

Takeaway 🔥

Even, when there are certain utility functions not available in the Az PowerShell module. We can rely on the `-Resource* cmdlets to work our way through.

Azure DevOps Tip - Job re-use within a Stage

Sun, 26 Apr 2020 09:53:50 +0530

Background

Azure DevOps introduced multi-stage yaml pipelines a while ago. It allows us to define our entire Build/Release landscape inside these yaml definitions.

To re-iterate of some terms used in this post:

A pipeline comprises one or more stages
Stage is collection of jobs
Job runs on an agent/ agentless
Job contains steps (task/script)
Steps are the atomic unit to perform a task

Challenge

Recently, working our multi-stage yaml pipelines, we hit an interesting behavior with our job template re-use.

We had quite few common steps we require to take in each stage (multiple times) to hit an external API. So, to re-use we extracted them out as a job template.

jobtemplate.yml

For this post, creating a sample job and taking an agentless delay task as an example but this job template can include series of steps.

Below is my extracted out job template definition (indues a delay) which I want to re-use within stages in my main azure-pipelines.yml pipeline definition later.

parameters:
  name: 'windows'
jobs:
  - job: commonJob
    displayName: Common Job
    pool: server
    steps:
    - task: Delay@1
      displayName: 'Delay by 1 minutes for ${{ parameters.name }}'
      inputs:
        delayForMinutes: 1

azure-pipelines.yml

Now, I want to call my Job template from above inside two stages Build & Deploy in the multi-stage yaml pipeline as below.

stages:
- stage: Build
  jobs:
  - template: ./jobtemplate.yml
  - job: Build1

- stage: Deploy
  jobs:
    - template: ./jobtemplate.yml
    - job: Deploy1

Kick off a build for this and you would see it in the AzDO portal, that it works out well.

But, if I try to use the same Job template multiple times within the same stage that is when things get interesting. Let’s modify our azure-pipelines.yml definition now.

stages:
- stage: Build
  jobs:
  - template: ./jobtemplate.yml
    parameters:
        name: Deepak
  - template: ./jobtemplate.yml
    parameters:
        name: Dhami
  - job: Build1

- stage: Deploy
  jobs:
    - template: ./jobtemplate.yml
    - job: Deploy1

Try running this new modified pipeline and you would be greeted with the below in the portal.

This is quite obvious but the job names in a stage should be unique.

Solution

To get to a solution we looked at generating a unique name for our jobs but it turns out there is a very simple solution to this problem.

One can remove the job identifier in the job template completely while extracting out jobs to re-use.

What this means is we had to modify our Job templates like below:

parameters:
  name: 'windows'
jobs:
  - job: # Identifier here is removed
    displayName: Common Job
    pool: server
    steps:
    - task: Delay@1
      displayName: 'Delay by 1 minutes for ${{ parameters.name }}'
      inputs:
        delayForMinutes: 1

Now, it works.

Conclusion

As a general practice, when I start extracting out steps as independent job templates I tend to not populate the Job identifier if it’s scope is to be re-used multiple times in the same stage.

But if there is a job which you want to ensure that it is allowed to run only once, you could stick with populating the field.

So, as every engineer building solution says:

It depends on the use-case

Azure DevOps Tip - Find private APIs

Tue, 14 Apr 2020 15:06:22 +0530

Problem

Often working with Azure DevOps, I hit a wall trying to automate some tasks but there are no REST APIs made public yet.

It was one of those task of automating creation of Environments in multi-stage YAML based pipelines in AzDO.

Quick research reveals that this has been requested in uservoice (please upvote). Let’s see one of the very simple ways to discover some of these APIs.

Developers Tools to rescue

Using your browser’s developers tools you can actually inspect the HTTP requests being made while performing an action in the web portal.

Let’s do this.

Let’s click on the “Create Environment” button, fill out some dummy values, hit create and keep an eye on the network tab in the developer tools.

We see some activity, it might take you some time to walk through what happened but in this case the top activity named “environments” has the required details.

See below and note the URL & method used:

Also, make note of the Json request in the payload.

That’s mostly it, fire up postman/PowerShell to make the API call to test this.

Invoke-RestMethod in Pwsh

Generate a Personal Access Token in AzDO, typically start with a short lived PAT token with full access and then nail down on the specific permissions you need.

Below is the code snippet, I used with AzDO to hit the REST API endpoint:

$url = 'https://dev.azure.com/ddhami/BITPro.AzDeploy/_apis/distributedtask/environments'
$cred = Get-Credential -UserName 'vsts' -Message 'Enter AzDO Personal Access Token with privs to create env'
$encodedValue = [Convert]::ToBase64String(
            [Text.Encoding]::ASCII.GetBytes(
                ("{0}:{1}" -f '', $cred.GetNetworkCredential().Password)
            )
        )

$body = @{
  name = 'test-pwsh-env';
  description = 'test environment from APR'
} | ConvertTo-Json
$headers = @{
  Accept = 'application/json;';
  Authorization = "Basic {0}" -f $encodedValue
}

Invoke-RestMethod -Method POST -Uri $url -Body $body -Headers $headers -ContentType 'application/json'

But when I execute the above code, it gives me an error.

The error thrown is below.

{
    "$id": "1",
    "innerException": null,
    "message": "No api-version was supplied for the \"POST\" request. The version must be supplied either as part of the Accept header (e.g. \"application/json; api-version=1.0\") or as a query parameter (e.g. \"?api-version=1.0\").",
    "typeName": "Microsoft.VisualStudio.Services.WebApi.VssVersionNotSpecifiedException, Microsoft.VisualStudio.Services.WebApi",
    "typeKey": "VssVersionNotSpecifiedException",
    "errorCode": 0,
    "eventId": 3000
}

Read the error message, it explains that the api-version is missing. Also, looking back at the capture and see where the api-version was specified.

Let’s make the same change in our code snippet to include API version in the header.

$url = 'https://dev.azure.com/ddhami/BITPro.AzDeploy/_apis/distributedtask/environments'
$cred = Get-Credential -UserName 'vsts' -Message 'Enter AzDO Personal Access Token with privs to create env'
$encodedValue = [Convert]::ToBase64String(
            [Text.Encoding]::ASCII.GetBytes(
                ("{0}:{1}" -f '', $cred.GetNetworkCredential().Password)
            )
        )

$body = @{
  name = 'test-pwsh-env';
  description = 'test environment from APR'
} | ConvertTo-Json
$headers = @{
  Accept = 'application/json;api-version=5.0-preview.1';
  Authorization = "Basic {0}" -f $encodedValue
}

Invoke-RestMethod -Method POST -Uri $url -Body $body -Headers $headers -ContentType = 'application/json'

Here I go, this finally works and using the similar API endpoint I can fetch the environments as well (GET request).

#1 Hello World

Tue, 14 Apr 2020 12:20:36 +0530

print("Hello World!")

Write-Host -Object "Hello World"

Engineer taking Dev?Ops notes

About

Subscribe

Kubernetes API Terms

Overview

API-Server

API terminology

Kind

API Group

Version

Resource

Points to note

Azure Vms Resource Graph Queries

Why use Resource Graph instead Az CLI/ PowerShell?

Prerequisite

Virtual Machine Queries

Find Virtual Machine with Name

Find Virtual machines with name regex matching

Find Virtual machines with a specific tag

Find Virtual machines deployed using MarketPlace images

Find Virtual machines deployed using custom Images

Find Virtual machines which do not have a Tag populated

Find Virtual machine with an IPAddress

Gather extra Virtual Machine information

Conclusion

PowerShell to C# & back - JSON Create, Beautify

Background

JSON Create object

Beautify JSON

Conclusion

Reference links 📖

PowerShell to C# & back - JSON Seriazlie & Deserialize

Background

Serialze to JSON

Deserialize from JSON

Bonus

Conclusion 🗞

Reference links 📖

PowerShell to C# & back - JSON parse & enumerate

Background

JSON parsing

JSON Enumerate

Conclusion 🗞

Reference links 📖

PowerShell Classes - Validating ARM parameters

Origin❓

Brain-storming 🤔

Solution 🚀

Convert Json to Class ☯

Add validation attributes 🔨

Add logic inside Empty Constructor ⌛

Secret-Sauce : Cast-Initialization 🍲

Conclusion ✅

Resource links 📚

.NET notes - create SHA256 hash

Background 🧐

Walkthrough ⚡

Using dotnet script

Using statements in scripts

Reading the docs

string to byte[] conversion

byte[] to string conversion

remove ‘-’ from string

Solution 😎

Reference links 📖

ARM templates - iterate & deploy resource

Background 🧐

Walkthrough 🏃

Scenario

Author linked template

Author the stitching logic

Adding barebone template

Add parameters

Add the variables (iteration logic)

Add the deployment resource

Dry run - Verify Variable iteration logic

Test the solution

TLDR; Solution 🗞

References 📚

AKS PowerShell Tip - Add Authorized Ip