Today’s topic is critical to complete full picture of software development for network automation. Today’s topic is what allows you to conduct your tasks within meaningful time frame, especially when you have a lot of network devices, servers, virtual machines to manage. Today’s topic is concurrency of code execution in Python and Golang.

What Other Programming Languages Makes Sense To Study?

There are more than 100 programming languages out there. Some of them are quite universal and allow development of almost any kind of application. Others are more specific. Python is probably the most universal programming language, from what I’ve worked with or heard of. It can be used in infrastructure management at scale (e.g., OpenStack is written in Python), web applications, data science and many more. Golang is much more low-level compared to Python and, therefore, way more performant. Various benchmarks available online suggests that the same business tasks could be 3-30 times quicker in Golang compared to Python; therefore Golang is suitable for system programing (e.g, Kubernetes and Docker are created in Go). That’s what we cover in our blogs. Apart from them there are a lot of other: C/C++ if you need even lower level than Golang. Rust is also quite popular for its speed as it is also low-level. Java is popular for enterprise applications. JavaScript/TypeScript is suitable for front-end development. So if you are willing to grow outside of infrastructure and network management, select the area and we advise on languages then.

The truth is, each new language is easier to learn than the previews one. Therefore, enroll today to our network automation programs to get started:

We offer the following training programs in network automation for you:

• Zero-to-Hero Network Automation Training
• High-scale automation with Nornir
• Ansible Automation Orchestration with Ansble Tower / AWX
• Expert-level training: Closed-loop Automation and Next-generation Monitoring

During these trainings you will learn the following topics:

• Success and failure strategies to build the automation tools.
• Principles of software developments and the most useful and convenient tools.
• Data encoding (free-text, XML, JSON, YAML, Protobuf).
• Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
• Full configuration templating with Jinja2 based on the source of truth (NetBox).
• Best programming languages (Python, Bash) for developing automation
• The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
• Network automation infrastructure (Linux, Linux networking, KVM, Docker).
• Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
• Collection network data via SNMP and streaming telemetry with Prometheus
• Building API gateways with Python leveraging Fast API
• Integration of alerting with Slack and your own APIs
• … and many more

Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data center networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.

Start your automation training today.

What Are We Going To Talk Today?

In modern business world, speed is critical to success. Implementation of desired changes to IT and network infrastructure must be quick, correct, and controlled. We talked about how to make interaction controlled and correct. Today we talk about speed of application.

Speed can be achieved on different levels, using different techniques. Today we focus specifically on concurrency and parallelism:

1. What are the different approaches exist to execute multiple parts of code simultaneously?
2. What is available in Python? What is available in Golang?
3. How to develop software leveraging concurrency?

Explanation

Why Concurrency Is Needed?

One of the distinct aspect of network and infrastructure management is scale. It is not uncommon to manage simultaneously hundreds and thousands of devices. At such scale, the software architecture and selected patterns start playing a critical role. Think about a simple scenario: duration of interaction with network device to perform the required pre-/post- check as well as relevant configuration is 30 seconds. And you need to configure 1000 network devices. In the most straightforward case, you will interact with devices one by one, which results in 30000 seconds, or about 8 hours 20 minutes. Way too long.

The idea of concurrency is to execute some parts of code simultaneously or near simultaneously, which is heavily dependent on the nature of your tasks.

Join our Zero-to-Hero Network Automation Training to dive into details of concurrency.

Concurrency 101

To understand how concurrency is implemented in Python and in Golang (Go), you first need to understand how the computer executes your code:

1. Your computer has some amount of CPUs and cores per CPU. Could be anything from 1x core CPU (probably not anymore, but it was the case 25 years ago) to 4x/12x core modern laptops to 2x/4x CPU servers with 24x cores each.
2. When you start your application, written in any programming language, including those written in Golang/Python, which we created in previous blogs, it is started on one core of one CPU.
3. Application can be actively doing things (e.g., computing anything with local data) or being idle (e.g., waiting on response from remote servers).

Based on the explanation above, we come to two types of concurrency existing in software development:

• Concurrency for IO-bound tasks, that is a collection of techniques to utilize efficiently application idle time, whilst waiting the response
• Concurrency for CPU-bound tasks, that is a collection of techniques to leverage other available CPU cores (if any) for spreading computational tasks, when application isn’t idle.

From the network and IT infrastructure management we predominantly, but not always, deal with IO-bound tasks, whilst in data science including AI/ML, we typically deal with CPU-bound tasks

Concurrency in Python

For IO-bound concurrency, Python has two techniques:

• Multi-threading, which is an approach to spin up light threads on the same CPU core, where the Python process is started. It is relatively simple to implement and doesn’t require any specific support from libraries.
• Asyncio, which is the said to be the most scalable approach to IO-bound tasks as it asynchronously execute tasks on the same CPU core, where the Python process is started. More complicated to implement, but typically is more performant than threading. Also it requires Python libraries to implement async functionality, which is not the case for many Python packages.

Both of these approaches are used in IO-bound tasks as they use only one single core of one CPU.

For CPU-bound concurrency, Python implements:

• Multi-processing, which is an approach to spin up multiple Python processes on different CPU cores. It is as complicated as multi-threading, but has extra overhead on memory utilization as memory is to be copied from a parent to all child process, effectively multiplying its usage.

In theory it shall be possible to combine multi-threading and multi-processing, but I haven’t used it as it makes code quite complicated and, therefore, difficult to maintain.

Typically, you select between asyncio and multi-threading for infrastructure management, and the result will depend on many parameters. As such, testing is King.

Concurrency in Golang (Go)

Golang was developed later than Python (almost 18 year later) in the era of multi-core CPUs; therefore, it was created by speed. The approach to concurrency in Golang is called goroutines, which is similar to threads in Python with a major difference: Golang distributes goroutines across all the available cores, potentially placing multiple goroutines on a single core, which allows to achieve much better performance as all available CPU resources are utilized. In my experience, such approach on the one hand drives your overall CPU utilization across all cores up to 100%, which could lead to impact of other applications. On the other hand, you get 3x-30x, or even more, gain in performance, which allows you to complete your tasks much faster.

Example

In this scenario we will extend the code from our previous blog, which included GNMI/YANG and REST API towards NetBox.

In that blog, our example application, which is created both in Python and Go (Golang), fetches inventory from NetBox via REST API, and then connect to devices one by one using for-loop.

What we are going to do today is to modify the logic of that loop so that it can concurrently interact with multiple network devices, what will significantly improve application performance with any amount of network devices more than one.

Python

In Python we will use multi-threading approach. Library, which is needed for this is called “concurrent” and it is a part of standard package distribution in Python, what means you don’t need to install. As such, the only dependencies you need are the same as in the previous blog post:

1
$ pip install pygnmi httpx

Here is the code of the network automation application written in Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
"""From Python to Go: Python: 020 - Concurency."""



# Modules

import datetime

import os

import sys

from dataclasses import dataclass

from typing import List, Tuple

import difflib

from concurrent.futures import ThreadPoolExecutor

import pygnmi.client

import httpx





# Classes

@dataclass

class Credentials:

    """Class to store credentials."""

    username: str

    password: str





@dataclass

class InventoryCredentials:

    """Class to store credentials."""

    url: str

    token: str





@dataclass

class Instruction:

    """Class to store instructions."""

    command: List[str]

    config: List[tuple]





@dataclass

class Result:

    """Class to store command execution data."""

    instruction: Instruction

    diff: str

    timestamp: datetime.datetime





class Device:

    """Class to interact with netowrk device."""

    def __init__(self, hostname: str, ip_address: str, port: int, platform: str, credentials: Credentials):

        self.hostname = hostname

        self.ip_address = ip_address

        self.port = port

        self.platform = platform

        self.credentials = credentials



        self.results: List[Result] = []



    def execute_change(self, instruction: Instruction) -> None:

        """Method to execute change."""



        # Connect to device

        with pygnmi.client.gNMIclient(

            target=(self.ip_address, self.port),

            username=self.credentials.username,

            password=self.credentials.password,

            skip_verify=True,

            timeout=5,

        ) as gconn:

            # Get state before change

            before = gconn.get(path=instruction.command, datatype="config")

            before_stringified = self.dict_to_xpath(before)



            # Apply change

            config_result = gconn.set(update=instruction.config, encoding="json_ietf")

            print(f"{config_result=}")



            # Get state after change

            after = gconn.get(path=instruction.command, datatype="config")

            after_stringified = self.dict_to_xpath(after)



            # Diff

            diff = "\n".join(

                difflib.context_diff(

                    before_stringified,

                    after_stringified,

                    lineterm="",

                )

            )



            self.results.append(

                Result(

                    instruction=instruction,

                    diff=diff,

                    timestamp=datetime.datetime.now(),

                )

            )



    def dict_to_xpath(self, data: dict) -> list:

        """Method to convert dict to xpath."""

        result = []



        if isinstance(data, str):

            return data



        for key, value in data.items():

            if isinstance(value, list):

                for ind, item in enumerate(value):

                    tr = self.dict_to_xpath(item)

                    result.extend([f"{key}/{ind}/{_}" for _ in tr])



            elif isinstance(value, dict):

                tr = self.dict_to_xpath(value)

                result.extend([f"{key}/{_}" for _ in tr])



            else:

                result.append(f"{key} = {value}")



        return result





# Functions

def load_inventory(inventory: InventoryCredentials, credentials: Credentials) -> List[Device]:

    """Function to load inventory data."""

    # Create HTTP client and set headers

    hclient = httpx.Client(

        base_url=inventory.url.rstrip("/"),

        headers={"Authorization": f"Token {inventory.token}"},

    )

    # Retrieve data from REST API

    try:

        response = hclient.get(

            "/api/dcim/devices/",

            params={

                "site": "ka-blog",

            }

        )

        response.raise_for_status()

        data = response.json()



    except Exception as e:

        print(e)

        sys.exit(1)



    # Populate list of devices

    result = []

    for device in data["results"]:

        result.append(

            Device(

                hostname=device["name"],

                ip_address=device["primary_ip"]["address"].split("/")[0],

                port=device["custom_fields"].get("gnmi_port", 50051),

                platform=device["platform"]["slug"],

                credentials=credentials,

            )

        )



    return result





def get_credentials() -> Tuple[Credentials, InventoryCredentials]:

    """Function to get credentials."""

    return (

        Credentials(

            os.getenv("AUTOMATION_USER"),

            os.getenv("AUTOMATION_PASS"),

        ),

        InventoryCredentials(

            os.getenv("AUTOMATION_INVENTORY_URL"),

            os.getenv("AUTOMATION_INVENTORY_TOKEN"),

        ),

    )





# Main code

if __name__ == "__main__":

    # Get credentials

    credentials, inventory_credentials = get_credentials()



    # Load inventory

    devices = load_inventory(inventory_credentials, credentials=credentials)



    # Config

    instruction = Instruction(

        command=["/interfaces"],

        config=[

            (

                "/interfaces",

                {

                    "interface": [

                        {

                            "name": "Loopback 23",

                            "config": {

                                "name": "Loopback 23",

                                "description": "Test-gnmi-python-24",

                            }

                        },

                    ],

                },

            ),

        ],

    )



    # Execute command

    with ThreadPoolExecutor(max_workers=10) as executor:

        execution_results = executor.map(

            lambda device: device.execute_change(instruction),

            devices,

        )



    # Print results

    for device in devices:

        print(f"Device: {device.hostname}")

        for result in device.results:

            print(f"Config: {result.instruction.config}", f"Impact: {result.diff}", f"Timestamp: {result.timestamp}", sep="\n")

We suggest to read previous blogs to get better understanding of this code, as we omit many details in this post.

Here is what is changed:

1. Class ThreadedPoolExecutor is imported from concurrent.future library. This class is used instantiate an object executor, which has capabilities spin up threads for tasks.
2. Using method map() of the object executor, which has two arguments (can have more than two):
   • Name of function (so called, callable), which is to be executed in a threaded way
   • Arguments to this callable, which shall be an iterator (list, dictionary, etc). There could be more than one argument that you need; hence, it is possible to provide multiple iterators.
3. Ultimately, we call in the threaded fashion method execute_change() of the device object, with list devices being the iterator for creating threads.
4. The results of execution are stored in the object execution_results, which is iterator itself. We don’t use them however in this scenario, as we store results within the object device itself.
5. We loop through results in the same way, as in the previous blog.

As you can see, adding multi-threading to Python is very straightforward. The syntax is different to for-loops, but it follows it in spirit, as you have what to do (callable) and iterator against which elements the callable it is to be executed.

Let’s see the execution of this Python application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
$ python main.py

ssl_target_name_override is applied, should be used for testing only!

ssl_target_name_override is applied, should be used for testing only!

config_result={'timestamp': 1745696412856245684, 'prefix': None, 'response': [{'path': 'interfaces', 'op': 'UPDATE'}]}

config_result={'timestamp': 1745696415292591043, 'prefix': None, 'response': [{'path': 'interfaces', 'op': 'UPDATE'}]}

Device: ka-blog-001

Config: [('/interfaces', {'interface': [{'name': 'Loopback 23', 'config': {'name': 'Loopback 23', 'description': 'Test-gnmi-python-23'}}]})]

Impact: ***

---

***************

*** 1,4 ****

! notification/0/timestamp = 1745696415279571842

  notification/0/prefix = None

  notification/0/alias = None

  notification/0/atomic = False

--- 1,4 ----

! notification/0/timestamp = 1745696415677702589

  notification/0/prefix = None

  notification/0/alias = None

  notification/0/atomic = False

***************

*** 48,58 ****

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/openconfig-if-ip:ipv4/addresses/address/0/config/prefix-length = 31

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/openconfig-if-ip:ipv4/addresses/address/0/ip = 10.0.0.1

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/openconfig-if-ip:ipv4/config/enabled = True

! notification/0/update/0/val/openconfig-interfaces:interface/3/config/description = Test-gnmi-python-2

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/loopback-mode = FACILITY

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/name = Loopback23

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/type = iana-if-type:softwareLoopback

  notification/0/update/0/val/openconfig-interfaces:interface/3/name = Loopback23

! notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/config/description = Test-gnmi-python-2

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/index = 0

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/openconfig-if-ip:ipv4/config/enabled = True

--- 48,58 ----

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/openconfig-if-ip:ipv4/addresses/address/0/config/prefix-length = 31

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/openconfig-if-ip:ipv4/addresses/address/0/ip = 10.0.0.1

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/openconfig-if-ip:ipv4/config/enabled = True

! notification/0/update/0/val/openconfig-interfaces:interface/3/config/description = Test-gnmi-python-23

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/loopback-mode = FACILITY

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/name = Loopback23

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/type = iana-if-type:softwareLoopback

  notification/0/update/0/val/openconfig-interfaces:interface/3/name = Loopback23

! notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/config/description = Test-gnmi-python-23

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/index = 0

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/openconfig-if-ip:ipv4/config/enabled = True

Timestamp: 2025-04-26 20:40:17.240665

Device: ka-blog-002

Config: [('/interfaces', {'interface': [{'name': 'Loopback 23', 'config': {'name': 'Loopback 23', 'description': 'Test-gnmi-python-23'}}]})]

Impact: ***

---

***************

*** 818,824 ****

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/openconfig-if-ip:ipv6/state/mtu = 1500

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/state/index = 0

! notification/0/update/0/val/openconfig-interfaces:interface/5/config/description = Test-gnmi-python-2

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/arista-intf-augments:load-interval = 300

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/loopback-mode = True

--- 818,824 ----

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/openconfig-if-ip:ipv6/state/mtu = 1500

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/state/index = 0

! notification/0/update/0/val/openconfig-interfaces:interface/5/config/description = Test-gnmi-python-23

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/arista-intf-augments:load-interval = 300

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/loopback-mode = True

***************

*** 833,839 ****

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/name = Loopback23

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/openconfig-vlan:tpid = openconfig-vlan-types:TPID_0X8100

! notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/description = Test-gnmi-python-2

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/index = 0

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/index = 0

--- 833,839 ----

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/name = Loopback23

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/openconfig-vlan:tpid = openconfig-vlan-types:TPID_0X8100

! notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/description = Test-gnmi-python-23

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/index = 0

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/index = 0

Timestamp: 2025-04-26 20:40:17.138377

Go (Golang)

In Golang there is no need to use any extra packages for concurrency, it is implemented via built-in functions. As such, like in Python, all we need is to install the same dependencies for gnmi and structs comparison as in the previous blog:

1
2
3
$ go get github.com/google/go-cmp/cmp

$ go get github.com/openconfig/gnmic/api

$ go get google.golang.org/protobuf/encoding/prototext

The following software written in Go (Golang) adds concurrency to interacting with network devices via GNMI/YANG (suitable for any protocol):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
/* From Python to Go: Go: 020 - Concurency. */



package main



// Imports

import (

    "context"

    "encoding/json"

    "fmt"

    "io"

    "log"

    "net/http"

    "os"

    "strings"

    "time"



    "github.com/google/go-cmp/cmp"

    "github.com/openconfig/gnmic/pkg/api"

    "google.golang.org/protobuf/encoding/prototext"

)



// Types and Receivers

type Arguments struct {

    /* Class to starte CLI arguments */

    Inventory string

}



type Crendetials struct {

    /* Struct to store credentials. */

    Username string

    Password string

}



type InventoryCredentials struct {

    /* Struct to store inventory credentails */

    Url   string

    Token string

}



type Instruction struct {

    Command string

    Config  struct {

        Path  string

        Value any

    }

}



type Result struct {

    /* Struct to store command execution result. */

    Instruction Instruction

    Diff        string

    Timestamp   time.Time

}



type Device struct {

    /* Struct to interact with netowrk device. */

    Hostname    string `yaml:"hostname"`

    IpAddress   string `yaml:"ip_address"`

    Port        uint   `yaml:"port"`

    Platform    string `yaml:"platform"`

    Crendetials Crendetials

    Result      []Result

}



func (d *Device) executeChange(i Instruction) {

    /* Method to execute command */



    // Create GNMI Target

    gnmiTarget, err := api.NewTarget(

        api.Name(d.Hostname),

        api.Address(fmt.Sprintf("%s:%d", d.IpAddress, d.Port)),

        api.Username(d.Crendetials.Username),

        api.Password(d.Crendetials.Password),

        api.SkipVerify(true),

    )

    if err != nil {

        log.Fatal("Cannot create GNMI Target: ", err)

    }



    // Create context

    ctx, cancel := context.WithCancel(context.Background())

    defer cancel()



    // Create GNMI client

    err = gnmiTarget.CreateGNMIClient(ctx)

    if err != nil {

        log.Fatal("Cannot create GNMI Client: ", err)

    }

    defer gnmiTarget.Close()



    // Get state before change

    getReq, err := api.NewGetRequest(

        api.Path(i.Command),

        api.DataType("config"),

        api.Encoding("json_ietf"),

    )

    if err != nil {

        log.Fatal("Cannot create Get request: ", err)

    }

    beforeGetResponse, err := gnmiTarget.Get(ctx, getReq)

    if err != nil {

        log.Fatal("Cannot make a Get request: ", err)

    }

    beforeStruct := OpenConfigInterfaces{}

    err = json.Unmarshal(beforeGetResponse.Notification[0].Update[0].Val.GetJsonIetfVal(), &beforeStruct)

    if err != nil {

        log.Fatal("Cannot unmarshall JSON: ", err)

    }



    // Make change

    setReq, err := api.NewSetRequest(

        api.Update(

            api.Path(i.Config.Path),

            api.Value(i.Config.Value, "json_ietf"),

        ),

    )

    if err != nil {

        log.Fatal("Cannot create Set request: ", err)

    }

    setResp, err := gnmiTarget.Set(ctx, setReq)

    if err != nil {

        log.Fatal("Cannot make a Set request: ", err)

    }

    log.Println(prototext.Format(setResp))



    // Get state after change

    afterGetResponse, err := gnmiTarget.Get(ctx, getReq)

    if err != nil {

        log.Fatal("Cannot make a Get request: ", err)

    }

    afterStruct := OpenConfigInterfaces{}

    err = json.Unmarshal(afterGetResponse.Notification[0].Update[0].Val.GetJsonIetfVal(), &afterStruct)

    if err != nil {

        log.Fatal("Cannot unmarshall JSON: ", err)

    }

    // Diff

    diff := cmp.Diff(beforeStruct, afterStruct)



    // Update the result

    (*d).Result = append((*d).Result, Result{

        Instruction: i,

        Diff:        diff,

        Timestamp:   time.Now(),

    })

}



// Functions

func loadInventory(iC InventoryCredentials) *[]Device {

    /* Function to load inventory data. */



    // Create HTTP client

    hclient := &http.Client{}



    // Prepare request

    NetboxRequest, err := http.NewRequest("GET", iC.Url+"/api/dcim/devices/", nil)

    if err != nil {

        fmt.Println("Error during preparing HTTP Request ", err)

        os.Exit(1)

    }



    // Set headers

    NetboxRequest.Header.Add("Authorization", fmt.Sprintf("Token %s", iC.Token))



    // Set URL params

    q := NetboxRequest.URL.Query()

    q.Add("site", "ka-blog")

    NetboxRequest.URL.RawQuery = q.Encode()



    // Get data

    resp, err := hclient.Do(NetboxRequest)

    if err != nil {

        fmt.Println("Erorr during executing HTTP query ", err)

        os.Exit(1)

    }

    defer resp.Body.Close()



    body, err := io.ReadAll(resp.Body)

    if err != nil {

        fmt.Println("Error during reading body of HTTP response ", err)

        os.Exit(1)

    }



    td := NetboxDcimDevices{}

    err = json.Unmarshal(body, &td)

    if err != nil {

        fmt.Println("Error during parsing JSON ", err)

        os.Exit(1)

    }



    // Load inventory

    result := &[]Device{}



    for _, v := range td.Results {

        *result = append(*result, Device{

            Hostname:  v.Name,

            Platform:  v.Platform.Slug,

            IpAddress: strings.Split(v.PrimaryIp4.Address, "/")[0],

            Port:      uint(v.CustomFields.GnmiPort),

        })

    }



    // Return result

    return result

}



func getCredentials() (Crendetials, InventoryCredentials) {

    /* Function to get credentials. */

    return Crendetials{

            Username: os.Getenv("AUTOMATION_USER"),

            Password: os.Getenv("AUTOMATION_PASS"),

        },

        InventoryCredentials{

            Url:   os.Getenv("AUTOMATION_INVENTORY_URL"),

            Token: os.Getenv("AUTOMATION_INVENTORY_TOKEN"),

        }

}



// Main

func main() {

    /* Core logic */

    // Get credentials

    sshCreds, invCreds := getCredentials()



    // Load inventory

    inventory := loadInventory(invCreds)



    // Config

    instruction := Instruction{

        Command: "/interfaces",

        Config: struct {

            Path  string

            Value any

        }{

            Path: "/interfaces",

            Value: map[string]any{

                "interface": []map[string]any{

                    {

                        "name": "Loopback 23",

                        "config": map[string]any{

                            "name":        "Loopback 23",

                            "description": "Test-gnmi-golang-23",

                        },

                    },

                },

            },

        },

    }



    // Create communication channel

    c := make(chan Device)



    // Execute commands

    for i := 0; i < len(*inventory); i++ {

        // Set credentals

        (*inventory)[i].Crendetials = sshCreds



        // Launch goroutines

        go func(d Device, ins Instruction, c chan<- Device) {

            // Execute task

            d.executeChange(ins)



            // Send device back

            c <- d

        }((*inventory)[i], instruction, c)

    }



    // Collect results

    iventory_with_results := make([]Device, 0)

    for i := 0; i < len(*inventory); i++ {

        iventory_with_results = append(iventory_with_results, <-c)

    }



    // Print results

    for i := 0; i < len(iventory_with_results); i++ {

        for j := 0; j < len((iventory_with_results)[i].Result); j++ {

            fmt.Printf(

                "Config: %v\nImpact: %v\nTimestamp: %v\n",

                (iventory_with_results)[i].Result[j].Instruction.Config,

                (iventory_with_results)[i].Result[j].Diff,

                (iventory_with_results)[i].Result[j].Timestamp,

            )

        }

    }

}

We suggest to read previous blogs to get better understanding of this code, as we omit many details in this post.

Some Details On Goroutine

Implementation of concurrency in Golang is more complicated than in Python, which requires two components:

• Goroutine, which is an instruction to start task as a separate goroutine (read thread). The syntax is as follows:

1
2
3
go func (a int){

  a++

}(10)

• The keyword “go” in front of function name, or in this case anonymous function, same as lambda-funciton in Python instructs for function start as goroutine.
• Second component is called channel, and this is pretty much a single mechanism to pass data from one goroutine to another one. Channel is a variable, which you create using funciton make(), same as you do for slices and maps. the content of the channel is the data type you want to transfer from one goroutine to another prepended by the instruction “chan“.
• Here is how full example with goroutine and channel looking like:

1
2
3
4
5
6
7
8
c := make (chan int)



go func (a int, c chan int){

  a++

  c <- a

}(10)



newData := <- c

Two important notes: even if your gouroutine is a method (receiver function) of pointer, you HAVE to communicate it back; original pointer won’t change (1). If you are using gorotine to modify shared variable, you need mutex (2)

Back to explanation

Based on the quick explanation above, let’s see what’s changed in the code:

• The channel c is created with Device data type:

1
2
    // Create communication channel

    c := make(chan Device)

• Within the for loop, the goroutine is started to interact with every device. The inputs to goroutine are:
  • Device, which the interaction is done against
  • Instruction, which interaction shall it be
  • channel Device, to communicte back the changed state with results

1
2
3
4
5
6
7
8
9
10
11
12
13
14
    // Execute commands

    for i := 0; i < len(*inventory); i++ {

        // Set credentals

        (*inventory)[i].Crendetials = sshCreds



        // Launch goroutines

        go func(d Device, ins Instruction, c chan<- Device) {

            // Execute task

            d.executeChange(ins)



            // Send device back

            c <- d

        }((*inventory)[i], instruction, c)

    }

• New slice inventory_with_results are created by created as an empty slice, which is populated with date from c channel:

1
2
3
4
5
    // Collect results

    iventory_with_results := make([]Device, 0)

    for i := 0; i < len(*inventory); i++ {

        iventory_with_results = append(iventory_with_results, <-c)

    }

• The rest of the code is identical

Such a code allows you to start interactions with all the devices in your inventory simultaneousely and then to process results as they are ready. For example, if you have a few devices, which are slow to reponse, and a lot of others, which are quick, the duration of this code execution will be defined by the slowest device alone.

Let’s execute this Go (Golang) application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
$ go run .

2025/04/26 20:56:36 response: {

  path: {

    elem: {

      name: "interfaces"

    }

  }

  op: UPDATE

}

timestamp: 1745697392374689242



2025/04/26 20:56:36 response: {

  path: {

    elem: {

      name: "interfaces"

    }

  }

  op: UPDATE

}

timestamp: 1745697394881050173



Config: {/interfaces map[interface:[map[config:map[description:Test-gnmi-golang-23 name:Loopback 23] name:Loopback 23]]]}

Impact:   main.OpenConfigInterfaces{

        Interface: []main.OpenConfigInterface{

                ... // 3 identical elements

                {Name: "Loopback51", Config: {Name: "Loopback51", Description: "pytest-update-test-33"}},

                {Name: "Loopback0", Config: {Name: "Loopback0", Enabled: true}},

                {

                        Name: "Loopback23",

                        Config: struct{ Name string "xml:"name,omitempty" json:"name,omitempty""; Description string "xml:"description,omitempty" json:"description,omitempty""; Enabled bool "xml:"enabled,omitempty" json:"enabled,omitempty"" }{

                                Name:        "Loopback23",

-                               Description: "Test-gnmi-golang-3",

+                               Description: "Test-gnmi-golang-23",

                                Enabled:     true,

                        },

                },

        },

  }



Timestamp: 2025-04-26 20:56:36.703237967 +0100 BST m=+1.125671022

Config: {/interfaces map[interface:[map[config:map[description:Test-gnmi-golang-23 name:Loopback 23] name:Loopback 23]]]}

Impact:   main.OpenConfigInterfaces{

        Interface: []main.OpenConfigInterface{

                {Name: "Management1", Config: {Name: "Management1"}},

                {Name: "Ethernet2", Config: {Name: "Ethernet2"}},

                {Name: "Ethernet1", Config: {Name: "Ethernet1"}},

                {

                        Name: "Loopback23",

                        Config: struct{ Name string "xml:"name,omitempty" json:"name,omitempty""; Description string "xml:"description,omitempty" json:"description,omitempty""; Enabled bool "xml:"enabled,omitempty" json:"enabled,omitempty"" }{

                                Name:        "Loopback23",

-                               Description: "Test-gnmi-golang-3",

+                               Description: "Test-gnmi-golang-23",

                                Enabled:     false,

                        },

                },

        },

  }



Timestamp: 2025-04-26 20:56:37.002338539 +0100 BST m=+1.424771596

Lessons in GitHub

You can find the final working versions of the files from this blog at out GitHub page.

Conclusion

This blog post concludes our introduction to Golang for network and IT infrastructure management with Python serving as a scaffolding. Within the course of 21 blog posts, we’ve covered everything you need to start developing and using software for IT and network infrastructure management with both Golang and Python. The range of topics spans basics, such as data types, working with variables and code flow control to object-oriented programming and parsing/serializing XML/JSON/YAML to handling exceptions and user input to existing protocols for interacting with servers and network devices (SSH, NETCONF/YANG, GNMI/YANG) to templating and concurrency.

We may create second series in future, where we will cover more advanced protocols and external components, such as various databases, message brokers and advanced APIs. If you want to make this happen, please, star our GitHub repo and/or repost our posts so that we see it is needed for you and for the community.

Thank you so much for reading us. Take care and good bye.

Support us

Support new interop and automation articles at karneliuk.com
EUR

I want to support with:
9.99 EUR4.99 EUR24.99 EUR49.99 EUR99.99 EUR199.99 EUR

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk 

So far we’ve covered all means to interact with network devices, which are meaningful in our opinion: SSH, NETCONF/YANG, and GNMI/YANG. There is one more protocol, which exists for managing network devices, which is called RESTCONF, which is application of REST API to network devices. From our experience, its support across network vendors is very limited; therefore, we don’t cover it. However, REST API itself is immensely important, as it is still the most widely used protocol for applications to talk to each other. And this is the focus for today’s blog.

I See Everywhere Stop Learning Code, Why Do You Teach It?

Generative AI, Agentic AI, all other kinds of AI is absolutely useful things. The advancements there are very quick and we ourselves using them in our projects. At the same time, if you don’t know how to code, how to solve algorithmic tasks, how can you reason if the solution provided by AI is correct? If that optimal? And moreover, when it breaks, because every software breaks sooner or later, how can you fix it? That’s why we believe it is absolutely important to learn software development, tools and algorithms. Perhaps, more than ever as various software are used more and more around the world

And our trainings are designed to teach you foundation of software development of network and IT infrastructure management. Enroll today:

We offer the following training programs in network automation for you:

• Zero-to-Hero Network Automation Training
• High-scale automation with Nornir
• Ansible Automation Orchestration with Ansble Tower / AWX
• Expert-level training: Closed-loop Automation and Next-generation Monitoring

During these trainings you will learn the following topics:

• Success and failure strategies to build the automation tools.
• Principles of software developments and the most useful and convenient tools.
• Data encoding (free-text, XML, JSON, YAML, Protobuf).
• Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
• Full configuration templating with Jinja2 based on the source of truth (NetBox).
• Best programming languages (Python, Bash) for developing automation
• The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
• Network automation infrastructure (Linux, Linux networking, KVM, Docker).
• Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
• Collection network data via SNMP and streaming telemetry with Prometheus
• Building API gateways with Python leveraging Fast API
• Integration of alerting with Slack and your own APIs
• … and many more

Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data center networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.

Start your automation training today.

What Are We Going To Talk Today?

REST API is de-facto standard for application talking to each other since beginning of 2010s, when it was introduced and started replacing SOAP/XML. A lot of existing applications nowadays have REST API to allow users programmatically to interact with them, including tools in IT and network infrastructure management. That’s why we are talking about the REST API today, focusing on:

1. How REST API operates?
2. Which popular tools in IT management has REST API?
3. How to interact with external applications via REST API in your applications?

Explanation

REST API is the most popular way for application to talk to each other. It walked long a way from its first inception back in 2000.

We covered REST API in depth earlier in our blog: Part 1, Part 2, and Part 3.

Let’s do a quick refresher, on how REST API works.

First of all, REST is a short of REpresentational State Transfer API. In real world it is based on HTTP transport, and utilizes various HTTP methods to perform actions tailored to data lifecycle (CRUD):

Authentication in REST API is done using HTTP headers; in fact, HTTP headers for much more: they are used to signal various metadata, such as provided/expected content types, encoding, etc.

Typically REST API uses JSON as data serialization, although I saw some applications using XML. Usage of JSON, which is easily human-readable, and clean stateless API structure made REST API easy to implement and, therefore, very popular nowadays.

It is not ideal, though. There are a number of pitfalls, which some other API techniques tried to address:

• Retrieving joint data. In a nutshell, REST API, and any other API, including the ones we reviewed previously in our blog, are developed to interact with data, managing its full lifecycle. If you are familiar with SQL, which we may cover in some future blogs, it allows you to join data from multiple tables in a single response. You cannot do it using REST API, as typically each API endpoint represents a dedicated table, and you need to do multiple GET’s from different REST API endpoints and to perform join on the client side. GraphQL addresses this issue; however, it comes on costs of the excessive resources utilization on the server side, as clients now can create complicated nested join requests, which will take compute/memory.
• Strict schema adherence. One of the issues in any application to application communication is a stable contract, that is defined by API schema. It shall be clear, which fields are permitted, which are mandatory, which data types shall they have, etc. Some advancements were done in REST API world via development of self-documenting frameworks, e.g. swagger-ui, which generates the documentation of APIs. However, when you create an application, which talks to specific API, it is easy to do typo in the key name or any other mistake. Another issue is how to deal with scenarios, where the key isn’t provided. GRPC/Protobuf addresses this issue via implementation of strict contracts in form or Proto-files, which are used both in server and client applications. That though comes on cost of complexity of code and till the recent times in the absence of tool for manual experimentation. The latter however was partially solved by introduction of grpcurl.

Despite these drawbacks, REST API is still very much used. I saw an interesting insight in LinkedIn, that some companies even going back from GRPC to REST API for some services.

Practical Applications

NetBox

One of the best tools in the world in my opinion for network and IT network infrastructure management is NetBox.

If you want to master NetBox, join our Zero-to-Hero Network Automation Training.

It has a great a UI and REST API allowing you to manage all data points about the IT/network infrastructure.

We have quite a few blogs about NetBox, which we encourage you to read if you aren’t familiar with NetBox.

Other Applications

Service Now, JIRA, Infoblox, Slack, Microsoft Team, Ansible Tower (Ansible Automation Controller), even Google Sheets – all these applications have REST API, and this is just a tip of the iceberg. Therefore, if you are in software development, you absolutely should know how to programmatically interact with REST API and how to create REST API services yourself.

Example

We’ll continue our logic of build on top of what we’ve build so far in previous blogs. To show you how to use REST API, we will:

1. Replace local static inventory YAML file with integration to public demo NetBox website.
2. Use this inventory to connect to network device using GNMI.

Python

To interact with REST API services, we need libraries, which are able to perform HTTP requests. There are some libraries, such as urllib, which are included in standard Python distribution. However, they are outdated. The most advanced, which is also gaining more popularity, is httpx. You need to install, in addition to pygnmi, which is used to interact with network devices.

1
$ pip install pygnmi httpx

And now the code of application in Python, which gets via REST API from NetBox inventory and then connects to those devices using GNMI:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
"""From Python to Go: Python: 017 - NETCONF."""



# Modules

import argparse

import datetime

import os

import sys

from dataclasses import dataclass

from typing import List, Tuple

import difflib

import pygnmi.client

import yaml

import httpx





# Classes

@dataclass

class Credentials:

    """Class to store credentials."""

    username: str

    password: str





@dataclass

class InventoryCredentials:

    """Class to store credentials."""

    url: str

    token: str





@dataclass

class Instruction:

    """Class to store instructions."""

    command: List[str]

    config: List[tuple]





@dataclass

class Result:

    """Class to store command execution data."""

    instruction: Instruction

    diff: str

    timestamp: datetime.datetime





class Device:

    """Class to interact with netowrk device."""

    def __init__(self, hostname: str, ip_address: str, port: int, platform: str, credentials: Credentials):

        self.hostname = hostname

        self.ip_address = ip_address

        self.port = port

        self.platform = platform

        self.credentials = credentials



        self.results: List[Result] = []



    def execute_change(self, instruction: Instruction) -> None:

        """Method to execute change."""



        # Connect to device

        with pygnmi.client.gNMIclient(

            target=(self.ip_address, self.port),

            username=self.credentials.username,

            password=self.credentials.password,

            skip_verify=True,

            timeout=5,

        ) as gconn:

            # Get state before change

            before = gconn.get(path=instruction.command, datatype="config")

            before_stringified = self.dict_to_xpath(before)



            # Apply change

            config_result = gconn.set(update=instruction.config, encoding="json_ietf")

            print(f"{config_result=}")



            # Get state after change

            after = gconn.get(path=instruction.command, datatype="config")

            after_stringified = self.dict_to_xpath(after)



            # Diff

            diff = "\n".join(

                difflib.context_diff(

                    before_stringified,

                    after_stringified,

                    lineterm="",

                )

            )



            self.results.append(

                Result(

                    instruction=instruction,

                    diff=diff,

                    timestamp=datetime.datetime.now(),

                )

            )



    def dict_to_xpath(self, data: dict) -> list:

        """Method to convert dict to xpath."""

        result = []



        if isinstance(data, str):

            return data



        for key, value in data.items():

            if isinstance(value, list):

                for ind, item in enumerate(value):

                    tr = self.dict_to_xpath(item)

                    result.extend([f"{key}/{ind}/{_}" for _ in tr])



            elif isinstance(value, dict):

                tr = self.dict_to_xpath(value)

                result.extend([f"{key}/{_}" for _ in tr])



            else:

                result.append(f"{key} = {value}")



        return result





# Functions

def load_inventory(inventory: InventoryCredentials, credentials: Credentials) -> List[Device]:

    """Function to load inventory data."""

    # Create HTTP client and set headers

    hclient = httpx.Client(

        base_url=inventory.url.rstrip("/"),

        headers={"Authorization": f"Token {inventory.token}"},

    )

    # Retrieve data from REST API

    try:

        response = hclient.get(

            "/api/dcim/devices/",

            params={

                "site": "kblog",

            }

        )

        response.raise_for_status()

        data = response.json()



    except Exception as e:

        print(e)

        sys.exit(1)



    # Populate list of devices

    result = []

    for device in data["results"]:

        result.append(

            Device(

                hostname=device["name"],

                ip_address=device["primary_ip"]["address"].split("/")[0],

                port=device["custom_fields"].get("gnmi_port", 50051),

                platform=device["platform"]["slug"],

                credentials=credentials,

            )

        )



    return result





def get_credentials() -> Tuple[Credentials, InventoryCredentials]:

    """Function to get credentials."""

    return (

        Credentials(

            os.getenv("AUTOMATION_USER"),

            os.getenv("AUTOMATION_PASS"),

        ),

        InventoryCredentials(

            os.getenv("AUTOMATION_INVENTORY_URL"),

            os.getenv("AUTOMATION_INVENTORY_TOKEN"),

        ),

    )





# Main code

if __name__ == "__main__":

    # Get credentials

    credentials, inventory_credentials = get_credentials()



    # Load inventory

    devices = load_inventory(inventory_credentials, credentials=credentials)



    # Config

    instruction = Instruction(

        command=["/openconfig-interfaces:interfaces"],

        config=[

            (

                "/openconfig-interfaces:interfaces",

                {

                    "interface": [

                        {

                            "name": "Loopback 23",

                            "config": {

                                "name": "Loopback 23",

                                "description": "Test-gnmi-python-2",

                            }

                        },

                    ],

                },

            ),

        ],

    )



    # Execute command

    for device in devices:

        device.execute_change(instruction)



    # Print results

    for device in devices:

        print(f"Device: {device.hostname}")

        for result in device.results:

            print(f"Config: {result.instruction.config}", f"Impact: {result.diff}", f"Timestamp: {result.timestamp}", sep="\n")

Read previous blogs to get better understanding of this code.

What’s changed since the previous version:

1. New data class InventoryCredentials is introduced to store URL and token of NetBox inventory.
2. Function get_credentials() now returns two credentials: in addition to devices’ credentials it returns also inventory for NetBox, which it reads from environment variable.
3. Function load_inventory() underwent the major rework:
   • Using class Client from httpx library the object is instantiated to talk to NetBox. As arguments to this class both the NetBox base URL as well “Authorization” header with Token are passed.
   • Using get() method, inventory data is retrieved from NetBox via REST API. Arguments this method is specific URL, which contains all the devices, and the arguments, which allows to filter specific subset of data.
   • Retrieved data is converted from a string in JSON format to a dictionary using json() method of the received response.
   • Then the result is populated , which uses unaltered Device data class, which we use for all past examples.

That’s all the changes. As you can see, the modularity we created in code, allows us to replace content of one of the functions with other actions so long we maintains the result structure, which is used as an input to other functions.

Let’s execute the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
$ python main.py 

ssl_target_name_override is applied, should be used for testing only!

config_result={'timestamp': 1744970788637880096, 'prefix': None, 'response': [{'path': 'interfaces', 'op': 'UPDATE'}]}

Device: ka-blog-dev-001

Config: [('/openconfig-interfaces:interfaces', {'interface': [{'name': 'Loopback 23', 'config': {'name': 'Loopback 23', 'description': 'Test-gnmi-python-2'}}]})]

Impact: *** 

--- 

***************

*** 687,700 ****

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-fcs-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-multicast-pkts = 0

! notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-octets = 43751

! notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-unicast-pkts = 463

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-broadcast-pkts = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-discards = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-multicast-pkts = 0

! notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-octets = 20010

! notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-unicast-pkts = 144

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/openconfig-platform-port:hardware-port = Port97

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/ifindex = 999001

--- 687,700 ----

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-fcs-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-multicast-pkts = 0

! notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-octets = 47298

! notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/in-unicast-pkts = 493

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-broadcast-pkts = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-discards = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-multicast-pkts = 0

! notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-octets = 50683

! notification/0/update/0/val/openconfig-interfaces:interface/2/state/counters/out-unicast-pkts = 176

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/openconfig-platform-port:hardware-port = Port97

  notification/0/update/0/val/openconfig-interfaces:interface/2/state/ifindex = 999001

***************

*** 818,824 ****

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/openconfig-if-ip:ipv6/state/mtu = 1500

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/state/index = 0

! notification/0/update/0/val/openconfig-interfaces:interface/5/config/description = Go_Test_2

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/arista-intf-augments:load-interval = 300

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/loopback-mode = True

--- 818,824 ----

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/openconfig-if-ip:ipv6/state/mtu = 1500

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/4/subinterfaces/subinterface/0/state/index = 0

! notification/0/update/0/val/openconfig-interfaces:interface/5/config/description = Test-gnmi-python-2

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/arista-intf-augments:load-interval = 300

  notification/0/update/0/val/openconfig-interfaces:interface/5/config/loopback-mode = True

***************

*** 833,839 ****

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/name = Loopback23

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/openconfig-vlan:tpid = openconfig-vlan-types:TPID_0X8100

! notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/description = Go_Test_2

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/index = 0

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/index = 0

--- 833,839 ----

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/name = Loopback23

  notification/0/update/0/val/openconfig-interfaces:interface/5/state/openconfig-vlan:tpid = openconfig-vlan-types:TPID_0X8100

! notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/description = Test-gnmi-python-2

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/config/index = 0

  notification/0/update/0/val/openconfig-interfaces:interface/5/subinterfaces/subinterface/0/index = 0

Timestamp: 2025-04-18 11:06:30.556257

Go (Golang)

In contrast to Python, Golang has a first-class great library for HTTP requests in net/http package. As such, we need only to install libraries for GNMI and to compare objects:

1
2
3
$ go get github.com/google/go-cmp/cmp

$ go get github.com/openconfig/gnmic/api

$ go get google.golang.org/protobuf/encoding/prototext

Next contrast to Python, as Golang is strict typed language (we omit reflection use cases), we need to create struct for NetBox response so that we can use it later:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
/* From Python to Go: Go: 019 - REST API and GNMI: modules */



package main



// Import



// Data types

type OpenConfigInterface struct {

    Name   string `xml:"name,omitempty"`

    Config struct {

        Name        string `xml:"name,omitempty" json:"name,omitempty"`

        Description string `xml:"description,omitempty" json:"description,omitempty"`

        Enabled     bool   `xml:"enabled,omitempty" json:"enabled,omitempty"`

    } `xml:"config" json:"config"`

}

type OpenConfigInterfaces struct {

    Interface []OpenConfigInterface `xml:"openconfig-interfaces:interface,omitempty" json:"openconfig-interfaces:interface,omitempty"`

}



type NetboxDcimDevices struct {

    /* Struct to store data from NetBox */

    Count   uint64 `json:"count"`

    Results []struct {

        Name       string `json:"name"`

        PrimaryIp4 struct {

            Address string `json:"address"`

        } `json:"primary_ip4"`

        Platform struct {

            Slug string `json:"slug"`

        } `json:"platform"`

        CustomFields struct {

            GnmiPort uint64 `json:"gnmi_port"`

        } `json:"custom_fields"`

    } `json:"results"`

}

The new type NetboxDcimDevices of struct is created, which parses a few fields in the response. We don’t parse the entire message but only what is really needed for our inventory use case.

More on JSON parsing in Golang.

And not the code of main application written in Go (Golang):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
/* From Python to Go: Go: 019 - REST API and GNMI. */



package main



// Imports

import (

    "context"

    "encoding/json"

    "fmt"

    "io"

    "log"

    "net/http"

    "os"

    "strings"

    "time"



    "github.com/google/go-cmp/cmp"

    "github.com/openconfig/gnmic/pkg/api"

    "google.golang.org/protobuf/encoding/prototext"

)



// Types and Receivers

type Arguments struct {

    /* Class to starte CLI arguments */

    Inventory string

}



type Crendetials struct {

    /* Struct to store credentials. */

    Username string

    Password string

}



type InventoryCredentials struct {

    /* Struct to store inventory credentails */

    Url   string

    Token string

}



type Instruction struct {

    Command string

    Config  struct {

        Path  string

        Value any

    }

}



type Result struct {

    /* Struct to store command execution result. */

    Instruction Instruction

    Diff        string

    Timestamp   time.Time

}



type Device struct {

    /* Struct to interact with netowrk device. */

    Hostname    string `yaml:"hostname"`

    IpAddress   string `yaml:"ip_address"`

    Port        uint   `yaml:"port"`

    Platform    string `yaml:"platform"`

    Crendetials Crendetials

    Result      []Result

}



func (d *Device) executeChange(i Instruction) {

    /* Method to execute command */



    // Create GNMI Target

    gnmiTarget, err := api.NewTarget(

        api.Name(d.Hostname),

        api.Address(fmt.Sprintf("%s:%d", d.IpAddress, d.Port)),

        api.Username(d.Crendetials.Username),

        api.Password(d.Crendetials.Password),

        api.SkipVerify(true),

    )

    if err != nil {

        log.Fatal("Cannot create GNMI Target: ", err)

    }



    // Create context

    ctx, cancel := context.WithCancel(context.Background())

    defer cancel()



    // Create GNMI client

    err = gnmiTarget.CreateGNMIClient(ctx)

    if err != nil {

        log.Fatal("Cannot create GNMI Client: ", err)

    }

    defer gnmiTarget.Close()



    // Get state before change

    getReq, err := api.NewGetRequest(

        api.Path(i.Command),

        api.DataType("config"),

        api.Encoding("json_ietf"),

    )

    if err != nil {

        log.Fatal("Cannot create Get request: ", err)

    }

    beforeGetResponse, err := gnmiTarget.Get(ctx, getReq)

    if err != nil {

        log.Fatal("Cannot make a Get request: ", err)

    }

    beforeStruct := OpenConfigInterfaces{}

    err = json.Unmarshal(beforeGetResponse.Notification[0].Update[0].Val.GetJsonIetfVal(), &beforeStruct)

    if err != nil {

        log.Fatal("Cannot unmarshall JSON: ", err)

    }



    // Make change

    setReq, err := api.NewSetRequest(

        api.Update(

            api.Path(i.Config.Path),

            api.Value(i.Config.Value, "json_ietf"),

        ),

    )

    if err != nil {

        log.Fatal("Cannot create Set request: ", err)

    }

    setResp, err := gnmiTarget.Set(ctx, setReq)

    if err != nil {

        log.Fatal("Cannot make a Set request: ", err)

    }

    log.Println(prototext.Format(setResp))



    // Get state after change

    afterGetResponse, err := gnmiTarget.Get(ctx, getReq)

    if err != nil {

        log.Fatal("Cannot make a Get request: ", err)

    }

    afterStruct := OpenConfigInterfaces{}

    err = json.Unmarshal(afterGetResponse.Notification[0].Update[0].Val.GetJsonIetfVal(), &afterStruct)

    if err != nil {

        log.Fatal("Cannot unmarshall JSON: ", err)

    }

    // Diff

    diff := cmp.Diff(beforeStruct, afterStruct)



    // Update the result

    (*d).Result = append((*d).Result, Result{

        Instruction: i,

        Diff:        diff,

        Timestamp:   time.Now(),

    })

}



// Functions

func loadInventory(iC InventoryCredentials) *[]Device {

    /* Function to load inventory data. */



    // Create HTTP client

    hclient := &http.Client{}



    // Prepare request

    NetboxRequest, err := http.NewRequest("GET", iC.Url+"/api/dcim/devices/", nil)

    if err != nil {

        fmt.Println("Error during preparing HTTP Request ", err)

        os.Exit(1)

    }



    // Set headers

    NetboxRequest.Header.Add("Authorization", fmt.Sprintf("Token %s", iC.Token))



    // Set URL params

    q := NetboxRequest.URL.Query()

    q.Add("site", "kblog")

    NetboxRequest.URL.RawQuery = q.Encode()



    // Get data

    resp, err := hclient.Do(NetboxRequest)

    if err != nil {

        fmt.Println("Erorr during executing HTTP query ", err)

        os.Exit(1)

    }

    defer resp.Body.Close()



    body, err := io.ReadAll(resp.Body)

    if err != nil {

        fmt.Println("Error during reading body of HTTP response ", err)

        os.Exit(1)

    }



    td := NetboxDcimDevices{}

    err = json.Unmarshal(body, &td)

    if err != nil {

        fmt.Println("Error during parsing JSON ", err)

        os.Exit(1)

    }



    // Load inventory

    result := &[]Device{}



    for _, v := range td.Results {

        *result = append(*result, Device{

            Hostname:  v.Name,

            Platform:  v.Platform.Slug,

            IpAddress: strings.Split(v.PrimaryIp4.Address, "/")[0],

            Port:      uint(v.CustomFields.GnmiPort),

        })

    }



    // Return result

    return result

}



func getCredentials() (Crendetials, InventoryCredentials) {

    /* Function to get credentials. */

    return Crendetials{

            Username: os.Getenv("AUTOMATION_USER"),

            Password: os.Getenv("AUTOMATION_PASS"),

        },

        InventoryCredentials{

            Url:   os.Getenv("AUTOMATION_INVENTORY_URL"),

            Token: os.Getenv("AUTOMATION_INVENTORY_TOKEN"),

        }

}



// Main

func main() {

    /* Core logic */

    // Get credentials

    sshCreds, invCreds := getCredentials()



    // Load inventory

    inventory := loadInventory(invCreds)



    // Config

    instruction := Instruction{

        Command: "/openconfig-interfaces:interfaces",

        Config: struct {

            Path  string

            Value any

        }{

            Path: "/openconfig-interfaces:interfaces",

            Value: map[string]any{

                "interface": []map[string]any{

                    {

                        "name": "Loopback 23",

                        "config": map[string]any{

                            "name":        "Loopback 23",

                            "description": "Test-gnmi-golang-3",

                        },

                    },

                },

            },

        },

    }



    // Execute commands

    for i := 0; i < len(*inventory); i++ {

        (*inventory)[i].Crendetials = sshCreds

        (*inventory)[i].executeChange(instruction)

    }



    // Print results

    for i := 0; i < len(*inventory); i++ {

        for j := 0; j < len((*inventory)[i].Result); j++ {

            fmt.Printf(

                "Config: %v\nImpact: %v\nTimestamp: %v\n",

                (*inventory)[i].Result[j].Instruction.Config,

                (*inventory)[i].Result[j].Diff,

                (*inventory)[i].Result[j].Timestamp,

            )

        }

    }

}

We suggest to read previous blogs to get better understanding of this code, as we omit many details in this post.

Changes:

1. New custom data type of struct type is created, which is named InventoryCredentials and is used to store URL and Token to connect to NetBox.
2. In the same manner as we did in Python, we modify function getCredentials() to read from environment variables with NetBox connectivity data.
3. Function loadInventory() is reworked to fetch inventory data from NetBox using REST API:
   • Pointer to struct Client from http package is created.
   • Using function NewRequest(), which takes URL and Method name, the REST API request is prepared.
   • To this request the authorization header is added using Add() Receiver function
   • Also the query parameters are added to URL.
   • Using Do receiver function of Client, the request is made.
   • Content of the body is read as a slice of bytes
   • Using Unmarshall function from encoding/json package, the content is parsed into struct of NetboxDcimDevices type.
   • Finally, the result, which is a pointer to slice of Device is populated.

Executing the code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ go run .

2025/04/18 13:57:59 response: {

  path: {

    elem: {

      name: "openconfig-interfaces:interfaces"

    }

  }

  op: UPDATE

}

timestamp: 1744981077614602933



Config: {/openconfig-interfaces:interfaces map[interface:[map[config:map[description:Test-gnmi-golang-3 name:Loopback 23] name:Loopback 23]]]}

Impact:   main.OpenConfigInterfaces{

        Interface: []main.OpenConfigInterface{

                ... // 3 identical elements

                {Name: "Loopback51", Config: {Name: "Loopback51", Description: "pytest-update-test-33"}},

                {Name: "Loopback0", Config: {Name: "Loopback0", Enabled: true}},

                {

                        Name: "Loopback23",

                        Config: struct{ Name string "xml:"name,omitempty" json:"name,omitempty""; Description string "xml:"description,omitempty" json:"description,omitempty""; Enabled bool "xml:"enabled,omitempty" json:"enabled,omitempty"" }{

                                Name:        "Loopback23",

-                               Description: "Test-gnmi-python-2",

+                               Description: "Test-gnmi-golang-3",

                                Enabled:     true,

                        },

                },

        },

  }



Timestamp: 2025-04-18 13:57:59.50417202 +0100 BST m=+1.361305621

Lessons in GitHub

You can find the final working versions of the files from this blog at out GitHub page.

Conclusion

As you see, the interaction with REST API in Python and Golang are slightly different due to Golang being strict typed programming language and, thefore, requires more work on data processing. However, it comes also with benefit of clean data with right types. By now we’ve covered all main APIs to interact with network devices and other applications, which is part of network and IT infrastructure automation journey: SSH, NETCONF/YANG, GNMI/YANG, and now REST. There is one more topic we are to cover for us to complete this guide. Take care and good bye.

Support us

Support new interop and automation articles at karneliuk.com
EUR

I want to support with:
9.99 EUR4.99 EUR24.99 EUR49.99 EUR99.99 EUR199.99 EUR

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk 

Within past blog posts we covered how to interact with network devices (in fact, many servers support that as well) using SSH/CLI and NETCONF/YANG. Those two protocols allow you to confidently cover almost all cases for managing devices, whether you prefer more human-like approach, that is templating CLI commands and pushing them via SSH or using structured XML data and send it via NETCONF. However, there are more protocols and today we are going to talk about the most modern to the date, which is called GNMI (generalized network management interface).

How Does Automation Help Real Business?

Talking to students at our trainings and with customers and peers at various events, there is often a concern arise that small and medium businesses don’t have time and/or need to invest in automation. There is no time as engineers are busy solving “real” problems, like outages, customer experience degradation, etc. There is no need because why to bother, we have engineers to tackle issue. From my experience automation allows to save so much time on doing operational tasks and to free it up for improving user experience and developing new problems, that is difficult to overestimate. It really is that important and the sooner you start using it, the better it will be for your business

And our trainings are designed to teach you how to solve real world problems with automation. Enroll today:

We offer the following training programs in network automation for you:

• Zero-to-Hero Network Automation Training
• High-scale automation with Nornir
• Ansible Automation Orchestration with Ansble Tower / AWX
• Expert-level training: Closed-loop Automation and Next-generation Monitoring

During these trainings you will learn the following topics:

• Success and failure strategies to build the automation tools.
• Principles of software developments and the most useful and convenient tools.
• Data encoding (free-text, XML, JSON, YAML, Protobuf).
• Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
• Full configuration templating with Jinja2 based on the source of truth (NetBox).
• Best programming languages (Python, Bash) for developing automation
• The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
• Network automation infrastructure (Linux, Linux networking, KVM, Docker).
• Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
• Collection network data via SNMP and streaming telemetry with Prometheus
• Building API gateways with Python leveraging Fast API
• Integration of alerting with Slack and your own APIs
• … and many more

Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data center networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.

Start your automation training today.

What Are We Going To Talk Today?

GNMI is the newest management protocol, which is used in networking world. It was developed originally by Google to manage their network infrastructure. In today’s blog post we’ll answer the following questions:

1. How GNMI is different to other management protocols?
2. In which use cases you shall consider using GNMI?
3. How to configure network devices using GNMI?

Explanation

GNMI has an interesting story. All the management protocols, whether it is SSH, NETCONF or any other (e.g., RESTCONF, which we don’t include in our blog series due to limited real-world implementations) are IETF standards and have associated RFCs. GNMI doesn’t have RFC covering its operation; yet its adoption across network vendors is very high. Cisco IOS-XE / IOS XR / NX-OS, Arista EOS, Juniper JUNOS, Nokia SR OS and many more supports GNMI. There are multiple reasons why this happened; what matters though, you can use it configure network devices, collect operational data, and many more.

In our network automation trainings, we cover reasons behind GNMI popularity as well as its operation in-depth.

From the perspective of configuration or data retrieval, GNMI relies on YANG modules, the very same YANG modules we’ve introduced in the previous blogpost. However, it doesn’t use XML serialisation. Instead, it uses Protobuf, which is another important Google’s invention. However, the real killer feature of GNMI is support of streaming telemetry, the mechanism which revolutionized the world of collecting operational data from network and infrastructure devices.

Join our Zero-to-Hero Network Automation Training to learn how us streaming telemetry in a programmable way.

Let’s briefly compare GNMI to NETCONF to get some sense of it:

RPC stands for remote procedure call

If you can see our video overview of GNMI from UKNOF conference.

Since Protobuf is not really human-readable, we’ll show you how to use GNMI in code.

Example

Same as in the previous blog, we are going to use the very same logic of execution production grade change:

1. Collection of the operational state of the network device before the change.
2. Execution of the change.
3. Collection of the operational state of the network device after the change.
4. Comparing the difference of the operational state before and after the change.

Python

Let’s start with Python. You need to install the following two libraries:

1
$ pip install pyyaml pygnmi

By the way, pygnmi library was written by myself to address the lack of solid Python library for managing network devices with GNMI. It is great to see its wide adoption across the industry.

And now the code of Python application to manage network device with GNMI:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
"""From Python to Go: Python: 017 - NETCONF."""



# Modules

import argparse

import datetime

import os

import sys

from dataclasses import dataclass

from typing import List

import difflib

import pygnmi.client

import yaml

import pygnmi

# from scrapli_netconf.driver import NetconfDriver

# import xmltodict





# Classes

@dataclass

class Credentials:

    """Class to store credentials."""

    username: str

    password: str





@dataclass

class Instruction:

    """Class to store instructions."""

    command: List[str]

    config: List[tuple]





@dataclass

class Result:

    """Class to store command execution data."""

    instruction: Instruction

    diff: str

    timestamp: datetime.datetime





class Device:

    """Class to interact with netowrk device."""

    def __init__(self, hostname: str, ip_address: str, port: int, platform: str, credentials: Credentials):

        self.hostname = hostname

        self.ip_address = ip_address

        self.port = port

        self.platform = platform

        self.credentials = credentials



        self.results: List[Result] = []



    def execute_change(self, instruction: Instruction) -> None:

        """Method to execute change."""



        # Connect to device

        with pygnmi.client.gNMIclient(

            target=(self.ip_address, self.port),

            username=self.credentials.username,

            password=self.credentials.password,

            skip_verify=True,

            timeout=5,

        ) as gconn:

            # Get state before change

            before = gconn.get(path=instruction.command, datatype="config")

            before_stringified = self.dict_to_xpath(before)



            # Apply change

            config_result = gconn.set(update=instruction.config, encoding="json_ietf")

            print(f"{config_result=}")



            # Get state after change

            after = gconn.get(path=instruction.command, datatype="config")

            after_stringified = self.dict_to_xpath(after)



            # Diff

            diff = "\n".join(

                difflib.context_diff(

                    before_stringified,

                    after_stringified,

                    lineterm="",

                )

            )



            self.results.append(

                Result(

                    instruction=instruction,

                    diff=diff,

                    timestamp=datetime.datetime.now(),

                )

            )



    def dict_to_xpath(self, data: dict) -> list:

        """Method to convert dict to xpath."""

        result = []



        if isinstance(data, str):

            return data



        for key, value in data.items():

            if isinstance(value, list):

                for ind, item in enumerate(value):

                    tr = self.dict_to_xpath(item)

                    result.extend([f"{key}/{ind}/{_}" for _ in tr])



            elif isinstance(value, dict):

                tr = self.dict_to_xpath(value)

                result.extend([f"{key}/{_}" for _ in tr])



            else:

                result.append(f"{key} = {value}")



        return result





# Functions

def read_args() -> argparse.Namespace:

    """Helper function to read CLI arguments."""

    parser = argparse.ArgumentParser(description="User input.")

    parser.add_argument("-i", "--inventory", type=str, help="Path to inventory file.")

    return parser.parse_args()





def load_inventory(filename: str, credentials: Credentials) -> List[Device]:

    """Function to load inventory data."""

    # Open file

    try:

        with open(filename, "r", encoding="utf-8") as f:

            data = yaml.safe_load(f)



    except FileNotFoundError as e:

        print(e)

        sys.exit(1)



    # Populate list of devices

    result = []

    for device in data:

        result.append(Device(credentials=credentials, **device))



    return result





def get_credentials() -> Credentials:

    """Function to get credentials."""

    username = os.getenv("AUTOMATION_USER")

    password = os.getenv("AUTOMATION_PASS")

    return Credentials(username, password)





# Main code

if __name__ == "__main__":

    # Read CLI arguments

    args = read_args()



    # Get credentials

    credentials = get_credentials()



    # Load inventory

    devices = load_inventory(args.inventory, credentials=credentials)



    # Config

    instruction = Instruction(

        command=["/openconfig-interfaces:interfaces"],

        config=[

            (

                "/openconfig-interfaces:interfaces",

                {

                    "interface": [

                        {

                            "name": "Loopback 23",

                            "config": {

                                "name": "Loopback 23",

                                "description": "Test-gnmi-python-2",

                            }

                        },

                    ],

                },

            ),

        ],

    )



    # Execute command

    for device in devices:

        device.execute_change(instruction)



    # Print results

    for device in devices:

        print(f"Device: {device.hostname}")

        for result in device.results:

            print(f"Config: {result.instruction.config}", f"Impact: {result.diff}", f"Timestamp: {result.timestamp}", sep="\n")

It is strongly recommended to read previous blogs as concepts explained before aren’t repeated.

Let’s break what’s new and how GNMI is used:

1. Import “pygnmi.client” to get access to GNMI client.
2. Data class “Instruction” is amended so that its field “command” became list of strings instead of dictionary and the field “config” became list of tuples instead of dictionary. These changes are made to map the pygnmi data structures.
3. Differences are made within method “execute_change()” from “Deivce” class:
   • Instance of class “pygnmi.client.gNMIclient” is created, which handles gNMI connection with the device
   • Using “get()” method of this class the operational data is retrieved. As an argument you pass here list of GNMI Path to fetch data from in a format of simple strings.
   • Using “set()” method of this class the configuration is pushed to network device. There are three options how you can do it:
     • “update” will create new one or modify existing one by overriding provided key/value pairs.
     • “replace” will completely replace the content of the existing key/container and all its further nested elements.
     • “delete” will delete the key/container including all nested elements.
   • Using “dict_to_xpath()” the nested dictionaries are converted to list of key/value pairs, which is then compared.
4. Method “dict_to_xpath()” created in the previous blog to convert the nested keys in XPath was proven to be extremely useful and we continue using it here as well to convert nested GNMI data to XPath.
5. The input object “instruction” is modified to match changes in “Instruction” class as outlined above.

As you can see, we used the absolutely identical logic to change and code structure for SSH, NETCONF and now for GNMI. What slightly changes is an input data. In real production application you would typically have some normalization layer, which will handle these differences.

The result of execution is the following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
$ python main.py -i ../data/inventory.yaml

ssl_target_name_override is applied, should be used for testing only!

config_result={'timestamp': 1743270193131674380, 'prefix': None, 'response': [{'path': 'interfaces', 'op': 'UPDATE'}]}

Device: go-blog-arista

Config: [('/openconfig-interfaces:interfaces', {'interface': [{'name': 'Loopback 23', 'config': {'name': 'Loopback 23', 'description': 'Test-gnmi-python-2'}}]})]

Impact: ***

---

***************

*** 97,110 ****

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-fcs-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-multicast-pkts = 0

! notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-octets = 49384793

! notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-unicast-pkts = 649097

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-broadcast-pkts = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-discards = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-multicast-pkts = 0

! notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-octets = 18386266

! notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-unicast-pkts = 63350

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/openconfig-platform-port:hardware-port = Port97

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/ifindex = 999001

--- 97,110 ----

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-fcs-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-multicast-pkts = 0

! notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-octets = 49390958

! notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/in-unicast-pkts = 649163

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-broadcast-pkts = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-discards = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-errors = 0

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-multicast-pkts = 0

! notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-octets = 18434988

! notification/0/update/0/val/openconfig-interfaces:interface/0/state/counters/out-unicast-pkts = 63463

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/openconfig-platform-port:hardware-port = Port97

  notification/0/update/0/val/openconfig-interfaces:interface/0/state/ifindex = 999001

***************

*** 742,748 ****

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/openconfig-if-ip:ipv6/state/mtu = 1500

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/state/index = 0

! notification/0/update/0/val/openconfig-interfaces:interface/3/config/description = Test-gnmi-python

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/arista-intf-augments:load-interval = 300

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/loopback-mode = True

--- 742,748 ----

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/openconfig-if-ip:ipv6/state/mtu = 1500

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/2/subinterfaces/subinterface/0/state/index = 0

! notification/0/update/0/val/openconfig-interfaces:interface/3/config/description = Test-gnmi-python-2

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/arista-intf-augments:load-interval = 300

  notification/0/update/0/val/openconfig-interfaces:interface/3/config/loopback-mode = True

***************

*** 757,763 ****

  notification/0/update/0/val/openconfig-interfaces:interface/3/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/3/state/name = Loopback23

  notification/0/update/0/val/openconfig-interfaces:interface/3/state/openconfig-vlan:tpid = openconfig-vlan-types:TPID_0X8100

! notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/config/description = Test-gnmi-python

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/config/index = 0

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/index = 0

--- 757,763 ----

  notification/0/update/0/val/openconfig-interfaces:interface/3/state/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/3/state/name = Loopback23

  notification/0/update/0/val/openconfig-interfaces:interface/3/state/openconfig-vlan:tpid = openconfig-vlan-types:TPID_0X8100

! notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/config/description = Test-gnmi-python-2

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/config/enabled = True

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/config/index = 0

  notification/0/update/0/val/openconfig-interfaces:interface/3/subinterfaces/subinterface/0/index = 0

Timestamp: 2025-03-29 17:43:15.708137

The content of line, which are different starts with the exclamation mark. So you see that name of interface is changed following execution of our change.

Now onto Golang.

Go (Golang)

For Golang we use the following external packages:

1
2
3
4
$ go get github.com/google/go-cmp/cmp

$ go get gopkg.in/yaml.v3

$ go get github.com/openconfig/gnmic/api

$ go get google.golang.org/protobuf/encoding/prototext

Finally I had an opportunity to work with gnmic library created by Nokia’s Roman Dodin and Karim Radhouani.

Using gnmic, the following Go (Golang) application was developed to manage network devices using GNMI:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
/* From Python to Go: Go: 018 - GNMI. */



package main



// Imports

import (

    "context"

    "encoding/json"

    "flag"

    "fmt"

    "log"

    "os"

    "time"



    "github.com/google/go-cmp/cmp"

    "github.com/openconfig/gnmic/pkg/api"

    "google.golang.org/protobuf/encoding/prototext"

    "gopkg.in/yaml.v3"

)



// Types and Receivers

type Arguments struct {

    /* Class to starte CLI arguments */

    Inventory string

}



type Crendetials struct {

    /* Struct to store credentials. */

    Username string

    Password string

}



type Instruction struct {

    Command string

    Config  struct {

        Path  string

        Value any

    }

}



type Result struct {

    /* Struct to store command execution result. */

    Instruction Instruction

    Diff        string

    Timestamp   time.Time

}



type Device struct {

    /* Struct to interact with netowrk device. */

    Hostname    string `yaml:"hostname"`

    IpAddress   string `yaml:"ip_address"`

    Port        uint   `yaml:"port"`

    Platform    string `yaml:"platform"`

    Crendetials Crendetials

    Result      []Result

}



func (d *Device) executeChange(i Instruction) {

    /* Method to execute command */



    // Create GNMI Target

    gnmiTarget, err := api.NewTarget(

        api.Name(d.Hostname),

        api.Address(fmt.Sprintf("%s:%d", d.IpAddress, d.Port)),

        api.Username(d.Crendetials.Username),

        api.Password(d.Crendetials.Password),

        api.SkipVerify(true),

    )

    if err != nil {

        log.Fatal("Cannot create GNMI Target: ", err)

    }



    // Create context

    ctx, cancel := context.WithCancel(context.Background())

    defer cancel()



    // Create GNMI client

    err = gnmiTarget.CreateGNMIClient(ctx)

    if err != nil {

        log.Fatal("Cannot create GNMI Client: ", err)

    }

    defer gnmiTarget.Close()



    // Get state before change

    getReq, err := api.NewGetRequest(

        api.Path(i.Command),

        api.DataType("config"),

        api.Encoding("json_ietf"),

    )

    if err != nil {

        log.Fatal("Cannot create Get request: ", err)

    }

    beforeGetResponse, err := gnmiTarget.Get(ctx, getReq)

    if err != nil {

        log.Fatal("Cannot make a Get request: ", err)

    }

    beforeStruct := OpenConfigInterfaces{}

    err = json.Unmarshal(beforeGetResponse.Notification[0].Update[0].Val.GetJsonIetfVal(), &beforeStruct)

    if err != nil {

        log.Fatal("Cannot unmarshall JSON: ", err)

    }



    // Make change

    setReq, err := api.NewSetRequest(

        api.Update(

            api.Path(i.Config.Path),

            api.Value(i.Config.Value, "json_ietf"),

        ),

    )

    if err != nil {

        log.Fatal("Cannot create Set request: ", err)

    }

    setResp, err := gnmiTarget.Set(ctx, setReq)

    if err != nil {

        log.Fatal("Cannot make a Set request: ", err)

    }

    log.Println(prototext.Format(setResp))



    // Get state after change

    afterGetResponse, err := gnmiTarget.Get(ctx, getReq)

    if err != nil {

        log.Fatal("Cannot make a Get request: ", err)

    }

    afterStruct := OpenConfigInterfaces{}

    err = json.Unmarshal(afterGetResponse.Notification[0].Update[0].Val.GetJsonIetfVal(), &afterStruct)

    if err != nil {

        log.Fatal("Cannot unmarshall JSON: ", err)

    }

    // Diff

    diff := cmp.Diff(beforeStruct, afterStruct)



    // Update the result

    (*d).Result = append((*d).Result, Result{

        Instruction: i,

        Diff:        diff,

        Timestamp:   time.Now(),

    })

}



// Functions

func readArgs() Arguments {

    /* Helper function to read CLI arguments */

    result := Arguments{}



    flag.StringVar(&result.Inventory, "i", "", "Path to the inventory file")



    flag.Parse()



    return result

}



func loadInventory(p string) *[]Device {

    /* Function to load inventory data. */



    // Open file

    bs, err := os.ReadFile(p)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Load inventory

    result := &[]Device{}



    err = yaml.Unmarshal(bs, result)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Return result

    return result

}



func getCredentials() Crendetials {

    /* Function to get credentials. */

    return Crendetials{

        Username: os.Getenv("AUTOMATION_USER"),

        Password: os.Getenv("AUTOMATION_PASS"),

    }

}



// Main

func main() {

    /* Core logic */

    // Read CLI arguments

    cliArgs := readArgs()



    // Get credentials

    sshCreds := getCredentials()



    // Load inventory

    inventory := loadInventory(cliArgs.Inventory)



    // Config

    instruction := Instruction{

        Command: "/openconfig-interfaces:interfaces",

        Config: struct {

            Path  string

            Value any

        }{

            Path: "/openconfig-interfaces:interfaces",

            Value: map[string]any{

                "interface": []map[string]any{

                    {

                        "name": "Loopback 23",

                        "config": map[string]any{

                            "name":        "Loopback 23",

                            "description": "Test-gnmi-golang-3",

                        },

                    },

                },

            },

        },

    }



    // Execute commands

    for i := 0; i < len(*inventory); i++ {

        (*inventory)[i].Crendetials = sshCreds

        (*inventory)[i].executeChange(instruction)

    }



    // Print results

    for i := 0; i < len(*inventory); i++ {

        for j := 0; j < len((*inventory)[i].Result); j++ {

            fmt.Printf(

                "Config: %v\nImpact: %v\nTimestamp: %v\n",

                (*inventory)[i].Result[j].Instruction.Config,

                (*inventory)[i].Result[j].Diff,

                (*inventory)[i].Result[j].Timestamp,

            )

        }

    }

}

And structs for OpenConfig, which we re-use from previous blog with some minor modifications:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
/* From Python to Go: Go: 018 - GNMI: modules */



package main



// Import



// Data types

type OpenConfigInterface struct {

    Name   string `xml:"name,omitempty"`

    Config struct {

        Name        string `xml:"name,omitempty" json:"name,omitempty"`

        Description string `xml:"description,omitempty" json:"description,omitempty"`

        Enabled     bool   `xml:"enabled,omitempty" json:"enabled,omitempty"`

    } `xml:"config" json:"config"`

}

type OpenConfigInterfaces struct {

    Interface []OpenConfigInterface `xml:"openconfig-interfaces:interface,omitempty" json:"openconfig-interfaces:interface,omitempty"`

}

It is strongly recommended to read previous blogs as concepts explained before aren’t repeated.

Here what happens:

1. In structs we add mapping to JSON keys. The reason we do this is because many vendors choose to send serialized JSON as a string within Protobuf message. Using JSON allows us to parse it and to deal with structured data.
2. Fields “Command” and “Config” of type “Instruction” are modified to be list of strings and list of structs respectively, matching the pattern we had with pygnmi before.
3. Also like in Python part, the major changes happened within receiver function “executeChange()“:
   • Using function “NewTarget()” from gnmic library the gnmi target is created. Target here is an abstract definition of the device, where the connectivity will be established to.
   • Using receiver function “CreateGNMIClient()” of Target struct, connectivity is established.
   • Using function “NewGetRequest()” and “NewSetRequest()” the Protobus messages are generated.
   • Using receiver functions “Get()” and “Set()” of Target these messages are sent are responses receveid.
   • Responses are parsed from JSON strings embedded in Protobuf messages into structs using parsing methodology from JSON to struct we covered before.
4. The struct “instruction” created out ouf “Instruction” type is populated with details needed to do configuration.

Now let’s execute this Go (Golang) code to perform change on network device:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
$ go run . -i ../data/inventory.yaml

2025/03/29 20:48:28 response:  {

  path:  {

    elem:  {

      name:  "openconfig-interfaces:interfaces"

    }

  }

  op:  UPDATE

}

timestamp:  1743281306005644545



Config: {/openconfig-interfaces:interfaces map[interface:[map[config:map[description:Test-gnmi-golang-3 name:Loopback 23] name:Loopback 23]]]}

Impact:   main.OpenConfigInterfaces{

        Interface: []main.OpenConfigInterface{

                {Name: "Management1", Config: {Name: "Management1", Enabled: true}},

                {Name: "Ethernet2", Config: {Name: "Ethernet2", Enabled: true}},

                {Name: "Ethernet1", Config: {Name: "Ethernet1", Enabled: true}},

                {

                        Name: "Loopback23",

                        Config: struct{ Name string "xml:"name,omitempty" json:"name,omitempty""; Description string "xml:"description,omitempty" json:"description,omitempty""; Enabled bool "xml:"enabled,omitempty" json:"enabled,omitempty"" }{

                                Name:        "Loopback23",

-                               Description: "Test-gnmi-golang-2",

+                               Description: "Test-gnmi-golang-3",

                                Enabled:     true,

                        },

                },

                {Name: "Loopback0", Config: {Name: "Loopback0", Enabled: true}},

                {Name: "Loopback51", Config: {Name: "Loopback51", Description: "pytest-update-test-33"}},

        },

  }



Timestamp: 2025-03-29 20:48:28.7119283 +0000 GMT m=+0.725116820

You can see the similar logic as we had in the previous blog, where comparing of two structs using third-party libraries provides very preciese difference and we can assess if our change reached the goal.

Lessons in GitHub

You can find the final working versions of the files from this blog at out GitHub page.

Conclusion

GNMI is no doubts a new and powerful protocol. However, there are still talks though in industry, where it will have wide production usage outside of Google network. Time will answer this question. From our perspective, though, it is important to know how you can use it both in Python and Go (Golang). Take care and good bye.

Support us

Support new interop and automation articles at karneliuk.com
EUR

I want to support with:
9.99 EUR4.99 EUR24.99 EUR49.99 EUR99.99 EUR199.99 EUR

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk 

We continue exploring programmable network management using Python and Go (Golang) as programming languages. In today’s blog post we’ll cover how to interact with network devices using NETCONF.

How To Chose Which API To Use?

There are many APIs (Application Programmable Interfaces) out there. We already covered SSH and now covering NETCONF. And there are a few more existing, which we are going to cover. Cannot we just stick to a single API for all use cases. The truth is that each API has its own advantages and disadvantages, as well as design patterns and areas, where it shall be used. As such, each of them is important and valuable.

And in our training programs we do deep-dive in all these APIs. Enrol today:

We offer the following training programs in network automation for you:

• Zero-to-Hero Network Automation Training
• High-scale automation with Nornir
• Ansible Automation Orchestration with Ansble Tower / AWX
• Expert-level training: Closed-loop Automation and Next-generation Monitoring

During these trainings you will learn the following topics:

• Success and failure strategies to build the automation tools.
• Principles of software developments and the most useful and convenient tools.
• Data encoding (free-text, XML, JSON, YAML, Protobuf).
• Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
• Full configuration templating with Jinja2 based on the source of truth (NetBox).
• Best programming languages (Python, Bash) for developing automation
• The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
• Network automation infrastructure (Linux, Linux networking, KVM, Docker).
• Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
• Collection network data via SNMP and streaming telemetry with Prometheus
• Building API gateways with Python leveraging Fast API
• Integration of alerting with Slack and your own APIs
• … and many more

Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data center networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.

Start your automation training today.

What Are We Going To Talk Today?

NETCONF is a truly programmable way to manage network devices. NETCONF is a standard, covered in a number of RFCs; check 6241 and 6242 to start with. Being standard, which exists already for a while, it is wide spreaded with majority of vendors supporting it. I personally like it a lot and use it a lot across various projects. In today’s blog we are going to show you, how you can use NETCONF, namely:

1. How to prepare XML messages and parse incoming ones?
2. How to get configuration from network devices using NETCONF?
3. How to configure network devices using NETCONF?

Explanation

NETCONF is a great protocol to manage network devices in a programmable way. It uses SSH transport, as we discussed previously. However, it uses XML serialisation, what makes it simpler to deal with from code perspective compared to CLI commands we sent the last time. It is simpler, because XML serialises structured data, which with minimal efforts can be converted in maps/dictionaries or data classes/structs. The model of structured data is defined using YANG modules.

Join our Zero-To-Hero Network Automation Training to master YANG.

Here is an example of NETCONF message:

1
2
3
4
5
6
7
8
9
10
11
12
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">

  <capabilities>

    <capability>urn:ietf:params:netconf:base:1.0</capability>

    <capability>urn:ietf:params:netconf:base:1.1</capability>

    <capability>urn:ietf:params:xml:ns:netconf:base:1.0</capability>

    <capability>urn:ietf:params:xml:ns:yang:ietf-netconf-monitoring</capability>

    <capability>urn:ietf:params:netconf:capability:writable-running:1.0</capability>

    <capability>urn:ietf:params:netconf:capability:candidate:1.0</capability>

    <capability>urn:ietf:params:netconf:capability:url:1.0?scheme=file,flash,ftp,http</capability>

    <capability>http://openconfig.net/yang/network-instance-l3?module=openconfig-network-instance-l3&revision=2018-11-21</capability>

  </capabilities>

</hello>

This can fit in the following set of Python data classes:

1
2
3
4
5
6
7
8
9
10
from dataclasses import dataclass

from typing import List



@dataclass

class Capability:

    capability: List[str]



@dataclass

class Capabilities:

    capabilities: Capability

Or into the following Go (Golang) struct:

1
2
3
4
5
6
import "encoding/xml"



type Capabilities struct {

   XMLName     xml.Name   `xml:"capabilities"`

   Capability  []string   `xml:"capability"`

}

One question that you may have in your mind now, how do we convert XML data to structured data? Luckily, we already discussed that in our blog series: there is a great third party Python library, which does conversion into Python dictionary for you. Afterwards, you populate data class from dictionary.

In case of Go (Golang), we use built-in package “encoding/xml”, which does for us marshalling (conversion of structs to XML bytes) and unmarshalling (conversion of XML bytes to structs).

To create messages, apart from YANG modules, we also need to know basic NETCONF operations (officially they are called RPCs – remote procedure calls). We’ll call here a few, some of which we are to use in our example:

Enrol into Zero-to-Hero Network Automation Training to learn more about NETCONF operations and how to use them.

“Talk is cheap. Show me the code”, as Linus Torvalds is used to day. Let’s take a look at the examples.

Example

We’ll re-use the scenario from the previous blog post, with replacing SSH/CLI by NETCONF/YANG:

• Get the config of interfaces before the change
• Apply new configuration
• Get the configuration of interfaces after the change
• Compare the configuration

As we are dealing with NETCONF, though, we will need to ensure we do conversion between dictionaries or structs to XML strings and vice versa.

For YANG modules, we will OpenConfig, which is a multi-vendor YANG model. It is supported by Arista EOS, Cisco IOS XR, Cisco NX-OS and many others.

Start Zero-to-Hero Network Automation Training to get up to speed with different YANG modules, and where they shall be used.

Python

We traditionally start with Python code, so we do today. First of all, you need to install the following dependencies:

1
$ pip install pyyaml scrapli scrapli-netconf xmltodict ssh2-python

If you are not familiar how to convert to/from XML, we recommend to read this blog first.

Now on to coding:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
"""From Python to Go: Python: 017 - NETCONF."""



# Modules

import argparse

import datetime

import os

import sys

from dataclasses import dataclass

from typing import List

import difflib

import yaml

from scrapli_netconf.driver import NetconfDriver

import xmltodict





# Classes

@dataclass

class Credentials:

    """Class to store credentials."""

    username: str

    password: str





@dataclass

class Instruction:

    """Class to store instructions."""

    command: dict

    config: dict





@dataclass

class Result:

    """Class to store command execution data."""

    instruction: Instruction

    diff: str

    timestamp: datetime.datetime





class Device:

    """Class to interact with netowrk device."""

    def __init__(self, hostname: str, ip_address: str, platform: str, credentials: Credentials):

        self.hostname = hostname

        self.ip_address = ip_address

        self.platform = platform

        self.credentials = credentials



        self.results: List[Result] = []



    def execute_change(self, instruction: Instruction) -> None:

        """Method to execute change."""



        # Connect to device

        with NetconfDriver(

            host=self.ip_address,

            port=830,

            auth_username=self.credentials.username,

            auth_password=self.credentials.password,

            auth_strict_key=False,

            transport="ssh2",

        ) as conn:

            filter_ = xmltodict.unparse(instruction.command).splitlines()[1]

            # Get state before change

            before = conn.get_config(source="running", filter_=filter_, filter_type="subtree")

            before_stringified = self.dict_to_xpath(xmltodict.parse(before.result))



            # Apply change

            config_ = "\n".join(xmltodict.unparse(instruction.config).splitlines()[1:])

            change_result = conn.edit_config(target="candidate", config=config_)



            if change_result.failed:

                print(f"Error: {change_result.result}")

            else:

                commit_result = conn.commit()

                if commit_result.failed:

                    print(f"Error: {commit_result.result}")



            # Get state after change

            after = conn.get_config(source="running", filter_=filter_, filter_type="subtree")

            after_stringified = self.dict_to_xpath(xmltodict.parse(after.result))



            # Diff

            diff = "\n".join(

                difflib.context_diff(

                    before_stringified,

                    after_stringified,

                    lineterm="",

                )

            )



            self.results.append(

                Result(

                    instruction=instruction,

                    diff=diff,

                    timestamp=datetime.datetime.now(),

                )

            )



    def dict_to_xpath(self, data: dict) -> list:

        """Method to convert dict to xpath."""

        result = []

        for key, value in data.items():

            if isinstance(value, list):

                for ind, item in enumerate(value):

                    tr = self.dict_to_xpath(item)

                    result.extend([f"{key}/{ind}/{_}" for _ in tr])



            elif isinstance(value, dict):

                tr = self.dict_to_xpath(value)

                result.extend([f"{key}/{_}" for _ in tr])



            else:

                result.append(f"{key} = {value}")



        return result





# Functions

def read_args() -> argparse.Namespace:

    """Helper function to read CLI arguments."""

    parser = argparse.ArgumentParser(description="User input.")

    parser.add_argument("-i", "--inventory", type=str, help="Path to inventory file.")

    return parser.parse_args()





def load_inventory(filename: str, credentials: Credentials) -> List[Device]:

    """Function to load inventory data."""

    # Open file

    try:

        with open(filename, "r", encoding="utf-8") as f:

            data = yaml.safe_load(f)



    except FileNotFoundError as e:

        print(e)

        sys.exit(1)



    # Populate list of devices

    result = []

    for device in data:

        result.append(Device(credentials=credentials, **device))



    return result





def get_credentials() -> Credentials:

    """Function to get credentials."""

    username = os.getenv("AUTOMATION_USER")

    password = os.getenv("AUTOMATION_PASS")

    return Credentials(username, password)





# Main code

if __name__ == "__main__":

    # Read CLI arguments

    args = read_args()



    # Get credentials

    credentials = get_credentials()



    # Load inventory

    devices = load_inventory(args.inventory, credentials=credentials)



    # Config

    instruction = Instruction(

        command={"interfaces": {"@xmlns": "http://openconfig.net/yang/interfaces"}},

        config={

            "config": {

                "interfaces": {

                    "@xmlns": "http://openconfig.net/yang/interfaces",

                    "interface": [

                        {

                            "name": "Loopback 23",

                            "config": {

                                "name": "Loopback 23",

                                "description": "Test-netconf-python-2",

                            }

                        },

                    ],

                },

            },

        },

    )



    # Execute command

    for device in devices:

        device.execute_change(instruction)



    # Print results

    for device in devices:

        print(f"Device: {device.hostname}")

        for result in device.results:

            print(f"Config: {result.instruction.config}", f"Impact: {result.diff}", f"Timestamp: {result.timestamp}", sep="\n")

We strongly recommend to read previous blogs, if you aren’t sure about some parts of the code, as we will focus our explanation only on what is relevant to NETCONF.

Assuming you have read previous blogs, here is what is NETCONF-specific happening in this snippet:

1. Instruction data class now uses dictionaries as data types, as we will use them to store configuration and operational filter, so that we can convert it to XML using xmltodict library.
2. The method execute_change() of Device class is re-written to use NETCONF protocol leveraging scrapli-netconf library:
   • Class NetconfDriver() is used to establish NETCONF connectivity. We use transport “ssh2“, which is available for us after installing the aforementioned third party libraries.
   • Using unparse() function from xmltodict library, the dictionary is converted to a multi-line string with XML syntax. The string starts with XML declaration, which needs to be removed before it can be sent to network device.
   • Running configuration is collected from the remote device using get_config() method, which performs under the hood NETCONF get-config RPC. The result is parsed to dict using parse() function of xmltodict library and then processed by the new method (see below point 3).
   • Then configuration is pushed to network device, with message being converted from dict to XML using same method as explained above.
   • If result isn’t failed, then configuration is commit using commit() method of scrapli-netconf.
   • If there are no errors, the running configuration is collected again and processed.
   • Finally, the pre- and post- configurations are compared and difference is identified
3. In addition, there is a new method dict_to_xpath(), which converts nested dictionaries and lists in a flat key/value structure, where key combines all the nested sub-keys using recursion. This allows us to simplify visualisation of changes made on the device.

As you can see, the logic is the same as we have in the previous blog post with SSH/CLI; we’ve added a few extra helper functions.

Let’s execute this script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
$ python main.py -i ../data/inventory.yaml

Device: go-blog-nexus

Config: {'config': {'interfaces': {'@xmlns': 'http://openconfig.net/yang/interfaces', 'interface': [{'name': 'Loopback 23', 'config': {'name': 'Loopback 23', 'description': 'Test-netconf-python-2'}}]}}}

Impact: *** 

--- 

***************

*** 1,6 ****

  rpc-reply/@xmlns = urn:ietf:params:xml:ns:netconf:base:1.0

! rpc-reply/@message-id = 101

! rpc-reply/data/@time-modified = 2025-03-22T16:53:24.80910925Z

  rpc-reply/data/interfaces/@xmlns = http://openconfig.net/yang/interfaces

  rpc-reply/data/interfaces/interface/0/name = Management1

  rpc-reply/data/interfaces/interface/0/config/enabled = true

--- 1,6 ----

  rpc-reply/@xmlns = urn:ietf:params:xml:ns:netconf:base:1.0

! rpc-reply/@message-id = 104

! rpc-reply/data/@time-modified = 2025-03-22T16:53:25.701759874Z

  rpc-reply/data/interfaces/@xmlns = http://openconfig.net/yang/interfaces

  rpc-reply/data/interfaces/interface/0/name = Management1

  rpc-reply/data/interfaces/interface/0/config/enabled = true

***************

*** 134,140 ****

  rpc-reply/data/interfaces/interface/2/subinterfaces/subinterface/ipv6/config/enabled = false

  rpc-reply/data/interfaces/interface/2/subinterfaces/subinterface/ipv6/config/mtu = 1500

  rpc-reply/data/interfaces/interface/3/name = Loopback23

! rpc-reply/data/interfaces/interface/3/config/description = Test-netconf-golang-2

  rpc-reply/data/interfaces/interface/3/config/enabled = true

  rpc-reply/data/interfaces/interface/3/config/load-interval/@xmlns = http://arista.com/yang/openconfig/interfaces/augments

  rpc-reply/data/interfaces/interface/3/config/load-interval/#text = 300

--- 134,140 ----

  rpc-reply/data/interfaces/interface/2/subinterfaces/subinterface/ipv6/config/enabled = false

  rpc-reply/data/interfaces/interface/2/subinterfaces/subinterface/ipv6/config/mtu = 1500

  rpc-reply/data/interfaces/interface/3/name = Loopback23

! rpc-reply/data/interfaces/interface/3/config/description = Test-netconf-python-2

  rpc-reply/data/interfaces/interface/3/config/enabled = true

  rpc-reply/data/interfaces/interface/3/config/load-interval/@xmlns = http://arista.com/yang/openconfig/interfaces/augments

  rpc-reply/data/interfaces/interface/3/config/load-interval/#text = 300

***************

*** 148,154 ****

  rpc-reply/data/interfaces/interface/3/hold-time/config/down = 0

  rpc-reply/data/interfaces/interface/3/hold-time/config/up = 0

  rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/index = 0

! rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/config/description = Test-netconf-golang-2

  rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/config/enabled = true

  rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/config/index = 0

  rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/ipv4/@xmlns = http://openconfig.net/yang/interfaces/ip

--- 148,154 ----

  rpc-reply/data/interfaces/interface/3/hold-time/config/down = 0

  rpc-reply/data/interfaces/interface/3/hold-time/config/up = 0

  rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/index = 0

! rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/config/description = Test-netconf-python-2

  rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/config/enabled = true

  rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/config/index = 0

  rpc-reply/data/interfaces/interface/3/subinterfaces/subinterface/ipv4/@xmlns = http://openconfig.net/yang/interfaces/ip

Timestamp: 2025-03-22 16:53:26.975625

If we haven’t introduced new function for making nested keys flat, it will be very difficult to compare two configurations in Python. With our improvement though it is fairly trivial and easy understandable by humans.

Go (Golang)

Now we’ll demonstrate same application in Go (Golang). Here are the external packages, which need to be installed:

1
2
3
$ go get github.com/google/go-cmp/cmp

$ go get gopkg.in/yaml.v3

$ go get github.com/scrapli/scrapligo

Those are all the same packages, which were used in the previous blog about SSH. In contrast to Python, Go (Golang) version of scrapli – scrapligo, includes NETCONF driver and it doesn’t have to be installed separately.

Before jumping to the code, we’ll highlight one important difference we are going to implement in Go (Golang) code: instead of using map any types, we will create a set of structs, which will be used to prepare configuration as well as to parse the received messages. Here are those structs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
$ cat modules.go

/* From Python to Go: Go: 017 - NETCONF : modules */



package main



// Import

import (

    "encoding/xml"

)



// Data types

type OpenConfigInterface struct {

    Name   string `xml:"name,omitempty"`

    Config struct {

        Name        string `xml:"name,omitempty"`

        Description string `xml:"description,omitempty"`

        Enabled     bool   `xml:"enabled,omitempty"`

    } `xml:"config,omitempty"`

}

type OpenConfigInterfaces struct {

    XMLName   xml.Name              `xml:"interfaces"`

    Interface []OpenConfigInterface `xml:"interface,omitempty"`

}

type NetconfConfig struct {

    XMLName    xml.Name `xml:"config,omitempty"`

    Interfaces OpenConfigInterfaces

}

type NetconfData struct {

    XMLName    xml.Name `xml:"data,omitempty"`

    Interfaces OpenConfigInterfaces

}



type RPCResponse struct {

    XMLName xml.Name `xml:"rpc-reply"`

    Data    NetconfData

}

Type OpenConfigInterfaces follows OpenConfig Interfaces YANG module. We don’t implement all the keys, but only the tiny structure we need to configure network device for our use case. On top of examples of XML conversation we did before, there are a few new concepts we’d like to highlight:

1. You may see the instruction “omitempty” after mapping the field to XML key. This instruction means that if we don’t set explicitly any value in struct, there will be no key created in XML struct. This is very important as we want to change value of some keys within the struct, no the entire struct.
2. You see field “XMLName” of xml.Name data type. This data type is used, when we need to attribute key to some namespace, as it is struct with a few fields and “Namespace” is one of them.

We put these data types to a separate module to show that you can split code in Go (Golang) across multiple files and so long they have the same “package” value, they all will be built together.

Now let’s take a look in the code of application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
/* From Python to Go: Go: 017 - NETCONF. */



package main



// Imports

import (

    "encoding/xml"

    "flag"

    "fmt"

    "log"

    "os"

    "time"



    "github.com/google/go-cmp/cmp"

    "github.com/scrapli/scrapligo/driver/netconf"

    "github.com/scrapli/scrapligo/driver/options"

    "gopkg.in/yaml.v3"

)



// Types and Receivers

type Arguments struct {

    /* Class to starte CLI arguments */

    Inventory string

}



type Crendetials struct {

    /* Struct to store credentials. */

    Username string

    Password string

}



type Instruction struct {

    Command OpenConfigInterfaces

    Config  NetconfConfig

}



type Result struct {

    /* Struct to store command execution result. */

    Instruction Instruction

    Diff        string

    Timestamp   time.Time

}



type Device struct {

    /* Struct to interact with netowrk device. */

    Hostname    string `yaml:"hostname"`

    IpAddress   string `yaml:"ip_address"`

    Platform    string `yaml:"platform"`

    Crendetials Crendetials

    Result      []Result

}



func (d *Device) executeChange(i Instruction) {

    /* Method to execute command */



    // Get netowrk driver

    dr, err := netconf.NewDriver(

        (*d).IpAddress,

        options.WithAuthNoStrictKey(),

        options.WithAuthUsername(d.Crendetials.Username),

        options.WithAuthPassword(d.Crendetials.Password),

    )

    if err != nil {

        log.Fatalln("failed to fetch network driver from the platform; error ", err)

    }



    // Open session

    err = dr.Open()

    if err != nil {

        log.Fatalln("failed to open driver; error: ", err)

    }

    defer dr.Close()



    // Get change before start

    before, err := dr.GetConfig("running")

    if err != nil {

        log.Fatalln("failed to send command; error: ", err)

    }

    beforeStruct := RPCResponse{}

    err = xml.Unmarshal((*before).RawResult, &beforeStruct)

    if err != nil {

        log.Panic("Cannot parse received response: ", err)

    }



    // Apply change

    configXmlBs, err := xml.Marshal(i.Config)

    if err != nil {

        log.Fatalln("Cannot convert config to XML: ", err)

    }

    configXmlStr := string(configXmlBs)



    changeResponse, err := dr.EditConfig("candidate", configXmlStr)

    if err != nil {

        log.Fatalln("failed to send config; error: ", err)

    } else if changeResponse.Failed != nil {

        log.Fatalln("Return error from device during config; error: ", err)

    }



    commitResponse, err := dr.Commit()

    if err != nil {

        log.Fatalln("failed to commit config; error: ", err)

    } else if commitResponse.Failed != nil {

        log.Fatalln("return error from device during commit; error: ", err)

    }



    // Get state after change

    after, err := dr.GetConfig("running")

    if err != nil {

        log.Fatalln("failed to send command; error: ", err)

    }

    afterStruct := RPCResponse{}

    err = xml.Unmarshal((*after).RawResult, &afterStruct)

    if err != nil {

        log.Panic("Cannot parse received response: ", err)

    }



    // Diff

    diff := cmp.Diff(beforeStruct, afterStruct)



    // Update the result

    (*d).Result = append((*d).Result, Result{

        Instruction: i,

        Diff:        diff,

        Timestamp:   time.Now(),

    })

}



// Functions

func readArgs() Arguments {

    /* Helper function to read CLI arguments */

    result := Arguments{}



    flag.StringVar(&result.Inventory, "i", "", "Path to the inventory file")



    flag.Parse()



    return result

}



func loadInventory(p string) *[]Device {

    /* Function to load inventory data. */



    // Open file

    bs, err := os.ReadFile(p)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Load inventory

    result := &[]Device{}



    err = yaml.Unmarshal(bs, result)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Return result

    return result

}



func getCredentials() Crendetials {

    /* Function to get credentials. */

    return Crendetials{

        Username: os.Getenv("AUTOMATION_USER"),

        Password: os.Getenv("AUTOMATION_PASS"),

    }

}



// Main

func main() {

    /* Core logic */

    // Read CLI arguments

    cliArgs := readArgs()



    // Get credentials

    sshCreds := getCredentials()



    // Load inventory

    inventory := loadInventory(cliArgs.Inventory)



    // Config

    instruction := Instruction{

        Command: OpenConfigInterfaces{

            XMLName: xml.Name{

                Space: "http://openconfig.net/yang/interfaces",

                Local: "interfaces",

            },

        },

        Config: NetconfConfig{

            Interfaces: OpenConfigInterfaces{

                XMLName: xml.Name{

                    Space: "http://openconfig.net/yang/interfaces",

                    Local: "interfaces",

                },

                Interface: make([]OpenConfigInterface, 0),

            },

        },

    }

    instruction.Config.Interfaces.Interface = append(instruction.Config.Interfaces.Interface, OpenConfigInterface{

        Name: "Loopback 23",

        Config: struct {

            Name        string "xml:"name,omitempty""

            Description string "xml:"description,omitempty""

            Enabled     bool   "xml:"enabled,omitempty""

        }{

            Name:        "Loopback 23",

            Description: "Test-netconf-golang-2",

        },

    })



    // Execute commands

    for i := 0; i < len(*inventory); i++ {

        (*inventory)[i].Crendetials = sshCreds

        (*inventory)[i].executeChange(instruction)

    }



    // Print results

    for i := 0; i < len(*inventory); i++ {

        for j := 0; j < len((*inventory)[i].Result); j++ {

            fmt.Printf(

                "Config: %v\nImpact: %v\nTimestamp: %v\n",

                (*inventory)[i].Result[j].Instruction.Config,

                (*inventory)[i].Result[j].Diff,

                (*inventory)[i].Result[j].Timestamp,

            )

        }

    }

}

We strongly recommend to read previous blogs, if you aren’t sure about some parts of the code, as we will focus our explanation only on what is relevant to NETCONF.

Let’s analyse the NETCONF specifics:

1. Instruction struct uses structs NetconfConfig and OpenConfigInterface as data type.
2. When instruction is created, the corresponding structs are populated as well. Pay attention how we populate slice of interface
3. The receiver function (aka method) executeChange() is modified to perform interaction with network devices using NETCONF:
   • Using function NewDriver() from Netconf Scrapli driver “github.com/scrapli/scrapligo/driver/netconf“, the struct is created, which is used to interact with network devices.
   • Using receiver function GetConfig() of NetconfDriver struct, the actual running configuration is collected.
   • This data is then converted to struct RPCResponse using Unmarshall() function from “encoding/xml” package.
   • Then the configuration for change from NetconfConfig struct is converted to XML string using Marshall() from “encoding/xml” followed by string() type casting, as scrapligo requires string as input and marhsalling gives you “[]byte” type.
   • Configuration is send to network device using EditConfig() receiver function and is applied using Commit().
   • Then configuration is collected and parsed again
4. In contrast to previous blogpost, where compared two slices of strings, here we comparing two structs and, as you will see shortly in the execution snippet, it is really fantastic.

Let’s execute this application written in Go (Golang):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
$ go run . -i ../data/inventory.yaml 

Config: {{ } {{http://openconfig.net/yang/interfaces interfaces} [{Loopback 23 {Loopback 23 Test-netconf-golang-2 false}}]}}

Impact:   main.RPCResponse{

        XMLName: {Space: "urn:ietf:params:xml:ns:netconf:base:1.0", Local: "rpc-reply"},

        Data: main.NetconfData{

                XMLName: {Space: "urn:ietf:params:xml:ns:netconf:base:1.0", Local: "data"},

                Interfaces: main.OpenConfigInterfaces{

                        XMLName: {Space: "http://openconfig.net/yang/interfaces", Local: "interfaces"},

                        Interface: []main.OpenConfigInterface{

                                {Name: "Management1", Config: {Name: "Management1", Enabled: true}},

                                {Name: "Ethernet2", Config: {Name: "Ethernet2", Enabled: true}},

                                {Name: "Ethernet1", Config: {Name: "Ethernet1", Enabled: true}},

                                {

                                        Name: "Loopback23",

                                        Config: struct{ Name string "xml:"name,omitempty""; Description string "xml:"description,omitempty""; Enabled bool "xml:"enabled,omitempty"" }{

                                                Name:        "Loopback23",

-                                               Description: "Test-netconf-python-2",

+                                               Description: "Test-netconf-golang-2",

                                                Enabled:     true,

                                        },

                                },

                                {Name: "Loopback0", Config: {Name: "Loopback0", Enabled: true}},

                                {Name: "Loopback51", Config: {Name: "Loopback51", Description: "pytest-update-test-33"}},

                        },

                },

        },

  }



Timestamp: 2025-03-22 20:14:17.95016661 +0000 GMT m=+3.170702314

You see the structured data following all the marshalling/unmarshalling we do in our code. Also, the comparison part shows you precisely which key has changed and which values were before and after.

Lessons in GitHub

You can find the final working versions of the files from this blog at out GitHub page.

Conclusion

NETCONF/YANG provides a solid programmable framework for network devices management. It is complementing templating and SSH. In fact, I’d use NETCONF/YANG if it is available and will use SSH/CLI only as last resort. In the next blog post we’ll cover one more API, the most modern one, to manage network devices. Take care and good bye.

Support us

Support new interop and automation articles at karneliuk.com
EUR

I want to support with:
9.99 EUR4.99 EUR24.99 EUR49.99 EUR99.99 EUR199.99 EUR

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk 

This blog continues discussion of how to manage devices (network switches and routers, servers, virtual machines, etc) using SSH, which we started in previous blog. In this discussion we’ll cover advanced interaction with devices, which include multiple commands, change of contexts and validations. Let’s dive in.

If You Say “From Python to Go”, Shall I Learn Python?

Each programming language has its strong and weak capabilities. Golang, by virtue being a low level (or at least much lower level compared to Python) is very good, when performance and efficiency are paramount. However, you don’t need it for all applications. Python give quicker time to market, possibility to iteratively develop your code with Jupyter and vast ecosystem of existing libraries. Both programming languages are important and both of them play crucial role in IT and network infrastructure management. So if you are good with Python, learn Go (Golang) using our blog series.

And if you are not, or you want to have good intro to IT and network automation holistically, enroll to our training programms:

We offer the following training programs in network automation for you:

• Zero-to-Hero Network Automation Training
• High-scale automation with Nornir
• Ansible Automation Orchestration with Ansble Tower / AWX
• Expert-level training: Closed-loop Automation and Next-generation Monitoring

During these trainings you will learn the following topics:

• Success and failure strategies to build the automation tools.
• Principles of software developments and the most useful and convenient tools.
• Data encoding (free-text, XML, JSON, YAML, Protobuf).
• Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
• Full configuration templating with Jinja2 based on the source of truth (NetBox).
• Best programming languages (Python, Bash) for developing automation
• The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
• Network automation infrastructure (Linux, Linux networking, KVM, Docker).
• Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
• Collection network data via SNMP and streaming telemetry with Prometheus
• Building API gateways with Python leveraging Fast API
• Integration of alerting with Slack and your own APIs
• … and many more

Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data center networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.

Start your automation training today.

What Are We Going To Talk Today?

As aforementioned, we continue discussion how to interact with IT and network devices (routers, switches, firewalls, servers, virtual machines, containers, etc) using SSH. In previous blog we showed a simple scenario, where you send one command and receive one output. Whilst it was possible to extend it easily for Python, it was more problematic for Go (Golang). In today’s blog we will show how to significantly improve the capability:

1. How is that possible to send multiple commands to the SSH remote host?
2. What are the associated problems?
3. What is general approach to do changes on IT/network equipment with SSH?
4. What are the purpose-built libraries out there to improve experience?

Explanation

When we do real production-grade interaction with remote devices, for example for the purpose of making change, we normally send a lot of commands as well as receiving a lot of output. Lets take a look at few examples:

Configuration of new BGP neighbor

In this change you would need to:

• Create a route policy, which filters incoming/outgoing network reachability information (NLRI) updates (simply, route updates) towards the new neighbor.
• Possibly these route policies will require prefix lists and/or community lists to be created/modified as well.
• Then to enter the BGP routing configuration context, create a neighbor with all relevant details including created policies

This configuration alone will easily be a few dozens of configuration lines long. As we talk though about production-grade changes, you would also want to capture operational state of BGP before and after the change to ensure you achieved the relevant result.

Installation of New Packages in Linux

In this change you would need to:

• Upload the relevant packages on Linux servers
• Unpack them and make relevant permission adoptions
• Perform installation
• Make adjustment to existing services or create a new one.

Same as above for network example, you may need quite a few commands here, as well as logic to capture the state of the Linux before and after the change if you do in production systems.

Ansible is very good for server management, and you can learn it in-dept at our network automation trainings.

Challenges with SSH

We briefly mentioned in the previous blog post, but it is worth reiterating it here as well. The nature of SSH communication requires client to send requests to server, wait for response and then to process it. There could be multiple ways how client can identify if the response is received in full; however, the most robust is to parse the received text from server until the specific CLI pattern is found. Whilst it sounds easy in straightforward, it is a little bit more challenging when dealing with network equipment, as CLI pattern changes during the configuration, when you traverse the configuration contexts, e.g.:

1
2
3
4
karneliuk-router-01> enable

karneliuk-router-01# configure terminal

karneliuk-router-01(config)# interface Loopback 10

karneliuk-router-01(config-if)#

The snippet above is the one you will see in Cisco IOS / IOS XE, Cisco IOS XR and Cisco NX-OS operating system as well as in Arista EOS. Nokia SR OS and Junipers are slightly different, but they have the same idea. This means that your application, which interacts with network devices via SSH be able to dynamically adapt to changing prompt.

Another challenge is, let’s admit, SSH communication happen over network, which is not always stable. The remote host you communicating with also may experience issues, so your application need some timeout handling to detect that communication is not happening anymore; otherwise, there is a to get stuck on a dead session.

All this mean that you need to have good tooling for SSH to handle all these cases.

Purpose-built SSH Libraries

Good news is that such libraries to exist. There is a number of such libraries in Python, less in Go (Golang). There is though one, which is from my experience is quite good and it exists both in Python and Go (Golang). Imagine, same-ish syntax and capabilities in both Python and Go (Golang), isn’t it beautiful?

The library we are going to use in today’s blog is called Scrapli:

• Python library
• Go (Golang) package

The reason we are going to use this library is because it addresses all the aforementioned challenges, has good documentation and extensive support, both from maintainer and from community.

Example

Picture costs thousand words; so we will demonstrate you how to change the script from the previous blog post to be production-grade leveraging scrapli in both Python and Go (Golang). Namely, we would:

1. Capture state of device using specific show command before and after the change.
2. Perform multiline configuration change.
3. Compare two captured state to generate a difference to simplify engineering job analyzing the impact.

Python

There is number of 3rd party libraries, which shall be installed for this code to work:

1
$ pip install pyyaml scrapli

Once libraries are installed, let’s progress with developing our application with Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
"""From Python to Go: Python: 016 - Advanced SSH."""



# Modules

import argparse

import datetime

import os

import sys

import difflib

from dataclasses import dataclass

from typing import List

import yaml

from scrapli import Scrapli





# Classes

@dataclass

class Credentials:

    """Class to store credentials."""

    username: str

    password: str





@dataclass

class Instruction:

    """Class to store instructions."""

    command: str

    config: List[str]





@dataclass

class Result:

    """Class to store command execution data."""

    instruction: Instruction

    diff: str

    timestamp: datetime.datetime





class Device:

    """Class to interact with netowrk device."""

    def __init__(self, hostname: str, ip_address: str, platform: str, credentials: Credentials):

        self.hostname = hostname

        self.ip_address = ip_address

        self.platform = platform

        self.credentials = credentials



        self.results: List[Result] = []



    def execute_change(self, instruction: Instruction) -> None:

        """Method to execute change."""



        # Connect to device

        with Scrapli(

            host=self.ip_address,

            auth_username=self.credentials.username,

            auth_password=self.credentials.password,

            platform=self.platform,

            auth_strict_key=False,

        ) as conn:

            # Get state before change

            before = conn.send_command(instruction.command)



            # Apply change

            conn.send_configs(instruction.config)



            # Get state after change

            after = conn.send_command(instruction.command)



            # Diff

            diff = "\n".join(

                difflib.context_diff(

                    before.result.splitlines(),

                    after.result.splitlines(),

                    lineterm="",

                )

            )



            self.results.append(

                Result(

                    instruction=instruction,

                    diff=diff,

                    timestamp=datetime.datetime.now(),

                )

            )





# Functions

def read_args() -> argparse.Namespace:

    """Helper function to read CLI arguments."""

    parser = argparse.ArgumentParser(description="User input.")

    parser.add_argument("-i", "--inventory", type=str, help="Path to inventory file.")

    return parser.parse_args()





def load_inventory(filename: str, credentials: Credentials) -> List[Device]:

    """Function to load inventory data."""

    # Open file

    try:

        with open(filename, "r", encoding="utf-8") as f:

            data = yaml.safe_load(f)



    except FileNotFoundError as e:

        print(e)

        sys.exit(1)



    # Populate list of devices

    result = []

    for device in data:

        result.append(Device(credentials=credentials, **device))



    return result





def get_credentials() -> Credentials:

    """Function to get credentials."""

    username = os.getenv("AUTOMATION_USER")

    password = os.getenv("AUTOMATION_PASS")

    return Credentials(username, password)





# Main code

if __name__ == "__main__":

    # Read CLI arguments

    args = read_args()



    # Get credentials

    credentials = get_credentials()



    # Load inventory

    devices = load_inventory(args.inventory, credentials=credentials)



    # Config

    instruction = Instruction(

        command="show interfaces description",

        config=["interface Loopback 23", "description Test"],

    )



    # Execute command

    for device in devices:

        device.execute_change(instruction)



    # Print results

    for device in devices:

        print(f"Device: {device.hostname}")

        for result in device.results:

            print(f"Config: {result.instruction.config}", f"Impact: {result.diff}", f"Timestamp: {result.timestamp}", sep="\n")

It is strongly recommended to read previous blog post about SSH, as there is a number of concepts and structures explained there, which are omitted in this blog post for brevity.

The key new concepts are the following:

1. Two new libraries are imported:
   • scrapli for advanced SSH interaction with remote network devices or servers
   • difflib to compare two text files (output of captured states) and outline differences
2. The new data class “Instruction” is introduced to store both the validation command as well as configuration, which is to be applied to network devices. In this example this class is populated with:
   • command “show interfaces description” to get the description of all the interfaces configured on the target network device.
   • config commands, which is a list of strings, which contain the configuration , which shall be pushed to network devices.
3. From the previous example, method “execute_command()” is renamed to “execute_change()” and it expects now the object of Instruction class as an argument.
4. Within this function we do the interaction with network device:
   • Using class “Scrapli()” we create object for SSH connectivity with the device. In addition to IP address/FQDN and credentials, it requires to provide a platform value, which shall be from the core supported ones.
   • Object is created using “with … as …” context manager, which means the SSH channel will be “open()” and “close()” automatically.
   • To send non-configuration command, method “send_command()” is used. If there are multiple commands are to be sent, then “send_commands()” can be used instead.
   • To send configuration, method “send_config()” is used. If there are multiple commands are to be sent, then “send_configs()” can be used instead.
   • Those four commands return Response (or MultiResponse) objects from “scrapli” library, which has field “result“, storing the string output from network device.
   • Finally, using function “context_diff()” from “difflib” library, the difference between two outputs is identified.
   • Afterwards the result are stored on in the modified Result, which then is printed back in the main body

To execute this script, much like in the previous blog, you need to set environment variables with credentials (username and password), so that your application can read it:

1
2
$ export AUTOMATION_USER='karneliuk'

$ export AUTOMATION_PASS='***'

Then execute the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ python main.py -i ../data/inventory.yaml

UserWarning:



**************************************************************************************************************************** Authentication Warning! *****************************************************************************************************************************

                                    scrapli will try to escalate privilege without entering a password but may fail.

Set an 'auth_secondary' password if your device requires a password to increase privilege, otherwise ignore this message.

**********************************************************************************************************************************************************************************************************************************************************************************



  warn(warning_message)

Device: dev-pygnmi-eos-001

Command: ['interface Loopback 23', 'description Test']

Impact: ***

---

***************

*** 2,6 ****

--- 2,7 ----

  Et1                            up             up

  Et2                            up             up

  Lo0                            up             up

+ Lo23                           up             up                 Test

  Lo51                           admin down     down               pytest-update-test-33

  Ma1                            up             up

Timestamp: 2025-03-07 21:24:11.267377

Execution of change went successfully and you see diff, which shows that new interface was added (identified by “+” symbol).

Now let’s implement the same use case in Go (Golang).

Go (Golang)

The following packages are to be installed:

1
2
3
$ go get github.com/google/go-cmp/cmp

$ go get gopkg.in/yaml.v3

$ go get github.com/scrapli/scrapligo

In addition scrapli and yaml, we need to installed 3rd go-cmp party for comparing texts. It is possible to complete use case without it; however, it will take more time to write logic for comparing texts.

Now the code for this application, which performs advanced SSH management:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
/* From Python to Go: Go: 015 - Basic SSH. */



package main



import (

    "flag"

    "fmt"

    "log"

    "os"

    "strings"

    "time"



    "github.com/google/go-cmp/cmp"

    "github.com/scrapli/scrapligo/driver/options"

    "github.com/scrapli/scrapligo/platform"

    "gopkg.in/yaml.v3"

)



// Imports



// Types and Receivers

type Arguments struct {

    /* Class to starte CLI arguments */

    Inventory string

}



type Crendetials struct {

    /* Struct to store credentials. */

    Username string

    Password string

}



type Instruction struct {

    Command string

    Config  []string

}



type Result struct {

    /* Struct to store command execution result. */

    Instruction Instruction

    Diff        string

    Timestamp   time.Time

}



type Device struct {

    /* Struct to interact with netowrk device. */

    Hostname    string `yaml:"hostname"`

    IpAddress   string `yaml:"ip_address"`

    Platform    string `yaml:"platform"`

    Crendetials Crendetials

    Result      []Result

}



func (d *Device) executeChange(i Instruction) {

    /* Method to execute command */



    // Get platform

    p, err := platform.NewPlatform(

        (*d).Platform,

        (*d).IpAddress,

        options.WithAuthNoStrictKey(),

        options.WithAuthUsername(d.Crendetials.Username),

        options.WithAuthPassword(d.Crendetials.Password),

    )

    if err != nil {

        log.Fatalln("failed to create platform; error: ", err)

    }



    // Get netowrk driver

    dr, err := p.GetNetworkDriver()

    if err != nil {

        log.Fatalln("failed to fetch network driver from the platform; error ", err)

    }



    // Open session

    err = dr.Open()

    if err != nil {

        log.Fatalln("failed to open driver; error: ", err)

    }

    defer dr.Close()



    // Get change before start

    before, err := dr.SendCommand(i.Command)

    if err != nil {

        log.Fatalln("failed to send command; error: ", err)

    }



    // Apply change

    _, err = dr.SendConfigs(i.Config)

    if err != nil {

        log.Fatalln("failed to send config; error: ", err)

    }



    // Get state after change

    after, err := dr.SendCommand(i.Command)

    if err != nil {

        log.Fatalln("failed to send command; error: ", err)

    }



    // Diff

    diff := cmp.Diff(strings.Split(before.Result, "\n"), strings.Split(after.Result, "\n"))



    // Update the result

    (*d).Result = append((*d).Result, Result{

        Instruction: i,

        Diff:        diff,

        Timestamp:   time.Now(),

    })

}



// Functions

func readArgs() Arguments {

    /* Helper function to read CLI arguments */

    result := Arguments{}



    flag.StringVar(&result.Inventory, "i", "", "Path to the inventory file")



    flag.Parse()



    return result

}



func loadInventory(p string) *[]Device {

    /* Function to load inventory data. */



    // Open file

    bs, err := os.ReadFile(p)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Load inventory

    result := &[]Device{}



    err = yaml.Unmarshal(bs, result)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Return result

    return result

}



func getCredentials() Crendetials {

    /* Function to get credentials. */

    return Crendetials{

        Username: os.Getenv("AUTOMATION_USER"),

        Password: os.Getenv("AUTOMATION_PASS"),

    }

}



// Main

func main() {

    /* Core logic */

    // Read CLI arguments

    cliArgs := readArgs()



    // Get credentials

    sshCreds := getCredentials()



    // Load inventory

    inventory := loadInventory(cliArgs.Inventory)



    // Config

    instruction := Instruction{

        Command: "show interfaces description",

        Config: []string{

            "interface Loopback 23",

            "description Go_Test_2",

        },

    }



    // Execute commands

    for i := 0; i < len(*inventory); i++ {

        (*inventory)[i].Crendetials = sshCreds

        (*inventory)[i].executeChange(instruction)

    }



    // Print results

    for i := 0; i < len(*inventory); i++ {

        for j := 0; j < len((*inventory)[i].Result); j++ {

            fmt.Printf(

                "Config: %v\nImpact: %v\nTimestamp: %v\n",

                (*inventory)[i].Result[j].Instruction.Config,

                (*inventory)[i].Result[j].Diff,

                (*inventory)[i].Result[j].Timestamp,

            )

        }

    }

}

It is strongly encouraged to read the previous blog about basic SSH interaction with network devices and servers with Python and Go (Golang) to get grasp on the code before reading explanation below.

The details of what’s new:

1. The relevant third party libraries (ones mentioned above) are imported.
2. Struct Instruction is added with the same keys and data types as Instruction data class in Python.
3. Receiver function “func (d *Device) executeChange(i Instruction)” replaced “func (d *Device) executeCommand(i string)“, following the same logic as explained in Python section.
4. Within that function interaction with network device is done:
   • First of all, using function NewPlatform(), the pointer to struct for platform is created. Same as with Scrapli() class in Python, NewPlatform requires actual platform name, which is supported by Scrapli package.
   • From the Platform pointer the new device driver is obtained using receiver function (method) GetNetworkDriver(), which is also pointer.
   • Using Open() function the SSH session is established and using the “defer Close()” the closure is scheduled right before the exit out of the function.
   • Much as in Python, there are four main methods to interact with the remote host:
     • “func (d *network.Driver) sendCommand(command string)” to execute single non-configuration command.
     • “func (d *network.Driver) sendCommands(commands []string)” to execute multiple non-configuration command.
     • “func (d *network.Driver) sendConfig(config string)” to execute single configuration command.
     • “func (d *network.Driver) sendConfigs(config []string)” to execute multiple configuration command.
   • All these methods returns pointer to Response struct, which has key “Result” storing output from network device.
   • Using “Diff()” function from “go-cmp” package the two slices of strings are compared and it is identified what is different.
   • Then results are printed.

You shall have credentials set in your environment by now. See Python part for details.

Execution of this newly developed Go (Golang) application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
 go run . -i ../data/inventory.yaml 

Config: [interface Loopback 23 description Go_Test_23]

Impact:   []string{

        ... // 2 identical elements

        "Et2                            up             up",

        "Lo0                            up             up",

        strings.Join({

                "Lo23                           up             up                ",

                " Go_Test_2",

+               "3",

        }, ""),

        "Lo51                           admin down     down              "...,

        "Ma1                            up             up",

  }



Timestamp: 2025-03-08 17:36:35.170185362 +0000 GMT m=+2.091754052

The output is slightly more cryptic here compared to Python:

• In Python the entire line was highlighted, which was new or different
• In Go (Golang) only the new characters are shown

Lessons in GitHub

You can find the final working versions of the files from this blog at out GitHub page.

Conclusion

In this blog post you have learned two things, critical to IT and network automation: what is the correct process to implement change including pre and post checks as well how to interact with remote hosts via SSH at advanced level. Coupling this knowledge with templating, and in general foundational skills you learn before in this blog series, you are in a good position to start automating. Take care and good bye.

Support us

Support new interop and automation articles at karneliuk.com
EUR

I want to support with:
9.99 EUR4.99 EUR24.99 EUR49.99 EUR99.99 EUR199.99 EUR

P.S.

If you have further questions or you need help with your networks, I’m happy to assist you, just send me message. Also don’t forget to share the article on your social media, if you like it.

BR,

Anton Karneliuk 

As mentioned in the previous blogpost, we started talking about practical usage of Python and Go (Golang) for network and IT infrastructure automation. Today we’ll take a look how we can interact with any SSH-speaking device, whether it is a network device, server, or anything else.

You Put So Much Content For Free Online, Why To Join Trainings Then?

Our ultimate goal is to make you successful with software developing for IT infrastructure management. Out blogs are the first step so that you can get up to speed if you already well equipped with fundamentals as protocols, data formats, etc. We believe that sharing is caring, hence we share back our knowledge with you, so that your path could be a little bit easier and quicker, so that you have more time to focus on what matters. If that’s enough for you to move forward, that’s great.

At the same time, if you feel you need more, you want to have finely-curated labs, slack support and deep dive not just in coding, but really in fundamentals, our training programs are here for you:

We offer the following training programs in network automation for you:

• Zero-to-Hero Network Automation Training
• High-scale automation with Nornir
• Ansible Automation Orchestration with Ansble Tower / AWX
• Expert-level training: Closed-loop Automation and Next-generation Monitoring

During these trainings you will learn the following topics:

• Success and failure strategies to build the automation tools.
• Principles of software developments and the most useful and convenient tools.
• Data encoding (free-text, XML, JSON, YAML, Protobuf).
• Model-driven network automation with YANG, NETCONF, RESTCONF, GNMI.
• Full configuration templating with Jinja2 based on the source of truth (NetBox).
• Best programming languages (Python, Bash) for developing automation
• The most rock-solid and functional tools for configuration management (Ansible) and Python-based automation frameworks (Nornir).
• Network automation infrastructure (Linux, Linux networking, KVM, Docker).
• Orchestration of automation workflows with AWX and its integration with NetBox, GitHub, as well as custom execution environments for better scalability.
• Collection network data via SNMP and streaming telemetry with Prometheus
• Building API gateways with Python leveraging Fast API
• Integration of alerting with Slack and your own APIs
• … and many more

Moreover, we put all mentions technologies in the context of real use cases, which our team has solved and are solving in various projects in the service providers, enterprise and data center networks and systems across the Europe and USA. That gives you opportunity to ask questions to understand the solutions in-depth and have discussions about your own projects. And on top of that, each technology is provided with online demos and labs to master your skills thoroughly. Such a mixture creates a unique learning environment, which all students value so much. Join us and unleash your potential.

Start your automation training today.

What Are We Going To Talk Today?

Nowadays, SSH is the most popular protocol to interact with network devices as well as compute nodes, such as servers, virtual machines (VMs) and to a degree even containers. System and network engineers use SSH daily to get data from devices for analysis, for health checking or to perform configuration change. Having such a wide spread, it is no wonder that we put it first to the queue of network management.

Today we are going to discuss:

1. How to connect to network devices using SSH with Python and Go (Golang)?
2. How to retrieve operational status?

Explanation

Interaction with network and IT infrastructure devices via SSH meaning opening the interactive terminal session remotely, which gives look and feel as if you are directly connected to the device console. This interactive session lasts for as long as you interact with the devices and involves exchange of data between you / your application and destination node, which raises two important questions to address:

1. How to ensure that you is you when session is established? In other words, there shall be some authentication and authorization mechanisms in place.
2. How to ensure that data exchange between your host and node, even if eavesdropped, aren’t altered and/or its content visible to the attacker? This requires some encryption techniques.

Both these issues were not addressed in telnet, which made it very unfavorable for network and IT infrastructure management, although this is typically the first protocol engineers learn to use. And both these issues are addressed in SSH.

Once the SSH session is established between the destination node and your application, free form text is sent back and force. Typically, you send some command and receive some response to them. Albeit seems extremely simple, the important part of this process is to have mechanism, which will detect, when you have received the output in full so that you can send new instruction. Some SSH client have this functionality built-in, whilst others would require you do it yourself.

Join Zero-to-Hero Network Automation Training to learn the details of SSH, which are important for building scalable network and IT automation solutions.

Examples

In this blog we’ll show the basics of SSH interaction. Basics mean that we will actually execute one off commands and collect their output, whilst in the next blog post we will show advanced interaction with network and IT infrastructure devices using SSH.

The today lab’s scenario:

• Connect to network device using SSH.
• Send command and receive output in full.
• Display the collected information in CLI (stdout).

Python

First comes Python as our series is called from Python to Go. Before we can create an appliation in Python, which will be connecting to remote devices using SSH, we need to install corresponding package. We’ll use today paramiko, which is by far the most popular and widely used SSH library. Some other great libraries, such as netmiko is based on paramiko. We’ll also install pyyaml to parse input YAML file:

1
$ pip install pyyaml paramiko

Once libraries are available, we can develop our software with Python:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
"""From Python to Go: Python: 015 - Basic SSH."""



# Modules

import argparse

import datetime

import os

import sys

import time

import re

from dataclasses import dataclass

from typing import List

import yaml

import paramiko





# Classes

@dataclass

class Credentials:

    """Class to store credentials."""

    username: str

    password: str





@dataclass

class Result:

    """Class to store command execution data."""

    command: str

    output: str

    timestamp: datetime.datetime





class Device:

    """Class to interact with netowrk device."""

    def __init__(self, hostname: str, ip_address: str, credentials: Credentials):

        self.hostname = hostname

        self.ip_address = ip_address

        self.credentials = credentials



        self.results: List[Result] = []



    def execute_command(self, command: str) -> None:

        """Method to execute a command."""

        # Create a new SSH client

        client = paramiko.SSHClient()

        client.set_missing_host_key_policy(paramiko.AutoAddPolicy())



        # Connect to the device

        client.connect(

            self.ip_address,

            username=self.credentials.username,

            password=self.credentials.password,

            look_for_keys=False,

            allow_agent=False,

        )



        # Invoke the session

        session = client.invoke_shell()

        session.recv(65535)



        # Execute the command

        session.send(command + "\n")

        output = ""

        print(f"{regex=}")

        while not regex.search(output):

            time.sleep(.1)

            output += session.recv(65535).decode("utf-8")



        # Store the result

        self.results.append(Result(command, output, datetime.datetime.now()))



        # Close the connection

        session.close()





# Functions

def read_args() -> argparse.Namespace:

    """Helper function to read CLI arguments."""

    parser = argparse.ArgumentParser(description="User input.")

    parser.add_argument("-i", "--inventory", type=str, help="Path to inventory file.")

    return parser.parse_args()





def load_inventory(filename: str, credentials: Credentials) -> List[Device]:

    """Function to load inventory data."""

    # Open file

    try:

        with open(filename, "r", encoding="utf-8") as f:

            data = yaml.safe_load(f)



    except FileNotFoundError as e:

        print(e)

        sys.exit(1)



    # Populate list of devices

    result = []

    for device in data:

        result.append(Device(credentials=credentials, **device))



    return result





def get_credentials() -> Credentials:

    """Function to get credentials."""

    username = os.getenv("AUTOMATION_USER")

    password = os.getenv("AUTOMATION_PASS")

    return Credentials(username, password)





# Main code

if __name__ == "__main__":

    # Read CLI arguments

    args = read_args()



    # Get credentials

    credentials = get_credentials()



    # Load inventory

    devices = load_inventory(args.inventory, credentials=credentials)



    # Execute command

    for device in devices:

        device.execute_command("show version")



    # Print results

    for device in devices:

        print(f"Device: {device.hostname}")

        for result in device.results:

            print(f"Command: {result.command}", f"Output: {result.output}", f"Timestamp: {result.timestamp}", sep="\n")

This code is long and it is assumes that you have read and practiced all our previous blog posts in “From Python to Go” series or have relevant knowledge already.

We’ll focus our explanation on “Device” class as all other things were already explained:

• Object-oriented programming and classes
• Reading environment variables
• Reading files
• Parsing YAML
• Processing CLI arguments
• Handling exceptions

Break down for this blog post:

1. All the user inputs (credentials via environment variables and inventory via YAML file) is passed for class Device, which contains method “execute_command” for interacting with network devices.
2. This method takes a single external argument, which is the command, which shall be executed on the remote device and returns the result of execution.
3. Within this method, the new SSH client is created via instantiation of object of “SSHClient” class from paramiko library.
4. Once the client is created, the policy policy is set to automatically add SSH keys for device, which allows to connect to devices for the first time, before its SSH key is known to your host “client.set_missing_host_key_policy(paramiko.AutoAddPolicy())“.
5. Then the SSH session is established to the remote node (network device, server) using “connect()” method of “SSHClient” class. This method takes as arguments connectivity details, such as target node IP address or FQDN, port, credentials, etc.
6. After the SSH session is established, you need to open interactive shell using “invoke_shell()” method, which allows you to send and receive information from the endpoint. The interactive shell is a new object, which you need to store.
7. Finally, you can send and receive messages. Be mindful though that this is a raw SSH session and you need to handle sending and receiving on buffer level, using “send()” and “recv()” methods. For the latter you need to specify how many bytes you want to read from the buffer.
8. The tricky part is to detect when the output of your command is received in full. In network devices and in servers this is typically signaled with prompt with hostname followed by some special character. So the following approach is taken:
   • Regular expression is created, which is looking for hostname followed by “#” or “>” symbol.
   • This regular expression is capable to read multiline strings.
   • the output is read from buffer and is added to the received data. It is then checked against regular expression and if there is no match, the data read again. It is repeated unlimited amount of time until the pattern is matched.
9. Then result is sent back to the caller.

Let’s test this application. First of all, here is the inventory:

1
2
3
4
$ cat data/inventory.yaml

---

- hostname: dev-pygnmi-eos-001

  ip_address: 192.168.51.79

Set the credentials to environment:

1
2
$ export AUTOMATION_USER="user"

$ export AUTOMATION_PASS="password"

And execute the Python application:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
$ python main.py -i ../data/inventory.yaml

Device: dev-pygnmi-eos-001

Command: show version

Output: show version

show version

dev-pygnmi-eos-001>show version

 vEOS

Hardware version: 

Serial number: 

Hardware MAC address: b239.c742.24ec

System MAC address: b239.c742.24ec



Software image version: 4.26.0.1F

Architecture: i686

Internal build version: 4.26.0.1F-21994874.42601F

Internal build ID: e41b7ab2-f5ed-45cb-ba9c-f320cb81332f



Uptime: 0 weeks, 6 days, 7 hours and 58 minutes

Total memory: 2006640 kB

Free memory: 1188532 kB



dev-pygnmi-eos-001>

Timestamp: 2025-02-22 17:31:21.631414

It works!

Go (Golang)

Same as with Python, we need to install extra package to be able to establish SSH connectivity, which is called “crypto/ssh” and it is also one of the foundational ones. Same as with Python, we install package yaml for parsing YAML files:

1
2
$ go get golang.org/x/crypto/ssh

$ go get gopkg.in/yaml.v3

And here is code for our application written in Go (Golang) to communicate with network devices (switches, routers, firewalls, load balancers) as well as other IT infrastructure devices (servers, virtual machines, containers, etc) using SSH:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
/* From Python to Go: Go: 015 - Basic SSH. */



package main



import (

    "bytes"

    "flag"

    "fmt"

    "log"

    "os"

    "time"



    "golang.org/x/crypto/ssh"

    "gopkg.in/yaml.v3"

)



// Imports



// Types and Receivers

type Arguments struct {

    /* Class to starte CLI arguments */

    Inventory string

}

type Crendetials struct {

    /* Struct to store credentials. */

    Username string

    Password string

}



type Result struct {

    /* Struct to store command execution result. */

    Command   string

    Output    string

    Timestamp time.Time

}



type Device struct {

    /* Struct to interact with netowrk device. */

    Hostname    string `yaml:"hostname"`

    IpAddress   string `yaml:"ip_address"`

    Crendetials Crendetials

    Result      []Result

}



func (d *Device) executeCommand(c string) {

    /* Method to execute command */

    interactiveAuth := ssh.KeyboardInteractive(

        func(user, instruction string, questions []string, echos []bool) ([]string, error) {

            answers := make([]string, len(questions))

            for i := range answers {

                answers[i] = (*d).Crendetials.Password

            }



            return answers, nil

        },

    )



    // Create a new SSH client

    sshClientConfig := &ssh.ClientConfig{

        User:            (*d).Crendetials.Username,

        Auth:            []ssh.AuthMethod{interactiveAuth},

        HostKeyCallback: ssh.InsecureIgnoreHostKey(),

    }

    sshClient, err := ssh.Dial("tcp", fmt.Sprintf("%v:22", (*d).IpAddress), sshClientConfig)

    if err != nil {

        log.Fatalln("Failed to dial: ", err)

    }

    defer sshClient.Close()



    // Create session

    session, err := sshClient.NewSession()

    if err != nil {

        log.Fatalln("Failed to open the session: ", err)

    }

    defer session.Close()



    // Execute the command

    buffer := bytes.Buffer{}

    session.Stdout = &buffer

    if err := session.Run(c); err != nil {

        log.Fatalln("Failed to execute command: ", err)

    }



    // Update the result

    (*d).Result = append((*d).Result, Result{

        Command:   c,

        Output:    buffer.String(),

        Timestamp: time.Now(),

    })

}



// Functions

func readArgs() Arguments {

    /* Helper function to read CLI arguments */

    result := Arguments{}



    flag.StringVar(&result.Inventory, "i", "", "Path to the inventory file")



    flag.Parse()



    return result

}



func loadInventory(p string) *[]Device {

    /* Function to load inventory data. */



    // Open file

    bs, err := os.ReadFile(p)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Load inventory

    result := &[]Device{}



    err = yaml.Unmarshal(bs, result)

    if err != nil {

        fmt.Println("Get error ", err)

        os.Exit(1)

    }



    // Return result

    return result

}



func getCredentials() Crendetials {

    /* Function to get credentials. */

    return Crendetials{

        Username: os.Getenv("AUTOMATION_USER"),

        Password: os.Getenv("AUTOMATION_PASS"),

    }

}



// Main

func main() {

    /* Core logic */

    // Read CLI arguments

    cliArgs := readArgs()



    // Get credentials

    sshCreds := getCredentials()



    // Load inventory

    inventory := loadInventory(cliArgs.Inventory)



    // Execute commands

    for i := 0; i < len(*inventory); i++ {

        (*inventory)[i].Crendetials = sshCreds

        (*inventory)[i].executeCommand("show version")

    }



    // Print results

    for i := 0; i < len(*inventory); i++ {

        for j := 0; j < len((*inventory)[i].Result); j++ {

            fmt.Printf(

                "Command: %v\nOutput: %v\nTimestamp: %v\n",

                (*inventory)[i].Result[j].Command,

                (*inventory)[i].Result[j].Output,

                (*inventory)[i].Result[j].Timestamp,

            )

        }

    }

}