TLDR: I developed a new plugin for NetBox to manage phone numbers and more. Using NetBox as Voice and UC Source of Truth might be a good idea.

How and What Enterprises Document

I have seen many examples of how the documentation can be organized since 2011 when I started my career. I’ve been managing and consulting on Voice and UC deployments serving thousands of users and consisting of hundreds of voice devices and circuits in total. However, regardless of the region and size, they all shared one thing in common: the documentation related to Voice and UC relied on Microsoft Office and PDF files stored in a more or less structured fashion.

It is a good question, how representative this sample is. Based on discussions with other engineers, this is quite typical. Dedicated domain-specific documenting solutions for Voice and UC are not that common in the enterprise sector. “What is the problem with that?” you may ask. Let’s try to figure this out.

To begin with, I would break the pieces of the information we are usually interested in and keeping track of as follows:

1. Solution design documents and implementation summaries. How it was designed and expected to work. You can typically find such files in most organizations.
2. Network and domain-specific diagrams. How it looks visually. This might be a part of #1.
3. Inventory information. What physical and virtual boxes we have: Voice and UC equipment, virtual machines, PRI and DSP modules, etc. Related models, part IDs, serial numbers typically fall into this category.
4. Cabling and device interconnection details. Cable paths across patch panels and between ports. Those who spent hours with a tone generator and wire tracker trying to identify the actual cross-connections from a traditional PBX will likely sigh at this point. *Sighs*. In VoIP environments, this information is still useful to know.
5. IP addressing. How we assign IP-addresses to our devices.
6. Voice Circuits. What voice interconnections we have: SIP-trunks, PRIs, CO lines. It is necessary to know what their requirements are and how many lines they offer.
7. Phone numbers. What PSTN pools and internal extensions we have.
8. Call routing. How we route phone numbers and patterns.
9. Configuration details and templates. What configuration elements translate #1 into a working infrastructure. The actual configurations on the devices and VMs are often the only source of such information.
10. Management access details. How do we get access to our infrastructure elements.
11. External contracts and agreements. What external services we have and what we billed for. Searching phone numbers through PDF scans of the service provider agreements is adventurous. *Sighs again*.

The list demonstrates some general categories. It might change from one organization to another. There is no single format convention nor standard templates across different organizations. Furthermore, some additional challenges arise with regular text files and spreadsheets:

1. There are no clear relations between each piece of the information. Especially cross-document.
2. It is hard to search and index the information across distributed files.
3. It heavily relies on your knowledge of each file’s location and purpose.
4. Some information may be duplicated across the documents.
5. It is hard to validate the information and compare it with the actual infrastructure state.
6. Reporting requires extra effort and manual data transformation and analysis.
7. A whole picture may be missing. UC and Network teams may operate independently. The larger organization is, the higher chances are. Cross-functional tasks like setting up and maintaining Voice QoS may become much harder in such cases.

The information we store in our documentation should be up-to-date and consistent to help us operate our infrastructures reliably and efficiently. Some documentation is usually better than no documentation at all. However, blindly trusting the documentation may lead to even worse scenarios. The need to double-check the actual state every time eventually pushes us to avoid opening to the documentation and forget to update it afterwards. As long as the configuration changes can be applied unreflected in the documentation, sooner or later, they will.

Text files and spreadsheets are not necessarily a poor choice. General design documents are more self-sufficient and tend to stay valid in a long-term perspective. Regular file formats are typically acceptable and suitable for them. Frequently updated operational documentation, in contrast, requires different approaches to overcome the limitations listed above.

UC Infrastructure-as-Code and UC Source-of-Truth

One of the solutions the industry has come to is Infrastructure-as-Code (IaC) paired with Single-Source of-Truth. It redefines the idea of how we treat our infrastructures and perform changes:

• An infrastructure representation moves toward machine-readable scripted or declarative configurations. Using this approach, you can apply the advantages and best practices of software development, testing, and DevOps.
• Single Source of Truth (or simply Source-of-Truth, SoT) is an idea of identifying a single place for each piece of the information about our infrastructure where it can be found and must be managed. It does not mean all the data has to be stored in a single place. It also does not assume the individual pieces of data can not exist in multiple copies anymore. The key requirement is that the primary source for each piece of data must be unique.
• Changes are applied to the infrastructure components by the teams indirectly. You change the data in your Souce-of-Truth first. Then it gets validated, applied to the IaC model, tested, and eventually delivered to the target devices and VMs by the Automation stack of your choice.

Souce-of-Truth contains a desirable state of your infrastructure. The purpose of the automation stack is to bring the actual configurations in compliance with the Source-of-Truth based on Infrastructure object models you store as code and scripts. It eliminates the problem of inconsistent documentation as now you change your infrastructure by updating the documentation.

It is vital to notice that moving from the traditional workflows to the Source-of-Trust and Infrastructure-as-Code does not has to be done in a single breaking step. It is possible to migrate the processes and procedures gradually. Rebuilding the documentation framework might be a good starting point.

Well, with enough time and perseverance, you can implement the ultimately complex scenarios inside MS Excel as its formulas are Turing-complete. Supporting and maintaining such a solution might be a nightmare. An ability to integrate it with the outer systems will also be limited. That leads us to the need for specialized tools.

These tools and concepts are already widely used by DevOps engineers.
Being applied to Networks, NetDevOps has greatly evolved during the past few years.
I believe all these practices are applicable to the core UC domain as well. Not to mention that UC components might even share the platforms with regular Network functions. Initial provisioning steps for an SBC and a Router or a Switch are very similar. Automating BGP peering is not too far from automating SIP trunking.
Many best practices and tools from the NetDevOps are transferable to the UC domain. One of such tools is NetBox.

Why NetBox

And an obvious question is “What is NetBox?” A comprehensive answer from its developers:

NetBox is an open source web application designed to help manage and document computer networks. Initially conceived by the network engineering team at DigitalOcean, NetBox was developed specifically to address the needs of network and infrastructure engineers. It encompasses the following aspects of network management:

• IP address management (IPAM) - IP networks and addresses, VRFs, and VLANs
• Equipment racks - Organized by group and site
• Devices - Types of devices and where they are installed
• Connections - Network, console, and power connections among devices
• Virtualization - Virtual machines and clusters
• Data circuits - Long-haul communications circuits and providers
• Secrets - Encrypted storage of sensitive credentials

NetBox is designed as the Network Source-of-Truth. Everything in NetBox is an object, and almost everything is accessible via API that provides great opportunities for NetBox integration with external systems. There are a relational database and data model behind NetBox objects. NetBox is well documented and has great community around it. Not surprisingly, the popularity of NetBox keeps growing worldwide. Chances are, your Network team is already aware of NetBox.

You might have noticed that the list above already covers several categories from the list of Voice and UC documentation. PBXs, SBCs, Gateways, MCUs, and many other Voice and UC boxes are Devices. You can usually find them installed into the Equipment Racks below the ToR switches. They are interconnected with Connections and may terminate some Data Circuits carrying Voice signaling and media from the Telephony Service Providers. Some of Voice and UC functions may be running as Virtual Machines. And all (unless you are using electromechanical PBXs) Voice and UC infrastructure components use IP-addresses.

But some critical points like Phone Numbers, Voice Circuits, and Call Routing are missing. Fortunately, NetBox provides extremely powerful extensibility feature called Plugins. It allows you to extend the core NetBox data schema and embed new features. Combined with native NetBox data models, such a plugin would provide you with a full set of necessary abstractions to describe your Voice and UC deployment.

Some of the primary benefits of this approach:

• It simplifies the modeling of the cross-domain relations and dependencies. UC is highly dependant on the underlying Network infrastructure.
• Unified documentation and Source of Truth framework for Network and Unified Communications provides you with full visibility of your infrastructure.
• Common tools may simplify cross-team communications.
• Reporting becomes more agile and powerful.

So I introduce the PhoneBox plugin for NetBox.

PhoneBox Plugin

This plugin is intended to bridge the gap between general NetBox network data models and those of Voice and UC world. I started development recently and already implemented Phone Numbers. It provides some instant value and allows you to manage PSTN DIDs and internal extensions using NetBox. Some feature requests from the NetBox community to add support for such abstractions were already there.

Each Phone Number consists of the following attributes:

• Number – An individual phone number.
• Tenant – A mandatory relation to Netbox Tenant object. It allows for number plan partitioning.
• Description – An optional text description for the number.
• Provider – An optional relation to NetBox native Provider object. It identifies the provider of the number.
• Region – An optional relation to NetBox native Region. It represents the geographic location of the number.
• Forward_To – an optional relation to another Number object. It represents a Call Forwarding scenario.
• Tags – An optional relation to NetBox native tag objects.

The plugin implements all Create, Read, Update, Delete (CRUD) operations for the Phone Numbers via NetBox web interface and REST API. It also supports bulk import from CSV for the Phone Numbers to simplify the migration of your data.

I am planning to add more Voice and UC abstractions to the plugin in the future. The complexity of selecting proper data models that fit into arbitrary infrastructure is hard to underestimate. I am going to write a separate post about it.

Please feel free to leave the comments below or to reach me on social media. Your feedback matters.

Disclaimer

This is a translation of my original article in Russian I posted in June 2018.
If you are willing to help to improve the translation, please DM me.
All listed code is published under MIT license and does not provide guarantees of 
any kind.

The solution is written in Python 3.7.
An understanding of the programming and networking basics is desirable for reading.

If you found a typo, please use Ctrl+Enter or ⌘+Enter to send it to the author.

Task Decomposition and Requirements Analysis

Considering the initial task summary, I would split it into two main parts:

• Extracting the routing tables from the text files into some representation in Python data structures.
• Analyzing routing paths based on that pre-processed routing data.

Such logic separation will allow us to import the routing data from different sources (e.g. APIs or SNPM) and not limiting the potential scope to text file input.

To improve route lookup performance, it is necessary to initialize the files just once on script startup. The solution will support Cisco IOS, IOS-XE, and ASA routing table output format for IPv4. Extensibility logic applies here as well.

As we know, a route selection during routing table lookup relies on a Longest Prefix Match (LPM) logic. Unlike with Access Control lists, it is not enough to pick the first match. We have to find the most specific one.

Fortunately, there are fast algorithms and approaches for LPM calculation. One of them is building a so-called prefix tree (prefix trees may also be referenced to as Subnet Tries, Patricia Tries, or Radix Trees in the general case) based on a routing table.

In prefix trees, the lookup speed does not depend on the tree size (the number of prefixes), it only depends on a tree depth (maximum prefix length) which is a maximum of 32 for IPv4 (programmers would say that the search time complexity is O(k), where k is the maximum prefix length). In other words, route lookup using prefix trees will work with the same average speed on routing tables containing 500 and 500,000 routes.

Initial tree building time does linearly depend on a number of prefixes and their length (programmers would say that the build time complexity is O(n*k) where n is the number of prefixes and k is the maximum prefix length), but we do this just once. Any subsequent search request will work super fast.

A detailed explanation of this algorithm deserves a dedicated article.
Please let me know if you are interested.

As we are not limited in usage of an external dependencies, we can use some existing Python library implementing this. Some of them are:

• SubnetTree. I wrote my original solution based on this library. It works really fast. But its internal code is written in C++ which causes some compatibility issues and dependency hell across operating systems.
• PyTricia. This library looks best in terms of performance and compatibility in early 2021. It is written in C, so it should work more seamlessly in different OSs. So I migrated my solution to this library. Thanks to the selected code design, it was a matter of a few changed lines of code.

We should also keep in mind that the analyzed network segment may contain routing loops. They should be detected. It should not break the script execution.
The routers may also have no route to the destination. That’s another point to note.

Routing tables for VRFs, if present, should be saved into dedicated text files as each VRF instance represents a separate logical device from the routing and topology perspective.

Hardware performance limitations for script execution, the potential size of the routing tables and the network segments were not in a list of initial requirements. However, let’s take them into account. Most modern network platforms support over 1M routing entries on average. IPv4 BGP Full View size is around 814,000 prefixes as of January 2021.

My rough estimations and tests showed that in-memory processing of each 1M prefixes requires ~500MB of RAM for this scenario. Even a mid-level laptop with 8GB RAM should allow you to process a topology consisting of 17-18 routers with Full View on each of them (~12-13M prefixes in total). I believe this is enough for most of the cases. For larger network segments, the analysis can either be split into smaller scopes or moved into an external out-of-memory database.
640 kB ought to be enough for anybody. Let’s stick on an in-memory processing option.

Parsing Source Files and Selecting Data Structures

Let’s store all our text files with routing tables in a separate variable-defined sub-directory:

RT_DIRECTORY = "./routing_tables"

Here is a reference of IOS and IOS-XE routing table output format:

show ip route

S* 0.0.0.0/0 [1/0] via 10.220.88.1
10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C 10.220.88.0/24 is directly connected, FastEthernet4
L 10.220.88.20/32 is directly connected, FastEthernet4
     1.0.0.0/32 is subnetted, 1 subnets
S       1.1.1.1 [1/0] via 212.0.0.1
                [1/0] via 192.168.0.1
D EX     10.1.198.0/24 [170/1683712] via 172.16.209.47, 1w2d, Vlan910
                       [170/1683712] via 172.16.60.33, 1w2d, Vlan60
                       [170/1683712] via 10.25.20.132, 1w2d, Vlan220
                       [170/1683712] via 10.25.20.9, 1w2d, Vlan20
     4.0.0.0/16 is subnetted, 1 subnets
O E2    4.4.0.0 [110/20] via 194.0.0.2, 00:02:00, FastEthernet0/0
     5.0.0.0/24 is subnetted, 1 subnets
D EX    5.5.5.0 [170/2297856] via 10.0.1.2, 00:12:01, Serial0/0
     6.0.0.0/16 is subnetted, 1 subnets
B       6.6.0.0 [200/0] via 195.0.0.1, 00:00:04
     172.16.0.0/26 is subnetted, 1 subnets
i L2    172.16.1.0 [115/10] via 10.0.1.2, Serial0/0
     172.20.0.0/32 is subnetted, 3 subnets
O       172.20.1.1 [110/11] via 194.0.0.2, 00:05:45, FastEthernet0/0
O       172.20.3.1 [110/11] via 194.0.0.2, 00:05:45, FastEthernet0/0
O       172.20.2.1 [110/11] via 194.0.0.2, 00:05:45, FastEthernet0/0
     10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
C       10.0.1.0/24 is directly connected, Serial0/0
D       10.0.5.0/26 [90/2297856] via 10.0.1.2, 00:12:03, Serial0/0
D       10.0.5.64/26 [90/2297856] via 10.0.1.2, 00:12:03, Serial0/0
D       10.0.5.128/26 [90/2297856] via 10.0.1.2, 00:12:03, Serial0/0
D       10.0.5.192/27 [90/2297856] via 10.0.1.2, 00:12:03, Serial0/0
     192.168.0.0/32 is subnetted, 1 subnets
D       192.168.0.1 [90/2297856] via 10.0.1.2, 00:12:03, Serial0/0
O IA 195.0.0.0/24 [110/11] via 194.0.0.2, 00:05:45, FastEthernet0/0
O E2 212.0.0.0/8 [110/20] via 194.0.0.2, 00:05:35, FastEthernet0/0
C    194.0.0.0/16 is directly connected, FastEthernet0/0

Cisco ASA looks very similar. The difference is ASA displays full subnet masks instead of prefix lengths:

show route

S    10.1.1.0 255.255.255.0 [3/0] via 10.86.194.1, outside
C    10.86.194.0 255.255.254.0 is directly connected, outside
S*   0.0.0.0 0.0.0.0 [1/0] via 10.86.194.1, outside

The examples show that, despite the multitude of options, all routing table entries have a predictable format. So they can be processed with regular expressions.
There are two common groups based on route entry format: Local+Connected types and all the rest.

The existence of multi-line routes for multi-path routing cases makes them harder to extract. We can not use simple line iteration through the content of the files because of that. One of the workarounds is to iterate through regular expression matches covering multiple lines.

Let’s write such regular expressions:

# Local and Connected route strings matching.
REGEXP_ROUTE_LOCAL_CONNECTED = re.compile(
    r'^(?P<routeType>[L|C])\s+'
    + r'((?P<ipaddress>\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?)'
    + r'\s?'
    + r'(?P<maskOrPrefixLength>(\/\d\d?)?'
    + r'|(\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?)?))'
    + r'\ is\ directly\ connected\,\ '
    + r'(?P<interface>\S+)',
    re.MULTILINE
)

# Static and dynamic route strings matching.
REGEXP_ROUTE = re.compile(
    r'^(\S\S?\*?\s?\S?\S?)'
    + r'\s+'
    + r'((?P<subnet>\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?)'
    + r'\s?'
    + r'(?P<maskOrPrefixLength>(\/\d\d?)?'
    + r'|(\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?)?))'
    + r'\s*'
    + r'(?P<viaPortion>(?:\n?\s+(\[\d\d?\d?\/\d+\])\s+'
    + r'via\s+(\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?)(.*)\n?)+)',
    re.MULTILINE
)

Both regular expressions contain (named groups) to make them more readable and maintainable. You can reference the named group value within the regular expression match by its key: subnet/interface/maskOrPrefixLength for the prefix info and viaPortion/interface for the route destination info in our case.

The regular expression covers subnet mask and prefix length representations at once. It can be extracted by maskOrPrefixLength key. For a further processing, let’s bring it to a common format of the prefix length as it is shorter:

def convert_netmask_to_prefix_length(mask_or_pref):
    """
    Gets subnet_mask (XXX.XXX.XXX.XXX) of /prefix_length (/XX).
    For subnet_mask, converts it to /prefix_length and returns the result.
    For /prefix_length, returns as is.
    For empty input, returns "" string.
    """
    if not mask_or_pref:
        return ""
    if re.match(r"^\/\d\d?$", mask_or_pref):
        return mask_or_pref
    if re.match(r"^\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?$",
                mask_or_pref):
        return (
            "/"
            + str(sum([bin(int(x)).count("1") for x in mask_or_pref.split(".")]))
        )
    return ""

Let’s also write a regular expression for next-hop extraction from the viaPortion group and a regular expression for IPv4 address format check in a file and user input:

# Route string VIA portion matching.
REGEXP_VIA_PORTION = re.compile(
    r'.*via\s+(\d\d?\d?\.\d\d?\d?\.\d\d?\d?\.\d\d?\d?).*'
)
# RegEx template string for IPv4 address matching.
REGEXP_IPv4_STR = (
    r'((25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.'
    + r'(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.'
    + r'(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.'
    + r'(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))'
)
# IPv4 CIDR notation matching in user input.
REGEXP_INPUT_IPv4 = re.compile(r"^" + REGEXP_IPv4_STR + r"(\/\d\d?)?$")

Now let’s translate our network representation into Python data structures. All the prefixes we extract from the routing tables will be used as prefix tree keys. Each prefix tree object will be inherited from the PyTricia module. Search result on a prefix tree will return a list of available next-hops and a full-text representation of the matched routing entry. Another list will store a list of local interfaces with their IP-addresses for each router. Each router will be represented by a dictionary object containing all above.

# Example data structures
route_tree = pytricia.PyTricia()
route_tree[’subnet’] = ((next_hop_1, next_hop_n), raw_route_string)
interface_list = ((interface_1, ip_address_1), (interface_n, ip_address_n))
connected_networks = ((interface_1, subnet_1), (interface_n, subnet_n))
router = {
    ‘routing_table’: route_tree,
    ‘interface_list’: interface_list,
    ‘connected_networks’: connected_networks,
}

Now we can implement a route lookup function:

def route_lookup(destination, router):
    if destination in router['routing_table']:
        return router['routing_table'][destination]
    else:
        return (None, None)

To distinguish between the routers, it is important to assign some unique router identifier (RID) for each of them. Router ID generation and selection algorithms might be different. In our case, let’s use a filename as a RID for simplicity.
Let’s put all resulting routers into a dictionary with RIDs as keys and corresponding router objects as values:

ROUTERS = {
    ‘router_id_1’: router_1,
    ‘router_id_n’: router_n,
}

We also need to implement some next-hop RID resolution mechanism by known next-hop IP-address (think of ARP). Let’s create one more prefix tree containing IP-addresses of all discovered router as keys and RID with interface types as corresponding values:

# Example
GLOBAL_INTERFACE_TREE = pytricia.PyTricia()
GLOBAL_INTERFACE_TREE[‘ip_address’] = (router_id, interface_type)

# Returns RouterID by Interface IP address which it belongs to.
def get_rid_by_interface_ip(interface_ip):
    if interface_ip in GLOBAL_INTERFACE_TREE:
        return GLOBAL_INTERFACE_TREE[interface_ip][0]

Now let’s combine our IOS/IOS-XE/ASA format parsers into a single function. It will take a text routing table as an input and return a router dictionary object of a format we discussed earlier:

def parse_show_ip_route_ios_like(raw_routing_table):
    """
    Parser for routing table text output.
    Compatible with both Cisco IOS(IOS-XE) 'show ip route'
    and Cisco ASA 'show route' output format.
    Processes input text file and write into Python data structures.
    Builds internal PyTricia search tree in 'route_tree'.
    Generates local interface list for a router in 'interface_list'
    Returns 'router' dictionary object with parsed data.
    """
    router = {}
    route_tree = pytricia.PyTricia()
    interface_list = []
    # Parse Local and Connected route strings in text.
    for raw_route_string in REGEXP_ROUTE_LOCAL_CONNECTED.finditer(raw_routing_table):
        subnet = (
            raw_route_string.group('ipaddress')
            + convert_netmask_to_prefix_length(
                raw_route_string.group('maskOrPrefixLength')
            )
        )
        interface = raw_route_string.group('interface')
        route_tree[subnet] = ((interface,), raw_route_string.group(0))
        if raw_route_string.group('routeType') == 'L':
            interface_list.append((interface, subnet,))
    if not interface_list:
        print('Failed to find routing table entries in given output')
        return None
    # parse static and dynamic route strings in text
    for raw_route_string in REGEXP_ROUTE.finditer(raw_routing_table):
        subnet = (
            raw_route_string.group('subnet')
            + convert_netmask_to_prefix_length(
                raw_route_string.group('maskOrPrefixLength')
            )
        )
        via_portion = raw_route_string.group('viaPortion')
        next_hops = []
        if via_portion.count('via') > 1:
            for line in via_portion.splitlines():
                if line:
                    next_hops.append(REGEXP_VIA_PORTION.match(line).group(1))
        else:
            next_hops.append(REGEXP_VIA_PORTION.match(via_portion).group(1))
        route_tree[subnet] = (next_hops, raw_route_string.group(0))
    router = {
        'routing_table': route_tree,
        'interface_list': interface_list,
    }
    return router

To improve extensibility, let’s wrap all parsers into another function:

def parse_text_routing_table(raw_routing_table):
    """
    Parser functions wrapper.
    Add additional parsers for alternative routing table syntaxes here.
    """
    router = parse_show_ip_route_ios_like(raw_routing_table)
    if router:
        return router

Finally, we need a function to go through a directory containing our routing table text files. It will take a directory path as an input and return a dictionary of all discovered routers:

def do_parse_directory(rt_directory):
    """
    Go through the specified directory and parse all .txt files.
    Generate router objects based on parse result if any.
    Populate new_routers with those router objects.
    The default key for each router object is FILENAME.
    Return new_routers.
    """
    new_routers = {}
    if not os.path.isdir(rt_directory):
        print(
            "{} directory does not exist.".format(rt_directory)
            + "Check rt_directory variable value."
        )
        return None
    start_time = time()
    print("Initializing files...")
    for FILENAME in os.listdir(rt_directory):
        if FILENAME.endswith('.txt'):
            file_init_start_time = time()
            with open(os.path.join(rt_directory, FILENAME), 'r') as f:
                print('Opening {}'.format(FILENAME))
                raw_table = f.read()
                new_router = parse_text_routing_table(raw_table)
                router_id = FILENAME.replace('.txt', '')
                if new_router:
                    new_routers[router_id] = new_router
                    if new_router['interface_list']:
                        for iface, addr in new_router['interface_list']:
                            GLOBAL_INTERFACE_TREE[addr] = (router_id, iface,)
                else:
                    print('Failed to parse ' + FILENAME)
            print(
                FILENAME
                + " parsing has been completed in {} sec".format(
                    "{:.3f}".format(time() - file_init_start_time)
                )
            )
    else:
        if not new_routers:
            print(
                "Could not find any valid .txt files with routing tables"
                + " in {} directory".format(rt_directory)
            )
        else:
            print(
                "\nAll files have been initialized"
                + " in {} sec".format("{:.3f}".format(time() - start_time))
            )
            return new_routers

Once we have the structured data, we can move to the routing paths analysis part of the task.

Analyzing Routing Paths

In general, the task at this stage is to analyze the network graph. Routers are graph vertices and L3-links are graph edges. ROUTERS dictionary stores Router IDs as keys and next-hop IP-addresses as values. GLOBAL_INTERFACE_TREE returns RIDs by next-hop IP-addresses at the same time. So ROUTERS and GLOBAL_INTERFACE_TREE together define a graph adjacency table.

If we draw parallels with real routers, to find a path, you need to reproduce their high-level work logic (not taking RIB/FIB/ASIC and different optimizations into account) during the packet processing: from routing table lookup to ARP-request (router_id in our case) and further packet forwarding or drop depending on the result.

To achieve this, let’s implement a recursive path search algorithm. Each path segment will be represented by a list containing router_id (RID) and raw_route_string (matched route string). The current path will be stored in a path tuple. As we might have multiple paths, the resulting list of them will be stored in a paths tuple. Individual path will be appended to paths once the current path analysis reached the end (the destination or no route to the destination at some point) or on routing loop detection. The function will take a RID we start from and a target IP we are searching path to as an input and return resulting paths.

def trace_route(source_router_id, target_ip, path=[]):
    """
    Performs recursive path search from source Router ID (RID) to target subnet.
    Returns tuple of path tuples.
    Each path tuple contains a sequence of Router IDs with matched route strings.
    Multiple paths are supported.
    """
    if not source_router_id:
        return [path + [(None, None)]]
    current_router = ROUTERS[source_router_id]
    next_hop, raw_route_string = route_lookup(target_ip, current_router)
    path = path + [(source_router_id, raw_route_string)]
    paths = []
    if next_hop:
        if nexthop_is_local(next_hop[0]):
            return [path]
        for nh in next_hop:
            next_hop_rid = get_rid_by_interface_ip(nh)
            if next_hop_rid not in [r[0] for r in path]:
                inner_path = trace_route(next_hop_rid, target_ip, path)
                for p in inner_path:
                    paths.append(p)
            else:
                path = path + [(next_hop_rid+"<<LOOP DETECTED", None)]
                return [path]
    else:
        return [path]
    return paths


def nexthop_is_local(next_hop):
    """
    Check if next-hop points to the local interface.
    Will be True for Connected and Local route strings on Cisco devices.
    """
    interface_types = (
        'Eth', 'Fast', 'Gig', 'Ten', 'Port',
        'Serial', 'Vlan', 'Tunn', 'Loop', 'Null'
    )
    for type in interface_types:
        if next_hop.startswith(type):
            return True

Let’s also implement a function to provide interactive path lookup ability to our script user. It will perform a path search to the given IP-address from all the discovered routers:

def do_user_interactive_search():
    """
    Provides interactive search dialog for users.
    Asks user for target subnet or host in CIDR notation.
    Validates input. Prints error and goes back to start for invalid input.
    Executes path search to given target from each router in global ROUTERS.
    Prints formatted path search results.
    Goes back to start.
    """
    while True:
        print('\n')
        target_subnet = input('Enter Target Subnet or Host: ')
        if not target_subnet:
            continue
        if not REGEXP_INPUT_IPv4.match(target_subnet.replace(' ', '')):
            print("incorrect input")
            continue
        lookup_start_time = time()
        for rtr in ROUTERS.keys():
            subsearch_start_time = time()
            result = trace_route(rtr, target_subnet)
            if result:
                print("\n")
                print("PATHS TO {} FROM {}".format(target_subnet, rtr))
                n = 1
                print('Detailed info:')
                for r in result:
                    print("Path {}:".format(n))
                    print([h[0] for h in r])
                    for hop in r:
                        print("ROUTER: {}".format(hop[0]))
                        print("Matched route string: \n{}".format(hop[1]))
                    else:
                        print('\n')
                    n += 1
                else:
                    print(
                        "Path search on {} has been completed in {} sec".format(
                           rtr, "{:.3f}".format(time() - subsearch_start_time)
                        )
                    )
        else:
            print(
                "\nFull search has been completed in {} sec".format(
                   "{:.3f}".format(time() - lookup_start_time),
                )
            )

Bringing the logic together:

def main():
    global ROUTERS
    ROUTERS = do_parse_directory(RT_DIRECTORY)
    if ROUTERS:
        do_user_interactive_search()

if __name__ == "__main__":
    main() 

And here is a complete solution!

How do we not it is working? Of course, let’s do some testing.

Testing

I used a small topology consisting of four Cisco CSR-1000v routers for testing:

They are interconnected with GigabitEthernet 2 and 3 interfaces. All adjacent routers are EIGRP neighbors. All Connected networks are being advertised, including Loopback addresses behind each router. Besides, csr1000v-01 and csr1000v-04 have a pair of GRE tunnels between them. Both of them have a static route for 10.0.0.0/8 subnet pointing to remote GRE tunnel IP forming a routing loop.

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

S     10.0.0.0/8 [1/0] via 192.168.142.2
                 [1/0] via 192.168.141.2
      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks
C        172.16.114.0/24 is directly connected, GigabitEthernet2
L        172.16.114.5/32 is directly connected, GigabitEthernet2
      192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.2.0/24 is directly connected, GigabitEthernet1
L        192.168.2.201/32 is directly connected, GigabitEthernet1
      192.168.12.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.12.0/24 is directly connected, GigabitEthernet2
L        192.168.12.201/32 is directly connected, GigabitEthernet2
      192.168.13.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.13.0/24 is directly connected, GigabitEthernet3
L        192.168.13.201/32 is directly connected, GigabitEthernet3
D     192.168.24.0/24 [90/3072] via 192.168.12.202, 00:06:56, GigabitEthernet2
D     192.168.34.0/24 [90/3072] via 192.168.13.203, 00:06:56, GigabitEthernet3
      192.168.141.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.141.0/30 is directly connected, Tunnel141
L        192.168.141.1/32 is directly connected, Tunnel141
      192.168.142.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.142.0/30 is directly connected, Tunnel142
L        192.168.142.1/32 is directly connected, Tunnel142
      192.168.201.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.201.0/24 is directly connected, Loopback201
L        192.168.201.201/32 is directly connected, Loopback201
D     192.168.202.0/24 
           [90/130816] via 192.168.12.202, 00:05:44, GigabitEthernet2
D     192.168.203.0/24 
           [90/130816] via 192.168.13.203, 00:06:22, GigabitEthernet3
D     192.168.204.0/24 
           [90/131072] via 192.168.13.203, 00:06:56, GigabitEthernet3
           [90/131072] via 192.168.12.202, 00:06:56, GigabitEthernet2

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.2.0/24 is directly connected, GigabitEthernet1
L        192.168.2.202/32 is directly connected, GigabitEthernet1
      192.168.12.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.12.0/24 is directly connected, GigabitEthernet2
L        192.168.12.202/32 is directly connected, GigabitEthernet2
D     192.168.13.0/24 [90/3072] via 192.168.12.201, 00:46:17, GigabitEthernet2
      192.168.24.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.24.0/24 is directly connected, GigabitEthernet3
L        192.168.24.202/32 is directly connected, GigabitEthernet3
D     192.168.34.0/24 [90/3072] via 192.168.24.204, 00:46:15, GigabitEthernet3
D     192.168.201.0/24 
           [90/130816] via 192.168.12.201, 00:36:59, GigabitEthernet2
      192.168.202.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.202.0/24 is directly connected, Loopback202
L        192.168.202.202/32 is directly connected, Loopback202
D     192.168.203.0/24 
           [90/131072] via 192.168.24.204, 00:06:31, GigabitEthernet3
           [90/131072] via 192.168.12.201, 00:06:31, GigabitEthernet2
D     192.168.204.0/24 
           [90/130816] via 192.168.24.204, 00:37:26, GigabitEthernet3

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

      192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.2.0/24 is directly connected, GigabitEthernet1
L        192.168.2.203/32 is directly connected, GigabitEthernet1
D     192.168.12.0/24 [90/3072] via 192.168.13.201, 00:46:12, GigabitEthernet3
      192.168.13.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.13.0/24 is directly connected, GigabitEthernet3
L        192.168.13.203/32 is directly connected, GigabitEthernet3
D     192.168.24.0/24 [90/3072] via 192.168.34.204, 00:46:12, GigabitEthernet2
      192.168.34.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.34.0/24 is directly connected, GigabitEthernet2
L        192.168.34.203/32 is directly connected, GigabitEthernet2
D     192.168.201.0/24 
           [90/130816] via 192.168.13.201, 00:36:56, GigabitEthernet3
D     192.168.202.0/24 
           [90/131072] via 192.168.34.204, 00:05:51, GigabitEthernet2
           [90/131072] via 192.168.13.201, 00:05:51, GigabitEthernet3
      192.168.203.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.203.0/24 is directly connected, Loopback203
L        192.168.203.203/32 is directly connected, Loopback203
D     192.168.204.0/24 
           [90/130816] via 192.168.34.204, 00:37:22, GigabitEthernet2

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

S     10.0.0.0/8 [1/0] via 192.168.142.1
                 [1/0] via 192.168.141.1
      192.168.2.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.2.0/24 is directly connected, GigabitEthernet1
L        192.168.2.204/32 is directly connected, GigabitEthernet1
D     192.168.12.0/24 [90/3072] via 192.168.24.202, 00:46:17, GigabitEthernet3
D     192.168.13.0/24 [90/3072] via 192.168.34.203, 00:46:19, GigabitEthernet2
      192.168.24.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.24.0/24 is directly connected, GigabitEthernet3
L        192.168.24.204/32 is directly connected, GigabitEthernet3
      192.168.34.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.34.0/24 is directly connected, GigabitEthernet2
L        192.168.34.204/32 is directly connected, GigabitEthernet2
      192.168.141.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.141.0/30 is directly connected, Tunnel141
L        192.168.141.2/32 is directly connected, Tunnel141
      192.168.142.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.142.0/30 is directly connected, Tunnel142
L        192.168.142.2/32 is directly connected, Tunnel142
D     192.168.201.0/24 
           [90/131072] via 192.168.34.203, 00:37:02, GigabitEthernet2
           [90/131072] via 192.168.24.202, 00:37:02, GigabitEthernet3
D     192.168.202.0/24 
           [90/130816] via 192.168.24.202, 00:05:57, GigabitEthernet3
D     192.168.203.0/24 
           [90/130816] via 192.168.34.203, 00:06:34, GigabitEthernet2
      192.168.204.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.204.0/24 is directly connected, Loopback204
L        192.168.204.204/32 is directly connected, Loopback204

Let’s save show ip route files into separate files named by hostnames inside ./routing_tables/ directory. Now we can run the script:

$ python3.7 traceroute_by_routing_tables.py
Initializing files...
Opening csr1000v-01.txt
csr1000v-01.txt parsing has been completed in 0.001 sec
Opening csr1000v-02.txt
csr1000v-02.txt parsing has been completed in 0.001 sec
Opening csr1000v-03.txt
csr1000v-03.txt parsing has been completed in 0.001 sec
Opening csr1000v-04.txt
csr1000v-04.txt parsing has been completed in 0.001 sec

All files have been initialized in 0.003 sec


Enter Target Subnet or Host:

All the files are processed as expected. The script expects an IP-address input to analyze paths. Let’s put several IP-addresses subsequently and compare the output with the data from our routers:

Enter Target Subnet or Host: 192.168.204.204


PATHS TO 192.168.204.204 FROM csr1000v-04
Detailed info:
Path 1:
['csr1000v-04']
ROUTER: csr1000v-04
Matched route string: 
L        192.168.204.204/32 is directly connected, Loopback204


Path search on csr1000v-04 has been completed in 0.000 sec


PATHS TO 192.168.204.204 FROM csr1000v-03
Detailed info:
Path 1:
['csr1000v-03', 'csr1000v-04']
ROUTER: csr1000v-03
Matched route string: 
D     192.168.204.0/24 
           [90/130816] via 192.168.34.204, 00:37:22, GigabitEthernet2

ROUTER: csr1000v-04
Matched route string: 
L        192.168.204.204/32 is directly connected, Loopback204


Path search on csr1000v-03 has been completed in 0.000 sec


PATHS TO 192.168.204.204 FROM csr1000v-02
Detailed info:
Path 1:
['csr1000v-02', 'csr1000v-04']
ROUTER: csr1000v-02
Matched route string: 
D     192.168.204.0/24 
           [90/130816] via 192.168.24.204, 00:37:26, GigabitEthernet3
ROUTER: csr1000v-04
Matched route string: 
L        192.168.204.204/32 is directly connected, Loopback204


Path search on csr1000v-02 has been completed in 0.000 sec


PATHS TO 192.168.204.204 FROM csr1000v-01
Detailed info:
Path 1:
['csr1000v-01', 'csr1000v-03', 'csr1000v-04']
ROUTER: csr1000v-01
Matched route string: 
D     192.168.204.0/24 
           [90/131072] via 192.168.13.203, 00:06:56, GigabitEthernet3
           [90/131072] via 192.168.12.202, 00:06:56, GigabitEthernet2
ROUTER: csr1000v-03
Matched route string: 
D     192.168.204.0/24 
           [90/130816] via 192.168.34.204, 00:37:22, GigabitEthernet2

ROUTER: csr1000v-04
Matched route string: 
L        192.168.204.204/32 is directly connected, Loopback204


Path 2:
['csr1000v-01', 'csr1000v-02', 'csr1000v-04']
ROUTER: csr1000v-01
Matched route string: 
D     192.168.204.0/24 
           [90/131072] via 192.168.13.203, 00:06:56, GigabitEthernet3
           [90/131072] via 192.168.12.202, 00:06:56, GigabitEthernet2
ROUTER: csr1000v-02
Matched route string: 
D     192.168.204.0/24 
           [90/130816] via 192.168.24.204, 00:37:26, GigabitEthernet3
ROUTER: csr1000v-04
Matched route string: 
L        192.168.204.204/32 is directly connected, Loopback204


Path search on csr1000v-01 has been completed in 0.000 sec

Full search has been completed in 0.001 sec

csr1000v-01#show ip route 192.168.204.204
Routing entry for 192.168.204.0/24
  Known via "eigrp 200", distance 90, metric 131072, type internal
  Redistributing via eigrp 200
  Last update from 192.168.13.203 on GigabitEthernet3, 00:02:15 ago
  Routing Descriptor Blocks:
    192.168.13.203, from 192.168.13.203, 00:02:15 ago, via GigabitEthernet3
      Route metric is 131072, traffic share count is 1
      Total delay is 5020 microseconds, minimum bandwidth is 1000000 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 1/255, Hops 2
  * 192.168.12.202, from 192.168.12.202, 00:02:15 ago, via GigabitEthernet2
      Route metric is 131072, traffic share count is 1
      Total delay is 5020 microseconds, minimum bandwidth is 1000000 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 1/255, Hops 2

csr1000v-02#show ip route 192.168.204.204
Routing entry for 192.168.204.0/24
  Known via "eigrp 200", distance 90, metric 130816, type internal
  Redistributing via eigrp 200
  Last update from 192.168.24.204 on GigabitEthernet3, 00:08:48 ago
  Routing Descriptor Blocks:
  * 192.168.24.204, from 192.168.24.204, 00:08:48 ago, via GigabitEthernet3
      Route metric is 130816, traffic share count is 1
      Total delay is 5010 microseconds, minimum bandwidth is 1000000 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 1/255, Hops 1

csr1000v-3#show ip route 192.168.204.204
Routing entry for 192.168.204.0/24
  Known via "eigrp 200", distance 90, metric 130816, type internal
  Redistributing via eigrp 200
  Last update from 192.168.34.204 on GigabitEthernet2, 00:08:45 ago
  Routing Descriptor Blocks:
  * 192.168.34.204, from 192.168.34.204, 00:08:45 ago, via GigabitEthernet2
      Route metric is 130816, traffic share count is 1
      Total delay is 5010 microseconds, minimum bandwidth is 1000000 Kbit
      Reliability 255/255, minimum MTU 1500 bytes
      Loading 1/255, Hops 1

csr1000v-04#show ip route 192.168.204.204
Routing entry for 192.168.204.204/32
  Known via "connected", distance 0, metric 0 (connected)
  Routing Descriptor Blocks:
  * directly connected, via Loopback204
      Route metric is 0, traffic share count is 1

Enter Target Subnet or Host: 10.10.10.0/24

PATHS TO 10.10.10.0/24 FROM csr1000v-04
Detailed info:
Path 1:
['csr1000v-04', 'csr1000v-01', 'csr1000v-04<<LOOP DETECTED']
ROUTER: csr1000v-04
Matched route string: 
S     10.0.0.0/8 [1/0] via 192.168.142.1
                 [1/0] via 192.168.141.1

ROUTER: csr1000v-01
Matched route string: 
S     10.0.0.0/8 [1/0] via 192.168.142.2
                 [1/0] via 192.168.141.2

ROUTER: csr1000v-04<<LOOP DETECTED
Matched route string: 
None


Path 2:
['csr1000v-04', 'csr1000v-01', 'csr1000v-04<<LOOP DETECTED']
ROUTER: csr1000v-04
Matched route string: 
S     10.0.0.0/8 [1/0] via 192.168.142.1
                 [1/0] via 192.168.141.1

ROUTER: csr1000v-01
Matched route string: 
S     10.0.0.0/8 [1/0] via 192.168.142.2
                 [1/0] via 192.168.141.2

ROUTER: csr1000v-04<<LOOP DETECTED
Matched route string: 
None


Path search on csr1000v-04 has been completed in 0.000 sec


PATHS TO 10.10.10.0/24 FROM csr1000v-03
Detailed info:
Path 1:
['csr1000v-03']
ROUTER: csr1000v-03
Matched route string: 
None


Path search on csr1000v-03 has been completed in 0.000 sec


PATHS TO 10.10.10.0/24 FROM csr1000v-02
Detailed info:
Path 1:
['csr1000v-02']
ROUTER: csr1000v-02
Matched route string: 
None


Path search on csr1000v-02 has been completed in 0.000 sec


PATHS TO 10.10.10.0/24 FROM csr1000v-01
Detailed info:
Path 1:
['csr1000v-01', 'csr1000v-04', 'csr1000v-01<<LOOP DETECTED']
ROUTER: csr1000v-01
Matched route string: 
S     10.0.0.0/8 [1/0] via 192.168.142.2
                 [1/0] via 192.168.141.2

ROUTER: csr1000v-04
Matched route string: 
S     10.0.0.0/8 [1/0] via 192.168.142.1
                 [1/0] via 192.168.141.1

ROUTER: csr1000v-01<<LOOP DETECTED
Matched route string: 
None


Path 2:
['csr1000v-01', 'csr1000v-04', 'csr1000v-01<<LOOP DETECTED']
ROUTER: csr1000v-01
Matched route string: 
S     10.0.0.0/8 [1/0] via 192.168.142.2
                 [1/0] via 192.168.141.2

ROUTER: csr1000v-04
Matched route string: 
S     10.0.0.0/8 [1/0] via 192.168.142.1
                 [1/0] via 192.168.141.1

ROUTER: csr1000v-01<<LOOP DETECTED
Matched route string: 
None


Path search on csr1000v-01 has been completed in 0.003 sec

Full search has been completed in 0.004 sec

csr1000v-01#show ip route 10.10.10.0 255.255.255.0
Routing entry for 10.0.0.0/8
  Known via "static", distance 1, metric 0
  Routing Descriptor Blocks:
  * 192.168.142.2
      Route metric is 0, traffic share count is 1
    192.168.141.2
      Route metric is 0, traffic share count is 1

csr1000v-04#show ip route 10.10.10.0 255.255.255.0
Routing entry for 10.0.0.0/8
  Known via "static", distance 1, metric 0
  Routing Descriptor Blocks:
    192.168.142.1
      Route metric is 0, traffic share count is 1
  * 192.168.141.1
      Route metric is 0, traffic share count is 1

PATHS TO 10.10.10.0/24 FROM csr1000v-01
Path 1:
['csr1000v-01', 'csr1000v-04', 'csr1000v-01<<LOOP DETECTED']
Path 2:
['csr1000v-01', 'csr1000v-04', 'csr1000v-01<<LOOP DETECTED']

PATHS TO 10.10.10.0/24 FROM csr1000v-04
Path 1:
['csr1000v-04', 'csr1000v-01', 'csr1000v-04<<LOOP DETECTED']
Path 2:
['csr1000v-04', 'csr1000v-01', 'csr1000v-04<<LOOP DETECTED']

csr1000v-02#show ip route 10.10.10.0 255.255.255.0
% Network not in table

csr1000v-3#show ip route 10.10.10.0 255.255.255.0
% Network not in table

All common test cases are covered. The script result matches the output we get from the actual network devices using CLI.

Conclusion

The resulting solution is not perfect but it provides great lookup speed thanks to efficient algorithms and successfully solves the original task. The source code is published on my GitHub page.

The solution has some room for improvement and adding new analysis features. Additional parsers for alternative routing table syntax can easily be added by design. IPv6 is supported by PyTricia library natively.

I also tested the script on a BGP Full View routing table output with 700,000+ prefixes. On my good old MacBook Pro with Intel Core i5 and 8GB RAM, an initialization time takes less than 10 seconds. Memory consumption is around 320-350MB. Once it is initialized, any lookup to a routing table of that size still takes milliseconds as expected.

In 2021, some full-blown network analysis tools like PyATS and Batfish could be a better starting point for more complex scenarios. However, being able to develop custom tools for such corner cases is still relevant.

Hope it helps someone solve some real issues or inspire to develop some automation on his or her own. Thank you for reading.

Disclaimer

This is a translation of my original article in Russian I posted in May 2020. If you are willing to help to improve the translation, please DM me. The article does not aim to cover all the possible scenarios yet does describe the general approaches based on the specific case. All above and below is an author’s personal subjective opinion if not stated otherwise. All listed code is published under MIT license and does not provide guarantees of any kind.

The solution is written in Python and JavaScript. An understanding of the programming and networking basics is desirable for reading.

If you found a typo, please use Ctrl+Enter or ⌘+Enter to send it to the author.

The Task

The final task description was the following:

There is a network consisting of various L2/L3 network devices running IOS/IOS-XE. You have a list of management IP-addresses of all the devices. All devices are available by their IP addresses. You have access rights to execute any ‘show’ commands. You are free to use any data gathering methods. But trust us, you unlikely need SNMP. We are not ought to limit your fantasy though.

The primary task is to identify the physical topology and device interconnections based on LLDP data and visualize it in a human-friendly format (yeah, we all find visual diagrams more readable). Then you should save the result in a format suitable for further analysis by the computer (yeah, machines are not that good at reading visual diagrams).

The topology view should include:

• Different icons for each device type (routers and switches may have the same icon).
• Device hostnames.
• Interface names (you may use a shortened format, e.g. Gi0/0 for GigabitEthernet0/0/0).

It is allowed to implement filters limiting or hiding some of the information. A bonus task is to identify the topology changes (by comparing the current and the previous version) and visualize them in a human-friendly format.

A summary: IP-addresses and credentials as an input, a visualized topology as an output (and a great space for experiments and options somewhere in between).

I also denoted some additional personal considerations to follow while choosing the solution toolset:

• Feature-rich vs Simple balance.
  The solution should be balanced in terms of available features and the ease of their usage and implementation. Some ready-to-use open-source free tools might be used.
• Familiarity with the selected toolset.
  We have had three days to complete the task. The first day I had to spend on some side emergencies. So to be able to provide a working solution within such a limited time frame some known tools were a must.
• Solution reusability.
  The task is potentially applicable to many production networks. One should keep this in mind.
• Support for multiple network platforms.
  Real-world networks consist of many platforms. This is a thing to note as well.
• The documentation must be available for the chosen toolset.
  I believe this is a mandatory requirement for any reusable solution.

Existing solutions

It is always a good idea to check whether the wheel is already invented before reinventing it by yourself. I have made some research on available existing solutions for network visualization. Not surprisingly, no solution could fit all the requirements out of the box. Most such solutions were built into much larger (and typically far not free) enterprise-grade network monitoring and analysis systems which would significantly reduce reusability and customization potential.

Task decomposition and toolset selection

I would divide an abstract network visualization task into the following layers and steps:

Let’s focus on each of them while keeping our requirement in mind:

1. Network Equipment
   The initial task requires us to support IOS and IOS-XE. Real-world networks tend to be much more heterogeneous. Let’s try to consider this.

2. Topology Data Sources
   The task statement suggests us to use LLDP (Link Layer Discovery Protocol) protocol. This is an L2 (OSI Link Layer) protocol described in IEEE 802.1AB standard. It is widely supported by modern network platforms (including IOS and IOS-XE) and operating systems (including Linux and Window), so it suits our case.
   The topology data can also be enriched by various outputs from the network devices, such as routing and switching tables, routing protocols data, and so on. Let’s just mark it for future improvements.

3. Data Access Protocols and Standards The most modern platforms usually support shiny and chrome NETCONF, REST APIs, RESTCONF with YANG models and data structures. The existence of legacy equipment and platforms will usually force us to revert to SSH, Telnet, and good ol’ CLI.

4. Protocol- and Vendor-Specific Drivers or Plugins The core logic of the solution will be written in Python because of two primary reasons: Python has a pretty comprehensive set of handy modules and libraries for network automation and this is a programming language I am most experienced in.
   API-driven network automation in Python can be done using the requests module or some specialized modules.
   Bare SSH/Telnet access to network equipment from Python commonly relies on netmiko, paramiko, or scrapli modules. They let you emulate the standard CLI: sending some text commands to the session and expecting back the text output of the more or less predictable readability level and formatting.
   There are also several high-level Python frameworks allowing for additional features on top of the tools I mentioned above. The two most useful of them in our case are NAPALM and Nornir. NAPALM provides vendor-neutral GETTERs for getting structured data from the network devices. Nornir implements many useful abstractions and multithreading out of the box.
   As for SNMP, let’s leave it for network monitoring purposes.

5. Unstructured Data -> Data Normalization Toolset -> Structured Data
   Data gathering with API usually allows you to get structured output straight away. A text output you get from network equipment CLI is natively inapplicable for further machine processing. A traditional way to extract the data from the CLI output in Python is re module and regular expressions. Modern approaches are TextFSM framework developed by Google and brand new TTP (Template Text Parser) developed by dmulyalin. Both tools perform data parsing with more usable templates in comparison to regular expressions.
   NAPALM module mentioned above performs unstructured data normalization internally for supported GETTERs and returns the structured output. This may make things some easier in our case.

6. Data Processing and Analysis -> Topology Representation in Python Datastructures
   Once we get the structured topology data pieces from all our devices, all we need to do is to bring it to common representation, analyze and assemble a final puzzle.

7. Topology Representation in Visualization Engine Format
   Depending on visualization engine selection, you may need to transform a final topology data format according to what the tool supports as an input.

8. Visualization Engine
   This point was the most not obvious for me and I had no prior experience in such development. Google search and discussions in DevNet Marathon telegram channel with colleagues introduced me to several Python (pygraphviz, matplotlib, networkx) and JavaScript (JS D3.js, vis.js.) frameworks. Then I found JavaScript+HTML5 NeXt UI Toolkit I once bookmarked while digging through Cisco DevNet labs before. This is a specialized network visualization toolkit developed by Cisco. It has many features and decent documentation.

9. Visualized Topology
   Our final goal. A view may vary from a simple static image or an HTML document to something more advanced and interactive.

Here is a summary of the most common tools we have got:

Based on the requirements above I have selected the following tools for my target solution:

• LLDP is a topology data source.
• SSH and CLI for interaction with the network devices.
• Nornir for multithreading, more useful data gathering result handling and processing, and keeping the information about our devices in a structured Inventory.
• NAPALM to abstract from manual CLI scrapping.
• Python3 for writing the core logic.
• NeXt UI (JS+HTML5) for topology visualization based on result we get from Python peace of code.
  

I have already used NAPALM and Nornir successfully for network audits and data gathering from hundreds of network devices before. Default NAPALM GETTERs support LLDP on Cisco IOS/IOS-XE, IOS-XR, NX-OS, Juniper JunOS, and Arista EOS.
Furthermore, the logic separation discussed above would allow us to add more data sources and network connectors without affecting the whole codebase.
Next UI was a thing to familiarize with and figure out how it works on the run. However, the examples looked promising.

Preparation

Test lab

I used Cisco Modeling Labs as a test lab. This is a new version of the VIRL network emulator. Cisco DevNet Sandbox allows using it for free for a limited time window. You just have to register and proceed with a reservation which is a matter of a few mouse clicks (and a few more to connect to the lab using AnyConnect VPN once you receive the reservation details back to your email). In the old days we would have to use a production network a bare metal homelab or to have fun with GNS3.

Lab topology on the CML web interface looks as follows (we should get the similar picture as result):
It consists of Cisco devices of any kind: IOS (edge-sw01), IOSXE (internet-rtr01, distr-rtr01, distr-rtr02), NXOS (dist-sw01, dist-sw02), IOSXR (core-rtr01, core-rtr02), ASA (edge-firewall01). LLDP is enabled on all of them. SSH access is available on IOS, IOSXE, and NXOS nodes.

Installing and initializing Nornir

Nornir is an open-source Python framework. It is available on PyPI for Python 3.6.2 and above. Nornir has a dozen of dependencies, including NAPALM and netmiko. It is recommended to use Python virtual environments (venv to isolate the dependencies. My local development environment used Nornir 2.4.0 with Python 3.7 on MacOS 10.15. This should work on Linux and Windows as well. Nornir installation is straightforward:

$ mkdir ~/testenv
$ python3.7 -m venv ~/testenv/
$ source ~/testenv/bin/activate
(testenv)$ pip install nornir==2.4.0

Important: Nornir has had some massive changes in the 3.X release. Some of them were not backward compatible with 2.X versions. Nornir related configurations and code are relevant to 2.X versions.

Nornir supports various inventory plugins. They all provide a convenient way to structure and operate your network devices’ information programmatically. For this solution, the standard SimpleInventory plugin is sufficient.
General Nornir settings are listed in a set of YAML files. Configuration file names can be arbitrary but you should point Nornir to their exact names during initialization from Python.

nornir_config.yaml:

---
core:
    num_workers: 20
inventory:
    plugin: nornir.plugins.inventory.simple.SimpleInventory
    options:
        host_file: "inventory/hosts_devnet_sb_cml.yml"
        group_file: "inventory/groups.yml"

A sample Nornir primary configuration file you can see above contains references to two more YAML files: hosts file and groups file. These files define the SimpleInventory plugin configuration. The Hosts file contains a list of our network devices (hosts) with their attributes. Groups file contains a list of groups and their attributes. An individual host can be included in one or more groups. The host inherits the attributes of all groups it belongs to. Hosts and groups file names and locations can be arbitrary as well.

inventory/hosts_devnet_sb_cml.yml has the following structure:

---

internet-rtr01:
    hostname: 10.10.20.181
    platform: ios
    groups:
        - devnet-cml-lab

dist-sw01:
    hostname: 10.10.20.177
    platform: nxos_ssh
    transport: ssh
    groups:
        - devnet-cml-lab

Just two hosts are being displayed for brevity. Both hosts have IP-address, platform attributes. dist-sw01 has transport type assigned specifically. For internet-rtr01, transport type will be chosen based on platform type (it is SSH for IOS by default) by Nornir. Both hosts belong to the ‘devnet-cml-lab’ group.

groups.yml will define all group settings for them:

---

devnet-cml-lab:
    username: cisco
    password: cisco
    connection_options:
        napalm:
            extras:
                optional_args:
                    secret: cisco

Group attributes above contain access credentials and enable secret for Cisco devices. These attributes will be inherited by all group members.
Important: Never store credentials (and any sensitive data) in your production environment in clear text configs like this. This simple config is used for demonstrational and lab purposes only.
Those are all general Nornir configuration steps. All we need to do now is to initialize it from a Python code.

Downloading NeXt UI

For local usage and testing, it is enough to download the NeXt UI source code from GitHub. Let’s put the sources into ./next_sources in our project root directory.

Our progress upon download completion:

$ tree . -L 2
.
├── inventory
│   ├── groups.yml
│   └── hosts_devnet_sb_cml.yml
├── next_sources
│   ├── css
│   ├── doc
│   ├── fonts
│   └── js
├── nornir_config.yml

Age of Topology Discovery

The main logic will be written in a Python script named generate_topology.py.

Initializing Nornir

Once our Nornir config is ready, it can be initialized in Python as simply as:

from nornir import InitNornir
from nornir.plugins.tasks.networking import napalm_get

NORNIR_CONFIG_FILE = "nornir_config.yml"

nr = InitNornir(config_file=NORNIR_CONFIG_FILE)

That’s it. Nornir is ready to work. napalm_get imported above allows us to use NAPALM straight from Nornir.

LLDP at-a-glance

LLDP-enabled devices exchange periodic LLDP messages consisting of TLV-fields with their direct neighbors. LLDP messages are not normally relayed.
Mandatory TLV fields: Chassis ID, Port ID, Time-to-Live. Optional TLV fields: System Name and Description; Port Name and Description; VLAN Name; IP Management address; System Capabilities (switching, routing, etc.), and more. As the examined topology segment is under our control, let’s consider System Name and Port Name TLV fields required and advertisable internally.
It does not cause significant security risks but allows us to identify multi-chassis devices with a shared control plane (e.g. stacked switches) and device interconnections uniquely.

In this case, a topology analysis task as a whole can be reduced to the analysis of the neighborship data received on each device. This allows us to identify unique devices and their interconnections (i.e. Vertices and Edges of the topology Graph).
By the way, OSPF LSA exchange and analysis work in a very similar way. Visualizing routing protocol data may also be a good use case (I’d recommend to check out the Topolograph service released in October 2020 by @Vadims06). But let’s focus on LLDP for now.

In our lab environment, all edge, core, and distribution layer devices should see their direct LLDP neighbors. internet-rtr01 is isolated from the rest of the network so it should not have any LLDP neighbors.

Here is a manual “show lldp neighbors” output from dist-rtr01:

dist-rtr01#show lldp neighbors
Capability codes:
    (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
    (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

Device ID           Local Intf     Hold-time  Capability      Port ID
dist-rtr02.devnet.laGi6            120        R               Gi6
dist-sw01.devnet.labGi4            120        B,R             Ethernet1/3
dist-sw02.devnet.labGi5            120        B,R             Ethernet1/3
core-rtr02.devnet.laGi3            120        R               Gi0/0/0/2
core-rtr01.devnet.laGi2            120        R               Gi0/0/0/2

Total entries displayed: 5

Five neighbors. Looks good.
The same output from core-rtr02:

RP/0/0/CPU0:core-rtr02#show lldp neighbors
Sun May 10 22:07:05.776 UTC
Capability codes:
        (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
        (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other

Device ID       Local Intf          Hold-time  Capability     Port ID
core-rtr01.devnet.la Gi0/0/0/0           120        R               Gi0/0/0/0
edge-sw01.devnet.lab Gi0/0/0/1           120        R               Gi0/3
dist-rtr01.devnet.la Gi0/0/0/2           120        R               Gi3
dist-rtr02.devnet.la Gi0/0/0/3           120        R               Gi3

Total entries displayed: 4

Four neighbors. That’s correct as well.
Please note that the output contains incomplete hostnames in a Device ID column in both cases.
CLI automation always comes along with such issues.
In our given case the workaround is to use a detailed output format.
As an example:

dist-rtr01#show lldp neighbors detail
------------------------------------------------
Local Intf: Gi6
Chassis id: 001e.e57c.cf00
Port id: Gi6
Port Description: L3 Link to dist-rtr01
System Name: dist-rtr02.devnet.lab

System Description: 
Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 28-May-19 12:45

Time remaining: 91 seconds
System Capabilities: B,R
Enabled Capabilities: R
Management Addresses:
    IP: 172.16.252.18
Auto Negotiation - not supported
Physical media capabilities - not advertised
Media Attachment Unit type - not advertised
Vlan ID: - not advertised
          
------------------------------------------------
Local Intf: Gi4
Chassis id: 5254.0007.5d59
Port id: Ethernet1/3
Port Description: L3 link to dist-rtr01
System Name: dist-sw01.devnet.lab

System Description: 
Cisco Nexus Operating System (NX-OS) Software 9.2(3)
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2019, Cisco Systems, Inc. All rights reserved.

Time remaining: 108 seconds
System Capabilities: B,R
Enabled Capabilities: B,R
Management Addresses:
    IP: 10.10.20.177
    Other: 52 54 00 07 5D 59 00
Auto Negotiation - not supported
Physical media capabilities - not advertised
Media Attachment Unit type - not advertised
Vlan ID: - not advertised
          
------------------------------------------------
Local Intf: Gi5
Chassis id: 5254.0007.b7e6
Port id: Ethernet1/3
Port Description: L3 link to dist-rtr01
System Name: dist-sw02.devnet.lab

System Description: 
Cisco Nexus Operating System (NX-OS) Software 9.2(3)
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2019, Cisco Systems, Inc. All rights reserved.

Time remaining: 97 seconds
System Capabilities: B,R
Enabled Capabilities: B,R
Management Addresses:
    IP: 10.10.20.178
    Other: 52 54 00 07 FF FF 00
Auto Negotiation - not supported
Physical media capabilities - not advertised
Media Attachment Unit type - not advertised
Vlan ID: - not advertised
          
------------------------------------------------
Local Intf: Gi3
Chassis id: 02c7.9dc0.0c06
Port id: Gi0/0/0/2
Port Description: L3 Link to dist-rtr01
System Name: core-rtr02.devnet.lab

System Description: 
Cisco IOS XR Software, Version 6.3.1[Default]
Copyright (c) 2017 by Cisco Systems, Inc., IOS XRv Series

Time remaining: 94 seconds
System Capabilities: R
Enabled Capabilities: R
Management Addresses:
    IP: 172.16.252.26
Auto Negotiation - not supported
Physical media capabilities - not advertised
Media Attachment Unit type - not advertised
Vlan ID: - not advertised

------------------------------------------------
Local Intf: Gi2
Chassis id: 0288.15c0.0c06
Port id: Gi0/0/0/2
Port Description: L3 Link to dist-rtr01
System Name: core-rtr01.devnet.lab

System Description: 
Cisco IOS XR Software, Version 6.3.1[Default]
Copyright (c) 2017 by Cisco Systems, Inc., IOS XRv Series

Time remaining: 110 seconds
System Capabilities: R
Enabled Capabilities: R
Management Addresses:
    IP: 172.16.252.22
Auto Negotiation - not supported
Physical media capabilities - not advertised
Media Attachment Unit type - not advertised
Vlan ID: - not advertised


Total entries displayed: 5

dist-sw01# show lldp neighbors detail
Capability codes:
  (R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
  (W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID            Local Intf      Hold-time  Capability  Port ID  

Chassis id: 5254.0007.b7e4
Port id: Ethernet1/1
Local Port id: Eth1/1
Port Description: VPC Peer Link
System Name: dist-sw02.devnet.lab
System Description: Cisco Nexus Operating System (NX-OS) Software 9.2(3)
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2019, Cisco Systems, Inc. All rights reserved.
Time remaining: 112 seconds
System Capabilities: B, R
Enabled Capabilities: B, R
Management Address: 10.10.20.178
Management Address IPV6: not advertised
Vlan ID: 1


Chassis id: 5254.0007.b7e5
Port id: Ethernet1/2
Local Port id: Eth1/2
Port Description: VPC Peer Link
System Name: dist-sw02.devnet.lab
System Description: Cisco Nexus Operating System (NX-OS) Software 9.2(3)
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2019, Cisco Systems, Inc. All rights reserved.
Time remaining: 112 seconds
System Capabilities: B, R
Enabled Capabilities: B, R
Management Address: 10.10.20.178
Management Address IPV6: not advertised
Vlan ID: 1


Chassis id: 001e.7a2a.3900
Port id: Gi4
Local Port id: Eth1/3
Port Description: L3 Link to dist-sw01
System Name: dist-rtr01.devnet.lab
System Description: Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 28-May-19 12:45
Time remaining: 109 seconds
System Capabilities: B, R
Enabled Capabilities: R
Management Address: 172.16.252.2
Management Address IPV6: not advertised
Vlan ID: not advertised


Chassis id: 001e.e57c.cf00
Port id: Gi4
Local Port id: Eth1/4
Port Description: L3 Link to dist-sw01
System Name: dist-rtr02.devnet.lab
System Description: Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Tue 28-May-19 12:45
Time remaining: 108 seconds
System Capabilities: B, R
Enabled Capabilities: R
Management Address: 172.16.252.6
Management Address IPV6: not advertised
Vlan ID: not advertised

Total entries displayed: 4

Gathering the data from the devices

We are going to gather the LLDP data from the devices running IOS (edge-sw01), IOSXE (internet-rtr01, distr-rtr01, distr-rtr02), and NXOS (dist-sw01, dist-sw02).
IOS-XR-based core routers (core-rtr01, core-rtr02) will be intentionally restricted from management access.
Thereby the following scenarios will be covered:

1. Full mesh neighborship handling for all distribution layer devices. All unique nodes and links should be discovered properly.
2. Device access or connectivity issues handling for core-rtr01 and core-rtr02. This should not affect the ability to work with the rest of the devices.
3. Building the topology based on partial data from discontiguous network segments. Both edge switch and distribution routers “see” core-rtr01 and core-rtr02 from different sides.
   This should be enough to build the full picture.

---

internet-rtr01:
    hostname: 10.10.20.181
    platform: ios
    site: devnet_sandbox
    groups:
        - devnet-cml-lab

edge-sw01:
    hostname: 10.10.20.172
    platform: ios
    site: devnet_sandbox
    groups:
        - devnet-cml-lab

core-rtr01:
    # Device access is restricted for the test
    hostname: 10.10.20.173
    platform: iosxr
    groups:
        - devnet-cml-lab

core-rtr02:
    # Device access is restricted for the test
    hostname: 10.10.20.174
    platform: iosxr
    groups:
        - devnet-cml-lab

dist-rtr01:
    hostname: 10.10.20.175
    platform: ios
    groups:
        - devnet-cml-lab

dist-rtr02:
    hostname: 10.10.20.176
    platform: ios
    groups:
        - devnet-cml-lab

dist-sw01:
    hostname: 10.10.20.177
    platform: nxos_ssh
    transport: ssh
    groups:
        - devnet-cml-lab

dist-sw02:
    hostname: 10.10.20.178
    platform: nxos_ssh
    transport: ssh
    groups:
        - devnet-cml-lab

NAPALM GETTERs to use:

• GET_LLDP_NEIGHBORS_DETAILS.
  A detailed output version is chosen as it provides more consistent data.
• GET_FACTS.
  It collects some extended device information such as FQDN, model, serial number, etc.
  

Let’s wrap the data gathering task into a Nornir Task function.
This is one of useful method of grouping actions on the individual hosts.

def get_host_data(task):
    """Nornir Task for data collection on target hosts."""
    task.run(
        task=napalm_get,
        getters=['facts', 'lldp_neighbors_detail']
    )

Now we can run the Task and save the result into variable for a further processing.
Default execution scope is all devices.

get_host_data_result = nr.run(get_host_data)

You may also use simple and complex inventory filters to limit the execution scope to individual hosts or groups.

Processing the data received from devices

get_host_data_result variable contains a get_host_data task execution results for each target device.

>>> get_host_data_result
AggregatedResult (get_host_data): {'internet-rtr01': MultiResult: [Result: "get_host_data", Result: "napalm_get"], 'edge-sw01': MultiResult: [Result: "get_host_data", Result: "napalm_get"], 'core-rtr01': MultiResult: [Result: "get_host_data", Result: "napalm_get"], 'core-rtr02': MultiResult: [Result: "get_host_data", Result: "napalm_get"], 'dist-rtr01': MultiResult: [Result: "get_host_data", Result: "napalm_get"], 'dist-rtr02': MultiResult: [Result: "get_host_data", Result: "napalm_get"], 'dist-sw01': MultiResult: [Result: "get_host_data", Result: "napalm_get"], 'dist-sw02': MultiResult: [Result: "get_host_data", Result: "napalm_get"]}

Every host result object has failed method returning a boolean value. False means no errors occured during task execution on a specific host.
The global task result is iterable as a dictionalry object:

>>> for device, result in get_host_data_result.items():
...     print(f'{device} failed: {result.failed}')
... 
internet-rtr01 failed: False
edge-sw01 failed: False
core-rtr01 failed: True
core-rtr02 failed: True
dist-rtr01 failed: False
dist-rtr02 failed: False
dist-sw01 failed: False
dist-sw02 failed: False

Looks expectedly.

Some complete result outputs for the reference:

>>> get_host_data_result['dist-rtr01'][1].result
{'facts': {'uptime': 6120, 'vendor': 'Cisco', 'os_version': 'Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)', 'serial_number': '9JDCOVUDSWN', 'model': 'CSR1000V', 'hostname': 'dist-rtr01', 'fqdn': 'dist-rtr01.devnet.lab', 'interface_list': ['GigabitEthernet1', 'GigabitEthernet2', 'GigabitEthernet3', 'GigabitEthernet4', 'GigabitEthernet5', 'GigabitEthernet6', 'Loopback0']}, 'lldp_neighbors_detail': {'GigabitEthernet6': [{'remote_chassis_id': '001e.e57c.cf00', 'remote_port': 'Gi6', 'remote_port_description': 'L3 Link to dist-rtr01', 'remote_system_name': 'dist-rtr02.devnet.lab', 'remote_system_description': 'Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}], 'GigabitEthernet4': [{'remote_chassis_id': '5254.0007.5d59', 'remote_port': 'Ethernet1/3', 'remote_port_description': 'L3 link to dist-rtr01', 'remote_system_name': 'dist-sw01.devnet.lab', 'remote_system_description': 'Cisco Nexus Operating System (NX-OS) Software 9.2(3)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['bridge', 'router'], 'parent_interface': ''}], 'GigabitEthernet5': [{'remote_chassis_id': '5254.0007.b7e6', 'remote_port': 'Ethernet1/3', 'remote_port_description': 'L3 link to dist-rtr01', 'remote_system_name': 'dist-sw02.devnet.lab', 'remote_system_description': 'Cisco Nexus Operating System (NX-OS) Software 9.2(3)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['bridge', 'router'], 'parent_interface': ''}], 'GigabitEthernet3': [{'remote_chassis_id': '02c7.9dc0.0c06', 'remote_port': 'Gi0/0/0/2', 'remote_port_description': 'L3 Link to dist-rtr01', 'remote_system_name': 'core-rtr02.devnet.lab', 'remote_system_description': 'Cisco IOS XR Software, Version 6.3.1[Default]', 'remote_system_capab': ['router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}], 'GigabitEthernet2': [{'remote_chassis_id': '0288.15c0.0c06', 'remote_port': 'Gi0/0/0/2', 'remote_port_description': 'L3 Link to dist-rtr01', 'remote_system_name': 'core-rtr01.devnet.lab', 'remote_system_description': 'Cisco IOS XR Software, Version 6.3.1[Default]', 'remote_system_capab': ['router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}]}}

>>> get_host_data_result['dist-sw01'][1].result
{'facts': {'uptime': 6090, 'vendor': 'Cisco', 'os_version': '9.2(3)', 'serial_number': '9P5OMCCMSQ4', 'model': 'Nexus9000 9000v Chassis', 'hostname': 'dist-sw01', 'fqdn': 'dist-sw01.devnet.lab', 'interface_list': ['mgmt0', 'Ethernet1/1', 'Ethernet1/2', 'Ethernet1/3', 'Ethernet1/4', 'Ethernet1/5', 'Ethernet1/6', 'Ethernet1/7', 'Ethernet1/8', 'Ethernet1/9', 'Ethernet1/10', 'Ethernet1/11', 'Ethernet1/12', 'Ethernet1/13', 'Ethernet1/14', 'Ethernet1/15', 'Ethernet1/16', 'Ethernet1/17', 'Ethernet1/18', 'Ethernet1/19', 'Ethernet1/20', 'Ethernet1/21', 'Ethernet1/22', 'Ethernet1/23', 'Ethernet1/24', 'Ethernet1/25', 'Ethernet1/26', 'Ethernet1/27', 'Ethernet1/28', 'Ethernet1/29', 'Ethernet1/30', 'Ethernet1/31', 'Ethernet1/32', 'Ethernet1/33', 'Ethernet1/34', 'Ethernet1/35', 'Ethernet1/36', 'Ethernet1/37', 'Ethernet1/38', 'Ethernet1/39', 'Ethernet1/40', 'Ethernet1/41', 'Ethernet1/42', 'Ethernet1/43', 'Ethernet1/44', 'Ethernet1/45', 'Ethernet1/46', 'Ethernet1/47', 'Ethernet1/48', 'Ethernet1/49', 'Ethernet1/50', 'Ethernet1/51', 'Ethernet1/52', 'Ethernet1/53', 'Ethernet1/54', 'Ethernet1/55', 'Ethernet1/56', 'Ethernet1/57', 'Ethernet1/58', 'Ethernet1/59', 'Ethernet1/60', 'Ethernet1/61', 'Ethernet1/62', 'Ethernet1/63', 'Ethernet1/64', 'Ethernet1/65', 'Ethernet1/66', 'Ethernet1/67', 'Ethernet1/68', 'Ethernet1/69', 'Ethernet1/70', 'Ethernet1/71', 'Ethernet1/72', 'Ethernet1/73', 'Ethernet1/74', 'Ethernet1/75', 'Ethernet1/76', 'Ethernet1/77', 'Ethernet1/78', 'Ethernet1/79', 'Ethernet1/80', 'Ethernet1/81', 'Ethernet1/82', 'Ethernet1/83', 'Ethernet1/84', 'Ethernet1/85', 'Ethernet1/86', 'Ethernet1/87', 'Ethernet1/88', 'Ethernet1/89', 'Ethernet1/90', 'Ethernet1/91', 'Ethernet1/92', 'Ethernet1/93', 'Ethernet1/94', 'Ethernet1/95', 'Ethernet1/96', 'Ethernet1/97', 'Ethernet1/98', 'Ethernet1/99', 'Ethernet1/100', 'Ethernet1/101', 'Ethernet1/102', 'Ethernet1/103', 'Ethernet1/104', 'Ethernet1/105', 'Ethernet1/106', 'Ethernet1/107', 'Ethernet1/108', 'Ethernet1/109', 'Ethernet1/110', 'Ethernet1/111', 'Ethernet1/112', 'Ethernet1/113', 'Ethernet1/114', 'Ethernet1/115', 'Ethernet1/116', 'Ethernet1/117', 'Ethernet1/118', 'Ethernet1/119', 'Ethernet1/120', 'Ethernet1/121', 'Ethernet1/122', 'Ethernet1/123', 'Ethernet1/124', 'Ethernet1/125', 'Ethernet1/126', 'Ethernet1/127', 'Ethernet1/128', 'Port-channel1', 'Loopback0', 'Vlan1', 'Vlan101', 'Vlan102', 'Vlan103', 'Vlan104', 'Vlan105']}, 'lldp_neighbors_detail': {'Ethernet1/1': [{'remote_chassis_id': '5254.0007.b7e4', 'remote_port': 'Ethernet1/1', 'remote_port_description': 'VPC Peer Link', 'remote_system_name': 'dist-sw02.devnet.lab', 'remote_system_description': 'Cisco Nexus Operating System (NX-OS) Software 9.2(3)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['bridge', 'router'], 'parent_interface': ''}], 'Ethernet1/2': [{'remote_chassis_id': '5254.0007.b7e5', 'remote_port': 'Ethernet1/2', 'remote_port_description': 'VPC Peer Link', 'remote_system_name': 'dist-sw02.devnet.lab', 'remote_system_description': 'Cisco Nexus Operating System (NX-OS) Software 9.2(3)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['bridge', 'router'], 'parent_interface': ''}], 'Ethernet1/3': [{'remote_chassis_id': '001e.7a2a.3900', 'remote_port': 'Gi4', 'remote_port_description': 'L3 Link to dist-sw01', 'remote_system_name': 'dist-rtr01.devnet.lab', 'remote_system_description': 'Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}], 'Ethernet1/4': [{'remote_chassis_id': '001e.e57c.cf00', 'remote_port': 'Gi4', 'remote_port_description': 'L3 Link to dist-sw01', 'remote_system_name': 'dist-rtr02.devnet.lab', 'remote_system_description': 'Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}]}}

Result object is a dictionary with a keys matching selected GETTER names: ‘facts’ and ‘lldp_neighbors_detail’.
Dictionary values contain a structured data processed and returned by NAPALM.
Let’s compare a neighbor sets:

>>> for neighbor in get_host_data_result['dist-rtr01'][1].result['lldp_neighbors_detail'].items():
...     print(neighbor)
...     print('\n')
... 
('GigabitEthernet6', [{'remote_chassis_id': '001e.e57c.cf00', 'remote_port': 'Gi6', 'remote_port_description': 'L3 Link to dist-rtr01', 'remote_system_name': 'dist-rtr02.devnet.lab', 'remote_system_description': 'Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}])

('GigabitEthernet4', [{'remote_chassis_id': '5254.0007.5d59', 'remote_port': 'Ethernet1/3', 'remote_port_description': 'L3 link to dist-rtr01', 'remote_system_name': 'dist-sw01.devnet.lab', 'remote_system_description': 'Cisco Nexus Operating System (NX-OS) Software 9.2(3)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['bridge', 'router'], 'parent_interface': ''}])

('GigabitEthernet5', [{'remote_chassis_id': '5254.0007.b7e6', 'remote_port': 'Ethernet1/3', 'remote_port_description': 'L3 link to dist-rtr01', 'remote_system_name': 'dist-sw02.devnet.lab', 'remote_system_description': 'Cisco Nexus Operating System (NX-OS) Software 9.2(3)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['bridge', 'router'], 'parent_interface': ''}])

('GigabitEthernet3', [{'remote_chassis_id': '02c7.9dc0.0c06', 'remote_port': 'Gi0/0/0/2', 'remote_port_description': 'L3 Link to dist-rtr01', 'remote_system_name': 'core-rtr02.devnet.lab', 'remote_system_description': 'Cisco IOS XR Software, Version 6.3.1[Default]', 'remote_system_capab': ['router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}])

('GigabitEthernet2', [{'remote_chassis_id': '0288.15c0.0c06', 'remote_port': 'Gi0/0/0/2', 'remote_port_description': 'L3 Link to dist-rtr01', 'remote_system_name': 'core-rtr01.devnet.lab', 'remote_system_description': 'Cisco IOS XR Software, Version 6.3.1[Default]', 'remote_system_capab': ['router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}])

>>> for neighbor in get_host_data_result['dist-sw01'][1].result['lldp_neighbors_detail'].items():
...     print(neighbor)
...     print('\n')
... 
('Ethernet1/1', [{'remote_chassis_id': '5254.0007.b7e4', 'remote_port': 'Ethernet1/1', 'remote_port_description': 'VPC Peer Link', 'remote_system_name': 'dist-sw02.devnet.lab', 'remote_system_description': 'Cisco Nexus Operating System (NX-OS) Software 9.2(3)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['bridge', 'router'], 'parent_interface': ''}])

('Ethernet1/2', [{'remote_chassis_id': '5254.0007.b7e5', 'remote_port': 'Ethernet1/2', 'remote_port_description': 'VPC Peer Link', 'remote_system_name': 'dist-sw02.devnet.lab', 'remote_system_description': 'Cisco Nexus Operating System (NX-OS) Software 9.2(3)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['bridge', 'router'], 'parent_interface': ''}])

('Ethernet1/3', [{'remote_chassis_id': '001e.7a2a.3900', 'remote_port': 'Gi4', 'remote_port_description': 'L3 Link to dist-sw01', 'remote_system_name': 'dist-rtr01.devnet.lab', 'remote_system_description': 'Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}])

('Ethernet1/4', [{'remote_chassis_id': '001e.e57c.cf00', 'remote_port': 'Gi4', 'remote_port_description': 'L3 Link to dist-sw01', 'remote_system_name': 'dist-rtr02.devnet.lab', 'remote_system_description': 'Cisco IOS Software [Gibraltar], Virtual XE Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.11.1b, RELEASE SOFTWARE (fc2)', 'remote_system_capab': ['bridge', 'router'], 'remote_system_enable_capab': ['router'], 'parent_interface': ''}])

Five neighbors for dist-rtr01 and four neighbors for dist-sw01 which is exactly what we saw in CLI outputs before.
The rest of the data is also valid.

For ease of processing let’s split LLDP and facts data into separate entities.
Any devices may be also present in multiple outputs. To distinguish between them, it is necessary to use some unique node identifiers. Let’s choose it in the following descending order:

• Device FQDN if available (may be further referred to as hostname for simplicity).
• Device hostname if available.
• Device host object name from Nornir Inventory.

LLDP relies on the first two steps too.

def normalize_result(nornir_job_result):
    """
    get_host_data result parser.
    Returns LLDP and FACTS data dicts
    with hostname keys.
    """
    global_lldp_data = {}
    global_facts = {}
    for device, output in nornir_job_result.items():
        if output[0].failed:
            # Write default data to dicts if the task is failed.
            # Use the host inventory object name as a key.
            global_lldp_data[device] = {}
            global_facts[device] = {
                'nr_ip': nr.inventory.hosts[device].get('hostname', 'n/a'),
            }
            continue
        # Use FQDN as unique ID for devices withing the script.
        device_fqdn = output[1].result['facts']['fqdn']
        if not device_fqdn:
            # If FQDN is not set use hostname.
            # LLDP TLV follows the same logic.
            device_fqdn = output[1].result['facts']['hostname']
        if not device_fqdn:
            # Use host inventory object name as a key if
            # neither FQDN nor hostname are set
            device_fqdn = device
        global_facts[device_fqdn] = output[1].result['facts']
        # Populate device facts with its IP address or hostname as per Inventory data
        global_facts[device_fqdn]['nr_ip'] = nr.inventory.hosts[device].get('hostname', 'n/a')
        global_lldp_data[device_fqdn] = output[1].result['lldp_neighbors_detail']
    return global_lldp_data, global_facts

Then we should extract a list of all neighbors known by devices and build based on that:

• A list of unique hosts.
• A list of unique links between them.

To ensure unambiguous identification of links, we will store them in a following format:
((source_device_id, source_port_name), (destination_device_id, destination_port_name))

It is also necessary to keep in mind that:

• A link may be visible from two sides if we collect the data from both devices it is connected to.
  We have to check for side A and side B permutations to filter out the duplicates.
• A port name may be formatted differently in the LLDP announcement and local outputs. For example, GigabitEthernet4 locally and Gi4 in LLDP Port name.

To ensure data consistency, we will translate the port names into a full format for the analysis stage. At the same time, let’s implement a name shortening function to provide better visual experience during visualization.
Automatic icon selection can be implemented based on device capabilities advertised in LLDP. Let’s extract them into a separate {“hostname”: “primary_capability”} dictionary.
The same code wise:

interface_full_name_map = {
    'Eth': 'Ethernet',
    'Fa': 'FastEthernet',
    'Gi': 'GigabitEthernet',
    'Te': 'TenGigabitEthernet',
}

def if_fullname(ifname):
    for k, v in interface_full_name_map.items():
        if ifname.startswith(v):
            return ifname
        if ifname.startswith(k):
            return ifname.replace(k, v)
    return ifname

def if_shortname(ifname):
    for k, v in interface_full_name_map.items():
        if ifname.startswith(v):
            return ifname.replace(v, k)
    return ifname


def extract_lldp_details(lldp_data_dict):
    """
    LLDP data dict parser.
    Returns set of all the discovered hosts,
    LLDP capabilities dict with all LLDP-discovered host,
    and all discovered interconnections between hosts.
    """
    discovered_hosts = set()
    lldp_capabilities_dict = {}
    global_interconnections = []
    for host, lldp_data in lldp_data_dict.items():
        if not host:
            continue
        discovered_hosts.add(host)
        if not lldp_data:
            continue
        for interface, neighbors in lldp_data.items():
            for neighbor in neighbors:
                if not neighbor['remote_system_name']:
                    continue
                discovered_hosts.add(neighbor['remote_system_name'])
                if neighbor['remote_system_enable_capab']:
                    # In case of multiple enable capabilities pick first in the list
                    lldp_capabilities_dict[neighbor['remote_system_name']] = (
                        neighbor['remote_system_enable_capab'][0]
                    )
                else:
                    lldp_capabilities_dict[neighbor['remote_system_name']] = ''
                # Store interconnections in a following format:
                # ((source_hostname, source_port), (dest_hostname, dest_port))
                local_end = (host, interface)
                remote_end = (
                    neighbor['remote_system_name'],
                    if_fullname(neighbor['remote_port'])
                )
                # Check if the link is not a permutation of already added one
                # (local_end, remote_end) equals (remote_end, local_end)
                link_is_already_there = (
                    (local_end, remote_end) in global_interconnections
                    or (remote_end, local_end) in global_interconnections
                )
                if link_is_already_there:
                    continue
                global_interconnections.append((
                    (host, interface),
                    (neighbor['remote_system_name'], if_fullname(neighbor['remote_port']))
                ))
    return [discovered_hosts, global_interconnections, lldp_capabilities_dict]

Initializing NeXt UI application

Topology visualization logic will be implemented in next_app.js script based on Next UI.
Let’s start with the basics:

(function (nx) {
    /**
     * NeXt UI based application
     */
    // Initialize topology
    var topo = new nx.graphic.Topology({
        // View dimensions
        width: 1200,
        height: 700,
        // Dataprocessor is responsible for spreading 
        // the Nodes across the view.
        // 'force' data processor spreads the Nodes so
        // they would be as distant from each other
        // as possible. Follow social distancing and stay healthy.
        // 'quick' dataprocessor picks random positions
        // for the Nodes.
        dataProcessor: 'force',
        // Node and Link identity key attribute name
        identityKey: 'id',
        // Node settings
        nodeConfig: {
            label: 'model.name',
            iconType:'model.icon',
        },
        // Link settings
        linkConfig: {
            // Display Links as curves in case of 
            // multiple links between Node Pairs.
            // Set to 'parallel' to use parallel links.
            linkType: 'curve',
        },
        // Display Node icon. Displays a dot if set to 'false'.
        showIcon: true,
    });

    var Shell = nx.define(nx.ui.Application, {
        methods: {
            start: function () {
                // Read topology data from variable
                topo.data(topologyData);
                // Attach it to the document
                topo.attach(this);
            }
        }
    });

    // Create an application instance
    var shell = new Shell();
    // Run the application
    shell.start();
})(nx);

The topology data structure will be stored in a topologyData variable. Let’s move it into a separate topology.js file. The format details will be discussed below.

A final visualization result will be displayed in a local HTML form with attached JS components:

<!DOCTYPE html>

<html>
    <head>
        <meta charset="utf-8">
        <link rel="stylesheet" href="next_sources/css/next.css">
        <link rel="stylesheet" href="styles_main_page.css">
        <script src="next_sources/js/next.js"></script>
        <script src="topology.js"></script>
        <script src="next_app.js"></script>
    </head>
    <body>
    </body>
</html>

Generating NeXt UI topology in Python

We have already written required data gathering result handlers and normalized it using Python data structures.
Let’s apply this:

GLOBAL_LLDP_DATA, GLOBAL_FACTS = normalize_result(get_host_data_result)
TOPOLOGY_DETAILS = extract_lldp_details(GLOBAL_LLDP_DATA)

General NeXt UI topology representation looks as follows:

// Two nodes connected with two links
var topologyData = {
    "links": [
        {
            "id": 0,
            "source": 0,
            "target": 1,
        }, {
            "id": 1,
            "source": 0,
            "target": 1,
        }
    ],
    "nodes": [
        {
            "icon": "router",
            "id": 0,
        },
        {
            "icon": "router",
            "id": 1,
        }
    ]

As you can see, this is a JSON object which can be mapped to a Python data structure of the following format: {‘nodes’: [], ‘links’: []}.
We will bring all our data together into that.
In addition, let’s take device model into account for node icon selection to handle missing LLDP capabilities case.
Node objects will also be populated with some extended attributes derived from GET_FACTS (model, S/N, etc) to enrich the topology view.

icon_capability_map = {
    'router': 'router',
    'switch': 'switch',
    'bridge': 'switch',
    'station': 'host'
}

icon_model_map = {
    'CSR1000V': 'router',
    'Nexus': 'switch',
    'IOSXRv': 'router',
    'IOSv': 'switch',
    '2901': 'router',
    '2911': 'router',
    '2921': 'router',
    '2951': 'router',
    '4321': 'router',
    '4331': 'router',
    '4351': 'router',
    '4421': 'router',
    '4431': 'router',
    '4451': 'router',
    '2960': 'switch',
    '3750': 'switch',
    '3850': 'switch',
}

def get_icon_type(device_cap_name, device_model=''):
    """
    Device icon selection function. Selection order:
    - LLDP capabilities mapping.
    - Device model mapping.
    - Default 'unknown'.
    """
    if device_cap_name:
        icon_type = icon_capability_map.get(device_cap_name)
        if icon_type:
            return icon_type
    if device_model:
        # Check substring presence in icon_model_map keys
        # string until the first match
        for model_shortname, icon_type in icon_model_map.items():
            if model_shortname in device_model:
                return icon_type
    return 'unknown'

def generate_topology_json(*args):
    """
    JSON topology object generator.
    Takes as an input:
    - discovered hosts set,
    - LLDP capabilities dict with hostname keys,
    - interconnections list,
    - facts dict with hostname keys.
    """
    discovered_hosts, interconnections, lldp_capabilities_dict, facts = args
    host_id = 0
    host_id_map = {}
    topology_dict = {'nodes': [], 'links': []}
    for host in discovered_hosts:
        device_model = 'n/a'
        device_serial = 'n/a'
        device_ip = 'n/a'
        if facts.get(host):
            device_model = facts[host].get('model', 'n/a')
            device_serial = facts[host].get('serial_number', 'n/a')
            device_ip = facts[host].get('nr_ip', 'n/a')
        host_id_map[host] = host_id
        topology_dict['nodes'].append({
            'id': host_id,
            'name': host,
            'primaryIP': device_ip,
            'model': device_model,
            'serial_number': device_serial,
            'icon': get_icon_type(
                lldp_capabilities_dict.get(host, ''),
                device_model
            )
        })
        host_id += 1
    link_id = 0
    for link in interconnections:
        topology_dict['links'].append({
            'id': link_id,
            'source': host_id_map[link[0][0]],
            'target': host_id_map[link[1][0]],
            'srcIfName': if_shortname(link[0][1]),
            'srcDevice': link[0][0],
            'tgtIfName': if_shortname(link[1][1]),
            'tgtDevice': link[1][0],
        })
        link_id += 1
    return topology_dict

Then we should just write this Python topology dictionary into the topology.js file. A standard ** json ** module will work for this perfectly providing a readable and formatted output:

import json

OUTPUT_TOPOLOGY_FILENAME = 'topology.js'
TOPOLOGY_FILE_HEAD = "\n\nvar topologyData = "

def write_topology_file(topology_json, header=TOPOLOGY_FILE_HEAD, dst=OUTPUT_TOPOLOGY_FILENAME):
    with open(dst, 'w') as topology_file:
        topology_file.write(header)
        topology_file.write(json.dumps(topology_json, indent=4, sort_keys=True))
        topology_file.write(';')

TOPOLOGY_DICT = generate_topology_json(*TOPOLOGY_DETAILS)
write_topology_file(TOPOLOGY_DICT)

var topologyData = {
    "links": [
        {
            "id": 0,
            "source": 7,
            "srcDevice": "edge-sw01.devnet.lab",
            "srcIfName": "Gi0/2",
            "target": 5,
            "tgtDevice": "core-rtr01.devnet.lab",
            "tgtIfName": "Gi0/0/0/1"
        },
        {
            "id": 1,
            "source": 7,
            "srcDevice": "edge-sw01.devnet.lab",
            "srcIfName": "Gi0/3",
            "target": 3,
            "tgtDevice": "core-rtr02.devnet.lab",
            "tgtIfName": "Gi0/0/0/1"
        },
        {
            "id": 2,
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi3",
            "target": 3,
            "tgtDevice": "core-rtr02.devnet.lab",
            "tgtIfName": "Gi0/0/0/2"
        },
        {
            "id": 3,
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi4",
            "target": 1,
            "tgtDevice": "dist-sw01.devnet.lab",
            "tgtIfName": "Eth1/3"
        },
        {
            "id": 4,
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi6",
            "target": 0,
            "tgtDevice": "dist-rtr02.devnet.lab",
            "tgtIfName": "Gi6"
        },
        {
            "id": 5,
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi5",
            "target": 2,
            "tgtDevice": "dist-sw02.devnet.lab",
            "tgtIfName": "Eth1/3"
        },
        {
            "id": 6,
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi2",
            "target": 5,
            "tgtDevice": "core-rtr01.devnet.lab",
            "tgtIfName": "Gi0/0/0/2"
        },
        {
            "id": 7,
            "source": 0,
            "srcDevice": "dist-rtr02.devnet.lab",
            "srcIfName": "Gi3",
            "target": 3,
            "tgtDevice": "core-rtr02.devnet.lab",
            "tgtIfName": "Gi0/0/0/3"
        },
        {
            "id": 8,
            "source": 0,
            "srcDevice": "dist-rtr02.devnet.lab",
            "srcIfName": "Gi4",
            "target": 1,
            "tgtDevice": "dist-sw01.devnet.lab",
            "tgtIfName": "Eth1/4"
        },
        {
            "id": 9,
            "source": 0,
            "srcDevice": "dist-rtr02.devnet.lab",
            "srcIfName": "Gi5",
            "target": 2,
            "tgtDevice": "dist-sw02.devnet.lab",
            "tgtIfName": "Eth1/4"
        },
        {
            "id": 10,
            "source": 0,
            "srcDevice": "dist-rtr02.devnet.lab",
            "srcIfName": "Gi2",
            "target": 5,
            "tgtDevice": "core-rtr01.devnet.lab",
            "tgtIfName": "Gi0/0/0/3"
        },
        {
            "id": 11,
            "source": 1,
            "srcDevice": "dist-sw01.devnet.lab",
            "srcIfName": "Eth1/1",
            "target": 2,
            "tgtDevice": "dist-sw02.devnet.lab",
            "tgtIfName": "Eth1/1"
        },
        {
            "id": 12,
            "source": 1,
            "srcDevice": "dist-sw01.devnet.lab",
            "srcIfName": "Eth1/2",
            "target": 2,
            "tgtDevice": "dist-sw02.devnet.lab",
            "tgtIfName": "Eth1/2"
        }
    ],
    "nodes": [
        {
            "icon": "router",
            "id": 0,
            "model": "CSR1000V",
            "name": "dist-rtr02.devnet.lab",
            "serial_number": "9YZKNQKQ566",
            "layerSortPreference": 7,
            "primaryIP": "10.10.20.176",
            "dcimDeviceLink": "http://localhost:32768/dcim/devices/?q=dist-rtr02.devnet.lab"
        },
        {
            "icon": "switch",
            "id": 1,
            "model": "Nexus9000 9000v Chassis",
            "name": "dist-sw01.devnet.lab",
            "serial_number": "9MZLNM0ZC9Z",
        },
        {
            "icon": "switch",
            "id": 2,
            "model": "Nexus9000 9000v Chassis",
            "name": "dist-sw02.devnet.lab",
            "serial_number": "93LCGCRUJA5",
        },
        {
            "icon": "router",
            "id": 3,
            "model": "n/a",
            "name": "core-rtr02.devnet.lab",
            "serial_number": "n/a",
        },
        {
            "icon": "router",
            "id": 4,
            "model": "CSR1000V",
            "name": "dist-rtr01.devnet.lab",
            "serial_number": "9S78ZRF2V2B",
        },
        {
            "icon": "router",
            "id": 5,
            "model": "n/a",
            "name": "core-rtr01.devnet.lab",
            "serial_number": "n/a",
        },
        {
            "icon": "router",
            "id": 6,
            "model": "CSR1000V",
            "name": "internet-rtr01.virl.info",
            "serial_number": "9LGWPM8GTV6",
        },
        {
            "icon": "switch",
            "id": 7,
            "model": "IOSv",
            "name": "edge-sw01.devnet.lab",
            "serial_number": "927A4RELIGI",
        }
    ]
};

Now let’s finally run main.html and see our visualization Hello World:
Looks correct. All known nodes and links between them are displayed.
The nodes are draggable in any direction. On mouse click on nodes and links, a Next UI tooltip menu appears. It contains all the attributes we previously passed into node and link topology objects in Python:

Not so bad. There is also room for improvement. We will return to this some later on. Let’s implement a solution for the second part of the task for now.

Detecting and visualizing topology changes

A bonus task is to detect and visualize topology changes.
To get it done, some additions will be required:

• A topology cache file cached_topology.js to store a previous topology state. generate_topology.py script will be reading this cache file on each run and rewriting with a newer state if necessary.
• diff_topology.js topology file to store a diff topology.
• diff_page.html page to display a visualized topology diff.

Diff HTML-form will look as follows:

<!DOCTYPE html>

<html>
    <head>
        <meta charset="utf-8">
        <link rel="stylesheet" href="next_sources/css/next.css">
        <link rel="stylesheet" href="styles_main_page.css">
        <script src="next_sources/js/next.js"></script>
        <script src="diff_topology.js"></script>
        <script src="next_app.js"></script>
    </head>
    <body>
        <a href="main.html"><button>Display current topology</button></a>
        </p>
    </body>
</html>

All we need to read and write topology cache files:

CACHED_TOPOLOGY_FILENAME = 'cached_topology.json'

def write_topology_cache(topology_json, dst=CACHED_TOPOLOGY_FILENAME):
    with open(dst, 'w') as cached_file:
        cached_file.write(json.dumps(topology_json, indent=4, sort_keys=True))

def read_cached_topology(filename=CACHED_TOPOLOGY_FILENAME):
    if not os.path.exists(filename):
        return {}
    if not os.path.isfile(filename):
        return {}
    cached_topology = {}
    with open(filename, 'r') as file:
        try:
            cached_topology = json.loads(file.read())
        except:
            return {}
    return cached_topology

Topology diff analysis steps:

1. Extract the node and link attributes for comparison from the current and cached topology dictionaries.
   Node format:
   (node object with all attributes, (hostname,))
   Link format:
   (link object with all attributes, (source_hostnme, source_port), (dest_hostname, dest_port))
   Both node and link format allows further extension.

2. Compare the extracted node and link objects. Link format permutations should be considered.
   Diff result for nodes and links will be written into two dictionaries in the following format:
   • diff_nodes = {‘added’: [], ‘deleted’: []}
   • diff_links = {‘added’: [], ‘deleted’: []}
3. Merge current and cached topology with the diff data.
   The resulting topology will be written to diff_merged_topology dictionary.
   Deleted node and link objects will be extended with the is_dead attribute. For a better visual experience, deleted node icons will be customized (Next UI changes for that will be discussed below).
   New node and link objects will be extended with the is_new attribute.
   

Let’s code:

def get_topology_diff(cached, current):
    """
    Topology diff analyzer and generator.
    Accepts two valid topology dicts as an input.
    Returns:
    - dict with added and deleted nodes,
    - dict with added and deleted links,
    - dict with merged input topologies with extended
      attributes for topology changes visualization
    """
    diff_nodes = {'added': [], 'deleted': []}
    diff_links = {'added': [], 'deleted': []}
    diff_merged_topology = {'nodes': [], 'links': []}
    # Parse links from topology dicts into the following format:
    # (topology_link_obj, (source_hostnme, source_port), (dest_hostname, dest_port))
    cached_links = [(x, ((x['srcDevice'], x['srcIfName']), (x['tgtDevice'], x['tgtIfName']))) for x in cached['links']]
    links = [(x, ((x['srcDevice'], x['srcIfName']), (x['tgtDevice'], x['tgtIfName']))) for x in current['links']]
    # Parse nodes from topology dicts into the following format:
    # (topology_node_obj, (hostname,))
    # Some additional values might be added for comparison later on to the tuple above.
    cached_nodes = [(x, (x['name'],)) for x in cached['nodes']]
    nodes = [(x, (x['name'],)) for x in current['nodes']]
    # Search for deleted and added hostnames.
    node_id = 0
    host_id_map = {}
    for raw_data, node in nodes:
        if node in [x[1] for x in cached_nodes]:
            raw_data['id'] = node_id
            host_id_map[raw_data['name']] = node_id
            raw_data['is_new'] = 'no'
            raw_data['is_dead'] = 'no'
            diff_merged_topology['nodes'].append(raw_data)
            node_id += 1
            continue
        diff_nodes['added'].append(node)
        raw_data['id'] = node_id
        host_id_map[raw_data['name']] = node_id
        raw_data['is_new'] = 'yes'
        raw_data['is_dead'] = 'no'
        diff_merged_topology['nodes'].append(raw_data)
        node_id += 1
    for raw_data, cached_node in cached_nodes:
        if cached_node in [x[1] for x in nodes]:
            continue
        diff_nodes['deleted'].append(cached_node)
        raw_data['id'] = node_id
        host_id_map[raw_data['name']] = node_id
        raw_data['is_new'] = 'no'
        raw_data['is_dead'] = 'yes'
        raw_data['icon'] = 'dead_node'
        diff_merged_topology['nodes'].append(raw_data)
        node_id += 1
    # Search for deleted and added interconnections.
    # Interface change on some side is considered as
    # one interconnection deletion and one interconnection insertion.
    # Check for permutations as well:
    # ((h1, Gi1), (h2, Gi2)) and ((h2, Gi2), (h1, Gi1)) are equal.
    link_id = 0
    for raw_data, link in links:
        src, dst = link
        if not (src, dst) in [x[1] for x in cached_links] and not (dst, src) in [x[1] for x in cached_links]:
            diff_links['added'].append((src, dst))
            raw_data['id'] = link_id
            link_id += 1
            raw_data['source'] = host_id_map[src[0]]
            raw_data['target'] = host_id_map[dst[0]]
            raw_data['is_new'] = 'yes'
            raw_data['is_dead'] = 'no'
            diff_merged_topology['links'].append(raw_data)
            continue
        raw_data['id'] = link_id
        link_id += 1
        raw_data['source'] = host_id_map[src[0]]
        raw_data['target'] = host_id_map[dst[0]]
        raw_data['is_new'] = 'no'
        raw_data['is_dead'] = 'no'
        diff_merged_topology['links'].append(raw_data)
    for raw_data, link in cached_links:
        src, dst = link
        if not (src, dst) in [x[1] for x in links] and not (dst, src) in [x[1] for x in links]:
            diff_links['deleted'].append((src, dst))
            raw_data['id'] = link_id
            link_id += 1
            raw_data['source'] = host_id_map[src[0]]
            raw_data['target'] = host_id_map[dst[0]]
            raw_data['is_new'] = 'no'
            raw_data['is_dead'] = 'yes'
            diff_merged_topology['links'].append(raw_data)
    return diff_nodes, diff_links, diff_merged_topology

get_topology_diff implements comparison of two arbitrary topology dictionaries of a valid format
This allows us to implement a topology cache versioning in the future.
Let’s also implement a console diff print function:

def print_diff(diff_result):
    """
    Formatted get_topology_diff result
    console print function.
    """
    diff_nodes, diff_links, *ignore = diff_result
    if not (diff_nodes['added'] or diff_nodes['deleted'] or diff_links['added'] or diff_links['deleted']):
        print('No topology changes since last run.')
        return
    print('Topology changes have been discovered:')
    if diff_nodes['added']:
        print('')
        print('^^^^^^^^^^^^^^^^^^^^')
        print('New Network Devices:')
        print('vvvvvvvvvvvvvvvvvvvv')
        for node in diff_nodes['added']:
            print(f'Hostname: {node[0]}')
    if diff_nodes['deleted']:
        print('')
        print('^^^^^^^^^^^^^^^^^^^^^^^^')
        print('Deleted Network Devices:')
        print('vvvvvvvvvvvvvvvvvvvvvvvv')
        for node in diff_nodes['deleted']:
            print(f'Hostname: {node[0]}')
    if diff_links['added']:
        print('')
        print('^^^^^^^^^^^^^^^^^^^^^^')
        print('New Interconnections:')
        print('vvvvvvvvvvvvvvvvvvvvvv')
        for src, dst in diff_links['added']:
            print(f'From {src[0]}({src[1]}) To {dst[0]}({dst[1]})')
    if diff_links['deleted']:
        print('')
        print('^^^^^^^^^^^^^^^^^^^^^^^^^')
        print('Deleted Interconnections:')
        print('vvvvvvvvvvvvvvvvvvvvvvvvv')
        for src, dst in diff_links['deleted']:
            print(f'From {src[0]}({src[1]}) To {dst[0]}({dst[1]})')
    print('')

Finally, let’s summarize the code pieces we have written above into a dedicated main() function.
Here are a fairly self-documenting code and my personal answer to a question “why not Ansible”:

def good_luck_have_fun():
    """Main script logic"""
    get_host_data_result = nr.run(get_host_data)
    GLOBAL_LLDP_DATA, GLOBAL_FACTS = normalize_result(get_host_data_result)
    TOPOLOGY_DETAILS = extract_lldp_details(GLOBAL_LLDP_DATA)
    TOPOLOGY_DETAILS.append(GLOBAL_FACTS)
    TOPOLOGY_DICT = generate_topology_json(*TOPOLOGY_DETAILS)
    CACHED_TOPOLOGY = read_cached_topology()
    write_topology_file(TOPOLOGY_DICT)
    write_topology_cache(TOPOLOGY_DICT)
    print('Open main.html in a project root with your browser to view the topology')
    if CACHED_TOPOLOGY:
        DIFF_DATA = get_topology_diff(CACHED_TOPOLOGY, TOPOLOGY_DICT)
        print_diff(DIFF_DATA)
        write_topology_file(DIFF_DATA[2], dst='diff_topology.js')
        if topology_is_changed:
            print('Open diff_page.html in a project root to view the changes.')
            print("Optionally, open main.html and click 'Display diff' button")
    else:
        # write current topology to diff file if the cache is missing
        write_topology_file(TOPOLOGY_DICT, dst='diff_topology.js')

if __name__ == '__main__':
    good_luck_have_fun()

Testing

To begin with, let’s restrict the access to dist-rtr01 and run the script. Resulting topology:
Then let’s recover the access to dist-rtr02, restrict the access to edge-sw01, and execute the script again.
The previous version becomes cached. The current topology looks as follows:

var topologyData = {
    "links": [
        {
            "id": 0,
            "is_dead": "no",
            "is_new": "yes",
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi3",
            "target": 3,
            "tgtDevice": "core-rtr02.devnet.lab",
            "tgtIfName": "Gi0/0/0/2"
        },
        {
            "id": 1,
            "is_dead": "no",
            "is_new": "yes",
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi4",
            "target": 1,
            "tgtDevice": "dist-sw01.devnet.lab",
            "tgtIfName": "Eth1/3"
        },
        {
            "id": 2,
            "is_dead": "no",
            "is_new": "yes",
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi6",
            "target": 0,
            "tgtDevice": "dist-rtr02.devnet.lab",
            "tgtIfName": "Gi6"
        },
        {
            "id": 3,
            "is_dead": "no",
            "is_new": "yes",
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi5",
            "target": 2,
            "tgtDevice": "dist-sw02.devnet.lab",
            "tgtIfName": "Eth1/3"
        },
        {
            "id": 4,
            "is_dead": "no",
            "is_new": "yes",
            "source": 4,
            "srcDevice": "dist-rtr01.devnet.lab",
            "srcIfName": "Gi2",
            "target": 5,
            "tgtDevice": "core-rtr01.devnet.lab",
            "tgtIfName": "Gi0/0/0/2"
        },
        {
            "id": 5,
            "is_dead": "no",
            "is_new": "no",
            "source": 0,
            "srcDevice": "dist-rtr02.devnet.lab",
            "srcIfName": "Gi3",
            "target": 3,
            "tgtDevice": "core-rtr02.devnet.lab",
            "tgtIfName": "Gi0/0/0/3"
        },
        {
            "id": 6,
            "is_dead": "no",
            "is_new": "no",
            "source": 0,
            "srcDevice": "dist-rtr02.devnet.lab",
            "srcIfName": "Gi4",
            "target": 1,
            "tgtDevice": "dist-sw01.devnet.lab",
            "tgtIfName": "Eth1/4"
        },
        {
            "id": 7,
            "is_dead": "no",
            "is_new": "no",
            "source": 0,
            "srcDevice": "dist-rtr02.devnet.lab",
            "srcIfName": "Gi5",
            "target": 2,
            "tgtDevice": "dist-sw02.devnet.lab",
            "tgtIfName": "Eth1/4"
        },
        {
            "id": 8,
            "is_dead": "no",
            "is_new": "no",
            "source": 0,
            "srcDevice": "dist-rtr02.devnet.lab",
            "srcIfName": "Gi2",
            "target": 5,
            "tgtDevice": "core-rtr01.devnet.lab",
            "tgtIfName": "Gi0/0/0/3"
        },
        {
            "id": 9,
            "is_dead": "no",
            "is_new": "no",
            "source": 1,
            "srcDevice": "dist-sw01.devnet.lab",
            "srcIfName": "Eth1/1",
            "target": 2,
            "tgtDevice": "dist-sw02.devnet.lab",
            "tgtIfName": "Eth1/1"
        },
        {
            "id": 10,
            "is_dead": "no",
            "is_new": "no",
            "source": 1,
            "srcDevice": "dist-sw01.devnet.lab",
            "srcIfName": "Eth1/2",
            "target": 2,
            "tgtDevice": "dist-sw02.devnet.lab",
            "tgtIfName": "Eth1/2"
        },
        {
            "id": 11,
            "is_dead": "yes",
            "is_new": "no",
            "source": 7,
            "srcDevice": "edge-sw01.devnet.lab",
            "srcIfName": "Gi0/2",
            "target": 5,
            "tgtDevice": "core-rtr01.devnet.lab",
            "tgtIfName": "Gi0/0/0/1"
        },
        {
            "id": 12,
            "is_dead": "yes",
            "is_new": "no",
            "source": 7,
            "srcDevice": "edge-sw01.devnet.lab",
            "srcIfName": "Gi0/3",
            "target": 3,
            "tgtDevice": "core-rtr02.devnet.lab",
            "tgtIfName": "Gi0/0/0/1"
        }
    ],
    "nodes": [
        {
            "icon": "router",
            "id": 0,
            "is_dead": "no",
            "is_new": "no",
            "model": "CSR1000V",
            "name": "dist-rtr02.devnet.lab",
            "serial_number": "9YZKNQKQ566",
        },
        {
            "icon": "switch",
            "id": 1,
            "is_dead": "no",
            "is_new": "no",
            "model": "Nexus9000 9000v Chassis",
            "name": "dist-sw01.devnet.lab",
            "serial_number": "9MZLNM0ZC9Z",
        },
        {
            "icon": "switch",
            "id": 2,
            "is_dead": "no",
            "is_new": "no",
            "model": "Nexus9000 9000v Chassis",
            "name": "dist-sw02.devnet.lab",
            "serial_number": "93LCGCRUJA5",
        },
        {
            "icon": "router",
            "id": 3,
            "is_dead": "no",
            "is_new": "no",
            "model": "n/a",
            "name": "core-rtr02.devnet.lab",
            "serial_number": "n/a",
        },
        {
            "icon": "router",
            "id": 4,
            "is_dead": "no",
            "is_new": "yes",
            "model": "CSR1000V",
            "name": "dist-rtr01.devnet.lab",
            "serial_number": "9S78ZRF2V2B",
        },
        {
            "icon": "router",
            "id": 5,
            "is_dead": "no",
            "is_new": "no",
            "model": "n/a",
            "name": "core-rtr01.devnet.lab",
            "serial_number": "n/a",
        },
        {
            "icon": "unknown",
            "id": 6,
            "is_dead": "no",
            "is_new": "no",
            "model": "CSR1000V",
            "name": "internet-rtr01.virl.info",
            "serial_number": "9LGWPM8GTV6",
        },
        {
            "icon": "dead_node",
            "id": 7,
            "is_dead": "yes",
            "is_new": "no",
            "model": "IOSv",
            "name": "edge-sw01.devnet.lab",
            "serial_number": "927A4RELIGI",
        }
    ]
};

A console output on the last run:

$ python3.7 generate_topology.py 
Open main.html in a project root with your browser to view the topology

Topology changes have been discovered:

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
New network devices:
vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Hostname: dist-rtr01.devnet.lab

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Deleted devices:
vvvvvvvvvvvvvvvvvvvvvvvvvvvvv
Hostname: edge-sw01.devnet.lab

^^^^^^^^^^^^^^^^^^^^^
New interconnections:
vvvvvvvvvvvvvvvvvvvvv
From dist-rtr01.devnet.lab(Gi3) To core-rtr02.devnet.lab(Gi0/0/0/2)
From dist-rtr01.devnet.lab(Gi4) To dist-sw01.devnet.lab(Eth1/3)
From dist-rtr01.devnet.lab(Gi6) To dist-rtr02.devnet.lab(Gi6)
From dist-rtr01.devnet.lab(Gi5) To dist-sw02.devnet.lab(Eth1/3)
From dist-rtr01.devnet.lab(Gi2) To core-rtr01.devnet.lab(Gi0/0/0/2)

^^^^^^^^^^^^^^^^^^^^^^^^^
Deleted interconnections:
vvvvvvvvvvvvvvvvvvvvvvvvv
From edge-sw01.devnet.lab(Gi0/2) To core-rtr01.devnet.lab(Gi0/0/0/1)
From edge-sw01.devnet.lab(Gi0/3) To core-rtr02.devnet.lab(Gi0/0/0/1)

Open diff_page.html to view the changes.
Optionally, open main.html and click the 'Display diff' button

Everything looks correct. The output matches the changes.
To visualize the diff topology properly, we will make some adjustments in next_app.js below.

Enhancing NeXt UI application

Most of the improvements below were made based on the examples in the Next UI documentation and tutorials.

Adding interface labels

In order to add the interface labels, let’s extend the standard nx.graphic.Topology.Link class:

    nx.define('CustomLinkClass', nx.graphic.Topology.Link, {
        properties: {
            sourcelabel: null,
            targetlabel: null
        },
        view: function(view) {
            view.content.push({
                name: 'source',
                type: 'nx.graphic.Text',
                props: {
                    'class': 'sourcelabel',
                    'alignment-baseline': 'text-after-edge',
                    'text-anchor': 'start'
                }
            }, {
                name: 'target',
                type: 'nx.graphic.Text',
                props: {
                    'class': 'targetlabel',
                    'alignment-baseline': 'text-after-edge',
                    'text-anchor': 'end'
                }
            });
            return view;
        },
        methods: {
            update: function() {
                this.inherited();
                var el, point;
                var line = this.line();
                var angle = line.angle();
                var stageScale = this.stageScale();
                line = line.pad(18 * stageScale, 18 * stageScale);
                if (this.sourcelabel()) {
                    el = this.view('source');
                    point = line.start;
                    el.set('x', point.x);
                    el.set('y', point.y);
                    el.set('text', this.sourcelabel());
                    el.set('transform', 'rotate(' + angle + ' ' + point.x + ',' + point.y + ')');
                    el.setStyle('font-size', 12 * stageScale);
                }
                if (this.targetlabel()) {
                    el = this.view('target');
                    point = line.end;
                    el.set('x', point.x);
                    el.set('y', point.y);
                    el.set('text', this.targetlabel());
                    el.set('transform', 'rotate(' + angle + ' ' + point.x + ',' + point.y + ')');
                    el.setStyle('font-size', 12 * stageScale);
                }
            }
        }
    });

A customized link class can now be listed in the topo topology object properties.
Let’s also highlight the links on the diff topology. New links will be green, deleted links will be red and dashed.

linkConfig: {
    linkType: 'curve',
    sourcelabel: 'model.srcIfName',
    targetlabel: 'model.tgtIfName',
    style: function(model) {
        if (model._data.is_dead === 'yes') {
            // Deleted links contain 'is_dead' attribute set to 'yes'.
            // Make them dashed.
            return { 'stroke-dasharray': '5' }
        }
    },
    color: function(model) {
        if (model._data.is_dead === 'yes') {
            // Deleted links contain 'is_dead' attribute set to 'yes'
            // Make them red.
            return '#E40039'
        }
        if (model._data.is_new === 'yes') {
            // New links contain 'is_new' attribute set to 'yes'
            // Make them green.
            return '#148D09'
        }
    },
},
// Use extended link class version with interface labels enabled
linkInstanceClass: 'CustomLinkClass' 

Adding custom node icons

Next UI already contains a broad set of default icons for network devices.
However, you are free to add custom icons based on your requirements. For deleted nodes, we need something special.
To add a new icon, you should put an image into a directory accessible by Next UI and initialize it in topo object.

// the image is saved to ./img/dead_node.png
topo.registerIcon("dead_node", "img/dead_node.png", 49, 49);

Getting diff visualization done

We have actually done everything we need. Let’s open diff_page.html and see how the changes we have done before look like:

A topology view speaks for itself, doesn’t it?

Updating a node tooltip

By default, a node tooltip contains excessive information like internal node id and coordinates.
In NeXt UI, it is possible to customize it for better readability and usability.
Let’s include the following information for now:

• Device hostname.
  Let’s also make hostname a customizable link to an arbitrary resource. It might be a device page in Netbox or a monitoring system.
  The link template will be stored in the dcimDeviceLink variable.
  It can be added during the topology generation process. In the case of missing value, the hostname will just be just a simple text.
• Device IP-Address, serial number, and model.

In order to implement this, let’s extend a standard nx.ui.Component class and build a simle HTML form inside of it:

    nx.define('CustomNodeTooltip', nx.ui.Component, {
        properties: {
            node: {},
            topology: {}
        },
        view: {
            content: [{
                tag: 'div',
                content: [{
                    tag: 'h5',
                    content: [{
                        tag: 'a',
                        content: '{#node.model.name}',
                        props: {"href": "{#node.model.dcimDeviceLink}"}
                    }],
                    props: {
                        "style": "border-bottom: dotted 1px; font-size:90%; word-wrap:normal; color:#003688"
                    }
                }, {
                    tag: 'p',
                    content: [
                        {
                        tag: 'label',
                        content: 'IP: ',
                    }, {
                        tag: 'label',
                        content: '{#node.model.primaryIP}',
                    }
                    ],
                    props: {
                        "style": "font-size:80%;"
                    }
                },{
                    tag: 'p',
                    content: [
                        {
                        tag: 'label',
                        content: 'Model: ',
                    }, {
                        tag: 'label',
                        content: '{#node.model.model}',
                    }
                    ],
                    props: {
                        "style": "font-size:80%;"
                    }
                }, {
                    tag: 'p',
                    content: [{
                        tag: 'label',
                        content: 'S/N: ',
                    }, {
                        tag: 'label',
                        content: '{#node.model.serial_number}',
                    }],
                    props: {
                        "style": "font-size:80%; padding:0"
                    }
                },
            ],
            props: {
                "style": "width: 150px;"
            }
        }]
        }
    });

    nx.define('Tooltip.Node', nx.ui.Component, {
        view: function(view){
            view.content.push({
            });
            return view;
        },
        methods: {
            attach: function(args) {
                this.inherited(args);
                this.model();
            }
        }
    });

Now the custom class version can be listed in topo topology object properties:

tooltipManagerConfig: {
    nodeTooltipContentClass: 'CustomNodeTooltip'
},

This is what we see on mouse click on the node after that:

Customizing node layout

As discussed before, the default Next UI data processor is ‘force’. It relies on the best effort algorithm spreading the Nodes so they would be as distant from each other as possible.

This logic produces a proper layout even for complex hierarchical network topologies but the topology layers may not be oriented as desired. For sure, you can drag them manually afterward. However, this is not our way.

Fortunately, there are some built-in tools to work with layers in Next UI.
Let’s use a new numeric layerSortPreference attribute inside nodes in our Next UI application.
The logic defining this value can be implemented outside the visualization application on the topology object generation stage in this case. Next UI would just order the layers in a way we tell it to. This is a more scalable approach.

Let’s add some functions to be able to switch between layouts:

    var currentLayout = 'auto'
    horizontal = function() {
        if (currentLayout === 'horizontal') {
            return;
        };
        currentLayout = 'horizontal';
        var layout = topo.getLayout('hierarchicalLayout');
        layout.direction('horizontal');
        layout.levelBy(function(node, model) {
            return model.get('layerSortPreference');
        });
        topo.activateLayout('hierarchicalLayout');
    };
    vertical = function() {
        if (currentLayout === 'vertical') {
            return;
        };
        currentLayout = 'vertical';
        var layout = topo.getLayout('hierarchicalLayout');
        layout.direction('vertical');
        layout.levelBy(function(node, model) {
          return model.get('layerSortPreference');
        });
        topo.activateLayout('hierarchicalLayout');
    };

Map these function to button elements on main.html and diff_page.html:

<button onclick='horizontal()'>Horizontal layout</button>
<button onclick="vertical()">Vertical layout</button>

Let’s improve the generate_topology.py script and add some additional attributes to Nornir hosts file to implement an automatic node hierarchy calculation.
The script will define an ordered list of human-friendly layer names and make a conversion into numeric values:

# Topology layers would be sorted
# in the same descending order
# as in the tuple below
NX_LAYER_SORT_ORDER = (
    'undefined',
    'outside',
    'edge-switch',
    'edge-router',
    'core-router',
    'core-switch',
    'distribution-router',
    'distribution-switch',
    'leaf',
    'spine',
    'access-switch'
)

def get_node_layer_sort_preference(device_role):
    """Layer priority selection function
    Layer sort preference is designed as a numeric value.
    This function identifies it by NX_LAYER_SORT_ORDER
    object position by default. With numeric values,
    the logic may be improved without changes on the NeXt app side.
    0(null) causes an undefined layer position in the NeXt UI.
    Valid indexes start with 1.
    """
    for i, role in enumerate(NX_LAYER_SORT_ORDER, start=1):
        if device_role == role:
            return i
    return 1

A numeric layer sort order for the layer will be defined by its relative position in NX_LAYER_SORT_ORDER.
Important: NeXt UI interprets 0(null) as undefined. Valid layer indexes start with 1.

The device layer will be based on its role attribute inside the Nornir Hosts Inventory file.
data field allows us to specify a list of attributes with arbitrary names:

dist-rtr01:
    hostname: 10.10.20.175
    platform: ios
    groups:
        - devnet-cml-lab
    data:
        role: distribution-router

Any attribute inside host data can then be called in Python as a dict key as follows:

nr.inventory.hosts[device].get('some_attribute_name')

To reflect the changes, let’s update our Python code. A new nr_role node attribute will be appended along with others into global_facts inside the normalize_result function:

# Full function is omitted here for brevity
global_facts[device_fqdn]['nr_role'] = nr.inventory.hosts[device].get('role', 'undefined')

Then we should read this attribute during node object generation inside the generate_topology_json function:

# Full function is omitted here for brevity
device_role = facts[host].get('nr_role', 'undefined')
topology_dict['nodes'].append({
    'id': host_id,
    'name': host,
    'primaryIP': device_ip,
    'model': device_model,
    'serial_number': device_serial,
    'layerSortPreference': get_node_layer_sort_preference(
        device_role
    ),
    'icon': get_icon_type(
        lldp_capabilities_dict.get(host, ''),
        device_model
    )
})

And now we can control the chaos align the layers horizontally and vertically on button click. This is how it looks like:

Resulting project structure

The final project structure looks as follows:

$ tree . -L 2
.
├── LICENSE
├── README.md
├── diff_page.html
├── diff_topology.js
├── generate_topology.py
├── img
│   └── dead_node.png
├── inventory
│   ├── groups.yml
│   └── hosts_devnet_sb_cml.yml
├── main.html
├── next_app.js
├── next_sources
│   ├── css
│   ├── doc
│   ├── fonts
│   └── js
├── nornir_config.yml
├── requirements.txt
├── samples
│   ├── sample_diff.png
│   ├── sample_layout_horizontal.png
│   ├── sample_link_details.png
│   ├── sample_node_details.png
│   └── sample_topology.png
├── styles_main_page.css
└── topology.js

Conclusion

First of all, thank you for reading. I hope you enjoyed it.

In this article, I tried to reproduce and document the solution creation stages and my considerations behind them.

A full source code is available on my GitHub page.

The solution took first place on the Marathon based on attendees’ and organizers’ votes. And equally important, it has a potential for reuse and scalability(spoiler: I developed a Netbox plugin reusing the core code from this project).

How do you like the solution? What could be improved? How would you solve this task? Please feel free to share your own experience and thoughts.