Network Inventory unreachable after Prime 3.9 Upgrade

Network Inventory unreachable after Prime 3.9 Upgrade

Recently I upgraded a Cisco Prime Infrastructure deployment for a customer and after the normal wait of the database migrations, restart of the ACS appliance I ran into the issue that quite a few network devices were set to unreachable. 

 After some troubleshooting I found out that it were only the devices that were configured with a hostname and not IP-address in the inventory.  That brought me to troubleshooting on the CLI, which gave me the following output:

prime-server-1/admin# ping
% Error: Error invoking ping for the provided host

So something is wrong, and that was odd as the DNS Servers were up, runing and reachable. I managed to analyse this further and found out that Cisco Prime 3.9 has implemented a new feature, DNSSEC and has enabled it by default. It can result in DNS errors in ACS (upon which Prime runs), resulting the above output.

Cisco registered this as caveat CSCvx06532 


The workaround is relatively easy, just disable dnssec. How? Just perform the following steps

  1. Login to the CLI of your Prime Deployment
  2. Hit config mode
  3. Disable dnssec
  4. Save config


Below is the output of the change I performed. 

prime-server-1/admin# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
prime-server-1/admin(config)# no ip dnssec 
prime-server-1/admin(config)# end
prime-server-1/admin# write memory 
prime-server-1/admin# ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=253 time=0.536 ms
64 bytes from icmp_seq=2 ttl=253 time=0.751 ms
64 bytes from icmp_seq=3 ttl=253 time=0.574 ms
64 bytes from icmp_seq=4 ttl=253 time=0.579 ms

--- ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.536/0.610/0.751/0.083 ms

And voila, DNS is working and inventory is recovering. 

I hope this quick tip helps you when you upgrade Prime and run into this issue.

Enabling Intent-Based with Airwall

Enabling Intent-Based with Airwall

One of the things presented by Tempered Networks at XFD3 was the extensive programmability capabilities of Airwall. Everything you do in Conductor can also be preformed leveraging a REST API. I think this is a really good feature that can really demonstrate the capabilities and principles of Intent-Based Networking .  In this post, I’ll demonstrate what happens when you combine several technologies and trends together.

Background & use case description

Zero Trust is a paradigm/principle within Security that has gained quite some momentum. The core principle behind Zero Trust is that every single transaction should only be executed if it is properly authenticated and authorized. This can be an employee sending an email (workforce), a computer accessing the network (workspace), or two services exchanging data (workloads). 

You could say that access to a resource is denied unless there is an explicit permit based on authentication (who are you) and authorization (are you allowed to). 

Another trend in networking and security field is leveraging automation. In other words, instead of performing changes manually on-box, a script or program is defined that will perform the changes automatically for you on all devices, making sure that the change is performed consistently. There are of course requirements and disadvantages to automation, but that is for another post.

The success of the automation trend is only achievable by the programmability of the network devices; it is effectively a requirement for automation and any SDx solution. It means that you can perform changes in the infrastructure leveraging frameworks and API’s (often RESTful) using programming languages like Python and Swift. 

As Airwall is fully programmable, I think it has a very powerfull intent-based Zero-Trust use case…

In network infrastructures, it is very common to have a centralized authentication and authorization server, which validates if you are allowed access to a specific switch, router or firewall. This is often implemented via the RADIUS protocol or TACACS+ protocol. This means that, in general, you as a network engineer, have network management credentials and are able to login to devices 24×7 to perform monitoring tasks and execute configuration changes.

However, in an ideal world, based on Zero-Trust, from a security perspective, you’d only want to allow access to the (often critical) network infrastructure when there is an actual incident or change request that involves that devices. Some say that this is just-in-time rights management, providing you only rights when needed. It is basically an implementation / interpretation of Zero-trust. 

Unfortunately, a validation-check with an IT Service Management tool and CMDB is not a feature that is currently in Radius server implementations and would require some hacking or manual changes.

Proof of concept

But why not leverage Airwall and its programmability capabilities to achieve this. In other words, only allow access from a network management station to a managed device if there is a change or incident open in the IT Service Management tools. 

You could also state the use case as: “I have an intent to perform changes to a specific managed device (can also be a server) only when there is a specific change request or incident open. “

This is definitely something which is possible with Airwall, as I will show below. I do not have access to a personal ITSM tool with full extensive API’s, a full Radius server setup and all other requirements, but I do have XCode (to build iOS / MacOS apps), access to conductor and a server that I manage via Airwall.

For the POC, I will use the following setup / construct:

  • I have an overlay network (security policy) that allows my devices to connect to the server agent
  • The overlay network is disabled by default
  • Simple app instead of ITSM 

I will create a very simple iPad/iPhone/Mac application that asks for the hostname, credentials and overlay network name. It will have two options, enable communication and disable communication, simulating what an ITSM can do.

  • I will use ping and ssh to validate that communication is allowed or denied
  • I will use the Mac Application as simulator of the ITSM tool, where I can enable/disable the access with a tip of a switch. 


In conclusion, this POC shows that you can definitely implement Zero-Trust for the management of infrastructure, not only network but also server infrastructure. 

I also think that this POC/demo is not only showing how powerful Airwall itself is, but it is a true demonstration of the possibilities are of Intent-Based Networking if you leverage these programmability capabilities and give them to software engineers. It has only taken me 1,5 hours to build this macOS app that can also run on an iPhone or iPad to validate the POC (granted, I can already code and demo’d some apps in Swift 😉 ). 

Using Airwall: Secure Remote Access

Using Airwall: Secure Remote Access

In this blog post I would like to share the configuration I have been trying out with Airwall from Tempered Networks over the past months. Airwall is a new approach to security leveraging the HIP protocol defined in RFC7401. 

If you would like to read more about my experiences with Airwall, check out my first posts, Tampering Visibility with by Tempered and Configuring Airwall.


Network topology

The following topology describes how I have included the Airwall gateway in my home network. 

In this setup, I created a seperate VLAN for the untrusted VLAN that is effectively a subinterface on my FTD device. It allows traffic to the outside only. I have configured PAT to allow the necessary inbound Airwall communication ports. 

The trusted port of the Airwall gateway is directly connected to the client VLAN in my network. Why? I wanted to test out bonjour and L2 extension / forwarding too. 

Secure remote access

Probably one of the most visible use cases for Airwall is allowing secure access as if it was a VPN/SASE client allowing your users connecting securely to inside resources. And that is definitely possible with Airwall. You can leverage the gateway function to effectively connect secure (inside) hosts and networks via the secure ports.

For this use case, I want to securely allow access from my laptop to several management IP’s in my internal network, being:

  • Firepower Management Center (FMC)
  • C9800 Wireless Controller
  • My Raspberry PI with Domoticz to see real-time power consumption
  • My laser printer
  • My distribution switch (SSH)

As the gateway has its secure port to my clients VLAN, I just need to define these devices (and yes, IP scanning is possible) as local devices and add static routing for my network management. All this is configured for the specific Airwall via the Conductor (Airwalls -> Select the right Airwall). 

Add local devices to an Airwall gateway

Configure Overlay IP routing via the gateway.

I had some connectivity issues when I had my Mac conencted to the same VLAN and was not able to connect to my FMC. This is solvable by either setting a different overlay IP address on my Airwall Agents or configure PAT on the gateway. I’ve chosen the first option, and it has worked quite well. 

Now that the Airwall components have the right (local) connectivity, it is time to configure a security policy for secure access. Within Airwalls paradigm, these security policies are overlay networks. And they work very easy. Within the conductor the tab “Overlays” is where you’ll configure all overlay networks. I have created a hub-spoke overlay network with my own devices (which I grouped) as hub. Why would I do that? It means that all the other devices are spokes and I can automatically communicate with them, once I add them. 

Adding/Removing devices is like really easy. Just hit the plus and add any device you have defined in the Conductor, and that’s it. 

Once I add a device in the overlay, it can communicate with the hub. And when I am outside, or even when I’m at home, I am actually connecting to these devices via the Airwall secure network, as can be seen via the below traceroute.

In summary, I have been very satisfied with the way Airwall is working. This common use case is really easy to configure and manage. And as long as the underlay ports can communicate, it is fast and reliable. But you can do more with Airwall. One of the things from Security Field Day that got me triggered was the full programmability, which I will describe in another post. 

Configuring Airwall

Configuring Airwall

In a previous post I have already shared some information about the Airwall solution from Tempered Networks. In this blog, I will share my experiences in configuring and operating the Airwall solution. I have been running an Airwall setup to test for a number of months.

Airwall components

Before moving into the configuration aspect, let’s first describe the components of the Airwall solution.

Airwall Conductor

The conductor is the “heart” of the Airwall solution. It is the SDN controller (running on-prem or in a cloud) with which you manage all Airwall components and configure the different policies.

Airwall Agent

The airwall agent is a software component that is running on your user endpoint devices, such as your smartphone, tablet, laptop or desktop. It is effectively the client that connects to gateways and Airwall Server agents. 

Airwall Server agent

The Airwall Server agent is the software component that you can install on individual servers. You might say that it is comparable with the Airwall agent, except it runs faceless and is optimized for running on your servers. I have been running one on a virtual server I have hosted in the cloud. 

Airwall Gateway

The airwall gateway is a special component within the solution. You can use it to secure networks behind a gateway with agents. E.g., if you allow a connection from an agent to a gateway, you are able to securely connect from the agent through the gateway to multiple hosts or networks behind the gateway. The gateway comes in different flavors, a virtual machine, a cloud instance, a physical gateway for wired connections, and a phyisical gateway that connects to the unsecure networks via WiFi. The gateway is also capable of acting as a relay agent between agents that cannot communicate directly with each other, but only if you have specified and allow that communication in the conductor.  

Overlay networks

Overlay networks within Airwall are a bit different compared to the traditional overlay networks seen with VPN solutions, which is also what makes Airwall unique in its approach. An overlay network is basically a secured IP network that can be configured between agents, servers, and gateways. And the unique twist is that you can manage completely different overlay networks and configure a policy that a single agent can communicate via these different overlay networks.

My personal experience is that I initially misinterpreted overlay networks and thought of it as a single network with unique policies; it is easier to see each overlay network as a security policy that defines who can securely communicate with who at which moment in time. 

Underlay network

The underlay networks is your regular network, or the Internet. It is the insecure network via which you connect the different Airwall components. The conductor of course needs to be reachable via the underlay network.

Getting started

The team at Tempered networks was kind enough to provide me with a complete set of components to test with and try out use cases. My kit consisted of agents for my iOS devices, my laptops, 1 virtual gateway (Cloud), 2 physical gateways (one wired, one wireless), a server agent and a conductor.

Getting started with the conductor is pretty much straight forward, after running through the provisioning guide, you have the link to the conductor portal, which provides a good overview of your environment. The tab labels itself speak for itself. Now that the conductor is setup correctly, it is time to download and provision agents.


For the physical gateway, it was effectively the same setup. Hook a serial console to the device, configure the hostname and port for the conductor, activate the device and you can define your policies.

I used the provided download links to download an agent to my Mac and used the App-Store to download the iOS apps. Once they are downloaded, you provide the agent with a profile that contains the hostname and port to your conductor. It took me a while to figure out that I had to manually approve the license before further configuration was possible. So after you’ve configured your agent, go back to the conductor and hit the settings -> Licensing tab. Activate the just registered device and a license will be consumed. With that action, the airwall agent/gateway becomes active.

Remember, the gateway has different behavior compared to the agent with more functionality. Make sure that the communication ports you defined in the Conductor are available via the underlay networks. It took me a while to figure that out (I assumed too much routing in my mind), and once I had the PAT configured correctly, the gateway was working as expected and the agent on my mac could communicate with it. If you use NAT, you can configure the external IP address for the gateway via the conductor (auto-detect and auto-set would be great when you have dynamic outbound IP-addresses and relay on PAT). The screen shot below shows the setting for my gateway.

In my next post, I will give you a sample underlay network that I have been using very succesfull (including demos and firepower trainings) over the past months.

Configuring Airwall

Finding out which MIBs are supported on your Cisco device

Recently fellow champion Ioannis Theodoridis asked around how to find out what SNMP MIB’s are supported by a specific Cisco network device. And although model driven telemetry is gaining momentum, many monitoring systems still rely on SNMP. And finding those MIB’s can be a challenge. I found out a very easy method to quickly find which MIBs are supported. 

To get started, you need your CCO credentials by hand and an outbound FTP client. In this blog I’m using the terminal app on my mac.

1. Connect via FTP to and login with your own CCO credentials, set FTP to passive

					pr-mbp15-001:Desktop nefkensp$ ftp
Connected to
220-	Cisco Systems File Transfer Service.
--- SNIP ---
220 FTP Server (Apache/2.2) ready.
Name ( nefkensp
331 Password required for nefkensp
230 User nefkensp logged in
ftp> passive
Passive mode on.

2. now go to the directory “/pub/mibs/supportlists”


					ftp> cd /pub/mibs/supportlists
250 CWD command successful.

3. Do a directory listing (long) and enter the directory for your device. The device listing is long, I selected a new Cat9k switch in this example 

					ftp> ls
200 PORT: Command successful
150 Opening ASCII mode data connection for file list
drwxrwxr-x    2 swdsadm  cisco        4096 Jan  7  2002 2948g-13
drwxrwxr-x    2 swdsadm  cisco        4096 Jan  8  2002 4908g-13
drwxrwxr-x    2 swdsadm  cisco        4096 Dec 20  2000 90i
drwxrwxr-x    2 swdsadm  cisco        4096 Apr 19  2011 CTXSystem
drwxrwxr-x    2 swdsadm  cisco        4096 Jan  6  2003 ONS15530
drwxrwxr-x    3 swdsadm  cisco        4096 Dec 20  2000 accessProEC
drwxrwxr-x    3 swdsadm  cisco        4096 Dec 20  2000 accessProRC
drwxrwxr-x    2 swdsadm  cisco        4096 Aug 13  2008 ace
--- SNIP ---
drwxrwxr-x    3 swdsadm  cisco        4096 Oct 27  2006 wsc6009
drwxrwxr-x    2 swdsadm  cisco        4096 Oct 27  2006 wsc6503
drwxrwxr-x    2 swdsadm  cisco        4096 Oct 27  2006 wsc6504
drwxrwxr-x    3 swdsadm  cisco        4096 Oct 27  2006 wsc6506
drwxrwxr-x    3 swdsadm  cisco        4096 Oct 27  2006 wsc6509
drwxrwxr-x    2 swdsadm  cisco        4096 Oct 27  2006 wsc6513
drwxrwxr-x    2 swdsadm  cisco        4096 Dec 20  2000 wsc8510csr
drwxrwxr-x    2 swdsadm  cisco        4096 Apr 11  2001 wsc8510msr
drwxrwxr-x    2 swdsadm  cisco        4096 Dec 20  2000 wsc8540csr
drwxrwxr-x    2 swdsadm  cisco        4096 Apr 11  2001 wsc8540msr
drwxrwxr-x    2 swdsadm  cisco        4096 Jan  9  2008 xr12000
226 Transfer complete.
ftp> cd cat9300
250 CWD command successful.

4. Do a directory listing and fetch that HTML file and logout

					ftp> ls
227 Entering Passive Mode (72,163,7,54,102,29)
150 Opening ASCII mode data connection for file list
-rw-r--r--    1 swdsadm  cisco       15983 Jan 29  2018 CAT9300.html
226 Transfer complete.
ftp> get CAT9300.html
227 Entering Passive Mode (72,163,7,54,102,32)
150 Opening ASCII mode data connection for CAT9300.html
226 Transfer complete.
16159 bytes received in 0.169 seconds (93.1 kbytes/s)

5. Open the HTML file in your favorite browser and voila, you have the supported MIB directory for that network device type

Below are two samples of the supported MIBs for the Cat9300 family and the Nexus 3000 family. 


I have found this method very easy to find which MIBS are supported by any cisco device. It is an easy way to get that comprehensive list of supported MIBs. If you want to download them for your own NMS, you can go to the SNMP Object navigator and download the supported MIBS as next step.

Configuring Airwall

Cisco C9800-CL sits idle at GRUB Loading Stage2…

I have been using the Cisco Catalyst 9800-CL (Wireless Controller for cloud) for a while now. Recently, I accidentally powered off the wrong VMWare server, resulting in a wireless disruption. Priority 1 at home! And of course, just before I had a session with Shawn preparing for CiscoLive Barcelona…

After restarting the vSphere server, my C9800-CL wasn’t booting up, with a message: “GRUB Loading stage2… ” And it just sat there, for minutes..  Eventually, during the WebEx Call, I managed to fix it and got my controller back up and running. 

Steps to fix the issue

These are the steps that I used for fixing this issue.

First, power off the VM in vSphere. We need to change some settings in the BIOS.

Next, go and select “Edit Settings” of your VM and click “VM Options” at the top to view some advanced settings and click “Boot options” open. Change the Boot delay to “8000” milliseconds, so that you have enough time when you boot the VM.

Hit Save after you have changed the settings.

Just to make sure, open the settings of the VM again, and click open the first CD/DVD Drive. Check that there is an image named “_deviceImage-0.iso” and that it is connected at powerup.

When I used the vCenter convertor to move the VM off to a new server, I found that this iso wasn’t copied with the controller and it is needed.

Hit Save when you know the ISO image is there.

Follow the next steps to get the C9800-CL booting up again

  1. Open up the console of the VM in the browser (it saves you time)
  2. Power on the VM
  3. Once the Bios is shown, click in the console and hit “ESC
  4. The boot order menu is shown, like the image on the left
  5. Scroll down to highlight “CD-ROM Drive” 
  6. And hit “Enter
  7. Now the VM will boot normally and your controller will start as expected.


It seems that Grub (the bootloader on the first disk) is not configured correctly the C9800-CL, which leads to a VM / Appliance that is not booted because it cannot find any kernel to load. By selecting the CD image, the right bootloader is selected and the controller is started with the correct configuration. 

I do assume this is a caveat/bug in the Cloud version and will be fixed in a newer release. I do hope you can use this info to fix your C9800-CL deployment sooner.