Creating custom indexes to data with Swift Sets

Creating custom indexes to data with Swift Sets

Introduction & Background

I have been working on a flow visualization tool for a while now. It is part of a project where a datacenter has evolved over a long time and has grown into a complex myriad of connections, applications and “solutions” to make things work. Goal of the project is to visualize certain environments and how they communicate with other services. Surely we’ve looked at Tetration as a Service and other visualization tools. The business case wasn’t viable enough for commercial solutions, so decided to go with Elastiflow as a basis.

Personal note: It’s been way overdue for a blog post. Even though Tom (@networkingnerd) mentioned that many bloggers have found it hard to find the time and drive to write blog posts during COVID19, WFH and everything that comes with it, I have been frustrated over myself I haven’t found the drive to write for way too long. 

It works well for forensics but for the original purpose (visualization, research & analysis) it was lacking a few bits. It was thus decided that a custom-made application that takes the samples from ElastiFlow and work from there was the most viable road. And so I started writing a Mac Application, using Core Data and GraphViz to fetch samples, process them (dedup), and were able to visualize flows. 

Fast forward to just before summer break; the flows in the database have grown to millions of records, which led to huge delays (300s) for the visualization of the flows on a larger dataset of server IPs. The reason for that is twofold, database queries are always more expensive than in-memory searches and the way I organized the data (in-memory) was inefficient. I’ve solved the latter by optimizing the code structure and algorithm and reduced the visualization time from 300s to 25s per visualization iteration. But the first cause of delays (roundtrips to the database) isn’t solved with that optimization and it bit me in ingesting new samples (as that also checks the database for duplicate samples). The solution to that is to use an in-memory database (this is quite common; Cisco ISE has an in-memory database (I think Cisco Prime Infrastructure does too), and big data platforms like SAP/HANA). After a quick research for in-memory Swift based databases, I decided to build my own Swift based memory manager for just the flows.

Sets in Swift, Hashable, and my idea

Swift has a number of ways to create collections of data elements (classes or structures). The most common approach is an array, which is an ordered collection. It is very easy to use an array, as the example below shows.

Swift Arrays are not fixed in size, and it’s easy to add or remove elements to that array. Ideal for data manipulation and search. But.. the search is linear and takes, on average, half of the iterations to find the entry, but worst case it will take n-iterations (where n is the size of the array) to find the record you’re looking for. 

So to optimize the search speed, we need to reduce the number of iterations needed to find a specific flow. This is where the Set class comes into play. The Swift Set class is used to store unique data elements. The uniqueness is determined by the hash of a unique identifier (the key). Hashing is a mathematical method that creates a unique consistent value for a piece of data. It’s a common principle in not only encryption (integrity validation), but also in database search optimizations.

Instead of comparing all attributes in a record (let’s say a flow), the Set class checks the unique identifier if the element is inside the collection:

Using the Set swift class allowed me to do that optimization from 300seconds to 25seconds for visualization. So why not use the Set Class to store the flows. So I can definitely use that in the in-memory database of flows. In my application I use four categories of queries / searches on those flows:

  1. Full flows: Search for all attributes of a flow (protocol, source IP and Port, destination IP and Port) 
  2. Similar flows: Search for all flows for the same protocol, same source IP and to a specific destination IP and Port
  3. IP conversations: Search for flows only based on source and destination IP
  4. Server applications: Search for all flows matching a specifc destination IP and Port

Effectively these four types of searches are similar to four indices in a database, and I should be able to implement them inside Swift. And this is where the power of class inheritance and protocols come into play inside the Swift language. The Set class requires that the element inside the collection conforms to the Hashable and Equatable protocol (Hashable inherits from Equatable). It means that every element you want to store in a Set needs to minimally have the following functions:

func hash(into: inout Hasher)
static func  == (Self,Self) -> Bool

All common types in Swift (String, Int, Date) have a default implementation of these methods, so you can compare types ( == keyword is actually a function ) and use the Set class in your code. The default hash and == function implementations take all attributes and use them for comparison. 

But it is possible to override that behavior by specifically implementing the Hashable protocol and write your own implementation (and thus specify which properties are used for uniqueness)

The problem

So I wrote the following code to create the indices:

// Base class for storing network flows in memory
class FMMFlowEntry : Hashable, CustomStringConvertible {
    var proto : String = ""
    var srcIP : String = ""
    var srcPort : Int = 0
    var dstIP : String = ""
    var dstPort : Int = 0
    var firstSeen : Date?
    var lastSeen : Date?
    
    static func == (lhs: FMMFlowEntry, rhs: FMMFlowEntry) -> Bool {
        let response = dstIP == rhs.dstIP && proto == proto && dstPort == dstPort && srcIP == srcIP && srcPort == rhs.srcPort
        return response
    }
        
    public func hash(into hasher: inout Hasher) {
        hasher.combine(proto)
        hasher.combine(srcIP)
        hasher.combine(srcPort )
        hasher.combine(dstIP)
        hasher.combine(dstPort)
    }
    
    var description : String {
        let response : String = "\(proto) \(srcIP):\(srcPort) -> \(dstIP):\(dstPort)"
        return response
    }
    
    init(proto: String, srcIP: String, srcPort: Int, dstIP : String, dstPort: Int) {
        self.proto = proto
        self.srcIP = srcIP
        self.srcPort = srcPort
        self.dstIP = dstIP
        self.dstPort = dstPort
    }
    
    
    init() {
        
    }
}

// Class that represents a flow record based on source IP address and destination IP address
class FMMIPFlow  : FMMFlowEntry  {
    var refCount : Int = 1
    init(from: FMMFlowEntry) {
        super.init(proto : from.proto, srcIP : from.srcIP, srcPort : from.srcPort, dstIP : from.dstIP, dstPort : from.dstPort)

    }

    override public func hash(into hasher: inout Hasher) {
        hasher.combine(srcIP)
        hasher.combine(dstIP)
    }
    
    static func == (lhs: FMMIPFlow, rhs: FMMIPFlow) -> Bool {
        let response = lhs.dstIP == rhs.dstIP && lhs.srcIP == rhs.srcIP
        return response
    }
    
    override var description : String {
        let response : String = "\(srcIP) -> \(dstIP) [\(refCount)]"
        return response
    }
}

In the above code there is the base class (FMMFlowEntry) that has all necessary attributes. The hash function specifies which properties of the flow are used to determine if it is unique). The == function also uses those attributes to check for equality.

The class FMMIPFlow inherits every method and property from FMMFlowEntry. It then overrides the hash function that only the srcIP and dstIP attributes are used for hashing and comparison. 

This should mean that if I’d compile and run the code below, the variable uniqueIPFlows would only have one entry (as the srcIP and dstIP are the same for the two added flows). When I run it in Playground, this is the result.

That means that both flow1 and flow4 are added to the Set. I was not expecting that, I created a custom hash function and so it should not add it, right? For debugging I ran it again, printing the hash value too:

So even while they have the same hash value (which changes every time I run the code), both flows are still added to the index. That is not the purpose, but why?

Cause of the problem

The answer to that took me quite a while and would like to share it here. The problem lies in the comparison function. It is defined as a static function (conform to the protocol). The static keyword also means final and cannot be overridden. So even if I have a new static == function in the subclass, it will never be called because static implies final (and the compiler will just ignore that other method). We can validate that by adding a print in our code:

static func == (lhs: FMMFlowEntry, rhs: FMMFlowEntry) -> Bool {
    print("FMMFlowEntry == func called")
    let response = lhs.dstIP == rhs.dstIP && lhs.proto == rhs.proto &&
         lhs.dstPort == rhs.dstPort && lhs.srcIP == rhs.srcIP &&
         lhs.srcPort == rhs.srcPort
    return response
}

And rerun the code

So the hash is used for searching, but to determine if an element needs to be added to the Set, there is an equality check. And because the two flows are different (on source port), both are added. 

The solution

Now that we know the cause, we can create a solution to work around that blockade. Instead of performing the comparison check inside the static function, I can create an instance function that is called on the lhs parameter and validates against the rhs, so that I am able to override that function in a subclass. The code in FMMFlowEntry is changed:

static func == (lhs: FMMFlowEntry, rhs: FMMFlowEntry) -> Bool {
    return lhs.equals(to: rhs)
}
    
func equals(to rhs: FMMFlowEntry) -> Bool {
    let response = dstIP == rhs.dstIP && proto == rhs.proto && 
        dstPort == rhs.dstPort && srcIP == rhs.srcIP && 
        srcPort == rhs.srcPort
    return response
}

and the code inside the FMMIPFlow Class is changed to:

override func equals(to rhs: FMMFlowEntry) -> Bool {
    guard let rhs = rhs as? FMMIPFlow else { return false }
    let response = self.dstIP == rhs.dstIP && self.srcIP == rhs.srcIP
    return response
}

Now when we try to run the following code, the static compares in the base class should call the overriden equals function of the instance (FMMIPFlow in this case) and will only check for the srcIP and dstIP property, which should lead to only one entry in the Set:

Succes! Now I can create indexes / search types very quickly by just subclassing the base record and override the equals function for that specific index. Which allows me to to code the indexes I need with only a few lines of code and still be able to work with the Set class and have speed in search queries. One step closer to my in-memory database that is capable of storing millions of flows.

 

The key lesson here is that the keyword static in Swift means that it is a final method and cannot be overridden in a subclass. The method around it is to call an Instance method in the base class and override this in the subclass.

The Swift Playground used in this post is available on github

Network Inventory unreachable after Prime 3.9 Upgrade

Network Inventory unreachable after Prime 3.9 Upgrade

Recently I upgraded a Cisco Prime Infrastructure deployment for a customer and after the normal wait of the database migrations, restart of the ACS appliance I ran into the issue that quite a few network devices were set to unreachable. 

 After some troubleshooting I found out that it were only the devices that were configured with a hostname and not IP-address in the inventory.  That brought me to troubleshooting on the CLI, which gave me the following output:

prime-server-1/admin# ping switch1.domain.com
% Error: Error invoking ping for the provided host

So something is wrong, and that was odd as the DNS Servers were up, runing and reachable. I managed to analyse this further and found out that Cisco Prime 3.9 has implemented a new feature, DNSSEC and has enabled it by default. It can result in DNS errors in ACS (upon which Prime runs), resulting the above output.

Cisco registered this as caveat CSCvx06532 

Workaround

The workaround is relatively easy, just disable dnssec. How? Just perform the following steps

  1. Login to the CLI of your Prime Deployment
  2. Hit config mode
  3. Disable dnssec
  4. Save config

 

Below is the output of the change I performed. 

prime-server-1/admin# conf t
Enter configuration commands, one per line.  End with CNTL/Z.
prime-server-1/admin(config)# no ip dnssec 
prime-server-1/admin(config)# end
prime-server-1/admin# write memory 
prime-server-1/admin# ping switch1.domain.com
PING switch1.domain.com (10.1.1.22) 56(84) bytes of data.
64 bytes from 10.1.1.22: icmp_seq=1 ttl=253 time=0.536 ms
64 bytes from 10.1.1.22: icmp_seq=2 ttl=253 time=0.751 ms
64 bytes from 10.1.1.22: icmp_seq=3 ttl=253 time=0.574 ms
64 bytes from 10.1.1.22: icmp_seq=4 ttl=253 time=0.579 ms

--- switch1.domain.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3001ms
rtt min/avg/max/mdev = 0.536/0.610/0.751/0.083 ms
prime-server-1/admin# 

And voila, DNS is working and inventory is recovering. 

I hope this quick tip helps you when you upgrade Prime and run into this issue.

Whiteboarding From Home difficult?

Whiteboarding From Home difficult?

So many of us are working from home now; COVID19 has changed the way we work in a very rapid way and Work From Home has become a common fact for many of us. Personally I have been working from home almost since the start of my career , way back. Reason was simple, we had a home office and were a family business. One thing that has always made me want to go to a customer or talk with a team is to have the ability to whiteboard a design or solution to a problem.

And that is something which is in my opinion very difficult with Webex and Zoom. Sure, there are corporate solutions around, like Desk pro, Webex Board, etc. And I will be honest, they look great. But they just don’t work in a smaller work from home environment/situation.

 

And until recently I just accepted the fact that Whiteboarding from Home as a feature wasn’t going to happen soon, and just had to accept it. Well, until I asked around and my fellow Cisco Champion Matt Ouellette shared his solution with me. The solution is based on the Apple ecosystem and what you need are:

 

  • MacBook Pro with latest MacOS
  • iPad that supports an Apple Pencil
  • Apple Pencil
  • Your favorite drawing app on you iPad (I am using Paper, but you could use any tool, like MS OneNote)

And here’s how to do it:

  1. First connect (physically) your iPad to your Mac and trust the computer
  2. Now that the Mac is trusted, your iPad screen has become a media source within Quicktime, so launch Quicktime Player.
  3. Click File -> New Movie Recording and click on the small triangle just right next to the record button
  4. Select your iPad as source
  5. Now go to your favorite video conferincing tool and share your desktop or juist quicktime player
  6. Go back to the iPad, start your favorite drawing App and start drawing

Agreed, it is not a collaborative way of Whiteboarding, that someone in the meeting can co-whiteboard with me, but it allows me to draw designs and solutions based on what we are talking about like I am standing in front a virtual whiteboard . What I like is that you don’t even need to record it, you can just show it. Save the diagrams for after the meeting and you have great sketches to start writing documentation with.

I am really happy with this, because it allows me to create videos and explain while drawing in the courses that I am preparing too, so thanks Matt for the tip. I have been using this a few times already over the past days!

Cisco C9800-CL sits idle at GRUB Loading Stage2…

Cisco C9800-CL sits idle at GRUB Loading Stage2…

I have been using the Cisco Catalyst 9800-CL (Wireless Controller for cloud) for a while now. Recently, I accidentally powered off the wrong VMWare server, resulting in a wireless disruption. Priority 1 at home! And of course, just before I had a session with Shawn preparing for CiscoLive Barcelona…

After restarting the vSphere server, my C9800-CL wasn’t booting up, with a message: “GRUB Loading stage2… ” And it just sat there, for minutes..  Eventually, during the WebEx Call, I managed to fix it and got my controller back up and running. 

Steps to fix the issue

These are the steps that I used for fixing this issue.

First, power off the VM in vSphere. We need to change some settings in the BIOS.

Next, go and select “Edit Settings” of your VM and click “VM Options” at the top to view some advanced settings and click “Boot options” open. Change the Boot delay to “8000” milliseconds, so that you have enough time when you boot the VM.

Hit Save after you have changed the settings.

Just to make sure, open the settings of the VM again, and click open the first CD/DVD Drive. Check that there is an image named “_deviceImage-0.iso” and that it is connected at powerup.

When I used the vCenter convertor to move the VM off to a new server, I found that this iso wasn’t copied with the controller and it is needed.

Hit Save when you know the ISO image is there.

Follow the next steps to get the C9800-CL booting up again

  1. Open up the console of the VM in the browser (it saves you time)
  2. Power on the VM
  3. Once the Bios is shown, click in the console and hit “ESC
  4. The boot order menu is shown, like the image on the left
  5. Scroll down to highlight “CD-ROM Drive” 
  6. And hit “Enter
  7. Now the VM will boot normally and your controller will start as expected.

Summary

It seems that Grub (the bootloader on the first disk) is not configured correctly the C9800-CL, which leads to a VM / Appliance that is not booted because it cannot find any kernel to load. By selecting the CD image, the right bootloader is selected and the controller is started with the correct configuration. 

I do assume this is a caveat/bug in the Cloud version and will be fixed in a newer release. I do hope you can use this info to fix your C9800-CL deployment sooner. 

FDM Application fails after upgrade

FDM Application fails after upgrade

This is just a quick blog post for those that might have FDM issues after upgrading your FTD software.

I have recently updated my Firepower appliance from 6.5.0 to 6.5.0.2. One of the reasons to update is not only that 6.5.0 is a .0 release, but also that I noticed some failed rule-update deployments that set snort to block all traffic.

Unfortunately, after upgrading, FDM reported an error that it could not be launched with an application failure error. The suggested action was to remove the manager, add a new local manager and begin from scratch. This is the error: “The Firepower Device Manager application cannot be opened. Please try again”

While googling for a possible caveat of this behavior on 6.5.0.2, I came across a caveat in 6.2.3 that has the same behavior. 

That caveat has supported me in fixing my solution. What I did was executing the following commands:

 

> expert
**************************************************************
NOTICE - Shell access will be deprecated in future releases
         and will be replaced with a separate expert mode CLI.
**************************************************************
admin@na-grm-ftd01:~$ sudo su -
Password: 
root@my-ftd01:httpd# cd /ngfw/var/cisco/ngfwWebUi/
root@my-ftd01:ngfwWebUi# ls -a
.   .bootstrap-failed  clifile    deploy                      ha_pkg  lina_cli_sqlite_stores   pjb_output  sslCiphers  variables.ftd_onbox
..  bin                clisyncer  ftd_onbox_6.5.0.2_previous  libs    ngfw_onbox_bootstrap.sh  sru         tomcat      version

root@my-ftd01:ngfwWebUi# rm .bootstrap-failed 
root@my-ftd01:pmtool disablebyid tomcat
root@my-ftd01:pmtool enablebyid tomcat

Basically, you go into expert mode, find the tomcat directory used for FDM and then remove a status file and try to restart it.

With me, this worked and helped me get back access to FDM. Should you run into issues with FDM after an upgrade, this “hack” might help you.

Disclaimer: You are entering expert mode of FTD, it means you can DESTROY your FTD configuration and box. Be aware of what you are doing and make sure you have a backup. 

Hitting 100% CPU after restore on Firepower Management Center

I recently purchased a new microserver to reduce my power footprint at home. And I had to move the FMC (Firepower Management Center) from the OpenStack deployment that I previously ran at home. However, I ran into several bugs in OpenStack that were fixed in later versions, but I couldn’t upgrade because of another bug. Essentially I hit a catch-22 and had to deploy a new FMC on that new microserver and use a restore to move the data and policies. In that process I did hit a bug for which I’d like to share some info on. (more…)