Updating DNA-Center experiences

So I am in the lucky position that in the past few weeks I’ve been working with the DNA-Center solution in different ways in a primarily automation focused project. One of the new philosophies of the Cisco DNA-C BU is to deliver updates more frequently and in a micro-update based strategy. E.g. if a package has a bug, only that package can be updated, on the fly. So DNA-Center needs to have a connection to the Internet. And it’s time to share my experiences.

DNA-Center overview

DNA-Center is essentially based on the container-based platform of APIC-EM. In this platform, grapevine has been replaced with a different container-based infrastructure platform named maglev. The figure below gives a schematic approach on how the DNA-C appliance is essentially structured. Maglev introduces the possiblilty to cluster multiple DNA-C appliances for a HA solution. For HA you do need three DNA-C appliances with the primary reason that the integrity of the data (NDP) and config can be validated as there are still two copies, a common practice also seen in HyperFlex or vSAN deployments.

So in essence, the DNAC solution consists of four different layers , being

Ubuntu based Appliance This is the core OS upon which the container platform (with docker and kubernetes) is running. In general, you shouldn’t touch the OS, except for a minor config thing which I’ll cover later

System Package Although it is called a package, it is the primary underlay for all other DNAC packages and applications, such as automation, path-trace, network visibility, etc, are running and talking to. You could see this as the hypervisor.

Application packages This is the core of the DNAC appliance, it contains all the applications, services, packages and intelligence that you can use. Packages can contain different microservices that provide some part of the intelligence, for example a syslog collector in assurance that receives syslog data and parses it. Or the SDA Lan Automation is part of a package and contains different micro-apps that all can talk with each other

User interface I’ve drawn the user interface as a separate layer, as I suspect that the ui modules in essence call api’s of the application packages so that a consistent UI can be created and other developers can reuse the API’s for their own applications.

Now that there’s a generic schematic overview of DNAC appliance, it’s time to perform some updates, right?

Updating DNA-Center

Updating DNAC system and applications is really easy. Just go to the DNAC web interface, click on the cog-wheel, select System Settings and then “App management”.

And there you’ll see which applications you have running in DNAC and which applications have updates.  You might see that DNAC has already checked for you if there are updates available, just like your computer does. So DNAC checks periodically for updates. That could come in handy..

Click on the checkbox left of the application and click download, after downloading you can install the new package. But WAIT!! According to the release notes, some packages need to be downloaded en installed before others, so ALWAYS check the release notes.

Troubleshooting tips

Of course, sometimes things don’t go as well as expected and you need to do some troubleshooting. The following tips might help you further along the road.

URL's used

If you have DNAC behind a strict firewall, like a next-gen firewall, make sure you allow DNAC to connect the following URL’s to succesfully download software packages and provide feedback

So make sure your next-gen firewall allows these url’s and domains to pass for DNAC.

Stuck in "downloading"

And what would happen if your DNAC server is behind a proxy, and you didn’t configure one..  Well, your download of the package gets stuck in downloading. It is a known issue.

Fortunately, Console access comes into the rescue for that.

Login to the DNAC console via SSH to the DNAC IP address on port 2222. Login with maglev and the admin password. Now that you are in maglev, execute the following command
$ maglev catalog package display
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]

NAME                           VERSION         STATE      INFO
--------------------------------------------------------------------
application-policy             2.1.6.170003    PARTIAL    Package needs to be pulled/downloaded
assurance                      1.0.5.670       PARTIAL    Package needs to be pulled/downloaded
automation-core                2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
base-provision-core            2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
command-runner                 2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
device-onboarding              2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
image-management               2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
ncp-system                     2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
ndp-base-analytics             1.0.7.873       PARTIAL    Package needs to be pulled/downloaded
ndp-platform                   1.0.7.818       PARTIAL    Package needs to be pulled/downloaded
ndp-ui                         1.0.7.949       READY
network-visibility             2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
path-trace                     2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
sd-access                      2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
sensor-assurance               1.0.5.347       PARTIAL    Package needs to be pulled/downloaded
sensor-automation              2.1.6.60014     PARTIAL    Package needs to be pulled/downloaded
system                         1.0.4.779       PARTIAL    PluginType artifact is not available yet

This command provides the following output. It says that different packages are not READY but in PARTIAL, which can mean that they’re stuck downloading. You can delete that stuck download process for each package with the following command

maglev catalog package delete <packagename>:<version>


[Mon Mar 26 19:32:42 UTC] maglev@10.137.187.70 (maglev-master-1) ~
$ maglev catalog package delete application-policy:2.1.6.170003
ok
[Mon Mar 26 19:33:00 UTC] maglev@10.137.187.70 (maglev-master-1) ~
$ maglev catalog package delete assurance:1.0.5.670
ok

This will delete the package download for you and if you do this for each package, you’re going to cleanup before you can do a system package install. The output would then become

$ maglev catalog package display
maglev-1 [main - https://kong-frontend.maglev-system.svc.cluster.local:443]
NAME                           VERSION         STATE      INFO      
--------------------------------------------------------------------
application-policy             2.1.4.170015    READY                
assurance                      1.0.5.621       READY                
automation-core                2.1.4.60025     READY                
base-provision-core            2.1.4.60025     READY                
command-runner                 2.1.4.60025     READY                
device-onboarding              2.1.4.60025     READY                
image-management               2.1.4.60025     READY                
ncp-system                     2.1.4.60025     READY                
ndp-base-analytics             1.0.7.871       READY                
ndp-platform                   1.0.7.803       READY                
ndp-ui                         1.0.7.949       READY                
network-visibility             2.1.4.60025     READY                
path-trace                     2.1.4.60025     READY                
sd-access                      2.1.4.60025     READY                
sensor-automation              2.1.4.60025     READY                
system                         1.0.4.779       PARTIAL    PluginType artifact is not available yet

Proxy configuration

Now, if you run into the situation that your DNAC server is behind a proxy, you can configure a proxy server within maglev. I didn’t find an option to specify a username and password, so if corporate IT requires authentication, you need to talk with them and ask for exceptions of the URLs that I provided earlier.

  1. Again, login to the DNAC console via SSH on port 2222
  2. Issue the following command
    magctl service setenv catalogserver MAGLEV_HTTPS_PROXY http://<yourproxy>:<yourproxyport>
  3. I’ve heard different responses on whether to reboot or not after this change, but better be safe than sorry and issue a reboot command
    sudo reboot

Logging locations

!!Now a very very important caution here, you are logging into the console of your DNAC appliance and executing superuser commands. This can seriously break an appliance. And I mean seriously break it. So use these at your own risk and make sure you know what you’re doing. Information is provided as-is and no claim can be made. !!

As DNAC runs on Ubuntu, you can use standard linux tools like grep and less to search for problems. I found the following commands quite usefull in troubleshooting

sudo tail -f /var/log/syslogThis command puts a tail on the syslog, if you watch it in a separate session and something like a download goes wrong, you can see the error message for that URL in this log. Be aware that it takes some time for DNAC to settle down and have all services that are dependent of each other started. Until that time syslog will be overflowing with long log messages
sudo egrep <whatyoulookfor> /var/log/containers/*logThis command searches in the /var/log/containers/ directory for whatever you want to look for. Check for the specific package name, or a URL, or something else to see if you can find the problem. It won’t be easy, but can be helpfull

DNAC system update & proxy

Did you remember the four layers I described earlier in this post? Well, a DNAC system update is actually also a package, but you cannot delete it with maglev command. Also, as this is a different layer in the schematic approach, it does NOT use the maglev proxy settings. It is actually a docker process that does this communication, and that requires a different configuration file. At the install I was working on, the proxy configuration for docker was missing. It could be that this file is only created when you specify a proxy during first time setup. Or it was something else. The key thing is that the proxy config is stored in two places, within maglev and within a docker config file.

So again a caution, you are changing a system level file and you MUST know what you’re doing. Make sure you keep track of your changes and know how to connect (CIMC) and recover if things go bad..

  1. Login to the ssh console of DNAC on port 2222 user maglev
  2. Issue the command sudo su - and provide the maglev admin password
  3. Go to the directory /etc/systemd/docker.service.d/
  4. Create the following file “http-proxy.conf“
  5. And make sure the following content is in the file
    [Service]
Environment=NO_PROXY=.svc.cluster.local,localhost,127.0.0.1,localhost,127.0.0.1
Environment=no_proxy=.svc.cluster.local,localhost,127.0.0.1,localhost,127.0.0.1
Environment=HTTP_PROXY=http://<yourproxyip>:8080
Environment=HTTPS_PROXY=http://<yourproxyip>:8080
Environment=http_proxy=http://<yourproxyip>:8080
Environment=https_proxy=http://<yourproxyip>:8080
  6. And reboot the DNAC appliance via the command reboot . It is possible to just restart systemd and docker, but I had the feeling that DNAC didn’t like that too much.

Automatic downloads

Oh and last but not least, remember that DNAC checks automatically for new versions and will try to download those packages for you. So they can get “stuck” again, also after a reboot. So be sure to check if a package is stuck in a download frequently and delete them if needed.

Summary

In summary, updating DNAC should be a simple and easy process. But as with any modern enterprise, security measurements need to be taken into account and you need to configure that proxy correctly and check.

Also, although you can now more rapidly update your DNAC software, you should follow your own procedures with regards to changes and executing updates. To me, that is reading release notes, reading upgrade guides, backing up DNAC, testing the upgrade in a test environment, allocate a service window and do the update in a planned way. And, according to more and more best practices, wait with major updates until it’s proven stable enough for production…

Share this

8 Responses

  1. Added some more URL’s that are being used and modified the http_proxy config file to include registry.maglev-system.svc.cluster.local

    Reply

  2. Edited the http_proxy.conf file to have both upper and lowercase values, as apparently that is needed.

    Reply

  3. Using DNAC 1.1.6 in the lab… Is there a way to wipe all assurance data without reinstalling the appliance?

    Reply

    1. I’m sorry, but I do not know (by heart) how to recover/reset the maglev password without admin access. Only thing that you could try is raise a tac case or reboot with the install DVD and check if that install has options. But raising a TAC case would be the best step.

      Reply

  5. Perfect Guide!!!
    I had this issue since my appliance is going to internet via internal proxy, I saw the traffic going from the server without being marked with proxy so started to looking up how to configure the ubuntu and came to here.

    Just needed to do this and it started working

    Again, login to the DNAC console via SSH on port 2222
    Issue the following command
    magctl service setenv catalogserver MAGLEV_HTTPS_PROXY http://:
    I’ve heard different responses on whether to reboot or not after this change, but better be safe than sorry and issue a reboot command
    sudo reboot

    Thanks for this post.
    Best Regards
    Pedro Reforço

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Categories

Tags