What can I test using Netpicker?

Table of contents

Check if one line is present in a configuration
Check if lines are NOT present in your configuration
Check lines that can contain variable values
Check if lines are present in your configuration, while others aren’t
Validate the output of a command run on your device
Validate multiple lines
Conditional rules
Verifying access lists

This guide details all the ways in which you can use NetPicker’s policies in order to test your devices.

Check if one line is present in a configuration

This is the most basic use case. Suppose you have a Cisco IOS device and you want to check its version. The version is defined on a single line in the configuration, so we need a simple rule with nothing extras. Go to the policies section, click on your policy and create a Simple rule for the Cisco IOS platform:

For this use case you only need to fill in the textfield “Must include the text”; you don’t need to worry about the other options. This test will succeed if the line you provided exists anywhere within the configuration. On any line, it doesn’t matter if the matching line contains more text than just the matched string. Click on “Save” and the test is finished.

Check if lines are NOT present in your configuration

Suppose there was once a user account in your configuration that has been deleted, and you want to check whether it really has been deleted across all your Cisco devices. In this use case, we don’t want the following line to be present in our configuration:

username obsoleteuser password 7 045D19030B

For this, we can use the ‘invert-checkbox:

You’ll see that the label of the text field has changed from “Must include the text” to “Must not include the text”. Click on save, and any Cisco IOS device that still contains this line will be marked as non-compliant. 

Check lines that can contain variable values

Suppose that you require all your devices to have hostnames that follow a same pattern. For example, they should first have six letters, followed by a dash and then followed by two numbers, and you want to validate that the hostname defined within your configuration follows these constraints. For this you can use the regular expressions option:

Standard regular expression rules apply. For a detailed guide we refer to here: https://www.dataquest.io/blog/regex-cheatsheet/

The basic guide is that a-z matches all lowercase characters, A-Z uppercase, 0-9 all numbers. Between square brackets are all characters that can be matched and between the curly braces is the amount of characters that it should match. * means any amount of characters and + means one or more characters should match. /s stands for all whitespace characters, and /w matches all alphanumeric characters (and underscores).

Check if lines are present in your configuration, while others aren’t

So suppose you want to check whether your users are defined in your Cisco IOS configuration, but you also want to make sure that there are no other users that have been sneakily defined that could grant a malicious actor access. For that you can click on “+ But not the text”-link below the text input field:

Here we check if a user named admin is defined. If any other user is defined, then the test will fail. Here we again use regular expressions in order to match any wildcard user. You could also get away with simply entering “username” here, but be careful, because the entire configuration is checked here. If the name “username” occurs anywhere else in the config (for example in a banner or in a description somwhere), it will also be matched. It is therefore recommended for you to be as specific as possible when writing your tests, so that you won’t get any false positives. 

Validate the output of a command run on your device

Configurations are big, and they can get quite complex when sometimes the only thing you want to check is whether the clock is set correctly. In these cases it is also possible to run a show command on your device, and test against the output of this command, and let us run the following command at our sample cisco device:

show hostname
hostname SWITCH-01

Let’s say we want to validate that the output of ‘show hostname’ is indeed ‘SWITCH-01. For this you need the Type field of the rule form. By default it is configuration, but for this example we want to switch it to ‘command’:

When a test is run, we retrieve the configuration real-time. At the same time we retrieve this configuration we also execute any commands from command rules and group them together, so both configuration and command outputs are up to date. 

Validate multiple lines

Suppose you have a strict definition of the users in your Cisco IOS device. The user admin should be present with admin rights, and the users alice and bob should be present, without admin rights. No other users should be present. So we need to validate the following lines in your configuration:

username admin privilege 15 password 7 1419171F15072F
username alice password 7 05481C5C235E1A194D015D
username bob password 7 045D19030B

Validating multiple lines is not possible with simple rules. For that, we need python rules. These are rules you can fully customise using python:

This will create a python rule with a basic script to start with:

What is automatically generated here is a rule that checks if the string “text” is present in a configuration for a Cisco IOS device. It gives this a medium priority. Don’t worry about some of the other bits that you see here, for now we’ll focus on the code we need to validate our users. To start with, here is the code we need to validate the admin user:

@medium(
    name='rule_textblock',
    platform=['cisco_ios'],
)
def rule_textblock(configuration, commands, device):
    assert 'username admin privilege' in configuration

To test more lines, simply add more assert lines:

@medium(
    name='rule_textblock',
    platform=['cisco_ios'],
)
def rule_textblock(configuration, commands, device):
    assert 'username admin privilege' in configuration
    assert 'username alice password' in configuration
    assert 'username bob password' in configuration

And now we need to make sure that there are no users defined. We know that to define a user, you need a line that starts with the word ‘username’. So if we count the amount of times the string ‘username’ appears in the configuration, we know that if the answer is more than three, some extra user that does not belong has been defined:

@medium(
    name='rule_textblock',
    platform=['cisco_ios'],
)
def rule_textblock(configuration, commands, device):
    assert 'username admin privilege' in configuration
    assert 'username alice password' in configuration
    assert 'username bob password' in configuration

    total_usernames = configuration.count('username')

    # Check if there are exactly 3 instances of 'username' (the ones we've just asserted)
     assert total_usernames == 3

Save, and the rule should work.

Conditional rules

Python rules support all of the basic python functionality, including conditional statements. Suppose we have a device where we want to confirm that its hostname in its configuration is the same as its name in Netpicker. Let’s pick a device with hostname cisco_device as an example:

The device parameter is a variable that contains a number of attributes of a device we can test against. Here we use the device’s hostname. The if-statement works the same as in python: if (and only if) the device’s hostname in Netpicker is equal to ‘cisco_device’, it will try to check if the line ‘hostname cisco_device’ is present in the configuration. Otherwise it won’t test anything, and the test is automatically compliant.

In real life, you’d want to make this a bit more generic. Here is a test that checks for every Cisco IOS device if its configuration contains the hostname as defined in Netpicker:

Here we put the variable ‘device.hostname’ directly into the string that we want to assert.

Verifying access lists

Access lists are tricky to validate, since not only do they need to be in a specific order, they also cannot contain lines in between. Suppose our Cisco device has this access list that we want to verify:

access-list 198 permit tcp any any eq 22
access-list 198 permit tcp 212.19.192.0 0.0.31.255 any eq telnet
access-list 198 permit udp 172.16.0.0 0.0.255.255 any eq snmp
access-list 199 dynamic Cluster-NAT permit ip any any

We want to verify that all these lines exist, with these ip addresses, and at the same time that there aren’t any sneaky extra lines in between them. in this simple example it would be easy to count the amount of access lists, but in real life situations you’re dealing with very long access lists that counting them will just be impractical, so let’s come up with a solution that checks the configuration line by line. Let’s start by just checking if the first line in our access list is present:

@medium(
    name='rule_access_list',
    platform=['cisco_ios'],
)
def rule_access_list(configuration, commands, device):
    acl_index = -1
    for index, line in configuration.lines:
        if line == "access-list 198 permit tcp any any eq 22":
            acl_index = index
            break

    # Assert if the line was not found
    assert acl_index != -1, "Access list 198 was not found"

What happens here is that in the line for index, line in configuration.lines: we loop over all of the lines in the configuration. When we encounter the line we want, we mark it in the variable acl_index. If acl_index is still equal to -1 after the loop ends, we know that this access list entry is not present. Break is there to jump out of the loop, since it’s not necessary to continue checking the other lines when we do find it.

Now we need to define the consecutive lines that we need to check:

expected_lines = [
    "access-list 198 permit tcp 212.19.192.0 0.0.31.255 any eq telnet",
    "access-list 198 permit udp 172.16.0.0 0.0.255.255 any eq snmp",
    "access-list 199 dynamic Cluster-NAT permit ip any any"
]

And then we check if each of them is directly following the first line we found:

The function enumerate is a built-in python function that gives an index to our expected_lines. This allows us to pinpoint the positions of these lines in our configuration. The first index starts at 0, so if we want to check the line after our first match, we need to add 1 to it. We then assert whether these lines match each other, and if not the test fails, returning with an explanation, plus the line we actually found.

Verifying banners

A common use case is verifying if the banners on your device still are what you expect. These are long strings that contain weird characters at strange places, but the advantage is that they usually don’t have that much variability. Where in access lists you might want to customise certain values and ip addresses on each line, you know that a banner can only be one thing and one thing only. Instead of looping over lines we can compare the entire thing at once:

Checking multiple port configurations

Let’s now create a bit of a more complex example to show how powerful netpicker’s python rules can get. Let’s take our cisco device and look at the way it defines its ports:

interface FastEthernet0/2
 description test 2
 mtu 1548
 duplex full
 speed 100
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 1,1002-1005
 switchport mode trunk
!

They start with the label interface, followed by the name of the port. After this you see a number of lines, indented by one space and it’s closed off by an un-indented exclamation mark. Suppose for this use case, we want to make sure that all ports on the device are active and available. So we don’t want any ports defined like this:

interface FastEthernet0/3
 shutdown
!

So for our rule, we need to loop over every configuration block. And for each of the blocks, we need check if it doesn’t contain the word ‘shutdown’. This is how you can tackle it:

@medium(
    name='rule_interfaces',
    platform=['cisco_ios'],
)
def rule_interfaces(configuration, commands, device):
    in_interface_block = False
    found_shutdown = False

    for line in configuration.lines:
        # Check if the line marks the beginning of an interface block
        if line.startswith('interface '):
            in_interface_block = True
            # Check if the line marks the end of an interface block
        elif line.strip() == '!':
            in_interface_block = False
            # If in an interface block, check for 'shutdown'
        elif in_interface_block and line.strip() == 'shutdown':
            found_shutdown = True

    # Assert that no 'shutdown' command was found in any interface block
    assert not found_shutdown, "Found 'shutdown' command in an interface block."

This code first defines two boolean variables to keep track of the status of the line being examined, and then it loops over every line in the configuration. If it encounters a line that starts with ‘interface’, the in_interface_block value is set to True. From that point on it checks any mention of the word ‘shutdown’, until it encounters an exclamation mark again. If it does encounter a shut off port, the found_shutdown boolean is set to true. If this value remains false after all of the lines have been examined, we can be sure that all ports are functioning.

But we can do better than this. This is nice to see if any port has shut down at all, but you don’t get any information on which actual port this has been applied to, let alone if there are multiple ports. It would be nice to see the actual port name in the test result:

@medium(
    name='rule_interfaces',
    platform=['cisco_ios'],
)
def rule_interfaces(configuration, commands, device):
    in_interface_block = False
    interface_name = ""
    shutdown_interfaces = []

    for line in configuration.lines:
        if line.startswith('interface '):
            in_interface_block = True
            interface_name = line.strip()
        elif line.strip() == '!':
            in_interface_block = False
            interface_name = ""
        elif in_interface_block and line.strip() == 'shutdown':
            shutdown_interfaces.append(interface_name)

    assert not shutdown_interfaces, f"Shutdown command found in interfaces: {', '.join(shutdown_interfaces)}."

Here we store the name of the interface in the array shutdown_interfaces. This we can then print on the assert at the end, so that you know which interfaces you need to deal with.

Checking live commands

You can also execute live show commands in your python code and test their output. Suppose we just want to see whether the interface FastEthernet0/1 is shut down or not. You can send a request through the device parameter in the python function:

The cli function of the device will return the output of the command sent through its parameters. This can then be compared in the same way as the configurations we’ve been doing.

Checking redundant configurations

Netpicker’s python tests also allow you to use the configurations of multiple devices. Suppose you have two Cisco IOS devices that need to be redundant. For the sake of example, let us verify that their interface FastEthernet0/1 settings match each other. This is the rule that can do that for you:

What you see here is an extra argument called device_tags. This tells netpicker to only perform this test on devices that have the device tag of “load_balancer”. You can set these tags in the device form:

You can find your devices in the devices parameter. This is a dictionary that you can loop over, with the keys being the hostnames of the devices. You can then send a command to all devices, and then compare the outputs against each other to see if they are equal to each other. If they are not equal, then both devices will show up in your reports as not being compliant.

In this example we restrict ourselves to only two devices, with a single interface, but you can easily extend this rule so that it checks for four devices, all running on the same configuration. By using the right show commands, you can check against every interface and vlan, allowing you to validate every aspect of your devices.