Dynatrace Extension 2.0 SNMP framework requires that you provide exact OIDs (Object Identifiers) for metrics you wish to monitor. This approach is different to other common SNMP managers where you can present an entire MIB file.

Terminology:

OID (Object Identifier)

An OID is a sequence of digits separated by ‘.' that refers to a particular metric on the device.

For example, the OID 1.3.6.1.2.1.1.3.0 refers to sysUpTime which provides the Uptime of the device in Timeticks.

MIB (Management Information Base):

A MIB is a file, provided by device manufacturer, that contains OIDs accepted by the device. A device usually has multiple MIBs focussing on certain set of metrics. For example, there can be a MIB with OIDs for resource metrics like CPU, Memory etc and another for interface metrics, and so on.

There are 2 types of MIBs:

• Generic MIBs that are common to all devices irrespective of make and model.
• Device specific MIBs that provide device specific metrics.

All MIBs accepted by a device are usually available to download from device management UI and/or the manufacturer website.

Reading MIB files

MIB files usually come with .txt, .my or .mib extensions, but can also depend on manufacturer. The common factor is that all MIBs should be readable in a text editor.

The MIB files usually provide OIDs in string format. In order to get the numeric format you may have to use a MIB browser software.

iReasoning is an example for a popular MIB browser. Please visit iReasoning Website for details.

Gathering required OIDs

Once you have all the relevant MIB files, its time to gather OIDs.

TIP: Recommend writing down a list of metrics that are critical to monitor for your device before you begin. MIB files can contain thousands of OIDs and you can easily get lost looking though it. I have found writing down critical metrics helps maintain focus.

There are broadly two types of OIDs:

1. OIDs that return single values.
2. OIDs that return a table.

OIDs that provide single value

These are the OIDs that provide a single return value - like Uptime, CPU, Memory etc. These metrics should be retrivable using a snmpget command.

Example:

snmpget -v 2c -c public 127.0.0.1 1.3.6.1.2.1.1.3.0

This particular query returns the UpTime of a device.

Breaking down the query:

snmpget: snmp command to gather a metric

-v 2c: Mentions we are using SNMP authenticaion version 2c

-c public: Community string for authenticaion

127.0.0.1: IP address of device

1.3.6.1.2.1.1.3.0: OID to gather details

TIP: Recommend running both snmget and snmwalk on all OIDs. This way you can easily identify if the OID returns a table or single values. Sometimes you will find that you have to append a .0 or .1 to get a single value OID to get the value. This does not make it a table OID.

OIDs that return a table

These are the OIDs that return a table of data when queried. An example for this OID is the interface root OID 1.3.6.1.2.1.2.2.1. These metrics are retrievable using snmwalk command.

Example:

snmpwalk -v 2c -c public 127.0.0.1 1.3.6.1.2.1.2.2.1

This particular query returns values like Index, Description, Type, MTU, Speed, InOctets, OutOctets etc for each interface on a device. If the device has 2 interfaces, each metric will be returned for both interfaces, but if the devices has 20 interfaces, each metric will be returned for all 20 interfaces and so on. The table size (values returned) depends on characteristics of the device.

You will notice that an snmpget on this oid will return an error message as below.

Device MIB documentation is the best way to identify if an OID returns a single value or a table.

OID details to note down during research

1. OID in number format. Eg: 1.3.6.1.2.1.1.3.0
2. Whether the OID returns a single value or a table
3. The return variable type for each value (metric) returned Some common variable types are:
   • Counter32
   • Counter64
     
   • INTEGER
   • Gauge32
   • Gauge64
   • STRING
     
   • Timeticks
   This can be seen in the result of an snmpget or snmwalk command. See highlighted in below result.

Dynatrace Extension Framework 2.0 provides the capability to define extension configuration as a .yaml file. In this page, you will learn how to prepare extension.yaml file rquired for SNMP monitoring.

Please read official Extensions 2.0 concepts for how the extension works.

This documentation does not require a deep understanding of yaml file structure but recommend light reading on the topic.

Extension folder structure

Please structure your extension working directory as below (required):

SNMP
 |
 --> Cisco                      # OK to rename with your device name
        |
        --extension             # DO NOT rename
             |
             -- extension.yaml  # DO NOT rename

YAML file

The extension.yaml file follows below basic structure:

name: <value>               
version: <value>
minDynatraceVersion: <value>
author:
  name: <value>

metrics:
- key: <value>
   metadata:
      displayName: <value>
      description: <value>
      unit: <value>
- key: <value>
   metadata:
      displayName: <value>
      description: <value>
      unit: <value>

snmp:
- group: <value>
  interval: <value>
  
  dimensions:
  - key: <value>
    value: <value>

  subgroups:
    - subgroup: <value>
      table: false    
      metrics:
        - key: <value>
          value: oid:<value>
          type: <value>
    - subgroup: <value>
      table: true
      dimensions:
        - key: <value>
          value: <value>
      metrics:
        - key: <value>
          value: oid:<value>
          type: <value>

Each parameter in order shown above:

name: A name for the extension. This have to be in the format custom:. Example: custom:cisco_router_snmp

version: This is the your extension iteration version. Example: 1.0.0 and go up as 1.1.0, 1.2.0 and so on as you update the extension.

minDynatraceVersion: Earliest version of Dynatrace Server this extension can run on. For the purpose of this documentation please use "1.217"

author

name: Company/your name.

metrics: In this section you define metadata like name, description, unit for your metrics.

key: This is a unique identifier for your metric. Example: custom.cisco.switch.cpu.usage.5second. Please see Metric key for format restrictions.

name: A friendly name for your metric. Example "CPU Usage % - 5secs"

description: A description for your metric

unit: Unit for your metric. Example Percentage, Byte, Count etc. For full list of available Units see Unit parameter under Response body > Response parameters (tab) > The MetricDescriptor object > unit > the element can hold these values

snmp: In this section all required metrics are organized in groups and subgroups. Indivual and Table OIDs have to specified in separate subgroups.

group: A group of metrics. Please see Groups and subgroups for details.

interval: Frequecy of SNMP polling. Examples: 1m, 5m etc. Please see Interval for details.

dimensions: Your dimension value. An analogy for dimension is - If ‘City' is the Dimension, then ‘London', ‘New York', ‘Auckland' etc are some dimension values. A common dimension for SNMP is the device IP address.

A dimension provides the ability to filter metrics. For example, if you want to see CPU breakdown per device, you can specify IP address as a dimension and then use it to ‘split' or ‘filter' results. Please see Dimension for more details.

Multiple dimensions can be defined.

key: A unique key for your dimension. Examples: this:device.address, device:name. Please note that keys cannot have Uppercase characters.

value: oid:: This is where you tell extension where to pick up Dimension values. Examples: this:device.address - value based on extension configuration & 1.3.6.1.2.1.1.5.0 - value from OID

subgroups: Required OIDs are organized into separate subgroups.

metrics

key: The unique values here are the same ‘keys' from root metrics where you defined metadata for the key. So the same key appears twice in the extension.

1. under root metrics and
2. under snmp > group > subgroups > metrics

Other Unique items in this section that were not discussed earlier are table and type

table: This tells the extension if the OIDs are to be treated as individual OIDs or table OIDs. Acceptable values are true or false

type: Dynatrace supports two metric types - gauge and count metrics. Please see Payload for details on these types. Below is a generic rule I follow.

If snmpget or snmpwalk of an OID returns the variable type as Counter32 or Counter64, then ‘type' is Count and if the return variable type is anything EXCEPT STRING, type is Gauge

STRING variable types can ONLY be dimensions. String values cannot be sent to Dynatrace as a metric.

Example extension

Here is a full example of an extension to monitor Cisco Network Switches.

Please note that this extension has not been tested for all Cisco switches. The OIDs may vary between models.

name:  custom:custom.snmp.cisco.switch
version: 1.0.0
minDynatraceVersion: "1.217"
author:
  name: "Monitoring team"
metrics:
  - key: custom.snmp.cisco.switch.uptime
    metadata:
      displayName: System uptime
      description: System uptime
      unit: Count
  - key: custom.snmp.cisco.switch.cpu.usage.5second
    metadata:
      displayName: CPU Usage % - 5secs
      description: The overall CPU busy percentage in the last 5 second period.
      unit: Percent
  - key: custom.snmp.cisco.switch.cpu.usage.1minute
    metadata:
      displayName: CPU Usage % - 1min
      description: The overall CPU busy percentage in the last 1 minute period.
      unit: Percent
  - key: custom.snmp.cisco.switch.cpu.usage.5minutes
    metadata:
      displayName: CPU Usage % - 5mins
      description: The overall CPU busy percentage in the last 5 minute period.
      unit: Percent
  - key: custom.snmp.cisco.switch.memory.used
    metadata:
      displayName: Memory Used
      description: The overall CPU wide system memory which is currently under use.
      unit: KiloByte
  - key: custom.snmp.cisco.switch.memory.free
    metadata:
      displayName: Memory Free
      description: The overall CPU wide system memory which is currently free.
      unit: KiloByte  
  - key: custom.snmp.cisco.switch.ifHCInOctets
    metadata:
      displayName: Interface Traffic - In (IfXTable)
      description: The total number of octets received on the interface.
      unit: Byte
  - key: custom.snmp.cisco.switch.ifHCOutOctets
    metadata:
      displayName: Interface Traffic - Out (IfXTable)
      description: The total number of octets transmitted out of the interface.
      unit: Byte  
  - key: custom.snmp.cisco.switch.ifOperStatus
    metadata:
      displayName: Interface operational status 
      description: The current operational state of the interface. (1)up,(2)down,(3)testing,(4)unknown,(5)dormant,(6)notPresent,(7)lowerLayerDown
      unit: State
  - key: custom.snmp.cisco.switch.ifInOctets
    metadata:
      displayName: Interface traffic - In 
      description: The total number of octets received on the interface.
      unit: Byte
  - key: custom.snmp.cisco.switch.ifOutOctets
    metadata:
      displayName: Interface traffic - Out
      description: The total number of octets transmitted out of the interface.
      unit: Byte  
  
snmp:
- group: Cisco Switch
  interval: 5m
  dimensions:
  - key: device:address  
    value: this:device.address
  - key: device:name    
    value: oid:1.3.6.1.2.1.1.5.0
  subgroups:
    - subgroup: Health
      table: false      
      metrics:
        - key: custom.snmp.cisco.switch.uptime
          value: oid:1.3.6.1.2.1.1.3.0
          type: gauge
        - key: custom.snmp.cisco.switch.cpu.usage.5second
          value: oid:1.3.6.1.4.1.9.9.109.1.1.1.1.6.1
          type: gauge
        - key: custom.snmp.cisco.switch.cpu.usage.1minute
          value: oid:1.3.6.1.4.1.9.9.109.1.1.1.1.7.1
          type: gauge
        - key: custom.snmp.cisco.switch.cpu.usage.5minutes
          value: oid:1.3.6.1.4.1.9.9.109.1.1.1.1.8.1
          type: gauge
        - key: custom.snmp.cisco.switch.memory.used
          value: oid:1.3.6.1.4.1.9.9.109.1.1.1.1.12.1
          type: gauge
        - key: custom.snmp.cisco.switch.memory.free
          value: oid:1.3.6.1.4.1.9.9.109.1.1.1.1.13.1
          type: gauge   
    

    - subgroup: IF XTable
      table: true
      dimensions:
        - key: cisco:interface-name
          value: oid:1.3.6.1.2.1.31.1.1.1.1
        - key: cisco:interface-alias
          value: oid:1.3.6.1.2.1.31.1.1.1.18
      metrics:
        - key: custom.snmp.cisco.switch.ifHCInOctets
          value: oid:1.3.6.1.2.1.31.1.1.1.6
          type: count
        - key: custom.snmp.cisco.switch.ifHCOutOctets
          value: oid:1.3.6.1.2.1.31.1.1.1.10
          type: count
    
    - subgroup: IF Table
      table: true
      dimensions:
        - key: cisco:interface-index
          value: oid:1.3.6.1.2.1.2.2.1.1
        - key: cisco:interface-descr
          value: oid:1.3.6.1.2.1.2.2.1.2        
      metrics:
        - key: custom.snmp.cisco.switch.ifOperStatus
          value: oid:1.3.6.1.2.1.2.2.1.8
          type: gauge        
        - key: custom.snmp.cisco.switch.ifInOctets
          value: oid:1.3.6.1.2.1.2.2.1.10
          type: count
        - key: custom.snmp.cisco.switch.ifOutOctets
          value: oid:1.3.6.1.2.1.2.2.1.16
          type: count

Once extension.yaml is ready, its time to package the extension to a zip file.

The next step in extension creation process is to Sign your extension with a certificate and make it ready for use.

NOTE: This page describes generating and signing an extension using Dynatrace Open Source dt-cli utility. Please see Sign extension official documentation for manual step by step instructions.

Python install

dt-cli is a Python utility and therefore requires Python installed on your computer before you can use it.

This document describes Python installation on Windows OS but steps are similar for other Operating Systems.

1. Download Python - v3.9+

Dowload Python v3.9+ from Python official Downloads page

1. Install Python Open downloaded .exe file, select ‘Add Python 3.10 to PATH' checkbox and then click Install Now Wait for the installation to complete successfully and click close.
2. Open a PowerShell.exe terminal and confirm installation successful.

dt-cli install

1. From your PowerShell terminal run pip install dt-cli. This will download the dt-cli utility onto your computer.
2. Confirm success by running dt --help

Generate certficates and sign extension

Change terminal path to your directory that has the ‘extension' folder. Then run dt extension gencerts - Leave all entries blank by hitting ‘Enter' key.

This generates the CA and Developer Certificates and Keys required to sign the extension.

Next, sign extension by running dt extension build. This command uses the generated certificates to sign the extension and generate a zip file.

Upload ca.pem to ActiveGates

The CA certificate used to sign an extension has to be present on all ActiveGates from which the extension is to run.

If there are multiple ActiveGates in a tenancy, Dynatrace chooses an ActiveGate to run the extension based on ActiveGate group specified in Monitoring configuration (we will be looking at this later).

Copy ca.pem file to following ActiveGate directory on respective OS.

Windows: %PROGRAMDATA%\dynatrace\oneagent\agent\config\certificates\

Linux: /var/lib/dynatrace/oneagent/agent/config/certificates

Assign a group to your ActiveGate

During SNMP monitoring setup, there is the option to select ActiveGate ‘Group' to run the extension. All ActiveGates out of the box belong to ‘default' group. If you would like the extension to run from a subset of your ActiveGates, a group has to be defined.

To add an ActiveGate to a group, specify ‘group' parameter in ActiveGate custom.properties file and restart Dynatrace ActiveGate service.

Location of custom.properties file:

Windows: %PROGRAMDATA%\dynatrace\gateway\config\custom.properties

Linux: /var/lib/dynatrace/gateway/config/custom.properties

Content to add in custom.properties:

[collector]
group = snmp_extension      # Feel free to chose another group name 

Once ActiveGate service is restarted, the new group name should appear in ActiveGate properties under Deployment Status > ActiveGates > .

Upload ca.pem to Dynatrace Tenancy

ca.pem file has to be uploaded to tenancy as well. This is for Dynatrace to verify the integrity of extensions that are uploaded.

Navigate to Settings > Web and mobile monitoring > Credential vault and click Add new credential.

Extension upload and monitoring configuration can be done via both UI and API. This document covers the UI method. If you are interested in doing these through API please refer to Manage Extension 2.0 lifecycle

Upload extension

1. Navigate to Manage > Hub > Extension 2.0 in your environment and click on Upload custom Extesion 2.0
2. Upload your extension to the page with relevant details
3. Upload extension takes you to the configuration page.

Configure extension endpoints

You get to the configuration page right after you upload the extension or by navigating to Manage > Hub > Extension 2.0 in your environment > Your extension

Once on configuration page, follow below steps providing relevant details on each page.

Once saved the configuration should show as active.

Add more monitoring configuration/devices as required.

Once extension successfully connects to your device, metrics should start appearing under Metrics view in UI. You can focus on just your metrics by filtering with metric key name.

Please note metrics can take upto 5 minutes to appear after first setup.

Metadata for ‘count' metrics

You will notice that metrics for which you had defined ‘type: count' (i.e Counter metrics) don't show the metadata you had defined for it in the extension. Instead, they just show up with the full key name.

This is a known issue and can be fixed by manually editing the metadata for the metric as below.

Chart metrics

From this point all metrics can be charted as usual using Dynatrace Data Explorer.

1. Issues during upload Fix: Upload ca.pem certificate to tenancy
2. Metrics don't show after endpoints are setup Reason: SNMP Poll to device is not running as expected. Fix: Check logs at below location to learn more (On ActiveGates in Extension group) Windows: %PROGRAMDATA%\dynatrace\remotepluginmodule\log\extensions Linux: /var/lib/dynatrace/remotepluginmodule/log/extensions

For more details refer to Extensions 2.0 official documentation