Services HOT Session: Monitoring as Code

Welcome to the APAC Services Hands on Training session.

ace logo

👉 Ensure you can login to your Dynatrace Managed tenant

👉 Ensure you can SSH into your VM.

👉 Ensure you can reach: http://52.200.197.144.nip.io/p/mac

👉 Once logged in. Go to Settings > Integration > Dynatrace API > Token Settings and enable the new API format.

👉 Generate an API Token with the following permissions:

api token permissions

New API Key format: Good article from Barbara Schachner

👁️‍ Retrieve an Auto Tag

I have created an auto-tag called myFirstTag. Please help me retrieve the JSON representation of this tag.

The Dynatrace customer you've been assigned to sells SaaS to their end customers. Currently they have 3 customers: Customer A, Customer B and Customer C.

Your customer is currently running on AWS but they have plans to extend their service to Azure and / or GCP in the near future.

Your customer is planning a large marketing event in just over 1 month. They expect to have over 1000 customers within the first few months.

Your customer would like to use a single monitoring tool (Dynatrace) to monitor the health of their entire customer environment. They need to see statistics, health and metrics of individual customers views in addition to a "global" health dashboard for all customers.

Your job is to help them achieve this goal. You have less than 1 month to deliver this outcome.

Each participant should have received access to a Dynatrace environment. In this environment you'll find a single host connected.

You've received the following architecture diagram from your customer.

customer-architecture

http://staging.customera.VMIP.nip.io
http://customera.VMIP.nip.io
http://staging.customerb.VMIP.nip.io
http://customerb.VMIP.nip.io
http://staging.customerc.VMIP.nip.io
http://customerc.VMIP.nip.io

👉 Please visit each customer website and ensure they are available.

👁️‍ Take 10 minutes and use the chat box to discuss with colleagues:

👁️‍ Has anyone delivered Dynatrace configuration as code to your customers? What was involved?
  1. Pull the existing kubernetes annotations and labels. Apply as Dynatrace tags
  2. Tag the process groups per customer
  3. Tag the services per customer
  4. Create friendly process group names by prepending the customer name
  5. Create friendly service names by prepending the customer name
  6. An application with URL rules, one per customer environment
  7. A management zone per customer

Customer Delivery of Configuration

We will deliver all of this configuration to the customer as code.

The customer can then:

Dynatrace can then:

infra as code meme

Due to the scale that this customer expects, every configuration we deliver must be automated. That said, we can build the template manually then use automation to build subsequent configuration.

Introducing Monaco

Monaco is an open source project from Dynatrace to apply Monitoring As Code at scale.

The monaco tool is idempotent. It will only make changes if required. Re-run as many times as you wish.

👉 Download Monaco

The monaco binary is already available on the VM as ./monaco so either run it from the VM or download the Windows binary and run it locally. Your choice.

Go to https://github.com/dynatrace-oss/dynatrace-monitoring-as-code/releases and download the latest monaco release for your operating system.

👉 Setup Monaco

For ease during this demo, please rename the binary to ./monaco (if on windows) or monaco if on macOS or linux.

Status Check

You should now have:

file and folder structure

Define Your Environment

Open environments.yaml and type the following:

mac_training_environment:
  - name: "mac_training_environment"
  - env-url: "{{ .Env.MAC_TRAINING_ENVIRONMENT }}"
  - env-token-name: "TOKEN_MAC_TRAINING_ENVIRONMENT"

Copy & Paste version here.

Set Dynatrace URL

Open a new command window and set your Dynatrace URL as an environment variable.

Notice that the name of this variable matches the env-url variable you used in environments.yaml

set MAC_TRAINING_ENVIRONMENT=https://dtmanaged.dynatrace.training/e/***

Set Dynatrace API Token

Repeat the process by setting your API token. Again notice that the name corresponds to the value used in environments.yaml:

set TOKEN_MAC_TRAINING_ENVIRONMENT=***

Run Monaco

In the same command window, run monaco:

./monaco --environments=environments.yaml --dry-run

You should see:

> ./monaco --environments=environments.yaml --dry-run
  INFO  Dynatrace Monitoring as Code v1.0.1
  INFO  Executing projects in this order:
  INFO  Processing environment mac_training_environment...
  INFO  Validation finished without errors

File and Folder Structure

Monaco has the following concepts:

  1. Environments
  2. Projects
  3. Configuration Types

Environments

Environments are simply Dynatrace environments, SaaS or Managed. monaco can push configuration to one or more environments.

As we've already seen, environments are defined in environments.yaml.

Projects

Projects are distinct units of configuration. The most basic is to create a single project which holds your configuration. You can then push this identical configuration out to one (or multiple) environments.

Projects are created by creating a new folder inside the projects folder.

Configuration Types

A single configuration item in Dynatrace. The monaco tool currently supports the following configuration types:

k8s labels and annotations as Dynatrace Tags

✅ This has already been configured in your demo environments.

The customer has already annotated their Deployment with the following labels which we can leverage automatically in Dynatrace.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prod-web
  namespace: customer-a
  labels:
    app: web
    stage: prod

This is simple, give the default service account permission to view metadata about the customer-a, customer-b and customer-c namespaces.

kubectl -n customer-a create rolebinding default-view --clusterrole=view --serviceaccount=customer-a:default
kubectl -n customer-b create rolebinding default-view --clusterrole=view --serviceaccount=customer-b:default
kubectl -n customer-c create rolebinding default-view --clusterrole=view --serviceaccount=customer-c:default

k8s labels

Namespace = Customer Tag

Recall that your client creates a new namespace for each of their customers.

👉 Create an auto-tag rule via the UI which matches this definition:

customer-auto-tag

👁️ What will this tag do?

👉 Get JSON Representation

Use the Dynatrace API to retrieve the JSON representation of that tag.

Use the Automatically Applied Tags endpoint in the Configuration Api to GET all tag IDs:

{
  "values": [{
      "id": "afd026e7-2abc-492f-873e-66bb571a572a",
      "name": "customer"
    }]
}

Use the other GET endpoint by passing this ID to retrieve the full JSON.

GET /api/config/v1/autoTags/afd026e7-2abc-492f-873e-66bb571a572a

Save this JSON as a file.

baseconfig project

👉 Inside projects, create a folder called baseconfig. This folder will hold all of our configuration.

👉 Create auto-tag folder for Monaco

Inside projects/baseconfig, create a new folder called auto-tag.

YAML & JSON Explanation

Inside each configuration type folder you need one YAML file and one (or more) JSON files.

Every configuration needs a YAML containing required and optional content.

A minimal viable config looks like this:

config:
    - {config name} : "{path of config json template}"

{config name}:
    - name: "{a unique name}"

The JSON file(s) inside this folder essentially hold the content that you'd normally POST to Dynatrace.

The only difference is that you can use variables inside the JSON which reference the YAML file.

Let's see that in action...

👉 Create tags.yaml

Create projects/baseconfig/auto-tag/tags.yaml:

config:
  - customer_tag: "customertag.json"

customer_tag:
  - name: "customer"

Copy & Paste version here.

👉 Create customertag.json

Create projects/baseconfig/auto-tag/customertag.json

Paste in the JSON content that you retrieved from the Dynatrace API.

👉 Cleanup the JSON

There are a few sections we don't need in customertag.json.

Remove the metadata section and id variable. Your JSON should now look like this:

{
  "name": "customer",
  "rules": [
    {
      "type": "PROCESS_GROUP",
      "enabled": true,
      "valueFormat": "{ProcessGroup:KubernetesNamespace}",
      "propagationTypes": [
        "PROCESS_GROUP_TO_SERVICE"
      ],
      "conditions": [
        {
          "key": {
            "attribute": "PROCESS_GROUP_PREDEFINED_METADATA",
            "dynamicKey": "KUBERNETES_NAMESPACE",
            "type": "PROCESS_PREDEFINED_METADATA_KEY"
          },
          "comparisonInfo": {
            "type": "STRING",
            "operator": "EXISTS",
            "value": null,
            "negate": false,
            "caseSensitive": null
          }
        }
      ]
    }
  ]
}

👉 Template the JSON

Notice that we have certain hard coded values in the JSON. We can template these.

Templates in monaco are denoted with double curly braces.

Replace customer with {{ .name }}:

{
  "name": "{{ .name }}",
  "rules": [...]
  ...
}

Explanation of templates

When monaco executes, it will look at each configuration in the YAML and process them sequentially. We have one configuration:

- customer_tag: "customertag.json"

monaco then looks at the JSON and the properties for customer_tag. The properties are:

customer_tag:
  - name: "customer"

monaco replaces anything inside curly braces with the corresponding value (denoted by .VariableName).

In this way, "{{ .name }}" becomes "customer".

👉 Apply Configuration as Code

Delete any previous tag rules you have in the tenant.

Apply your configuration:

./monaco --environments=environments.yaml

You should see that one configuration has been applied:

./monaco --environments=environments.yaml
  INFO  Dynatrace Monitoring as Code v1.0.1
  INFO  Executing projects in this order:
  INFO        1: projects\baseconfig (1 configs)
  INFO  Processing environment mac_training_environment...
  INFO        Processing project projects\baseconfig...
  INFO  Deployment finished without errors

customer-tag

Prepend customer name to process

We need a way to quickly distinguish the processes (pods) for each customer. This configuration will prepend the namespace (aka customer name) to the process group.

👉 Create Manually

Go to Settings > Processes and containers > Process group naming. Add a new rule:

prepend process group

Name format is: {ProcessGroup:KubernetesNamespace} - {ProcessGroup:DetectedName}

👉 Get PG Naming Rules via API

Hint: Dynatrace API > Configuration API > Conditional Naming

{
  "values": [{
      "id": "c6e41ebd-726a-4d79-9fd6-d3bcc37b836b",
      "name": "Prepend Customer Name to PG"
    }]
}

👉 Get Definition

{
  "metadata": {
    "configurationVersions": [
      0
    ],
    "clusterVersion": "1.206.95.20201116-094826"
  },
  "id": "c6e41ebd-726a-4d79-9fd6-d3bcc37b836b",
  "type": "PROCESS_GROUP",
  "nameFormat": "{ProcessGroup:KubernetesNamespace} - {ProcessGroup:DetectedName}",
  "displayName": "Prepend Customer Name to PG",
  "enabled": true,
  "rules": [
    {
      "key": {
        "attribute": "PROCESS_GROUP_PREDEFINED_METADATA",
        "dynamicKey": "KUBERNETES_NAMESPACE",
        "type": "PROCESS_PREDEFINED_METADATA_KEY"
      },
      "comparisonInfo": {
        "type": "STRING",
        "operator": "EXISTS",
        "value": null,
        "negate": false,
        "caseSensitive": null
      }
    }
  ]
}

👉 Cleanup & Use Variables

Remove metadata and id. Use variables in JSON.

{
  "type": "PROCESS_GROUP",
  "nameFormat": "{ProcessGroup:KubernetesNamespace} - {ProcessGroup:DetectedName}",
  "displayName": "{{ .name }}",
  "enabled": true,
  "rules": [{
      "key": {
        "attribute": "PROCESS_GROUP_PREDEFINED_METADATA",
        "dynamicKey": "KUBERNETES_NAMESPACE",
        "type": "PROCESS_PREDEFINED_METADATA_KEY"
      },
      "comparisonInfo": {
        "type": "STRING",
        "operator": "EXISTS",
        "value": null,
        "negate": false,
        "caseSensitive": null
      }
    }]
}

👉 Create Folder and File Structures

  1. Create projects/baseconfig/conditional-naming-processgroup
  2. Create rules.yaml
  3. Create prepend-customer-name.json

👉 Complete YAML

config:
  - prepend_customer_name: "prepend-customer-name.json"
  
prepend_customer_name:
  - name: "Prepend Customer Name to PG"

👉 Complete JSON

prepend-customer-name.json should have this content:

{
  "type": "PROCESS_GROUP",
  "nameFormat": "{ProcessGroup:KubernetesNamespace} - {ProcessGroup:DetectedName}",
  "displayName": "{{ .name }}",
  "enabled": true,
  "rules": [{
      "key": {
        "attribute": "PROCESS_GROUP_PREDEFINED_METADATA",
        "dynamicKey": "KUBERNETES_NAMESPACE",
        "type": "PROCESS_PREDEFINED_METADATA_KEY"
      },
      "comparisonInfo": {
        "type": "STRING",
        "operator": "EXISTS",
        "value": null,
        "negate": false,
        "caseSensitive": null
      }
    }]
}

👉 Delete Manually Created Configuration

Delete the manually created process group naming rule from the Dynatrace UI.

delete manual pg naming rule

👉 Run Monaco

Run monaco with the --dry-run flag to check for syntax errors:

./monaco --environments=environments.yaml --dry-run

When satisfied, run without the --dry-run flag to push your new config:

./monaco --environments=environments.yaml

Notice there are now two configurations applied:

./monaco --environments=environments.yaml
  INFO  Dynatrace Monitoring as Code v1.0.1
  INFO  Executing projects in this order:
  INFO        1: projects\baseconfig (2 configs)
  INFO  Processing environment mac_training_environment...
  INFO        Processing project projects\baseconfig...
  INFO  Deployment finished without errors

pg naming rule applied

For fun, run the monaco tool again as many times as you wish. Notice it is perfectly safe.

Build configuration to prepend the customer name to the service.

Hints:

You have 10 minutes to complete this task

We have now built the following:

We have 6 websites:

  1. Customer A Staging
  2. Customer A Production
  3. Customer B Staging
  4. Customer B Production
  5. Customer C Staging
  6. Customer C Production

Basic XHR detection and JQuery support should be enabled by default for all websites. Customer wants 20% of traffic to be captured in staging environments and 100% capture in production.

👉 Use the monaco tool to define applications for each of these websites.

⚠️ Do not define the application URL rules yet, we'll do that together next.

Time: 10 minutes

Your customer made a mistake! Their clients are paying for Dynatrace and some have complained that they're not seeing all their visits.

They should actually be capturing the following traffic amounts for each customer:

👉 Please correct the traffic capture levels for your customer using monaco.

Time: 5 minutes

Time to apply URL rules to the applications. So far we've dealt with configurations that stand alone. They do not reference, depend on or relate to any other configurations.

URL rules relate to the application for which they're assigned. So we have this:

Customer A - Staging

http://staging.customera

Customer B - Staging

http://staging.customerb

Customer C - Staging

http://staging.customerc

Customer A - Production

http://customera

Customer B - Production

http://customerb

Customer C - Production

http://customerc

👉 Create Configuration Type

YAML File Content

config:
  - customer-a-staging: "rule.json"
  - customer-a-production: "rule.json"
  - customer-b-staging: "rule.json"
  - customer-b-production: "rule.json"
  - customer-c-staging: "rule.json"
  - customer-c-production: "rule.json"

customer-a-staging:
  - name: "/projects/baseconfig/application/customer-a-staging.name"
  - application_id: "/projects/baseconfig/application/customer-a-staging.id"
  - pattern: "http://staging.customera"

customer-b-staging:
  - name: "/projects/baseconfig/application/customer-b-staging.name"
  - application_id: "/projects/baseconfig/application/customer-b-staging.id"
  - pattern: "http://staging.customerb"
  
customer-c-staging:
  - name: "/projects/baseconfig/application/customer-c-staging.name"
  - application_id: "/projects/baseconfig/application/customer-c-staging.id"
  - pattern: "http://staging.customerc"
  
customer-a-production:
  - name: "/projects/baseconfig/application/customer-a-prod.name"
  - application_id: "/projects/baseconfig/application/customer-a-prod.id"
  - pattern: "http://customera"

customer-b-production:
  - name: "/projects/baseconfig/application/customer-b-prod.name"
  - application_id: "/projects/baseconfig/application/customer-b-prod.id"
  - pattern: "http://customerb"
  
customer-c-production:
  - name: "/projects/baseconfig/application/customer-c-prod.name"
  - application_id: "/projects/baseconfig/application/customer-c-prod.id"
  - pattern: "http://customerc"

JSON File Content

{
  "applicationIdentifier": "{{ .application_id }}",
  "filterConfig": {
    "pattern": "{{ .pattern }}",
    "applicationMatchType": "BEGINS_WITH",
    "applicationMatchTarget": "URL"
  }
}

👉 Apply with Monaco

./monaco --environments=environments.yaml

url rule output

👉 We need to update the customer auto tag rule to include the application.

Hint: You only need to adjust the auto-tag/customertag.json file. You probably need to experiment in the UI first :)

update customer tag

Time: 15mins

Status Check

You should now have code which creates the following components:

Create a management zone for each customer and environment which encompasses all processes, process groups, services and applications.

You should end up with 6 management zones:

Time: 20mins

So far we've only created configuration. What about deletion?

Create a file in the root folder (alongside ./monaco) called delete.yaml

delete.yaml has the following syntax:

delete:
  - "ConfigurationType/ConfigurationName"

For example to delete an auto-tag which looks like this:

config:
  - customer_tag: "customertag.json"

customer_tag:
  - name: "customer"

Your delete.yaml would look like this:

delete:
  - "auto-tag/customer"

Run ./monaco as normal and it will delete your config:

./monaco --environments=environments.yaml
  INFO Dynatrace Monitoring as Code v1.0.1
  INFO Executing projects in this order:
  INFO   1: projects\baseconfig (15 configs)
  INFO Processing environment mac_training_environment...
  INFO   Processing project projects\baseconfig...
  INFO Deployment finished without errors
  INFO Deleting 1 configs for environment mac_training_environment...

To prevent configuration deletion, just rename the delete.yaml to something else.

You now have a very stable base setup for your customer. Moreover your customer can take this configuration as code and apply it from their build pipelines.

You can share this configuration with customers and colleagues so they can easily understand what and how you have built things.

You have built this in a completely future-proof and flexible way that your customer can easily understand and amend to any new requirements.

You have successfully built the following:

❓ Thoughts, ideas, comments or questions?

If you'd like to extend this demo, there are a number of configurations you could still apply as code. Here are some ideas: