Select Page

Azure alerts to Microsoft Teams (and Slack)

I found myself in the position of having to get Azure Monitor alerts notifications into Microsoft Teams. So I thought it would be as easy as creating an Incoming Webhook in Teams and adding its URL to an Azure Action Group, right? Wrong!

After trying that in vain, I’ve ended up “googling” on this subject only to find out that the best way to achieve this is using an Azure Logic App. I’ve also seen others using Function Apps while the easiest approach of them all is that of relying on an alert management system similar to PagerDuty (etc.) or a better monitoring solution.
The only problem with the easiest approach is that the organisation needs to be willing to pay for the tool unless you’re in the fortunate position to have one already, which I wasn’t. Hence, let’s go down the complicated route, the way Microsoft likes to do things anyway (private joke).

One of the most common results when looking for a Logic App approach is this article by Bruno Gabrielli, https://techcommunity.microsoft.com/t5/core-infrastructure-and-security/azure-monitor-alert-notification-via-teams/ba-p/2507676
However, like other attempts on this subject, I don’t think that guide is stupid-proof as I still wasted hours until I’ve eventually figured out how to get it done. So I thought will write down something which I hope might help me (or someone else) if I have to do this again at a later stage.

Logic App

Let’s create a Logic App based on a Consumption plan (I’ve seen reports that this is not going to work with a Standard plan):

Once created and accessed, we are greeted by a Logic App Designer with a few proposed triggers to choose from. Let’s pick “When a HTTP request is received”.

Past the following JSON schema and click Next step (you might want other payloads depending on the type of alert you want to send to Teams):

{
    "properties": {
        "data": {
            "properties": {
                "context": {
                    "properties": {
                        "condition": {
                            "properties": {
                                "allOf": {
                                    "items": {
                                        "properties": {
                                            "dimensions": {
                                                "items": {
                                                    "properties": {
                                                        "name": {
                                                            "type": "string"
                                                        },
                                                        "value": {
                                                            "type": "string"
                                                        }
                                                    },
                                                    "required": [
                                                        "name",
                                                        "value"
                                                    ],
                                                    "type": "object"
                                                },
                                                "type": "array"
                                            },
                                            "metricName": {
                                                "type": "string"
                                            },
                                            "metricValue": {
                                                "type": "integer"
                                            },
                                            "operator": {
                                                "type": "string"
                                            },
                                            "threshold": {
                                                "type": "string"
                                            },
                                            "timeAggregation": {
                                                "type": "string"
                                            }
                                        },
                                        "required": [
                                            "metricName",
                                            "dimensions",
                                            "operator",
                                            "threshold",
                                            "timeAggregation",
                                            "metricValue"
                                        ],
                                        "type": "object"
                                    },
                                    "type": "array"
                                },
                                "windowSize": {
                                    "type": "string"
                                }
                            },
                            "type": "object"
                        },
                        "conditionType": {
                            "type": "string"
                        },
                        "description": {
                            "type": "string"
                        },
                        "id": {
                            "type": "string"
                        },
                        "name": {
                            "type": "string"
                        },
                        "portalLink": {
                            "type": "string"
                        },
                        "resourceGroupName": {
                            "type": "string"
                        },
                        "resourceId": {
                            "type": "string"
                        },
                        "resourceName": {
                            "type": "string"
                        },
                        "resourceType": {
                            "type": "string"
                        },
                        "subscriptionId": {
                            "type": "string"
                        },
                        "timestamp": {
                            "type": "string"
                        }
                    },
                    "type": "object"
                },
                "properties": {
                    "properties": {
                        "key1": {
                            "type": "string"
                        },
                        "key2": {
                            "type": "string"
                        }
                    },
                    "type": "object"
                },
                "status": {
                    "type": "string"
                },
                "version": {
                    "type": "string"
                }
            },
            "type": "object"
        },
        "schemaId": {
            "type": "string"
        }
    },
    "type": "object"
}

In the Next step search for Condition. Pick Control and choose Condition, select a status dynamic content and set it is equal to Activated.

Continue and add Teams actions (Post message in a chat or channel) to both True and False conditions (you can do the same with Slack and also send notifications to multiple channels at the same time).

Now, here you can go ahead and complete the Message with dynamic content and/or expressions.
In my case, I wanted a clickable link to the Azure alert. The URL is given by the portalLink dynamic content, however, that doesn’t come clickable in Teams and I’ve only been able to render that clickable by editing the Logic App’s messageBody section via the code view.
Essentially, my messageBody in the True condition looks like this:
"messageBody": "<p>🚨 Azure <strong></strong><strong>@{triggerBody()?['data']?['context']?['name']}</strong><strong></strong><br>\n@{triggerBody()?['data']?['context']?['description']}.<br>\n<a href=\"@{triggerBody()?['data']?['context']?['portalLink']}\">@{triggerBody()?['data']?['context']?['portalLink']}</a></p>",
Then for False I simply have:
"messageBody": "<p>The Azure <strong>@{triggerBody()?['data']?['context']?['name']}</strong> is now <span style=\"color: rgb(65,168,95)\"><strong>Resolved ✔</strong></span><br>\n<a href=\"@{triggerBody()?['data']?['context']?['portalLink']}\">@{triggerBody()?['data']?['context']?['portalLink']}</a></p>",

With all that Saved, we can move on to the Alert.

Alert Rule

For this exercise, I have created a ping URL test from within an App Insights’ Availability and by default that created me an alert for it. Let’s edit this alert rule and add an Action Group:

Now you can either choose an existing Action Group or create a new one which is what I’m doing. Either way, we need to go to the Actions tab and choose Webhook.

You might be tempted to pick the Logic App instead of a Webhook and that’s what you’ll also find in some guides. However, I’ve wasted a lot of time on that and never had it to work as the alerts, although sent to Teams, were coming through empty.
Once Webhook is selected, we have to paste the “HTTP POST URL” from the first Logic App step.

Please note that you do not need to enable the “common alert schema”:

With the alert rule now saved you have it bound to an Action Group coupled with the Webhook pointing at the Logic App HTTP POST URL trigger.
So let’s do some testing.

Teams

As mentioned, in my example I’m using a simple ping URL test against an App Service. I’m going to stop the App Service and once the thresholds are reached I am expecting a notification to my chosen Teams channel.
Here we go:

Starting the App Service back should now trigger a “resolved” notification back to my Teams channel:

Collect Magento logs with Papertrail

I’ve used Papertrail before its acquisition from SolarWinds and was impressed by its simple interface and the logs scrapping capabilities. At the time, I have set Papertrail to monitor a few auth logs and trigger Slack notifications whenever SSH access occurred from a different IP than those whitelisted.

Well, years have passed since I’ve last played with it and now I have a new use case to monitor Magento errors recorded in its var/log/system.log and var/log/exception.log files. As I’ve forgotten how I did this at the time I thought I’ll put down my steps which might help later on.

First thing I’ve gone straight into reading the docs. That brought me into the app log files aggregate page.

Papertrail Setup Logging

The first step is to download the latest remote_syslog2 script from their GitHub repository. The only problem I have with this method is that you will struggle to keep things up-to-date without manually checking the repo for a new version and update. Hopefully one day we would be able to install via an OS package.
Ok, let’s download the latest current version and install. As I’m using Ubuntu I’ve done this:
wget https://github.com/papertrail/remote_syslog2/releases/download/v0.20/remote-syslog2_0.20_amd64.deb
sudo dpkg -i remote-syslog2_0.20_amd64.deb

The second indicated step is that of configuring and starting remote-syslog. The example only shows you how to do it with a single log so without too much fuss I’ve instead downloaded the custom config file and replaced the content of /etc/log_files.yml with:

files:
  - /home/user/myMagentoWebsite.co.uk/public/var/log/system.log
  - /home/user/myMagentoWebsite.co.uk/public/var/log/exception.log
destination:
  host: logs7.papertrailapp.com
  port: 77777
  protocol: tls
exclude_patterns:
  - main.INFO
pid_file: /var/run/remote_syslog.pid

Figured out that we can add multiple files just under the first example line. Also, the original /etc/log_files.yml had an exclude_patterns example which I’ve used to prevent main.INFO records from filling Papertrail as these are outside the scope.

The remote-syslog GitHub page has obviously more examples and explanations if we want to dig further into how it works.

Finally, the sudo remote_syslog command should get us set.

Linux disk usage monitor script

I found that Azure has a hard time to monitor disk usage on a Linux box. There are ways, but it seems to be too complicated to achieve such a basic thing. So I came up with the following script I call in a cron job as needed.

#!/bin/bash
CURRENT=$(df / | grep / | awk '{ print $5}' | sed 's/%//g')
THRESHOLD=90
PUBLICIP=$(curl ipecho.net/plain)
LOCALIP=$(ips=($(hostname -I))
for ip in "${ips[@]}"
do
 echo $ip
done)

if [ "$CURRENT" -gt "$THRESHOLD" ] ; then
    mail -a From:[email protected] -s "Disk Space Alert on ${HOSTNAME} ($PUBLICIP / $LOCALIP)" [email protected] << EOF
The $HOSTNAME root partition remaining free space is critically low. Used: $CURRENT%
EOF
fi

Make sure to adjust the THRESHOLD and emails as needed. Ideally, you’ll configure a mail server (e.g. Postfix) with SMTP credentials (e.g. SendGrid) and proper SPF records to make sure your emails don’t end up in Junk and miss them.

There are plenty of proper monitoring solutions and I always love those that give you a free tier such as HetrixTools Uptime Monitors or Datadog. However, not always these can be used.