VMware vROps - vROps Self-Monitoring Alerts and Notifications

As I mentioned in my previous vROps Self-Monitoring Dashboard post, you need a way to get notified when vROps self-monitoring alerts get triggered. This way you don't have to keep staring at the dashboard all day and can rest assured that if something goes awry you will find out. Even though vROps is deployed as a Virtual Appliance (VA) in most environments, it is important to remember that under the hood it is still just another application with services running in an Operating System. Therefore, just like any other application, it is susceptible to failure at some point. In this post, I will go over how to setup simple email notifications for vROps self-monitoring alerts if there is an issue with one of the vROps objects such as Cluster, Nodes, or one of the application services.

First of all, vROps also has a Rest API notification method, but for simplicity, flexibility, and the lowest common denominator sake, we will only cover the SMTP option here since not all organizations are ready to consume alerts programmatically. However, they all should at least set up vROps self-monitoring alert notifications to stay on top of the health of their vROps implementation.

Setting up email notifications is fairly easy and involves two main steps: creating an outbound alerts instance and a notification rule. But before we get into that, let's look at the available vROps self-health related alert definitions and decide which alerts we want to see. vROps comes with 28 out-of-the-box health related alert definitions; you can find them by going to Content, selecting Alert Definitions and then filtering on Adapter Type field for vRealize Operations Adapter. Review this list and decide which alerts you want vROps to notify you about. This naturally leads to another bigger picture discussion regarding how you're going to handle other vROps alerts pertaining to VMs, Hosts, Datastores, etc. You obviously don't want to open up a flood gate on all alerts, get inundated, desensitized, and stop caring about the problems in your environment because you couldn't keep up with the noise in your mailbox. However, this is mostly an organizational strategy and process discussion, and well beyond the scope of this post. So let's refocus and look at the details for setting-up your vROps Self-Monitoring Alert Notifications. (Note: for now we will be forwarding all of those alerts and tweak it over time to exclude this that are not critical.)

First go to Administration (1) and click on Outbound Settings (2), then click Add (3) to create a new outbound alerts instance.

On the Add/Edit Outbound Alert Instance click the Plugin Type drop-down (1) and select Standard Email Plugin (2). This will expose further settings in the Add/Edit Outbound Alert Instance window.

Provide Instance Name (1); it can be anything you want but making it something meaningful will be helpful in future. Enter your Email Relay FQDN or IP in the SMTP Host field (2). Type 25 in the SMTP Port field (3). Put a meaningful email address sender name in the Sender Email Address (4) and Sender Name (5) fields respectively. Click Test and Save (6) to verify the email relay address and save the configuration. (Note: depending on your company security policy you may need additional SMTP configuration options from your messaging team.)

Now your Outbound Settings should look similar to this:

Next, go to Content (1) and click on Notifications (2) and click Add, (2) which will start the Add Rule wizard.

On the Add Rule wizard window enter the rule Name (1); make it something descriptive like vROps Self-Monitoring Alert Notification, and select Standard Email Plugin (2) from the Method (Select Plug-in Type) drop-down.

Selecting the email plugin adds email configuration options to the window. From the Select Instance (1) dropdown select the Outbound Alert Instance you created at the beginning of this guide, in my case it was "My company Email". Enter your or your team's email address in the Recipient(s) (2) field. You can also configure the other options to your desire, and since they are pretty self-explanatory, I'm not going to cover them here.

Moving on to the Filtering Criteria (this is important so pay attention), this where the most mistakes are made and no alert notifications get delivered. The first option we have to define is Scope. From the Scope dropdown select Object. This will add a Select an Object... button to the right of the dropdown.

Click the Select an Object... button to reveal the Object window.

In the Select Object start typing vRealize Oper... (1) and from the dropdown list select vRealize Operations Manager Self Monitoring (2). 

Once the Select Object field is populated, click Include Descendants (1) checkbox, then click the Select All button (2). This will populate the Descendants field with all relevant vROps objects. Click Select (3) to save.

The Select Object button will now show vRealize Operations Manager Self Monitoring and ## children selected. (Note that the ## will reflect the actual number of vROps objects in your environment which depends on the number of vROps Nodes and Remote Collectors you have deployed.) Click the Notification Trigger dropdown (1) and select Impact (2). 

In the dropdown (1) that appeared next to the Notification Trigger select Health, Risk and Efficiency (2). (Note: as mentioned before, for now we will be forwarding all alert types and tweak it over time to exclude this that are not critical.)

Click the Critically dropdown and select Info, Warning, Immediate and Critical. (Note: as mentioned before, for now we will be forwarding all alert types and tweak it over time to exclude this that are not critical.)

Confirm all your selections and click Save. You should now have a working alert notification for all vROps Self-Monitoring alerts. That's great, but we're not finished yet. There is one more Alert Notification we need to create.

But, before we get to create another alert notification rule, let me point out a common mistake made when creating this self-monitoring alert notification. If you instead select Object Type and then choose vRealize Operations Manager Self Monitoring under Container in the dropdown, you will not be getting any vROps self-monitoring alerts. It does not work as you expect. The reason is that when you select the Realize Operations Manager Self Monitoring under Container Object Type, it only applies to alerts at the container level and not at the various vROps Objects that we want to be notified about.

OK, so finally there is one more Alert Notification rule you need to create. It has to do with the data collection from vCenter(s). Since this post is already getting long, this will be the slightly abridged version of screenshots, so pay attention.

Just as before, start by creating a new notification rule, but this time in the Filtering Criteria we're only going to set up a different trigger. Select Alert Definition from the Notification Trigger dropdown, then click on Select an Alert Definition... 

In the Alert Definitions pop-up window, filter for "collect" and pick vCenter data collection is slow from the list and click Select.

Review the new rule setting and click Save.

So now you will finally be notified when there is an issue with vROps. Let's look at an example of one of the email alert notifications.

One last thing you could do, since these emails are strictly formatted, is forward them to your ITSM solution and have them parsed by a workflow, which could do a CMDB lookup since the object name is included. Once the owner is determined based on the object name in CMDB, a ticket can be created and assigned to the owner's queue. When the ticket gets resolved, a Rest API call can be made back to vROps to cancel the alert based on the Alert ID (UUID) included in the initial notification.

In conclusion, Alerts and Notifications are a very powerful feature of vROps, but you need to plan wisely before opening up the valve.

For more information about vROps, see the following resources:

vROps Extensibility Options:
You can extend vROps functionality by installing additional management packs:
VMware Management Packs include options for vRA, NSX, vRO, Log Insight, AWS, etc.
Blue Medora Management Packs include options for NetApp, Oracle, SAP, UCS, Citrix, etc.

Books:
VMware Performance and Capacity Management - Second Edition by Iwan 'e1' Rahabok
VMware vRealize Operations Managers Essentials by Matthew Steiner
Mastering vRealize Operations Manager by Scott Norris
VMware vRealize Operations Manager Capacity and Performance Management by Iwan 'e1' Rahabok

Official VMware:
VMware Professional Services
Official vROps Documentation
VMware Operations Management White Papers
vROps product page

Blogs:
VMignite by Lan Nguyen
vXpress by @Sunny_Dua 
virtual red dot by @e1_ang
Virtualise Me by @auScottNorris
Elastic Sky Labs by @JAGaudreau
i'm all vIRTUAL by @LiorKamrat