You are here

Monitoring and Notifications

Overview

AWS provides flexible tools decide which parts of your infrastructre are monitored, and Burris Builder arrives with configurations and methods to expand on those services. This section outlines the basics of understanding these features and how to be notified when certain events occur.

Accessing Monitoring

Monitoring for your web server can be found by traversing to "Ec2" from your console dashboard. Select "Instances" from the left menu then click "Running Instances," assuming you have at least one running instance. Select the small box to the left of the instance you are running, which will reveal a selection of tabs below.

Status Checks

The Status Checks are fundamental reports of system stability; baseline assurance to make sure an instance is available. Until this reaches 2/2 with a green checkbox in the instance table, your server will be unavailable. It may take several minutes for a "2/2" state to be reached during a launch or restart. Changes to this status during normal operation are rare, but possible. In the event of a sudden change in status to an operating server, restarts are may be required to recover from such a state, which can be an automated or manual process, covered in the CloudWatch section of this documentation

Ec2 Monitoring

The monitoring tab covers all AWS provided metrics into graphs. You may also present your own metrics into this panel, but that is outside the scope of this documentation. Clicking on each graph will enlarge the graph to give you better details and more options. What to watch and creating alarms based on those metrics are covered later in this document.

RDS Monitoring

Any RDS instance created also has metrics which may be watched. From the console dashboard, select "RDS" then select "DB Instances." In instances dashboard, select the box next to the instance you want, then click "Show Monitoring" next to the Launch DB Instance button. Note in the panel that opens, there are two pages of graphs. In the upper right, select "Show All" to reveal all of the available graphs.

Next, we'll setup notifications based on some of these metrics.

Notifications with SNS

In order for monitoring to be effective, notifications are sent out to the humans which maintain the infrastructure when events or thresholds are reached. Depending on the notification, the person may act on the event, or let the automated processes in place take action.

SNS, or Simple Notification Service, is the AWS solution to send alerts or messages to you, or even to other services. You provide AWS with methods of contact and then decide the rules which trigger that contact. SNS is a flexible platform used for sending messages for virtually any occasion or event within AWS. We're going to use it for infrastructure alerts, where you may receive messages by text message (SMS) or email.

Getting Started

The first step is creating an alert topic. An alert topic is a categorized event. People (or potentially other services) are notified based on a subscription list. For this documentation, our topic will be web server issues.

To begin, login into your AWS Console and click "SNS" under "Mobile Services," once you arrive at your dashboard, anc click "Create Topic". The "Name" is machine readable must not include any spaces, the Description will show up in your alerts as subject lines in emails. For this example, we are using "web_server" as the topic name and "Web Server" as the description.

Step 2: Subscribing

Once your topic is created, you need to provide methods of contact; click "Create Subscription." For infrastructure alerts, there are two primary methods of contact, email and SMS. From the Protocol drop-down, select "Email" and type in your preferred email as the Endpoint, and then click Create Subscription

Optionally, you may create a text message alert by clicking Create subscription again, and selecting SMS from the Protocol drop-down. Type in your complete phone number, with starting with your international code. An example of this format is provided in the placeholder of the Endpoint field.

You will need to confirm each subscription. You will receive an email message at the email provided or a text message at the phone number provided. Follow instructions to confirm your subscription.Once finished, a subscription ID should be available under the Subscriptions table. You may have to click the "refresh" button. Once all of your subscriptions have subscription IDs, you're done. Now alert events can be setup.

Monitoring Basics

What to monitor and how to setup your alerts are personal decisions, based on the needs and resources of your infrastructure. This section is intended to get you started by introducing general concepts for monitoring a web server.

The T2 Instance Type

The T2 instance type is recommended for most general uses of delivering websites for Ec2 instances, RDS instances and ElastiCache. They are designed to deliver cost effective compute power by allowing bursts of CPU which are earned as CPU credits. For many websites, this lower cost method of obtaining CPU power is great match for website delivery, where traffic is often sporadic or peaks in certain times of the day; therefore, consistent CPU power is not a requirement.

Since CPU credits allow the full potential of your T2 instance to be reached, monitoring the amount of credits you have available will make sure your instances will perform as expected. T2 instances show two additional metrics, "CPU Credit Usage" and "CPU Credit Balance".

CPU Credit Usage

As your website handles spikes in traffic, or if it performs intensive tasks such as a large update, you will start to use CPU credits. General usage of these credits is fine, as your instances will regain those credits, but if you are seeing unusual spikes in usage, it may help you predict future infrastructure needs, or save you from performance problems.

CPU Credit Balance

The larger your instance, the more CPU credits will have available, as well as a higher baseline CPU performance, making it less likely to need the credits to begin with. With CPU Credit Usage being watched for spikes, this metric can be set

CloudWatch

CloudWatch is where you will create alarms, which then send notifications to the topic(s) created in the Notifications section.

Internal Monitoring

Resources:

  • Burris Builder Monit Documentation

Along with external monitoring performed by AWS services, Burris Builder installs "Monit" for sending alerts and taking action based on certain events.

By default, Monit configuration profiles are installed for Nginx, PHP-FPM, and if they're installed, Redis and Solr.

Monit Mail Server

Although monit will restart services which fail to respond by default, it will not yet send email messages without an email server. The AWS SES is an ideal solution for sending these messages and making sure they land in your inbox. If SES is not yet setup, follow the SES setup link above and return here when finished.

Monit CFG File

SSH into your web server and run the following commands:

cd /etc/monit.d
sudo wget http://bucket.burrisdigital.com/monit/mail.cfg
sudo nano mail.cfg

You should see a file which looks like the following:

set mailserver email-smtp.us-east-1.amazonaws.com port 587
    username "YOURUSERNAMEHERE"
    password "YOURPASSWORDHERE"
    using TLSV1 with timeout 30 seconds

set mail-format {
    from: approvedaddress@example.com
    reply-to: approvedaddress@example.com
    subject: $SERVICE $EVENT at $DATE

    message: Monit $ACTION $SERVICE at $DATE on $HOST: $DESCRIPTION.

    Yours truly,

    monit
}
        

You will need to make a few adjustments to this file:

  • You may need to adjust the mail server for your region. The default listed here is for US East (N. Virginia) The servername can be found under your console's SES Settings
  • Replace username and password, between the quotes, with your credentials created for SES.
  • Replace both the "from" and "reply to' addresses with email addresses approved by SES.

After you have made the changes, type CMD + X on your keyboard. You will be asked if you want to save changes. Type Y then hit Enter

Then, at the terminal command prompt, reload monit by typing sudo service monit reload. You should get an email notifying you of the change made to monit. Your alerts are now setup. You will notice now if you type sudo service nginx resart to restart nginx, for instance, you will receive a notification of a PID change, indicating the service has been restarted.

If you are not receiving messages, check the monit logs located at /var/log/monit, or check CloudWatch Logs if your server was setup with a role.