Monitor infrastructure statistics with AWS SDK

by   AWS PHP


Thursday, 01 September 2016



Link to Github repo

There are various ways to manage your cloud infrastructure with Amazon Web Services (AWS). One of these is with the AWS SDK. In this post, I will demonstrate how you can use this service to retrieve relevant statistics about your AWS infrastructure components, for example your instance CPU utilization. Data will be displayed as Google charts and inserted into your admin portal, so that you can quickly gain insights about how well your various services are doing and eventually take further measures to improve them.

Context

AWS use CloudWatch to collect all kind of statistics around your various AWS components. There are literally tons of it, and unless you updated the service to get more frequent insights, it will automatically collect data every 5 minutes. Here we'll focus on some very specific AWS component that most customers are dealing with: Elastic Compute Cloud (EC2) and Amazon Relational Database Service (RDS). For these services, we decided to monitor a single selected variable: CPU utilization. Of course, you can opt for other measures by changing a single line in your code, as we'll see later. To get the full list of available statistics, visit the AWS website. So, where do we start?

Well, I have a scenario for you. Let's say you have a blogging platform that you're hosting on an amazon EC2 instance. All of your post content is saved in a RDS MySQL database. Quite a common architecture, isn't it? For simplicity, let's put aside questions related to high availability and fault tolerance. Now, you would like to get (near) live insights about how well you app is doing by retrieving critical metrics and displaying them in your admin portal. This way you won't have to connect to AWS each time you'd like to learn about your infrastructure performance.

Guess what? I have everything prepared for you to start from scratch, assuming you have a valid AWS account. Be aware that, as we'll be launching an EC2 and a RDS instance, the whole process will incur some costs (although these can be considered negligible if you just keep the instances running for an hour or two - $0.035/hour in the Frankfurt region as I'm checking prices right now). So if that doesn't make you look elsewhere, let's start with the set up. You can of course skip this part and go directly to "Retrieve data from CloudWatch" if you simply want to experiment the code with an existing EC2 instance. However, it needs to have an IAM role attached to it, as you can't assign a role to an existing instance.

Process

If you happen to have read some of my previous posts, you should know that I like to start working with a fresh new basic Laravel 5.2 blog. This post is no exception, sorry. So, I suggest that we set up an infrastructure for a simple blog. You can see how it looks like on the blog repo page. The entire process can be described in 3 simple steps:

  1. Set up AWS infrastructure
  2. Retrieve data from CloudWatch
  3. Display data as Google charts

So again, if you already have a running EC2 instance and don't necessarily want to install the blog, go ahead, skip the next part and simply implement the code from there on.

Set up AWS infrastructure

The architecture for the blog is pretty standard: A single EC2 instance linked to a database inside a custom VPC, with an internet gateway attached to it. Simply note that we define an IAM role to allow access to CloudWatch metrics from the EC2 instance. architecture


Launch a new stack

Open up your AWS console and head over to the CloudFormation Service. Click "Create Stack" and under "Specify an Amazon S3 template URL" type the following url: https://s3.eu-central-1.amazonaws.com/jeanquark/blog-home_cloudformation.json. Provide a stack name, a DB username and password (remember the latter two, because you'll need them later), as well as the key name of your KeyPair to SSH access the instance (in case you haven't created such a KeyPair so far, visit this page). It will take a few minutes for AWS to set up the new infrastructure.

Configure EC2 instance

Once completed, SSH access to the EC2 instance with the name "BlogHome" (you can click "Connect" on the top panel of your instances page to have the whole connection procedure explained). Next, there are a few configuration steps to complete. Go to the root of the blog-home project with your terminal (path is "/var/www/blog-home"). Install dependencies by typing sudo composer install. Then, edit the .env file to enter your DB_HOST (it is the endpoint of your DB instance, less :3306 at the end), as well as the DB_NAME and DB_PASSWORD that you previously chose.

your .env file


Finally, generate an application key with sudo php artisan key:generate. When this is done, visit the public DNS of your EC2 instance. An error message will appear, stating that the table "BlogHome.posts" cannot be found.

error message


If you made it this far, you're on the right track! There simply remains to create the database tables and fill them with data. To do that, type sudo php artisan migrate and sudo php artisan db:seed at the root of your blog.

Phew... there you have it: A complete blog platform with some interesting dummy posts;-) Visit the admin portal with the credentials provided in the login page.

blog-home


Hopefully, some people will be interested in reading your posts. As they do, your server will be soon saturated with requests and before it all goes nuts, it is wise to get informed on how it performs. To this end, you decide to monitor your infrastructure statistics.

Retrieve data from CloudWatch

We first need to make sure we can access the CloudWatch service from our web application. If you've launched the blog following the directives from the previous part, you should be good to go because an IAM role with proper rules has been attached to the EC2 instance (you can verify this by selecting the EC2 instance and check for the IAM role in the description pane). If you're experimenting from an already existing EC2 instance, make sure you attach the "CloudWatchReadOnlyAccess" policy to your IAM role. Naturally, another option would be to insert your AWS credentials as environment variables, but it's not considered best practice for security reasons (learn more about this problematichere).

Since we're interacting with AWS programmatically, we need to install the AWS SDK for PHP through composer. So just add this single line at the end of the require section of your composer file.

composer.json

{
    "require": {
        "aws/aws-sdk-php": "2.*"
    }
}

Next, run sudo composer update to install the new dependencies. You may also need to generate a new application key afterwards: sudo php artisan key:generate.

Now you can invoke the CloudWatch namespace on top of the admin controller

app/Http/Controllers/AdminController.php

use Aws\CloudWatch\CloudWatchClient;

We then create a new route to the statistics page under the already existing route group with middleware admin:

app/Http/routes.php

Route::group(['middleware' => 'admin'], function () {
    Route::get('admin/statistics', array('as' => 'admin.statistics', 'uses' => 'AdminController@statistics'));
});

And add a link to this new routing in the admin template, right at the end of the sidebar menu items unordered list:

resources/views/layoutBack.blade.php

<li class="{{ active_class(if_route(['admin.statistics'])) }}">
    <a href="{{ route('admin.statistics') }}"><i class="fa fa-fw fa-bar-chart"></i> Statistics</a>
</li>

We build a new method named statistics() in the already existing AdminController to retrieve relevant data. As mentioned in the introduction, we'll focus here on a single important metrics: the percentage of CPU utilization for the EC2 instance and RDS instance. Feel free to go with other measures as you like. We follow indications from the CloudWatch API for PHP to build the method. Here are the few modifications you need to apply to this code:

  1. Define a client by specifying the region where we launched our services. I happen to launch all my services in the Frankfurt region, so I specified "eu-central-1". Use your own region here (what is the code for my region?) [1]
  2. Retrieve your EC2 instance id and place it in the $cpu array. [2]
  3. Choose your metrics. You can simply keep CPU Utilization. [3]
  4. Place your database name in the $rds array. [4]

app/Http/Controllers/AdminController.php

public function statistics() {
    $client = CloudWatchClient::factory(array('region'  => 'eu-central-1', 'version' => 'latest')); // [1]
    $cpu = array( 
        array('Name' => 'InstanceId', 'Value' => 'i-d63fbf6b'), // [2]
    );
    $cpu1 = $client->getMetricStatistics(array(
        'Namespace' => 'AWS/EC2',
        'MetricName' => 'CPUUtilization', // [3]
        'Dimensions' => $cpu,
        'StartTime' => strtotime('-1 day'),
        'EndTime' => strtotime('now'),
        'Period' => 300,
        'Statistics' => array('Maximum', 'Minimum'),
    ));

    $rds = array(
        array('Name' => 'DBInstanceIdentifier', 'Value' => 'bloghomedb'), // [4]
    );
    $rds1 = $client->getMetricStatistics(array(
        'Namespace' => 'AWS/RDS',
        'MetricName' => 'CPUUtilization',
        'Dimensions' => $rds,
        'StartTime' => strtotime('-1 day'),
        'EndTime' => strtotime('now'),
        'Period' => 300,
        'Statistics' => array('Maximum', 'Minimum'),
    ));

    $cpu_util = $cpu1['Datapoints'];
    $rds_util = $rds1['Datapoints'];

    return View::make('admin.statistics')
        ->with('cpu', $cpu)
        ->with('rds', $rds)
        ->with('cpu_util', $cpu_util)
        ->with('rds_util', $rds_util);
}

Display data as Google charts

Create a new view called statistics.blade.php and have it placed in the admin folder. It will contain both the html as the placeholder for the Javascript rendered charts.

resources/views/admin/statistics.blade.php

@extends('layoutBack')

@section('css')

@stop

@section('content')
    <div class="well">
        <h3 class="">Amazon Web Services - Virtual servers performance</h3>
        <div class="row">
            <h4 style="margin-left: 15px;"> CPU Utilization Statistics for EC2 instance {{ $cpu[0]['Value'] }}</h4>
            <div class="col-md-12">
                <div id="chart_cpu_util"></div>
            </div>
        </div>
        <hr>
        <div class="row">
            <h4 style="margin-left: 15px;">DB Utilization Statistics for RDS instance {{ $rds[0]['Value'] }}</h4>
            <div class="col-md-12">
                <div id="chart_rds_util"></div>
            </div>
        </div>
    </div><!-- /.well -->
@stop

Right below the html, we retrieve data that was sent from the controller and convert it to javascript variables to be used in Google charts.

resources/views/statistics.blade.php

@section('scripts')
    <script>
        var cpu = <?php 
            echo json_encode($cpu_util);
        ?>;
        var rds = <?php 
            echo json_encode($rds_util);
        ?>;
    </script>
    // directly followed by the next code snippet

Next we create the charts with those two variables as inputs.

resources/views/statistics.blade.php

    <script type="text/javascript" src="https://www.gstatic.com/charts/loader.js"></script>
    <script>
        google.charts.load('current', {packages: ['corechart', 'line']});
        google.charts.setOnLoadCallback(cpu_util);
        google.charts.setOnLoadCallback(rds_util);


        // CPU Utilization graph 
        function cpu_util() {

            var data = new google.visualization.DataTable();
            data.addColumn('string', 'X');
            data.addColumn('number');


            function compare(a,b) {
              if (a.Timestamp < b.Timestamp)
                return -1;
              else if (a.Timestamp > b.Timestamp)
                return 1;
              else 
                return 0;
            }

            cpu.sort(compare);

            for(i = 0; i < cpu.length; i++) {
                var date = Date.parse(cpu[i].Timestamp);

                var t = new Date(date);
                var monthNames = [
                  "January", "February", "March",
                  "April", "May", "June", "July",
                  "August", "September", "October",
                  "November", "December"
                ];
                function addZero(i) {
                    if (i < 10) {
                        i = "0" + i;
                    }
                    return i;
                }
                var day = t.getDate();
                var monthIndex = t.getMonth();
                var year = t.getFullYear();
                var hour = addZero(t.getHours());
                var min = addZero(t.getMinutes());
                var date = String((day + ' ' + monthNames[monthIndex] + ' at ' + hour + ':' + min));

                data.addRow([date, parseFloat(cpu[i].Maximum)])
            };

            var options = {
                title: 'CPU Utilization (%)',
                legend: {position: 'none'},
                height: 400,
                hAxis: {
                    title: 'Last day span',
                    textStyle: {fontSize: 8}
                },
                vAxis: {
                    title: 'CPU Utilization (%)'
                }
            };

            var chart = new google.visualization.LineChart(document.getElementById('chart_cpu_util'));

            chart.draw(data, options);
        }

        // Database CPU Utilization graph
        function rds_util() {

            var data = new google.visualization.DataTable();
            data.addColumn('string', 'X');
            data.addColumn('number');


            function compare(a,b) {
              if (a.Timestamp < b.Timestamp)
                return -1;
              else if (a.Timestamp > b.Timestamp)
                return 1;
              else 
                return 0;
            }

            rds.sort(compare);

            for(i = 0; i < rds.length; i++) {
                var date = Date.parse(rds[i].Timestamp);

                var t = new Date(date);
                var monthNames = [
                  "January", "February", "March",
                  "April", "May", "June", "July",
                  "August", "September", "October",
                  "November", "December"
                ];
                function addZero(i) {
                    if (i < 10) {
                        i = "0" + i;
                    }
                    return i;
                }
                var day = t.getDate();
                var monthIndex = t.getMonth();
                var year = t.getFullYear();
                var hour = addZero(t.getHours());
                var min = addZero(t.getMinutes());
                var date = String((day + ' ' + monthNames[monthIndex] + ' at ' + hour + ':' + min));

                data.addRow([date, parseFloat(rds[i].Maximum)])
            };

            var options = {
                title: 'Database CPU Utilization (%)',
                legend: {position: 'none'},
                height: 400,
                hAxis: {
                    title: 'Last day span',
                    textStyle: {fontSize: 8}
                },
                vAxis: {
                    title: 'DB CPU Utilization (%)'
                }
            };

            var chart = new google.visualization.LineChart(document.getElementById('chart_rds_util'));

            chart.draw(data, options);
        }
    </script>
@stop

End Result

There you have it: Visit your site, sign in as an administrator and head over to the statistics page.

statistics page


To reverse the stack creation process, go to CloudFormation, select the relevant stack and click "erase". This will delete all of the components created for the blog, thus stopping any ongoing cost related to it.

Get the code
If you need to, you can download the final AdminController.php and statistics.blade.php files from the Github repo.

Blog Search

About

Hi there! My name is Jean-Marc Kleger and I'm a web developer. Welcome to my blog where a share some tips on how to deal with a selection of challenges encountered in my day-to-day coding workflow. Most articles are related to their own Github repo so that you can quickly experiment the code. Don't hesitate also to make use of the comment section at the end of each post to share your knowledge on the topic. Enjoy the visit!