Sunday, March 06, 2016

Monitor EC2 Memory Usage using AWS CloudWatch

CloudWatch an AWS service provides rich set of tools to monitor health and resource utilization of various services on AWS. These metrics can be used to trigger an alarm, send notifications, even trigger actions based on these alarms.

 

One main metric that is missing from default CloudWatch metrics is Memory utilization of EC2 instances. Since AWS does not have access to EC2 instance at the Operating System level, only CPU, Network utilization, IO and other metrics that can be monitored through Hypervisor layer are available by default in AWS Console.

 

AWS provides a set of scripts that utilize CloudWatch API to send Memory Utilization info and make use of CloudWatch features to monitor and take action if alarms are raised.

The process of sending these custom metrics are different for Linux and Windows Instances. Even the process of installing pre-requisites on different Linux distributions is slightly different.

 

In this post we will see the process of installing and configuring CloudWatch Custom Metrics for Redhat derived Linux Distros, mainly RHEL, Amazon Linux and CentOS along with Windows to monitor Memory Utilization of the EC2 instance

 

Updated (27-March-2016) – Added Commands for Ubuntu Linux

 

List of all commands run on Linux are also available at the end of the post for easy

reference.

 

Step –1

Create an IAM user with access key. We do not need password or console access to this user.

 

image_thumb[32]

 

Copy and keep the Access key handy as we would need this every time we configure custom metrics to be sent to CloudWatch

 

image_thumb[37]

 

Create and Attach an Inline Policy to the user with below actions.

cloudwatch:PutMetricData
cloudwatch:GetMetricStatistics
cloudwatch:ListMetrics
ec2:DescribeTags

 

Policy would like below. You may copy paste this for your use.

 

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["cloudwatch:PutMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "ec2:DescribeTags"],
      "Resource": "*"
    }
  ]
}

 

Go to IAM Users, click on the user just created and click on “click here” hyperlink in Inline Policies

 

image_thumb[39]

 

In next page select custom Policy and click select

 

image_thumb[40]

 

Give a name for Policy and paste policy in Policy Document and click on Apply Policy

 

image_thumb[41]

 

Now the user would have policy attached to send custom metrics using API to CloudWatch

 

image_thumb[42]

 

You can verify the policy using Show Policy Actions in IAM console for the user.

 

Step –2

Installing pre-requisites, all the following commands are to be run with root privilege, you can use sudo as well, except inside CPAN shell.

 

CEntOS 6.x -

 

Command -1

yum -y install perl-DateTime perl-Sys-Syslog openssl openssl-devel cpan unzip

 

Command -2

perl -MCPAN -e shell

 

If this command prompts for input, accept defaults

 

image_thumb[26]

 

The above command when complete would show CPAN prompt

 

image_thumb[28]

 

Run below command at CPAN prompt

 

Command -3

install Bundle::LWP6 LWP

 

This would prompt for input multiple times, accept all default values or answer yes. This would take some time to complete, so, have patience.

 

image_thumb[31]

 

One the command finishes successfully, the CPAN prompt number would change, type quit and hit enter to exit to shell prompt

 

image_thumb[45]

 

Run below command at shell prompt and accept all defaults or answer yes

 

Command -4

perl -MCPAN -e 'install LWP::Protocol::https'

 

Once this finishes successfully, you will have shell prompt waiting for next steps.

 

Step –3

Downloading Scripts

 

Run below commands to download, extract, delete the downloaded zip file and move scripts to /opt folder, and open awscreds.conf file to configure credentials.

 

wget http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip

unzip CloudWatchMonitoringScripts-1.2.1.zip

rm -f CloudWatchMonitoringScripts-1.2.1.zip

mv aws-scripts-mon /opt/

cd /opt/aws-scripts-mon/

cp awscreds.template awscreds.conf

vi awscreds.conf

 

image_thumb[50]

 

I have used vi for text editing, but you can use text editor of your choice.

 

Once awscreds.conf file is open in the text editor, key in Accesskey ID and Secret Key that we got when we created IAM user in Step-1

 

image_thumb[54]

 

Save and exit text editor.

 

Now run below command to check if everything is OK.

 

/opt/aws-scripts-mon/mon-put-instance-data.pl --mem-util --verify –verbose

 

image_thumb[57]

 

If you see the output as below, everything is properly configured and we can proceed to configure Cron to send Memory Metrics to CloudWatch every 5 minutes.

 

Verification completed successfully. No actual metrics sent to CloudWatch.

 

Step –4

Configure Crontab

 

Type crontab –e at the shell prompt and append below line to the end of the file.

 

*/5 * * * * /opt/aws-scripts-mon/mon-put-instance-data.pl --mem-util --from-cron

 

Note that the file would be empty if there are no existing cron jobs

 

image_thumb[60]

 

Now save and exit.

 

After 10 or 15 minutes you would see Memory utilization of this instance in CloudWatch console.

 

image_thumb[62]

Windows Instance

On your Windows Server EC2 Instance, download and install latest EC2Config utility

 

Goto - http://aws.amazon.com/developertools/5562082477397515

 

and click on download to get the latest Amazon Windows EC2Config Service

 

image_thumb[65]

 

Extract the downloaded zip file.

 

image_thumb[67]

 

Get into the extracted folder and double click to run the EXE

 

CustomCloudWatchMetrics-03_thumb[1]

 

Follow onscreen instructions to complete the installation.

 

image_thumb[69]

 

CustomCloudWatchMetrics-05_thumb[2]

 

CustomCloudWatchMetrics-06_thumb[2]

 

Now open the AWS.EC2.Windows.CloudWatch.json file in Notepad or Notepad++ with administrative privilege

 

Complete path of the file is

JSON Settings File

C:\Program Files\Amazon\Ec2ConfigService\Settings\AWS.EC2.Windows.CloudWatch.json

 

image_thumb[71]

 

In AWS.EC2.Windows.CloudWatch.json file look for the section “Id”: “PerformanceCounter”

 

image_thumb[74]

 

Modify the Parameters “DimensionName” and “DimensionValue” as shown in the screenshot with values “InstanceId” and “{instance_id}” respectively.

 

image_thumb[77]

 

Now go to section “Id”: “CloudWatch”

 

image_thumb[79]

 

Key in IAM user AccessKey and SecretKey at respective place holders as shown below

 

image_thumb[82]

 

Change the Region to the AWS Region where the EC2 Instance is running. In my case it was Singapore – ap-southeast-1

 

Change NameSpace to “System/Windows” . By default this would be “Windows/Default”

Now go to Flows section of the file, Generally this would be at the End of File.

 

CustomCloudWatchMetrics-12_thumb[1]

 

Replace "(ApplicationEventLog,SystemEventLog),CloudWatchLogs" with

"PerformanceCounter,CloudWatch"

 

image_thumb[84]

 

Save and close the file.

 

Now go to start and look for EC2ConfigService Settings.

 

Windows 2008 R2

image_thumb[86]

 

Windows 2012 R2

 

image_thumb[88]

 

Check Enable CloudWatch Logs integration

 

image_thumb[90]

 

Click on Apply and OK to save and close

 

Go to services and restart Ec2Config Service

 

image_thumb[93]

 

After 10 to 15 minutes, you will be able to see the Memory Metrics in CloudWatch Console with default polling interval of 5 minutes.

 

image_thumb[95]

 

You can monitor below log file for any errors.

 

Log File

C:\Program Files\Amazon\Ec2ConfigService\Logs\Ec2ConfigLog.txt

 

Commands for Instance Reference

These steps are almost similar on RHEL (RedHat Enterprise Linux) and Amazon Linux only slight difference with pre-requisite installation.

 

CEntOS 6.x Linux

yum -y install perl-DateTime perl-Sys-Syslog openssl openssl-devel cpan unzip

perl -MCPAN -e shell

install Bundle::LWP6 LWP

perl -MCPAN -e 'install LWP::Protocol::https'

 

Amazon Linux

yum -y install perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https cpan

perl -MCPAN -e shell

install Bundle::LWP6 LWP

 

RedHat Linux

yum -y install install perl-DateTime perl-Sys-Syslog zip unzip perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA

perl -MCPAN -e shell

install Bundle::LWP6 LWP

 

Ubuntu Linux

apt-get update
apt-get install unzip
apt-get install libwww-perl libdatetime-perl

 

Downloading and Configuring Scripts

wget http://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.1.zip

unzip CloudWatchMonitoringScripts-1.2.1.zip

rm -f CloudWatchMonitoringScripts-1.2.1.zip

mv aws-scripts-mon /opt/

cd /opt/aws-scripts-mon/

cp awscreds.template awscreds.conf

vi awscreds.conf

crontab -e

*/5 * * * * /opt/aws-scripts-mon/mon-put-instance-data.pl --mem-util --from-cron

 

Error-

If you ever get timezone.pm error, run below command.

 

yum install perl-DateTime-TimeZone

 

References and Download Links -

EC2 Linux Instance

http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/mon-scripts.html

Download - http://aws.amazon.com/code/8720044071969977

 

EC2 Windows Instance

https://blogs.aws.amazon.com/application-management/post/Tx1KG4IKXZ94QFK/Using-CloudWatch-Logs-with-Amazon-EC2-Running-Microsoft-Windows-Server

Download - http://aws.amazon.com/developertools/5562082477397515

 

Other References

http://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/ec2-configuration-cwl.html
http://www.tothenew.com/blog/setup-custom-cloudwatch-metrics-on-ec2-instance/
https://forums.aws.amazon.com/message.jspa?messageID=592245

10 comments:

  1. Now EC2Config version has been changed in Win Server 2012 R2 and new version doesn't show 'Enable Cloudwatch logs integration" option in EC2config properties , due to which memory metric is not getting reflected to cloudwatch console.
    Please help how i can get it worked.

    Thanks in advance...
    Shubh

    ReplyDelete
  2. EC2Config does not come with Windows Server 2016. It says to use the EC2 Launch Powershell. How does one use this to monitor the memory?

    ReplyDelete
  3. I have the same problem like Shubham and Craig.. Any news about it?

    ReplyDelete
  4. I am adding memory as well as disk metrics on windows server 2012 R2, but for disk metrics, in the Ec2ConfigLog.txt i am getting error -

    [Error] Host not configured, no data will be uploaded to CloudWatch

    ReplyDelete
  5. nice tutorial.
    http://amazonwebservicesforum.com

    ReplyDelete
  6. A Detour: From security perspective,
    In Step 1, rather than creating a user, create a role with policy and attach it to ec2 instance.
    Then, in Step 3, no need to add credentials to awscreds.conf

    ReplyDelete
  7. The configuration for monitoring windows machines results with "Available Memory". Is it possible to get "Memory Utilized"?

    ReplyDelete
  8. Nice Tutorial, Thanks for sharing the steps

    ReplyDelete
  9. "Enable Cloudwatch logs integration"not showing up
    Logs as below:

    I am getting the following warning in the log file:

    Warning: Unable to Publish to WMI. | System.Management.Instrumentation.WmiProviderInstallationException: Exception of type 'System.Management.Instrumentation.WMIInfraException' was thrown. at System.Management.Instrumentation.InstrumentationManager.Publish(Object value) at Ec2Config.LegacyConfiguration.LegacyConfigurator.PublishWmiInstance()

    Failed to fetch instance metadata http://169.254.169.254/latest/user-data with exception The remote server returned an error: (404) Not Found. Failed to get metadata/user-data The remote server returned an error: (404) Not Found.

    ReplyDelete

Share what you feel