AWS Monitoring with amazon-cloudwatch-agent | Daniel Tinizaray

Why amazon-cloudwatch-agent

Out of the box, CloudWatch delivers basic EC2 metrics: average CPU usage, network traffic, status checks. Enough to know if an instance is alive, not enough to understand how it's behaving.

amazon-cloudwatch-agent is AWS's official collector that runs inside the instance and sends real operating system metrics: used/free memory, swap, per-partition disk usage, per-process CPU, and custom logs. Without it, you're monitoring blind to everything happening outside the hypervisor.

The difference is simple: hypervisor metrics tell you the instance is powered on. OS-level metrics tell you why it crashed.

1. Installation on EC2 (Amazon Linux 2023 / Ubuntu)

Installation varies by OS, but the flow is identical:

Amazon Linux 2023

sudo yum install amazon-cloudwatch-agent -y

Ubuntu 22.04 / 24.04

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb
sudo dpkg -i amazon-cloudwatch-agent.deb

Then configure it using an interactive wizard or —recommended— with a JSON file:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

The wizard generates the config at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json. I prefer writing it manually from scratch —more control, fewer prompts.

2. Essential Configuration

The config file has two main sections: metrics (OS metrics) and logs (log files).

Here's a base config covering 90% of use cases:

{
  "agent": {
    "region": "us-east-1",
    "run_as_user": "root",
    "collect_metrics_interval": "60s",
    "omit_hostname": false
  },
  "metrics": {
    "namespace": "CWAgent",
    "metrics_collected": {
      "cpu": {
        "measurement": [
          "cpu_usage_idle",
          "cpu_usage_user",
          "cpu_usage_system",
          "cpu_usage_iowait"
        ],
        "metrics_collection_interval": 60,
        "totalcpu": true
      },
      "mem": {
        "measurement": [
          "mem_used_percent",
          "mem_available_percent"
        ],
        "metrics_collection_interval": 60
      },
      "disk": {
        "measurement": [
          "disk_used_percent",
          "disk_inodes_free"
        ],
        "metrics_collection_interval": 120,
        "resources": ["/", "/var", "/data"]
      },
      "swap": {
        "measurement": ["swap_used_percent"],
        "metrics_collection_interval": 120
      },
      "netstat": {
        "measurement": ["tcp_established", "tcp_time_wait"],
        "metrics_collection_interval": 120
      }
    },
    "append_dimensions": {
      "InstanceId": "${aws:InstanceId}",
      "AutoScalingGroupName": "${aws:AutoScalingGroupName}",
      "ImageId": "${aws:ImageId}"
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/syslog",
            "log_group_name": "syslog",
            "log_stream_name": "{instance_id}",
            "timestamp_format": "%b %-d %H:%M:%S"
          }
        ]
      }
    }
  }
}

Key takeaways from this configuration:

• collect_metrics_interval: 60s — Enough granularity without driving up costs.
• append_dimensions — Every metric gets tagged with InstanceId and ASG, enabling dashboard filtering.
• disk.resources — Specifying concrete partitions avoids collecting tmpfs and irrelevant bind mounts.
• iowait — The most ignored metric and the first one to spike when storage is the bottleneck.

To apply the configuration:

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config -m ec2 \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json \
  -s

The -s flag restarts the service with the new config. Without -s, changes won't apply until the next restart.

3. Per-Process Metrics with procstat

When you need to know not just how much CPU the instance is using, but which process is consuming it, the procstat plugin is the answer:

"procstat": [
  {
    "pattern": "nginx",
    "measurement": ["cpu_usage", "memory_rss", "pid_count"],
    "metrics_collection_interval": 60
  },
  {
    "pattern": "python",
    "measurement": ["cpu_usage", "memory_rss"],
    "metrics_collection_interval": 60
  }
]

pattern accepts a regex against the process name. If the process is nginx: worker process, "nginx" as pattern captures all workers. The pid_count metric tells you how many processes matched —useful to verify the expected Nginx or Gunicorn worker count.

If in production you see CPU at 95% but don't know whether it's Nginx, the database, or a zombie process, procstat is the difference between a 5-minute debug and a 2-hour one.

4. Custom Logs → CloudWatch Logs

The agent also ships logs to CloudWatch Logs. The example config includes syslog, but you can add any file:

"logs": {
  "logs_collected": {
    "files": {
      "collect_list": [
        {
          "file_path": "/var/log/nginx/access.log",
          "log_group_name": "nginx-access",
          "log_stream_name": "{instance_id}",
          "timestamp_format": "%d/%b/%Y:%H:%M:%S %z",
          "timezone": "UTC"
        },
        {
          "file_path": "/var/log/nginx/error.log",
          "log_group_name": "nginx-error",
          "log_stream_name": "{instance_id}",
          "timestamp_format": "%Y/%m/%d %H:%M:%S",
          "timezone": "UTC"
        }
      ]
    }
  }
}

Once in CloudWatch Logs, you can create metric filters to extract patterns (e.g., HTTP 5xx above threshold) without modifying a single line of application code. These run over the log stream in real time and emit metrics to CloudWatch.

5. Costs: How Not to Break the Bank

CloudWatch's Achilles' heel is the cost of custom metrics. Each unique combination of metric name + dimension is billed separately. With append_dimensions, every instance generates its own combination.

Practical cost rules:

• Use the default CWAgent namespace. Don't create per-project namespaces unless strictly necessary.
• Don't collect everything. Out of 60+ metrics the agent can collect, choose 10-15 you actually need.
• 60s intervals or higher. 60s costs 1 metric unit per minute. 10s costs 6. The factor is linear.
• Group dimensions wisely. InstanceId is necessary, but avoid volatile dimensions like hostname or ip that multiply combinations.
• Summarize in dashboards. Use AVG, SUM, p95 in the dashboard on raw metrics —don't create pre-aggregated metrics.
• Clean up terminated instances. Dead instance metrics still incur costs if they have associated alarms. Review periodically.

A typical instance with 15 metrics at 60s intervals costs ~$3-5 USD/month in custom metrics. Compared to third-party solutions, it's economical —but without control, it can scale to $50+/month per instance.

6. Integration with Alarms and Dashboards

Once the agent sends data, metrics appear automatically in CloudWatch. You can create alarms directly:

• Memory > 90% for 5 minutes → SNS notification to Slack/Email.
• Disk used > 85% on root → scale EBS volume or auto-cleanup.
• iowait > 20% sustained → EBS volume is likely IOPS-throttled. Check gp2 vs gp3.
• nginx pid_count == 0 → the process isn't running. Immediate alarm.

In the dashboard, InstanceId and AutoScalingGroupName dimensions let you filter by server or group. A well-designed dashboard answers in seconds: "which of the 20 instances has high memory?" without checking one by one.

7. amazon-cloudwatch-agent vs Third-Party Providers

Why use the native AWS agent over Datadog, New Relic, or Grafana Agent?

• Cost: CloudWatch Agent has no additional license. You only pay for metric storage and API calls. No third-party is cheaper at scale.
• Latency: Metrics travel within AWS's network without going to the internet. On instances with sensitive data, you avoid exfiltrating metrics to an external SaaS.
• Native integration: CloudWatch Dashboards, Alarms, Logs Insights, and Lambda are AWS's natural ecosystem. No exporters or bridges needed.
• Fewer moving parts: One agent, one config file. No DaemonSets, no sidecars, no complex OpenTelemetry pipelines.

The downside: CloudWatch lacks the smart alerting ecosystem and flexible dashboards of Grafana. It also has no native SLI/SLO tracking. For small-to-medium teams (< 50 instances), it's more than enough. For complex orchestration, Grafana + Prometheus remains the standard —but you can use CloudWatch + Grafana via AWS Managed Grafana without giving up the best of both.

Conclusion

amazon-cloudwatch-agent is the missing piece between hypervisor metrics and what's actually happening inside the server. Without it, you're flying blind: you know the instance is on, but not if it's running out of memory, if the disk is saturated, or if Nginx crashed 20 minutes ago.

The configuration presented here covers 90% of production monitoring scenarios for teams running web applications, APIs, workers, or batch processing on EC2. You don't need an ELK stack, a Prometheus cluster, or a Datadog contract. You need a 10 MB agent, a 50-line JSON file, and knowing which metrics matter.

Everything else is noise.