Automating AWS Resource Tracking with Shell Scripting

Introduction

In the dynamic realm of DevOps, effective resource management is paramount for maintaining operational efficiency and cost control. On Day 11 of the 100 Days of DevOps Challenge, we dive into a practical AWS DevOps project, leveraging shell scripting and AWS CLI to automate resource tracking.

Project Highlights: AWS Resource Management

Cloud platforms like AWS offer immense scalability and flexibility, but without proper oversight, they can lead to unnecessary expenses. This project focuses on building a shell script to monitor and report AWS resource usage, helping organizations optimize their cloud expenditure.

Why Automate Resource Tracking?

Cost Control: Avoid expenses from unused or underutilized resources like idle EC2 instances or unattached EBS volumes.
Operational Efficiency: Automated scripts save time compared to manual tracking, reducing the risk of oversight.
Timely Reporting: Integration with cron jobs ensures reports are generated and shared regularly without manual intervention.

Step-by-Step Implementation

1. Setting Up AWS CLI

Before diving into the script, configure the AWS CLI:

aws configure

Provide your AWS access key, secret key, default region, and output format.

2. Writing the Shell Script

Here’s an outline of the script:

Shebang: Define the shell environment (#!/bin/bash).
Commands: Use AWS CLI commands like aws ec2 describe-instances to gather data.
Output Formatting: Tools like jq can refine JSON output for better readability.

Example:

#!/bin/bash

echo "Fetching EC2 Instances..."
aws ec2 describe-instances --query 'Reservations[*].Instances[*].[InstanceId,State.Name]' --output table

echo "Listing S3 Buckets..."
aws s3 ls

echo "Retrieving Lambda Functions..."
aws lambda list-functions --query 'Functions[*].[FunctionName]' --output table

echo "Checking IAM Users..."
aws iam list-users --query 'Users[*].[UserName]' --output table

3. Adding Automation with Cron Jobs

Schedule the script using cron:

crontab -e

Add a line to run the script daily at 6 PM:

0 18 * * * /path/to/script.sh >> /path/to/logfile.log 2>&1

4. Testing and Enhancing the Script

Run the script manually to ensure functionality.
Redirect output to a file for review and sharing.

Add print statements for clarity, such as:

  echo "Report generated successfully!"

Final Thoughts

By automating AWS resource tracking, DevOps engineers can proactively manage resources, reduce unnecessary costs, and streamline reporting processes. This project not only reinforces essential shell scripting skills but also highlights the importance of automation in modern cloud environments.