Effortless Disk Cleanup Automation For Ec2

Maintaining optimal performance and efficiency of your EC2 instances is crucial in today's cloud-centric world. Disk space management is a common challenge that can impact the performance of your applications and services if not handled proactively. Fortunately, AWS provides powerful tools like Systems Manager (SSM), Lambda, and EventBridge to automate routine maintenance tasks like disk cleanup.

In this blog, we will walk you through setting up an automated disk cleanup solution for your EC2 instances. By leveraging AWS SSM for command execution, Lambda for serverless processing, and EventBridge for event-driven automation, you can ensure your EC2 instances remain clutter-free and performant with minimal manual intervention. Whether you’re managing a small fleet of instances or a large-scale infrastructure, this guide will help you implement a scalable and efficient disk cleanup strategy.

Let’s dive into the details and get started on automating your disk cleanup tasks effortlessly!

Business Use Case: Automated Disk Cleanup on EC2 Instances

Scenario:

A large e-commerce company runs several web servers on Amazon EC2 instances to handle its online traffic. Over time, these servers accumulate temporary files, logs, and other unnecessary data, which can lead to decreased performance and wasted storage space. To ensure optimal performance and efficient use of resources, the company needs to regularly perform disk cleanup on these instances.

Solution:

The company decides to implement an automated disk cleanup process using AWS Lambda and AWS Systems Manager (SSM). The Lambda function will trigger an SSM Run Command that executes a predefined disk cleanup script on the specified EC2 instances. This setup allows the company to automate the maintenance task without manual intervention, ensuring that their servers are always running efficiently.

Aws Services Used in this solution:

AWS Systems Manager Run Command

Using Run Command, a capability of AWS Systems Manager, you can remotely and securely manage the configuration of your managed nodes. A managed node is any Amazon Elastic Compute Cloud (Amazon EC2) instance or non-EC2 machine in your hybrid and multicloud environment that has been configured for Systems Manage.

AWS System Manager Agent (SSM Agent)

AWS Systems Manager Agent (SSM Agent) is Amazon software that runs on Amazon Elastic Compute Cloud (Amazon EC2) instances, edge devices, on-premises servers, and virtual machines (VMs). SSM Agent makes it possible for the Systems Manager to update, manage, and configure these resources. The agent processes requests from the Systems Manager service in the AWS Cloud and then runs them as specified in the request

AWS Lambda

AWS Lambda is a compute service that runs your code in response to events and automatically manages the compute resources, making it the fastest way to turn an idea into a modern, production, serverless application.

Amazon EventBridge

EventBridge Pipes is a serverless integration resource that helps you build point-to-point integrations by providing a simpler and consistent way to integrate event producers with event consumers without writing additional code.

Steps to Implement the Solution:

Create an SSM Document for Disk Cleanup:

Choose Create Document at the top of the console. This allows you to create your own document, note that documents can be written in either JSON or YAML.

Document Structure

A. schemaVersion: the schema version to use.

• It is strongly recommended to use v 2.2. Automation documents also support versioning.

B. Description: Information you provide to describe the purpose of the document.

C. Parameters: The parameters the document accepts, both required and optional. To easily reference parameters you use often, use Parameter Store parameters in the following format: {{ssm:parameter_name}}. For more information, see AWS Systems Manager Parameter Store.

D. mainSteps: An object that can include multiple steps (plugins). Steps include one or more actions, such as Run command, a unique name of the action, and inputs (parameters) for those actions.

---
schemaVersion: '2.2'
description: "Document to clean up disk space if usage exceeds 80%"
assumeRole: "arn:aws:iam::account-id:role/YourSSMRole"
mainSteps:
  - name: CheckDiskSpaceAndCleanup
    action: 'aws:runShellScript'
    inputs:
      runCommand:
        - "echo 'Checking disk space...'"
        - "DISK_USAGE=$(df / | grep / | awk '{print $5}' | sed 's/%//')"
        - "if [ $DISK_USAGE -gt 80 ]; then"
        - "  echo 'Disk usage is above 80%. Performing cleanup...'"
        - "  sudo rm -rf /var/log/*.log"  # Example command, replace with your actual cleanup commands
        - "  sudo rm -rf /tmp/*"  # Example command, replace with your actual cleanup commands
        - "  echo 'Disk cleanup completed.'"
        - "else"
        - "  echo 'Disk usage is below 80%. No cleanup needed.'"
        - "fi"

Details of SSM document can be found on this Github repo:

GitHub - Ashishkasaudhan/Ec2-Disk-cleanup: This repo holds ssm document and lambda code to cleanup…

Install SSM agent:

Follow this link to install SSM agent on ec2 instance. We can also automate this part by using user-data script.

Manually installing and uninstalling SSM Agent on EC2 instances for Linux

Run commands on your Amazon EC2 instance at launch

Create a Lambda Function to Trigger the SSM Run Command:

The Lambda function will be scheduled to run at regular intervals (e.g., daily, weekly) using Amazon CloudWatch Events.

1. Configure Permissions:

• Ensure that the Lambda function has the necessary IAM permissions to invoke the SSM Run Command and access the specified EC2 instances.

2. Create lambda function with schedule actions:

Follow the details mentioned in the following document to create lambda with the event bridge schedule.

Tutorial: Create an EventBridge scheduled rule for AWS Lambda functions

• Create lambda function

Lambda code and details can be found on this Github repo:

GitHub - Ashishkasaudhan/Ec2-Disk-cleanup: This repo holds ssm document and lambda code to cleanup…

• Create a trigger for lambda by following event bridge. Provide names, descriptions and schedule expressions.

The final Trigger configuration will look like this:

Final Demo Time

Increase the disk space:

To increase disk space in Linux, we are going to use “fallocate” command. It's an amazing command to create large size of files in Linux.

$ fallocate -l 4096M file1.log — this will create a file with 4 GB of size.

Repeat the same command multiple times to make sure the disk space reaches beyond 90% utilization.

Now wait for lambda to perform the magic with SSM:

Now lambda will take care of disk cleanup as per your defined scheduled time. It will run the automation and the output of automation execution will look like this:

You can check this output or execution status in AWS Systems Manager — -> Run Command section in the console.

Benefits:

1. Automated Maintenance:

• Ensures disk cleanup is performed regularly without manual intervention.
• Reduces the risk of performance degradation due to accumulated junk files.

2. Scalability:

• Easily scales to manage disk cleanup across multiple EC2 instances.

3. Cost Efficiency:

• Frees up storage space, potentially reducing costs associated with additional storage.

4. Reliability:

• Uses AWS’s managed services, ensuring a reliable and highly available solution.

Conclusion

This solution provides an automated way to perform disk cleanup on EC2 instances based on specific tags. By leveraging AWS Lambda and SSM, you can efficiently manage your instances and ensure they have sufficient disk space. This approach is flexible and can be extended to include more complex maintenance tasks as needed.