AWS Cost-Cutting Mistakes I Made, and What I Should Have Done From the Start

Last March, around 4 PM on a Friday, our team's engineering manager dropped a message in Slack. "Have you seen this month's AWS bill? Take a look." I opened the Billing Dashboard, saw the numbers, and nearly slid out of my chair. Roughly a 40% increase month over month. To be precise, thousands of extra dollars had been draining from the account allocated to our team's project in just one month. A significant portion of the cause was a few GPU instances I had spun up two months earlier and forgotten about.

From that day on, I unofficially became the team's "cloud cost auditor." Not by choice. When I, a data scientist, said I'd look into infrastructure costs, the manager just said, "You're good with numbers." Over the course of about four months, I managed to bring the bill down, but I made quite a few foolish mistakes along the way. Looking back, there were stretches where trying to save money actually cost us more.

I Cut Whatever Looked Expensive First

The first thing I did was open Cost Explorer and sort by highest cost. EC2 was the biggest line item, so I figured I should start there. Sounds reasonable, but this is where the mistakes began. I scanned the instance list and posted "Who's using this?" in Slack, then shut down four instances that nobody claimed after two days.

Monday morning, a colleague's cron job was dead. One of those instances had been running a data pipeline every Monday at dawn. Asking "Whose is this?" in Slack was the wrong approach entirely. Instead of asking people, I should have first checked the tags on the instance, the associated security groups, and the CloudTrail logs. Who created the instance, who last SSH'd into it: it's all there in the logs.

What I should have done was check at least two weeks' worth of CloudWatch metrics before shutting anything off. Even if CPU utilization is near 0%, a pattern of network traffic spiking at certain times means something is running. I jumped too quickly to the equation "low utilization = unnecessary."

Buying Reserved Instances Too Hastily

After confirming that EC2 costs were significant, I learned that buying Reserved Instances could save money. The math showed a substantial discount compared to on-demand with a one-year reservation, so I went ahead and reserved based on the instance types our team used the most. The problem was that I determined those "most-used instance types" from only three weeks of data.

Our team's workload fluctuates considerably from quarter to quarter. Instance configurations change depending on model training cycles, data collection campaign timing, and events like internal hackathons. A three-week snapshot was an absurdly short window for making an annual reservation decision. Two months later, we needed to switch instance types, and the reserved instances sat there unused.

If I could go back, I would have looked at least three months (ideally six months) of usage patterns before deciding whether to reserve. Instead of reserving everything, I would have reserved only enough for the baseline load and covered the rest with Spot Instances or on-demand in a blended strategy. AWS provides its own reservation recommendation tool, and at the time I didn't even know it existed.

Ignoring S3 Costs Because They Seemed Small

Compared to EC2, S3 costs looked negligible. They made up a small share of the total bill, so I brushed them off with "I'll deal with it later." That was the third mistake.

When I eventually took a closer look, our S3 buckets held terabytes of intermediate output files from a project two years prior: model checkpoints, preprocessing artifacts, logs that were written once and never read again. The storage cost itself was a problem, but the real issue was elsewhere. Some files had unnecessary Cross-Region Replication configured. Someone had turned it on while testing a disaster recovery scenario and never turned it off. Transfer costs for the replicated data had been quietly accumulating every month.

The reason S3 costs "felt" small was that I only looked at storage costs and never separately checked data transfer and API request costs. Had I broken things down by Usage Type rather than by service in Cost Explorer, the transfer costs would have jumped out immediately.

Once I started cleaning up, I set up an S3 Lifecycle Policy to automatically transition objects not accessed for over 90 days to Glacier, and delete them after 180 days. I should have done this from the start. After the cleanup, S3-related costs dropped by about 60% on a monthly basis. The absolute amount wasn't as large as EC2, but it was more than enough to regret dismissing as a "small cost."

I Should Have Understood the Structure Before Relying on Tools

The biggest lesson from those four months was that I had approached cost optimization as a question of "which tool to use." I thought I could solve it by bringing in a third-party cost analysis tool, building dashboards, setting up alerts, and automating reports. I actually connected a free-tier tool and started getting weekly reports, but all those reports told me was "You spent $X more on this service than last week." Only someone who understood the context of our workloads could judge why spending went up and whether the increase was justified or wasteful.

Before adopting any tool, here's what I should have done first:

• Categorize the workloads our team runs on AWS by type and map which resources each one uses.
• Determine whether each workload's usage pattern is steady-state, burst, or scheduled.
• Get the tagging policy in order. Tags need to be consistently applied by project, team, and environment (dev/staging/prod) for cost breakdowns to be meaningful.

Without this groundwork, attaching a tool gives you a pretty dashboard that's useless for decision-making.

Our team's tagging situation was dismal. More than half the instances had no project tag, and the ones that did were inconsistent (project-alpha, proj_alpha, Alpha). It took two full days just to clean up the tags, and honestly, those two days were the most cost-effective work I did in the entire four months. Once the tags were consistent, I could see which project was spending how much in five minutes, and I immediately found that already-completed projects were still consuming resources.

Don't Touch Infrastructure on a Friday Afternoon

This one isn't exclusive to cloud costs, but it's worth writing down. I first shut down those instances on a Friday afternoon, and the problem wasn't discovered until Monday morning. If you make changes right before a window when nobody is monitoring all weekend, recovery takes longer when something breaks. Cost-cutting can wait until Tuesday or Wednesday morning.

Now, I always start cost-related changes between Tuesday and Thursday mornings, and I monitor the relevant metrics for at least 24 hours after any change. This isn't some grand principle. It's just common sense. But common sense is the first thing to go when you feel a sense of urgency.

After four months, our team's monthly AWS costs were down roughly 35% from when I first started the audit. Not a bad result, but if you subtract the unnecessary incidents I caused (the pipeline outage, the stranded reserved instances, the time wasted on messy tags) I could have achieved the same result in half the time. If I ever do something like this again, I'll spend the first two weeks just observing without turning anything off. Resisting the urge to shut things down is the hardest first step.