How I cut the Axxos AWS bill by 38% in a quarter without touching a single line of product code. The audit, the tools, and the surprises along the way.
Our AWS bill at Axxos crept up the way these things always do: a few cents at a time, on services nobody remembered turning on. By the time someone forwarded me the invoice with a polite "is this normal?", we were spending almost three times what the workload actually justified.
I spent two weeks on what I now think of as the most boring high-leverage engineering work of my year. No new features. No refactors. Just a methodical walk through the bill, asking the same question of every line item: who is this for, and would we miss it if it stopped existing tomorrow? This post is what I found, what I cut, and what surprised me.
Step one: stop guessing, start tagging
The first hour of the audit was useless because I could not tell which resources belonged to which thing. Half of our EC2 instances had no tags. The other half had tags like env: prod and Name: api-2-final-OK. Nobody knew who turned on the SageMaker notebook from 2024.
Before I deleted a single thing, I forced tagging into the org. I wrote a small script that listed every untagged resource and dumped it to a CSV. We had a meeting where the team sat down and went row by row: ownership, environment, whether it was still needed. Boring. Necessary.
# List every EC2 instance missing the "owner" tag
aws ec2 describe-instances \
--query 'Reservations[].Instances[?!not_null(Tags[?Key==`owner`].Value)] | [].[InstanceId,State.Name,InstanceType,LaunchTime]' \
--output tableThen I enforced it. I added an SCP at the org level that denies ec2:RunInstances and rds:CreateDBInstance if the request does not include owner, env, and project tags. Nobody likes friction at create time, but unowned infra is how bills triple. New resources cannot be created today without telling AWS who is going to take care of them.
Stack técnica
AWS Cost Explorer
The starting point for any cost audit. Group by service, then by tag, then by usage type. It is rough around the edges but free and live.
AWS Compute Optimizer
Free recommendations on right-sizing EC2, EBS, RDS, and Lambda. Most of our easy wins came from following its suggestions verbatim.
Cost and Usage Report + Athena
For anything Cost Explorer cannot answer. CUR dumps every line item to S3, Athena queries it like SQL. Indispensable when you need to slice spend by tag combos.
Service Control Policies
Org-level guardrails. We use them to deny untagged resources, deny certain regions, and deny creation of resource types we have no reason to use.
The biggest line item was the one we forgot existed
Cost Explorer broke down our spend like this: 41% EC2, 19% RDS, 14% NAT Gateway, 11% S3, 8% data transfer, and a 7% miscellaneous bucket. I started with the largest categories and got ready to be clever about reserved instances and savings plans.
The actual biggest single dollar amount was a c5.4xlarge that had been running for 19 months with 4% average CPU. It had been spun up for a one-time data backfill, nobody turned it off, and it slowly faded into the wallpaper of our infrastructure. Killing it saved more in a year than every clever right-sizing recommendation combined.
This was the pattern that repeated. The wins were not in optimizing what we used. They were in deleting what we did not use. The rough breakdown of what I cut in the first week:
Three idle EC2 instances from old experiments. Two of them had been "temporary" for over a year. One RDS instance that was a copy of prod restored for an investigation in 2024 and never destroyed. Six EBS volumes attached to nothing, plus eleven EBS snapshots from instances that no longer existed. Two NAT Gateways in dev VPCs we had not deployed to in months. A SageMaker notebook running on a ml.m5.2xlarge that had been idle for 200+ days.
The total saved from that round alone was roughly 28% of the monthly bill. Zero engineering trade-offs.
NAT Gateways: the silent fee
NAT Gateways are the cost that bit me the hardest because the bill does not show you what is actually expensive about them. The hourly cost is small: about $33 a month per gateway. The data processing fee is $0.045 per GB. That second number is what gets you.
We had pods making outbound calls to S3 through the NAT Gateway because nobody had set up the S3 VPC endpoint. S3 traffic does not need to leave AWS: there is a free gateway endpoint that routes it without ever touching NAT. We were paying NAT processing fees on tens of TB a month of perfectly internal traffic.
# Create a Gateway VPC Endpoint for S3 (free, no processing fees)
aws ec2 create-vpc-endpoint \
--vpc-id vpc-0abcd1234 \
--service-name com.amazonaws.us-east-1.s3 \
--route-table-ids rtb-private-1 rtb-private-2 \
--vpc-endpoint-type GatewayTwo-line change in the route table. Cut our NAT data transfer spend by close to 60% the following month. The same trick applies to DynamoDB. For anything else (Secrets Manager, KMS, ECR), the Interface endpoints are not free, but at our usage levels they were cheaper than the NAT processing they replaced.
Storage classes: the slow win
S3 was 11% of the bill. Most of that was backups, build artifacts, and old logs sitting in Standard storage. Standard is the right class for hot data. For anything older than 30 days that we read maybe once a quarter, it is overpriced.
I wrote a lifecycle policy for each bucket. The shape is the same across all of them:
{
"Rules": [
{
"Id": "tier-down-old-objects",
"Status": "Enabled",
"Filter": { "Prefix": "" },
"Transitions": [
{ "Days": 30, "StorageClass": "STANDARD_IA" },
{ "Days": 90, "StorageClass": "GLACIER_IR" },
{ "Days": 365, "StorageClass": "DEEP_ARCHIVE" }
],
"Expiration": { "Days": 2555 }
}
]
}The trap with lifecycle policies is the transition fee. Moving an object between classes costs a few cents per thousand. If you have hundreds of millions of tiny objects, the transition itself can cost more than a year of cheaper storage. I learned this by transitioning a bucket of CloudFront access logs (1 KB each, 200 million objects) and getting a transition charge that erased six months of savings.
The fix is to filter lifecycle rules by object size. Objects under 128 KB are excluded from transitions: they stay in Standard because tiering them never pays back. AWS added the ObjectSizeGreaterThan filter for exactly this reason.
Impacto em números
Steady after three months of cleanup
EC2, RDS, EBS, NAT, SageMaker combined
After adding S3 + DynamoDB endpoints
The habits I built to keep the bill honest
Cost cleanup is not a project. It is a routine. The bill will creep back up the moment you stop watching. Three habits stuck after the initial cleanup:
Monthly bill review. First Monday of each month, I open Cost Explorer, group by service, and look at the deltas. Anything that grew more than 10% month-over-month gets investigated before I close the tab. Most of the time it is fine: traffic grew, so RDS grew. When it is not fine, I want to know in week one, not in the quarterly audit.
Idle detection cron. A scheduled Lambda that, once a week, lists all EC2 instances with CPU under 5% for the past 7 days, all RDS instances with under 5 connections/day, and all EBS volumes in available state. The output gets posted to a Slack channel. We do not auto-delete, I learned that one too, but visibility is enough.
Budget alerts that actually fire. Not the "you've used 80% of your budget" kind, which everyone mutes. The "this service grew by X% over baseline" kind. AWS Budgets has actions for anomaly detection now; I have one set on every service that ever appeared in our top 5. The signal-to-noise is high enough that I read them.
The bill is part of the product. It funds the next thing we build. Treating it as someone else's problem is how startups end up with a runway problem they could have prevented with two days of work. The most senior thing I have done in the last quarter is delete things, and I would do it again.