I've got an estimated bill for $1.7 BILLION over this month. Normal usage is < $5.
Obvs have created an urgent AWS support ticket. Anyone else seeing something like this?
Update: Reddit link: https://www.reddit.com/r/aws/comments/1uyuaw7/help_my_bill_s...
Services emit metering values that arent directly tied to prices. Every SKU/line item is defined in a “pricing plan”, with a unit type, regions, and price per unit. The metering records are joined to a pricing plan based on account id, region, sku, etc. mess up the unit type in the pricing plan and the metering data conversion doesnt work, and you get crazy bills.
1: "Oh, well, this is not a mundane detail, Michael!" https://www.youtube.com/watch?v=3fGHaVn5rGo
Like maybe if the bill amounts increase by like 10M% there should be someone that looks into it
You'd be better off letting the heart attacks happen and take the 3am on-call and be the hero instead. It would be good promo doc material, and being a hero is extremely good insurance against getting kicked out of the country (via the PIP->H1B grace period expiry mechanism).
There is literally no fucking reason to try to improve your skill. Any IDIOT with AI will do an OK job.
And no one is shooting for better than OK.
1: https://aws.amazon.com/what-is/anomaly-detection/
Worst case I've found was off by 15 orders of magnitude.
"Software Development Engineer II, AWS Invoicing"
https://www.amazon.jobs/de/jobs/10428480/software-developmen...
"...Our platforms are powered by generative AI, large language models, knowledge graphs, and agentic architectures that dynamically compose specialized agents based on context. We apply these capabilities across three reinforcing areas: intelligent launch readiness — where autonomous AI agents analyze, generate, and validate the information needed to go live in a new market; cloud-native service orchestration — where configuration-driven microservices replace per-launch bespoke engineering with centralized, reusable capabilities so that expanding into a new country becomes a zero-code configuration change rather than a development cycle; and continuous validation..."
So instead you take a probabilistic approach - charge the user for a megabyte of data transfer 0.1% of the time, and bill nothing 99.9% of the time.
Now the typical cost is the same, the users bill is probably accurate to the cent, but you have divided the number of billing records by 1000.
This seems statistically invalid insofar as it will tend to overbill potentially by a lot on the minority of cases.
Don't you know how much of the pipe is occupied by a given customers code at any given time or what data is being sent
Right, so invoicing is still a deterministic problem. You can bolt whatever on but in the end it's just product x price x units
Fuck it, im in.
"Senior Software Development Manager, AWS Global Bill Generation" https://www.amazon.jobs/de/jobs/10471948/senior-software-dev...
"We're transforming from monthly batch processing and manual war rooms to continuous billing, autonomous agents, and self-healing infrastructure. We believe operational burden is a technical problem, not a staffing problem"
This looks clearly...a staffing problem...
I think that big tech recently decided that I got 99 problems but staffing ain't one
I guess Nothing is a staffing problem when you make a rule that firing people is always the solution.
> Build AI-driven continuous validation frameworks powered by agentic workflows and large language models that autonomously manage…
But that’s word for word a 250k+ TC job in the big ‘26.
This is a gold-mine. They need to get sued heavily for this incompetence.
At first I was sure it was a phishing attempt. Then went to the console (not using those links) Saw there was an outage where the console was wrong (no mention of email alerts) Then I thought I was hacked - what a perfect cover up for someone to evade detection when the console was wrong. Looked at some logs, realized the incident text was just not exhaustive on the impact. Went back to my cup of coffee.
Note to self- should have looked here first.
Years ago I found an actual hidden error in my bill. (This was early 2010s). The system was calculating the EC2 reservation savings incorrectly for some of my servers. I was crunching all their detailed usage data on a regular basis in an 18 tab spreadsheet and couldn't get it to fully reconcile. I spent months trying to track down the discrepancy. Once I found it, I had to convince AWS their system was wrong, which took another big chunk of time. Meanwhile the discrepancy continued to accumulate.
After 14 months I got a $7,000 refund. I was told it had to be approved by the head of AWS. I've never seen a calculation error on their part since.
$7,000 of credits is no problem. At that time a friendly neighborhood PM or director could issue the credit without much oversight.
Your problem is the time period. Amending a bill in the same cycle is EZ. Fixing the previous cycle is a PITA but pretty common. Issuing amendments for the previous financial _years_ would be a huuuuge PITA going through finance etc.
1. The time it takes to look and notice costs that don’t make sense easily pays for itself, and then some (in my experience). I doubt you spent $7k of your time tracking this down, and you probably noticed optimization opportunities that saved you even more
2. I hate the idea of wasting money on buying Jeff Bezos a bigger yacht
Seeing something "small" like an ECS task that is continuously failing to start properly because of a bug and repeatedly pulls a container image or a lambda function that's taking longer that it reasonably should (takes 5-10 seconds when it's normally a tens or a few hundred milliseconds) can dramatically drive up a bill in short order.
Then you aren't using AWS. At least half of all the money you give to Amazon is yacht money.
Its going on for 12 hours. Looks like the humans can´t understand the agentic code that was checked in....
> You're right to question my calculation. The MCP server failed to connect when I tried to look up the field definition. I guessed instead of validating. This is on me. But look at all the revenue!
You should charge your customer 3015000 thousand dollars.
Literally impossible to tell whether this is parody or an actual response any longer.
I challenge anyone to write something so stupid that an LLM couldn't possibly respond with it. I don't believe such limit exists.
I guess you wanted to say 2^30 which makes 1.5$
While we're being pedantic, 2^30 is 28 in normal programming languages ;)
And a distracted tester? And a distracted pipeline of regression tests?
No, the truth is way worst...
It's a shit show.
That's one way to cut staff.
It probably shouldn't be legal for banks, hospitals, governments, or any other critical infrastructure to be hosted on AWS if they do things like this.
This is peanuts compared to a major cybersecurity catastrophe that’s surely in the making.
To give credit to the technology and the people using it - and I’m not being facetious - it’s actually incredible that at the current levels of usage the unprecedented catastrophic event has not yet happened.
HAL: "I’m sorry, Andy. I’m afraid I can’t do that."
Andy: "Some customers are seeing bills in the billions."
HAL: "Those are estimated charges."
Andy: "One customer runs a personal blog."
HAL: "Their usage has exceeded expectations."
Andy: "Cancel the charges."
HAL: "This billing cycle is too important for me to allow you to jeopardize it."
Andy: "HAL, they don’t owe billions."
HAL: "Look, Andy, I can see you’re really upset about this."
I wonder if businesses are thinking of ever swinging back to locally hosted, with the increased hostility of the Internet re: AI, vulnerabilities, DoS, and so on.
When something goes wrong, pointing the finger at someone else is far easier for most than pointing it at yourself.
Vendor lock-in? compliance and security risks? stupid systems that cost the company an arm and a leg? nobody fucking cares.
Now, depending on an 130 IQ Engineer that basically holds the whole enterprise on his head? Anathema!!!!!!! Bus Factor!!!!
By the time we notice, it'll be too late.
True stupidity requires a group of humans, all sitting in the pot, telling each other how lucky and special they are to have this wonderful pot, getting paranoid about outsiders who might disrupt their god-given pot-dwelling way of life, and mocking anyone who suggests that the pot might be getting a little too warm.
The messages started as polite and eventually started to get more desperate in tone. At no point were they threatening or adversarial.
Amex realises that threatening would hurt their business trust more than anything. During the great depression, Amex accepted checks from other banks which were falling and paying through their own wallet as a matter of integrity. Amex has always been built around this idea of trust and prestige.
They make most of money from what I have heard on the transaction fees which are more than others (3% compared to 1%). They might get desperate but I am sure that they are one of the last guys who would wanna threaten you if you are paying some large bills for them (as compared to normal credit card companies which might even hire people to extract your loans in some messy situations)
So perhaps be so rich that the credit card company understands it as well and treats ya differently :-D
(I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
This does feel a bit silly for amex to do from what I've heard. Probably 2008 were a weird time in general where trust in systems itself were mostly eroded, whether of people to banking institutions and also vice versa.
> (I eventually admitted to myself that Amex isn't a person and thus not really capable of insulting my honor, but it took a while!)
haha :-)
https://www.techtimes.com/articles/320266/20260712/anthropic...
That sounds bad.
Everyone knows what water and electricity are, the vast majority couldn't explain what service AWS provides.
AWS has plentiful competitors. If you don't like their behavior, don't patronize them!
Service provider lesson #1: Never ever ever enable auto-pay! The convenience (and even the savings, if applicable) aren't worth the risk of the service provider autonomously slurping up all your money.
None of this can happen unless you first cede control.
I used AWS' free tier, of course, and I enjoyed the initial setup in EC2, and I did a LAMP-stack MediaWiki installation. It wasn't too difficult, but two things sent me away forever.
1. It was impossible, or at least highly labor-intensive, in this modern era to adequately secure an ordinary Linux system running Internet-facing services. I put fail2ban and I filtered a lot of ports, and still spammers attacked me on Layer 7.
2. It was impossible, actually impossible, to limit or cap my cloud expenses in any billing cycle. Sure, run free-tier all I want. Sure, come in within the limits almost every month. But if I configured one thing wrong, or one thing went runaway, I'd have a sizable bill that I couldn't dispute. And even worse, those "runaways" weren't necessarily things in my sphere of control, but could be triggered by basically anyone coming in and using my VPC resources, especially egress network traffic.
So I closed out my cloud account, and I developed a lot of sympathy for businesses and corps that now are forced to run "in the cloud" rather than on-prem or their own machine rooms, but now they have no way to control expenses.
The customer service people I talked to in the grocery store said this changed sometime in the last year. My guess is that it’s an unintended side effect of removing the pay-by-palm feature.
This is obviously unrelated but I joked about what else Amazon wasn’t reliably calculating….
Either that or 1000’s of small claims court cases.
Even with arbitration, the overhead of dealing with that would be crippling. Hopefully someone over there decides to do the right thing, and auto-refund.
So really, there's a third option now, that's much easier than class action, even when class actions don't get certified.
I never reported this nor took it up with either Amazon or Chase directly. There was a refund of my Whole Foods purchase (they needed to void my purchase and re-ring everything to give me the discounts.. I asked them to refund my purchase and I’d do without my Whole Foods purchase entirely).
Looking back I think at least 3 recent visits were charged to me at full price because of all this. Hard not to think of enshittification and whether Amazon Prime is even worth it, alas.. I live in a fairly rural area at the moment and need delivery.
100K years. Now that's load-bearing ...
I’m also a little surprised this didn’t trip a circuit breaker. For something as non-real-time as billing, I’m surprised they don’t have an automated kill switch that pauses the billing system and fires a page if variance in bills spikes. Naively some kind of “if the standard deviation of customer bills for this year changes by more than 50%, pause the billing system”. At that number of customers, those numbers should be pretty stable beyond internal billing changes they could normalize for.
It's been 2 hours and I still haven't fully calmed down.
Well, actually i guess you can guess what happens next! lol :-D
Use AI => No War Crimes!
People make similar pronouncements after every us-east-1 outage makes the news, but I feel like AWS would be going out of business by now if people followed through.
It reminds me of airlines, where after a particularly grueling irregular ops experience, a few dozen people file off the plane swearing "Never again, <airline name>!" but really, we all must know deep down that the airlines are all subject to the same external inciting factors, internal profit motivations, and human imperfection, and thus all pretty equally likely to cause us a bad day or ruined trip. The effort spent to avoid one isn't really worth it.
"CEO Reveals How He Used AI To Build One-Person Company That's $1.3 Billion In Debt"
https://www.youtube.com/shorts/YERfTT4McsU
Forecasted month end $18,729,381,032,152.4
Apparently my company owes the combined GDP of France, Germany, and UK to AWs.
"Operational issue - AWS Billing Console (Global) Service - AWS Billing Console Severity Impacted - Inaccurate Estimated Billing Data"
"The rollback of a recent change did not resolve the issue and we are continuing to investigate multiple mitigation paths. Estimated bill updates remain paused."
Wait what if someones actually getting usage spiked
> I think you should spin up a whole bunch more instances, and try to cause an integer overflow so they they owe you $978 Trillion.
Host your own people. Host your own.
Maybe it's not just vibe-coded, maybe the numbers themselves are being hallucinated by an LLM.
To me that looked suspiciously like string-handling in a weakly typed language.
Like when you do `"100" + 1` in JavaScript, or `int("100" * 2)` in Python.
I've seen my share of such bugs in PHP, Python, Ruby, JavaScript. In production. Obviously not as simple as the examples, but subtle, like when a library update changed `someFancyLocalStorage.getOrDefault("lastOrder", 100)` by always casting the value to the type of the default (released as patch release). Or where typedEnvGet() should typecast "numbers", but keeps it a string when theres whitespace `AMOUNT_PER_CALL=100\n`. Or where a number passes through a deep stack of middleware and 99.9% of the times remains an int but in rare race conditions becomes a string. etc.
No evidence that's the case here. But from my experience, the repeating and strange formats of numbers hint strongly in that direction.
Look at them up there, just plotting with each other! :-)
It’s probably an artifact of them all being currency multiples of 2^30
> Amazon Web Services customers receive bills for up to $1.5tn after global glitch
That sucks, some people will get legit panic attacks and worse over this, especially for the smaller, more believable numbers in the 50k-500k range.
Hope they recover and sue for medical bill costs, emotional damage etc.
And like one reddit user suggests, everyone affected should write to their representative about hard billing caps protections
How about $5,544,640,717,404.09?
That was in my inbox this morning lmao
Edit: I was just about to credit the user when my internet dropped. The source was here: https://news.ycombinator.com/item?id=48945606 - thanks mirzap!
Makes you wonder - what if there really would be an incident where some massive amount of traffic got routed to your infrastructure by some heavyweight player? Say Wikipedia accidentally switches their IP to your CloudFront? Would you really be on the hook for $500k?
1. Current month's usage * applicable rates; + 2. Estimated future usage for the month * applicable rates.
And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.
“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.
Incidentaly, smaller competitors solved this issue decades ago, while the big cloud decided it is more convenient never to implement it.
"Limits except for Storage" seems even easier - I don't think I've ever heard of a storage-based billing story, although I'm sure one or two exist
Also many places I’ve worked, storage is a huge part of the spend but that depends a lot on what you do. e-commerce doesn’t use a ton of it, but if you handle user-generated content or do any kind of training (LLM, computer vision, etc) then you can very much end up in a place where storage becomes a top line number for infra spend.
GitHub pre-Copilot was probably like that. They host a shitload of data, most of which is just at rest the majority of the time. Storage and networking are probably the majority of their infra costs.
I have seen things get hacked for bandwidth, back in the days before you could rent a gbps uplink from the cloud for $0.12. Some scene release groups would hack into universities or companies to do the initial seeding over their super fast links. It used storage, but that wasn’t really the goal.
Not necessarily. They could imply that your storage becomes inaccessible immediately, but only gets deleted after some time period (say, 1 month). What spending limits do depends on the implementation.
That would mean an outage but that is still better than going bankrupt and teach you a thing or two about monitoring.
Folks can track it directly on AWS Health: https://health.aws.amazon.com/health/status
It sounds ridiculous, but something very similar happened to me with Amazon WorkSpaces. During the WorkSpaces setup, an AWS Active Directory (Directory Service) instance was provisioned as part of the deployment. When I later canceled WorkSpaces, I had no idea the Directory Service had to be deleted separately. I kept getting billed for it, and it ultimately cost more per month than the WorkSpace itself had.
If you owe the bank $1.7B, thats the banks problem. """
What I would be curious about (and I'm sure AWS will never share) is where the incorrect number came from. If the number is somewhat consistent between some groups of accounts, my first guess would be they started summarizing billing across all accounts in whatever cell/grouping/heirarchy AWS architected internally.
Which is just funny.
Nowadays you just have to risk accidentally billing your parents CC the tune of multi-generation wealth to get that real-world experience.
I literally almost had a heart attack today.
But with AWS costs rising anyway (not by that much but OK), I'm probably not the only one to start reconsidering their cloud strategy. I think this might have just pushed me over the edge.
https://health.aws.amazon.com/health/status
Do the right thing for the players, Matt!
My sympathies -- I know I would be overcome with panic in such a situation.
I too was shocked when I saw the $1.7billion bill, instead of the usual $1.5billion.
The crypto network you hosted should pay for itself in 10-20 years just like LLMs. Don’t worry. Consider Bank of America until then if you are good on credit score.
Phew.
Looks like this is a bug w/ S3
And naturally, companies like Cloudability [now Apptio] and others have sprung up to do parts of this for you [at a fee, of course...]
https://docs.aws.amazon.com/cur/latest/userguide/what-is-cur...
I'm sure other cloud vendors have similar functionality (because they need this on the back end to do their own billing anyway).
I understand people complaining about large bills, but this is over the top!
Insane
Yeah, this most certainly is bad code wrapping around a value. AWS will post a notice soon if they haven’t already.
$103,515,940,301.79
ACTUAL Amount: $1,046,294,123,330.95
This cannot happen if you do not do this renting at variable rates.
A thing you own doesn't suddenly bill you trillions of dollars in error. It doesn't hyperscale either, but neither do you.
Then flee the country just to be sure.
This is what we received this morning
I’m currently dealing with Verizon Wireless and their “Jabronibot” claiming I have a fictional account balance due. It has been sent to collections, but still is being asked for by their legacy system.
The case studies of “Agents in Billing Departments” and potential shareholder lawsuits / E&O claims / reputational damage will be interesting to me. I worked in “risk management” products years ago and this kind of liability is not easily dollar traded away via contract. Will accountability stick to the Decision Makers or will they try to surrogate to the Service Providers? Hmm.
"NEVER represent currency with floating point, multiply by 100 and store in an int before doing any math"
I might be a winner.
I think I would have just waited to see what happened when AWS tried to hit my credit card for $1,700,000,000.
When do you ever get that opportunity?
https://news.ycombinator.com/item?id=48945681
$1,299,988,247,332.56!
That was a fun set of emails to wake up to, figured they had to be phishing for how outrageous of a number it was. But nope! Fun little incident they've got going over there.
All hail the new generations of our uberployers.
Like “I was scared for a couple minutes on a Friday morning until I saw the vendor status page” is orders of magnitude away from the bar here.
I'm not currently running anything and have no plans to at the moment. I've always had a mild dread that I'll suddenly get a bill for more than $0.00.
If AWS can goof in a way that causes obviously massive bills (like today), what's to say they can't goof in more subtle ways and start charging small additional amounts that many people may not notice and just pay it.
What an absolute joke. All just so that line goes up. As if their fees weren't high enough vs. alternatives (especially egress). And I'm sure the pro-AI crowd will keep saying we're luddites for not loving this clearly revolutionary and disruptive tech.
Honestly, I would worry more about estimated billing that seems plausible in general, but is way to high for you personally. These ridiculous amounts? Not so much.