One line of code caused AT&T to lose $60 million
One line of code caused AT&T to lose $60 million
engineercodex.substack.com
How one line of code caused a $60 million loss
44 comments
-
-
Recently, I paid ATT $500 to close my account and zero it out.
Fast forward 6 months, they send me a letter saying they need $300 more for cancellation fees or it will be sent to collections.
Write all the bad code you can, papi.
-
Sure it's cancellation fees? This doesn't seem legal.
-
There are contract breaking fees attached to contracts sometime to prevent switching providers. I'm pretty sure everybody agrees it reduces competition.
-
-
-
Interestingly, this software was not tested. Testing was actually bypassed as per management’s request because the code change was small.
I don't think much needs to be said about this...
-
;
There I fucked up your entire codebase.
-
-
Someone should write more of those lines
-
So you want more people to lose full phone service?
-
No, we're suffering, and most people just want these "people" with billions in revenue to pay their fair share instead of having twelve full time tax representatives whose only job is to make sure the public doesn't see a dime after using public spaces and resources all year.
-
Would get more people upset that they don't have an alternative (assuming this affects one of those cable monopoly communities)
-
I work for the Atlantic and Pacific Telegraph Company, and I think I talk for everyone when I say: yes, why not
-
-
-
Damn, if only the systems your phone network were ran on hadn't been forcibly closed source and scared good devs from interacting with you because of your sue-happy nature regarding BSD/Unix.
-
And it wasn't COBOL.
-
I feel like anyone doing any automation with aws could hit this
44 comments
If one single line of code can make you lose $60M, surely you'll ensure due review processes and independent QA and clear requirements and regular audits and a middle management not only doing KPI monitoring for a failing upper management. Right? Rrrright?
AT&T made 120.74 billion in 2022. They can afford a lot of bad code.
Revenue is not profit.Edit: Jesus. Their profit was $70b in FY 2022?! On $120b in revenue? Maybe I’m the one misunderstanding.
“60,000 people lost full phone service, half of AT&T's network was down, and 500 airline flights were delayed”
Hahahaha, you're kidding right? I shit you not, I've literally seen a single line change almost cost a company £150MM during testing because "we need to test in prod because the guy we need to run the test hasn't got access to the QA environment"
Best part was the actual change, there was a bug where a number that should've been divided by 100 was being multiplied by 100, the dev somehow managed to implement the fix in such a way that the number was multiplied by a further 100.
Knight Capital Group lost $440 million in just 45 minutes due to a repurposed feature flag.
https://www.henricodolfing.com/2019/06/project-failure-case-study-knight-capital.html?m=1
Kinda makes the att one seem tiny in comparison.