For those interested, there was a 2009 outage of the FAA Telecommunications Infrastructure that at a surface level seems like it could be a similar failure mode. At the very least, it is an interesting read on how the FAAs systems operate, since a lot of the discussion so far has been wild speculation from people with little knowledge on how the national airspace system actually operates: https://www.oig.dot.gov/library-item/29670
Of particular note is this section:
Risks of FTI Outages for Existing Operational Services
As of December 2009, Harris reported that there are a total of 20,982 services operating on the FTI network. While FAA has transitioned all 1,808 IP services supporting FAA flight plans to the new FTI fiber optic network, additional risks will be introduced when the Agency begins transitioning other FAA services, such as En Route Automation Modernization (ERAM) and NextGen platforms, which also require IP technology to operate. Moreover, there are 19,174 existing services that may be transitioned to the new FTI fiber optic network. According to FAA, transitioning these remaining services is important because FTI now connects older FAA systems that provide safety-critical voice and surveillance radar information by utilizing Time Division Multiplex (TDM) technology, instead of IP services.
TBH TDM is far superior and it’s sad that the world discarded it. Packet switching is the worst thing to ever happen to phone calls. Kids today have no idea how good it used to be (albeit at $0.50/min).
Here's what they said caused it in the White House briefing:
Sec Mullin: "What it appears to be right now is, as Sec Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology. And that new equipment and new technology, it was overwhelming the older system. Which is why Sec Duffy made it very clear, the patchwork doesn't work. We need one complete system."
They also kept noting that the new IP based routers were not showing any inconsistency. To me that says that they were bridging IP traffic over the legacy systems, probably a protocol with retransmits like TCP, and a storm of retransmits was overloading the legacy systems, and older components using a less durable protocol were drowned out and dropped packets.
Duffy also kept saying they were working as quickly as they possibly could trying to rollout the new system. Not exactly reassuring.
I think the problem with a lot of the push to "modernize" is we forget the legacy system has been real-world tested for 40 years now. It's the new stuff that needs to prove itself in terms of stability.
We have a rule in my dev shop: if a piece of code (or hardware) has been working without fault in production, you do not touch it. It doesn't matter if it's old, if you don't like the tests, if it's in a programming language you despise; if there haven't been any problems in production, you don't touch it.
And then you have the common reality: "a piece of code has been working in production with known faults & known costs". And you have to weigh up costs and risks.
Sounds like the new system effectively performed a DoS on the old system. They probably don't have a development environment where they can hook the two systems up. Even if they do, it might actually have been an out-of-the-way component that failed first, impacting another, and so on. If they have a test environment with the two systems, a complex failure like that would unlikely to be found (and would explain why it took hours to resolve).
As someone who works under a different umbrella in the federal government, how do you propose testing something so archaic and complex without replicating an exact replica of a 40+ YO technology?
I've actually worked a system where the test env was scavenged old eq set up to mirror exactly the ancient prod env. It's possible, it just takes a lot of work. Scaling it might be touchy though.
Not only it takes lot of work, but to have the replica connect to the prod in some way to mirror identical behavior will possibly mean adjusting ATO, which is not a simple process in itself.
Emulate, computing is cheap. In an application as important as air traffic control it seems like it wold be worth it to _always_ have a working copy of the original system specifically for testing changes.
Not aware of specific system, but critical infrastructure systems have a quality assurance system that replicates production albeit not perfectly. You may replicate the core system but not all its interfaces, you might replicate it but scaled down etc. There are ways for solid tests to fail. For systems with downtime, a test on production is useful iif you have solid rollback procedures. There’s lots of art in the operation of these systems.
Let's set that aside and instead ask why tf they rolled out anything on a Friday? Or was it that they rolled it out on Tuesday but failed to notice it was close to overload state until it actually overloaded on Friday?
Because if it doesn't work, they call it out for bids
again, select their cronies and get kickbacks. If it does
work, then they've missed an opportunity for grift.
The parts to maintain the legacy copper infrastructure are no longer manufactured and they are cannibalizing used parts for spares.
Not to mention the expertise for these systems have mostly either retired or in some cases passed away.
My grandparents still have a landline and the number of outages in the last 12 months has been insane. Some were in excess of a week.
Many such cases.
Sometimes the telcos have been shrugging their shoulders and refusing to perform repairs. This is common after a car accident takes out a pole, cabinet, etc. They accelerate the retirement to right now and force the customers onto new technologies.
Not really. The problem with keeping old technology is that you need people who have knowledge of how to maintain the old systems. Ignoring the cynical view that many of those people were likely fired, even if they were kept on, they will eventually retire. There's no amount of training that can replace that institutional knowledge.
If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.
Sure, but the purpose of the tech policy office is to
argue why old systems (which have fixed maintenance costs)
should be replaced with new systems which goose the
economy when the federal government pays 10x for what they
should be paying.
This is only one of many recent “lip sores” indicating the a herpes virus that is all over US critical infrastructure.
Over a decade ago I was an intelligence officer and the alarm bells were going off and it was exceptionally clear to the National security community that US infrastructure is barely holding on.
I can't tell if you're being ironic or not. Sure, "one complete system", and I want a pony.
I watched Sec Duffy's presentation, and it was appalling, but par for the course with this administration: in the middle of a major nationwide outage, he tried to blame someone else.
Even if Duffy got everything he wished for, how do you think they would upgrade a system as old and complicated as ATC. They would have to do it piecemeal anyway. This was a classic upgrade that broke shit. The problem was with the new upgrade, not the old shit that worked.
I was just presenting the full quote from the Secretary of Homeland Security explaining what went wrong. I wasn't commenting on it. I added quotation marks just now to make that more clear.
Since the threads keep getting closed about Hawaii DoT confusing everyone by claiming SpaceX debris was contributing to their shutdown despite the Starship launch being in Texas, successfully reaching orbit, and having a booster land in the Atlantic:
Notably, the problem was described as "routers overloaded with data". When directly questioned if this could be a cyber attack or rogue ai agent the Sec/Homeland responded "there's nothing completely certain now."
Or a non-technical person is relating the information as they understand it to a non-technical press corps, so everything is generic and hand-wavy. Honestly, have you never dealt with non-technical C-suite staff?
At the press conference (current, live), they are blaming old hardware that is failing. They’re in an endless loop of repeatedly saying that it’s Congress’s fault for lack of funding and that this is the result of old hardware. I just gave up on watching because they aren’t saying anything informative.
Not only was there a lack of useful information (other than the fact that the grounding was lifted), it was an astounding demonstration of poor leadership during a crisis. No ownership, no focus on the problem/root cause/next steps.
Usually I refuse to give any detailed postmortem right after an incident is mitigated/resolved until I have had a chance to collect my thoughts and make sure I can write a clear problem/root cause/next steps. The communication that goes out with the mitigation signal is often short and vague and the detailed postmortem follows a day or so later.
Let's be clear: it was a bunch of idiot political hacks being unbelievably obsequious to Trump. So embarrassing for the country. I don't trust that the ask for $50B is anything more than grift for their friends. Was the grounding a purposeful overreaction to increase odds of getting money?
I don't doubt that the system needs upgrades, but these people should not be involved.
I thought you were being a jerk, but no this dude has an Associate in Applied Science in plumbing from the Oklahoma State University Institute of Technology. Huh.
I'm pretty sure saying that Congress needs to give them the money to replace the ancient hardware is putting ownership directly where it belongs and identifying the problem, the root cause, and the next steps.
That if the ancient hardware was working and just the new software they tried to put on it did not worked with that hardware - it still means they messed up.
There's still time for that. The mayhem is ongoing and will be felt for days.
I hope by Monday a few "bosses" will say thanks to all the staff who ran overtime to deal with desperate grounded passengers, do all the rebooking snd schedule changes.
Ah sorry for the misleading writing. My brain has been fried trying to work while also trying to dig up any useful info to help my wife who is stranded at the airport figure out what to do. I usually can last longer watching press conferences but this one was… just painful. Maybe someone else can give an update if they actually talk about what’s going on.
I didn’t say it isn’t. Just that they aren’t really giving any details and more importantly, they aren’t giving info for what people who are stranded should be doing right now. (And now that I’ve settled down a bit, it’s odd that if it’s old hardware failing, how is it suddenly not failing? Was it just some systems, did they have replacements, or if it’s what the other commenters wrote, then did they just turn off the AI updates to stop overloading the hardware? Which would mean that hardware is fine. It’s just underprovisioned.)
Not with the budget the FAA has to apply to the problem.
Yes, you have a point. But the importance of this system could get circuit boards manufactured to old drawings. Specific 'electronics boxes' replaced with new ones designed to replace the antiquated hardware.
FAA has been doing this for decades. A single electronics box, or even a couple should not drop the system. Maybe for the time it takes to remove and replace.
After hearing the press conference "some new stuff was plugged in and the legacy system could not handle it" or alternatively 'a mistake was made that just so happens involved non today equipment, but we want more money'
I'm being a little bit tongue in cheek, I'm sorry your wife is stranded.
I expect it to be a combination of extremely poorly funded and sidelined IT infrastructure trying to use a bloated AI system from a hype company. Which results in everything breaking and poor schmucks already at the end of their rope working to fix something they barely understand.
Because everyone [in the specific field of technical professional operating systems like this] is using AI and not everyone is smart about how they use AI?
System worked mostly fine for decades, then in the middle of integrating a new AI system it completely goes down.
It doesn't take a genius to see the problem is the new AI system. Whether or it's the "AI" part of the system isn't really relevant system this new system was vibe-coded by AI and built around the "AI" features.
There is zero evidence of this at all. The stuff going around on social media has zero correlation to the system in question. Unless you have other evidence others are not privy to?
> I expect it to be a combination of extremely poorly funded
Highly doubt this. Unless you mean sort of like education in many areas where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
> where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
Yeah of course I mean this, I mean the guy who had to serial connect to whatever shit show was running and was sitting there trying to parse error logs.
Given that the old hardware has been around a long time without a problem like this, it seems very odd to blame the old thing for the new problem. My money is on a poorly planned transition, one where they didn't take the old hardware seriously enough.
It's certainly possible - but for whatever reason this administration (with the help of that guy from the robot taxi company) seems to not like air traffic controllers and has been trying to find ways around them.
It's possible this is the standard "privatize a government function and skim the profits" move but some of this stinks of union busting and trying to reduce the amount of influence air traffic controllers have whenever the organized criminals that run our government want to have a shutdown but keep their private jets flying.
Suffice it to say - there's no reason to trust any of these people when it comes to taking care of critical government functions.
They were also insisting its not a problem with the new upgrades they were installing, but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
Not sure if this was the the exact quote, but I see AP wrote, "introduction of some new technology into an old air traffic control system overwhelmed it." Despite that, the whole conference, on loop, they were insisting that the problem is entirely the old hardware and this is why they need billions to overhaul it and rebuild from scratch.
>but one of them slipped up and said something along the lines of the "new hardware overwhelmed the old hardware" which seems like the real issue.
The secretary also mentioned that the new system should be brought altogether, and not piece by piece. Which would lend credence to an incompatibility between both systems.
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system"
I assume that this was a boring, routine update that went awry.
But my completely unfounded and more conspiratorial theory, based on zero knowledge of air traffic control systems, is that to better accommodate SMART, L3Harris was strong-armed into to rolling out a hastily written patch on a Friday, which accidentally took down the system.
They are trying desperately to not name the AI system that got turned on yesterday that they commissioned.
Is it more likely that decades old hardware failed coincidentally or that the first rollout of a new AI tech went wrong within a number of hours because the test environment didn't have sufficiently scaled load testing?
The press should be asking why this could be the case when the GOP has held congress and the white house for two years ? Just what was the party in charge thinking ?
I think the supplemental appropriation that the FAA got as part of the OBB is probably too big to be spent.
Unless you just hand it all to a contractor, it takes time and oversight to spend billions. Legally and practically.
"It's the safest system in the world, I mean, uh, except that it's horribly outdated with switches from the 80s and radars from the 70s, so we really need 50 billion dollars from congress STAT or we're in big trouble, but it's completely safe. Also Congress needs to give Trump the Builder this money _now_. We couldn't get it in the past couple of years even though we controlled the entire government. I mean, sure, we got 12 billion, but that's not enough to unfuck what Joe Biden did. Your loved ones are completely safe by the way, but we can't promise this won't happen again."
We need a despot that will ignore all bureaucratic bloat and get things done, quite frankly. Otherwise this country is done for. Spiraling into the trash can.
Transportation secretary saying that their networking routers were being overloaded by increased communications traffic. But its not a cyberattack or software failure.
They just said in the press conference that it couldn't possibly be related to SMART, since the systems were disconnected, but the timing and lack of a concrete answer as to the cause or likely cause, combined with asking for money from congress (which currently has a republican majority...) for "more modern infrastructure" has me doubting their version of events
I acknowledge this is borderline conspiracy. Sec Mullen's statement is that FAA was quote "installing new equipment, new technology and that the new equipment new technology was overwhelming the system"
Transportation secretary Duffy says that the smart system is unrelated. That the smart system is a dashboard for displaying all the data from across the air traffic system. My questions is obviously where is the smart system getting its data from...
I suspect FAA pushed for the new AI system, didnt take into account the increased traffic demand to feed the system. Aging infra isnt sufficient to handle the increased demand. Now Duffy is trying to deflect blame unto someone else.
I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was...
Should be an easy answer. We saw increased traffic from X and we've implemented Y to ensure it doesn't happen again.
Instead it was vague hand waving around old hardware and funding.
Even if they don't have all the details organized, what was done to fix the problem? How was the 12b spent thus far and what where does the additional 50b number come from?
Or maybe don't start with demanding money. What happened? How was it fixed? What is being done to ensure it doesn't happen again? Then mention this shit needs fixed and we'll be working with Congress for additional funding.
It was interesting that Mullen mention new technology contributing at one point. Sounds like someone made a mistake somewhere and this press conference was purposefully vague or they really don't understand what happened yet (doubtful).
Plenty of critical systems operate reliably on old tech as long as you work within the limitations.
This press conference had maybe 30s of actual info and the rest was this antiquated system is fucked even though it's been working for years and the only way to move forward is $50b. :shrug:
> I kept waiting for someone to ask what the source of the traffic that overwhelmed the legacy systems was
They did hurriedly mention it was "new patches" in the midst of their deranged ranting and finger-pointing (at...their own party, a point Duffy was visibly taken aback by when told by a reporter). When they vehemently insisted it was 'not SMART' one has to presume they meant that the hasty execution of their haphazard AI rollout was performed unintelligently.
maybe, what is confirmed is that they were trying to upgrade some physical TDM switches with IP routers (as they have been doing for a while), and the "new systems overwhelmed the old ones".
It is possible that this overwhelming was caused by a software or configuration error, no doubt.
I think that if that's the case, rather than failing upwards and going for a full simultaneous upgrade with a 50B congress bill, the brakes should be stepped on a little bit, continue the partial migration as long as there are no issues. Or at least wait until the start of a 4year administration to go for a rehaul, so that you can stay around if things go fubar.
Unclear what happened but they have old equipment. Only Duffy can fix it, he needs $50B from Congress to do it and they can't fix it overnight. They can't guarantee it won't happen again.
Not much info yet. Even ops.group [1], which is for people who do flight planning, has nothing yet.
LAX hasn't restarted yet. Departures are either "Cancelled" or "Delayed".
All information about this is from PR teams. Nothing from reporters who cover airports. Nobody from airport management, airline operations, FAA operations people, or somebody who just got off shift at a control tower. People who might actually know something.
There was a briefing at the White House, and it was all cabinet secretaries. "Instead of reassuring an anxious public, and providing concrete information, this briefing appears to be intended for President Trump’s approval" wrote a New York Times reporter.
FAA policy prohibits anybody but PR people talking to the public. The details of that policy are not public.[2]
So they were doing a network upgrade and putting in new equipment. This new equipment was misconfigured and caused the existing working system to stop working. And now they are saying in the press briefing that this is a problem of the existing working system which they call the "legacy system" and are using this as a political move to call for the congress the regime no longer controls to provide funding to the FAA/DHS. Framing the manufactured issue they caused with the new system as if it was congress' problem instead of their own breakage they caused and are deflecting from. Demanding an emergency funding bill now despite ignoring it in all the years they were in control.
It’s possible the interfaces to the legacy system aren’t documented well and/or it could be incompetence on behalf of the network engineers working this
Maybe things are moving fast today and they didn't have time to understand and communicate accurately, but at some point soon I think we deserve a more in depth explanation.
There is an actual bill for $30B that's been stalled for months in the Senate called the Modern Skies Act, and there is absolutely widespread antiquation within the US air infrastructure which this bill seeks to address. So he's not just making this up. He's talking about exactly that bill albeit with all the Trump BS injected into it.
Modern Skies Act is for $30B, not $50B. A separate $12.5B from "Big Beautiful" is already being spent on ATC modernization, and only $10B of MSA is toward ATC
Sure, but blaming Democrats when they’ve controlled Congress and the presidency for the past two years and their highest priority has been cutting funding via DOGE is pretty disingenuous
Their press conference was horrendous but it seemed he was just blaming Congress overall and not Democrats specifically (since it is Republicans actually holding it up)
Assuming I'm looking at the right bill (S. 5572, the Modern Skies Act), "stalled for months" seems a tad overstated considering it was introduced Sep 25th 2026, just three days before Congress went on recess for the entire month of October (elections). The Senate wasn't back in session until Nov 9th, so the bill has been pending for a grand total of seven in-session days (even emergency legislation would rarely pass that quickly).
The Senate will be in session for four more days next week before the Thanksgiving holiday, and then back for the first three weeks of December before the 119th Congress adjourns for good on Dec 18th. So basically, after two full years with control of both branches of Congress and the WH, this bill was introduced on the very last week of the regular Congress calendar. After which the lame duck Congress would only meet for about four weeks until adjournment.
Not really stalled, more like procrastinated until days before the election (when everyone is scared to vote on anything), hoping that the lame duck Congress (notoriously unproductive period generally) would force it through in a rush?
Doesn't seem like a great strategy if they actually wanted it to pass. Especially since a budget CR is needed before Dec 11 which will dominate members' attention in the remaining time (plus the permitting bill BAAJA).
Doesn’t it feel wrong to anyone else that there’s a single point of failure for all fights in the US? I can understand if there are regional zones and the East coast gets grounded. But everything at once??
Given that many flights are between regions, it makes sense to me that you would need a nationwide system. Even a flight within a region may depend on a plane or crew coming in from another one.
Having it degrade into regional systems would be a nice property, but risks making a complex system more complex. Put all of your eggs in one basket and then make sure it's a really good basket.
It's a miracle that US air traffic control isn't having incidents like this more often. Admiral Cloudberg has a great write-up [1] that touches on some of the systemic problems in American aviation, oh boy it's a mess.
I guess if you consider safety the top priority, you might let a local failure pause the whole system. So I’m not sure quite how much to criticize this point
Is it a bad idea? Absolutely. But also it's to be expected. People like to have a single point of control. You can't do that without introducing single points of failure.
I wish more was being done on partition tolerance. Consistency is over rated.
I took the opportunity to ask Fable what the 50% over/under on my 7am flight being delayed tomorrow, and it used 33% of my four hour window to come up with an answer of 15 minutes. Somehow, I doubt it.
"we" being him (ie. not the reporter) because he's part of the executive branch; that's the limit of the authority of the executive order, after all...
Ahh the great Kubernetes cluster in the sky finally finished reconciling and got the new nodes up and running. I bet this time it only took 3 faxes and 9 different signatures from heads of the redundant departments of redundancy.
They jumped from 3 airports to nationwide in one step, and are now framing it as the existing hardware not being compatible with their new software, rather than the other way around.
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system".
So, it was a careless update, probably the one deploying their new 875 millions AI tool.
And no journalist at the conference picked that up.
Of particular note is this section: Risks of FTI Outages for Existing Operational Services As of December 2009, Harris reported that there are a total of 20,982 services operating on the FTI network. While FAA has transitioned all 1,808 IP services supporting FAA flight plans to the new FTI fiber optic network, additional risks will be introduced when the Agency begins transitioning other FAA services, such as En Route Automation Modernization (ERAM) and NextGen platforms, which also require IP technology to operate. Moreover, there are 19,174 existing services that may be transitioned to the new FTI fiber optic network. According to FAA, transitioning these remaining services is important because FTI now connects older FAA systems that provide safety-critical voice and surveillance radar information by utilizing Time Division Multiplex (TDM) technology, instead of IP services.
Sec Mullin: "What it appears to be right now is, as Sec Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology. And that new equipment and new technology, it was overwhelming the older system. Which is why Sec Duffy made it very clear, the patchwork doesn't work. We need one complete system."
I think the problem with a lot of the push to "modernize" is we forget the legacy system has been real-world tested for 40 years now. It's the new stuff that needs to prove itself in terms of stability.
> if a piece of code (or hardware) has been working without fault in production
If something can't be turned back on after it gets turned off, that's just a latent, undetected fault.
Not only it takes lot of work, but to have the replica connect to the prod in some way to mirror identical behavior will possibly mean adjusting ATO, which is not a simple process in itself.
https://static.wingify.com/gcp/uploads/abtasty/test-in-produ...
> Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
And as a tip, people tend to downvote comments whining about downvotes.
The parts to maintain the legacy copper infrastructure are no longer manufactured and they are cannibalizing used parts for spares.
Not to mention the expertise for these systems have mostly either retired or in some cases passed away.
My grandparents still have a landline and the number of outages in the last 12 months has been insane. Some were in excess of a week.
Many such cases.
Sometimes the telcos have been shrugging their shoulders and refusing to perform repairs. This is common after a car accident takes out a pole, cabinet, etc. They accelerate the retirement to right now and force the customers onto new technologies.
If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.
In your mind maybe. Most are dead or retired.
New techs have been trained on splicing fiber and VoIP over the last two decades.
Nobody knows about vacuum tubes anymore either, that doesn't mean they were fired and had their pensions taken away.
Over a decade ago I was an intelligence officer and the alarm bells were going off and it was exceptionally clear to the National security community that US infrastructure is barely holding on.
Clapper in 2013:
https://nsarchive.gwu.edu/document/21491-document-90
Dan Coates in 2018:
https://www.euronews.com/2018/07/13/us-intel-chief-warns-of-...
Tip of the iceberg folks. Hold on to your butts…
I watched Sec Duffy's presentation, and it was appalling, but par for the course with this administration: in the middle of a major nationwide outage, he tried to blame someone else.
Even if Duffy got everything he wished for, how do you think they would upgrade a system as old and complicated as ATC. They would have to do it piecemeal anyway. This was a classic upgrade that broke shit. The problem was with the new upgrade, not the old shit that worked.
I agree with your view.
Hawaii was the early/late splashdown site and pre-cleared: https://www.cbsnews.com/news/spacex-launches-starship-flight...
SpaceX released the early return around 4 p.m. Pacific: https://x.com/TrackingTheSB/status/2121392533418786992
HDOT phrased it badly, but there was a window today where Starship could have aborted over the west coast in the middle of this shutdown.
Notably, the problem was described as "routers overloaded with data". When directly questioned if this could be a cyber attack or rogue ai agent the Sec/Homeland responded "there's nothing completely certain now."
I don't doubt that the system needs upgrades, but these people should not be involved.
What am I missing?
As a postmortem, once the dust has settled, is the time to ask for more funding/support
> old system give us money
> probably not a cyberattack btw
> not SMART related
> we've opened the airspace but this is gonna be slow
just ended actually as i was typing this, that's pretty much it
They didn't.
FAA has been doing this for decades. A single electronics box, or even a couple should not drop the system. Maybe for the time it takes to remove and replace.
After hearing the press conference "some new stuff was plugged in and the legacy system could not handle it" or alternatively 'a mistake was made that just so happens involved non today equipment, but we want more money'
I expect it to be a combination of extremely poorly funded and sidelined IT infrastructure trying to use a bloated AI system from a hype company. Which results in everything breaking and poor schmucks already at the end of their rope working to fix something they barely understand.
It doesn't take a genius to see the problem is the new AI system. Whether or it's the "AI" part of the system isn't really relevant system this new system was vibe-coded by AI and built around the "AI" features.
Highly doubt this. Unless you mean sort of like education in many areas where the system is robustly funded but the money doesn't actually reach the people doing the work on the ground.
Yeah of course I mean this, I mean the guy who had to serial connect to whatever shit show was running and was sitting there trying to parse error logs.
Do you really need the government to solve that problem for you? Just book another flight.
It's possible this is the standard "privatize a government function and skim the profits" move but some of this stinks of union busting and trying to reduce the amount of influence air traffic controllers have whenever the organized criminals that run our government want to have a shutdown but keep their private jets flying.
Suffice it to say - there's no reason to trust any of these people when it comes to taking care of critical government functions.
Not sure if this was the the exact quote, but I see AP wrote, "introduction of some new technology into an old air traffic control system overwhelmed it." Despite that, the whole conference, on loop, they were insisting that the problem is entirely the old hardware and this is why they need billions to overhaul it and rebuild from scratch.
The secretary also mentioned that the new system should be brought altogether, and not piece by piece. Which would lend credence to an incompatibility between both systems.
(https://news.ycombinator.com/item?id=50492957)
But my completely unfounded and more conspiratorial theory, based on zero knowledge of air traffic control systems, is that to better accommodate SMART, L3Harris was strong-armed into to rolling out a hastily written patch on a Friday, which accidentally took down the system.
Is it more likely that decades old hardware failed coincidentally or that the first rollout of a new AI tech went wrong within a number of hours because the test environment didn't have sufficiently scaled load testing?
Speculation on my part, they just rolled out their SMART air traffic routing system. https://www.flyingmag.com/faa-smart-ai-now-active-at-22-artc...
Bet the increased strain on data polling from this system is the culprit for increased traffic.
Transportation secretary Duffy says that the smart system is unrelated. That the smart system is a dashboard for displaying all the data from across the air traffic system. My questions is obviously where is the smart system getting its data from...
I suspect FAA pushed for the new AI system, didnt take into account the increased traffic demand to feed the system. Aging infra isnt sufficient to handle the increased demand. Now Duffy is trying to deflect blame unto someone else.
Should be an easy answer. We saw increased traffic from X and we've implemented Y to ensure it doesn't happen again.
Instead it was vague hand waving around old hardware and funding.
Even if they don't have all the details organized, what was done to fix the problem? How was the 12b spent thus far and what where does the additional 50b number come from?
Or maybe don't start with demanding money. What happened? How was it fixed? What is being done to ensure it doesn't happen again? Then mention this shit needs fixed and we'll be working with Congress for additional funding.
It was interesting that Mullen mention new technology contributing at one point. Sounds like someone made a mistake somewhere and this press conference was purposefully vague or they really don't understand what happened yet (doubtful).
Plenty of critical systems operate reliably on old tech as long as you work within the limitations.
This press conference had maybe 30s of actual info and the rest was this antiquated system is fucked even though it's been working for years and the only way to move forward is $50b. :shrug:
They did hurriedly mention it was "new patches" in the midst of their deranged ranting and finger-pointing (at...their own party, a point Duffy was visibly taken aback by when told by a reporter). When they vehemently insisted it was 'not SMART' one has to presume they meant that the hasty execution of their haphazard AI rollout was performed unintelligently.
It is possible that this overwhelming was caused by a software or configuration error, no doubt.
I think that if that's the case, rather than failing upwards and going for a full simultaneous upgrade with a 50B congress bill, the brakes should be stepped on a little bit, continue the partial migration as long as there are no issues. Or at least wait until the start of a 4year administration to go for a rehaul, so that you can stay around if things go fubar.
Unclear what happened but they have old equipment. Only Duffy can fix it, he needs $50B from Congress to do it and they can't fix it overnight. They can't guarantee it won't happen again.
LAX hasn't restarted yet. Departures are either "Cancelled" or "Delayed".
All information about this is from PR teams. Nothing from reporters who cover airports. Nobody from airport management, airline operations, FAA operations people, or somebody who just got off shift at a control tower. People who might actually know something. There was a briefing at the White House, and it was all cabinet secretaries. "Instead of reassuring an anxious public, and providing concrete information, this briefing appears to be intended for President Trump’s approval" wrote a New York Times reporter.
FAA policy prohibits anybody but PR people talking to the public. The details of that policy are not public.[2]
[1] https://www.ops.group/blog
[2] https://www.faa.gov/regulations_policies/orders_notices/inde...
Maybe things are moving fast today and they didn't have time to understand and communicate accurately, but at some point soon I think we deserve a more in depth explanation.
https://www.transportation.gov/sites/dot.gov/files/2026-09/M...
The Senate will be in session for four more days next week before the Thanksgiving holiday, and then back for the first three weeks of December before the 119th Congress adjourns for good on Dec 18th. So basically, after two full years with control of both branches of Congress and the WH, this bill was introduced on the very last week of the regular Congress calendar. After which the lame duck Congress would only meet for about four weeks until adjournment.
Not really stalled, more like procrastinated until days before the election (when everyone is scared to vote on anything), hoping that the lame duck Congress (notoriously unproductive period generally) would force it through in a rush?
Doesn't seem like a great strategy if they actually wanted it to pass. Especially since a budget CR is needed before Dec 11 which will dominate members' attention in the remaining time (plus the permitting bill BAAJA).
https://tilde.zone/@sirwumpus/117611840050306362
EVENT TIME: 14/0116 - 14/0130 DESTINATION AIRPORT:ALL FACILITIES INCLUDED: ALL EXPECT UPDATE:0400Z REASON: REMARKS: THE FAA COMMAND CENTER NATIONWIDE GROUND STOP HAS BEEN CANCELLED. EFFECTIVE TIME:
140119-140200 SIGNATURE:
26/11/14 01:19
BTW, Uber surge pricing felt absurd in this situation.
Having it degrade into regional systems would be a nice property, but risks making a complex system more complex. Put all of your eggs in one basket and then make sure it's a really good basket.
[1] https://admiralcloudberg.medium.com/reaping-the-whirlwind-in...
I wish more was being done on partition tolerance. Consistency is over rated.
So it was the AI?
Who could have seen this coming?
"What it appears to be right now is as secretary Duffy and FAA was going in and putting patches into the system, they were putting new equipment, new technology, and, that new equipment new technology it was overwhelming the older system".
So, it was a careless update, probably the one deploying their new 875 millions AI tool.
And no journalist at the conference picked that up.
"They were putting new equipment um and uh new um......technology"
Unrelated I only fly ocunus because I don't want a blue haired pilot.