Charity Majors: Generative AI Is Not Going To Build Your Engineering Team For You
[L]arge software systems are unpredictable and nondeterministic, with emergent behaviors. The mere existence of users injects chaos into the system. Components can be automated, but complexity can only be managed.
Even if systems could be fully automated and managed by AI, the fact that we cannot understand how AI makes decisions is a huge, possibly insurmountable problem. Running your business on a system that humans can't debug or understand seems like a risk so existential that no security, legal or finance team would ever sign off on it. Maybe some version of this future will come to pass, but it's hard to see it from here. I would not bet my career or my company on it happening.
Charity's article is essentially one apt, quotable point about software and teams after another, and I recommend reading it.
I like how this quote ties understandability and debuggability of systems to business risk.
RSS