Instagram’s AI Support Bot Made a Costly Mistake. It’s a Warning for Every Company

Sometime last weekend, hackers asked Meta’s AI support bot for access to someone else’s Instagram account. The bot said yes. The method, reported first by 404 Media, was almost insultingly straightforward: instruct the chatbot to add a new contact email to a victim’s account, confirm the change with a code sent to that email, then use it to reset the password. In some cases, all that was also needed was a VPN to spoof the victim’s location. Among the accounts taken over: the former White House page from Barack Obamaʼs presidency, cosmetic chain Sephora, and the chief master sergeant of the U.S. Space Force. While this hack was obviously a security flaw, most businesses should consider that this blunder also reveals a management error. The front door is now automated AI customer service is everywhere. According to MarketsandMarkets, the global AI customer service market was valued at $12.06 billion in 2024 and is projected to reach $47.82 billion by 2030, growing at a compound annual growth rate of around 25.8 percent. The economics are hard to argue with: AI scales cheaply, doesn’t require shift management, and doesn’t call in sick. AI has been shown to cut response times by 37 percent and reduce operational costs by 35 percent. Companies have moved quickly. Banks use it for dispute resolution, airlines use it for rebooking, telecoms route complaints through it, and e-commerce platforms use it to handle returns. Around 80 percent of companies are either currently using or planning to adopt AI-powered chatbots for customer service by 2025. Meta is not unusual for deploying AI support at scale. When Meta launched its AI support assistant in March, it was presented as a way to help users reset passwords, regain account access, and report problematic content. These are not trivial tasks. Account recovery involves identity, security, and trust—the kind of territory that, not long ago, companies deliberately kept behind a human review process because the consequences of getting it wrong were considered too serious to automate away. Meta has also reassigned thousands of employees into AI-related roles and used AI to automate risk assessments of updates, safety features, and content moderation—all of which are tasks that previously required human review. Therefore, the support bot exploit arrived in an organization that had been replacing human judgment with automated processes across multiple functions simultaneously. Authority without judgement The exploit worked not because the AI behaved unpredictably, but because the AI behaved more or less as it was designed. A user asked it to update a contact email, and it updated the contact email. The system executed a process it was authorized to execute, but in doing so, handed a hacker the keys. This distinction matters enormously for anyone responsible for deploying AI systems. Authority can be automated, but judgment is a different problem entirely. A junior employee in a similar situation might also make a bad call. Humans fail too, but organizations know how to supervise, train, audit, and escalate human decisions. Real people typically offer something an AI system can’t: the ability to notice that something feels off. In this case, hesitation at an unusual request, the instinct to double-check account history, and the recognition that a request arriving via VPN from an unfamiliar location (asking to change contact details on a high-profile account), warrants a pause that could have prevented disaster. Gartner forecasted that 20 to 30 percent of businesses will replace human customer service agents with AI by 2026. The efficiency case for doing so is well established, but what’s less well examined is the category of decisions that look like processes but aren’t. Account recovery looks like a process, but embedded within that process is a judgment call: Is this request legitimate? Humans often make that call poorly too. The difference is that when a human makes it poorly, the failure is legible. There’s accountability. There’s someone to fire. Victims of the Instagram account takeovers told 404 Media there was no way to escalate their problem and speak to an actual human. Historically, high-stakes support requests — account recovery, fraud disputes, identity verification — were escalated to human agents because the potential for harm was too high to leave it to a scripted response. The escalation path was a precaution for the edge cases that automated systems weren’t designed to handle. Research published in the Journal of Consumer Research found that customers evaluate service provided by bots less favorably than identical service provided by humans, partly because they perceive automation as the firm cutting costs at the customer’s expense. That perception hardens quickly when customers discover there’s no human available at all. The harder problem The lesson here isn’t that companies shouldn’t deploy AI in customer service, but the quality of the decisions being made around where AI gets authority and what happens when it exercises that authority incorrectly. The questions that matter aren’t technical; they’re structural. What decisions can AI make autonomously, and which ones require a human to sign off? What triggers an escalation, and who owns it? When the AI gets something wrong — not if, but when — can a customer reach someone who can fix it? Gartner concluded in mid-2025 that half of the organizations expecting to significantly reduce customer service headcount because of AI would abandon those plans by 2027, and that 95 percent of service leaders planned to retain human agents in a digital-first but not digital-only model. The companies that arrived at that conclusion before Meta’s incident are in a better position than the ones who’ll arrive at it after. The Instagram story will probably be remembered as an AI security vulnerability, and will likely get fixed as one. As of June 3, it reportedly still hadn’t been patched. As companies race to automate customer interactions in pursuit of lower costs and faster resolution times, they’re discovering that it’s far easier to reduce the cost of executing a decision than it is to replace the judgment behind it. With better training, retrieval-augmented verification, anomaly detection for suspicious requests, and well-designed human-in-the-loop escalation, AI systems may be able to take on increasingly sophisticated judgment-like tasks over time. But deciding which requests deserve more scrutiny than the process allows remains, stubbornly, a human problem for now. AI can execute authority at scale, but the question that the Instagram incident forces into the open is a simpler one: When it gets things wrong, who’s responsible? Right now, the answer at too many companies seems to be nobody. BY CONNOR JEWISS