We Don’t Need Experts

At 1:04 p.m., every phone in the call center went quiet at the same moment.

Not dead quiet. Hold-music quiet, the maddening kind, built out of a voice that will never tire and never apologize, telling a member their call is important and that someone will be with them in approximately two minutes.

Thomas Hudson was three sips into a coffee that had gone lukewarm an hour ago when the first Teams message landed, from Brian Meyers, who ran the Contact Center floor and did not usually message this fast.

Brian Meyers: anyone seeing IVR self-service down? getting flooded

He set the coffee down without drinking it.

By 1:06, the channel had eleven people in it. By 1:09, it had thirty. The messages had stopped being sentences and started being fragments. Same here. Confirmed. All queues. Not just us. Nobody left typing a complete thought. There wasn’t time.

Tom pulled up NICE CXone, the platform that ran the call center’s routing and its IVR menus. Call volume looked normal — members were still dialing in, same as any Tuesday. But the self-service numbers had fallen off a cliff exactly four minutes ago, a sheer vertical line on the dashboard, like something had been switched off. Every call that should have been resolved by the IVR in under a minute — a claim status, a prior authorization, an address verification — was instead landing in a queue with a two-minute wait and a live agent who had no idea why their day had just gotten longer.

The team had a name for that queue. IVR Jail.

It existed for a real reason. If the IVR couldn’t get a valid response back from NICE Voice Biometrics — the voice-verification layer that confirmed a caller actually was who they claimed to be — it didn’t guess. It didn’t take a chance on a stranger authorizing a transfer. It punted the caller to a human, every time, no exceptions. A security feature, not a bug.

Which meant somewhere upstream of NICE Voice Biometrics, something had just stopped answering. And every caller into the plan, for the last five minutes, had been treated like a security risk instead of a member asking whether their claim had processed.

“Talk to me,” Nate said, pulling up beside him without waiting to be asked.

Nate Bishop, the Product Owner, and the one person on this floor who’d argue with Tom about architecture as readily as he’d defend him to the business. He’d appeared without either of them acknowledging it. The building itself seemed to be holding its breath.

Nate was young for the seat, well under forty, and called himself an introvert with a straight face every time it came up, a claim Tom believed for about the first month and never quite again after that. What Tom noticed instead, almost immediately, was how he talked — measured, exact, the kind of professional Tom found himself quietly holding other people to afterward, whether or not he ever said so out loud. Nate wasn’t a yes-man. He pushed back plainly when he disagreed. But he leaned hard, always, toward whatever kept the business whole and satisfied, an instinct Tom understood the moment he learned where Nate had actually come from — not IT originally, not really, a background in sales and marketing before any of this, a brain that ran on a completely different track than most people at this table. It made him strange to work with sometimes, in the best way. He’d float an idea so far outside the lines that Tom’s first reaction was some version of we don’t have time for that, and half the time, sitting with it another five minutes, it turned out to be exactly right. He got frustrated more than most people on the floor ever saw, and Tom saw more of it than most, but he never once actually lost it in front of anyone. He’d walk it off instead, find Tom or Maya after a bad call, and let out the one sound that meant more from him than a full sentence would have from anyone else. Ay ay ay. Every flavor of it, exhausted, annoyed, disbelieving, somehow always came out exactly the same three syllables.

“NICE CXone is calling ivr-rs-v1 and getting nothing back,” Tom said, not looking up from his second monitor. “Timeout, not an error. Just nothing.”

“One route.”

“One route, one target. Same as always.”

Nate didn’t say anything to that. He didn’t have to. Every API call at Concordia — cloud to cloud, cloud to on-prem, it didn’t matter — was required to route through Apigee, the company’s API gateway.

“You’ve flagged this before,” Nate said.

“More than once. It’s policy — every call routes through Apigee, no exceptions. I never fought that part. What I’ve never stopped raising is that we’re not actually owning it. No high availability. No real observability. I pushed to bring the vendor in and close the gap properly. It stalled every time.”

“So when something breaks—”

“Apigee’s the first place I look. Not because I trust what I’ll find there. Because it sits in front of everything, whether or not anyone ever built it to tell you when it fails.”

He pulled up Datadog and started searching — his own team’s backend API logs first, the ones that fed the softphone and the CRM integrations, then Apigee’s proxy logs layered in right alongside them, everything that touched this chain, for the last ten minutes.

Nothing.

Not nothing relevant. Nothing. No errors, no warnings, no 5xx responses. No traffic at all, in either direction. Like the whole chain between here and wherever ivr-rs-v1 actually lived had simply stopped existing for the last ten minutes.

“That can’t be right,” Nate said, reading over his shoulder.

“It’s right,” Tom said. “That’s the problem.”

“So it’s not Apigee.”

“If nothing downstream of Apigee is even seeing traffic — not erroring, not timing out, just never arriving — then whatever’s broken isn’t broken anywhere I have visibility into.” Tom was already thinking out loud, half to Nate and half to himself. “It has to be sitting between NICE CXone and the Apigee proxy. The proxy, and the backend behind it — Architecture required both be stood up years ago, same mandate. Neither one’s ours.”

“Whose is it?”

“Core backend API team. Haven’t talked to them in months.” He was already reaching for his phone.

There was a version of the next hour that involved opening a ticket, waiting for Apigee’s platform team to triage it, waiting for someone in IT Data Management to confirm they’d even seen the alert — assuming there was an alert, which, given what he was looking at, he doubted. A version where he did this by the book. Filed the ticket. Waited for triage. Built a record that would look clean in an incident report later.

He didn’t do that. Something about the shape of it, a single unmonitored target sitting behind a gateway that couldn’t see past its own front door, snagged on a memory he didn’t have time to chase down properly, a conversation from a few weeks back that had ended in a silence he still hadn’t liked. He’d sort out later whether that was instinct or just recognition. Right now it didn’t matter which. It was fast enough to act on.

Instead, he picked up his phone and called a guy named Wesley, on the core backend API team — someone he’d worked with two reorgs ago and hadn’t spoken to in four months. Wesley’s team happened to own both the proxy and the actual backend ivr-rs-v1 pointed to. Tom happened to remember that.

“Hey. Random ask,” Tom said, when Wesley picked up. “Can you pull Datadog on your side for the last fifteen minutes? I think something’s timing out on your end and I can’t see it from mine.”

A pause. The sound of a keyboard.

“Oh,” Wesley said. “Yeah. Yeah, we’ve got a wall of these. JWT validation failures. Token expired at—” another pause “—12:58.”

“Expired, or rotated wrong?”

“Expired. Just expired. Like nobody renewed it.”

“Whose token. Somebody calling you, or you calling somebody.”

“Us calling somebody. It’s not inbound, it’s not Apigee’s side of this at all. It’s the token this instance uses to authenticate itself outbound, when it calls the other API behind it. The second that expired, every one of our own outbound calls started failing at once. Nothing coming in ever had a reason to fail. Everything going back out did.”

“How many instances are you running?”

“One.”

“One?”

“One. It was supposed to be phase two — stand up a second instance, put a load balancer in front, all of it. Phase two never happened.” Wesley’s voice flattened, someone realizing something in real time and not liking where it was going. “And the token on that one instance expired. So it’s down. The only one there was.”

Tom closed his eyes for exactly one second.

That was the whole incident, right there, in two sentences from a guy he’d called on a hunch. Not a partial outage. Not one bad instance silently compensated for while a healthy one carried the load. There had never been a healthy one to fall back to. There had never been a second instance at all. No load balancer, no failover path, nothing standing behind it. One target, carrying everything, since the day it was deployed, waiting for a clock to run out. And Apigee, sitting directly in front of all of it, had never been configured to notice. No health check against whatever was actually alive behind that VIP. No service discovery telling it which upstream was answering and which wasn’t. Just requests thrown forward and hoped for. It hadn’t logged a single line saying anything was wrong, because as far as Apigee could tell, nothing was. It had done exactly the job it was configured to do. Nobody had ever configured it to do the job that would have actually caught this.

“Okay,” Tom said. “Redeploy with a fresh token. I’ll get Apigee looped in so they know what’s coming back online.”

“On it.”

He hung up. The room came back into focus. Nate was still standing there, and Grant Cole had joined him at some point without Tom noticing, quiet enough that even a room this small hadn’t registered him arriving. Grant had a way of doing that — tall, lean, unhurried in a way that read as competence rather than distance, the kind of calm that made the people around him calmer just by proximity. Tom’s manager, and before that, someone this whole team had trusted for years, for good reason. Grant was, by any honest measure, one of the best managers Tom had ever worked for, sharp, fair, genuinely liked by nearly everyone who’d ever reported to him. He also hated conflict enough to avoid it even when avoiding it cost him something. Whatever his own manager wanted, Grant delivered, no pushback, unless somebody else pushed first and handed him a reason he could stand behind instead of building one himself. Both of them were watching him with the expression of people waiting to hear whether this was actually over.

“Fixed,” Tom said. “Four minutes, maybe five.”

Nate exhaled. Grant didn’t, not quite.

“How’d you find it?” Grant asked.

“I called a guy,” Tom said. “That’s it. That’s the whole system. I called a guy who happened to remember he owned the API, who happened to be at his desk, who happened to pick up.”

On his monitor, the self-service line held flat for another few seconds, then twitched, climbed, climbed again — the clean vertical drop from twenty minutes ago reversing itself in jagged little steps as the single instance came back online and started actually answering. Somewhere out past his cubicle wall, the floor’s ambient hum changed pitch. Fewer holds. More actual conversations.

The calls started clearing at 1:19. By 1:25, IVR Jail was empty again, queue depth back to its normal, unremarkable near-zero. Somewhere in a call center forty feet from Tom’s desk, an agent who’d spent the last twenty minutes fielding claim-status questions that should never have reached a human being was finally getting a break between calls.

Tom pulled up his email to write the incident summary, and stopped with his cursor blinking in the subject line. Before he started typing, he went looking for the conversation that had been nagging at him for the last twenty minutes, the one he hadn’t had time to place in the middle of it.

He found it. Three weeks old.