I've always been a huge fan of Reddit. It's real people talking about real things. You follow the topics you care about and you often hear the news there before anywhere else.
So on Saturday, September 26, when I started seeing chatter about an unpublished NetScaler® vulnerability, I had to dig in. The more I read, the more interested I got. Organizations were shutting down their NetScalers, and the vendor hadn't said anything yet.
That evening I was at the wedding of one of my favorite people. Between the toasts and the dancing, I kept checking my phone for anything new.
Getting in front of it
By Sunday morning it was clearly heating up. I opened a conversation with Claude and asked a simple question: how do we get in front of this, troubleshoot it, and do real-time incident response?
More information came out over the day. Some sources estimated up to eight vulnerabilities, and at least a few were already being exploited. Citrix® soon confirmed them, published a security bulletin, and released fixed firmware.
That bulletin, plus everything the security community was posting, told us what to look for. An initial IOC script had already been published, and I started there. Then I pulled in other public sources and began building our own check script. It runs live on an appliance, and it can also read collected log files offline to confirm whether a box was hit.
Following the evidence
The NetScaler show tech command collects a full bundle of logs and configuration. We gathered these bundles from our own environment and from our customers. Patterns showed up quickly: the same attack methods and the same source IP addresses, appliance after appliance.
From that, two things grew together:
- A firewall blocklist of attacker infrastructure, which kept growing.
- The check script, which got more thorough with every bundle we reviewed.
Others online were writing their own scripts and finding new things to look for. Whenever someone published new indicators or a new script, we reviewed it and folded in anything useful. Some days we released a new version every 10 to 15 minutes. It felt less like a support case and more like a fast, live development project.
Everywhere we ran the script, the output showed more of the same patterns. The picture became clear: attackers were planting code in the logs. A routine process on the appliance would then reread those logs and run that code. Citrix's patch closed that hole. On an unpatched appliance, though, the planted code was a time bomb.
Sharing what we learned
This went on for days. We put everything (scripts, blocklists, change log) in a shared folder for our customers and for anyone who asked. Our peer community, M7 Global Partners, was great. Members shared data, tested the scripts in their own environments, and helped make them better.
It was rewarding to see the community build on each other's work. Independent researchers folded some of what we found into their public checkers and credited Gotham for it and we did the same with theirs. In a week like this, nobody has the whole picture, but together we got close.
The second wave: forced reboots
Then a new problem appeared. NetScaler Gateways started crashing and rebooting, including fully patched ones. Someone was sending a crafted sign-in request that crashed the authentication service, and repeated crashes forced the appliance to restart.
Here's why that mattered. One of the original vulnerabilities ran the planted log code when the appliance booted. My working theory is that attackers were planting a payload in the logs and then forcing a reboot to set it off right away. On patched devices, the effect was mostly an outage. On unpatched devices, we don't really know the full impact. As far as we knew, every customer we work with had already patched.
We updated the check script to detect the crash pattern. We also wrote a new script that builds a mitigation for each appliance. Citrix was working on its own guidance at the same time. We combined their approach and ours into one script: a configuration backup, IP blocking with ACLs, responder policies to drop the malicious requests, and a matching rollback. As soon as customers applied it, their appliances stabilized. That was a great thing to see.
What I took away from this
This week opened my eyes. I'm an infrastructure guy. I wouldn't have expected to determine whether systems were compromised, write mitigations, and run incident response across several organizations in a week. With Claude working alongside me, I could go well beyond what I knew on my own.
There's an irony here. AI was likely used to find these vulnerabilities and develop an exploit. It's also amazing to see how we can use AI to protect ourselves and to become more resilient against the attackers out there.
Whenever we build something with AI, the most rewarding part is seeing it used and knowing someone else got real value out of it. That's also how we keep humans in the loop. AI did a lot of the heavy lifting, but the ideas, the judgment calls, and the direction came from people: our own experience, customers sending us their logs, and a community comparing notes in real time. We guided it, corrected it. and pushed it in new directions. Together, we got to a better place than any of us would have reached alone.
Need help checking or protecting your NetScalers? Contact your Gotham Account Manager or email us at info@gothamtg.com.
Per trademark guidelines: Citrix and NetScaler are registered trademarks of Cloud Software Group.