Wait One Damn Moment, Please

Wait One Damn Moment, Please

By Ed Bratter
Posted in Infrastructure
On April 03, 2020

Just wait one damn moment, please.

I wonder how many of my fellow IT professionals have experienced some variation of this: You walk into the office on a sunny morning thinking about the tasks you need to do for the day. Before you get to your desk, you run into your manager, who says she needs you to attend a meeting in the conference room in 30 minutes. You grab a cup of coffee, do a couple of quick things, and stroll down the hall wondering what this could be about. You walk into the conference room, where your manager, a project manager, and a couple of strangers are seated. You are introduced to these strangers who work for a company named CoolCo. Coolco is a software development company that wrote an application that the business wants to deploy. You are informed that this shiny new app (let’s call it SoSmart) has an aggressive deployment timeline. At that point, the project manager hands you the project plan that CoolCo prepared that maps out the tasks and milestones to deploy SoSmart. You read down the list and something jumps out at you. You see a single task that requires that a supporting system be deployed that is a dependency for SoSmart. At that point you should think to yourself – wait one damn minute!

Ok, perhaps I embellished a little – but here is the point: As a consultant, I see a disturbing trend that as IT professionals and the gatekeepers of our fiefdoms we need to put the brakes on, or not willingly participate in. That trend is rapidly deploying one system without proper design and thought about what is a dependency for supporting that shiny new app the business wants. I have seen this happen all too often particularly with Active Directory Certificate Services, Active Directory Federation Services (ADFS), and SQL Server – my life revolves around the Microsoft mother ship. As you probably know, all three of these products, as standalones, do nothing useful, and are always deployed to support other applications.

Here are two of my favorite examples (true stories) of how a situation like this played out.

We were preparing to do a Skype for Business (SfB) deployment at a financial services firm. I was in a meeting to discuss the requirements to deploy SfB. In attendance was someone from the security team. As I was going down the list of requirements, I mentioned that it was common to utilize an internal PKI (Public Key Infrastructure) to generate certificates for SfB. At that point, the security person interrupted me and said the organization did not have an internal PKI and that he did want to implement one. His concern was that a PKI can have significant security implications if not deployed properly. Proper deployments can be costly and introduce operational challenges. I understood his concerns and told him that several third party certificates could be purchased as an alternative to support SfB. Life was good.

Fast-forward to the deployment phase. As I was preparing to kick off the deployment, I discovered that there was a Microsoft Certificate Authority integrated into their Active Directory infrastructure. I immediately went back to the individual who said they did not have a CA to inform him that there was, indeed, a CA in place. At first, he did not believe me. When I provided the proof of said CA, he was furious (though not at me). I was instructed to stay on plan and not use it. Several days later he came back to thank me for discovering the presence of the CA and reporting it to him. It turned out the CA had been installed by a vendor who needed it for a system they were deploying. Of course, I asked him what he was going to do with the CA. His response was that the vendor was going to have to rectify the situation. I told him that decommissioning a CA can be a difficult proposition because once a CA is deployed; many systems will discover it and pull certificates automatically. He was aware of that and told me that was one reason why he would not authorize installing one in the first place.  

In another example, I was working with a client who was having trouble setting up something that pointed to an issue with Active Directory. As I started to dig in, I discovered that the domain controller that hosted the five special roles (FSMOs) that AD environments require was not accessible to a handful of DCs located in remote offices. I asked if there were firewalls within the internal network. I was told no. After further digging, I discovered this DC was sitting in the company DMZ – and it was indeed firewalled off from the DCs in the remote offices. At that point I inquired why a DC (which also was hosting the FSMO roles) was needed in the DMZ. I was told it was there to support an ADFS deployment. Now the sirens were deafening in my head. I inquired what ADFS was being used for and if I could take a look. After doing so, I discovered that there was a single ADFS server installed on that same DC that also hosted the Web Application Proxy role (WAP). For simplicity, I will describe this ADFS deployment as follows: !@#$%^&*!. It turned out ADFS was deployed as part of a project to allow external access to Citrix ShareFile.

Both these examples illustrate how two supporting systems were hastily deployed in an environment to satisfy a task in a project plan that was developed to roll out that really CoolApp. As such, these systems were deployed in a manner that was problematic on many fronts and extremely risky from a security perspective.

So, going back to the encounter above – here is what you should do: pick up the closest chair, hurl it across the room and ask for a sample of what the people who wrote the project plan are smoking. OK, perhaps that’s a little overly theatrical (but it would be fun). So, try this instead: I think we need to pull that task out of the project plan and create a supporting project that addresses properly implementing that supporting system. To show off, maybe ask: What are the requirements for the supporting system? Does it need to be highly available? What is the business continuity plan for it? What is the RCO? What is the RTO? You get the drift.

Now grab your swords and go fix that troubled system!

Ed Bratter

Ed Bratter

Ed has over 15 years’ experience in the IT industry as a Systems Consultant, Systems Engineer, and Technology Specialist. He architects, designs, and manages Active Directory, Exchange, Citrix, VMware, and RSA SecurID solutions for Gotham’s clients, and provides technical expertise for Active Directory, Exchange, and Citrix.