Employing DevOps in Classified Environments (Part 1)
Organizations of all sizes are increasingly deploying Development Operations (DevOps) tools to enhance productivity, streamline workflows, reduce time to market, deliver better quality software, and minimize costs associated with the software development life-cycle. Deploying and utilizing open-source and enterprise DevOps tools in classified environments where access to the internet is restricted, additional security certifications are required and cleared DevOps engineers are needed adds a level of complexity that many are unwilling to tackle. This case study describes Ascolta’s experience in using the latest DevOps tools successfully in support of a U.S. Air Force contract on Secret and Top Secret Department of Defense (DoD) networks.
Before we tackle the challenges associated with deploying DevOps solutions in classified environments, let’s establish why DevOps is important in these environments in the first place. With a team composed of cross-functional members working in collaboration, DevOps organizations can deliver with maximum speed, functionality, and innovation. The Technical benefits include improved infrastructure, security, continuous delivery, less complexity, and faster resolution of problems. Cultural benefits include happier, more productive teams, higher employee engagement, and greater professional development opportunities. Finally, the mission and operational benefits include faster delivery of features, more stable and scalable operating environments, improved communication and collaboration and more time to innovate rather than fix and maintain.
As in most problems, the solution involves aspects of people, processes and technology. In this and future blogs we’ll be examining the challenges and solutions associated with each. This week we’ll start with what I believe is the most important, people.
Organizations trying to implement DevOps solutions face problems of finding, training and keeping skilled DevOps engineers; finding DevOps Engineers with active DoD clearances is even harder. According to a March 2019 Federal News Network story about 103,000 federal employees and contractors are waiting for an initial background investigation. In industry alone, 37,000 people are waiting for a secret clearance, and 25,000 are waiting for a top secret clearance. In general, expect a confidential or secret clearance to take between one to three months and a top-secret clearance to take between four to eight months. However, some individuals have been waiting for their top secret/sensitive compartmentalized information (TS/SCI) clearances for more than a year. Workers that already possess a clearance are difficult to find and more expensive to hire. In addition to an active security clearance the DoD in many cases, based on individual Service or Agency, requires that engineers that access – hands on keyboard – classified computer systems meet the requirements established in DoD Manual 8570.01-M, Information Assurance Workforce Improvement Program for Information Assurance Technical (IAT) personnel. Generally, this entails the individual maintaining an IAT level II status which requires initial training, an IA baseline certification (i.e. Security +, CCNA Security, GICSP, etc.), and annual continuous education.Anecdotally, out of a pool of 100 qualified DevOps engineers maybe ten either have a clearance or are clearable. Of those ten remaining, two have the required IA qualifications, and if you’re lucky, one of them is looking for a new job and fits your budget.
How to overcome these challenges: In most cases involving classified contracts, the production data is classified, the software is not. Uncleared DevOps engineers can be utilized to build and test solutions, and a smaller number of cleared DevOps engineers can deploy and configure them on classified networks. To solve this problem for our Air Force customer Ascolta maintains a mix of TS/SCI cleared Developers and DevOps engineers on staff with the requisite IAT training. Our matrixed staffing approach allowed us to assign personnel where needed and transition smoothly between unclassified development environments to classified operational environments.
Stay tuned for next week’s blog where we’ll address processes.