DevSecOps with GitHub
A DevSecOps architecture built on top of GitHub Actions and Advanced Security. The key objectives of this software development solution are to improve developer velocity, “fail-fast”, and “shift-left”. Project/product managers, delivery leads, and tech leads are the perfect audience for this article. It is not a “One size fits all” solution, but it could give you some relevant inspiration on land DevSecOps in your organization.
We strongly recommend you to read and use the official DevSecOps architectures from Microsoft and GitHub as reference. This article will focus more on the developer point of view and how DevSecOps should integrate with your branching strategy.
Prerequisite
This article is intended for an audience of business and technical people who are already clear on why they need DevSecOps and are looking for a possible architecture to implement it. We assume there is no need to explain why security is essential in today’s landscape (“Global cyber attacks increased by 29%, as hackers continue to exploit the COVID-19 pandemic and shift to remote work”, Check Point).
We assume you are proficient in the following concepts/technologies:
- DevOps and DevSecOps
- git, branch, GitHub
- Container, docker and Azure Container Registry
- Project management tools for software development and Azure Boards
- PaaS, compute service in Azure and App Service
- OWASP
- [Recommended] V for Vendetta
Objective
In late 2021 majority of companies developing software have implemented some DevOps. The most common challenge in becoming a mature DevOps organization is improving developers productivity by working on developer velocity and including security as part of the CI/CD.
One of my key takeaways from my Accenture days is to do your best to discover a bug, a vulnerability or a design gap as soon as possible during the development lifecycle. And the best moment is, of course, during the design phase.
Our objective translates into three significant challenges: improve developer velocity, “fail-fast”, and “shift-left”.
Improve developer velocity by automating everything (code build, scan and deployment) provides excellent benefit to your organization like up to 200 times more frequent code deployment, up to 100 times faster to set up a dev environment, up to 7 times less few failures on deployments, and so on.
The point of fail-fast in DevOps is not to maximize failure but rather to encourage development teams to experiment in a structured environment. Thus, the quicker they fail, the faster they can discover ways to improve systems and products.
By using a shift-left strategy, DevSecOps redirects the security focus. Instead of pointing toward auditing at the end, it shifts to development in the beginning. Besides producing robust code, this fails fast approach helps resolve problems early on when they’re easy to fix. Try to imagine the difference in cost between fixing a vulnerability discovered in development vs during a breach?
Bonus 1: Open-source security
As measured by McKinsey, the top-performing organizations showed that open-source adoption is the biggest differentiator. These organizations are seeing three times more impact from adopting open source than the rest of the industry.
When decomposing most organization software, up to 95% of the code comes from the open-source community. Most of the time, the security scanning focuses on the code that your company is writing and not on the millions of lines that you might include with a third-party library. An additional objective of this architecture is to secure and scan all your software and not only the code that you write.
Bonus 2: Hacker don’t sleep
In several companies, the security scan is performed between UAT and production; then, vulnerabilities are fixed and (maybe) the product is secure today. What about tomorrow or in 3 months? What if nobody is performing any scan periodically?
Our architecture should periodically scan the code, find vulnerabilities and alert the development team.
Components
The architecture that we will introduce uses several components that you can discover by checking below definition and material:
- GitHub package / release
It is the GitHub public (or private) repository for package, libraries and applications, similar to other products like Jfrog, npm or Maven; key tool to improve reusability and modularity in your software. - GitHub Action
The engine that is enables your CI/CD to work, similar to Jenkins or Azure DevOps - GitHub Advanced Security
It requires an additional license. Advance Security provide features to improve and maintain the quality of your code in three major areas:
Code scanning — Search for potential security vulnerabilities and coding errors in your code.
Secret scanning — Detect secrets, for example keys and tokens, that have been checked into the repository.
Dependency review — Show the full impact of changes to dependencies and see details of any vulnerable versions before you merge a pull request. - GitHub — Azure Boards integration
By connecting Azure Boards with GitHub repositories, you enable linking between GitHub commits, pull requests, and issues to work items. You can use GitHub for software development while using Azure Boards to plan and track your work. Azure Boards provides the scalability to grow as your organization and business needs grow. - SonarCloud
SonarCloud’s static analysis detects Bugs and Code Smells in your repositories and provides the feedback you need to write better code.
Each component plays a critical role in our architecture; We recommend reading more about those components you are not familiar with.
Everything should be made as simple as possible, but not simpler.
- Albert Einstein
A DevSecOps Architecture
After talking with several customers, we saw the need to envision and present a DevSecOps Architecture that could leverage several Microsoft products and bring the best from each.
At the heart of this architecture, GitHub (the Enterprise Cloud license or the Enterprise Server license are prerequisites to leverage on GitHub Advanced Security) is where your code lives.
The GitHub repositories are integrated with Azure Boards to facilitate project management by linking commits and PR with tasks or bugs and your favourite IDEs to simplify the developer experience while committing his code. In addition, GitHub Package allows a developer to write reusable modules that can be referenced within their organization.
When GitHub Advanced Security (GHAS) is enabled, it automatically scans your repository over time. In addition, GHAS created a dependency graph of your code and checked that all your dependencies were not exposing vulnerabilities. If this is the case, GHAS dependabot will make a PR to upgrade your libraries.
The architecture shows an example of CI/CD for containers enabled by the GitHub Actions to configure workflows to build, test, scan and deploy your code. That activity can run in parallel or sequence, and it can be configured in multiple ways based on your organization needs. In the following section, “DevSecOps in action”, we will deep dive into our recommendations to implement an efficient branch strategy & CI/CD.
GitHub Action offers a marketplace with more than 10,000 available tasks for your code, but don’t worry if you need any specific command. You can always leverage on the Linux bash or Windows PowerShell.
The GitHub Advanced Security Code Scanning is triggered during the CI, and it runs on top of GitHub Actions. Code scanning is a feature you use to analyze the code in a GitHub repository to find security vulnerabilities and coding errors. Any problems identified by the analysis are shown in GitHub Security tab. You can use code scanning to find, triage, and prioritize fixes for existing or new issues in your code. If code scanning finds a potential vulnerability or error in your code, GitHub displays an alert in the repository. After fixing the code that triggered the alert, GitHub closes the alert.
When a vulnerability is found, GitHub will help you understand it (figure 3) and also recommend how to fix it with coding examples (figure 4).
DevSecOps in action
Every organization might have different processes or needs when talking about DevSecOps based on your building, testing and deploying strategies.
In figure 5, we are illustrating how a branching strategy based on a rolling master branch can help your project/product to:
- Build a container image only once and then promote it across environments
- Map your live production environment with a specific branch (release)
- Avoid that fix are overwritten by a new feature merged in the master branch
- Avoid that new features are deployed with the last-minute production bug fix
The rolling master branch requires a branch manager who will create new release branches, protect them and ensure that the branches are periodically merged from an older release to more recent and never in the opposite direction.
The branch strategy should be based on a strict naming convention that the workflows will map to trigger CI, CD or security scan:
- Release/R* :
the release branch where all the features are merged and designed to become the main branch when his code is deployed in production - Features/F[0–9]+ (exclude Features/F*/*):
The feature branch of a feature. It is used as a collaboration branch between developers - Features/F[0–9]/Item[0–9]+:
Branch created by a developer to implement a task related to a feature - Hotfix/*:
Branch created for production hotfix.
In figure 6, we illustrate a possible way of how the DevSecOps workflows should be configured for the different branches. We will explore two scenarios:
1) Release/R* :
When a PR is created targeting this branch, several workloads are triggered to build the code and run tests, analyze the code quality, validate that the overall quality didn’t drop, and finally run the code scanning for static analysis of the vulnerability. You should configure to prevent merging the branch if any check fails (figure 7). The PR will require two reviewers to be merged.
As soon as the code is merged in this branch, automatically (CI), it should build the image and upload it to container registry. When a new container is available, it will scan it and deploy it (CD) to the development environment. After the deployment an automated pentest validation is recommended. The promotion of the container to any other environment can be manual or automatic (after passing a UI or API testing).
GitHub allows created environments owners (i.e. testing lead) who can approve or deny deployments to their environments, as shown in figure 8.
2) Features/F[0–9]+:
When a PR is created targeting this branch, there are several workloads that you can trigger to build the code and run tests, analyze the code quality and run the security scan. Some might fail since you are still in development. The PR can require just a reviewer to be merged.
As soon as the code is merged in this branch, automatically (CI), it should build the image and upload it to the container registry. When a new container is available, it will scan it and deploy it (CD) to a dedicated development environment for this feature.
How does it work in practice?
Several different types of events can trigger GitHub Actions workflows. In the above branching example, we used only two events: A and B from figure 9.
The branch manager will need to configure those pipeline files properly for each pipeline.
In figure 9, you can also see case C that will help you to trigger a security scan every X days or Y weeks based on your configuration. In this case, we start a security scan every Wednesday at 11:42 AM.
A developer day
Let’s do a bit of role-play here. Let’s assume Evey Hammond is one of your junior javascript developers. Evey’s day is illustrated in Figure 10.
Evey starts her day with Azure Boards. Her manager assigned her to fix a vulnerability discovered in her code by CodeQL (the GitHub Advanced Security AI for code scanning).
She opens the link inside the Azure Board Task, and she jumps to GitHub Security, where she learns how to fix the vulnerability (Figure 3 and 4).
She created a new branch called Features/F653/Item778, and she developed her task in Visual Studio Code. Every time she commits her code, she adds a reference to the item in the commit message AB#778, and the commit appears in the item history inside Azure Board (figure 12).
When she creates the PR and share the link with her manager for approval, her job is over. The PR will trigger all the validations (please check DevSecOps in the action section). And when the PR is approved, the task will be automatically marked as completed.
Conclusion
We have seen as this GitHub DevSecOps architecture (Figure 1) helps automate your software development lifecycle by providing a great engine to build, scan, test, deploy and secure your code from day 1 of your development journey. A similar architecture implemented in your organization will enable your team to experiment more and rely less on people for Ops. The effort required to implement CI/CD pipeline quickly pays back in terms of productivity, time to market, and cost savings for system administrators.
The great deal about this architecture is consolidating everything under one single service (GitHub) that acts as a dashboard and orchestrator for several different services that usually require context switch and complex integrations.
DevOps is not a Goal, But a never-ending process of continual improvement
Jez Humble
(Author “The DevOps Handbook, Lean Enterprise”)
Special thanks to Adrian Lee and CoderDave for answering all my questions on GitHub.
About the author
Since 2012, Francesco managed seven mobile projects as a Mobile Lead or CTO. He covered several industries, from banking to lifestyle. Francesco led teams from one to seventy developers on four continents. He used technologies such as JavaScript, PhoneGap, Objective-C, Java, Kotlin, Swift, ReactNative and Flutter in his project. His most successful project has almost 8 million active users a month.
Today, Francesco is working as Cloud Solution Architect for one of the major cloud providers, providing advice and supporting the customer journeys on application innovation and developer velocity.
Disclaimer
This article reflects my personal opinions, and it should not be attributed to the view of any of my past or present employers.
