Application Security: Lessons in DevSecOps

Application security (AppSec) can be very daunting, especially for teams just staring to tackle it. There are countless tools available, but you may be asking “where do I begin”? Hopefully some of my experience will benefit you and get you on your way.
Successful AppSec cannot be merely a bolt-on task tacked on the end of the development workflow (DevOps + Sec). Developers must be fully engaged, and held accountable for writing and maintaining secure code (DevSecOps).
Over the last year, my security team has completely overhauled our approach to application security. This has yielded some significant wins, including improved timeliness in resolving security findings, developers having a better understanding of how their code directly impacts security, and most importantly, developers empowered to take ownership of application security by leveraging tools directly within their deployment workflow.
Application development is an iterative process. Development teams are primarily driven by deliverables. New features must be introduced and tested. Bugs are typically identified during testing and hopefully mitigated. The application update is finally deployed, and the process repeats itself.
Depending on the size of the organization and the amount of change control involved in deploying code into production, any bug can quickly introduce lengthy delays into the cycle. Not only does it take time for the developers to resolve the issue at the code level, but subsequent functional testing is also required to ensure no other issues have been unintentionally introduced. All this takes measurable time and effort.
Dynamic Application Scanning Tools
Previously, much of our AppSec effort was focused on dynamic application scanning tools (DAST). While they certainly have their place, which I’ll get into more later, a mature AppSec program needs a broader approach.
The first challenge we found with use of DAST tools, was that dynamic application security testing could not begin until the code had been deployed into a testing or staging environment. Typically, at this phase of the development cycle, the application has moved from the developers over to quality assurance for functional and regression testing. Resolving any security findings at this point requires moving the application back into development phase, thus restarting most, if not all, of the testing. This quickly becomes a time consuming road block to deploying the application, with potential negative impact on timelines and deliverables.
Other challenges include the difficulty and level of effort required to automate dynamic testing into the build tools. Most dynamic application scanners require some assistance stepping through the application, often with valid test data, depending on the nature of the application. Typically this is done either by manually exercising the application from a web browser, or developing automation scripts that walk the scanner through the workflow of the application.
Finally, and perhaps the most significant challenge we encountered, was related to understanding and communicating the nature of the security findings. Because the DAST scanner is testing the application externally, security findings are reported by the behavior that was observed. This can be difficult for developers to fully understand how the reported behavior relates to their code. We found in some cases hours and days were spent fully dissecting findings, replicating and demonstrating the behavior for the developers. We needed results that the developers could easily relate to.
Static Application Scanning Tools
Perhaps you’re thinking that static application scanning tools (SAST) are great, but you’ve already looked into some commercial products and they are cost-prohibitive. This can be true, but first let’s take a look at one free option that yielded tremendous results for us.
OWASP Dependency Checker
When writing modern web applications, developers no longer begin with a blank slate writing code. Typically web application frameworks, such as Spring, Grails, etc are used as a foundation to build the application upon. This means that developers no longer need to write code for functionalities common to most applications, (for example user authentication, parameter validation, etc). These reusable functions, often thousands of lines of code, are contained within the framework that the developer can easily integrate into their new application as a code dependency.
The good news for security folks, is that because of the prevalence of these application frameworks, there are many eyes constantly evaluating them, discovering and reporting vulnerabilities.
The OWASP Dependency Checker tool focuses on OWASP top 10 risk A9: Using Components with known vulnerabilities. It identifies dependencies in Java and .NET applications, checking them against vulnerability repositories to determine if there are any known, publicly-disclosed vulnerabilities. These findings are especially present when developers revert to linking in older framework libraries, without obtaining the latest version release. Often resolving these dependency findings is as easy as updating the application with the latest framework versions.
We found that the effort of ensuring that all frameworks and third-party libraries are up to date has resulted in substantially fewer security findings from both DAST and SAST tools.
Once third-party code has been evaluated with this free OWASP tool, results from DAST and SAST tools will now focus primarily on findings related to code written by the application developers.
Preventing an Equifax Breach – Some security shops attempted to use host vulnerability scanners to detect for Apache Struts. This proved to be difficult and unreliable at best, because Apache Struts is a framework incorporated into application code and not a server application component such as Apache web server or OpenSSL.
The OWASP Dependency Check tool was perfect for quickly evaluating all of our application source code repositories to determine if the vulnerable Apache Struts dependency was present within any applications.
Commercial SAST Tools
While third party frameworks and libraries constitute large percentages of applications, it is still critically important to be testing the custom code that developers are writing into their applications. This is where commercial SAST products play an important role. Many of these tools look for vulnerabilities against all of the OWASP Top Ten risks, including SQL or OS command injection, faulty authentication or session management, cross-site scripting, and other security flaws.
As noted above, a significant benefit we found from implementation of SAST provided findings that were reported in was that were meaningful to developers, in language they understand. For example, looks at this highlighted line of code, and make this change to address a specific vulnerability. While some false positives still existed, conversations around the finding are abbreviated as the developers have a much better understanding of what the potential issue is, and where it exists within their code.
Automation
In order for modern security tools to keep up with the pace of DevOps and rapid deployments, we must effectively integrate automated security into the SDLC workflow and application deployment process. By introducing these tools as additional steps in the automated build process with tools such as Atlassian Bamboo or Jenkins, security scans can be completed each time a developer checks code back into the repository and runs a build. The build can even be configured to fail if a clean dependency check or static scan is not reported.
Integration at this level is easily understood by developers. They already have a good understanding of the build process, and they are getting feedback from the security tools while they are still actively engaged and working on their app changes.
Many SAST products also offer deeper integration and automation features, such as integration directly within the developers’ IDE environment. This integration offers evaluation of code as the developer is writing it, not just at build time.
SAST IDE integration provides immediate feedback to the developer with suggestions for resolving findings, which can also server as ongoing secure coding education for the developers.
Summary
To accomplish true DevSecOps, and for application security to be effective, security testing must be automated into the deployment workflows. In my experience, static based tools such as the OWASP Dependency Check tool and, if budgets allow, commercial SAST scanners, are ideal candidates for automation. They only require access to application source code, and do not require automation scripting or test data which complicates the automation efforts.
Before you throw out DAST tools completely, they are great as an additional layer for conducting more thorough testing against web applications. However, realize that they can often be difficult to automate into the build process. For penetration testing, or other focused testing that allows for manual interaction with the application, DAST tools certainly provide worthwhile benefits: both in catching vulnerabilities that static tools may have missed, or in validating whether a static finding is indeed a false positive, or non-exploitable in certain circumstances.
If you don’t know where to begin, start small. Start running some dependency checks on your source branches. You may be surprised by the number of vulnerable libraries that are reported. Once the development teams are accustomed to keeping their dependencies up to date, consider conducting a proof of concept evaluation on a commercial SAST product.
You may find that the benefits easily justify the budget, especially if you can demonstrate a significant win from the dependency check effort.
Whatever steps you take, you will begin to get your arms around your overall application security posture and have a better understanding where you should be focusing efforts.