The Product Security Playbook
How to Burn Down a 5,000-Finding SCA Backlog
By Chad Butler ·
Three large-scale attacks have hit the npm ecosystem in the last 12 months. You’ve heard of Shai Hulud, Shai Hulud 2.0, and ChainDrop. The root cause of these attacks is more complicated than it looks, but it comes down to one principle.
You shouldn’t run code that you haven’t vetted, no matter how popular it is. And you shouldn’t allow automatic updates of that code without vetting it first. Most organizations can’t do either one. They can’t tell you what’s in their dependency tree, and they can’t patch the vulnerabilities they already know about.
Many of the organizations I have worked for or with are dealing with a large backlog of outdated and vulnerable packages. It doesn’t take much analysis to realize that if you can’t update your packages with known vulnerabilities, you probably aren’t in a position to effectively defend against the next software supply chain attack.
Scanning for those issues is easy. I have run programs using Snyk, Semgrep, Wiz, and Dependabot. The temptation is always to throw that work over the wall without thinking about it. Every time I’ve seen a security team take that approach, the developers push back hard.
Upgrading those packages is risky. And ensuring that the upgrades happen safely takes work. Manual work. And the work estimates are non-intuitive. The security risk rating does not determine how hard something is to fix. The difficulty comes from the type of upgrade (patch, minor, major).
I have worked with a number of SCA programs over my 26 years in the industry. The numbers below are synthetic, but they are proportional and realistic. In other words, I’ve seen numbers like these in every program I’ve run.
An organization runs an SCA scan. They ignore the low risk and informational findings and come up with a distribution as follows:
- Critical: 500 (10%)
- High: 1500 (30%)
- Moderate: 3000 (60%)
But the amount of work required for the development team is not driven by the security risk. The security risk tells you the potential impact if all of the exploitation pre-requisites are true and an adversary exploits it. The remediation difficulty comes from the upgrade factor. It is much easier to apply a patch level upgrade (e.g. version 1.2.3 to 1.2.4) than it is to apply a major version upgrade (e.g. version 1.2.3 to 2.0.0). The higher the version jump, the more potential breaking changes there are that you have to mitigate.
This organization categorized by upgrade complexity to arrive at the following:
- Patch level (1.2.3 to 1.2.4): 40% = 2,000
- Minor level (1.2.3 to 1.3.0): 35% = 1,750
- Major level (1.2.3 to 2.0.0): 25% = 1,250
Each pull request that upgrades one of these packages requires some checking. The estimates below include the time required to read release notes, search the codebase for problem areas, apply the changes, wait on CI, review test results, and approve the PR.
Per upgrade time estimates (in minutes)
| Complexity | Best | Likely | Worst |
|---|---|---|---|
| Patch | 10 | 25 | 60 |
| Minor | 25 | 60 | 180 |
| Major | 90 | 240 | 960 |
Here is the background on those estimates:
- Patch: The likely case requires minimal verification and checking tests. The worst case is the patch that shouldn’t have broken anything but did. Hopefully caught before deployment.
- Minor: Minor upgrades should be backward compatible, but that rule is frequently broken. So you can’t afford to skip verification. The likely case is an hour to read the change log, check for warnings on APIs you use, build and test. The worst case may require refactoring.
- Major: The likely case (4 hours) is reading (or creating) the migration guide, understanding the changes, grepping your code for every use of the changes, refactoring code, developing new tests, running full regression. The worst case is a deeply integrated package with an upgrade that forces peer dependency updates that quickly turn into an upgrade program.
If you extrapolate those estimates, you get a likely total of 5.3 years to process that backlog:
| Scenario | Total hours | Engineer-months (120 focused hrs/mo) | Engineer-years |
|---|---|---|---|
| Best | ~2,940 | ~24.5 | ~2.0 |
| Likely | ~7,580 | ~63 | ~5.3 |
| Worst | ~27,250 | ~227 | ~19 |
The good news is that there’s likely some duplication in those figures. But if you assume a 5:1 de-duplication factor, you still have a 13 engineer-month effort.
Go ahead and put those estimates and issue counts in front of the engineering team and let me know how it goes. Many of those packages have been vulnerable for years. If you have hundreds of critical risk vulnerabilities that have been in the code that long, are they really critical risk? Probably not.
How do you actually remediate these findings?
Across the repositories I’ve scanned, roughly 80% of those findings can be confidently ignored because they aren’t exploitable. They should still be upgraded, but the risk is lower.
The approach that works is equal parts marketing and technical work.
Reduce false positives
This starts by understanding the exploitation pre-requisites. Read the CVE and any linked information. Search for exploit PoCs, researcher writeups, and any info that didn’t make it into the CVE. Search through your code to determine if it applies. Your goal is to answer the following questions:
- Are we using the vulnerable version?
- Are we using the API/methods that are vulnerable?
- Are we using the API/methods in a vulnerable way?
- Is it reachable?
Here’s an example from one of my past programs. We had a critical CVE for a Python SSH library. I read the CVE and determined that the vulnerability was exploited through a listening SSH server. But, if you were using the package to make outbound SCP file transfers, the vulnerability wasn’t exploitable. It took about 10 minutes to confirm that we only used it for outbound SCP transfers. That was an easy suppression.
Share the good news
One of the best ways to gain trust with the development team is to save them work. Silently saving it earns you nothing. Tell them you dropped the backlog from 5,000 findings to 800. They will do the math.
Simplify
Group those findings by the affected package and share that message. In one of my past engagements, I was able to share that 90% of the Critical and High risk vulnerabilities were caused by 10 packages. That’s 90% of the risk retired by ten upgrade plans, applied over and over. Not bad.
De-risk
This is the part that largely falls to developers, but security teams can help. You need to understand the diffs between your current version and the upgrade version. Look for upgrade risks and breaking changes. Search your code to find problem areas. Those problem areas will probably require refactoring, new test cases, and a full regression to upgrade safely.
My co-founder and I built Lunir SCR (lunir.io) around this approach. We follow the process I described above and take the manual work out of it. It shouldn’t take years to determine what you can confidently ignore and how to safely fix what is left. We do it in minutes.