Evaluating Open Source Software#
FOSS software is not automatic more secure that commercial software. But having the source code is available means it is easier to evaluate security aspects.
As a software developer, before using open source software (OSS) dependencies or tools, identify candidates and evaluate the leading ones against your needs. To evaluate a potential OSS dependency for security and sustainability, consider these questions (all tools or services listed are merely examples):
Can you avoid adding it? Can you use an existing (possibly indirect) dependency instead? Every new dependency increases the attack surface (a subversion of the new dependency, or its transitive dependencies, may subvert the system).
Are you evaluating the intended version? Ensure you are evaluating the intended version of the software, not a personal fork nor an attacker-controlled fork. These techniques help to counter the common “typosquatting” attack (where an attacker creates an “almost-correct” name).
Check its name and the project website for the link.
Verify the fork relation on GitHub/GitLab.
Check if the project is affiliated with a foundation (in this case, you should be able to access the official source from the foundation’s website).
Check its creation time, and check its popularity.
Is it maintained? Unmaintained software is a risk; most software needs continuous maintenance. If it’s unmaintained, it’s also likely to be insecure.
Has significant recent activity (e.g., commits) occurred within the last year?
When was its last release (was it less than a year ago)?
Is there more than one maintainer, ideally from different organizations?
Are there recent releases or announcements from its maintainer(s)?
Does its version string indicate instability (e.g., begin with “0”, include “alpha” or “beta”, etc.)
Is there evidence that its developers work to make it secure?
Determine whether the project has earned (or is well on the way to) an Open Source Security Foundation (OpenSSF) Best Practices badge.
Examine information on https://deps.dev, including its OpenSSF Scorecards score and any known vulnerabilities.
Determine whether the package dependencies are (relatively) up to date.
Determine whether there is documentation explaining why it’s secure (aka an “assurance case”).
Are there automated tests included in its CI pipeline? What is its test coverage?
Does the project fix bugs (especially security bugs) in a timely manner? Do they release security fixes for older releases? Do they have an LTS (Long Time Support) version?
Do the developers use code hosting security features where applicable (e.g., if they’re on GitHub or GitLab, do they use branch protection)?
Identify security audits and whether any problems found were fixed. Security audits are relatively uncommon, but see OpenSSF’s “Security Reviews”.
Use SAFECode’s guide Principles for Software Assurance Assessment (2019), a multi-tiered approach for examining the software’s security.
Is the current version free of known important vulnerabilities (especially long-known ones)? Organizations may want to implement the OpenChain Security Assurance Specification 1.1 to systemically check for known vulnerabilities on ingest and as new vulnerabilities are publicly revealed.
Do they apply many practices in the Concise Guide for Developing More Secure Software?
Is it easy to use securely?
Are the default configuration and “simple examples” secure (e.g., encryption turned on by default in network protocols)? If not, avoid it.
Is its interface/API designed to be easy to use securely (e.g., if the interface implements a language, does it support parameterized queries)?
Is there guidance on how to use it securely?
Are there instructions on how to report vulnerabilities?
Does it have significant use? Software with many users or large users may be inappropriate for your use. However, widely used software is more likely to offer useful information on how to use it securely, and more people will care about its security. Check if a similar name is more popular - that could indicate a typosquatting attack.
What is the software’s license? Licenses are technically not security, but licenses can have a significant impact on security and sustainability. Ensure every component has a license, that it’s a widely-used OSI license if it’s OSS, and that it’s consistent with your intended use. Projects that won’t provide clear license information are less likely to follow other good practices that lead to secure software.
What are the results of code evaluation? Even a brief review of software (by you, someone you hire, or someone else), along with recent changes to it, can give you some insight. Here are things to consider:
When reviewing its source code, is there evidence in the code that the developers were trying to develop secure software (such as rigorous input validation of untrusted input and the use of parameterized statements)?
Is there evidence of insecure/ incomplete software (e.g., many TODO statements)?
What are the “top” problems reported by static analysis tools?
Is there evidence that the software is malicious? Per Backstabber’s Knife Collection, check the installation scripts/routines for maliciousness, check for data exfiltration from ~/.ssh and environment variables, and look for encoded/ obfuscated values that are executed. Examine the most recent commits for suspicious code (an attacker may have added them recently).
Consider running the software in a sandbox to attempt to trigger and detect malicious code.
Consider running all defined test cases to ensure the software passes them.
See the OpenSSF’s list of security reviews.
Other resources you may wish to consider include:
The Tidelift guide to choosing packages well (February 2021), Tidelift
How to Evaluate Open Source Software / Free Software (OSS/FS) Programs
Attribution: This text is derived from the this OSSF repository on github.