How Buyers Can Use Performance Measures to Drive Better Security in Software Products
Every modern organization relies on software and systems that it cannot create for itself. Whether that be small office automation software, larger enterprise applications, or hundreds of Internet-based services, the organization depends on that software to meet its business goals. The number of applications in use in a typical company has expanded significantly in the last years: in 2023, the typical company used 112 software-as-a-service applications, up from 16 in 2017. Moreover, artificial intelligence (AI) systems typically include not only parameters (i.e., model weights) and data, but also software, and so software security risks can be magnified as AI systems are used to automate increasingly complex tasks – for example driving.
This article expands on a recent Lawfare blog that explains how a software buyer “window sticker” could be distilled from multiple best practice frameworks and verification information into a single, repeatable view of a product’s overall security performance. It works to map theory to practice – applying different risk types (malicious software, vulnerabilities, leaked secrets) and practical implementation challenges to explain the pros and cons of individual product security evaluation factors.
To frame out these pros and cons, we need to establish some underlying context – these include: (1) the nature and complexity of software, and (2) what the software development lifecycle looks like. A few key points are explained here and a more comprehensive treatment can be found in Software Supply Chain Security, by Schneider Electric Vice President for Supply Chain Security Cassie Crossley.
Nature and Complexity of Software. We need to know a few things about software attributes (type of software and relevant language) to understand threats and defenses in an actionable way. Developers use various human-readable programming languages to develop software – some of the most popular are C, C++, JavaScript, HTML/CSS, SQL, Python and Java. In human-readable form, the code behind a piece of software is referred to as “source code.” And source code can be made up of open source, commercial or proprietary code.
- Open Source. To streamline development, most developers today supplement their own hand-crafted code with code downloaded from open-source libraries. Open source code is software developed and maintained, mostly by volunteers, “via open collaboration, and made available, typically at no cost, for anyone to use, examine, alter and redistribute however they like.” Some of the most popular open source ecosystems include Maven for Java, npm for JavaScript, PyPI for Python, NuGet and Ruby Gems.
- Commercial. Commercial code is software purchased for use within a larger software product. Commercial code can and often does include open-source code.
- Proprietary. Proprietary code is code custom-developed by the software vendor in question.
Modern software has grown massively both in volume and complexity. The Space Shuttle’s flight control system required 2 million lines of code. Goldman Sachs has predicted that automobiles will require as many as 650 million lines of code per car in 2025. Most developers are utilizing open-source software as ingredients into their codebase. Software security firm Synopsys reports not only that 96 percent of all codebases scanned by its tools contained open source code, but also that open source represented on average 76 percent of the entire codebase. Thus, one basic question around security practices is whether they can handle both the scale and complexity of the codebase in question.
Software Lifecycle: From Source to Package. For the most part, machines do not process code in human readable form. Instead, source code needs to be compiled or interpreted into “machine language,” which is what a computer’s processor understands. These are low level languages expressed in “binary” form, like zeros and ones. Machine code is reflected in executable files, which is what computers actually run when a person uses software. Further, binary code “packers” often further compress or encrypt code to reduce its size and protect it from tampering.
The end-result is a software package, which is defined as “an identifiable unit of software intended for distribution.” Software packages include executable code (both the core application itself as well as installation functionality) and artifacts (configuration files, read-me help files, supporting data sets for artificial intelligence software products, etc.). According to research by Reversing Labs, post-build artifacts can increase the size of the total software package from 10 to 1,000 times or more with many different file types compared to the code alone.
The journey between source code and a software package also involves intermediate processing steps, in particular merging multiple open source, commercial and proprietary source code libraries together through software “build” processes.
Product Security Good Practice
So what does good practice look like in the context of secure software product development? The illustration below offers a food production analogy (expanding on one offered by the Supply Chain Levels for Software Artifacts framework (SLSA) described later in the article) to summarize secure software lifecycle practices.
Spices and other ingredients (akin to open source and commercial software components) are acquired, inventoried (on a bill of materials) and stored securely; they are then manufactured into a food product in clean, secure factories (akin to a software build environment) and distributed to consumers in BPA-free packages (akin to the lack of harmful artifacts and ingredients introduced during the delivery phase) with tamper-resistant seals (akin to build and release security and integrity practices), and with recall procedures in place (akin to vulnerability management practices). Infrastructure security measures defend against infiltration attempts by adversaries. Product safety testing mechanisms verify the absence of harmful ingredients or adulteration (akin to software composition analysis, binary analysis and pen testing, which are described below).
As explained in the Lawfare article, it is critical to ensure that software security practices address likely threats. We will focus on three vectors that can result in bad security outcomes in final software products: (1) software vulnerabilities (software flaws caused by an unintended error in the design or coding of software, which for these purposes encompass both actively exploitable vulnerabilities and weak coding practices that set the stage for future exploitability), (2) intentional malicious code such as malware (software specifically coded to enable adverse impact on a system’s confidentiality, integrity or availability of an information system) or tampering (an intentional act resulting in the modification of a system or data), and (3) hard-coded secrets (sensitive information – i.e., plain-text passwords or API keys – that are embedded into source code). In addition, certain product security design features can also weaken defenses and inhibit response – for example lack of strong product-level authentication, very limited logging and features that are prone to user error.
Likewise, best practice adoption in name only (i.e., pro forma), legacy code complexity and product-by-product variance can also obfuscate security problems.
The earlier article summarized five main approaches for direct technical observability into software product security performance: (1) software composition analysis (for open-source vulnerabilities), (2) complex binary file analysis, (3) penetration testing, (4) software bill of materials analysis (for open source and in commercial code vulnerabilities) and (5) Supply Chain Levels for Software Artifacts framework (SLSA) attestation verification. This article explains how they work and their limitations.
Software Composition Analysis (SCA) tools analyze code to create inventories of software libraries (generally limited to open source), dependencies, and related vulnerabilities, and there is no need for access to source code. If binary code is in 0s and 1s, how do these tools work?
SCA tools look for readable metadata and file headers that provide information about libraries and dependencies. Binaries may also include readable symbols (e.g., “SSL_connect”) and other strings which are not obfuscated. Open-source libraries may also have unique patterns, based on how the code was compiled, which can effectively serve as a “fingerprint.” Some binaries also explicitly include “manifest files” that list libraries and dependencies (although these manifest files are entirely dependent on what developers chose to list in the manifest).
Complex Binary File Analysis. Like some SCA, complex binary analysis tools examine software in its binary form. As noted above, there is no dependency on the software product company providing source code access. Unlike SCA, complex binary analysis tools examine whether threat behavior such as malware or tampering is indicated. These tools conduct automatic analysis of every file in the final software package, including core executable binary files, plus related artifacts such as installers, configuration files and related read-me files and other documentation.
The process includes steps such identifying file formats (e.g., binary versus text), decomposing files to extract components and metadata (headers, strings, function names, library dependencies), with unpacking ideally tailored to different file formats. Digital signatures are reviewed (e.g., for bad signatures, impersonation attempts, tampering, etc.); file structures are analyzed for malformations; and metadata is evaluated. Known threat behavior patterns can also be checked against file formats and metadata. Change detection comparing different versions of a binary can also identify tampering.
Wouldn’t standard anti-malware scanners identify malware? Unlikely because scanners aren’t conducting deeper-level code inspection. They may also be challenged in unpacking and de-obfuscating binaries (which is often done by adversaries to make it harder to detect malware).
Binary analysis also attempts to discover all open-source and commercial third-party software components that are embedded inside of a software package.
How can hard-coded secrets be detected from a binary? Hard-coded secrets and private keys are often represented as distinct binary objects or plain-text strings in a binary, and where code can be unpacked, hard-coded secrets, tokens and private keys can be revealed.
Penetration Testing is defined by NIST as a “test methodology in which assessors, typically working under specific constraints, attempt to circumvent or defeat the security features of an information system.” The goal of penetration testing is to “determine whether a mature security posture can also withstand an intrusion attempt from an advanced attacker with a specific goal.” Penetration testing can be thought of as verifying a configuration when you believe it to be secure (using pen testing against systems already known to be insecure would be akin to shooting ducks in a barrel). Application-related penetration testing comes in different flavors, including testing specific to web and mobile applications. Value can vary widely depending on how tests are scoped, plus the skill and experience of the tester.
Software Bill of Materials. A software bill of materials is defined as “a formal, machine-readable inventory of software Components and Dependencies, information about those Components, and their relationships. SBOMs can help inform pre-purchase evaluation and post-purchase asset and vulnerability management and supply chain risks. They can include open source or commercially licensed software and can be widely available or access-restricted to protect proprietary or sensitive information.”
SBOMs need to be machine-readable to be practically useful over time given the volume and complexity of modern codebases. They should also be in industry-standard formats, such SPDX and CycloneDX, and meet related minimum elements. When software is updated, SBOMs should be too.
SBOMs can be developed by the providers of software, which benefits from increased accuracy based on access to the source code and build environment. This is easier for newer classes of software, where modern processes can enable automated SBOM generation, but harder for older legacy software, where manual methods may be needed to generate an SBOM. This process also assumes the trustworthiness of the software provider.
SBOMs can also be generated by third-party tools, without a need for access to the build process or the provider. The disadvantage of this approach is that it “may be prone to omissions, errors or approximations if the tool is unable to decompose or recognize the software components precisely.”
Supply Chain Levels for Software Artifacts Framework. SLSA is an industry-developed specification for (1) characterizing and measuring the security of software supply chain infrastructure and (2) generating provenance to support security attestations that can be verified. Provenance is defined as the “verifiable information about software artifacts describing where, when and how something was produced,” or put another way the “chain of custody of the software and all of the constituent components that comprise that software, capturing information about the authors and locations from where the components were obtained.”
Provenance metadata is verified in the form of an “attestation” from the software producer. SLSA’s current focus is on verifying the integrity of the software build process (i.e., the sequence of steps that translate code from source to a software package). In terms of how this is implemented, a provider’s security team would evaluate a build platform to ensure that it meets the specified SLSA Build Track requirements. Public keys are then used by the build platform to sign provenance, attesting that a software package was actually built in that platform. Downstream users would verify the provenance signed by the public key to determine that an artifact meets the SLSA level.
SLSA encompasses Levels 0 through 3, where ascending levels indicate increasingly hardened security practices and related provenance. Level 1 reflects simply that provenance exists; Level 2 reflects that provenance is actually signed (by the trusted build platform); and Level 3 also verifies the hardening of the build platform. Future SLSA versions are expected to cover other parts of the software supply chain, such as how source code is managed. As with software provider-generated SBOMs, SLSA assumes the trustworthiness of the SLSA attestor.
The following table describes how the buyer options for direct technical observability align to the key threat vectors described above.
| Malware or Tampering | Code Vulnerabilities | Hard-Coded Secrets | |
| Software Composition Analysis | SCA is used to identify and inventory software components and does not directly inspect the code, so SCA would not be expected to identify malware or tampering. | Like SBOMs, SCA is used to identify vulnerabilities in software based on matching software libraries in the code to vulnerabilities. Focus is open source software. | As noted above, SCA generates inventories and does not inspect the code itself, so SCA would not be used to identify hard-coded secrets. |
| Penetration Testing | Purpose is to attempt to circumvent or defeat the security features of an information system. | May look for hard-coded secrets as a way to subvert system and environment security features. | |
| Complex Binary File Analysis | Software binary analysis can identify forms of malware and tampering. Examples of static binary analysis product differentiation include (1) the library of malicious behaviors to compare against, as well as (2) the comprehensiveness of being able to address nuances of open source versus commercial versus proprietary code; (3) depth of unpacking capability. | Binary analysis could be better at identifying custom code vulnerabilities, whereas SCA is designed to flag third-party (open source, commercial code) code with known vulnerabilities. | Software binary analysis can also identify hard-coded secrets. Differentiation can come from the library of hard-coded secret indicators to compare against. |
| Software Bill of Materials | SBOMs are inventories of software components and do not inspect the code itself, so SBOMS (even where generated by a third party) would not be expected to identify malware. | SBOMs for open source and commercial code inventories can be cross-referenced with vulnerability datasets. Examples of SBOM interpretation product differentiation include: (1) the ability to identify not only lagging indicators (basically known vulnerabilities that are already formally documented national vulnerability repositories) but also leading indicators (qualities about the nature of an open source project – for example the frequency of how well maintained it is, the number of committers, country-of-concern risks, etc.); and (2) the ability to host multiple SBOM manifests, so that customers can more easily search across their software products to look for instances of newly reported open source vulnerabilities that apply across products. Assumes trustworthiness of provider, where SBOMs are generated by provider. | SBOMs are inventories and do not inspect the code itself, so SBOMs would not be used to identify hard-coded secrets. |
| Supply Chain Levels for Software Artifacts | A SLSA provenance attestation would build confidence that malware had not been introduced through tampering either after (SLSA Level 2) or during (SLSA Level 3) the build process. Future versions of SLSA will extend this to other parts of the software development lifecycle such as source code management. Assumes trustworthiness of provider. | A SLSA provenance attestation would build confidence that vulnerabilities had not been introduced through tampering either after (SLSA Level 2) or during (SLSA Level 3) the build process. Future versions of SLSA will extend this to other parts of the software development lifecycle such as source code management. Assumes trustworthiness of provider. | A SLSA provenance attestation would build confidence that hard-coded secrets had not been introduced through tampering either after (SLSA Level 2) or during (SLSA Level 3) the build process. Future versions of SLSA will extend this to other parts of the software development lifecycle such as source code management. Assumes trustworthiness of provider. |
How do these technical observability capabilities address (or not) the practical challenges described earlier – pro-forma adoption of product security practices, legacy code complexity and product-by-product variance? The following table outlines the pros and cons of each:
| Pro Forma Adoption | Legacy Code Complexity | Product-By-Product Variance | |
| Software Composition Analysis | SCA can help identify when the supplier is only paying lip service to product security because the product has critical issues due to outdated and vulnerable open source software. | Not dependent on legacy code complexity, except that tools may perform better on some code languages. Focus is open source. | Should be applied on a product-by-product basis; can scale without vendor cooperation. |
| Penetration Testing | Effective pen-testing that is right-sized for the product and its associated risk can help identify systemically bad design and development practices, depending on how testing is time-bound and scoped, plus qualifications of tester. | Effective at highlighting vulnerabilities notwithstanding code complexity, depending on how testing is time-bound and scoped, plus qualifications of tester. Some technology stacks (e.g., embedded, IoT, web app) might be harder to pen test. | Should be applied on a product-by-product basis. Some technology stacks (e.g., embedded, IoT, web app) might be harder to pen test. |
| Complex Binary File Analysis | Effective at highlighting vulnerabilities, malware and hard-coded secrets resulting from pro-forma security practices. | Not dependent on legacy code complexity, except that tools may perform better on some code languages. | Should be applied on a product-by-product basis; can scale without vendor cooperation. |
| Software Bill of Materials | Effective at highlighting vulnerabilities resulting from pro-forma security practices (needs to be machine readable in standard formats such as SPDX or CycloneDX). SBOM accuracy maybe dependent on vendor coooperation. | Harder for older legacy software, where manual methods may be needed to generate an SBOM. | Should be applied on a product-by-product basis. Effective at highlighting vulnerabilities that are common across products (where machine readable in standard formats such as SPDX or CycloneDX). |
| Supply Chain Levels for Software Artifacts | Higher levels of SLSA attestation indicate more mature security practices in build process. Dependent on vendor coooperation for underlying attestation. | Higher levels of SLSA likely require significant re-engineering of build environment. | Should be applied a product-by-product basis, but efficiencies may result from common build environment/ approach. |
Adam Isles is principal and head of cybersecurity at The Chertoff Group, where he helps clients build and sustain cybersecurity programs. Adam drafted the Firm’s security risk management methodology, which is SAFETY Act designated by the U.S. Department of Homeland Security (DHS). Adam previously served as DHS Deputy Chief of Staff where he coordinated policy decisions, including technology standards, regulations, and business rules, for numerous border, travel security, cybersecurity and critical infrastructure protection programs. He started his legal career as a trial attorney in the Criminal Division at the U.S. Department of Justice. Adam is a Certified Information Systems Security Professional (CISSP), and a graduate of Harvard Law School.
