Building a PSIRT Organization to Secure Connected Products
The Need for Device Security
IoT products continue to be plagued by basic vulnerabilities such as using insecure network services, shared default account credentials and unsigned firmware. The availability of best practice guidance has done little to help developers avoid these common mistakes. Yet, organizations continue to demand connected products. Medical devices, automobiles, robotic platforms, airplanes, farm-equipment and even clothing are becoming cyber-physical. As time progresses, consumers and the general public will develop an increased reliance on the availability and security of these systems. Without a strategic approach to cyber security, the connected devices that form the foundation of tomorrow will expose an increased quantity of vulnerabilities making it impossible to develop assured business and mission systems.
Product vendors must invest in a holistic cyber security strategy. Not doing so will not only expose products to exploitation, but also invite pressure from outside entities. Cyber insurance companies are beginning to assert their influence on the market and the Security and Exchange Commission (SEC) is beginning to require public companies to report on any vulnerabilities that could lead to reputational damage.
This article examines two approaches that an organization can adopt in order to begin building out a strategic cyber security capability. The first approach is top-down. An executive is recruited and hired to define and oversee a product portfolio security strategy. The second approach is bottoms-up. A team of product security engineers is assembled that can proactively hunt for vulnerabilities in products while simultaneously working with 3rd parties to collect and respond to submitted defects. This Product Security Incident Response Team (PSIRT) works closely with each product development team to create and route security tasks based on identified issues.
Finding budget for either of these approaches is challenging. Larger organizations may be able to staff for both approaches, however small and midsize device makers may find it useful to weigh the pros and cons of each approach. Even with sufficient budget, finding and hiring personnel with the right skills and expertise for these roles is a challenging task.
Two Approaches for Building Cybersecurity Strategy
1) The Product Security Executive
Just as the Chief Information Security Officer (CISO) role is accountable for the security of an organization’s data and assets, a Product Security executive can be made accountable for the cyber security of the portfolio of products sold by the organization. A Product Security Executive establishes the vision and goals for the cyber security program and works with product teams to ensure proper training in cybersecurity risks and mitigations. She or her designee establishes secure coding standards, maintains a listing of approved 3rd party libraries, and defines the minimum requirements for code reviews and security tests. This includes sufficient use of static and dynamic security testing, hardware security testing, radio frequency (RF) security testing, fuzz testing, and penetration testing. This executive also monitors the compliance state of each product in the portfolio to ensure compliance with defined security guidelines. This role is also accountable for handling all cyber security incidents associated with any of the organization’s products.
Finding the right person to staff this role can be difficult. The role of the Product Security Executive is dynamic and challenging and requires a diverse set of skills. The executive must be able to liaise with multiple functions. He must work with the board of directors, audit committee, and legal team to understand the ramifications of vulnerabilities in products. He must work with Public Relations to craft press releases describing the nature of vulnerabilities, impacts to users and steps to remediation. He must also work with the C-Suite to make the case for the overall cyber security budget and strategy.
Collaboration with the technical teams requires an executive that understands the technologies used across the portfolio of products. She must also have the ability to understand and recommend changes to development methodologies, understand the threat landscape associated with each of the technologies used across product teams, and to be able to identify, and recommend corrective actions for product vulnerabilities. This executive must also be able to work closely with product safety engineers to communicate impact of cyber security threats to product and environmental safety mechanisms. The executive must also be comfortable communicating with the security research community and able to host discussions with researches responsibly disclosing product vulnerabilities.
A Product Security Executive holds significant responsibilities. She must be provided with technology that enables success in the position. This requires a solution that can minimize the amount of work required to hunt for and understand vulnerabilities. The solution should also provide expert inputs regarding the actions needed to mitigate the vulnerabilities. VDOO’s Vision™ platform for example can provide detailed and prioritized reporting on weaknesses as well as step-by-step guidance describing how to mitigate or remediate the weaknesses. Vision™ also provides an objective rating of each product based on a comparison identified weaknesses against other similar products.
Using Vision™, a Product Security Executive can test all existing firmware files to find security issues related to products in the field and prioritize the mitigation plan based on available resources. This is a great approach for an executive stepping into a role with already deployed products. Scans are completed quickly. A one (1) gigabyte firmware file can be scanned in approximately one hour. For new products, Vision™ can be integrated directly into the build pipeline. As new builds and versions are completed, they are tested, and reports are sent to the Product Security Executive and the development teams. The Product Security Executive can then work with the development teams to prioritize issues and ensure they are fixed prior to shipment of the product. Once all issued are remedied, VDOO’s on-device self-certification CertIoT™ can be applied to assert the security state of the product.
Vision™ can also be used by development teams for regression testing to ensure that any change made to a product satisfies the required security threshold.
2) The Product Security Incident Response Team
An alternate approach to hiring a Product Security Executive focuses on building up a team of technical staff focused on the cyber security of products. This internal center of excellence uses a matrixed approach to staff developers and security engineers from across the product lines. Typically, these engineers focus part time within the PSIRT and part time on their development team. Their goal is to proactively identify and respond to vulnerabilities in products. The team actively scans product files for weaknesses and works with the research community to welcome responsible disclosure of vulnerabilities.
The 2016 ENISA’s report Strategies for Incident Response and Cyber Crisis Coordination defines Incident Response and Management as: “The protection of an organization's information by developing and implementing an incident response infrastructure (e.g. plans, defined roles, training, communications, management oversight) in order to quickly discover an attack and then effectively contain the damage, eradicate the attacker's presence, and restore the integrity of the network and systems.”
This same definition can be applied at the product level. While a Computer Security Incident Response Team (CSIRT) focuses on internal organizational technology, a PSIRT focuses on identifying and remediating vulnerabilities and exploits associated specifically with products sold by organization. Vulnerabilities are then prioritized and sent to the development teams for remediation. The PSIRT then tracks each to closure.
Although the PSIRT conducts its own security analysis of each product, the team also works closely with the external cyber security research community. They communicate their desire to receive vulnerability reports and provide a mechanism to accept those reports in a secure manner. For example, the team might publish a Pretty Good Privacy (PGP) public key that allows researchers to encrypt reports to them. Once accepted, the PSIRT team works to establish a CVE ID and manage the vulnerability using CVSS¹.
A primary responsibility of the PSIRT is to prioritize and triage vulnerabilities and work with the development teams to remediate the vulnerabilities. The PSIRT may set deadlines based on compliance requirements or organizational policies. For any incidents that occur, after action reports are created that include a detailed Root Cause Analysis (RCA).
The PSIRT is charged with ensuring that the organization maintain compliance with applicable laws and regulations. This requires that the right mix of skills is made available on the team. For example, the PSIRT should include legal representation that can provide guidance based on the scope of the vulnerability. The varied functions of a PSIRT are described below.
Just like the Product Security Executive, a PSIRT is actively engaged in evaluating the security of the organization’s products. This includes both legacy and new products. For legacy products that are already deployed, the PSIRT uses tools such as VDOO’s Vision™ to test the firmware files of existing products in the field to create a risks assessment and prioritize future security tasks that will result in product updates in the field.
The PSIRT creates and publishes internal product security guidelines that are used by development teams for any new product releases. The PSIRT also ensures that appropriate test tools are integrated into the continuous integration system for each product team. Tools like VDOO’s Vision™ are used to test new products, identify weaknesses, and determine the appropriate mitigations. Product teams should get the green light from the PSIRT prior to launching a new product or an update to a product.
Building a Security Organization
Whether an organization decides to recruit and onboard a Product Security Executive or establish a PSIRT, decisions must be made on their role and placement within the hierarchy of the organization. There are options for each approach.
A Product Security Executive is accountable for the cyber security of all products sold by the organization. This responsibility requires a commensurate level of authority. Although persuasive skills are required for any individual to be successful in this role, placement within the top executive ranks of the company provides a degree of influence that can help with accomplishing the role’s mission. Just as the placement of CISOs within an organizational structure is open to intense debate, the same can be said of the placement of the Product Security Executive.
The figure above advocates for a direct reporting relationship between the Product Security Executive and the Chief Executive Officer (CEO). At a minimum, a dotted line reporting relationship should be established between these two roles to ensure a level of independence for the cyber security function within the organization. Ideally, legal and public relations functions would have a dotted line reporting relationship to the Product Security Executive. Product Owners across the organization would also have this dotted line reporting structure. A look at the org chart above shows that anyone in the Product Security Executive role must be able to influence decision making even when dealing with those not in the direct chain of command. A challenging task and one reason why finding the right candidate may prove difficult.
There are options for establishing a PSIRT within an organization as well. A PSIRT can be centralized or decentralized. The ability to implement a centralized PSIRT will depend heavily on budget availability for staffing and tooling a team that operates separately from the product teams. A decentralized PISRT on the other hand pulls staff from across the product teams on a part-time basis. This matrixing of team members can be cost effective. This team is allocated a percentage of time to perform PSIRT duties. There are a number of benefits to this approach, for example allowing cross-pollination of ideas and skills across different products.
A decentralized or matrixed PSIRT is recommended since each member comes from a product team that they can directly educate and influence to make informed security decisions.
Establish the PSIRT as a service center and require all product teams to pass through the service center as a gating function. In this role, the PSIRT reviews new products and updates for security flaws prior to release. Using tools such as VDOO’s Vision™, the PSIRT team can perform initial scanning and provide direction to the product teams for remediation. Product teams can then use their version of VDOO Vision™ to perform a more detailed analysis in support of closing any identified weaknesses. This includes more in-depth and time-consuming scans such as binary analysis using approaches such as symbolic execution.
Executives within the organization should monitor PSIRT activities using metrics to ensure that the PSIRT continues to add value. For example – quantity and responsiveness to externally submitted vulnerability reports, number of internally-identified vulnerabilities, and time-to-resolution.
Arm your PSIRT and product teams with security technology that will allow them to efficiently identify, evaluate and manage product vulnerabilities. Standard business communication tools such as Slack are valuable for collaboration and allow channels to be created for individual product teams as well. Issue management tools such as JIRA should also be used to record and track all identified vulnerabilities to closure.
A PSIRT should be capable of identifying vulnerabilities on their own as well. This requires a range of scanning and analysis tools that depend on the IoT products being analyzed. For example, the table below provides a listing of useful tools that support hardware security analysis, radio frequency testing, and application/network testing.
The VDOO Vision™ and ERA™ tools provide capabilities that can help a PSIRT, Product Security Executive and development teams better understand and mitigate their product’s unique weaknesses. VDOO Vision™ can be used by either a PSIRT or Product Security Executive to quickly scan all firmware binaries prior to authorizing release. The scan process takes up to 45 minutes and outputs recommended mitigations in a clear and easy-to-understand manner that do not require a trained security engineer to interpret.
Vision™ is ideal for quickly scanning a new product or an update to the product to identify vulnerabilities. For these roles, Vision™ provides highly useful metrics that show the relative security state of the product compared with other products of similar features. VDOO’s CertIoT™ on-device self-certification can then be used to assert the secure state of the device.
For development teams, Vision™ provides substantial detail that not only identifies a vulnerability but also guides the team towards the proper approach to mitigation.
VDOO ERA™ provides additional flexibility to both the Product Security Executive and a PSIRT. VDOO ERA™ can be quickly generated automatically for any device model to deal with its specific potential threats, known and unknown, based on the insights of VDOO Vision™. It allows on-device detection and prevention and enables predictive security and control. ERA’s features range from exploit mitigation to active protection of the device’s critical assets.
A more in-depth binary analysis scan can also be conducted on each firmware file using VDOO Vision™. The binary analysis process requires additional time. However, it can result in the identification of zero-day vulnerabilities located within the firmware.
There are pros and cons to hiring a Product Security Executive or establishing a PSIRT. The Product Security Executive can help guide an organization towards a comprehensive security strategy. This may take time however as someone in this role must familiarize themselves with the intricacies of each product line in order to make decisions that have the least impact on product schedule and cost. It may also take a substantial amount of time to hire the executive for this role as the skills required make qualified candidates hard to come by.
The PSIRT team is much more organic. Using a matrixed organizational model, an organization can begin standing up a PSIRT immediately. Training will be required to get all participants to a relatively similar skill level, but that training will eventually pay off as the participants become product security evangelists for their respective teams.
Whichever approach you take, make sure to invest in establishing a cyber security function within your organization. All products should be developed using a Security Development Lifecycle (SDL or SDLC) and security testing should be conducted prior to any release. These processes should be standardized across product teams. The best way to accomplish this is to create a centralized role or center of excellence for security within your organization. This can be done by hiring a Product Security Executive or by standing up a PSIRT.
VDOO was established in 2017 to pioneer embedded systems security, with an end-to-end solution of security automation, certification, and protection. The VDOO founders’ backgrounds include an endpoint cybersecurity startup acquired by Palo Alto Networks, as well as notable experience serving in the Israeli Intelligence Elite Unit. For additional information, please contact us at firstname.lastname@example.org or visit our website at vdoo.com.
Share this post