Vulnerability Research
May 6, 2020
By Maor Vermucht

Smarter Black-box Fuzzing of Industrial Communication Protocols

Industrial control systems (ICS) can become a significant vulnerability for major large-scale industrial enterprises. Attacks that utilize them can yield incredible physical damage, which malicious actors can leverage for financial or political gain. This is demonstrated by last year’s high profile ICS attacks which were launched by a range of international players including Iranian hacker groups, Chinese state-sponsored hackers, and more.

From the attacker’s perspective, the communication protocols between the control systems and their implementations are particularly interesting. They can be used as an attack vector if vulnerabilities in the implementation are found and can also serve for lateral motion in the network under attack, propagating the effects of the attack between different systems.

Traditionally, computer networks that connect industrial control systems were kept strictly isolated. Since the systems comprising these networks are not replaced often, many of the protocols and their implementations were rarely updated in order to keep them compatible with all the devices on the network. As a result of these two factors, industrial control systems often contain legacy code which was not developed with a defensive programming style in mind and has not been updated since it was implemented. Such devices tend to contain software implementation bugs, some of which pose significant security vulnerabilities.

One approach commonly used to locate software bugs and potential vulnerabilities in industrial devices is black-box fuzzing where the device is fed large amounts of partially random inputs over one of its network communication ports, and is then monitored for various kinds of misbehavior.

This approach is popular due to its relative simplicity. It can be used in a fire and forget mode, where the device under test is connected to a PC which generates random protocol packets and is monitored for crashes over time. This can yield important results without requiring in-depth research of the protocol and its implementation.

Although this approach can be sensitive to non-trivial problems, which would be challenging to locate otherwise, not all bugs with potential security implications will be discovered using this method. The resulting crashes seldom provide actionable information regarding the root cause since researchers generally do not have access to the internal state of the device and the precise failure scenarios. As a result, it can be challenging to triage, and even reproduce, the crashes produced using the pure black-box fuzzing approach.

In this article, we discuss a smarter approach to black-box fuzzing, which allowed us to successfully discover the CVE-2019-14462 and CVE-2019-14463 vulnerabilities in the libmodbus implementation of the Modbus protocol.

What is the Modbus protocol?

The MODBUS Protocol is a messaging structure widely used to establish master-slave communication between industrial devices. Although the basic Modbus protocol does not employ any authentication, authorization or encryption mechanisms, it serves as the primary transport protocol for many industrial devices. There are versions of the Modbus protocol over RS232/RS485 (Modbus RTU) and Ethernet (Modbus TCP).

The diagram below shows a typical Modbus architecture incorporating Modbus TCP and Modbus RTU versions:

fuzzing modbus architecture

As can be seen from the diagram, the typical architecture may include the next components:

  • Modbus clients (master) such as Human-Machine Interface (HMI), Distributed Control System (DCS) and Supervisory Control and Data Acquisition (SCADA)
  • Modbus servers (slave) such as PLC
  • An industrial firewall for securing communications across the entire network
  • Switches used as barriers for splitting the structure into separate zones

The following diagram shows the general outline of the request or response structure:

fuzzing blog diagram 1 (1)

Fuzzing set-up for a Modbus-facing device: The standard approach

Black-box fuzzing is a common approach used by industrial equipment manufacturers where the tested device is investigated as is by sending it random communication packets and looking for anomalies in its behavior (such as indications of software hangs or crashes).

The device under test (2) is connected to a computer (1) over the tested communication port using the examined communication protocol. The computer generates communication packets, usually adhering to the required protocol but with randomized content and monitors the device status by requesting responses from the device and examining their validity.

fuzzing device validity

Fuzzing set-up for a Modbus-facing device: Improving the efficiency of fuzzing

In this report, we implement a fuzzer for libmodbus library which is a specific implementation of the Modbus protocol. While the naïve approach to fuzzing would ideally not require any prior knowledge of the protocol or its implementation, understanding how to use some of its features, as well as intuition regarding the inner workings of the protocol, can significant improve the efficiency of the process.

Based on knowledge and intuition, we introduced three tweaks to the standard approach described above which led to discovery of two new vulnerabilities in the library.

fuzzing blog diagram 3 (1)

Tweak 1: Replacing the device under test

First of all, since libmodbus is open source we can recompile it at will and are free to set up our own testbench device to directly run the code we want to evaluate. More importantly, we can then use this new compilation to include stricter sanity check conditions than the ones found in the default compilation of the real device. As a result, the testbench device becomes much more sensitive to memory access violations which may indicate a vulnerability in the implementation.

We used two sanitizers: AddressSanitizer (ASAN) which excels in detecting memory corruption bugs such as heap/stack overflows and memory leaks, and MemorySanitizer (MSAN) which excels at detecting uninitialized memory reads in C/C++ programs. The MSAN quietly monitors the spread of uninitialized data and alerts where it finds use of this data. As described in detail later in this article, the MSAN helped us find the uninitialized memory after running Modbus writing and reading commands.

When compiling using Clang, including both ASAN And MSAN is as simple as adding –fsanitize=address and –fsanitize=memory compilation keys.

Tweak 2: Choosing the right inputs

Understanding the protocol and its possible implementations may provide hints that some fields, such as packet length of TCP modbus packet, write payload size and read payload size, could be more prone to errors than others. Therefore, it is important to examine the consequences of injecting these specific fields with random faulty values.

It would also be interesting to test the response of the protocol to random TCP traffic which does not adhere to the Modbus protocol at all.

In fact, we can determine that the most promising fields for fuzzing in a Modbus packet are the fields which run any kind of non-trivial parsing logic (as opposed to address fields which may be handled in simpler ways): fuzzing fields no trivial parsing logic

Tweak 3: Monitoring the device more effectively

While software crashes upon reception of a packet from the fuzzer is an obvious sign of an implementation bug, many bugs are more subtle and may go unnoticed if only the crashes are monitored. In order to detect other misbehavior patterns without their tracks being overwritten by later commands, we can issue a read command after every packet that is generated by the fuzzer and verify the validity of the response.

Good candidates of read commands for this purpose would return as much information as possible about the internal state of the device. In the case of Modbus we used Read Holding Registers or Read Coil Status commands.

In addition, error codes of all kinds generated by the device should be logged and validated by comparing them to the expected errors resulting from faulty field values in the issued communication packets.

Fuzzing results

Running the fuzzing set-up described above quickly causes the fuzzed code to crash when running on Raspberry Pi. The crash is caused by the MSAN sanitation condition check (i.e., the real device would not crash for this input):

Uninitialized value was stored to memory at#0 0x55c6bfac5f28 in modbus_reply /home/workspace/modbus_demo/Release/../modbus.c:887:46
#1 0x55c6bfae95ed in main /home/workspace/modbus_demo/Release/../modbus_demo_main.c:63:13
#2 0x7f14dd84682f in __libc_start_main /build/glibc-Cl5G7W/glibc-2.23/csu/../csu/libc-start.c:291
 
Uninitialized value was created by an allocation of 'query' in the stack frame of function 'main'
#0 0x55c6bfae88b0 in main /home/workspace/modbus_demo/Release/../modbus_demo_main.c:25
 
SUMMARY: 
MemorySanitizer: use-of-uninitialized-value /home/workspace/modbus_demo/Release/../modbus-tcp.c:169:12 in _modbus_tcp_send

The additional details provided by the MSAN printout help us pinpoint the reason for the crash which can be clearly seen in the source code:

for (i = address, j = 6; i < address + nb; i++, j += 2)
 {
      /* 6 and 7 = first value */
      mb_mapping->tab_registers[i] =
               (req[offset + j] << 8) + req[offset + j + 1];
 }

In this code, req is the received Modbus payload, and nb is the requested count of registers to write. Since nb is controlled from the received packet independently from the length of req, by providing a large enough nb, the attacker causes memory contents after req to be saved to Modbus register units.

The contents can be later read out, resulting in a memory exfiltration vulnerability. It should be noted that MSAN’s detection of uninitialized memory used is triggered only when reading the Modbus register units (upon read command). This highlights the importance of reading from the device after each packet sent by the fuzzer.

Conclusions

The two discovered vulnerabilities received a CVE score of 9.8 and are classified as information leak vulnerabilities, which are usually perceived as less critical in the context of industrial control systems since they cannot be used directly to cause physical damage. However, in more complex attack scenarios, there are many benefits that attackers can gain from them.

For example, when combined with a stack overflow vulnerability, an information leak vulnerability may be used to defeat address space layout randomization (ASLR) protection mechanisms by obtaining information about the memory layout.

VDOO has responsibly disclosed the vulnerabilities to the project's maintainer who responded very promptly and uploaded their fixes into libmodbus version 3.1.6.

Written with Ilya Khivrich

Share this post

You may also like