Technical
June 9, 2019
By Leo Dorrendorf

Monitoring Embedded Linux Device Integrity with AIDE

With guest writer Donald Tevault

Introduction

This is the second article in the series of articles on isolation and integrity techniques, which we outlined in the previous blog entry. The series as a whole focuses on Linux-based Internet of Things (IoT) devices, but much of the material we will cover is suited for Linux in general, including embedded and regular machines such as desktops and servers.

Securing network-enabled devices involves two layers of defense: one that deals with preventing attackers from breaking into the devices, and another that deals with monitoring and discovering attacks as they take place. No matter how good the first layer gets, the second layer will always be necessary, because of new zero-day vulnerabilities and attacks which are discovered daily, and also because successful attacks have to be detected as soon as possible, to allow an update or recovery of the devices to a secure state.

In this article, we will examine one popular approach to file integrity monitoring on Linux, using the AIDE (Advanced Intrusion Detection Environment) solution. AIDE can detect attacks taking place on the device by monitoring for file modifications. We will look into its installation and operation in depth, and once it's clear how it operates, we will discuss the downsides of this approach.

AIDE, which was forked from the open-source version of Tripwire when Tripwire went commercial, is a host-based intrusion detection system. It works by creating cryptographic checksums of all important files on a Linux system, and then storing those checksums in a database file. A cron job will run a filesystem scan on a periodic basis, and will detect when files have been added, deleted, or changed. This can help detect if someone has replaced any important system files with malicious trojaned versions, or if someone has added files that could be malicious.

Since most of our customers prefer to run Debian or Ubuntu-type operating systems, we'll concentrate on them for this procedure.

Logs and Reports

When AIDE runs a system check, it automatically sends a report to a designated user. In order for this to work, a mail server daemon, such as Postfix, must be installed on the devices. If Postfix isn't installed already, just run:

$ sudo apt install postfix

Configure Postfix as either a server for just the local machine, or as a server for your Local Area Network, as your needs dictate. Optionally, you can configure the /etc/aliases file to forward the root user's mail to an account of your own choosing. Open the file in your text editor, and at the bottom of the file, add a line such as one of the following:

root:    donnie
root:    donnie@xyzwidgets.com

The first line is good for if you just want to check mail on each local device. The second is good for when you want to forward mail to a central email server on your network. After you edit the aliases file, run:

$ sudo newaliases

Now, the reason we said that configuring the aliases file is optional, is because you can also configure AIDE to send reports to a specific address. By default, AIDE sends reports to the machine's root user account.

AIDE also stores its scan results in a set of log files within the /var/log/aide directory. You can scan the logs on each device manually, or you can install an automatic log file monitoring solution such as Filebeat.

Note: We'll look at how to interpret the log files in just a bit.

Installation

To install AIDE, just run:

$ sudo apt install aide
$ sudo aideinit

Note: The sudo aide --init command that you might be used to using doesn't work on Debian/Ubuntu distros. Instead, you have to use sudo aideinit.

The good news is that this sets up pretty much everything you need, with a default configuration that will work for most cases. You'll probably still want to tweak things a bit in order to exclude things that you don't need to scan, or to customize a scan for your own custom files and directories.) The aideinit command creates the initial database of file checksums, and stores it in the /var/lib/aide/aide.db file.

Note: This aide.db file is a compressed binary file, which includes seven different checksums, using seven different hashing algorithms, for each file. You can expect the file to grow rather large. Here on our Raspberry Pi, it weighs in at about 56 MBytes with all seven of the checksums enabled.

How AIDE Operates

To get a feel for how AIDE operates, you'll need to look through its various configuration files, paying careful attention to the comments.

The Cron Job Script

First, look in the /etc/cron.daily directory. There, you'll see the aide file, which is a shell script that runs once every day. Open the file and go down to line 368.

368         # Filter package upgrades/installations
369 
370         # Figure out where the dpkg log file is
371         DPKGLOG="$(< /etc/dpkg/dpkg.cfg grep "^log" | head -n 1 | cut -d ' ' -f 2)"
372 
373         if ( [ "$FILTERUPDATES" = "yes" ] || [ "$FILTERINSTALLATIONS" = "yes" ] ) && [ -s "$DPKGLOG" ]; then
374 
375             # Create a list of files modified by system updates
376             if ( [ "$FILTERUPDATES" = "yes" ] && [ "$FILTERINSTALLATIONS" = "yes" ] ) ; then FILTER="install|upgrade"
377             elif [ "$FILTERUPDATES" = "yes" ]; then FILTER="upgrade"
378             else FILTER="install"
379             fi
380             PKG_FILE_LIST="$(mytempfile pkg_file_list)"
381             REGEX="^([^ ]+ [^ ]+) ("$FILTER") ([^ ]+) [^ ]+ [^ ]+$"
382             pkgs=
383             while read line; do
384                 if [[ $line =~ $REGEX ]] && [[ "$DATABASEDATE" < ${BASH_REMATCH[1]} ]]; then
385                     if dpkg-query -L ${BASH_REMATCH[3]} > /dev/null 2>&1; then
386                         pkgs+="${BASH_REMATCH[3]} (${BASH_REMATCH[2]})\n"
387                         dpkg-query -L ${BASH_REMATCH[3]} | sed -e "/^$/d" -e "/\/\./d" >> "$PKG_FILE_LIST"
388                         if ! ls /var/lib/dpkg/info/${BASH_REMATCH[3]}.* >> "$PKG_FILE_LIST" 2>/dev/null; then
389                             ls /var/lib/dpkg/info/${BASH_REMATCH[3]%:*}.* >> "$PKG_FILE_LIST"
390                         fi
391                     fi
392                 fi
393             done < "$DPKGLOG"

Even if you can't read the actual code, you can see from the comment lines that we have a handy routine that detects if files were added or changed due to performing either a legitimate package installation, or a legitimate system update. That's good, because it allows us to choose whether or not we want to see reports about files that changed as a result of these legitimate "apt" or "aptitude" actions.

The "Default" File

Next, let's look in the /etc/default/aide file. Everything here is explained really well in the comments, so we'll just point out a couple of things of especial interest. First, on line 23, we have the MAILTO line, which looks like this:

MAILTO=root

If you don't want to mess around with reconfiguring the /etc/aliases file, you can instead configure this line with the user address that you want to receive the daily reports.

On line 39, we see:

COMMAND=update

This causes AIDE to create a new database file each time that the cron job runs. However, instead of overwriting the old database file, it just saves a new file as aide.db.new. In order to replace the old database, an administrator will have to delete the original aide.db file, and rename the aide.db.new file to aide.db. But, at line 48, we see:

COPYNEWDB=no

This line allows us to change this behavior. By setting this parameter to yes, AIDE will automatically overwrite the old aide.db file with the new one. As you can see in the comments, though, this isn't recommended. The other option for this parameter is:

COPYNEWDB=ifnochange

This will cause AIDE to overwrite the old aide.db file only if no changes were detected.

Note: In its default configuration, AIDE won't automatically overwrite the aide.db file with a new one. But, every time that the cron job runs, AIDE will overwrite the existing aide.db.new file with a new one. So, you'll never have more than one aide.db.new file.

By default, AIDE will report if any files have been updated or added by a legitimate system update or package management operation. On lines 58 and 64 you see:

FILTERUPDATES=no
FILTERINSTALLATIONS=no

Change those values to yes if you don't want AIDE to report on files that were changed, added, or deleted due to legitimate system management actions that you would perform with "apt" or "aptitude".

Configuration and Rules Files

We'll now turn our attention to the files in the /etc/aide directory. First, there's the aide.conf file, which defines the actions which can apply to different types of files, and where the aide.db file will be located. (Normally, you'll never need to modify this file.)

In the /etc/aide/aide.conf.d directory, we see ruleset files that apply the actions from aide.conf to the various files for different programs and services. The cool part about this, is that you can create other rules files here that will apply the appropriate actions to either new files and directories, or to existing files and directories. For example, let's say that we want to exclude certain directories from getting scanned, because it's expected that stuff in these directories will change. For this, we created the 10_aide_excludes file, which looks like this:

!/var/lib/lxcfs/
!/run/
!/tmp/

The trailing / at the end of each line means that we also want to exclude all subdirectories from getting scanned. (In truth, there's more that we'd like to exclude, but this is enough to serve as an example.) Now, the next time that the nightly scan runs, our new rules will get inserted into the aide.conf.autogenerated file that gets created in the /var/lib/aide directory. To see if the rules are there, let's run:

$ sudo grep '^!' /var/lib/aide/aide.conf.autogenerated | less

The new rules should appear at the top, like this:

!/@@{ROOTPREFIX}var/lib/lxcfs/
!/@@{ROOTPREFIX}run/
!/@@{ROOTPREFIX}tmp/

For the most part, the ruleset files within /etc/aide/aide.conf.d just apply actions that are defined within the /etc/aide/aide.conf file. Let's look in the /etc/aide/aide.conf file to find an appropriate action for our new application. On lines 44 and 45, we see the Full rule, which looks like this:

44 # Check everything
45 Full = InodeData+StaticFile

This references two other rules, which are defined in lines 36 through 38.

36 # Files that stay static
37 InodeData = OwnerMode+n+i+Size+l+X
38 StaticFile = m+c+Checksums

In turn, the InodeData rule references the OwnerMode rule, which is defined on lines 30 and 31.

 30 # check permissions, owner, group and file type
 31 OwnerMode = p+u+g+ftype

The Checksums rule that is referenced by the StaticFile rule is defined in lines 21 through 24.

21 # if you want to sacrifice security for speed, remove some of these
22 # checksums. Whirlpool is broken on sparc and sparc64 (see #429180,
23 # #420547, #152203).
24 Checksums = sha256+sha512+rmd160+haval+gost+crc32+tiger

As you can see, each file for which the StaticFile rule is applied has seven different checksums created for it. Now, you can leave this as is if you feel that you really need to have seven different checksums for each file. But, if you believe that's a bit of overkill, you can reduce AIDE's performance impact on your device by removing all but one that you trust. For our example, let's just use the SHA512 algorithm.

21 # if you want to sacrifice security for speed, remove some of these
22 # checksums. Whirlpool is broken on sparc and sparc64 (see #429180,
23 # #420547, #152203).
24 # Checksums = sha256+sha512+rmd160+haval+gost+crc32+tiger
25 Checksums = sha512

As you can see, we left the original line in place and just commented it out. This will make it easier to revert back to the original configuration, should we ever choose to do so.

To see what the codes that are assigned to these rules mean, just run:

man aide.conf

To see an example of how this works for a custom application of your own making, let's create the newapp directory within the /opt directory. Within this new directory, let's create the newapp.sh shell script. (The contents of it don't matter. Put anything in it that you want to.) If you want for the Full rule to apply to your new application, you don't have to do anything at all, because the /etc/aide/aide.conf.d/99_aide_root file already applies this default action to the entire filesystem. The contents of this file look like this:

/ Full

The other ruleset files in /etc/aide/aide.conf.d just modify or replace this default action for certain files and directories. So, if you don't want the Full rule to apply to your newapp.sh application, you could create a new ruleset file and call it 31_aide_newapp. In reality though, the Full rule is probably what you'll want for new applications.

Finally, there's the /etc/aide/aide.settings.d directory, which contains files with other miscellaneous program settings. (You'll probably never need to modify anything here.)

Normal Operation

So, out-of-the-box, you already have pretty much everything you need, with a working configuration. If you might need to reconfigure anything at all, it would be to add custom rulesets for any custom applications that you might create and for which you don't want the Full rule to apply, to add rulesets for anything that you might want to exclude, or to cut down on the number of hashing algorithms.

At some time every night, the daily cron job will run. It will create a new aide.db.new file and send a report of any changes that AIDE has detected. If you want to, you can also run a scan manually by doing sudo aideinit. But, scanning manually will only create a new aide.db.new file. It won't create either a report or any log entries about what has changed.

Interpreting the Results

By default, the report that gets mailed to you is truncated, so you'll need to look at the /var/log/aide/aide.log file to see the whole results. You'll see what files have been added, deleted, or modified. For example, here we see where we added the newapp directory and the newapp.sh file:

---------------------------------------------------
Added entries:
---------------------------------------------------
...
d++++++++++++++++: /opt/newapp
f++++++++++++++++: /opt/newapp/newapp.sh
...

Now, if you just leave the original aide.db file in place, you'll see the report about these additions after every nightly scan. But, you'll never see any reports about any modifications that get made to either the directory or to the file. To monitor for any modifications, you'll need to replace the original aide.db file with the new aide.db.new file.

cd /var/lib/aide
sudo rm aide.db
sudo mv aide.db.new aide.db

The next time that the nightly scan runs, you'll now see if any changes get made to the new file. To demonstrate, we opened the "newapp.sh" file in a text editor and added a few lines. Here's what came up after the next scan:

---------------------------------------------------
Changed entries:
---------------------------------------------------
...
d =.... mc.. .. .: /opt/newapp
f >.... mci.C.. .: /opt/newapp/newapp.sh
...

To see what this all means, do man aide.conf to view the code definitions. In both lines of the above output, we see mc. The m means that the modification time has changed, and the c means that the change time has changed. In the second line, we also see an i and a C. The i means that the inode for the file has changed, and the C means that at least one checksum for the file has changed. The > means that the file has grown larger. If you know for a fact that this file change was not authorized, then you'll need to activate your organization's incident response team. If you know that the change was authorized, then just replace the aide.db file with the aide.db.new file so that you won't keep seeing this in future scan reports.

The Downsides

You might think that AIDE is pretty cool, and in a way it is. But, there are a few downsides.

First, there's the simple fact that since AIDE is a daily scanner instead of a continuous monitoring solution, your devices could go as long as a full day with files that have been tampered with before AIDE will find them. Needless to say, that's not a good thing.

Secondly, since AIDE is installed on the devices that it scans, how do we guarantee that the AIDE executable files, configuration files, and database files haven't been tampered with? An attacker who tampers with the device's system could tamper with AIDE just as well. In effect, AIDE cannot protect itself, and needs protection from another system if the device is to achieve a high level of security.

Finally, there is a performance hit with the default configuration. A scan on our Raspberry Pi, with all seven checksums enabled, takes over 40 minutes to complete. With only the SHA512 hashing enabled, a scan takes about 20 minutes. During that time, you could see a bit of a performance hit for other applications. So far though, that seems to be fairly minimal. Disabling all but the SHA512 algorithm is the way to go, and it also cuts down the size of the aide.db file from 56 MBytes to about 20 MBytes.

Summary

AIDE is easy to set up and easy to use. While it does have its uses, its biggest downside, as we noted above, is that it only runs as a periodic scanner. This means that you could go a long time before it will catch any files that have been tampered with.

To address AIDE's downsides, a real-time monitoring solution is needed. Such a solution is possible - it requires an agent module to receive notifications of system operations as they happen, to process them and to monitor them for exceptions which indicate unauthorized or malicious activity. Unlike a periodic monitoring solution such as AIDE, a run-time agent could prevent unwanted operations from happening, instead of reporting damage after the fact. Real-time monitoring also means that any reports of attacks taking place would reach the device's administrators in a timely manner, rather than retroactively. We are offering VDOO ERA™ - Embedded Runtime Agent - as one such solution.

Share this post

You may also like