How Cape Protects Keys from Attacks Trying to Exploit Memory Dumps

In this post, we'll be exploring one of the many hazards of multi-tenancy, and just how hard it can be to get security right! Have you ever wondered how secure data is in memory? Today I will show you that you don't have to exploit Spectre and Meltdown to steal private information from a running program. You can do that with basic debugging utilities included in modern operating systems.

The Attack

When a program runs, it has various memory segments, namely, the stack, heap, data, and code segments. For this attack, we'll be specifically looking at the heap segment. Heap memory needs to be "allocated" to the process. This is usually done using a call to malloc, which in turn calls the mmap system call under the hood. To get a better understanding of how this attack works, we'll be using mmap directly. Let's start by looking at a simple C program that allocates some heap memory, writes to it, pauses for user input, then unmaps this memory.

/* mmap.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/mman.h>
#include <ctype.h>

int main() {
  const char *key = "my-private-key";
  const size_t mmap_size = 1 + strlen(key);
  char *buf = mmap(
    NULL,
    mmap_size,
    PROT_READ | PROT_WRITE,
    MAP_PRIVATE | MAP_ANON,
    -1,
    0
  );
  if (buf == MAP_FAILED) {
    perror("mmap");
    exit(EXIT_FAILURE);
  }

  buf[mmap_size - 1] = '\0';
  for (size_t i = 0; key[i] != '\0'; i++) {
    buf[i] = toupper(key[i]);
  }

  puts("wrote to buf, now do the \"before\" core dump!");
  fflush(stdout);
  getchar();

  puts("unmapping memory, now do the \"after\" core dump!");
  fflush(stdout);
  munmap(buf, mmap_sie);
  getchar();

  return 0;
}

Now we'll compile and run this program

$ cc mmap.c -o mmap
$ ./mmap
wrote to buf, now do the "before" core dump!

At this point, keep this program running, and open a new terminal and type

$ ps -ef | grep "./mmap" | grep -v grep | awk '{ print $2 }' | xargs sudo gcore -o before

This will dump the memory of the running program mmap into a core dump file. If the process ID (or PID) of the mmap process is 511787, then the name of the core dump file will look like this. Don't worry if your file has a different name, you'll almost certainly have a different number for the pid. This is what my core dump file looks like.

before.511787

Notice that in the C code, we have a string literal "my-private-key" and we copy the uppercase version of this string into the buffer that we allocate with mmap. The reason for this is that string literals are stored in the data segment of memory, but for the purposes of this attack, we want to look at data stored in heap memory, so we are altering this string literal into something that will be easy to find, but guaranteed to be in the heap and not in the read only data segment. Let's go ahead and inspect that core dump file to see what's in there. We can use the strings utility to extract things that look like strings from a binary file. We know that the uppercase variant of "my-private-key" will only exist in heap memory, so let's look for that.

$ strings before.511787 | grep MY-PRIVATE-KEY
MY-PRIVATE-KEY

And there we go! We have extracted the string "MY-PRIVATE-KEY" from the core dump file. Let's now return to the other terminal where we are still running ./mmap and we still have this prompt:

$ ./mmap
wrote to buf, now do the "before" core dump!

Let's hit enter to get to the next prompt.

$ ./mmap
wrote to buf, now do the "before" core dump!
unmapping memory, now do the "after" core dump!

Let's follow those instructions and take the "after" core dump!

$ ps -ef | grep "./mmap" | grep -v grep | awk '{ print $2 }' | xargs sudo gcore -o after

And again, inspect with strings

$ strings after.511787 | grep MY-PRIVATE-KEY

Now that this buffer has been unmapped, it is no longer part of the memory image of the process, and therefore it is inaccessible to gcore. It is easy to reason about this in C, where one can deterministically allocate and free memory, but when looking at a garbage collected language such as Go, this can be trickier to reason about. Let's take a look at a Go program that has a secret key in memory.

// exploitable.go
package main

import (
  "bufio"
  "fmt"
  "os"
  "strings"
)

func main() {
  reader := bufio.NewReader(os.Stdin)

  key := make([]byte, 100)
  copy(key, strings.ToUpper(os.Args[1]))
  key[0] += 1
  fmt.Printf("First byte of my key: %d\n", key[0])

  fmt.Println("now do the \"before\" core dump!")
  reader.ReadString('\n')

  // attempt to "ero" key
  key = []byte{}

  fmt.Println("now do the \"after\" core dump!")
  reader.ReadString('\n')
}

Let's see what's going on in this code. We are reading in a key from input arguments, then upper-casing it and storing it into a byte array. We do this to ensure that the uppercase version will be in heap memory, then we increment the first byte of the byte array to be able to distinguish which string belongs to which buffer. Let's build this and run it, dumping core along the way.

$ go build exploitable.go
$ ./exploitable my-private-key
First byte of my key: 78
now do the "before" core dump!
now do the "after" core dump!

Let's take another look at the new core dumps! Remember, we both upper-cased the string and incremented it when we stored it in our key buffer, hence an ascii ‘M' becomes an ‘N' when we read it from memory.

$ strings before.1367438 | grep PRIVATE-KEY
NY-PRIVATE-KEY
MY-PRIVATE-KEY
$ strings after.1367438 | grep PRIVATE-KEY
NY-PRIVATE-KEY
MY-PRIVATE-KEY

Inspecting the core dumps, we can see that they both contain the same private keys! Changing the variable key did nothing to change the private information in memory! Well in this case, we are dealing with a garbage collected language, which means that the exact moment when memory gets unmapped is non-deterministic. Even though there is nothing referencing the chunk of memory that this string is in, it still has not been unmapped from the process's memory. Don't get me wrong, garbage collectors are an amazing technology that accelerates development speed, but there are also consequences in the sense that it is difficult to reason about memory usage and whether you still have sensitive data still in memory at any given point in time. Something we can do is to loop through the byte array, and overwrite each item with zero. This will clear out any sensitive data from the buffer. We can replace the "zero"ing code with the following:

 // attempt to "zero" key
 for i := 0; i < len(key); i++ {
 key[i] = 0
}

Now when we search for this string, we see that it is no longer visible in memory. We've effectively erased it!

$ strings after.1444047 | grep NY-PRIVATE-KEY

It can be quite cumbersome, however, to get this right every single time, and as a codebase evolves over the weeks and months and years, it can be difficult to ensure that this is always being caught. At Cape, we are being prudent about clearing sensitive information before invoking any user code, but as always with security, it's best to take a multi-layered approach in case something goes wrong at another layer. Enter capejail Within AWS Nitro Enclaves, there are restrictions to what code is allowed to do. For example, we have not yet gotten Docker containers or other similar containerization and jailing technology working within AWS Nitro Enclaves. Because we have our Runtime process running inside of the AWS Nitro Enclave alongside arbitrary user code, we needed to have a way to protect the Runtime's private information such as AWS credentials and private keys. This is why we created capejail. Capejail launches a process in a new PID namespace and in a chroot environment, closed off from the rest of the system and runs as a user that has locked down permissions. On top of this, capejail applies seccomp filters that block certain system calls that can be used to escape a chroot jail or attack other processes within the enclave. Let's see what happens if the worst case scenario happens – a malicious user exploits a zero day vulnerability in the Linux kernel to gain root access within the enclave.

$ sudo capejail -u root -r ~/chroot-gcore -- bash
[jail]# gcore 1457587
/usr/bin/gcore: line 96: 1460414 Bad system call (core dumped) "$binary_path/gdb" --nx --batch --readnever -ex "set pagination off" -ex "set height 0" -ex "set width 0" "${dump_all_cmds[@]}" -ex "attach $pid" -ex "gcore $prefix.$pid" -ex detach -ex quit < /dev/null
gcore: failed to create core.1457587
[jail]#

We see that even though the attacker had root access, they still were not able to gcore the Runtime process. Capejail blocked it from doing so! Without capejail, this zero day exploit would have left the Runtime vulnerable to the gcore attack that we described above, but with this layered approach, we are able to protect against this attack, in this case, by blocking the ptrace system call that gcore depends on. Of course, not allowing user code to become root is the first line of defense, but exploits do happen, and having more layers between your secret information and malicious actors is always the best approach!

Check out the Getting Started Docs to try Cape for free. We'd love to hear what you think.