Core Dumb - HitCon 2019

Note: challenge was solved together with Disconnected.

Damn it my flag checker is so buggy it destroyed the program itself 😱
All I left is a core dump file :(
Could you help me recover the flag ? Q_Q

Let's start by checking the file that we are given in this challenge:

λ file core-3c5a47af728e9968fd7a6bb41fbf573cd52677bc
core-3c5a47af728e9968fd7a6bb41fbf573cd52677bc: ELF 64-bit LSB core file, x86-64, version 1 (SYSV), SVR4-style, from '/home/ctf/flag_checker'

So we are sure that this is a core dump, but what I did not know that such file can be opened in Ida or Ghidra without any problems. I always had the impression that only gdb can handle those. Anyway, during the competition we did use Ida but I'll use Ghidra with the decompiler during the writeup.

Static analysis with Ghidra

After opening the file in Ghidra and analyzing it we can see a couple of function in the memory range of 0x55555555xxxx but only one of them could be considered interesting - at least from the first look: 0x555555554c7e.

The fact that it contains strings like "Please enter the flag:" and "Congratz! The flag is hitcon{%s}" :)\n" was also a strong indicator that it's the right place to start. From the string usages we can quickly identify that FUN_555555554730 is some kind of printf or puts and FUN_555555554740 reads the flag - up to 0x37 characters and stores it in variable local_d8. Later we see the check that in fact limits the flag length to 0x34 characters.

iVar1 = FUN_55555555488a(flag);
if (iVar1 != 0x34) {
  FUN_555555554949();
}

After that we see a series of calls

FUN_5555555548bc(&local_98,flag,10,flag);
FUN_555555554a38(local_128,local_120,&local_98,10);
FUN_55555555490f(&local_98,0x37);
FUN_5555555548bc(&local_98,flag + 10,8,flag + 10);
FUN_555555554b0c(local_118,local_110,&local_98,local_118);
FUN_55555555490f(&local_98,0x37);
FUN_5555555548bc(&local_98,flag + 0x12,0x12,flag + 0x12);
FUN_555555554a38(local_108,local_100,&local_98,0x12);
FUN_55555555490f(&local_98,0x37);
FUN_5555555548bc(&local_98,flag + 0x24,0xc,flag + 0x24);
FUN_555555554a38(local_f8,local_f0,&local_98,0xc);
FUN_55555555490f(&local_98,0x37);
FUN_5555555548bc(&local_98,flag + 0x30,4,flag + 0x30);
FUN_555555554b0c(local_e8,local_e0,&local_98,local_e8);

that is followed with printing the success message with the flag . We just need to analyze and understand what's going on in those functions. A bit closer inspection reveals that there are 5 groups by 3 calls.
Let's write them like that:

FUN_5555555548bc(&local_98,flag,10,flag);
FUN_555555554a38(local_128,local_120,&local_98,10);
FUN_55555555490f(&local_98,0x37);

FUN_5555555548bc(&local_98,flag + 10,8,flag + 10);
FUN_555555554b0c(local_118,local_110,&local_98,local_118);
FUN_55555555490f(&local_98,0x37);

FUN_5555555548bc(&local_98,flag + 0x12,0x12,flag + 0x12);
FUN_555555554a38(local_108,local_100,&local_98,0x12);
FUN_55555555490f(&local_98,0x37);

FUN_5555555548bc(&local_98,flag + 0x24,0xc,flag + 0x24);
FUN_555555554a38(local_f8,local_f0,&local_98,0xc);
FUN_55555555490f(&local_98,0x37);

FUN_5555555548bc(&local_98,flag + 0x30,4,flag + 0x30);
FUN_555555554b0c(local_e8,local_e0,&local_98,local_e8);

Each group starts with a call to FUN_555555555548bc and closes with FUN_55555555490f (except for the last one). The call in the middle is not always the same but it alternating between FUN_555555554a38 and FUN_55555555b0c.

The first one - FUN_5555555548bc - after simple renaming looks like a memncpy

ulong _memncpy(undefined *dst,undefined *src,int cnt)    
{
  undefined *_src;
  uint i;
  undefined *_dst;
  i = 0;
  _src = src;
  _dst = dst;
  while ((int)i < cnt) 
  {
    *_dst = *_src;
    i = i + 1;
    _src = _src + 1;
    _dst = _dst + 1;
  }
  return (ulong)i;
}

Before we analyze the function in the middle let's look at the last one (in the group): FUN_55555555490f. Again with simple renaming we can we the purpose of this function is to clear the memory:

void memset(undefined *mem,int cnt) 
{
  int i; 
  i = 0;
  while (i < cnt) 
  {
    mem[i] = 0;
    i = i + 1;
  }
  return;
}

Ok, so now is the time to tackle those functions in the middle.

Let's start with FUN_555555554a38. It's a bit more complex but the most interesting part is:

  FUN_555555554963(param_1,param_2,local_518,param_1);
  iVar1 = (*(code *)local_518)(param_3,(ulong)param_4,param_3,(ulong)param_4);
  if (iVar1 == 0) {
    FUN_555555554949();
  }

What we can see here is that we execute one function and than after that we should be able to call content of the buffer local_518 and treat is as a function that takes 4 arguments.

Now we need to find out from where those params are being passed from. To do that we need to get back to the main function where we began.

If w scroll a bit above the code from reading the flag we can spot the following code (after a bit of cleaning).

  data[0].data = &DAT_555555756020;
  data[1].data = &DAT_555555756140;
  data[2].data = &DAT_555555756300;
  data[3].data = &DAT_555555756600;
  data[4].data = &DAT_555555756a00;
  i = 1;
  while (i < 6) {
    data[i].header.key = (&xor_keys)[(long)(i + -1) * 4];
    data[i].header.size = (&sizes)[i + -1];
    i = i + 1;
  }
  data[0].key_size.key = DWORD_555555756af4;
  data[0].key_size.size = DAT_555555756b14;

We can se the data for decrypt methods are being prepared. If we would inspect those DAT_555555756xxx addresses we would find arrays of bytes to be decoded and then later executed to verify the flag.

decrypt_and_run(data[0].data,data[1].header,&buf,10);

Data and header are being passed to decode(data,header,(char *)&local_518); and inside the key is used to xor bytes in the data.

buf[i] = src[i] ^ *(byte *)((long)&local_24 + (long)(int)((i + uVar1 & 3) - uVar1));

After the decrypting, the call is made with part of the flag and the len.

Ok, so we understand the first of the functions. The second one (FUN_555555554b0c) is quite similar with one difference in calling the decrypted code - in this case only with one argument.

So we need to decrypt those data and start looking what is inside. We can do that with the following script:

With that we got 5 files part[0-4].raw which we need to analyze. This is a good part during the CTF that could be done in parallel. We will see that in part 2.

Part 0

This was a simple xor with a static key.

Part 1

Modified XTEA.

Part 2

Custom (or at least I don't know its name) algorithm.

Part 3

Modified RC4.

Part 4

Small, bruteforce-able checksum calculation algorithm that uses parts of 1-byte CRC calculation.