YARA Malware Scanning For Mail: Rspamd, Milter, ICAP

Mailstrix owl mascot for YARA malware scanning in mail, ICAP and clamd paths — Mailstrix: the owl that finds malware hiding in your mail. Project home at mailstrix.com.

Somewhere out there a user is about to open Invoice_FINAL_v3.xlsm, and the only thing standing between them and a ransom note is a mail filter that actually bothered to look inside the attachment. That .xlsm hides its VBA source inside a compressed OLE2 file, which is itself tucked inside a ZIP container. A raw scan sees packed bytes and shrugs. The dangerous part is in there, wearing three coats, a fake moustache, and the confident smile of someone who has done this before.

Mailstrix is the small scanner that does the dismantling. YARA malware scanning works only when the rules can see something, so Mailstrix unwraps containers, decompresses macros, decodes the obfuscation, carves out the payload, and only then runs the rules. It’s open source on GitHub and ships as a container on Docker Hub.

It runs out of process, which is the whole design. libyara is a C library and a zoo of file parsers is exactly the code you don’t want sharing an address space with your mail daemon. Keep it outside and a parser crash costs you one scan instead of the queue.

A macro is not a text file

Old Word and Excel files such as .doc and .xls use Microsoft’s Compound File Binary Format, mercifully shortened to OLE2: a tiny filesystem crammed into one file, with named streams, storage directories, eldritch sector chains and several decades of forensic trauma. A macro project lives in those streams and its source is MS-OVBA-compressed, because of course it is. Modern .docm and .xlsm files dress better, but the macro project is still an OLE2 file buried in a ZIP like a body in a basement.

So a rule hunting for AutoOpen, WScript.Shell or a dodgy URL sails straight past a raw compressed macro and finds nothing. This is not YARA failing. This is YARA being shown a locked box and asked to describe the contents.

Mailstrix magic-sniffs the attachment, opens the OLE2 or OOXML, locates the VBA streams, decompresses them, and scans the recovered source as a separate buffer. It scans the original file too, because format-level exploits live there. For the extracted source it sets YARA’s VBA external variable, so macro rules can be precise instead of treating every newsletter containing the word “Shell” as the opening scene of a heist film.

YARA malware scanning in Rspamd sends messages and attachments to the out-of-process Mailstrix scanner — Any of six front ends asks an out-of-process scanner over HTTP, so attachment analysis can never block the mail worker’s event loop. Block the event loop and the only thing that gets quarantined is your weekend.

What suspicious VBA actually looks like

A macro running on open is not proof of malice. Plenty of finance departments run workbooks that resemble an archaeological dig with a quarterly budget. But a macro that auto-runs, writes a file to disk, and launches it has stopped being an annoying spreadsheet and become a line item in next week’s incident report.

Document_Open
  + ADODB.Stream or SaveToFile
  + WScript.Shell, ShellExecute, or PowerShell
  = a very bad Tuesday for someone who is not you

The local Maldoc_AutoExec_Write_Execute rule demands all three categories at once. That restraint is deliberate. A lone CreateObject is normal in half the macros ever written. A lone AutoOpen is everywhere. But an auto-execution trigger plus a file-drop primitive plus an execute primitive in the same buffer is a real signal, and it lets your filter add a score that means something instead of crying wolf at every monthly sales report.

Malicious VBA loves hiding its strings in pieces, like a ransom note assembled from magazine cuttings. It builds commands with Chr() and ChrW(), shuffles characters through Replace(), reverses literals with StrReverse(), XORs byte arrays, and occasionally trots out the Dridex-style string encoding that has been irritating analysts since roughly 2015. Mailstrix folds those forms back into readable text and scans the result again.

It also keeps going. The decode pass recurses up to four layers deep, so a payload that is base64 wrapped around hex wrapped around a reversed string comes apart instead of stopping at the outer skin. That matters because one layer is where most scanners quit and every dropper author knows it. A URL chopped into twenty innocent-looking fragments is still a URL once you scrape the glue off.

One limit, stated bluntly so nobody files a bug about it later: this is static analysis. Mailstrix doesn’t run the macro. It applies bounded decoding and structural checks, and full VBA interpretation still belongs to oletools and the parallel Olefy scanner. Two tools, one mail stream, zero faith in attachments.

The Office tricks that skip the macro entirely

Some document attacks don’t bother embedding a macro at all, because why work harder than the victim. An OOXML relationship can point Word at a remote template or OLE object, which is the remote-template-injection family: a clean-looking document politely asks Office to fetch someone else’s content the moment it opens. Mailstrix reads the _rels/*.rels parts and surfaces remote http, https, SMB and UNC targets. The attachment looks empty because the interesting part is still in transit.

DDE and DDEAUTO fields are another ancient problem that refuses to die quietly. A field instruction can convince Office to ask another program to do work on its behalf. Mailstrix pulls those instructions out of OOXML document, header and footer XML, including the split text runs that try to hide the letters “DDE” from anyone searching lazily, and it recognises the same idea in RTF, SYLK, CSV and Excel 2003 XML. Formula injection isn’t exotic when a cell starts with =; DDE is just the version where someone showed up with a crowbar and a grudge.

Excel 4.0 macros, usually called XLM, earn their own pile of suspicion. They predate VBA and live in macro sheets that can be hidden, or “very hidden”, which is Microsoft’s way of admitting the first hidden wasn’t sincere. A workbook can carry no visible VBA project at all while an XLM sheet fires a command through a formula language older than some of your colleagues. Mailstrix finds those sheets in OOXML, legacy BIFF, .xlsb and SLK, reassembles the formula strings from their raw tokens, then runs a bounded emulator over them: cell evaluation, GOTO, SET.VALUE, with five separate fuses to stop a runaway sheet. That’s why an obfuscated cell reference resolves at all. It still doesn’t pretend to be a full Excel interpreter. That ambition has eaten better-funded projects whole.

Then there’s VBA stomping, the trick for people who read the detection docs and got ideas. A document can carry compiled p-code that Office happily executes while the visible source is empty. Mailstrix weighs “substantial p-code present” against “implausibly tiny decompressed source” and raises VBA_Stomped. It’s a heuristic, not a disassembler, and that distinction stays boring right up until someone treats a score as a verdict and quarantines the chief accountant’s beloved 2004 workbook. Then it becomes your distinction to explain, slowly, in a meeting.

The attachment zoo is much larger than Office

RTF is the format people call harmless about four seconds before it hands them a hexadecimal object blob. Mailstrix decodes {\*\objdata ...}, follows the embedded OLE packages, and feeds every extracted child back through the same dispatcher, so an RTF object can unfold into a PDF, a script or yet another Office document under one shared size-and-depth budget. The budget is not optional. Skip it and an attachment becomes a small, well-dressed decompression bomb.

PDF gets suspicion rather than trust, having earned every ounce of it. Mailstrix inflates Flate streams and flags the actions that travel with trouble: JavaScript wired to /OpenAction, /Launch, embedded files, deliberately obfuscated names, and the historic /JBIG2Decode path. A PDF has no business behaving like a program loader. The specification disagrees, at length, which is how we got here.

The less glamorous routes are the more common ones, because attackers are nothing if not practical: Outlook .msg attachments, OneNote files, .lnk shortcuts carrying command lines, MSI payloads, VBE and JSE encoded scripts, Ole10Native package streams, Java .jar and Android .apk members, and the full ZIP / 7z / RAR / gzip / tar family. Every carved child returns to the same format detector. A PDF inside a message attachment inside a ZIP doesn’t get a hall pass because the attacker had patience and you, briefly, did not.

Two evasions deserve naming because they beat extension checks outright. A polyglot file can be a valid PE and a valid ZIP at once, so your gateway parses the archive while the endpoint runs the executable; Polyglot_PE_ZIP catches the overlap. And Renamed_Container fires when the type Mailstrix actually parsed contradicts a harmless-looking extension, which is the oldest dropper trick there is and still works on anyone trusting the filename.

Password-protected archives are the honest problem here. A scanner can’t read what it can’t decrypt, and the attacker knows it, which is why the password is helpfully written in the mail body. Set MAILSTRIX_ARCHIVE_PW=1 and Mailstrix scrapes candidates from the subject and body, tries them against ZipCrypto, WinZip-AES, 7z and RAR members, and scans whatever falls out. A password shipped alongside its own ciphertext is itself the tell.

Six places to bolt it into a mail path

The scanner is one service. How you reach it depends on where in the mail flow you want the CPU spent, and the repo ships all six front ends rather than making you write glue.

Rspamd. An async Lua plugin POSTs each message or part at SMTP time and turns hits into score symbols. Scoring and rejection, at the cost of scanning mail you were going to reject anyway.
SpamAssassin. A Mailstrix.pm plugin doing the same job through the same service.
Dovecot and Sieve. The lean strix-scan client scans at delivery and a Sieve rule quarantines a match. Less mail, off the connection’s critical path.
Postfix or Sendmail milter. strix-milter runs on the MTA host and stamps the verdict as a header. It always accepts; your milter_header_checks decides policy. A scanner outage can’t hold your mail hostage.
ICAP. Set MAILSTRIX_ICAP_ADDR and the same daemon speaks RFC 3507 REQMOD and RESPMOD, so Squid or c-icap can push web bodies through the identical engine.
clamd streams. Opt-in listeners accept INSTREAM from clamd clients, which is how you drop it into something that only ever learned to speak one antivirus protocol.

Here’s the opinion, and it will annoy somebody: YARA scanning probably does not belong at SMTP time. It’s genuinely CPU-hungry, the MTA hot path is the most latency-sensitive place you own, and at SMTP time you’re burning that CPU on mail you’ll reject on score alone. Scanning at delivery, after Rspamd has already thrown out the obvious junk, inspects far less and never makes a connection wait. Both paths work. Pick the one your mailflow can afford, and there is no trophy for making the fastest SMTP conversation also do the most work.

YARA malware scanning starts with evidence

Only after Mailstrix has produced clean, useful buffers does ordinary YARA matching get to do its best work. The service loads roughly 10,000 curated public rules from YARA-Forge, signature-base, ANY.RUN, Didier Stevens, bartblaze, InQuest, CAPEv2 and YARAify. Those rules see raw files, decoded macro source, extracted objects, and recovered scripts. They also get the attachment filename and extension, which matters enormously for rules expecting a lure named invoice.pdf.exe by someone who genuinely believed nobody would notice the second extension. Someone always notices.

Public rules catch known samples. For the ones nobody has filed yet, Mailstrix ships over a hundred rules of its own, and they score structure rather than bytes: oleid and pdfid parity indicators, XLM dropper shapes, HTML and SVG smuggling, the PowerShell and VBScript and JScript dropper families that arrive in waves, carved-PE entropy and overlay checks, and a weight-of-evidence rule that fires when five independently unremarkable structural markers show up in one document. None of those need a sample to have been seen before.

That rule set is a starting point, not a divine oracle. Mailstrix returns the rule name, its tags and the source namespace, sorted into score tiers so a confident family hit outranks a structural suspicion. Start every deployment at score 0.0, read the logs for a while, and tune the noisy rules before you let them reject anything. Detection-only mode is not cowardice. It’s how you avoid discovering, at 09:04 on a Monday, that payroll was your test corpus and the CFO is on line one.

Three optional abuse.ch feeds add evidence from outside the corpus. URLhaus checks URLs found in the message and extracted content, including defanged disguises such as hxxp and [.]; MalwareBazaar checks an attachment’s SHA-256; ThreatFox adds URL and domain IOCs from live campaigns. All three cache locally and fail open. Losing mail because a third party had a bad afternoon is how trust in a filter dies.

Updating rules without trusting the network

Detection rules go stale fast, so Mailstrix can pull a fresh compiled bundle with strixd fetch-rules and swap it in on SIGHUP, no new image and no restart. Which raises the obvious question: what stops somebody feeding your scanner a rule bundle of their own design?

The manifest has to carry a valid ed25519 signature from a public key compiled into the binary. No signature, wrong key, truncated file, and the update is refused before the compiler ever sees it. There’s no environment variable to turn that off, which is deliberate: a scanner that accepts unsigned detection logic from the internet is a remote code path wearing a security product’s badge, and you’d never find out until it mattered. A botched reload also keeps the previous compiled set live, so the worst case for a broken edit is stale rules rather than none.

What it does when things break

The separation from the mail daemon buys real operational peace. A scan timeout, a parse error, or an honest-to-goodness libyara panic reports “no match” and logs the failure instead of dropping the message. Verdicts are cached by SHA-256, and concurrent copies of the same campaign coalesce into a single scan. Your CPU did not consent to inspect the same poisoned invoice 500 times just because the sender blind-copied the entire org chart.

Under load there’s one effort dial from 1 to 10 that scales decode depth, the XLM and PDF clamps, the reputation feeds and the scan timeout together. Set it to auto and it sheds a level at a time as the admission gate fills, then climbs back as the queue drains. Degrading depth beats degrading availability. The alternative is a filter that’s thorough right until it falls over and becomes a filter that is nothing.

If you want a sandbox verdict in the mix there’s an optional CAPEv2 adapter, and it is deliberately report-only: the policy knob accepts exactly one value, so a pending detonation can never hold, tempfail or quarantine a message. Mail follows the static verdict. Detonation is something you read afterwards, not something your queue waits on.

None of this replaces your filter’s judgement. Rspamd already fuses authentication results, URL reputation, fuzzy hashes, Bayes and its own rules into one decision, and the wider tour lives in Rspamd explained. Mailstrix just pries open the attachment so that judgement has something to read.

The whole thing is open source at github.com/myguard-labs/mailstrix, with extractor tests and the rule build included, and the ready-to-run image lives on Docker Hub. The current release is v1.2.0, with static amd64 and arm64 binaries and .deb packages on the release page for anyone who would rather not run a container. The full project tour, deployment paths, and syntax reference live at mailstrix.com. Treat the score as another set of eyes, keep Office patched, and stay permanently suspicious of Q2_forecast_FINAL_FINAL.xlsm. That filename has never, in the entire history of email, brought good news.

Frequently asked questions

Does Rspamd have built-in YARA scanning?

No. Rspamd has a strong asynchronous plugin model but no native YARA module. A small Lua plugin sends messages and attachments to an out-of-process YARA scanner such as Mailstrix, then turns the returned evidence into ordinary Rspamd symbols and scores.

Can Mailstrix run without Rspamd?

Yes. The same scanner is reachable six ways: the Rspamd Lua plugin, a SpamAssassin plugin, the strix-scan client for Dovecot and Sieve delivery, a strix-milter front end for Postfix and Sendmail, an RFC 3507 ICAP listener for proxies such as Squid, and opt-in clamd INSTREAM listeners. All six share one engine, rule set and verdict cache.

Should YARA scanning run at SMTP time or at delivery?

Scanning at delivery is usually the better trade. YARA is CPU-intensive, and at SMTP time you spend that CPU on mail you will reject on score alone while a connection waits. Scanning after the spam filter has dropped the obvious junk inspects far less and keeps the work off the critical path. Mailstrix supports both.

Why must a scanner decompress Office VBA macros?

The source in an old OLE2 document, or in the vbaProject.bin inside a DOCM or XLSM, is MS-OVBA-compressed. A raw scan only sees packed bytes, so macro keywords and URLs are invisible. Decompressing the VBA gives YARA and behavioural rules the actual text to inspect.

What is VBA stomping?

VBA stomping is a document trick where compiled p-code stays present but the visible VBA source is empty or misleading. Office can still execute the p-code. Mailstrix flags the suspicious mismatch as a heuristic; it does not disassemble the p-code or claim to prove malware from that signal alone.

Can YARA scanning catch macro-free Office attacks?

Yes. Mailstrix looks for external OOXML relationships, DDE and DDEAUTO fields, hidden XLM macrosheets, OLE2Link URL monikers, emailed XLL add-ins, and suspicious embedded objects. Those techniques can fetch or execute content without any obvious VBA project in the document.

Which attachments besides Office documents are inspected?

Mailstrix handles RTF object data, PDF streams and actions, Outlook MSG attachments, OneNote files, LNK shortcuts, encoded scripts, MSI and OLE package payloads, HTML and SVG smuggling carriers, Java JAR and Android APK members, and nested ZIP, 7z, RAR, gzip and tar archives. Each extracted child is scanned again within strict depth and byte limits.

How are rule updates protected?

A downloaded rules manifest must carry a valid ed25519 signature from a public key pinned into the binary, or the update is refused before compilation. Verification cannot be disabled by configuration, and a failed reload keeps the previously compiled rule set running.

Will a YARA scanner block mail when it fails?

It should not. Mailstrix fails open on timeouts, parse errors, rule problems, and backend failures, and the milter front end always accepts. A scanner outage becomes an observability problem rather than a reason to lose a legitimate message.

Mailstrix: YARA malware scanning for your whole mail path