Identifying Malicious Browser Extensions Using Feature Engineering, Contextual Publisher Analysis, and a Weighted Ensemble of Random Forest and XGBoost on 491,308 Extensions

Most security vendors lead with a single hero number: "99% accuracy detecting browser extension malware." We can quote that number too — our Risk Engine's latest training run hits 99.28% accuracy on a holdout set of 284,391 extension versions. We won't lead with it, because it's the number that matters least.

When we sliced the same model by extension age, the picture changed:

On extensions aged 90 to 365 days — the window where the most damaging supply-chain attacks tend to live — recall drops to 71.16%. Roughly 29 of every 100 malicious updates in that window are missed by the ML component of our Risk Engine alone.

This post explains how the Risk Engine works, how we measured that gap, why it happens, why our existing layered defenses don't fully close it, and what we're shipping next. We're publishing the number because it's the honest answer to the question every security team should be asking: where is your engine weak?

What we're actually detecting

The pipeline is a four-stage funnel that runs continuously over the entire Chrome Web Store. The third stage — the ML scorer — is what produces the headline numbers in this post.

We train on 1,421,952 extension versions spanning 491,308 unique extensions (1,159,131 benign versions, 262,821 malicious — a 4.41 : 1 imbalance). Each version is encoded as 123 features drawn from five sources:

• Static code features (40) — regex over HTML, JS, and CSS files: counts of eval, innerHTML, XMLHttpRequest, obfuscation patterns, data-URIs, hidden iframes, and the rest of the academic feature set.
• File composition and manifest features (12) — file counts by type, total JS size, manifest version, background-script presence, search-provider override.
• Permission analysis (17) — booleans for webRequest, webRequestBlocking, chrome.debugger, dynamic tabs.executeScript, broad host access, plus permission counts and DNR rule shapes.
• Reputation features (15) — publisher aggregates (active / obsolete / malicious / UWS sibling counts), permhash cluster membership, IOC list hits, CWS metadata (rating, user count, featured/unlisted flags, age).
• Version-delta and dataflow features (18) — diffs against the previous version (new domains, added JS, introduced eval / executeScript) and lightweight source-to-sink dataflow markers.

Two models train on this matrix: a RandomForest (175 trees, class_weight="balanced") and an XGBoost (300 trees, depth 6, scale_pos_weight = 4.41). We then combine them as a weighted ensemble: score = 0.7 × RF_prob + 0.3 × XGB_prob.

The blend isn't arbitrary. RandomForest alone is a precision specialist: 99.30% precision but only 94.64% recall. XGBoost alone is a recall specialist: 98.93% recall at the cost of 66.37% precision (it flags 11% of benign extensions as malicious). Neither ships well on its own. The weighted blend lands at 98.77% precision and 97.33% recall — strictly better than either model on F1, and the right operating point when "don't falsely flag a popular benign extension" and "don't miss a malware campaign" are equally expensive mistakes.

Which of the 123 features actually matter?

We pulled feature importances directly out of the deployed RandomForest. The answer is not what most extension-security writing assumes:

Publisher reputation dominates. The top five features — publisher_active_extensions, days_since_last_update, publisher_malicious_extensions, publisher_obsolete_rate, publisher_obsolete_extensions — together account for ~65% of the model's predictive power. The classic "look at the code" features (regex over JS/HTML/CSS) appear far down the list. permhash_malware_rate — the strongest signal in our first model two years ago — has fallen to sixth as the publisher-context features got reliable.

The implication for the rest of this post: when we say the ML scorer struggles on extensions aged 90 to 365 days, we're really saying it struggles when the publisher context is unrevealing. That's the lens for everything that follows.

That's the headline picture. Now for what's underneath it.

The weak spot: 90 to 365 days

The 97% global recall hides a U-shaped curve.

Random train/test splits sample uniformly across ages, so a 29% miss rate concentrated in one age band gets washed out by the easy buckets. The global recall number doesn't lie — it just averages across a population that isn't uniformly hard.

Why this specific window is hard: it's the regime where the most expensive supply-chain attacks happen. The pattern goes like this:

1. A developer ships a benign extension. Or buys one from an original benign developer.
2. It accumulates users for several months while behaving normally — so it's no longer a "new and suspicious" 0–90 day extension.
3. The developer ships a malicious update.
4. The Chrome Web Store eventually catches it and obsoletes it — but until that happens, the model's strongest signals don't fire:
   • Permhash reputation — its permission set hasn't been seen on confirmed malware yet, so permhash_malware_rate is zero.
   • Publisher reputation — the publisher has no malicious sibling extensions yet (they're all the same extension, pre-malicious), so publisher_malicious_extensions is zero.
   • User-count anomaly — it already has a healthy install base from its benign past.
   • Age — at six months old it looks established, not suspect.

The features that should catch this case are the version-delta features — delta_introduced_api_eval, delta_new_domain_count, delta_total_suspicious_api_additions, and friends. The 28.84% miss rate is the empirical evidence that they're real signals but insufficient. The Cyberhaven extension breach in late 2024 — and the smaller, less-reported attacks that follow this same pattern every month — happen in exactly this band.

Why our existing layers don't fully close it

We don't ship ML alone. Every flagged extension goes through additional layers — but a careful look shows none of them, individually or in combination, fully closes the 90-to-365-day gap.

Claude code analysis on Critical/High scores

When ML produces a critical or high ml_risk_score, we hand the extension's source code to Claude (Sonnet 4.6) for code-level review. We've measured Claude's performance against a labeled corpus:

The numbers are striking:

• Precision: 1.000 — every extension Claude labels as malicious truly is malicious.
• Recall on malware: 0.394 — Claude only labels 39.4% of confirmed malware as malicious. It silently clears 47.1% as false_positive.

This is by design. Claude isn't the Risk Engine's primary classifier — it's a precision-first filter layered on top of the ML model to suppress false positives before a human reviews them. A code_analysis row from a weaker LLM is worse than a NULL row, because it makes a version look reviewed when the verdict is unreliable.

But this design has a consequence for the weak spot:

Claude is on the wrong side of the funnel to fix recall. It only sees what ML has already flagged. The extensions ML misses in the 90-to-365-day window never get a critical or high score, so they never reach Claude at all. Adding more LLM review at the top of the funnel can reduce noise above the threshold — it can't recover false negatives that ML didn't surface.

Continuous monitoring

Our other major layer is continuous re-scoring: every new version of every extension on the Chrome Web Store gets ingested, hashed, and scored. This is the right architecture for catching delayed-malicious-update attacks — by definition, the attack lives in a new version of a previously-clean extension.

But "monitoring every new version" only converts to "catching every new version" if the model can actually distinguish the malicious update from the benign baseline. The 28.84% FNR in the weak window is exactly the measurement of how often it can't. Continuous monitoring is necessary but not sufficient.

What we're shipping to close it

The data points at three concrete improvements. Two are in active development; one is exploratory.

1. Stop wasting the publisher email signal we already have

Every publisher on the Chrome Web Store has a contact email. We enrich those emails with a reputation lookup that we store in extension_publishers.email_reputation as JSONB:

{
  "email": "...",
  "score": 100,
  "status": "VALID",
  "validations": {
    "syntax": true,
    "mx_records": true,
    "domain_exists": true,
    "is_disposable": false,
    "is_role_based": false,
    "mailbox_exists": true
  }
}

There's a numeric score, a status enum, and six independent validation booleans — about six useful bits per publisher. Today our ML model reads one of them: a single publisher_email_invalid boolean derived from the older email_is_invalid column.

82.3% of publishers have a populated email_reputation JSONB. The full reputation signal currently only flows into the LLM stage as prompt context — the ML model never sees is_disposable, is_role_based, mailbox_exists, or the numeric score. We're extracting these as four new features in the next training run. They're particularly relevant for the 90-to-365-day band, because a takeover attacker often updates the publisher contact to a thinly-validated address shortly before pushing the malicious update — a signal that's invisible to the current one-bit feature.

2. AST-level version deltas, not regex deltas

The current version-delta features compare two ZIPs at the regex level — counting added/removed lines of JS, added domains in string literals, introduced eval tokens. Regex catches the obvious cases. It misses restructured code: an attacker who refactors the bundle layout between versions can hide a new eval sink behind unchanged token counts. We're adding AST-level diffs that compare the syntactic structure across versions, so a sink that moves in the source is detected as a sink that appeared.

3. Behavioral signals (ex

exploratory)

The features above are all static — they look at the bytes of the extension without running it. Some attacks only manifest at runtime: a benign-looking content script that pulls its payload from a remote server on first execution, for example. We're exploring lightweight sandbox execution for the highest-risk flagged versions, capturing the runtime network calls and DOM mutations. This is research, not roadmap — sandbox execution at scale is expensive, and we want to measure whether the marginal recall lift justifies the cost before committing.

Why we're publishing the 29% number

There's an obvious commercial argument against telling buyers "our model misses 29% of the hardest cases." We're publishing it anyway, for three reasons.

It's the honest answer. Stratified metrics are the only metrics that matter once you ship a model in production. Global accuracy averages across an unrepresentative population; the question "where does this fail?" is what determines whether a buyer should trust it.

It's the product argument. The 29% number is precisely why a one-time scan of an extension catalog is not a defensible security posture. The extensions ML misses on day-one can be caught — by monitoring across versions, by enriching publisher reputation, by layering precision-first LLM review on top. None of those work without continuous re-evaluation. The layered architecture isn't a feature checklist; it's the response to a measured limitation.

It's a credibility filter for our buyers. A security tool vendor who publishes its blind spots is a vendor you can audit. A vendor who only publishes its headline metrics is one you'll learn to distrust the first time a missed extension shows up in your incident report.

We're committed to keeping this measurement public. Every retrain regenerates the stratified breakdown in our internal training_report.json; the charts in this post are produced by a script that consumes that file, so the next version of this post will show whether the 90-to-365-day band closed.

If you're running browser extension security at your organization and the "scan once, hope for the best" model has been quietly bothering you — that instinct is correct. The 29% number is the data behind it.

Try Extension Auditor

We continuously monitor every version of every Chrome Web Store extension and re-score on every change. That's the architecture this post is arguing for, and it's the only architecture that addresses the weak spot we just walked through.

See your organization's extensions — or read our earlier post, We Scanned 534,000 Chrome Extensions. Here's What We Found., for the dataset this all sits on top of.