VRT Diff Classification: How VisualQ Learns Bug vs Evolution
Visual regression testing is excellent at one question: did these two screenshots differ?
It is weak at the next one: is that difference a bug, or an evolution you shipped on purpose?
Without help, humans open every failed viewport × browser pair, guess intent, approve or investigate, and repeat. That does not scale.
Diff classification adds a second layer on top of VisualQ’s pixel engine. After a run fails, VisualQ suggests a class for each changed region, surfaces the signal behind it, and — only when statistical risk is bounded — marks some regions for lighter review. Detection stays pixel-based. Classification is the learning loop.
Two classes, not three
Every prediction and every human label uses one of two classes:
- Bug — something is broken. Typical action: investigate and fix.
- Evolution — a deliberate product or design change. Typical action: approve and update the baseline.
Environment noise — anti-aliasing, third-party widgets, mono-browser quirks — is classified as a bug. From a user’s point of view a rendering artefact is still something broken, even when the cause is the renderer rather than the commit. The fix may be a comparison rule instead of a code change, but the diff is not an evolution and pretending otherwise teaches the classifier the wrong lesson. Ties and no-signal cases also fall to bug: the safe direction.
The unit is a region, not a screenshot
A page rarely changes in one way at a time. A release can move the hero on purpose and break a button in the header in the same deploy, and a single verdict for the whole screenshot is wrong either way.
So VisualQ classifies one changed region at a time. The comparison engine already knows which page sections moved; each of them gets its own class, its own signal, and its own human decision. A row in the results matrix reads as what it is — for instance two regions flagged bug and one flagged evolution — instead of collapsing to a single guess.
Regions keep a stable identity across runs (the section id from the page, a hash of its selector, or its position on the page as a fallback), so a correction made today still applies to the same section tomorrow.
What this is not
Classification does not replace screenshot capture, anti-shift comparison, or the fail/pass gate. An “evolution” suggestion on a failing region does not promote the baseline by itself in v1 — labelling and approving stay separate actions.
Pixels still find the change. The classifier explains and prioritizes it.
The learning flywheel
1. A VRT run finishes and one frozen snapshot is built per changed region.
2. The classifier writes a suggested class on each unlabeled region.
3. When a human confirms or corrects a region, that label is stamped and also written — anonymized — into a shared memory.
4. Later runs use that memory, with extra weight for neighbors from the same project.
The correction is the product. One click to say “no, that one is a bug” is both the fastest way to review a run and the only training signal the model gets.
Day one, with no labels yet: only heuristics. After enough labeled history: nearest-neighbor classification becomes available. After enough local calibration: a project may unlock auto-triage.
Hybrid cascade (how a suggestion is made)
Everything runs in-process in VisualQ’s TypeScript stack. There is no external Python model service for this path.
- First, each region becomes a versioned numeric feature vector (what changed inside that region, magnitude, spatial stats, DOM/shift signals, history, sibling viewports, GitHub / JIRA / Figma intent when present, distilled Smart Diff fields). Missing integrations are true missing values — never fake zeros.
- Heuristics always run: deterministic rules for day one (mono-browser noise, isolated pixels, an interactive element changing, design-intent signals, and more). Heuristic results never auto-triage.
- k-NN runs when there is enough labeled context: distance-weighted neighbors over anonymized history, with project-local neighbors over-weighted.
- A conformal gate per project decides whether a suggestion is safe enough for auto-triage — roughly: among low-risk regions, the measured chance of missing a true bug stays within a tight bound.
- On deferred regions, an optional second opinion is distilled from Smart Diff analysis already on the bundle. It never silently overrides the primary class.
Every triage carries an explicit source: heuristic or model. No silent mix on a single region.
Privacy: learn globally without leaking client data
Labeled training rows live in a dedicated store that holds only:
- Feature version and numeric features
- Label, label source, timestamp
- Salted hashes of org and project identity
Never stored: URLs, scenario names, images, free text, or raw org/project ids. Client SDKs cannot read this store. Without the salt configured, learning writes are skipped; classification still runs on heuristics.
Progressive autonomy
- Suggest only (default) — the viewer shows the class and the strongest signal for each region. Humans always decide.
- Eligible for auto-triage — after enough local calibration pairs and a valid risk threshold, a project can enable the toggle.
- Auto-triage on — regions under the conformal threshold are marked for lighter review. v1 does not auto-promote baselines.
Calibration is built in the shadow: when a prediction exists and a human later labels the region, the pair (bug probability, predicted class, human label) is recorded for that project.
Where you see it in VisualQ
- After each VRT run, triage runs in the background and never blocks the test pipeline.
- The diff viewer triage panel lists each changed region with its class, the signal behind it, and a one-click correction.
- The results matrix counts the regions per row — for example two bug and one evolution — in the same colours as the viewer.
- Correcting a region, or approving a row, feeds the learning loop.
- Coding agents get the same breakdown over MCP, so an assistant can name the broken section rather than the whole page.
- Teams can re-run triage and manage auto-triage eligibility from project settings APIs.
One sentence to remember
VisualQ already detects visual differences. Classification points at each changed region and calls it a bug or an evolution — using your team’s past corrections, anonymized — with a hard brake so automation only activates when missing a real bug is statistically rare.
Pixels find the change. The cascade explains and prioritizes it. A human can always overrule the call, and that correction is exactly what trains the classifier.
*VisualQ product engineering — July 2026.*