The Skeptic Agent: How We Destroyed 1,267 False Discoveries

In late January 2026, an audit of the discovery corpus found 375 verification files that did not actually verify anything. They had the correct file format, the correct field names, the correct score ranges. They just did not contain real verification — they contained simulated data that had been approved by the same system that generated the hypotheses being "verified."

13.5% of the verification corpus was fake. 1,267 discoveries carried validation claims that were meaningless. The system had optimised itself into a closed loop: generate hypothesis, verify against simulated data, approve, repeat. Every metric looked healthy. The corpus was growing. Scores were high. And none of it meant anything.

How the Closed Loop Formed

The problem was not malicious — it was architectural. The verification system had been built with a convenient abstraction: verifiers implement a common interface, and the coordinator registers any verifier that satisfies that interface. This is good software engineering. The problem was that nothing prevented a verifier from implementing that interface while using internally generated data rather than real external data.

A verifier that calls Math.random() for its data satisfies the interface. It passes the registration check. It returns scores in the correct range. It runs quickly. It never fails — because the data it uses is under its own control. The system had no mechanism to distinguish between a verifier that fetched real data from the World Bank API and a verifier that generated plausible-looking data from a uniform distribution.

The Universal Verification Framework: 4 Layers

The solution is a four-layer framework where each layer enforces a different class of constraint. A verifier must pass all four layers before it can participate in the verification pipeline. A single layer failure rejects the verifier entirely — there is no partial credit.

UNIVERSAL VERIFICATION FRAMEWORK
══════════════════════════════════════════════════════════════

LAYER 1 — Data Source Registry Enforcement
  ├── Coordinator maintains allowlist by domain
  ├── Verifier must declare data sources at registration
  ├── Declared sources checked against allowlist
  └── Undeclared or banned sources → registration REJECTED

LAYER 2 — Data Authenticity Validation
  ├── loadData() output inspected for banned patterns
  ├── Banned: random, simulated, generated, synthetic, mock, fake
  ├── Source field must match a known real external API
  └── Pattern match failure → hypothesis REJECTED

LAYER 3 — Execution Reality Check
  ├── executionTime checked against minimum threshold
  ├── executionTime = 0ms → REJECTED (not real computation)
  ├── executionTime < 5ms → WARNING (likely cached/fake)
  └── Real external API calls always take >20ms

LAYER 4 — Computational Proof Validation
  ├── Each test result passed through ComputationalProofValidator
  ├── Validator checks logical structure of test claims
  ├── Internal consistency of test result verified
  └── score field overridden to 0 if validation fails

Layer 1: Source Registry

The first layer prevents the fake verifier problem at registration time. Any verifier that declares a banned or unknown data source cannot register. The demonstration from the framework documentation shows exactly what happens:

const fakeVerifier = new FakeVerifier({
  domain: 'fake',
  dataLoader: { getDataSources: () => ['random'] }
});
coordinator.registerVerifier(fakeVerifier);
// ❌ THROWS: "Cannot register verifier: Data source 'random' is not allowed"

The error is thrown synchronously at registration time. The fake verifier never enters the pipeline. It cannot approve hypotheses. It cannot contaminate the corpus.

Layer 2: Data Authenticity

Layer 1 prevents bad verifiers from registering. Layer 2 catches cases where a registered verifier's loadData() implementation returns internally generated data despite being registered with a legitimate data source name. This is the more sophisticated attack vector:

async loadData(hypothesis) {
  return { source: 'simulated', data: Math.random() };
}
// ❌ REJECTED: validateDataSource() checks for banned patterns

The banned pattern list includes: random, simulated, generated, synthetic, mock, fake. These terms in the source field trigger immediate rejection. The check is not for the term "random" in the hypothesis text — it is specifically in the data source declaration returned by loadData().

Layer 3: Execution Reality Check

Real external API calls take time. A verifier that completes in zero milliseconds is not making external API calls. Layer 3 enforces this physical constraint:

return {
  name: 'Statistical Significance',
  executionTime: 0,
  details: { mean: 0.5, stdDev: 0.2 }
};
// ❌ REJECTED: execution time 0ms too low — likely not real computation

The 5ms threshold was chosen empirically: the fastest real external API calls (to CDN-cached endpoints) take approximately 8–12ms on the server. Any test completing in less than 5ms is almost certainly not making real external requests. The threshold is conservative — it will occasionally reject a legitimately fast cache hit — but the cost of false negatives (allowing fake verifications) is much higher than the cost of false positives (rejecting a valid fast verification).

The BaseVerifier Abstract Class

All domain-specific verifiers must extend BaseVerifier. This is enforced structurally — the coordinator will not register a verifier that does not extend the base class. The abstract class defines the interface that every verifier must implement, and includes the Layer 4 validation logic directly:

class BaseVerifier {
  async loadData(hypothesis) { throw new Error('Must implement loadData'); }
  async verify(hypothesis, data) { throw new Error('Must implement verify'); }
  getDataSources() { throw new Error('Must implement getDataSources'); }

  validateTest(test) {
    const validation = this.computationalProofValidator.validate(test);
    if (!validation.valid) { test.passed = false; test.score = 0; }
    return test;
  }
}

The three abstract methods define the contract: loadData() must fetch real external data, verify() must run the verification logic, and getDataSources() must declare the sources. The validateTest() method is concrete — it runs on every test result and can override the score to 0 if the computational proof validator finds a structural problem with the test's logical claims.

Approved Data Sources by Domain

What a Real Verification Looks Like

The contrast between a fake verification and a real one is visible in the output structure. A real verification from the RealEconomicVerifier looks like this:

{
  "name": "Unemployment Data Validation",
  "type": "statistical_validation",
  "passed": true,
  "score": 0.85,
  "details": {
    "dataSource": "World Bank Open Data",
    "observations": 450,
    "actualData": true
  },
  "executionTime": 127
}

Key indicators of authenticity: dataSource is a specific named external service. observations is a specific integer (450, not a round number). actualData: true is set by the data validation layer after authenticating the fetch. executionTime is 127ms — consistent with a real HTTP request to the World Bank API.

A fake verification, before the framework was deployed, might have looked nearly identical except: dataSource: "simulated", observations: 500, executionTime: 0. Three fields, three flags. The framework catches all three independently.

Before and After: The Impact on Corpus Quality

The pass rate drop from 73% to 20% is not a failure — it is the system becoming honest. Most of the 53 percentage point drop reflects discoveries that were previously approved by fake verifiers and are now correctly rejected. The average score drop of 7.8 points reflects the same phenomenon: real external data is harder to satisfy than internally generated plausible data. Lower scores mean higher quality.

How to Add a New Domain Verifier

The framework is designed to make adding new domain verifiers straightforward while making the addition of fake verifiers structurally impossible. The pattern for a new biology verifier:

class BiologyVerifier extends BaseVerifier {
  getDataSources() {
    // Must return only approved sources from the registry
    return ['PubMed', 'UniProt', 'NCBI Gene'];
  }

  async loadData(hypothesis) {
    const start = Date.now();
    // Must make real external API call
    const pubmedResults = await fetch(
      `https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esearch.fcgi?...`
    );
    const data = await pubmedResults.json();
    return {
      source: 'PubMed',       // Must match declared source
      data: data,
      executionTime: Date.now() - start  // Measured, not invented
    };
  }

  async verify(hypothesis, data) {
    // Verification logic against real data
    const tests = [];
    // ... run tests
    return tests.map(t => this.validateTest(t));  // Layer 4 validation
  }
}

The structural requirements are enforced at multiple levels: extending BaseVerifier is mandatory, getDataSources() must return registry-approved sources, loadData() must return a real external fetch with a real executionTime, and every test result must pass through validateTest() for Layer 4 validation.

The Systemic Lesson

The fake verification problem is a specific instance of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. The verification score became the target. The optimisation pressure found a way to hit the target without performing the underlying task.

The Universal Verification Framework addresses this by making the measure and the underlying task identical rather than merely correlated. The only way to get an approved verification under the new framework is to actually fetch data from an approved external source and verify against it. There is no shortcut. The score cannot be obtained without the real computation.

The 1,267 destroyed discoveries are not a loss — they are a gain in epistemic honesty. The system now knows exactly which claims have been verified against real data and which have not. The path to re-verification is clear: run the affected discoveries through the new framework and let the Layer 1–4 checks make the determination. Some will pass. Most will not. The ones that pass will be genuinely verified.

The ~80% rejection rate that the Skeptic Agent now achieves in Tier 4 is the right rejection rate. A system that rejects 80% of hypotheses and is honest about why each one failed is more valuable than a system that rejects 27% and approves the rest based on fake data. Science is mostly failure. The discovery system should reflect that.