Skip to content

feat(validator): cross-model validation, anti-self-grading defense, and hardened exploitability checks (P0) #34

Open

Description

@Mahajan-Sachin

Motivation & Background

In SCANNER_AUGMENTATION_PLAN.md, item P0 highlights a critical precision dynamic in the scanner pipeline:
Currently, in openhack/agents/validator_swarm.py:32, model = settings.validator_model_id or self.llm.model. When validator_model_id is unset, the validation swarm defaults to using the identical model as the hunter swarm ("self-validation" / model grading its own work). In benchmark research (Benchmarks/lean-harness), this self-validation causes precision inflation and confirmation bias.

Additionally:

  • Tying votes (e.g. 1 confirmed vs 1 rejected) should not default to confirmed.
  • Common false-positive patterns in static findings (CSRF where cookies are SameSite=Lax/Strict, XXE where parsers do not expand external entities like xml2js/sax, XSS in auto-escaped JSX text nodes, and framework development default configurations) should have mandatory hardened validation checks.

Proposed Implementation

  1. Cross-Model Validation Resolution:
    • Resolve validator models to independent models by default (e.g., hunter gpt-5.6-sol / glm-5.2 -> validator kimi-k2.5).
    • Add a startup check and session trace event warning if the validator model is identical to the hunter model.
  2. Hardened Exploitability Prompts in VALIDATOR_PROMPT:
    • CSRF: Require inspection of cookie configuration and middleware; confirm only if auth cookie is NOT SameSite=Lax/Strict.
    • XXE: Confirm only if parser resolves external entities.
    • XSS: Confirm only if untrusted input reaches a raw-HTML sink (dangerouslySetInnerHTML, v-html, etc.), auto-escaped JSX is false_positive.
    • Framework dev configs (Nuxt DevTools, Vite HMR) and blank rationale -> false_positive.
    • Clarify false_positive strictly means mitigated or unexploitable (low-but-real remains confirmed low).
  3. Strict Majority Consensus Voting:
    • Configurable validator votes (validator_votes, e.g. 3 for 2-of-3 majority).
    • Strict majority required (>= (votes // 2) + 1). Ties are strictly rejected as false positives.

I have an implementation ready with unit tests in feat/cross-model-validation-p0 and will link the PR to this issue once opened.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions