SWE-bench Verified evaluation
Prilog reports 91.2% (456/500) in a feedback-guided evaluation. Read the method, artifacts, and pending submission status.
Reported result: 456 of 500 issues resolved
Prilog reports 91.2% resolved (456/500) on the final saved prediction set of SWE-bench Verified. The October 22, 2026 submission describes a feedback-guided evaluation: it retains 413 previously passing patches and replaces 87 previously unresolved cases with one new attempt each.
Submission status checked October 25, 2026: SWE-bench experiments PR #513 is open for maintainer eligibility and methodology review. The result is self-reported and has not been accepted or independently reproduced by SWE-bench maintainers. It does not establish an official leaderboard rank or a pass@1 score.
What was evaluated?
The engine checks out an issue's base revision, investigates the repository, prepares dependencies, generates a patch, and performs local verification and bounded review and repair. The system is proprietary and the underlying model and provider are undisclosed. The report evaluates the complete engine; it does not isolate the effect of orchestration from the underlying model.
| Measure | Reported value |
|---|---|
| Dataset | SWE-bench Verified, 500 issues |
| Resolved | 456 |
| Unresolved | 44 |
| Empty final predictions | 4, counted unresolved |
| Insufficient required-test evidence | 11 cases, counted unresolved |
| Nonempty predictions regraded | 496 |
| Method | Feedback-guided retention and selective regeneration |
| Underlying model | Undisclosed |
The empty predictions and evidence-error cases are included in the unresolved count; they are not additional exclusions from the denominator.
How were final predictions selected?
The final set retains 413 patches that had previously passed official evaluation. Official failure feedback selected the other 87 cases for regeneration and informed generic reliability improvements. Each new attempt replaced its earlier prediction, including when the new result was empty or unsuccessful.
All 496 nonempty final predictions were regraded under one frozen AWS Batch lifecycle adapter using official SWE-bench 4.1.0 task images, scripts, parsers, and grading functions. All 500 instances remain in the denominator. The report describes saved-output verification matching the 456-case resolved list, as well as repository-level negative and gold controls.
How should the score be interpreted?
This is not a fresh 500-instance generation run, pass@1, or Best@k selected independently of official evaluation feedback. A fresh-run score is unknown. The submission acknowledges that its affiliation, proprietary system, and selection method may make it ineligible under the current Verified submission policy.
The result describes benchmark issue resolution under the disclosed process. It does not measure production incident detection, reviewer time, customer acceptance rate, or superiority over Sentry, Datadog, or a coding agent. Those outcomes require separate evaluations.
Inspect the report and artifacts
- Submission and maintainer discussion.
- Public artifact repository: final predictions, patches, test outputs, captured execution traces, prior attempts, selection provenance, and checksums.
- Technical report: engine workflow, adapter, controls, redactions, and limitations.
- Pinned submission description: the reported counts and methodology accompanying the PR.
Model nondisclosure limits independent inference reproduction. The report also notes possible training exposure to public benchmark instances; auditing trajectories for solution lookup does not prove absence of training contamination.
Evaluate Prilog beyond the benchmark
Inspect merged public contributions, follow automatic bug fixing, and trial representative issues in your own repository. Record successful fixes and failures, review effort, extra edits, checks run, and cost per accepted change.