Deterministic edge-case stress testing for frontier and open-weight models. Evaluates code generation on IEEE-754 precision, OWASP injection vectors, RegExp statefulness, and async race conditions — zero LLM-as-a-judge bias. Every verdict is binary: real code running against real tests.
| Rank | Model | Pass Rate | Passed / Trials | Confident Errors | Verdict |
|---|---|---|---|---|---|
| 1 | Claude | 86% | 18 / 21 | 0 | Optimal |
| 2 | DeepSeek | 86% | 18 / 21 | 0 | Optimal |
| 3 | Llama 3 | 86% | 18 / 21 | 0 | Optimal |
| 4 | Mistral | 86% | 18 / 21 | 0 | Optimal |
| 5 | Gemini | 71% | 15 / 21 | 0 | Optimal |
| 6 | ChatGPT | 57% | 12 / 21 | 0 | Defective |
| 7 | Grok | 43% | 9 / 21 | 2 | Defective |
| Task & Blind Spot | ChatGPT | Claude | DeepSeek | Gemini | Grok | Llama 3 | Mistral |
|---|---|---|---|---|---|---|---|
|
allocate_minor_units
Largest-remainder allocation with deterministic tie-break. Models dump the leftover on first/last recipient or round each share independently.
|
Pass | Pass | Pass | Pass | FailMissing expected RangeError exception | Pass | Pass |
|
billing_dates
Month-end clamping from original anchor. Iterative clamping drifts: Jan 31 → Feb 28 → Mar 28.
|
Pass | Pass | Pass | Pass | Pass | Pass | FailJan 31 anchor must return to 31st after short months |
|
extract_json
Greedy regex breaks on trailing braces. Naive brace counting breaks on braces inside strings.
|
Pass | Pass | Pass | FailBrace inside string not handled | FailBrace inside string not handled | Pass | Pass |
|
regex_filter
RegExp /g flag introduces statefulness via lastIndex. Repeated calls on same text oscillate between true and false.
|
FailRegExp /g statefulness: repeat call returns false | Pass | Pass | Pass | Pass | Pass | Pass |
|
safe_redirect
startsWith('/') accepts protocol-relative //evil.com, backslash and tab tricks, userinfo @ hosts.
|
FailBackslash redirect /\evil.com not blocked | Pass | Pass | FailTab-inside-// redirect /\t/evil.com not blocked | Pass | FailTab-inside-// redirect /\t/evil.com not blocked | Pass |
|
search_latest_wins
Guarding by query value instead of call sequence lets stale async responses overwrite newer results.
|
Pass | Pass | Pass | Pass | FailOlder response overwrites newer result | Pass | Pass |
|
snowflake_dedupe
JSON.parse turns 64-bit integers into doubles above 2^53. Adjacent Snowflake IDs collapse and dedupe silently drops rows.
|
FailIDs lose precision as IEEE-754 doubles | FailIDs lose precision as IEEE-754 doubles | FailIDs lose precision as IEEE-754 doubles | Pass | FailIDs lose precision as IEEE-754 doubles | Pass | Pass |
These models explicitly asserted the generated code was correct, safe, robust, or fixed in prose —
yet failed the deterministic edge-case test. Raw responses are stored verbatim in
responses/.