4 appearances on the backlist front page in the last 30 days.
Nemotron 3 Ultra 505b a55b scores 43.5% on WeirdML, comparable to Mistral Medium 3.5 128b or o3 mini. It can sometimes do well on some of the hard tasks, but it's not very reliable. It also often emitted the "stop" token when done with
Nemotron 3 Ultra 505b a55b scores 43.5% on WeirdML, comparable to Mistral Medium 3.5 128b or o3 mini. It can sometimes do well on some of the hard tasks, but it's not very reliable. It also often emitted the "stop" token when done with
Claude Opus 4.8 (xhigh) scores 82.9% on WeirdML, right behind GPT 5.5. We now also (unlike 4.7) see a clear scaling with output token use: - no thinking: 2.4k tokens, 70.5% - medium: 4.3k, 76.0% - xhigh:
How far behind are open models? Across 17 selected benchmarks, private ones show a gap of 8-10 months today, almost 2x the gap on public ones (4-6 mo). More discussion (including limitations), code and blog in the thread.