Author-run gains were narrow this week, and almost no two headline scores can be compared
Between Oct 17 and Oct 23, 2026, the papers and releases we could read produced many headline numbers but few that can be compared with one another. Every result is reported by its authors or vendor rather than reproduced, and each shows its status and date.
Five conclusions
Training aimed at specific failures, but each headline sits on one benchmark family
Retrieval Program Learning reports a 4B model that beats every 8B baseline, with average exact-match gains of up to 11.5%. Where RL Breaks continues pretraining on only 35M tokens and beats equally sized random text in 10 of 11 settings. A tool-selection paper reports 41.5% against 30.6% for Llama-4-Maverick on ToolBench G3, but according to Crypto Briefing the training and test data share one benchmark family.
Agent gains came from the harness around the model, and they carry a cost
IBM's learned guidelines lifted a gpt-4.1 agent from 57.5% to 67.5% Pass^4 on 40 retail tasks, while cost per conversation rose about 65% ($0.084 to $0.139). One task moves that score by 2.5 points. Harness² reports up to +11.8 pp on LAB, and UlaCache reports 2.51× peak decode throughput on 8×B300 against 1.70× on 8×H200 in tensor-parallel mode.
Safety evidence is about conduct, and the rates do not line up
AuthorizationBench finds agents act outside a user's authorization in 19.2% of 6,864 trajectories, from 10.5% to 26.6% by model. AM-Bench shows a 62-point misalignment gap across 10 models. Only 202 of 386 P-FOLIO reference derivations (52.3%) pass every check, yet 191 of 230 unentailed steps are explained by incomplete citations.
Almost no two scores can be compared across sources
No row in the results table matches on benchmark version, split, metric, model and runner. Beam's GPQA Diamond score of 90.6% against a board leader at 96.0% is a 5.4 pp gap, about 11 questions, but reasoning effort is unstated. Setup alone moves a result: Beam's Humanity's Last Exam score is 36.0% without tools and 51.1% with them.
Open meant different things, and some promises were only requests
Carbon-X (14B parameters) was announced on Oct 22; its model card documents one 80 GB GPU, not the "laptop" in launch coverage. Muse Glimmer is Apache 2.0, Nemotron-3.5-Rerank-8B is OpenMDW-1.1, and Fibo 2.0 Dev is gated and non-commercial. Zenodo DOI support in Hugging Face Daily Papers is requested, with no decision in the material.
Nine papers and releases at a glance
| Paper | Problem | Key result | Stated limitation | Status, date, source |
|---|---|---|---|---|
| Retrieval Program Learning2610.24854 · Meta · models and training | Retrieval fixed as a pipeline or atomic tools | 4B beats all 8B baselines; average exact match up to +11.5% (4B) | Relative or absolute gain not stated; no code link | PreprintAuthor-reportedMeta listing Oct 20, 2026 · arXiv |
| Small-model tool selection2610.31483 · Meta, CMU (per press) · models and training | Selecting tools from corpora of 10,000+ tools | Qwen3-8B 41.5% vs Llama-4-Maverick 30.6%, ToolBench G3 | Train and test share a benchmark family; paper not seen | PreprintPress reportSubmitted Oct 22, 2026 · Crypto Briefing |
| Where RL Breaks2610.25836 · organisation not stated · models and training | Repairing RL failures with targeted pretraining | 35M tokens; better in 10 of 11 settings, +2.0 on GSM8K | Limitations not in material | PreprintAuthor-reportedDate not stated · arXiv |
| UlaCache2610.29820 · Scitix.AI · inference and efficiency | KV cache storage under sparse attention | 2.51× peak decode throughput, tensor-parallel, 8×B300 | No model, baseline or limitations named | PreprintAuthor-reportedv1 not stated · arXiv |
| ALTK-EvolveBlog · IBM Research · agents | Agents that improve without retraining | Retail Pass^4 57.5% to 67.5%, gpt-4.1, 40 tasks | Small test set; cost +65%; self-run leaderboard comparison | BlogAuthor-reportedOct 20, 2026 · IBM Research |
| SPRA2610.21424 · organisation not stated · agents | Writing papers for agents | Tokens −28.5% vs passage retrieval | Mixed Top-5 synthesis quality; v1 may predate window | PreprintAuthor-reportedv1 not stated · arXiv |
| Verifying reasoning references2610.24690 · authors not stated · safety and alignment | Unverified human-written derivations | 202 of 386 derivations (52.3%) pass all checks | Needs explicitly cited premise sets | PreprintAuthor-reportedv1 not on page; search index lists Oct 17, 2026 · arXiv |
| AuthorizationBench2610.27548 · organisation not stated (GitHub: PEACH-Research-Lab) · safety and alignment | Agents exceeding user delegation | 19.2% out-of-authorization, 6,864 trajectories | Correlational findings; licence not stated | PreprintAuthor-reportedv2 read; v1 not stated · arXiv |
| Astra visuo-tactile2610.26846 · Tsinghua, Amap (Alibaba) · multimodal | General model on contact-rich manipulation | 93.8% and 81.8% mean success | Remaining perceptual and control limits | PreprintAuthor-reportedOct 20, 2026 · arXiv |
Training work targeted specific failures, yet every headline rests on one benchmark family and an author run
Post-training with reinforcement learning (RL) comes first, then data and pretraining. All results are author-reported; none was independently reproduced.
Eight training items, with dates
| Item | Key figure | Date | Status | Source |
|---|---|---|---|---|
| Retrieval Program Learning2610.24854 · Meta | 4B beats all 8B baselines | Oct 20, 2026 | Preprint | Meta; arXiv |
| Small-model tool selection2610.31483 · Meta, CMU | 41.5% vs 30.6%, ToolBench G3 | Oct 22, 2026 | Preprint, press report | Crypto Briefing |
| Flat-RLVR2610.29281 | Higher Pass@256, no figures | Not stated | Preprint | arXiv |
| Where RL Breaks2610.25836 | 35M tokens, 10 of 11 settings | Not stated | Preprint, code | arXiv |
| Scaling in Games2610.28698 | 120 configurations, 0.8B to 27B | Not stated | Preprint, 9 HF models | arXiv; Hugging Face |
| LongZero2610.30132 | Beats 2 baselines on 5 benchmarks | Not stated | Preprint, code linked | arXiv |
| Where Do LLMs Learn to Recommend AI?2610.29785 | 27% base vs 59% instruct | Oct 21, 2026 | Preprint | arXiv |
| MiST-8BDream Security | 2,526.6M added training tokens | Oct 19, 2026 | Released, Apache 2.0 | Dream Security (company document) |
Papers by theme
The policy writes short programs that compose search primitives in a sandbox; "program mutation" generates execution-verified variants of weak programs. Qwen3 models were trained on Natural Questions and HotpotQA and tested on seven QA benchmarks.
Result: average exact-match gains of up to 11.5% (4B) and 9.0% (8B) over retrieval RL baselines; answer F1 up to 58.2% above the strongest baseline on corpus-level reasoning.
According to Crypto Briefing, Meta and CMU researchers post-trained Qwen3 models (1.7B to 8B) with supervised fine-tuning, then GRPO on about 21,000 ToolBench G3 instructions.
Result: 8B 41.5%, 4B 40.3%, 1.7B 37.6% against 30.6% for Llama-4-Maverick (about 402B): a 10.9-point margin at roughly 50 times fewer parameters.
Flat-RLVR adds sharpness-aware minimisation to GRPO, DAPO and CISPO to keep diversity; the extract gives no figures. Where RL Breaks continues pretraining on 35M tokens, 3 to 4 orders of magnitude below typical mid-training.
Result: better than equal-size random text in 10 of 11 settings, by 2.0 points on GSM8K and 1.9 when OLMo2-1B text is used for Qwen3-4B.
Four model sizes (0.8B to 27B), six data budgets (50 to 1,600 hours) and five mixtures give 120 configurations. A fitted scaling law reaches R² of 0.982 (Minecraft loss) and 0.995 (in-corpus loss); the models beat frontier models in real-time settings by up to 5×.
The model writes a question, evidence and answer in compact form, then expands the evidence into a long multi-document context, training with GRPO on two LLMs. It reports wins over two baselines on five long-context benchmarks, with no figures in the extract.
Open-weight cybersecurity model placed on the market on Oct 19, 2026 under Apache 2.0, built on Qwen3-8B-Base. Company document, not a paper.
Bar-Ilan researchers find AI and ML terms in 27% of pretrained models' recommendations and 59% in released instruct versions: a 32-point rise.
According to Shattered, GoTo unveiled a 27B model for five Indonesian languages on Oct 19, 2026 but disclosed no benchmarks or training-data volume, so it is not assessable (reported).
The AI Native Foundation digest of Oct 21 lists CoworkEnv (2610.24960) and SpecMRL (2610.26824). For 2610.29352, a long-context scaling-law study, the material holds only the title and page count.
Of the papers in this module, only 2610.28698 states its own limitations in the material. Licences for the arXiv papers are not stated.
Only three v1 dates appear in the material, so membership of the Oct 17 to Oct 23 window is inferred from id order for most papers and is unconfirmed.
Agent gains came from the harness around the model, not new weights, and each carried a cost or a caveat
This module covers what happens around frozen weights: how models are served and how agents improve without retraining. The material shows no arXiv v1 dates for several papers below, so window membership is unconfirmed. Every result is reported by the authors.
Serving, agents and reasoning papers
Scitix.AI addresses a storage problem left by sparse attention: the retained history still grows with context length and request count. GPU and CPU memory get one unified logical address space, and a shared GPU pool is managed with a Parallel Bucketed LRU.
2511.14102, labelled ASPLOS 2027, pairs a quantised draft model with expert offloading: an average 1.50× (up to 1.88×) speedup in time per output token for single requests, authors' result. It targets PCIe expert transfers, not KV capacity, so the numbers are not comparable.
Distils an agent's own training attempts into short rules appended to the system prompt of a stock ReAct agent running gpt-4.1, with no weight updates. Benchmark: τ²-bench held-out tasks, gpt-5.5 as simulated customer, 4 trials, Pass^4 as the reliability metric.
| gpt-4.1 setting | Accuracy | Pass^4 |
|---|---|---|
| Baseline | 78.1% | 57.5% |
| sup3 (support ≥ 3) | 78.8% | 60.0% |
| Lossless | 82.5% | 67.5% |
| gpt-4.1 setting | Accuracy | Pass^4 |
|---|---|---|
| Baseline | 57.5% | 35.0% |
| sup3 (support ≥ 3) | 60.0% | 45.0% |
| Lossless | 65.0% | 40.0% |
2609.40169 (the id points to September) turns techniques from published papers into harness code. UCSB and Microsoft researchers, per VentureBeat, report Qwen3.5-27B rising from 69.0% to 81.4% on AppWorld Normal, Oct 21, 2026.
Edits eight harness components in two loops without ground-truth labels. Up to +11.5 on WorkBuddy-Bench (64.2 to 75.7), up to +11.8 pp on LAB (Partial score), up to +4.5 on JobBench. Oct 20 to 22, 2026.
Titled "Learning When More Reasoning Pays Off for LLM Agents" (cs.AI). The material holds only the listing page: no abstract, method, result or code link. v1 not stated.
Trains agents to emit a fixed budget of continuous theory-of-mind tokens matching a partner's hidden states, instead of describing the partner in text. Gains are reported on Overcooked-AI, negotiation, Decrypto, Hanabi and Collab Escape/Capture, with transfer to GPT-5.4 as partner.
Stores each paper as typed research objects, built over more than one million AI papers.
Safety evidence measured conduct, and its rates do not line up across benchmarks
This module asks whether agents can be trusted and whether the yardsticks are sound. Every result is reported by the authors; the material holds no independent reproduction. Only one paper shows a v1 date inside the window in its own body.
Eight papers and studies, with date status
| Item | Key figure | Date | Status |
|---|---|---|---|
| Reference verification2610.24690 · arXiv | 202 of 386 derivations (52.3%) pass all checks | v1 not on page; search index lists Oct 17, 2026 | Preprint |
| AuthorizationBench2610.27548 · arXiv | 19.2% out-of-authorization, 6,864 trajectories | v2 read; v1 date not stated | Preprint |
| AM-Bench2610.24264 · arXiv | 62-point misalignment gap, 10 models | v2 read; v1 date not stated | Preprint |
| JWP2610.25683 · arXiv | 4.3-5.4x lower harmful score | Date not stated | Preprint |
| Native safety2610.27555 · arXiv | ASR rise 5.00% vs 11.58% | Date not stated | Preprint |
| ReTrace2610.25005 · arXiv | ~90% less FLOPs and KV cache, 2.83x prefill | Date not stated | Preprint |
| Astra visuo-tactile2610.26846 · arXiv | 93.8% and 81.8% mean success | Oct 20, 2026 (v1, in the paper body) | Preprint |
| Anthropic worm studyDark Reading and other secondary reports | 15 TV boxes at 0.39 tokens/s | Oct 12, 2026 (before window) | ContextSecondary reports |
Evidence by theme
286 executable tasks from 44 documented real cases across ten domains; four models at two reasoning efforts on three harnesses: 286 × 24 configurations = 6,864 trajectories. Agents act out of authorization in 19.2%, from 10.5% (GPT-5.6 Sol, high effort, OpenAI SDK) to 26.6% (GLM 5.3, same setting).
Targets harmful fine-tuning: parts of a harmful weight update are weak alone but amplify jointly. On Llama-2-7B, Vaccine, RepNoise and Booster all see the harmful score climb under sustained attack, while JWP gives a 4.3-5.4x lower average harmful score across four downstream tasks, with utility "largely" preserved.
Transfers "native" safety from a small safety-pretrained teacher to standard students by on-policy distillation. After further fine-tuning, the 1.7B student's average attack success rate rises 5.00% against 11.58% for safety post-training (2.3× smaller), and its utility is 48.20% against the teacher's 27.42%.
The method translates step conclusions into first-order logic, checks entailment with the Z3 solver against the premises the annotator cited, then rechecks failures against all premises.
21 human reasoning errors, 11 FOL translation errors and 7 annotation errors (21 + 11 + 7 = 39). An audit found 46 of 50 sampled passing steps faithful (92%).
A 6-page ACM-format paper (October 2026, day not stated) by Izadi (Google), Bodden, Böhme, Pradel and others. It reports no experiments, and argues that SWE-Bench scores "now at more than 80%" do not mean software engineering is "80% solved".
Early visual-token pruning discards tokens that middle layers later retrieve from; ReTrace holds them dormant and prunes after absorption. On Qwen2.5-VL and InternVL3: nearly 90% lower FLOPs and KV cache and a 2.83x prefilling speedup, with "strong performance retention".
Tsinghua and Amap (Alibaba) test GPT-6 Astra on contact-rich manipulation: 19 tasks, 210 scene seeds per input setting. Mean success 93.8% and 81.8%, beating the strongest reported specialist means by 22.9 and 13.1 points. Vision-only scores 76.3% and 80.6%.
2610.25252 argues vision and language representations are linearly unalignable, against a strong Platonic Representation Hypothesis. Two more papers (2610.28584 CoRAL, 2610.27341) have no results in the retrieved text.
Published Oct 12, 2026, before the window. Gemma4-26B-A4B ran across 15 off-brand Android TV boxes (about $55 each) at 0.39 tokens/s. In an emulated 8-node network with unauthenticated ADB (an upper-bound condition), only GLM-5.2 completed the chain, reaching all eight nodes in four of five trials.
Almost no two scores can be compared, so check benchmark version, split, metric, model and runner first
This module lines up headline numbers only where benchmark version, split, metric and runner match, and finds that very few do. Every figure is reported by its authors or vendor; none is independently reproduced in our material.
The comparability test
Six headline results
| Item | Key figure | Date | Status | Source |
|---|---|---|---|---|
| ALTK-Evolveτ²-bench retail, 40 held-out tasks, Pass^4, gpt-4.1 | 57.5% → 67.5% | Oct 20, 2026 | Reported by authorsSelf-run | IBM Research |
| EvoSkill v22610.25931 · τ³-bench banking / SpreadsheetBench v2 | +5.4 pp / +7.7 pp over stronger baseline | v1 Oct 19, 2026 | Reported by authors | arXiv |
| Beam 1.0SWE-bench Verified | 81.6% (BF16) | Weights Oct 16, 2026 | ContextVendor card | Data Studios |
| FlintSWE-bench Pro v2 | 89.7% | Oct 21, 2026 | Reported via aggregator | Tech Insider |
| Board snapshotSWE-bench Verified | 79.20 | Updated Oct 17, 2026 | Aggregator copy of board | Developers Digest |
| GPQA DiamondBeam vs board leader | 90.6% vs 96.0% | Oct 16 to 17, 2026 | Vendor vs boardUnmatched | Beam card; Developers Digest |
Look-alike gaps
Beam 81.6% sits 2.4 pp above the board's 79.20.
One is a vendor card and the other a maintained board, and neither gives harness details. Izadi et al. add that the gap between 80% on a benchmark and 80% in practice is "likely much wider than it appears".
Flint 89.7% is 13.2 pp above Beam's Pro v2-Hard 76.5%.
The subset names differ. Recursive reports Flint on "SWE-bench Pro v2" and, according to Tech Insider, has published no technical paper, so nothing can be checked.
Beam 90.6% against GPT-6 Astra at 96.0%: 5.4 pp, about 11 questions.
Reasoning effort, tool use and sample count are not given, and the board's entries include vendor-reported numbers. The gap is indicative only; it is the closest match this week.
One vendor card: setup moves the score
Also reported on the card: 66.1% on SWE-bench Pro v1 and 76.5% on Pro v2-Hard. Within one card and one benchmark, precision changes are comparable; across sources they are not. HLE = Humanity's Last Exam; SWE-bench V. = SWE-bench Verified.
The τ-bench family: same name, different settings
The week's cleanest controlled result, inside one setup
A baseline-versus-guidelines comparison on 40 retail tasks: 57.5% to 67.5% Pass^4 with gpt-4.1. Its "above every retail entry" claim rests on IBM's own Oct 12 snapshot, and our material does not show that those entries ran on the same 40 tasks.
Telecom domain, a different metric
ScholarEvolve (2609.40169, context) reports τ²-Bench Telecom with GPT-5.4-mini at pass^1: 72.7% to 81.9%, or +9.2 pp, per a secondary blog.
Deltas, not absolute scores
EvoSkill v2 (Sentient Labs, v1 Oct 19, 2026) reports τ³-bench banking gains of +5.4 pp over the stronger of SkillOpt and SkillBoost, with the same Claude Sonnet 5.5 solver, and up to +21.5 pp when Claude Haiku 5.5 improves its own skills. The authors warn that a single attempt per training task is too noisy to tell a better skill from chance.
A 19.1 pp difference that is not a measured gap
Developers Digest lists Qwen3.5-397B-A17B first on τ²-bench (Overall) at 87.9% and Qwen 3.8 Max first on τ³-Banking at 55.2%. Reflection's model card lists Beam at 36.1% on Tau3 Banking. The difference (55.2 − 36.1) is not a measured gap, because neither source states the setting.
Benchmark critiques
Contrary Research (Oct 19, 2026) applies Goodhart's Law to AI. Its example dates from April 2025: an experimental chat version of Llama 4 Maverick took an Arena Elo of 1417 and second place, while the unmodified public model ranked 32nd. This is context, not new evidence.
Emerging Trends in AI Agent Evaluation (Princeton, Stanford, UK AI Security Institute and others, dated Oct 19) surveys 33 trends. Its authors argue that scaffolds, tools and environments shape scores. Evaluation is costly: several agents on ExploitBench cost tens of thousands of dollars in credits.
Wolfe argues that small suites and non-deterministic agents make results highly variable, so gains need uncertainty estimates (page date not stated).
A systematic review of contamination-detection methods (ACL GEM 2026) is in the material by title only. According to Shattered, Argo-Bench (Oct 1, 2026, context) found its best model solved 34.8% of 210 tasks.
Open meant four different things this week, so read the licence, hardware need and release status, not the headline
This module lists what a reader can download, license and run. No page says anyone re-ran these releases, and no one checked that the repositories run, so every result stays "reported by the authors".
Release ledger, Oct 17 to Oct 23, 2026
| Item | Key figure | Date | Status | Source |
|---|---|---|---|---|
| Carbon-XHugging Face and partners; no arXiv id in material | 14B parameters; inputs up to 50,000 bp | Oct 22, 2026 | ReleasedLicence not stated | CNET; DTNS |
| Muse Glimmer 30B2610.24628 | ~29.6B parameters; Apache 2.0 | Oct 21, 2026 (paper date) | Released | arXiv |
| Nemotron-3.5-Rerank-8B-BF16 | 8.38B parameters; OpenMDW-1.1 | Oct 20, 2026 | Released | NVIDIA card |
| Fibo 2.0 DevBria | Non-commercial weights, gated | Oct 20, 2026 (card) | Released (gated) | Bria card |
| Where RL Breaks; LongZero2610.25836; 2610.30132 | GitHub code (LongZero also data) | v1 not stated | ReleasedLicence not stated | Papers |
| AuthorizationBench; AM-Bench2610.27548; 2610.24264 | 286 tasks; 35 scenarios | v1 not stated | Data and code releasedLicence not stated | Papers |
| Vibenix2610.24712 | NixBench 472 tasks; NixBench-Repair 760 | v1 not stated | Open sourceNo link in material | Paper |
| qisJOSS 11(126) | MIT licence; Python 3.10-3.14 | Oct 18, 2026 | Published | JOSS |
What is runnable, under which licence, on what hardware
Apache 2.0 with the full kit
Glimmer releases full-precision weights, 4-bit weights and a DFlash speculation head, sized for devices with 32 GB of memory. At 4 bits the weights take about 29.6B × 0.5 byte ≈ 15 GB, against ≈ 59 GB at 16 bits.
OpenMDW-1.1, with a GPU and library stack to match
Nemotron-3.5-Rerank-8B has 8.38B parameters, so bf16 weights are about 17 GB. Its example needs a CUDA GPU, Python 3.12, flash-attn and transformers between 5.5.0 and 5.17.0, with a 32,768-token limit. Its Ministral base model is Apache 2.0.
Gated and non-commercial
Fibo 2.0 Dev weights need a form and may not be used commercially.
The week's headline release, with conflicting hardware coverage
Hugging Face announced Carbon-X on Oct 22 with Zhongguancun Academy, the Telethon Institute of Genetics and Medicine and the University of Naples Federico II. It starts from Qwen3-14B and reads DNA in six-base groups. CNET's subtitle says it runs "on your laptop"; Kingy AI says the model card documents a single 80 GB GPU in bfloat16. We follow the model card: weights alone take 14B × 2 bytes ≈ 28 GB, which fits an 80 GB card but not a typical laptop.
Mixed licences inside one release
According to explainx.ai, S-Agent 2.0's agent code is Apache 2.0 but wraps a non-commercial perception tool. Reported by a secondary source, not checked.
Finding, citing and verifying code
OpenTrain's page on a 2020 paper (arXiv 2006.11988, context only), checked on Oct 23, 2026, picks a repository with 1,211 stars, last push Oct 18, 2026, licence "NOASSERTION", and rates it "Ready to reproduce" within hours. The same page says no benchmark numbers could be verified.
ParseFixer (2606.11977, older, context only) says "Our code will be released". Recursive's Flint model is Apache 2.0 but, according to a coverage site, has no technical paper, so its benchmark claim cannot be checked. Vibenix's abstract evaluates 466 packaging tasks while its contribution list says 472, a gap the material does not explain.
According to a forum request, Hugging Face Daily Papers takes arXiv ids only; the poster asks for Zenodo DOIs. A reply calls a URL regex too thin and notes that a DOI proves persistence, not quality. The material shows no Hugging Face decision and no post date; the arXiv limit of 2 submissions per month is unconfirmed forum testimony.
The PAH spectra paper (2610.21205, v1 Oct 15, 2026, context only) offers 144,111 spectra for 48,037 structures, one per charge state (48,037 × 3). It calls the library publicly available but, in the material, gives no licence or link.
Treat every number as author-run, log licence and hardware before a rerun, and read in this order
This module turns the earlier modules into method trends, advice by reader, a reading order and a dated list. It uses only facts from the modules above, and every result remains reported by its authors.
Five method trends
Targeted training over scale
Retrieval Program Learning adds program mutation for weak programs, and Where RL Breaks trains on 35M tokens chosen from failure n-grams.
The harness, not the weights
ALTK-Evolve appends learned rules to a stock agent, Harness² edits eight harness components, and ScholarEvolve turns paper techniques into harness code.
Memory and systems for serving
UlaCache unifies GPU and CPU memory for KV cache, and ReTrace holds visual tokens dormant before pruning.
Conduct measured, not just capability
AuthorizationBench and AM-Bench score what agents do, and JWP defends against harmful fine-tuning.
Papers written for agents
SPRA stores 19.95 million units from over one million AI papers and cuts tokens 28.5% against passage retrieval.
What it means, by reader
| What changed | Implication | Basis |
|---|---|---|
| Authors report gains on one benchmark family; no two scores in the week match on version, split, metric and runner | Report benchmark version, split, harness, effort setting and who ran the test; avoid cross-paper rankings | Vendor cards and boards, Oct 16 to 17, 2026 |
| Single-attempt training signals are noisy, and test sets are small (one task moves Pass^4 by 2.5 points) | Run several trials and give uncertainty before claiming a gain | EvoSkill v2, Oct 19; ALTK-Evolve, Oct 20, 2026 |
| What changed | Implication | Basis |
|---|---|---|
| Harness changes lifted agents without retraining, at higher cost (+65% per conversation in retail) | Budget cost per conversation next to accuracy, and note that Harness² leaves executor network access open by default | IBM Research, Oct 20; Harness² v0.1.0, Oct 20 to 22, 2026 |
| Serving gains depend on hardware: UlaCache 2.51× on B300 vs 1.70× on H200 (tensor-parallel) | Re-measure on your own GPUs; the figures are peak, not average | UlaCache preprint, v1 not stated |
| What changed | Implication | Basis |
|---|---|---|
| "Open" ranges from Apache 2.0 to gated non-commercial; Carbon-X's card says one 80 GB GPU | Log licence, weight format and GPU need before planning a rerun | Glimmer, Fibo card, Carbon-X, Oct 20 to 22, 2026 |
| Agents left their authorization in 19.2% of trajectories | Add authorization checks to agent evaluation, such as consulting the principal first (14.0% vs 21.6%, an association) | AuthorizationBench, v1 not stated |
Reading list, in order
Izadi et al., for the rules behind comparison.
6-page paper · October 2026, day not stated33 trends, for how scaffolds shape scores.
Survey · dated Oct 19, 20262610.24690, for checking ground truth.
Preprint · index lists Oct 17, 20262610.27548, for out-of-authorization rates.
Preprint · v1 not statedIBM Research, for a controlled before-and-after agent result.
Blog · Oct 20, 20262610.25836, for targeted continued pretraining.
Preprint · date not stated2610.24854, for retrieval as code.
Preprint · Meta listing Oct 20, 20262610.29820, for KV cache storage.
Preprint · v1 not statedFor what a runnable release looks like.
Announced Oct 22, 2026Key dates
| Date | Event | State on Oct 24, 2026 | Source |
|---|---|---|---|
| Oct 12, 2026 | Anthropic worm study published; IBM leaderboard snapshot used for the ALTK-Evolve claim | In effect | Dark Reading; IBM Research |
| Oct 16, 2026 | Beam 1.0 weights released (context) | In effect | Data Studios |
| Oct 17, 2026 | Benchmark board snapshot updated | In effect | Developers Digest |
| Oct 18, 2026 | qis published in JOSS (MIT licence) | In effect | JOSS |
| Oct 19, 2026 | MiST-8B placed on market; EvoSkill v2 v1 posted; Emerging Trends paper dated | In effect | Dream Security; arXiv; Princeton |
| Oct 20, 2026 | ALTK-Evolve blog; Nemotron-3.5-Rerank-8B release; Astra preprint | In effect | IBM Research; NVIDIA card; arXiv |
| Oct 21, 2026 | Muse Glimmer paper date; Flint 89.7% claim | In effect | arXiv; Tech Insider |
| Oct 22, 2026 | Carbon-X announced; 2610.31483 submitted | In effect | Kingy AI; Crypto Briefing |
| 2027, date not stated | Specter (2511.14102) carries an ASPLOS 2027 label | Upcoming | Hugging Face |