AI

The Failure a Pass Rate Hides: How the YouTube Ads Team Runs Production Evals

The idea that a well-written prompt makes an agent behave holds up right until the demo ends. In production, the same prompt and the same input produce different results run to run. A case that passed yesterday fails today, and one that failed yesterday passes. Deciding whether that system is ready to ship takes measurement, not a feeling.

Read More

Don't Ship Agent Skills Without Evals: Philipp Schmid's Testing Method and SkillsBench

If you use a coding agent like Claude Code, Gemini CLI, or Codex, you eventually end up writing skills: Markdown files that hand the agent your team’s coding conventions, a specific SDK’s usage patterns, or a deployment procedure. The problem is that almost nobody tests them. We would never ship code without tests, yet skills, which directly change how an agent behaves, get shipped after a few manual runs and a gut-level “looks fine.”

Read More

GPT-5.6 Sol, Terra, and Luna: A Model Routing Guide for Coding Agents

A claim like “Luna Max beats Terra High” circulates around GPT-5.6 discussions, and it does not hold up. Sol, Terra, and Luna are separate models, each a different capability tier. Max, high, xhigh, and ultra are settings within a given model that control how much reasoning time it uses and how many agents run in parallel. Collapsing a tier name and a settings name into one ranking compares two different axes as if they were one.

Read More

On-Policy Distillation: Closing the Gap Between RL and SFT

The two standard post-training methods each leave a gap. Supervised fine-tuning (SFT) has the student imitate sequences a teacher already produced, but training happens on states the student may never actually visit, so errors compound over long generations. Reinforcement learning (RL) samples from the student’s own rollouts, which fixes that mismatch, but the reward is usually a single bit or two per episode. A post Thinking Machines Lab published in October 2025 proposes combining the two: sample trajectories from the student, then have a strong teacher score every token in that trajectory.

Read More

AI Self-Improvement Starts Outside the Model

When people talk about AI self-improvement, the usual image is a model rewriting its own weights. Lilian Weng’s July 4, 2026 post on Lil’Log argues the near-term version looks different. The recursive self-improvement (RSI) we can actually observe today shows up first in the system around the model, not inside it. That system is what she calls the harness. This post walks through her argument: what a harness is, where the optimization target is moving, what a striking SWE-bench number actually means, and what breaks if you get the risk boundaries wrong.

Read More

Hermes Agent v0.18: When a Self-Improving Agent Gets MoA

Nous Research’s open-source agent Hermes Agent shipped v0.18.0 on July 1, 2026. The release notes call it “The Judgment Release.” Reading this as a routine feature update misses the point. The official docs describe Hermes Agent as “the self-improving AI agent,” and the project is built around a loop that accumulates memory and skills the more it gets used. This post covers how that loop was refined in v0.18, and what role Mixture of Agents (MoA), now a first-class model choice in the same release, plays inside it.

Read More

TradingAgents: Reading the Paper and Code Behind an LLM Trading Desk

Up front: this is not investment advice, and nothing here recommends buying or selling anything. It is a read of how you organize LLM agents into a single decision, looked at from the research and the code. Trading is just the domain the design happens to target.

Read More

Harness-1: Teaching Search Agents to Offload State

Search agents and the state problem A search agent is an AI system that answers a question by iterating through multiple searches. Unlike a one-shot retrieval lookup, it reads intermediate results, adjusts its search strategy, compares candidate documents, and checks whether specific claims are actually supported by what it found. Tasks like analyzing financial filings, tracing multi-hop facts across sources, or interpreting complex regulations need this kind of iterative work. A single query won’t get you there.

Read More

LLM Observability Without LangSmith: Five Open-Source Tools Compared

At some point in building LLM applications or agents, you need to know why a call failed, what the tool invocation looked like, or why the agent got stuck in a loop. LangSmith, LangChain’s commercial observability platform, has been the default answer for this: it covers trace visualization, prompt versioning, and evaluation in one place. Its usage-based pricing and cloud-hosted architecture are where teams start looking for alternatives. Traces carry raw user inputs and internal prompts, so shipping that data to an external SaaS is itself a problem for plenty of organizations.

Read More

DeepSWE: A Benchmark for Long-Horizon Coding Agents

SWE-bench has been the default coding-agent leaderboard for a while, but it has well-known weaknesses. Most tasks come from existing public issues and PR patches, so a high score might partly reflect memorization. Most tasks are also single-file bug fixes, which is not representative of the multi-file, long-horizon work that a coding agent does in practice.

Read More

Mixture of Agents: How Layering Open-Source LLMs Beat GPT-4 Omni

Instead of scaling a single model up, you can stack multiple models in layers and have each one refine the previous layer’s output. Together AI’s research team formalized that approach in June 2024 as Mixture of Agents (MoA) in arXiv:2406.04692. Using only open-source models, their MoA configuration scored 65.1% on AlpacaEval 2.0, versus 57.5% for GPT-4 Omni.

Read More

Open Knowledge Format: A Shared Vocabulary for Agent Knowledge

When AI agents fail in production, the model is often not the problem. The missing context is. Table schemas, metric definitions, runbooks, join paths between systems, and API deprecation notices are scattered across catalog vendors, internal wikis, code comments, and personal notes. Every agent developer solves the same context assembly problem from scratch.

Read More

Qwen3.6-35B-A3B: Community Reviews, Uncensored Variants, and MTP Benchmarks

Alibaba released Qwen3.6-35B-A3B in April 2026: a 35B-parameter MoE model with around 3B active per token, a 262K native context, and an official SWE-bench score of 73.4%. Two months in, it’s the most widely tested 35B-class model in the local LLM community.

Read More

Robot Learning: A Tutorial (From Classical Robotics to Generalist Policies)

“Robot Learning: A Tutorial” (arXiv:2510.12403) is a paper-length tutorial by Francesco Capuano, Caroline Pascal, Adil Zouitine, Thomas Wolf, and Michel Aractingi, from the University of Oxford and Hugging Face. It covers the full arc of robot learning methods, from classical dynamics-based control through reinforcement learning, imitation learning, and generalist vision-language-action models, using the Hugging Face LeRobot library throughout.

Read More

VibeThinker-3B: Packing Verifiable Reasoning into 3 Billion Parameters

“Small model beats big model” papers appear regularly. Usually the claim holds on a specific benchmark under specific conditions, not across the board. WeiboAI’s VibeThinker-3B, published June 15, 2026, follows a similar structure but draws a clearer boundary: the claim is not that a 3B model replaces a frontier generalist. The claim is that verifiable reasoning can be compressed into a small model, while open-domain knowledge and general dialogue still benefit from more parameters.

Read More

Future AGI: Evaluate, Observe, and Improve AI Agents in One Place

If you have shipped an AI agent, this will sound familiar. The demo runs fine. Then it hits production, the hallucinations start, and you can’t tell what went wrong or why. So you bolt on one tool for evals, another for tracing, another for guardrails. The real problem is that none of them talk to each other, so the loop you need to actually fix things never closes.

Read More
Categories
Tags