Coding-Agent

Don't Ship Agent Skills Without Evals: Philipp Schmid's Testing Method and SkillsBench

If you use a coding agent like Claude Code, Gemini CLI, or Codex, you eventually end up writing skills: Markdown files that hand the agent your team’s coding conventions, a specific SDK’s usage patterns, or a deployment procedure. The problem is that almost nobody tests them. We would never ship code without tests, yet skills, which directly change how an agent behaves, get shipped after a few manual runs and a gut-level “looks fine.”

Read More

GPT-5.6 Sol, Terra, and Luna: A Model Routing Guide for Coding Agents

A claim like “Luna Max beats Terra High” circulates around GPT-5.6 discussions, and it does not hold up. Sol, Terra, and Luna are separate models, each a different capability tier. Max, high, xhigh, and ultra are settings within a given model that control how much reasoning time it uses and how many agents run in parallel. Collapsing a tier name and a settings name into one ranking compares two different axes as if they were one.

Read More

AI Self-Improvement Starts Outside the Model

When people talk about AI self-improvement, the usual image is a model rewriting its own weights. Lilian Weng’s July 4, 2026 post on Lil’Log argues the near-term version looks different. The recursive self-improvement (RSI) we can actually observe today shows up first in the system around the model, not inside it. That system is what she calls the harness. This post walks through her argument: what a harness is, where the optimization target is moving, what a striking SWE-bench number actually means, and what breaks if you get the risk boundaries wrong.

Read More

DeepSWE: A Benchmark for Long-Horizon Coding Agents

SWE-bench has been the default coding-agent leaderboard for a while, but it has well-known weaknesses. Most tasks come from existing public issues and PR patches, so a high score might partly reflect memorization. Most tasks are also single-file bug fixes, which is not representative of the multi-file, long-horizon work that a coding agent does in practice.

Read More
Categories
Tags