Coding-Agent

Don't Ship Agent Skills Without Evals: Philipp Schmid's Testing Method and SkillsBench

Don't Ship Agent Skills Without Evals: Philipp Schmid's Testing Method and SkillsBench

If you use a coding agent like Claude Code, Gemini CLI, or Codex, you eventually end up writing skills: Markdown files that hand the agent your team’s coding conventions, a specific SDK’s usage patterns, or a deployment procedure. The problem is that almost nobody tests them. We would never ship code without tests, yet skills, which directly change how an agent behaves, get shipped after a few manual runs and a gut-level “looks fine.”

Read More
GPT-5.6 Sol, Terra, and Luna: A Model Routing Guide for Coding Agents

GPT-5.6 Sol, Terra, and Luna: A Model Routing Guide for Coding Agents

A claim like “Luna Max beats Terra High” circulates around GPT-5.6 discussions, and it does not hold up. Sol, Terra, and Luna are separate models, each a different capability tier. Max, high, xhigh, and ultra are settings within a given model that control how much reasoning time it uses and how many agents run in parallel. Collapsing a tier name and a settings name into one ranking compares two different axes as if they were one.

Read More
AI Self-Improvement Starts Outside the Model

AI Self-Improvement Starts Outside the Model

An AI agent can improve its behavior across runs while its model weights stay unchanged. It can diagnose failures in execution traces, revise its prompts, memory, tools, or workflow, and retain changes that pass a separate evaluation. Persistent, tested changes distinguish harness improvement from simply making more attempts. Darwin Gödel Machine (DGM) changes a coding agent’s code; GEPA changes prompts; Reflexion changes the memory supplied to later trials. Each changes the system around a model rather than retraining that model. (DGM, GEPA, Reflexion)

Read More
DeepSWE: A Benchmark for Long-Horizon Coding Agents

DeepSWE: A Benchmark for Long-Horizon Coding Agents

SWE-bench has been the default coding-agent leaderboard for a while, but it has well-known weaknesses. Most tasks come from existing public issues and PR patches, so a high score might partly reflect memorization. Most tasks are also single-file bug fixes, which is not representative of the multi-file, long-horizon work that a coding agent does in practice.

Read More
Categories
Tags