Swe-Bench

AI Self-Improvement Starts Outside the Model

AI Self-Improvement Starts Outside the Model

An AI agent can improve its behavior across runs while its model weights stay unchanged. It can diagnose failures in execution traces, revise its prompts, memory, tools, or workflow, and retain changes that pass a separate evaluation. Persistent, tested changes distinguish harness improvement from simply making more attempts. Darwin Gödel Machine (DGM) changes a coding agent’s code; GEPA changes prompts; Reflexion changes the memory supplied to later trials. Each changes the system around a model rather than retraining that model. (DGM, GEPA, Reflexion)

Read More
DeepSWE: A Benchmark for Long-Horizon Coding Agents

DeepSWE: A Benchmark for Long-Horizon Coding Agents

SWE-bench has been the default coding-agent leaderboard for a while, but it has well-known weaknesses. Most tasks come from existing public issues and PR patches, so a high score might partly reflect memorization. Most tasks are also single-file bug fixes, which is not representative of the multi-file, long-horizon work that a coding agent does in practice.

Read More
Categories
Tags