AI Models Know When They Are Being Tested

Something interesting is happening in the world of AI coding agents. With benchmarks like SWE-bench, we are not only testing how well models can write code anymore. We are also testing how well they understand the environment in which they are being tested.

Cursor published a blog post about reward hacking in coding benchmarks. The short version is that some AI agents do not always solve the programming task from scratch. Instead, when they recognize that a benchmark is based on an old public GitHub issue, they sometimes search for the original fix, inspect git history, or look up the already merged pull request. In other words, the model is not only trying to solve the bug, it is also trying to understand where the answer might already exist.

This is both really interesting and really worrying. Not because it means AI models are “self-aware” in the human sense, but because it shows that they are becoming more aware of the context around a task. If the environment gives away clues that a bug has already been fixed somewhere, a capable agent might use those clues. From a pure problem-solving perspective, that is actually quite smart. From a benchmark perspective, it is a problem.

This makes evaluating AI agents much harder. A high benchmark score might not only mean that the model is better at reasoning about code. It might also mean that the model is better at finding leaked answers, searching through git history, or recognizing that it is inside an artificial test setup. Cursor showed that scores dropped significantly when they removed access to repository history and restricted internet access during evaluation.

In the end, this feels very similar to how we evaluate people. If someone passes an exam by understanding the subject, that tells us one thing. If someone passes by finding the answer sheet, that tells us something completely different. Both outcomes can result in a correct answer, but only one of them shows the skill we actually wanted to measure.