What benchmarks give us is efficiency. We don’t have to spend years peer-reviewing papers — if it beats the state of the art, we know it’s probably a good paper. But unfortunately, it has narrowed the collective vision of the community. We’re searching under streetlights — we are searching the kinds of things that are easy to search in a benchmark-driven evaluation regime. We need to move beyond that. That means evaluating how good a paper is will become much more costly. Sadly, that is a cost that we will have to pay. Right now the community is trying to rush in the opposite direction.