A Scrabble move generator can get faster while leaving a harder research question unresolved: faster on which positions? In a LessWrong comment, Arjun Panickssery reports a tenfold improvement in listing legal moves through repeated prompting of a system he calls Astra. But he also identifies two important human interventions: suggesting measurement across a game rather than per random position, and proposing separate algorithms according to how many blank tiles remained in the bag.
Those interventions change more than the instructions. The first changes the workload against which an optimization is judged. The second introduces a distinction that may make different algorithms useful in different cases; Panickssery says blank tiles cause the number of moves to explode. The inference is narrow but consequential: an automated search can benefit from a person changing the test or dividing the problem, not merely asking for another attempt.
This is an account of side projects, not a controlled comparison of autonomous research systems. Panickssery also describes weak literature selection and experiments he considered uninformative. Those observations cannot establish how models generally perform. Even the reported speedup lacks enough implementation and measurement detail in the supplied snapshot to verify independently. The useful evidence here is the described division of work, not a verdict that AI either can or cannot conduct research.
For a hypothetical engineer optimizing a dispatch planner, the analogous choice would be whether to measure average planning time across ordinary days or examine days with unusually constrained vehicles separately. An agent might search many implementations against either test. If the test hides the difficult days, however, improving its score would not establish that dispatchers face fewer delays. Defining those cases is part of the research, not administrative preparation for it.
One reasonable reading of the Scrabble account is that repeated prompting already delivers substantial value. That may be true. The more useful product opportunity is to make expert interventions reusable: preserve why a workload was chosen and which cases justified a separate approach. That could let later searches test against those distinctions instead of requiring the expert to reconstruct them. It would not confer autonomous research judgment; it would turn one consequential piece of that judgment into something the next experiment can use.
