Is there a programming equivalent of the strawberry problem, i.e. a specific coding task that LLMs are consistently bad at?
Related Posts
The niche of short, environment-specific scripts, and the consequences for programming language design choices: https://brrt-to-the-future.blogspot.com/2021/03/why-bother-with-scripting.html
LLMs seem to handle dependency upgrades really well.
The task is well-specified, there's usually a build/test suite to check correctness of the modifications, and there's often a changelog they can consume too.
It's incredibly hard to explain adversarial problems to users. I see gamers sincerely asking "why doesn't the publisher just fix the cheater problem?".
This is exacerbated by the fact that sharing too many details of anticheat can make the problem worse.