The thing about this is that in a different codebase, even just that search for the trumpet guy could either burn a ton of tokens and come back with nothing or find it but light the ocean on fire in the process. This is because trumpet guy would be named "asset-ds1f23nbiq35421fsda.jpg", and it would be located in an images folder with thousands of other similar assets. So the agent starts by grepping for reasonable names, discovers nothing but finds an image directory, and then searches inside. In the worst case, I guess it starts analyzing each image. Or maybe it checks the article like he mentions, but the codebase uses a component library and the filename thing is still true, so it analyzes every image in the page. This is a bit of hyperbole, of course. Please allow it for the sake of comedic example.
Many of his other prompt examples also illustrate this. The author is massively leveraging the architecture of the organization through familiarity. It could well be the case that even an LLM was used to set that up, but I doubt it happened in one prompt or without critical thought. To be fair, I guess he's sort of trying to point that out, but by saying "Look how lazy I am while it still works" or something like that in order to look cool, I think all he's conveying is "Look how easy this is", when it's potentially not and could increasingly cost him/his company wildly varying amounts of money based on how he goes about it. When tokens were virtually free, no one cared. This is increasingly not true. Many are drawing comparisons to early cloud architecture.
I find this is sort of similar to the problematic idea that in theory anyone can clone your repo and get a local dev environment running in a few seconds. So then you try it with a non-technical staff member and quickly realize how unrealistic that is as you run into abstraction after abstraction that is completely foreign to them.
Loved this. It's funny how basic all of this can be. We have all these skills and and tricks and incantations whatever else, when it probably all just comes down to giving the agent the right info to set them up for success (just like you'd do with anyone you're delegating anything to).
I found it interesting that Thorsten mentions the importance of attaching screenshots.
In my experience, it varies widely by workload. Vision is great for some UI stuff but extremely terrible for my work, which involves visualizing connectivity and geometric stuff for road structure for autonomous vehicles. It can get super frustrating when I paste a screenshot and go "this polyline isn't connected when it should be" or "those polylines are self intersecting in an invalid way" and then the AI literally can't see what I'm talking about. It would have to then write debug code to dump out the coordinates of the polylines to convince itself that the bug even exists. Then it would often claim that it's fixed when it is clearly not.
I previously complained about GPT 5.6 Sol not being able to solve a simple "follow the lines" puzzle [1] without having to write custom Python code for it. GPT-6 Astra, however, is vastly improved, and could do it without writing Python. Still, for my work, it can occasionally struggle.
I think these AI platforms would guarantee bigger wins for themselves if they figured out how to give these models more of an object-conceptualization of their input than they currently have. Solving math puzzles is cool and all, but 99% of customers don't need to solve math puzzles like that; they're asking the model to answer questions about thins and to translate ideas into a real UI. A model that could reliably determine if/how its input/output isn't visually matching up to a design would be a big deal. Sure, we kind of have that now, but it's still in gimmick territory.
Slick presentation. His prompts look similar to my brain dumps. I’m in the habit of running these through Codex’s plan-mode first to ensure the AI is on the same page.
articles like this need to factor in the overall time/effort tradeoff in developing the set of prompts necessary to reach an overall goal... for example, if I can achieve the same results while being sloppy/lazy/vague with my prompts, I wouldn't bother with the extra discipline or time expended to craft "to spec" prompts
agreed. I'd also love to see more comparisons like this.
I do think that a well speced out prompt can be finished faster by an agent then if prompted lazily. I've also found measuring this to be challenging, since issues that come from lazy prompting can hide until its too late to pinpoint exactly what prompt introduced them. But maybe this should be the new "git blame"... gears are already turning for what evals could look like for this.
Many of his other prompt examples also illustrate this. The author is massively leveraging the architecture of the organization through familiarity. It could well be the case that even an LLM was used to set that up, but I doubt it happened in one prompt or without critical thought. To be fair, I guess he's sort of trying to point that out, but by saying "Look how lazy I am while it still works" or something like that in order to look cool, I think all he's conveying is "Look how easy this is", when it's potentially not and could increasingly cost him/his company wildly varying amounts of money based on how he goes about it. When tokens were virtually free, no one cared. This is increasingly not true. Many are drawing comparisons to early cloud architecture.
I find this is sort of similar to the problematic idea that in theory anyone can clone your repo and get a local dev environment running in a few seconds. So then you try it with a non-technical staff member and quickly realize how unrealistic that is as you run into abstraction after abstraction that is completely foreign to them.
In my experience, it varies widely by workload. Vision is great for some UI stuff but extremely terrible for my work, which involves visualizing connectivity and geometric stuff for road structure for autonomous vehicles. It can get super frustrating when I paste a screenshot and go "this polyline isn't connected when it should be" or "those polylines are self intersecting in an invalid way" and then the AI literally can't see what I'm talking about. It would have to then write debug code to dump out the coordinates of the polylines to convince itself that the bug even exists. Then it would often claim that it's fixed when it is clearly not.
I previously complained about GPT 5.6 Sol not being able to solve a simple "follow the lines" puzzle [1] without having to write custom Python code for it. GPT-6 Astra, however, is vastly improved, and could do it without writing Python. Still, for my work, it can occasionally struggle.
[1] https://news.ycombinator.com/item?id=49334444
I do think that a well speced out prompt can be finished faster by an agent then if prompted lazily. I've also found measuring this to be challenging, since issues that come from lazy prompting can hide until its too late to pinpoint exactly what prompt introduced them. But maybe this should be the new "git blame"... gears are already turning for what evals could look like for this.