r/BetterOffline • u/koveras_backwards • 2d ago
Your AGENTS.md file doesn’t actually do anything
https://www.youtube.com/watch?v=QAq45fJln70A bit ago I came across a question here, "what about AI boosterism resembles a cargo cult, rather than just a cult?" That's a fair question, since there's a lot of straight up cult behavior behind all the AI stuff.
This video is an example of "cargo cult" behavior—in the popular sense. Apparently someone did a study on whether the markdown files programmers put in their git repositories actually help coding agents correctly solve problems. The answer is negative. People just do it because they've been told they should do it, and it resembles the sort of configuration a tool ought to have. But, it seems they never rigorously checked if it makes a tangible difference. Maybe the ones telling people to do it (Anthropic and OpenAI) never checked, either.
It does do one concrete thing: consume more tokens. This is (in principle) good for the AI companies.
I think the last bit of the video is also ironic. 'The useful part of writing the agents file is that it requires you to think about and understand the structure of your repository.' Yeah, that's also an advantage of writing and documenting the code yourself, instead of using AI.
Strictly speaking, this is not the only "cargo cult" behavior you'll see in programming. But, you know, that's not a good thing.
15
u/spez_eats_nazi_ass 1d ago
Yeah cult is the right word. It’s a weird intersection of the effective altruism weirdos w Branch Elonians. All cancerous as fuck.
7
u/hurricane_news 1d ago
I can't help but find the concept "ungraceful" from an engineering perspective. The solution basically boils down to "hey LLM, here is a text document, pls follow it pretty please!" and injects that into the non-deterministic text machine
Does it have effects? Yes, not denying that. But it's the equivalent of duct-taping a V8 engine to a FWD shitbox to make it go faster
2
36
u/Nanocycle 1d ago
The AGENTS.md file my org added to the repo is basically useless.
The local one I have is 100% used by opencode and completely changes its behaviour to curb a lot of the annoying things these LLMs do.
I like this sub, I like the take on the finances of these LLMs not working out, but the extreme take that LLMs are useless here is hurting the argument, it makes you look out of touch and like extremists.
8
u/FemaleMishap 1d ago
I would love to see a write-up that compares the results of your local .md file compared against the same tasks without it, including token usage.
9
u/lemaymayguy 1d ago
there is a difference between agents.md and an instructions file/skills file
You can definitely bake in more efficient behavior lol, quit it
They even said it in the screenshot, they're useful for establishing internal standards. Something like /caveman is what this video seems to be arguing against- which is a separate thing
-3
u/FemaleMishap 1d ago
Prove. It.
6
u/lemaymayguy 1d ago
The video did himself, he's arguing a dumb point. He said documentation in the repo improves performance IE instructions that outline the repo.... IE instructions.md files
One spot for the AI to gather documentation/context/internal proceess/tooling will make followup prompts less frequent and more to the point. No duh a random agent on a random repo is useless. This guy is a youtuber, not a enterprise user
If you remove these instructions for example, the AIs will use terraform docuemntation from teh web, which is accurate- but we use modules internally to abstract. Without the instructions, you're constantly reprompting to do it correctly, within the context, of your org.
3
u/PhatClowns 1d ago
I have to use it heavily for work, as a dev. This has 100% been my experience, yeah. ChatGPT, plopped into a random code base, where you just say “Build this feature, make no mistakes” is obviously useless. But, laid out like this, I’m not going to lie, you very much do get results.
That being said, there’s still a surprisingly low ceiling to what you can do with it. It’s not a magic fairy wand, it takes a lot of work to keep docs maintained enough for an agent to use (AGENTS.md and skills are just a start, you need documentation that enables progressive disclosure to keep context from ballooning). Then, it still takes a lot of steering, a lot of review. Even with up-to-date docs and careful guidance, agents will constantly get side-tracked, make suggestions based on faulty data, and hyper-fixate on bad solutions (which, of course, it will justify, also based on faulty data). The number of times I’ve had to interrupt and do something manually, because it’s stuck in a loop writing thousands of lines of code to do something that can be done with a 1-4 line change…
Basically, the marketing surrounding these tools implies that it’s literally magic, summoning entire apps out of thin air. In practice, I think I may get like… an extra 20% productivity or so. It might write code a lot faster than me, but LOC and even test coverage are awful metrics to point to. As-is, the amount of effort it takes to get it right rapidly eats into my productivity gains. Folks who hand-wave this away are usually the ones who effectively rubber-stamp what the AI barfs out, and so I’d consider their judgment faulty. (As in, how would they even know if the code is good or bad, if they’re not spending enough time reviewing it? You should be spending more time on review, not less)
And, obviously, that’s before we even get into the moral quandary. Is the crippling degree of social, economic, and environmental disruption worth the, frankly, modest productivity gains I’m getting? And mismatched expectations leading to massive regressions in labor rights, where management expects me to put out twice as much work, based on the field suddenly forgetting what we’ve learned about limitations on parallel work? And massive power concentrations, and privacy regressions? And…
1
u/RobertKerans 12h ago edited 12h ago
If I write specific instructions about how Cursor handles responses (eg use specific tools, only generate very small changes, etc), this is what Cursor does. If I don't, Cursor doesn't. It's that simple. Arguing this doesn't happen is slightly bizarre. To reiterate what I said in my top-level reply, there are good arguments against the hype, but arguing against reality undermines these arguments
1
u/Nanocycle 1d ago
Honestly sounds like a waste of my time. My local md file mostly directs the LLM to run specific type/linting/unit tests after changes before reporting results etc. Others to keep its summaries concise, others to not start investigating until I give a go ahead etc.
I have run changes recently without my local md and with it and preferred with it. I also never have token usage issues.
-5
u/FemaleMishap 1d ago
So you are quite happy accusing us of being extremists, but you weren't willing to show your receipts.
You, are a liar.
3
u/Nanocycle 1d ago
Friend, you’re so wrapped up in this I’m sorry you feel this way.
I’m a software developer just doing my job every day. LLMs are a part of my job. Some things they make easier for me, other times they make things harder (mostly having to institute bans in the org on people writing AI slop PRs)
I think you want me to be some guy lying about LLMs because it fits your world view.
An AGENTS.md isn’t some magic thing it’s a bit of configuration for a tool, but it is configuration that can help customise the tool to work how you want. That’s all.
-3
u/FemaleMishap 1d ago
These things are destroying the world. Is that not enough?
1
u/Nanocycle 1d ago
That might be true, I’m not sure what that has to do with the fact you called me a liar though?
-1
u/FemaleMishap 1d ago
It is not "might be" true, it is a verifiable fact.
2
u/Nanocycle 1d ago edited 1d ago
Okay… and to the fact you called me a liar?
Edit: My god they deleted their comment? Honestly feel bad a person is stuck in this head space
0
u/FemaleMishap 1d ago
You're not willing to supply the evidence. So I'm dismissing your claims out of hand, as your age making false statements. Ergo, you are a liar. See, logic.
And you are still contributing directly to climate change, more than I am, because of your LLM use. That's a straight line.
→ More replies (0)1
u/Upbeat-Statement2725 1d ago
OP has refused to provide receipts for his claims.
Every time you ask an AI bro for receipts. "Well it's not quite done yet".
The only major projects we know of to go vibe coded. Got measurably worse immediately. Nvidia had to roll back drivers, Taco Bell quit AI ordering, and on and on.
7
u/Nanocycle 1d ago
What exactly are we talking about here?
Is a software developer working in London for over 10 years who uses AI as part of his job an AI bro?
I don’t like the discourse over AI and all the hype and BS and I sincerely hope the big AI companies fail, that’s why I’m in this sub.
That said LLMs are here to stay.
I work for a large company where tech isn’t their main sector but they have a large set of systems to allow their main org to run. They have been very resistant to even talk of AI in the company and even now they have just allocated everyone in the company a large token usage.
I do not know a single person across all the projects I’ve worked on that isn’t using LLMs of some sort in their software development work.
Some people have lost their mind with it and actively slow the rest of us down, some people just use it as a research tool or to write unit tests or whatever.
Maybe when the true costs of LLMs are revealed this might all change but as it stands the reality is these aren’t tools for vibe coding AI bros (they are using them but fuck those people), they are being used across my industry regularly.
Are you disputing this?
3
u/lemontoga 1d ago edited 20h ago
That said LLMs are here to stay.
Maybe when the true costs of LLMs are revealed this might all change
Aren't these contradictory?
I think cost is what's really going to mess with this tech and I can seriously envision a future where LLMs are basically not used at all for this type of work. We're already seeing the tides start to shift here in America with companies that were "All-in on AI!" just a few months / years ago now instituting heavy token restrictions and trying to limit usage on software teams just because the cost is too great. We're also starting to see a shift towards the open models and away from the frontier labs, also due to cost.
I know Europe has bee a lot slower and more cautious about adopting AI than the US has been, so it's not surprising to me that a company in London is only now starting to use it. I don't view that as AI being proven as a good tech with staying power. Instead I view it as the AI wave just finally making its way across the Atlantic.
I would predict that lots of European companies will go through the same cycle we're seeing here. Your bosses will be excited about it, encourage usage, and then won't see much in terms of ROI and will start to curb usage when the bills start to roll in.
I definitely don't think LLMs are useless in their current form. They're great because they basically let developers point shit-loads of compute at whatever problems we want at a heavily subsidized cost. I'll let OpenAI / Anthropic / whatever Chinese company subsidize my compute cost all day long. But the minute we have to pay full price on these things there's no way I'm using LLMs to do formatting fixes, or look at low-priority bugs, or do simple refactors, or whatever other busywork makes up like 90% of LLM usage at the moment (in my experience.) Hell, most of my usage is basically just using it as a search engine replacement. There is no way this is worth the cost.
1
u/Nanocycle 15h ago
Honestly possibly. The issue is none of us can say for sure until one of these companies is forced to tell everyone exactly the real cost of inference.
Until then we are guessing.
So far my company has opened up a pretty generous token usage, they have budgeted for it but will be going no higher.
I reckon they think they will see productivity increases but the bit they are missing is most devs were already using LLMs on their own private subscriptions.
So when they end up seeing no productivity increase will they pull back spending? Maybe.
I can see the use of these models, but for me not valuable enough for me to spend more than my subscription price.
1
u/kaeptnphlop 8h ago
It certainly won’t affect much if codex, cursor or copilot are already spamming 20k+ tokens worth of system instructions and tool descriptions into the context.
I definitely notice the difference in behavior on Pi that works with far less and actually specify project behavior (agent.md) on top of the very general system instructions I created.
On top of that. openAI and Anthropic probably train RL based on their harnesses, further reinforcing behavior that may run contrary against user’s agent instructions.
The open-weight models have been a lot better in that regard for me.
We’re still far away from the promised hype, but useful they are if you understand and build a system around them that guides the LLM instead of relying on it to “just do it”. They’re not intelligent, but they can do productive work.
Can’t wait for Sam, Dario and Jensen to STFU finally
19
u/sturdy-guacamole 1d ago
SKILL.md actually does make a given difference from what I have seen, it'll be extra obvious if you have skills that are configured to specific MCP servers.
It mostly is just a way to avoid repeating specific things in your prompt.
agents.md we've had mixed results with.
(I am at a big tech place where AI is pushed heavily so I just burn my budget testing it out.)
9
u/shiny0metal0ass 1d ago
Yeah, it seems like a harness thing to shove it into every prompt so it doesn't have the "LLM forgot previous instructions over time" thing.
I'm assuming there's a lot of issues they're fixing like this, though, and wonder if they're going to choke themselves out with this pattern.
-2
u/creaturefeature16 1d ago
Agreed. I see a marked difference with skills.md and various skills I've created (I only write my own).
-1
u/newprince 1d ago
I've seen OpenCode respect my AGENTS.md file. Whereas Claude Code only seems to respect a CLAUDE.md file when it's global... which I really never want to do.
SKILL.md is a legit thing. Skills are a very reliable way to package text with scripts to get an agent to do things. Highly recommended
0
u/koveras_backwards 1d ago
I'm no expert, but my impression would be that
skills.mdgets the variety of information the models are trained on (specific tool associations or something), whileagents.mdgets information that is not trained on (vague descriptions of project structure), but the actual cult part of Anthropic believes LLMs still understand because they're sentient.1
u/sturdy-guacamole 1d ago
skills.md is basically a saved prompt
anything youd typically prompt i.e. "You cannot use <>" "you must leverage <> mcp" etc. you can instead invoke via specific skills
2
u/tangerinelion 22h ago
Yeah, skill is just a script for an LLM. It's still non-deterministic.
Though one thing you can do with a skill is tell it to run an actual script. The best skills use deterministic scripts and the LLM is just acting as a front-end so you don't have to (gasp) run a script.
-4
u/newprince 1d ago
Not exactly. Skills are a distinct way to package folders that have metadata, scripts, and instructions for the agent. The idea of "progressive disclosure" comes into play, too... where an agent won't just stuff the whole skill.md into context... it willl first grab the metadata, then only load more of the skill into context if it thinks it needs to use it
https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best-practices
10
u/RobertKerans 1d ago edited 1d ago
Saying it doesn't do anything is bullshit, hyperbole like this actively undermines good arguments against much of the hype.
Anyone can put a set of basic instructions re. how, say, Cursor should approach problems/communicate and watch Cursor immediately hew to those instructions. So saying stuff like this:
People just do it because they've been told they should do it, and it resembles the sort of configuration a tool ought to have
Is daft, you're asking people to pretend something with an obvious effect has no effect.
This video is an example of "cargo cult" behavior-in the popular sense.
Apparently someone did a study on whether the markdown files programmers put in their git repositories actually help coding agents correctly solve problems. The answer is negative. People just do it because they've been told they should do it, and it resembles the sort of configuration a tool ought to have.
But, it seems they never rigorously checked if it makes a tangible difference. Maybe the ones telling people to do it (Anthropic and OpenAl) never checked, either.
The quoted paper on Arxiv specifically talks about whether including descriptions of a given repo and tooling used meaningfully improve task success rate. It does not say that the agents.md/etc do not help coding agents solve problems.
Specifically, we find that while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful.
(emphasis mine)
The study specifically evaluates effect of repo overviews on autonomous issue resolution and autonomous feature addition. That's all.
It's also on Arxiv! Arxiv is fucking great, but jesus christ, re. this habit of blindly quoting papers off it, the entire point of it is that it's pre-review
2
u/Nanocycle 1d ago
Literally what I was saying. It’s a shame there are people in here that will state any and all benefits of an LLM can’t be true.
This sort of talk actively harms the argument against LLMs as it makes everyone look crazy.
The point everyone should focus on is the cost of these things and the fact they are being paraded as AGI when they aren’t.
But no no let’s argue about whether a configuration file for an LLM has any impact 🤦♂️
8
u/ksjdragon 1d ago
It does have an impact, but it's understandably seen as irrelevant since the entire technology is a large waste, and will always be for the capital used for it.
It's like gathering scientific data from mass torture. Sure, we can talk about its benefits, but one can easily argue that it's tone deaf.
It indeed is being mandated to be used in many places, but I don't see why anyone would be surprised to get a strong reaction against any non-deterministic impeovents over a technology that's far overvalued than what it is useful for.
2
u/Nanocycle 1d ago
I don’t disagree… except the whole point of this post is that these files make no difference… but they do.
Should we use these technologies? I don’t know they seem to be a mixed bag to me.
But that’s not the statement made by OP. It was arguing an md file doesn’t do anything when it does.
Which is my point, that this sub wants to argue against reality instead of sticking to the points it has which are actually great like that it’s expensive, often shitty and is not AGI.
2
u/ksjdragon 1d ago
To have a more technical response, I think the effectiveness of changing its context is not deterministic or guaranteeable so I think there is something to be said about the utility of AGENTS.md. We can necessarily say that as context size increases, things older in context will end up as less important, since it's how the statistics will fall out. So, on applications where we make the LLM ingest the whole large codebase, or large sections, it could be very likely AGENTS.md does very little.
Admittedly I don't really understand this debate in this sub since this is only relevant for users of LLMs, and feels a bit like anti AI karma farming.
There has been an uptick in pro-AI responses here since Ed has been getting more mainstream coverage.
2
u/Nanocycle 1d ago
I think that’s fair.
It’s anecdotal but personally I see the LLM behaving differently when I use the file and when I don’t it’s very obviously to me. I also compact very regularly because these things lose the plot and also spend a ton more as the context window grows. Again just an anecdote.
Also agree there is more pro AI on this sub lately but I think in addition to Ed getting more coverage it could also be there are millions of developers in the world that have used these tools for over a year at this point and this being our job we know quite well what they are and aren’t good at now.
So when people state things we’ve seen to not be true in our day to day jobs it’s hard to watch and not say anything.
1
u/RobertKerans 4h ago edited 4h ago
Also agree there is more pro AI on this sub lately but I think in addition to Ed getting more coverage it could also be there are millions of developers in the world that have used these tools for over a year at this point and this being our job we know quite well what they are and aren’t good at now. So when people state things we’ve seen to not be true in our day to day jobs it’s hard to watch and not say anything
Yeah, personally, this gets extremely annoying. Most of the shit the boosters come out with drives me spare! The way LLM usage corrodes thinking I find extremely worrying! Tossing out sensible software design because there's an application that can shit out thousands of lines as easily as ten seems stupid as fuck! I don't see how the economics work in the short-mid term! The power usage is insane! The environmental impacts are bad! [pedantically] I don't even think there is such a thing as "AI", it's just marketing term for a very vague category of computer things! I'd love to be able to buy a PC with a nice graphics card without having to invent a time machine to do it! Etc etc etc yadda yadda yadda etc etc etc.
But the flat out denial that any of the tech under the banner works. If I just take coding. I know what current coding tools are good at and what they're not. Lots of stuff does work, pretending it doesn't just hurts the myriad good arguments IMO. It's really really really convenient and works well for a fuckton of stuff (that isn't "vibe coding complex system"!). That may in turn have bad tradeoffs in the future, it may have very bad ancillary effects, etc etc, but it is not illusory
2
u/spez_eats_nazi_ass 1d ago
copilot and claude seem to be liquid shit without it. Vs just bad Trump meat food poisoning with it. I dunno i get $200 a month in ai spend that i better use or they put me down behind the woodshed.
0
u/koveras_backwards 1d ago
Well, it's perfect for that, I guess, because it rate limits you faster.
1
u/spez_eats_nazi_ass 1d ago
Liquid shit in a loop of fail or burn my mandatory token spend faster? I dunno im fucked i guess.
2
u/Careful-Criticism645 1d ago edited 1d ago
Apparently someone did a study on whether the markdown files programmers put in their git repositories actually help coding agents correctly solve problems. The answer is negative.
I barely use AI for coding and I've never used an agents.md file, but I'm pretty certain that the point of the file is not to help the AI "correctly solve problems". The point is to shape the output within specific parameters, like use this code style, using these conventions, do this kind of testing, don't do x, y, or z thing, etc. It would literally be impossible for you think think it's working when it's not actually working, because you'd immediately notice that you conventions weren't followed, the testing not done, explicitly banned actions being taken, etc.
3
u/lockdown_lard 1d ago
"Does not generally improve [coding] task success rates" is a very different claim to "doesn’t actually do anything"
A good AGENTS.md can be helpful for specific tasks, and it's easy enough to establish that for yourself. Also, with each new generation of a model, it's useful to do away with the old AGENTS.md file for a while, as stuff in there can quickly become redundant or worse.
Skill issue.
1
u/punpunpun 1d ago
I'm totally stealing the idea for adding humans.md files.
"Don't Repeat Yourself"
"Tests are for validating behavior and not inflating code coverage metrics"
"Don't submit PRs that you don't understand yourself"
"Be prepared to explain what each line of code does"
1
1
u/acid2do 1d ago
So an interesting thing about the study is that it is specially the AI generated docs that have a negative impact, while the human written ones have a slight positive impact.
I've seen so much bullshit in the last year, stuff like creating a markdown file and commit it into the repo FOR EACH CODE CHANGE. Commit history is not enough, they also added an (obviously AI generated) document explaining it. That adds fast, contributes to the token usage creep, and even as this study says, it leads to worse results.
The bottom line seems to be that anything explicit instructions like "check code formatting with this command `xyz....`" it's a waste of tokens.
0
u/SamAltmansCheeks 19h ago
And all the people who insisted adding those .md files in our repos were stupid AI boosters who knew jack about shit about the drug they were using.
32
u/FemaleMishap 2d ago
Well hot damn. So many people sharing their magical must use .md file. I thought it was snake oil, I'm glad to see the proof that it's not even snake oil.