AGENTS.md outperforms skills in our agent evals (vercel.com)

Article summary

The article discusses an experiment where an AI coding agent was given access to framework-specific knowledge through two approaches: skills and AGENTS.md. The results showed that the AGENTS.md approach, which provides a compressed index of documentation, outperformed the skills approach in terms of pass rate. The article suggests that this is because the AGENTS.md approach eliminates the decision point for the agent to invoke a skill, provides consistent availability of information, and avoids ordering issues. The authors conclude that providing an AGENTS.md snippet can be a reliable way to give coding agents access to framework knowledge.

Main themes

• AI coding agents
• Framework knowledge
• Documentation access
• Skills vs AGENTS.md
• Model training
• Evaluation methodologies

What commenters say

• The results of the experiment may not be conclusive due to the small sample size and potential noise in the data.
• The AGENTS.md approach may be more effective because it eliminates the need for the agent to decide when to invoke a skill.
• The skills approach may improve as models become better trained to use them, but for now, the AGENTS.md approach is more reliable.
• The use of subagents and smaller models to curate skills and context may be a more efficient approach in the future.
• The inconsistency and non-determinism of LLMs make it difficult to draw meaningful conclusions from benchmarking results.
• The article's findings may be specific to the particular model and experiment used, and may not generalize to other situations.
• The AGENTS.md approach may be more effective because it provides a more direct and explicit way of accessing documentation, rather than relying on the agent to invoke a skill.
• The experiment's results may be influenced by the specific wording and structure of the prompts and documentation used.