The ecosystem of “AI skills” — modular instruction packs that extend an LLM with task-specific know-how, whether they’re called Skills, plugins, agents, MCP servers, system-prompt templates, or tool bundles — has exploded fast enough that “which one should I use?” has become the dominant question. The answer is rarely obvious from a README, and almost never from a star count.

This guide is opinionated. Its core claim is that skill evaluation is fundamentally a function of your tasks, not of the skill, and that most popular shortcuts (stars, recency, feature lists) are weak proxies that mislead more often than they help.

Table Of Contents

• 1. What “evaluating a skill” actually means
• 2. A framework that actually works
  • Trigger accuracy
  • Execution quality
  • Cost
  • Trust
  • How to actually run the evaluation
• 3. Choosing between skills that seem to do the same thing
  • They almost never actually do the same thing
  • Decision heuristics
  • When you should run both
• 4. GitHub stars: what they actually tell you
• 5. A consolidated checklist
• 6. The meta-point

1. What “evaluating a skill” actually means

A skill has to do two things, and they fail independently:

1. Activate at the right moment. The skill must be invoked when it’s relevant and stay quiet when it isn’t.
2. Produce correct output once activated. Given the right context, it must do the job well.

Most evaluations conflate these. A skill can have flawless execution and still be useless because it never fires; another can fire reliably and produce mediocre work. Always score both, separately.

There’s also a third dimension people forget: cost. Skills are not free. They consume context window, add latency, can conflict with other skills, and impose maintenance burden. A skill that’s 70% as good as the alternative but a quarter of the size may be the right pick.

So the real evaluation matrix is roughly:

If you can’t answer all four for a candidate skill, you haven’t evaluated it. You’ve read its README.

2. A framework that actually works

Trigger accuracy

The skill’s description is its trigger. The model decides whether to invoke a skill primarily by matching the user’s request against the description text. Two skills with identical capabilities but different descriptions will have radically different fire rates.

Two failure modes to measure:

• False negatives — the skill should have fired and didn’t. Symptoms: the model attempts the task with general knowledge, missing the constraints the skill encodes. Often caused by descriptions that are too narrow, too jargon-heavy, or missing the synonyms users actually employ.
• False positives — the skill fires for unrelated tasks, or for tasks adjacent enough that it derails the answer. Symptoms: irrelevant scaffolding appears in responses, token cost balloons, the model gets confused by conflicting instructions. Caused by descriptions that are too broad, that lean on common verbs (“create,” “analyze,” “help with”) without disambiguating nouns, or that don’t specify what’s out of scope.

To test, write a fixture of, say, 40 prompts: 20 that should trigger the skill, 20 that shouldn’t but are close enough to be tempting. Run them. Compute a confusion matrix. The skills that do well in the wild are usually the ones whose description spends as much energy saying “do not use this when…” as “use this when…”.

Execution quality

Once a skill fires, you’re evaluating its work product, not the model’s. The right tool here is a regression suite, not vibes:

• A handful of representative tasks with known-good outputs.
• A handful of edge cases — empty input, malformed input, input at the boundary of the skill’s stated scope.
• A handful of adversarial inputs — prompt injection embedded in user data, conflicting instructions, ambiguous goals.

What you’re measuring is not just “did it work” but the shape of the failure when it doesn’t. Skills that fail loudly and recoverably (refuse, ask, surface the problem) are dramatically more useful in production than skills that fail silently with plausible-looking but wrong output. The latter category is genuinely dangerous and is everywhere.

Run each test multiple times. LLM outputs vary. A skill with 80% success at low variance is often better than one with 90% success at high variance, because the second one is harder to build on top of.

Cost

Three costs are typically underweighted:

• Token tax. A skill’s instructions are loaded into context for every invocation (or every relevant conversation, depending on the system). A 4,000-token skill that fires on 30% of conversations is a substantial recurring expense. Read the skill file. If it’s bloated with examples, hedges, and prose where a sentence would do, that bloat is being paid by every user, every time.
• Conflict surface. Skills can contradict each other or layer instructions in ways that produce worse output than no skill at all. The more opinionated a skill is, the higher its conflict surface. Test it alongside the other skills you actually use, not in isolation.
• Maintenance debt. A skill that pins an external API, a library version, or a model version is a future bug. Note the dependencies; price in the rot.

Trust

For any skill you’d use beyond personal experimentation:

• Who wrote it, and what’s their track record?
• Does it execute code? Fetch URLs? Touch credentials? What’s the blast radius if it’s wrong or compromised?
• When was it last updated, and was that update substantive or cosmetic?
• Are there enough other eyes on it that a problem would have been caught?

A skill from a thoughtful individual who maintains it actively is usually safer than a skill from an anonymous repo with twice the popularity. “Many users” is not the same as “many auditors.”

How to actually run the evaluation

The single highest-leverage practice is keeping a fixture set — a folder of representative prompts and expected outputs that you rerun every time you change skills. It doesn’t have to be elaborate. Twenty prompts is plenty to start. The fixture is what turns evaluation from a vibe into a measurement, and it’s also what lets you re-evaluate when the skill author ships an update.

Without a fixture, every “evaluation” is a fresh impression, easily swayed by the last interaction you had.

3. Choosing between skills that seem to do the same thing

They almost never actually do the same thing

When two skills look like duplicates, the duplication is almost always at the surface. Look one layer down and they differ in one or more of:

• Scope. One skill creates PDFs; the other reads them; both call themselves “PDF skill.” One does flowcharts; the other does both flowcharts and architecture diagrams. The common case may be identical but the long tail diverges sharply.
• Prescriptiveness. One skill enforces a house style (specific fonts, layouts, defaults); the other gives the model freedom. Neither is wrong — they just optimize for different users. The opinionated one is better if its opinions match yours and worse if they don’t.
• Trigger boundary. One fires for “make a chart”; the other only fires for “make a chart from this data.” This is invisible from a feature comparison and decisive in practice.
• Failure modes. One skill, when uncertain, asks a clarifying question; the other guesses. Both can be defensible; you should know which you’re getting.

The exercise: open both SKILL.md files (or whatever the equivalent is) side by side and mark every sentence that diverges. The differences you find there are the real differences. The features list is marketing.

Decision heuristics

When the diff is small and you still have to pick:

1. Optimize for your most common case, then your worst case. Pick the skill that wins on the task you do every day. Then sanity-check it on the case where being wrong hurts most.
2. Prefer narrow + composable over broad + monolithic. A skill that does one thing well plays nicely with others. A skill that tries to own a whole domain tends to crowd the context and conflict with neighbors.
3. Smaller breaks ties. All else equal, the skill with a tighter SKILL.md is the better citizen.
4. Recent meaningful activity breaks ties. Not commits — meaningful commits. A repo with three issue replies in the last month tells you more than a repo with a hundred green-square commits that are all README polish.
5. Test on your worst input, not the demo input. The README’s example is the case the author optimized for. The interesting question is what happens on the case they didn’t.

When you should run both

Rarely, but it does happen: two skills cover overlapping but non-identical traffic, and the cost of either one missing its case is high. In that situation you can keep both, but you should explicitly engineer the trigger boundaries so they don’t fight each other — for instance, by editing the descriptions to make the partition clear. If you can’t make the partition clear, you have one skill too many.

4. GitHub stars: what they actually tell you

GitHub stars are a bookmark-and-attention metric, not a certificate of quality, production reliance, or maintenance. At best they hint at coarse popularity within a niche; they also persist on abandoned repos, spike with hype, skew by category and language community, and can be gamed. Glance if you like, then weight recency and substance of commits, issue response, contributors, dependents, releases, and tests.

5. A consolidated checklist

Before adopting a skill:

•  I have read the SKILL.md (or equivalent) end to end, not just the README.
•  I know what tasks it’s meant to fire on and what tasks it’s not meant to fire on.
•  I have a fixture of at least 20 prompts — half should-fire, half shouldn’t-fire — and I’ve run them.
•  I have a small set of known-good output tests for the execution path.
•  I’ve measured the token footprint and decided I can afford it on every relevant turn.
•  I’ve checked it doesn’t conflict with the other skills I rely on.
•  I know who the maintainer is and when they last shipped something meaningful.
•  If it executes code or touches credentials, I’ve read what it actually does.
•  I have not, at any point, made a final decision based on a star count.

6. The meta-point

The reason most skill evaluations are bad isn’t lack of effort — it’s that they’re trying to ask “is this skill good?” as if the answer were a property of the skill. It isn’t. A skill is a fit between a tool and a workload, and “good” is a relation, not an attribute. The same skill can be excellent for one team and a net negative for the next, with no contradiction.

The practical consequence is that the highest-value thing you can build for yourself is not a list of “the best skills” — it’s a small, durable test fixture that represents your work. Once you have that, evaluation becomes mechanical. You stop arguing about stars and start measuring outcomes, and the right skill — for you — becomes obvious.

Everything else in this guide is a way of buying time until you have that fixture.