Skill Following
Do AI agents actually use what they look up?
Paper by Seonghyeon Cho · Chanjun Park
Presented by Kevin Antony
Seminar: Retrieval Augmented AI
AI agents can look things up
Problem
Search
AI
Guide
Guide
Guide
Answer
Did the guides help?
A student with a formula sheet
Used the sheet
80%
easy problems
Didn't use it
40%
hard problems
+40 points. Looks like the sheet helps.
Same easy problems, no sheet
95%
The sheet actually cost 15 points.
The fix: test each problem twice
Problems where the AI searched
Without help
With help
1rescued
3broken
Score = 1 − 3 = −2
The paper calls this score RAE.
Old score vs. new score
Old score
New score (paired)
0
Every one of them flips direction.
One model, 45 changed answers
Llama-3.3-70B on math problems
3rescued
42broken
The old score said this model gained +14 points.
Why the old score was fooled
Same problems, tested with no help at all (illustration)
Problems it searched
Problems it skipped
17 / 20 solved
6 / 20 solved
The searched problems were already easier.
In the real data: up to 65 points easier
Looking it up is not using it
DeepSeek-V3.2 on coding problems
0%
looked up a guide
actually used the guide
used it and got it right
Why help breaks solved problems
2 in 3
ignored the guide
and still got confused by it
Other causes: wrong answer format, a guide forced onto the wrong problem, a small bug
Even junk text hurts
What search returned · score for DeepSeek-V3.2
Every version scores below zero.
Limits of the study
Only short coding and math problems
A small, fixed set of 17 guides
Not tested on long, multi-step agents
Takeaway
One thing to remember
Test the same problem with and without the help.
Count
What it rescues
Count
What it breaks
→ / click: next · ← back · P: presenter · N: notes · F: fullscreen
1 / 12
LIVE AUDIENCE VIEW
UPCOMING
00:00
--:--:--
Slide 1 / 12
SPEAKER NOTES (PRIVATE)
[Space / →] Next · [←] Back · [P] Presenter View · [F] Fullscreen