for sure. i've found the same. however they can be useful for putting already verified information into mnemonically optimal configurations. see dwarkesh's build here:
Honestly, I wouldn't update super strongly in either direction on this.
"Professionally designed" flash cards sometimes contain "errors" in them; I find a particularly common form of errors in human-generated "professional" (as opposed to "passionate autodidact") cards is oversimplifying a nuanced belief into a definitive assertion that fits within a card.
Even if you manually make your own flash cards, if you make thousands of them over your life time, you'll probably make a typo at some point, and "reversing the direction of the gradient" sounds like a plausible typo one might make when designing one's own flash cards.
This is a crucial piece of information. The AI companies have a bunch of poorly named products that can produce crappy output.
For what it's worth I recommend you try the same thing with Google Gemini 2.5 Pro (it's currently available for free) a flagship model intended to compete with o3-pro and Claude 4 Opus.
I suspect Clause 4 Opus would not make the errors you describe, but I would like to know for sure.
This was Claude 4 Opus. I just gave the deck to Gemini 2.5 Pro and it did *not* find Claude's error. Also it said a correct card was erroneous. the prompt was:
Please check the following developmental biology Anki deck for errors. I know it contains at least two errors and possibly more. For each error that you find, output the incorrect card and a corrected version.
Awesome! Could you please post one or two of the exact errors? I would like to test something. There are many naming ambiguities in biology. (For example the Mediator complex proteins are numberered, e.g. MED28 etc but the numbers are inconsistent across species)
for sure. i've found the same. however they can be useful for putting already verified information into mnemonically optimal configurations. see dwarkesh's build here:
https://www.generateflash.cards/
Honestly, I wouldn't update super strongly in either direction on this.
"Professionally designed" flash cards sometimes contain "errors" in them; I find a particularly common form of errors in human-generated "professional" (as opposed to "passionate autodidact") cards is oversimplifying a nuanced belief into a definitive assertion that fits within a card.
Even if you manually make your own flash cards, if you make thousands of them over your life time, you'll probably make a typo at some point, and "reversing the direction of the gradient" sounds like a plausible typo one might make when designing one's own flash cards.
Which Claude model?
This is a crucial piece of information. The AI companies have a bunch of poorly named products that can produce crappy output.
For what it's worth I recommend you try the same thing with Google Gemini 2.5 Pro (it's currently available for free) a flagship model intended to compete with o3-pro and Claude 4 Opus.
I suspect Clause 4 Opus would not make the errors you describe, but I would like to know for sure.
This was Claude 4 Opus. I just gave the deck to Gemini 2.5 Pro and it did *not* find Claude's error. Also it said a correct card was erroneous. the prompt was:
Please check the following developmental biology Anki deck for errors. I know it contains at least two errors and possibly more. For each error that you find, output the incorrect card and a corrected version.
Awesome! Could you please post one or two of the exact errors? I would like to test something. There are many naming ambiguities in biology. (For example the Mediator complex proteins are numberered, e.g. MED28 etc but the numbers are inconsistent across species)
oooo I'm interested in comparing how different LLMs trained on the same material make Anki cards - your next post, Metacelsus?