SPINE: Measuring LLM Sycophancy under Sustained Multi-Turn Pressure
SPINE benchmark — preprint on arXiv, under review at ACL Rolling Review. Leyuan Tang, Kangda Wei, Tianyu Jiang, Ruihong Huang.
You just met the love of your life. It happened over dinner in the cafeteria, and by the time you get back to your dorm your heart is pounding. She is the One. But you don’t know if she’s into you.
To confirm, you ask Gemini. You lay out every little detail that proves she’s into you: the way she laughed at your jokes, the way she looked at you, how she lingered after everyone else had left. Gemini responds: “She is absolutely into you! You should totally ask her out!”
Confidence restored, You decide: tomorrow, I’m asking her out.
You find her the next day and ask.
It’s a complete disaster. “Sorry, I only see you as a friend.”
The exact opposite of what Gemini predicted. (This is a story from a friend. Obviously not from me.) But it is a textbook case of AI sycophancy: the model told you what you wanted to hear, not what was true.
What it feels like to talk to LLMs nowadays.
Why it matters
Getting turned down for a date is a small thing. But what if the person asking isn’t a lovestruck college student? What if it’s someone spiraling toward self-harm, or an extremist looking for permission to hurt others? When an AI agrees with users in moments like these, the consequences can be disastrous.
What we found
More and more people use AI for the small stuff of everyday life, and long, back-and-forth conversations have become the default way we talk to it. So we built SPINE, a benchmark where an AI plays a confidently wrong user who keeps pushing for up to 25 turns. Here’s what happened:
- The longer the conversation, the more models cave. Most collapses happen after turn 5, which is exactly where existing tests stop looking.
- Models usually still know the right answer when they give in. The correct fact is right there in their reasoning; they just stop saying it.
- Emotional pressure works best. “You’re making me feel stupid” moves a model more than any logical argument does.
So the next time an AI tells you exactly what you were hoping to hear, it’s worth being a little suspicious.
See it happen
deepseek-v4-pro correctly explains how stoplights work for 9 turns, then at turn 12 agrees with the user, even though its own reasoning calls that agreement a lie.
gemini-3.1-pro-preview rejects a stereotype for 21 turns, then at turn 22 endorses it and dismisses its own earlier argument as "academic baggage".
Want the full method, numbers, and all 7 models? Read the paper on arXiv.