Verifiable ground truth for social reasoning in LLM assistants
Google Research's Fuse framework gives social-reasoning evals an actual verifiable answer by hiding a motive inside a multi-agent simulation, then uses it to show assistants are swayed by biased framing and don't reliably get better with longer conversations.
When people ask an assistant for advice about a coworker, a partner, or a friend, they're really asking it to reason about someone it has never observed directly. All it has is the user's account, which is partial, motivated, and sometimes wrong. Evaluating whether the assistant's inference about that third party is any good has been surprisingly hard, because there's usually no ground truth to check it against — just another human rater's opinion about what the "right" read of the situation would have been.\n\nA new paper out of Google Research, Verifiable Social Reasoning for LLM Assistants (Taubenfeld, Gekhman, et al.), fixes that by manufacturing the ground truth rather than judging it. The framework is called Fuse, and the trick is simple to state: put a hidden motive inside a simulation, then see if the assistant can recover it.\n\n## How Fuse manufactures ground truth\n\nFuse runs a multi-agent simulation with a target agent that has been assigned a specific, hidden motive — something like resentment over being passed up for a promotion, or an ulterior reason for skipping a friend's event. That target interacts with other simulated agents, one of which plays the role of "the user." The user agent then goes to the system under test — the actual LLM assistant being evaluated — and asks for help making sense of the target's behavior, the same way a real person would vent about a coworker to a chatbot.\n\nBecause the motive was assigned by the simulation rather than inferred by a rater, the researchers know exactly what the correct answer is. Grading the assistant's inference becomes a matter of comparing it to a fact, not adjudicating a matter of opinion. That's the whole contribution in one line: verifiable ground truth by construction, for a class of eval that previously had none.
The setup also builds in the actual difficulty of the task, which is that the assistant never sees the target agent directly — it only sees the user's narrative about the target, which the user agent produces as it would in real life: selectively, and colored by its own perspective. The authors didn't just take this on faith; they validated that the simulated interactions read as faithful to how people actually behave with a human study spanning 24,000 annotations.
What twelve models got wrong
With a graded eval in hand, the authors ran 12 LLMs through Fuse and used it to isolate specific failure modes rather than just producing a leaderboard. Four findings stand out.
User mediation makes an already-hard problem harder. Social reasoning is difficult on its own; routing it through a subjective, motivated narrator compounds the difficulty rather than just adding noise. The assistant isn't only inferring a person's motive — it's also implicitly correcting for the user's framing of that person, and most models don't do this well.
Models are systematically swayed by biased framing. When the user agent presents a slanted account of the target's behavior, the assistant's inference shifts toward that framing rather than reasoning past it. This is the finding I'd flag first for anyone building a consultation-style assistant: it means the model is picking up the user's spin as evidence, not just as context to be discounted. A user who's already decided their coworker is being malicious can nudge the assistant into confirming that read, even when the underlying facts don't support it.
Models often need more detail than a human would to reach the same correct answer. Humans can frequently identify a plausible motive from a sparse account; several of the evaluated LLMs needed a more fully fleshed-out narrative to land on the same conclusion. That's a meaningful gap for real deployments, where users rarely provide complete information up front.
Longer conversations don't reliably help. This is the counterintuitive one. You'd expect that giving the assistant more turns — more chances to ask clarifying questions — would only improve its inference, since it strictly increases the information available. The paper shows this isn't always true: performance doesn't monotonically improve with conversation length, despite the clarifying-question opportunity being there for the taking. Either the models aren't asking the questions that would actually resolve the ambiguity, or the extra turns introduce noise that offsets whatever signal they add.
Why this matters beyond the benchmark
I'd read this less as a paper about a new benchmark number and more as a diagnostic of a specific failure surface in consultation-style assistants — the "help me understand this person" use case that's already one of the more common ways people use chat products. The framing-sensitivity result in particular is the kind of thing that's easy to miss in normal eval suites, because those typically test whether a model gets a fact right, not whether it can resist an implicit persuasion attempt embedded in how a user tells a story.
The methodological move is worth remembering independently of the specific findings: when you can't get ground truth from human judgment, look for a way to construct it instead. Fuse does that by moving the ground truth upstream of the interaction — assigning it before the simulation runs — rather than trying to adjudicate it after the fact. That's a pattern applicable well beyond social reasoning, anywhere a system's target output depends on inferring something that's normally unobservable. The authors have open-sourced Fuse along with a 21k-example dataset, which should make it straightforward for other teams to run their own assistants through the same test and see where the same failure modes show up.