Our Multi-Agent System Got 9 More Cases Right, but We Weren't Satisfied

Preface

Wiggins began as a multi-agent AI detective built for a hackathon. We used Qwen Cloud to power five specialist agents, LangGraph to orchestrate investigation, verification, and synthesis, and an Evidence Board to stream every reasoning step in real time. Once the product was complete, we did not want to stop at “multi-agent looks impressive.” We wanted to know how much measurable value it actually delivered. So we built an evaluation pipeline to separate the contributions of accuracy, reasoning rules, and architecture. This article follows Wiggins as it moved from a hackathon project to a verifiable reasoning system.

Wiggins solves MuSR murder mysteries, each with two suspects. The system splits the reasoning among three specialist agents—motive, means, and opportunity—then adds a verification step and a final verdict. It reached 94.6% on the clean test set, beating both the single-call baseline and a much larger model.

But we did not stop at that score. We asked a question that few people measure: how much of this advantage came from the multi-agent architecture, and how much came from the reasoning rules we gave it? This is the story of how we separated the two—and what we discovered about when multi-agent systems are truly worth it.

Step 1: Strip the Architecture Down to Three Useful Mechanisms

The previous version, code-named find-the-gap, was packed with mechanisms: cross-dimension consistency checks with automatic reinvestigation, verbatim quote verification, a counter-evidence channel, and a patch that treated “silence” as a reasoning gap. After examining the cases one by one, we found that most of these mechanisms caused net harm or added pure noise. Forty-six of 50 reinvestigations went nowhere. Quote verification disrupted decisions that had originally been correct. The counter-evidence channel was even used by the model to force connections for the wrong suspect.

So we removed almost all of it, returned to the original architecture, and kept only three mechanisms that survived scrutiny:

• Three-tier strength labels: Each dimension is no longer a binary yes/no judgment. Instead, it is labeled strong (explicitly stated), weak (reasonably inferred), or none (completely unsupported). This gives the system a way to find the decoy suspect who is missing exactly one dimension.
• Loud-decoy rule: Dramatic but crime-irrelevant traits—aliases, strange hobbies, suspicious habits—do not count as evidence for any dimension. The real culprit must have all three dimensions; a none on any one of them rules a suspect out.
• Temporal discipline: The system separates “present at the time of the murder” from “present only because they were brought in for questioning afterward.” The latter does not count as opportunity.

Compared with the original architecture, we changed only three files. Everything else was restored.

Step 2: The Numbers

We first selected 20 difficult cases with a history of errors. To keep the comparison fair, we reran the original architecture and the single-call baseline on the same day, July 11, 2026, so that normal day-to-day model drift would not be mistaken for a real difference.

The multi-agent system did not just beat the single-call baseline. It also surpassed the larger reference model.

Step 3: Run the Full Set and Make Sure It Wasn't Luck

The hard subset contains only 20 cases, and its credibility comes from the same-day comparison above. We then ran the full clean test set of 92 cases to see the broader picture. Our version was evaluated on July 11, 2026, while the other versions use their historical results, with dates shown in the table. The same-day comparison in the previous step addresses the model-drift question.

The multi-agent system solved 87 cases, while the single-call baseline solved 78—9 more correct answers. In accuracy terms, that is 94.6% versus 84.8%, a gap of 9.8 percentage points. It also preserved every easy case that all five historical versions had answered correctly.

We also removed the few borderline cases where even the gold answer was debatable. The conclusion held on this stricter 90-case set:

The multi-agent system solved 9 more cases than the single-agent baseline, a gap of 10 percentage points.

Step 4: Where Did Those 9 Extra Correct Answers Come From?

Up to this point, the single-call baseline used a very simple prompt, while the three mechanisms—strength labels, the loud-decoy rule, and temporal discipline—were given only to the multi-agent system. To determine what drove those 9 extra correct answers, we first had to put both systems on equal footing.

So we created a fair version: we added the exact same three ideas to the single-call prompt. Both systems now had identical reasoning knowledge. The only remaining difference was architectural: whether the work was split into independent nodes and whether an independent verification step existed.

We retested the same 20 difficult cases that had historically produced errors:

• Naive single call: 10 / 20
• Single call with the same rules: 15 / 20
• Multi-agent system: 16 / 20

The result was clear. Adding the three rules alone raised the single-call baseline from 10 to 15, closing most of the gap. The architecture itself finished one case ahead.

Most of the nine-case advantage came from the reasoning rules, not the orchestration. But here is the key: we discovered those three rules by building the multi-agent system and diagnosing failures one dimension at a time. Splitting the reasoning into independent motive, means, and opportunity agents showed us exactly which dimension and which step had failed. That observability is what uncovered the effective rules. The architecture was our microscope.

Step 5: So What Did Multi-Agent Give Us?

If most of the accuracy gain came from the rules, what did the multi-agent architecture itself contribute to this task?

The answer is an auditable reasoning process. A single call with structured output can produce the same dimension-level fields, so readable output is not unique to multi-agent systems. The real difference is between a guarantee and a request. A single call can be asked to output separate dimensions, but it cannot guarantee that the reasoning was actually independent; those fields may simply be a reorganization of one entangled judgment. Multi-agent architecture turns “independent reasoning for each dimension, followed by independent verification” into a structural guarantee. Every step can be inspected on its own.

Our conclusion for this task is:

The multi-agent architecture did not produce a measurable accuracy advantage on this task. What it provided was an auditable, inspectable reasoning process, while the measurable gain came from the reasoning rules themselves.

Our Finding: The Value of Multi-Agent Systems Lies in Complexity

The architecture did not show an accuracy advantage here because the state space was too small. The entire case contains only six cells, and one call can hold all of them at once. Decomposition is unnecessary when everything fits inside one model's working context. Multi-agent specialization and verification become valuable when the problem no longer fits inside “one brain.”

Our view is that the value of multi-agent systems is governed by a task-complexity threshold. Push beyond that threshold—with more suspects and more conflicting evidence—and a single call will begin to lose track, allowing the advantage of decomposition to emerge.

Kim and Schuster (2023) found that what strains LLM entity tracking is the accumulation of state-changing operations. As updates accumulate, accuracy declines. Adding more suspects and mixing in conflicting clues creates exactly this kind of pressure.

Tang et al. (2025), from the opposite direction, use a depth × width framework to show that the advantage of multi-agent systems over single-agent systems grows with task complexity. A contemporary preprint by Li (2026) presents a close parallel to our result: a single agent equipped with skills can replace a multi-agent system while the selection problem remains below its capacity threshold, but fails once that threshold is crossed.

MuSR murder mysteries are fixed at two suspects and three dimensions (Sprague et al., 2024), placing our task near the shallow end of this curve. But “shallow” does not mean “easy.” Even the strongest GPT-4 configuration reached only 86%, below the human score of 94.1%. The state space is simply compact enough to fit within a single call.

Future Directions

Push the task beyond the threshold. The hypothesis that multi-agent value emerges with complexity has not yet been tested directly. Our next step is to generate harder MuSR-style cases with five or more suspects and more conflicting clues, then observe when a single call begins to lose track and when decomposition starts to show an advantage.

Test again with a weaker model. Complexity comes not only from the task, but also from model capability. With a smaller model, a single call may reach its limit at a lower level of task complexity, allowing the value of the architecture to emerge earlier. This is the second axis of the complexity threshold: how difficult the task is, and how capable the model is.

References

• Kim, N., & Schuster, S. (2023). Entity Tracking in Language Models. arXiv:2305.02363. https://arxiv.org/abs/2305.02363
• Tang, B., Liang, H., Jiang, K., & Dong, X. (2025). On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systems. arXiv:2510.04311. https://arxiv.org/abs/2510.04311
• Li, X. (2026). When Single-Agent with Skills Replace Multi-Agent Systems and When They Fail (preprint). arXiv:2601.04748. https://arxiv.org/abs/2601.04748
• Sprague, Z., Ye, X., Bostrom, K., Chaudhuri, S., & Durrett, G. (2024). MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning. ICLR 2024 (Spotlight). arXiv:2310.16049. https://arxiv.org/abs/2310.16049