Researchers just tested six major language models on a critical security task: piecing together how attackers moved through a system using fragmented logs and telemetry. The results are sobering. The best-performing models only got 39.6% of individual steps right across 849 reference actions.
The new benchmark, called DiagChain, was built by a team including Xuyang Liu, Yibin Han, and others specifically to diagnose where AI agents break down during incident response. Most existing tests just check the final answer right or wrong. DiagChain instead tracks each step of reasoning, revealing exactly how and why reconstructions fail.
Where AI stumbles on attack chains
The MAIN-69 test suite spans 69 real-world scenarios across different operating systems, noise levels in evidence, and chain complexity. Smaller models couldn't effectively use retrieved evidence at all. Larger ones had the opposite problem: they grabbed the right pieces of information but arranged them in the wrong order.
That distinction matters enormously for security teams. A model that misses evidence entirely is obviously unreliable. But one that finds the clues then botches the timeline might look promising at first glance, luring teams into false confidence.
DiagChain also introduced ECRAG, a method pairing evidence retrieval with an evolving structured representation of the chain being rebuilt. The approach helped, but nowhere near enough to make current AI agents ready for solo incident triage work.
The gap between what researchers hoped for and what's actually possible remains wide. Until these models clear much higher accuracy thresholds, security teams will need human analysts in the loop to verify any AI-assisted attack reconstruction.
This article is informational only and does not constitute security advice or a recommendation to deploy AI tools in your incident response workflow without proper validation.
