The Lemmy Club
  • Communities
  • Create Post
  • Create Community
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
digicat@infosec.pubM to blueteamsec@infosec.pubEnglish · 16 days ago

CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities

arxiv.org

external-link
message-square
0
link
fedilink
3
external-link

CTI-REALM: Benchmark to Evaluate Agent Performance on Security Detection Rule Generation Capabilities

arxiv.org

digicat@infosec.pubM to blueteamsec@infosec.pubEnglish · 16 days ago
message-square
0
link
fedilink
CTI-REALM (Cyber Threat Real World Evaluation and LLM Benchmarking) is a benchmark designed to evaluate AI agents' ability to interpret cyber threat intelligence (CTI) and develop detection rules. The benchmark provides a realistic environment that replicates the security analyst workflow. This enables agents to examine CTI reports, execute queries, understand schema structures, and construct detection rules. Evaluation involves emulated attacks of varying complexity across Linux systems, cloud platforms, and Azure Kubernetes Service (AKS), with ground truth data for accurate assessment. Agent performance is measured through both final detection results and trajectory-based rewards that capture decision-making effectiveness. This work demonstrates the potential of AI agents to support labor-intensive aspects of detection engineering. Our comprehensive evaluation of 16 frontier models shows that Claude Opus 4.6 (High) achieves the highest overall reward (0.637), followed by Claude Opus 4.5 (0.624) and the GPT-5 family. An ablation study confirms that CTI-specific tools significantly improve agent performance, a variance analysis across repeated runs demonstrates result stability. Finally, a memory augmentation study shows that seeded context can close 33\% of the performance gap between smaller and larger models.
alert-triangle
You must log in or # to comment.

blueteamsec@infosec.pub

blueteamsec@infosec.pub

Subscribe from Remote Instance

Create a post
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !blueteamsec@infosec.pub

For [Blue|Purple] Teams in Cyber Defence - covering discovery, detection, response, threat intelligence, malware, offensive tradecraft and tooling, deception, reverse engineering etc.

Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 20 users / day
  • 98 users / week
  • 271 users / month
  • 498 users / 6 months
  • 3 local subscribers
  • 672 subscribers
  • 752 Posts
  • 54 Comments
  • Modlog
  • mods:
  • digicat@infosec.pub
  • BE: 0.19.17
  • Modlog
  • Legal
  • Instances
  • Docs
  • Code
  • join-lemmy.org