01 / relevance
Relevance classification
Identify whether a sentence contains actionable cyber threat intelligence or background / non-actionable material.
binary classificationEMNLP 2026 Findings · Dataset & Benchmark
A unified benchmark for sentence-level cyber threat intelligence extraction.
SMAH-CTI is a human-annotated benchmark built from 100 public CTI reports, unifying relevance classification, CTI-specific named entity recognition, MITRE ATT&CK mapping, and grounded procedure descriptions in one sentence-level resource.
The benchmark
One resource, four tightly connected CTI tasks.
Each instance is anchored to a sentence from a source report, with immediately preceding context where available. Relevant sentences are annotated across complementary semantic layers so models can be evaluated independently or as a multi-task pipeline.
100
public CTI reports
12,304
sentence instances
4,302
relevant sentences
15,487
entity annotations
100%
validated NER spans
02 / entities
CTI named entities
Extract Action, Infrastructure Indicator, Malware/Tool, and Threat Actor spans with exact character offsets.
span-level NER03 / ATT&CK
Tactic & technique mapping
Assign one or more MITRE ATT&CK tactics and techniques/sub-techniques to sentences that express adversarial behavior.
multi-label mapping04 / procedures
Grounded procedures
Generate a concise, annotator-written procedure description grounded in the specific adversarial action expressed by the sentence.
grounded generationAnnotation anatomy
What one annotated sentence looks like.
The entity offsets are zero-based and end-exclusive relative to sentence_text. Every one of the 15,487 released spans was validated against the exact source substring.
Example · aptnotes
According to Mandiant, the compromise conducted by North Korean state-sponsored threat actors started when someone at 3CX downloaded an installer for the X_TRADER futures trading platform from Trading Technologies.
TACTIC TA0001 · Initial Access
TECHNIQUE T1195 · Supply Chain Compromise
Procedure: Download trojanized installer from a trusted vendor as part of a supply chain compromise.
ThreatActor · 1,842
Threat groups, actor names, or individuals associated with malicious activity.
Action · 6,546
Adversarial or security-relevant actions expressed in the sentence.
Infrastructure_Indicator · 4,404
Infrastructure, indicators, paths, domains, IPs, and other technical artifacts.
MalwareTool · 2,695
Malware families, tools, utilities, and exploit kits.
Data composition
Frozen splits and diverse public CTI sources.
The released partition is report-level: entire reports belong to exactly one of Train, Dev, or Test. No report identifier or sentence UID overlaps across the three splits.
Source collections
Sentence instances from five public CTI collections; the bars show the share of all 12,304 instances.
Frozen benchmark split
Report-level partition used by the associated paper.
| Split | Reports | Sentences | Relevant |
|---|---|---|---|
| Train | 70 | 8,635 | 3,006 |
| Dev | 10 | 1,238 | 434 |
| Test | 20 | 2,431 | 862 |
| Total | 100 | 12,304 | 4,302 |
138 relevant sentences intentionally contain tactic-level annotations without a specific technique where the evidence did not support a confident technique assignment.
Dataset explorer
Inspect the release in the browser.
The explorer is bundled directly into this page, so it works even when index.html is opened as a local file. Search sentence text and filter by source, relevance, and entity type. Nothing is sent to a server.
SMAH-CTI browser
12,304 records are bundled with this page
Open the explorer to begin browsing.
Release resources
Everything needed to reproduce the benchmark.
The public release includes the dataset itself, explicit report splits, a JSON Schema, provenance records, integrity validation, licenses, and the paper.
JSON · DATA
Main dataset
12,304 annotated sentence instances in a single JSON release.
JSON SCHEMADataset schema
Machine-readable field definitions, allowed values, and NER offset contract.
JSON · SPLITSFrozen report split
Exact Train, Dev, and Test report IDs used for benchmark evaluation.
XLSX · PROVENANCESource provenance
Report-level acquisition details, original source URLs, and access dates.
JSON · QAValidation summary
Integrity checks, UID uniqueness, distribution counts, and span validation.
PDF · PAPERAssociated paper
Methods, annotation design, benchmark formulation, and experimental results.
Citation
Use SMAH-CTI in your research.
If you use the benchmark, please cite the associated EMNLP 2026 Findings paper. The repository also includes CITATION.cff for machine-readable citation metadata.
@inproceedings{jaffal-etal-2026-smah-cti,
title = {{SMAH-CTI}: A Unified Sentence-Level Benchmark for Multi-Task Cyber Threat Intelligence Extraction},
author = {Jaffal, Niveen O. and Jarrar, Radi and Alkhanafseh, Mohammed and Mohaisen, David},
booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2026},
year = {2026}
}
License: dataset and associated data files are CC BY-NC 4.0; software/source code is MIT. See LICENSE-DATA and LICENSE-CODE for the applicable terms.