New Academic Benchmarks Highlight LLMs' Exploit Development Capabilities

Researchers Newton Cheng, Keane Lucas, Winnie Xiao, Nicholas Carlini, and Milad Nasr have assessed the exploit development capabilities of their model, Mythos Preview, through new benchmarks: ExploitBench and ExploitGym. These benchmarks enable a detailed evaluation of LLMs in creating end-to-end exploits rather than merely proof-of-concept demonstrations. Mythos Preview has shown superior performance across all three benchmarks, suggesting that the expertise needed to develop exploits may significantly diminish as these advanced capabilities become more accessible. ExploitBench focuses on the full exploit development process, decomposing it into 16 capabilities for granular analysis, which could reshape the understanding of vulnerabilities in software.