Project
About project
About project
About project
Modern coding LLMs achieve impressive scores on public benchmarks, but these results do not always reflect their ability to solve real-world software engineering tasks. Many existing benchmarks are limited to a single programming language, contain noisy annotations, or overlap with models' training data.
Our goal was to create an independent multilingual benchmark that enables fair evaluation of AI coding agents on realistic software engineering tasks, including bug fixing, API modifications, and improvements to existing codebases.
SOLUTION
SOLUTION
SOLUTION
We developed an extended version of SWE-Bench Verified with support for multiple programming languages and significantly stricter data quality standards.
The dataset was built from issue–merge request pairs collected from both public and private repositories. To identify high-quality tasks, we combined automated filtering with expert review. Each candidate was evaluated based on the size and scope of the code changes, the consistency between the issue description and the implemented fix, the quality of the accompanying test suite, and the complexity of the task.
The final benchmark includes tasks in C#, C++, Go, Java, JavaScript, Kotlin, PHP, Ruby, Rust, Scala, and TypeScript. For each task, we also prepared the required execution environment and supporting files for automated task reproduction and evaluation of AI-agent solutions.
RESULTS
RESULTS
RESULTS
We assembled one of the largest multilingual datasets for evaluating coding LLMs. More than 7,500 issue–merge request pairs were collected and annotated, with 1,500 tasks meeting our strict quality criteria and included in the final benchmark.
The benchmark enables objective evaluation of AI coding agents in scenarios that closely resemble real-world software development. Testing leading open-source and commercial models demonstrated that even state-of-the-art LLMs still struggle with complex, multi-file modifications in existing codebases, highlighting the importance of high-quality SWE benchmarks for advancing AI-assisted software engineering.
PROCESS AND TECHNOLOGIES
PROCESS AND TECHNOLOGIES
PROCESS AND TECHNOLOGIES
We designed a multi-stage pipeline for dataset collection, filtering, and validation.
The process began by collecting issue–merge request pairs from a diverse range of repositories. Each candidate was then evaluated to ensure consistency between the reported issue and the corresponding fix, while also assessing patch quality, test coverage, and task complexity.
To support reproducible evaluation, we prepared the complete execution infrastructure for every task, including environment configurations, supporting files, and automated validation workflows.
Model evaluation was performed using Harbor, the agent framework developed by the creators of Terminal Bench, together with the Multi-SWE-Bench evaluation framework. These tools automatically applied model-generated patches to repositories, executed the accompanying test suites, and evaluated the generated solutions.
