On the public citation graph

Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks

https://doi.org/10.1145/3786771
0 signed · 50 inferred · view as graph

Inferred edges come from two machine sources: sweeps of open repository metadata (mostly SSRN and Zenodo), and a bulk OpenAlex citation snapshot. Every row below is labelled with the one it came from. Neither is a complete citation index — absence here is not evidence of absence.

Is this your work? Sign in with ORCID and put your name on the claims that are yours.
Claim with ORCID
Reference health. See whether every reference in this work resolves to a known work and none has been retracted — and get a dated CiteStamped badge you can embed.
See reference health

What this work says

Outgoing claims: the papers this work cites, builds on, or argues with.

inferredcites →https://doi.org/10.1007/978-3-642-12578-2openalex-snapshot
Requirements Engineering
inferredcites →https://doi.org/10.1007/s10515-024-00452-xopenalex-snapshot
Automated requirement contradiction detection through formal logic and LLMs
inferredcites →https://doi.org/10.1007/s10664-021-10083-5openalex-snapshot
A fine-grained data set and analysis of tangling in bug fixing commits
inferredcites →https://doi.org/10.1007/s10664-023-10372-1openalex-snapshot
A syntax-guided multi-task learning approach for Turducken-style code generation
inferredcites →https://doi.org/10.1016/j.cose.2024.103971openalex-snapshot
Detecting command injection vulnerabilities in Linux-based embedded firmware with LLM-based taint analysis of library functions
inferredcites →https://doi.org/10.1016/j.infsof.2010.12.010openalex-snapshot
Identifying relevant studies in software engineering
inferredcites →https://doi.org/10.1016/j.infsof.2020.106312openalex-snapshot
Multiple fault localization of software programs: A systematic literature review
inferredcites →https://doi.org/10.1016/j.infsof.2021.106809openalex-snapshot
VUDENC: Vulnerability Detection with Deep Learning on a Natural Codebase for Python
inferredcites →https://doi.org/10.1016/j.ipm.2024.103809openalex-snapshot
Are LLMs good at structured outputs? A benchmark for evaluating structured output capabilities in LLMs
inferredcites →https://doi.org/10.1016/j.jss.2022.111423openalex-snapshot
A systematic literature review on benchmarks for evaluating debugging approaches
inferredcites →https://doi.org/10.1016/j.jss.2022.111537openalex-snapshot
On the use of deep learning in software defect prediction
inferredcites →https://doi.org/10.1016/j.jss.2023.111947openalex-snapshot
Log statements generation via deep learning: Widening the support provided to developers
inferredcites →https://doi.org/10.1016/j.jss.2024.112149openalex-snapshot
Project-specific code summarization with in-context learning
inferredcites →https://doi.org/10.1016/j.scico.2023.102994openalex-snapshot
TABASCO: A transformer based contextualization toolkit
inferredcites →https://doi.org/10.1109/aitest62860.2024.00022openalex-snapshot
User Centric Evaluation of Code Generation Tools (Invited Paper)
inferredcites →https://doi.org/10.1109/llm4code66737.2025.00024openalex-snapshot
Code Summarization Beyond Function Level
inferredcites →https://doi.org/10.1109/msr66628.2025.00043openalex-snapshot
Too Noisy To Learn: Enhancing Data Quality for Code Review Comment Generation
inferredcites →https://doi.org/10.1109/oncon62778.2024.10931625openalex-snapshot
Exploring LLMs for Verifying Technical System Specifications Against Requirements
inferredcites →https://doi.org/10.1109/saner60148.2024.00068openalex-snapshot
SparseCoder: Identifier-Aware Sparse Transformer for File- Level Code Summarization
inferredcites →https://doi.org/10.1109/tse.2015.2454513openalex-snapshot
The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs
inferredcites →https://doi.org/10.1109/tse.2017.2663435openalex-snapshot
Automated Test Case Generation as a Many-Objective Optimisation Problem with Dynamic Selection of the Targets
inferredcites →https://doi.org/10.1109/tse.2021.3087402openalex-snapshot
Deep Learning Based Vulnerability Detection: Are We There Yet?
inferredcites →https://doi.org/10.1109/tse.2022.3147265openalex-snapshot
Neural Transfer Learning for Repairing Security Vulnerabilities in C Code
inferredcites →https://doi.org/10.1109/tse.2022.3171202openalex-snapshot
Data Preparation for Software Vulnerability Prediction: A Systematic Literature Review
inferredcites →https://doi.org/10.1109/tse.2023.3267446openalex-snapshot
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
inferredcites →https://doi.org/10.1109/tse.2024.3368208openalex-snapshot
Software Testing With Large Language Models: Survey, Landscape, and Vision
inferredcites →https://doi.org/10.1109/tse.2025.3541166openalex-snapshot
Automated Test Case Repair Using Language Models
inferredcites →https://doi.org/10.1145/3428334openalex-snapshot
Magma
inferredcites →https://doi.org/10.1145/3502853openalex-snapshot
Correlating Automated and Human Evaluation of Code Documentation Generation Quality
inferredcites →https://doi.org/10.1145/3643681openalex-snapshot
VeriGen: A Large Language Model for Verilog Code Generation
inferredcites →https://doi.org/10.1145/3664597openalex-snapshot
Deep Learning for Code Intelligence: Survey, Benchmark and Toolkit
inferredcites →https://doi.org/10.1145/3695988openalex-snapshot
Large Language Models for Software Engineering: A Systematic Literature Review
inferredcites →https://doi.org/10.1145/3695991openalex-snapshot
CodeScore: Evaluating Code Generation by Learning Code Execution
inferredcites →https://doi.org/10.1186/s13638-023-02242-7openalex-snapshot
VulRep: vulnerability repair based on inducing commits and fixing commits
inferredcites →https://doi.org/10.18653/v1/2024.findings-acl.471openalex-snapshot
NaturalCodeBench: Examining Coding Performance Mismatch on HumanEval and Natural User Queries
inferredcites →https://doi.org/10.18653/v1/d18-1425openalex-snapshot
Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
inferredcites →https://doi.org/10.48550/arxiv.2310.16390openalex-snapshot
Evaluating Pre-trained Language Models for Repairing API Misuses
inferredcites →https://doi.org/10.48550/arxiv.2404.01558openalex-snapshot
Automated User Story Generation with Test Case Specification Using Large Language Model
inferredcites →https://doi.org/10.48550/arxiv.2504.20964openalex-snapshot
OSVBench: Benchmarking LLMs on Specification Generation Tasks for Operating System Verification

What points at this work

Incoming claims: where this work appears in other papers’ reference lists and arguments.

inferred← citeshttps://doi.org/10.1007/978-3-032-22375-3_16openalex-snapshot
Adoption of Large Language Models in Scrum Management: Insights from Brazilian Practitioners
inferred← citeshttps://doi.org/10.1016/j.cie.2026.112333openalex-snapshot
Engineering Reasoning and Instruction (ERI) benchmark: a large taxonomy-driven dataset for foundation models and agents
inferred← citeshttps://doi.org/10.1109/ase63991.2025.00161openalex-snapshot
Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning
inferred← citeshttps://doi.org/10.1109/cascon66301.2025.00128openalex-snapshot
A Fully Automated Agent for End-to-End Code Translation and Validation
inferred← citeshttps://doi.org/10.1109/esem64174.2025.00058openalex-snapshot
A Fully Automated Agent for End-to-End Code Translation and Validation
inferred← citeshttps://doi.org/10.1109/iicai70155.2026.11621026openalex-snapshot
Analyzing Stack Overflow Question Types and Answer Characteristics: Implications for Designing and Benchmarking Coding Assistant LLMs
inferred← citeshttps://doi.org/10.1145/3777730.3777793openalex-snapshot
Cross-Cultural Translation Style Recognition and Consistency Evaluation Model for Tourism Brands Using BERT+XGBoost
inferred← citeshttps://doi.org/10.1145/3832175openalex-snapshot
Hidden Licensing Risks in the PTMware Ecosystem
inferred← citeshttps://doi.org/10.1145/3832783.3834411openalex-snapshot
RealAIGC: Towards More Realistic Evaluation of AI-Generated Code Detection in Real-World Code Repository
inferred← citeshttps://doi.org/10.3390/systems14060598openalex-snapshot
A Methodology for the Systematic Evaluation of LLMs in Requirement Engineering Tasks with No Ground Truth
inferred← citeshttps://doi.org/10.4271/2026-01-0110openalex-snapshot
Culture Clashes: LLM Support in the Engineering of Safety-Critical Systems
Editorial review for independent work. ICSAC — the Institute for Complexity Science and Advanced Computing — runs a structured editorial panel for researchers publishing outside journals, and publishes on CiteStamp rails. Submission is free and goes through ORCID: icsacinstitute.org/submit.