On the public citation graph

Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering

0 signed · 69 inferred · view as graph

Inferred edges come from two machine sources: sweeps of open repository metadata (mostly SSRN and Zenodo), and a bulk OpenAlex citation snapshot. Every row below is labelled with the one it came from. Neither is a complete citation index — absence here is not evidence of absence.

Is this your work? Sign in with ORCID and put your name on the claims that are yours.
Claim with ORCID
Reference health. See whether every reference in this work resolves to a known work and none has been retracted — and get a dated CiteStamped badge you can embed.
See reference health

What this work says

Outgoing claims: the papers this work cites, builds on, or argues with.

inferredcites →https://doi.org/10.1145/3600006.3613165openalex-snapshot
Efficient Memory Management for Large Language Model Serving with PagedAttention
inferredcites →https://doi.org/10.1145/3728963openalex-snapshot
Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering
inferredcites →https://doi.org/10.1162/coli_a_00561openalex-snapshot
LLM-based NLG Evaluation: Current Status and Challenges
inferredcites →https://doi.org/10.18653/v1/2020.acl-main.703openalex-snapshot
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension
inferredcites →https://doi.org/10.18653/v1/2022.acl-long.499openalex-snapshot
UniXcoder: Unified Cross-Modal Pre-training for Code Representation
inferredcites →https://doi.org/10.18653/v1/d19-1053openalex-snapshot
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance
inferredcites →https://doi.org/10.3115/1613715openalex-snapshot
Proceedings of the Conference on Empirical Methods in Natural Language Processing - EMNLP '08

What points at this work

Incoming claims: where this work appears in other papers’ reference lists and arguments.

inferred← citeshttps://doi.org/10.1007/978-3-032-11733-5_26openalex-snapshot
RAG Evaluation: From Model-Centric Benchmarks to System-Level Metrics
inferred← citeshttps://doi.org/10.1007/978-3-032-28117-3_15openalex-snapshot
Proxy Analysis in Bias Testing
inferred← citeshttps://doi.org/10.1007/978-981-92-0068-9_12openalex-snapshot
How Smart is Smart Enough? Benchmarking LLMs with Embedding-Based Similarity in Python Code Generation
inferred← citeshttps://doi.org/10.1007/s10664-025-10792-1openalex-snapshot
XMutant: XAI-based fuzzing for deep learning systems
inferred← citeshttps://doi.org/10.1007/s10664-026-10919-yopenalex-snapshot
Evaluating foundation model integration strategies for detecting PII in java software engineering pipelines
inferred← citeshttps://doi.org/10.1007/s11192-026-05557-6openalex-snapshot
Co-Reviewer: can AI review like a human? An agentic framework for LLM-human alignment in peer review
inferred← citeshttps://doi.org/10.1007/s11219-026-09785-0openalex-snapshot
AI-as-a-Judge for software quality assessment: Consistency, attitude effects, and complexity sensitivity in multi-agent code evaluation
inferred← citeshttps://doi.org/10.1007/s12525-026-00890-3openalex-snapshot
LLM-as-a-Judge for mental support: A meta-evaluation using domain-specific platform data
inferred← citeshttps://doi.org/10.1007/s42979-026-05242-8openalex-snapshot
A Multi-Agent Benchmarking Framework for Evaluating the Performance of Large Language Models in Logic Programming
inferred← citeshttps://doi.org/10.1016/j.infsof.2025.107935openalex-snapshot
Usage patterns of software product metrics in assessing developers’ output: A comprehensive study
inferred← citeshttps://doi.org/10.1016/j.jss.2026.112852openalex-snapshot
Model-agnostic quality assessment for LLM-generated code via dynamic internal representation selection
inferred← citeshttps://doi.org/10.1038/s41598-025-27846-5openalex-snapshot
Measuring and mitigating debugging effectiveness decay in code language models
inferred← citeshttps://doi.org/10.1109/access.2025.3647111openalex-snapshot
GenScore: Agent-Based Short-Answer Question Generation and Scoring in Software Engineering Courses
inferred← citeshttps://doi.org/10.1109/acdsa67686.2026.11468166openalex-snapshot
Large Language Models as MT Evaluators: Statistical Evidence from a Low-Resource Setting
inferred← citeshttps://doi.org/10.1109/apsec66846.2025.00088openalex-snapshot
Towards Function-Level Correctness Assessment of System Software with LLMs: A Case Study
inferred← citeshttps://doi.org/10.1109/ase63991.2025.00020openalex-snapshot
LongCodeZip: Compress Long Context for Code Language Models
inferred← citeshttps://doi.org/10.1109/ase63991.2025.00046openalex-snapshot
Comprehend, Imitate, and then Update: Unleashing the Power of LLMs in Test Suite Evolution
inferred← citeshttps://doi.org/10.1109/ase63991.2025.00150openalex-snapshot
Multi-dimensional Assessment of Crowdsourced Testing Reports via LLMs
inferred← citeshttps://doi.org/10.1109/ase63991.2025.00200openalex-snapshot
An Agent-based Evaluation Framework for Complex Code Generation
inferred← citeshttps://doi.org/10.1109/ase63991.2025.00214openalex-snapshot
SE-Jury: An LLM-as-Ensemble-Judge Metric for Narrowing the Gap with Human Evaluation in SE
inferred← citeshttps://doi.org/10.1109/codit66093.2025.11321666openalex-snapshot
Evaluating Generative AI Models for Code Generation Tasks Using Embedding-Based Semantic Similarity
inferred← citeshttps://doi.org/10.1109/gaclm67198.2025.11232128openalex-snapshot
Querying Large Automotive Software Models: Agentic vs. Direct LLM Approaches
inferred← citeshttps://doi.org/10.1109/iaict71158.2026.11620949openalex-snapshot
Framework-Adaptive Agentic Pipelines for API Knowledge Extraction and Test Generation
inferred← citeshttps://doi.org/10.1109/mic.2025.3548143openalex-snapshot
User Feedback-Driven Generative Models-Based Methodology for Service Construction
inferred← citeshttps://doi.org/10.1109/models67397.2025.00014openalex-snapshot
MCeT: Behavioral Model Correctness Evaluation using Large Language Models
inferred← citeshttps://doi.org/10.1109/siie68580.2025.11368402openalex-snapshot
LLM-Assisted Assessment in Biology Education: A Comparative Study of ChatGPT-4o, Gemini 2.5 Flash and Teacher Grading in High-School Exams
inferred← citeshttps://doi.org/10.1109/zinc69910.2026.11655803openalex-snapshot
Human-Verifiable API Documentation Generation for Legacy Web Frameworks Using Multi-Agent LLMs
inferred← citeshttps://doi.org/10.1145/3744916.3787761openalex-snapshot
Agent-Based Ensemble Reasoning for Repository-Level Issue Resolution
inferred← citeshttps://doi.org/10.1145/3748522.3780008openalex-snapshot
Requirement Bot: Enhancing Software User Requirement List Through Retrieval-Augmented Human–AI Co-Authoring
inferred← citeshttps://doi.org/10.1145/3772318.3790537openalex-snapshot
Peeking Ahead of the Field Study: Exploring VLM Personas as Support Tools for Embodied Studies in HCI
inferred← citeshttps://doi.org/10.1145/3772318.3791034openalex-snapshot
Open-ended Structured Question Assessment with Human-LLM Collaboration
inferred← citeshttps://doi.org/10.1145/3772318.3791428openalex-snapshot
Who Does What? Archetypes of Roles Assigned to LLMs During Human-AI Decision-Making
inferred← citeshttps://doi.org/10.1145/3785022.3785109openalex-snapshot
A Novel Approach to Evaluating the Effectiveness of Large Language Models for Multimodal Analysis of Embodied Learning in Classrooms
inferred← citeshttps://doi.org/10.1145/3786580.3786985openalex-snapshot
Scaling Assessment of Student Models with LLMs: Integrating Feedback into Practice
inferred← citeshttps://doi.org/10.1145/3786583.3786915openalex-snapshot
Enterprise-Scale COBOL-to-Java Translation: LLMs Augmented with Program Analysis
inferred← citeshttps://doi.org/10.1145/3794763.3794796openalex-snapshot
AutoLogger: A Multi-Agent Framework for the End-to-End Automated Logging
inferred← citeshttps://doi.org/10.1145/3797071openalex-snapshot
Mining Long Tail Bugs: Identifying Rare and Overlooked Issues in Code
inferred← citeshttps://doi.org/10.1145/3797146openalex-snapshot
Validating LLM-Generated SQL Queries through Metamorphic Prompting
inferred← citeshttps://doi.org/10.1145/3803437.3804877openalex-snapshot
Reliable and Developer-Aligned Evaluation of Agents for Software Engineering
inferred← citeshttps://doi.org/10.1145/3803437.3805266openalex-snapshot
Dependency-Guided Repository-Level C-to-Rust Translation with Reinforcement Alignment
inferred← citeshttps://doi.org/10.1145/3803846.3807463openalex-snapshot
High Agreement, Shallow Reasoning: A Mixed-Method Study of LLMs in Refactoring Reviews
inferred← citeshttps://doi.org/10.1145/3808102openalex-snapshot
Bash-Commenter: Leveraging Syntax-Aware Preference Optimization to Reinforce Large Language Model for Bash Code Comment Generation
inferred← citeshttps://doi.org/10.1145/3811825openalex-snapshot
FURINA: A Multi-Agent Framework for Automated Patch Correctness Assessment
inferred← citeshttps://doi.org/10.1145/3828168openalex-snapshot
More than a Judge: An Empirical Study of Agent-Human Interaction in Crowdsourced Testing Assessment
inferred← citeshttps://doi.org/10.1145/3832245openalex-snapshot
BinRAG: An RAG-Based Decompilation Framework Fusing Name Prediction and Calling Context
inferred← citeshttps://doi.org/10.1145/3832264openalex-snapshot
Paired Code Smells and Test Smells: A Fine-Grained Longitudinal Empirical Study
inferred← citeshttps://doi.org/10.1145/3832783.3837478openalex-snapshot
FAiLGen: Calibrating LLM for Failing Test Generation with Monte Carlo Tree Search
inferred← citeshttps://doi.org/10.1145/3832783.3837558openalex-snapshot
Can Large Language Models Decompose User Stories into Tasks? Exploring the Role of Prompting Strategies and Models
inferred← citeshttps://doi.org/10.1145/3832783.3844575openalex-snapshot
Reconstructing Executable Programs from Informal Bug Reports
inferred← citeshttps://doi.org/10.1155/stc/8514242openalex-snapshot
A Digital Twin With Material Characterisation and LLM as High‐Level Orchestrator via Few‐Shot In‐Context Learning for Closed‐Loop Control in Robotic Adaptive 3DCP
inferred← citeshttps://doi.org/10.1200/cci-25-00361openalex-snapshot
Can Small Open-Source Language Models With Retrieval-Augmented Generation Match GPT-4 Performance in Breast Cancer Clinical Decision Support?
inferred← citeshttps://doi.org/10.2139/ssrn.5959496openalex-snapshot
More Than Human: The Capital-Market Effects of AI-Generated Corporate Disclosure
inferred← citeshttps://doi.org/10.2139/ssrn.6642016openalex-snapshot
ACREF: Enhancing Code Refinement Tasks for Large Language Models Using LLM-as-a-Judge
inferred← citeshttps://doi.org/10.2139/ssrn.7203093openalex-snapshot
Decoupling Safety Assessment from Code Generation: Iterative Refinement with External Signals
inferred← citeshttps://doi.org/10.2139/ssrn.7346712openalex-snapshot
ACREF: LLM-as-a-Judge Guided Data Curation and Semantic Evaluation for Automated Code Refinement
inferred← citeshttps://doi.org/10.3390/app16136540openalex-snapshot
Large Language Models for UML Class Diagram Modeling: A Preliminary Empirical Evaluation
inferred← citeshttps://doi.org/10.3390/ijerph23081035openalex-snapshot
Walkable Intelligent Parks: Can a Large Language Model Turn Urban Park Audit Findings into Actionable Recommendations?
inferred← citeshttps://doi.org/10.3390/info17010073openalex-snapshot
Foundation Models in Software Engineering: A Taxonomy, Systematic Review, and In-Depth Analysis of Testing Support
inferred← citeshttps://doi.org/10.3390/systems14060598openalex-snapshot
A Methodology for the Systematic Evaluation of LLMs in Requirement Engineering Tasks with No Ground Truth
inferred← citeshttps://doi.org/10.5753/cibse.2026.42437openalex-snapshot
LLM-Assisted INVEST Evaluation and Improvement of User Stories: An Industrial Replication Study
inferred← citeshttps://doi.org/10.7759/cureus.88167openalex-snapshot
Context-Aware Retrieval-Augmented Generation for Artificial Intelligence in Urology
showing the first 61 of 62 · page through all 62
Editorial review for independent work. ICSAC — the Institute for Complexity Science and Advanced Computing — runs a structured editorial panel for researchers publishing outside journals, and publishes on CiteStamp rails. Submission is free and goes through ORCID: icsacinstitute.org/submit.