On the public citation graph

Reproducibility Challenges in Information Retrieval Evaluation

0 signed · 89 inferred · view as graph

Inferred edges come from two machine sources: sweeps of open repository metadata (mostly SSRN and Zenodo), and a bulk OpenAlex citation snapshot. Every row below is labelled with the one it came from. Neither is a complete citation index — absence here is not evidence of absence.

Is this your work? Sign in with ORCID and put your name on the claims that are yours.
Claim with ORCID
Reference health. See whether every reference in this work resolves to a known work and none has been retracted — and get a dated CiteStamped badge you can embed.
See reference health

What this work says

Outgoing claims: the papers this work cites, builds on, or argues with.

inferredcites →https://doi.org/10.1002/asi.20583openalex-snapshot
TREC: Experiment and evaluation in information retrieval
inferredcites →https://doi.org/10.1007/978-3-031-02276-0openalex-snapshot
Information Retrieval Evaluation
inferredcites →https://doi.org/10.1007/978-3-319-11382-1_5openalex-snapshot
An Information Retrieval Ontology for Information Retrieval Nanopublications
inferredcites →https://doi.org/10.1007/978-3-319-16354-3_83openalex-snapshot
Rank-Biased Precision Reloaded: Reproducibility and Generalization
inferredcites →https://doi.org/10.1007/978-3-540-74999-8_2openalex-snapshot
Scientific Data of an Evaluation Campaign: Do We Properly Deal with Them?
inferredcites →https://doi.org/10.1007/978-3-642-04346-8_8openalex-snapshot
Managing the Knowledge Creation Process of Large-Scale Evaluation Campaigns
inferredcites →https://doi.org/10.1007/978-3-642-33247-0_11openalex-snapshot
DIRECTions: Design and Specification of an IR Evaluation Infrastructure
inferredcites →https://doi.org/10.1007/s00799-016-0172-8openalex-snapshot
Semantic representation and enrichment of information retrieval experimental data
inferredcites →https://doi.org/10.1016/j.ipm.2012.01.004openalex-snapshot
Using crowdsourcing for TREC relevance assessment
inferredcites →https://doi.org/10.1016/j.procs.2011.04.078openalex-snapshot
Harnessing the Scientific Data Produced by the Experimental Evaluation Search Engines and Information Access Systems
inferredcites →https://doi.org/10.1016/s0306-4573(00)00010-8openalex-snapshot
Variations in relevance judgments and the measurement of retrieval effectiveness
inferredcites →https://doi.org/10.1145/1148170.1148261openalex-snapshot
Evaluating evaluation metrics based on the bootstrap
inferredcites →https://doi.org/10.1145/2064227.2064247openalex-snapshot
Principles for robust evaluation infrastructure
inferredcites →https://doi.org/10.1145/2215676.2215681openalex-snapshot
DESIRE 2011
inferredcites →https://doi.org/10.1145/2422256.2422258openalex-snapshot
Panel on use of proprietary data
inferredcites →https://doi.org/10.1145/2701583.2701587openalex-snapshot
CLEF 15th Birthday
inferredcites →https://doi.org/10.1145/2888422.2888439openalex-snapshot
Report on the SIGIR 2015 Workshop on Reproducibility, Inexplicability, and Generalizability of Results (RIGOR)
inferredcites →https://doi.org/10.1145/290941.291014openalex-snapshot
How reliable are the results of large-scale information retrieval experiments?
inferredcites →https://doi.org/10.1145/2964797.2964808openalex-snapshot
Increasing Reproducibility in IR
inferredcites →https://doi.org/10.1145/3239570openalex-snapshot
Evaluation-as-a-Service for the Computational Sciences
inferredcites →https://doi.org/10.1561/1500000009openalex-snapshot
Test Collection Based Evaluation of Information Retrieval Systems
inferredcites →https://doi.org/10.1561/1500000027openalex-snapshot
Patent Retrieval
inferredcites →https://doi.org/10.21236/ada477388openalex-snapshot
Million Query Track 2007 Overview
inferredcites →https://doi.org/10.4230/dagrep.6.1.108openalex-snapshot
Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041)
inferredcites →https://doi.org/10.4230/dagrep.6.2.1openalex-snapshot
Data-Driven Storytelling (Dagstuhl Seminar 16061)

What points at this work

Incoming claims: where this work appears in other papers’ reference lists and arguments.

inferred← citeshttps://doi.org/10.1002/asi.23917openalex-snapshot
Theory and practice of data citation
inferred← citeshttps://doi.org/10.1007/978-3-030-22948-1_4openalex-snapshot
An Innovative Approach to Data Management and Curation of Experimental Data Generated Through IR Test Collections
inferred← citeshttps://doi.org/10.1007/978-3-030-72240-1_51openalex-snapshot
repro_eval: A Python Interface to Reproducibility Measures of System-Oriented IR Experiments
inferred← citeshttps://doi.org/10.1007/978-3-031-88717-8_10openalex-snapshot
A Reproducibility Study for Joint Information Retrieval and Recommendation in Product Search
inferred← citeshttps://doi.org/10.1007/978-3-031-88717-8_11openalex-snapshot
Towards Reproducibility of Interactive Retrieval Experiments: Framework and Case Study
inferred← citeshttps://doi.org/10.1007/978-3-319-56300-8_12openalex-snapshot
Proposal for an Evaluation Framework for Compliance Checkers for Long-Term Digital Preservation
inferred← citeshttps://doi.org/10.1007/978-3-319-56300-8_8openalex-snapshot
An Ontology to Make the DELOS Reference Model and the 5S Model Interoperable
inferred← citeshttps://doi.org/10.1007/978-3-319-61893-7_2openalex-snapshot
Digital Libraries: From Digital Resources to Challenges in Scientific Data Sharing and Re-Use
inferred← citeshttps://doi.org/10.1007/978-3-319-68130-6_2openalex-snapshot
The Road Towards Reproducibility in Science: The Case of Data Citation
inferred← citeshttps://doi.org/10.1007/978-3-319-73165-0_4openalex-snapshot
Thirty Years of Digital Libraries Research at the University of Padua: The Systems Side
inferred← citeshttps://doi.org/10.1007/978-3-319-98932-7_23openalex-snapshot
Overview of CENTRE@CLEF 2018: A First Tale in the Systematic Reproducibility Realm
inferred← citeshttps://doi.org/10.1007/978-3-319-98932-7_3openalex-snapshot
Using R Markdown for Replicable Experiments in Evidence Based Medicine
inferred← citeshttps://doi.org/10.1007/s10664-021-10113-2openalex-snapshot
The reproducibility of programming-related issues in Stack Overflow questions
inferred← citeshttps://doi.org/10.1007/s11257-021-09302-xopenalex-snapshot
Improving accountability in recommender systems research through reproducibility
inferred← citeshttps://doi.org/10.1016/j.ipm.2018.04.006openalex-snapshot
CLAIRE: A combinatorial visual analytics system for information retrieval evaluation
inferred← citeshttps://doi.org/10.1016/j.ipm.2019.102149openalex-snapshot
Effectiveness evaluation without human relevance judgments: A systematic analysis of existing methods and of their combinations
inferred← citeshttps://doi.org/10.1016/j.ipm.2023.103332openalex-snapshot
An in-depth investigation on the behavior of measures to quantify reproducibility
inferred← citeshttps://doi.org/10.1016/j.jbi.2017.07.010openalex-snapshot
Reproducibility of studies on text mining for citation screening in systematic reviews: Evaluation and checklist
inferred← citeshttps://doi.org/10.1080/1528008x.2018.1563016openalex-snapshot
A Systematic Literature Review: Information Accuracy Practices in Tourism
inferred← citeshttps://doi.org/10.1108/ftinr-05-2025-0087openalex-snapshot
Reproducibility in Information Retrieval
inferred← citeshttps://doi.org/10.1109/access.2022.3158675openalex-snapshot
Reproducibility in Computing Research: An Empirical Study
inferred← citeshttps://doi.org/10.1109/mmul.2021.3066779openalex-snapshot
On the User-Centric Comparative Remote Evaluation of Interactive Video Search Systems
inferred← citeshttps://doi.org/10.1111/exsy.12871openalex-snapshot
A replication study on implicit feedback recommender systems with application to the data visualization recommendation
inferred← citeshttps://doi.org/10.1145/3106371openalex-snapshot
What Does Affect the Correlation Among Evaluation Measures?
inferred← citeshttps://doi.org/10.1145/3110217openalex-snapshot
AWARE
inferred← citeshttps://doi.org/10.1145/3121050.3121110openalex-snapshot
LEARning Next gEneration Rankers (LEARNER 2017)
inferred← citeshttps://doi.org/10.1145/3197026.3197037openalex-snapshot
Evaluation of Conformance Checkers for Long-Term Preservation of Multimedia Documents
inferred← citeshttps://doi.org/10.1145/3239573openalex-snapshot
Reproduce and Improve
inferred← citeshttps://doi.org/10.1145/3241064openalex-snapshot
Reproduce. Generalize. Extend. On Information Retrieval Evaluation without Relevance Judgments
inferred← citeshttps://doi.org/10.1145/3242180openalex-snapshot
To Clean or Not to Clean
inferred← citeshttps://doi.org/10.1145/3268408openalex-snapshot
Introduction to the Special Issue on Reproducibility in Information Retrieval
inferred← citeshttps://doi.org/10.1145/3268410openalex-snapshot
Introduction to the Special Issue on Reproducibility in Information Retrieval
inferred← citeshttps://doi.org/10.1145/3274784.3274786openalex-snapshot
SIGIR Initiative to Implement ACM Artifact Review and Badging
inferred← citeshttps://doi.org/10.1145/3274784.3274789openalex-snapshot
The Dagstuhl Perspectives Workshop on Performance Modeling and Prediction
inferred← citeshttps://doi.org/10.1145/3343413.3377988openalex-snapshot
Update Delivery Mechanisms for Prospective Information Needs
inferred← citeshttps://doi.org/10.1145/3477495.3531738openalex-snapshot
ir_metadata: An Extensible Metadata Schema for IR Experiments
inferred← citeshttps://doi.org/10.1145/3539618.3591913openalex-snapshot
DECAF: A Modular and Extensible Conversational Search Framework
inferred← citeshttps://doi.org/10.1145/3674127openalex-snapshot
Information Retrieval
inferred← citeshttps://doi.org/10.1145/3674127.3674128openalex-snapshot
Preface
inferred← citeshttps://doi.org/10.1145/3674127.3674129openalex-snapshot
Introduction
inferred← citeshttps://doi.org/10.1145/3674127.3674130openalex-snapshot
Neural Information Retrieval
inferred← citeshttps://doi.org/10.1145/3674127.3674131openalex-snapshot
Recommender Systems
inferred← citeshttps://doi.org/10.1145/3674127.3674132openalex-snapshot
Evaluation of IR Systems
inferred← citeshttps://doi.org/10.1145/3674127.3674133openalex-snapshot
Query-driven Search Functionality
inferred← citeshttps://doi.org/10.1145/3674127.3674134openalex-snapshot
Knowledge Graphs and Search
inferred← citeshttps://doi.org/10.1145/3674127.3674135openalex-snapshot
Conversational Search
inferred← citeshttps://doi.org/10.1145/3674127.3674136openalex-snapshot
Cross-language Retrieval
inferred← citeshttps://doi.org/10.1145/3674127.3674137openalex-snapshot
Temporal Extraction and Retrieval
inferred← citeshttps://doi.org/10.1145/3674127.3674138openalex-snapshot
Bias in Retrieval Systems
inferred← citeshttps://doi.org/10.1145/3674127.3674139openalex-snapshot
Privacy in Information Retrieval
inferred← citeshttps://doi.org/10.1145/3674127.3674140openalex-snapshot
eCommerce Search and Discovery
inferred← citeshttps://doi.org/10.1145/3674127.3674141openalex-snapshot
Professional Search
inferred← citeshttps://doi.org/10.1145/3674127.3674142openalex-snapshot
Searching Personal Collections
inferred← citeshttps://doi.org/10.1145/3674127.3702959openalex-snapshot
Audio-based Music Retrieval
inferred← citeshttps://doi.org/10.1145/3674127.3702960openalex-snapshot
IR in Biomedicine
inferred← citeshttps://doi.org/10.1145/3674127.3702961openalex-snapshot
Bibliography
inferred← citeshttps://doi.org/10.1145/3674127.3702962openalex-snapshot
Authors’ Biography/Index
inferred← citeshttps://doi.org/10.1145/3772275openalex-snapshot
Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing
inferred← citeshttps://doi.org/10.1177/0165551519866551openalex-snapshot
An intrinsic evaluation of the Waterloo spam rankings of the ClueWeb09 and ClueWeb12 datasets
inferred← citeshttps://doi.org/10.1177/24056456261497567openalex-snapshot
Enhancing Web Search Effectiveness and Robustness Using BERT- and Co-Occurrence-Based Pruned Conflation Sets
inferred← citeshttps://doi.org/10.18517/ijaseit.12.6.13215openalex-snapshot
Evaluation of Average Term Occurrences Weighting Technique for Arabic Textual Information Retrieval
inferred← citeshttps://doi.org/10.4000/books.aaccademia.2403openalex-snapshot
A Reproducible Approach with R Markdown to Automatic Classification of Medical Certificates in French
inferred← citeshttps://doi.org/10.4230/dagman.7.1.96openalex-snapshot
From Evaluating to Forecasting Performance: How to Turn Information Retrieval, Natural Language Processing and Recommender Systems into Predictive Sciences (Dagstuhl Perspectives Workshop 17442)
inferred← citeshttps://doi.org/10.48550/arxiv.2011.00479openalex-snapshot
Cheap IR Evaluation: Fewer Topics, No Relevance Judgements, and Crowdsourced Assessments
Editorial review for independent work. ICSAC — the Institute for Complexity Science and Advanced Computing — runs a structured editorial panel for researchers publishing outside journals, and publishes on CiteStamp rails. Submission is free and goes through ORCID: icsacinstitute.org/submit.