On the public citation graph

Examining the Limits of Crowdsourcing for Relevance Assessment

0 signed · 48 inferred · view as graph

Inferred edges come from two machine sources: sweeps of open repository metadata (mostly SSRN and Zenodo), and a bulk OpenAlex citation snapshot. Every row below is labelled with the one it came from. Neither is a complete citation index — absence here is not evidence of absence.

Is this your work? Sign in with ORCID and put your name on the claims that are yours.
Claim with ORCID
Reference health. See whether every reference in this work resolves to a known work and none has been retracted — and get a dated CiteStamped badge you can embed.
See reference health

What this work says

Outgoing claims: the papers this work cites, builds on, or argues with.

inferredcites →https://doi.org/10.1007/978-3-642-20161-5_17openalex-snapshot
In Search of Quality in Crowdsourcing for Search Engine Evaluation
inferredcites →https://doi.org/10.1007/978-3-642-22511-6_12openalex-snapshot
Training Students to Evaluate Search Engines
inferredcites →https://doi.org/10.1109/cvprw.2008.4562953openalex-snapshot
Utility data annotation with Amazon Mechanical Turk
inferredcites →https://doi.org/10.1145/1242572.1242639openalex-snapshot
Navigating the intranet with high precision
inferredcites →https://doi.org/10.1145/1390334.1390347openalex-snapshot
The good and the bad system
inferredcites →https://doi.org/10.1145/1390334.1390447openalex-snapshot
Relevance assessment
inferredcites →https://doi.org/10.1145/1458082.1458160openalex-snapshot
How evaluator domain expertise affects search result relevance judgments
inferredcites →https://doi.org/10.1145/1571941.1572138openalex-snapshot
Topic set size redux
inferredcites →https://doi.org/10.1145/1924475.1924481openalex-snapshot
Crowdsourcing for search evaluation
inferredcites →https://doi.org/10.1145/290941.291017openalex-snapshot
Variations in relevance judgments and the measurement of retrieval effectiveness
inferredcites →https://doi.org/10.1145/792550.792552openalex-snapshot
A taxonomy of web search
inferredcites →https://doi.org/10.3115/1613715.1613751openalex-snapshot
Cheap and fast---but is it good?

What points at this work

Incoming claims: where this work appears in other papers’ reference lists and arguments.

inferred← citeshttps://doi.org/10.1002/asi.23625openalex-snapshot
Assessing geographic relevance for mobile search: A computational model and its validation via crowdsourcing
inferred← citeshttps://doi.org/10.1007/978-3-030-00668-6_12openalex-snapshot
The Computer Science Ontology: A Large-Scale Taxonomy of Research Areas
inferred← citeshttps://doi.org/10.1007/978-3-030-58219-7_2openalex-snapshot
s-AWARE: Supervised Measure-Based Methods for Crowd-Assessors Combination
inferred← citeshttps://doi.org/10.1016/j.cageo.2025.105875openalex-snapshot
Automatic detection of ditches and natural streams from digital elevation models using deep learning
inferred← citeshttps://doi.org/10.1016/j.comnet.2015.05.018openalex-snapshot
Hybrid human–machine information systems: Challenges and opportunities
inferred← citeshttps://doi.org/10.1016/j.websem.2014.10.001openalex-snapshot
An Overview of Semantic Search Evaluation Initiatives
inferred← citeshttps://doi.org/10.1109/iccpeic.2017.8290392openalex-snapshot
The crowdsourcing systems survey work
inferred← citeshttps://doi.org/10.1109/icde.2018.00194openalex-snapshot
CrowdSheet: Instant Implementation and Out-of-Hand Execution of Complex Crowdsourcing
inferred← citeshttps://doi.org/10.1109/sai.2014.6918273openalex-snapshot
Exploring relevance assessment using crowdsourcing for faceted and ambiguous queries
inferred← citeshttps://doi.org/10.1109/taffc.2023.3273916openalex-snapshot
Crowdsourcing Affective Annotations Via fNIRS-BCI
inferred← citeshttps://doi.org/10.1109/tmc.2014.2316517openalex-snapshot
Utilize Signal Traces from Others? A Crowdsourcing Perspective of Energy Saving in Cellular Data Communication
inferred← citeshttps://doi.org/10.1109/tpds.2015.2457924openalex-snapshot
A Crowdsourcing Worker Quality Evaluation Algorithm on MapReduce for Big Data Applications
inferred← citeshttps://doi.org/10.1145/2854946.2854968openalex-snapshot
A Comparison of Primary and Secondary Relevance Judgements for Real-Life Topics
inferred← citeshttps://doi.org/10.1145/3002172openalex-snapshot
On Crowdsourcing Relevance Magnitudes for Information Retrieval Evaluation
inferred← citeshttps://doi.org/10.1145/3020165.3022127openalex-snapshot
Automatic Summarization of Domain-specific Forum Threads
inferred← citeshttps://doi.org/10.1145/3110217openalex-snapshot
AWARE
inferred← citeshttps://doi.org/10.1145/3209978.3210033openalex-snapshot
Crowd vs. Expert
inferred← citeshttps://doi.org/10.1145/3269206.3271779openalex-snapshot
Studying Topical Relevance with Evidence-based Crowdsourcing
inferred← citeshttps://doi.org/10.1145/3406522.3446027openalex-snapshot
An Intent Taxonomy for Questions Asked in Web Search
inferred← citeshttps://doi.org/10.1145/3498366.3505815openalex-snapshot
The Crowd is Made of People
inferred← citeshttps://doi.org/10.1145/3503516.3503526openalex-snapshot
Crowdsourcing Backstories for Complex Task-Based Search
inferred← citeshttps://doi.org/10.1145/3600227openalex-snapshot
On the Ordering of Pooled Web Pages, Gold Assessments, and Bronze Assessments
inferred← citeshttps://doi.org/10.1145/3626772.3657707openalex-snapshot
Large Language Models can Accurately Predict Searcher Preferences
inferred← citeshttps://doi.org/10.1145/3657054.3657071openalex-snapshot
Crowdsourcing of public service: Decision-making and organizational characteristics with digital technology as a key variable: Take C district of Guangdong Province as an example
inferred← citeshttps://doi.org/10.1145/3726302.3729916openalex-snapshot
LLM-Assisted Relevance Assessments: When Should We Ask LLMs for Help?
inferred← citeshttps://doi.org/10.1155/2014/135641openalex-snapshot
Creation of Reliable Relevance Judgments in Information Retrieval Systems Evaluation Experimentation through Crowdsourcing: A Review
inferred← citeshttps://doi.org/10.1162/dint_a_00055openalex-snapshot
The Computer Science Ontology: A Comprehensive Automatically-Generated Taxonomy of Research Areas
inferred← citeshttps://doi.org/10.1609/hcomp.v6i1.13332openalex-snapshot
All That Glitters Is Gold — An Attack Scheme on Gold Questions in Crowdsourcing
inferred← citeshttps://doi.org/10.1613/jair.1.11332openalex-snapshot
Adversarial Attacks on Crowdsourcing Quality Control
inferred← citeshttps://doi.org/10.1613/jair.1.12012openalex-snapshot
Annotator Rationales for Labeling Tasks in Crowdsourcing
inferred← citeshttps://doi.org/10.17760/d20194416openalex-snapshot
Optimally selecting and combining assessment and assessor types for information retrieval evaluation
inferred← citeshttps://doi.org/10.24963/ijcai.2019/850openalex-snapshot
Quality Control Attack Schemes in Crowdsourcing
inferred← citeshttps://doi.org/10.3233/ssw200037openalex-snapshot
Ontology Extraction and Usage in the Scholarly Knowledge Domain1
inferred← citeshttps://doi.org/10.48550/arxiv.1501.06370openalex-snapshot
Augmented Test Collections: A Step in the Right Direction
inferred← citeshttps://doi.org/10.48550/arxiv.1806.00755openalex-snapshot
Mix and Match: Collaborative Expert-Crowd Judging for Building Test Collections Accurately and Affordably
inferred← citeshttps://doi.org/10.7892/boris.98845openalex-snapshot
Digital Preservation through Digital Sustainability
Editorial review for independent work. ICSAC — the Institute for Complexity Science and Advanced Computing — runs a structured editorial panel for researchers publishing outside journals, and publishes on CiteStamp rails. Submission is free and goes through ORCID: icsacinstitute.org/submit.