On the public citation graph

Empirical methodology for crowdsourcing ground truth

0 signed · 65 inferred · view as graph

Inferred edges come from two machine sources: sweeps of open repository metadata (mostly SSRN and Zenodo), and a bulk OpenAlex citation snapshot. Every row below is labelled with the one it came from. Neither is a complete citation index — absence here is not evidence of absence.

Is this your work? Sign in with ORCID and put your name on the claims that are yours.
Claim with ORCID
Reference health. See whether every reference in this work resolves to a known work and none has been retracted — and get a dated CiteStamped badge you can embed.
See reference health

What this work says

Outgoing claims: the papers this work cites, builds on, or argues with.

inferredcites →https://doi.org/10.1002/pds.4850openalex-snapshot
Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program.
inferredcites →https://doi.org/10.1007/978-3-319-11915-1_3openalex-snapshot
Conference v2.0: An Uncertain Version of the OAEI Conference Benchmark
inferredcites →https://doi.org/10.1007/978-3-319-11915-1_31openalex-snapshot
CrowdTruth: Machine-Human Computation Framework for Harnessing Disagreement in Gathering Annotated Data
inferredcites →https://doi.org/10.1007/978-3-319-48472-3_13openalex-snapshot
Human-in-the-Loop Web Resource Classification
inferredcites →https://doi.org/10.1007/978-3-642-35173-0_15openalex-snapshot
Query Driven Hypothesis Generation for Answering Queries over NLP Graphs
inferredcites →https://doi.org/10.1007/bf02295996openalex-snapshot
Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
inferredcites →https://doi.org/10.1007/bf02769550openalex-snapshot
On the definition of “Picture”
inferredcites →https://doi.org/10.1007/s00778-013-0328-8openalex-snapshot
Hybrid entity clustering using crowds and data
inferredcites →https://doi.org/10.1016/j.jbi.2012.04.004openalex-snapshot
The EU-ADR corpus: Annotated drugs, diseases, targets, and their relationships
inferredcites →https://doi.org/10.1038/nature21054openalex-snapshot
A solution to the single-question crowd wisdom problem
inferredcites →https://doi.org/10.1080/09298215.2016.1200631openalex-snapshot
The Problem of Limited Inter-rater Agreement in Modelling Music Similarity
inferredcites →https://doi.org/10.1093/nar/gkh061openalex-snapshot
The Unified Medical Language System (UMLS): integrating biomedical terminology
inferredcites →https://doi.org/10.1109/icde.2014.6816717openalex-snapshot
Combining information extraction and human computing for crowdsourced knowledge acquisition
inferredcites →https://doi.org/10.1145/1357054.1357127openalex-snapshot
Crowdsourcing user studies with Mechanical Turk
inferredcites →https://doi.org/10.1145/1629911.1630023openalex-snapshot
Human computation
inferredcites →https://doi.org/10.1145/1743384.1743478openalex-snapshot
How reliable are annotations via crowdsourcing
inferredcites →https://doi.org/10.1145/1837885.1837906openalex-snapshot
Quality management on Amazon Mechanical Turk
inferredcites →https://doi.org/10.1145/2464464.2464482openalex-snapshot
Mechanical turk as an ontology engineer?
inferredcites →https://doi.org/10.1145/2488388.2488403openalex-snapshot
Reactive crowdsourcing
inferredcites →https://doi.org/10.1145/2488388.2488489openalex-snapshot
Pricing mechanisms for crowdsourcing markets
inferredcites →https://doi.org/10.1145/2615569.2615644openalex-snapshot
Crowdsourcing knowledge-intensive tasks in cultural heritage
inferredcites →https://doi.org/10.1145/3025453.3026044openalex-snapshot
Revolt
inferredcites →https://doi.org/10.1145/3152889openalex-snapshot
Crowdsourcing Ground Truth for Medical Relation Extraction
inferredcites →https://doi.org/10.1162/coli_a_00074openalex-snapshot
What Determines Inter-Coder Agreement in Manual Annotations? A Meta-Analytic Investigation
inferredcites →https://doi.org/10.1186/1471-2105-12-486openalex-snapshot
Constructing a semantic predication gold standard from the biomedical literature
inferredcites →https://doi.org/10.13016/m21j97c2popenalex-snapshot
Annotating named entities in Twitter data with crowdsourcing
inferredcites →https://doi.org/10.15346/hc.v1i1.3openalex-snapshot
The Three Sides of CrowdTruth
inferredcites →https://doi.org/10.15346/hc.v2i1.2openalex-snapshot
Crowdsourcing and the Semantic Web: A Research Manifesto
inferredcites →https://doi.org/10.1609/aimag.v36i1.2564openalex-snapshot
Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation
inferredcites →https://doi.org/10.1609/aimag.v39i4.2834openalex-snapshot
Reports of the Workshops Held at the Sixth AAAI Conference on Human Computation and Crowdsourcing
inferredcites →https://doi.org/10.1609/hcomp.v1i1.13075openalex-snapshot
Volunteering Versus Work for Pay: Incentives and Tradeoffs in Crowdsourcing
inferredcites →https://doi.org/10.1609/hcomp.v1i1.13091openalex-snapshot
Crowdsourcing Multi-Label Classification for Taxonomy Creation
inferredcites →https://doi.org/10.1609/hcomp.v2i1.13167openalex-snapshot
To Re(label), or Not To Re(label)
inferredcites →https://doi.org/10.1609/hcomp.v3i1.13266openalex-snapshot
Linguistic Wisdom from the Crowd
inferredcites →https://doi.org/10.1609/hcomp.v4i1.13287openalex-snapshot
Why Is That Relevant? Collecting Annotator Rationales for Relevance Judgments
inferredcites →https://doi.org/10.2196/jmir.2426openalex-snapshot
Web 2.0-Based Crowdsourcing for High-Quality Gold Standard Development in Clinical Natural Language Processing
inferredcites →https://doi.org/10.3115/1608829.1608840openalex-snapshot
The reliability of anaphoric annotation, reconsidered
inferredcites →https://doi.org/10.3115/1613715.1613751openalex-snapshot
Cheap and fast---but is it good?
inferredcites →https://doi.org/10.3115/1690219.1690287openalex-snapshot
Distant supervision for relation extraction without labeled data
inferredcites →https://doi.org/10.3115/v1/n15-1089openalex-snapshot
Early Gains Matter: A Case for Preferring Generative over Discriminative Crowdsourcing Models
inferredcites →https://doi.org/10.3115/v1/p14-1078openalex-snapshot
Medical Relation Extraction with Manifold Models
inferredcites →https://doi.org/10.3115/v1/p14-2083openalex-snapshot
Linguistically debatable or just plain wrong?
inferredcites →https://doi.org/10.48550/arxiv.1310.4546openalex-snapshot
Distributed Representations of Words and Phrases and their Compositionality
inferredcites →https://doi.org/10.48550/arxiv.1506.03140openalex-snapshot
On-the-Job Learning with Bayesian Decision Theory
inferredcites →https://doi.org/10.48550/arxiv.cmp-lg/9602004openalex-snapshot
Assessing agreement on classification tasks: the kappa statistic
inferredcites →https://doi.org/10.6084/m9.figshare.679997openalex-snapshot
Crowd Truth: Harnessing disagreement in crowdsourcing a relation extraction gold standard
inferredcites →https://doi.org/10.63317/2294e6wj5mvyopenalex-snapshot
Temporal Information Annotation: Crowd vs. Experts
inferredcites →https://doi.org/10.63317/3asvpdmppif3openalex-snapshot
The VU Sound Corpus: Adding More Fine-grained Annotations to the Freesound Database
inferredcites →https://doi.org/10.63317/44rwjpkoe68wopenalex-snapshot
Crowdsourcing Salient Information from News and Tweets

What points at this work

Incoming claims: where this work appears in other papers’ reference lists and arguments.

inferred← citeshttps://doi.org/10.1017/9781009505468openalex-snapshot
AI and Image
inferred← citeshttps://doi.org/10.1109/bigdata50022.2020.9378188openalex-snapshot
Evaluating the Accuracy of Cloud NLP Services Using Ground-Truth Experiments
inferred← citeshttps://doi.org/10.1145/3269206.3271779openalex-snapshot
Studying Topical Relevance with Evidence-based Crowdsourcing
inferred← citeshttps://doi.org/10.1145/3308558.3320096openalex-snapshot
Crowdsourcing Inclusivity: Dealing with Diversity of Opinions, Perspectives and Ambiguity in Annotated Data
inferred← citeshttps://doi.org/10.1145/3555636openalex-snapshot
It's All Relative! A Method to Counter Human Bias in Crowdsourced Stance Detection of News Articles
inferred← citeshttps://doi.org/10.1145/3613904.3642834openalex-snapshot
If in a Crowdsourced Data Annotation Pipeline, a GPT-4
inferred← citeshttps://doi.org/10.1145/3637366openalex-snapshot
Linguistically Differentiating Acts and Recalls of Racial Microaggressions on Social Media
inferred← citeshttps://doi.org/10.1162/tacl_a_00586openalex-snapshot
Design Choices for Crowdsourcing Implicit Discourse Relations: Revealing the Biases Introduced by Task Design
inferred← citeshttps://doi.org/10.1177/00220345241272052openalex-snapshot
Publicly Available Dental Image Datasets for Artificial Intelligence
inferred← citeshttps://doi.org/10.14778/3407790.3407842openalex-snapshot
Detecting and preventing confused labels in crowdsourced data
inferred← citeshttps://doi.org/10.1609/hcomp.v11i1.27547openalex-snapshot
Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection
inferred← citeshttps://doi.org/10.18653/v1/2022.findings-naacl.51openalex-snapshot
LMTurk: Few-Shot Learners as Crowdsourcing Workers in a Language-Model-as-a-Service Framework
inferred← citeshttps://doi.org/10.2196/72524openalex-snapshot
Performance of Large Language Models in the Cognitive Analysis of Misinformation: Evaluation Study
inferred← citeshttps://doi.org/10.3390/math9080875openalex-snapshot
A Conceptual Probabilistic Framework for Annotation Aggregation of Citizen Science Data
inferred← citeshttps://doi.org/10.48550/arxiv.1906.07254openalex-snapshot
Crowdsourcing in the Absence of Ground Truth -- A Case Study
inferred← citeshttps://doi.org/10.48550/arxiv.2102.08482openalex-snapshot
Finding the Ground-Truth from Multiple Labellers: Why Parameters of the Task Matter
Editorial review for independent work. ICSAC — the Institute for Complexity Science and Advanced Computing — runs a structured editorial panel for researchers publishing outside journals, and publishes on CiteStamp rails. Submission is free and goes through ORCID: icsacinstitute.org/submit.