On the public citation graph

Crowdsourcing Subjective Annotations Using Pairwise Comparisons Reduces Bias and Error Compared to the Majority-vote Method

0 signed · 70 inferred · view as graph

Inferred edges come from two machine sources: sweeps of open repository metadata (mostly SSRN and Zenodo), and a bulk OpenAlex citation snapshot. Every row below is labelled with the one it came from. Neither is a complete citation index — absence here is not evidence of absence.

Is this your work? Sign in with ORCID and put your name on the claims that are yours.
Claim with ORCID
Reference health. See whether every reference in this work resolves to a known work and none has been retracted — and get a dated CiteStamped badge you can embed.
See reference health

What this work says

Outgoing claims: the papers this work cites, builds on, or argues with.

inferredcites →https://doi.org/10.1007/978-3-319-18818-8_43openalex-snapshot
Crowdsourcing Disagreement for Collecting Semantic Annotation
inferredcites →https://doi.org/10.1007/bf02769550openalex-snapshot
On the definition of “Picture”
inferredcites →https://doi.org/10.1007/s10115-020-01475-yopenalex-snapshot
Label similarity-based weighted soft majority voting and pairing for crowdsourcing
inferredcites →https://doi.org/10.1007/s10559-020-00216-xopenalex-snapshot
Improving Text Generation Through Introducing Coherence Metrics
inferredcites →https://doi.org/10.1016/j.comnet.2020.107587openalex-snapshot
A spam worker detection approach based on heterogeneous network embedding in crowdsourcing platforms
inferredcites →https://doi.org/10.1016/j.cviu.2018.10.009openalex-snapshot
Pros and cons of GAN evaluation measures
inferredcites →https://doi.org/10.1016/s0898-1221(96)90463-0openalex-snapshot
Advances in neural information processing systems 7
inferredcites →https://doi.org/10.1017/pan.2019.32openalex-snapshot
Crowdsourcing Reliable Local Data
inferredcites →https://doi.org/10.1037//0003-066x.52.1.45openalex-snapshot
Structure mapping in analogy and similarity.
inferredcites →https://doi.org/10.1038/111566b0openalex-snapshot
The Meaning of Meaning: a Study of the Influence of Language upon Thought and of the Science of Symbolism
inferredcites →https://doi.org/10.1080/10584609.2010.542360openalex-snapshot
From Incivility to Outrage: Political Discourse in Blogs, Talk Radio, and Cable News
inferredcites →https://doi.org/10.1109/iccv.2011.6126281openalex-snapshot
Relative attributes
inferredcites →https://doi.org/10.1109/iccv.2013.426openalex-snapshot
Attribute Adaptation for Personalized Image Search
inferredcites →https://doi.org/10.1109/icip.2011.6116320openalex-snapshot
Crowdsourcing subjective image quality evaluation
inferredcites →https://doi.org/10.1109/icme51207.2021.9428220openalex-snapshot
Toward Effective Automated Content Analysis via Crowdsourcing
inferredcites →https://doi.org/10.1109/icpr.2014.756openalex-snapshot
Systematic Labeling Bias: De-biasing Where Everyone is Wrong
inferredcites →https://doi.org/10.1109/snams.2018.8554744openalex-snapshot
Inter-Rater Agreement for Social Computing Studies
inferredcites →https://doi.org/10.1109/snams.2018.8554954openalex-snapshot
Online Hate Interpretation Varies by Country, But More by Individual: A Statistical Analysis Using Crowdsourced Ratings
inferredcites →https://doi.org/10.1145/2433396.2433420openalex-snapshot
Pairwise ranking aggregation in a crowdsourced setting
inferredcites →https://doi.org/10.1145/2441776.2441923openalex-snapshot
The future of crowd work
inferredcites →https://doi.org/10.1145/2675133.2675214openalex-snapshot
Flock
inferredcites →https://doi.org/10.1145/2756406.2757741openalex-snapshot
Debugging a Crowdsourced Task with Low Inter-Rater Agreement
inferredcites →https://doi.org/10.1145/2818048.2820016openalex-snapshot
Parting Crowds
inferredcites →https://doi.org/10.1145/3025453.3026044openalex-snapshot
Revolt
inferredcites →https://doi.org/10.1145/3152889openalex-snapshot
Crowdsourcing Ground Truth for Medical Relation Extraction
inferredcites →https://doi.org/10.1145/3278721.3278729openalex-snapshot
Measuring and Mitigating Unintended Bias in Text Classification
inferredcites →https://doi.org/10.1145/3290605.3300637openalex-snapshot
Understanding and Mitigating Worker Biases in the Crowdsourced Collection of Subjective Judgments
inferredcites →https://doi.org/10.1145/3295750.3298954openalex-snapshot
Online Hate Ratings Vary by Extremes
inferredcites →https://doi.org/10.1145/3308560.3317083openalex-snapshot
Crowdsourcing Subjective Tasks: The Case Study of Understanding Toxicity in Online Discussions
inferredcites →https://doi.org/10.1145/3359164openalex-snapshot
Efficient Elicitation Approaches to Estimate Collective Crowd Answers
inferredcites →https://doi.org/10.1145/3394486.3403102openalex-snapshot
Towards Fair Truth Discovery from Biased Crowdsourced Answers
inferredcites →https://doi.org/10.1145/3415243openalex-snapshot
Sketchy: Drawing Inspiration from the Crowd
inferredcites →https://doi.org/10.1145/3491101.3519614openalex-snapshot
Debiased Label Aggregation for Subjective Crowdsourcing Tasks
inferredcites →https://doi.org/10.1177/1745691610393521openalex-snapshot
Is It All Relative?
inferredcites →https://doi.org/10.1214/20-ba1223openalex-snapshot
Improving Multilevel Regression and Poststratification with Structured Priors
inferredcites →https://doi.org/10.1371/journal.pone.0233154openalex-snapshot
Enhancing big data in the social sciences with crowdsourcing: Data augmentation practices, techniques, and opportunities
inferredcites →https://doi.org/10.14778/3055540.3055547openalex-snapshot
Truth inference in crowdsourcing
inferredcites →https://doi.org/10.1609/aaai.v31i1.10634openalex-snapshot
Pairwise HITS: Quality Estimation from Pairwise Comparisons in Creator-Evaluator Crowdsourcing Process
inferredcites →https://doi.org/10.1609/aimag.v36i1.2564openalex-snapshot
Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation
inferredcites →https://doi.org/10.1609/hcomp.v3i1.13231openalex-snapshot
Crowdsourcing Feature Discovery via Adaptively Chosen Comparisons
inferredcites →https://doi.org/10.1609/hcomp.v4i1.13270openalex-snapshot
MicroTalk: Using Argumentation to Improve Crowdsourcing Accuracy
inferredcites →https://doi.org/10.1609/hcomp.v6i1.13330openalex-snapshot
Capturing Ambiguity in Crowdsourcing Frame Disambiguation
inferredcites →https://doi.org/10.1609/hcomp.v7i1.5272openalex-snapshot
Second Opinion: Supporting Last-Mile Person Identification with Crowdsourcing and Face Recognition
inferredcites →https://doi.org/10.1613/jair.1.12752openalex-snapshot
Learning from Disagreement: A Survey
inferredcites →https://doi.org/10.24963/ijcai.2018openalex-snapshot
Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence
inferredcites →https://doi.org/10.24963/ijcai.2018/304openalex-snapshot
Experimental Design under the Bradley-Terry Model
inferredcites →https://doi.org/10.3115/v1/p14-2062openalex-snapshot
Experiments with crowdsourced re-annotation of a POS tagging data set
inferredcites →https://doi.org/10.48550/arxiv.1911.02471openalex-snapshot
Designing Evaluations of Machine Learning Models for Subjective Inference: The Case of Sentence Toxicity
inferredcites →https://doi.org/10.48550/arxiv.2202.04823openalex-snapshot
Decreasing Annotation Burden of Pairwise Comparisons with Human-in-the-Loop Sorting: Application in Medical Image Artifact Rating
inferredcites →https://doi.org/10.7717/peerj-cs.296openalex-snapshot
Democratizing AI: non-expert design of prediction tasks

What points at this work

Incoming claims: where this work appears in other papers’ reference lists and arguments.

inferred← citeshttps://doi.org/10.1002/asi.70102openalex-snapshot
Methodological challenges in content‐based citation analysis: Expertise, reliability, and the primacy of citance identification
inferred← citeshttps://doi.org/10.1007/978-3-031-91398-3_4openalex-snapshot
Semi-supervised Learning with Pairwise Instance Comparisons for Medical Instance Classification
inferred← citeshttps://doi.org/10.1007/s12555-024-0702-xopenalex-snapshot
Crowdsourcing Task Design Using Multi-agent Systems and an Enhanced Genetic Algorithm
inferred← citeshttps://doi.org/10.1007/s13278-026-01596-2openalex-snapshot
Do not review my code harshly: identifying counterproductive behavior in code review comments
inferred← citeshttps://doi.org/10.1016/j.compind.2025.104328openalex-snapshot
Vision-based hand pose estimation methods for Augmented Reality in industry: Crowdsourced evaluation on HoloLens 2
inferred← citeshttps://doi.org/10.1016/j.eswa.2025.130660openalex-snapshot
A systematic survey on video frame interpolation: Advances, challenges, and future directions
inferred← citeshttps://doi.org/10.1038/s41598-026-74798-5openalex-snapshot
Ranking handwriting legibility with foundation-model embeddings at human-level consistency on the HaLO dataset
inferred← citeshttps://doi.org/10.1109/coginfocom66819.2025.11200596openalex-snapshot
Stochastic VR Reconstruction Advisor: A Probabilistic Workflow Recommendation Engine
inferred← citeshttps://doi.org/10.1109/iscas56072.2025.11043827openalex-snapshot
HEALSound: Healthcare Education And Labeling for Cardiopulmonary Sounds
inferred← citeshttps://doi.org/10.1128/mbio.01167-25openalex-snapshot
Osmotic conditions shape fitness gains and resistance mechanisms during E. coli and T4 phage co-evolution
inferred← citeshttps://doi.org/10.1145/3613905.3650930openalex-snapshot
Exploring Collective Theory of Mind on Pedestrian Behavioral Intentions
inferred← citeshttps://doi.org/10.1145/3686987openalex-snapshot
(De)Noise: Moderating the Inconsistency Between Human Decision-Makers
inferred← citeshttps://doi.org/10.1145/3696410.3714587openalex-snapshot
Query Design for Crowdsourced Clustering: Effect of Cognitive Overload and Contextual Bias
inferred← citeshttps://doi.org/10.1145/3706599.3720244openalex-snapshot
Can LLMs be used to Quantify the Emotional Salience of Text Statements using an Elo Rating System?
inferred← citeshttps://doi.org/10.1145/3715275.3732146openalex-snapshot
The Value of Disagreement in AI Design, Evaluation, and Alignment
inferred← citeshttps://doi.org/10.1145/3715928.3737470openalex-snapshot
Value-Centered Framing Supports Inclusive Idea Convergence in Participatory Civic Design
inferred← citeshttps://doi.org/10.1145/3772363.3799172openalex-snapshot
Framing the Crowd: How Task Design Shapes Collective Expectation in Crowdsourcing Pedestrian Behavior Change
inferred← citeshttps://doi.org/10.1145/3805689.3806510openalex-snapshot
The Consensus Trap: Dissecting Subjectivity and the “Ground Truth” Illusion in Data Annotation
inferred← citeshttps://doi.org/10.1177/01925121241293459openalex-snapshot
Experts, coders and crowds: An analysis of substitutability
inferred← citeshttps://doi.org/10.3390/electronics15081666openalex-snapshot
Impact of Environmental Control on Subjective Video Quality Assessment in Crowdsourced QoE Experiments
Editorial review for independent work. ICSAC — the Institute for Complexity Science and Advanced Computing — runs a structured editorial panel for researchers publishing outside journals, and publishes on CiteStamp rails. Submission is free and goes through ORCID: icsacinstitute.org/submit.