On the public citation graph

Multilingual Tokenization Efficiency in Large Language Models: A Study on Indian Languages

https://doi.org/10.2139/ssrn.5201887

0 signed · 14 inferred · view as graph

Inferred edges come from two machine sources: sweeps of open repository metadata (mostly SSRN and Zenodo), and a bulk OpenAlex citation snapshot. Every row below is labelled with the one it came from. Neither is a complete citation index — absence here is not evidence of absence.

Is this your work? Sign in with ORCID and put your name on the claims that are yours.

Claim with ORCID

Reference health. See whether every reference in this work resolves to a known work and none has been retracted — and get a dated CiteStamped badge you can embed.

See reference health

What this work says

Outgoing claims: the papers this work cites, builds on, or argues with.

inferredcites →https://doi.org/10.1109/icassp.2012.6289079openalex-snapshot

Japanese and Korean voice search

inferredcites →https://doi.org/10.1145/3578707openalex-snapshot

Impact of Tokenization on Language Models: An Analysis for Turkish

inferredcites →https://doi.org/10.1162/tacl_a_00448openalex-snapshot

Canine : Pre-training an Efficient Tokenization-Free Encoder for Language Representation

inferredcites →https://doi.org/10.1162/tacl_a_00461openalex-snapshot

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models

inferredcites →https://doi.org/10.1609/aaai.v34i05.6451openalex-snapshot

Neural Machine Translation with Byte-Level Subwords

inferredcites →https://doi.org/10.18653/v1/2021.acl-long.243openalex-snapshot

How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

inferredcites →https://doi.org/10.18653/v1/2022.emnlp-main.616openalex-snapshot

Few-shot Learning with Multilingual Generative Language Models

inferredcites →https://doi.org/10.18653/v1/d18-2012openalex-snapshot

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

inferredcites →https://doi.org/10.18653/v1/p18-1007openalex-snapshot

Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

inferredcites →https://doi.org/10.3929/ethz-b-000446004openalex-snapshot

Character-Level Translation with Self-attention

inferredcites →https://doi.org/10.48550/arxiv.2005.14165openalex-snapshot

Language Models are Few-Shot Learners

inferredcites →https://doi.org/10.48550/arxiv.2106.12672openalex-snapshot

Charformer: Fast Character Transformers via Gradient-based Subword Tokenization

inferredcites →https://doi.org/10.48550/arxiv.2204.14268openalex-snapshot

How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training?

inferredcites →https://doi.org/10.48550/arxiv.2308.00683openalex-snapshot

CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code

Editorial review for independent work. ICSAC — the Institute for Complexity Science and Advanced Computing — runs a structured editorial panel for researchers publishing outside journals, and publishes on CiteStamp rails. Submission is free and goes through ORCID: icsacinstitute.org/submit.