Introduction
1.1 Characteristics and Challenges of Japanese NLP
Japanese presents unique challenges in natural language processing. Its main characteristics and challenges are as follows.
Morpheme boundary ambiguity: Unlike English, Japanese text does not place spaces between words, making "word segmentation" — cutting individual words (morphemes) out of text — a necessity. This morphological analysis is the most fundamental and important processing step in Japanese NLP, and affects the accuracy of all downstream NLP tasks.
Diversity of character types: Japanese uses multiple writing systems simultaneously, including hiragana, katakana, kanji, Latin characters, and numerals. This complicates preprocessing such as character normalization, conversion, and encoding.
Diversity of honorifics and writing styles: The complexity of the Japanese honorific system and the distinction between formal (desu/masu) and plain (da/de aru) styles also poses challenges for Japanese NLP.
Flexible word order: Japanese follows a basic SOV (Subject-Object-Verb) order, but because grammatical relationships are indicated by particles, word order is relatively free, making syntactic parsing difficult.
Polysemy and multiple readings of kanji: The same kanji character often has multiple readings and meanings, requiring reading disambiguation based on context (resolution of homographic ambiguity).
1.2 Objectives and Contributions of This Survey
The objective of this survey is to comprehensively investigate and classify the open-source resources available in the field of Japanese NLP, and to make the following contributions to researchers and practitioners:
- Provision of a comprehensive resource catalog: Systematic classification of 898 GitHub repositories and 497 Hugging Face repositories
- Quantitative comparative analysis: Evaluation of the popularity and activity level of each tool based on download counts, Star counts, and update frequency
- Analysis of technology trends: An overview of the technological transition from rule-based methods to deep learning and further to LLMs
- Overview of multilingual implementations: Comparison of implementations in Python, C++, Rust, Go, Java, JavaScript, and more
- Practical selection guidelines: Providing a basis for deciding which tool to choose for each task
1.3 Survey Methodology
This survey used the awesome-japanese-nlp-resources repository (CC0-1.0 license) maintained by taishi-i as its foundational data. As of November 17, 2026, the repository contains information on 898 GitHub repositories and 497 Hugging Face repositories, and provides the following quantitative information for each:
- Weekly download count (downloads/week)
- Total download count (total downloads)
- GitHub Star count
- Date of last commit
This survey uses this data while also analyzing the technical characteristics, design philosophy, and usage scenarios of representative tools in each category.
1.4 Data Sources
This site was created based on information from taishi-i/awesome-japanese-nlp-resources.
Recent Updates
As of November 17, 2026. See the linked sections for full details on each topic.
HuggingFace Model Trends
Speech recognition (ASR): wav2vec2 jumps to 16M and stays on top
wav2vec2-large-xlsr-53-japanese grew sharply from 2M to 16M downloads and remains #1 among all HF models. wav2vec2-large-xlsr-japanese-hiragana, which outputs hiragana, also surged from 14k to 153k (#12). Meanwhile kotoba-whisper-v2.2 went from 221k to 16k (⭐134) and anime-whisper from 37k to 33k (⭐165). → 13.1 Speech Recognition
LLMs: llm-jp-4.1 reasoning models and GGUF builds grow
japanese-gpt-neox-small (rinna) rose from 242k to 656k and is now #2 overall. The newly released llm-jp-4.1-8b-thinking (80k, ⭐18) and the GGUF build llm-jp-4-33b-thinking-gguf (271k, #10) are growing, while last time's leaders llm-jp-4-8b-thinking (21k → 3k) and LFM2.5-1.2B-JP-202606 (20k → 4k) have settled. A community-modified GGUF model with refusals suppressed, Spark-X2.5-4B-Heretic-jp-gguf, also reached 589k (#4), pointing to wider use of GGUF for local inference. → 10.1 LLM Development Projects
Sentence embeddings: PLaMo-Embedding-1B surges to 486k
ruri-v3-310m went from 782k to 634k (⭐89, #3) and remains the top sentence-embedding model. plamo-embedding-1b (Preferred Networks) jumped from 21k to 486k (#6), while GLuCoSE-base-ja-v2 fell from 205k to 6k. → 14.1 Sentence Embedding Models
Reranking: japanese-reranker-cross-encoder-small-v1 retakes the lead
japanese-reranker-cross-encoder-small-v1 (235k → 156k, #11) and ruri-v3-reranker-310m (286k → 137k, #13) swapped places again at the top of the text-ranking category. → 14.3 Text Reranking
BERT and classification: whole-word-masking leads fill-mask at 462k
bert-base-japanese-whole-word-masking grew from 388k to 462k (#7) and now leads the fill-mask category. Last time's #2 overall, japanese-roberta-base (rinna), fell from 2M to 9k, and deberta-v2-large-japanese-char-wwm from 480k to 76k. In text-classification, bert-finetuned-japanese-sentiment surged from 24k to 329k (#8). → 9.2 BERT-family Models, 5.3 BERT Fine-tuning Approaches
Spoken dialogue and translation: LFM2.5-Audio at #5, OPUS-MT leads translation
LFM2.5-Audio-1.5B-JP-GGUF (Liquid AI), which handles speech recognition and speech synthesis in a single model, reached 506k (#5). In translation, vntl-llama3-8b-v2-gguf fell from 714k to 20k and Sugoi-14B-Ultra-GGUF from 172k to 10k, leaving opus-mt-ja-en (37k, ⭐77) at the top of the category. → 13.1 Speech Recognition, 8.3 Translation Evaluation
Images and OCR: manga-ocr-base at 299k, #9
manga-ocr-base went from 424k to 299k (⭐192, #9), and meiki.txt.recognition.v0 for game screens has settled at 42k (from 45k). → 7.1 Manga OCR
New Repository Additions (August–November 2026)
Agent skills that polish Japanese prose
yomiyasu (nanaism, ⭐3,300) is an agent skill that revises AI-generated Japanese into natural Japanese, and natural-japanese (coji, ⭐2,100) is an Agent Skill for writing and fixing work Japanese so that it reads clearly. suiko (nwiizo, ⭐114) is a Rust CLI that reproducibly diagnoses how natural and readable a Japanese document is; tools that deal with the "quality of Japanese" keep arriving in the LLM era. → 10.2 ChatGPT/LLM Applications
Manga translation and OCR
koharu (koharu-rs, ⭐6,700) is an automatic manga translation tool written in Rust that uses LLMs. ndlocr-lite (National Diet Library, ⭐1,400) is the application for the lightweight NDLOCR, and Kindai-OCR (deepapps91, ⭐153) is an OCR for modern-era Japanese magazines. → 7. OCR, 8. Machine Translation
Dictionaries, input, and moved repositories
Meltype (yksr-melt, ⭐1,600) is a Windows resident tool that lets you type Japanese and English without pressing the half-width/full-width key, and jmdict-yomitan (yomidevs, ⭐332) provides JMdict, JMnedict, and KANJIDIC dictionaries for Yomitan. fcitx5-hazkey has been renamed hazkey (⭐254), and the morphological analyzer lindera has moved to lindera/lindera (⭐708). → 11.3 Input Methods (IME), 2.5 Comparison of Multilingual Implementations
Speech synthesis
AivisSpeech Engine (aivis-project, ⭐186) is an AI speech synthesis engine, and sanoTTS-jp (ayutaz, ⭐71) synthesizes Japanese speech in real time on an ESP32-S3 with a 559K-parameter TTS model. Among multilingual tools, mlx-audio (⭐8,000), a speech synthesis and recognition library for Apple Silicon, was also added. → 13.2 Text-to-Speech (TTS)
Evaluation benchmarks
New additions include JFBench and pfgen-bench (Preferred Networks) for instruction following, J-tau-bench (SB Intuitions) for tool-using agents, J-MeetEval (Nishika) for meeting tasks, the linguistic acceptability corpus JCoLA (osekilab), and the financial text embedding benchmark JFinTEB (retarfi). → 10.3 Evaluation Benchmarks
Hugging Face: llm-jp-4.1 and a large pretraining corpus
NII LLM-jp released the llm-jp-4.1 reasoning models (llm-jp-4.1-8b-thinking, llm-jp-4.1-33b-thinking and others, each with a GGUF build) and the vision-language model llm-jp-4-vl-9b. Among datasets, ClimbLab-Ja (15k), a 300-billion-token Japanese pretraining corpus filtered from LLM-jp Corpus v4, was added, and in speech recognition Haas-ja-v1.0 (⭐79). → 10.1 LLM Development Projects, 12.6 LLM Training Corpora
New Repository Additions (July 28, 2026)
SDKs for Japanese Data APIs and PII Masking
shirabe-sdk (techwell-inc-jp) and shirabe-sdk-python are the official SDKs for the Shirabe Japan data APIs, covering Japanese name splitting and reading estimation (with JMnedict-backed candidates), address normalization, corporate number lookup, and the rokuyo calendar. They ship with ready-made tool definitions for LangChain, the OpenAI Agents SDK, and the Vercel AI SDK, targeting LLM agents that need Japan-specific data processing. fuseji (sserada) is Japanese-specific PII detection and masking middleware for LLM observability. → 4.2 Text Normalization, 10.2 ChatGPT/LLM Applications
Dictionary Data and Prompt Compression
JMdictSQLite (seanmcbroom) provides JMdict and Kanjidic as a SQLite database with automatic daily updates, making Japanese-English dictionary data directly queryable from SQL. genshijin (interfacex-co-jp, ⭐263) is an ultra-compressed communication skill for AI agents such as Claude Code and Codex; based on the Japanese version of caveman, it is optimized for stripping redundant expressions specific to Japanese. → 11.2 Electronic Dictionaries, 4.1 Character Conversion
New Repository Additions (June 30, 2026)
LLM-powered Japanese Input Methods Emerge
rakukan (fukuyori) is a Windows Japanese IME powered by a local LLM, and Sumibi (kiyoka) is a Japanese input method powered by the ChatGPT API — signaling a new trend of applying large language models to kana-kanji conversion. Violet/japanesekeyboard (kazumaproject) additionally offers a fully offline Japanese keyboard app. → 11.3 Input Method (IME)
Expansion of Medical-domain Speech & Evaluation Resources
medvoice-jp-asr (nikotora) releases a whisper-small + LoRA ASR model specialized for Japanese medical speech along with a first-of-its-kind domestic medical ASR benchmark, and medLLM_QA_benchmark (AIST) is a trilingual (English/Japanese/Chinese) QA benchmark for medical LLMs. → 13.1 Automatic Speech Recognition, 10.3 Evaluation Benchmarks
Foundations for Spoken Dialogue & Multimodal Evaluation
moshi-finetune (Nagoya U. Dialogue Lab) fine-tunes Moshi/J-Moshi on custom spoken-dialogue data, simple-evals-mm (llm-jp) is a lightweight library for evaluating vision-language models on English and Japanese benchmarks, and jvs_nonpara_kana (CyberAgent AI Lab) provides katakana annotations of the JVS nonpara corpus for G2P evaluation. → 13. Speech Processing, 15.1 VLMs
Expansion of the Rust Morphological-analysis Ecosystem
mecab-dic-converter (tomokane) is a Rust crate that reads, parses, and reconstructs compiled MeCab dictionaries into dictionaries usable by MeCab-compatible tokenizers such as Vibrato and Lindera, easing migration of existing MeCab assets into the Rust ecosystem. → 2.5 Other-language Implementations of Morphological Analysis
New Repository Additions (June 7, 2026)
LLM-powered Japanese Novel Writing Tool
novel2hermes_jp (kgmkm, ⭐83) is a skills.md for planning/plotting/writing novels in Japanese using the powerful memory-capable hermes-agent and the Japanese-search-optimized external memory vecmemori. → 10.2 ChatGPT/LLM Applications
azooKey Repositories Migrate to Official Organization
azooKey (iOS, ⭐710), azooKey-Desktop (macOS, ⭐927), and AzooKeyKanaKanjiConverter (⭐148) have moved from the personal ensan-hcl account to the official azooKey organization. URLs have been updated accordingly. → 11.3 Input Method (IME)
New Repository Additions (June 2, 2026)
Translation Evaluation Benchmark Added
jp-tl-bench (shisa-ai) is an anchored pairwise LLM evaluation framework for bidirectional Japanese-English translation quality assessment. → 8. Machine Translation
New Repository Additions (May 2026)
Concentrated Addition of LLM Evaluation Benchmarks & Fact-Checking Datasets
On May 13, 2026, a cluster of LLM evaluation benchmarks and datasets were added. japanese-llm-benchmark (shi3z) is a general-purpose benchmark tool for evaluating various LLMs' Japanese capabilities; JTruthfulQA (Waseda NLP) is a Japanese truthfulness evaluation dataset built from scratch (not translated from TruthfulQA, Lin+ 2022). LCTG-Bench (CyberAgentAILab) benchmarks LLM-controlled text generation; EDINET-Bench (SakanaAI, ICLR 2026) evaluates LLM performance on challenging Japanese financial tasks; japanese-dataset-for-automated-fact-checking (FujitsuResearch) is a Japanese dataset for automated fact-checking (JAD-AFC); and llm-jp-longbench (llm-jp) is a Japanese version of LongBench leveraging the jemhop dataset. → 10.3 Evaluation Benchmarks, 12.4 QA & Reasoning Corpora
Expansion of Multimodal and Speech Datasets
LookVQA (riken-grp, LREC-COLING 2024) is a gaze-grounded VQA dataset for clarifying ambiguous Japanese questions, and Kokoro-Speech-Dataset (kaiidams) is a public-domain single-speaker Japanese speech dataset. Open resources for multimodal and speech research continue to expand. → 15. Multimodal Processing, 13. Speech Processing
Rust Morphological Analyzer Extension
lindera-sqlite (lindera) integrates Lindera as a SQLite FTS5 extension, allowing Japanese morphological analysis and full-text search to be embedded directly in databases. → 2.5 Other Language Implementations
New Repository Additions (March–April 2026)
Multimodal and LLM Evaluation Benchmarks
Several evaluation benchmarks targeting vision-language models (VLMs) and long-context LLMs were added. eval_vertical_ja (llm-jp, added 2026-03-08) evaluates multimodal LLM performance on vertically written Japanese text. modelvista-3lang (kuramitsulab, 2026-03-22) benchmarks software diagram understanding in three languages (Japanese, English, Korean). business-slide-questions (Stockmark) is a VQA benchmark (BusinessSlideVQA) for business presentation slides. japanese-hr-niah (kufu) uses the Needle-In-A-Haystack (NIAH) format to measure long-context processing in the HR and labor domain. → 15.1 VLMs, 10.3 Evaluation Benchmarks
Expansion of Dialogue Corpora
Several dialogue corpora capturing distinctly Japanese cultural and linguistic phenomena were added. HOTATE (Ehime University NLP Group) is a dialogue dataset incorporating honne (true feelings) and tatemae (public facade), concepts central to Japanese social interaction. ETCDataset (UEC Inaba Lab) contains approximately 1,000 dialogues where each speaker writes their own emotional state during the conversation. diafill-toolkit (SB Intuitions) is a Python toolkit for synthesizing natural short-utterance dialogue rich in fillers (e.g., "eto", "ano") using LLMs. JMultiWOZ-TC (llm-jp) provides agent function-calling evaluation in multi-turn dialogue, and Novel2DialCorpus (ganbon) offers a method for constructing casual dialogue corpora from novel text. → 12.3 Dialogue Corpora
Domain-Specific Evaluation Datasets
JMedWiC (Ehime University NLP Group) is a word-sense disambiguation dataset for the Japanese medical domain. JBE-QA (hancules) covers Japanese bar exam Q&A. JSTS-Neg (reiko-y) is a semantic similarity dataset specifically designed to evaluate negation understanding. Doppelganger-JC (0017-alt) benchmarks cross-lingual homograph misuse between Chinese and Japanese in LLMs — testing whether models correctly distinguish words that look identical but have different meanings (e.g., 手紙 means "letter" in Japanese but "toilet paper" in Chinese). E2U (sociocom) provides data for converting text into more accessible forms for readers with cognitive or linguistic barriers. → 12.4 QA & Reasoning Corpora, 10.3 Evaluation Benchmarks
Digitization of Classical and Historical Materials
nijl-manyoshutei (National Institute of Japanese Literature, added 2026-03-29) published TEI/XML data of the Hirose-hon Man'yoshu held at Kansai University under a CC-BY license. jhpt (nict-astrec-att) provides a segment-aligned parallel corpus of historical Japanese texts with modern Japanese translations. The steady growth of digitized classical Japanese resources is expanding the foundation for classical-era NLP research. → 12.1 Corpora, 12.2 Parallel Corpora
Dictionary and IME Additions
kamuskita (matbahasa, added 2026-04-06) is an open Malay-Japanese dictionary created by a Malay language study group. dvorakjp-roman-table (shinespark, added 2026-04-12) is a DvorakJP roman table for azooKey and Google Japanese Input, supporting Japanese typing for non-QWERTY keyboard users. → 11.2 Electronic Dictionaries, 11.4 Specialized Dictionaries & IME Dictionaries
LLM Quantization Tools
OneCompression (FujitsuResearch, added 2026-04-21, ⭐398) is a post-training quantization (PTQ) pipeline for LLMs developed by Fujitsu Laboratories. It integrates QEP (NeurIPS 2025), ILP mixed precision, rotation preprocessing, and a vLLM plugin into a single comprehensive framework for efficient local LLM inference. → 10.1 LLM Development Projects