JapaneseTextParser
in package
Japanese text parsing using MeCab.
Handles splitting, previewing, and tokenization for Japanese text. Produces ParsedToken objects consumed by TokenPersistence — no scratch table involved.
Tags
Table of Contents
Methods
- buildTokensFromMecab() : array<string|int, ParsedToken>
- Convert raw MeCab output into ParsedToken objects.
- displayJapanesePreview() : void
- Display preview HTML for Japanese text.
- splitJapaneseSentences() : array<string|int, string>
- Split Japanese text into sentences (split-only mode).
- tokenize() : array<string|int, ParsedToken>
- Tokenize Japanese text with MeCab into ParsedToken objects.
Methods
buildTokensFromMecab()
Convert raw MeCab output into ParsedToken objects.
public
static buildTokensFromMecab(string $mecabed) : array<string|int, ParsedToken>
Kept separate from the MeCab subprocess call so it can be unit-tested with captured/synthetic MeCab output. Mirrors the former SQL pipeline: build staging rows, drop the final EOP order group, then group rows sharing a TiOrder into one token (text concatenated in TiCount order).
Parameters
- $mecabed : string
-
Raw MeCab output ("word\ttype\tnode" per line)
Return values
array<string|int, ParsedToken>displayJapanesePreview()
Display preview HTML for Japanese text.
public
static displayJapanesePreview(string $text) : void
Parameters
- $text : string
-
Preprocessed text
splitJapaneseSentences()
Split Japanese text into sentences (split-only mode).
public
static splitJapaneseSentences(string $text) : array<string|int, string>
Parameters
- $text : string
-
Preprocessed text
Tags
Return values
array<string|int, string> —Array of sentences
tokenize()
Tokenize Japanese text with MeCab into ParsedToken objects.
public
static tokenize(string $text) : array<string|int, ParsedToken>
Parameters
- $text : string
-
Preprocessed text (character substitutions applied)