TextParsing
in package
Text parsing and processing utilities (facade).
Delegates tokenization to JapaneseTextParser / StandardTextParser and persistence to TokenPersistence. Parsing happens entirely in PHP — there are no scratch tables involved.
Tags
Table of Contents
Methods
- checkText() : array{sentences: int, words: int, unknownPercent: float, preview: string}
- Check/preview text and return parsing statistics without saving.
- parseAndDisplayPreview() : void
- Parse text and display preview HTML for validation.
- parseAndSave() : void
- Parse text and save to database.
- splitIntoSentences() : array<string|int, string>
- Split text into sentences without database operations.
- preprocess() : array{0: string, 1: bool}|null
- Apply the language's text preprocessing (escaping + character substitutions) and report whether it uses the MeCab parser.
- tokenize() : array<string|int, ParsedToken>
- Tokenize a text into ParsedToken objects (no database writes).
Methods
checkText()
Check/preview text and return parsing statistics without saving.
public
static checkText(string $text, int $lid) : array{sentences: int, words: int, unknownPercent: float, preview: string}
Does not output any HTML or save to database.
Parameters
- $text : string
-
Text to parse
- $lid : int
-
Language ID
Return values
array{sentences: int, words: int, unknownPercent: float, preview: string}parseAndDisplayPreview()
Parse text and display preview HTML for validation.
public
static parseAndDisplayPreview(string $text, int $lid) : void
Outputs HTML directly to show parsed sentences and word statistics.
Parameters
- $text : string
-
Text to parse
- $lid : int
-
Language ID
parseAndSave()
Parse text and save to database.
public
static parseAndSave(string $text, int $lid, int $textId) : void
Parameters
- $text : string
-
Text to parse
- $lid : int
-
Language ID
- $textId : int
-
Text ID (must be positive)
Tags
splitIntoSentences()
Split text into sentences without database operations.
public
static splitIntoSentences(string $text, int $lid) : array<string|int, string>
Parameters
- $text : string
-
Text to parse
- $lid : int
-
Language ID
Tags
Return values
array<string|int, string> —Array of sentences
preprocess()
Apply the language's text preprocessing (escaping + character substitutions) and report whether it uses the MeCab parser.
private
static preprocess(string $text, int $lid) : array{0: string, 1: bool}|null
Parameters
- $text : string
-
Raw text
- $lid : int
-
Language ID
Return values
array{0: string, 1: bool}|null —[preprocessed text, isMecab] or null if language missing
tokenize()
Tokenize a text into ParsedToken objects (no database writes).
private
static tokenize(string $text, int $lid) : array<string|int, ParsedToken>
Parameters
- $text : string
-
Text to parse
- $lid : int
-
Language ID