Documentation

StandardTextParser
in package

Standard text parsing with sentence splitting.

Handles language settings retrieval, text transformations, splitting, previewing, and database insertion for non-Japanese text.

Tags
since
3.0.0

Table of Contents

Methods

applyInitialTransformations()  : string
Apply initial text transformations (before display preview).
applyWordSplitting()  : string
Apply word-splitting transformations (after display preview).
displayStandardPreview()  : void
Display preview HTML for standard text.
echoPreview()  : void
Echo the preview HTML for a (character-substituted) standard text.
getLanguageSettings()  : array{removeSpaces: string, splitSentence: string, noSentenceEnd: string, termchar: string, rtlScript: mixed, splitEachChar: bool}|null
Get language settings for parsing.
splitSentences()  : array<string|int, string>
Split a (character-substituted) standard text into sentences only.
splitStandardSentences()  : array<string|int, string>
Split standard text into sentences (split-only mode).
tokenize()  : array<string|int, ParsedToken>
Tokenize a (character-substituted) standard text into ParsedToken objects.
buildTokenBlob()  : string
Build the tab-delimited token blob from preprocessed text.
parseBlob()  : array<string|int, ParsedToken>
Turn the token blob into ParsedToken objects.
quoteChars()  : string
Build the Unicode quotation-mark character class fragment used in regex patterns.

Methods

applyInitialTransformations()

Apply initial text transformations (before display preview).

public static applyInitialTransformations(string $text, bool $splitEachChar) : string
Parameters
$text : string

Raw text

$splitEachChar : bool

Whether to split each character

Return values
string —

Text after initial transformations

applyWordSplitting()

Apply word-splitting transformations (after display preview).

public static applyWordSplitting(string $text, string $splitSentence, string $noSentenceEnd, string $termchar) : string
Parameters
$text : string

Text after initial transformations

$splitSentence : string

Sentence split regex

$noSentenceEnd : string

Exception patterns

$termchar : string

Word character regex

Tags
psalm-suppress

InvalidReturnType, InvalidReturnStatement, PossiblyNullArgument

Return values
string —

Preprocessed text ready for parsing

displayStandardPreview()

Display preview HTML for standard text.

public static displayStandardPreview(string $text, bool $rtlScript) : void
Parameters
$text : string

Preprocessed text (after initial transformations)

$rtlScript : bool

Whether text is right-to-left

echoPreview()

Echo the preview HTML for a (character-substituted) standard text.

public static echoPreview(string $text, int $lid) : void
Parameters
$text : string

Preprocessed text (character substitutions applied)

$lid : int

Language ID

getLanguageSettings()

Get language settings for parsing.

public static getLanguageSettings(int $lid) : array{removeSpaces: string, splitSentence: string, noSentenceEnd: string, termchar: string, rtlScript: mixed, splitEachChar: bool}|null
Parameters
$lid : int

Language ID

Return values
array{removeSpaces: string, splitSentence: string, noSentenceEnd: string, termchar: string, rtlScript: mixed, splitEachChar: bool}|null —

Language settings or null if not found

splitSentences()

Split a (character-substituted) standard text into sentences only.

public static splitSentences(string $text, int $lid) : array<string|int, string>
Parameters
$text : string

Preprocessed text (character substitutions applied)

$lid : int

Language ID

Tags
psalm-return

non-empty-list

Return values
array<string|int, string>

splitStandardSentences()

Split standard text into sentences (split-only mode).

public static splitStandardSentences(string $text, string $removeSpaces) : array<string|int, string>
Parameters
$text : string

Preprocessed text

$removeSpaces : string

Space removal setting

Tags
psalm-return

non-empty-list

Return values
array<string|int, string> —

Array of sentences

tokenize()

Tokenize a (character-substituted) standard text into ParsedToken objects.

public static tokenize(string $text, int $lid) : array<string|int, ParsedToken>
Parameters
$text : string

Preprocessed text (character substitutions applied)

$lid : int

Language ID

Return values
array<string|int, ParsedToken>

buildTokenBlob()

Build the tab-delimited token blob from preprocessed text.

private static buildTokenBlob(string $text, string $termchar, string $removeSpaces) : string

Produces one line per token as "\t", where a term ending in "\r" marks the end of a sentence. This is the same serialization the old LOAD DATA path consumed; parseBlob() turns it into ParsedToken objects.

Parameters
$text : string

Text after word-splitting transformations

$termchar : string

Word character regex

$removeSpaces : string

Space removal setting

Return values
string

parseBlob()

Turn the token blob into ParsedToken objects.

private static parseBlob(string $blob) : array<string|int, ParsedToken>

Replicates the semantics of the former LOAD DATA SET clause: each line is "\t"; a term ending with "\r" ends the current sentence (that token still belongs to the current sentence, the next one starts a new sentence). Order is a global 1-based counter.

Unlike the old saveWithSqlFallback(), this does NOT trim the line, so the "\r" sentence markers and trailing-space tokens are preserved (that bug caused LOAD-DATA-less installs to parse every text as one sentence).

Parameters
$blob : string

Token blob from buildTokenBlob()

Return values
array<string|int, ParsedToken>

quoteChars()

Build the Unicode quotation-mark character class fragment used in regex patterns.

private static quoteChars() : string

Contains: RIGHT DOUBLE QUOTE, close-paren, LEFT/RIGHT SINGLE QUOTE, single angle quotes, LEFT DOUBLE QUOTE, DOUBLE LOW-9 QUOTE, guillemets, CJK brackets.

Return values
string —

Character class content (without surrounding brackets)

On this page

Search results