Documentation

TokenPersistence
in package

FinalYes

Turns a parsed token stream into `sentences` and `word_occurrences` rows, detecting multi-word expressions along the way — all in PHP.

This replaces the old scratch-table pipeline (temp_word_occurrences + tempexprs + numbers + the LOAD DATA path + the stateful @-variable multi-word detection SQL). Sentences are inserted first (to satisfy the FK on word_occurrences), their real SeIDs are read back, and word occurrences — single words and multi-word expressions — are inserted referencing them.

Tags
since
3.2.2

Table of Contents

Constants

CHUNK  : int = 500

Methods

echoCheckValid()  : void
Echo the sentence list and per-word JSON for the check-text preview.
echoStatistics()  : void
Echo the multi-word statistics JSON for the check-text preview.
save()  : void
Save parsed tokens as sentences + word occurrences for a text.
stats()  : array{sentences: int, words: int, unknownPercent: float, preview: string}
Compute preview statistics for the check-text UI (no output).
chunkedInsert()  : void
Execute a multi-row INSERT in chunks of CHUNK rows.
detectMultiWords()  : array<int, array{id: int, sentence: int, order: int, n: int, text: string}>
Detect multi-word expression occurrences in the token stream.
distinctWordLowercase()  : array<int, string>
Distinct lowercased word-token texts.
groupBySentence()  : array<int, array<int, ParsedToken>>
Group tokens by their sentence index, preserving order.
insertSentences()  : array<int, int>
Insert sentences and return a map of local sentence index -> real SeID.
insertWordOccurrences()  : void
Insert word-occurrence rows in chunks.
lc()  : string
Lowercase a string (UTF-8).
multiWordTerms()  : array<int, array<string, array{id: int, text: string, tr: string}>>
Load multi-word terms grouped by word count.
sentenceText()  : string
Concatenate a sentence's token texts.
singleWordTerms()  : array<string, array{id: int, tr: string}>
Load single-word terms for the given lowercased words.

Constants

Methods

echoCheckValid()

Echo the sentence list and per-word JSON for the check-text preview.

public static echoCheckValid(array<string|int, ParsedToken> $tokens, int $lid) : void
Parameters
$tokens : array<string|int, ParsedToken>

Tokens for the whole text

$lid : int

Language ID

echoStatistics()

Echo the multi-word statistics JSON for the check-text preview.

public static echoStatistics(array<string|int, ParsedToken> $tokens, int $lid, bool $rtlScript) : void
Parameters
$tokens : array<string|int, ParsedToken>

Tokens for the whole text

$lid : int

Language ID

$rtlScript : bool

Whether the language is right-to-left

save()

Save parsed tokens as sentences + word occurrences for a text.

public static save(array<string|int, ParsedToken> $tokens, int $lid, int $textId) : void
Parameters
$tokens : array<string|int, ParsedToken>

Tokens for the whole text

$lid : int

Language ID

$textId : int

Text ID

stats()

Compute preview statistics for the check-text UI (no output).

public static stats(array<string|int, ParsedToken> $tokens, int $lid) : array{sentences: int, words: int, unknownPercent: float, preview: string}
Parameters
$tokens : array<string|int, ParsedToken>

Tokens for the whole text

$lid : int

Language ID

Return values
array{sentences: int, words: int, unknownPercent: float, preview: string}

chunkedInsert()

Execute a multi-row INSERT in chunks of CHUNK rows.

private static chunkedInsert(string $prefix, string $rowPlaceholder, int $colsPerRow, array<int, mixed> $params) : void
Parameters
$prefix : string

SQL up to and including "VALUES "

$rowPlaceholder : string

Placeholder for one row, e.g. "(?, ?)"

$colsPerRow : int

Number of columns per row

$params : array<int, mixed>

Flat parameter list

detectMultiWords()

Detect multi-word expression occurrences in the token stream.

private static detectMultiWords(array<int, array<int, ParsedToken>> $bySentence, array<int, array<string, array{id: int, text: string, tr: string}>> $mwTerms) : array<int, array{id: int, sentence: int, order: int, n: int, text: string}>

For each sentence and each known multi-word length n, slide a window of n words and match the concatenated span (words + intervening separators) against the language's n-word terms.

Parameters
$bySentence : array<int, array<int, ParsedToken>>

Tokens by sentence

$mwTerms : array<int, array<string, array{id: int, text: string, tr: string}>>

Multi-word terms

Return values
array<int, array{id: int, sentence: int, order: int, n: int, text: string}>

distinctWordLowercase()

Distinct lowercased word-token texts.

private static distinctWordLowercase(array<string|int, ParsedToken> $tokens) : array<int, string>
Parameters
$tokens : array<string|int, ParsedToken>

Tokens

Return values
array<int, string>

groupBySentence()

Group tokens by their sentence index, preserving order.

private static groupBySentence(array<string|int, ParsedToken> $tokens) : array<int, array<int, ParsedToken>>
Parameters
$tokens : array<string|int, ParsedToken>

Tokens

Return values
array<int, array<int, ParsedToken>>

insertSentences()

Insert sentences and return a map of local sentence index -> real SeID.

private static insertSentences(array<int, array<int, ParsedToken>> $bySentence, int $lid, int $textId) : array<int, int>
Parameters
$bySentence : array<int, array<int, ParsedToken>>

Tokens grouped by sentence

$lid : int

Language ID

$textId : int

Text ID

Return values
array<int, int>

insertWordOccurrences()

Insert word-occurrence rows in chunks.

private static insertWordOccurrences(array<int, array{0: ?int, 1: int, 2: int, 3: int, 4: int, 5: int, 6: string}> $rows) : void
Parameters
$rows : array<int, array{0: ?int, 1: int, 2: int, 3: int, 4: int, 5: int, 6: string}>

Rows

lc()

Lowercase a string (UTF-8).

private static lc(string $s) : string
Parameters
$s : string

Input

Return values
string

multiWordTerms()

Load multi-word terms grouped by word count.

private static multiWordTerms(int $lid) : array<int, array<string, array{id: int, text: string, tr: string}>>
Parameters
$lid : int

Language ID

Return values
array<int, array<string, array{id: int, text: string, tr: string}>>

sentenceText()

Concatenate a sentence's token texts.

private static sentenceText(array<int, ParsedToken> $sTokens) : string
Parameters
$sTokens : array<int, ParsedToken>

Sentence tokens

Return values
string

singleWordTerms()

Load single-word terms for the given lowercased words.

private static singleWordTerms(int $lid, array<int, string> $lowerWords) : array<string, array{id: int, tr: string}>
Parameters
$lid : int

Language ID

$lowerWords : array<int, string>

Distinct lowercased words to look up

Return values
array<string, array{id: int, tr: string}>
On this page

Search results