From 0e986341d48347642685bb2c85287a01ffa1480c Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Sun, 21 Jun 2026 14:30:33 -0400 Subject: [PATCH 01/13] OPENNLP-1850 Document Unicode normalization, the UAX #29 tokenizer, and DL handling Add the Text Normalization manual chapter (CharClass engine, normalizer pipeline, the Term model, and the Aligned offset variants that return an AlignedText carrying an Alignment), extend the tokenizer chapter with the UAX #29 segmenter, and document the DL components' Unicode-aware chunking and opt-in whitespace/dash folding with offset-safe findInOriginal. All embedded ONNX snippets are self-contained and compile. --- opennlp-docs/src/docbkx/doccat.xml | 21 +- opennlp-docs/src/docbkx/introduction.xml | 3 +- opennlp-docs/src/docbkx/namefinder.xml | 46 +- opennlp-docs/src/docbkx/normalizer.xml | 536 +++++++++++++++++++++++ opennlp-docs/src/docbkx/opennlp.xml | 1 + opennlp-docs/src/docbkx/tokenizer.xml | 91 +++- 6 files changed, 691 insertions(+), 7 deletions(-) create mode 100644 opennlp-docs/src/docbkx/normalizer.xml diff --git a/opennlp-docs/src/docbkx/doccat.xml b/opennlp-docs/src/docbkx/doccat.xml index 0d45acd4b3..cbbca0330d 100644 --- a/opennlp-docs/src/docbkx/doccat.xml +++ b/opennlp-docs/src/docbkx/doccat.xml @@ -165,12 +165,31 @@ String bestCategory = categorizer.getBestCategory(outcomes);]]> File vocab = new File("/path/to/vocab.txt"); Map categories = new HashMap<>(); String[] inputText = new String[]{"My input text is great."}; -final DocumentCategorizerDL myCategorizer = new DocumentCategorizerDL(model, vocab, categories); +final DocumentCategorizerDL myCategorizer = new DocumentCategorizerDL( + model, vocab, categories, new AverageClassificationScoringStrategy(), new InferenceOptions()); double[] outcomes = myCategorizer.categorize(inputText); String category = myCategorizer.getBestCategory(outcomes);]]> For additional examples, refer to the DocumentCategorizerDLEval class. + + Like NameFinderDL, long input is split into overlapping chunks on the full + Unicode White_Space set rather than Java's \s, so text copied + from PDFs, the web, or multilingual sources tokenizes consistently. Optional + preprocessing through InferenceOptions is off by default: + setNormalizeWhitespace(true) maps each Unicode whitespace code point to an + ASCII space, and setNormalizeDashes(true) maps Unicode dashes to the ASCII + hyphen-minus. Both are one-to-one replacements that preserve character offsets. See + for the shared CharClass engine and the + full normalization library. + + + + diff --git a/opennlp-docs/src/docbkx/introduction.xml b/opennlp-docs/src/docbkx/introduction.xml index a96346b873..51afe02a7f 100644 --- a/opennlp-docs/src/docbkx/introduction.xml +++ b/opennlp-docs/src/docbkx/introduction.xml @@ -303,7 +303,8 @@ Arguments description: and Document Categorizer. This allows models trained by other frameworks such as PyTorch and Tensorflow to be used by OpenNLP. The documentation for each of the OpenNLP components that supports ONNX models describes how to - use ONNX models for inference. + use ONNX models for inference. DL inference uses Unicode-aware text chunking and + optional input normalization; see . diff --git a/opennlp-docs/src/docbkx/namefinder.xml b/opennlp-docs/src/docbkx/namefinder.xml index 81991bb6c7..08eb9780b6 100644 --- a/opennlp-docs/src/docbkx/namefinder.xml +++ b/opennlp-docs/src/docbkx/namefinder.xml @@ -155,13 +155,51 @@ Span[] nameSpans = nameFinder.find(sentence);]]> categories = new HashMap<>(); -String[] tokens = new String[]{"George", "Washington", "was", "president", "of", "the", "United", "States", "."}; -NameFinderDL nameFinderDL = new NameFinderDL(model, vocab, false, getIds2Labels()); -Span[] spans = nameFinderDL.find(tokens);]]> +// Maps the model's output indices to its BIO labels, e.g. "O", "B-PER", "I-PER". +Map ids2Labels = new HashMap<>(); +SentenceDetector sentenceDetector = + new SentenceDetectorME(new SentenceModel(new File("/path/to/en-sent.bin"))); +String[] tokens = {"George", "Washington", "was", "president", "of", "the", "United", "States", "."}; +NameFinderDL nameFinderDL = new NameFinderDL(model, vocab, ids2Labels, sentenceDetector); +// findInOriginal returns spans in the original input's coordinates. +Span[] spans = nameFinderDL.findInOriginal(tokens);]]> For additional examples, refer to the NameFinderDLEval class. + + Long input text is split into overlapping chunks on the full Unicode + White_Space set before WordPiece tokenization, so spacing such as a + no-break space or the CJK ideographic space is recognized as a delimiter. After + inference, reconstructed entity text is matched back to the caller's original input + with a Unicode-aware cursor scan (not a regular expression), so + Span#getCoveredText(...) returns the source text even when WordPiece + rejoins sub-tokens with spaces or when the source uses non-ASCII whitespace between + tokens. + + + Optional preprocessing of the joined input text is available through + InferenceOptions and is off by default: + setNormalizeWhitespace(true) folds each Unicode whitespace character to + an ASCII space, and setNormalizeDashes(true) folds Unicode dashes to the + ASCII hyphen-minus. Both transforms are one code point to one character and preserve + offsets. Full details, the underlying CharClass engine, and the broader + normalization pipeline are documented in . + + + ids2Labels = new HashMap<>(); // the model's BIO labels +SentenceDetector sentenceDetector = + new SentenceDetectorME(new SentenceModel(new File("/path/to/en-sent.bin"))); +String[] tokens = {"George", "Washington", "was", "president", "."}; + +InferenceOptions options = new InferenceOptions(); +options.setNormalizeWhitespace(true); +options.setNormalizeDashes(true); +NameFinderDL finder = new NameFinderDL(model, vocab, ids2Labels, options, sentenceDetector); +// findInOriginal maps spans back to the original input even when a fold changes its length. +Span[] spans = finder.findInOriginal(tokens);]]> + diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml new file mode 100644 index 0000000000..b96f7ed154 --- /dev/null +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -0,0 +1,536 @@ + + + + + + + Text Normalization + +
+ Introduction + + The package opennlp.tools.util.normalizer provides Unicode-aware text + normalization for matching, search, and tokenization preprocessing. It cleans up the + kinds of inconsistency that real text carries when it is copied from the web, PDFs, + office documents, or multilingual sources: spacing that is not an ordinary space, the + many dash and quotation variants, decomposed versus precomposed accents, non-ASCII + digits, and invisible control characters. + + + The implementation follows three principles: + + + + + Standards-sourced. Membership sets come from the + Unicode Character Database (for example the White_Space and + Dash properties), not from the JVM's locale-dependent or quirky + character predicates. The library never relies on + Character.isWhitespace, which disagrees with the Unicode standard. + + + + + Cursor-based, no regular expressions. Every + operation is a single forward pass over the input that tests membership in O(1) + and advances by code point. This avoids the allocation and the catastrophic + backtracking (ReDoS) risk of regular expressions, and it correctly recognizes + Unicode characters that Java's \s does not. + + + + + Offset-preserving. The original text is always + the source of truth. Normalization produces a derived form for matching while the + original character offsets are kept, so a search hit can be reported and + highlighted against the source even when the normalized form has a different + length. + + + + + Two engines underpin everything: the CharSequenceNormalizer family offers + ready-made, composable normalizers, and the CharClass engine is the low-level, + configurable building block they are made of. Built on these are three higher-level + features documented below: a layered term model that projects a token through a + configurable stack of transforms while keeping every intermediate form (see + ), per-language profiles that select the transforms + appropriate to a language (see ), and confusable + folding that reduces lookalike characters for matching (see + ). + +
+ +
+ The normalizer family + + Each normalizer implements the existing + opennlp.tools.util.normalizer.CharSequenceNormalizer interface + (CharSequence normalize(CharSequence)) and is a shared, stateless singleton + obtained through getInstance(). They can therefore be combined with the + existing AggregateCharSequenceNormalizer, or with the + TextNormalizer builder described below. + + + + + + + Normalizer + Effect + + + + + WhitespaceCharSequenceNormalizer + Collapses each run of Unicode whitespace to a single ASCII space and + trims the edges. + + + DashCharSequenceNormalizer + Maps every Unicode dash to the ASCII hyphen-minus. The mathematical + minus signs and the soft hyphen are not affected. + + + QuoteCharSequenceNormalizer + Folds typographic single quotes and apostrophes to ' and + double quotes (including guillemets) to ". + + + DigitCharSequenceNormalizer + Maps Unicode decimal digits (Arabic-Indic, Devanagari, fullwidth, ...) + to ASCII 0-9 by their numeric value. + + + EllipsisCharSequenceNormalizer + Expands the horizontal ellipsis to ... and the two-dot + leader to .. + + + BulletCharSequenceNormalizer + Replaces unambiguous list bullets with a space; the Catalan middle dot + is left alone. + + + InvisibleCharSequenceNormalizer + Removes invisible format and bidirectional control characters (BOM, + zero width space, bidi marks/overrides/isolates, ...). The zero width + joiner and non-joiner and variation selectors are kept. + + + NfcCharSequenceNormalizer + Applies Unicode Normalization Form C (canonical composition); a safe, + lossless baseline for matching. + + + NfkcCharSequenceNormalizer + Applies Unicode Normalization Form KC (compatibility composition); + folds fullwidth forms, ligatures, and super/subscripts. + + + CaseFoldCharSequenceNormalizer + Lower cases for case-insensitive matching, using + Locale.ROOT. + + + AccentFoldCharSequenceNormalizer + Folds diacritics in a script-aware way (see below). + + + GermanUmlautCharSequenceNormalizer + Transliterates German umlauts and the eszett (a-umlaut to ae, + eszett to ss; DIN 5007-2). + + + ConfusableSkeletonCharSequenceNormalizer + Reduces lookalike characters to a confusable skeleton for matching + (UTS #39); see below. + + + + + + + A single normalizer is applied directly: + + + + +
+ +
+ Composing a pipeline + + TextNormalizer is a fluent builder that composes the rungs, in the order + they are added, into a single CharSequenceNormalizer: + + + + + + A conservative search-oriented chain (strip invisibles, NFC, collapse whitespace, fold + quotes and dashes, case fold, then script-gated accent fold) is available directly: + + + + + + Any custom CharSequenceNormalizer can be inserted with + with(...). The TextNormalizer pipeline and the individual + CharSequenceNormalizer implementations are not applied automatically by + statistical OpenNLP components; callers compose them explicitly when preprocessing text + for search or matching. The DL components described in the next section use a narrower, + built-in subset of this machinery. + +
+ +
+ Use in DL components + + NameFinderDL and DocumentCategorizerDL share Unicode-aware text + handling through AbstractDL. Long inputs are split into overlapping chunks + on the full Unicode White_Space set (no-break space, ideographic space, line + and paragraph separators, and the other members listed under + ), not on Java's six-character + \s subset. Empty tokens from leading, trailing, or repeated whitespace are + not produced. + + + NameFinderDL additionally locates reconstructed entity text in the original + input with a cursor-based matcher: a space in the reconstructed span matches zero or more + Unicode whitespace code points in the source, and every other code point is compared + case-insensitively. This replaces the previous regular-expression approach and correctly + handles spacing copied from PDFs, the web, or non-Latin sources when resolving + Span#getCoveredText(...). + + + Optional input folding is controlled through InferenceOptions and is + off by default so existing models keep their prior inputs unless + you opt in: + + + + + setNormalizeWhitespace(true) maps each Unicode whitespace code point + to a single ASCII space before inference. The transform is one code point to one + space, so character offsets stay aligned with the input. + + + + + setNormalizeDashes(true) maps each dash in the default + CharClass.dashes() set to the ASCII hyphen-minus. Mathematical minus + signs and the soft hyphen are not affected unless you extend the set explicitly. + This replacement is also one code point to one character for Basic Multilingual + Plane dashes. + + + + + Run-collapsing normalization (for example WhitespaceCharSequenceNormalizer, + which collapses whitespace runs to a single space) is not enabled + through these flags because it would shift character offsets. Use the + CharSequenceNormalizer pipeline directly when you need that behavior on text + that does not require offset-preserving span lookup. See also + and + . + + + ASCII space +options.setNormalizeDashes(true); // opt-in: en dash, em dash, ... -> hyphen-minus + +NameFinderDL finder = new NameFinderDL(model, vocab, ids2Labels, options, sentenceDetector);]]> + +
+ +
+ Diacritic folding and multilingual safety + + AccentFoldCharSequenceNormalizer folds accents for search, but does so in a + script-aware way that a Latin-only folding filter cannot. It decomposes the text, then + drops nonspacing combining marks only for base characters whose script is configured for + folding (Latin, Greek, and Cyrillic by default). Combining marks on other scripts are + left untouched, because there they are essential orthography rather than decoration: + dropping an Indic vowel sign or virama, an Arabic harakat, a Hebrew point, or a Thai + vowel would change the word. + + + alpha) +fold.normalize("का"); // unchanged (Devanagari is left intact)]]> + + + Atomic Latin letters that do not decompose are mapped to an ASCII approximation by + default: for example the stroke letters and ligatures, eszett, and thorn + (ø -> o, æ -> ae, ß -> ss, + þ -> th). Both behaviors are configurable through the constructor: + + + + + + Diacritic folding is a recall optimization, not a linguistically correct transform, so it + is intended for a search or matching form rather than for display. Language-specific case + and letter rules (for example German DIN umlaut expansion, or the Turkish + dotless-i) are out of scope for the default folder and should be applied with an explicit + locale upstream. + +
+ +
+ The CharClass engine and code point sets + + The set-based normalizers are built on CharClass, a configurable class of + Unicode code points paired with a single canonical replacement, backed by a + CodePointSet with O(1) membership. You choose both the membership and the + replacement code point with CharClass.of(members, replacement); whitespace and + dashes are the two built-in presets, and any other class is one more configured instance: + + + U+0020 +CharClass dash = CharClass.dashes(); // Unicode Dash (curated) -> U+002D + +ws.collapse("a b"); // "a b" (runs -> one space) +ws.trim(" hi "); // "hi" +String[] tokens = ws.split("one two"); // ["one", "two"] (offset-aware via splitSpans) +dash.normalize("a—b"); // "a-b"]]> + + + A class applies its replacement three ways, which differ in whether they collapse runs and + whether they preserve character offsets: + + + + + normalize(text) replaces each member one-for-one with the replacement, + so it is length- and offset-preserving; use it when you still need spans back into + the original text. + + + + + collapse(text) reduces each maximal run of members to a single + replacement; it changes length, so it is a search and match transform. + + + + + collapsePreserving(text, keep, keepReplacement) collapses runs but emits + keepReplacement for any run containing a kept code point, which is how + you squish horizontal whitespace while keeping line breaks. + + + + + So the replacement is your choice and the method picks the behavior. Folding tabs and + newlines to a single newline, for example, is one configured class: + + + + + + When you need the normalized form together with a map back to the original, the + normalizeAligned, collapseAligned, + collapsePreservingAligned, trimAligned, and + removeAllAligned variants return an AlignedText that carries an + Alignment. The alignment maps spans between the two forms with + toOriginalSpan and toNormalizedSpan, staying correct across + deletions and length-changing folds, and composes with andThen. + + + A CodePointSet can be built explicitly, as a range, by union, or loaded from + a user definitions file so that delimiters can be extended without a code change. The + file is line oriented and parsed with the same cursor approach (no regular expression): a + [name] line opens a section, a # begins a comment, and each + remaining line is a hex code point or an inclusive range. + + + + + + + +
+ +
+ The layered term model + + TermAnalyzer tokenizes text and gives each token a + stack of normalization layers while keeping its source span. It is the + offset-preserving entry point for matching and BM25-style search: the normalized form is + what you index or query, and the span ties every layer back to the original text for + highlighting, even when normalization changes a token's length. A Term is one + token projected through an ordered chain of + Dimensions: original, NFC, NFKC, whitespace, dash, case fold, accent fold, + confusable fold, stem, and lemma. The order is fixed because the transforms do not commute + (case folding then accent folding differs from the reverse). The original is always kept, + so aggressive folding stays safe and a match on any layer maps back to the source through + the token's Span. + + + "Running" +// term.normalized() -> "run" (the final configured dimension, here STEM) +// term.peel() -> "running" (the layer below the top, O(1)) +// term.at(Dimension.NFC) -> computed lazily on first request, then cached]]> + + + Segmentation uses the word tokenizer, so the input + does not need to be pre-tokenized. The dimensions named in the builder are computed eagerly; + any other dimension is computed on first request, applied on top of the final form, and + cached, so querying a configured layer or peeling the last one is O(1) and adding an + unrequested dimension costs one transform. The character-level dimensions have built-in + defaults; STEM and LEMMA require a + Stemmer or Lemmatizer (and LEMMA a part-of-speech + tag), and fail loudly if requested without them. An analyzer configured with a stemmer is + not thread-safe, because the Snowball stemmers are stateful. + + + Each dimension's transform is configurable on the builder. Beyond the no-argument methods + that enable a dimension with its default, there are convenience methods for the common + knobs, and a general transform(dimension, normalizer) escape hatch for any + character-level dimension: + + + + + + The whitespace and dash methods take any CharSequenceNormalizer, so a + CharClass method reference (::normalize for one-for-one, + ::collapse for run-collapsing) selects both the fold target and the behavior. + The case-fold method takes a Locale for language-specific rules such as the + Turkish dotted/dotless i, and the accent-fold method takes the scripts to fold and whether + to fold stroke letters. + +
+ +
+ Confusable (homoglyph) folding + + Confusables reduces text to its Unicode confusable + skeleton following + UTS #39: it decomposes the + text, replaces each code point with its prototype, and decomposes again. Two strings are + confusable exactly when their skeletons are equal, which catches spoofing where Cyrillic or + Greek letters imitate Latin ones. + + + + + + The skeleton changes length and offsets, so like accent folding it is a derived, + matching-only form. It is also available as + ConfusableSkeletonCharSequenceNormalizer and as the + CONFUSABLE_FOLD term dimension. The mapping comes from the bundled Unicode + security data file confusables.txt. + +
+ +
+ Per-language profiles + + NormalizationProfiles selects per-language settings the same way OpenNLP + already selects a Snowball stemmer by language: ask for a language, or detect it with a + LanguageDetector when it is unspecified. Each + NormalizationProfile pairs a language with its Snowball stemmer and the + diacritic fold appropriate for that language, and builds a search-oriented + TermAnalyzer. + + + + + + The diacritic fold is the generic accent fold for English and the major Romance languages, + the German-specific fold (a-umlaut to ae, eszett to ss, following + DIN 5007-2) for German, and none for the Nordic languages and non-Latin scripts, where + folding distinct letters is language-wrong. As stated in + , this is a search-recall choice, not + linguistic correctness; a caller that wants different behavior builds a + TermAnalyzer directly. + +
+ +
+ Reference data + + The underlying Unicode data is also available directly as immutable reference tables, + with O(1) membership tests that match the Unicode standard: + + + + + UnicodeWhitespace lists the 25 characters carrying the + White_Space property, plus the related look-alike format characters + (zero width space, byte order mark, ...) that are not + whitespace. It exposes isWhitespace(int), + byCodePoint(int), and helpers for the line breaks and the + non-breaking spaces. + + + + + UnicodeDash lists every code point carrying the Dash + property, distinguishing the mathematical minus signs that are excluded from the + default normalization set. + + + +
+ + diff --git a/opennlp-docs/src/docbkx/opennlp.xml b/opennlp-docs/src/docbkx/opennlp.xml index 693248da60..5d0166c5e4 100644 --- a/opennlp-docs/src/docbkx/opennlp.xml +++ b/opennlp-docs/src/docbkx/opennlp.xml @@ -101,6 +101,7 @@ under the License. + diff --git a/opennlp-docs/src/docbkx/tokenizer.xml b/opennlp-docs/src/docbkx/tokenizer.xml index 0aebd511ff..8ab16c9e4d 100644 --- a/opennlp-docs/src/docbkx/tokenizer.xml +++ b/opennlp-docs/src/docbkx/tokenizer.xml @@ -23,7 +23,16 @@ The OpenNLP Tokenizers segment an input character sequence into tokens. Tokens are usually words, punctuation, numbers, etc. - + + + The statistical tokenizers in this chapter assume conventional whitespace-separated training + and test data. When input contains Unicode spacing or dash variants (no-break space, + ideographic space, en dash, and similar characters from PDFs or the web), use the + Unicode-aware preprocessing described in . The DL + components apply that machinery automatically for document chunking; see + . + + + +
+ Unicode Word Segmentation (UAX #29) + + The package opennlp.tools.tokenize.uax29 provides a tokenizer that follows the + Unicode Text Segmentation algorithm + (UAX #29), word boundary + rules WB1 through WB999. It is rule based and needs no trained model, it works directly over + a CharSequence, and it reports character offsets so the original text is + preserved for downstream processing such as the normalization described in + . The boundary data comes from the bundled Unicode + Character Database (currently Unicode 17.0) and the implementation passes the official + WordBreakTest conformance suite for that release. + +
+ Word Segmenter + + WordSegmenter finds the word boundaries. It is a single forward cursor pass + with constant-time property look-ups and no regular expression. Every segment is + reported, including whitespace and punctuation runs, so the segments are contiguous and + together cover the whole text. + + segments = WordSegmenter.segments("The quick brown fox.");]]> + + For allocation-free processing of large inputs, stream the segments to a callback instead + of collecting them. + + { + // handle the segment [start, end) +});]]> + + +
+
+ Word Tokenizer + + WordTokenizer builds on the segmenter. It keeps the segments that are words + (letters, digits, ideographs, kana, Hangul, a Southeast Asian script, or emoji), drops + whitespace and punctuation, and classifies each token. It implements the standard + Tokenizer interface, so it can be used wherever a tokenizer is expected. + + + + The tokens array contains "The", "quick", "brown", and "fox"; the trailing period and the + spaces are dropped. The tokenizeTyped method additionally returns the + category of each token as a WordType. + + + + The categories are ALPHANUMERIC, NUMERIC, + IDEOGRAPHIC, HIRAGANA, KATAKANA, + HANGUL, SOUTHEAST_ASIAN, and EMOJI. + + + A streaming overload reports each token to a handler with no per-token allocation, which + is the fastest option when the tokens are consumed on the fly. + + { + // handle the token [start, end) of the given WordType +});]]> + + A token longer than the maximum token length is emitted as consecutive pieces without + splitting a surrogate pair. The maximum defaults to + WordTokenizer.DEFAULT_MAX_TOKEN_LENGTH and can be set through the + constructor. + + + + +
+
From 8d850d363c54d6101c83c7a0648b1ee638d6816a Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Sun, 21 Jun 2026 19:20:44 -0400 Subject: [PATCH 02/13] OPENNLP-1850 Document the offset-aware normalization pipeline (buildAligned) Add an "Offset-aware pipelines" section to the normalizer chapter covering TextNormalizer.Builder.buildAligned(), the OffsetAwareNormalizer capability interface, mapping a match back to the source with AlignedText/Alignment, and the fail-loud rejection of rungs that cannot report edits (NFC/NFKC). List the new line-break-preserving whitespace rung in the normalizer family table. --- opennlp-docs/src/docbkx/normalizer.xml | 49 ++++++++++++++++++++++++-- 1 file changed, 47 insertions(+), 2 deletions(-) diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml index b96f7ed154..74e69285b2 100644 --- a/opennlp-docs/src/docbkx/normalizer.xml +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -97,6 +97,12 @@ Collapses each run of Unicode whitespace to a single ASCII space and trims the edges. + + LineBreakPreservingWhitespaceCharSequenceNormalizer + Collapses horizontal whitespace runs to a single ASCII space but keeps a + run containing a line break as a single newline, so paragraph structure + survives; trims the edges. + DashCharSequenceNormalizer Maps every Unicode dash to the ASCII hyphen-minus. The mathematical @@ -203,8 +209,47 @@ String t = search.normalize("“CafÉ”").toString(); // "\"cafe\""]]> with(...). The TextNormalizer pipeline and the individual CharSequenceNormalizer implementations are not applied automatically by statistical OpenNLP components; callers compose them explicitly when preprocessing text - for search or matching. The DL components described in the next section use a narrower, - built-in subset of this machinery. + for search or matching; to recover original character offsets from the composed result, + see . The DL components described in + use a narrower, built-in subset of this machinery. + + + +
+ Offset-aware pipelines + + build() returns a normalizer that yields only the cleaned text. When you also + need to map a match found in the normalized text back to the original (to highlight a search + hit in the source, say), build the pipeline with buildAligned() instead. It + returns an OffsetAwareNormalizer whose normalizeAligned(text) returns + an AlignedText: the normalized string together with an Alignment that + maps spans between the two forms, composed across every stage, so a match maps straight back to + the source even when a stage collapsed a run or folded a multi-unit character. + + + "find the-match" +Span hit = aligned.toOriginalSpan(5, 14); // "the-match" in the normalized text +// hit.getCoveredText(original) -> "the—match" (back to the source, em dash and all)]]> + + + OffsetAwareNormalizer is a capability interface: it extends + CharSequenceNormalizer and adds normalizeAligned, so a caller tests + for it with a plain instanceof, the same pattern the name finder uses for + OffsetMappingNameFinder. Only the cursor-based rungs implement it: whitespace, the + line-break-preserving whitespace rung, dashes, and invisible-control stripping. Rungs that + delegate to java.text.Normalizer (NFC and NFKC) or to a fold table cannot report + their per-character edits, so a chain that contains one is rejected by + buildAligned() with an IllegalStateException that names the rung, + rather than returning a pipeline that would hand back an offset that does not hold. The + per-stage maps and their composition use the Alignment machinery described in + .
From 301bb2b1b55ff3e97d75158b96e6fe4063d5fb30 Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Sun, 21 Jun 2026 19:30:31 -0400 Subject: [PATCH 03/13] OPENNLP-1850 Name the OffsetMappingNameFinder capability interface in the manual Document that NameFinderDL.findInOriginal comes from the OffsetMappingNameFinder capability interface, detectable with a plain instanceof check, so the name-finder chapter matches how the normalizer chapter presents OffsetAwareNormalizer. --- opennlp-docs/src/docbkx/namefinder.xml | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/opennlp-docs/src/docbkx/namefinder.xml b/opennlp-docs/src/docbkx/namefinder.xml index 08eb9780b6..c008edee17 100644 --- a/opennlp-docs/src/docbkx/namefinder.xml +++ b/opennlp-docs/src/docbkx/namefinder.xml @@ -176,6 +176,13 @@ Span[] spans = nameFinderDL.findInOriginal(tokens);]]> rejoins sub-tokens with spaces or when the source uses non-ASCII whitespace between tokens. + + findInOriginal is declared by the OffsetMappingNameFinder + capability interface that NameFinderDL implements, so a caller holding a + plain TokenNameFinder can detect the offset-mapping capability with a + finder instanceof OffsetMappingNameFinder check (no reflection) and fall + back to token-index spans otherwise. + Optional preprocessing of the joined input text is available through InferenceOptions and is off by default: From 1253cafabe86629ee84ccc0210b3a5179ead88e8 Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Sun, 21 Jun 2026 20:18:01 -0400 Subject: [PATCH 04/13] OPENNLP-1850 Document the offset-aware substitution folds (quotes, digits, ellipsis, bullets, umlaut) --- opennlp-docs/src/docbkx/normalizer.xml | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml index 74e69285b2..45f2eb7d1b 100644 --- a/opennlp-docs/src/docbkx/normalizer.xml +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -242,10 +242,12 @@ Span hit = aligned.toOriginalSpan(5, 14); // "the-match" in the normalized tex OffsetAwareNormalizer is a capability interface: it extends CharSequenceNormalizer and adds normalizeAligned, so a caller tests for it with a plain instanceof, the same pattern the name finder uses for - OffsetMappingNameFinder. Only the cursor-based rungs implement it: whitespace, the - line-break-preserving whitespace rung, dashes, and invisible-control stripping. Rungs that - delegate to java.text.Normalizer (NFC and NFKC) or to a fold table cannot report - their per-character edits, so a chain that contains one is rejected by + OffsetMappingNameFinder. Every per-code-point fold implements it: whitespace, the + line-break-preserving whitespace rung, dashes, invisible-control stripping, quotes, digits, + ellipsis, bullets, and the German umlaut transliteration. The folds that route through + java.text.Normalizer (NFC, NFKC, accent folding, and confusable folding) or + through JDK case mapping (case folding) cannot report their per-character edits, so they do + not implement the interface, and a chain that contains one is rejected by buildAligned() with an IllegalStateException that names the rung, rather than returning a pipeline that would hand back an offset that does not hold. The per-stage maps and their composition use the Alignment machinery described in From 7f883eeb1fa9ff591b72278b098b392ac35495eb Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Sun, 21 Jun 2026 21:49:34 -0400 Subject: [PATCH 05/13] OPENNLP-1850 Document the supplementary-dash offset shift in the DL fold options Note in the normalizer manual that, with dash folding enabled, a dash in the supplementary planes shrinks from two UTF-16 units to one and shifts later offsets, so find reports offsets into the normalized text in that case while findInOriginal maps them back to the original input. The one-for-one whitespace fold versus the run-collapsing whitespace rung is already covered in the same section. --- opennlp-docs/src/docbkx/normalizer.xml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml index 45f2eb7d1b..133ad3ef72 100644 --- a/opennlp-docs/src/docbkx/normalizer.xml +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -293,7 +293,10 @@ Span hit = aligned.toOriginalSpan(5, 14); // "the-match" in the normalized tex CharClass.dashes() set to the ASCII hyphen-minus. Mathematical minus signs and the soft hyphen are not affected unless you extend the set explicitly. This replacement is also one code point to one character for Basic Multilingual - Plane dashes. + Plane dashes. A dash in the supplementary planes shrinks from two UTF-16 units + to one, which shifts later offsets, so with this fold enabled find + reports offsets into the normalized text in that case while + findInOriginal maps them back to the original input. From c9208be61730e6303a2798e216b5afec16bb975c Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Sun, 21 Jun 2026 23:59:32 -0400 Subject: [PATCH 06/13] OPENNLP-1850 Tighten normalizer manual wording (review nits) Scope the "never relies on Character.isWhitespace" statement to the normalization engine rather than the whole library. Note that getInstance() gives the default shared instance and that case and accent folding also offer configured forms. Refer to the conformance file by its full name WordBreakTest.txt. --- opennlp-docs/src/docbkx/normalizer.xml | 7 ++++--- opennlp-docs/src/docbkx/tokenizer.xml | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-) diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml index 133ad3ef72..268509a33f 100644 --- a/opennlp-docs/src/docbkx/normalizer.xml +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -36,7 +36,7 @@ Standards-sourced. Membership sets come from the Unicode Character Database (for example the White_Space and Dash properties), not from the JVM's locale-dependent or quirky - character predicates. The library never relies on + character predicates. The normalization engine never relies on Character.isWhitespace, which disagrees with the Unicode standard. @@ -77,8 +77,9 @@ Each normalizer implements the existing opennlp.tools.util.normalizer.CharSequenceNormalizer interface - (CharSequence normalize(CharSequence)) and is a shared, stateless singleton - obtained through getInstance(). They can therefore be combined with the + (CharSequence normalize(CharSequence)) and is, by default, a shared stateless instance + obtained through getInstance() (case and accent folding also offer + configured forms). They can therefore be combined with the existing AggregateCharSequenceNormalizer, or with the TextNormalizer builder described below. diff --git a/opennlp-docs/src/docbkx/tokenizer.xml b/opennlp-docs/src/docbkx/tokenizer.xml index 8ab16c9e4d..509fdafd05 100644 --- a/opennlp-docs/src/docbkx/tokenizer.xml +++ b/opennlp-docs/src/docbkx/tokenizer.xml @@ -464,7 +464,7 @@ DetokenizationDictionary dict = new DetokenizationDictionary(tokens, operations) preserved for downstream processing such as the normalization described in . The boundary data comes from the bundled Unicode Character Database (currently Unicode 17.0) and the implementation passes the official - WordBreakTest conformance suite for that release. + WordBreakTest.txt conformance suite for that release.
Word Segmenter From a865d73fb1cf5d584017c99d6869aea1df1cceee Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Mon, 22 Jun 2026 01:34:29 -0400 Subject: [PATCH 07/13] OPENNLP-1850 Mirror the Extended_Pictographic emoji caveat in the tokenizer manual The Word Tokenizer section said it drops punctuation and keeps emoji without noting that emoji means any Extended_Pictographic code point, so symbol-like characters such as the copyright, trademark, and double-exclamation signs are kept. Match the WordTokenizer class javadoc. --- opennlp-docs/src/docbkx/tokenizer.xml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/opennlp-docs/src/docbkx/tokenizer.xml b/opennlp-docs/src/docbkx/tokenizer.xml index 509fdafd05..47f516b840 100644 --- a/opennlp-docs/src/docbkx/tokenizer.xml +++ b/opennlp-docs/src/docbkx/tokenizer.xml @@ -491,7 +491,9 @@ List segments = WordSegmenter.segments("The quick brown fox.");]]> WordTokenizer builds on the segmenter. It keeps the segments that are words (letters, digits, ideographs, kana, Hangul, a Southeast Asian script, or emoji), drops - whitespace and punctuation, and classifies each token. It implements the standard + whitespace and punctuation, and classifies each token. Here emoji means any + Extended_Pictographic code point, so symbol-like characters such as the copyright, trademark, + and double-exclamation signs are kept rather than dropped. It implements the standard Tokenizer interface, so it can be used wherever a tokenizer is expected. Date: Tue, 23 Jun 2026 08:24:43 -0400 Subject: [PATCH 08/13] OPENNLP-1850 Docs review nits: populate ids2Labels example; rule-based hyphenation Show a concrete, exhaustive ids2Labels BIO mapping in the ONNX name-finder example instead of an empty map (an unmapped predicted index raises IllegalStateException at runtime), and note the exhaustiveness requirement. Hyphenate 'rule-based' and split the comma splice in the UAX #29 tokenizer section. --- opennlp-docs/src/docbkx/namefinder.xml | 12 +++++++++++- opennlp-docs/src/docbkx/tokenizer.xml | 4 ++-- 2 files changed, 13 insertions(+), 3 deletions(-) diff --git a/opennlp-docs/src/docbkx/namefinder.xml b/opennlp-docs/src/docbkx/namefinder.xml index c008edee17..6bb2c03d88 100644 --- a/opennlp-docs/src/docbkx/namefinder.xml +++ b/opennlp-docs/src/docbkx/namefinder.xml @@ -155,8 +155,18 @@ Span[] nameSpans = nameFinder.find(sentence);]]> ids2Labels = new HashMap<>(); +ids2Labels.put(0, "O"); +ids2Labels.put(1, "B-PER"); +ids2Labels.put(2, "I-PER"); +ids2Labels.put(3, "B-ORG"); +ids2Labels.put(4, "I-ORG"); +ids2Labels.put(5, "B-LOC"); +ids2Labels.put(6, "I-LOC"); +ids2Labels.put(7, "B-MISC"); +ids2Labels.put(8, "I-MISC"); SentenceDetector sentenceDetector = new SentenceDetectorME(new SentenceModel(new File("/path/to/en-sent.bin"))); String[] tokens = {"George", "Washington", "was", "president", "of", "the", "United", "States", "."}; diff --git a/opennlp-docs/src/docbkx/tokenizer.xml b/opennlp-docs/src/docbkx/tokenizer.xml index 47f516b840..a4fe95d5a6 100644 --- a/opennlp-docs/src/docbkx/tokenizer.xml +++ b/opennlp-docs/src/docbkx/tokenizer.xml @@ -459,8 +459,8 @@ DetokenizationDictionary dict = new DetokenizationDictionary(tokens, operations) The package opennlp.tools.tokenize.uax29 provides a tokenizer that follows the Unicode Text Segmentation algorithm (UAX #29), word boundary - rules WB1 through WB999. It is rule based and needs no trained model, it works directly over - a CharSequence, and it reports character offsets so the original text is + rules WB1 through WB999. It is rule-based and needs no trained model. It works directly over + a CharSequence and reports character offsets so the original text is preserved for downstream processing such as the normalization described in . The boundary data comes from the bundled Unicode Character Database (currently Unicode 17.0) and the implementation passes the official From 3455f0840dbefa816543e5500196f0b4d2b98176 Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Thu, 25 Jun 2026 13:14:29 -0400 Subject: [PATCH 09/13] OPENNLP-1850 Docs review nits: declare xmlns:xlink; populate second ids2Labels example Declare the xlink namespace on the normalizer and tokenizer chapter roots -- both use (UAX #29 and UTS #39 references) but did not bind the prefix. Populate the ids2Labels map in the second NameFinderDL example (the InferenceOptions/findInOriginal one), which previously left it empty so the example would have located nothing. --- opennlp-docs/src/docbkx/namefinder.xml | 9 +++++++++ opennlp-docs/src/docbkx/normalizer.xml | 2 +- opennlp-docs/src/docbkx/tokenizer.xml | 2 +- 3 files changed, 11 insertions(+), 2 deletions(-) diff --git a/opennlp-docs/src/docbkx/namefinder.xml b/opennlp-docs/src/docbkx/namefinder.xml index 6bb2c03d88..87a04bb7be 100644 --- a/opennlp-docs/src/docbkx/namefinder.xml +++ b/opennlp-docs/src/docbkx/namefinder.xml @@ -206,6 +206,15 @@ Span[] spans = nameFinderDL.findInOriginal(tokens);]]> ids2Labels = new HashMap<>(); // the model's BIO labels +ids2Labels.put(0, "O"); +ids2Labels.put(1, "B-PER"); +ids2Labels.put(2, "I-PER"); +ids2Labels.put(3, "B-ORG"); +ids2Labels.put(4, "I-ORG"); +ids2Labels.put(5, "B-LOC"); +ids2Labels.put(6, "I-LOC"); +ids2Labels.put(7, "B-MISC"); +ids2Labels.put(8, "I-MISC"); SentenceDetector sentenceDetector = new SentenceDetectorME(new SentenceModel(new File("/path/to/en-sent.bin"))); String[] tokens = {"George", "Washington", "was", "president", "."}; diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml index 268509a33f..e99d32148b 100644 --- a/opennlp-docs/src/docbkx/normalizer.xml +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -13,7 +13,7 @@ OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> - + Text Normalization diff --git a/opennlp-docs/src/docbkx/tokenizer.xml b/opennlp-docs/src/docbkx/tokenizer.xml index a4fe95d5a6..7dd09a4897 100644 --- a/opennlp-docs/src/docbkx/tokenizer.xml +++ b/opennlp-docs/src/docbkx/tokenizer.xml @@ -13,7 +13,7 @@ OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. --> - + Tokenizer From 7930824b28823be8507a5ce4829519b458527dd1 Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Sat, 27 Jun 2026 08:15:23 -0400 Subject: [PATCH 10/13] OPENNLP-1850 Review nits: manual uses defaultChain()/matchingAnalyzer(); drop BM25/search framing Update the Text Normalization chapter examples for the searchDefault()->defaultChain() and searchAnalyzer()->matchingAnalyzer() renames, and drop the 'BM25-style search' phrasing. --- opennlp-docs/src/docbkx/normalizer.xml | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml index e99d32148b..41bbd38d9a 100644 --- a/opennlp-docs/src/docbkx/normalizer.xml +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -200,10 +200,10 @@ String term = pipeline.normalize("CAFÉ").toString(); // "cafe"]]> quotes and dashes, case fold, then script-gated accent fold) is available directly: - +String t = chain.normalize("“CafÉ”").toString(); // "\"cafe\""]]> Any custom CharSequenceNormalizer can be inserted with @@ -447,7 +447,7 @@ CharClass wsPlus = CharClass.whitespace().withAdditional(extra);]]> TermAnalyzer tokenizes text and gives each token a stack of normalization layers while keeping its source span. It is the - offset-preserving entry point for matching and BM25-style search: the normalized form is + offset-preserving entry point for matching: the normalized form is what you index or query, and the span ties every layer back to the original text for highlighting, even when normalization changes a token's length. A Term is one token projected through an ordered chain of @@ -540,11 +540,11 @@ Confusables.skeleton("paypal"); // a matching key, not readable tex +NormalizationProfiles.detect(text, languageDetector).map(NormalizationProfile::matchingAnalyzer);]]> The diacritic fold is the generic accent fold for English and the major Romance languages, From f1938e08234484c39c69f91b3c5f2df7c9103df7 Mon Sep 17 00:00:00 2001 From: Kristian Rickert Date: Tue, 30 Jun 2026 08:47:57 -0400 Subject: [PATCH 11/13] OPENNLP-1850 Align normalizer.xml DOCTYPE to the OPENNLP-1854 local DTD catalog form main's OPENNLP-1854 switched every chapter to the local-catalog public id/URL (-//OASIS//DTD DocBook XML 5.0//EN, http://docbook.org/xml/5.0/dtd/docbook.dtd); the new normalizer chapter predated that, so align it so the docs build resolves the DTD locally rather than over the network. --- opennlp-docs/src/docbkx/normalizer.xml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/opennlp-docs/src/docbkx/normalizer.xml b/opennlp-docs/src/docbkx/normalizer.xml index 41bbd38d9a..ae38ee75a8 100644 --- a/opennlp-docs/src/docbkx/normalizer.xml +++ b/opennlp-docs/src/docbkx/normalizer.xml @@ -1,6 +1,6 @@ -