OPENNLP-1850: Document Unicode normalization and the UAX #29 tokenizer (4-docs/7)

[krickert]

Part 4-docs (7 of 7) of the OPENNLP-1850 stack: Developer Manual coverage of Unicode normalization and the UAX #29 tokenizer — a new "Text Normalization" chapter plus tokenizer / doccat / namefinder / introduction updates and the master opennlp.xml.

Base: OPENNLP-1850-3-dl (#1105).

[krickert]

OPENNLP-1850 stacked PRs (review independently; merge bottom-up, re-targeting each base to main as the one below lands):

1. OPENNLP-1850: Unicode normalization foundation — CharClass engine, rungs, Dimension (1/4) #1103 — Unicode normalization foundation (CharClass engine, rungs, Dimension)
2. OPENNLP-1850: UAX #29 word tokenizer and the layered Term model (2/4) #1104 — UAX OPENNLP-910: Add checkstyle #29 word tokenizer + layered Term model
3. OPENNLP-1850: Offset-safe input normalization in the DL components (3-dl/7) #1105 — Offset-safe input normalization in the DL components
4. OPENNLP-1850: Document Unicode normalization and the UAX #29 tokenizer (4-docs/7) #1106 — Documentation

Supersedes #1101.

[Copilot]

Pull request overview

Adds/updates Developer Manual documentation for Unicode-aware normalization and UAX #29 tokenization, and connects those docs to the DL (ONNX) components’ Unicode chunking/normalization behavior.

Changes:

• Adds a new “Text Normalization” manual chapter and includes it in the master DocBook.
• Extends the tokenizer chapter with guidance about Unicode preprocessing and a new UAX #29 tokenizer/segmenter section.
• Updates NameFinderDL/DocumentCategorizerDL and introduction docs to reference Unicode-aware DL chunking and normalization options.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
opennlp-docs/src/docbkx/tokenizer.xml	Adds Unicode preprocessing guidance and documents the UAX #29 tokenizer/segmenter APIs.
opennlp-docs/src/docbkx/opennlp.xml	Includes the new normalizer chapter in the book build.
opennlp-docs/src/docbkx/normalizer.xml	New “Text Normalization” chapter covering normalizers, pipelines, term model, and reference data.
opennlp-docs/src/docbkx/namefinder.xml	Updates NameFinderDL constructor usage and documents Unicode-aware DL chunking and normalization options.
opennlp-docs/src/docbkx/introduction.xml	Links DL inference Unicode handling to the normalizer documentation.
opennlp-docs/src/docbkx/doccat.xml	Documents Unicode-aware DL chunking/normalization and adds ONNX usage snippet updates.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[rzo1]

Thx for the PR. Here are some suggestions:

• Declare xmlns:xlink explicitly on the chapter roots of normalizer.xml and tokenizer.xml. Both use xlink:href (normalizer.xml:457, tokenizer.xml:461) but rely on the DocBook 5.0 DTD's #FIXED default to bind the prefix. The Maven build resolves the DTD so it works, but it breaks under any non-validating namespace-aware tool (IDE linters, xmllint --nonet), and every other chapter declares it explicitly:
  normalizer.xml: <chapter xml:id="tools.normalizer" xmlns:xlink="http://www.w3.org/1999/xlink">
  tokenizer.xml: <chapter xml:id="tools.tokenizer" xmlns:xlink="http://www.w3.org/1999/xlink">
  Note that tokenizer.xml newly introduces xlink usage, so this is the first chapter to add it there.

Otherwise the content is accurate.

[krickert]

Status since the last review. Normalizer chapter gains an "Offset-aware pipelines" section for buildAligned() and the capability interface with a worked dash-fold span example, the line-break-preserving rung in the fold table, and the supplementary-dash offset note in the DL fold options. Name-finder chapter names OffsetMappingNameFinder behind findInOriginal. docbkx HTML builds clean. Rebased onto the updated stack.

[rzo1]

• normalizer.xml and the modified tokenizer.xml use xlink:href without declaring xmlns:xlink on the chapter root, unlike the other xlink-using chapters (e.g. chunker.xml). It still builds (the DocBook DTD declares it #FIXED), but add xmlns:xlink for convention.
• The second namefinder.xml ONNX snippet uses an empty ids2Labels map, which would throw IllegalStateException at runtime and contradicts the surrounding warning that the map must be exhaustive. Mirror the populated BIO map from the first snippet.

[krickert]

@rzo1 Both fixed (tip 121c2588).

xmlns:xlink declaration. Declared xmlns:xlink="http://www.w3.org/1999/xlink" on the normalizer.xml and tokenizer.xml chapter roots, matching the other xlink-using chapters — both use <link xlink:href> (the UTS #39 and UAX #29 references) and previously leaned on the DocBook DTD's #FIXED default, which non-validating namespace-aware tools don't honor.

Empty ids2Labels in the second snippet. Populated it with the same BIO map as the first snippet (O, B-PER/I-PER, B-ORG/I-ORG, B-LOC/I-LOC, B-MISC/I-MISC), so the findInOriginal example is runnable and consistent with the "map must be exhaustive" warning.