OPENNLP-1850: UAX #29 word tokenizer — WordSegmenter, WordTokenizer, WordType (2a/7)

[krickert]

Part 2a of the OPENNLP-1850 stack. Splits the former tokenizer PR (#1104) into the UAX #29 tokenizer (this PR), the Term model (2b), and the NormalizationProfile registry (2c), as requested in review.

Self-contained: the Unicode-conformant WordSegmenter/WordTokenizer/WordType/WordToken over the bundled Word_Break and Extended_Pictographic data, the official WordBreakTest.txt conformance suite (1944/1944), and the Unicode data LICENSE/NOTICE/rat-excludes.

WordBreakProperty and ExtendedPictographic load their data lazily and recoverably (double-checked accessor, no classpath resource I/O in a static {} block), per the same review point as on the foundation — so a resource the loader cannot see is a catchable exception at call time, not a class-poisoning ExceptionInInitializerError.

Base: OPENNLP-1850-1b-alignment (#1109). Stack: 1a → 1b → 2a (this) → 2b → 2c → DL → docs.

[krickert]

@rzo1 Both points on the tokenizer PR are addressed.

Resource loading (done). WordBreakProperty and ExtendedPictographic no longer load in a static {} block — each now loads lazily on first use via a double-checked accessor, so a resource the loader can't see is a catchable exception at call time rather than an ExceptionInInitializerError that poisons the class (and would otherwise take the whole WordSegmenter/WordTokenizer down). The getResourceAsStream in WordBoundaryConformanceTest is left as-is (test-only).

Split into 2a / 2b / 2c (done). Along the three concepts you identified:

• OPENNLP-1850: UAX #29 word tokenizer — WordSegmenter, WordTokenizer, WordType (2a/7) #1110 — UAX OPENNLP-910: Add checkstyle #29 tokenizer (WordSegmenter/WordTokenizer/WordType/WordToken/WordBreak/WordBreakProperty/ExtendedPictographic + bundled data + the conformance suite). Self-contained, the bulk of the work.
• OPENNLP-1850: Layered Term model — Term, TermAnalyzer (2b/7) #1111 — Term model (Term/TermAnalyzer), on 2a.
• OPENNLP-1850: Per-language NormalizationProfile registry (2c/7) #1112 — NormalizationProfile registry, on 2b.

#1105 (DL) now bases on #1112; I closed #1104 pointing here. Each layer builds and tests green on its own.

[rzo1]

• Loader symmetry: WordBreakProperty.parse fails loud on a missing ;, while ExtendedPictographic.parse silently treats a line with no ; as whole-content, and only the former has a malformed-data/missing-resource test. Worth making the two read and test alike.
• WordType.of fixes a token's script from its first script code point only; fine for single-script UAX#29 segments but a heuristic for mixed-script runs. A one-line javadoc note would help.

[krickert]

@rzo1 Both addressed (tip f2d1d8cc).

Loader symmetry. I kept the two loaders deliberately different but documented why, and closed the test gap. The difference is real rather than an oversight: WordBreakProperty.txt always has a code ; property shape, so a missing ; is corruption and fails loud. ExtendedPictographic.txt is a filtered single-property file (only Extended_Pictographic, with the property column stripped), so a line with no ; is the normal, well-formed case — the code points are taken whole. Forcing it to fail on a missing ; would reject valid data; I added a comment on ExtendedPictographic.parse spelling that out. For the test asymmetry: ExtendedPictographic.parse is now package-visible and has a malformed-data test (parseFailsLoudOnMalformedHex) asserting that a non-hex code-point column fails loud with IllegalArgumentException naming the resource — the same fail-loud contract WordBreakProperty already had.

WordType.of leading-script heuristic. Added a note on WordType.of: the script category is taken from the first script code point in the range; UAX #29 word segments are single-script in practice, so for an unusual mixed-script run this reports the leading script rather than a per-character determination.