pplkit db317c358b feat: add language-aware sentence tokenization (#1269) ...

* feat: add language-aware sentence tokenization

* feat: add missing punkt languages

---------

Co-authored-by: pulkit <129310466+p1kit@users.noreply.github.com>
Co-authored-by: Barabazs <31799121+Barabazs@users.noreply.github.com>

2025-10-21 15:57:26 +02:00

23 KiB

Raw Permalink Blame History

View Raw