Skip to content

Support UTF8 in fulltext analyzers #26850

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Support UTF8 in fulltext analyzers #26850

wants to merge 4 commits into from
+106 −26

Conversation

kunga
Copy link
Member

@kunga kunga commented Oct 14, 2025

Changelog entry

...

Changelog category

  • Not for changelog (changelog entry is not required)

Description for reviewers

Fixes ru texts tokenization

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

⚪ 2025-10-14 13:17:12 UTC Pre-commit check linux-x86_64-release-asan for 307182a has started.
⚪ 2025-10-14 13:17:16 UTC Artifacts will be uploaded here
⚫ 2025-10-14 13:18:23 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

🟢 2025-10-15 12:41:08 UTC The validation of the Pull Request description is successful.

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

⚪ 2025-10-14 13:17:59 UTC Pre-commit check linux-x86_64-relwithdebinfo for 307182a has started.
⚪ 2025-10-14 13:18:03 UTC Artifacts will be uploaded here
⚫ 2025-10-14 13:18:23 UTC Check cancelled

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

⚪ 2025-10-14 13:20:23 UTC Pre-commit check linux-x86_64-release-asan for 7a23765 has started.
⚪ 2025-10-14 13:20:37 UTC Artifacts will be uploaded here
⚪ 2025-10-14 13:24:14 UTC ya make is running...
🟡 2025-10-14 15:39:18 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17452 17010 0 173 249 20

🟢 2025-10-14 15:39:25 UTC Build successful.
🟢 2025-10-14 15:39:50 UTC ydbd size 3.8 GiB changed* by -182.9 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 6212018 merge: 7a23765 diff diff %
ydbd size 4 031 469 520 Bytes 4 031 282 224 Bytes -182.9 KiB -0.005%
ydbd stripped size 1 497 304 384 Bytes 1 497 312 000 Bytes +7.4 KiB +0.001%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 14, 2025 •
edited
Loading

⚪ 2025-10-14 13:20:24 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7a23765 has started.
⚪ 2025-10-14 13:20:38 UTC Artifacts will be uploaded here
⚪ 2025-10-14 13:24:16 UTC ya make is running...
🟡 2025-10-14 14:49:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40595 37658 0 8 2903 26

⚪ 2025-10-14 14:49:23 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-10-14 15:10:10 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
879 (only retried tests) 780 0 0 76 23

🟢 2025-10-14 15:10:13 UTC Build successful.
🟢 2025-10-14 15:10:31 UTC ydbd size 2.3 GiB changed* by -123.1 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: 6212018 merge: 7a23765 diff diff %
ydbd size 2 423 191 896 Bytes 2 423 065 824 Bytes -123.1 KiB -0.005%
ydbd stripped size 515 835 656 Bytes 515 836 104 Bytes +448 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kunga kunga self-assigned this Oct 14, 2025
@kunga kunga mentioned this pull request Oct 14, 2025
Open
12 tasks
@kunga kunga changed the title Support russian in fulltext index Support russian in fulltext analyzers Oct 14, 2025
@kunga kunga marked this pull request as ready for review October 14, 2025 15:10
@kunga kunga requested a review from vitalif October 14, 2025 15:10
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 15, 2025 •
edited
Loading

⚪ 2025-10-15 07:48:46 UTC Pre-commit check linux-x86_64-release-asan for 1e56e10 has started.
⚪ 2025-10-15 07:48:59 UTC Artifacts will be uploaded here
⚪ 2025-10-15 07:52:40 UTC ya make is running...
🟡 2025-10-15 09:58:46 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
17480 17033 0 194 237 16

🟢 2025-10-15 09:58:53 UTC Build successful.
🟢 2025-10-15 09:59:18 UTC ydbd size 3.8 GiB changed* by -189.9 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: a116ac4 merge: 1e56e10 diff diff %
ydbd size 4 032 935 280 Bytes 4 032 740 848 Bytes -189.9 KiB -0.005%
ydbd stripped size 1 497 801 344 Bytes 1 497 803 200 Bytes +1.8 KiB +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 15, 2025 •
edited
Loading

⚪ 2025-10-15 07:51:10 UTC Pre-commit check linux-x86_64-relwithdebinfo for 1e56e10 has started.
⚪ 2025-10-15 07:51:23 UTC Artifacts will be uploaded here
⚪ 2025-10-15 07:54:58 UTC ya make is running...
🟡 2025-10-15 09:25:24 UTC Some tests failed, follow the links below. Going to retry failed tests...

Ya make output | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
40651 37877 0 4 2744 26

⚪ 2025-10-15 09:25:34 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-10-15 09:39:54 UTC Tests successful.

Ya make output | Test bloat | Test bloat

TESTS PASSED ERRORS FAILED SKIPPED MUTED?
876 (only retried tests) 854 0 0 0 22

🟢 2025-10-15 09:39:59 UTC Build successful.
🟢 2025-10-15 09:40:15 UTC ydbd size 2.3 GiB changed* by -120.5 KiB, which is <= 0 Bytes vs main: OK

ydbd size dash main: a116ac4 merge: 1e56e10 diff diff %
ydbd size 2 423 988 336 Bytes 2 423 864 904 Bytes -120.5 KiB -0.005%
ydbd stripped size 515 997 032 Bytes 515 997 992 Bytes +960 Bytes +0.000%

*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation

@kunga kunga changed the title Support russian in fulltext analyzers Support UTF8 in fulltext analyzers Oct 15, 2025
@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 15, 2025 •
edited
Loading

⚪ 2025-10-15 12:40:17 UTC Pre-commit check linux-x86_64-relwithdebinfo for 4cf6bc4 has started.
⚪ 2025-10-15 12:40:31 UTC Artifacts will be uploaded here
⚪ 2025-10-15 12:44:10 UTC ya make is running...

@github-actions GitHub Actions
Copy link

github-actions bot commented Oct 15, 2025 •
edited
Loading

⚪ 2025-10-15 12:40:37 UTC Pre-commit check linux-x86_64-release-asan for 4cf6bc4 has started.
⚪ 2025-10-15 12:40:52 UTC Artifacts will be uploaded here
⚪ 2025-10-15 12:44:33 UTC ya make is running...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vitalif vitalif vitalif approved these changes

Assignees

@kunga kunga

Labels

not-for-changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kunga @vitalif