Half-width katakana characters are not displaying correctly with Shift-JIS encoding

[nhp65]

This is about Japanese character encoding issues with Shift-JIS.
Japanese text includes hiragana, katakana, and kanji characters in addition to alphanumeric characters. These are all full-width characters, but among them, only katakana also has half-width versions, similar to alphanumeric characters.
When displaying a text file containing these half-width katakana in UTF-8 encoding, it displays correctly.

However, when a text file containing half-width katakana is saved in Shift-JIS encoding,
it is incorrectly identified as cp1252 encoding and doesn't display properly.

I expect text containing half-width katakana to display without issues, just like with UTF-8.

Please copy and test the following text.

あいうえおｱｲｳｴｵ

The character code table for half-width katakana is as follows.

This test was executed in the following environment.

macOS Sequoia 15.1.1
cudatext-macos-cocoa-aarch64-1.219.1.0

[Alexey-T]

However, when a text file containing half-width katakana is saved in Shift-JIS encoding,
it is incorrectly identified as cp1252 encoding and doesn't display properly.

this is not a bug. this is missing feature - detection of Shift-JIS coding. Cud cannot detect it. so user must choose Shift-JIS coding using the statusbar click.

PS. Cud can detect UTF8 by signature and by byte content.
PS. if you have Pascal code to detect shift-jis, tell me pls.

[Alexey-T]

I found new bug when experimented with your Shift-JIS sample text. I saved sample

あいうえおｱｲｳｴｵ

to UTF8, then converted it to Shift-JIS, then loaded Shift-JIS file. and I see editor became 'locked' forever (hourglass icon). fixed!

[Alexey-T]

@nhp65
@rasberryrabbit
Another possible issue with shift-jis...
If I save sample あいうえおｱｲｳｴｵ as UTF8 and convert file to shift-jis, then I restart Cud and reopen that file - Cud cannot convert last few chars and shows this:

is it normal for sample text and for shift-jis? Other editors do the same? If other editors are showing all chars normal (not ??) can you try to repeat the issue in the SynEdit in the Lazarus IDE? we may have incomplete shift-jis coding in Lazarus?

[rasberryrabbit]

Some special CJK text doesn't shown on other languages.
It look like normal behavier.

[Alexey-T]

@rasberryrabbit I did not understand your comment. BTW. i made the bugreport at Lazarus tracker

https://gitlab.com/freepascal.org/lazarus/lazarus/-/issues/41277

can you help with that report, ie fix LConvEncoding somehow? we have not much Asian users who can help.
I also tag @rich2014 who is Asian user, can he help?

[rasberryrabbit]

I submit patch about converting cp932 to utf8 above link.

[Alexey-T]

Applied @rasberryrabbit 's patch. it is in the release 1.220 now. thanks.

[nhp65]

@Alexey-T
@rasberryrabbit

I tried 1.220.
All my problems were solved.
As a result, the shift-jis Encoding detection also worked perfectly.
Many thanks to both of you. Thank you.

[Alexey-T]

the shift-jis Encoding detection also worked perfectly.

Detection is not supported, as before. but if you save file as shift-jis, file will be loaded in shift-jis next time.