Terminal hangs or prints repetitive characters when using Japanese IME #18836

https://user-images.githubusercontent.com/19299/200154562-a6ece9bd-f0c3-461f-98b3-6f62a9094d59.mp4

@hnakamur commented on GitHub (Nov 6, 2022):

The same problem occurs with PowerShell version 5.1.22621.608.

@hnakamur commented on GitHub (Nov 6, 2022): The same problem occurs with PowerShell version 5.1.22621.608. https://user-images.githubusercontent.com/19299/200154562-a6ece9bd-f0c3-461f-98b3-6f62a9094d59.mp4

claunia commented

@lhecker commented on GitHub (Nov 7, 2022):

Input some Japanese characters using Microsoft IME at the right side edge of the terminal window.

Just to be sure: You're saying this never happens if you are not at the right side edge? Because if that's true, then that would narrow down the issue quite a lot and make fixing it much easier.

@lhecker commented on GitHub (Nov 7, 2022): > Input some Japanese characters using Microsoft IME at the right side edge of the terminal window. Just to be sure: You're saying this never happens if you are _not_ at the right side edge? Because if that's true, then that would narrow down the issue quite a lot and make fixing it much easier.

claunia commented

https://user-images.githubusercontent.com/19299/200470370-a19b6c63-54e2-4948-b70b-a51189c81bc7.mp4

@hnakamur commented on GitHub (Nov 8, 2022):

I tested this again and found out this also happens when I am not at the right side edge.

Steps to reproduce:
Input some Japanese texts and delete them and repeat that.

@hnakamur commented on GitHub (Nov 8, 2022): I tested this again and found out this also happens when I am not at the right side edge. Steps to reproduce: Input some Japanese texts and delete them and repeat that. https://user-images.githubusercontent.com/19299/200470370-a19b6c63-54e2-4948-b70b-a51189c81bc7.mp4

claunia commented

@hnakamur commented on GitHub (Nov 18, 2022):

I uninstalled version 1.15.2875.0 and installed https://github.com/microsoft/terminal/releases/download/v1.14.2281.0/Microsoft.WindowsTerminal_Win11_1.14.2282.0_8wekyb3d8bbwe.msixbundle and it seems the problem go away.

@hnakamur commented on GitHub (Nov 18, 2022): I uninstalled version 1.15.2875.0 and installed https://github.com/microsoft/terminal/releases/download/v1.14.2281.0/Microsoft.WindowsTerminal_Win11_1.14.2282.0_8wekyb3d8bbwe.msixbundle and it seems the problem go away.

claunia commented

@dogatana commented on GitHub (Feb 1, 2023):

Unfortunately, 1.16 still has this issue.
As a workaround, I'm using 1.14 now, but sometimes it's updated to 1.15 automatically.
And I cannot re-install 1.14 onto 1.15 with the following message.

アプリをインストールできませんでした。エラーメッセージ: API Logging data because access was denied for file: \?\C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.14.2281.0_x64__8wekyb3d8bbwe\CascadiaCode.ttf, user SID: S-1-5-18 からのエラー 0x0 が発生したため、AppX 展開操作に失敗しました (0x80070005)

Although this is a Japanese message, I hope you can get information from this. 😅

Every time this happens, I need to uninstall 1.15, install 1.14 and recover setting.json.

I checked the settings, but I cloud not found the setting about auto-update.
Any suggestions are most welcome.

@dogatana commented on GitHub (Feb 1, 2023): Unfortunately, 1.16 still has this issue. As a workaround, I'm using 1.14 now, but sometimes it's updated to 1.15 automatically. And I cannot re-install 1.14 onto 1.15 with the following message. >アプリをインストールできませんでした。エラーメッセージ: API Logging data because access was denied for file: \\?\C:\Program Files\WindowsApps\Microsoft.WindowsTerminal_1.14.2281.0_x64__8wekyb3d8bbwe\CascadiaCode.ttf, user SID: S-1-5-18 からのエラー 0x0 が発生したため、AppX 展開操作に失敗しました (0x80070005) Although this is a Japanese message, I hope you can get information from this. 😅 Every time this happens, I need to uninstall 1.15, install 1.14 and recover setting.json. I checked the settings, but I cloud not found the setting about **auto-update**. Any suggestions are most welcome.

claunia commented

https://superuser.com/questions/1676001/windows-stop-killing-my-terminals-for-an-update

@dogatana commented on GitHub (Feb 2, 2023):

1.17 has this too.
I should use the following workaround.

@dogatana commented on GitHub (Feb 2, 2023): 1.17 has this too. I should use the following workaround. https://superuser.com/questions/1676001/windows-stop-killing-my-terminals-for-an-update

claunia commented

@lhecker commented on GitHub (Sep 7, 2023):

I genuinely can't figure out how to reproduce the issue. I've tried writing some variation of "どうぞよろしくお願いいたします。" over and over again for about ~~15min~~ an hour and it didn't happen.

But clearly this regression is caused by either of these two commits, both of which first shipped in v1.15:

I don't think either of the two commits themselves is "buggy", but rather that they make bugs in TSF (Text Services Framework) obvious. This is because backspacing like in hnakamur's videos causes the TermControl::_CharacterHandler event handler to be called (the character is \b). This will cause TSFInputControl().ClearBuffer() to be called which should instruct TSF to flush its entire buffer and reset its state.

I think what's happening here is that TSF is either:

ignoring this request
getting into a bad state
running out of process in the "Text Input Management Service" and that may result in yet-unknown, subtle race conditions? ¹

2c922e105c could have triggered the regression because it drops clamping the signed range values from TSF. I believe at the time I didn't consider it possible that TSF would actually pass either negative indices or a range where StartCaretPosition is greater than EndCaretPosition. I'll be more cautious in the code I write from now on.

ed800dc72d could have triggered the regression because it changed this:

_activeTextStart = ::base::ClampMin(_activeTextStart, ::base::ClampedNumeric<size_t>(range.StartCaretPosition));

into this:

_activeTextStart = std::min(_activeTextStart, _inputBuffer.size());

I think the latter is technically more correct ², but if TSF is in a bad state where the internal state didn't properly reset, then this may be off.

Microservices in your OS! Got a problem? Now it's a problem and a distributed problem! 100% more value! /s ↩︎
The design of the TSF API implies that it is random access. When the buffer contains "foo" and the _activeTextStart is 3 (= at the end), then we shouldn't move the start to 0 just because TSF replaced the entire string with "foobar". It should remain at 3, because the active string is "bar". It's unclear how TSF is supposed to be used in this case because TSF lacks documentation and is not designed to be used for a terminal. ↩︎

@lhecker commented on GitHub (Sep 7, 2023): I genuinely can't figure out how to reproduce the issue. I've tried writing some variation of "どうぞよろしくお願いいたします。" over and over again for about ~15min~ an hour and it didn't happen. But clearly this regression is caused by either of these two commits, both of which first shipped in v1.15: * 2c922e105c6468a8c2469946d44b98390ff94f1d * ed800dc72de9e51201becf1f752093f4c1b4e155 I don't think either of the two commits themselves is "buggy", but rather that they make bugs in TSF ([Text Services Framework](https://learn.microsoft.com/en-us/windows/win32/tsf/text-services-framework)) obvious. This is because backspacing like in hnakamur's videos causes the `TermControl::_CharacterHandler` event handler to be called (the character is `\b`). This will cause `TSFInputControl().ClearBuffer()` to be called which _should_ instruct TSF to flush its entire buffer and reset its state. I think what's happening here is that TSF is either: * ignoring this request * getting into a bad state * running out of process in the "Text Input Management Service" and that may result in yet-unknown, subtle race conditions? [^1] --- 2c922e105c6468a8c2469946d44b98390ff94f1d could have triggered the regression because it drops clamping the signed `range` values from TSF. I believe at the time I didn't consider it possible that TSF would actually pass either negative indices or a range where `StartCaretPosition` is greater than `EndCaretPosition`. I'll be more cautious in the code I write from now on. ed800dc72de9e51201becf1f752093f4c1b4e155 could have triggered the regression because it changed this: ```cpp _activeTextStart = ::base::ClampMin(_activeTextStart, ::base::ClampedNumeric<size_t>(range.StartCaretPosition)); ``` into this: ```cpp _activeTextStart = std::min(_activeTextStart, _inputBuffer.size()); ``` I think the latter is technically more correct [^2], but if TSF is in a bad state where the internal state didn't properly reset, then this may be off. [^1]: Microservices in your OS! Got a problem? Now it's a problem _and_ a distributed problem! 100% more value! /s [^2]: The design of the TSF API implies that it is random access. When the buffer contains "foo" and the `_activeTextStart` is 3 (= at the end), then we shouldn't move the start to 0 just because TSF replaced the entire string with "foobar". It should remain at 3, because the active string is "bar". It's unclear how TSF is supposed to be used in this case because TSF lacks documentation and is not designed to be used for a terminal.

claunia commented

@lhecker commented on GitHub (Sep 7, 2023):

I've marked #15938 as closing this issue, but I'll have to unfortunately rely on you to validate it, as I was entirely unable to reproduce the issue. 😟

@lhecker commented on GitHub (Sep 7, 2023): I've marked #15938 as closing this issue, but I'll have to unfortunately rely on you to validate it, as I was entirely unable to reproduce the issue. 😟

claunia commented

2026-01-31 06:25:53 +00:00

@hnakamur commented on GitHub (Sep 7, 2023):

Can I download the binary built with #15938 included?
Then I can validate it.

As of 1.17.11461.0, I just reproduce the issue just by inputting and deleting "日本語を入力して" three times.
I'm using the US keyboard on Japanese Windows 11.

@hnakamur commented on GitHub (Sep 7, 2023): Can I download the binary built with [#15938](https://github.com/microsoft/terminal/pull/15938) included? Then I can validate it. As of 1.17.11461.0, I just reproduce the issue just by inputting and deleting "日本語を入力して" three times. I'm using the US keyboard on Japanese Windows 11.

claunia commented

@zadjii-msft commented on GitHub (Sep 7, 2023):

@hnakamur sure can!

You should be able to go to: https://dev.azure.com/ms/terminal/_build/results?buildId=494745&view=artifacts&pathAsName=false&type=publishedArtifacts

Then navigate to the build-x64-release > _unpackaged > WindowsTerminalDev_0.0.1.0_x64.zip file. (or for a shortcut: I think this link will work?)

If there are subsequent commits, then this link:

will take you to the latest CI run

@zadjii-msft commented on GitHub (Sep 7, 2023): @hnakamur sure can! You should be able to go to: https://dev.azure.com/ms/terminal/_build/results?buildId=494745&view=artifacts&pathAsName=false&type=publishedArtifacts Then navigate to the `build-x64-release` > `_unpackaged` > `WindowsTerminalDev_0.0.1.0_x64.zip` file. (or for a shortcut: [I think this link will work?](https://artprodwus21.artifacts.visualstudio.com/Aea964927-e753-4550-bf48-b63004033439/20953fb4-d427-4ad6-b78c-13142d0c5462/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL21zL3Byb2plY3RJZC8yMDk1M2ZiNC1kNDI3LTRhZDYtYjc4Yy0xMzE0MmQwYzU0NjIvYnVpbGRJZC80OTQ3NDUvYXJ0aWZhY3ROYW1lL2J1aWxkLXg2NC1SZWxlYXNl0/content?format=file&subPath=%2F_unpackaged%2FWindowsTerminalDev_0.0.1.0_x64.zip)) If there are subsequent commits, then this link: ![image](https://github.com/microsoft/terminal/assets/18356694/78619425-2d40-4cde-b93e-6e73bf59921c) will take you to the latest CI run

claunia commented

2026-01-31 06:25:53 +00:00

@hnakamur commented on GitHub (Sep 7, 2023):

Thanks, your shortcut link works.
I'm afraid the issue is not resolved.

https://github.com/microsoft/terminal/assets/19299/04d8ce3b-5d6f-4878-9397-37a7f7a8eccb

@hnakamur commented on GitHub (Sep 7, 2023): Thanks, your shortcut link works. I'm afraid the issue is not resolved. https://github.com/microsoft/terminal/assets/19299/04d8ce3b-5d6f-4878-9397-37a7f7a8eccb

claunia commented

2026-01-31 06:25:53 +00:00

@hnakamur commented on GitHub (Sep 7, 2023):

I review my Windows settings and I found 「以前のバージョンのMicrosoft IMEを使う」 was on.
I changed it off and the issue seems go away on both Windows Terminal 1.17.11461.0 and WindowsTerminalDev_0.0.1.0_x64.zip!

@hnakamur commented on GitHub (Sep 7, 2023): I review my Windows settings and I found 「以前のバージョンのMicrosoft IMEを使う」 was on. I changed it off and the issue seems go away on both Windows Terminal 1.17.11461.0 and WindowsTerminalDev_0.0.1.0_x64.zip! ![image](https://github.com/microsoft/terminal/assets/19299/d0c57db6-129d-459b-b831-a49c48502439)

claunia commented

@hnakamur commented on GitHub (Sep 7, 2023):

I tried again with「以前のバージョンのMicrosoft IMEを使う」 on, the issue was reproduced.
So it seems there is a problem in the combination of the old version Microsoft IME and Windows Terminal.

As for me, I am fine with the current version of Microsoft IME.

@hnakamur commented on GitHub (Sep 7, 2023): I tried again with「以前のバージョンのMicrosoft IMEを使う」 on, the issue was reproduced. So it seems there is a problem in the combination of the old version Microsoft IME and Windows Terminal. As for me, I am fine with the current version of Microsoft IME.

claunia commented

@lhecker commented on GitHub (Sep 7, 2023):

Hmm, I think that's fairly good news for us (Windows Terminal) at least, because it implies that this is indeed a bug in TSF. (An application shouldn't notice a difference in behavior between the legacy and new IME after all.) Would it be fine for us to only support the new IME?

BTW I tried to reproduce the issue with the legacy IME and again couldn't get it to reproduce it. 😟
I don't speak Japanese, so maybe I'm just not using the right input? I saw you write "kokodenennkannn" and tried that and that didn't do anything for me. Do you use any specific input that reliably reproduces the issue for you after a few tries?

@lhecker commented on GitHub (Sep 7, 2023): Hmm, I think that's fairly good news for us (Windows Terminal) at least, because it implies that this is indeed a bug in TSF. (An application shouldn't notice a difference in behavior between the legacy and new IME after all.) Would it be fine for us to only support the new IME? BTW I tried to reproduce the issue with the legacy IME and again couldn't get it to reproduce it. 😟 I don't speak Japanese, so maybe I'm just not using the right input? I saw you write "kokodenennkannn" and tried that and that didn't do anything for me. Do you use any specific input that reliably reproduces the issue for you after a few tries?

claunia commented

@hnakamur commented on GitHub (Sep 7, 2023):

I suppose the support for the old IME would be nice, but I am fine without it.

I was using 「日本語を入力して」"nihongowonyuuryokusite" to reproduce the issue.

@hnakamur commented on GitHub (Sep 7, 2023): I suppose the support for the old IME would be nice, but I am fine without it. I was using 「日本語を入力して」"nihongowonyuuryokusite" to reproduce the issue.

claunia commented

@lhecker commented on GitHub (Sep 7, 2023):

I was using 「日本語を入力して」"nihongowonyuuryokusite" to reproduce the issue.

Thanks! But I still can't get it to reproduce the issue. 🥲 I just genuinely can't figure this out.
And I noticed I haven't said this before: I'm sorry that it took so long for us to try and address this bug.

I suppose the support for the old IME would be nice, but I am fine without it.

One of my main tasks has been to improve the Internationalization of this project for a while now and I don't intend to give up just yet. 🙂 I was hoping to unify conhost's IME with the one in Windows Terminal at some point. conhost's IME implementation has two benefits: It uses TSF directly (without the WinRT CoreTextEditContext wrapper; less code = less bugs), and it writes the text during composition directly into the terminal viewport, so when you finish the composition it doesn't look quite as weird as it does now:

vs.

(I prefer the latter a lot.)

I'm just not sure when I can get to that. There's a lot of remaining Internationalization bugs in this project after all... But there's a good chance that using TSF directly without the WinRT wrapper will allow us to properly support the legacy IME and fix all those weird edge cases.
(Korean and Vietnamese in particular have a very difficult IME. By fixing those 2 IMEs, I broke the Japanese legacy IME. 😟 The conhost TSF/IME implementation never had those issues to begin with, thanks to its tight TSF integration.)

@zadjii-msft Should we close this bug, or repurpose it to track the Japanese legacy IME being broken? I think the latter might be good. This issue would get fixed properly if we ever manage to integrate the conhost TSF implementation into WT.

@lhecker commented on GitHub (Sep 7, 2023): > I was using 「日本語を入力して」"nihongowonyuuryokusite" to reproduce the issue. Thanks! But I still can't get it to reproduce the issue. 🥲 I just genuinely can't figure this out. And I noticed I haven't said this before: I'm sorry that it took so long for us to try and address this bug. > I suppose the support for the old IME would be nice, but I am fine without it. One of my main tasks has been to improve the Internationalization of this project for a while now and I don't intend to give up just yet. 🙂 I was hoping to unify conhost's IME with the one in Windows Terminal at some point. conhost's IME implementation has two benefits: It uses TSF directly (without the WinRT `CoreTextEditContext` wrapper; less code = less bugs), and it writes the text during composition directly into the terminal viewport, so when you finish the composition it doesn't look quite as weird as it does now: ![image](https://github.com/microsoft/terminal/assets/2256941/e5f7670c-e8ba-4807-acea-2ae04e72d340) vs. ![image](https://github.com/microsoft/terminal/assets/2256941/7fa3194b-78e6-4a4a-bfc0-2437078c4cfb) (I prefer the latter a lot.) I'm just not sure when I can get to that. There's a lot of remaining Internationalization bugs in this project after all... But there's a good chance that using TSF directly without the WinRT wrapper will allow us to properly support the legacy IME and fix all those weird edge cases. (Korean and Vietnamese in particular have a very difficult IME. By fixing those 2 IMEs, I broke the Japanese legacy IME. 😟 The conhost TSF/IME implementation never had those issues to begin with, thanks to its tight TSF integration.) --- @zadjii-msft Should we close this bug, or repurpose it to track the Japanese legacy IME being broken? I think the latter might be good. This issue would get fixed properly if we ever manage to integrate the conhost TSF implementation into WT.

claunia commented