[PR #871] Fix signatures of some callback functions #24358

Open
opened 2026-01-31 09:02:47 +00:00 by claunia · 0 comments
Owner

Original Pull Request: https://github.com/microsoft/terminal/pull/871

State: closed
Merged: Yes


Summary of the Pull Request

This PR fixes a bunch of callback function signatures so that they align with the signatures expected by Windows API. Most changes are from __cdecl to __stdcall, which matters on x86, but not on x64 or ARM64.

PR Checklist

  • Closes #xxx
  • CLA signed. If not, go over here and sign the CLA
  • Tests added/passed
  • Requires documentation to be updated
  • I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx

Detailed Description of the Pull Request / Additional comments

I find it hard to believe that conhost.exe as shipped in Windows would use callback functions with incorrect calling convention. Perhaps the version that ships with Windows is compiled with /Gz (__stdcall-by-default)? But OpenConsole.sln isn't.

All of these functions were found manually, so it's likely that there are more issues like this that I didn't find. It would be great if MSVC had a warning similar to GCC's -Wcast-function-type.

The only one signature incompatibility I found that wasn't eliminated was with FONTENUMPROC (I only fixed the calling convention, which is most important anyway), since a slightly different signature is being used that is similar to EnumFontFamProc rather than EnumFontFamExProc/FONTENUMPROC, and a slightly more invasive change would be needed.

**Original Pull Request:** https://github.com/microsoft/terminal/pull/871 **State:** closed **Merged:** Yes --- ## Summary of the Pull Request This PR fixes a bunch of callback function signatures so that they align with the signatures expected by Windows API. Most changes are from `__cdecl` to `__stdcall`, which matters on x86, but not on x64 or ARM64. <!-- Please review the items on the PR checklist before submitting--> ## PR Checklist * [ ] Closes #xxx * [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA * [ ] Tests added/passed * [ ] Requires documentation to be updated * [ ] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #xxx <!-- Provide a more detailed description of the PR, other things fixed or any additional comments/features here --> ## Detailed Description of the Pull Request / Additional comments I find it hard to believe that conhost.exe as shipped in Windows would use callback functions with incorrect calling convention. Perhaps the version that ships with Windows is compiled with `/Gz` (`__stdcall`-by-default)? But OpenConsole.sln isn't. All of these functions were found manually, so it's likely that there are more issues like this that I didn't find. It would be great if MSVC had a warning similar to GCC's `-Wcast-function-type`. The only one signature incompatibility I found that wasn't eliminated was with `FONTENUMPROC` (I only fixed the calling convention, which is most important anyway), since a slightly different signature is being used that is similar to [`EnumFontFamProc`](https://docs.microsoft.com/en-us/previous-versions/dd162621(v%3Dvs.85)) rather than [`EnumFontFamExProc`/`FONTENUMPROC`](https://docs.microsoft.com/en-us/previous-versions//dd162618(v=vs.85)), and a slightly more invasive change would be needed.
claunia added the pull-request label 2026-01-31 09:02:47 +00:00
Sign in to join this conversation.
No Label pull-request
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#24358