Does CreatePseudoConsole accept Socket handle? #12795

Closed
opened 2026-01-31 03:25:07 +00:00 by claunia · 11 comments
Owner

Originally created by @skyline75489 on GitHub (Mar 2, 2021).

I brought this up to Dustin yesterday. Feel like should also open a thread here.

Background

We know that on Linux file descriptor is a "universal" thing that can be used both for socket & file. However, on Windows file handle(HANDLE) & socket handle (SOCKET) are two different things. Despite the fact that socket handle from a Winsock provider can be used with other non-Winsock functions, we can't do the other way around, that is passing a file handle into a function that accepts socket handle.

I'm trying to port tmux on Windows which uses libevent a lot. Most of Libevent's APIs utilizes file descriptor, which make it suitable for both file & networking programming on Linux. In tmux, the IO of tty device (/dev/tty) is also managed by libevent using the same FD interface. Underneath the interface, libevent uses recv on Linux (again, works for both file & socket), and uses WSARecv on Windows (which only works for SOCKET handle).

ConPTY Interface

The CreatePseudoConsole interface is defined as this:

HRESULT WINAPI CreatePseudoConsole(
    _In_ COORD size,
    _In_ HANDLE hInput,
    _In_ HANDLE hOutput,
    _In_ DWORD dwFlags,
    _Out_ HPCON* phPC
);

The example in the documentation uses HANDLE created by CreatePipe. The documentation does not mention restriction about passing a SOCKET handle.

Question

So here's the problem. I can not use HANDLE created by CreatePipe because it won't work in libevent. I may be able to use SOCKET instead, which works in libevent. But I don't know if it works in ConPTY.

I've tried to pass TCP/UDP sockets like this:

// conpty_fd is a TCP/UDP socket.
hr = CreatePseudoConsole(coord, conpty_fd, conpty_fd, 0, &hPC);

CreateProcess does work. But the process exits immediately after creation (cmd.exe, powershell.exe).

I've also tried this:

// pipe is created by `CreatePipe`
hr = CreatePseudoConsole(coord, pipe, conpty_fd, 0, &hPC);

CreateProcess still works, and the process survives. But I'm not seeing anything written into the socket.

Originally created by @skyline75489 on GitHub (Mar 2, 2021). I brought this up to Dustin yesterday. Feel like should also open a thread here. ## Background We know that on Linux file descriptor is a "universal" thing that can be used both for socket & file. However, on Windows file handle(`HANDLE`) & socket handle (`SOCKET`) are two different things. Despite the fact that [socket handle from a Winsock provider can be used with other non-Winsock functions](https://docs.microsoft.com/en-us/windows/win32/winsock/socket-handles-2), we can't do the other way around, that is passing a file handle into a function that accepts socket handle. I'm trying to port [tmux](https://github.com/tmux/tmux) on Windows which uses libevent a lot. Most of Libevent's APIs utilizes file descriptor, which make it suitable for both file & networking programming on Linux. In tmux, the IO of tty device (`/dev/tty`) is also managed by libevent using the same FD interface. Underneath the interface, libevent uses `recv` on Linux (again, works for both file & socket), and uses `WSARecv` on Windows (which only works for SOCKET handle). ## ConPTY Interface The `CreatePseudoConsole` interface is defined as this: ```cpp HRESULT WINAPI CreatePseudoConsole( _In_ COORD size, _In_ HANDLE hInput, _In_ HANDLE hOutput, _In_ DWORD dwFlags, _Out_ HPCON* phPC ); ``` The example in the documentation uses HANDLE created by `CreatePipe`. The documentation does not mention restriction about passing a `SOCKET` handle. ## Question So here's the problem. I can not use HANDLE created by `CreatePipe` because it won't work in libevent. I may be able to use SOCKET instead, which works in libevent. But I don't know if it works in ConPTY. I've tried to pass TCP/UDP sockets like this: ```cpp // conpty_fd is a TCP/UDP socket. hr = CreatePseudoConsole(coord, conpty_fd, conpty_fd, 0, &hPC); ``` `CreateProcess` does work. But the process exits immediately after creation (cmd.exe, powershell.exe). I've also tried this: ```cpp // pipe is created by `CreatePipe` hr = CreatePseudoConsole(coord, pipe, conpty_fd, 0, &hPC); ``` `CreateProcess` still works, and the process survives. But I'm not seeing anything written into the socket.
Author
Owner

@zadjii-msft commented on GitHub (Mar 2, 2021):

Well, this is my API so I should know the answer here 😅 Unfortunately, I don't really know what makes SOCKETs and HANDLEs different on Windows. Guess I maybe should have investigated that more in 2018 😬

@miniksa usually does know more about these sorts of things. Any ideas?

@zadjii-msft commented on GitHub (Mar 2, 2021): Well, this is my API so I _should_ know the answer here 😅 Unfortunately, I don't really know what makes `SOCKET`s and `HANDLE`s different on Windows. Guess I maybe should have investigated that more in 2018 😬 @miniksa usually does know more about these sorts of things. Any ideas?
Author
Owner

@miniksa commented on GitHub (Mar 2, 2021):

Nothing in particular is checking the type of the HANDLE that is coming through to the psuedoconsole threads, as far as I'm aware. I believe they're just calling ReadFile and WriteFile on whatever handle they're given. The only caveat is that they can't be async (require OVERLAPPED). That's #262 and it just hasn't been a big enough problem to-date to implement.

You could.... replace your conhost.exe in system32 with a build of openconsole.exe, set up a post-mortem debugger in VS or WinDBG, and use the DebugLaunch flag (see srvinit.cpp ~ line 246) to make it break on startup into your debugger and trace around to see what might be going wrong.... that's what I'd do if I had time to dig at the moment.

@miniksa commented on GitHub (Mar 2, 2021): Nothing in particular is checking the type of the `HANDLE` that is coming through to the psuedoconsole threads, as far as I'm aware. I believe they're just calling `ReadFile` and `WriteFile` on whatever handle they're given. The only caveat is that they can't be async (require `OVERLAPPED`). That's #262 and it just hasn't been a big enough problem to-date to implement. You could.... replace your `conhost.exe` in system32 with a build of `openconsole.exe`, set up a post-mortem debugger in VS or WinDBG, and use the `DebugLaunch` flag (see srvinit.cpp ~ line 246) to make it break on startup into your debugger and trace around to see what might be going wrong.... that's what I'd do if I had time to dig at the moment.
Author
Owner

@eryksun commented on GitHub (Mar 2, 2021):

The only caveat is that they can't be async (require OVERLAPPED). That's #262 and it just hasn't been a big enough problem to-date to implement.

Winsock IFS sockets created with socket and accept are "\Device\Afd" files opened in asynchronous mode. See, for example, the documentation of the hFile parameter of ReadFile[Ex] and WriteFile[Ex].

@eryksun commented on GitHub (Mar 2, 2021): > The only caveat is that they can't be async (require OVERLAPPED). That's #262 and it just hasn't been a big enough problem to-date to implement. Winsock IFS sockets created with `socket` and `accept` are "\Device\Afd" files opened in asynchronous mode. See, for example, the documentation of the `hFile` parameter of `ReadFile[Ex]` and `WriteFile[Ex]`.
Author
Owner

@miniksa commented on GitHub (Mar 2, 2021):

The only caveat is that they can't be async (require OVERLAPPED). That's #262 and it just hasn't been a big enough problem to-date to implement.

Winsock IFS sockets created with socket and accept are "\Device\Afd" files opened in asynchronous mode. See, for example, the documentation of the hFile parameter of ReadFile[Ex] and WriteFile[Ex].

Welp, there we go then. It just takes doing #262 to be able to deal with an OVERLAPPED and it'd probably work.

@miniksa commented on GitHub (Mar 2, 2021): > > The only caveat is that they can't be async (require OVERLAPPED). That's #262 and it just hasn't been a big enough problem to-date to implement. > > Winsock IFS sockets created with `socket` and `accept` are "\Device\Afd" files opened in asynchronous mode. See, for example, the documentation of the `hFile` parameter of `ReadFile[Ex]` and `WriteFile[Ex]`. Welp, there we go then. It just takes doing #262 to be able to deal with an `OVERLAPPED` and it'd probably work.
Author
Owner

@skyline75489 commented on GitHub (Mar 2, 2021):

Thanks guys for the comments. You guys are awesome.

Is it possible for me to workaround this by making the socket blocking?

@skyline75489 commented on GitHub (Mar 2, 2021): Thanks guys for the comments. You guys are awesome. Is it possible for me to workaround this by making the socket blocking?
Author
Owner

@eryksun commented on GitHub (Mar 3, 2021):

Thanks guys for the comments. You guys are awesome.

Is it possible for me to workaround this by making the socket blocking?

I haven't experimented with using sockets with conpty. But you can get a synchronous-mode socket by calling WSASocketW directly, which allows creating a socket without the WSA_FLAG_OVERLAPPED flag. At the system I/O level, this corresponds to opening the file object in FILE_SYNCHRONOUS_IO_NONALERT mode. accept(s, ...) and WSAAccept(s, ...) will return a socket with the same synchronous or asynchronous mode as s.

@eryksun commented on GitHub (Mar 3, 2021): > Thanks guys for the comments. You guys are awesome. > > Is it possible for me to workaround this by making the socket blocking? I haven't experimented with using sockets with conpty. But you can get a synchronous-mode socket by calling [`WSASocketW`](https://docs.microsoft.com/en-us/windows/win32/api/winsock2/nf-winsock2-wsasocketw) directly, which allows creating a socket without the `WSA_FLAG_OVERLAPPED` flag. At the system I/O level, this corresponds to opening the file object in [`FILE_SYNCHRONOUS_IO_NONALERT`](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/ns-ntifs-_file_mode_information) mode. `accept(s, ...)` and `WSAAccept(s, ...)` will return a socket with the same synchronous or asynchronous mode as `s`.
Author
Owner

@skyline75489 commented on GitHub (Mar 3, 2021):

Thanks Eryk. You’re amazing! It works actually. Socket with conpty works as long as the socket has no overlapped attribute.

My next question would be is it possible to WSARecv on a file handle? For example, I need to pass the stdhandle into libevent. Inside libevent it will use WSA apis to manipulate the handle. I think it doesn't work as WSA api would complains about the file socket not being a valid socket. I wonder if there’s some kind of bridge that can connect file handle & socket handle.

获取 Outlook for iOShttps://aka.ms/o0ukef

@skyline75489 commented on GitHub (Mar 3, 2021): Thanks Eryk. You’re amazing! It works actually. Socket with conpty works as long as the socket has no overlapped attribute. My next question would be is it possible to WSARecv on a file handle? For example, I need to pass the stdhandle into libevent. Inside libevent it will use WSA apis to manipulate the handle. I think it doesn't work as WSA api would complains about the file socket not being a valid socket. I wonder if there’s some kind of bridge that can connect file handle & socket handle. 获取 Outlook for iOS<https://aka.ms/o0ukef>
Author
Owner

@miniksa commented on GitHub (Mar 3, 2021):

My next question would be is it possible to WSARecv on a file handle?

What do you mean, specifically? Are you saying that you're doing GetStdHandle(STD_OUTPUT_HANDLE) and the return type is HANDLE while WSARecv wants a SOCKET but you know that the object is a socket for another reason?

@miniksa commented on GitHub (Mar 3, 2021): > My next question would be is it possible to WSARecv on a file handle? What do you mean, specifically? Are you saying that you're doing `GetStdHandle(STD_OUTPUT_HANDLE)` and the return type is `HANDLE` while `WSARecv` wants a `SOCKET` but you know that the object is a socket for another reason?
Author
Owner

@eryksun commented on GitHub (Mar 3, 2021):

My next question would be is it possible to WSARecv on a file handle? For example, I need to pass the stdhandle into libevent. Inside libevent it will use WSA apis to manipulate the handle. I think it doesn't work as WSA api would complains about the file socket not being a valid socket. I wonder if there’s some kind of bridge that can connect file handle & socket handle.

Short story: Winsock socket operations, such as recv and select, do not work on a non-socket file. They will fail with the last error set to WSAENOTSOCK.

Very long story: I'll outline the problem and the context, as far as I know, but this doesn't include any simple way to wrap a non-socket file for use with socket functions...

A socket is a handle for a file object opened on the "Afd" device (created by the "Ancillary Function Driver"), which is associated with Winsock protocol and connection state. (The latter is the reason for WSADuplicateSocket. Technically, the socket context is registered with AFD, from which it can be retrieved later. So it's possible to use handle inheritance and DuplicateHandle. The main problem is/was Winsock layered service providers, but those are deprecated.)

A file object is created when a device object is opened. The driver for the device is sent an IRP_MJ_CREATE request that includes any remaining path in the namespace of the device. Exactly how the remaining path should be parsed is up to the device.

You're probably used to accessing files on a volume device (e.g. "C:\Windows\explorer.exe" might be "\Device\HarddiskVolume2\Windows\explorer.exe"). For a volume device that's mounted by a filesystem, the system uses the volume parameter block (VPB) of the device object to refer to the filesystem device that controls access to the volume and its namespace. The filesystem driver typically supports either opening the entire volume or a regular file or directory named by a path in the filesystem (e.g. "\Windows\explorer.exe"). The operations supported on the file object are entirely up to what's directly supported and allowed by the filesystem device.

Any other device type usually manages its own namespace and operations. It can implement a filesystem or a fixed set of virtual files. For example, "\Device\NamedPipe\" (note the trailing backslash) is an NPFS (named-pipe filesystem) directory that contains named pipes, "\Device\Mup" (Multiple UNC Provider) mounts UNC shares in its namespace (with the remaining path managed by a filesystem redirector device, e.g. for SMB shares), and the console's "ConDrv" device supports virtual files such as "\Device\ConDrv\Input". The AFD driver implements an "Endpoint" virtual file to create an enpoint. Winsock calls NtCreateFile on the path "\Device\Afd\Endpoint" in order to create the file object for a socket.

The I/O manager provides a set of common I/O functions for file objects. For example, NtReadFile sends an IRP_MJ_READ request to the driver that's associated with the file object. The base set of services is extended by IOCTL and FSCTL device and filesystem control operations -- some standard and some custom for a particular device or filesystem. These extended requests are respectively sent by calling NtDeviceIoControlFile and NtFsControlFile, which sends an IRP_MJ_DEVICE_CONTROL or IRP_MJ_FILE_SYSTEM_CONTROL request to the driver.

You probably see where this is going. The "receive" operation for a socket WSARecv is the IOCTL code 0x12017, which is supported only by the AFD driver. The IOCTL code is comprised of (FILE_DEVICE_NETWORK << 12 | 5 << 2 | METHOD_NEITHER), where FILE_DEVICE_NETWORK (18) is for any network device type such as an NDIS device (but AFD is actually of type FILE_DEVICE_NAMED_PIPE), the function code for the receive operation is 5, and METHOD_NEITHER (3) tells the I/O manager that it should neither buffer nor lock/map user buffers in system space.

0x12017 is a non-standard code. The standard format for a control code has the 16-bit device type in the upper word, the 2-bit required access left shifted by 14 bits, the 12-bit function code left shifted by 2 bits, and the buffering method in the lower 2 bits. When parsed normally, 0x12017 looks like a code for function 2053 on a beep-type device (FILE_DEVICE_BEEP). That said, since AFD is a private driver for Winsock, no one cares about its weird IOCTL codes.

If a non-socket file handle is passed to a Winsock function such as recv or WSARecv, it won't even get as far as sending the receive device control request. Since Winsock doesn't have any association with the handle, it tries to look up the (presumably) previously stored socket context using the control code 0x12043. This code won't be supported by any driver other than AFD, so the request will fail with STATUS_INVALID_DEVICE_REQUEST, and Winsock will fail the call with the last error set to WSAENOTSOCK.

@eryksun commented on GitHub (Mar 3, 2021): > My next question would be is it possible to WSARecv on a file handle? For example, I need to pass the stdhandle into libevent. Inside libevent it will use WSA apis to manipulate the handle. I think it doesn't work as WSA api would complains about the file socket not being a valid socket. I wonder if there’s some kind of bridge that can connect file handle & socket handle. Short story: Winsock socket operations, such as `recv` and `select`, do not work on a non-socket file. They will fail with the last error set to `WSAENOTSOCK`. Very long story: I'll outline the problem and the context, as far as I know, but this doesn't include any simple way to wrap a non-socket file for use with socket functions... A socket is a handle for a file object opened on the "Afd" device (created by the "Ancillary Function Driver"), which is associated with Winsock protocol and connection state. (The latter is the reason for `WSADuplicateSocket`. Technically, the socket context is registered with AFD, from which it can be retrieved later. So it's possible to use handle inheritance and `DuplicateHandle`. The main problem is/was Winsock layered service providers, but those are deprecated.) A [file object](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ns-wdm-_file_object) is created when a [device object](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/wdm/ns-wdm-_device_object) is opened. The driver for the device is sent an `IRP_MJ_CREATE` request that includes any remaining path in the namespace of the device. Exactly how the remaining path should be parsed is up to the device. You're probably used to accessing files on a volume device (e.g. "C:\\Windows\\explorer.exe" might be "\\Device\\HarddiskVolume2\\Windows\\explorer.exe"). For a volume device that's mounted by a filesystem, the system uses the volume parameter block (VPB) of the device object to refer to the filesystem device that controls access to the volume and its namespace. The filesystem driver typically supports either opening the entire volume or a regular file or directory named by a path in the filesystem (e.g. "\\Windows\\explorer.exe"). The operations supported on the file object are entirely up to what's directly supported and allowed by the filesystem device. Any other device type usually manages its own namespace and operations. It can implement a filesystem or a fixed set of virtual files. For example, "\\Device\\NamedPipe\\" (note the trailing backslash) is an NPFS (named-pipe filesystem) directory that contains named pipes, "\\Device\\Mup" (Multiple UNC Provider) mounts UNC shares in its namespace (with the remaining path managed by a filesystem redirector device, e.g. for SMB shares), and the console's "ConDrv" device supports virtual files such as "\\Device\\ConDrv\\Input". The AFD driver implements an "Endpoint" virtual file to create an enpoint. Winsock calls `NtCreateFile` on the path "\\Device\\Afd\\Endpoint" in order to create the file object for a socket. The I/O manager provides a set of common I/O functions for file objects. For example, `NtReadFile` sends an `IRP_MJ_READ` request to the driver that's associated with the file object. The base set of services is extended by IOCTL and FSCTL device and filesystem control operations -- some standard and some custom for a particular device or filesystem. These extended requests are respectively sent by calling [`NtDeviceIoControlFile`](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-zwdeviceiocontrolfile) and [`NtFsControlFile`](https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/ntifs/nf-ntifs-zwfscontrolfile), which sends an `IRP_MJ_DEVICE_CONTROL` or `IRP_MJ_FILE_SYSTEM_CONTROL` request to the driver. You probably see where this is going. The "receive" operation for a socket `WSARecv` is the IOCTL code 0x12017, which is supported only by the AFD driver. The IOCTL code is comprised of `(FILE_DEVICE_NETWORK << 12 | 5 << 2 | METHOD_NEITHER)`, where `FILE_DEVICE_NETWORK` (18) is for any network device type such as an NDIS device (but AFD is actually of type `FILE_DEVICE_NAMED_PIPE`), the function code for the receive operation is 5, and `METHOD_NEITHER` (3) tells the I/O manager that it should neither buffer nor lock/map user buffers in system space. 0x12017 is a non-standard code. The standard format for a control code has the 16-bit device type in the upper word, the 2-bit required access left shifted by 14 bits, the 12-bit function code left shifted by 2 bits, and the buffering method in the lower 2 bits. When parsed normally, 0x12017 looks like a code for function 2053 on a beep-type device (`FILE_DEVICE_BEEP`). That said, since AFD is a private driver for Winsock, no one cares about its weird IOCTL codes. If a non-socket file handle is passed to a Winsock function such as `recv` or `WSARecv`, it won't even get as far as sending the receive device control request. Since Winsock doesn't have any association with the handle, it tries to look up the (presumably) previously stored socket context using the control code 0x12043. This code won't be supported by any driver other than AFD, so the request will fail with `STATUS_INVALID_DEVICE_REQUEST`, and Winsock will fail the call with the last error set to `WSAENOTSOCK`.
Author
Owner

@skyline75489 commented on GitHub (Mar 3, 2021):

Thanks @eryksun so much for such a detailed writeup! I have no word but to thank you for the effort. And I’ve learned a lot from it.

Unfortunately this “file handle to socket handle” issue will be a huge blocker for tmux to work on Windows. I’ll close this issue now. I think people in the future will have the reference they need if they are tackling similar issues.

@skyline75489 commented on GitHub (Mar 3, 2021): Thanks @eryksun so much for such a detailed writeup! I have no word but to thank you for the effort. And I’ve learned a lot from it. Unfortunately this “file handle to socket handle” issue will be a huge blocker for tmux to work on Windows. I’ll close this issue now. I think people in the future will have the reference they need if they are tackling similar issues.
Author
Owner

@skyline75489 commented on GitHub (Mar 3, 2021):

@miniksa yeah basically I’m trying to do the impossible, to use the StdHandle as if it is on Linux and recv/select on it.

@skyline75489 commented on GitHub (Mar 3, 2021): @miniksa yeah basically I’m trying to do the impossible, to use the StdHandle as if it is on Linux and recv/select on it.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#12795