starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-03 21:25:34 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

Reading extended attributes (RGB colors) from the screen buffer #431

New Issue
Open
opened 2026-01-30 21:52:15 +00:00 by claunia · 15 comments
claunia commented 2026-01-30 21:52:15 +00:00
Owner
Copy Link

Originally created by @alabuzhev on GitHub (Oct 21, 2018).

TL;DR: How?

Scenario: an app needs to read a part of the screen buffer, output something, and then, when needed, restore the original buffer content.

ReadConsoleOutput always worked seamlessly for that, but now Windows supports 256 colours, true colours and underline text, but CHAR_INFO.Attributes is still WORD and obviously can't hold all that luxury.

I remember reading somewhere that you don't have plans to extend output APIs (WriteConsoleOutput and friends) because it can be achieved with VT sequences (with limitations, inconvenience, more work, but, technically - yes, it can), but what about reading?

I tried to search first and found this comment, but I don't think that #57 would help here - no 3rd-party terminals involved in this case, we're reading purely static data.

screen

Originally created by @alabuzhev on GitHub (Oct 21, 2018). TL;DR: How? Scenario: an app needs to read a part of the screen buffer, output something, and then, when needed, restore the original buffer content. ReadConsoleOutput always worked seamlessly for that, but now Windows supports 256 colours, true colours and underline text, but CHAR_INFO.Attributes is still WORD and obviously can't hold all that luxury. I remember reading somewhere that you don't have plans to extend output APIs (WriteConsoleOutput and friends) because it can be achieved with VT sequences (with limitations, inconvenience, more work, but, technically - yes, it can), but what about reading? I tried to search first and found [this](https://github.com/Microsoft/console/issues/88#issuecomment-387777464) comment, but I don't think that #57 would help here - no 3rd-party terminals involved in this case, we're reading purely static data. ![screen](https://user-images.githubusercontent.com/11453922/47342037-8cc79d80-d69a-11e8-83b8-d9d12747d7b4.png)
claunia added the Issue-FeatureProduct-ConhostArea-Server labels 2026-01-30 21:52:16 +00:00
claunia commented 2026-01-30 21:52:16 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Oct 22, 2018):

You're definitely right that this isn't something that is possible with VT sequences alone. In fact, there aren't any VT sequences (that I'm aware of) that allow the client application to query the contents of the buffer. The entire concept of having a query-able buffer is unique to Windows console applications.

We're trying to move the Windows Console environment to be more akin to that of linux terminals. If you were to write a command-line application that needed to query the buffer contents to work properly, then that application would be pretty much impossible to port to *nix (should the developer ever choose to).

My advice for best-practice when using the console API is to only read input to the commandline application and only write output - DON'T be writing input to the console or reading output from the console. If you follow those guidelines, your applications will end up typically simpler and more portable to other platforms.

Is there a reason you need to read part of the buffer to be able to restore it? Couldn't you reconstruct the output from the data you previously output to the console?

@zadjii-msft commented on GitHub (Oct 22, 2018): You're definitely right that this isn't something that is possible with VT sequences alone. In fact, there aren't _any_ VT sequences (that I'm aware of) that allow the client application to query the contents of the buffer. The entire concept of having a query-able buffer is unique to Windows console applications. We're trying to move the Windows Console environment to be more akin to that of linux terminals. If you were to write a command-line application that **needed** to query the buffer contents to work properly, then that application would be pretty much impossible to port to *nix (should the developer ever choose to). My advice for best-practice when using the console API is to only read input to the commandline application and only write output - DON'T be writing input to the console or reading output from the console. If you follow those guidelines, your applications will end up typically simpler and more portable to other platforms. Is there a reason you need to read part of the buffer to be able to restore it? Couldn't you reconstruct the output from the data you previously output to the console?
claunia commented 2026-01-30 21:52:17 +00:00
Author
Owner
Copy Link

@oising commented on GitHub (Oct 22, 2018):

As an aside, the VT420 had a DECCRA sequence which could copy rectangular areas of the buffer, but I don't think it was defined in the spec - nor ever implemented in any terminals - to keep the colour information.

I could imagine putting a kind of a "event sourcing" buffer in front of conpty, but that would be a lot of work to create aggregates and snapshots. But a special kind of masochistic fun work nonetheless :)

@oising commented on GitHub (Oct 22, 2018): As an aside, the VT420 had a DECCRA sequence which could copy rectangular areas of the buffer, but I don't think it was defined in the spec - nor ever implemented in any terminals - to keep the colour information. I could imagine putting a kind of a "event sourcing" buffer in front of conpty, but that would be a lot of work to create aggregates and snapshots. But a special kind of masochistic fun work nonetheless :)
claunia commented 2026-01-30 21:52:17 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Oct 22, 2018):

@oising That is true - though that seems to only be able to copy the buffer contents to another region in the buffer. It doesn't seem to return the contents to the client application in a usable way. There's another function I came across that IIRC calculated a hash of a buffer region, so that a client application could compare two regions to see if they're the same, or if the contents of a region changed, but again, the client app couldn't reverse engineer the actual contents of the buffer.

@zadjii-msft commented on GitHub (Oct 22, 2018): @oising That is true - though that seems to only be able to copy the buffer contents to another region in the buffer. It doesn't seem to return the contents to the client application in a usable way. There's another function I came across that IIRC calculated a hash of a buffer region, so that a client application could compare two regions to see if they're the same, or if the contents of a region changed, but again, the client app couldn't reverse engineer the actual contents of the buffer.
claunia commented 2026-01-30 21:52:17 +00:00
Author
Owner
Copy Link

@oising commented on GitHub (Oct 22, 2018):

Certain terminals do support printing the contents, like xterm - so somehow it's being done. It's such a pity we're all being held to this ancient stream oriented way of dealing with the screen. With respect to DECCRA, I guess it could be used to copy the visible contents into the scrollback buffer as a kind of poor man's save/restore of a given area, but you're right - there's no way to serialize it back out.

@oising commented on GitHub (Oct 22, 2018): Certain terminals do support printing the contents, like xterm - so somehow it's being done. It's such a pity we're all being held to this ancient stream oriented way of dealing with the screen. With respect to DECCRA, I guess it could be used to copy the visible contents into the scrollback buffer as a kind of poor man's save/restore of a given area, but you're right - there's no way to serialize it back out.
claunia commented 2026-01-30 21:52:17 +00:00
Author
Owner
Copy Link

@oising commented on GitHub (Oct 22, 2018):

Btw, technically you could use the hashing to test each cell of an area against a pre-computed "rainbow table" of known cell/attribute vales. Disgusting, but feasible.

@oising commented on GitHub (Oct 22, 2018): Btw, technically you could use the hashing to test each cell of an area against a pre-computed "rainbow table" of known cell/attribute vales. Disgusting, but feasible.
claunia commented 2026-01-30 21:52:17 +00:00
Author
Owner
Copy Link

@alabuzhev commented on GitHub (Oct 22, 2018):

@zadjii-msft , thanks, that's a very good point.

It is gratifying to hear that all these improvements are not just to support more programs and increase market share, but Windows Console team also has a non-NIH, non-vendor-lock-in attitude and is concerned about porting 3rd-party apps to other platforms :)

However, it's a bit too late to stay pure. 30 years too late actually. Reading the buffer has been supported since forever, as well as other unique features of Windows Console (and that's why we love it). Apps use them and you can't drop anything because compatibility.

So it all comes down to this:

  • There's a well-designed, easy-to-use API, which has been there forever and is not going anywhere, but now it cannot work with the full range of possible values - it basically corrupts the data (but only sometimes), and there's no replacement.

Is there a reason you need to read part of the buffer to be able to restore it? Couldn't you reconstruct the output from the data you previously output to the console?

Ok, why exactly we do this and what exactly we do there:
The app is a classic OFM and conceptually can be considered as a command processor with advanced features on top of it (directory listing, text editor, dialogs etc.)
It uses the console buffer to render its interface.
At some point the user wants to use the "command processor" feature to launch something:

  • We hide the interface, "prepare the canvas" and create a child process.
  • The child process does its things and exits. It might output something. We don't know what and where and how much.
  • We read the visible area of the console buffer ("take snapshot") into memory and restore the interface.
  • If the user wants to inspect the output of the program - they press a hotkey ( Ctrl+O or Ctrl+F1 or Ctrl+Up etc.), we z-reorder interface layers or hide some of them, making the snapshot partially of fully visible.

Now, if the launched program decides to use true colours we still can take a snapshot, and it even might be readable (thanks to the approximation you're doing there), but obviously all the colours will be messed up.

@alabuzhev commented on GitHub (Oct 22, 2018): @zadjii-msft , thanks, that's a very good point. It is gratifying to hear that all these improvements are not just to support more programs and increase market share, but Windows Console team also has a non-NIH, non-vendor-lock-in attitude and is concerned about porting 3rd-party apps to other platforms :) However, it's a bit too late to stay pure. 30 years too late actually. Reading the buffer has been supported since forever, as well as other unique features of Windows Console (and that's why we love it). Apps use them and you can't drop anything because compatibility. So it all comes down to this: - There's a well-designed, easy-to-use API, which has been there forever and is not going anywhere, but now it cannot work with the full range of possible values - it basically corrupts the data (but only sometimes), and there's no replacement. > Is there a reason you need to read part of the buffer to be able to restore it? Couldn't you reconstruct the output from the data you previously output to the console? Ok, why exactly we do this and what exactly we do there: [The app](https://farmanager.com) is a classic [OFM](https://en.wikipedia.org/wiki/Orthodox_file_manager) and conceptually can be considered as a command processor with advanced features on top of it (directory listing, text editor, dialogs etc.) It uses the console buffer to render its interface. At some point the user wants to use the "command processor" feature to launch something: - We hide the interface, "prepare the canvas" and create a child process. - The child process does its things and exits. It might output something. We don't know what and where and how much. - We read the visible area of the console buffer ("take snapshot") into memory and restore the interface. - If the user wants to inspect the output of the program - they press a hotkey ( <kbd>Ctrl</kbd>+<kbd>O</kbd> or <kbd>Ctrl</kbd>+<kbd>F1</kbd> or <kbd>Ctrl</kbd>+<kbd>Up</kbd> etc.), we z-reorder interface layers or hide some of them, making the snapshot partially of fully visible. Now, if the launched program decides to use true colours we still can take a snapshot, and it even might be readable (thanks to the approximation you're doing there), but obviously all the colours will be messed up.
claunia commented 2026-01-30 21:52:18 +00:00
Author
Owner
Copy Link

@eryksun commented on GitHub (Oct 23, 2018):

The comparison to the Windows console is a stretch of the imagination, but the Linux console bears mentioning. It's implemented as virtual consoles named /dev/tty[1-63] that emulate a VT102 terminal. From an X11 session, they're commonly accessed via Ctrl+Alt+F[1-6]. /dev/console and /dev/tty0 reference the current virtual console, which is not necessarily the controlling terminal of the current process (i.e. /dev/tty). Of interest to this issue are the /dev/vcs[a][1-63] devices. A vcs[a] device allows reading the screen buffer of the corresponding virtual console, with or without [a]ttributes.

Currently, the Windows console provides access to its input buffer and screen buffers via files on \Device\ConDrv, such as Console (reads from the input buffer, and writes to the active screen buffer), CurrentIn, and CurrentOut. These files are surfaced in the Windows API as the DOS devices CON, CONIN$ and CONOUT$. Special functions are required to write to the input buffer (WriteConsoleInput) and read from the screen buffer (ReadConsoleOutput, ReadConsoleOutputCharacter, ReadConsoleOutputAttribute). I'd like to see an alternative that uses VT streams with generic I/O functions instead of INPUT_RECORD and CHAR_INFO arrays with specialized functions.

Another set of ConDrv files could be added that explicitly support reading and writing of VT sequences as UTF-8 text via ReadFile and WriteFile (but not ReadConsole or WriteConsole) . For the sake of discussion, call these files VirtualTerminal (reads from the input buffer, and writes to the screen buffer), VTCurrentIn, and VTCurrentOut. Surface them as the Windows devices TTY, TTYIN, and TTYOUT, respectively, but require the \\.\ or \\?\ local-device prefix rather than adding legacy DOS devices. To read the screen as VT text, open "\\?\TTYOUT" with read access (or read-write) and call ReadFile. Use the lpOverlapped parameter to set an initial offset, else use the current cursor position.

@eryksun commented on GitHub (Oct 23, 2018): The comparison to the Windows console is a stretch of the imagination, but the [Linux console](https://en.wikipedia.org/wiki/Linux_console) bears mentioning. It's implemented as virtual consoles named /dev/tty[1-63] that emulate a VT102 terminal. From an X11 session, they're commonly accessed via Ctrl+Alt+F[1-6]. /dev/console and /dev/tty0 reference the current virtual console, which is not necessarily the controlling terminal of the current process (i.e. /dev/tty). Of interest to this issue are the /dev/vcs[a][1-63] devices. A vcs[a] device allows reading the screen buffer of the corresponding virtual console, with or without [a]ttributes. Currently, the Windows console provides access to its input buffer and screen buffers via files on \Device\ConDrv, such as Console (reads from the input buffer, and writes to the active screen buffer), CurrentIn, and CurrentOut. These files are surfaced in the Windows API as the DOS devices CON, CONIN$ and CONOUT$. Special functions are required to write to the input buffer (`WriteConsoleInput`) and read from the screen buffer (`ReadConsoleOutput`, `ReadConsoleOutputCharacter`, `ReadConsoleOutputAttribute`). I'd like to see an alternative that uses VT streams with generic I/O functions instead of `INPUT_RECORD` and `CHAR_INFO` arrays with specialized functions. Another set of ConDrv files could be added that explicitly support reading and writing of VT sequences as UTF-8 text via `ReadFile` and `WriteFile` (but not `ReadConsole` or `WriteConsole`) . For the sake of discussion, call these files VirtualTerminal (reads from the input buffer, and writes to the screen buffer), VTCurrentIn, and VTCurrentOut. Surface them as the Windows devices TTY, TTYIN, and TTYOUT, respectively, but require the \\\\.\ or \\\\?\ local-device prefix rather than adding legacy DOS devices. To read the screen as VT text, open "\\\\?\TTYOUT" with read access (or read-write) and call `ReadFile`. Use the `lpOverlapped` parameter to set an initial offset, else use the current cursor position.
claunia commented 2026-01-30 21:52:18 +00:00
Author
Owner
Copy Link

@HBelusca commented on GitHub (Oct 26, 2018):

@eryksun : Note that then this means that the conhost has to maintain in memory an extended representation of the buffer that can be read from (and/or written to??) sequentially and that mirrors the very internal buffer (that certainly stores the info in a completely different way) the conhost uses to display in a terminal window.

Re. the condrv files, there are some that are "exposed" in the embedded strings of the driver:
"\Input", "\Output", "\Display", "\ScreenBuffer" ; it would be interesting to hear whether they (in particular the last two) couldn't be used for such a purpose.

@HBelusca commented on GitHub (Oct 26, 2018): @eryksun : Note that then this means that the conhost has to maintain in memory an extended representation of the buffer that can be read from (and/or written to??) sequentially and that mirrors the very internal buffer (that certainly stores the info in a completely different way) the conhost uses to display in a terminal window. Re. the condrv files, there are some that are "exposed" in the embedded strings of the driver: "\Input", "\Output", "\Display", "\ScreenBuffer" ; it would be interesting to hear whether they (in particular the last two) couldn't be used for such a purpose.
claunia commented 2026-01-30 21:52:18 +00:00
Author
Owner
Copy Link

@alabuzhev commented on GitHub (Oct 27, 2018):

To read the screen as VT text

@eryksun, reading the screen as VT text is probably the last thing I'd like to do here:

  • Implementing a VT parser in every single program that needs this feature? It's not like it's not doable, but there are better things in life.
  • The console host will need to keep a VT representation of the whole buffer somehow, and update/patch it every time someone outputs something into random places using VT or Console API.
  • VT attributes are sticky. What if the output is something like \033[38;2;42;117;94blablablafoobar and I need to read the foo part only? How should I know that there was some colour override somewhere earlier? Or should the host simply take the attributes of the initial offset / cursor position, convert them to VT and prepend to the result string?

If we're unlucky and ReadConsoleOutputEx will never happen, I'd prefer ReadFile(console_handle, ...) to return an array of fixed-length structures instead, CHAR_INFO-like (character + attributes).

@alabuzhev commented on GitHub (Oct 27, 2018): > To read the screen as VT text @eryksun, reading the screen *as VT text* is probably the last thing I'd like to do here: - Implementing a VT parser in every single program that needs this feature? It's not like it's not doable, but there are better things in life. - The console host will need to keep a VT representation of the whole buffer somehow, and update/patch it every time someone outputs something into random places using VT or Console API. - VT attributes are sticky. What if the output is something like `\033[38;2;42;117;94blablablafoobar` and I need to read the `foo` part only? How should I know that there was some colour override somewhere earlier? Or should the host simply take the attributes of the initial offset / cursor position, convert them to VT and prepend to the result string? If we're unlucky and ReadConsoleOutput**Ex** will never happen, I'd prefer `ReadFile(console_handle, ...)` to return an array of fixed-length structures instead, CHAR_INFO-like (character + attributes).
claunia commented 2026-01-30 21:52:18 +00:00
Author
Owner
Copy Link

@HBelusca commented on GitHub (Oct 27, 2018):

As an alternative I don't have an idea whether it would be possible to obtain some kind of shared memory buffer from conhost and available in the (attached) console app that requests it, that translates the contents of the screen.

@HBelusca commented on GitHub (Oct 27, 2018): As an alternative I don't have an idea whether it would be possible to obtain some kind of shared memory buffer from conhost and available in the (attached) console app that requests it, that translates the contents of the screen.
claunia commented 2026-01-30 21:52:18 +00:00
Author
Owner
Copy Link

@oising commented on GitHub (Dec 1, 2018):

I suspect Microsoft are going to surface their VT parser/buffer they built for conpty at some point in the future when it is stabilized, and this will make life far easier. It needs to get to at least VT420+ level to cover the majority of uses, imo. Oh, and obviously have dot net bindings, and be using spans, pipelines and as much allocation free code they can muster.

@oising commented on GitHub (Dec 1, 2018): I suspect Microsoft are going to surface their VT parser/buffer they built for conpty at some point in the future when it is stabilized, and this will make life far easier. It needs to get to at least VT420+ level to cover the majority of uses, imo. Oh, and obviously have dot net bindings, and be using spans, pipelines and as much allocation free code they can muster.
claunia commented 2026-01-30 21:52:19 +00:00
Author
Owner
Copy Link

@miniksa commented on GitHub (May 18, 2019):

@oising, good suspicion.

@miniksa commented on GitHub (May 18, 2019): @oising, good suspicion.
claunia commented 2026-01-30 21:52:19 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Jul 17, 2023):

horrifying thoughts:

  • GetConsoleScreenBufferInfoEx always had a dwSize we could have always added this!
  • oh no we can't the size is literally checked in conclnt, and fails if it's not the right size. Doesn't even pass the request to the console server.
  • oh no we still can't because conclnt unpacks the result from the API and then repacks into the result struct.
@zadjii-msft commented on GitHub (Jul 17, 2023): horrifying thoughts: * `GetConsoleScreenBufferInfoEx` always had a `dwSize` we could have always added this! * oh no we can't the size is literally checked in conclnt, and fails if it's not the right size. Doesn't even pass the request to the console server. * oh no we still can't because conclnt unpacks the result from the API and then repacks into the result struct.
claunia commented 2026-01-30 21:52:19 +00:00
Author
Owner
Copy Link

@alabuzhev commented on GitHub (Jul 17, 2023):

@zadjii-msft please see a related discussion.
TL;DR: we read the output only to save it, write something on top and then restore it, similarly to what cmd.exe does on F7 & F9. We don't really need to see the actual characters, colors, styles etc., so fully opaque APIs or sequences to take / restore / drop snapshots would do fine.

@alabuzhev commented on GitHub (Jul 17, 2023): @zadjii-msft please see a [related discussion](https://github.com/microsoft/terminal/issues/10810#issuecomment-898581819). TL;DR: we read the output only to save it, write something on top and then restore it, similarly to what cmd.exe does on F7 & F9. We don't really need to see the actual characters, colors, styles etc., so fully opaque APIs or sequences to take / restore / drop snapshots would do fine.
claunia commented 2026-01-30 21:52:20 +00:00
Author
Owner
Copy Link

@zadjii-msft commented on GitHub (Jul 18, 2023):

Oh, for sure. I remember that thread, that had some good solutions. I was mostly jotting down quick notes we had that came up during another discussion. There's probably folks out there where the "layers" / rectangular operations might not work. For them, full rgb data returned is still probably useful. And doing so probably would have been a lot easier than we thought 🤦

@zadjii-msft commented on GitHub (Jul 18, 2023): Oh, for sure. I remember that thread, that had some good solutions. I was mostly jotting down quick notes we had that came up during another discussion. There's probably folks out there where the "layers" / rectangular operations might not work. For them, full rgb data returned is still probably useful. And doing so probably would have been a lot easier than we thought 🤦
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Server Issue-Feature Product-Conhost
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#431
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?