Incorrect colors returned by the ReadConsoleOutputAttribute API #8187

Open
opened 2026-01-31 01:22:54 +00:00 by claunia · 0 comments
Owner

Originally created by @j4james on GitHub (May 16, 2020).

Environment

Windows build number: Version 10.0.18362.719

Steps to reproduce

Compile and run the following C++ code.

#include <windows.h>
#include <conio.h>
#include <stdio.h>

void main()
{
  constexpr auto numColors = 24;
  HANDLE handle = GetStdHandle(STD_OUTPUT_HANDLE);

  // Enable VT mode.
  DWORD mode;
  GetConsoleMode(handle, &mode);
  mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
  SetConsoleMode(handle, mode);

  printf("\033[m\033[H\033[2J\033[3J"); // clear screen and scrollback
  for (auto i = 0; i < numColors; i++) {
    auto c = i*10 + 8;
    printf("\033[48;2;%d;%d;%dm ", c, c, c);
  }
  printf("\033[m <- Colors produced with VT sequences\n");

  CHAR_INFO info[numColors];
  SMALL_RECT region = { 0, 0, numColors, 1 };
  ReadConsoleOutput(handle, info, { numColors, 1 }, { 0, 0 }, &region);

  for (auto i = 0; i < numColors; i++) {
    SetConsoleTextAttribute(handle, info[i].Attributes);
    printf(" ");
  }
  printf("\033[m <- Colors seen by ReadConsoleOutput\n");

  WORD attr[numColors];
  DWORD read;
  ReadConsoleOutputAttribute(handle, attr, numColors, { 0, 0 }, &read);

  for (auto i = 0; i < numColors; i++) {
    SetConsoleTextAttribute(handle, attr[i]);
    printf(" ");
  }
  printf("\033[m <- Colors seen by ReadConsoleOutputAttribute\n");
}

This test case writes out a line of spaces in varying shades of gray using VT sequences to set the colors with RGB values. It then reads back those colors as legacy attributes using the ReadConsoleOutput and ReadConsoleOutputAttribute APIs, and writes out what they see.

Expected behavior

The two lines rendered with legacy attributes should at least approximate the RGB shades of gray.

Actual behavior

The ReadConsoleOutput API produces reasonable results (given the limitations of the 16 color palette), but the colors returned by the ReadConsoleOutputAttribute API are quite obviously wrong.

image

The problem is that ReadConsoleOutputAttribute is using the TextAttribute::GetLegacyAttributes method, which only works correctly for the first 16 indexed colors. ReadConsoleOutput uses the Settings::GenerateLegacyAttributes method, which produces decent results, although it can be quite slow.

I'd like to have a go at refactoring this code for a number of reasons:

  1. We obviously should be using a consistent legacy color calculation, that produces reasonable results, for all the console APIs.
  2. I think there are a few places in the code that are using TextAttribute::GetLegacyAttributes that don't actually need legacy attributes at all, and could avoid the color narrowing.
  3. I think the GenerateLegacyAttributes implementation could be made faster with a simple table lookup, instead of the current search for the closest HSL match in the color table (I suspect this might be responsible for issues like #783 and #3950).
Originally created by @j4james on GitHub (May 16, 2020). # Environment Windows build number: Version 10.0.18362.719 # Steps to reproduce Compile and run the following C++ code. ```cpp #include <windows.h> #include <conio.h> #include <stdio.h> void main() { constexpr auto numColors = 24; HANDLE handle = GetStdHandle(STD_OUTPUT_HANDLE); // Enable VT mode. DWORD mode; GetConsoleMode(handle, &mode); mode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING; SetConsoleMode(handle, mode); printf("\033[m\033[H\033[2J\033[3J"); // clear screen and scrollback for (auto i = 0; i < numColors; i++) { auto c = i*10 + 8; printf("\033[48;2;%d;%d;%dm ", c, c, c); } printf("\033[m <- Colors produced with VT sequences\n"); CHAR_INFO info[numColors]; SMALL_RECT region = { 0, 0, numColors, 1 }; ReadConsoleOutput(handle, info, { numColors, 1 }, { 0, 0 }, &region); for (auto i = 0; i < numColors; i++) { SetConsoleTextAttribute(handle, info[i].Attributes); printf(" "); } printf("\033[m <- Colors seen by ReadConsoleOutput\n"); WORD attr[numColors]; DWORD read; ReadConsoleOutputAttribute(handle, attr, numColors, { 0, 0 }, &read); for (auto i = 0; i < numColors; i++) { SetConsoleTextAttribute(handle, attr[i]); printf(" "); } printf("\033[m <- Colors seen by ReadConsoleOutputAttribute\n"); } ``` This test case writes out a line of spaces in varying shades of gray using VT sequences to set the colors with RGB values. It then reads back those colors as legacy attributes using the `ReadConsoleOutput` and `ReadConsoleOutputAttribute` APIs, and writes out what they see. # Expected behavior The two lines rendered with legacy attributes should at least approximate the RGB shades of gray. # Actual behavior The `ReadConsoleOutput` API produces reasonable results (given the limitations of the 16 color palette), but the colors returned by the `ReadConsoleOutputAttribute` API are quite obviously wrong. ![image](https://user-images.githubusercontent.com/4181424/82122093-6213cd00-9789-11ea-81a2-512efd94a696.png) The problem is that `ReadConsoleOutputAttribute` is using the `TextAttribute::GetLegacyAttributes` method, which only works correctly for the first 16 indexed colors. `ReadConsoleOutput` uses the `Settings::GenerateLegacyAttributes` method, which produces decent results, although it can be quite slow. I'd like to have a go at refactoring this code for a number of reasons: 1. We obviously should be using a consistent legacy color calculation, that produces reasonable results, for all the console APIs. 2. I think there are a few places in the code that are using `TextAttribute::GetLegacyAttributes` that don't actually need legacy attributes at all, and could avoid the color narrowing. 3. I think the `GenerateLegacyAttributes` implementation could be made faster with a simple table lookup, instead of the current search for the closest HSL match in the color table (I suspect this might be responsible for issues like #783 and #3950).
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#8187