Add support for OSC 8, Hyperlink, to the console infrastructure #300

@lypanov commented on GitHub (Jan 14, 2020):

@zadjii-msft My current specific interest could reduce this list even further. I'd like to add support for the OSC hyperlink extension to the wezterm (https://github.com/wez/wezterm) Windows port.

Reading over the changes from https://github.com/microsoft/terminal/pull/891/commits to add support for OSC 10/11 sounds like it has direct parallels in the changes I'll need for OSC 8.

I understand from the maintainer of wezterm that the VT sequences themselves are not directly passed on to wezterm. Is this due to the lack of "parse and re-emit" support in this repository? Am I correct in saying that OutputStateMachineEngine would be the correct place to add such support to? Is this source file part of "conhost"? I've read the sentence "pass them through to the terminal" here from you and also several times in the docs but don't understand exactly how this can relate to 3rd party users of ConPTY.

Final question... how can wezterm know about my changes to conhost? Do I have to somehow embed the open source fork? (My assumption is that wezterm right now is using a conhost provided by Windows 10).

Please correct any horrible assumptions / failures in understanding I might have. Thank you so much!

@lypanov commented on GitHub (Jan 14, 2020): @zadjii-msft My current specific interest could reduce this list even further. I'd like to add support for the OSC hyperlink extension to the wezterm (https://github.com/wez/wezterm) Windows port. Reading over the changes from https://github.com/microsoft/terminal/pull/891/commits to add support for OSC 10/11 sounds like it has direct parallels in the changes I'll need for OSC 8. I understand from the maintainer of wezterm that the VT sequences themselves are not directly passed on to wezterm. Is this due to the lack of "parse and re-emit" support in this repository? Am I correct in saying that OutputStateMachineEngine would be the correct place to add such support to? Is this source file part of "conhost"? I've read the sentence "pass them through to the terminal" here from you and also several times in the docs but don't understand exactly how this can relate to 3rd party users of ConPTY. Final question... how can wezterm know about my changes to conhost? Do I have to somehow embed the open source fork? (My assumption is that wezterm right now is using a conhost provided by Windows 10). Please correct any horrible assumptions / failures in understanding I might have. Thank you so much!

claunia commented

@egmontkob commented on GitHub (Jan 14, 2020):

Experiences from implementing it in gnome-terminal:

It was freaking easy to implement a demo, which was feature complete and everything, except the "tiny" issue that each character cell contained the entire target URL (in addition to the Unicode character and the SGR properties such as colors, boldness etc.), so the memory consumption went up from ~12 bytes per charcell to ~2k per charcell. Needless to say, this was a proof of concept, not meant to ever hit the codebase.

And then it was freaking hard to come up with a solution that consumes reasonable amount of memory, assuming reasonable use of the feature. The details probably don't really matter because gnome-terminal stores the onscreen data and the scrollback buffer in totally different formats, so whatever we did there probably wouldn't apply to WT. But basically for the onscreen bits we have 2 bytes per cell storing an index to a per-terminal pool of hyperlinks (with garbage collecting and stuff), and for the scrollback we store the URLs in runlength-encoding (i.e. once per continuous run of the same URL) in the corresponding disk file (just as we do the SGR attributes). Converting back and forth between the two, handling if the user scrolls back across the boundary etc. was a true PITA. I hope WT's architecture allows for something cleaner for you guys, e.g. 4 bytes per cell indexing to the pool which is common for the writable part as well as the scrollback (still need to figure out refcounting and/or GC, etc.).

Would be lovely to hear how much trouble it was to other terminals that implemented it :)

@egmontkob commented on GitHub (Jan 14, 2020): Experiences from implementing it in gnome-terminal: It was freaking easy to implement a demo, which was feature complete and everything, except the "tiny" issue that each character cell contained the entire target URL (in addition to the Unicode character and the SGR properties such as colors, boldness etc.), so the memory consumption went up from ~12 bytes per charcell to ~2k per charcell. Needless to say, this was a proof of concept, not meant to ever hit the codebase. And then it was freaking hard to come up with a solution that consumes reasonable amount of memory, assuming reasonable use of the feature. The details probably don't really matter because gnome-terminal stores the onscreen data and the scrollback buffer in totally different formats, so whatever we did there probably wouldn't apply to WT. But basically for the onscreen bits we have 2 bytes per cell storing an index to a per-terminal pool of hyperlinks (with garbage collecting and stuff), and for the scrollback we store the URLs in runlength-encoding (i.e. once per continuous run of the same URL) in the corresponding disk file (just as we do the SGR attributes). Converting back and forth between the two, handling if the user scrolls back across the boundary etc. was a true PITA. I hope WT's architecture allows for something cleaner for you guys, e.g. 4 bytes per cell indexing to the pool which is common for the writable part as well as the scrollback (still need to figure out refcounting and/or GC, etc.). Would be lovely to hear how much trouble it was to other terminals that implemented it :)

claunia commented

@lypanov commented on GitHub (Jan 14, 2020):

I've been wondering about these things myself @egmontkob. The implementation I'm looking at is written in rust which I can imagine simplifies the GCing requirements you had. Based on a very quick perusal it looks like Hyperlinks are not instantiated per cell but shared and referenced via pointers. Not sure it uses a pool / arena as, well, I have no Rust skills.

@lypanov commented on GitHub (Jan 14, 2020): I've been wondering about these things myself @egmontkob. The implementation I'm looking at is written in rust which I can imagine simplifies the GCing requirements you had. Based on a very quick perusal it looks like Hyperlinks are not instantiated per cell but shared and referenced via pointers. Not sure it uses a pool / arena as, well, I have no Rust skills.

claunia commented

@egmontkob commented on GitHub (Jan 14, 2020):

To complete the picture, there are two more reasons which made it quite cumbersome in vte (gnome-terminal).

One is that we're paranoid about long-time memory fragmentation, so we try to avoid malloc/free (new/delete, etc.) as much as possible. E.g. even in the hyperlink pool, allocated memory for each hyperlink is grown on demand but isn't shrunk or freed until the terminal is closed.

The other one is: due to the way our scrollback is stored on disk, we don't really know what data (incl. hyperlinks) scrolls out for good at the top of the scrollback. I mean, knowing it would require an extra read which we don't want to do for performance reasons.

If a terminal doesn't suffer from these constraints, some simple refcounted pointer (e.g. std::shared_ptr) could be a good solution. One reference by the parser as long as it's the active URL, and one per character cell that got populated with it.

@egmontkob commented on GitHub (Jan 14, 2020): To complete the picture, there are two more reasons which made it quite cumbersome in vte (gnome-terminal). One is that we're paranoid about long-time memory fragmentation, so we try to avoid malloc/free (new/delete, etc.) as much as possible. E.g. even in the hyperlink pool, allocated memory for each hyperlink is grown on demand but isn't shrunk or freed until the terminal is closed. The other one is: due to the way our scrollback is stored on disk, we don't really know what data (incl. hyperlinks) scrolls out for good at the top of the scrollback. I mean, knowing it would require an extra read which we don't want to do for performance reasons. If a terminal doesn't suffer from these constraints, some simple refcounted pointer (e.g. std::shared_ptr) could be a good solution. One reference by the parser as long as it's the active URL, and one per character cell that got populated with it.

claunia commented

cc9d2ca9e3/src/terminal/parser/OutputStateMachineEngine.cpp (L752-L757)

@zadjii-msft commented on GitHub (Jan 14, 2020):

Okay yea lets do this.

Right now, all VT output goes through OutputStateMachine, who tries to parse it into actions to perform on the console buffer. When we do find a sequence we understand, we call methods on AdaptDispatch, which handles actually doing something to conhost. Since you're really only worried about adding support in another terminal, you won't need to worry about that too much.

When the parser encounters a string it doesn't understand, it's supposed to flush the string through to the connected terminal, if there is one. So, since conhost doesn't yet understand this sequence, and it's an OSC sequence, it should be hitting this block of code:

That's the code that should tell the state machine to "pass the string through" to the terminal, whatever application that might be. (You might be hitting #2011/#2665 while debugging this, but any subsequent printable character should cause the frame to flush, so I doubt you're hitting this too badly.)

I'd start debugging there.

It's definitely possible to use the OSS-built conhost to host conptys instead of the OS one - that's what the Windows Terminal does today. However, I don't think we've really had time to polish that story for external consumers right now. We're currently just building a .lib & .dll for the conpty APIs, and making sure to link that first, and that's a little janky. @DHowett-MSFT can go into greater details on that topic.

@egmontkob thanks for the great tips. That's pretty much the way I envisioned it - using a single ID in a `TextAttributeRun` to host the URL. We don't really have any difference between scrollback and viewport in the TextBuffer, so we don't need to worry about that too much. @miniksa is also interested in this, so I'm cc'ing him.

@zadjii-msft commented on GitHub (Jan 14, 2020): Okay yea lets do this. Right now, all VT output goes through `OutputStateMachine`, who tries to parse it into actions to perform on the console buffer. When we do find a sequence we understand, we call methods on `AdaptDispatch`, which handles actually doing something to conhost. Since you're really only worried about adding support in another terminal, you won't need to worry about that too much. When the parser encounters a string it _doesn't_ understand, it's supposed to flush the string through to the connected terminal, if there is one. So, since conhost doesn't yet understand this sequence, and it's an OSC sequence, it should be hitting this block of code: https://github.com/microsoft/terminal/blob/cc9d2ca9e393b64478771b6b7c60c7302ac57d08/src/terminal/parser/OutputStateMachineEngine.cpp#L752-L757 That's the code that should tell the state machine to "pass the string through" to the terminal, whatever application that might be. (You might be hitting #2011/#2665 while debugging this, but any subsequent printable character should cause the frame to flush, so I doubt you're hitting this too badly.) I'd start debugging there. It's definitely possible to use the OSS-built conhost to host conptys instead of the OS one - that's what the Windows Terminal does today. However, I don't think we've really had time to polish that story for external consumers right now. We're currently just building a .lib & .dll for the conpty APIs, and making sure to link that first, and that's a little janky. @DHowett-MSFT can go into greater details on that topic. <hr> @egmontkob thanks for the great tips. That's pretty much the way I envisioned it - using a single ID in a `TextAttributeRun` to host the URL. We don't really have any difference between scrollback and viewport in the TextBuffer, so we don't need to worry about that too much. @miniksa is also interested in this, so I'm cc'ing him.

claunia commented

@miniksa commented on GitHub (Jan 15, 2020):

This sounds reasonable. I think we'd also want to consider some sort of heuristic-based detection of hyperlinks (in addition to being able to surface ones declared with this VT code) to meet our goal of the hyperlink feature we dreamed of for our announce video.

I also agree that the concept of having a table of links and hanging some sort of attribute off the relevant area of the buffer would be sufficient.

I don't know if we'd do this in InteractivityWin32 though because we're probably not going to add interactivity features to the conhost UI. We'd probably surface the actual click behavior through the new Terminal UI.

@miniksa commented on GitHub (Jan 15, 2020): This sounds reasonable. I think we'd also want to consider some sort of heuristic-based detection of hyperlinks (in addition to being able to surface ones declared with this VT code) to meet our goal of the hyperlink feature we dreamed of for our announce video. I also agree that the concept of having a table of links and hanging some sort of attribute off the relevant area of the buffer would be sufficient. I don't know if we'd do this in `InteractivityWin32` though because we're probably not going to add interactivity features to the conhost UI. We'd probably surface the actual click behavior through the new Terminal UI.

claunia commented

@egmontkob commented on GitHub (Jan 15, 2020):

I think we'd also want to consider some sort of heuristic-based detection of hyperlinks

Sure thing :) But that's quite a different story.

@egmontkob commented on GitHub (Jan 15, 2020): > I think we'd also want to consider some sort of heuristic-based detection of hyperlinks Sure thing :) But that's quite a different story.

claunia commented

@edburns commented on GitHub (Jan 30, 2020):

FWIW, iTerm2 does this on the mac. When you hold down command, any hyperlinks it recognizes get underlined and blue.

@edburns commented on GitHub (Jan 30, 2020): FWIW, iTerm2 does this on the mac. When you hold down command, any hyperlinks it recognizes get underlined and blue.

claunia commented

@jquast commented on GitHub (Feb 3, 2020):

This feature is demonstrated in the blessed python library documentation, as @edburns mentions,

a dotted underline is displayed and cannot be clicked,
the ⌘ key is then held to show URL at bottom of the window,
and the mouse cursor indicates it is now clickable, 👆

@jquast commented on GitHub (Feb 3, 2020): This feature is demonstrated in the *blessed* python library [documentation](https://blessed.readthedocs.io/en/latest/terminal.html#hyperlinks), as @edburns mentions, - a dotted underline is displayed and cannot be clicked, - the ⌘ key is then held to show URL at bottom of the window, - and the mouse cursor indicates it is now clickable, 👆 ![](https://dxtz6bzwq9sxx.cloudfront.net/demo_basic_hyperlink.gif)

claunia commented

@jzabroski commented on GitHub (Apr 14, 2020):

FWIW, iTerm2 does this on the mac. When you hold down command, any hyperlinks it recognizes get underlined and blue.

@edburns Not only does iTerm2 do it, but according to the Symfony blog, iTerm2 collaborated with GNOME Terminal on a hyperlinking standard. Might be worth asking them where that standard is published.

Rendering clickable hyperlinks is one of the most important missing features of Console apps and commands. Although most of the terminal emulators auto-detect URLs and allow to click on them with some key combination, it's not possible to render clickable text that points to some arbitrary URL.

However, some terminal emulators led by iTerm2 and GNOME Terminal, are working on a new specification to add support for this feature.

@jzabroski commented on GitHub (Apr 14, 2020): > FWIW, iTerm2 does this on the mac. When you hold down command, any hyperlinks it recognizes get underlined and blue. @edburns Not only does iTerm2 do it, but according to the Symfony blog, [iTerm2 collaborated with GNOME Terminal on a hyperlinking standard](https://symfony.com/blog/new-in-symfony-4-3-console-hyperlinks). Might be worth asking them where that standard is published. > Rendering clickable hyperlinks is one of the most important missing features of Console apps and commands. Although most of the terminal emulators auto-detect URLs and allow to click on them with some key combination, it's not possible to render clickable text that points to some arbitrary URL. > > *However, some terminal emulators led by iTerm2 and GNOME Terminal, are working on a new specification to add support for this feature.*

claunia commented

2026-01-30 21:48:20 +00:00

@DHowett-MSFT commented on GitHub (Apr 14, 2020):

Might be worth asking them

Fortunately, @egmontkob's already weighed in on this issue with his authority and some implementation details 😄

@DHowett-MSFT commented on GitHub (Apr 14, 2020): > Might be worth asking them Fortunately, @egmontkob's already weighed in on this issue with his authority and some implementation details :smile:

claunia commented

@jzabroski commented on GitHub (Apr 14, 2020):

😆 I just realized the first link in this post is about what I just posted about. Sorry.

@jzabroski commented on GitHub (Apr 14, 2020): 😆 I just realized the first link in this post is about what I just posted about. Sorry.

claunia commented

2026-01-30 21:48:20 +00:00

@andrzejnovak commented on GitHub (May 5, 2020):

So. this happening at some point? Probably last thing holding me back from switching.

@andrzejnovak commented on GitHub (May 5, 2020): So. this happening at some point? Probably last thing holding me back from switching.

claunia commented

2026-01-30 21:48:20 +00:00

@DHowett-MSFT commented on GitHub (May 5, 2020):

Well, I've got good news: if it weren't happening, this issue wouldn't still be open. 😄

@DHowett-MSFT commented on GitHub (May 5, 2020): Well, I've got good news: if it weren't happening, this issue wouldn't still be open. :smile:

claunia commented

@jzabroski commented on GitHub (May 5, 2020):

I have so many cool ideas for this feature once it comes.

@jzabroski commented on GitHub (May 5, 2020): I have so many cool ideas for this feature once it comes.

claunia commented

@jantari commented on GitHub (Jun 2, 2020):

I have so many cool ideas for this feature once it comes.

Are we thinking the same thing? Terminal Duck Hunt! 🦆🦆🦆

EDIT: oh I forgot we don't need hyperlinks to make mouse-clicks work, nvm

@jantari commented on GitHub (Jun 2, 2020): > I have so many cool ideas for this feature once it comes. Are we thinking the same thing? Terminal Duck Hunt! 🦆🦆🦆 EDIT: oh I forgot we don't need hyperlinks to make mouse-clicks work, nvm

claunia commented

@jzabroski commented on GitHub (Jun 2, 2020):

No. I run a project called FluentMigrator that lets you run database migrations. One user story would be able to deep link stack traces as hyperlinks to lines of code where the stack trace occurred, so that you could click the error and jump directly to your Visual Studio solution where the stack trace occurred. And this is such a primitive idea that any .NET Core CLI tool could benefit from it. I have many more user stories.

@jzabroski commented on GitHub (Jun 2, 2020): No. I run a project called FluentMigrator that lets you run database migrations. One user story would be able to deep link stack traces as hyperlinks to lines of code where the stack trace occurred, so that you could click the error and jump directly to your Visual Studio solution where the stack trace occurred. And this is such a primitive idea that any .NET Core CLI tool could benefit from it. I have many more user stories.

claunia commented