starred/terminal
1
0
Fork 0
mirror of https://github.com/microsoft/terminal.git synced 2026-02-04 05:35:20 +00:00
Code Issues 19.0k Packages Projects Releases 20 Wiki Activity

[Terminal Chat] The LLM receives confusing information about shell #22488

New Issue
Open
opened 2026-01-31 08:14:58 +00:00 by claunia · 5 comments
claunia commented 2026-01-31 08:14:58 +00:00
Owner
Copy Link

Originally created by @vbrozik on GitHub (Nov 3, 2024).

Originally assigned to: @PankajBhojwani on GitHub.

Windows Terminal version

1.23.3061.0

Windows build number

10.0.22631.0

Other Software

  • Ubuntu 22.04.5 LTS - installed as the default Ubuntu distribution in WLS2, then gradually updated
  • GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)
  • GitHub Copilot

Steps to reproduce

  1. Use the default Ubuntu distribution in WSL.
  2. Open a terminal for it.
  3. Start a Terminal Chat.
  4. After few interactions it could mention that it has information that I am using a shell named ubuntu.exe.
  5. It seems that a sequence of these two user messages lead to responses like below: Hello, Please explain the error.

Image

Expected Behavior

The chatbot should not behave is if it received wrong information about the shell being used. It should either have information that I am using Ubuntu distribution of Linux (and maybe version) or that I am using Bash shell in Ubuntu Linux.

It should probably be allowed to modify parts of the initial prompt including information about the shell and operating system. This setting should be per profile.

Actual Behavior

The Terminal Chat probably receives confusing information about the shell in the terminal window in its preconfigured initial prompt.

When I am using the default Ubuntu WSL distribution with the default bash shell the GitHub Copilot responds as if it received information that I am using shell named ubuntu.exe:

Image

With Ubuntu 24.04 GitHub Copilot is confused even more:

Image

Originally created by @vbrozik on GitHub (Nov 3, 2024). Originally assigned to: @PankajBhojwani on GitHub. ### Windows Terminal version 1.23.3061.0 ### Windows build number 10.0.22631.0 ### Other Software - Ubuntu 22.04.5 LTS - installed as the default Ubuntu distribution in WLS2, then gradually updated - GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu) - GitHub Copilot ### Steps to reproduce 1. Use the default Ubuntu distribution in WSL. 2. Open a terminal for it. 3. Start a Terminal Chat. 4. After few interactions it could mention that it has information that I am using a shell named `ubuntu.exe`. 5. It seems that a sequence of these two user messages lead to responses like below: `Hello`, `Please explain the error.` ![Image](https://github.com/user-attachments/assets/efd246b4-67bf-4f0e-943b-979006b409e9) ### Expected Behavior The chatbot should not behave is if it received wrong information about the shell being used. It should either have information that I am using Ubuntu distribution of Linux (and maybe version) or that I am using Bash shell in Ubuntu Linux. It should probably be allowed to modify parts of the initial prompt including information about the shell and operating system. This setting should be per profile. ### Actual Behavior The Terminal Chat probably receives confusing information about the shell in the terminal window in its preconfigured initial prompt. When I am using the default Ubuntu WSL distribution with the default `bash` shell the GitHub Copilot responds as if it received information that I am using shell named `ubuntu.exe`: ![Image](https://github.com/user-attachments/assets/efd246b4-67bf-4f0e-943b-979006b409e9) With Ubuntu 24.04 GitHub Copilot is confused even more: ![Image](https://github.com/user-attachments/assets/eb803aec-d4ac-4a24-9001-f1f0804d9e4f)
claunia added the Issue-BugNeeds-Tag-FixProduct-TerminalArea-Chat labels 2026-01-31 08:14:58 +00:00
claunia commented 2026-01-31 08:14:59 +00:00
Author
Owner
Copy Link

@zcobol commented on GitHub (Nov 4, 2024):

@vbrozik LLM hallucination 😀

@zcobol commented on GitHub (Nov 4, 2024): @vbrozik LLM hallucination 😀
claunia commented 2026-01-31 08:14:59 +00:00
Author
Owner
Copy Link

@vbrozik commented on GitHub (Nov 4, 2024):

@zcobol we can see the hallucination in the second case: ubuntu2404.exe vs "Ubuntu 20.04"

...but I think we can be almost sure that the LLM receives information about the command used to start the WSL2 session like ubuntu.exe or ubuntu2404.exe and it is being wrongly presented to it as a shell running inside the terminal.

@vbrozik commented on GitHub (Nov 4, 2024): @zcobol we can see the hallucination in the second case: `ubuntu2404.exe` vs "Ubuntu 20.04" ...but I think we can be almost sure that the LLM receives information about the command used to start the WSL2 session like `ubuntu.exe` or `ubuntu2404.exe` and it is being wrongly presented to it as a shell running inside the terminal.
claunia commented 2026-01-31 08:14:59 +00:00
Author
Owner
Copy Link

@DHowett commented on GitHub (Nov 4, 2024):

This is one of the annoying limitations we've run into - Terminal actually cannot know what is running "behind" the single executable that it started (in this case, ubuntu.exe or ubuntu2404.exe). There's no interface by which this data is communicated, so it makes a "best guess" and inserts the name of the root process of the tab.

It's not ideal.

@DHowett commented on GitHub (Nov 4, 2024): This is one of the annoying limitations we've run into - Terminal actually cannot know what is running "behind" the single executable that it started (in this case, `ubuntu.exe` or `ubuntu2404.exe`). There's no interface by which this data is communicated, so it makes a "best guess" and inserts the name of the root process of the tab. It's not ideal.
claunia commented 2026-01-31 08:14:59 +00:00
Author
Owner
Copy Link

@vbrozik commented on GitHub (Nov 6, 2024):

Yes, I expected that Terminal lacking the information was the cause. I think that presenting ubuntu.exe to LLMs as a shell could cause bad experience for the users. I really do not know what is optimal to present to LLMs but I would certainly consider one or more of following solutions:

  • Present the command like ubuntu.exe as a command used to start the environment inside the Terminal, not as a shell.
  • Alternatively do not present the command at all to the LLM.
  • Create a configuration of initial prompt for the LLMs a part of the profile configuration. One of the parameters would be the default shell used with the profile. It will be the place to specify bash, POSIX shell, PowerShell etc.
  • When a profile is created the initial value of the default shell could be automatically inferred. Examples: ubuntu.exe -> bash or bash in Ubuntu Linux; powershell.exe -> PowerShell
  • Later there could be some kind of integration of the Terminal with shells. It would be really good to cooperate with the VS Code terminal team. For example the shell could communicate about itself to the Terminal using control sequences. This way the Terminal Chat could work correctly even if you start a new shell on a different machine for example through SSH.
@vbrozik commented on GitHub (Nov 6, 2024): Yes, I expected that Terminal lacking the information was the cause. I think that presenting `ubuntu.exe` to LLMs as a shell could cause bad experience for the users. I really do not know what is optimal to present to LLMs but I would certainly consider one or more of following solutions: - Present the command like `ubuntu.exe` as a command used to start the environment inside the Terminal, not as a shell. - Alternatively do not present the command at all to the LLM. - Create a configuration of initial prompt for the LLMs a part of the profile configuration. One of the parameters would be the default shell used with the profile. It will be the place to specify `bash`, `POSIX shell`, `PowerShell` etc. - When a profile is created the initial value of the default shell could be automatically inferred. Examples: `ubuntu.exe` -> `bash` or `bash in Ubuntu Linux`; `powershell.exe` -> `PowerShell` - Later there could be some kind of integration of the Terminal with shells. It would be really good to cooperate with the VS Code terminal team. For example the shell could communicate about itself to the Terminal using control sequences. This way the Terminal Chat could work correctly even if you start a new shell on a different machine for example through SSH.
claunia commented 2026-01-31 08:15:00 +00:00
Author
Owner
Copy Link

@PankajBhojwani commented on GitHub (Nov 18, 2024):

Those are some solid suggestions, thank you! We actually do have a work item for allowing the user to modify/add to the system prompt that we send to the LLM, once that is implemented we can remove the 'default' addition that we do right now entirely and just use the user-provided one, which should fix this :)

@PankajBhojwani commented on GitHub (Nov 18, 2024): Those are some solid suggestions, thank you! We actually do have a work item for allowing the user to modify/add to the system prompt that we send to the LLM, once that is implemented we can remove the 'default' addition that we do right now entirely and just use the user-provided one, which should fix this :)
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
⛺ Reserved
A11yCO
A11yMAS
A11ySev1
A11ySev2
A11ySev3
A11yTTValidated
A11yUsable
A11yVoiceAccess
A11yWCAG
Area-Accessibility
Area-AtlasEngine
Area-AzureShell
Area-Build
Area-Build
Area-Chat
Area-CmdPal
Area-CodeHealth
Area-Commandline
Area-CookedRead
Area-DefApp
Area-Extensibility
Area-Fonts
Area-GroupPolicy
Area-i18n
Area-Input
Area-Interaction
Area-Interop
Area-Localization
Area-Output
Area-Performance
Area-Portable
Area-Quality
Area-Remoting
Area-Rendering
Area-Schema
Area-Server
Area-Settings
Area-SettingsUI
Area-ShellExtension
Area-ShellExtension
Area-ShellExtension
Area-Suggestions
Area-Suggestions
Area-TerminalConnection
Area-TerminalControl
Area-Theming
Area-UserInterface
Area-VT
Area-Windowing
Area-WPFControl
AutoMerge
Blocking-Ingestion
Culprit-Centennial
Culprit-WinUI
Disability-All
Disability-Blind
Disability-LowVision
Disability-Mobility
External-Blocked-WinUI3
Fixed
Gathering-Data
good first issue
HCL-E+D
HCL-WindowsTerminal
Help Wanted
Impact-Compatibility
Impact-Compliance
Impact-Correctness
Impact-Visual
In-PR
InclusionBacklog
InclusionBacklog-Windows TerminalWin32
InclusionCommitted-202206
Issue-Bug
Issue-Docs
Issue-Feature
Issue-Feature
Issue-Question
Issue-Samples
Issue-Scenario
Issue-Task
Needs-Attention
Needs-Author-Feedback
Needs-Bisect
Needs-Discussion
Needs-Repro
Needs-Tag-Fix
Needs-Tag-Fix
Needs-Triage
No-Recent-Activity
Priority-0
Priority-1
Priority-2
Priority-3
Product-Cmd.exe
Product-Colortool
Product-Colortool
Product-Colortool
Product-Conhost
Product-Conpty
Product-Meta
Product-Powershell
Product-Terminal
Product-WSL
pull-request
Mirrored from GitHub Pull Request
 Resolution-Answered
Resolution-By-Design
Resolution-Duplicate
Resolution-External
Resolution-Fix-Available
Resolution-Fix-Committed
Resolution-No-Repro
Resolution-Won't-Fix
Severity-Blocking
Severity-Crash
Severity-DataLoss
spam
this-will-be-a-breaking-change
Tracking-External
WindowsTerminal_Win32
Work-Item
zAskModeBug
zInbox-Bug
No Label Area-Chat Issue-Bug Needs-Tag-Fix Product-Terminal
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
claunia
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/terminal#22488
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?