Files
terminal-microsoft/src/terminal/parser/ut_parser/StateMachineTest.cpp

247 lines
8.5 KiB
C++
Raw Normal View History

// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license.
#include "precomp.h"
#include "WexTestClass.h"
#include "../../inc/consoletaeftemplates.hpp"
#include "stateMachine.hpp"
using namespace WEX::Common;
using namespace WEX::Logging;
using namespace WEX::TestExecution;
namespace Microsoft
{
namespace Console
{
namespace VirtualTerminal
{
class StateMachineTest;
class TestStateMachineEngine;
};
};
};
using namespace Microsoft::Console::VirtualTerminal;
class Microsoft::Console::VirtualTerminal::TestStateMachineEngine : public IStateMachineEngine
{
public:
void ResetTestState()
{
printed.clear();
passedThrough.clear();
csiParams.reset();
}
bool ActionExecute(const wchar_t /* wch */) override { return true; };
bool ActionExecuteFromEscape(const wchar_t /* wch */) override { return true; };
bool ActionPrint(const wchar_t /* wch */) override { return true; };
bool ActionPrintString(const std::wstring_view string) override
{
printed += string;
return true;
};
bool ActionPassThroughString(const std::wstring_view string) override
{
passedThrough += string;
return true;
};
Refactor VT control sequence identification (#7304) This PR changes the way VT control sequences are identified and dispatched, to be more efficient and easier to extend. Instead of parsing the intermediate characters into a vector, and then having to identify a sequence using both that vector and the final char, we now use just a single `uint64_t` value as the identifier. The way the identifier is constructed is by taking the private parameter prefix, each of the intermediate characters, and then the final character, and shifting them into a 64-bit integer one byte at a time, in reverse order. For example, the `DECTLTC` control has a private parameter prefix of `?`, one intermediate of `'`, and a final character of `s`. The ASCII values of those characters are `0x3F`, `0x27`, and `0x73` respectively, and reversing them gets you 0x73273F, so that would then be the identifier for the control. The reason for storing them in reverse order, is because sometimes we need to look at the first intermediate to determine the operation, and treat the rest of the sequence as a kind of sub-identifier (the character set designation sequences are one example of this). When in reverse order, this can easily be achieved by masking off the low byte to get the first intermediate, and then shifting the value right by 8 bits to get a new identifier with the rest of the sequence. With 64 bits we have enough space for a private prefix, six intermediates, and the final char, which is way more than we should ever need (the _DEC STD 070_ specification recommends supporting at least three intermediates, but in practice we're unlikely to see more than two). With this new way of identifying controls, it should now be possible for every action code to be unique (for the most part). So I've also used this PR to clean up the action codes a bit, splitting the codes for the escape sequences from the control sequences, and sorting them into alphabetical order (which also does a reasonable job of clustering associated controls). ## Validation Steps Performed I think the existing unit tests should be good enough to confirm that all sequences are still being dispatched correctly. However, I've also manually tested a number of sequences to make sure they were still working as expected, in particular those that used intermediates, since they were the most affected by the dispatch code refactoring. Since these changes also affected the input state machine, I've done some manual testing of the conpty keyboard handling (both with and without the new Win32 input mode enabled) to make sure the keyboard VT sequences were processed correctly. I've also manually tested the various VT mouse modes in Vttest to confirm that they were still working correctly too. Closes #7276
2020-08-18 19:57:52 +01:00
bool ActionEscDispatch(const VTID /* id */) override { return true; };
Refactor VT control sequence identification (#7304) This PR changes the way VT control sequences are identified and dispatched, to be more efficient and easier to extend. Instead of parsing the intermediate characters into a vector, and then having to identify a sequence using both that vector and the final char, we now use just a single `uint64_t` value as the identifier. The way the identifier is constructed is by taking the private parameter prefix, each of the intermediate characters, and then the final character, and shifting them into a 64-bit integer one byte at a time, in reverse order. For example, the `DECTLTC` control has a private parameter prefix of `?`, one intermediate of `'`, and a final character of `s`. The ASCII values of those characters are `0x3F`, `0x27`, and `0x73` respectively, and reversing them gets you 0x73273F, so that would then be the identifier for the control. The reason for storing them in reverse order, is because sometimes we need to look at the first intermediate to determine the operation, and treat the rest of the sequence as a kind of sub-identifier (the character set designation sequences are one example of this). When in reverse order, this can easily be achieved by masking off the low byte to get the first intermediate, and then shifting the value right by 8 bits to get a new identifier with the rest of the sequence. With 64 bits we have enough space for a private prefix, six intermediates, and the final char, which is way more than we should ever need (the _DEC STD 070_ specification recommends supporting at least three intermediates, but in practice we're unlikely to see more than two). With this new way of identifying controls, it should now be possible for every action code to be unique (for the most part). So I've also used this PR to clean up the action codes a bit, splitting the codes for the escape sequences from the control sequences, and sorting them into alphabetical order (which also does a reasonable job of clustering associated controls). ## Validation Steps Performed I think the existing unit tests should be good enough to confirm that all sequences are still being dispatched correctly. However, I've also manually tested a number of sequences to make sure they were still working as expected, in particular those that used intermediates, since they were the most affected by the dispatch code refactoring. Since these changes also affected the input state machine, I've done some manual testing of the conpty keyboard handling (both with and without the new Win32 input mode enabled) to make sure the keyboard VT sequences were processed correctly. I've also manually tested the various VT mouse modes in Vttest to confirm that they were still working correctly too. Closes #7276
2020-08-18 19:57:52 +01:00
bool ActionVt52EscDispatch(const VTID /*id*/, const gsl::span<const size_t> /*parameters*/) override { return true; };
Add support for VT52 emulation (#4789) ## Summary of the Pull Request This PR adds support for the core VT52 commands, and implements the `DECANM` private mode sequence, which switches the terminal between ANSI mode and VT52-compatible mode. ## References PR #2017 defined the initial specification for VT52 support. PR #4044 removed the original VT52 cursor ops that conflicted with VT100 sequences. ## PR Checklist * [x] Closes #976 * [x] CLA signed. If not, go over [here](https://cla.opensource.microsoft.com/microsoft/Terminal) and sign the CLA * [x] Tests added/passed * [ ] Requires documentation to be updated * [x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan. Issue number where discussion took place: #2017 ## Detailed Description of the Pull Request / Additional comments Most of the work involves updates to the parsing state machine, which behaves differently in VT52 mode. `CSI`, `OSC`, and `SS3` sequences are not applicable, and there is one special-case escape sequence (_Direct Cursor Address_), which requires an additional state to handle parameters that come _after_ the final character. Once the parsing is handled though, it's mostly just a matter of dispatching the commands to existing methods in the `ITermDispatch` interface. Only one new method was required in the interface to handle the _Identify_ command. The only real new functionality is in the `TerminalInput` class, which needs to generate different escape sequences for certain keys in VT52 mode. This does not yet support _all_ of the VT52 key sequences, because the VT100 support is itself not yet complete. But the basics are in place, and I think the rest is best left for a follow-up issue, and potentially a refactor of the `TerminalInput` class. I should point out that the original spec called for a new _Graphic Mode_ character set, but I've since discovered that the VT terminals that _emulate_ VT52 just use the existing VT100 _Special Graphics_ set, so that is really what we should be doing too. We can always consider adding the VT52 graphic set as a option later, if there is demand for strict VT52 compatibility. ## Validation Steps Performed I've added state machine and adapter tests to confirm that the `DECANM` mode changing sequences are correctly dispatched and forwarded to the `ConGetSet` handler. I've also added state machine tests that confirm the VT52 escape sequences are dispatched correctly when the ANSI mode is reset. For fuzzing support, I've extended the VT command fuzzer to generate the different kinds of VT52 sequences, as well as mode change sequences to switch between the ANSI and VT52 modes. In terms of manual testing, I've confirmed that the _Test of VT52 mode_ in Vttest now works as expected.
2020-06-01 22:20:40 +01:00
bool ActionClear() override { return true; };
bool ActionIgnore() override { return true; };
bool ActionOscDispatch(const wchar_t /* wch */,
const size_t /* parameter */,
const std::wstring_view /* string */) override
{
if (pfnFlushToTerminal)
{
pfnFlushToTerminal();
return true;
}
return true;
};
bool ActionSs3Dispatch(const wchar_t /* wch */,
const gsl::span<const size_t> /* parameters */) override { return true; };
Improve support for VT character sets (#4496) This PR improves our VT character set support, enabling the [`SCS`] escape sequences to designate into all four G-sets with both 94- and 96-character sets, and supports invoking those G-sets into both the GL and GR areas of the code table, with [locking shifts] and [single shifts]. It also adds [`DOCS`] sequences to switch between UTF-8 and the ISO-2022 coding system (which is what the VT character sets require), and adds support for a lot more characters sets, up to around the level of a VT510. [`SCS`]: https://vt100.net/docs/vt510-rm/SCS.html [locking shifts]: https://vt100.net/docs/vt510-rm/LS.html [single shifts]: https://vt100.net/docs/vt510-rm/SS.html [`DOCS`]: https://en.wikipedia.org/wiki/ISO/IEC_2022#Interaction_with_other_coding_systems ## Detailed Description of the Pull Request / Additional comments To make it easier for us to declare a bunch of character sets, I've made a little `constexpr` class that can build up a mapping table from a base character set (ASCII or Latin1), along with a collection of mappings for the characters the deviate from the base set. Many of the character sets are simple variations of ASCII, so they're easy to define this way. This class then casts directly to a `wstring_view` which is how the translation tables are represented in most of the code. We have an array of four of these tables representing the four G-sets, two instances for the active left and right tables, and one instance for the single shift table. Initially we had just one `DesignateCharset` method, which could select the active character set. We now have two designate methods (for 94- and 96- character sets), and each takes a G-set number specifying the target of the designation, and a pair of characters identifying the character set that will be designated (at the higher VT levels, character sets are often identified by more than one character). There are then two new `LockingShift` methods to invoke these G-sets into either the GL or GR area of the code table, and a `SingleShift` method which invokes a G-set temporarily (for just the next character that is output). I should mention here that I had to make some changes to the state machine to make these single shift sequences work. The problem is that the input state machine treats `SS3` as the start of a control sequence, while the output state machine needs it to be dispatched immediately (it's literally the _Single Shift 3_ escape sequence). To make that work, I've added a `ParseControlSequenceAfterSs3` callback in the `IStateMachineEngine` interface to decide which behavior is appropriate. When it comes to mapping a character, it's simply an array reference into the appropriate `wstring_view` table. If the single shift table is set, that takes preference. Otherwise the GL table is used for characters in the range 0x20 to 0x7F, and the GR table for characters 0xA0 to 0xFF (technically some character sets will only map up to 0x7E and 0xFE, but that's easily controlled by the length of the `wstring_view`). The `DEL` character is a bit of a special case. By default it's meant to be ignored like the `NUL` character (it's essentially a time-fill character). However, it's possible that it could be remapped to a printable character in a 96-character set, so we need to check for that after the translation. This is handled in the `AdaptDispatch::Print` method, so it doesn't interfere with the primary `PrintString` code path. The biggest problem with this whole process, though, is that the GR mappings only really make sense if you have access to the raw output, but by the time the output gets to us, it would already have been translated to Unicode by the active code page. And in the case of UTF-8, the characters we eventually receive may originally have been composed from two or more code points. The way I've dealt with this was to disable the GR translations by default, and then added support for a pair of ISO-2022 `DOCS` sequences, which can switch the code page between UTF-8 and ISO-8859-1. When the code page is ISO-8859-1, we're essentially receiving the raw output bytes, so it's safe to enable the GR translations. This is not strictly correct ISO-2022 behavior, and there are edge cases where it's not going to work, but it's the best solution I could come up with. ## Validation Steps Performed As a result of the `SS3` changes in the state machine engine, I've had to move the existing `SS3` tests from the `OutputEngineTest` to the `InputEngineTest`, otherwise they would now fail (technically they should never have been output tests). I've added no additional unit tests, but I have done a lot of manual testing, and made sure we passed all the character set tests in Vttest (at least for the character sets we currently support). Note that this required a slightly hacked version of the app, since by default it doesn't expose a lot of the test to low-level terminals, and we currently identify as a VT100. Closes #3377 Closes #3487
2020-06-04 20:40:15 +01:00
bool ParseControlSequenceAfterSs3() const override { return false; }
bool FlushAtEndOfString() const override { return false; };
bool DispatchControlCharsFromEscape() const override { return false; };
bool DispatchIntermediatesFromEscape() const override { return false; };
// ActionCsiDispatch is the only method that's actually implemented.
Refactor VT control sequence identification (#7304) This PR changes the way VT control sequences are identified and dispatched, to be more efficient and easier to extend. Instead of parsing the intermediate characters into a vector, and then having to identify a sequence using both that vector and the final char, we now use just a single `uint64_t` value as the identifier. The way the identifier is constructed is by taking the private parameter prefix, each of the intermediate characters, and then the final character, and shifting them into a 64-bit integer one byte at a time, in reverse order. For example, the `DECTLTC` control has a private parameter prefix of `?`, one intermediate of `'`, and a final character of `s`. The ASCII values of those characters are `0x3F`, `0x27`, and `0x73` respectively, and reversing them gets you 0x73273F, so that would then be the identifier for the control. The reason for storing them in reverse order, is because sometimes we need to look at the first intermediate to determine the operation, and treat the rest of the sequence as a kind of sub-identifier (the character set designation sequences are one example of this). When in reverse order, this can easily be achieved by masking off the low byte to get the first intermediate, and then shifting the value right by 8 bits to get a new identifier with the rest of the sequence. With 64 bits we have enough space for a private prefix, six intermediates, and the final char, which is way more than we should ever need (the _DEC STD 070_ specification recommends supporting at least three intermediates, but in practice we're unlikely to see more than two). With this new way of identifying controls, it should now be possible for every action code to be unique (for the most part). So I've also used this PR to clean up the action codes a bit, splitting the codes for the escape sequences from the control sequences, and sorting them into alphabetical order (which also does a reasonable job of clustering associated controls). ## Validation Steps Performed I think the existing unit tests should be good enough to confirm that all sequences are still being dispatched correctly. However, I've also manually tested a number of sequences to make sure they were still working as expected, in particular those that used intermediates, since they were the most affected by the dispatch code refactoring. Since these changes also affected the input state machine, I've done some manual testing of the conpty keyboard handling (both with and without the new Win32 input mode enabled) to make sure the keyboard VT sequences were processed correctly. I've also manually tested the various VT mouse modes in Vttest to confirm that they were still working correctly too. Closes #7276
2020-08-18 19:57:52 +01:00
bool ActionCsiDispatch(const VTID /*id*/, const gsl::span<const size_t> parameters) override
{
// If flush to terminal is registered for a test, then use it.
if (pfnFlushToTerminal)
{
pfnFlushToTerminal();
return true;
}
else
{
csiParams.emplace(parameters.begin(), parameters.end());
return true;
}
}
// This will only be populated if ActionCsiDispatch is called.
std::optional<std::vector<size_t>> csiParams;
// Flush function for pass-through test.
std::function<bool()> pfnFlushToTerminal;
// Passed through string.
std::wstring passedThrough;
// Printed string.
std::wstring printed;
};
class Microsoft::Console::VirtualTerminal::StateMachineTest
{
TEST_CLASS(StateMachineTest);
TEST_CLASS_SETUP(ClassSetup)
{
return true;
}
TEST_CLASS_CLEANUP(ClassCleanup)
{
return true;
}
TEST_METHOD(TwoStateMachinesDoNotInterfereWithEachother);
TEST_METHOD(PassThroughUnhandled);
TEST_METHOD(RunStorageBeforeEscape);
TEST_METHOD(BulkTextPrint);
TEST_METHOD(PassThroughUnhandledSplitAcrossWrites);
};
void StateMachineTest::TwoStateMachinesDoNotInterfereWithEachother()
{
auto firstEnginePtr{ std::make_unique<TestStateMachineEngine>() };
// this dance is required because StateMachine presumes to take ownership of its engine.
const auto& firstEngine{ *firstEnginePtr.get() };
StateMachine firstStateMachine{ std::move(firstEnginePtr) };
auto secondEnginePtr{ std::make_unique<TestStateMachineEngine>() };
const auto& secondEngine{ *secondEnginePtr.get() };
StateMachine secondStateMachine{ std::move(secondEnginePtr) };
firstStateMachine.ProcessString(L"\x1b[12"); // partial sequence
secondStateMachine.ProcessString(L"\x1b[3C"); // full sequence on second parser
firstStateMachine.ProcessString(L";34m"); // completion to previous partial sequence on first parser
std::vector<size_t> expectedFirstCsi{ 12u, 34u };
std::vector<size_t> expectedSecondCsi{ 3u };
VERIFY_ARE_EQUAL(expectedFirstCsi, firstEngine.csiParams);
VERIFY_ARE_EQUAL(expectedSecondCsi, secondEngine.csiParams);
}
void StateMachineTest::PassThroughUnhandled()
{
auto enginePtr{ std::make_unique<TestStateMachineEngine>() };
// this dance is required because StateMachine presumes to take ownership of its engine.
auto& engine{ *enginePtr.get() };
StateMachine machine{ std::move(enginePtr) };
// Hook up the passthrough function.
engine.pfnFlushToTerminal = std::bind(&StateMachine::FlushToTerminal, &machine);
machine.ProcessString(L"\x1b[?999h 12345 Hello World");
VERIFY_ARE_EQUAL(String(L"\x1b[?999h"), String(engine.passedThrough.c_str()));
VERIFY_ARE_EQUAL(String(L" 12345 Hello World"), String(engine.printed.c_str()));
}
void StateMachineTest::RunStorageBeforeEscape()
{
auto enginePtr{ std::make_unique<TestStateMachineEngine>() };
// this dance is required because StateMachine presumes to take ownership of its engine.
auto& engine{ *enginePtr.get() };
StateMachine machine{ std::move(enginePtr) };
// Hook up the passthrough function.
engine.pfnFlushToTerminal = std::bind(&StateMachine::FlushToTerminal, &machine);
// Print a bunch of regular text to build up the run buffer before transitioning state.
machine.ProcessString(L"12345 Hello World\x1b[?999h");
// Then ensure the entire buffered run was printed all at once back to us.
VERIFY_ARE_EQUAL(String(L"12345 Hello World"), String(engine.printed.c_str()));
VERIFY_ARE_EQUAL(String(L"\x1b[?999h"), String(engine.passedThrough.c_str()));
}
void StateMachineTest::BulkTextPrint()
{
auto enginePtr{ std::make_unique<TestStateMachineEngine>() };
// this dance is required because StateMachine presumes to take ownership of its engine.
auto& engine{ *enginePtr.get() };
StateMachine machine{ std::move(enginePtr) };
// Print a bunch of regular text to build up the run buffer before transitioning state.
machine.ProcessString(L"12345 Hello World");
// Then ensure the entire buffered run was printed all at once back to us.
VERIFY_ARE_EQUAL(String(L"12345 Hello World"), String(engine.printed.c_str()));
}
void StateMachineTest::PassThroughUnhandledSplitAcrossWrites()
{
auto enginePtr{ std::make_unique<TestStateMachineEngine>() };
// this dance is required because StateMachine presumes to take ownership of its engine.
auto& engine{ *enginePtr.get() };
StateMachine machine{ std::move(enginePtr) };
// Hook up the passthrough function.
engine.pfnFlushToTerminal = std::bind(&StateMachine::FlushToTerminal, &machine);
// Broken in two pieces (test case from GH#3081)
machine.ProcessString(L"\x1b[?12");
VERIFY_ARE_EQUAL(L"", engine.passedThrough); // nothing out yet
VERIFY_ARE_EQUAL(L"", engine.printed);
machine.ProcessString(L"34h");
VERIFY_ARE_EQUAL(L"\x1b[?1234h", engine.passedThrough); // whole sequence out, no other output
VERIFY_ARE_EQUAL(L"", engine.printed);
engine.ResetTestState();
// Three pieces
machine.ProcessString(L"\x1b[?2");
VERIFY_ARE_EQUAL(L"", engine.passedThrough); // nothing out yet
VERIFY_ARE_EQUAL(L"", engine.printed);
machine.ProcessString(L"34");
VERIFY_ARE_EQUAL(L"", engine.passedThrough); // nothing out yet
VERIFY_ARE_EQUAL(L"", engine.printed);
machine.ProcessString(L"5h");
VERIFY_ARE_EQUAL(L"\x1b[?2345h", engine.passedThrough); // whole sequence out, no other output
VERIFY_ARE_EQUAL(L"", engine.printed);
engine.ResetTestState();
// Split during OSC terminator (test case from GH#3080)
machine.ProcessString(L"\x1b]99;foo\x1b");
VERIFY_ARE_EQUAL(L"", engine.passedThrough); // nothing out yet
VERIFY_ARE_EQUAL(L"", engine.printed);
machine.ProcessString(L"\\");
VERIFY_ARE_EQUAL(L"\x1b]99;foo\x1b\\", engine.passedThrough);
VERIFY_ARE_EQUAL(L"", engine.printed);
}