From d42b2971281b2626eb147449fb355ddb24b80274 Mon Sep 17 00:00:00 2001 From: Matt Jarvis Date: Sun, 13 Mar 2022 12:54:41 -0400 Subject: [PATCH 1/2] initial documentation on parser --- doc/parsing-ast.md | 55 +++++++++ doc/parsing-extensions.md | 158 ++++++++++++++++++++++++++ doc/parsing-overview.md | 227 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 440 insertions(+) create mode 100644 doc/parsing-ast.md create mode 100644 doc/parsing-extensions.md create mode 100644 doc/parsing-overview.md diff --git a/doc/parsing-ast.md b/doc/parsing-ast.md new file mode 100644 index 00000000..3945ff33 --- /dev/null +++ b/doc/parsing-ast.md @@ -0,0 +1,55 @@ +# The Abstract Syntax Tree + +If successful, the `MarkdownParser.Parse(...)` method returns the abstract syntax tree (AST) of the source text. + +This will be an object of the `MarkdownDocument` type, which is in turn derived from a more general block container and is part of a larger taxonomy of classes which represent different semantic constructs of a markdown syntax tree. + +This document will discuss the different types of elements within the Markdig representation of the AST. + +## Structure of the AST + +Within Markdig, there are two general types of node in the markdown syntax tree: `Block`, and `Inline`. Block nodes may contain inline nodes, but the reverse is not true. Blocks may contain other blocks, and inlines may contain other inlines. + +The root of the AST is the `MarkdownDocument` which is itself derived from a container block but also contains information on the line count and starting positions within the document. Nodes in the AST have links both to parent and children, allowing the edges in the tree to be traversed efficiently in either direction. + +Different semantic constructs are represented by types derived from the `Block` and `Inline` types, which are both `abstract` themselves. These elements are produced by `BlockParser` and `InlineParser` derived types, respectively, and so new constructs can be added with the implementation of a new block or inline parser and a new block or inline type, as well as an extension to register it in the pipeline. For more information on extending Markdig this way refer to the [Extensions/Parsers](parsing-extensions.md) document. + +The AST is assembled by the static method `MarkdownParser.Parse(...)` using the collections of block and inline parsers contained in the `MarkdownPipeline`. For more detailed information refer to the [Markdig Parsing Overview](parsing-overview.md) document. + +## Block Elements + +Block elements all derive from `Block` and may be one of two types: + +1. `ContainerBlock`, which is a block which holds other blocks (`MarkdownDocument` is itself derived from this) +2. `LeafBlock`, which is a block that has no child blocks, but may contain inlines + +Block elements in markdown refer to things like paragraphs, headings, lists, code, etc. Most blocks may contain inlines, with the exception of things like code blocks. + +### Properties of Blocks + +All blocks have a reference to a parent (`Parent`) of type `ContainerBlock?`, which allows for efficient traversal up the abstract syntax tree. The parent will be `null` in the case of the root node (the `MarkdownDocument`). + +Additionally, all blocks have a reference to a parser (`Parser`) of type `BlockParser?` which refers to the instance of the parser which created this block. + +Blocks have an `IsOpen` boolean flag which is set true while they're being parsed **(is this true?)** and then closed when parsing is complete. + +Blocks are either breakable or not, specified by the `IsBreakable` flag. **(What is the significance of breakable blocks? Can't be split? Is anything besides `FencedCodeBlock` not breakable?)** + +**(Is there anything special that should be documented for the ParagraphBlock or any other specific type of blocks?)** + +## Inline Elements + +Inline elements may be one of two types: + +1. `ContainerInline`, which contains other inlines, and whose parent may be a `LeafBlock` or another `ContainerInline` (**is this true?**) +2. `Inline`, whose parent is always a `ContainerInline` (**is this true?**) + +Inlines in markdown refer to things like embellishments (italics, bold, underline, etc), links, urls, inline code, images, etc. + +**(Is there anything special worth documenting about inlines or types of inlines?)** + +## The SourceSpan Struct + +If the pipeline was configured with `.UsePreciseSourceLocation()`, all elements in the abstract syntax tree will contain a reference to the location in the original source where they occurred. This is done with the `SourceSpan` class, a custom Markdig `struct` which provides a start and end location. + +All objects derived from `MarkdownObject` contain the `Span` property, which is of type `SourceSpan`. diff --git a/doc/parsing-extensions.md b/doc/parsing-extensions.md new file mode 100644 index 00000000..35c71764 --- /dev/null +++ b/doc/parsing-extensions.md @@ -0,0 +1,158 @@ +# Extensions and Parsers + +Markdig was [implemented in such a way](http://xoofx.com/blog/2016/06/13/implementing-a-markdown-processor-for-dotnet/) as to be extremely pluggable, with even basic behaviors being mutable and extendable. + +The basic mechanism for extension of Markdig is the `IMarkdownExtension` interface, which allows any implementing class to be registered with the pipeline builder and thus to directly modify the collections of `BlockParser` and `InlineParser` objects which end up in the pipeline. + +This document discusses the `IMarkdownExtension` interface, the `BlockParser` abstract base class, and the `InlineParser` abstract base class, which together are the foundation of extending Markdig's parsing machinery. + +## Creating Extensions + +Extensions can vary from very simple to very complicated. + +A simple extension, for example, might simply find a parser already in the pipeline and modify a setting on it. An example of this is the `SoftlineBreakAsHardlineExtension`, which locates the `LineBreakInlineParser` and modifies a single boolean flag on it. + +A complex extension, on the other hand, might add an entire taxonomy of new `Block` and `Inline` types, as well as several related parsers and renderers, and require being added to the the pipeline in a specific order in relation to other extensions which are already configured. The `FootnoteExtension` and `PipeTableExtension` are examples of more complex extensions. + +For extensions that don't require order considerations, the implementation of the extension itself is adequate, and the extension can be added to the pipeline with the generic `Use()` method on the pipeline builder. For extensions which do require order considerations, it is best to create an extension method on the `MarkdownPipelineBuilder` to perform the registration. See the following two sections for further information. + +### Implementation of an Extension + +The [IMarkdownExtension.cs](https://github.com/xoofx/markdig/blob/master/src/Markdig/IMarkdownExtension.cs) interface specifies two methods which must be implemented. + +The first, which takes only the pipeline builder as an argument, is called when the `Build()` method on the pipeline builder is invoked, and should set up any modifications to the parsers or parser collections. These parsers will then be used by the main parsing algorithm to process the source text. + +```csharp +void Setup(MarkdownPipelineBuilder pipeline); +``` + +The second, which takes the pipeline itself and a renderer, is used to set up a rendering component in order to convert any special `MarkdownObject` types associated with the extension into an output. This is not relevant for parsing, but is necessary for rendering. + +```csharp +void Setup(MarkdownPipeline pipeline, IMarkdownRenderer renderer); +``` + +The extension can then be registered to the pipeline builder using the `Use()` method. A skeleton example is given below: + +```csharp +public class MySpecialBlockParser : BlockParser +{ + // ... +} + +public class MyExtension : IMarkdownExtension +{ + void Setup(MarkdownPipelineBuilder pipeline) + { + pipeline.BlockParsers.AddIfNotAlready(); + } + + void Setup(MarkdownPipeline pipeline, IMarkdownRenderer renderer) { } +} +``` + +```csharp +var builder = new MarkdownPipelineBuilder() + .Use(); +``` + +### Pipeline Builder Extension Methods + +For extensions which require specific ordering and/or need to perform multiple operations to register with the builder, it's recommended to create an extension method. + +```csharp +public static class MyExtensionMethods +{ + public static MarkdownPipelineBuilder UseMyExtension(this MarkdownPipelineBuilder pipeline) + { + // Directly access or modify pipeline.Extensions here, with the ability to + // search for other extensions, insert before or after, remove other extensions, + // or modify their settings. + + // ... + + return pipeline; + } +} + +``` + +### Simple Extension Example + +An example of a simple extension which does not add any new parsers, but instead creates a new, horrific emphasis tag, marked by triple percentage signs. This example is based on [CitationExtension.cs](https://github.com/xoofx/markdig/blob/master/src/Markdig/Extensions/Citations/CitationExtension.cs) + +```csharp +/// +/// An extension which applies to text of the form %%%text%%% +/// +public class BlinkExtension : IMarkdownExtension +{ + // This setup method will be run when the pipeline builder's `Build()` method is invoked. As this + // is a simple, self-contained extension we won't be adding anything new, but rather finding an + // existing parser already in the pipeline and adding some settings to it. + public void Setup(MarkdownPipelineBuilder pipeline) + { + // We check the pipeline builder's inline parser collection and see if we can find a parser + // registered of the type EmphasisInlineParser. This is the parser which nominally handles + // bold and italic emphasis, but we know from its documentation that it is a general parser + // that can have new characters added to it. + var parser = pipeline.InlineParsers.FindExact(); + + // If we find the parser and it doesn't already have the % character registered, we add + // a descriptor for 3 consecutive % signs. This is specific to the EmphasisInlineParser and + // is just used here as an example. + if (parser is not null && !parser.HasEmphasisChar('%')) + { + parser.EmphasisDescriptors.Add(new EmphasisDescriptor('%', 3, 3, false)); + } + } + + // This method is called by the pipeline before rendering, which is a separate operation from + // parsing. This implementation is just here for the purpose of the example, in which we + // daisy-chain a delegate specific to the EmphasisInlineRenderer to cause an unconscionable tag + // to be inserted into the HTML output wherever a %%% annotated span was placed in the source. + public void Setup(MarkdownPipeline pipeline, IMarkdownRenderer renderer) + { + if (renderer is not HtmlRenderer) return; + + var emphasisRenderer = renderer.ObjectRenderers.FindExact(); + if (emphasisRenderer is null) return; + + var previousTag = emphasisRenderer.GetTag; + emphasisRenderer.GetTag = inline => + (inline.DelimiterCount == 3 && inline.DelimiterChar == '%' ? "blink" : null) + ?? previousTag(inline); + } +} +``` + +## Parsers + +Markdig has two types of parsers, both of which derive from `ParserBase`. + +Block parsers, derived from `BlockParser`, identify block elements from lines in the source text and push them onto the abstract syntax tree. Inline parsers, derived from `InlineParser`, identify inline elements from `LeafBlock` elements and push them into an attached container. + +Both inline and block parsers are regex-free, and instead work on finding opening characters and then making fast read-only views into the source text. + +### Block Parser + +**(The contents of this section I am very unsure of, this is from my reading of the code but I could use some guidance here)** + +**(Does `CanInterrupt` specifically refer to interrupting a paragraph block?)** + +In order to be added to the parsing pipeline, all block parsers must be derived from `BlockParser`. + +Internally, the main parsing algorithm will be stepping through the source text, using the `HasOpeningCharacter(char c)` method of the block parser collection to pre-identify parsers which *could* be opening a block at a given position in the text based on the active character. Thus any derived implementation needs to set the value of the `char[]? OpeningCharacter` property with the initial characters that might begin the block. + +If a parser can potentially open a block at a place in the source text it should expect to have the `TryOpen(BlockProcessor processor)` method called. This is a virtual method that must be implemented on any derived class. The `BlockProcessor` argument is a reference to an object which stores the current state of parsing and the position in the source. + +**(What are the rules concerning how the `BlockState` return type should work for `TryOpen`? I see examples returning `None`, `Continue`, `BreakDiscard`, `ContinueDiscard`. How does the return value change the algorithm behavior?)** + +**(Should a new block always be pushed into `processor.NewBlocks` in the `TryOpen` method?)** + +As the main parsing algorithm moves forward, it will then call `TryContinue(...)` on blocks that were opened in `TryOpen(..)`. + +**(Is this where/how you close a block? Is there anything that needs to be done to perform that beyond `block.UpdateSpanEnd` and returning `BlockState.Break`?)** + + +### Inline Parser \ No newline at end of file diff --git a/doc/parsing-overview.md b/doc/parsing-overview.md new file mode 100644 index 00000000..e45f28f2 --- /dev/null +++ b/doc/parsing-overview.md @@ -0,0 +1,227 @@ +# Markdig Parsing + +Markdig provides efficient, regex-free parsing of markdown documents directly into an abstract syntax tree (AST). The AST is a representation of the markdown document's semantic constructs, which can be manipulated and explored programmatically. + +* This document contains a general overview of the parsing system and components and their use +* The [Abstract Syntax Tree](parsing-ast.md) document contains a discussion of how Markdig represents the product of the parsing operation +* The [Extensions/Parsers](parsing-extensions.md) document explores extensions and block/inline parsers within the context of extending Markdig's parsing capabilities + +## Introduction + +Markdig's parsing machinery consists of two main components at its surface: the `MarkdownParser` and the `MarkdownPipeline`. The parsed document is represented by a `MarkdownDocument` object, which is a tree of objects derived from `MarkdownObject`, including block and inline elements. + +The `MarkdownParser` is a static class which contains the `Parse(...)` method, the main algorithm for parsing a markdown document. The `Parse(...)` method in turn uses a `MarkdownPipeline`, which is a sealed internal class which maintains some configuration information and the collections of parsers and extensions. The `MarkdownPipeline` determines how the parser behaves and what its capabilities are. The `MarkdownPipeline` can be modified with built-in as well as user developed extensions. + +### Glossary of Relevant Types + +The following is a table of some of the types relevant to parsing and mentioned in the related documentation. For an exhaustive list refer to API documentation (coming soon). + +|Type|Description| +|-|-| +|`MarkdownParser`|Static entry point to the parsing algorithm via the `Parse(...)` method| +|`MarkdownPipeline`|Configuration object for the parser, contains collections of block and inline parsers and registered extensions| +|`MarkdownPipelineBuilder`|Responsible for constructing the `MarkdownPipeline`, used by client code to configure pipeline options and behaviors| +|`IMarkdownExtension`|Interface for [Extensions](#extensions-imarkdownextension) which alter the behavior of the pipeline, this is the standard mechanism for extending Markdig| +|`BlockParser`|Base type for an individual parsing component meant to identify `Block` elements in the markdown source| +|`InlineParser`|Base type for an individual parsing component meant to identify `Inline` elements within a `Block`| +|`Block`|A node in the AST representing a markdown block element, can either be a `ContainerBlock` or a `LeafBlock`| +|`Inline`|A node in the AST representing a markdown inline element| +|`MarkdownDocument`|The root node of the AST produced by the parser, derived from `ContainerBlock`| +|`MarkdownObject`|The base type of all `Block` and `Inline` derived objects (as well as `HtmlAttributes`)| + +### Simple Examples + +*The following are simple examples of parsing to help get you started, see the following sections for an in-depth explanation of the different parts of Markdig's parsing mechanisms* + +The `MarkdownPipeline` dictate how the parser will behave. The `MarkdownParser.Parse(...)` method will construct a default pipeline if none is provided. A default pipeline will be CommonMark compliant but nothing else. + +```csharp +var markdownText = File.ReadAllText("sample.md"); + +// No pipeline provided means a default pipeline will be used +var document = MarkdownParser.Parse(markdownText); +``` + +Pipelines can be created and configured manually, however this must be done using a `MarkdownPipelineBuilder` object, which then is configured through a fluent interface composed of extension methods. + +```csharp +var markdownText = File.ReadAllText("sample.md"); + +// Markdig's "UseAdvancedExtensions" option includes many common extensions beyond +// CommonMark, such as citations, figures, footnotes, grid tables, mathematics +// task lists, diagrams, and more. +var pipeline = new MarkdownPipelineBuilder() + .UseAdvancedExtensions() + .Build(); + +var document = MarkdownParser.Parse(markdownText, pipeline); +``` + +Extensions can also be added individually: + +```csharp +var markdownText = File.ReadAllText("sample.md"); + +var pipeline = new MarkdownPipelineBuilder() + .UseCitations() + .UseFootnotes() + .UseMyCustomExtension() + .Build(); + +var document = MarkdownParser.Parse(markdownText, pipeline); +``` + +## The Parser and the Pipeline + +As metioned in the [Introduction](#introduction), Markdig's parsing machinery involves two surface components: the `MarkdownParser`, and the `MarkdownPipeline`. The main parsing algorithm (not to be confused with individual `BlockParser` and `InlineParser` components) lives in the `MarkdownParser.Parse(...)` static method. The `MarkdownPipeline` is responsible for configuring the behavior of the parser. + +These two components are covered in further detail in the following sections. + +### The MarkdownPipeline + +The `MarkdownPipeline` is a sealed internal class which dictates what features the parsing algorithm has. The pipeline must be created by using a `MarkdownPipelineBuilder` as shown in the examples above. + +The `MarkdownPipeline` holds configuration information and collections of extensions and parsers. Parsers fall into one of two categories: + +* Block Parsers (`BlockParser`) +* Inline Parsers (`InlineParser`) + +Extensions are classes implementing `IMarkdownExtension` which are allowed to add to the list of parsers, or modify existing parsers and/or renderers. They are invoked to perform their mutations on the pipeline when the pipeline is built by the `MarkdownPipelineBuilder`. + +Lastly, the `MarkdownPipeline` contains a few extra elements: + +* A configuration setting determining whether or not trivial elements, referred to as *trivia*, (whitespace, extra heading characters, unescaped strings, etc) are to be tracked +* A configuration setting determining whether or not nodes in the resultant abstract syntax tree will refer to their precise original locations in the source +* An optional delegate which will be invoked when the document has been processed. +* An optional `TextWriter` which will get debug logging from the parser + +### The MarkdownParser + +`MarkdownParser` is a stateless static class which contains the overall parsing algorithm but not the actual parsing components, which instead are contained within the pipeline. + +The `MarkdownParser.Parse(...)` method takes a string containing raw markdown and returns a `MarkdownDocument`, which is the root node in the abstract syntax tree. The `Parse(...)` method optionally takes a pre-configured `MarkdownPipeline`, but if none is given will create a default pipeline which has minimal features. + +Within the `Parse(...)` method, the following sequence of operations occur: + +1. The block parsers contained in the pipeline are invoked on the raw markdown text, creating the initial tree of block elements +2. If the pipeline is configured to track markdown trivia (trivial/non-contributing elements), the blocks are expanded to absorb neighboring trivia +3. The inline parsers contained in the pipeline are now invoked on the blocks, populating the inline elements of the abstract syntax tree +4. If a delegate has been configured for when the document has completed processing, it is now invoked +5. The abstract syntax tree (`MarkdownDocument` object) is returned + +## The Pipeline Builder and Extensions + +The `MarkdownPipeline` determines the behavior and capabilities of the parser, and *extensions* added via the `MarkdownPipelineBuilder` determine the configuration of the pipeline. + +This section discusses the pipeline builder and the concept of *extensions* in more detail. + +### Extensions (IMarkdownExtension) + +Extensions are the primary mechanism for modifying the parsers in the pipeline. + +An extension is any class which implements the `IMarkdownExtension` interface found in [IMarkdownExtension.cs](https://github.com/xoofx/markdig/blob/master/src/Markdig/IMarkdownExtension.cs). This interface consists solely of two `Setup(...)` overloads, which both take a `MarkdownPipelineBuilder` as the first argument. + +When the `MarkdownPipelineBuilder.Build()` method is invoked as the final stage in pipeline construction, the builder runs through the list of registered extensions in order and calls the `Setup(...)` method on each of them. The extension then has full access to modify both the parser collections themselves (by adding new parsers to it), or to find and modify existing parsers. + +Because of this, *some* extensions may need to be ordered in relation to others, for instance if they modify a parser that gets added by a different extension. The `OrderedList` class contains convenience methods to this end, which aid in finding other extensions by type and then being able to added an item before or after them. + +For a discussion on how to implement an extension, refer to the [Extensions/Parsers](parsing-extensions.md) document. + +### The MarkdownPipelineBuilder + +Because the `MarkdownPipeline` is a sealed internal class, it cannot (and *should* not be attempted to) be created directly. Rather, the `MarkdownPipelineBuilder` manages the requisite construction of the pipeline after the configuration has been provided by the client code. + +As discussed in the [section above](#the-markdownpipeline), the `MarkdownPipeline` primarily consists of a collection of block parsers and a collection of inline parsers, which are provided to the `MarkdownParser.Parse(...)` method and thus determine its features and behavior. Both the collections and some of the parsers themselves are mutable, and the mechanism of mutation is the `Setup(...)` method of the `IMarkdownExtension` interface. This is covered in more detail in the section on [Extensions](#extensions-imarkdownextension). + +#### The Fluent Interface + +A collection of extension methods in the [MarkdownExtensions.cs](https://github.com/xoofx/markdig/blob/master/src/Markdig/MarkdownExtensions.cs) source file provides a convenient fluent API for the configuration of the pipeline builder. This should be considered the standard way of configuring the builder. + +##### Configuration Options + +There are several extension methods which apply configurations to the builder which change settings in the pipeline outside of the use of typical extensions. + +|Method|Description| +|-|-| +|`.ConfigureNewLine(...)`|Takes a string which will serve as the newline delimiter during parsing| +|`.DisableHeadings()`|Disables the parsing of ATX and Setex headings| +|`.DisableHtml()`|Disables the parsing of HTML elements| +|`.EnableTrackTrivia()`|Enables the tracking of trivia (trivial elements like whitespace)| +|`.UsePreciseSourceLocation()`|Maps syntax objects to their precise location in the original source, such as would be required for syntax highlighting| + +```csharp +var builder = new MarkdownPipelineBuilder() + .ConfigureNewLine("\r\n") + .DisableHeadings() + .DisableHtml() + .EnableTrackTrivia() + .UsePreciseSourceLocation(); + +var pipeline = builder.Build(); +``` + +##### Adding Extensions + +All extensions which ship with Markdig can be added through a dedicated fluent method, while user code which implements the `IMarkdownExtension` interface can be added with one of the `Use()` methods, or via a custom extension method implemented in the client code. + +Refer to [MarkdownExtensions.cs](https://github.com/xoofx/markdig/blob/master/src/Markdig/MarkdownExtensions.cs) for a full list of extension methods: + +```csharp +var builder = new MarkdownPipelineBuilder() + .UseFootnotes() + .UseFigures(); +``` + +For custom/user-provided extensions, the `Use(...)` methods allow either a type to be directly added or an already constructed instance to be put into the extension container. Internally they will prevent two of the same type of extension from being added to the container. + +```csharp +public class MyExtension : IMarkdownExtension +{ + // ... +} + +// Only works if MyExtension has an empty constructor (aka new()) +var builder = new MarkdownPipelineBuilder() + .Use(); +``` + +Alternatively: + +```csharp +public class MyExtension : IMarkdownExtension +{ + public MyExtension(object someConfigurationObject) { /* ... */ } + // ... +} + +var instance = new MyExtension(configData); + +var builder = new MarkdownPipelineBuilder() + .Use(instance); +``` + +##### Adding Extensions with the Configure Method + +The `MarkdownPipelineBuilder` has one additional method for the configuration of extensions worth mentioning: the `Configure(...)` method, which takes a `string?` of `+` delimited tokens specifying which extensions should be dynamically configured. This is a convenience method for the configuration of pipelines whose extensions are only known at runtime. + +Refer to [MarkdownExtensions.cs's `Configure(...)`](https://github.com/xoofx/markdig/blob/983187eace6ba02ee16d1443c387267ad6e78f58/src/Markdig/MarkdownExtensions.cs#L538) code for the full list of extensions. + + +```csharp +var builder = new MarkdownPipelineBuilder() + .Configure("common+footnotes+figures"); + +var pipeline = builder.Build(); +``` + +#### Manual Configuration + +Internally, the fluent interface wraps manual operations on the three primary collections: + +* `MarkdownPipelineBuilder.BlockParsers` - this is an `OrderedList` of the block parsers +* `MarkdownPipelineBuilder.InlineParsers` - this is an `OrderedList` of the inline element parsers +* `MarkdownPipelineBuilder.Extensions` - this is an `OrderedList` of the extensions + +All three collections are `OrderedList`, which is a collection type custom to Markdig which contains special methods for finding and inserting derived types. With the builder created, manual configuration can be performed by accessing these collections and their elements and modifying them as necessary. + +***Warning**: be aware that it should not be necessary to directly modify either the `BlockParsers` or the `InlineParsers` collections directly during the pipeline configuration. Rather, these can and should be modified whenever possible through the `Setup(...)` method of extensions, which will be deferred until the pipeline is actually built and will allow for ordering such that operations dependent on other operations can be accounted for.* From 25db6cb4141649dc1b5b497f1e6d423d198e053b Mon Sep 17 00:00:00 2001 From: Matt Jarvis Date: Tue, 15 Mar 2022 21:01:45 -0400 Subject: [PATCH 2/2] updated with initial comments --- doc/parsing-ast.md | 99 ++++++++++++++++++++++++++++++++++++----- doc/parsing-overview.md | 24 +++++----- 2 files changed, 101 insertions(+), 22 deletions(-) diff --git a/doc/parsing-ast.md b/doc/parsing-ast.md index 3945ff33..cbffc6b1 100644 --- a/doc/parsing-ast.md +++ b/doc/parsing-ast.md @@ -1,6 +1,6 @@ # The Abstract Syntax Tree -If successful, the `MarkdownParser.Parse(...)` method returns the abstract syntax tree (AST) of the source text. +If successful, the `Markdown.Parse(...)` method returns the abstract syntax tree (AST) of the source text. This will be an object of the `MarkdownDocument` type, which is in turn derived from a more general block container and is part of a larger taxonomy of classes which represent different semantic constructs of a markdown syntax tree. @@ -14,7 +14,69 @@ The root of the AST is the `MarkdownDocument` which is itself derived from a con Different semantic constructs are represented by types derived from the `Block` and `Inline` types, which are both `abstract` themselves. These elements are produced by `BlockParser` and `InlineParser` derived types, respectively, and so new constructs can be added with the implementation of a new block or inline parser and a new block or inline type, as well as an extension to register it in the pipeline. For more information on extending Markdig this way refer to the [Extensions/Parsers](parsing-extensions.md) document. -The AST is assembled by the static method `MarkdownParser.Parse(...)` using the collections of block and inline parsers contained in the `MarkdownPipeline`. For more detailed information refer to the [Markdig Parsing Overview](parsing-overview.md) document. +The AST is assembled by the static method `Markdown.Parse(...)` using the collections of block and inline parsers contained in the `MarkdownPipeline`. For more detailed information refer to the [Markdig Parsing Overview](parsing-overview.md) document. + +### Quick Examples: Descendants API + +The easiest way to traverse the abstract syntax tree is with a group of extension methods that have the name `Descendants`. Several different overloads exist to allow it to search for both `Block` and `Inline` elements, starting from any node in the tree. + +The `Descendants` methods return `IEnumerable` or `IEnumerable` as their results. Internally they are using `yield return` to perform edge traversals lazily. + +#### Depth-First Like Traversal of All Elements + +```csharp +MarkdownDocument result = Markdown.Parse(sourceText, pipeline); + +// Iterate through all MarkdownObjects in a depth-first order +foreach (var item in result.Descendants()) +{ + Console.WriteLine(item.GetType()); + + // You can use pattern matching to isolate elements of certain type, + // otherwise you can use the filtering mechanism demonstrated in the + // next section + if (item is ListItemBlock listItem) + { + // ... + } +} +``` + +#### Filtering of Specific Child Types + +Filtering can be performed using the `Descendants()` method, in which T is required to be derived from `MarkdownObject`. + +```csharp +MarkdownDocument result = Markdown.Parse(sourceText, pipeline); + +// Iterate through all ListItem blocks +foreach (var item in result.Descendants()) +{ + // ... +} + +// Iterate through all image links +foreach (var item in result.Descendants().Where(x => x.IsImage)) +{ + // ... +} +``` + +#### Combined Hierarchies + +The `Descendants` method can be used on any `MarkdownObject`, not just the root node, so complex hierarchies can be queried. + +```csharp +MarkdownDocument result = Markdown.Parse(sourceText, pipeline); + +// Find all Emphasis inlines which descend from a ListItem block +var items = document.Descendants() + .Select(block => block.Descendants()); + +// Find all Emphasis inlines whose direct parent block is a ListItem +var other = document.Descendants() + .Where(inline => inline.ParentBlock is ListItemBlock); +``` ## Block Elements @@ -27,29 +89,46 @@ Block elements in markdown refer to things like paragraphs, headings, lists, cod ### Properties of Blocks +The following are properties of `Block` objects which warrant elaboration. For a full list of properties see the generated API documentation (coming soon). + +#### Block Parent All blocks have a reference to a parent (`Parent`) of type `ContainerBlock?`, which allows for efficient traversal up the abstract syntax tree. The parent will be `null` in the case of the root node (the `MarkdownDocument`). -Additionally, all blocks have a reference to a parser (`Parser`) of type `BlockParser?` which refers to the instance of the parser which created this block. +#### Parser -Blocks have an `IsOpen` boolean flag which is set true while they're being parsed **(is this true?)** and then closed when parsing is complete. +All blocks have a reference to a parser (`Parser`) of type `BlockParser?` which refers to the instance of the parser which created this block. + +#### IsOpen Flag + +Blocks have an `IsOpen` boolean flag which is set true while they're being parsed and then closed when parsing is complete. + +Blocks are created by `BlockParser` objects which are managed by an instance of a `BlockProcessor` object. During the parsing algorithm the `BlockProcessor` maintains a list of all currently open `Block` objects as it steps through the source line by line. The `IsOpen` flag indicates to the `BlockProcessor` that the block should remain open as the next line begins. If the `IsOpen` flag is not directly set by the `BlockParser` on each line, the `BlockProcessor` will consider the `Block` fully parsed and will no longer call its `BlockParser` on it. + +#### IsBreakable Flag + +Blocks are either breakable or not, specified by the `IsBreakable` flag. If a block is non-breakable it indicates to the parser that the close condition of any parent container do not apply so long as the non-breakable child block is still open. + +The only built-in example of this is the `FencedCodeBlock`, which, if existing as the child of a container block of some sort, will prevent that container from being closed before the `FencedCodeBlock` is closed, since any characters inside the `FencedCodeBlock` are considered to be valid code and not the container's close condition. + +#### RemoveAfterProcessInlines -Blocks are either breakable or not, specified by the `IsBreakable` flag. **(What is the significance of breakable blocks? Can't be split? Is anything besides `FencedCodeBlock` not breakable?)** -**(Is there anything special that should be documented for the ParagraphBlock or any other specific type of blocks?)** ## Inline Elements +Inlines in markdown refer to things like embellishments (italics, bold, underline, etc), links, urls, inline code, images, etc. + Inline elements may be one of two types: -1. `ContainerInline`, which contains other inlines, and whose parent may be a `LeafBlock` or another `ContainerInline` (**is this true?**) -2. `Inline`, whose parent is always a `ContainerInline` (**is this true?**) +1. `Inline`, whose parent is always a `ContainerInline` +2. `ContainerInline`, derived from `Inline`, which contains other inlines. `ContainerInline` also has a `ParentBlock` property of type `LeafBlock?` -Inlines in markdown refer to things like embellishments (italics, bold, underline, etc), links, urls, inline code, images, etc. **(Is there anything special worth documenting about inlines or types of inlines?)** ## The SourceSpan Struct -If the pipeline was configured with `.UsePreciseSourceLocation()`, all elements in the abstract syntax tree will contain a reference to the location in the original source where they occurred. This is done with the `SourceSpan` class, a custom Markdig `struct` which provides a start and end location. +If the pipeline was configured with `.UsePreciseSourceLocation()`, all elements in the abstract syntax tree will contain a reference to the location in the original source where they occurred. This is done with the `SourceSpan` type, a custom Markdig `struct` which provides a start and end location. All objects derived from `MarkdownObject` contain the `Span` property, which is of type `SourceSpan`. + diff --git a/doc/parsing-overview.md b/doc/parsing-overview.md index e45f28f2..1a489102 100644 --- a/doc/parsing-overview.md +++ b/doc/parsing-overview.md @@ -8,9 +8,9 @@ Markdig provides efficient, regex-free parsing of markdown documents directly in ## Introduction -Markdig's parsing machinery consists of two main components at its surface: the `MarkdownParser` and the `MarkdownPipeline`. The parsed document is represented by a `MarkdownDocument` object, which is a tree of objects derived from `MarkdownObject`, including block and inline elements. +Markdig's parsing machinery consists of two main components at its surface: the `Markdown.Parse(...)` method and the `MarkdownPipeline` type. The parsed document is represented by a `MarkdownDocument` object, which is a tree of objects derived from `MarkdownObject`, including block and inline elements. -The `MarkdownParser` is a static class which contains the `Parse(...)` method, the main algorithm for parsing a markdown document. The `Parse(...)` method in turn uses a `MarkdownPipeline`, which is a sealed internal class which maintains some configuration information and the collections of parsers and extensions. The `MarkdownPipeline` determines how the parser behaves and what its capabilities are. The `MarkdownPipeline` can be modified with built-in as well as user developed extensions. +The `Markdown` static class is the main entrypoint to the Markdig API. It contains the `Parse(...)` method, the main algorithm for parsing a markdown document. The `Parse(...)` method in turn uses a `MarkdownPipeline`, which is a sealed internal class which maintains some configuration information and the collections of parsers and extensions. The `MarkdownPipeline` determines how the parser behaves and what its capabilities are. The `MarkdownPipeline` can be modified with built-in as well as user developed extensions. ### Glossary of Relevant Types @@ -18,7 +18,7 @@ The following is a table of some of the types relevant to parsing and mentioned |Type|Description| |-|-| -|`MarkdownParser`|Static entry point to the parsing algorithm via the `Parse(...)` method| +|`Markdown`|Static class with the entry point to the parsing algorithm via the `Parse(...)` method| |`MarkdownPipeline`|Configuration object for the parser, contains collections of block and inline parsers and registered extensions| |`MarkdownPipelineBuilder`|Responsible for constructing the `MarkdownPipeline`, used by client code to configure pipeline options and behaviors| |`IMarkdownExtension`|Interface for [Extensions](#extensions-imarkdownextension) which alter the behavior of the pipeline, this is the standard mechanism for extending Markdig| @@ -33,13 +33,13 @@ The following is a table of some of the types relevant to parsing and mentioned *The following are simple examples of parsing to help get you started, see the following sections for an in-depth explanation of the different parts of Markdig's parsing mechanisms* -The `MarkdownPipeline` dictate how the parser will behave. The `MarkdownParser.Parse(...)` method will construct a default pipeline if none is provided. A default pipeline will be CommonMark compliant but nothing else. +The `MarkdownPipeline` dictate how the parser will behave. The `Markdown.Parse(...)` method will construct a default pipeline if none is provided. A default pipeline will be CommonMark compliant but nothing else. ```csharp var markdownText = File.ReadAllText("sample.md"); // No pipeline provided means a default pipeline will be used -var document = MarkdownParser.Parse(markdownText); +var document = Markdown.Parse(markdownText); ``` Pipelines can be created and configured manually, however this must be done using a `MarkdownPipelineBuilder` object, which then is configured through a fluent interface composed of extension methods. @@ -54,7 +54,7 @@ var pipeline = new MarkdownPipelineBuilder() .UseAdvancedExtensions() .Build(); -var document = MarkdownParser.Parse(markdownText, pipeline); +var document = Markdown.Parse(markdownText, pipeline); ``` Extensions can also be added individually: @@ -68,12 +68,12 @@ var pipeline = new MarkdownPipelineBuilder() .UseMyCustomExtension() .Build(); -var document = MarkdownParser.Parse(markdownText, pipeline); +var document = Markdown.Parse(markdownText, pipeline); ``` ## The Parser and the Pipeline -As metioned in the [Introduction](#introduction), Markdig's parsing machinery involves two surface components: the `MarkdownParser`, and the `MarkdownPipeline`. The main parsing algorithm (not to be confused with individual `BlockParser` and `InlineParser` components) lives in the `MarkdownParser.Parse(...)` static method. The `MarkdownPipeline` is responsible for configuring the behavior of the parser. +As metioned in the [Introduction](#introduction), Markdig's parsing machinery involves two surface components: the `Markdown.Parse(...)` method, and the `MarkdownPipeline` type. The main parsing algorithm (not to be confused with individual `BlockParser` and `InlineParser` components) lives in the `Markdown.Parse(...)` static method. The `MarkdownPipeline` is responsible for configuring the behavior of the parser. These two components are covered in further detail in the following sections. @@ -95,11 +95,11 @@ Lastly, the `MarkdownPipeline` contains a few extra elements: * An optional delegate which will be invoked when the document has been processed. * An optional `TextWriter` which will get debug logging from the parser -### The MarkdownParser +### The Markdown.Parse Method -`MarkdownParser` is a stateless static class which contains the overall parsing algorithm but not the actual parsing components, which instead are contained within the pipeline. +`Markdown.Parse` is a static method which contains the overall parsing algorithm but not the actual parsing components, which instead are contained within the pipeline. -The `MarkdownParser.Parse(...)` method takes a string containing raw markdown and returns a `MarkdownDocument`, which is the root node in the abstract syntax tree. The `Parse(...)` method optionally takes a pre-configured `MarkdownPipeline`, but if none is given will create a default pipeline which has minimal features. +The `Markdown.Parse(...)` method takes a string containing raw markdown and returns a `MarkdownDocument`, which is the root node in the abstract syntax tree. The `Parse(...)` method optionally takes a pre-configured `MarkdownPipeline`, but if none is given will create a default pipeline which has minimal features. Within the `Parse(...)` method, the following sequence of operations occur: @@ -131,7 +131,7 @@ For a discussion on how to implement an extension, refer to the [Extensions/Pars Because the `MarkdownPipeline` is a sealed internal class, it cannot (and *should* not be attempted to) be created directly. Rather, the `MarkdownPipelineBuilder` manages the requisite construction of the pipeline after the configuration has been provided by the client code. -As discussed in the [section above](#the-markdownpipeline), the `MarkdownPipeline` primarily consists of a collection of block parsers and a collection of inline parsers, which are provided to the `MarkdownParser.Parse(...)` method and thus determine its features and behavior. Both the collections and some of the parsers themselves are mutable, and the mechanism of mutation is the `Setup(...)` method of the `IMarkdownExtension` interface. This is covered in more detail in the section on [Extensions](#extensions-imarkdownextension). +As discussed in the [section above](#the-markdownpipeline), the `MarkdownPipeline` primarily consists of a collection of block parsers and a collection of inline parsers, which are provided to the `Markdown.Parse(...)` method and thus determine its features and behavior. Both the collections and some of the parsers themselves are mutable, and the mechanism of mutation is the `Setup(...)` method of the `IMarkdownExtension` interface. This is covered in more detail in the section on [Extensions](#extensions-imarkdownextension). #### The Fluent Interface