Document parser authoring contracts and migration risks

doc/parser-authoring-api-breaking-changes.md

@@ -0,0 +1,77 @@
+# Parser + AST API Migration Notes (Potential Breaking Changes)
+
+This document tracks compatibility risks for consumers adopting the new parser/AST authoring APIs and for future releases that may tighten contracts.
+
+## Current Release Impact
+
+The implemented changes are additive/public-surface expansions:
+
+- `InlineProcessor.ReplaceParentContainer(...)` is now public.
+- `BlockProcessor.TakeLinesBefore()` is public.
+- `BlockProcessor.IsOpen(Block)` is public.
+- `BlockProcessor.TryDiscard(Block)` is new.
+- `InlineProcessor.GetParserState(...)` and `InlineProcessor.Emit(...)` are new.
+- `Block.Remove()`, `Block.ReplaceBy(...)`, `ContainerBlock.TransferChildrenTo(...)`, and `ContainerInline.TransferChildrenTo(...)` are new.
+- `MarkdownPipelineBuilder.TrackTrivia` setter is public.
+- Typed metadata helpers (`DataKey<T>`, `MarkdownObjectDataExtensions`) are new.
+
+No runtime behavior change is required for existing consumers unless they opt into these APIs.
+
+## Potential Future Breaking Changes
+
+These are not implemented in this release, but consumers should avoid relying on ambiguous behavior.
+
+### 1. `ReplaceParentContainer` replacement limits
+
+Current contract allows a single replacement request per leaf pass. If this evolves to support multiple requests, error behavior and ordering guarantees may change.
+
+Guidance:
+
+- Keep replacements localized and deterministic.
+- Avoid depending on repeated replacement attempts in one pass.
+
+### 2. Mutation-helper metadata behavior
+
+Mutation helpers currently do not recalculate spans/trivia automatically. A future strict mode might validate or enforce metadata consistency.
+
+Guidance:
+
+- Explicitly set/update `Span`, `Line`, `Column`, and trivia fields in transforms where source fidelity matters.
+
+### 3. Internal helper cleanup
+
+Internal compatibility shims may be removed in a later major release after migration (for example, internal aliases kept during transition).
+
+Guidance:
+
+- Use public methods (`TakeLinesBefore`, `TryDiscard`, etc.) directly.
+
+### 4. Typed metadata API naming and overload resolution
+
+Some helper overloads share names with existing instance methods. Advanced consumers should prefer explicit generic invocations to avoid ambiguity.
+
+Guidance:
+
+- Prefer explicit generic calls:
+  - `node.SetData<MyType>(value)`
+  - `node.SetData<MyType>(key, value)`
+  - `node.GetData<MyType>(key)`
+  - `node.TryGetData<MyType>(key, out var value)`
+
+## Migration Checklist for Extension Authors
+
+1. Replace custom open-stack discard patterns with `TryDiscard` where appropriate.
+2. Use `TakeLinesBefore()` instead of internal trivia helpers.
+3. For inline-driven parent replacement:
+   - replace the parent container in the AST first,
+   - then call `ReplaceParentContainer(old, @new)`.
+4. Replace manual child-move loops (`RemoveAt(0)` patterns) with transfer helpers.
+5. Add regression tests for transformed AST shape (parent/child ownership and ordering).
+
+## Versioning Guidance
+
+For a future major version, consider:
+
+- deprecating internal transition shims,
+- optionally adding strict validation mode for parser invariants and mutation metadata,
+- documenting any tightened contracts as migration steps in this file.

doc/parser-authoring-api.md

@@ -0,0 +1,139 @@
+# Parser + AST Authoring API Guide
+
+This guide documents the parser authoring contracts and the new public APIs added for extension parity and safe AST manipulation.
+
+## Scope
+
+The authoring model remains two-phase:
+
+1. Block parsing (`BlockProcessor` + `BlockParser`).
+2. Inline parsing (`InlineProcessor` + `InlineParser` + `IPostInlineProcessor`).
+
+Performance characteristics are unchanged: parser dispatch still relies on opening-character maps and pooled processors.
+
+## Public API Additions
+
+### `InlineProcessor.ReplaceParentContainer(...)`
+
+Use this when an inline parser replaces a parent container block during inline processing.
+
+Contract:
+
+- Call only while processing the current leaf (`ProcessInlineLeaf` pass).
+- Replace the container in the AST first, then call `ReplaceParentContainer(old, @new)` to synchronize traversal state.
+- Only one replacement request is allowed per leaf processing pass.
+
+### `BlockProcessor.TakeLinesBefore()`
+
+Use this in `TryOpen` when `TrackTrivia` is enabled and the new block should own pending leading blank/trivia lines.
+
+Contract:
+
+- Returns current `LinesBefore` and clears it.
+- Returns `null` when there is no pending list.
+
+### `BlockProcessor.IsOpen(Block)` and `BlockProcessor.TryDiscard(Block)`
+
+Use these to safely reason about/modify the open block stack.
+
+Contract:
+
+- `IsOpen` is a read-only stack membership check.
+- `TryDiscard` removes an open non-root block from both parent container and open stack, returning `true` only when a discard happened.
+
+### `MarkdownPipelineBuilder.TrackTrivia`
+
+`TrackTrivia` is now publicly settable and flows into the built pipeline/processors.
+
+## Block Parser Contract
+
+### `BlockState` semantics
+
+- `None`: no match for this parser/block.
+- `Skip`: skip parser for this line and continue with others.
+- `Continue`: block stays open; leaf blocks append line content.
+- `ContinueDiscard`: block stays open; line consumed but not appended.
+- `Break`: block closes; current line remains available for further parsing.
+- `BreakDiscard`: block closes; current line is consumed.
+
+### `NewBlocks` invariants
+
+- Every pushed block must have `Parser` set to the creating parser.
+- Leaf blocks must be pushed last.
+- Push order is outer container to inner container to leaf (LIFO pop in processor).
+
+### Trivia rules
+
+When `TrackTrivia` is enabled:
+
+- Use `TakeLinesBefore()` in `TryOpen` to assign pending leading blank/trivia lines.
+- Use `UseTrivia(end)` when a parser needs exact trivia slices around syntax markers.
+
+## Inline Parser Contract
+
+### `Match` behavior
+
+- On success: advance `slice` and set `processor.Inline` when emitting a node.
+- On failure: return `false` without mutating parser output state.
+
+### Emission behavior
+
+- If a matched parser sets a parentless `processor.Inline`, the processor appends it to the deepest open inline container.
+- For explicit emission, use `processor.Emit(inline)`.
+
+### Parser state
+
+Use `processor.GetParserState<TState>(this)` or `processor.GetParserState(this, factory)` for per-leaf parser state.
+Parser states are cleared at the beginning of each `ProcessInlineLeaf`.
+
+### Block transforms from inline parsing
+
+- `processor.BlockNew` replaces the current leaf block after the current leaf pass returns.
+- `processor.ReplaceParentContainer(old, @new)` keeps traversal coherent when a parent container has already been replaced in the AST.
+
+## Hook Selection
+
+| Goal | Preferred hook |
+|---|---|
+| Add block syntax | `BlockParser` in `MarkdownPipelineBuilder.BlockParsers` |
+| Add inline syntax | `InlineParser` in `MarkdownPipelineBuilder.InlineParsers` |
+| Deferred inline resolution | `IPostInlineProcessor` |
+| Replace current leaf block | `InlineProcessor.BlockNew` |
+| Replace parent container from inline | `InlineProcessor.ReplaceParentContainer` |
+| Literal post-processing | `LiteralInlineParser.PostMatch` |
+| Post-close block transform | `BlockParser.Closed` |
+| Per-block inline begin/end behavior | `Block.ProcessInlinesBegin` / `Block.ProcessInlinesEnd` |
+| Post-document transform | `MarkdownPipelineBuilder.DocumentProcessed` |
+
+## AST Mutation Helpers
+
+### Block-level
+
+- `Block.Remove()`: remove from parent container.
+- `Block.ReplaceBy(replacement, moveChildren: true)`: replace in parent container and optionally move children.
+- `ContainerBlock.TransferChildrenTo(destination)`: move child blocks in order.
+
+### Inline-level
+
+- `ContainerInline.TransferChildrenTo(destination)`: move child inlines in order.
+
+Mutation helpers do not auto-recompute spans/trivia; callsites remain responsible for source metadata correctness when required.
+
+## Typed Metadata Helpers
+
+For extension state attached to AST nodes:
+
+- `SetData<T>(value)` / `GetData<T>()`.
+- `TryGetData<T>(key, out value)` / `GetData<T>(key)` for explicit object keys.
+- `DataKey<T>` for collision-resistant typed keys.
+
+Example:
+
+```csharp
+var key = new DataKey<MyState>();
+block.SetData<MyState>(key, state);
+if (block.TryGetData<MyState>(key, out var existing))
+{
+    // use existing
+}
+```

src/Markdig/Parsers/BlockState.cs

@@ -10,32 +10,34 @@ namespace Markdig.Parsers;
 public enum BlockState
 {
     /// <summary>
-    /// A line is not accepted by this parser.
+    /// The parser does not accept the line for this block.
+    /// No line content is consumed by this result.
     /// </summary>
     None,
 
     /// <summary>
-    /// The parser is skipped.
+    /// Skips this parser for the current line and continues with the next parser/candidate block.
     /// </summary>
     Skip,
 
     /// <summary>
-    /// The parser accepts a line and instruct to continue.
+    /// The parser accepts the line and keeps the block open.
+    /// For leaf blocks, the current line is appended to the block.
     /// </summary>
     Continue,
 
     /// <summary>
-    /// The parser accepts a line, instruct to continue but discard the line (not stored on the block)
+    /// The parser accepts the line and keeps the block open, but the line is consumed and not appended.
     /// </summary>
     ContinueDiscard,
 
     /// <summary>
-    /// The parser is ending a block, instruct to stop and keep the line being processed.
+    /// The parser ends the block and keeps the current line available for further parsing.
     /// </summary>
     Break,
 
     /// <summary>
-    /// The parser is ending a block, instruct to stop and discard the line being processed.
+    /// The parser ends the block and consumes the current line.
     /// </summary>
     BreakDiscard
-}
+}