Lazy line scanning #173
Description
Behavior
Now:
The input is split into lines and each line is scanned into one of different line structs
Dave did this to be able to scan lines in parallel which must bring performance gains for normal markdown I guess.
However, I removed the parallel scanning, when I realized how much has to be done later.
If there was a performance penalty, it was never noted/reported.
In the Near Future ;)
Lazily scan lines by extracting line by line (up to ~r/\n\r?/) and only after processing the scanned line scanning the next iff needed.
Rationale
When stumbling upon text which will need to access future lines, we need to grab into the succeeding line structs.
This makes some actions cumbersome, e.g. multiline inline code
It also makes parsing with tools like NimbleParsec extremely costly (we need to rejoin all following lines and then remove the parsed lines, and we are not even sure that line boundaries are where the parser finishes).
This point derives directly from @bradhanks musings about Refactoring HTML Parsing
While these two points hint that scanning lines lazily would make the code simpler, there are no guarantees that there are no obstacles, maybe even road blockers by applying this approach before closer analysis.
It goes however without saying, that I am not aware of any right now :)