Skip to content

lexer: fixed syntax highlighting breaks on longer files - #1357

Open
jorbakk wants to merge 14 commits into
martanne:masterfrom
jorbakk:lex_string
Open

jorbakk wants to merge 14 commits into
martanne:masterfrom
jorbakk:lex_string

Conversation

@jorbakk

@jorbakk jorbakk commented May 28, 2026 •

Copy link
Copy Markdown
Contributor

This patch fixes issue #1253

Problem description

If lexer starts parsing at a file offset determined by the horizon variable and this offset happens to be within a string, then all non-strings are highlighted as strings and vice versa.

This occurs with strings that have identical characters for their opening and closing delimiters, e.g. single quotes ', double quotes " or quotes like """. Paired delimiters that use different types of characters for start and end of the string are not affected. Most programming languages use identical characters as string delimiters. Exceptions are:

  • m4 macro processor uses backtick ` at the start and apostrophe ' at the end
  • Tcl allows quotes and braces {}
  • postscript uses parantheses

The current approach also fails when the horizon lies completely within a very big token. This is for example the case with big comments, when the starting delimiter of the comment is before the horizon. Then the lexer treats the comment as un-commented program text.

The lexer currently fails quite frequently on files with:

  • large multi-line strings, e.g. for documentation, SQL statements, statements in embedded DSLs
  • stb-style header-only libraries in C/C++, where there is usually a very large comment in the beginning of the file, which contains the API documentation of the library

Patch description

With this patch, if the horizon variable is set to -1, the whole file buffer up to the end of the viewport is run through the lexer. Performance is still reasonably good, as no syntax highlighting takes place outside of the viewport.

However, with this simple patch, performance degrades with big files when the viewport is close to the end of the file. A set of patches will follow shortly that overcome this limitation while also yielding the same correct results as this simple patch. It also achieves generally a big performance boost in almost all cases compared to the current horizon approach.

Edit 2026-09-06: more explicit and accurate problem description, including examples

@jorbakk

jorbakk commented May 31, 2026 via email

Copy link
Copy Markdown
Contributor Author

@jorbakk

jorbakk commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

This next set of patches provides incremental lexing of a file while introducing a cache for already lexed tokens. This makes the process of lexing the text into tokens for syntax highlighting much more efficient.

In order to achieve this, an event is needed that signals changes in the text, in particular where and when these changes occur. Then only the affected lexer tokens need an update, so the lexer can start from the position of the change until the end position of the current viewport. This is the minimal possible change of the token cache that also preserves as much tokens as possible and maintains a correct list of tokens that reflect the document's syntax.

Any newly created tokens should only depend on the position of changes in the document and the position of the viewport. They should not depend on a fixed sized horizon, as this leads to incorrect results and should not be required in any text editor. In addition to a correct token list, this approach also gives a performance boost, e.g. when scrolling down through a freshly opened document. Then only the new viewport has to be lexed. After hitting the bottom of the text, the whole document is in the cache on no lexing is required when moving through it, until the document is changed. If this happens, only the tokens from the cursor, where the changes take place, to the end of the viewport need to be lexed, again.

Each window maintains its own token cache, so a different syntax can be set for the same document in different windows.

@Nomarian

Nomarian commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

I don't understand any of this but localize your variables

@jorbakk

jorbakk commented Sep 8, 2026

Copy link
Copy Markdown
Contributor Author

I don't understand any of this but localize your variables

Yep, missed a few, hope I got them all now.

@Nomarian

Nomarian commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

run luacheck on it

@jorbakk

jorbakk commented Sep 9, 2026

Copy link
Copy Markdown
Contributor Author

run luacheck on it

Okay, should be clean now. Thanks!

Comment thread lua/vis-std.lua Outdated
local i1 = #tokens
if pos < tokens[i0]-1 then return i0 end
if pos > tokens[i1]-1 then return i1 end
repeat

@Nomarian Nomarian Sep 9, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

put local i repeat here and remove lines 67-70, repeat runs at least once so local i can be unnassigned

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great twist, much better!

@fischerling

Copy link
Copy Markdown
Collaborator

Hey thanks for your work!

I will test it and will report back.

Especially, the modification event looks really promising to me. I had this kind of event on the to-do list for vis-lspc for a long time.

@jorbakk

jorbakk commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Hey thanks for your work!

I will test it and will report back.

Especially, the modification event looks really promising to me. I had this kind of event on the to-do list for vis-lspc for a long time.

Thanks! I can imagine that the modification event would be useful for interactive use of an LSP client, like auto-complete. It actually sounds like a basic event to be provided by an editor. Looking forward to your feedback!

@rnpnr

rnpnr commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Now that this was at the top of my inbox again I took a quick look. I don't remember why I didn't comment on the original but this is looking quite good.

One thing I think should be changed is that we should disable lexing for "large" files. I don't see any reasonable way of handling them without doing partial lexing and I don't want your cache to blow up in size just because someone opened a "large" file.

Right now a file being large or not seems to be determined in lua but it's a property that's set during the creation of the window in C. I think it should just be exported in window_index() in vis-lua.c.

Also if you can leave a comment documenting how the cache is actually organized that would be helpful. I have a plugin which maintains a cache and I know that vis-spellcheck also has a token cache and probably we would both prefer to just access this one directly (even if we aren't explicitly making it public API).

EDIT: What I mean by the last part is that both these plugins are trying to avoid re-lexing the file over and over. They want access to the token list directly. I suppose an explicit API might be better but even just access to this cache would be sufficient.

@Nomarian

Nomarian commented Sep 12, 2026 •

Copy link
Copy Markdown
Contributor

I don't know the C side but from what I understand, Disabling syntax highlighting is just done by setting win.syntax = nil, statusbar event already sets a win.large boolean, it could be a one line change to just add a win.syntax = nil to the conditional, the other PR already disables syntax highlighting for the window as well via vis.command"set syntax off"... But I do think a cache would be nice, maybe both must be used, win.huge files and win.large, with huge files disabling many facilities, like the syntax highlighting and etc and large turning on the cache mode.

@jorbakk

jorbakk commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@rnpnr: I agree that lexing and syntax highlighting should be turned off on large files, so I had a closer look to the current way large files are handled:

  • there is View.large_file (C) and win.large (lua), they are set independently, albeit using the same threshold: file size > 2^25 or selection.col > 2^16
  • there's also a ui option UI_OPTION_LARGE_FILE that sets View.large_file using win_options_set(), however it cannot be set with a :set command.
  • the thresholds are hard coded (same value, but in two different places)
  • View.large_file and win.large are checked by polling file/line size on each STATUS_BAR update

I think that "large file" should be a property/function of the file, without storing it as additional state that needs to be updated. So I made the following changes:

  • removed UI_OPTION_LARGE_FILE property.
  • introduced one single function that checks for large files and depends only on the text state.
  • made this function available in the lua API.
  • introduced thresholds for file and line size that can be set as ui options per window to adjust the large file check.
  • Turn off lexing and syntax highlighting when check for large files is true or no syntax is set for the window.

I also added documentation for the token cache.

@Nomarian: do you think there should be two thresholds for large and huge files? For now, I think one adjustable threshold should suffice and is simpler. Keep in mind that the token cache should be enabled whenever syntax highlighting is enabled, for performance reasons.

@rnpnr

rnpnr commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

One issue I have found is that when you yank some text from another window and then paste it the cache after the pasted text is not invalidated. For example open 2 of the c files in vis and yank a comment from one and paste it above some other code in the other.

@Nomarian

Copy link
Copy Markdown
Contributor

@Nomarian: do you think there should be two thresholds for large and huge files? For now, I think one adjustable threshold should suffice and is simpler. Keep in mind that the token cache should be enabled whenever syntax highlighting is enabled, for performance reasons.

If your cache system works then there should be no need for it, I was merely stating this because its a quick addition and fix on the nosyntaxhighlighting PR and it seemed to be what Randy wanted.

@jorbakk

jorbakk commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

@Nomarian: do you think there should be two thresholds for large and huge files? For now, I think one adjustable threshold should suffice and is simpler. Keep in mind that the token cache should be enabled whenever syntax highlighting is enabled, for performance reasons.

If your cache system works then there should be no need for it, I was merely stating this because its a quick addition and fix on the nosyntaxhighlighting PR and it seemed to be what Randy wanted.

Okay. I'll need to take a closer look at your PR, but I would guess that this patch here only reads win.syntax and your patch sets it. Let me know if there's any potential conflict, and I'm sure we can fix it.

I also replied the following (with little edits) to your other comment (via email), but it bounced, the github notification bot seems to be busy:

Sorry, but I can't follow. The WIN_STATUS event callback in vis-std.lua did set win.large before I changed win.large: bool to a function win:large() -> bool. I also changed the WIN_STATUS event callback, so there should be no conflict. Please let me know if I missed s.th. here.

@jorbakk

jorbakk commented Sep 14, 2026

Copy link
Copy Markdown
Contributor Author

One issue I have found is that when you yank some text from another window and then paste it the cache after the pasted text is not invalidated. For example open 2 of the c files in vis and yank a comment from one and paste it above some other code in the other.

@rnpnr : Sorry, but I have a hard time to reproduce this ...?! I could not manage to corrupt the token cache by yanking and pasting any code snippet from one window to another ... .

@rnpnr

rnpnr commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

@rnpnr : Sorry, but I have a hard time to reproduce this ...?! I could not manage to corrupt the token cache by yanking and pasting any code snippet from one window to another ... .

My apologies, please disregard. I was testing on a different system from usual on top of my personal work branch and I didn't realize that it must have been out of date.

@rnpnr

rnpnr commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

I do have a similar example which is still an issue:

  1. Open an existing C file
  2. Start typing #inclu (this is correctly not recognized)
  3. Finish typing so you now have #include (this is incorrectly not recognized until you do something which invalidates the cache).

Note that in order to reproduce the file needs to have more than one token. I you open a new file t.c and only do these steps this first token seems to work as intended.

@jorbakk

jorbakk commented Sep 15, 2026 •

Copy link
Copy Markdown
Contributor Author

I do have a similar example which is still an issue:

  1. Open an existing C file
  2. Start typing #inclu (this is correctly not recognized)
  3. Finish typing so you now have #include (this is incorrectly not recognized until you do something which invalidates the cache).

Note that in order to reproduce the file needs to have more than one token. I you open a new file t.c and only do these steps this first token seems to work as intended.

@rnpnr Great that you spotted this! I should have stumbled across it also, but the devil's in the details ...

So here's a small patch with a long answer. I'd like to start with your example and the corresponding lexer outputs:

File contents:

int main();

lexer output:

{ type, 4, whitespace.c, 5, function, 9, operator, 10, operator, 11, operator, 12, whitespace.c, 14 }

File contents:

int main();
#inc

lexer output:

{ type, 4, whitespace.c, 5, function, 9, operator, 10, operator, 11, operator, 12, whitespace.c, 13, default, 14, identifier, 17, whitespace.c, 18 }

File contents:

int main();
#include

lexer output with min_token_cache_entries = 4, looking back 2 tokens:

{ type, 4, whitespace.c, 5, function, 9, operator, 10, operator, 11, operator, 12, whitespace.c, 13, default, 14, identifier, 21, whitespace.c, 22 }

The hash sign (now identified as 'default') is ignored and only the string 'include' and the following whitespace are re-lexed. This is why it's not identified as a preprocessor token.

lexer output with min_token_cache_entries = 6, looking back 3 tokens:

{ type, 4, whitespace.c, 5, function, 9, operator, 10, operator, 11, operator, 12, whitespace.c, 13, preprocessor, 21, whitespace.c, 22 }

The three tokens '#' = 'default', 'include' = 'identifier', and whitespace = 'whitespace.c' are re-lexed and correctly identified as 'preprocessor' and 'whitespace.c'.

Note that the incremental lexing performed here always starts at token boundaries, currently two tokens back from the modification position (min_token_cache_entries = 4). The question is how many tokens do we need to go back for incremental lexing with any grammar? This is obviously related to the complexity of the grammar, so I made a quick analysis of all grammars in lua/lexers and came to the conclusion that 8 tokens (that's 16 entries in the token cache) should cover all grammars, even the weird ones.

edit: some small changes

@jorbakk

jorbakk commented Sep 19, 2026

Copy link
Copy Markdown
Contributor Author

While testing it as my daily driver, I found another edge case which I need to take care of. Please bear with me ...

@jorbakk

jorbakk commented Sep 24, 2026

Copy link
Copy Markdown
Contributor Author

With the latest patch, I use the same heuristic as scintilla/scintillua-based editors like scite and textadept for determining the position from where to start lexing when the document has been modified.

Please note that lpeg is not intended to be a fully incremental lexer, so syntax highlighting while modifying a document without lexing it from the beginning requires heuristics and won't be 100% perfect. In order to achieve this, something like tree-sitter would be needed.

An example for incorrect highlighting is when you type this into a C file:

foo();
bar
();

Then, while typing, foo is colored as a function name, bar is not. Note, that to observe this with textadept you need to set a color for function names first (e.g. in ~/.textadept/themes/light.lua). By default function names are not colored.

However, the current approach should yield the same results as textadept, which is where also the lexer and grammars for vis come from.

@rnpnr

rnpnr commented Sep 24, 2026

Copy link
Copy Markdown
Collaborator

The latest changes fix the issue I was running into as well but wasn't able to write up yet. I will continue with more testing but I think this is getting to be in pretty good shape.

Comment thread lua/vis-std.lua Outdated
local lexer = vis.lexers.load(win.syntax, nil, true)
if not lexer then return {} end
if win.token_cache == nil then return {} end
if win.token_cache == nil or start == nil or finish == nil then return {} end

@Nomarian Nomarian Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could do this here

if -- check nothing is falsy
  not( lexer and win.token_cache and start and finish )
then
  return {}
end

But its harder to read so 😮‍💨

At the very least, the if above can be incorporated into this one

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, at least merge the two statements into one, either negated or not ... but I guess this is more concise as it doesn't need nil for every boolean expression. I'll do that but will wait a little for other changes to accumulate ... . Thanks for the hint!

@rnpnr rnpnr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have some local changes pending that will conflict with this so I would like to merge this first. Can you please address the comment I have below and the other one that was left.

I can't really comment on the TEXT_MODIFIED event. It looks reasonable to me but I can't really think of any use for it other than what you are using it for here (which doesn't need most of the arguments). I worry that it is not as useful as one might hope. From a plugin's perspective how useful is it to know where a modification happened? Besides the most simple cases making use of that information would require the plugin maintain its own representation of the entire text state.

@fischerling maybe you can comment on if this TEXT_MODIFIED event covers the uses you had in mind?

If its not usable as it is I would suggest documenting it as used internally but experimental and subject to change (and removing the argument documentation).

Comment thread view.c Outdated
bool view_large_file(View *view) {
Selection *sel = view_selections_primary_get(view);
size_t col = view_cursors_col(sel);
return text_size(view->text) > view->large_file_size << 10 /* in KiB */ ||

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead can you introduce some macros in util.h:

#define KiB(a)     ((u64)(a) << 10ULL)
#define MiB(a)     ((u64)(a) << 20ULL)
#define GiB(a)     ((u64)(a) << 30ULL)

Then you don't need the comment. I'm also worried about the current version overflowing so a cast to u64 is needed regardless.

Comment thread view.h Outdated
int wrapcolumn; /* wrap lines at minimum of window width and wrapcolumn (if != 0) */
int wrapcol; /* used while drawing view content, column where word wrap might happen */
bool prevch_breakat; /* used while drawing view content, previous char is part of breakat */
int large_file_size; /* threshold for large files in KiB */

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
int large_file_size; /* threshold for large files in KiB */
int large_file_size_kib; /* threshold for large files in KiB */

This way it is clear wherever it is used that it is in KiB.

@fischerling fischerling mentioned this pull request Oct 1, 2026
@jorbakk

jorbakk commented Oct 2, 2026

Copy link
Copy Markdown
Contributor Author

Thanks to @rnpnr and @fischerling for having a closer look and your feedback!

The following patch includes the changes proposed by @Nomarian and @rnpnr.

Regarding the modification event, generally speaking, any plugin that analyses the code / text or triggers a function based on the typed text while typing could make use of it. Additionally to syntax highlighting, the following ideas of use cases might be relevant:

  • code analysis: (syntax) errors, linter
  • text / code snippets suggestions
  • auto complete (lsp) / AI agentic tools
  • collaborative work on shared documents, e.g. pair programming when coding or just some clients working on one document
  • specific auto save function that depends on the content
  • spell checking while typing (as @fischerling already pointed out)
  • real time vizualisations, e.g. rendering of markup/markdown text
  • other vizualisations or update of code / text metrics

Some thoughts on your concerns, Randy. I agree that useless parameters should be removed. However, the purpose of these parameters is to narrow down the area of change as much as possible in order for plugins to react in an efficient manner. Even for the syntax highlighting, there could be some more optimizations, e.g. if we know on insert that only n characters are affected, the lexing could be restraint to this area of change. This is especially possible for real incremental lexers. Currently, the complete text from insertion point until the end of the viewport is lexed, but most of it has already been lexed and is available in the cache. So, for this optimization both, the modification type and len parameters of the event would be required. For example treesitter has functionality to determine the affected nodes in the parse tree given position and length of the changed text, so the parse tree can be modified accordingly with very few changes. So, in scenarios like this, also the type of modification, position and len is required and that's all of the currently proposed parameters (and of course the file, which is modified). The underlying text of the changes then should be accessed through the currently available API. The event just provides the hook when the event happens and where it happens (as precise as possible).

To give an example of how the current events look like and giving an answer to @fischerlings question:

Text substitution, replace word with 4 characters by word with 5 characters in file /tmp/main.c by marking the word in visual mode, substituting using s and typing the five inserted characters:

FILE_MODIFIED /tmp/main.c, DELETE, 39, 4
FILE_MODIFIED /tmp/main.c, INSERT, 39, 1
FILE_MODIFIED /tmp/main.c, INSERT, 40, 1
FILE_MODIFIED /tmp/main.c, INSERT, 41, 1
FILE_MODIFIED /tmp/main.c, INSERT, 42, 1
FILE_MODIFIED /tmp/main.c, INSERT, 43, 1

Pasting 3 characters from a register:

FILE_MODIFIED /tmp/main.c, INSERT, 86, 3

@rnpnr

rnpnr commented Oct 2, 2026

Copy link
Copy Markdown
Collaborator

For example treesitter has functionality to determine the affected nodes in the parse tree given position and length of the changed text, so the parse tree can be modified accordingly with very few changes.

This was exactly my point. The fact that treesitter needs to maintain it's own internal copy of the entire text state is really not ideal. In your example the text state can be compressed because there is some sort of semantic tree structure to the file but that doesn't change the fact that it is just duplicating work.

What I really envision is some scenario where TEXT_MODIFIED better matches what vis is actually doing internally along with some method of accessing the text structure from lua. This seems much more useful then some compressed representation which only highlights a position and length.

I'm not going to stall on merging this waiting for such a thing but I hope it better describes my thinking.

@Nomarian

Nomarian commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

There are 2 ideas here, and it seems the cache is still in the planning/testing stages, can a second PR be done that is just for the file_modified event or would that be too much of a bother?

@jorbakk

jorbakk commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

Yes, I agree that breaking down this PR into different ones is a good idea if it will get accepted, not only the event patch but there was also e.g. a separate bugfix or changes to the big file handling along the lines. However, I'm not sure if the token cache is in question, but the concerns raised where more about the parameters of the modify event and if it should be public api or not. Please correct me if I missed s.th. here ...

@rnpnr

rnpnr commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

However, I'm not sure if the token cache is in question ...

I still run into issues with invalid highlighting when I yank and paste blocks of code. The idea of the cache is not in question, I quite like it and it does solve the issue with markdown files, but I'm still not 100% on the implementation. Also the theme I use is very simple. I worry that if I was using a more elaborate theme like zenburn or solarized the invalid highlighting would be very jarring.

@jorbakk

jorbakk commented Oct 8, 2026

Copy link
Copy Markdown
Contributor Author

That's interesting. I'm using zenburn and didn't encounter any highlighting errors since the last patch. If you manage to reproduce this behavior let me know and I would like to try to track it down.

@fischerling

fischerling commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

This conflicts with f4c08c3.
@jorbakk would you please rebase it onto the current master?

@rnpnr Are you still encountering any weird highlighting?

@rnpnr

rnpnr commented Oct 30, 2026

Copy link
Copy Markdown
Collaborator

@rnpnr Did you still encountering any weird highlighting?

Yes very often. The issue with the markdown slowdown should be mostly resolved with: orbitalquark/scintillua@670597e

@fischerling

Copy link
Copy Markdown
Collaborator

@jorbakk So this should probably be split up to be merged in separately if you have time to do so.

@jorbakk

jorbakk commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Now rebased to current master.

The only thing I noticed meanwhile is: when changing a file from an external process that is also opened in vis and then reloading it in vis with :e, the token cache is not invalidated. This is a small patch that I added now. Otherwise, it seems that I successfully circumvented any issues or syntax highlighting errors in my daily usage, so I still would appreciate hints on how they could be reproduced.

@fischerling, I could make a proposal on how to split this PR up, but I'm still not sure if and how this is going to be merged, so I would need some input here ...

@fischerling

Copy link
Copy Markdown
Collaborator

@fischerling, I could make a proposal on how to split this PR up, but I'm still not sure if and how this is going to be merged, so I would need some input here ...

I think the lua_get_vis fix and the large file updates can be merged seperatly and already improve the status quo.

@jorbakk

jorbakk commented Nov 2, 2026

Copy link
Copy Markdown
Contributor Author

I think the lua_get_vis fix and the large file updates can be merged seperatly and already improve the status quo.
Okay, I pushed two PRs accordingly!

@rnpnr

rnpnr commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

The only thing I noticed meanwhile is: when changing a file from an external process that is also opened in vis and then reloading it in vis with :e, the token cache is not invalidated. This is a small patch that I added now. Otherwise, it seems that I successfully circumvented any issues or syntax highlighting errors in my daily usage, so I still would appreciate hints on how they could be reproduced.

This could be related, but let me test. I have also noticed it with vis-spellcheck. I will try to remember to document what I'm doing next time I run into it.

@fischerling

Copy link
Copy Markdown
Collaborator

The only thing I noticed meanwhile is: when changing a file from an external process that is also opened in vis and then reloading it in vis with :e, the token cache is not invalidated. This is a small patch that I added now. Otherwise, it seems that I successfully circumvented any issues or syntax highlighting errors in my daily usage, so I still would appreciate hints on how they could be reproduced.

This could be related, but let me test. I have also noticed it with vis-spellcheck. I will try to remember to document what I'm doing next time I run into it.

vis-spellcheck definitly has to change if this gets merged.
But it is due to an rewrite anyway, I just did not have the time to do it.
And I am not willing to review or maintain an LLM generated rewrite.

Using a limited horizon for lexing yields to incorrect results in certain
situations. By removing the horizon parameter, files are always lexed
from the beginning to the end of the viewport.
Limit lexing to the parts of a file that have changed only or are viewed for the
first time. Store already lexed tokens in a cache. Make sure that the tokens in
the cache reflect a correctly lexed document.
- Removed UI_OPTION_LARGE_FILE property to avoid additional state that needs to be synced with file changes.
- Introduced one single function that checks for large files and depends only on the text state.
- Made this function available in the lua API.
- Introduced thresholds for file and line size that can be set as ui options per window to adjust the large file check.
- Turn off lexing and syntax highlighting when check for large files is true or no syntax is set on the file.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants