Repository navigation
Conversation
|
Without the patch, this would just set the horizon to -1 which means
that lexing of the file begins at
viewport.start - (-1) = viewport.start + 1
which is a one byte offset into the current viewport.
With the patch, the exact same command sets the horizon to the beginning
of the file. So, you also need this command to set the horizon with the
patch to activate it. I made it configurable so that you can opt out if
you might have performance issues on very slow machines.
Currently, it's deactivated by default and it follows the old behaviour
for backward compatibility. However, lexing is fast and otherwise syntax
highlighting is simply not correct and I see no other way to fix this
issue other than to parse the file from the beginning. So it might be
reasonable to activate it by default and remove the horizon variable. I
would be happy to send a patch to do so.
…On 2026-05-30 22:53, Joe M wrote:
*4d30* left a comment (martanne/vis#1357)
<#1357 (comment)>
Cannot the horizon command do this?
|:set horizon -1 |
—
Reply to this email directly, view it on GitHub
<#1357?email_source=notifications&email_token=AWXEBLIV3AKRBADWW7ELNLD45NC5HA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTINJYGQ3TENBXGA22M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-4584724705>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AWXEBLO4SZBPGXMCKNYPACL45NC5HAVCNFSM6AAAAACZQQWGF6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHM2DKOBUG4ZDINZQGU>.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
|
This next set of patches provides incremental lexing of a file while introducing a cache for already lexed tokens. This makes the process of lexing the text into tokens for syntax highlighting much more efficient. In order to achieve this, an event is needed that signals changes in the text, in particular where and when these changes occur. Then only the affected lexer tokens need an update, so the lexer can start from the position of the change until the end position of the current viewport. This is the minimal possible change of the token cache that also preserves as much tokens as possible and maintains a correct list of tokens that reflect the document's syntax. Any newly created tokens should only depend on the position of changes in the document and the position of the viewport. They should not depend on a fixed sized horizon, as this leads to incorrect results and should not be required in any text editor. In addition to a correct token list, this approach also gives a performance boost, e.g. when scrolling down through a freshly opened document. Then only the new viewport has to be lexed. After hitting the bottom of the text, the whole document is in the cache on no lexing is required when moving through it, until the document is changed. If this happens, only the tokens from the cursor, where the changes take place, to the end of the viewport need to be lexed, again. Each window maintains its own token cache, so a different syntax can be set for the same document in different windows. |
|
I don't understand any of this but localize your variables |
Yep, missed a few, hope I got them all now. |
|
run luacheck on it |
Okay, should be clean now. Thanks! |
| local i1 = #tokens | ||
| if pos < tokens[i0]-1 then return i0 end | ||
| if pos > tokens[i1]-1 then return i1 end | ||
| repeat |
There was a problem hiding this comment.
put local i repeat here and remove lines 67-70, repeat runs at least once so local i can be unnassigned
There was a problem hiding this comment.
Great twist, much better!
|
Hey thanks for your work! I will test it and will report back. Especially, the modification event looks really promising to me. I had this kind of event on the to-do list for vis-lspc for a long time. |
Thanks! I can imagine that the modification event would be useful for interactive use of an LSP client, like auto-complete. It actually sounds like a basic event to be provided by an editor. Looking forward to your feedback! |
|
Now that this was at the top of my inbox again I took a quick look. I don't remember why I didn't comment on the original but this is looking quite good. One thing I think should be changed is that we should disable lexing for "large" files. I don't see any reasonable way of handling them without doing partial lexing and I don't want your cache to blow up in size just because someone opened a "large" file. Right now a file being large or not seems to be determined in lua but it's a property that's set during the creation of the window in C. I think it should just be exported in Also if you can leave a comment documenting how the cache is actually organized that would be helpful. I have a plugin which maintains a cache and I know that vis-spellcheck also has a token cache and probably we would both prefer to just access this one directly (even if we aren't explicitly making it public API). EDIT: What I mean by the last part is that both these plugins are trying to avoid re-lexing the file over and over. They want access to the token list directly. I suppose an explicit API might be better but even just access to this cache would be sufficient. |
|
I don't know the C side but from what I understand, Disabling syntax highlighting is just done by setting win.syntax = nil, statusbar event already sets a win.large boolean, it could be a one line change to just add a win.syntax = nil to the conditional, the other PR already disables syntax highlighting for the window as well via vis.command"set syntax off"... But I do think a cache would be nice, maybe both must be used, win.huge files and win.large, with huge files disabling many facilities, like the syntax highlighting and etc and large turning on the cache mode. |
|
@rnpnr: I agree that lexing and syntax highlighting should be turned off on large files, so I had a closer look to the current way large files are handled:
I think that "large file" should be a property/function of the file, without storing it as additional state that needs to be updated. So I made the following changes:
I also added documentation for the token cache. @Nomarian: do you think there should be two thresholds for large and huge files? For now, I think one adjustable threshold should suffice and is simpler. Keep in mind that the token cache should be enabled whenever syntax highlighting is enabled, for performance reasons. |
|
One issue I have found is that when you yank some text from another window and then paste it the cache after the pasted text is not invalidated. For example open 2 of the c files in vis and yank a comment from one and paste it above some other code in the other. |
If your cache system works then there should be no need for it, I was merely stating this because its a quick addition and fix on the nosyntaxhighlighting PR and it seemed to be what Randy wanted. |
Okay. I'll need to take a closer look at your PR, but I would guess that this patch here only reads win.syntax and your patch sets it. Let me know if there's any potential conflict, and I'm sure we can fix it. I also replied the following (with little edits) to your other comment (via email), but it bounced, the github notification bot seems to be busy:
|
@rnpnr : Sorry, but I have a hard time to reproduce this ...?! I could not manage to corrupt the token cache by yanking and pasting any code snippet from one window to another ... . |
My apologies, please disregard. I was testing on a different system from usual on top of my personal work branch and I didn't realize that it must have been out of date. |
|
I do have a similar example which is still an issue:
Note that in order to reproduce the file needs to have more than one token. I you open a new file |
@rnpnr Great that you spotted this! I should have stumbled across it also, but the devil's in the details ... So here's a small patch with a long answer. I'd like to start with your example and the corresponding lexer outputs: File contents: lexer output: File contents: lexer output: File contents: lexer output with The hash sign (now identified as 'default') is ignored and only the string 'include' and the following whitespace are re-lexed. This is why it's not identified as a preprocessor token. lexer output with The three tokens '#' = 'default', 'include' = 'identifier', and whitespace = 'whitespace.c' are re-lexed and correctly identified as 'preprocessor' and 'whitespace.c'. Note that the incremental lexing performed here always starts at token boundaries, currently two tokens back from the modification position ( edit: some small changes |
|
While testing it as my daily driver, I found another edge case which I need to take care of. Please bear with me ... |
|
With the latest patch, I use the same heuristic as Please note that lpeg is not intended to be a fully incremental lexer, so syntax highlighting while modifying a document without lexing it from the beginning requires heuristics and won't be 100% perfect. In order to achieve this, something like An example for incorrect highlighting is when you type this into a C file: foo();
bar
();Then, while typing, However, the current approach should yield the same results as |
|
The latest changes fix the issue I was running into as well but wasn't able to write up yet. I will continue with more testing but I think this is getting to be in pretty good shape. |
| local lexer = vis.lexers.load(win.syntax, nil, true) | ||
| if not lexer then return {} end | ||
| if win.token_cache == nil then return {} end | ||
| if win.token_cache == nil or start == nil or finish == nil then return {} end |
There was a problem hiding this comment.
You could do this here
if -- check nothing is falsy
not( lexer and win.token_cache and start and finish )
then
return {}
end
But its harder to read so 😮💨
At the very least, the if above can be incorporated into this one
There was a problem hiding this comment.
Yes, at least merge the two statements into one, either negated or not ... but I guess this is more concise as it doesn't need nil for every boolean expression. I'll do that but will wait a little for other changes to accumulate ... . Thanks for the hint!
rnpnr
left a comment
There was a problem hiding this comment.
I have some local changes pending that will conflict with this so I would like to merge this first. Can you please address the comment I have below and the other one that was left.
I can't really comment on the TEXT_MODIFIED event. It looks reasonable to me but I can't really think of any use for it other than what you are using it for here (which doesn't need most of the arguments). I worry that it is not as useful as one might hope. From a plugin's perspective how useful is it to know where a modification happened? Besides the most simple cases making use of that information would require the plugin maintain its own representation of the entire text state.
@fischerling maybe you can comment on if this TEXT_MODIFIED event covers the uses you had in mind?
If its not usable as it is I would suggest documenting it as used internally but experimental and subject to change (and removing the argument documentation).
| bool view_large_file(View *view) { | ||
| Selection *sel = view_selections_primary_get(view); | ||
| size_t col = view_cursors_col(sel); | ||
| return text_size(view->text) > view->large_file_size << 10 /* in KiB */ || |
There was a problem hiding this comment.
Instead can you introduce some macros in util.h:
#define KiB(a) ((u64)(a) << 10ULL)
#define MiB(a) ((u64)(a) << 20ULL)
#define GiB(a) ((u64)(a) << 30ULL)Then you don't need the comment. I'm also worried about the current version overflowing so a cast to u64 is needed regardless.
| int wrapcolumn; /* wrap lines at minimum of window width and wrapcolumn (if != 0) */ | ||
| int wrapcol; /* used while drawing view content, column where word wrap might happen */ | ||
| bool prevch_breakat; /* used while drawing view content, previous char is part of breakat */ | ||
| int large_file_size; /* threshold for large files in KiB */ |
There was a problem hiding this comment.
| int large_file_size; /* threshold for large files in KiB */ | |
| int large_file_size_kib; /* threshold for large files in KiB */ |
This way it is clear wherever it is used that it is in KiB.
|
Thanks to @rnpnr and @fischerling for having a closer look and your feedback! The following patch includes the changes proposed by @Nomarian and @rnpnr. Regarding the modification event, generally speaking, any plugin that analyses the code / text or triggers a function based on the typed text while typing could make use of it. Additionally to syntax highlighting, the following ideas of use cases might be relevant:
Some thoughts on your concerns, Randy. I agree that useless parameters should be removed. However, the purpose of these parameters is to narrow down the area of change as much as possible in order for plugins to react in an efficient manner. Even for the syntax highlighting, there could be some more optimizations, e.g. if we know on insert that only n characters are affected, the lexing could be restraint to this area of change. This is especially possible for real incremental lexers. Currently, the complete text from insertion point until the end of the viewport is lexed, but most of it has already been lexed and is available in the cache. So, for this optimization both, the modification type and len parameters of the event would be required. For example treesitter has functionality to determine the affected nodes in the parse tree given position and length of the changed text, so the parse tree can be modified accordingly with very few changes. So, in scenarios like this, also the type of modification, position and len is required and that's all of the currently proposed parameters (and of course the file, which is modified). The underlying text of the changes then should be accessed through the currently available API. The event just provides the hook when the event happens and where it happens (as precise as possible). To give an example of how the current events look like and giving an answer to @fischerlings question: Text substitution, replace word with 4 characters by word with 5 characters in file /tmp/main.c by marking the word in visual mode, substituting using Pasting 3 characters from a register: |
This was exactly my point. The fact that treesitter needs to maintain it's own internal copy of the entire text state is really not ideal. In your example the text state can be compressed because there is some sort of semantic tree structure to the file but that doesn't change the fact that it is just duplicating work. What I really envision is some scenario where I'm not going to stall on merging this waiting for such a thing but I hope it better describes my thinking. |
|
There are 2 ideas here, and it seems the cache is still in the planning/testing stages, can a second PR be done that is just for the file_modified event or would that be too much of a bother? |
|
Yes, I agree that breaking down this PR into different ones is a good idea if it will get accepted, not only the event patch but there was also e.g. a separate bugfix or changes to the big file handling along the lines. However, I'm not sure if the token cache is in question, but the concerns raised where more about the parameters of the modify event and if it should be public api or not. Please correct me if I missed s.th. here ... |
I still run into issues with invalid highlighting when I yank and paste blocks of code. The idea of the cache is not in question, I quite like it and it does solve the issue with markdown files, but I'm still not 100% on the implementation. Also the theme I use is very simple. I worry that if I was using a more elaborate theme like zenburn or solarized the invalid highlighting would be very jarring. |
|
That's interesting. I'm using zenburn and didn't encounter any highlighting errors since the last patch. If you manage to reproduce this behavior let me know and I would like to try to track it down. |
Yes very often. The issue with the markdown slowdown should be mostly resolved with: orbitalquark/scintillua@670597e |
|
@jorbakk So this should probably be split up to be merged in separately if you have time to do so. |
|
Now rebased to current master. The only thing I noticed meanwhile is: when changing a file from an external process that is also opened in vis and then reloading it in vis with @fischerling, I could make a proposal on how to split this PR up, but I'm still not sure if and how this is going to be merged, so I would need some input here ... |
I think the |
|
This could be related, but let me test. I have also noticed it with vis-spellcheck. I will try to remember to document what I'm doing next time I run into it. |
|
Using a limited horizon for lexing yields to incorrect results in certain situations. By removing the horizon parameter, files are always lexed from the beginning to the end of the viewport.
Limit lexing to the parts of a file that have changed only or are viewed for the first time. Store already lexed tokens in a cache. Make sure that the tokens in the cache reflect a correctly lexed document.
- Removed UI_OPTION_LARGE_FILE property to avoid additional state that needs to be synced with file changes. - Introduced one single function that checks for large files and depends only on the text state. - Made this function available in the lua API. - Introduced thresholds for file and line size that can be set as ui options per window to adjust the large file check. - Turn off lexing and syntax highlighting when check for large files is true or no syntax is set on the file.
This patch fixes issue #1253
Problem description
If lexer starts parsing at a file offset determined by the
horizonvariable and this offset happens to be within a string, then all non-strings are highlighted as strings and vice versa.This occurs with strings that have identical characters for their opening and closing delimiters, e.g. single quotes
', double quotes"or quotes like""". Paired delimiters that use different types of characters for start and end of the string are not affected. Most programming languages use identical characters as string delimiters. Exceptions are:`at the start and apostrophe'at the end{}The current approach also fails when the
horizonlies completely within a very big token. This is for example the case with big comments, when the starting delimiter of the comment is before the horizon. Then the lexer treats the comment as un-commented program text.The lexer currently fails quite frequently on files with:
Patch description
With this patch, if the horizon variable is set to -1, the whole file buffer up to the end of the viewport is run through the lexer. Performance is still reasonably good, as no syntax highlighting takes place outside of the viewport.
However, with this simple patch, performance degrades with big files when the viewport is close to the end of the file. A set of patches will follow shortly that overcome this limitation while also yielding the same correct results as this simple patch. It also achieves generally a big performance boost in almost all cases compared to the current
horizonapproach.Edit 2026-09-06: more explicit and accurate problem description, including examples