From a7e155de6dd9829f1455ac0888142a071ec3b249 Mon Sep 17 00:00:00 2001 From: Catalin Constantin Mititiuc Date: Fri, 20 Jun 2025 15:09:39 -0700 Subject: [PATCH] Add rough draft --- posts/2025-06-20-chroma.md | 183 +++++++++++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 posts/2025-06-20-chroma.md diff --git a/posts/2025-06-20-chroma.md b/posts/2025-06-20-chroma.md new file mode 100644 index 0000000..ea45cb9 --- /dev/null +++ b/posts/2025-06-20-chroma.md @@ -0,0 +1,183 @@ +{ + blurb: "Add a new lexer to chroma" +} + +## Intro + +Chroma doesn't have a MoonScript lexer. It does has a Python script that can +convert Pygments lexers, though, and Pygments has a MoonScript lexer. + +## Run MoonScript lexer generation script + +To create the lexer, in the Chroma root directory run: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt python bash -c \ +"pip install pystache pygments \ +&& python _tools/pygments2chroma_xml.py \ +pygments.lexers.scripting.MoonScriptLexer > lexers/embedded/moonscript.xml \ +&& pip list" +``` + +## Use the Chroma MoonScript lexer to highlight some code + +Create a file like this: + +::: filename-for-code-block +`main.go` +::: + +```go +package main + +import ( + "fmt" + "os" + + "github.com/alecthomas/chroma/v2/quick" +) + +func main() { + code := `package main + + func main() { } + ` + + fmt.Println(quick.Highlight(os.Stdout, code, "go", "html", "monokai")) +} +``` + +I did one of these: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go mod init main +``` + +Which gave me the `go.mod` file. + +::: filename-for-code-block +`go.mod` +::: + +``` +module main + +go 1.25 + +require ( + github.com/alecthomas/chroma/v2 v2.18.0 // indirect + github.com/dlclark/regexp2 v1.11.5 // indirect +) +``` + +Then I did one of these: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go run main.go +``` + +And that should output markup (and styles) for highlighting that block of Go +code to the console. But if we notice, it's importing the Chroma package from +the GitHub repo. We want to use our local version of chroma, so we use `go mod +edit` to [replace the chroma import with our local version](https://go.dev/ref/mod#go-mod-file-replace): + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go mod edit -replace github.com/alecthomas/chroma/v2@v2.18.0=./chroma +``` + +Which adds this line to our `go.mod` file: + +::: filename-for-code-block +`go.mod` +::: + +``` +... + +replace github.com/alecthomas/chroma/v2 v2.18.0 => ./chroma +``` + +Now we can put some MoonScript in `main.go`. + +```go +code := `print "Hello, #{@name}!"` + +fmt.Println(quick.Highlight(os.Stdout, code, "moonscript", "html", "monokai")) +``` + +And we have it: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go run main.go +go: downloading github.com/dlclark/regexp2 v1.11.5 +``` + +That should output syntax highlighting using our local version of chroma. + +## Create testdata + +Create a file in `lexers/testdata` called `moonscript.actual`. Add the tokens +from the language in this file. + +## Record test output + +Create another file called `lexers/testdata/moonscript.expected`. This is the +file we will record to. + +```console +$ RECORD=true go test ./lexers +``` + +Visually inspect and verify that the expected data is correct. + +## Run tests + +```console +$ go test ./lexers +``` + +## Bonus!: Use local `pygments` with `pygments2chroma_xml.py` + +These lines in `pygments2chroma_xml.py`: + +```python +import pystache +from pygments import lexer as pygments_lexer +from pygments.token import _TokenType +``` + +Import pygments from pip? How do we get it to load a local version of +`pygments`? + +In Pygments root directory: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt \ +-v ../gitea-syntax-highlight/chroma/_tools/pygments2chroma_xml.py:/opt/pygments2chroma_xml.py \ +python bash -c "pip install pystache && pip list \ +&& python pygments2chroma_xml.py pygments.lexers.scripting.LuaLexer" +``` + +Should see. + +```console +Package Version +-------- ------- +pip 25.0.1 +pystache 0.6.8 +``` + +That shows no remote pygments package is installed. After that you will see the +lexer markup output. + +```console + + +... +``` + +If you wanted to save