From a7e155de6dd9829f1455ac0888142a071ec3b249 Mon Sep 17 00:00:00 2001 From: Catalin Constantin Mititiuc Date: Fri, 20 Jun 2025 15:09:39 -0700 Subject: [PATCH 1/6] Add rough draft --- posts/2025-06-20-chroma.md | 183 +++++++++++++++++++++++++++++++++++++ 1 file changed, 183 insertions(+) create mode 100644 posts/2025-06-20-chroma.md diff --git a/posts/2025-06-20-chroma.md b/posts/2025-06-20-chroma.md new file mode 100644 index 0000000..ea45cb9 --- /dev/null +++ b/posts/2025-06-20-chroma.md @@ -0,0 +1,183 @@ +{ + blurb: "Add a new lexer to chroma" +} + +## Intro + +Chroma doesn't have a MoonScript lexer. It does has a Python script that can +convert Pygments lexers, though, and Pygments has a MoonScript lexer. + +## Run MoonScript lexer generation script + +To create the lexer, in the Chroma root directory run: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt python bash -c \ +"pip install pystache pygments \ +&& python _tools/pygments2chroma_xml.py \ +pygments.lexers.scripting.MoonScriptLexer > lexers/embedded/moonscript.xml \ +&& pip list" +``` + +## Use the Chroma MoonScript lexer to highlight some code + +Create a file like this: + +::: filename-for-code-block +`main.go` +::: + +```go +package main + +import ( + "fmt" + "os" + + "github.com/alecthomas/chroma/v2/quick" +) + +func main() { + code := `package main + + func main() { } + ` + + fmt.Println(quick.Highlight(os.Stdout, code, "go", "html", "monokai")) +} +``` + +I did one of these: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go mod init main +``` + +Which gave me the `go.mod` file. + +::: filename-for-code-block +`go.mod` +::: + +``` +module main + +go 1.25 + +require ( + github.com/alecthomas/chroma/v2 v2.18.0 // indirect + github.com/dlclark/regexp2 v1.11.5 // indirect +) +``` + +Then I did one of these: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go run main.go +``` + +And that should output markup (and styles) for highlighting that block of Go +code to the console. But if we notice, it's importing the Chroma package from +the GitHub repo. We want to use our local version of chroma, so we use `go mod +edit` to [replace the chroma import with our local version](https://go.dev/ref/mod#go-mod-file-replace): + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go mod edit -replace github.com/alecthomas/chroma/v2@v2.18.0=./chroma +``` + +Which adds this line to our `go.mod` file: + +::: filename-for-code-block +`go.mod` +::: + +``` +... + +replace github.com/alecthomas/chroma/v2 v2.18.0 => ./chroma +``` + +Now we can put some MoonScript in `main.go`. + +```go +code := `print "Hello, #{@name}!"` + +fmt.Println(quick.Highlight(os.Stdout, code, "moonscript", "html", "monokai")) +``` + +And we have it: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go run main.go +go: downloading github.com/dlclark/regexp2 v1.11.5 +``` + +That should output syntax highlighting using our local version of chroma. + +## Create testdata + +Create a file in `lexers/testdata` called `moonscript.actual`. Add the tokens +from the language in this file. + +## Record test output + +Create another file called `lexers/testdata/moonscript.expected`. This is the +file we will record to. + +```console +$ RECORD=true go test ./lexers +``` + +Visually inspect and verify that the expected data is correct. + +## Run tests + +```console +$ go test ./lexers +``` + +## Bonus!: Use local `pygments` with `pygments2chroma_xml.py` + +These lines in `pygments2chroma_xml.py`: + +```python +import pystache +from pygments import lexer as pygments_lexer +from pygments.token import _TokenType +``` + +Import pygments from pip? How do we get it to load a local version of +`pygments`? + +In Pygments root directory: + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt \ +-v ../gitea-syntax-highlight/chroma/_tools/pygments2chroma_xml.py:/opt/pygments2chroma_xml.py \ +python bash -c "pip install pystache && pip list \ +&& python pygments2chroma_xml.py pygments.lexers.scripting.LuaLexer" +``` + +Should see. + +```console +Package Version +-------- ------- +pip 25.0.1 +pystache 0.6.8 +``` + +That shows no remote pygments package is installed. After that you will see the +lexer markup output. + +```console + + +... +``` + +If you wanted to save -- 2.39.5 From a7634067019867a0c805172cc06c2bac1f5bcd3e Mon Sep 17 00:00:00 2001 From: Catalin Constantin Mititiuc Date: Fri, 20 Jun 2025 16:57:53 -0700 Subject: [PATCH 2/6] Add an index to draft post --- posts/2025-06-20-chroma.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/posts/2025-06-20-chroma.md b/posts/2025-06-20-chroma.md index ea45cb9..2da98bc 100644 --- a/posts/2025-06-20-chroma.md +++ b/posts/2025-06-20-chroma.md @@ -1,11 +1,13 @@ { blurb: "Add a new lexer to chroma" } +$index ## Intro -Chroma doesn't have a MoonScript lexer. It does has a Python script that can -convert Pygments lexers, though, and Pygments has a MoonScript lexer. +Gitea uses Chroma for syntax highlighting. Chroma doesn't have a MoonScript +lexer. It does has a Python script that can convert Pygments lexers, though, +and Pygments has a MoonScript lexer. ## Run MoonScript lexer generation script @@ -180,4 +182,3 @@ lexer markup output. ... ``` -If you wanted to save -- 2.39.5 From 4274618525b15b2a5aff40ebc7fb527f0eb1b0f9 Mon Sep 17 00:00:00 2001 From: Catalin Constantin Mititiuc Date: Sat, 21 Jun 2025 13:01:31 -0700 Subject: [PATCH 3/6] Edit post draft --- posts/2025-06-20-chroma.md | 218 ++++++++++++++++++++++++------------- 1 file changed, 145 insertions(+), 73 deletions(-) diff --git a/posts/2025-06-20-chroma.md b/posts/2025-06-20-chroma.md index 2da98bc..14f4bb3 100644 --- a/posts/2025-06-20-chroma.md +++ b/posts/2025-06-20-chroma.md @@ -1,62 +1,74 @@ { - blurb: "Add a new lexer to chroma" + blurb: "Add a new lexer to Chroma" } $index -## Intro +## Introduction -Gitea uses Chroma for syntax highlighting. Chroma doesn't have a MoonScript -lexer. It does has a Python script that can convert Pygments lexers, though, -and Pygments has a MoonScript lexer. +[Gitea](https://github.com/go-gitea/gitea) uses [Chroma](https://github.com/alecthomas/chroma) for syntax highlighting. Chroma is based on the Python +syntax highlighter, [Pygments](https://github.com/pygments/pygments), and includes a [script](https://github.com/alecthomas/chroma/blob/484750a96fc430f49d6b69cc2a2a8b7a67691446/_tools/pygments2chroma_xml.py) to help convert Pygments +lexers for use with Chroma. This post describes that process. -## Run MoonScript lexer generation script +## Convert a Pygments lexer to a Chroma lexer with `pygments2chroma_xml.py` -To create the lexer, in the Chroma root directory run: +In the Chroma root directory, we run: ```console $ docker run --rm -it -w /opt -v $PWD:/opt python bash -c \ -"pip install pystache pygments \ +"pip install pystache pygments && pip list \ && python _tools/pygments2chroma_xml.py \ -pygments.lexers.scripting.MoonScriptLexer > lexers/embedded/moonscript.xml \ -&& pip list" +pygments.lexers.scripting.LuaLexer > lexers/embedded/lua.xml" ``` -## Use the Chroma MoonScript lexer to highlight some code +As output, we should see this in our terminal: -Create a file like this: +``` +Package Version +-------- ------- +pip 25.0.1 +Pygments 2.19.2 +pystache 0.6.8 +``` + +This just helps us know what version of Pygments we generated our lexer from. +The file `lexers/embedded/lua.xml` should now contain all the tokenization +rules for the [Lua](https://www.lua.org) language. ::: filename-for-code-block -`main.go` +`lexers/embedded/lua.xml` ::: -```go -package main - -import ( - "fmt" - "os" - - "github.com/alecthomas/chroma/v2/quick" -) - -func main() { - code := `package main - - func main() { } - ` - - fmt.Println(quick.Highlight(os.Stdout, code, "go", "html", "monokai")) -} +```xml + + + Lua + ... ``` -I did one of these: +## Highlight some code with our new lexer + +Chroma provides a [simple example test file][1] we can modify to see what syntax +highlighting with our new lexer looks like. First, though, we need to create a +new Go module by running `go mod init`: ```console $ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ go mod init main +go: creating new go.mod: module main +go: to add module requirements and sums: + go mod tidy ``` -Which gave me the `go.mod` file. +We will need required modules, so let's go ahead and run `go mod tidy` as the +output suggests. + +```console +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ +go mod tidy +``` + +We should now have 2 additional files, `go.mod` and `go.sum`. `go.sum` has some +package hashes while `go.mod` should look like this: ::: filename-for-code-block `go.mod` @@ -67,23 +79,55 @@ module main go 1.25 -require ( - github.com/alecthomas/chroma/v2 v2.18.0 // indirect - github.com/dlclark/regexp2 v1.11.5 // indirect -) +require github.com/alecthomas/chroma/v2 v2.18.0 + +require github.com/dlclark/regexp2 v1.11.5 // indirect ``` -Then I did one of these: +Now we can create a `main.go` file and copy over the code from Chroma's example +test file, but we update the `code` variable and the lexer we pass into the +`Highlight` function for Lua: + + +::: filename-for-code-block +`main.go` +::: + +```go +package main + +import ( + "log" + "os" + + "github.com/alecthomas/chroma/v2/quick" +) + +func main() { + code := `print("hello")` + + err := quick.Highlight(os.Stdout, code, "lua", "html", "monokai") + if err != nil { + log.Fatal(err) + } +} +``` + +Now we can try running our `main.go` like this: ```console -$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ -go run main.go +$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm go run main.go +go: downloading github.com/alecthomas/chroma/v2 v2.18.0 +go: downloading github.com/dlclark/regexp2 v1.11.5 + +