ccm/miti.sh
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Publish post 'Add a Pygments Lexer to Chroma' #2

Merged
ccm merged 6 commits from ccm-chroma-post into trunk 2025-06-22 16:56:46 +00:00
Conversation 0 Commits 6 Files Changed 2 +60 -42
1 changed files with 60 additions and 42 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit ecfaf4edac - Show all commits

102
posts/2025-06-20-chroma.md
View File

@ -7,20 +7,47 @@ $index
[Gitea](https://github.com/go-gitea/gitea) uses [Chroma](https://github.com/alecthomas/chroma) for syntax highlighting. Chroma is based on the Python [Gitea](https://github.com/go-gitea/gitea) uses [Chroma](https://github.com/alecthomas/chroma) for syntax highlighting. Chroma is based on the Python
syntax highlighter, [Pygments](https://github.com/pygments/pygments), and includes a [script](https://github.com/alecthomas/chroma/blob/484750a96fc430f49d6b69cc2a2a8b7a67691446/_tools/pygments2chroma_xml.py) to help convert Pygments syntax highlighter, [Pygments](https://github.com/pygments/pygments), and includes a [script](https://github.com/alecthomas/chroma/blob/484750a96fc430f49d6b69cc2a2a8b7a67691446/_tools/pygments2chroma_xml.py) to help convert Pygments
lexers for use with Chroma. This post describes that process. lexers for use with Chroma. We describe how below.
## Setup
We're going to be using the `python` and `golang` [Docker][4] images. Docker
Desktop is _not_ required.
```console
$ docker pull python
$ docker pull golang
```
Let's set up some aliases to make running the commands easier.
```console
$ alias docker-run='docker run --rm -it -w /opt -v $PWD:/opt'
$ alias docker-run-go='docker-run golang'
$ alias docker-run-py='docker-run python'
```
[3]: https://docs.docker.com/engine/install/linux-postinstall/#manage-docker-as-a-non-root-user
[4]: https://docs.docker.com/engine/
## Convert a Pygments lexer to a Chroma lexer with `pygments2chroma_xml.py` ## Convert a Pygments lexer to a Chroma lexer with `pygments2chroma_xml.py`
```console
$ git clone https://github.com/alecthomas/chroma.git
$ cd chroma
```
In the Chroma root directory, we run: In the Chroma root directory, we run:
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt python bash -c \ $ docker-run-py bash -c \
"pip install pystache pygments && pip list \ "pip install pystache pygments && \
&& python _tools/pygments2chroma_xml.py \ python _tools/pygments2chroma_xml.py \
pygments.lexers.scripting.LuaLexer > lexers/embedded/lua.xml" pygments.lexers.scripting.LuaLexer > lexers/embedded/lua.xml && \
pip list"
``` ```
As output, we should see this in our terminal: We should see this in the output:
``` ```
Package Version Package Version
@ -45,15 +72,15 @@ rules for the [Lua](https://www.lua.org) language.
... ...
``` ```
## Highlight some code with our new lexer ## Highlight some code with a Chroma lexer
Chroma provides a [simple example test file][1] we can modify to see what syntax Chroma provides a [simple example test file][1] we can modify to see what syntax
highlighting with our new lexer looks like. First, though, we need to create a highlighting with our new lexer looks like. First, though, we need to create a
new Go module by running `go mod init`: new Go module by running `go mod init`:
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ $ cd ..
go mod init main $ docker-run-go go mod init main
go: creating new go.mod: module main go: creating new go.mod: module main
go: to add module requirements and sums: go: to add module requirements and sums:
go mod tidy go mod tidy
@ -63,8 +90,7 @@ We will need required modules, so let's go ahead and run `go mod tidy` as the
output suggests. output suggests.
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ $ docker-run-go go mod tidy
go mod tidy
``` ```
We should now have 2 additional files, `go.mod` and `go.sum`. `go.sum` has some We should now have 2 additional files, `go.mod` and `go.sum`. `go.sum` has some
@ -85,9 +111,8 @@ require github.com/dlclark/regexp2 v1.11.5 // indirect
``` ```
Now we can create a `main.go` file and copy over the code from Chroma's example Now we can create a `main.go` file and copy over the code from Chroma's example
test file, but we update the `code` variable and the lexer we pass into the test file, but we update the `code` variable with some Lua, `print("hello")`,
`Highlight` function for Lua: and the lexer we pass into the `Highlight` function is changed to `"lua"`:
::: filename-for-code-block ::: filename-for-code-block
`main.go` `main.go`
@ -116,7 +141,7 @@ func main() {
Now we can try running our `main.go` like this: Now we can try running our `main.go` like this:
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm go run main.go $ docker-run-go go run main.go
go: downloading github.com/alecthomas/chroma/v2 v2.18.0 go: downloading github.com/alecthomas/chroma/v2 v2.18.0
go: downloading github.com/dlclark/regexp2 v1.11.5 go: downloading github.com/dlclark/regexp2 v1.11.5
<html> <html>
@ -130,8 +155,8 @@ the GitHub repo. If we want to use a local version of Chroma, we have to use a
[`replace` directive][2] to import Chroma from our local directory: [`replace` directive][2] to import Chroma from our local directory:
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ $ docker-run-go go mod edit -replace \
go mod edit -replace github.com/alecthomas/chroma/v2@v2.18.0=./chroma github.com/alecthomas/chroma/v2@v2.18.0=./chroma
``` ```
Which adds this line to our `go.mod` file: Which adds this line to our `go.mod` file:
@ -150,7 +175,7 @@ Now, when we run `main.go`, we should no longer see Chroma being imported,
because it's using our local copy: because it's using our local copy:
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm go run main.go $ docker-run-go go run main.go
go: downloading github.com/dlclark/regexp2 v1.11.5 go: downloading github.com/dlclark/regexp2 v1.11.5
<html> <html>
<style type="text/css"> <style type="text/css">
@ -191,8 +216,7 @@ another file called `lexers/testdata/lua.expected`. This is the file we
will record to by running the following command from the Chroma root directory: will record to by running the following command from the Chroma root directory:
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt -e RECORD=true golang:tip-bookworm \ $ docker-run -e RECORD=true golang go test ./lexers
go test ./lexers
``` ```
Once test output is recorded in `lexers/testdata/lua.expected`, we should Once test output is recorded in `lexers/testdata/lua.expected`, we should
@ -204,8 +228,7 @@ As a final confirmation, we can run the tests to make sure we have not broken
anything: anything:
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt golang:tip-bookworm \ $ docker-run-go go test ./lexers
go test ./lexers
``` ```
## Conclusion ## Conclusion
@ -223,20 +246,23 @@ from pygments import lexer as pygments_lexer
from pygments.token import _TokenType from pygments.token import _TokenType
``` ```
import Pygments from the [Python Package Index](https://pypi.org/). But, if we are working on a import Pygments from the [Python Package Index](https://pypi.org/). But, if we
Pygments lexer locally, we might want to convert it to a Chroma lexer for want to convert a Pygments lexer from a local `git` repo, we can import it
testing. We can import a local version of Pygments when running by simply running the `pygments2chroma_xml.py` script from the repo root
`pygments2chroma_xml.py` by running the following from the Pygments root directory.
directory:
```console ```console
$ docker run --rm -it -w /opt -v $PWD:/opt \ $ git clone https://github.com/pygments/pygments.git
-v path/to/chroma/_tools/pygments2chroma_xml.py:/opt/pygments2chroma_xml.py \ $ cd pygments
python bash -c "pip install pystache && pip list \ $ docker-run \
&& python pygments2chroma_xml.py pygments.lexers.scripting.LuaLexer" -v ../chroma/_tools/pygments2chroma_xml.py:/opt/pygments2chroma_xml.py \
``` python bash -c \
"pip install pystache && \
python pygments2chroma_xml.py pygments.lexers.scripting.LuaLexer && \
pip list"
We should see ```
We should see the lexer output followed by
```console ```console
Package Version Package Version
@ -245,12 +271,4 @@ pip 25.0.1
pystache 0.6.8 pystache 0.6.8
``` ```
which indicates no remote Pygments package is installed. Following that, we which indicates no remote `pygments` package was installed.
should also see the lexer markup output.
```console
<lexer>
<config>
...
```