Compare commits

...

18 Commits

Author SHA1 Message Date
c800c8e775 Remove bonus content 2025-06-30 15:40:56 -07:00
3f494b037d Update blurb 2025-06-30 15:31:17 -07:00
b165c2112e Update post date 2025-06-30 15:25:51 -07:00
f9ef223cf1 Edit post 2025-06-30 15:25:14 -07:00
5d7896e9e5 Update post 2025-06-29 20:28:07 -07:00
0cc8f18302 Update post 2025-06-27 10:57:09 -07:00
be52884226 Update post 2025-06-26 18:14:32 -07:00
3cb0b83e8a Update image in post 2025-06-26 13:41:35 -07:00
ca15a32429 Edit post 2025-06-26 13:20:01 -07:00
82f4e9c92e Assert /posts/index.html exists before request in nginx spec 2025-06-26 13:19:15 -07:00
eb896676cc Add a renderer spec for escape and undescape double dollar signs ($$name) 2025-06-26 13:15:42 -07:00
766c9bad62 Use '_' instead of '.' for numbering cosmo escapes
because it gets syntax highlighted weird because '.' is not valid for
like a variable name so the syntax highlighter is splitting the escape
phrase at the '.' which means that phrase won't match when unescaped so
it fails to get unescaped

for example, this would fail:

```
 $$ct
```
2025-06-25 21:10:30 -07:00
c26453415d Clean up last spec 2025-06-25 15:49:54 -07:00
dce537b50c Delete duplicate post 2025-06-25 15:25:08 -07:00
b7e6a93bf9 Add post draft 2025-06-25 14:51:12 -07:00
95b3589d22 Add markdown file 2025-06-25 14:51:12 -07:00
4ad479f8f9 Update commands in README.md 2025-06-25 14:47:23 -07:00
0e848c5e7b Change cosmo '$' escape string in markdown renderer
so that we can put 2 selectors next to each other and still be able to
tell where one ends and the other one begins
2025-06-25 14:44:32 -07:00
10 changed files with 657 additions and 9 deletions

View File

@@ -56,9 +56,13 @@
$ make build
### build a single file
$ make build file=index.html
### start dev server
$ make
$ make serve
Visit `localhost:8080` in web browser

1
html/.gitignore vendored
View File

@@ -15,4 +15,5 @@ posts/resize-a-qemu-disk-image.html
posts/set-up-a-gitweb-server.html
posts/start-erlangs-dialyzer-with-gui-from-a-docker-container.html
posts/test-mix-task-file-modify.html
posts/test-nginx-conf-directives.html
pygments.css

Binary file not shown.

After

Width:  |  Height:  |  Size: 49 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 69 KiB

View File

@@ -0,0 +1,566 @@
{
title: "Test nginx Configuration Directives"
blurb: "We use MoonScript and some Lua packages to write tests for the
directives in our `nginx` configuration files."
}
$index
## Introduction
[`nginx`](https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#rewrite-uris-in-requests)
configuration can contain any number of important directives (redirects and
rewrites, for example) that need to be verified for correctness. We can write
tests for directives and run them against a test server to ensure they are
correct.
To do this, we'll use...
- [MoonScript](https://moonscript.org) and (by extension) [Lua](https://www.lua.org/) programming languages
- `nginx` we'll get from [OpenResty](https://openresty.org/en/), a web platform
created by Chinese developer, [Yichun Zhang](https://agentzh.org/)
- the [Busted testing framework](https://lunarmodules.github.io/busted/)
- the Lua package manager, [LuaRocks](https://luarocks.org/)
- a fantastic little library, [`luajit-curl`](https://bitbucket.org/senanetworksinc/luajit-curl/src/master/),
from Japanese developer [SENA Networks, Inc](https://www.sena-networks.co.jp)
- another great library, written by volunteers, [LuaSocket](https://github.com/lunarmodules/luasocket)
- our favorite container manager, [Docker Engine](https://docs.docker.com/engine/)
## Setup
Since we require LuaRocks, we'll use a Buildpack tag, which comes with it
already installed.
```console
$ docker pull openresty/openresty:bookworm-buildpack
```
Start a server on `localhost`:
```console
$ docker run --rm -it -p 80:80 openresty/openresty:bookworm-buildpack
```
We can visit `localhost` in our browser and we should see the OpenResty splash
page.
![OpenResty default nginx index page](/images/openresty-default-index-page.png)
## Get `nginx` running
First, let's [prepare the directory layout](https://openresty.org/en/getting-started.html#prepare-directory-layout).
```console
$ mkdir -p logs/ conf/conf.d/ html/
```
Next, we copy over [the default `nginx` config file](https://github.com/openresty/docker-openresty?tab=readme-ov-file#nginx-config-files).
```console
$ docker run --rm -it -w /opt -v $PWD:/opt openresty/openresty:bookworm-buildpack \
cp /etc/nginx/conf.d/default.conf /opt/conf.d/
```
Then, we update the root directive in `default.conf`:
::: filename-for-code-block
`conf/conf.d/default.conf`
:::
```diff
location / {
- root /usr/local/openresty/nginx/html;
+ root /var/www;
index index.html index.htm;
```
Now, let's add an index file.
::: filename-for-code-block
`html/index.html`
:::
```html
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<title></title>
</head>
<body>
hello world!
</body>
</html>
```
Last, we start `nginx`:
```console
$ docker run --rm -it -p 80:80 \
-v $PWD/conf/conf.d:/etc/nginx/conf.d -v $PWD/html:/var/www \
openresty/openresty:bookworm-buildpack
```
Then, in another console, this should output our index file.
```console
$ curl -v localhost
* Trying 127.0.0.1:80...
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: openresty/1.27.1.2
< ...
<
<!DOCTYPE html>
<html lang="en">
...
<body>
hello world!
</body>
</html>
```
## Test an HTTP request
If we want to write a test for that request, we need some packages from
LuaRocks. Let's add a `Dockerfile` to build an image with those packages
installed.
### Add a `Dockerfile`
```Dockerfile
FROM openresty/openresty:bookworm-buildpack
WORKDIR /opt/app
RUN luarocks install moonscript
RUN luarocks install busted
RUN luarocks install luajit-curl
RUN luarocks install luasocket
```
Now let's build our image:
```console
$ docker build -t test-nginx .
```
### Write the test
Let's first make a new directory where our tests will live.
```console
$ mkdir spec
```
Our test makes a cURL request against our test server:
::: filename-for-code-block
`spec/nginx_spec.moon`
:::
```moonscript
http = require "luajit-curl-helper.http"
req = (url) ->
request = http.init url
st = request\perform!
error request\lastError! if not st
request
describe "http://localhost", ->
it "sends /index.html", ->
request = req "http://localhost"
assert.same request\statusCode!, 200
assert.same request\statusMessage!, "OK"
assert.same request\body!\match("<body>%s+(.-)%s+</body>"), "hello world!"
```
### Run the test suite
Start the test server. We're going to use `text-nginx`, the image we just
built.
```console
$ ct=$(docker run --rm -d \
-v $PWD/conf/conf.d:/etc/nginx/conf.d \
-v $PWD/html:/var/www \
-v $PWD:/opt/app \
test-nginx)
```
Start the test run:
```console
$ docker exec -t $ct busted
1 success / 0 failures / 0 errors / 0 pending : 0.008246 seconds
```
Stop the test server.
```console
$ docker exec $ct openresty -s stop
```
## Create a `Makefile`
We now have a number of long `docker` commands, let's create a `Makefile`
to make running them easier.
::: filename-for-code-block
`Makefile`
:::
```Makefile
image = test-nginx
image-build:
docker build -t $(image) .
image-rm:
docker image rm $(image)
test:
@ct=$(shell docker run --rm -d \
-v $(PWD)/conf/conf.d:/etc/nginx/conf.d \
-v $(PWD)/html:/var/www \
-v $(PWD):/opt/app \
$(image)); \
docker exec -t $$ct busted; \
docker exec $$ct openresty -s stop
```
Now we can run the test suite with the command `make test`.
## Configure the domain name
Instead of `localhost` we'd like to use an actual domain name. We can do this
with the `--add-host` option. But before we do that, we want to make sure our
container does not have access to the internet, otherwise we might
unintentionally get a response from a domain's server on the internet rather
than from our test server.
### Ensure the test container is offline
We need to create a network that has no external access.
```console
$ docker network create --internal no-internet
```
Now we need to update our `Makefile` to add the test container to our
internal-only network:
```diff
test:
@ct=$(shell docker run --rm -d \
-v $(PWD)/conf/conf.d:/etc/nginx/conf.d \
-v $(PWD)/html:/var/www \
-v $(PWD):/opt/app \
+ --network no-internet \
$(image)); \
```
And now let's add a test in `spec/nginx_spec.moon` to make sure our test
environment is offline:
```moonscript
describe "test environment", ->
it "can't connect to the internet", ->
assert.has_error (-> req "http://example.org"),
"Couldn't resolve host name"
```
Let's run our tests:
```console
$ make test
●●
2 successes / 0 failures / 0 errors / 0 pending : 0.020207 seconds
```
### Replace `localhost` with a custom domain
To use a custom domain name instead of `localhost`, we will need to use the
`--add-host` option for the `docker run` command. Again, we edit `Makefile`:
```diff
test:
@ct=$(shell docker run --rm -d \
-v $(PWD)/conf/conf.d:/etc/nginx/conf.d \
-v $(PWD)/html:/var/www \
-v $(PWD):/opt/app \
--network no-internet \
+ --add-host=domain.abc=127.0.0.1 \
$(image)); \
```
Let's update our test to use the custom domain name:
::: filename-for-code-block
`spec/nginx_spec.moon`
:::
```diff
-describe "http://localhost", ->
+describe "http://domain.abc", ->
it "sends /index.html", ->
- request = req "http://localhost"
+ request = req "http://domain.abc"
assert.same request\statusCode!, 200
```
Verify our tests still pass.
```console
$ make test
●●
2 successes / 0 failures / 0 errors / 0 pending : 0.0224 seconds
```
## Test an HTTP redirect
We want our server to redirect all `http` requests to `https`.
### Write the test
Let's practice a bit of test-driven development and write our test first.
```moonscript
describe "http://domain.abc", ->
it "redirects to https", ->
request = req "http://domain.abc"
assert.same request\statusCode!, 301
assert.same request\statusMessage!, "Moved Permanently"
assert.same request\header!.Location, "https://domain.abc/"
```
We should now have one failing test.
```console
$ make test
●●◼
2 successes / 1 failure / 0 errors / 0 pending : 0.010449 seconds
Failure → .../luajit/lib/luarocks/rocks-5.1/busted/2.2.0-1/bin/busted @ 3
http://domain.abc redirects to https
spec/nginx_spec.moon:24: Expected objects to be the same.
Passed in:
(number) 301
Expected:
(number) 200
```
### Configure `nginx`
We're going to add the redirect directives, as well as a server name for our
domain and the directives for the SSL certificates we will generate.
```diff
+server {
+ listen 80;
+ return 301 https://$host$request_uri;
+}
server {
- listen 80;
+ listen 443 ssl;
+ server_name domain.abc;
+ ssl_certificate /etc/ssl/certs/domain.abc.pem;
+ ssl_certificate_key /etc/ssl/private/domain.abc.pem;
location / {
root /var/www;
index index.html index.htm;
}
```
### Generate self-signed SSL/TLS certs for testing
Add a command to our `Dockerfile` to generate self-signed certificates:
```Dockerfile
RUN openssl req -x509 -newkey rsa:4096 -nodes \
-keyout /etc/ssl/private/domain.abc.pem \
-out /etc/ssl/certs/domain.abc.pem \
-sha256 -days 365 -subj '/CN=domain.abc' \
-addext "subjectAltName=DNS:domain.abc"
```
Rebuild the image:
```console
$ make image-rm image-build
```
We need to update our previous test to use HTTPS instead of HTTP.
::: filename-for-code-block
`spec/nginx_spec.moon`
:::
```diff
-describe "http://domain.abc", ->
+describe "https://domain.abc", ->
it "sends /index.html", ->
- request = req "http://domain.abc"
+ request = req "https://domain.abc"
```
Run tests:
```console
$ make test
●●●
3 successes / 0 failures / 0 errors / 0 pending : 0.017065 seconds
```
## Test reverse proxy a subdomain request to a Unix socket
Let's say we have a running service that connects to a Unix socket. We want to
proxy the requests through `nginx` so that our service can respond to `https`
requests but can leave handling SSL/TLS to `nginx`.
### Configure `nginx`
We'll add another server block to `conf/conf.d/default.conf` for our subdomain,
`git.domain.abc`, with the proxy directives:
```nginx
server {
listen 443 ssl;
server_name git.domain.abc;
location / {
client_max_body_size 1024M;
proxy_pass http://unix:/run/gitea/gitea.socket;
proxy_set_header Connection $http_connection;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
### Add subdomain to SSL/TLS certs
Next, we need to add our subdomain to the generated SSL certs in the
`Dockerfile`:
```diff
RUN openssl req -x509 -newkey rsa:4096 -nodes \
-keyout /etc/ssl/private/domain.abc.pem \
-out /etc/ssl/certs/domain.abc.pem \
-sha256 -days 365 -subj '/CN=domain.abc' \
- -addext "subjectAltName=DNS:domain.abc"
+ -addext "subjectAltName=DNS:domain.abc,DNS:git.domain.abc"
```
### Add subdomain as a host
Let's assign the loopback address to a variable and then add our subdomain as a
host in our `Makefile`:
```diff
+loopback = 127.0.0.1
test:
@ct=$(shell docker run --rm -d \
-v $(PWD)/conf/conf.d:/etc/nginx/conf.d \
-v $(PWD)/html:/var/www \
-v $(PWD):/opt/app \
--network no-internet \
- --add-host=domain.abc=127.0.0.1 \
+ --add-host=domain.abc=$(loopback) \
+ --add-host=git.domain.abc=$(loopback) \
$(image)); \
```
### Add a test socket server
We need to start up a mock socket server for our test to ensure our request is
being proxied correctly. This is why we needed the LuaSocket library.
Copied and modified from [here](https://github.com/lunarmodules/luasocket/blob/4844a48fbf76b0400fd7b7e4d15d244484019df1/test/unixstreamsrvr.lua),
this should suit our purposes:
::: filename-for-code-block
`spec/unixstreamsrvr.moon`
:::
```moonscript
socket = require "socket"
socket.unix = require "socket.unix"
u = assert socket.unix.stream!
assert u\bind "/run/gitea/gitea.socket"
assert u\listen!
assert u\settimeout 1
c = assert u\accept!
while true
m = assert c\receive!
break if m == ""
print m
```
### Write the test
And now we can add our test:
::: filename-for-code-block
`spec/nginx_spec.moon`
:::
```moonscript
describe "https://git.domain.abc", ->
it "reverse-proxy's a subdomain request to a unix socket", ->
socket = fname: "unixstreamsrvr.moon", dir: "/run/gitea", owner: "nobody"
basepath = debug.getinfo(1).short_src\match"^(.*)/[^/]*$" or "."
seconds = 0.1
os.execute "install -o #{socket.owner} -d #{socket.dir}"
cmd = "su -s /bin/bash -c 'moon %s' %s"
server = io.popen cmd\format "#{basepath}/#{socket.fname}", socket.owner
os.execute "sleep #{seconds}" -- wait for server to start
f = io.popen "find #{socket.dir} -type s -ls", "r"
result = with f\read "*a"
f\close!
assert.truthy result\match "nobody%s+nogroup.+#{socket.dir}/gitea.socket"
req "https://git.domain.abc"
reqheader = with server\read "*a"
server\close!
assert.truthy reqheader\match "Host: git.domain.abc"
```
Because we modified the `Dockerfile`, we need to rebuild our image:
```console
$ make image-rm image-build
```
And if all went well, our test should pass.
```console
$ make test
●●●●
4 successes / 0 failures / 0 errors / 0 pending : 0.131619 seconds
```
## Conclusion
These are just a few examples of how to test `nginx` directives. Using these
tools, we can verify that changes to our server configuration are working the
way we intended.

View File

@@ -23,11 +23,11 @@ span.linenos.special { color: #50fa7b; background-color: #6272a4; padding-left:
.highlight .py-cpf { color: #6272a4 } /* Comment.PreprocFile */
.highlight .py-c1 { color: #6272a4 } /* Comment.Single */
.highlight .py-cs { color: #6272a4 } /* Comment.Special */
.highlight .py-gd { color: #8b080b } /* Generic.Deleted */
.highlight .py-gd { color: #ff5555 } /* Generic.Deleted */
.highlight .py-ge { color: #f8f8f2; text-decoration: underline } /* Generic.Emph */
.highlight .py-gr { color: #f8f8f2 } /* Generic.Error */
.highlight .py-gh { color: #f8f8f2; font-weight: bold } /* Generic.Heading */
.highlight .py-gi { color: #f8f8f2; font-weight: bold } /* Generic.Inserted */
.highlight .py-gi { color: #50fa7b } /* Generic.Inserted */
.highlight .py-go { color: #f8f8f2 } /* Generic.Output */
.highlight .py-gp { color: #50fa7b } /* Generic.Prompt */
.highlight .py-gs { color: #f8f8f2 } /* Generic.Strong */

View File

@@ -1,5 +1,70 @@
Path = require "sitegen.path"
dollar_temp = "z000sitegen_markdown00dollar0000"
-- a constructor for quote delimited strings
simple_string = (delim) ->
import P from require "lpeg"
inner = P("\\#{delim}") + "\\\\" + (1 - P delim)
inner = inner^0
P(delim) * inner * P(delim)
lua_string = ->
import P, C, Cmt, Cb, Cg from require "lpeg"
check_lua_string = (str, pos, right, left) ->
#left == #right
string_open = P"[" * P"="^0 * "["
string_close = P"]" * P"="^0 * "]"
valid_close = Cmt C(string_close) * Cb"string_open", check_lua_string
Cg(string_open, "string_open") *
(1 - valid_close)^0 * string_close
-- returns a pattern that parses a cosmo template. Can be used to have
-- pre-processors ignore text that would be handled by cosmo
parse_cosmo = ->
import P, R, Cmt, Cs, V from require "lpeg"
curly = P {
P"{" * (
simple_string("'") +
simple_string('"') +
lua_string! +
V(1) +
(P(1) - "}")
)^0 * P"}"
}
alphanum = R "az", "AZ", "09", "__"
P"$" * alphanum^1 * (curly)^-1
escape_cosmo = (str) ->
escapes = {}
import P, R, Cmt, Cs, V from require "lpeg"
counter = 0
cosmo = parse_cosmo! / (tpl) ->
counter += 1
key = "#{dollar_temp}_#{counter}"
escapes[key] = tpl
key
patt = Cs (cosmo + P(1))^0 * P(-1)
str = patt\match(str) or str, escapes
str, escapes
unescape_cosmo = (str, escapes) ->
import P, R, Cmt, Cs from require "lpeg"
escape_patt = P(dollar_temp) * P("_") * R("09")^1 / (key) ->
escapes[key] or error "bad key for unescape_cosmo"
patt = Cs (escape_patt + P(1))^0 * P(-1)
assert patt\match(str)
needs_shell_escape = (str) -> not not str\match "[^%w_-]"
shell_escape = (str) -> str\gsub "'", "''"
@@ -19,12 +84,13 @@ write_exec = (cmd, content) ->
fname
-- config command like this in site.moon:
-- require("renderers.markdown").cmd = "pandoc --mathjax >"
class PandocRenderer extends require "sitegen.renderers.markdown"
unescape_cosmo = @unescape_cosmo
escape_cosmo = @escape_cosmo
class PandocRenderer extends require "sitegen.renderers.html"
@escape_cosmo: escape_cosmo
@unescape_cosmo: unescape_cosmo
@parse_cosmo: parse_cosmo
source_ext: "md"
ext: "html"
cmd: "pandoc --mathjax --lua-filter pygments.lua >"
pandoc: (content) => Path.read_file write_exec @@cmd, content

View File

@@ -29,7 +29,6 @@ get_files = (path, prefix=path) ->
files = for file in *files
file\gsub "^#{escape_patt prefix}/?", ""
table.sort files
files
-- strip file extension from filename

View File

@@ -104,6 +104,10 @@ describe "https://miti.sh/posts/", ->
describe "https://miti.sh/posts", ->
it "sends /posts/index.html", ->
with require "sitegen.path"
assert .exists("html/posts/index.html"),
"missing html/posts/index.html (try `make build file=blog.html`)"
request = req "https://miti.sh/posts"
assert.same request\statusCode!, 200
assert.same request\statusMessage!, "OK"

View File

@@ -86,3 +86,11 @@ this code block has no label
assert.same [[<div class="sourceCode" id="cb1"><pre
class="sourceCode heex"><code class="sourceCode elixir"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a><span class="op">&lt;.</span>greet name<span class="op">=</span><span class="st">&quot;Jane&quot;</span><span class="op">/&gt;</span></span></code></pre></div>]], out
it "escapes and unescapes double dollar signs", ->
out = flatten_html render [[
```Makefile
$$name
```]]
assert.same [[<div class="highlight"><pre><span></span><code><span class="py-w"></span><span class="py-nv">$$name</span></code></pre></div>]], out