--- title: "Build A Static-Website Generator With Elixir, Part 1" blurb: "We take the first steps in designing and implementing the \"world's simplest static-website generator\". Building on tools and knowledge we acquired previously, and utilizing an incremental and iterative development process, we go through the entire software life-cycle from creating the initial project files to deploying to production. We spare nothing, from spelling out every command, to ensuring application integrity with tests, and even updating the README file. Grab a drink and some snacks, and dive right in!" ... { id: "build-static-website-generator-part-1" } ::: info This post was originally intended to be the first in a multi-part series. However, the deeper we got into this project, the more we realized we were basically implementing our own version of a web framework. Rather than continuing, we opted to change course and adopted an already-existing framework, [Phoenix](https://www.phoenixframework.org/), and simply added a Markdown-to-HTML conversion feature. As a consequence, there are no other parts to this post, but a description of the solution we chose instead can be found [here](/posts/publish-markdown-documents-as-static-web-pages-with-pandoc-and-phoenix). ::: This is one of our longer posts, so we've included a table of contents. ## Table of Contents - Introduction - Incremental Development Concepts - Iterative Development Concepts - Initial Planning - Deliverables - Initialization - Planning - Requirements - Analysis & Design - Implementation - Testing - Deployment - Evaluation - Project Control List - New Tasks - Completed Tasks - Conclusion ## Introduction We want to build the world's simplest static-website generator. Also, we want to use an iterative and incremental development method to do it. Finally, we want to use the Elixir programming language. Before we dive in, here's an extremely short primer on iterative and incremental development processes. We will combine these two ideas as we build this project. ### Incremental Development Concepts - The system is broken down into small pieces - Requirements are worked through in order of priority ### Iterative Development Concepts - The entire software development lifecycle is traversed in each iteration - Initialization, the initial "iteration", builds a base version of the system - All tasks and priorities are tracked in a Project Control List We are now in the initialization phase. We will go through each of the seven iteration steps to build a minimum base system. 1. Planning 2. Requirements 3. Analysis & Design 4. Implementation 5. Testing 6. Deployment 7. Evaluation ## Initial Planning ### Deliverables #### Functional Requirements * A way to publish written work online (ie blog) #### Non-Functional Requirements * Must use Elixir We love Elixir and functional programming. * Must be as simple as possible It's got to be dead simple. The app and the code. It's so easy to get carried away and over-engineer. Simplicity is a must-have requirement. * Must use an iterative and incremental development method In other words, short feedback loop/development life-cycle. Rather than releasing a minimum number of features as "version 1", we implement the smallest useful feature possible and release immediately. ## Initialization ### Planning We want to deliver the smallest increment of functionality possible, so we're going to start simple. We will first add an `index.html` serves as the initial page for our static website. Then, we will add a development server to serve the site to ourselves so we can see how it looks before we deploy it live. ### Requirements #### Functional 1. Post written text online This is the essence of the deliverable. Posting written text online is the first functionality we will deliver. #### Non-functional * A way to see what the output will look like locally before it is deployed This will make our lives easier during development and shorten the feedback loop. ### Analysis & Design #### Analysis The smallest bit of functionality we can offer is a static website with a single HTML file. We actually made a [proof of concept](https://github.com/webdevcat-me/docker_static_site_test) that does exactly this already and described the process in detail in an [earlier post](deploy-elixir-generated-html-with-docker-on-digitalocean.html). If we can add a development web server, that will pretty much meet all of our requirements. #### Design We could list our posts in a single HTML file, sorted in descending order of date, so that the most recent posts are at the top. Readers should have to scroll down to see older posts. Ideally, each post would have a title, a short message, and the date it was posted. The text should not stretch across the whole viewport on a wide screen, but should wrap at a point that facilitates readability. The markup should be [well-structured](https://developer.mozilla.org/en-US/docs/Learn/HTML/Introduction_to_HTML/Document_and_website_structure) and valid. The column of text should be centered in the viewport but the text itself should be left-aligned. ##### Name We've chosen "Serval" for the project name. ### Implementation #### 1. Create a new Mix project We're going to need a supervisor to start our development server later when we add it, so we use the `--sup` option when we run the `mix new` command to generate our initial project files. We can refer to our [explanation of this command](/posts/deploy-elixir-generated-html-with-docker-on-digitalocean.html#create-the-mix-project) previously, if necessary. ``` $ docker run --rm -w /opt -v $PWD:/opt -u $(id -u):$(id -u) elixir mix new serval --sup ``` Then, we initialize Git in our project directory. ``` $ cd serval $ git init ``` Configure the user name and email, if necessary. ``` $ git config user.email "webdevcat@proton.me" $ git config user.name "Catalin Mititiuc" ``` Lastly, create the initial commit. ``` $ git add . $ git commit -m "Initial commit" ``` #### 2. Create a GitHub repo After creating the remote repo on GitHub, we add it as the origin and push up our initial commit. ``` $ git remote add origin git@github.com:webdevcat-me/serval.git $ git push -u origin master ``` #### 3. Add an `index.html` file to the root directory Just a simple HTML boilerplate, for now. ::: filename-for-code-block `index.html` ::: ```html

Hello, Serval!

``` #### 4. Add a `Dockerfile` As we've mentioned before, we heavily favor using Docker for our projects when possible. We can copy [the `Dockerfile` from our proof of concept app](https://github.com/webdevcat-me/docker_static_site_test/blob/master/Dockerfile), but since we added a new file, `index.html`, we need to add a `COPY` command for it as well. ::: filename-for-code-block `Dockerfile` ::: ```dockerfile FROM elixir:slim WORKDIR /opt COPY lib ./lib COPY mix.exs ./mix.exs COPY index.html ./index.html ARG MIX_ENV=dev RUN mix deps.get --only $MIX_ENV && mix build ``` #### 5. Add a Mix task for building the site We can start with a copy of `lib/mix/tasks/build.ex` [from our concept app](https://github.com/webdevcat-me/docker_static_site_test/blob/master/lib/mix/tasks/build.ex), as it already mostly does what we want. Our build task isn't really going to "build" anything, just yet. For now, it's just going to copy our `index.html` file from our app's root directory to the `/public` directory in the Docker container. We'll build on this later to make it more dynamic. First, we want to add a variable for the output directory to the application environment. We do this in the `application()` function of the `mix.exs` file, as per the [documentation for the `mix compile.app` command](https://hexdocs.pm/mix/1.12/Mix.Tasks.Compile.App.html). ::: filename-for-code-block `mix.exs` ::: ```elixir def application do [ ... env: [output_directory: "/public"], ... ] end ``` Next, after we copy over `lib/mix/tasks/build.ex` from our proof-of-concept app, we can remove the `@markup` module attribute because we want the output directory to contain the contents of the `index.html` file from our root directory. Let's add a module attribute for our filename ```elixir @filename "index.html" ``` and replace the last line in `run()` with ```elixir def run(_args) do ... File.cp!(@filename, Path.join(output_directory, @filename)) end ``` And let's not forget to update the app name in the call to `fetch_env()`: ```elixir output_directory = Application.fetch_env!(:serval, :output_directory) ``` Here's what the build task looks like. We also updated the `@moduledoc` to reflect the changes in the task, as well as updated and added shell output to inform us when the task finishes. ::: filename-for-code-block `lib/mix/tasks/build.ex` ::: ```elixir defmodule Mix.Tasks.Build do @moduledoc "Creates a /public directory and copies an HTML index file from the root directory there" @shortdoc "Builds static HTML file" use Mix.Task @filename "index.html" @impl Mix.Task def run(_args) do Mix.shell().info("Building markup...\n " <> @filename) output_directory = Application.fetch_env!(:serval, :output_directory) File.mkdir_p!(output_directory) File.cp!(@filename, Path.join(output_directory, @filename)) Mix.shell().info("Done.") end end ``` So far, everything is pretty much just like our concept app that we're using as a reference. To make sure there aren't any problems, let's try building our Docker image. $ docker build -t serval_dev . And now let's start a container and check the contents of the HTML file in `/public`. $ docker run --rm -w /opt -v $PWD:/opt serval_dev bash -c "mix build && cat /public/index.html" Building markup... index.html Done.

Hello, Serval!

So far, so good! Let's add these Docker commands to our `README.md` for future reference. ::: filename-for-code-block `README.md` ::: ```markdown # Serval The world's simplest static-website generator! ## Build Docker image docker build -t serval_dev . ## Build static site docker run --rm -w /opt -v $PWD:/opt serval_dev mix build ``` Since we expanded on the `build` task that we copied from our reference project, next we'll add a test for the new functionality. #### 6. Add a test for our Mix build task To start, our POC app has [a test for the Mix `build` task](https://github.com/webdevcat-me/docker_static_site_test/blob/master/test/mix/tasks/build_test.exs) we can copy over to our project in `test/mix/tasks/build_test.exs`. We'll want to add a new test case to cover the changes we've made. ```elixir test "updates output files" ``` ##### First pass To test that our Mix task is updating the output files, we first need to create them so they exist already before the task runs. We need to create two fixtures in our test. The first is the file already existing in the output directory, and the second is the file in our root directory that will replace it when we run our build task. ```elixir test "updates output files", %{output_dir: output_dir} do output_path = Path.join(tmp_path("build"), output_dir) File.mkdir_p!(output_path) File.write!(Path.join(output_path, "index.html"), """ hello world """) File.mkdir_p!(tmp_path("build")) File.cd!(tmp_path("build")) File.write("index.html", """ welcome """) capture_io(fn -> Mix.Tasks.Build.run([]) end) assert File.read!(Path.join(output_path, "index.html")) =~ "welcome" File.cd!(__DIR__) end ``` ##### Refactoring ###### De-duplicate fixture Both tests in our file now need to use an `index.html` fixture, so we can move that markup into a module attribute, `@markup`, to avoid duplicating it in both tests. ```elixir @markup """ hello world """ ... test "creates output file", %{output_dir: output_dir} do ... File.write("index.html", @markup) ... end test "updates output files", %{output_dir: output_dir} do ... File.write!(Path.join(output_path, "index.html"), @markup) ... end ``` ###### De-duplicate directory setup Next, both our tests require setting up a temporary directory. ```elixir File.mkdir_p!(tmp_path("build")) File.cd!(tmp_path("build")) ... File.cd!(__DIR__) ``` The official Elixir code repository has [a test helper called `in_tmp()`](https://github.com/elixir-lang/elixir/blob/b829f1bae76f0fa847535083aef5e4e015ce0ab5/lib/mix/test/test_helper.exs#L122-L127) that does exactly this. We can copy that function into our test file. We're actually already using the function that it calls, `tmp_path()`, in our code. 😁 ```elixir defp in_tmp(which, function) do path = tmp_path(which) File.rm_rf!(path) File.mkdir_p!(path) File.cd!(path, function) end ``` And now we can remove the temp directory setup from our tests, and just wrap them in a call to `in_tmp()`. ```elixir test "creates output files", %{output_dir: output_dir} do in_tmp("build", fn -> ... end) end test "updates output files", %{output_dir: output_dir} do in_tmp("build", fn -> ... end) end ``` And since `in_tmp()` deletes any existing temp directory, we don't need to run our test teardown for every test. And since the output directory doesn't change between tests, we don't need to run our setup for every test, either. We can just do it all, once per test-run, by defining our setup block in a call to `setup_all()` instead of `setup()`. ```elixir setup_all do ``` ##### Cleanup - We use the string `"build"` in multiple places, so we can move that into the setup block and pass it into our tests. - We call `Path.join(output_path, "index.html")` twice, so we can just call it once in the setup block and pass it to our tests. - We use the string `"index.html"` in multiple places. Let's put that in a module attribute `@filename`. ##### Final form Here's what the final version of our test looks like. ::: filename-for-code-block `test/mix/tasks/build_test.exs` ::: ```elixir defmodule Mix.Tasks.BuildTest do use ExUnit.Case import ExUnit.CaptureIO @filename "index.html" @markup """ hello world """ setup_all do output_dir = "public" file_path = Path.join(output_dir, @filename) original = Application.fetch_env!(:serval, :output_directory) Application.put_env(:serval, :output_directory, "public") on_exit(fn -> Application.put_env(:serval, :output_directory, original) File.rm_rf!(tmp_path()) end) [context: "build", output_dir: output_dir, file_path: file_path] end test "creates output files", %{context: context, file_path: file_path} do in_tmp(context, fn -> File.write!(@filename, @markup) capture_io(fn -> Mix.Tasks.Build.run([]) end) assert File.exists?(file_path) end) end test "updates output files", %{file_path: file_path} = context do in_tmp(context[:context], fn -> File.mkdir_p!(context[:output_dir]) File.write!(file_path, @markup) File.write!(@filename, """ welcome """) capture_io(fn -> Mix.Tasks.Build.run([]) end) assert File.read!(file_path) =~ "welcome" end) end defp tmp_path, do: Path.expand("../tmp", __DIR__) defp tmp_path(extension), do: Path.join(tmp_path(), extension) defp in_tmp(which, function) do path = tmp_path(which) File.rm_rf!(path) File.mkdir_p!(path) File.cd!(path, function) end end ``` Let's try running our test. $ docker run --rm -w /opt -v $PWD:/opt serval_dev mix test .... Finished in 0.05 seconds (0.00s async, 0.05s sync) 1 doctest, 3 tests, 0 failures And let's not forget to add this Docker command to the `README`. ::: filename-for-code-block `README.md` ::: ```markdown ... ## Run tests docker run --rm -w /opt -v $PWD:/opt serval_dev mix test ``` We still need to add some markup and styles to our `index` file, but without a web server to show us the content, making and verifying changes will be tedious. Let's now focus on the development server and then we'll come back around to markup and styling. #### 7. Add a web server for development To add our development web server, we can take a look at the Plug library documentation, as it provides a [nice example](https://hexdocs.pm/plug/readme.html#hello-world-request-response) to get us started. We add `plug_cowboy` to `deps()` in our `mix.exs` file. Since the web server will only run during development, we use the `only: :dev` option. ```elixir defp deps do [ {:plug_cowboy, "~> 2.0", only: :dev}, ] end ``` Next, we copy the example plug module from the Plug documentation to `lib/serval/plug.ex`. ::: filename-for-code-block `lib/serval/plug.ex` ::: ```elixir defmodule Serval.Plug do import Plug.Conn def init(options) do # initialize options options end def call(conn, _opts) do conn |> put_resp_content_type("text/plain") |> send_resp(200, "Hello world") end end ``` We can update `lib/serval/application.ex` to start our web server under a supervisor when the application starts. We'll also add a call to our build task to make sure we're not looking at a stale page on startup. ::: filename-for-code-block `lib/serval/application.ex` ::: ```elixir defmodule Serval.Application do @moduledoc false use Application require Logger @impl true def start(_type, _args) do Mix.Tasks.Build.run([]) children = [ {Plug.Cowboy, plug: Serval.Plug, scheme: :http, options: [port: 4000]} ] opts = [strategy: :one_for_one, name: Serval.Supervisor] Logger.info("Plug now running on localhost:4000") Supervisor.start_link(children, opts) end end ``` Now, we can test our development server. First, we fetch our new dependencies with: $ docker run --rm -w /opt -v $PWD:/opt serval_dev mix deps.get Then, we can start the application. Since we want our server to stay running until we stop it, we use the `--no-halt` option. We also need to publish a port, since our browser is connecting to a server inside a container. The `-p 4000:4000` option binds the container's port `4000` to port `4000` on our system. $ docker run --rm -it -w /opt -v $PWD:/opt -p 4000:4000 serval_dev mix run --no-halt It's going to ask to install `hex` and then `rebar`. Mix requires the Hex package manager to fetch dependencies Shall I install Hex? (if running non-interactively, use "mix local.hex --force") [Yn] ... Could not find "rebar3", which is needed to build dependency :telemetry Shall I install rebar3? (if running non-interactively, use "mix local.rebar --force") [Yn] ... ==> serval Compiling 4 files (.ex) Generated serval app Building markup... index.html Done. 01:26:30.509 [info] Plug now running on localhost:4000 And when we visit `localhost:4000`, we should see a "Hello world" message in plain text. ![Our development server rendering the text "Hello world".](/images/serval-dev-server-hello-world.png) Great! So our web server is running and responding to requests! Before moving on, let's update our `README` with some new Docker commands. ::: filename-for-code-block `README.md` ::: ```markdown ... ## Run development server ### First, fetch the dependencies docker run --rm -w /opt -v $PWD:/opt serval_dev mix deps.get ### Start the server docker run --rm -it -w /opt -v $PWD:/opt -p 4000:4000 serval_dev mix run --no-halt Visit `localhost:4000` to view. ``` #### 8. Serve files from the output directory Our development web server currently responds to every call with "Hello world" in plain text. Now, lets see about making it serve our file from the container's `/public` directory. We update `lib/serval/plug.ex` to get the output directory path from the application environment variable. ```elixir def call(conn, _opts) do path = :serval |> Application.fetch_env!(:output_directory) |> Path.join("index.html") ... ``` Then, we replace the call to `send_resp` with `send_file` and pass the MIME type `"text/html"` instead of `"text/plain"` to the `put_resp_content_type()` call. ```elixir ... conn |> put_resp_content_type("text/html") |> send_file(200, path) end ``` Now when we start the development server, we should see the contents of the `index.html` file from our project root. ![Our development server rendering the `index.html` file from our project root directory with the text "Hello, Serval!".](/images/serval-hello-serval.png) If we make a change to `index.html`, we will not immediately see the results by refreshing the browser. We will first have to run our build task before we can see the new output. Since we added a call to the build task in `lib/serval/application.ex`, we could see the changes by simply restarting the dev server. To see the changes without restarting, we first need to find the name of the container running our app with `docker ps`. Then we can run a command in that container like so: $ docker exec CONTAINER_NAME mix build After that, refreshing the browser should show us any changes to `index.html`. That's also another one to add to `README.md`. ::: filename-for-code-block `README.md` ::: ```markdown ... ## View changes To see updates made to `index.html`, simply restart the dev server. To see changes without restarting, first find the name of the running container with `docker ps`. Then, run: docker exec CONTAINER_NAME mix build ``` #### 9. Rebuild the Docker image So, our dev server is running and it is serving the `index.html` file from our root directory. Awesome! If you have stopped and started the dev server a few times, you will notice that it asks to install Hex each time. Because we always run our commands in a new container, the Hex installation doesn't get saved between stops and restarts. Since our `Dockerfile` fetches dependencies and calls the build task, all we need to do to fix the issue is rebuild our Docker image. $ docker image rm serval_dev $ docker build -t serval_dev . Now when we start our app, we will no longer receive the `Shall I install Hex?` prompt. $ docker run --rm -it -w /opt -v $PWD:/opt -p 4000:4000 serval_dev mix run --no-halt Building markup... index.html Done. 03:24:57.061 [info] Plug now running on localhost:4000 #### 10. Avoid compiling the dev server in `:prod` (and running it in `:test`) We don't require any dependencies in production, but we do compile code that still refers to those non-existent dependencies. We have to edit `mix.exs` to ensure that our web server code is not compiled and our application is not started in production. ```elixir def project do [ ... elixirc_paths: elixirc_paths(Mix.env()), ... ] end def application do ... mod: mod(Mix.env()) ] end defp elixirc_paths(:prod), do: ["lib/mix/tasks"] defp elixirc_paths(_), do: ["lib"] defp mod(:dev), do: {Serval.Application, []} defp mod(_), do: [] ``` #### 12. Update markup Now that the dev server is working, viewing changes to our markup is much easier. We can go about improving what we have of our static website so far to make it more presentable. Our design calls for "title, a short message, and the date". Since we are building the world's simplest static-website generator, the simplest thing we can think of is a heading, a paragraph, and a date: We're going to keep it extremely simple. What is the smallest amount of information we can convey that is going to be useful? A page header would be good, so readers have something to identify us with. And then a heading, date, and a paragraph to satisfy our design. ```html

Page Heading

Post Heading

Lorem ipsum dolor sit amet, consectetur adipiscing elit.

``` After restarting our server, we can see our changes in the browser. ![Browser displaying a webpage with our HTML changes](/images/serval-update-markup.png) This is enough to get us started. Let's add some styling. #### 13. Add styles We're going to use white-space as much as possible do delineate different sections. Styling is pretty subjective, and we're still early in this project, so we don't want to spend too much time on the appearance now. We just want to make the text a bit more readabale. We add our style directly to the head of the `index.html` file. We will break the styles out into their own file later, for now we just want to do the minimum to initialize the project base. Here's what we've added to the bottom of the `` section in `index.html`: ```html ... ``` We can now see what our styles look like in a few different viewport widths. We want to make sure to test both super-narrow, for small mobile screens, and super-wide, for wide monitors. ![Narrow](/images/serval-style-initial-narrow.png) ![Medium](/images/serval-style-initial-medium.png) ![Wide](/images/serval-style-initial-wide.png) Our goal is legibility, and that looks well-legible enough to get us started. ### Testing Testing so far is pretty straight-forward: either our `index` page is viewable or it isn't. We're going to set up a staging environment to ensure that it's viewable once we deploy it. #### Deploy to a staging environment We already described our deployment process in [the earlier post](/posts/deploy-elixir-generated-html-with-docker-on-digitalocean.html) about the proof-of-concept for this project. We're going to follow the same steps to deploy Serval, but first we want to create a separate resource for staging, so that we can look everything over one last time before we commit to deploying for production. One difference from our POC instructions is, after we add our GitHub repository, DigitalOcean detects two resources: a Static Site and a Web Service. We want to delete the the Static Site resource and continue with the deployment steps using the resource that was detected as a Web Service. 1. For our Resource Name, we used, simply, `staging`. 2. We also want to set an HTTP Request Route for something like `/staging`, because the root route `/` will be for production. 3. On the info screen, we change the default app name to `serval` and click "Next". 4. Then we finish up by clicking "Create Resources". If all goes well, when deploying finishes, we should be able to visit the `/staging` route of the URL DigitalOcean gives for our app. Something like: https://serval-abc12.ondigitalocean.app/staging If everything looks alright, we can tag the release and proceed to deploying a production version. #### Tag the release And that's it! With our requirements met, we are ready to release. Let's add a tag to our repository to mark the release version and push it up to our remote repo. $ git tag v0.1.0 $ git push origin v0.1.0 ### Deployment #### Deploy to a production environment To deploy our production version of Serval, we're going to create a new resource for the DigitalOcean App we just created. 1. Go to the DigitalOcean Apps page. 2. Click on the Serval App. We named ours, appropriately, `serval`. 3. Click on "Create" and select "Create Resources from Source Code" from the menu. 4. As before, select our Serval repository. 5. As with our staging process above, it will detect two resources. We want to delete the Static Site one and edit the Web Service one as before. 6. We set our resource name as `production`. 7. Change the resource type from Web Service to Static Site and click "Save". 8. Again, we want to set our output directory to `/public` and click "Save". 9. Make sure we add a root route, `/`, under HTTP Request Routes and click "Save". 10. If we added a `MIX_ENV` with a value of `prod` under Global when deploying the staging resource, the we won't have to do anything else on the Environment Variables screen. Otherwise, we should make sure we add it, either under Global or under "production". 11. And if everything looks alright on the Review screen, we click "Create Resource". Once deployment finishes, our App will have two components: a "staging" one and a "production" one. We can now visit our production app at the URL DigitalOcean provides (or a custom URL, but that is outside the scope of this post). ### Evaluation We'll now take a moment to consider our project so far. On the functional side, what's most obvious is that we're not really "generating" anything, we're still just editing raw HTML. We'll adress this point soon. On the non-functional side, two things in particular stick out in our minds. 1. Having to run the build task to see a development change is repetitive and time-consuming 2. Having to refresh the browser to see a development change is, again, repetitive and time-consuming The consequences of having to do these repetitive steps over and over are a slower feedback loop and a longer development cycle. Luckily, these sorts of issues are easily remedied with a little automation. That sounds like a great next-step for this project and a great addition to our Project Control List. ## Project Control List Here's what we've accomplished in this iteration, and what we look forward to working on in the next one. ### New Tasks 1. Automate running the build task after code changes 2. Automate refreshing the browser after a rebuild ### Completed Tasks And here's what we've accomlished in this initial project step, building on the proof-of-concept we built before. - ~~Add a development web server~~ - ~~Add a root `index.html` file~~ ## Conclusion We have done the initialization phase and planted the "seed" from which the rest of our app will evolve. So far, we can publish a website with a single page and we can run a web server in our development environment to serve it to ourselves locally. Next time, we will make the build task run automatically whenever our markup changes. We'll see each other again, then. Take care and happy coding!