A Conservative Backend

Slimvoice – A Webapp Without JavaScript is a series where I document how I rebuilt my app, Slimvoice, using as little JavaScript as possible.

Prelude

I built the first two versions of Slimvoice on Node.js and MongoDB backend: in hindsight a massive mistake and learning experience. If I was going to revamp the UI and ditch JavaScript, I decided that it was time to rethink the whole backend as well. Just like my previous experiences with single-page applications, I have had a slew of backend nightmares as well.
I have had bosses who demand “we must use Firebase!" Why? "Well, Google can’t go down!" was their response. Turns out that isn’t really the case. Not to mention NoSQL being the wrong tool for most jobs, but being permanently tied to Firebase is also pretty bad. They would also tout TypeScript as solving all of JavaScript’s problems. I hardly think so.

The Backend

I rebuilt the backend in Go. Go forces me to handle errors at every corner. This is a good thing. try/catch is not your friend. I now know almost exactly every error that can possibly happen to a user. This not only helps me sleep at night knowing that I’ve done my due diligence, but user-facing errors are almost non-existent in Slimvoice. And when one does pop up, it won’t involve sorting through a deep, cryptic stack trace from a complex library I didn’t write.
I’ve done microservices in the past. I’ve done Docker, Mesos, Kubernetes, Rancher, you name it. The costs of managing microservices and the overhead you incur to make it happen do not pay off until your scale is unbelievably large and thus, Slimvoice is compiled into a single static binary. It’s so dang easy to deploy. At the bottom of the article I have some real-world graphs.

Gary Bernhardt
@garybernhardt

Computers exist to serve us, not the other way around. If it is not fast and reliable then it is wrong!
19:02 PM – 15 Jun 2018

128515

Go is pretty fast, both in execution and compilation. If I could have written the whole thing in C I would have (I prefer absolute control over the machine), but Go’s standard library is fantastic and makes it so much easier to productively write a web server (without worrying about buffer overflows and a host of other vulnerabilities from C). It’s a calculated trade off.
I’m using no backend web framework like Django; it’s little more than a bunch of route handlers. GET /something → return something. POST /something → do something and 303 redirect to a GET route. It’s so simple there isn’t even any code to share. I will say that instead of Go’s standard net/http package, I ended up using fasthttp from Aliaksandr Valialkin (the same guy who wrote quicktemplate, which is what the first post in this series is about).
For those that are afraid of writing a web server without a web framework and ORM, there are really only a few things you need to keep in mind:

Make sure that you validate which user can do what on every route.
Remember to use positional arguments instead of concatenating SQL strings to protect yourself from SQL injection attacks.
Make sure your templating language sanitizes user data rendered to HTML to prevent XSS attacks.
Know the cost of a database query.
Know what memory you’re allocating and when it’s freed.

Database

I made the mistake of following the hype and using MongoDB the first time around. MongoDB was "what everyone was using" back then, but completely the wrong choice. I lost all the benefits of a relational database and gained nothing but an "easy to use" JSON API. The new Slimvoice has no complicated ORM (Mongoose was a disaster in the previous versions of Slimvoice). Just plain old SQL with the rock-solid Postgres. Postgres does all of the hard work of enforcing constraints on the data, which was previously written by hand in JavaScript (I could only pray that those validations and constraints were correct and worked everywhere).
For example, I decided that the app needs to have the constraint that you can’t have duplicate invoice numbers per user.
CREATE UNIQUE INDEX ON invoices (user_id, number);

Why leave that up to the app, which could have a bug, or I could introduce one later? This way, I know that my data is always in a good state.
Here’s another example where I make sure that every invoice must have the same number of expense descriptions, expense quantities, and expense values. If the data is ever in a bad state, the database won’t save it. This has been built into SQL for a long time. By using NoSQL, you are throwing away most of these benefits.
ALTER TABLE invoices ADD CONSTRAINT invoices_check CHECK (cardinality(exp_descriptions) = cardinality(exp_quantities) AND cardinality(exp_quantities) = cardinality(exp_values));

Now that Postgres does all of heavy lifting of managing constraints and running calculations (like totaling invoices), my Go server is little more than a template-rendering HTTP wrapper around my database.

Development Environment

I develop solely on a $200 Celeron-based netbook with 4GB of RAM. It forces me to develop only fast software by keeping my frame of reference at the very bottom of the scale: if things run well on my machine, they will run fantastic on most everyone else’s device (see the previous article about avoiding JavaScript). This also is a huge factor in my distaste for long compilation times. If I’m going to be productive, I need to constantly be recompiling, making tweaks, and running tests. Webpack using all my RAM? No thanks. TypeScript? Slow on my machine. Using GCC to compile a C program? Fast. Go compiler? Fast enough.
Slimvoice builds everything with a Makefile. If you set up a Makefile properly, it’s fast! One of the most valuable things I’ve done in my programming career is to read the Make documentation. It ubiquitous and agnostic of your programming language. It’s quite powerful, encourages simplicity, and I use it for all kinds of projects. Here’s the beginning of my Makefile for compiling the server.
.PHONY: all
all: test bin/slimvoice css js images

GO_SOURCES = $(shell find . -type f -name "*.go")
TEMPLATE_SOURCES = $(shell find templates -type f -name "*.qtpl")

# The backend binary depends on all of the source code and templates
bin/slimvoice: $(GO_SOURCES) $(TEMPLATE_SOURCES)
mkdir -p $(@D)
go generate
go build -o $@

# Tests can be run with `make test`
.PHONY: test
test:
go test .

JavaScript is really simple, just copy it from the source directory to the destination directory.
JAVASCRIPT_SOURCES = $(shell find frontend/scripts -type f -name "*.js")

.PHONY: js
js: $(patsubst frontend/scripts/%.js,public/js/%.js,$(JAVASCRIPT_SOURCES))

public/js/%.js: frontend/scripts/%.js
mkdir -p $(@D)
cp -f $< $@ CSS is similar, but compiled from SASS. STYLESHEET_SOURCES = $(shell find frontend/stylesheets -maxdepth 1 -type f -name "*.scss") STYLESHEET_INCLUDES = $(shell find frontend/stylesheets/include -type f -name "*.scss") .PHONY: css css: $(patsubst frontend/stylesheets/%.scss,public/css/%.css,$(STYLESHEET_SOURCES)) public/css/%.css: frontend/stylesheets/%.scss $(STYLESHEET_INCLUDES) mkdir -p $(@D) sassc -I frontend/stylesheets/include -t compressed $< $@ Responsive Images Here's the solution I came up with for creating @2x images from @4x, and @1x images from @2x for all the retina-ready srcset attributes I used on Slimvoice. Now, when I update the high-res image, all of the smaller versions are created automatically. 4X_IMAGE_SOURCES = $(shell find frontend/images/high-dpi -type f -name "*@4x.png") 2X_IMAGE_SOURCES = $(shell find frontend/images/high-dpi -type f -name "*@2x.png") .PHONY: images images: $(patsubst frontend/images/high-dpi/%@4x.png,public/img/%@4x.png,$(4X_IMAGE_SOURCES)) \ $(patsubst frontend/images/high-dpi/%@4x.png,public/img/%@2x.png,$(4X_IMAGE_SOURCES)) \ $(patsubst frontend/images/high-dpi/%@4x.png,public/img/%@1x.png,$(4X_IMAGE_SOURCES)) \ $(patsubst frontend/images/high-dpi/%@2x.png,public/img/%@2x.png,$(2X_IMAGE_SOURCES)) \ $(patsubst frontend/images/high-dpi/%@2x.png,public/img/%@1x.png,$(2X_IMAGE_SOURCES)) public/img/%@4x.png: frontend/images/high-dpi/%@4x.png @ mkdir -p $(@D) cat $< | pngquant - > $@

public/img/%@2x.png: frontend/images/high-dpi/%@2x.png
@ mkdir -p $(@D)
cat $< | pngquant – > $@

public/img/%@2x.png: frontend/images/high-dpi/%@4x.png
@ mkdir -p $(@D)
convert $< -resize 50% – | pngquant – > $@

public/img/%@1x.png: frontend/images/high-dpi/%@4x.png
@ mkdir -p $(@D)
convert $< -resize 25% – | pngquant – > $@

public/img/%@1x.png: frontend/images/high-dpi/%@2x.png
@ mkdir -p $(@D)
convert $< -resize 50% – | pngquant – > $@

This is pretty verbose. If you think it can be done with less code, I’d love to hear from you!
The beauty of all this is that I don’t need any complicated build systems or a suite of additional software installed on my machine. GNU Make is pretty easy to come by.

Quick Iterations

At this point, things are looking pretty good. I can run make and everything is compiled and ready to go. But I’m missing two cool things that all the JavaScript kids are using these days.
The first is live reload on the server. When I make a change to the server code, I’d like to be automatically recompiled and restarted so that I don’t have to tab over to my terminal, type make, type ./bin/slimvoice, and then tab back to the browser to start testing my changes. You don’t need nodemon or any of that baloney to restart things when you make changes. You’ve got bash right? You’ve probably got inotify; if not it’s not hard to get on your system. Here’s a script I saved as dev.sh that will restart any binary when files change. Start it with ./dev.sh path/to/your/binary.
#!/bin/bash

make
while true; do
$@ &
PID=$!
inotifywait -e close_write -e create -e delete -r .
make
kill $PID
done

Simple! nodemon is currently at 1,102 commits on GitHub at the time of this writing, which is a little overkill. (What could they be fixing with all of those commits?)
Second, is live reload in the browser. This is extremely handy when making CSS changes. It would be ideal to pipe a signal from Make to the browser indicating that something has changed, but that doesn’t seem to be an available method to communicate with the browser. At the moment, I’m just using the Live Reload Firefox extension.

The Server

I run my very simple, single compiled binary at Digital Ocean on a $5 box, along with Postgres and nginx on the same machine. The machine is their cheapest offering, with 1GB RAM, 1 vCPU, and 25GB SSD.
I was pleasantly surprised to see that my previous post, A JavaScript-Free Frontend kind of blew up on the Internet: it made it to the front page of Hacker News, on Reddit, and racked up about 90,000 views here on dev.to in 24 hours. The perfect chance to stress-test my server! Here’s a screenshot of the Digital Ocean monitoring dashboard. Look at the bandwidth graph at the bottom and you can see when it got popular on Hacker News. Compare that to the CPU and memory graphs. Didn’t even break a sweat.

Conclusions

Slimvoice has been fun and easy to develop relative to other projects I’ve worked on who dug themselves into a huge, miserable hole. I’m very happy with both the dev experience and UX of the final product.

Don’t follow the hype.

Everything has a cost.

Make critical decisions about why one approach is better than another, despite what the marketing page says or what everyone else is doing. The people selling the cool new stuff are typically sweeping the downsides under the rug.

Link: https://dev.to/winduptoy/a-conservative-backend-4252