go/pkgweb (HowettNET GitLab)

I prefer to host my Go packages under the howett.net/ namespace. To do so, I needed to be able to respond to an unbounded set of requests with go-get metadata. Unfortunately, that rather conflicts with hosting a site here without having to write a bunch of rewriting rules. It also conflicts with having an unreliable web server, especially when you have a popular package or two.

Enter pkgweb. Pkgweb acts as the “front door” server to howett.net and responds to almost every request by redirecting it to www.howett.net–unless, of course, it’s come from go get. When a request comes in from go get, the URL is matched against a simple config file indicating which part of my package tree maps to which repository.

The config file can specify a number of things:

  • Hostnames to which to respond, and addresses on which to listen
  • Where to redirect bare requests
  • Which package namespaces to answer go get for
  • How to transform the incoming URL to a VCS URL
    • Namespaces can be mapped by depth, which is how many path components to preserve from a package name
    • Names can be transformed through the use of Go templates, with .Leaf available to collect all path components not covered by depth (above)

pkgweb supports the automatic generation and renewal of https certs using Let’s Encrypt and exposes a number of metrics that can be consumed by Prometheus.

On my reasonably underpowered front door server (kaus.howett.net, at the time of writing), pkgweb can respond to most requests in ~1ms. It will usually run for months and handily serves millions of package requests.

Grafana Dashboard displaying pkgweb metrics for kaus.howett.net (click to enlarge)

Configuration⌗ 

hostnames = ["example.com", "example.net"]

# only respond to requests from go get
goget_only = true
# redirect non-go-get responses to...
# (leave blank to reject all non-go-get requests)
redirect = "https://www.example.org"

# log redirects (NCSA-like)
log = true

[autocert]
# for autocert servers, [admin] will be the
# e-mail address of record for Let's Encrypt
admin = "admin@example.com"
# generated certificates and keys will be cached here
cache = "./autocert_cache"

# prometheus metrics
[metrics]
enabled = true
listen = ":9191"

# 1..n servers (usually one non-ssl, one ssl)
[[server]]
listen = ":80"

[[server]]
listen = ":443"
ssl = true
autocert = true # request certificates for [hostnames] from Let's Encrypt
cert = "" # path to certificate (ignored if autocert is enabled)
key = "" # path to key (ignored if autocert is enabled)

# 1..n namespaces
[[namespace]]
prefix = "example.com"
depth = 1 # packages under example.com are only one level deep:
          # example.com/a/cmd/b folds to example.com/a.
	  # .Leaf below will be "a"
repo = "https://github.com/example/go-{{.Leaf}}.git"
vcs = "git"

[[namespace]]
prefix = "example.net"
depth = 2 # packages under example.net can be two levels deep:
          # example.net/a/b/cmd/c folds down to example.net/a/b
	  # .Leaf below will be "b"
repo = "https://github.com/example/go-a-{{.Leaf}}.git"
vcs = "git"

[[namespace]]
prefix = "example.com/special"
depth = 1 # example.com/special doesn't have any children.
          # all requests for paths below .../special will
	  # result in the same repository URI
	  # Had we used depth=0, the imputed package name
	  # would be "example.com"
repo = "https://github.com/example/special.git"
vcs = "git"