Golang Optimization #

Once a self-hosted runner is on your own network, jobs that use self-hosted feel close to GitHub-hosted ones: same workflow syntax, less time spent in the hosted queue, a real ARM64 environment, and more CPU and RAM. The gap that shows up next is geography.

actions/setup-go can sit there for a long time downloading a Go toolchain of about 120MB over a slow path to GitHub.

Original write-up: GitHub Actions Self-Hosted Runner 优化——Golang 相关内网缓存.

GOPROXY #

Module downloads (go mod tidy and similar) should use an internal proxy. The runner image reads GOPROXY from the environment. The compose and Kubernetes examples set:

GOPROXY=http://goproxy.knat.network,https://proxy.golang.org,direct

That covers modules. It does not cover installing Go itself. Workflows still use actions/setup-go for the toolchain.

Baking every Go version into the image looks tempting, and it does not scale. You would end up maintaining tags such as github-runner:go1130 and github-runner:go1160, sizing each one by hand, and asking users to pick a label like runs-on: [self-hosted, X64, go1130]. That is closer to a Jenkins label matrix than to Actions.

Leave toolchain install as a workflow step. Composing the environment per job, including with a matrix, is the point of Actions. After setup-go installs a version once, later jobs on the same runner reuse that cache, unless the runner was started with --ephemeral.

`actions/setup-go` #

A typical step looks like this:

- uses: actions/setup-go@v2
  with:
    go-version: '1.16'

The action does not invent download URLs. Its test data points at release assets such as:

https://github.com/actions/go-versions/releases/download/1.12.17-20200616.21/go-1.12.17-darwin-x64.tar.gz

Those assets are listed in actions/go-versions versions-manifest.json:

[
  {
    "version": "1.17.5",
    "stable": true,
    "release_url": "https://github.com/actions/go-versions/releases/tag/1.17.5-1559554870",
    "files": [
      {
        "filename": "go-1.17.5-darwin-x64.tar.gz",
        "arch": "x64",
        "platform": "darwin",
        "download_url": "https://github.com/actions/go-versions/releases/download/1.17.5-1559554870/go-1.17.5-darwin-x64.tar.gz"
      },
      {
        "filename": "go-1.17.5-linux-x64.tar.gz",
        "arch": "x64",
        "platform": "linux",
        "download_url": "https://github.com/actions/go-versions/releases/download/1.17.5-1559554870/go-1.17.5-linux-x64.tar.gz"
      },
      {
        "filename": "go-1.17.5-win32-x64.zip",
        "arch": "x64",
        "platform": "win32",
        "download_url": "https://github.com/actions/go-versions/releases/download/1.17.5-1559554870/go-1.17.5-win32-x64.zip"
      }
    ]
  }
]

The action README says the same thing: it checks the local cache first, and if that version is missing it reads the main branch of go-versions. In src/installer.ts the lookup is:

  const releases = await tc.getManifestFromRepo(
    'actions',
    'go-versions',
    auth,
    'main'
  );

The cache path is then:

Download the packages listed in versions-manifest.json onto the internal network.
Fork actions/go-versions and replace each download_url with the internal URL.
Fork setup-go and point its manifest lookup at that fork.
Call the forked setup-go from the workflow.

Download the packages #

Set HOST_URL to the internal download base, STORE_PATH to the directory that will hold the archives, and GOLANG_VERSION_LIST to the versions you want. Save the script as download.py and run it.

import requests
from urllib.parse import urlparse
import os
import json

## ENV
HOST_URL = "http://download.nova.moe/download/github-actions/golang/"
STORE_PATH = "/path/to/download/github-actions/golang/"
GOLANG_VERSION_LIST = ['go-1.16','go-1.17','go-1.13']
## END ENV

def process_each_package(package_filename, package_url):
    package_path = STORE_PATH + package_filename
    if not os.path.isfile(package_path):
        print("Downloading: " + package_url)
        r = requests.get(package_url)
        with open(package_path, 'wb') as f:
            f.write(r.content)
    return package_path

if __name__ == '__main__':
    go_versions_url = 'https://raw.githubusercontent.com/actions/go-versions/main/versions-manifest.json'
    r = requests.get(go_versions_url).json()

    return_list = []
    golang_package_list = []
    for item in r:
        package_url = item['files'][1]['download_url']

        for version in GOLANG_VERSION_LIST:
            if version in package_url:
                golang_package_list.append(package_url)
                a = urlparse(package_url)
                package_filename = os.path.basename(a.path)

                process_each_package(package_filename, package_url)

                item['files'][1]['download_url'] = HOST_URL + package_filename
                return_list.append(item)

    with open('versions-manifest.json', 'w') as f:
        f.write(json.dumps(return_list, indent=2))

When it finishes, the .tar.gz files are in STORE_PATH, and the working directory has a versions-manifest.json whose download URLs point at the internal host.

Fork `go-versions` and `setup-go` #

Fork both repositories. Replace versions-manifest.json in the go-versions fork with the file the script just wrote. Editing that file in the GitHub web UI is enough.

setup-go is compiled into dist/index.js. The manifest lookup in that bundle looks like:

const releases = yield tc.getManifestFromRepo('actions', 'go-versions', auth, "main");

Point it at the fork:

const releases = yield tc.getManifestFromRepo('n0vad3v', 'go-versions-forked', auth, "master");

Use it from the runner #

Workflows then call the fork instead of actions/setup-go:

- uses: n0vad3v/setup-go-forked@master
  with:
    go-version: '1.16'

Toolchain downloads then come from the internal host.

Summary #

Caching these archives is close to a one-shot job. The set of Go versions you care about does not need to be refreshed on a short cycle, so this does not have to be automated. With the toolchains and modules both served from the internal network, runner jobs spend much less time waiting on GitHub.