CentOS Question posted in
The official documentation can be found here.

replace retool with tools.go for multi developer and CI environments using go modules – CentOS

I’d like to replace retool with go modules tools.go “tools as dependencies”. However I’m struggling to understand how this works when my developers and CI env all use different operating systems.

I want to make sure each environment uses the exact same version of tools.

For a concrete example, my app requires the protoc compiler to generate go code via github.com/golang/protobuf/protoc-gen-go. I have 3 OS, all needing to execute protoc with the protoc-gen-go plugin/generator:

  1. Ryan: uses MacOS
  2. Joe: Linux (Ubuntu)
  3. CI: Linux (CentOS)

I currently use retool to make sure ALL environments are locked in on the same version of tools (protoc-gen-go in this ex):

retool do build/bin/protoc -Ibuild/protoc/include -I. rpc/platform/platform.proto --go_out=.

My new go modules / “tools as dependencies” setup

tools.go:

// +build tools

package tools

import (
    _ "github.com/golang/protobuf/protoc-gen-go"
)

Set the path that go install will use:

export GOBIN=$PWD/bin

Install:

go install github.com/golang/protobuf/protoc-gen-go

If Ryan runs the go install .. a bin/protoc-gen-go MacOS executable is created.

Questions:

  1. At this point, why isprotoc-gen-go tool version (or git hash) NOT listed in go.mod?
  2. When Joe clones the app repo, how does he get and compile the same version of protoc-gen-go that Ryan used?
  3. How does protoc know to use the protoc-gen-go executable generator in my ./bin dir?
Tags:

2

Answers


  1. 0

    Go modules work with the imports of your .go files. If they find an import, they will automatically download the latest version that satisfies your requirements. You have to read https://github.com/golang/go/wiki/Modules and understand how modules work since Go 1.11 and later.

    At this point, why is protoc-gen-go tool version (or git hash) NOT listed in go.mod?

    This is because protoc-gen-go is just an external tool as far Go modules are concerned. You don’t import golang/protobuf/tree/master/protoc-gen-go but the code it generates.

    When Joe clones the app repo, how does he get and compile the same version of protoc-gen-go that Ryan used?

    Use:

    GIT_TAG="v1.2.0" # change as needed
go get -d -u github.com/golang/protobuf/protoc-gen-go
git -C "$(go env GOPATH)"/src/github.com/golang/protobuf checkout $GIT_TAG
go install github.com/golang/protobuf/protoc-gen-go

    to install a specific version on each machine of the users. Probably write a build script that automates the process.

    How does protoc know to use the protoc-gen-go executable generator in my ./bin dir?

    From the github docs:
    The compiler plugin, protoc-gen-go, will be installed in $GOPATH/bin unless $GOBIN is set. It must be in your $PATH for the protocol compiler, protoc, to find it.

    Login or Signup to reply.
  2. 0

    I was able to accomplish vendored tools build for protoc (and plugins like Twirp) following the Go Modules tools guidelines, plus a little Makefile-Fu for the protoc binary.

    A complete working example can be found in the Aspiration Labs pyggpot repo. Following are the essential details. Worth noting: getting the import path right for some of the tools was very fiddly, but ultimately, successful.

    For protoc itself, I vendor the binary release in the Makefile and set it up into a tools/bin dir:

    TOOLS_DIR := ./tools
TOOLS_BIN := $(TOOLS_DIR)/bin

# protoc
PROTOC_VERSION := 3.7.1
PROTOC_PLATFORM := osx-x86_64
PROTOC_RELEASES_PATH := https://github.com/protocolbuffers/protobuf/releases/download
PROTOC_ZIP := protoc-$(PROTOC_VERSION)-$(PROTOC_PLATFORM).zip
PROTOC_DOWNLOAD := $(PROTOC_RELEASES_PATH)/v$(PROTOC_VERSION)/$(PROTOC_ZIP)
PROTOC := $(TOOLS_BIN)/protoc

# protoc
$(PROTOC): $(TOOLS_DIR)/$(PROTOC_ZIP)
    unzip -o -d "$(TOOLS_DIR)" $< && touch $@  # avoid Prerequisite is newer than target `tools/bin/protoc'.

$(TOOLS_DIR)/$(PROTOC_ZIP):
    curl --location $(PROTOC_DOWNLOAD) --output $@

    The PROTOC_PLATFORM string can be automated with something like OS detecting makefile. The version of that we use is at https://github.com/aspiration-labs/pyggpot/blob/master/build/makefiles/osvars.mk.

    On to building the go tools. Create a tools.go something like

    // +build tools

package tools

import (
    // protocol buffer compiler plugins
    _ "github.com/golang/protobuf/protoc-gen-go"
    _ "github.com/twitchtv/twirp/protoc-gen-twirp"
    _ "github.com/twitchtv/twirp/protoc-gen-twirp_python"
    _ "github.com/thechriswalker/protoc-gen-twirp_js"
)

    Note: the // +build tools tag will keep go build from over-building tools imports in your final build.

    Finally, some make code to build your go tools:

    # go installed tools.go
GO_TOOLS := github.com/golang/protobuf/protoc-gen-go 
            github.com/twitchtv/twirp/protoc-gen-twirp 
            github.com/twitchtv/twirp/protoc-gen-twirp_python 
            github.com/thechriswalker/protoc-gen-twirp_js 

# tools
GO_TOOLS_BIN := $(addprefix $(TOOLS_BIN), $(notdir $(GO_TOOLS)))
GO_TOOLS_VENDOR := $(addprefix vendor/, $(GO_TOOLS))

setup_tools: $(GO_TOOLS_BIN)

$(GO_TOOLS_BIN): $(GO_TOOLS_VENDOR)
    GOBIN="$(PWD)/$(TOOLS_BIN)" go install -mod=vendor $(GO_TOOLS)

    And finally, a make setup target to run go mod vendor and process the targets above.

    setup: setup_vendor $(TOOLS_DIR) $(PROTOC) setup_tools

# vendor
setup_vendor:
    go mod vendor

$(TOOLS_DIR):
    mkdir -v -p $@
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search