Google Cloud Functions And Private Repos

Is it possible to use a private repo for a Google Cloud function? Yes, but…

Go Modules

Google Cloud Functions have support for Go modules. You can either use the traditional goroot or use a module. Whichever way you go, if you use any kind of library based on a private repo, Google Cloud Build will not be able to download the required package(s).

Vendor Directory Workaround

To get module support with private repos, you need to fall back on an older dependency system, the Vendor folder. By running go mod vendor, Go will produce a vendor folder containing all the source code for your dependencies. We can use this to pass the required dependencies to Cloud Build.

The vendor directory will not typically include code from your own repo — this can be a problem as if your repo is private, you will encounter the same issue as you would with dependencies — Cloud Build will be unable to retrieve the latest version of your own source code.

To get your application code into the vendor folder, use this neat trick.

1. Change your package name to something localised
2. Refer to any other packages within your repo with the full github path
3. The vendor system will now see your source code as a library and include it in the vendor folder

Here’s an example — imagine this is hosted at github.com/myuser/myproject

package myproject
func init() {
    ...
}

And then in your go.mod

module myproject
go 1.14
replace github.com/myuser/myjproject => ./

So you are effectively replacing your github project with the local directory. And naming your package myproject instead of github.com/myuser/myproject makes clear that your github source code is an external library to be included in the vendor directory.

Build Steps

Once you have set up the vendor directory workaround, you need to gather together the right files in the right places for Google Cloud Build. Here’s a formula I use in my Makefiles

.PHONY: build
build:
    @mkdir tmp
    @cp *.go tmp
    @cp .gcloudignore tmp/    
    @go mod vendor
    @mv ../vendor tmp/vendor
    @cd tmp; gcloud functions deploy mytestfunction
    --project mygcpproject --max-instances 3
    --entrypoint myproject.EntryPoint
    --env-vars-file env.yaml
    --memory 128MB
    --runtime go113
    --trigger-http
    --allow-unauthenticated
    --region europe-west2
    --timeout 5402

The key part above is that we are generating our vendor folder and then moving that plus all our source code into a tmp folder which we will present to Google Cloud Builder. The gcloud functions deploy command essentially makes an archive of the directory it is in and sends that off to Cloud Builder to be processed into a Cloud Function.

Important: don’t miss the step which copies the .gcloudignore file — the docs for Google Cloud Functions state that (in this order of precedence) if you have a go.mod file, modules will be used. If you have a vendor folder, that will be used. You need to make sure your go.mod file does not form part of your submission. Put these lines into your .gitignore:

go.mod
go.sum
.git
.gitignore

If you follow these steps, you should be able to successfully build Cloud Functions with one or more private repos. At the start of the article, I said Yes, but… There’s always a but.

If you follow the fairly standard Go directory layout of having an internal folder to hold your core code, this workaround will fail. A folder named internal is a special case. Internal is a reserved folder name. Folders called internal may not be imported by another package. And if you have used your go.mod to indicate your source code is in effect an external dependency, Cloud Builder will refuse to make use of the code in your internal folder. The solution? Well there is no solution to be honest — at this point you should really consider moving to Cloud Run instead.