Passing Credentials to GCP Cloud Run from a Github Action

Recently I had to automate a GCP Cloud Run deployment on commits in a Github repository. In cases like that the default solution is to define a Github action. Well not exactly.

 

The Cloud native way

In the case of Google Cloud (I can't be detailed about the others) the Cloud Build  is what's the closest to the execution environment. Using it utilizes the GCP's native containerization (Cloud Build's Docker runner), direct tagging and uploading to an Artifact Registry (formerly the Container Repository which was deprecated and is now retired) repository, and eventually that is the shortest path to Cloud Run itself.

Yet, the "cloud native" way is not always the right or the proper way.

In my case the Cloud Run application was a backend application the source for which resides in a monorepo along with the one for the accompanying frontend application. After a short research for my context I settled on the premise that the Cloud Build triggers can't distinguish if a commit comes solely for one of the applications and build and deploy everything possible in the repository tree (frontend, backend and whatever else Dockerized applications it finds in its repo scan). Not always convenient or necessary.

On the other hand the Github workflows can see not only branches but directory tree paths too. This was enough to tip the scales towards this solution - to build the application at it's source of truth and call the cloud operations from there to finish the job with deploying it (through the custom gcloud run deploy command).

 

Implementing the Action

For the basic case of building the application, authenticating with the Cloud, uploading the image and deploying it with the bare minimum of the said command there are more than enough tutorials. There are even readily available templates for the task in the Github's own workflows library. 

There are also many video tutorials with different levels of complexity. I especially liked this one:

because of it's graceful pace and to the point, grounded explanations - nothing redundant. Kudos.

Yet, my case was just a bit more complex than the minimum deployments. And it's nothing that is special. I even think almost anyone's Cloud Run use case includes that - the environment variables and the secrets.

The slightly specific sub-case arises when one needs to pass a service account credentials to the container being deployed. In the above video there is an example of defining a secret that contains such credentials but in his case is for a different purpose. These credentials are the ones authenticating the Github action (basically a runner - container with Ubuntu OS by default) with the Google Cloud Infrastructure for invoking the image upload and eventually the deployment. But I was already past that step and had the problem with passing the contents of the said secret to the container being deployed.

I won't get into details why I didn't use Workload Identity Federation, or resorted to the Base64-encoding method, let's just say it wasn't entirely my decision - I had to deal with just passing the secret.

 

Variables, Secrets and Execution Environments

Now, the variables are simple. Quick consulting with the official documentation and the commands reference it becomes clear that the --set-env-vars flag should be used - a flag per variable.

As for the secrets - they're basically the same with the main difference that one's no longer able to see their value anywhere on Github after creation. 

When it comes to putting a Service Account in a secret there's a certain gotcha that must be complied with. Let's keep the following in mind:

• In its essence a Service Account is a string in a JSON format that if kept as a local file is a multiline JSON document. 
• Github Secrets can take multiline values but preserve them in a single line string. How true is that can be arguable.

After long hours of frustration, chatbot circular hallucinations, short-lived moments of epiphany, and the mandatory existential dread I decided that I'll fight it off and make it actually work with the flags file method. 

In short the method includes the additional step (prior to executing the deployment command) that defines a YAML file on the fly and fills it with the environment variables assigning them the values of the secrets. The syntax looks like this:

Once created the file is passed to the command with the --env-vars-file flag.

Looks easy and why it didn't work out of the box cost me the longest time to figure out. It has to do with how the secrets keep the multiline values.

The telltale sign of the problem if easily visible (if you know what to look for) here:

What is seen is the Github secret actually preserves the multiline JSON format if one assigns it this way on creation/update. From here on it is the gcloud run deploy command that can't work it out this way because it can't recognize it as a proper YAML syntax.

So what could I do? The common sense suggested that when deployed from the Cloud Console the Service Accounts being passed as variables or secrets never pose this problem. 

So I refreshed all of the secrets' values with the ones taken directly from the Console (from the Edit screen but the fact that the variables are single lines can be seen also in the auto generated YAML of the deployment). 

The result in the runner's log changed to:

At this point all of the secrets being single-line strings are written as such in the generated flags file in proper YAML syntax, hence the successfully executed deployment.

Take Out

The imposing conclusion has the following dimensions:

• Keep it simple and if possible retain the whole process of the Github workflow to be visible on one page. The action and the flags file solution meet that requirement while the Base64 encoding and Workload Identity Federation methods although robust and proven in smaller use cases might be an overkill in terms of configuration and coding.
• Always double check the values of your environment variables and secrets, AND when used with the flags file make sure the values are strictly one-liners, no matter what they hold.

And probably, as a rule of thumb, check as much as needed with the Cloud Console the alignment between the two points of deployment because sometimes drift happens and may cause other hard-to-debug problems.