How To Use Workflows

Google Workflows is a powerful orchestrator for building data pipelines and process automation. It is a useful tool to consider if you need to handle dependent sequencing of:

1. Google BigQuery Data Transfer Service jobs (E.g., syncing S3, GCS, or Redshift data)

2. Run schedule BigQuery jobs with upstream or downstream dependencies

3. Handle more precise scheduling or query chaining than BigQuery Scheduled Queries allows

4. Call Google Cloud Functions, Google Cloud Run container jobs, or make HTTP-driven requests to authenticated or unauthenticated external APIs

Google Workflows uses a familiar YAML or JSON-based workflow DSL that allows you to connect various services easily, make fast incremental improvements and schedule workflow runs from the Google Cloud UI. You can additionally interface with Workflows via the gcloud CLI tool or Google’s Python, Java, or Go SDKs. For an overview of all things Workflows, you can check out Google’s documentation here.

How do I get started and deploy my first workflow?

CTA automatically enables the use of Google Workflows in all PAD projects. We have also provided you with a service account with the required IAM permissions to run workflows in PAD (workflows-partner@<YOUR_PROJECT_ID>.iam.gserviceaccount.com).

To begin using navigate to Workflows while in the Google Console:

1. On the Workflows page, click + Create

2. Enter the name for your new Workflow. Typically, these are written in either snake or camel case. EX: `my_first_workflow` or `myFirstWorkflow`

3. Choose a region for your workflow. We recommend `us-cental1` for most cases.

4. For service account, select your workflows-partner service account.

5. Select NEXT.

6. Type your YAML or JSON code into the workflow editor.

7. Click Deploy.

You can then execute your workflow:

1. On the Workflow details page, click Execute.

2. On the Execute workflow page, click Execute.

3. In the Output pane, your workflow's results are displayed.

Alternatively, if you are more command-line inclined, you can deploy workflows using the gcloud CLI. Google provides excellent documentation on this process here.

How much does it cost?

As with all PAD usage, Google Workflows counts towards the BigQuery usage included in your PAD contract. Workflows are pay-per-use. Google charges per step, including its failures, successes, and retries. Steps are $0.01 per increment of 1,000 steps. External calls–like a call to an HTTP endpoint, like http://www.randomnumberapi.com, that is not hosted on GCP are $0.025 per increment of 1,000 steps. There is no up-front cost, no pre-provisioning, and no infrastructure to run. 

How do I configure my workflow to run on a schedule?

Once you have deployed your workflow and confirmed that it works as you want it to, you can configure a Trigger on which your workflow should run. This trigger can be either a schedule via Cloud Scheduler or an event or Pub/Sub message via EventArc. 

To set your workflow to run on a schedule while on the Workflows page in PAD:

1. Select your workflow to go to its details page

2. Click Edit

3. Select Add new trigger -> Cloud Scheduler. This will open the Create a Scheduler Job pane on the side.

4. Define the Schedule:

   1. Choose a Name for your Cloud Scheduler job. It must be unique across the jobs in the same region

   2. Choose a region for your Cloud Scheduler job; for example, us-central1.

   3. In the Frequency field, specify a time interval that you define in a unix-cron format. For example, to schedule your workflow to execute every 5 minutes, type */5 * * * *.

   4. In the Timezone list, select the time zone that Cloud Scheduler should use to interpret the schedule you provide. You can search by country.

5. Click Continue.

6. Configure the workflow execution: 

   1. In the Workflow's argument field, specify any runtime arguments to pass to your workflow before execution. Arguments must be in JSON format. For example: {"firstName":"Sherlock", "lastName":"Holmes"}. If your workflow doesn't use runtime arguments, accept the default of {}, or leave the field blank.

   2. In the Workflow's call log level list, select the level of call logging that you want to apply during the workflow's execution. 

   3. In the Service account list, select your workflows-partner service account.

7. Click Create

For instructions on how to set your workflow to run on an event trigger, you can check out Google’s documentation here.

Where can I find more information about the Workflows API and some Sample Code?

Google provides documentation on the Workflows API and standard libraries, as well as code samples for basic use cases for each of the built-in connectors.

• Workflows API and reference documentation

• Workflows code samples