AWS Databricks Devops Engineer, Toronto

Title: Devops Engineer Databricks+AWS

Duration: Long term

Location: Remote /Canada

Job Description:

About the role

Client is migrating 100+ Databricks pipelines (batch and streaming) from the AWS West environment into AWS East Databricks workspace. Four Data Engineers are handling pipeline migration. This role owns the infrastructure, CI/CD, and deployment layer that underpins all of them.

What you'll own

• Design and maintain Terraform modules using the Databricks Terraform provider to provision and manage jobs, clusters, cluster policies, permissions, secret scopes, and instance profiles must-have
• Build and maintain GitHub Actions CI/CD workflows for automated Databricks job deployment across dev, staging, and production environments must-have
• Configure and manage Databricks Asset Bundles (DABs) as an IaC deployment mechanism alongside Terraform must-have
• Own AWS infrastructure supporting Databricks — IAM instance profiles, S3 bucket policies, cross-region access configurations, VPC/security group settings, and Secrets Manager integration must-have
• Manage cross-region S3 data access patterns during migration window (AWS West → East), including cross-region replication setup and cutover coordination
• Define and enforce cluster policies, autoscaling configs, and cost governance across the Databricks workspace
• Set up monitoring, alerting, and observability for pipeline infrastructure — job failure alerts, cluster health, cost anomalies
• Manage Databricks workspace configuration — Unity Catalog setup, secret scopes, workspace-level permissions, and RBAC nice to have
• Support Data Engineers with deployment issues, infrastructure debugging, and environment parity problems

Must-haves

• Terraform — specifically the databricks/databricks provider. Must have managed databricks_job, databricks_cluster, databricks_permissions, databricks_secret_scope resources in production
• GitHub Actions — built CI/CD pipelines for Databricks deployments, not just general application CI
• AWS — S3, IAM (instance profiles for Databricks clusters), Secrets Manager, VPC, cross-region access patterns
• Databricks platform knowledge — jobs, workflows, cluster types, Delta Lake basics, workspace administration
• Databricks Asset Bundles (DABs) — used for job/pipeline deployment
• Experience supporting a team of engineers — this role unblocks others, not just builds for yourself

Strong advantages

• AWS cross-region migration experience — S3 replication, IAM cross-account access, DNS/endpoint cutover
• Unity Catalog administration — workspace setup, catalog/schema permissions, external location config
• Databricks workspace-to-workspace migration — exporting/importing job configs via API, secret scope recreation
• Cost optimization on Databricks — spot instance strategy, cluster right-sizing, autoscaling policies
• Python or Bash scripting for automation and migration tooling