Senior DevOps Engineer

ABOUT THE ROLE 0G is hiring its first dedicated DevOps engineer to own our cloud platform end to end: infrastructure, deployments, networking, security hygiene and cost. Today that work is spread across several developers. Our platform runs on five providers: Google Cloud, Alibaba Cloud, AWS, Cloudflare and Vercel. It includes: - Validator, seed, RPC, and archive nodes for our mainnet and testnet - Confidential-computing (TEE) workloads, GPU machines and storage nodes - Kubernetes clusters and managed databases - A global RPC load balancer at the edge, and our web apps All together, that is more than 200 virtual machines. You will be the person who knows what runs where, why it runs there, and what it costs. WHAT YOU'LL OWN You own the platform, from the network edge to the bill. - Cloud operations: provisioning, upgrades, capacity and the full resource lifecycle on Google Cloud, Alibaba Cloud and AWS. That includes rightsizing and releasing what nobody uses. - Kubernetes: run, upgrade and scale our clusters. Keep them on supported versions and design node pools, autoscaling and Spot capacity. - Blockchain node fleet: operate validator, sentry, seed, RPC and archive nodes, including peering topology, snapshots, sync and client upgrades. - Networking and edge: VPC design, load balancers, DNS, Cloudflare Workers, Tunnels and WAF, and keeping internet egress deliberate and cheap. - Infrastructure as code and CI/CD: every environment defined in Terraform or Pulumi. You design and run CI/CD on Jenkins or an equivalent such as GitHub Actions or GitLab CI, covering builds, tests, security scans and releases from commit to production, with no manual steps. - Observability and on-call: metrics, logs, alerts and dashboards, including control of metric cardinality. You also run incident response and write the postmortems. - Security and access: least-privilege IAM across every provider, secrets management, and no databases or admin ports open to the internet. - Cost management: billing exports, budgets and alerts, tagging, a monthly cost review and commitment planning. - Developer enablement: templates, preview environments, documentation and support for the engineers who ship on the platform. OUR STACK Kubernetes: GKE, EKS, Helm, Ingress and Gateway API, autoscaling, RBAC, network policies Networking: TCP/IP, subnets and CIDR, routing, NAT, L4 and L7 load balancers, DNS, TLS, firewalls, VPC peering Google Cloud: Compute Engine, GKE, Cloud SQL, Cloud Run, IAM, Cloud Monitoring, billing export to BigQuery Infrastructure as code: Terraform, Pulumi CI/CD and containers: Jenkins or an equivalent (GitHub Actions, GitLab CI), Argo CD, Docker and BuildKit, container registries Observability: Prometheus, Grafana, Cloud Monitoring, CloudWatch, Alibaba CloudMonitor Linux: Systems administration, systemd, disks and filesystems, performance debugging Alibaba Cloud: ECS, VPC, NLB and ALB, ESSD disks, RDS, PolarDB, SAE, Cloud Firewall AWS: EC2, EKS, RDS, ALB and NLB, Route 53, S3, Organizations Cloudflare: DNS, Workers, Tunnels, WAF and rate limiting, Zero Trust Databases (operations): PostgreSQL, MySQL, MongoDB, managed Kafka Blockchain nodes: CometBFT-based chains, reth and other EVM clients, P2P networking Vercel: Projects, builds, ISR, team access Confidential computing: Intel TDX virtual machines on Google Cloud and Alibaba Cloud MUST-HAVES - We want 5+ years in DevOps, SRE or platform roles, including 3+ years running production Kubernetes and owning CI/CD pipelines. Kubernetes, in depth - Upgrading control planes and node pools with no downtime, and keeping clusters on supported versions. - Scheduling: requests and limits, affinity, taints and tolerations, priority, and eviction under node pressure. - Autoscaling with the HPA, cluster autoscaler or Karpenter, plus Spot or preemptible node pools for non-critical work. - Networking inside the cluster: CNI, Services, Ingress or Gateway API, network policies, and DNS within the cluster. - Security: RBAC, Pod Security, workload identity, and secrets that never sit in plain environment variables. - Stateful workloads: StatefulSets, persistent volumes and CSI drivers, and backup and restore. - Packaging and delivery with Helm or Kustomize, ideally through GitOps with Argo CD or Flux. - Debugging a failing pod, node or rollout from events, logs and metrics. CI/CD, in depth - Pipeline design: you have built and run production CI/CD on Jenkins or an equivalent (GitHub Actions, GitLab CI, CircleCI, Tekton). Pipelines are defined as code in a Jenkinsfile or workflow YAML, with shared libraries and reusable templates. - CI platform operations: running Jenkins controllers with ephemeral agents on Kubernetes, or self-hosted runners. That includes plugin and version upgrades, credential handling, and scaling build capacity. - Build pipelines: multi-service and monorepo builds with caching and parallel stages, and versioned artifacts. Container images built with Docker, BuildKit or Kaniko, including multi-architecture images, image scanning, SBOMs and signing. - Deployment pipelines: promotion from dev to staging to production with approvals and quality gates, plus blue-green, canary and fast rollback. GitOps with Argo CD or Flux, and database migrations as part of a release. - Infrastructure pipelines: Terraform or Pulumi plan and apply from CI, with policy checks and drift detection. - Pipeline security: short-lived credentials through OIDC federation or a secrets vault instead of long-lived keys, least-privilege deploy roles, and protected branches. - Release engineering for our chain: building and publishing node binaries and images, and coordinating rolling upgrades across validator and RPC fleets. - Delivery health: you track build time, build cost, flaky tests and DORA metrics (deployment frequency, lead time, change failure rate, time to restore), and you cut wasted builds with ignore rules and caching. Networking fundamentals - TCP/IP, subnetting and CIDR, routing, NAT, and the difference between L4 and L7 load balancing. - DNS, TLS and certificates, firewalls and security groups. - Private connectivity: VPC design, peering, private endpoints, and keeping internal traffic off public IPs. - Troubleshooting with dig, curl, mtr and tcpdump, and understanding what internet egress costs. Also required - Production experience on at least two of Google Cloud, AWS and Alibaba Cloud, and willingness to learn the third quickly. - Terraform or Pulumi in production, plus scripting in Bash and in Python or Go. - A track record of cutting a cloud bill with data, through rightsizing, cleanup or commitments. - A security-first habit with production credentials, and clear written communication. NICE TO HAVE - Running blockchain nodes in production, such as CometBFT validators and sentries, EVM archive or RPC nodes, and snapshot pipelines. - Cloudflare Workers, Tunnels and Zero Trust at scale. - Alibaba Cloud on the international site, including its billing and support model. - Confidential computing (Intel TDX) and GPU workloads. - CKA or CKS certification, a FinOps certification, or experience moving a team between CI systems, such as Jenkins to GitHub Actions. - Go or Rust, enough to read and patch the services you operate. WHAT WE OFFER - Purpose: The opportunity to contribute to making AI a public good - Growth: A self-directed environment where you can take initiative to shape your role and career - Compensation: Market-competitive compensation including, for most roles, exposure to pre-launch tokens In addition, 0G Labs is committed to the health and well being of all of our team members. To that end, we provide reimbursements towards a holistic set of experiences and courses: - Core self: Transcendental Meditation - Mind: Landmark Education https://www.landmarkworldwide.com - Emotion: Art of Communication - Presence: Speech Coach - Body: Fitness, gym and exercise memberships/classes