Staff Software Engineer, Visibility

Staff Engineer – Cloud Data Store (Visibility)

Temporal is an open-source programming model that can simplify code, make applications more reliable, and help developers focus on the important things like delivering features faster. We are on a mission to be the reliable foundation of every developer's toolbox, and are building the team that will make that happen.

Our values guide us — they are present in how we show up, make decisions, and work together to make an impact. We're curious, driven, collaborative, genuine, and humble.

Temporal is growing and we are looking for those who share our values, challenge "standard" thinking, and want to influence our future. If you have a passion for improving the developer experience, building world-class open-source software and communities, and want to be part of our amazing team, we'd love to hear from you!

⸻

Summary

Cloud Data Store (CDS) owns the storage, retrieval, and lifecycle of all workflow data at planet scale. We design the persistence APIs, build storage abstractions that run across cloud vendors, and deliver the observability that lets customers trust their state machines for years.

[To see more detail re: the Temporal CDS Eng team, click here [new window]

A core part of CDS is Temporal Visibility: the system that powers workflow listing, filtering, search, and observability across massive, long-lived state machines. Visibility has demanding characteristics — high write throughput, complex secondary indexing, low-latency queries, strict correctness guarantees, and the need to operate continuously while schemas and backends evolve.

As a Staff Engineer on CDS, you will take a leading role in a ground-up rewrite of the Visibility persistence layer, with the goal of making it dramatically more performant, scalable, and operable. This includes evaluating and driving deep integrations with modern analytical and search data stores such as ClickHouse and Elasticsearch, with a strong preference for self-hosted operation (while supporting managed offerings where appropriate).

You will design, build, and operate significant portions of our backend for highly scalable, multi-tenant services, and own large technical initiatives end-to-end — from initial design through live data migration, rollout, and long-term operational stewardship.

⸻

What You'll Do

Re-architect Temporal Visibility at scale
    •    Lead the design and implementation of a new persistence layer for Temporal Visibility, informed by its real-world access patterns (high-volume writes, time-based queries, filtering, sorting, and pagination across long-running workflows).
    •    Evaluate and select the most appropriate storage technologies (e.g., ClickHouse, Elasticsearch, or complementary systems), clearly articulating tradeoffs around indexing models, consistency, cost, latency, and operational complexity.
    •    Design schemas, APIs, and query models that make Visibility both powerful and intuitive for customers.

Deliver safe, large-scale data migrations
    •    Plan and execute online migrations of live Visibility data from existing persistence stores to new backends, at scale and without customer downtime.
    •    Design dual-write, backfill, validation, and cutover strategies that prioritize correctness, observability, and rollback safety.
    •    Build tooling and automation to validate data integrity and performance throughout the migration lifecycle.

Own performance, reliability, and operability
    •    Define and own SLOs for Visibility storage and query paths.
    •    Profile hot paths, design benchmarks, and lead systematic performance tuning efforts.
    •    Build operational playbooks, dashboards, and alerting that make the system understandable and debuggable for on-call engineers.
    •    Lead incident reviews and reliability improvements related to persistence and indexing systems.

Technical leadership and collaboration
    •    Break down large, ambiguous roadmap initiatives into concrete, executable phases.
    •    Author and steward design docs and RFCs through review with peers and stakeholders.
    •    Mentor and unblock other engineers working in the persistence and storage domain.
    •    Partner closely with Server, Cloud, and Developer Experience teams to land features end-to-end.

⸻

What You'll Bring

• 5 or more years of experience as an "Arranger" and/or "Builder/Enhancer" of highly scalable distributed systems. see HERE for more info re: "Arranger" and/or "Builder/Enhancer".
• Strong computer science fundamentals in distributed systems, including concurrency, consistency models, and failure modes
• Significant experience writing and operating concurrent production systems in Go, Java, or similar languages, at a high-end intermediate to expert level
• Experience writing concurrent code in production with languages like Go or Java or other applicable languages with skill level as "high end of Intermediate" and/or "Advanced" or "Expert" levels. see HERE for more info re: "high end of Intermediate" and/or "Advanced" or "Expert levels"
• Hands-on experience designing, operating, and tuning ClickHouse and/or Elasticsearch, ideally in self-hosted environments (managed services are a strong plus).
• Experience building and running services on AWS. Bonus: Azure and/or GCP experience.
• Demonstrated ability to lead large, multi-quarter technical initiatives, especially those involving core data infrastructure and live data migrations.

⸻

Nice to Have
    •    Prior contributions to Temporal, Cadence, or other workflow engines.
    •    Deep expertise in storage internals (e.g., columnar stores, LSM trees, inverted indexes, transactional logs).
    •    Experience operating multi-region services with ≥99.99% uptime.
    •    Strong background in operating and evolving Open Source systems.
    •    Experience building Kubernetes controllers and/or CRDs.

⸻