Lead end-to-end data engineering on GCP: design scalable data platforms and architectures (lake/lakehouse/warehouse), build and optimize BigQuery/Spark pipelines, ensure data quality/security, enable ML workflows with Data Science, mentor engineers, and own delivery and performance.
We are seeking a Senior Data Engineer (Lead) to drive and own end-to-end data engineering initiatives. This role will lead all data engineering efforts, working closely with Data Science and Analytics teams to design scalable data platforms, enable advanced analytics, and support machine learning use cases.
The ideal candidate will bring deep expertise in cloud data engineering (GCP), strong data modeling capabilities, and proven experience in leading enterprise-grade data solutions.
Responsibilities- Lead and manage end-to-end Data Engineering delivery across projects and initiatives
- Act as the primary technical owner for data pipelines, architecture, and platform design
- Mentor and guide a team of data engineers, ensuring best practices and coding standards
- Design, build, and optimize scalable data pipelines on GCP
- Define and implement modern data architectures (data lake, lakehouse, warehouse)
- Ensure high performance, reliability, and data quality across pipelines
- Partner closely with Data Science teams to enable ML/AI workflows
- Translate business and modeling requirements into optimized data structures
- Support feature engineering, model training, and deployment pipelines
- Design logical and physical data models for analytics and ML use cases
- Implement dimensional modeling (Star/Snowflake schemas) and data vault where applicable
- Optimize datasets for performance, scalability, and usability
- Build and manage solutions using GCP services such as: BigQuery, Cloud Composer (Airflow), Cloud Storage, Dataproc
- Ensure security, governance, and cost optimization on GCP
Required Qualification and Skills:
- 8+ years of experience in Data Engineering, with leadership experience
- Strong expertise in GCP ecosystem and services
- Proficiency in SQL, Python, and/or Scala
- Hands-on experience with ETL frameworks and distributed processing
- Solid experience in Dimensional modeling, Data warehousing concepts, Data structures for ML/analytics
- Experience with Apache Spark and Real-time and batch processing frameworks
- Experience working with cross-functional teams (Data Science, Analytics, Business)
- Proven ability to lead, mentor, and drive delivery
- Strong ownership mindset with leadership capabilities
- Excellent problem-solving and architectural thinking
- Ability to operate in a fast-paced, collaborative environment
Preferred Qualifications:
- Experience in ML data pipelines / feature stores
- Knowledge of data governance, lineage, and quality frameworks
- Exposure to healthcare/payor domain (nice to have)
- Certifications in GCP (Professional Data Engineer)
Similar Jobs
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Lead enterprise AI transformation and architecture engagements, advising C-suite on AI strategy, designing end-to-end AI and data architectures (RAG, knowledge graphs, data catalogs), defining governance and responsible AI frameworks, and driving adoption, enablement, and practice development across ServiceNow implementations.
Top Skills:
Agent-To-Agent (A2A)Agentic WorkflowsAi AgentsAi Control TowerApp EngineCsmFsmGenerative Ai Skill KitHrsdItsmKnowledge GraphsModel Context Protocol (Mcp)Now AssistRdfRetrieval-Augmented Generation (Rag)ServicenowSparql
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Lead and set technical direction for a cloud-native platform across multiple teams, solving multi-cluster, multi-cloud, control-plane, workload isolation, identity, and reliability problems. Drive architecture decisions, build critical control-plane components, mentor senior engineers, and influence hyperscaler strategy and platform standards at scale.
Top Skills:
AksAWSAzureCni (Container Network Interface)CrossplaneEksGCPGitopsGkeGoInfrastructure-As-CodeKata ContainersKubernetesKubernetes OperatorsMetricsOci BundlingService MeshSlosSpiffeSpireTracing
Artificial Intelligence • Cloud • HR Tech • Information Technology • Productivity • Software • Automation
Design, build, and maintain multicloud infrastructure and CI/CD for CPQ microservices. Develop and test Python applications, deploy containers on Docker/Kubernetes, manage GCP/Azure/AWS environments, integrate AI/ML microservices, and ensure security, monitoring, and operational reliability. Mentor engineers and provide production support for SaaS CPQ solutions.
Top Skills:
Ai/MlAWSAzure Government Community Cloud (Gcc)DockerGitGithub ActionsGoogle Cloud Platform (Gcp)KubernetesAzurePostgresPython
What you need to know about the Austin Tech Scene
Austin has a diverse and thriving tech ecosystem thanks to home-grown companies like Dell and major campuses for IBM, AMD and Apple. The state’s flagship university, the University of Texas at Austin, is known for its engineering school, and the city is known for its annual South by Southwest tech and media conference. Austin’s tech scene spans many verticals, but it’s particularly known for hardware, including semiconductors, as well as AI, biotechnology and cloud computing. And its food and music scene, low taxes and favorable climate has made the city a destination for tech workers from across the country.
Key Facts About Austin Tech
- Number of Tech Workers: 180,500; 13.7% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Dell, IBM, AMD, Apple, Alphabet
- Key Industries: Artificial intelligence, hardware, cloud computing, software, healthtech
- Funding Landscape: $4.5 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Live Oak Ventures, Austin Ventures, Hinge Capital, Gigafund, KdT Ventures, Next Coast Ventures, Silverton Partners
- Research Centers and Universities: University of Texas, Southwestern University, Texas State University, Center for Complex Quantum Systems, Oden Institute for Computational Engineering and Sciences, Texas Advanced Computing Center
