Design and build scalable data pipelines and distributed processing systems on Databricks. Lead technical design, develop advanced data models, champion CI/CD and IaC, ensure operational reliability, mentor engineers, and collaborate cross-functionally to support analytics and ML use cases.
Summary
As a Senior Data Engineer, you will design, build, and support the core systems that power our data platform to enable fast, data-driven decisions. You will create scalable data pipelines, self-service tools, and governance solutions that ensure trusted, accessible data across the organization. Working closely with business partners and the Data Platform team, you will support advanced analytics and machine learning use cases while sharing knowledge to elevate the team.
This role emphasizes scalable pipeline development, distributed data processing, strong data modeling, and engineering best practices. Success requires curiosity, experimentation, and a commitment to operational reliability and team mentorship.
- Design and build scalable data pipelines to ingest, transform, and curate data from APIs, databases, files, and event streams.
- Lead technical design reviews and translate complex business needs into enterprise-grade data solutions.
- Develop and optimize advanced data models (dimensional, data vault, domain-driven, canonical) to support analytics, BI, and productized datasets.
- Champion SDLC best practices, continuous delivery, and infrastructure automation using CI/CD and Infrastructure as Code.
- Optimize complex distributed workloads using SQL, Python; mentor others on tuning and scalable design patterns.
- Build reusable data frameworks, libraries, and reference architectures to accelerate team productivity and platform adoption.
- Perform root-cause analysis for major data incidents, lead long-term remediation, and drive operational reliability improvements.
- Provide technical mentorship, guide code reviews, and help shape engineering capability maturity.
- Collaborate with Architects, Data Leads, Product Owners, and cross-functional engineering teams to define long-term data strategies.
- Perform other duties as assigned.
- 5 to 7+ years of experience in data engineering or a related technical field.
- Expertise in SQL and advanced proficiency in at least one programming language, Python preferred.
- Strong experience designing and tuning distributed data processing systems at scale.
- Proven experience designing and implementing complex data models across multiple business domains.
- Strong knowledge of version control, CI/CD, DevOps/DataOps, automated testing, and engineering best practices.
- Ability to lead cross-functional engineering initiatives and influence technical roadmaps.
- Strong problem-solving, debugging, and analytical skills in complex, multi-system environments.
- Extensive hands‑on experience building scalable pipelines and workflows in Databricks (Delta Lake, Spark, Unity Catalog, Jobs, Workflows).
- DataOps experience (pipeline observability, monitoring, automated quality).
- Knowledge of metadata management or cataloging platforms (Purview, Collibra, Alation).
- Experience with streaming frameworks used with Spark Structured Streaming (Kafka, Event Hubs, Kinesis).
- Experience working in an Agile environment.
Engagement & Logistics
- Engagement Length: 12 months or more
- Time Zone: CST (9am - 5pm)
- Laptop: BYOD.
- Overtime Required: Very unlikely. In the unlikely event that they ask for some "on call" hours during the week the OT rate is 1.5x the regular rate
Selection process
- Meeting with Resilient Co. team.
- Client technical interview
- Project interview (Cultural fit and tech review)
- Final stage
Similar Jobs
Information Technology
Design, build, and maintain scalable cloud data pipelines and warehouses (primarily Snowflake and BigQuery). Optimize ETL/ELT, enable analytics/BI and ML use cases, implement governance/security, collaborate with distributed engineering and analytics teams, document lineage and metadata, and monitor and troubleshoot pipeline reliability.
Top Skills:
Apache AirflowBigQueryContentfulDagsterDatabricksDbtLookerPower BIPythonSegmentSnowflakeSnowflake CortexSnowparkSnowproSQLTableauTerraform
Fintech • Information Technology
Design, build, and operate Alpaca's scalable data platform and lakehouse infrastructure. Build low-latency streaming and CDC ingestion, batch pipelines landing in Iceberg, and tooling for BI and AI consumers. Manage IaC, Kubernetes deployments, monitoring/incident response, and partner with DevOps and analytics teams to meet growing data requirements.
Top Skills:
AirbyteAirflowAnsibleApache IcebergApache RangerArgocdCloud BuildCloud SqlCubeDatahubDataprocDbtDebeziumDockerGcsGoogle Cloud PlatformHelmHightouchKafkaKubernetesLookerOpenmetadataPrestoPythonRedpandaSQLTerraformTrino
Artificial Intelligence • Information Technology • Software • App development
Lead design and implementation of scalable data pipelines and cloud data warehousing solutions. Partner with stakeholders to translate requirements, build and optimize complex SQL, perform root-cause analysis, and deliver dashboards and reports for end-user analytics.
Top Skills:
C++JavaPower BIPythonRelational DatabasesScalaSnowflakeSQLTableau
What you need to know about the Austin Tech Scene
Austin has a diverse and thriving tech ecosystem thanks to home-grown companies like Dell and major campuses for IBM, AMD and Apple. The state’s flagship university, the University of Texas at Austin, is known for its engineering school, and the city is known for its annual South by Southwest tech and media conference. Austin’s tech scene spans many verticals, but it’s particularly known for hardware, including semiconductors, as well as AI, biotechnology and cloud computing. And its food and music scene, low taxes and favorable climate has made the city a destination for tech workers from across the country.
Key Facts About Austin Tech
- Number of Tech Workers: 180,500; 13.7% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Dell, IBM, AMD, Apple, Alphabet
- Key Industries: Artificial intelligence, hardware, cloud computing, software, healthtech
- Funding Landscape: $4.5 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Live Oak Ventures, Austin Ventures, Hinge Capital, Gigafund, KdT Ventures, Next Coast Ventures, Silverton Partners
- Research Centers and Universities: University of Texas, Southwestern University, Texas State University, Center for Complex Quantum Systems, Oden Institute for Computational Engineering and Sciences, Texas Advanced Computing Center
