Lead architecture, implementation, and optimization of scalable data platforms, ETL/ELT pipelines, and data governance. Ensure security and compliance, mentor junior engineers, drive CI/CD and IaC for data workflows, and collaborate with stakeholders to deliver actionable analytics and reporting.
Insomniac Design is a global digital agency headquartered in Washington D.C., with offices in London, Bucharest and Chisinau. We’re an agile, determined and innovative team organized by functional areas of expertise — Creative, Technology, Strategy, and Management. We specialize in human-centered design with deep focus on design thinking and digital transformation.
At Insomniac, we leverage AI to free our teams from routine tasks so they can focus on the aspects of our functions that are most valuable. We thrive on creative problem-solving, collaboration, and innovation. By thoughtfully integrating AI into our workflows, we’re not only improving productivity but also ensuring our people have the tools to do their best work. This empowers us to build smarter solutions and deliver stronger results for our clients.
As a Senior Data Engineer, you will architect, build, and optimize scalable data platforms that empower our clients with unified, actionable insights. You will lead the design of robust data pipelines, enforce best practices in data governance and security, and mentor junior engineers. Your expertise will bridge technical execution and strategic vision, ensuring high-performance, secure, and scalable data solutions that drive business impact.
Technical Leadership & Strategy:
- Architect and implement enterprise-grade data platforms, including data lakes, warehouses, and real-time pipelines.
- Define and enforce data engineering standards, including ETL/ELT frameworks, data modeling, and quality controls.
- Lead performance optimization of databases, queries, and pipelines (e.g., partitioning, indexing, cost-efficient cloud solutions).
- Stay updated with emerging trends, evaluating and integrating new technologies.
Data Architecture & Infrastructure:
- Design and implement scalable data architecture that ensures high availability, security, and performance.
- Define best practices for data modeling, storage, retrieval, and governance.
- Optimize database structures, schemas, indexing, and partitioning for efficient query performance.
- Implement CI/CD for data pipelines and automate infrastructure using Infrastructure-as-Code (IaC) tools.
Data Pipelines & ETL Development:
- Architect and develop end-to-end ETL/ELT pipelines for structured and unstructured data ingestion, processing, and transformation.
- Automate and optimize data workflows using tools like Apache Spark, Kafka, Airflow, and dbt.
- Monitor, troubleshoot, and optimize data pipeline performance.
Data Governance, Security & Compliance:
- Implement data governance policies, including metadata management, lineage tracking, and role-based access control (RBAC).
- Ensure compliance with data privacy regulations (e.g., GDPR, CCPA) and enforce security best practices.
- Develop frameworks for data quality validation, anomaly detection, and integrity checks.
Business & Stakeholder Engagement:
- Collaborate with stakeholders (clients, PMs) to translate business needs into technical specifications and scalable architectures.
- Develop interactive dashboards and reporting solutions to provide actionable insights.
Innovation & Mentorship:
- Lead technical discussions, workshops, and knowledge-sharing sessions within the team.
- Mentor junior engineers, conduct code reviews, and set best practices.
- Drive automation of data workflows, including CI/CD for pipelines and Infrastructure-as-Code (IaC) solutions.
Technical Skills & Experience:
- 7+ years of experience in data engineering with a proven track record of building scalable data solutions.
- Expertise in SQL and relational database management systems (MySQL, PostgreSQL, Oracle).
- Proficiency in Python (or similar languages) for data processing and automation.
- Deep understanding of cloud platforms (AWS, GCP, Azure) and cloud-native data services:AWS (Redshift, Glue, S3)Azure (Synapse, Data Factory)GCP (BigQuery, Dataflow)
- Hands-on experience with big data tools:Apache Spark, Kafka, Airflow, dbt, Snowflake, Databricks
- Expertise in data modeling.
- Strong knowledge of data security, governance, and compliance frameworks.
Leadership & Soft Skills:
- Proven track record of leading data projects, including migrations, warehousing, and real-time analytics.
- Ability to lead cross-functional teams and manage client expectations.
- Strong problem-solving skills for debugging distributed systems and resolving data anomalies.
- Experience with Agile/Scrum methodologies, including tools like Jira, Git.
- Excellent communication and stakeholder management skills.
Nice-to-Have (Bonus Skills):
- Certifications in cloud/data technologies (e.g., AWS Certified Data Analytics, Google Cloud Data Engineer).
- Experience with ML pipelines (e.g., feature stores, model deployment).
- Contributions to open-source data projects.
Insomniac Design offers a competitive salary and benefits package including health and life insurance as well as 401k contribution. Salary range: $115,000 - $150,000.
Applicant Eligibility: Please note, candidates who are eligible to work in the US without visa sponsorship are eligible to apply. We are not accepting applicants from recruiters or staffing agencies
Similar Jobs
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Design, build, and operate enterprise data pipelines and Lakehouse data products on Azure. Lead migrations, implement medallion architectures, optimize Spark and distributed SQL workloads, enforce CI/CD and observability, and support production incident response while partnering with architects, data scientists, and stakeholders.
Top Skills:
ArmAzure Ad / Entra IdAzure Data FactoryAzure Data Lake Storage Gen2Azure DatabricksAzure DevopsAzure Key VaultAzure MonitorAzure SqlAzure Synapse AnalyticsBashBicepDelta LakeKqlLog AnalyticsMicrosoft FabricPandasParquetPowershellPrivate EndpointsPysparkPytestPythonSpark SqlSsisSynapse Dedicated Sql PoolT-SqlTeradataTerraformUnity CatalogVnet
Artificial Intelligence • Cloud • Computer Vision • Hardware • Internet of Things • Software
Lead design and operation of large-scale data pipelines and data APIs. Build Spark/PySpark workflows on Databricks, optimize job performance, manage data quality and observability, and develop MCP servers and AI-agent integrations. Mentor engineers, define standards, support production incidents and on-call rotations, and collaborate with stakeholders to deliver scalable data platform products.
Top Skills:
Apache IcebergApi GatewayAWSAws LambdaAws Rds/AuroraAzureDatabricksDatadogDbtFastapiFivetranGCPGoogle BigqueryLlms/Ai AgentsMcp ServersMs Sql ServerMySQLOraclePostgresPysparkPythonS3SecretsmanagerSnowflakeSnsSparkSplunkSQLSqs
.png)
Cloud • Insurance • Payments • Software • Business Intelligence • App development • Big Data Analytics
Design, build, and optimize BigQuery-based data solutions and data models. Manage data storage, partitioning, clustering, ETL, and data quality. Collaborate with architects, data scientists, and engineers to deliver scalable, secure cloud data infrastructure and documentation; senior hires provide technical leadership, code review, and process improvement.
Top Skills:
Ansi Sql)BigQueryConfluenceGoogle Cloud Platform (Gcp)JIRAPythonSql (Bigquery Sql
What you need to know about the Austin Tech Scene
Austin has a diverse and thriving tech ecosystem thanks to home-grown companies like Dell and major campuses for IBM, AMD and Apple. The state’s flagship university, the University of Texas at Austin, is known for its engineering school, and the city is known for its annual South by Southwest tech and media conference. Austin’s tech scene spans many verticals, but it’s particularly known for hardware, including semiconductors, as well as AI, biotechnology and cloud computing. And its food and music scene, low taxes and favorable climate has made the city a destination for tech workers from across the country.
Key Facts About Austin Tech
- Number of Tech Workers: 180,500; 13.7% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Dell, IBM, AMD, Apple, Alphabet
- Key Industries: Artificial intelligence, hardware, cloud computing, software, healthtech
- Funding Landscape: $4.5 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Live Oak Ventures, Austin Ventures, Hinge Capital, Gigafund, KdT Ventures, Next Coast Ventures, Silverton Partners
- Research Centers and Universities: University of Texas, Southwestern University, Texas State University, Center for Complex Quantum Systems, Oden Institute for Computational Engineering and Sciences, Texas Advanced Computing Center
