Build and maintain large-scale data processing pipelines using Apache Spark (PySpark/Scala) and Python. Design ETL workflows, implement distributed data processing, use version control (Git), and work with cloud platforms (GCP/AWS/Azure).
Job Title: Data Engineer (PySpark / Scala / Python)
Location: Remote
Job Description:We are hiring a Data Engineer with strong hands-on experience in PySpark, Scala, and Python. You must have solid expertise in Apache Spark, as it will be the core technology used for building and managing large-scale data processing pipelines.
Experience with cloud platforms like Google Cloud Platform (GCP), Microsoft Azure, or AWS is a plus.
Required Skills:Strong hands-on experience with Apache Spark
Proficient in PySpark
Experience in Scala and Python
Knowledge of ETL processes and data pipeline design
Understanding of distributed data processing
Familiarity with version control tools like Git
Basic knowledge of cloud platforms (GCP, AWS, or Azure)
Experience with cloud-native data tools (e.g., Dataproc, Glue, EMR, BigQuery)
Familiarity with workflow/orchestration tools like Airflow or Cloud Composer
Experience with CI/CD for data engineering
Exposure to both structured and unstructured data
Similar Jobs
Artificial Intelligence • Cloud • Consumer Web • Productivity • Software • App development • Data Privacy
Build and operate production data pipelines and dimensional models using Spark, SparkSQL, and cloud lakehouse technologies. Own pipelines from requirements through deployment, monitoring, and iteration; improve data quality, lineage, reliability, and cost efficiency. Partner with data scientists, analysts, product managers, and engineers to support datamarts, KPIs, reporting, and analysis. Participate in business-hours on-call rotations and improve runbooks and alerting.
Top Skills:
AirflowC++DatabricksJavaKafkaKinesisMonte CarloPythonScalaSparkSparksqlSQLStructured Streaming
Financial Services
Develop and maintain secure, scalable data pipelines and production code using AWS, PySpark, Python, and ETL technologies. Extract and transform data, implement quality checks, optimize data processing workflows, support cloud data modernization, and ensure reliable data availability. Collaborate with cross-functional teams, troubleshoot technical issues, automate recurring remediation, and communicate with technical and non-technical stakeholders. The role also involves software testing, operational stability, continuous delivery, and potentially mentoring other engineers.
Top Skills:
Ab InitioAirflowAWSCi/CdData LakesDatabricksETLInformaticaJavaPysparkPythonSnowflakeSQL
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Build and operate production data pipelines supporting analytics, AI, and agentic workflows. Responsibilities include implementing canonical data models, maintaining Databricks or Snowflake platforms, monitoring reliability, responding to incidents, validating healthcare data mappings, improving performance and cost, and documenting architecture. Requires strong SQL and Python skills, cloud data platform experience, ETL/ELT orchestration expertise, and healthcare or pharmaceutical data experience.
Top Skills:
Ai/Ml WorkflowsDatabricksEltETLHedisOmopPythonSnowflakeSQL
What you need to know about the Austin Tech Scene
Austin has a diverse and thriving tech ecosystem thanks to home-grown companies like Dell and major campuses for IBM, AMD and Apple. The state’s flagship university, the University of Texas at Austin, is known for its engineering school, and the city is known for its annual South by Southwest tech and media conference. Austin’s tech scene spans many verticals, but it’s particularly known for hardware, including semiconductors, as well as AI, biotechnology and cloud computing. And its food and music scene, low taxes and favorable climate has made the city a destination for tech workers from across the country.
Key Facts About Austin Tech
- Number of Tech Workers: 180,500; 13.7% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Dell, IBM, AMD, Apple, Alphabet
- Key Industries: Artificial intelligence, hardware, cloud computing, software, healthtech
- Funding Landscape: $4.5 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Live Oak Ventures, Austin Ventures, Hinge Capital, Gigafund, KdT Ventures, Next Coast Ventures, Silverton Partners
- Research Centers and Universities: University of Texas, Southwestern University, Texas State University, Center for Complex Quantum Systems, Oden Institute for Computational Engineering and Sciences, Texas Advanced Computing Center
