Lead the development of data pipelines and transformations in Azure Databricks, converting Scala programs to PySpark while leveraging various Azure technologies.
Job Description :
Location - US (remote) . East coast preferable
Primary Skill :
Technical Lead -Data Engneering
- Azure databricks- Build Data Pipelines using ADB-Leveraging Python and Spark
- Build Data Transformation in ADB
- Convert Scala Pgms to Py Spark Pgms
- Understand the SCALA program in DataBricks
- ADF
- Secondary Skills:
- Azure data lake gen 2
- Synapse analytics (dedicated SQL pool)
- Exp in delta Lake
- Pyspark
Nice to Have : Tidal Scheduler , Exp in Healthcare Domain
Similar Jobs
Information Technology • Consulting
The Big Data Lead will implement ETL pipelines, ensure data integrity, troubleshoot PySpark applications, and integrate with existing frameworks while leading a team.
Top Skills:
HadoopHiveKafkaPysparkPythonSparkSQL
Information Technology • Consulting
The Big Data Lead will manage data projects utilizing technologies like Amazon Redshift, Azure Data Factory, and Apache Spark to optimize data processes.
Top Skills:
Amazon RedshiftSparkAzure Data FactoryDatabricksGoogle Cloud PlatformHadoopHiveScalaSnowflake
Information Technology • Consulting
The Big Data Lead will manage database development, ETL/ELT processes, and data warehousing, optimizing performance and ensuring data pipelines work reliably.
Top Skills:
AWSAws GlueAws S3AzureAzure BlobAzure Data FactoryAzure DevopsGitJenkinsOracleSnowflakeSQLTalendTeamcity
What you need to know about the Austin Tech Scene
Austin has a diverse and thriving tech ecosystem thanks to home-grown companies like Dell and major campuses for IBM, AMD and Apple. The state’s flagship university, the University of Texas at Austin, is known for its engineering school, and the city is known for its annual South by Southwest tech and media conference. Austin’s tech scene spans many verticals, but it’s particularly known for hardware, including semiconductors, as well as AI, biotechnology and cloud computing. And its food and music scene, low taxes and favorable climate has made the city a destination for tech workers from across the country.
Key Facts About Austin Tech
- Number of Tech Workers: 180,500; 13.7% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Dell, IBM, AMD, Apple, Alphabet
- Key Industries: Artificial intelligence, hardware, cloud computing, software, healthtech
- Funding Landscape: $4.5 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Live Oak Ventures, Austin Ventures, Hinge Capital, Gigafund, KdT Ventures, Next Coast Ventures, Silverton Partners
- Research Centers and Universities: University of Texas, Southwestern University, Texas State University, Center for Complex Quantum Systems, Oden Institute for Computational Engineering and Sciences, Texas Advanced Computing Center
