Senior Data Engineer (Python, Spark, Databricks)

Solvd Inc. is a rapidly growing AI-native consulting and technology services firm delivering enterprise transformation across cloud, data, software engineering, and artificial intelligence. We work with industry-leading organizations to design, build, and operationalize technology solutions that drive measurable business outcomes.

Following the acquisition of Tooploox, a premier AI and product development company, Solvd now offers true end-to-end delivery—from strategic advisory and solution design to custom AI development and enterprise-scale implementation. Our capability centers combine deep technical expertise, proven delivery methodologies, and sector-specific knowledge to address complex business challenges quickly and effectively.

We are looking for a Staff Data Platform Engineer (Python, Spark, Databricks) to join our team.
Project description

The project consists of building a scalable backend platform that runs data-driven analytical models in a controlled, automated, and auditable way. The backend must orchestrate model execution, enforce data validation, manage metadata and artefacts, and ensure full traceability of every processing step. The architecture is modular and cloud-oriented, with a strong focus on automation, versioning, and operational reliability.

• Build, maintain, and optimize data ingestion pipelines using Python, pandas, Spark, and Databricks

• Parse and transform complex Excel-based financial models into structured Delta Lake tables

• Implement data processing workflows and calculation logic in Python

• Develop and manage pipelines using Databricks Jobs, clusters, and SQL Warehouse

• Improve performance, scalability, and reliability of existing data pipelines

• Write clean, maintainable, and production-quality Python code

• Use SQL for data validation, transformation, and pipeline integration

• Senior-level engineer with 5–8+ years of experience

• Strong Python skills, with proven experience building production data pipelines

• Hands-on experience with pandas for data transformation and ingestion

• Experience working with Spark (PySpark or Scala) and Databricks

• Solid SQL skills and experience writing complex queries

• Experience designing, implementing, and maintaining data processing pipelines

• Experience working with medium to large datasets

• Strong ownership and ability to work hands-on in a data platform environment

• English fluency for daily communication with actuarial and engineering teams.

• Experience optimizing Python or Spark workloads for performance and scalability

• Experience working with Delta Lake

• Experience ingesting and transforming large Excel-based datasets

• Experience working with financial or structured modeling data

• Core: Python 3.x (pandas, numpy, pydantic, mlflow, databricks-sdk)

• Data Layer: PostgreSQL, Azure Blob Storage, Parquet

• Computation: Databricks Jobs / MLflow Projects

• Integration: NestJS API

• CI/CD & Infra: GitHub Actions, Terraform

When you join Solvd, you'll…

• Shape real-world AI-driven projects across key industries, working with clients from startup innovation to enterprise transformation.

• Be part of a global team with equal opportunities for collaboration across continents and cultures.

• Thrive in an inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.

Ready to make an impact?

If you're excited to build things that matter, champion responsible AI, and grow with some of the industry’s sharpest minds. Apply today and let’s innovate together.