Lead Data Engineer
BigCommerce is disrupting the e-commerce industry as the SaaS leader for fast- growing, mid-market businesses. We enable our customers to build intuitive and engaging stores to support every stage of their growth.
We are looking for a Lead Data Engineer to join our Data Engineering team. Leveraging your big data skills, you will help design and build a Realtime, centralized data distribution service for our core business systems to achieve faster and bigger results. This involves deciphering value from several systems with a variety of access patterns. You will report to Sr. Director of IT and work with cross-functional teams in a highly visible role. The Senior Data Engineer will mentor junior engineers and communicate with senior leadership.
What you’ll do
- Build scalable near real time data pipelines on AWS and GCP that collects, transforms, loads and curates data from various internal and external data sources
- Build a scalable distributed data store that will be central source of truth
- Create scalable and low latency solutions
- Implement machine learning pipelines and algorithms
- Evaluate new technologies and build prototypes for continuous improvements in Data Engineering
- Contribute to setting standards and development principles for the team and influence across the larger organization
Quality Skills
- Write automated unit, integration and acceptance tests to support our continuous integration pipelines
- Build performance and load tests written from scalability and resiliency standpoint
- Participate in peer code reviews and advocate for the best coding practices and principles
- Partners with Infrastructure and Engineering teams to ensure instrumentation, logging and monitoring is in place
Team / Leadership Skills
- Actively participate with our development team in all phases of the software development lifecycle, including requirements gathering, functional and technical design, development, testing and roll-out, and support
- Estimate work items and help maintain a scrum backlog
- Lead collaboration across different teams
- Distill and present research findings to both technical and non-technical leaders
- Plays a key role in defining data engineering strategy and communicating priorities to others
- Coach and mentor others, models the company values
- Partners with engineering management to determine hiring needs and helps with recruiting activities
Who you are
- BS or MS in Computer Science or equivalent experience or field
- 8+ years developing data and software solutions
- 4+ years of experience in Java, Scala and/or Python
- 4+ years of experience in building data pipelines in the cloud with tools like S3, Hadoop, Hive; or their equivalents
- 4+ years of experience in streaming technology such as Kafka
- Strong experience with AWS and/or GCP
- Strong experience with SQL, ETL, Data Warehousing and Data Orchestration
- Experience with scheduling frameworks, preferably Airflow
- Experience with monitoring near real time scalable data pipelines
- Experience with of all aspects of data systems including database design, ETL, aggregation strategy, performance optimization, and technology trade-offs
- Experience with Terraform, Puppet, and Jenkins
- Experience with shell scripting
- Focused developer with a strong sense of ownership
- Ability to independently drive projects consisting of many stories from inception through to completion and production deployment
- Capable of working closely across multiple teams to ensure data solutions line up with business initiatives and are of high quality
- Excellent written and verbal communication skills
Our Hiring Processes Might Include
We want to see your problem-solving and analytical skills. Be prepared to write good, clean, scalable code. You don’t need to know our entire stack, but we’re looking for practical experience, someone who can solve production problems in the cloud.
- Recruiter Phone Screen
- Hiring Manager Screening
- Final Team Interview
- Note: Visa Work Authorization Sponsorship Supported and Relocation Assistance Provided
#LI-GC1
Diversity, Equity & Inclusion at BigCommerce
We have the opportunity to build not only a great business but a great company, with soul. Our beliefs and commitment to diversity, equity and inclusion are a central part of achieving that.
Our dedication to DEI is grounded in two things: a moral belief in the dignity, value, and potential of every individual, and a practical belief that diverse, inclusive teams will create the best outcomes for our customers, partners, employees, and company. We welcome everyone to be a part of our journey.
Current BigCommerce Employees: Please use the internal job board to apply for openings