Lead Senior Data Engineer (Scala/Spark)

Lead Senior Data Engineer (Scala/Spark)

SemanticBits is a leading company specializing in the design and development of digital health services, and the work we do is just as unique as the culture we’ve created. We develop cutting-edge solutions to complex problems for commercial, academic, and government organizations. The systems we develop are used in finding cures for deadly diseases, improving the quality of healthcare delivered to millions of people, and revolutionizing the healthcare industry on a nationwide scale. There is a meaningful connection between our work and the real people who benefit from it; and, as such, we create an environment in which new ideas and innovative strategies are encouraged. We are an established company with the mindset of a startup and we feel confident that we offer an employment experience unlike any other and that we set our employees up for professional success every day.

SemanticBits is looking for a talented Lead Senior Data Engineer who is eager to apply industry experience in computer science, software engineering, databases, and distributed/parallel processing frameworks to prepare big data for the use of data analysts and data scientists while providing guidance, and mentorship to a team of developers. In this role you will drive contract initiatives to completion in a split role as a technical lead, team architect (IaC), and policy liaison with leadership and stakeholders. You will work closely with other technical leads on cross functional teams to solve challenging data delivery problems. You will advocate for best practices and technical debt reduction to streamline processing and reduce maintenance costs / cloud footprint. If you have experience with Scala and Spark and want your work to contribute to systems that collect healthcare data used by hundreds of thousands of daily users, we want to (virtually) meet you!

You will work on projects that support the Centers for Medicare and Medicaid Services (CMS) as we develop a next-generation analytics and reporting system that directly impacts healthcare quality. You will use Spark to build data processing pipelines that derive information from large sets of government data. You will be the go-to on your team for Spark, the Spark Engine, and the Spark Dataframe API. You will take a proactive approach in learning new technologies in distributed processing and cloud computing to pilot PoC for new development. We are a collaborative company, so we want you to use your knowledge of Spark to teach others, inform design decisions, and debug runtime problems.

Tools & Technology

    • Spark, Hadoop, Scala, Python, and AWS EMR
    • Terraform, Jenkins, Airflow, and AWS Step Functions
    • Jupyter and Zeppelin
    • AWS S3, AWS Redshift, AWS ECS, AWS SQS and Teradata
    • GSuite, Slack, Jira, Confluence, Git, and Github


    • Build scalable data processing pipelines in Spark
    • Debug Spark jobs and do performance tuning
    • Write unit and integration tests for all data processing code
    • Work with DevOps engineers on CI, CD, and IaC
    • Read specs and translate them into code and design documents
    • Perform code reviews and develop processes for improving code quality
    • Become intimately familiar with the medicare claims data model

Required Qualifications

    • Bachelor’s degree required in Computer Science or related field
    • Minimum of 7 years relevant work experience 
    • Highly Competent with Scala, Spark, the Spark Engine, and the Spark Dataframe API
    • Experience in a technical leadership role
    • Experience with Agile methodology, using test-driven development
    • Excellent command of written and spoken English
    • Candidate must reside in the United States
    • Flexible and willing to accept a change in priorities as necessary

Physical and emotional requirements for the job

    • This position is to be performed remotely from an individual’s home office and involves sedentary work. Employees in this role can be expected to exert up to 10 pounds of force on occasion in order to lift, carry, push, pull or otherwise move standard electronic equipment. Employees are expected to make decisions in a timely manner and display emotional intelligence during occasional stressful situations.
Competitive salary
Three weeks of PTO
Ten paid holiday days
Comprehensive health benefits (medical with HSA option, dental, and vision)
401k retirement plan with matching benefit
100% paid short-term and long-term disability
100% paid life insurance
Flexible Spending Accounts (FSA)
Casual working environment
Flexible working hours

SemanticBits, LLC is an equal opportunity, affirmative action employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, protected veteran status, disability, or any other characteristic protected by law. We are also a veteran-friendly employer.

If you are an individual with a disability and require a reasonable accommodation to complete any part of the application process, or are limited in the ability or unable to access or use this online application process and need an alternative method for applying, you may contact 703-787-9656 x257 or for assistance.
Logos/outerjoin logo full

Outer Join is the premier job board for remote jobs in data science, analytics, and engineering.