Senior Data Engineer

Senior Data Engineer

This job is no longer open
How you will help
You will support the engineering team’s data endeavors, diving in to fix issues, optimize processes, and automate what you do more than once.  You’ll use the best tools for the job, whether modern and revolutionary or time tested and proven, to deliver elegant, scalable solutions that meet business and technical needs. 

What you will do
• Work with internal stakeholders to load data into HealthVerity's data warehouse
• Troubleshoot and resolve issues relating to data integrity
• Help establish procedures and best practices for transforming and storing data
• Lead requirements gathering around data pipeline automation improvements
• Work with some of the most exciting open-source tools like Spark, Hadoop, Docker, Airflow, Zeppelin
• Leverage distributed computing and serverless architecture such as AWS EMR & AWS Lambda, to develop pipelines for transforming data
• Perform root cause analysis on internal and external data and process to answer specific business questions and identify opportunities for improvement
• Research and implement new technologies with a team of developers to execute strategies and implement solutions
• Solve complex problems related to the real-time discovery of large data
• Build POCs to evaluate and make right technology decisions

About you
• Experienced in writing scalable applications on distributed architectures
• Data driven, testing and measuring as much as you can
• Eager to both review peer code and have your code reviewed
• Comfortable on the command line and consider it an essential tool
• Confident in SQL, you know it, write smart queries, it’s no big deal
• Passionate designing and leading the implementation of resilient, distributed data platforms and large-scale infrastructures
• Advanced knowledge of data, design and platform architecture disciplines

Required skills and experience
• 5+ years of work experience
• 3+ years of data engineering experience with Python or Scala
• 3+ years of experience with PySpark and Spark-SQL (writing, testing, debugging spark routines)
• 2+ years of experience in custom ETL design, implementation and maintenance with AWS EMR, AWS S3 service
• Comfortable using AWS CLI and boto3
• Comfortable using *nix command line (shell scripting, AWK, SED)
• Experience with advanced SQL performance tuning and end-to-end process optimization
• Experience of handling various data sets Structured, Semi Structured, Data at rest/ Motion

Desired skills and experience
• Experience with data pipeline and workflow management tools: Apache Airflow, AWS Step Function, etc
• Bonus: Experience with Spark applications in Databricks Notebooks and Delta tables
• Bonus: Experience with healthcare data
Base salary for the role is commensurate with experience and can range between $63,000 - 250,000 + annual bonus opportunity.

All HealthVerity team members must be vaccinated against COVID-19 unless granted an exemption based on religious or medical reasons. Determinations for exemptions are evaluated on an individualized basis and are role-dependent.

About HealthVerity
At HealthVerity we are actively solving some of the greatest challenges in healthcare through innovative technology and data solutions. Our customers and partners including pharmaceutical manufacturers, payers and government organizations look to HealthVerity to partner on their  most complicated use cases, leveraging our transformative technologies and real-world data infrastructure. The HealthVerity IPGE platform, based on the foundational elements of Identity, Privacy, Governance and Exchange, enables the discovery of RWD across the broadest healthcare data ecosystem, the building of more complete and accurate patient journeys and the ability to power best-in-class analytics and applications with flexibility and ease. To learn more about the HealthVerity IPGE platform, visit

Why you'll love working here

We are making a difference – Our technology is at the forefront of some of the biggest healthcare challenges in the world. 
We are one team – Our people define our culture and always will. We take time out to celebrate each other at the end of every week through company-wide shout outs, and acknowledge the value that each of us adds towards our greater mission. Come share all you have to offer.
We are learners – Every team member is continually learning, no matter if we've been in a role for one year or much longer. We are committed to learning and implementing what is best for our clients, partners, and each other.

Benefits & Perks
• Compensation: competitive base salary & annual bonus opportunity (for non-commissioned roles)
• Benefits: comprehensive benefits with coverage on Day 1, medical, dental, vision, 401k, stock options
• Flexible location: our HQ is in Philadelphia with 50% of the team distributed across 25+ states
• Generous PTO: Take time off as needed, targeted at 4 weeks per year, including vacation, personal and sick time, plus paid maternity and paternity leave.
• Comprehensive and individualized onboarding: mentorship program, departmental talks, and a library of resources are available beginning day 1 for each new team member to minimize the stress of starting a new job
• Professional development: biweekly 1:1s, hands-on leadership that is goal-and growth-oriented for each team member, and an annual budget to support professional development pursuits

HealthVerity is an equal opportunity employer devoted to inclusion in the workplace. We believe incorporating different ideas, perspectives and backgrounds make us stronger and encourages an environment where ageism, racism, sexism, ableism, homophobia, transphobia or any other form of discrimination are not tolerated. At HealthVerity, we’re working towards an innovative and connected future for healthcare data and believe the future is better together. We can only do that if everyone has a seat at the table. Read our Equity Inclusion and Diversity Statement.

If you require a reasonable accommodation in completing this application, interviewing, completing any pre-employment testing, or otherwise participating in the employee selection process, please direct your inquiries to

HealthVerity offers in-office and remote options, so you can work from anywhere within the US! #LI-Remote
This job is no longer open
Logos/outerjoin logo full

Outer Join is the premier job board for remote jobs in data science, analytics, and engineering.