We are looking for a highly motivated data engineer, who will combine software engineering skills with a strong understanding of data driven problem-solving. This job is all about coming to grips with the messy world of user-submitted data at scale (hundreds of millions of documents) and finding optimal ways to process and enrich that data. We process job postings, resumes, and professional profiles to better answer questions about what skills are available in the workforce, what colleges should be training students for, and where the opportunities lie for job-seekers. We're looking for people who have the technical skills and an interest in solving problems like: what are the best ways to process 2 billion job postings and load them into Elasticsearch or how best to deduplicate 1 billion professional profiles from multiple providers?

Responsibilities:

• Be familiar with our data clients, both in-house and external, and determine how best to get our data into their tools
• Build and maintain data processing pipelines for several major data sources for job postings and professional profiles
• Create and preserve knowledge with well-written documentation
• Maintain Elasticsearch clusters and the servers they run on

Specific Requirements:

• 2+ years experience in a professional setting
• Strong software engineering, programming, and QA skills
• Proficiency with Linux (or other POSIX)
• Proficiency with a systems programming language
• Experience with Docker preferred
• Experience with Terraform preferred
• Experience with batch processing (AWS Batch) and/or Apache Spark (AWS EMR) preferred
• Gitlab, AWS ecosystem preferred
• Experience with Scala languages preferred

Lightcast is proud to be an equal opportunity workplace and is committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or Veteran status. Lightcast has always been, and always will be, committed to diversity, equity, and inclusion. We seek dynamic professionals from all backgrounds to join our team, and we encourage our employees to bring their authentic, original, and best selves to work.

#LI-Remote