Data Pipeline Development and Maintenance:

• Design, implement, and maintain robust, scalable data pipelines for the ingestion, processing, and transformation of large volumes of structured and unstructured data.
• Optimize data workflows to ensure efficiency, reliability, and data quality throughout the pipeline.

Database Management:

• Develop and maintain databases, data warehouses, and data lakes, ensuring proper organization, indexing, and optimization for fast data retrieval and analytics.
• Implement and manage ETL (Extract, Transform, Load) processes to integrate data from various sources into the appropriate storage solutions.

Data Modeling and Architecture:

• Design and implement efficient data models and architecture to support both operational and analytical use cases, balancing performance and scalability requirements.
• Collaborate with data scientists and analysts to define data requirements and ensure the data infrastructure meets analytical needs.

Data Quality and Governance:

• Implement data quality checks, validation processes, and monitoring systems to ensure data accuracy, completeness, and consistency.
• Adhere to data governance policies and procedures, maintaining data privacy, security, and compliance with relevant regulations.

Performance Optimization:

• Identify and resolve performance bottlenecks in data processing and retrieval, optimizing the system for improved speed, efficiency, and scalability.

Collaboration and Documentation:

• Work closely with cross-functional teams, including data scientists, analysts, and software engineers, to understand data requirements and support their initiatives.
• Create comprehensive documentation for data engineering processes, data models, and system architecture to ensure knowledge sharing and efficient onboarding of new team members.

Common Software Engineering Requirements

• You collaborate with cross-functional teams, including product managers, designers, and other engineers, to understand business requirements and translate them into efficient and scalable software solutions.
• You design, develop, test, deploy, support and maintain high-quality software applications using industry best practices and modern technologies. You own issues, including initial troubleshooting, identification of root cause and issue resolution/escalation.(Alternatively, to address verbosity, this can be  replaced with “You own your code end-to-end.”, but it may not be explicitly clear.)
• You write clean and maintainable code that adheres to industry coding standards and contributes to the overall stability of our systems. You participate in code reviews and provide constructive feedback to team members to ensure code quality and promote knowledge sharing.
• You proactively find and address technical debt, inefficient practices/tools and performance bottlenecks and bugs, continuously improving the reliability and performance of our software through building observability and other features to help trouble-shoot/triage issues.
• You demonstrate proficient usage of tools, techniques and architecture/coding patterns. You have demonstrated you are able to understand trade-offs of various architectural and design choices. Your solutions are focused on solving the needs of your customer.
• You stay up-to-date with emerging technologies and industry trends, and proactively propose and implement innovative solutions to enhance our products and services through continuous evolution and refinement of current tools and applications.

• Bachelor's or Master's degree in Computer Science, Data Science, or a related field.
• Proven experience as a Data Engineer or in a similar role.
• Strong proficiency in programming languages like Python, Java, or Scala.
• Expertise in working with big data technologies such as Hadoop, Spark, and Kafka.
• Proficiency in SQL and experience with database technologies (e.g., PostgreSQL, MySQL, NoSQL databases).
• Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services (e.g., AWS Redshift, S3, Azure SQL Data Warehouse).
• Knowledge of data warehousing concepts, data modeling, and ETL processes.
• Excellent problem-solving skills and attention to detail.
• Strong communication and collaboration abilities.