Company description
Twitter is what’s happening and what people are talking about right now. For us, life's not about a job, it's about purpose. We believe real change starts with conversation. Here, your voice matters. Come as you are and together we'll do what's right (not what's easy) to serve the public conversation.
Job description
Who we are
Cortex empowers internal teams to efficiently leverage ML by providing a platform and by unifying, educating, and advancing the state of the art in ML technologies within Twitter.
We win when our customers win by helping our users stay informed, share and discuss what matters; by serving the public conversation. We’re building an AI-first company and every major initiative is increasingly dependent on the successful application of machine learning. Cortex is at the nexus of this evolution. Our team of ML software engineers is constructing one of the strongest machine learning platforms in the world, based on the latest ML industry practices, deep learning, engineering excellence, and powered by Twitter data at scale.
The ML Feature Management (MLFM) team is part of the Cortex Platform group. Together with the other four Cortex Platform teams, we develop tools and infrastructure that standardizes machine learning development at Twitter on a common platform. This allows ML engineers in our customer teams to do their work better and faster and with a more modern stack that follows broad industry trends.
More specifically, our team’s vision is to maximize the velocity of Feature Engineering. To date, we’ve built and heavily invested in Twitter’s Feature Store, to share features across different ML product teams in production and thereby to broaden the impact of feature engineering investments across the company. This has been a wildly successful mission over the last two years. The Feature Store is running in production in many of Twitter’s ML stacks, serving millions of requests per second at super low latencies which has impacted top-line metrics like daily active users. These wins have prompted all remaining ML teams to migrate to the Feature Store. We’re assisting with this migration which we expect to complete in 2021.
Our other investments for 2021 include:
Core API improvements include new functionality to deliver a more complete solution to our customers.
Simplifying our architecture with an eye toward expandability and maintainability.
Systematic performance optimization and increasing production resilience.
Automating workflows for feature registration and for migration to the Feature Store as much as possible.
Support new and different data formats for feature data.
We’re participating in Cortex’s virtual team to explore ML tooling available in the Google Cloud Platform (GCP). For our team, this means integrating data processing engines such as BigQuery and Dataflow and their application to feature engineering use cases.
Seed projects to broach new frontiers:
Prototyping tooling to accelerate feature experimentation and exploration.
Building ML-as-a-service, a system that can handle dataset creation, training and online feature hydration purely from configurations and model specification, with the purpose of taking successful notebook experiments to an A/B test in production quickly.
Systematically tracking and analyzing metadata of all stages of tooling for answering questions around feature-model lineage, data privacy protection and easier model performance debugging.
Supporting trained embeddings as first class features throughout our tooling ecosystem.
What you will do
We’re looking for a senior engineer to join us in our core Feature Store workstream who can help us simplify the code base while adding new core functionality to tackle complex use cases, but without compromising performance. If thinking in architectural frameworks, like primitives, design patterns, functional paradigms, or the commonality-variability principle is your middle name, then we want to talk to you. If chasing down latency losses, drilling into performance traces and dissecting the JVM memory model is up your alley, give us a call.
Qualifications
Who you are
Do you identify with the majority of the following traits? Yes? We believe they will make you successful in this role.
You’re fluent in concepts like design patterns, composition vs inheritance, commonality variability principle, primitives, etc.
You have a solid grasp on type systems and functional programming, ideally in Scala or Java and a dynamic language like Python.
You have educated opinions about what good library and API design in distributed systems looks like, both from formal study and from lived experience with past design decisions you made.
You’re able to articulate such opinions and concepts persuasively both in spoken and written form, with the intent to teach and enroll others, while remaining open to a constructive dialogue. Your thought process intrigues others with its clarity.
You take a principled, disciplined approach to software engineering, characterized by high volume and quality, solid testing practices, data-driven decision making, and a low rate of incidents in production.
You take pride in the developer experience, the ease of use, documentation and the durability of the products you build, together with the team.
You have a passion for machine learning and high-leverage infrastructure platforms.
You bring partners together across organizational and functional boundaries.
You’re motivated by shipping high-impact software to production and you have a track record of doing so.
By nature of the problem domain, we expect you to have experience in:
5+ years building and delivering working software through an iterative, agile process.
2+ years in Sr engineering or architect capacity with a track record of resilient design decisions shipped to production.
4+ years of work experience in software engineering in the areas of distributed data processing, in high traffic or large data production environments.
2+ years with large Scala or Java code bases with many contributors
M.S. or Ph.D. degree in computer science or a related field or equivalent work experience.
Additional information
All your information will be kept confidential according to EEO guidelines.
Equal Opportunity Statement
We are committed to an inclusive and diverse Twitter. Twitter is an equal opportunity employer. We do not discriminate based on race, color, ethnicity, ancestry, national origin, religion, sex, gender, gender identity, gender expression, sexual orientation, age, disability, veteran status, genetic information, marital status or any legally protected status
We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.