Sr. Engineer - Data & ML Platform (Hybrid, IND)

What matters most

• Building a Scalable ML Experimentation Platform

  The core mandate is to design and build a platform from the ground up that supports thousands of users. You must prioritize creating self-service capabilities for ML pipelines, ensuring the infrastructure can handle massive telemetry data volumes while remaining accessible to data scientists and threat analysts.

• Standardizing ML Development and Deployment

  You are expected to modularize complex ML code and establish repeatable patterns for development, deployment, and monitoring. This requires creating standardized interfaces and CI/CD workflows that bridge the gap between experimental model development and reliable production operations.

• Cross-Functional Collaboration and Stakeholder Management

  Success depends on synthesizing the needs of Data Scientists, Threat Analysts, and Data Platform engineers. You must translate these diverse technical requirements into unified software interfaces and processes that improve overall organizational efficiency.

Likely challenges to think through

• Managing Hyper-Scale Data Complexity

  The platform operates on a 200 PB Data Lakehouse. The challenge is ensuring that ML pipelines remain performant and cost-effective at this scale. Prepare by demonstrating experience with distributed computing frameworks like Spark or Flink and explaining how you handle data partitioning and resource management in cloud-first environments.

• Bridging the Gap Between Research and Production

  Data scientists often prioritize model performance over production-grade software engineering. You will face the challenge of enforcing best practices without stifling innovation. Prepare by discussing how you implement 'guardrails'—such as automated code reviews and standardized deployment patterns—that maintain high quality without creating friction for researchers.

• Designing for Evolving AI Use Cases

  With future plans involving generative AI and complex attack path modeling, the platform must be highly extensible. The challenge is avoiding technical debt while building for unknown future requirements. Prepare by explaining your approach to modular architecture and how you select technologies that allow for long-term flexibility.

Topics to prepare

• MLOps Lifecycle and Pipeline Orchestration

  Focus on the transition from model development to production. Exercise: Design a conceptual architecture for a self-service ML pipeline that includes automated testing, versioning, and deployment stages. This is relevant because the role requires building an ML experimentation platform that bridges the gap between data scientists and production systems.

• Distributed Data Processing at Scale

  Review the principles of distributed computing using frameworks like Apache Spark or Flink. Exercise: Analyze how to optimize a data pipeline for a 200 PB scale environment, specifically focusing on partitioning strategies and data locality. This is relevant as the team manages massive telemetry data and requires expertise in building scalable data platform features.

Skills to revise

• Infrastructure-as-Code (Terraform and FluxCD)

  Practice defining and managing cloud infrastructure and Kubernetes configurations. Exercise: Write a Terraform module to provision a managed Kubernetes cluster and use FluxCD to automate the deployment of a sample application. This is a core requirement for maintaining the platform's infrastructure.

• Python for Platform Engineering

  Focus on writing clean, modular, and reusable Python code. Exercise: Create a Python wrapper or interface that simplifies a complex ML library for end-users, ensuring it follows software development best practices. This directly addresses the need to provide standardized interfaces for data scientists.

• ML Platform Tooling (MLFlow and Ray)

  Gain hands-on experience with experiment tracking and distributed model training. Exercise: Set up an MLFlow experiment tracking server and configure a simple Ray cluster to parallelize a model training task. These tools are explicitly listed as key components of the ML platform stack.

• Architect a Scalable ML Platform Design

  Prepare a high-level design document for a self-service ML experimentation platform. Focus on how you would integrate tools like Ray, MLFlow, and Kubernetes to support data scientists, ensuring the architecture handles massive telemetry data volumes and provides standardized, repeatable pipelines.

• Demonstrate Infrastructure-as-Code Proficiency

  Create a portfolio example or walkthrough showing how you have used Terraform and FluxCD to manage production-grade infrastructure. Explain how you ensure consistency and scalability when deploying ML platform components in a cloud-first environment.

• Prepare Distributed Data Processing Examples

  Review your experience with Apache Spark or Flink. Be ready to discuss specific challenges you solved regarding data ingestion, feature engineering, or performance optimization when working with large-scale datasets in a distributed computing environment.

• Develop Standardized Python Interfaces

  Prepare to discuss your approach to building Python-based abstractions that simplify complex platform tools for end-users like Data Scientists. Provide examples of how you have balanced ease-of-use with the need for robust, production-ready code.

• Formulate Strategic Questions for the Team

  Prepare thoughtful questions regarding the current bottlenecks in their ML experimentation lifecycle, how they currently handle model monitoring at scale, and how the team balances the need for rapid innovation with the stability requirements of a cybersecurity platform.