Bitdeer is a world-leading technology company for Bitcoin mining and AI cloud, operating globally with datacenters across multiple countries. The Senior AI Observability & Telemetry Engineer will play a critical role in architecting and scaling high-cardinality telemetry infrastructure to support AI-native workloads. Responsibilities include building highly available time-series database systems, integrating hardware-level exporters like NVIDIA DCGM, and developing eBPF-based diagnostic tools to trace distributed training bottlenecks. You will also design automated alerting pipelines, manage multi-tenant billing metrics based on GPU utilization, and collaborate with GPU Systems and Scheduling teams to establish observability standards. Required qualifications include a Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, or a related field, and at least 6 years of experience in software or site reliability engineering. Candidates must possess advanced proficiency in Go, deep expertise in the Prometheus/OpenTelemetry ecosystem, and hands-on experience with kernel-level tracing and Linux performance tuning. Strong technical leadership and the ability to influence architectural decisions are essential. Experience in high-velocity, high-growth engineering environments is strongly preferred.
For full details about this role, click “Apply on company site” to view the employer’s original posting.
A focused preparation guide, built around this job’s responsibilities and requirements.
✦ AI-generated guide
Preparation suggestions, not the employer’s actual interview questions. Always check the original posting for current requirements.
01
Understand the role
What matters most
Scalable High-Cardinality Telemetry Architecture
The role requires designing and maintaining time-series database systems (VictoriaMetrics, Thanos, or Mimir) capable of handling massive ingestion rates from global AI datacenters. You must demonstrate the ability to manage high-cardinality data while ensuring system availability and performance.
Hardware-Level Integration and AI-Native Observability
You must integrate low-level hardware metrics (NVIDIA DCGM, IPMI, Redfish) into the Kubernetes stack. This is critical for providing visibility into GPU utilization, power states, and memory bandwidth, which are essential for monitoring AI training job efficiency.
Automated Diagnostic and Billing Pipelines
You are responsible for building eBPF-based tools to identify kernel-level bottlenecks and creating automated alerting pipelines. Additionally, you must design metric pipelines for accurate multi-tenant billing, requiring precise tracking of resource consumption.
Likely challenges to think through
Balancing High-Performance Monitoring with System Overhead
Implementing deep kernel-level tracing (eBPF) and high-frequency hardware polling can introduce performance overhead on production AI workloads. Prepare to discuss how you minimize the impact of your observability tools on the very systems they are monitoring.
Managing Multi-Tenant Billing Accuracy
Translating raw GPU and network utilization metrics into accurate billing data for multiple tenants is complex due to potential data gaps or hardware failures. Prepare by reviewing strategies for data reconciliation and ensuring high-fidelity metric collection in distributed environments.
Standardizing Observability in a High-Growth Environment
As a senior engineer, you must influence cross-functional teams to adopt new observability standards. The challenge lies in aligning GPU Systems and Scheduling teams on these standards while maintaining the velocity required by a fast-growing company.
03
Practice your answers
Technical practice
Scaling High-Cardinality Telemetry
What to coverExplain your approach to managing high-cardinality data in systems like VictoriaMetrics, Thanos, or Mimir. Discuss strategies for label management, downsampling, and storage optimization to maintain performance under massive ingestion rates.
eBPF for Distributed Training Diagnostics
What to coverDescribe how you would utilize eBPF or BCC to identify network congestion or kernel-level I/O latency during distributed AI training jobs. Focus on how you isolate bottlenecks that are not visible through standard application-level metrics.
Hardware-Level Integration in Kubernetes
What to coverDetail the process of integrating NVIDIA DCGM or IPMI/Redfish data into a Kubernetes-native observability stack. Explain how you ensure these hardware metrics are effectively correlated with container-level performance data.
Multi-tenant Billing Metric Pipelines
What to coverHow would you design a reliable pipeline to capture and aggregate GPU and network utilization for multi-tenant billing? Address challenges regarding data accuracy, handling missing data points, and ensuring the system is tamper-proof.
Proactive Hardware Cordoning
What to coverDiscuss the logic required for an automated alerting pipeline that cordons degraded hardware. Explain how you define 'degraded' states for GPUs and how you balance aggressive automated responses with the need to avoid unnecessary job interruptions.
Go for Custom Kubernetes Operators
What to coverDescribe your experience writing custom Kubernetes operators in Go to manage observability resources. Focus on how you handle reconciliation loops and ensure your controllers are performant at scale.
Linux Performance Tuning for HPC
What to coverWhat specific Linux kernel parameters or subsystems do you prioritize when tuning for high-performance computing environments? Provide examples of how you have optimized system performance to support high-throughput telemetry collection.
AI-Native Observability Standards
What to coverWhat key performance indicators (KPIs) do you consider essential for 'AI-native' workloads? Suggest how you would collaborate with scheduling teams to ensure these metrics drive better job placement and resource efficiency.
Behavioural practice
Influencing Architectural Decisions
What to coverDescribe a situation where you had to advocate for a specific observability architecture against competing technical viewpoints. Explain how you presented your case, addressed stakeholder concerns, and reached a consensus.
Mentoring and Technical Leadership
What to coverProvide an example of how you have mentored team members on high-performance telemetry best practices. Focus on how you balanced your own delivery responsibilities with the need to elevate the team's technical capabilities.
Translating Requirements into Milestones
What to coverTell me about a time you had to break down a complex, ambiguous system requirement into manageable engineering milestones. How did you track progress and communicate status to non-technical or cross-functional stakeholders?
Operating in High-Velocity Environments
What to coverShare an experience working in a high-growth, fast-paced engineering environment where priorities shifted rapidly. How did you maintain the stability and quality of your telemetry infrastructure during such periods?
02
Sharpen your knowledge
Topics to prepare
High-Cardinality Telemetry Architecture
The role requires managing massive ingestion rates for AI-native workloads. Exercise: Design a high-level architecture for a multi-tenant telemetry system using Thanos or Mimir, focusing on how to handle high-cardinality labels without exhausting storage or query performance. This is relevant to the requirement for scaling telemetry infrastructure for AI cloud services.
AI Hardware Observability Metrics
Understanding GPU-specific performance indicators is critical. Exercise: Review the NVIDIA DCGM exporter documentation and map specific metrics (e.g., SM utilization, memory bandwidth, power states) to potential failure modes in distributed training. This is relevant to the responsibility of building alerting pipelines that proactively cordon degraded hardware.
Skills to revise
Go for Kubernetes Operators
Advanced Go proficiency is required for custom tooling. Exercise: Write a simple Kubernetes controller in Go that watches for a custom resource and triggers an action based on a simulated metric threshold. This is relevant to the requirement for developing custom Kubernetes metric exporters and operators.
eBPF Kernel Tracing
Hands-on kernel-level diagnostics are essential for identifying bottlenecks. Exercise: Use BCC tools to trace I/O latency or network packet drops on a Linux system. This is relevant to the responsibility of building eBPF-based tools to diagnose distributed training bottlenecks.
Linux Performance Tuning
Deep system-level knowledge is needed for HPC environments. Exercise: Analyze a Linux system's performance using tools like perf or iostat to identify resource contention. This is relevant to the requirement for deep performance tuning of Linux systems in high-performance computing environments.
04
Your preparation checklist
Prepare a technical walkthrough of a high-cardinality telemetry system
Create a brief presentation or document detailing how you architected a system using VictoriaMetrics, Thanos, or Mimir to handle massive ingestion rates. Focus on how you managed cardinality, storage retention, and query performance in a multi-tenant environment.
Develop a sample eBPF-based diagnostic tool
Prepare a code snippet or project summary demonstrating your use of eBPF or BCC to trace kernel-level I/O latency or network congestion. Explain how this tool helped identify or resolve a specific performance bottleneck in a Linux-based environment.
Draft a design for a GPU-aware observability pipeline
Outline an architectural approach for integrating NVIDIA DCGM metrics into a Kubernetes-native observability stack. Include how you would use these metrics to trigger automated alerts or proactive hardware cordoning for AI training workloads.
Prepare examples of Go-based Kubernetes operators
Gather code samples or architectural diagrams of custom Kubernetes operators or exporters you have developed in Go. Be ready to explain how these tools automated infrastructure tasks or provided specialized telemetry for hardware components like IPMI or Redfish.
Formulate strategic questions for the engineering leadership
Prepare thoughtful questions regarding their current AI-native observability standards, the specific challenges they face with GPU utilization billing, and how the observability team collaborates with GPU Systems and Scheduling teams to improve job efficiency.
Availability last checked: 18 Oct 2026. Status can change after a check; confirm on the employer’s website. Report an issue · Our checking policy
Share this opportunityKnow someone who would be a great fit?