About the role
Company Overview
Electronic Arts creates next-level entertainment experiences that inspire players and fans around the world. Here, everyone is part of the story, part of a community that connects across the globe. It is a place where creativity thrives, new perspectives are invited, and ideas matter. The team is where everyone makes play happen.
At EA, games are considered powerful because they bring together multiple ways people engage: play, watch, create, and connect. Increasingly, the biggest entertainment platforms are places where communities build. Creator-made content is a proven part of EA's history—from community creation tools in Battlefield to The Gallery in The Sims 4. The company believes new creative technologies and tools will expand how players engage with and contribute to experiences, supported by thoughtful product design, safety systems, and global reach. The focus is on enabling more players to participate in creative expression by making creation easier, safer, and more rewarding.
Role Overview
As a Principal ML Engineer, Training Pipelines, you will own the systems that prepare and deliver multimodal game data for large-scale model training. You will set the architecture for these training input pipelines and improve data throughput, accelerator utilization, reproducibility, and recovery from failures.
Your responsibilities will initially focus on hands-on individual contribution and technical leadership, with potential to take on people-management responsibilities as the team grows. You will partner with researchers and central data and compute teams to turn evolving model requirements into reusable training capabilities.
Responsibilities
- Architect and build shared training input pipelines from certified game capture to accelerators, covering storage layout, decoding, sharding, sampling, packing, batching, caching, prefetching, and host-to-device delivery across multiple training frameworks.
- Define schemas, data contracts, encodings, compression, and standards for dataset versioning, lineage, quality, coverage, acceptance criteria, and reproducible splits.
- Build deterministic loaders and checkpoint-safe iteration that recover from worker or hardware failures and corrupt data, including elastic restarts and changes to cluster size or parallelism.
- Design reproducible dataset mixtures and balanced batches across modalities, resolutions, sequence lengths, titles, and capture conditions, accounting for worker/rank locality and distributed parallelism.
- Benchmark and enforce throughput, data-wait time, step time, and accelerator utilization and stalls. Model FLOPs utilization (MFU) where applicable, per-rank skew, scaling efficiency, restart time, reliability, and cost targets through thousands of accelerators.
- Diagnose bottlenecks and intermittent failures across storage, network I/O, CPU, memory, decoding, preprocessing, and accelerators, including stragglers, faulty hardware, corrupt samples, and imbalanced batches.
- Integrate compilers, runtimes, optimized kernels, low-precision training, and hardware-accelerated media processing. Maintain efficient data delivery and recovery under FSDP/HSDP, tensor, context, and pipeline parallelism across heterogeneous accelerator environments.
- Lead training-pipeline architecture reviews, mentor engineers, and align priorities with research leads. Translate model and data-access requirements into scalable systems building on shared infrastructure services, while researchers retain model-research ownership.
Qualifications
- 20+ years of development experience, including owning production systems for distributed-training input, with measurable improvements in throughput, accelerator utilization, step time, reliability, recovery time, or cost.
- Expertise building custom PyTorch training and ingestion systems or adapting TorchTitan, JAX/XLA with MaxText, Megatron Core/NeMo, or equivalent stacks for single jobs spanning thousands of accelerators.
- Expertise in FSDP2/HSDP, tensor/context/pipeline parallelism, mixed-precision training, and optimization on one or more accelerator platforms such as GPUs, TPUs, or AWS Trainium.
- Pre-training experience and an understanding of how model architecture, data representation, batching, and orchestration affect training-system design and efficiency.
- Expertise building large-scale multimodal streaming or hardware-accelerated ingestion with Ray Data, Mosaic Streaming, WebDataset, NVIDIA DALI, NVDEC, or equivalent technologies.
- Deep knowledge of sharding, sampling, packing, caching, prefetching, deterministic data mixing, checkpoint-safe iteration, and elastic restart; experience with schemas, dataset versioning, lineage, quality, and reproducible training/evaluation.
- Experience profiling I/O, CPU, memory, network, distributed workers, and accelerators, including diagnosing cross-layer failures and intermittent faults.
Compensation and Benefits (North America)
Pay offered will be determined based on a number of relevant business and candidate factors (e.g. education, qualifications, certifications, experience, skills, geographic location, or business needs).
Pay Ranges
- British Columbia (depending on location e.g. Vancouver vs. Victoria): $204,600 - $287,500 CAD
- California (depending on location e.g. Los Angeles vs. San Francisco): $227,300 - $344,700 USD
US Benefits
- Paid time off (3 weeks per year to start).
- 80 hours per year of sick time.
- 16 paid company holidays per year.
- 10 weeks paid time off to bond with baby.
- Medical/dental/vision insurance.
- Life insurance and disability insurance.
- 401(k).
- Potential eligibility for bonus and other incentive programs.
Canada Benefits
- Vacation (3 weeks per year to start).
- 10 days per year of sick time.
- Paid top-up to EI/QPIP benefits up to 100% of base salary when you welcome a new child (12 weeks for maternity, and 4 weeks for parental/adoption leave).
- Extended health/dental/vision coverage.
- Life insurance and disability insurance.
- Retirement plan.
- Potential eligibility for bonus and other incentive programs.
Additional Information
EA values adaptability, resilience, creativity, and curiosity. The company adopts a holistic approach to wellness, emphasizing physical, emotional, financial, career, and community support. Packages are tailored to local needs and may include healthcare coverage, mental well-being support, and complimentary games.
Electronic Arts is an equal opportunity employer. All employment decisions are made without regard to race, color, national origin, ancestry, sex, gender, gender identity or expression, sexual orientation, age, genetic information, religion, disability, medical condition, pregnancy, marital status, family status, veteran status, or any other characteristic protected by law. EA considers qualified applicants with criminal records in accordance with applicable law and makes workplace accommodations for qualified individuals with disabilities.