Senior Principal Site Reliability Engineer (Chaos Engineering)
Software Engineering · Full-time
Kuala Lumpur, Malaysia · Hong Kong · Abu Dhabi - United Arab Emirates
About Us
- Chaos Engineering Platform Architecture & Development (50%)
- Design and build an enterprise-grade chaos engineering platform supporting multi-cluster (K8s + EC2 hybrid), multi-region, and multi-environment (testnet/mainnet) deployments
- Core capability development:
- Fault Injection Engine: Pod-level / Node-level / AZ-level fault simulation, network latency / packet loss / partition, dependency timeout / error injection
- Production Safety Assurance: Blast radius control, one-click Kill Switch, automatic rollback, real-time impact monitoring
- Traffic Isolation: Experiment traffic tagging and isolation to ensure fault injection does not impact real users
- Fault Isolation: Precise impact scoping at service / cluster / AZ granularity
- Design experiment orchestration capabilities supporting complex fault scenario composition (e.g., simultaneous network latency + downstream timeout + cache invalidation)
- Deep integration with existing monitoring, alerting, and SLO systems to achieve an automated closed loop: inject fault → observe impact → determine pass/fail
- Production Resilience Validation Framework (30%)
- Define safety standards and approval workflows for mainnet fault injection
- Design and drive routine chaos experiments:
- Daily patrol-level experiments: Low-risk experiments executed automatically on a daily/weekly basis
- Periodic validation experiments: Monthly/quarterly resilience verification of critical paths
- Large-scale drills: Cross-AZ / cross-region disaster recovery failover validation
- Establish a resilience scoring system to quantify system health based on experiment results
- Deliver improvement recommendations and drive business teams to remediate identified weaknesses
- Technology Selection & Team Enablement (20%)
- Evaluate and select the technology foundation (Chaos Mesh / Litmus / custom components — hybrid strategy)
- Develop chaos engineering best practices and playbooks to enable SRE teams and application developers
- Mentor and grow the team (2–3 engineers) in chaos engineering capabilities
- Stay current with industry developments and introduce cutting-edge practices (e.g., AI-driven fault scenario discovery)
- 8+ years of backend / infrastructure engineering experience, with 3+ years dedicated to chaos engineering or stability engineering
- Hands-on experience with large-scale fault injection in production environments (not just test environments), with deep understanding of production safety constraints
- Expert-level proficiency in Kubernetes fault injection (Chaos Mesh / Litmus / custom solutions), familiar with CRD / Operator development
- Proficient in at least one backend language (Go preferred), with platform-level system architecture design capability
- Deep understanding of distributed system failure modes (network partitions, split-brain, cascading failures, data inconsistency, etc.)
- Familiarity with observability tech stack (Prometheus / Grafana / Thanos / OpenTelemetry)
- Excellent technical documentation and solution design skills
- Experience in financial / trading system stability (understanding of transaction consistency and fund safety constraints)
- Experience building SLO / Error Budget frameworks
- Experience building automated fault recovery (self-healing) systems
- Familiarity with AWS infrastructure (EC2 / EKS / Multi-AZ / Multi-Region)
- Knowledge of Netflix Chaos Engineering / AWS Fault Injection Simulator / Gremlin
- Open-source community contributions (Chaos Mesh / Litmus or similar projects)
- Ability to balance "safety" and "validation depth" — not afraid of production injection, while maintaining strict risk control
- Strong cross-team collaboration and influence — chaos engineering requires buy-in from business teams; this role demands persuasion skills
- Self-driven, capable of independently planning and executing in ambiguous situations
- 混沌工程平台架构设计与建设(50%)
- 设计并构建企业级混沌工程平台,支持多集群(K8s + EC2混合)、多区域、多环境(testnet/mainnet)
- 核心能力建设:
- 故障注入引擎: Pod级/Node级/AZ级故障模拟,网络延迟/丢包/分区,依赖超时/错误注入
- 生产环境安全保障: 爆炸半径控制、一键停止(Kill Switch)、自动回滚、影响范围实时监控
- 流量隔离: 实验流量标记与隔离,确保故障注入不影响真实用户
- 故障隔离: 按服务/集群/AZ粒度精准控制影响范围
- 设计实验编排能力:支持复杂故障场景的组合编排(如:同时注入网络延迟+下游超时+缓存失效)
- 与现有监控、告警、SLO体系深度集成,实现"注入故障→观测影响→判定通过/失败"的自动化闭环
- 生产环境韧性验证体系(30%)
- 制定主网故障注入的安全规范和审批流程
- 设计和推动常态化混沌实验:
- 日常巡检级实验(每日/每周自动执行低风险实验)
- 定期验证级实验(月度/季度核心链路韧性验证)
- 大规模演练(跨AZ/跨Region容灾切换验证)
- 建立韧性评分体系:基于实验结果量化系统健康度
- 输出改进建议并推动业务方修复发现的弱点
- 技术方案选型与团队赋能(20%)
- 评估并选择技术底座(Chaos Mesh / Litmus / 自研组件的组合策略)
- 制定混沌工程最佳实践和Playbook,赋能SRE团队和业务研发
- 培养团队(2-3人)的混沌工程能力
- 与业界保持同步,引入前沿实践(如:AI驱动的故障场景发现)
- 8年+后端/基础设施开发经验,3年+混沌工程或稳定性工程专项经验
- 有生产环境大规模故障注入的实操经验(非仅测试环境),深刻理解生产安全约束
- 精通Kubernetes故障注入(Chaos Mesh/Litmus/自研方案),熟悉CRD/Operator开发
- 精通至少一种后端语言(Go优先),具备平台级系统架构设计能力
- 深入理解分布式系统故障模式(网络分区、脑裂、级联故障、数据不一致等)
- 熟悉可观测性技术栈(Prometheus/Grafana/Thanos/OpenTelemetry)
- 优秀的技术文档和方案设计能力
- 有金融/交易系统稳定性保障经验(理解交易一致性、资金安全约束)
- 有SLO/Error Budget体系建设经验
- 有自动化故障恢复(Self-healing)系统建设经验
- 熟悉AWS基础设施(EC2/EKS/Multi-AZ/Multi-Region)
- 了解Netflix Chaos Engineering / AWS Fault Injection Simulator / Gremlin
- 有开源社区贡献经验(Chaos Mesh/Litmus等项目)
- 能在"安全"和"验证深度"之间做好平衡——既不畏惧生产注入,也能严格控制风险
- 有推动跨团队协作的能力——混沌工程需要业务方配合,这个岗位需要说服力
- 自驱型,能在方向模糊时独立规划和推进
Why Join Us
At Bybit, we are committed to fostering a supportive and enriching work environment.
Our benefits include:
- Study Growth Fund: We support your professional development and continuous learning.
- Internal Events: Participate in regular team-building activities, workshops, and events designed to promote collaboration and innovation.
- Global Collaboration: Be part of a diverse, international team, working alongside colleagues from around the world.
- Career Advancement: Access opportunities for growth and advancement within a rapidly expanding global company.
- Internal Mobility: Grow with us- Your long-term development is important to us. We offer internal job opportunities to help build your career path.