Senior Production Reliability Engineer
We are seeking a highly technical Senior Production Reliability Engineer to lead our local Excellence in Engineering team. This is not a software development role: its focus is the reliability, diagnosability and operational effectiveness of complex, largely non-cloud production systems. The successful candidate will combine deep knowledge of system architecture, data flows and databases with advanced debugging, monitoring and troubleshooting skills. They will lead difficult investigations through to root cause, drive AI-enabled automation and operational improvement, and provide technical direction and mentoring to the local team. Infrastructure, capacity and performance are important supporting areas, but cloud-platform engineering is not the primary remit.
Key Responsibilities
Advanced Incident Investigation & Problem Management
•Lead complex, high-impact and multi-system production investigations from triage and containment through recovery, root cause and corrective action.
•Diagnose issues across applications, services, operating systems, databases, messaging and networks using logs, traces, metrics, configuration, queries and controlled reproduction.
•Retain end-to-end ownership while coordinating effectively with software engineering, product, infrastructure and customer-facing teams.
AI, Automation & Operational Improvement
•Champion practical use of AI and automation to accelerate incident investigation, correlate telemetry, surface likely causes and recommend next actions.
•Automate diagnostics, health checks, evidence collection, recovery activities and recurring operational workflows using scripts, tools and governed AI-assisted solutions.
•Build searchable operational knowledge from incidents, runbooks, architecture information and known errors, and measure reductions in manual effort and time to recovery.
•Improve system supportability by influencing logging, telemetry, diagnostic interfaces and non-functional engineering requirements.
System Architecture, Data Flows & Databases
•Maintain a deep understanding of end-to-end architecture, component responsibilities, dependencies and critical business flows.
•Trace transactions and data across APIs, services, message brokers, batch processes and databases to isolate defects and performance bottlenecks.
•Diagnose database query, locking, contention, indexing, connectivity, data-quality and throughput issues, and identify resilience gaps and operational risks.
Local Team Leadership & Operational Readiness
•Provide day-to-day technical leadership for the local team, setting priorities, allocating investigations and maintaining high standards of ownership and follow-through.
•Coach and mentor engineers in architecture, debugging, observability, automation and effective incident practice, while fostering collaboration with global teams.
Observability, Monitoring & Reliability
•Help develop actionable observability across services, databases, interfaces and infrastructure using platforms such as Dynatrace.
•Help direct improvements to dashboards, alerts and diagnostic views covering service health, dependencies, latency, throughput, saturation and errors.
•Help define meaningful indicators and thresholds, reduce alert noise and use telemetry to shorten diagnosis and recovery times.
Skills, Experience & Qualifications Required
•4–7 years of progressively responsible experience in site reliability, production engineering, application support, systems engineering or a similarly deep technical operations role, ideally in financial services.
•Proven ability to lead complex investigations across distributed systems using structured diagnosis and root-cause analysis.
•Strong understanding of system architecture, APIs, microservices, messaging, data flows and multi-tier failure modes.
•Strong relational database and SQL skills, including query performance, locking, indexing, connectivity and data integrity.
•Hands-on experience with observability platforms such as Dynatrace and with logs, metrics, traces, dashboards and alerts.
•Strong commitment to leveraging AI and automation safely and pragmatically to improve diagnostics, operations and reliability outcomes.
•Scripting capability in PowerShell, Python, Bash or SQL; working knowledge of Java, C#, Kubernetes, Kafka and API gateways is advantageous, but regular feature development is not expected.
•Understanding of performance, capacity, resilience, networking and operating-system fundamentals; on-premises or hybrid experience is strongly valued, while Azure familiarity is useful but secondary.
•Experience providing technical leadership, prioritising team activity, mentoring engineers and coordinating across engineering, product, infrastructure and customer-facing teams.
About State Street
Across the globe, institutional investors rely on us to help them manage risk, respond to challenges, and drive performance and profitability. We keep our clients at the heart of everything we do, and smart, engaged employees are essential to our continued success.
