Lead Software Engineer - Agentic Platform

Role Summary

You will be the first line of operational oversight for all agentic components deployed within the SocGen AI platform. The role centres on proactive monitoring, disciplined execution of run-books, and incident triage in line with the established ITSM process across every business use case running on the platform.

This is a hands-on operations role, delivered through a 24/7 rotating shift model to ensure uninterrupted platform oversight. You will continuously monitor dashboards and alerts, log into servers to verify pod and service health, inspect logs, identify anomalies before they escalate, and ensure every incident is recorded, triaged, and routed to the right resolution path. You will also contribute to the platform feedback loop by surfacing recurring issues and supporting continuous improvement of monitoring and run-books.

What You Will Do

Platform Monitoring

• Continuously monitor health dashboards, alerts, and operational metrics across all deployed agents and agentic workflows
• Proactively track agent execution, throughput, latency, token consumption, and error patterns; raise early warnings before incidents escalate
• Monitor user-facing interfaces, tool calling layers, and downstream integration touchpoints for availability and expected behaviour
• Maintain a visible daily check cadence across all agentic components to ensure continuous platform reliability

Infrastructure and Log Checks

• Log into application servers to verify service health, pod status, and container availability within the hosting environment
• Retrieve, review, and interpret application logs to isolate symptoms and form a first-level hypothesis of the underlying issue
• Execute scheduled and ad hoc run-book checks, maintaining clear evidence of completion and any deviations observed
• Perform basic environment sanity checks on connectors, credentials, and scheduled jobs as part of the daily operational routine

Incident Management (ITSM)

• Acknowledge incidents raised by delivery teams, IT support, and business users via the designated channels and take ownership of the issue
• Diagnose incidents, gather diagnostic inputs (logs, screenshots, reproduction steps), and record the incident in the Unity ticketing system
• Assign severity (S1, S2, S3) and owner in accordance with the defined triage matrix
• Resolve Severity 3 incidents directly where scope permits; escalate Severity 1 and Severity 2 incidents to L2 and L3 with a complete diagnostic packet
• Maintain issue cadence, ensure continuity of product support, and communicate clear status updates to stakeholders throughout the incident lifecycle

Continuous Improvement and Platform Feedback

• Track recurring incidents and contribute pattern analysis to the platform feedback loop covering incident analysis, prompt quality, test quality, resolution rate, and user feedback
• Flag candidates for run-book updates, monitoring rule enhancements, or prompt-level improvements based on real production evidence
• Participate in post-incident reviews, contribute to root cause classification, and support preventive action tracking through to closure
• Maintain clean and up-to-date operational documentation, run-books, and support knowledge articles

What You Bring

• 7 - 9 years of experience in an L1 or application support role, preferably within financial services or an enterprise technology environment
• Proven experience operating in a 24/7 shift-based support model with demonstrated discipline in handover, escalation, and continuous coverage
• Strong comfort with Linux and Unix command line operations, including SSH access, directory navigation, log retrieval, and standard text processing utilities
• Working familiarity with containerised environments; able to check pod and service status, interpret basic deployment state, and execute standard restarts where run-books permit
• Solid understanding of ITSM principles, including ticket creation, severity classification, escalation discipline, and service level awareness
• Structured, methodical approach to executing monitoring checks and documenting findings with a high attention to detail
• Strong written and verbal English communication; able to document incidents clearly, concisely, and accurately across shifts and stakeholders
• Ability to remain calm, organised, and decisive under operational pressure, particularly during live incidents outside business hours

Nice to Have

• Basic Python skills: able to read scripts, run simple diagnostic snippets, and understand common libraries used in AI and data pipelines
• Conceptual familiarity with large language models, agentic architectures, prompt-based systems, and retrieval-augmented generation
• Exposure to monitoring and observability tooling such as Grafana, Kibana, Splunk, or Prometheus
• Prior experience in back-office or middle-office banking operations in any domain
• Experience supporting AI or machine learning applications in a production environment