Senior Observability Engineer – New Relic Consolidation

We are looking for a Senior Observability Engineer to design and implement a consolidated New Relic observability layer across multiple project tenants. The role will focus on scalable New Relic administration, cross-account telemetry aggregation, observability-as-code, alerting integration, and standardized telemetry and metric design for large-scale AI agent environments.

Our client is a large global enterprise operating across multiple markets, with a complex technology landscape and a strong focus on digital transformation and innovation. The organization is actively investing in modern cloud, data, and AI capabilities to enable scalable, secure, and highly automated solutions across its business.

The project is focused on building an enterprise-grade AI Agent Platform from the ground up. The platform will provide a standardized foundation for developing, deploying, orchestrating, securing, and observing AI agents across multiple teams and use cases.

The initiative covers the full agent lifecycle and brings together agent orchestration, observability, security, governance, integrations, evaluation, and platform engineering. Engineers joining the project will have an opportunity to influence key architectural and technical decisions and contribute to a new platform rather than maintaining an existing solution.

• Design and operate a multi-account New Relic observability environment, consolidating telemetry from multiple project tenants.
• Build cross-account NRQL queries, dashboards, and reporting for centralized operational visibility.
• Implement dashboards-as-code and alert-policy-as-code using Terraform and Crossplane.
• Configure and manage New Relic alert policies and routing to Opsgenie, with future integration into ServiceNow.
• Support ingestion of telemetry into New Relic through OpenTelemetry/ADOT, OTLP, and Kinesis Firehose pipelines.
• Define scalable metric dimensions and low-cardinality facets for large AI agent estates.
• Establish observability patterns that support Technical, Business, and Audit telemetry segregation.
• Support cross-tenant monitoring, reporting, and operational governance.
• Collaborate on observability for GenAI workloads, including AWS Bedrock AgentCore telemetry and cost/token attribution.
• Automate provisioning and lifecycle management of observability resources using Terraform and Crossplane.

• New Relic administration at scale (management account, cross-account setup)

• NRQL — cross-account queries aggregating telemetry from multiple project tenants

• Dashboards-as-code and alert-policy-as-code (Terraform New Relic provider, Crossplane)

• New Relic alert policies routed to Opsgenie (ServiceNow later)

• OpenTelemetry / ADOT consumption fundamentals (OTLP and Firehose ingestion into New Relic)

• Metric dimension design for large agent estates (stable, low-cardinality facets)

• 6+ years of observability or SRE engineering

• New Relic (or equivalent SaaS observability) operated as a multi-tenant consolidation layer

• Cross-account / cross-tenant telemetry aggregation and reporting

• Working within a Technical / Business / Audit observability segregation model

• AWS Bedrock AgentCore observability (Transaction Search, GenAI Observability)

• Kinesis Firehose metric/log forwarding pipelines (CloudWatch → New Relic)

• Cost/token attribution modeling for GenAI workloads

• Crossplane-based provisioning of observability resources