Principal Datadog Engineer

Job Title: Principal Datadog Engineer

We are supporting a high-visibility federal program dedicated to modernizing software infrastructure for a key governmental department. This role focuses on establishing and scaling enterprise-wide observability using Datadog. The Principal Datadog Engineer will serve as the technical authority in designing, implementing, and governing complex monitoring solutions, influencing platform architecture, and guiding organizations toward best practices in observability. This position offers an opportunity to shape enterprise monitoring strategies, contribute to a cutting-edge tech environment, and lead critical technical initiatives.

Key Responsibilities:

• Define and drive high-level observability strategies, addressing unique and complex challenges that impact system design and future innovations.
• Architect and implement scalable Datadog solutions—dashboards, monitors, alerts, APM, synthetic testing, and log analysis—to enhance visibility across applications, infrastructure, and security layers.
• Lead large-scale enterprise integrations and oversee the deployment of Datadog across multiple systems, ensuring consistency, reliability, and security.
• Identify workflow inefficiencies and develop tools or methodologies to improve operational productivity, system reliability, and cost efficiency.
• Serve as the escalation point for advanced Datadog performance and monitoring issues, performing root cause analysis with expertise in tracing, logs, and dashboards.
• Mentor and develop engineering teams, promoting best practices and fostering continuous improvement through training and documentation.
• Collaborate with cross-functional teams—including DevOps, SRE, Cloud, Security, and Application Development—to embed observability into all facets of operations.
• Influence architecture decisions and lead technical governance efforts, ensuring alignment with program goals and industry standards.
• Communicate complex technical concepts effectively to non-technical stakeholders, representing the organization’s technical capabilities externally when needed.

Qualifications:

• Bachelor’s Degree and a minimum of 10 years of relevant experience, or 14+ years of relevant experience in lieu of a degree.
• Proven expertise in large-scale, enterprise-level software environments with deep, recent experience in Datadog architecture, configuration, and optimization.
• Datadog Fundamentals Certification (or willing to obtain) and hands-on proficiency with dashboards, APM, synthetic tests, and integrations.
• Strong engineering background with proficiency in Python, Bash, PowerShell, YAML; experience building complex automation scripts and Lambda functions.
• Demonstrated success in implementing infrastructure and application monitoring solutions using Datadog under the hood.
• Cloud experience, preferably AWS, with Infrastructure-as-Code tools such as Terraform.
• Prior experience as a Cloud DevOps or Automation Engineer with a track record of delivering scalable observability solutions.
• Excellent technical communication skills, capable of explaining complex concepts to diverse audiences.
• Analytical mindset with strong problem-solving skills and a strategic approach to future-proofing observability capabilities.
• Ability to lead initiatives independently, influencing architecture and operational practices at an enterprise level.

Nice to Have:

• Additional Datadog certifications or willingness to obtain certifications.
• Experience with DevSecOps practices and integrating security monitoring into observability platforms.
• Background in modern, innovative engineering environments (e.g., high-tech companies known for rapid development cycles).

Preferred Skills (Not required):

• Experience scripting in Python for automation, monitoring, and alerting solutions.
• Exposure to advanced security automation and monitoring practices.
• Previous work in highly modernized, cutting-edge technology settings.