Datadog is a a service that helps you monitor your infrastructure and applications.
Dashboards lets you create queries, tables, and graphs to show you important metrics (e.g. network latency, disk usage, out of memory, etc). Most of the time I’ll add conditionals (if memory free is < X, then make red instead of green)
You can add an integration (e.g. AWS MSK) and you’ll get metrics you need (e.g. say cpu, memory, disk space) usage.
Other integrations include say Processes to collect resource usage metrics for specific running processes
on any host: CPU, memory, I/O, number of threads. You can configure different thresholds so that you get an
alert when you get unexpected numbers (aka a Service Check).
Monitors allow you to detect changes in a metric, then alerts your team.
You can install independent agents to run on your host to collect events and metrics, then send this back to Datadog.
Agents are installed on your OS (e.g. Ubuntu) and run with say sudo service datadog-agent start
There is a collector that gathers all standard metrics every 15 seconds Then a forwarder sends the metrics over HTTPS to Datadog
You can setup and run datadog-agent with datadog-agent as a CLI. Usage is:
https://docs.datadoghq.com/agent/basic_agent_usage/?tab=agentv6
Datadog has an embedded python environment. You can find that here:
/opt/datadog-agent/embedded/ has your embedded Python environment with common libraries like python and pip.
Install Python packages using the embedded pip with:
sudo -Hu dd-agent /opt/datadog-agent/embedded/bin/pip install <package_name>
/etc/datadog-agent/datadog.yaml has the configuration files and folders for the Agent
/etc/datadog-agent/conf.d/ has datadog’s built-in integrations (e.g. process.d, gunicorn.d, docker.d, statsd.d)
/etc/datadog-agent/checks.d/ has your custom Agent checks (e.g. https://docs.datadoghq.com/developers/write_agent_check/?tab=agentv6)
In order to setup an agent yourself (i.e. one that is not a built in datadog integration), you just need to
add to conf.d with your configuration and then add in your check in checks.d.