New Coinbase Watchdog Service Launches As Automated Monitoring Tool For GitHub Changes
Coinbase Introduces New ‘Watchdog’ Tool For Monitoring GitHub Changes
Coinbase just unveiled a new tool called Coinbase Watchdog. It’s a GitHub app and a Golang service that uses the Datadog API to watch for changes in Datadog.
In a blog post announcing the launch, the San Francisco-based crypto exchange giant described the tool as “the best of both a code and UI-driven approach” for monitoring changes in Datadog.
“When Coinbase Watchdog sees a change, it automatically creates a Pull Request (PR) with the changes in a dedicated Datadog GitHub repository.”
The tool also has built-in “control and consensus mechanisms” that help Coinbase guarantee that a sufficient number of people have reviewed a change before it can land on the master.
“If a PR was not approved and closed by a customer, Watchdog will call Datadog APIs to restore the components from the master branch in source control.”
The end result is a UI-driven codification bot. All changes made in the Datadog UI are automatically picked up by the bot and corresponding Pull Requests will be created.
Coinbase Watchdog can also detect if a user modified the code and apply the change to Datadog.
There are two types of configurations available for Coinbase Watchdog, including a system configuration and a user configuration:
System Configuration: This configuration includes all required parameters like Datadog API/APP keys, GitHub application private key, GitHub project URL, GitHub app installation ID, etc.
User Configuration: This is the configuration used by customers. It includes simple YAML files with a list of Datadog component IDs and metadata about the team, project name, and other details.
What Problems Does Coinbase Watchdog Solve?
Coinbase created Watchdog to solve several crucial problems within its organization.
Coinbase uses Datadog to collect system and application metrics, implement SLIs and SLOs, create dashboards, and more.
Coinbase had seen the number of dashboards and monitors grow over time, which is when the company realized they needed to codify these dashboards and monitors.
“We were worried that we didn’t have tools to detect accidental or malicious modification. Imagine a production incident that was not noticed by engineers because of an accidentally muted monitor.”
By codifying dashboards and monitors, modifications are explicit (through code) and stored in version control, where they benefit from notification and code review systems.
As mentioned above, Coinbase Watchdog solves these problems through both a code-driven and UI-driven approach. It takes the best features from both systems and combines them into an effective tool.
How Coinbase Watchdog Detects Changes
Coinbase Watchdog can detect changes in two ways: full sync and incremental.
Full Sync: When Watchdog is launched for the first time, it queries all components by ID and checks against components stored in GitHub. If some component files are different, new Pull Requests will be created based on the user configuration file.
Incremental: Watchdog can also watch for incremental changes. Watchdog does this in several ways. It can poll the Datadog APIs every 10 minutes (or any customized length of time) to check for modified files. Or, Watchdog can use a git implementation written in Golang to pull the latest changes.
Through either of these two methods, Watchdog can create a new Pull Request, then notify the relevant team to review the Pull Request. To do that, Watchdog uses the GitHub CODEOWNERS feature.
In the future, Coinbase plans to add more features to Watchdog, including a way to automatically revert changes when a Pull Request expires.
You can view more technical details about Coinbase Watchdog in the blog post announcing the launch here.