5.1 KiB
Sentry at DISQUS
Since Sentry’s conception at DISQUS in early 2010, its use inside DISQUS has grown considerably. The entire DISQUS engineering team (16 people and counting) use Sentry in one way or another, deploying 20 Sentry instances across a wide range of services.
So as not to be deadbeat parents, we recently caught up with DISQUS engineers Mike Clarke and Ted Kaemming for an update.
A little history: As DISQUS started taking off in 2009 (by 2010 it would be handling 200M unique visitors), traditional logging became problematic. A need existed for real aggregation, but also the power of introspection. In early 2010, Sentry was conceived at DISQUS to solve exception logging within a Django application (DISQUS’ back end is written primarily in Django). Today, DISQUS is the #1 online discussion and commenting service, serving 1.1 billion unique monthly visitors at 2.5 million sites. It logs on average 1-2 events per second in Sentry, with spikes upwards of 100-500 errors a second.
As the open source Sentry (GitHub repo) matured, gaining functionality and support for most popular platforms and languages (Python, PHP, Java, Node.js, Javascript, etc.), its use inside DISQUS grew, encompassing the entire engineering team. They primarily access Sentry via Python, but also Django, flask, Javascript and some in-house custom software.
On the operations side, Sentry is used primarily as an early warning system, allowing for rapid response to issues as they arise. DISQUS engineer Mike Clarke explains, “As we ship code, which we do at least daily, I’m constantly staring at Sentry. Watching the trending to see if something new is broken versus something that’s been broken for a while.”
On the development side, Sentry helps prioritize resources. “When we’re building stuff, there’s always going to be errors. It helps to find out how frequently those errors are occurring to prioritize which things get tackled first. Being able to look at Sentry over a time period, or just overall, tells us what’s really most impactful to the user”, explains developer Ted Kaemming.
Sentry is also used by support. When a user gets an error, they’re given a hash they can include in a support email. This hash gives the support person a clue as to what might be broken, and who in engineering to talk to. If the email becomes a support ticket for engineering, engineering can then use Sentry to debug, using the search and lookup features (among others) to examine the specifics of the error.
DISQUS also uses Sentry for diagnostic logging, using it to see how errors in one part of the system affect other systems. It logs warnings in addition to errors, and even uses Sentry for auditing and security tasks. “If you’re masquerading as another user doing super admin type things, we’ll log that to Sentry so you can follow if someone is unexpectedly doing bad things to somebody else. The stringing and grouping is so effective that when something out of the ordinary does happen, regardless of it’s an error or not, it’s a convenient way of visualizing it”, explains Mike Clarke.
In all, DISQUS has 20 projects in Sentry. They report that setup is surprisingly easy for new projects, and adoption shows no signs of slowing. “Sentry is great in terms of visualizing stack traces and that sort of thing, but there’s many ways we can know when our system is broken. How Sentry consolidates that into a single location is what makes it truly useful for us”, explains Mike Clarke.