Honeycomb

A great fit for anyone struggling to make sense out of their systems

Comments: At QuintoAndar we have dozens of engineering teams maintaining over a hundred services. Honeycomb became the go-to solution when we need to dive in and understand what is happening with them. Our biggest use case is by far troubleshooting incidents. Thanks to Honeycomb's blazing fast query times, once we know something is wrong, we are often able to quickly pinpoint exactly what is going on and fix it faster than before. But we also started to use it to keep track of Service Level Objectives and even help onboarding new engineers, for example by showing some ways user requests map into our infrastructure. Finally, the customer support was great. Since we started the trial we had a direct channel with both their sales and engineering teams who were always very quick to answer any questions and act on any feedback.

Pros:

Honeycomb has made it very easy to ingest any kind of structured data and "slice and dice it" to figure out what is really happening with our systems. We have extracted a lot of value from database query logs, load balancer logs, CDN logs, even Kubernetes events! However, it's at analyzing application traces where it really shines. The game changer is the "bubble up" feature: it tells you which dimension of an event is more likely to be responsible for an anomaly, and that is super useful when you are troubleshooting an incident or want to answer tricky questions such as "which API calls are causing us to not meet our Service Level Objectives?". Finally, I have to add: Honeycomb's pricing model (event-based instead of machine/CPU/license based) is probably the most cloud-friendly option out there.

Cons:

Some aspects of the UI/UX could be improved. It is easy to jump on and start asking meaningful questions about your data, but it can also be a bit overwhelming to new users who are more accustomed to more traditional "point-and-click" interfaces. The online interactive tutorials are very well made and do help a lot though.

Honeycomb

Comments: Honeycomb is what you get when you put really smart Ops people together with really smart Dev people to solve both of their problems. The APIs have been very reliable. I haven't lost a byte. The bytes I expect to be there are always there. Outages are well communicated and quickly remediated. Plenty of transparency. Every single person I have interacted with at Honeycomb over the years have been great to work with.

Pros:

The query interface is a joy to work with, literally. It is literally the product I've wanted for decades. I even started building pieces of it a very long time ago internally, but Honeycomb makes every prototype I've ever built look like the toys they were. I can't say enough good things about it. The query history and social debugging features are GREAT and are a good example of how the overall Honeycomb philosophy of promoting engineer happiness shine through. I trust that the product choices they make are as much to make my job easier as it is to make buckets of money. I could be here writing for hours and I don't know what the character limit is here, so just suffice it to say that the Query UI is amazing, the data ingestion APIs are very good, everything is well and thoughtfully designed and I can use them in a myriad of fully supported ways. Nirvana!

Cons:

The only con I can think of is a lack of off-the-shelf support for PHP, which we still use quite heavily. Support is getting there through OpenTelemetry, etc. Getting data INTO Honeycomb can be a challenge, but in the year and more that we've been using it even that con is shrinking, and quickly.

Answer all the questions!

Pros:

Honeycomb does exactly what it says it does. The flexible data model means we can freely instrument our code without worrying about massaging it into a form that it will accept. In fact, our codebase was already instrumented using OpenTracing before we adopted Honeycomb. We were able to point our OT client to Honeycomb (via a proxy they provide) and it just worked. The query engine is flexible and fast, which makes it easy to explore our data without demanding intimate familiarity -- this is critical when debugging issues in a distributed service where I don't always know exactly how the events are structured, or what exactly I'm looking for. Honeycomb's event-based pricing model means we can shove whatever information we think might be useful into our events without worrying about blowing the budget. This is critical to maintaining a well-instrumented codebase, as we don't want our developers having to make value judgements about what to report during development. Combined, these attributes mean that any question I can think to ask can almost certainly be answered by Honeycomb. It does not take long to become dependent on this ability. In fact, when our trial period was about to expire, a member of my team said, "you can pry Honeycomb from my cold, dead hands!"

Cons:

Honeycomb is excellent for exploring the system to get to the bottom of an incident. However, it is less useful for more classic monitoring use-cases. It does have dashboard and trigger features, but they are fairly rudimentary at this time, and insufficient for proactively monitoring the health of a complex service. In Honeycomb's defense, they do not position themselves as a monitoring solution, so this isn't particularly surprising, but it does mean that a separate monitoring solution is also required.

Opens up world of opportunities

Comments: I am now requiring Honeycomb integration on all software I build and help maintain. It is the highest praise I can give.

Pros:

It opens up new avenues in software engineering: what if you could ask your system how it's doing? Instrumenting everything, sampling sanely, and all the relevant data is in Honeycomb at a reasonable price. From then on forward, all the querying options are available. Bubble Up is there to help massively during incidents to find the outliers and where the problem lies. Once this is implemented properly( not an easy feast), it feels like living in the future.

Cons:

Coming from a classic( and wrong) observability=logs+metrics+traces mindset it is rather hard to get accustomed to Honeycomb. Once the mental hurdle is passed it's all sunshine and rainbows, but it takes a whole lot to get there. Fortunately, their Solution Engineers are very patient and helpful.

Best Tool Ever

Comments: Very easy to set up and get data into Honeycomb. We leverage AWS Lambda extensively and send all of our Cloudwatch log groups to a Kinesis stream, which another Lambda consumes & publishes to Honeycomb. We've been able to find patterns that would have been difficult to surface in other tools and the SLO functionality provides a good target for your application.

Pros:

The ability to explore your application data in ways you did not know you needed, but once you get a taste you're hooked.

Cons:

I can't think of anything about Honeycomb that I do not like. One thing to keep in mind when implementing it is to ensure your log data is well-structured with standardized field names, otherwise you won't get the most from it.