Since our models will be deployed on customers’ production clusters, they must be lightweight and performant ideally fast enough to handle data at line rate with minimal CPU overhead. Now let’s dive in and discuss how we got to this point. Both of these are actionable insights we can debug further. Immediately, we see that the “email” cluster of requests has significantly higher average p99 latency than the other clusters, and we see that the “product” cluster has occasional latency spikes. Using this view, you can quickly determine the three broad categories of requests coming into the frontend service, as well as the latency profiles of those requests. Here you can see a plot of the average 99th percentile response latency for requests for each semantic cluster. For example, we can look at a graph of the network connections within the Online Boutique application: Once we have Pixie deployed to a kubernetes cluster running Online Boutique, we can start to explore. We will use Pixie to explore a demo application called Online Boutique. To demonstrate this, let’s walk through the end result and then we’ll come back to how we got to this point. Instead of forcing the developer to sift through many individual HTTP requests by hand, we can instead cluster the HTTP requests semantically and then show them a timeseries of latencies for each type of semantically clustered request. Suppose a developer using Pixie wanted to get an idea of which types of HTTP requests are particularly slow. In this article, we’d like to share our experience and efforts here, in the hopes they are useful to inform your thinking on similar problems. To achieve this, we leverage state-of-the-art NLP techniques to learn the structure of the data. We foresee a future where this unstructured machine data can be queried as easily and efficiently as the structured data. For this blog post, we will focus on the vast amounts of unstructured data we collect in Pixie such as HTTP request/response bodies.

These are just two examples, we collect many other types of data, as well. For example, we collect structured information about CPU and memory usage for each process in their system, as well as many types of unstructured data (for example, the body of an HTTP request, or the error message from a program). We achieve this by providing developers easy access to an assortment of metric and log data from their production system. A guest post by James Bartlett and Zain Asgar of Pixie.Īt Pixie, our goal is to enable developers to quickly understand and debug production systems.

Pixie 4 software