Golang as an Observability Tool

Go, for good reason, has become an incredibly popular language for everything from web applications to DevOps tools. We lean on Golang heavily here at Honeycomb, and naturally we are observability nerds, so I got ridiculously excited to use the new pprof web UI to gain more insight into the performance of our Golang programs, both internal and open source.

honeyelb

Using the New Golang pprof UI

Recently we announced the new Honeycomb AWS Elastic Load Balancer integration. This is a tool to process the otherwise opaque access logs Amazon gives users to understand the request patterns that are hitting their load balancers. You can now use our open source tool to send this information to Honeycomb for later querying, allowing you to answer questions like “Which servers are performing poorly?” or “When are my applications returning a lot of errors?”.

Lucky for us, when we announced the tool our lovely customers jumped in and started trying it out right away. In the process of doing so, one identified a memory usage issue: the Go runtime was running out of memory and killing the program!

pprofpprof

Installing the pprof Tool

pprof
$ go get github.com/google/pprof
pprof-httpgraphvizsudo apt-get install -y graphvizbrew install graphviz
pprofhoneyelbinit
go func() {
  time.Sleep(120 * time.Second)
  memprof, err := os.Create("mem.pprof")
  if err != nil {
    logrus.Fatal(err)
  }
  pprof.WriteHeapProfile(memprof)
  memprof.Close()
}()
go func()mem.pprof
pproflocalhost:38687

Once booted up I saw something like this:

Generally speaking, the larger a box is, the higher the memory usage of that section of the program is. You can click around in this graph, zoom in and out, etc.

NewEventlibhoney-go

This shows us a list of the most memory-intensive functions in the program, sorted by flat usage.

runtime.hashGrowruntime.mapassignmap
libhoney.NewEvent
libhoney.NewEventPublisher
runtime.makechanbytes.makeSlicepprof

pprof UI and the Observability Mindset

pprof

Many thanks to rakyll and the rest of the Golang team for their great work! I’m looking forward to seeing more in this area such as a built-in flame graph implementation.

Until next time, observe all the things! And please do sign up to try out Honeycomb (for free!) if you haven’t already. :)