Don’t Just Read the Docs; Read the Source
Writing this article feels like confessing to a crime, but, here we go. I made an assumption about a third party library, and now it’s crashing my app in production. It’s not the libs fault; I read the documentation, but I did not read the source.
My application is an API service written in golang that at its core depends on Redis PubSub. I researched which golang Redis libraries supported PubSub and made a well-informed decision. I read the documentation and implemented a working MVP. Despite what felt like “putting in the effort” I now have an application that crashes hard after a day or so of light use.
The breakdown occurs when a user visits a page that initiates a long-lived SSE connection (Server Sent Events; a persistent connection that allows the server to send data to a client). The route handler for that page subscribes to a Redis queue via a second long-lived connection and listens for messages. When a message gets published to the queue, it’s picked up by the route handler and sent to the page via the SSE connection. However, each time the page is visited the route handler never exits, even after the tab/browser has closed.
The assumption was that if no messages were published the subscription connection to Redis would eventually timeout and the Redis library would return. Then, the route handler would loop, resubscribe to Redis, or exit if a client-side hang up occurred.
Naturally, this does not happen. It took me less than ten minutes of tracing through the library code to discover that the subscription method had a loop of its own and that when the timeout occurred it would sleep for x seconds, then reconnect to Redis to keep listening. It does this until a message is received. As people used the service, it would open more connections to Redis but being unable to close them the service would eventually panic.
Thankfully, nobody uses my app (it’s not ready for the public, clearly) so I’m getting away with just a slap on the wrist. As programmers, we know we should investigate how imported libraries work, but it’s easy to overlook. When you import a library for the first time; read the code, understand how it works, don’t make assumptions. Just a friendly reminder.
