Nearing the end of a recent project at work, we ran across an issue that didn't rear its ugly head until our new code was put under a load by a few (patient) customers, who were willing to put our app through the paces before it rolled out to everyone. Apps can always benefit from more unit testing, load testing, and all kinds of other testing, but in the end nothing can perfectly substitute for "the real world". It's awesome that some of our customers are willing to try new things out, and I'm grateful they are.
What was the issue? Well, as part of some new requirements, we found ourselves calling the SSRS API, which is pretty convenient if you find yourself having to programmatically access SSRS, but it's not too flexible. We wanted to verify that a certain property on a scheduled report wasn't being reused, so we needed to get a list of all subscriptions first and check their properties. The problem is that there's no option to say, "I only want those subscriptions where x property is set to y value". It's an all or nothing sort of thing, and in no time we hit some kind of limit.
The API call threw an exception that suggested it was returning too much data, and the limit was 64 kb. That's a pretty small amount IMO, especially since it seemed to occur after only 50 subscriptions had been created. That's about 1311 bytes per record, an easy limit to hit when each record includes report parameters and their values, meta data about the reports, meta data about when they're scheduled to run, etc. We weren't even sure if the limit was strictly an SSRS one, per se, or maybe a limit for SOAP API calls in general.
We got to work seeing how we could increase the limit, researching the error from multiple angles. Could it be fixed by passing some extra value to the SOAP API? It didn't seem like it. Maybe something in the app when it starts up? No, probably not. How about a setting in SSRS? Maybe, but we'd need a fix we could deploy that wouldn't require every customer manually changing a setting. Even if we figured it out, what was a reasonable limit? 1 MB? 10 MB? Increase it to 10 GB so we never have to worry about it again?
Wait a sec. Maybe we're focused on the wrong fix. Or a fix for the wrong (original) solution.
Sure, we were solving the problem in front of us, but what if our original solution was wrong? The SOAP API was convenient, and we used it in a few other places where it made complete sense, but in no time at all it had become a golden hammer. We were using it because we were already familiar with it, and it could do what we wanted, albeit poorly in this instance.
The app already had access to the database the API was hitting, so what if we just wrote a stored procedure that let us specify exactly what we were interested in, and returned only the piece of data we needed? It was written a couple hours later, deployed and testable a couple hours after that, and proved to be a better solution that's more flexible and resource-friendly.
Sometimes there's a place for focusing on trees. The devil's in the details, as they say. Executives and upper management look at the "big picture", and then it trickles down and down, where people get more and more focused. Eventually, someone's gotta flesh out the finer details, and that's up to the more technical folks. We're paid to sweat those details.
Other times, there's a place for focusing on the forest. I need to know what the larger end goal is too, so I can make better suggestions during planning and, ideally, notice things that won't scale well in the long run. Even without the system throwing some error about a limit, it was a subpar solution. Eventually it'd be returning a list 100 subscriptions, then 1000, then maybe 10,000.. all so we could filter it down to one potential match.
Solving the immediate problem isn't always enough. In this case, we were so focused on the tree in front of us (the immediate problem), and of reaching the finish line (looming large in front of us), that we forgot there was a forest around us that needed tending to, too.