10 tips and tricks for solving software problems
As software consultants, we are often faced with bugs or problems the client cannot fix themselves. We are expected to quickly figure out the problem, and propose or implement a solution. Through my career I have developed a sort of mindset to get to the bottom of issues without wasting time hunting deadends or non-issues, and deliver results and value more efficiently.
Here is my list of 10 tips for how I go about solving software problems
A common problem I saw in myself, and I see very often in software developers, is a lack of focus.
If you are trying to solve too many issues at once, you will probably fail at all of them.
The first thing you must do, is figure out what you are actually trying to solve, and then stay on target. Usually my clients, have some sort description of what should be happening, and what isn’t happening. The description of the issues are often vague, but can be boiled down to tangible issues.
As developers, we have a tendency to hunt any bug we find. If for example you are dealing with communication issues between two services, it might not be relevant to your current task, to start debugging the datalayer or refactoring a calculation algorithm.
A side effect to lack of focus, is that your git history becomes unclear and kind of a mess. All of a sudden you have two or more half-done fixes staged, which means that either you can’t commit the first fix, because the second half-done fix is not ready, or you get commits and PR’s that resembles
git commit -m "Fixed everything".
Stay focused, and solve the task at hand. This will make you more efficient, and your team mates will love your commit history.
2. Pinpoint method / Exclusion method
I am personally a big fan of what I call the “pinpoint method”, which kind of is derived from the exclusion method. The “pinpoint method” is great at helping you narrow down exactly what the root cause of a problem is. It is a method that I have had great success teaching the junior developers that I mentor, and they have become more productive as a result of this, so I think it is worth sharing.
- Figure out what the core problem actually is
- We need to know exactly what is not happening as it is suppose to.
- Bug reports are very often lacking critical details about what actually happened, and is usually just reported as “X feature doesn’t work”, or worse “There is a problem”!
- Create end-to-end map
- Once you know what the problem is, you should figure what broad area the problem exists in.
- Create an end-to-end map of what systems are touching the feature that doesn’t work.
- Figure out what is known!
- Once you have an end-to-end map, you should go into the code, and follow the code along to figure out what you know happened correctly. You can usually do this by looking in logs or databases to figure out if the code got this far without error.
- By following the code, and excluding everything you KNOW happened correctly, you should soon find yourself with a pinpoint of where the problem occurs. This does not necessarily mean you have the root cause yet, but it should give you the information you need to figure out what the root cause actually is.
At one of my clients, an employee that I was mentoring were faced with an issue, where some users were not receiving an email.
The employee, lets call him John, quickly started signing in to Mailgun (cloud mail sending service), and checking through logs if he could find any errors on their platform. He spent a good hour on this before asking me for help.
So I first asked him to figure out step 1 & 2 in the “pinpoint method”. And the result was:
What is the core problem: Some customers are not receiving emails.
End-to-end map: When an email must be sent, it is put into a queue in the database, with the selected template id and user id that should receive it. A different service, listens on the queue, loads out the template and user to get the email address and then sends it to the mailgun server.
Great I said, now lets figure out what we know is happening. We can tell from the code that we are logging when we are queuing the mail, when we start the mail job for that email, and when the email has been sent. That means we should quickly be able to pinpoint what part of the code the problem is most likely to occur in.
In the logs we saw:
- The email got queued without error.
- The job for that email was correctly started.
- No log entry was ever created for that email being sent.
From the above information, we can with great certainty say that everything up to the point where the job started, was working perfectly. So we pinpoint the problem to be in the email job.
John started a debugging session for the email job, and quickly realized that the email job, were getting a stale cached version of the users email. This mean we now know the cause for the problem, but have not solved the root cause yet, we then start the pinpoint method over to find out why a stale version of the users email was still in the cache. Long story short, the invalidation logic was buggy for the user service.
The pinpoint method, basically forces you to focus on your task, and use your time analyzing and debugging in an efficient manner. It forces you to sort out the areas of the systems, that are not relevant, so in a way it is a binary tree sort for problem areas.
3. Do not assume
When you ass-u-me, you make an ass out of u and me. When you assume, you are basing your actions on hunches, which basically means you are gambling with your time, and putting it “all on black”. If your hunch is correct, then JACKPOT, but most often you are probably wrong and you just wasted a good amount of time hunting down the wrong assumption.
In my previous example, Johns big flaw was basing his actions on a hunch, that it must have been the mailgun service that was failing, without knowing if that was actually true. He spent a good hour on basically looking through their logs to determine that it probably wasn’t there. John could have spent his time much more efficiently.
Now, there is nothing wrong with the fact that John signed in to mailgun to quickly check if the logs showed any errors, that basically means that he did go through the pinpoint method. By knowing that mailgun is part of the end-to-end map, and knowing that an email can get stuck in the spam filters and therefor would show up in the logs, was a good idea. But he kept assuming that is was Mailgun, and therefor wasted his own time.
If you do rely on assumptions, test them! That way they become knowledge. See Test your assumptions.
4. Test your assumptions
At some point, you will have to make assumptions, and that is okay, if you handle them the right way. Don’t just go off on a tangent, test your assumptions.
Depending on the issue you are hunting, testing your assumptions, can mean that you need to log in to mailgun and check the log quickly, to see if you find the error you are assuming to find.
Other times it means you need to built a proof of concept application, that proofs that what you are assuming actually causes the bug, this same proof of concept, can then also be used to prove that you actually fixed the problem.
5. Research and proof of concepts
As developers we are quick to jump into the code, at the first chance we get. But most often it is worth doing some research before hand, that includes finding best practices, and allowing yourself time to do proof of concepts. These proof of concepts can be thrown out, once they have proven the idea you would work in your context.
Obviously Google and StackOverflow is great starting points, but I would also look to the leaders and MVPs in your industry-segment. Reading blogs such as this one, or even reaching out to developers who have faced the same challenge as you before is a great way to do research. Must people have Twitter these days, and will often respond with helpful resources or point you in the right direction.
In general I think it is a good idea to be pro-active about your learning and read blogs such as this one regularly, at least to have a mental overview of what is available and what the industry as a whole is doing.
- Scott Hanselmann
- Works on ASP.NET and Azure Cloud for Microsoft
- Jeffrey Fritz – Twitch / Youtube
- Program Manager for Microsoft on ASP.NET and .NET Community outreach teams
Latest proof of concepts I have been doing:
- OAuth2 Identity Server for ASP.NET Core
- ASP.NET Core
- Docker with .NET Core
- CI/CD pipeline improvements
Doing proof of concepts, helps to de-risk a project, so you as a developer have a better chance of estimating your tasks correctly.
6. Finding best practices
There is no reason to re-invent the wheel, if you don’t have to. Most often you will find that there is a whole community around what you are trying to do, and they often have years of battle-hardened experience implementing what you are trying to do.
Listen to their best practices, and figure out if they will work for you.
Examples of best practices I was researching at some point:
- Git branching strategies
- I have introduced a few clients to Gitflow now
- CI/CD pipelines
- I absolutely recommend this book: Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
- OAuth2 for multi-tenant applications
Best practices does not always fit your needs completely, but they are a great starting point to learn some of the pitfalls and gotchas that certain tools have.
7. Ask for help!
As a consultant, I am often dropped into projects and expected to be productive in a very short amount of time. That means I need to get up to speed immediately and add value within a short time frame. It has taught me that you should never be afraid to ask for help, we are not super-heroes and no one cares if you were brilliant enough to figure something out all by yourself.
When you get stuck, never be afraid to ask your co-workers or others for help, often they have knowledge from the project or organisation as a whole that can help you greatly when researching issues.
A good example of this was a few years ago, when I couldn’t figure out where the Protobuf files were for our protocol layer. I spent a few hours trying to find them, before giving up and asking a colleague. Within minutes I had the files, because he knew exactly where those sort of things are kept.
Learn to never be afraid to ask, and know that everyone can teach you something. You are not batman, you do not need to solve all the crimes of Gotham on your own!
Always focus on value, rather than ego.
8. Use your tools
This should be a given, but I often see developers not using the tools, or caring to learn the tools they have at hand.
I especially see younger developers that are afraid of using the debugger.
Valuable tools for C# developers:
- Version control!
- Even if you are a one man band, version control is a MUST!
- Visual Studio!
- Let’s be honest there is no C# without the best editor in the world Visual Studio
- USE THE DEBUGGER!
- Visual Studio Code
- When the Worlds best Editor is a bit too much.
- Chrome DevTools
- API calls made easy
- WebEssentials plugin for Visual Studio
- Build and deployment automation
- If you are still doing deployments by hand, you should be ashamed of yourself.
- When you are seeing performance issues, nothing beats a good profiler! Chrome has it too, use it!
- If you can script it, you can automate it!
Do not be afraid to learn how to use your tools, they will make your more productive, and maybe even earn you an extra coffee break, or some time on that pet project of yours!
9. Create your own tools
Sometimes you are doing the same things over and over again, and you start to wonder whether that task be automated. And in many cases it can!
Consider rolling your own tools from time to time, to help you become more productive.
Example of tools I have created:
- Certificate Viewer
- Viewing Certificate information and testing signatures
- SSL Certificate Creator
- Small Powershell script
- I built a realtime dashboard for a previous client, that shows a lot of metrics of what is going on inside their system. The dashboard is always shown on their 50 inch TVs in the office.
- Redis PubSub Viewer
- I used this to debug a clients Redis PubSub communication.
- Deployment scripts
- Everytime I build a new piece of software I also build a script to do deployments, even with my raspberry pi projects.
Start small, but grow your tools as you need them, at some point you will have an entire tool belt that makes your life a lot easier.
10. Remember the bigger picture
Finally, you have too see all issues in the bigger picture. Some issues are not worth spending time on right now.
This is something I see often with junior developers, where they start hunting a bug that doesn’t really matter, to avoid working on the big critical issue that the customer is complaining about.
I think this often happens because the developer finds a specific task daunting in combination with the fact that the developer doesn’t have the “business” in mind. Especially on teams where resources are limited, and there is fires to be put out, you need to focus on the tasks that brings in the most value.
John from earlier, had two tasks he could start on. One was the issue where a lot of customer were experiencing issues when logging in to the website, and another were administrators would experience that deleting supporter accounts were really slow.
John chose to focus on the Administrator task.
When I noticed he was working on that, we had a small talk about the value of tasks. On one hand, we have a critical issue where our business is being hurt by users not being able to signin, this issue is affecting a lot of people, and fixing it would bring in a lot of value.
In comparison, administrators only deletes supporter accounts when someone changes jobs, and that is a very rare occurrence. On top of that, the feature was working, but it was slow.
John needed to think about the bigger picture here, and solve the task that was most important.
If you have any tips on how you are solving problems, please them in the comments below!