A Series on Profiling, Benchmarking, Code Coverage, and Other Helpful Development Tools
As our team was nearing the first big release of an ecommerce project we ran into what seemed like an insurmountable usability bug. Annoyingly, every time we made an update to the shopping cart there was suddenly a significant pause. It was an unacceptable user experience, and we had to root out the issue quickly.
Throughout development we had assumed this delay was just an issue with latency in the network, but after looking at how quickly the data was being returned to us, we realized that speed of data access was not the problem. The bulk of our delays could be blamed on the front-end user interface code. All of our tests were passing, our logs appeared normal, and the debugger showed our code executing precisely as expected. It was time to reach for another tool, one that would help us diagnose and fix the performance problem before other, even more pressing release preparations took over.
In that instance, a profiling tool in our web browser helped us discover that our user interface was completely re-rendered hundreds of times, which caused noticeable lag whenever a customer changed their product selections. Like that profiling tool, there are several other software development tools that, while they may not be what we reach for multiple times a day, are indispensable when the time comes. This is the first in a series of blog posts about these tools, the crime scene detectives of the development world. We’ll start with a survey in order to introduce each of the most common types of tools.
PROFILING/PERFORMANCE ANALYSIS TOOLS
Reach for this when…
your program is experiencing significant latency (delays), is using an abnormal amount of energy, memory, or is having an issue with throughput (scalability).
Works by…
observing the execution of a program, sometimes using statistics to estimate and summarize performance.
Typical usage is…
compiling your program with profiling support (not always needed) and running it with a profiler (hypervisor) attached to the process to examine CPU, GPU, memory, or energy use of the entire program or a sub-component.
Examine the results by…
(typically) looking at a graphical rendering of the processes within your program alongside their resource usage then investigating the heavier processes at a method call or source code level to check if the usage is expected.
Act on the results by…
changing the code in sections that are using more resources than expected and profiling again or using another tool.
Find it…
from the relevant programming language vendor/community, operating system vendor, or CPU/GPU vendor.
BENCHMARKING TOOLS
Reach for this when…
you want to track the performance of key activities of your software over time, typically to identify regressions—making sure that everything is still performing as it should.
Works by…
observing a repeatable execution of program activities, sometimes using statistics to estimate and summarize performance.
Typical usage is…
adding instrumentation instructions before and after a section of code that can be executed repeatedly to gauge its performance and then measuring and recording benchmarking results for each software build executed on your CI (continuous integration) server. Tip: benchmarking should not be performed during the normal execution of your program.
Examine the results by…
comparing performance results over time and identifying any deviations outside of an expected range, usually defined statistically.
Act on the results by…
identifying the specific code changes that correspond to a regression by either using profiling or guessing and checking to determine if and how the code should be adjusted to meet the previous standard.
Find it…
from the relevant programming language vendor/community: Google’s Caliper (Java), Go’s native benchmarks, and BenchmarkDotNet (C#/F#) for example.
SYSTEM CALL TRACING
Reach for this when…
you want to view the pattern of operating system kernel interactions (system calls) made by a program so you can learn why it is behaving unexpectedly or get a baseline for what behavior to expect. This can help you improve security by preventing unexpected behavior.
Works by…
observing interactions between the kernel and a program. Here’s a comic with a beginner-friendly explanation.
Typical usage is…
invoking a system call tracer on the command line and passing in the program you wish to trace as an argument.
Examine the results by…
combing through the trace output line by line or viewing a summary produced by the tracer itself and noting either expected or unexpected system calls.
Act on the results by…
investigating the cause of unexpected system calls or providing the list of expected system calls to a security sandboxing tool.
Find it…
from the relevant operating system vendor: dtrace, truss, dtruss, ktrace, and strace are among the most common implementations.
DISTRIBUTED TRACING
Reach for this when…
you have multiple networked programs and you want to see the path an individual request takes through the system and where errors are generated.
Works by…
tagging each request with an ID as it enters the system, propagating that ID from program to program, and using a separate program to continuously collect information from each program.
Typical usage is…
modifying your code to be compatible with the trace collection service in order to provide key request details that you can later use to find or examine the request.
Examine the results by…
accessing the trace collection service and searching for: • the longest request/response times • a particular request that showed up in a log • requests with errors
Act on the results by…
reaching for a profiling tool and analyzing programs that are responsible for poor performance or by debugging programs that produce errors.
Find it…
by looking for supported tracers or from your cloud infrastructure provider.
MONITORING/METRICS APPLICATION
Reach for this when…
you are running services subject to variable load in terms of quantity or impact, especially load from the Internet, and you want to be alerted when your services are not behaving normally.
Works by…
using a dedicated service to continuously collect metrics from one or more services.
Typical usage is…
integrating your program and hosting platform with a dedicated monitoring service.
Examine the results by…
looking at the data offered by the monitoring service and determining which of the metrics and what ranges of their values warrant sending an alert.
Act on the results by…
investigating alerts from the monitoring application and taking actions necessary to keep your service operational.
Find it…
by searching for a monitoring, metrics, or application performance management (APM) tool that works well in your environment or from your cloud infrastructure provider.
CODE COVERAGE
Reach for this when…
you are conducting automated testing of your program and wish to assess how much of your codebase is being tested overall and identify which parts are or are not being tested.
Works by…
observing test execution and mapping it back to the source code that was executed.
Typical usage is…
running unit tests and passing in a special flag that will generate a coverage report or by invoking a dedicated coverage tool on the command line and passing in the unit test program as an argument.
Examine the results by…
converting the report to a human-readable format, if needed, and browsing through the coverage results.
Act on the results by…
modifying tests to call additional methods and use different method arguments to check scenarios not covered by the current tests.
Find it…
from the relevant programming language vendor/community.
STATIC ANALYSIS/LINTING
Reach for this when…
you are writing code, so you can learn and follow conventions and best practices and catch bugs as early as possible in the development cycle.
Works by…
integrating with IDEs, editors, or build tools to analyze source code or compiled code directly, without executing it—this acts in a similar way to a spellcheck functionality.
Typical usage is…
running a linter and some static analysis tools as you are programming to provide immediate feedback and run deeper analysis as part of the build process.
Examine the results by…
reading through each warning or error generated.
Act on the results by…
researching any suggestions you do not understand and then either making code improvements or instructing the analysis tool to ignore a particular section of code in future analyses.
Find it…
from the relevant programming language vendor/community.
BISECTING WITH VERSION CONTROL
Reach for this when…
your code is in version control and you want to identify the code changes responsible for a particular behavior, for example determining when a bug or regression was introduced.
Works by…
optimally searching through revision history to find a particular change.
Typical usage is…
searching between the last known ‘good’ revision and the current one and specifying a script to execute on each revision in the search without manual intervention.
Examine the results by…
noting the revision identifier found by the bisection.
Act on the results by…
examining code changes in the revision found by the bisection.
Find it…
through your source control vendor. Here is documentation from Git and Mercurial.
USE THE RIGHT TOOL FOR THE RIGHT JOB
Logging and debugging are the mainstays of software development – so much so that most programming language introductions begin with a “hello world” example that teaches a rudimentary form of both. Remember though—even MacGyver, who always had his trusty Swiss Army knife, made use of chewing gum sometimes. Be ready to reach for more specialized tools like the ones mentioned in this post and you will be far better equipped to meet your own unique challenges. Feel free to reach out to us at findoutmore@credera.com with any questions or comments. Stay tuned for our next episode!
Contact Us
Ready to achieve your vision? We're here to help.
We'd love to start a conversation. Fill out the form and we'll connect you with the right person.
Searching for a new career?
View job openings
