Beautiful Testing: Leading Professionals Reveal How They Improve Software (Theory in Practice)
by Adam Goucher and Tim Riley · 13 Oct 2009 · 351pp · 123,876 words
have its testing practices evolved in order to handle these complexities. We now use more sophisticated techniques such as reference leak testing, automated continuous builds, Valgrind,‡ and Coverity§ or Klocwork‖ to supplement standard testing. Every one of these techniques helps add to our confidence that each release of Python is at
…
is not always followed, we have found it to be a valuable one and try to stick with it. † http://www.python.org/ ‡ http://www.valgrind.org/ § http://www.coverity.com/ ‖ http://www.klocwork.com/ # http://www.python.org/dev/committers * http://www.vimeo.com/1093745 120 CHAPTER NINE The Python
…
and 2.6.2) also go through rigorous testing with a no new feature policy. Only bug fixes are allowed in micro releases.§ Dynamic Analysis Valgrind is a dynamic analysis‖ tool. This means that it analyzes an application while the application runs. The memcheck tool that is part of
…
Valgrind detects memory leaks and errors, including invalid memory reads and writes. Using dynamic analysis is very powerful because it can find issues that are otherwise
…
only once in a million times and doesn’t always cause a noticeable problem. Such an issue can be detected with little effort when using Valgrind. Python has used dynamic analysis tools for over 10 years, to minimize the amount of memory leaked. Memory leaks can be very detrimental to applications
…
have a chance to find them. When there was a massive upheaval of code in the transition from Python 2 to Python 3, we ran Valgrind periodically to ensure that at each stage there weren’t too many errors. Since we’ve been running these tools and fixing the problems, stability
…
continuous refleak checkers help identify those problems early. We run both static analysis tools (such as Coverity and Klocwork) and dynamic analysis tools (such as Valgrind and Fusil) for even more comprehensive coverage. Each and every one of these testing tools has proven its worth and caught problems that otherwise would
…
function graph, we should be using dynamic binary analysis (DBA) and instrumentation tools. One of the widely used open source DBA tools is Valgrind. There is a Valgrind extension that is of particular interest to us: Callgrind. Callgrind generates a reliable and comprehensive call graph from a debug or optimized build of
…
an executable. Make sure that you have set up the Valgrind environment on the test box properly before you execute the Callgrind extension. Callgrind generates an output file in the form of a text document. With
…
Let us say that your program is called “foobar” and you have 20 test cases in your test base. Now, we execute the following command: valgrind -tool=callgrind ./foobar test1 This generates an output file callgrind.out.pid, where the “pid” is a number. This number will be unique for each
…
,261 foobar.c:Draw [/home/your/path/project/foobar] ... * For more details on how to execute Callgrind and KCachegrind tools, see the following resources: http: //valgrind.org/, http://kcachegrind.sourceforge.net/cgi-bin/show.cgi/KcacheGrindCalltreeFormat, http:// kcachegrind.sourceforge.net/cgi-bin/show.cgi/KcacheGrindIndex, http://kcachegrind.sourceforge.net/cgi-bin
…
how to generate caller–callee dependencies among the functions in the source code of the executable using dynamic binary analysis, with the help of the Valgrind tool suite. We also have identified the test case to source files mapping from the Callgrind tool. To do this, we need a debug or
…
more optimal and easier than the static approach, especially when the problem is reproducible. The ClamAV project most often makes use of Valgrind, Electric Fence and DUMA, and Mudflap. Valgrind Valgrind is both powerful and easy to use. Its major advantage is that the application you want to test doesn’t need to
…
be modified at all, and Valgrind works directly with binaries, meaning that your program can be written in any language (also interpreted ones!). It’s not only a memory debugger; in
…
synchronization problems. The original version only supports Linux on some specific hardware platforms (x86, amd64, ppc32/64), but there exist unofficial ports to other systems. Valgrind emulates an environment similar to the one the program is supposed to run in. Since it also emulates a CPU, the execution is significantly slower
…
to 50 times slower). That’s why it’s very helpful to have some fast box at hand. See Example 20-2. EXAMPLE 20-2. Valgrind reporting memory corruption in one of the unpackers for compressed executable files in ClamAV 0.92 ==18030== Invalid write of size 1 ==18030== at 0x4E6D92A
…
) 0x405737: checkfile (manager.c:651) 0x40675A: scanfile (manager.c:1093) 0x40733D: scanmanager (manager.c:371) 0x404EA5: main (clamscan.c:213) Electric Fence and DUMA Besides Valgrind, we make use of Electric Fence and its fork, DUMA. They both are malloc() debuggers and use the virtual memory hardware of the system to
…
systems it’s possible to avoid recompilation/relinking by preloading the libraries. These tools usually don’t cause as much of a runtime slowdown as Valgrind (although significant slowdowns may occur when the code does lots of small allocations) but primarily make the applications much more memory-hungry; therefore, their use
…
memory leaks may occur at very rare execution paths, which highly complicates their detection. When looking for memory leaks, we usually run our scanner under Valgrind and test large collections of files. We have also developed our own tool that wraps calls to memory and file functions and checks for memory
…
and descriptor leaks. Since its overhead is very low, it’s more practical than Valgrind when we’re just looking for leaks and need to scan huge amounts of data. Since memory management errors may occur in any code unit
…
, 166 teamwork, 157 testing according to regulations, 168–169 memory checkers, 273–275 Electric Fence and DUMA, 274 INDEX 325 limitations of, 275 Mudflap, 274 Valgrind, 273 memory leaks, 124 Mersenne Twister algorithm, 131 message stanzas, 86 Microsoft Office customer improvement program, 56 mindmaps, 242 Mothra, 250 Mozilla Calendar Project, 27
…
, 94–97 XMPP request-response protocols, 89–93 upstream bug tracking, 72 User Community Modeling Language (UCML), 45 user expectations, 55 user satisfaction, 58 V Valgrind, 120, 124, 149, 273 variance test, 136 VersionResponder test, 90 logic test, 92 with structured XML, 91 volunteers, 28 keeping engaged, 32 recruiting, 31 W
ZeroMQ
by Pieter Hintjens · 12 Mar 2013 · 1,025pp · 150,187 words
for memory management, here’s a short tutorial on using valgrind, which, among other things, will report on any leaks your programs have: To install valgrind, such as on Ubuntu or Debian, issue: sudo apt-get install valgrind By default, ØMQ will cause valgrind to complain a lot. To remove these warnings, create a
…
file called valgrind.supp that contains this: { <socketcall_sendto> Memcheck:Param socketcall.sendto(msg
…
exit cleanly after Ctrl-C. For any application that exits by itself, that’s not needed, but for long-running applications, this is essential. Otherwise, valgrind will complain about all currently allocated memory. Build your application with -DDEBUG, if it’s not your default setting. That ensures
…
valgrind can tell you exactly where memory is being leaked. Finally, run valgrind as follows (all on one line) valgrind --tool=memcheck --leak-check=full --suppressions=valgrind.supp someprog After fixing any errors it reports, you should get the pleasant message
…
ØMQ applications; you need to see the message flows to make any sense of what is going on. For testing, I always try to use valgrind, which catches memory leaks and invalid memory accesses. In C, this is a major concern, as you can’t delegate to a garbage collector. Using
…
Design Cut: The Protocol for Zyre project, Writing the Unprotocol–Writing the Unprotocol up-front testing, On Up-Front Testing–On Up-Front Testing V valgrind, Detecting Memory Leaks–Detecting Memory Leaks W Weather Update Proxy example, Transport Bridging–Transport Bridging Weather Update Server example, Getting the Message Out–Getting the
Building Secure and Reliable Systems: Best Practices for Designing, Implementing, and Maintaining Systems
by Heather Adkins, Betsy Beyer, Paul Blankinship, Ana Oprea, Piotr Lewandowski and Adam Stubblefield · 29 Mar 2020 · 1,380pp · 190,710 words
/Linux version of the OpenSSL library provides one notorious example of a major failure caused by a small change. An open source developer noticed that Valgrind, a standard tool for debugging memory problems, was reporting warnings about memory used prior to initialization. To eliminate the warnings, the developer removed two lines
…
continuous build and test automation harness. The list of pitfalls to check is language-dependent. This section presents some solutions for C++ and Go. C++: Valgrind or Google Sanitizers C++ allows for low-level memory management. As we mentioned earlier, memory management errors are a leading cause of security issues, and
…
(buffer overflow attack scenario) Reading uninitialized memory Memory leaks when a system loses the address of allocated memory or doesn’t deallocate unused memory early Valgrind is a popular framework that allows developers to catch those sorts of errors, even if unit tests don’t catch them
…
benefit of providing a virtual machine that interprets a user’s binary, so users don’t need to recompile their code to use it. The Valgrind tool Helgrind can additionally detect common synchronization errors such as these: Misuses of the POSIX pthreads API (e.g., unlocking a not-locked mutex, or
…
caused by accessing memory without adequate locking or synchronization Alternatively, the Google Sanitizers suite offers various components that can detect all the same issues that Valgrind’s Callgrind (a cache and branch prediction profiler) can detect: AddressSanitizer (ASan) detects memory errors (buffer overflows, use after free, incorrect initialization order). LeakSanitizer (LSan
…
-point types that will overflow the destination). The main advantage of the Google Sanitizers suite is speed: it’s up to 10 times faster than Valgrind. Popular IDEs like CLion also provide first-class integration with Google Sanitizers. The next chapter provides more details on sanitizers and other dynamic program analysis
…
issues in programs) and code coverage report generators are the best-known types of dynamic analysis. The previous chapter introduced the dynamic program analysis tool Valgrind, which provides a virtual machine and various tools to interpret a binary and check whether an execution exhibits various common bugs. This section focuses on
…
Versus Bugs buildsdefined, Concepts and Terminology verifiable (see verifiable builds) C C++, How Fuzz Engines Workfor publicly trusted CA, Programming Language Choice sanitizing code, C++: Valgrind or Google Sanitizers CA/Browser Forum Baseline Requirements, Background on Publicly Trusted Certificate Authorities California Department of Forestry and Fire Protection, Handovers, Culture of Sustainability
…
layers threat modeling, Risky APIs Google design document template, Example: Google Design Document Google Front End (GFE), Example: Google’s frontend design Google Sanitizers, C++: Valgrind or Google Sanitizers, Dynamic Program Analysis Google Search, Threat modeling insider risk governmentsas attackers, Governments and Law Enforcement-Protecting your systems from nation-state actors
…
Productivity and Usability-Increase Productivity and Usability user experience, failures' effect on, User experience user productivity, least privilege and, Impact on User Productivity V Valgrind, Evolution, C++: Valgrind or Google Sanitizers validation, continuous (see continuous validation) velocity, initial versus sustained, Initial Velocity Versus Sustained Velocity-Initial Velocity Versus Sustained Velocity verifiable builds
Coders at Work
by Peter Seibel · 22 Jun 2009 · 1,201pp · 233,519 words
, I run it under strace and see exactly what's happening. If I could only have one tool, it would probably be that. All the Valgrind tools, Callgrind and all that, those are good. But a lot of times lately, if there's something weird going on, I'm like, “OK
…
of automatic verification of some claims you're making in your code. You won't get all of them, right? And the dynamic tools like Valgrind and its race detectors, that's great too. There's no silver bullet, as Brooks said, but there are better languages and we should migrate
…
what's respected in academic research circles and what's already practice in the industry. That's compilers and VM stuff, debuggers even—things like Valgrind—profiling tools. Underinvested-in and not sexy for researchers, maybe not novel enough, too much engineering, but there's room for breakthroughs. We're working
…
you have to really start looking at harder-to-use tools, which have only come to the fore recently, thanks to those gigahertz processors, like Valgrind and Purify. Instrumenting and having a checked model of the entire memory hierarchy is big. Robert O'Callahan, our big brain in New Zealand, did
…
his own debugger based on the Valgrind framework, which efficiently logs every instruction so he can re-create the entire program state at any point. It's not just a time-traveling
…
that's one way I recognize talent. And I think that's why we're hiring superhackers. I think we're hiring up all the Valgrind hackers. Some of those guys can do anything; they don't fuck around. Seibel: So is that something you often do in interviews: get them
Autotools
by John Calcote · 20 Jul 2010 · 555pp · 119,733 words
a logfile—in order to debug memory block overruns, for instance. This sort of technique is used by such widely known debugging aids as the valgrind package.[82] In the following example, the LD_PRELOAD environment variable is set on the same command line used to execute the df program. This
…
soft references, which don't need to be fully resolved until the program is executed. [82] For more information on the Valgrind tool suite, see the Valgrind Developers' website at http://valgrind.org/. [83] Unix-like (POSIX) systems will retain deleted files for which outstanding file handles exist within running processes. From the
The Debian Administrator's Handbook, Debian Wheezy From Discovery to Mastery
by Raphaal Hertzog and Roland Mas · 24 Dec 2013 · 678pp · 159,840 words
). Indeed, the Debian maintainer had made a change so that applications using it would no longer generate warnings when analyzed by memory testing tools like valgrind. Unfortunately, this change also meant that the RNG was employing only one source of entropy corresponding to the process number (PID) whose 32,000 possible
…
OpenSSL project, as well as with the Debian package maintainer. A widely used library like OpenSSL should — without modifications — not generate warnings when tested by valgrind. Furthermore, the code (especially the parts as sensitive as the RNG) should be better commented to prevent such errors. The Debian maintainer, for his part
Masterminds of Programming: Conversations With the Creators of Major Programming Languages
by Federico Biancuzzi and Shane Warden · 21 Mar 2009 · 496pp · 174,084 words
I use a debugger, frequently it’s only to do a where to find where the code is crashing. For C code, a tool like Valgrind or Purify is essential. What is the role of comments in the source code? Roberto: Very small. I usually consider that if something needs comments
The Art of UNIX Programming
by Eric S. Raymond · 22 Sep 2003 · 612pp · 187,431 words
vendor's system and run the lint utility over your software. Run tools that look for memory leaks and other runtime errors; Electric Fence and Valgrind are two good ones available in open source. For Python projects, the PyChecker program can be a useful check. It often catches nontrivial errors. If