Episode Details

This week on BSDNow, Allan is currently at EuroBSDCon! However due to the magic of video (or time travel), you still get a new episode. (You’re Welcome!). Stay tuned

This episode was brought to you by

href="http://www.digitalocean.com/" title="DigitalOcean"> href="http://www.tarsnap.com/bsdnow" title="Tarsnap">

Headlines

Performance Improvements for FreeBSD Kernel Debugging

• “We previously explored FreeBSD userspace coredumps. Backtrace’s debugging platform supports FreeBSD kernel coredumps too, and their traces share many features. They are constructed somewhat differently, and in the process of adding support for them, we found a way to improve performance for automated programs accessing them.”
• “A kernel core is typically only generated in exceptional circumstances. Unlike userspace processes, kernel routines cannot fault without sacrificing the machine’s availability. This means things like page faults and illegal instructions inside the kernel stop the machine, instead of just one process. At that point, in most cases, it is only usable enough to inspect its state in a debugger, or to generate a core file.”
• No one likes it when this happens. This is why backtrace.io is focused on being able to figure out why it is happening
• “A FreeBSD kernel core file can be formatted in several different ways. This depends on which type of dump was performed. Full core dumps are ELF files, similar in structure to userspace core files. However, as RAM size grew, this became more difficult to manage. In 2006, FreeBSD introduced minidumps, which are much smaller without making the core file useless. This has been the default dump type since FreeBSD 6.0.”
• The article goes into detail on the minidump format, and some basic debugging techniques
• “Libkvm will first determine whether the virtual address lies within the kernel or direct maps. If it lies in the kernel map, libkvm will consult the page table pages to discover the corresponding physical address. If it lies in the direct map, it can simply mask off the direct map base address. If neither of these applies, the address is illegal. This process is encapsulated by va_to_pa, or “virtual address to physical address”. Once the physical address is determined, libkvm consults the core file’s bitmap to figure out where in the core file it is located.”
• “minidumps include a sparse bitmap indicating the pages that are included. These pages are dumped sequentially in the last section. Because they are sparse in a not entirely predictable way, figuring the offset into the dump for a particular physical address cannot be reduced to a trivial formula.”
• The article goes into detail about how lookups against this map are slow, and how they were improved
• “For typical manual debugger use, the impact of this change isn’t noticeable, which is probably why the hash table implementation has been in use for 10 years. However, for any automated debugging process, the extra latency adds up quickly.”
• “On a sample 8GB kernel core file (generated on a 128GB server), crashinfo improves from 44 seconds to 9 seconds, and uses 30% less memory”
• “Backtrace began shipping a version of this performance improvement in ptrace in February 2016. This enables us to also offer significantly faster tracing of FreeBSD kernel cores to customers running curr