[libvirt-users] converting save/dump output into physical memory image
A lot of people in the security community, myself included, are interested in memory forensics these days. Virtualization is a natural fit with memory forensics because it allows one to get access to a guest's memory without having to introduce any extra software into the guest or otherwise interfere with it. Incident responders are particularly interested in getting memory dumps from systems they're investigating. Virsh has "save" and "dump" commands for storing the state of a guest to a file on disk, but memory of KVM guests doesn't get saved in the "standard" input format for memory forensics tools, which is a raw physical memory image. (This is what you'd get via the classical "dd /dev/mem" approach or the contemporary equivalent using the crash driver; and VMware Server and Workstation produce .vmem files, which are such raw physical memory images, when a guest is paused or snapshotted.) In order to analyze the memory of Libvirt/KVM guests with my Linux memory forensics software, Second Look, I've created a tool for converting Libvirt-QEMU-save files (output of virsh save command) or QEMU-savevm files (output of virsh dump command) to raw physical memory images. I've got a basic working capability, though I'm still tracking down some problems with a guest allocated 8GB RAM--not all the memory seems to be present in the save or dump file. And I haven't tested very extensively yet, version support is limited to what I myself am currently running, etc. I'd like to know if this is a capability that others are interested in. Is this something that would be of interest to the Libvirt project if I were to contribute the code, or to the KVM project, or do you think it best exists as a separate project? I've also got a proof-of-concept tool for converting hibernate images to raw physical memory images. Perhaps a collection of tools for converting various memory dump formats would be a good project. Anyone else interested in this kind of stuff? As an author of commercial memory forensics software I've got a vested interest in availability of good memory acquisition capabilities. But there are a number of people working on FOSS Linux memory analysis tools, too... Andrew
On 05/05/2011 11:56 AM, Andrew Tappert wrote:
A lot of people in the security community, myself included, are interested in memory forensics these days. Virtualization is a natural fit with memory forensics because it allows one to get access to a guest's memory without having to introduce any extra software into the guest or otherwise interfere with it. Incident responders are particularly interested in getting memory dumps from systems they're investigating.
Virsh has "save" and "dump" commands for storing the state of a guest to a file on disk, but memory of KVM guests doesn't get saved in the "standard" input format for memory forensics tools, which is a raw physical memory image. (This is what you'd get via the classical "dd /dev/mem" approach or the contemporary equivalent using the crash driver; and VMware Server and Workstation produce .vmem files, which are such raw physical memory images, when a guest is paused or snapshotted.)
Libvirt also has the virDomainMemoryPeek API; right now, it is not exposed by virsh, but we could add a command-line-interface for it if that proves useful. Does that API fit your needs any better than converting a qemu dump image back into raw memory? -- Eric Blake eblake@redhat.com +1-801-349-2682 Libvirt virtualization library http://libvirt.org
On 05/05/2011 02:33 PM, Eric Blake wrote:
On 05/05/2011 11:56 AM, Andrew Tappert wrote:
Virsh has "save" and "dump" commands for storing the state of a guest to a file on disk, but memory of KVM guests doesn't get saved in the "standard" input format for memory forensics tools, which is a raw physical memory image. (This is what you'd get via the classical "dd /dev/mem" approach or the contemporary equivalent using the crash driver; and VMware Server and Workstation produce .vmem files, which are such raw physical memory images, when a guest is paused or snapshotted.)
Libvirt also has the virDomainMemoryPeek API; right now, it is not exposed by virsh, but we could add a command-line-interface for it if that proves useful. Does that API fit your needs any better than converting a qemu dump image back into raw memory?
Before starting to write the LibvirtQemuSave/QemuSavevm conversion tool I did explore the possibility of writing a Libvirt memory dump program using that API. But given the way it writes to a temporary file it did not seem like it would be feasible/efficient for full memory dumps. I just signed up to your mailing list today, but I saw in your last post a mention of qemu raw monitor passthrough commands. Maybe directly invoking qemu's pmemsave is the way to go. The nice thing about the qemu savevm format, though, is that it does contain cpu state, which is actually quite relevant to "memory" forensics... it's not all about getting a raw physical memory image, actually. Andrew
On Thu, May 05, 2011 at 01:56:10PM -0400, Andrew Tappert wrote:
A lot of people in the security community, myself included, are interested in memory forensics these days. Virtualization is a natural fit with memory forensics because it allows one to get access to a guest's memory without having to introduce any extra software into the guest or otherwise interfere with it. Incident responders are particularly interested in getting memory dumps from systems they're investigating.
Virsh has "save" and "dump" commands for storing the state of a guest to a file on disk, but memory of KVM guests doesn't get saved in the "standard" input format for memory forensics tools, which is a raw physical memory image. (This is what you'd get via the classical "dd /dev/mem" approach or the contemporary equivalent using the crash driver; and VMware Server and Workstation produce .vmem files, which are such raw physical memory images, when a guest is paused or snapshotted.)
In order to analyze the memory of Libvirt/KVM guests with my Linux memory forensics software, Second Look, I've created a tool for converting Libvirt-QEMU-save files (output of virsh save command) or QEMU-savevm files (output of virsh dump command) to raw physical memory images.
I've got a basic working capability, though I'm still tracking down some problems with a guest allocated 8GB RAM--not all the memory seems to be present in the save or dump file. And I haven't tested very extensively yet, version support is limited to what I myself am currently running, etc.
KVM only allocates memory on demand, when the guest touches the page, so the data will almost certainly be sparse. FWIW, the 'crash' tool has been taught how to understand the output of 'virsh dump' for Linux guests. I agree, it could be nice to have a raw memory dump though, instead of the KVM save format, at least as an option. Daniel -- |: http://berrange.com -o- http://www.flickr.com/photos/dberrange/ :| |: http://libvirt.org -o- http://virt-manager.org :| |: http://autobuild.org -o- http://search.cpan.org/~danberr/ :| |: http://entangle-photo.org -o- http://live.gnome.org/gtk-vnc :|
Andrew Tappert <andrew@...> writes:
A lot of people in the security community, myself included, are interested in memory forensics these days. Virtualization is a natural fit with memory forensics because it allows one to get access to a guest's memory without having to introduce any extra software into the guest or otherwise interfere with it. Incident responders are particularly interested in getting memory dumps from systems they're investigating.
I am definitely interested in this concept! I haven't played much yet with memory forensic tools (like Volatility) as atm I'm focusing on other things but I'm planning to switch to it in a very near future. For what I know it's not possible to analyze memory contained in libvirt snapshots taken through qemu savevm. This feature is really powerful under my perspective as it atomically takes the state of the whole machine: disk, memory, processes; everything is saved in the same moment. I am already able to get all the useful information from the disk state: through the qemu convert I can get a qcow2 disk image containing the state of the disk at the moment of the snapshot, through libguestfs I can mount and read this image to see FS changes. I am looking for a similar solution for memory, given a snapshot taken through the savevm command I want to get a file to read through Volatility.
On 06/02/2012 12:20 PM, NoxDaFox wrote:
For what I know it's not possible to analyze memory contained in libvirt snapshots taken through qemu savevm.
qemu 'savevm' dumps in an internal migration format. But you can use the 'migrate' command to do a migration to disk, which at least some memory analysis tools have learned how to reconstruct guest memory from. There is also a pending patch to libvirt to learn to use qemu's new 'dump-guest-memory' command which creates an ELF file corresponding to guest memory at a given point in time. The idea is that virDomainCoreDump() API will map to both qemu monitor commands for dumping guest memory. With a bit more help from qemu, it may even someday be possible to take a memory snapshot while still allowing the guest to run, and libvirt's virDomainCoreDump() would gain a new flag to expose that capability. -- Eric Blake eblake@redhat.com +1-919-301-3266 Libvirt virtualization library http://libvirt.org
participants (4)
-
Andrew Tappert -
Daniel P. Berrange -
Eric Blake -
NoxDaFox