I should probably mention that I'm using Ubuntu 12.04.3 server (64-bit). Also, I noticed this page: http://wiki.libvirt.org/page/Failed_to_connect_to_the_hypervisor, but the user is part of the libvirtd group.


On Fri, Jan 31, 2014 at 2:09 PM, Joshua McKee <josjunk@gmail.com> wrote:
Hello,

I am having an issue with libvirt in my Nimbus cloud setup. When I use the Nimbus VM creation test scripts on a VMM node (using libvirt), the VM is created without a problem. However, when I attempt to create a new VM using Nimbus (which then tries to create the VM on the VMM node), I encounter the following error:

---------------------------------------------------------------------------------------------------------------------------------------------
$ ./nimbus-cloud-client-022/bin/cloud-client.sh --run --name testimage --hours 1

Launching workspace.

Workspace Factory Service:

Creating workspace "vm-035"... done.


       IP address: 172.29.99.20
         Hostname: pub02
       Start time: Fri Jan 31 13:48:57 MST 2014
    Shutdown time: Fri Jan 31 14:48:57 MST 2014
 Termination time: Fri Jan 31 14:50:57 MST 2014

Waiting for updates.

Problem with vm-035: Unexpected issue
STDERR: 2014-01-31 13:48:44,412 - WARNING - no qemu_img configuration, copy-on-write support is disabled
libvir: QEMU error : Domain not found: no domain with matching name 'wrksp-33'
libvir: error : An error occurred, but the cause is unknown
2014-01-31 13:48:45,587 - ERROR - Problem creating the VM: An error occurred, but the cause is unknown
2014-01-31 13:48:45,587 - ERROR - An error occurred, but the cause is unknown
Traceback (most recent call last):
  File "/opt/nimbus/src/python/workspacecontrol/defaults/lvrt/lvrt_common.py", line 123, in create
    newvm = self._vmm().createXML(xml, 0)
  File "/usr/lib/python2.7/dist-packages/libvirt.py", line 2189, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: An error occurred, but the cause is unknown
2014-01-31 13:48:45,589 - ERROR - Problem creating the VM: An error occurred, but the cause is unknown
Traceback (most recent call last):
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 155, in _common_withnetsecurity
    _common_withtmplease(nic_set, kernel, local_file_set, c, netsecurity, platform, justprint)
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 174, in _common_withtmplease
    platform.create(local_file_set, nic_set, kernel)
  File "/opt/nimbus/src/python/workspacecontrol/defaults/lvrt/lvrt_common.py", line 128, in create
    raise UnexpectedError(shorterr)
UnexpectedError: Problem creating the VM: An error occurred, but the cause is unknown
2014-01-31 13:48:45,589 - ERROR - Creation problem: going to back out tmp space lease
2014-01-31 13:48:45,589 - ERROR - Backed out tmp space lease
2014-01-31 13:48:45,589 - ERROR - Problem creating the VM: An error occurred, but the cause is unknown
Traceback (most recent call last):
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 137, in _common_withnetbootstrap
    _common_withnetsecurity(vm_name, nic_set, kernel, local_file_set, p, c, netsecurity, platform, justprint)
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 164, in _common_withnetsecurity
    raise e
UnexpectedError: Problem creating the VM: An error occurred, but the cause is unknown
2014-01-31 13:48:45,589 - ERROR - Creation problem: going to back out net security
2014-01-31 13:48:45,606 - ERROR - Backed out net security
2014-01-31 13:48:45,606 - ERROR - Problem creating the VM: An error occurred, but the cause is unknown
Traceback (most recent call last):
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 119, in _common_withnics
    _common_withnetbootstrap(vm_name, nic_set, kernel, local_file_set, p, c, netsecurity, platform, justprint)
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 146, in _common_withnetbootstrap
    raise e
UnexpectedError: Problem creating the VM: An error occurred, but the cause is unknown
2014-01-31 13:48:45,607 - ERROR - Creation problem: going to back out net bootstrap
2014-01-31 13:48:45,607 - ERROR - Backed out net bootstrap
2014-01-31 13:48:45,607 - ERROR - Problem creating the VM: An error occurred, but the cause is unknown
Traceback (most recent call last):
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 96, in _common
    _common_withnics(vm_name, nic_set, kernel, local_file_set, p, c, localnet, netbootstrap, netsecurity, platform, justprint)
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 128, in _common_withnics
    raise e
UnexpectedError: Problem creating the VM: An error occurred, but the cause is unknown
2014-01-31 13:48:45,607 - ERROR - Creation problem: going to back out net leases
2014-01-31 13:48:45,607 - ERROR - Backed out net leases
2014-01-31 13:48:45,608 - ERROR - Problem creating the VM: An error occurred, but the cause is unknown
Traceback (most recent call last):
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core.py", line 79, in core
    _core(vm_name, action, p, c)
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core.py", line 168, in _core
    wc_core_creation.create(vm_name, p, c, async, editing, images, kernels, localnet, netbootstrap, netlease, netsecurity, persistence, platform)
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 16, in create
    _create_local(vm_name, p, c, async, editing, images, kernels, localnet, netbootstrap, netlease, netsecurity, persistence, platform)
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 31, in _create_local
    _common(local_file_set, vm_name, p, c, editing, kernels, localnet, netbootstrap, netlease, netsecurity, persistence, platform, justprint=justprint)
  File "/opt/nimbus/src/python/workspacecontrol/main/wc_core_creation.py", line 107, in _common
    raise e
UnexpectedError: Problem creating the VM: An error occurred, but the cause is unknown
Problem executing: Problem creating the VM: An error occurred, but the cause is unknown

Exiting with error code: 4

Workspace "vm-035" did NOT reach target state "Running"

Problem running 'vm-035'.
---------------------------------------------------------------------------------------------------------------------------------------------

I checked /var/log/libvirtd.log, and found the following:

---------------------------------------------------------------------------------------------------------------------------------------------
2014-01-31 20:48:39.311+0000: 2888: info : libvirt version: 0.9.8
2014-01-31 20:48:39.311+0000: 2888: error : virNetSocketReadWire:1006 : End of file while reading data: Input/output error
---------------------------------------------------------------------------------------------------------------------------------------------

I don't know if this is relevant, but I also noticed this warning when creating the VM through the test scripts mentioned above:

---------------------------------------------------------------------------------------------------------------------------------------------
2014-01-31 05:41:44.018+0000: 1351: warning : qemuDomainObjTaint:1134 : Domain id=1 name='control-test' uuid=95f2d398-6b98-5359-7597-b0f596aae996 is tainted: high-privileges
---------------------------------------------------------------------------------------------------------------------------------------------

I assumed this was a permissions issue, but I double checked and everything seems in order. I did not have this issue on my previous Nimbus cloud setup.

Thanks,
Josh