[libvirt] [PATCH] Revert "qemu: Fix integer/boolean logic in qemuSetUnprivSGIO"
by John Ferlan
This reverts commit 69b850fe2a19d0c32ae2f209e8d8463df6ead665.
This change broke the ability to "clear" or reset unfiltered back
to filtered.
Signed-off-by: John Ferlan <jferlan(a)redhat.com>
---
This is a <facepalm> event - have to wonder what I was thinking
at the time. The patch pushed removes the ability to actually
clear the value in the file. Although it's possible to check what's
in the file first, it's still important to "attempt" the check if
someone set sgio on a disk and unpriv_sgio isn't supported. So, just
reverting this restores the former functionality of setting to
either 0 or 1.
src/qemu/qemu_conf.c | 10 +++-------
1 file changed, 3 insertions(+), 7 deletions(-)
diff --git a/src/qemu/qemu_conf.c b/src/qemu/qemu_conf.c
index 1ce459f..658a50e 100644
--- a/src/qemu/qemu_conf.c
+++ b/src/qemu/qemu_conf.c
@@ -1433,7 +1433,7 @@ qemuSetUnprivSGIO(virDomainDeviceDefPtr dev)
virDomainHostdevDefPtr hostdev = NULL;
char *sysfs_path = NULL;
const char *path = NULL;
- bool val;
+ int val = -1;
int ret = -1;
/* "sgio" is only valid for block disk; cdrom
@@ -1475,12 +1475,8 @@ qemuSetUnprivSGIO(virDomainDeviceDefPtr dev)
* whitelist is enabled. But if requesting unfiltered access, always call
* virSetDeviceUnprivSGIO, to report an error for unsupported unpriv_sgio.
*/
- if (!val || !virFileExists(sysfs_path)) {
- ret = 0;
- goto cleanup;
- }
-
- if (virSetDeviceUnprivSGIO(path, NULL, 1) < 0)
+ if ((virFileExists(sysfs_path) || val == 1) &&
+ virSetDeviceUnprivSGIO(path, NULL, val) < 0)
goto cleanup;
ret = 0;
--
2.1.0
9 years, 3 months
- 2
- 1
[libvirt] ANNOUNCE: libvirt 1.2.18.1 maintenance release
by Cole Robinson
libvirt 1.2.18.1 is now available. This is a maintenance release of
libvirt 1.2.18 with additional bugfixes that have accumulated
upstream since the initial release.
This release can be downloaded at:
http://libvirt.org/sources/stable_updates/libvirt-1.2.18.1.tar.gz
Changes in this version:
* test driver: don't unlock pool after freeing it
* libxl: fix AttachDeviceConfig on hostdev type
* security_selinux: Take @privileged into account
* selinux: fix compile errors
* security_selinux: Add SetDirLabel support
* security: Add virSecurityDomainSetDirLabel
* security_selinux: Use proper structure to access socket data
* security_selinux: Replace SELinuxSCSICallbackData with proper struct
* virSecuritySELinuxSetSecurityAllLabel: drop useless
virFileIsSharedFSType
* virSecurityManager: Track if running as privileged
* qemu: hotplug: Properly clean up drive backend if frontend hotplug
fails
* xen: fix race in refresh of config cache
* libxl: don't end job for ephemeal domain on start failure
* conf: fix crash when parsing a unordered NUMA <cell/>
* qemu: Check virGetLastError return value for migration finish failure
* libxl: don't overwrite error from virNetSocketNewConnectTCP()
* domain-conf: escape string for socket attribute
* src: Check for symbols ordering in ADMIN_SYM_FILES
* src: Cleanup libvirt_admin.syms
* src: Check libvirt_admin.syms for exported symbols
* util: fallback to ioctl(SIOCBRDELBR) if netlink RTM_DELLINK fails
* util: fallback to ioctl(SIOCBRADDBR) if netlink RTM_NEWLINK fails
* libxl: acquire a job when receiving a migrating domain
* libxl: don't attempt to resume domain when suspend fails
* libxl: fix ref counting of libxlMigrationDstArgs
* libvirt_lxc: Claim success for --help
* virt-aa-helper: Improve valid_path
* qemu: Emit correct audit message for memory hot unplug
* qemu: Emit correct audit message for memory hot plug
* hostdev: skip ACS check when using VFIO for device assignment
* Start daemon only after filesystems are mounted
* virt-aa-helper: add NVRAM store file for read/write
* qemu: Update blkio.weight value after successful set
* Eliminate incorrect and unnecessary check for changed IP address
* virt-aa-helper: allow access to /usr/share/ovmf/
* virt-aa-helper: Simplify restriction logic
* virt-aa-helper: document --probing and --dry-run
* Add generated libvirt_admin.syms into .gitignore
* libvirt-admin: Generate symbols file
* daemon: Use $(NULL) for libvird_admin's flags
* qemu: Add check for invalid iothread_id in qemuDomainChgIOThread
* virsh: Reset global error after successfull domain lookup
* build: fix mingw build
* Detect location of qemu-bridge-helper
* Check if qemu-bridge-helper exists and is executable
* qemu: Use numad information when getting pin information
* qemu: Keep numad hint after daemon restart
* conf: Pass private data to Parse function of XML options
* qemu: Fix segfault when parsing private domain data
* domain: Fix crash if trying to live update disk <serial>
* virNetSocketCheckProtocols: handle EAI_NONAME as IPv6 unavailable
* util: don't overwrite stack when getting ethtool gfeatures
* conf: Don't try formating non-existing addresses
* admin: Drop 'internal.h' include from libvirt-admin.h
* qemu: fail on attempts to use <filterref> for non-tap network
connections
* network: validate network NAT range
* virNetDevBandwidthParseRate: Reject negative values
* network: verify proper address family in updates to <host> and <range>
* conf: more useful error message when pci function is out of range
* virDomainDefParseXML: Check for malicious cpu ids in <numa/>
* numa_conf: Introduce virDomainNumaGetMaxCPUID
* Allow vfio hotplug of a device to the domain which owns the iommu
* qemu: Forbid image pre-creation for non-shared storage migration
* virsh: fix domfsinfo output in quiet mode
* tests: extend workaround for gnutls private key loading failure
* qemu: fix some api cannot work when disable cpuset in conf
* storage: only run safezero if allocation is > 0
* qemu: command: Report stderr from qemu-bridge-helper
* qemu: Fix reporting of physical capacity for block devices
* remoteClientCloseFunc: Don't mangle connection object refcount
* storage: Correct the 'mode' check
* storage: Handle failure from refreshVol
* virfile: Introduce virFileUnlink
* Revert "LXC: show used memory as 0 when domain is not active"
For info about past maintenance releases, see:
http://wiki.libvirt.org/page/Maintenance_Releases
Thanks,
Cole
9 years, 3 months
- 1
- 0
[libvirt] [PATCH 00/13] Memory alignment vs. migration fixes
by Peter Krempa
The refactorings that I've done in preparation for memory hotplug had the right
idea in regards to handling of the various memory size fields in libvirt, but
the execution of the refactors was suboptimal for migration compatibilty.
This patchset fixes problems when migrating from older libvirt versions that
allowed specifying a config where contents <memory> were not equal to the total
size of all NUMA nodes. Additionally a few
Patches 01-05/13 are small cleanups to various parts of the code.
Patch 06/13 adds a XML parser flag that will be used later on.
Patches 07-08 do a few more cleanups in the memory parsing code.
Patch 12/13 in this series isn't entirely relevant to this series, but depends
on 09-11/13 so I've included it here. The patch modifies the alignment for PPC
machines that started requiring alignment to 256MiB in some cases. As we now
have a way to keep migrations intact we can switch the alignment always.
Patch 13/13 adds a test case based on an existing test to check that the sizing
code did not regress (again ...).
Peter Krempa (13):
libxl: vz: Use accessor instead of direct access for max_balloon
conf: Add helper to determine whether memory hotplug is enabled for a
vm
qemu: Make memory alignment helper more universal
conf: Drop VIR_DOMAIN_DEF_PARSE_CLOCK_ADJUST flag
conf: Document all VIR_DOMAIN_DEF_PARSE_* flags
conf: Add XML parser flag that will allow us to do incompatible
updates
conf: Split memory related post parse stuff into separate function
conf: Rename max_balloon to total_memory
conf: Pre-calculate initial memory size instead of always calculating
it
conf: Don't always recalculate initial memory size from NUMA size
totals
qemu: command: Align memory sizes only on fresh starts
qemu: ppc64: Align memory sizes to 256MiB blocks
test: Add test to validate that memory sizes don't get updated on
migration
src/conf/domain_conf.c | 113 +++++++++++++++------
src/conf/domain_conf.h | 29 ++++--
src/hyperv/hyperv_driver.c | 2 +-
src/libvirt_private.syms | 2 +
src/libxl/libxl_driver.c | 4 +-
src/lxc/lxc_driver.c | 2 +-
src/lxc/lxc_native.c | 4 +-
src/phyp/phyp_driver.c | 2 +-
src/qemu/qemu_command.c | 12 ++-
src/qemu/qemu_domain.c | 44 +++++---
src/qemu/qemu_domain.h | 3 +-
src/qemu/qemu_driver.c | 14 ++-
src/qemu/qemu_hotplug.c | 4 +-
src/qemu/qemu_migration.c | 8 +-
src/test/test_driver.c | 2 +-
src/uml/uml_driver.c | 2 +-
src/vbox/vbox_common.c | 4 +-
src/vmx/vmx.c | 2 +-
src/vz/vz_driver.c | 2 +-
src/vz/vz_sdk.c | 2 +-
src/xen/xm_internal.c | 2 +-
src/xenapi/xenapi_driver.c | 2 +-
src/xenconfig/xen_common.c | 2 +-
src/xenconfig/xen_sxpr.c | 2 +-
.../qemuxml2argv-migrate-numa-unaligned.args | 13 +++
.../qemuxml2argv-migrate-numa-unaligned.xml | 33 ++++++
.../qemuxml2argv-pseries-cpu-compat.args | 2 +-
.../qemuxml2argvdata/qemuxml2argv-restore-v1.args | 2 +-
tests/qemuxml2argvtest.c | 13 ++-
tests/qemuxml2xmltest.c | 3 +-
30 files changed, 238 insertions(+), 93 deletions(-)
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-migrate-numa-unaligned.args
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-migrate-numa-unaligned.xml
--
2.4.5
9 years, 3 months
- 2
- 29
[libvirt] [PATCH 0/3] Rework our enum parsing
by Michal Privoznik
Only the first two patches are intended for review. The 3rd one is just an
example to show how the code base will shrink in size.
Michal Privoznik (3):
virutil: Rename virEnum{From,To}String to virType{From,To}String
virutil: Introduce virEnumFromString
conf: Example for the new pattern
src/conf/domain_conf.c | 34 ++++++++++---------------------
src/libvirt_private.syms | 3 ++-
src/util/virutil.c | 53 ++++++++++++++++++++++++++++++++++++++----------
src/util/virutil.h | 40 +++++++++++++++++++++++++++++-------
4 files changed, 88 insertions(+), 42 deletions(-)
--
2.4.6
9 years, 3 months
- 3
- 5
[libvirt] [PATCH v2 sandbox] virt-sandbox-image: switch to use URI to identify templates
by Daniel P. Berrange
Currently the CLI syntax is somewhat docker specific requiring
inclusion of --registry arg to identify the docker download
server. Other app containers have a notion of download server,
but don't separate it from the template name.
This patch removes that docker-ism by changing to use a URI
for identifying the template image. So instead of
virt-sandbox-image download \
--source docker --registry index.docker.io
--username dan --password 123456 ubuntu:15.04
You can use
virt-sandbox-image download docker://dan:123456@index.docker.io/ubuntu?tag=15.04
The only mandatory part is the source prefix and image name, so
that can shorten to just
virt-sandbox-image download docker:///ubuntu
to pull down the latest ubuntu image, from the default registry
using no authentication.
---
Changed in v2:
- Rebase against master, instead of (unpushed) docker volume patch
libvirt-sandbox/image/cli.py | 71 +++++--------
libvirt-sandbox/image/sources/DockerSource.py | 142 ++++++++++++++------------
libvirt-sandbox/image/sources/Source.py | 29 +++---
libvirt-sandbox/image/template.py | 110 ++++++++++++++++++++
4 files changed, 228 insertions(+), 124 deletions(-)
create mode 100644 libvirt-sandbox/image/template.py
diff --git a/libvirt-sandbox/image/cli.py b/libvirt-sandbox/image/cli.py
index 1718cc5..4d02289 100755
--- a/libvirt-sandbox/image/cli.py
+++ b/libvirt-sandbox/image/cli.py
@@ -3,7 +3,7 @@
# Authors: Daniel P. Berrange <berrange(a)redhat.com>
# Eren Yagdiran <erenyagdiran(a)gmail.com>
#
-# Copyright (C) 2013 Red Hat, Inc.
+# Copyright (C) 2013-2015 Red Hat, Inc.
# Copyright (C) 2015 Universitat Politècnica de Catalunya.
#
# This program is free software; you can redistribute it and/or modify
@@ -34,6 +34,8 @@ import subprocess
import random
import string
+from libvirt_sandbox.image import template
+
if os.geteuid() == 0:
default_template_dir = "/var/lib/libvirt/templates"
default_image_dir = "/var/lib/libvirt/images"
@@ -44,15 +46,6 @@ else:
debug = False
verbose = False
-import importlib
-def dynamic_source_loader(name):
- name = name[0].upper() + name[1:]
- modname = "libvirt_sandbox.image.sources." + name + "Source"
- mod = importlib.import_module(modname)
- classname = name + "Source"
- classimpl = getattr(mod, classname)
- return classimpl()
-
gettext.bindtextdomain("libvirt-sandbox", "/usr/share/locale")
gettext.textdomain("libvirt-sandbox")
try:
@@ -73,11 +66,10 @@ def info(msg):
def download(args):
try:
- dynamic_source_loader(args.source).download_template(templatename=args.template,
- templatedir=args.template_dir,
- registry=args.registry,
- username=args.username,
- password=args.password)
+ tmpl = template.Template.from_uri(args.template)
+ source = tmpl.get_source_impl()
+ source.download_template(template=tmpl,
+ templatedir=args.template_dir)
except IOError,e:
print "Source %s cannot be found in given path" %args.source
except Exception,e:
@@ -85,17 +77,21 @@ def download(args):
def delete(args):
try:
- dynamic_source_loader(args.source).delete_template(templatename=args.template,
- templatedir=args.template_dir)
+ tmpl = template.Template.from_uri(args.template)
+ source = tmpl.get_source_impl()
+ source.delete_template(template=tmpl,
+ templatedir=args.template_dir)
except Exception,e:
print "Delete Error %s", str(e)
def create(args):
try:
- dynamic_source_loader(args.source).create_template(templatename=args.template,
- templatedir=args.template_dir,
- connect=args.connect,
- format=args.format)
+ tmpl = template.Template.from_uri(args.template)
+ source = tmpl.get_source_impl()
+ source.create_template(template=tmpl,
+ templatedir=args.template_dir,
+ connect=args.connect,
+ format=args.format)
except Exception,e:
print "Create Error %s" % str(e)
@@ -103,19 +99,22 @@ def run(args):
try:
if args.connect is not None:
check_connect(args.connect)
- source = dynamic_source_loader(args.source)
+
+ tmpl = template.Template.from_uri(args.template)
+ source = tmpl.get_source_impl()
+
name = args.name
if name is None:
randomid = ''.join(random.choice(string.lowercase) for i in range(10))
- name = args.template + ":" + randomid
+ name = tmpl.path[1:] + ":" + randomid
- diskfile = source.get_disk(templatename=args.template,
+ diskfile = source.get_disk(template=tmpl,
templatedir=args.template_dir,
imagedir=args.image_dir,
sandboxname=name)
format = "qcow2"
- commandToRun = source.get_command(args.template, args.template_dir, args.args)
+ commandToRun = source.get_command(tmpl, args.template_dir, args.args)
if len(commandToRun) == 0:
commandToRun = ["/bin/sh"]
cmd = ['virt-sandbox', '--name', name]
@@ -129,7 +128,7 @@ def run(args):
params.append('-N')
params.append(networkArgs)
- allEnvs = source.get_env(args.template, args.template_dir)
+ allEnvs = source.get_env(tmpl, args.template_dir)
envArgs = args.env
if envArgs is not None:
allEnvs = allEnvs + envArgs
@@ -151,7 +150,7 @@ def run(args):
def requires_template(parser):
parser.add_argument("template",
- help=_("name of the template"))
+ help=_("URI of the template"))
def requires_name(parser):
parser.add_argument("-n","--name",
@@ -163,23 +162,10 @@ def check_connect(connectstr):
raise ValueError("URI '%s' is not supported by virt-sandbox-image" % connectstr)
return True
-def requires_source(parser):
- parser.add_argument("-s","--source",
- default="docker",
- help=_("name of the template"))
-
def requires_connect(parser):
parser.add_argument("-c","--connect",
help=_("Connect string for libvirt"))
-def requires_auth_conn(parser):
- parser.add_argument("-r","--registry",
- help=_("Url of the custom registry"))
- parser.add_argument("-u","--username",
- help=_("Username for the custom registry"))
- parser.add_argument("-p","--password",
- help=_("Password for the custom registry"))
-
def requires_template_dir(parser):
global default_template_dir
parser.add_argument("-t","--template-dir",
@@ -196,8 +182,6 @@ def gen_download_args(subparser):
parser = subparser.add_parser("download",
help=_("Download template data"))
requires_template(parser)
- requires_source(parser)
- requires_auth_conn(parser)
requires_template_dir(parser)
parser.set_defaults(func=download)
@@ -205,7 +189,6 @@ def gen_delete_args(subparser):
parser = subparser.add_parser("delete",
help=_("Delete template data"))
requires_template(parser)
- requires_source(parser)
requires_template_dir(parser)
parser.set_defaults(func=delete)
@@ -213,7 +196,6 @@ def gen_create_args(subparser):
parser = subparser.add_parser("create",
help=_("Create image from template data"))
requires_template(parser)
- requires_source(parser)
requires_connect(parser)
requires_template_dir(parser)
parser.add_argument("-f","--format",
@@ -226,7 +208,6 @@ def gen_run_args(subparser):
help=_("Run an already built image"))
requires_name(parser)
requires_template(parser)
- requires_source(parser)
requires_connect(parser)
requires_template_dir(parser)
requires_image_dir(parser)
diff --git a/libvirt-sandbox/image/sources/DockerSource.py b/libvirt-sandbox/image/sources/DockerSource.py
index c374a0c..10f8537 100644
--- a/libvirt-sandbox/image/sources/DockerSource.py
+++ b/libvirt-sandbox/image/sources/DockerSource.py
@@ -2,6 +2,7 @@
# -*- coding: utf-8 -*-
#
# Copyright (C) 2015 Universitat Politècnica de Catalunya.
+# Copyright (C) 2015 Red Hat, Inc
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -28,6 +29,8 @@ import traceback
import os
import subprocess
import shutil
+import urlparse
+
class DockerConfParser():
@@ -47,12 +50,6 @@ class DockerConfParser():
class DockerSource(Source):
- www_auth_username = None
- www_auth_password = None
-
- def __init__(self):
- self.default_index_server = "index.docker.io"
-
def _check_cert_validate(self):
major = sys.version_info.major
SSL_WARNING = "SSL certificates couldn't be validated by default. You need to have 2.7.9/3.4.3 or higher"
@@ -62,43 +59,38 @@ class DockerSource(Source):
if (major == 2 and sys.hexversion < py2_7_9_hexversion) or (major == 3 and sys.hexversion < py3_4_3_hexversion):
sys.stderr.write(SSL_WARNING)
- def download_template(self, templatename, templatedir,
- registry=None, username=None, password=None):
- if registry is None:
- registry = self.default_index_server
-
- if username is not None:
- self.www_auth_username = username
- self.www_auth_password = password
-
+ def download_template(self, template, templatedir):
self._check_cert_validate()
- tag = "latest"
- offset = templatename.find(':')
- if offset != -1:
- tag = templatename[offset + 1:]
- templatename = templatename[0:offset]
+
try:
- (data, res) = self._get_json(registry, "/v1/repositories/" + templatename + "/images",
- {"X-Docker-Token": "true"})
+ (data, res) = self._get_json(template,
+ None,
+ "/v1/repositories" + template.path + "/images",
+ {"X-Docker-Token": "true"})
except urllib2.HTTPError, e:
- raise ValueError(["Image '%s' does not exist" % templatename])
+ raise ValueError(["Image '%s' does not exist" % template])
registryendpoint = res.info().getheader('X-Docker-Endpoints')
token = res.info().getheader('X-Docker-Token')
checksums = {}
for layer in data:
pass
- (data, res) = self._get_json(registryendpoint, "/v1/repositories/" + templatename + "/tags",
- { "Authorization": "Token " + token })
+ (data, res) = self._get_json(template,
+ registryendpoint,
+ "/v1/repositories" + template.path + "/tags",
+ { "Authorization": "Token " + token })
cookie = res.info().getheader('Set-Cookie')
+ tag = template.params.get("tag", "latest")
if not tag in data:
- raise ValueError(["Tag '%s' does not exist for image '%s'" % (tag, templatename)])
+ raise ValueError(["Tag '%s' does not exist for image '%s'" % (tag, template)])
imagetagid = data[tag]
- (data, res) = self._get_json(registryendpoint, "/v1/images/" + imagetagid + "/ancestry",
- { "Authorization": "Token "+token })
+ (data, res) = self._get_json(template,
+ registryendpoint,
+ "/v1/images/" + imagetagid + "/ancestry",
+ { "Authorization": "Token "+ token })
if data[0] != imagetagid:
raise ValueError(["Expected first layer id '%s' to match image id '%s'",
@@ -118,8 +110,11 @@ class DockerSource(Source):
datafile = layerdir + "/template.tar.gz"
if not os.path.exists(jsonfile) or not os.path.exists(datafile):
- res = self._save_data(registryendpoint, "/v1/images/" + layerid + "/json",
- { "Authorization": "Token " + token }, jsonfile)
+ res = self._save_data(template,
+ registryendpoint,
+ "/v1/images/" + layerid + "/json",
+ { "Authorization": "Token " + token },
+ jsonfile)
createdFiles.append(jsonfile)
layersize = int(res.info().getheader("Content-Length"))
@@ -128,12 +123,15 @@ class DockerSource(Source):
if layerid in checksums:
datacsum = checksums[layerid]
- self._save_data(registryendpoint, "/v1/images/" + layerid + "/layer",
- { "Authorization": "Token "+token }, datafile, datacsum, layersize)
+ self._save_data(template,
+ registryendpoint,
+ "/v1/images/" + layerid + "/layer",
+ { "Authorization": "Token "+token },
+ datafile, datacsum, layersize)
createdFiles.append(datafile)
index = {
- "name": templatename,
+ "name": template.path,
}
indexfile = templatedir + "/" + imagetagid + "/index.json"
@@ -152,9 +150,11 @@ class DockerSource(Source):
shutil.rmtree(d)
except:
pass
- def _save_data(self,server, path, headers, dest, checksum=None, datalen=None):
+
+ def _save_data(self, template, server, path, headers,
+ dest, checksum=None, datalen=None):
try:
- res = self._get_url(server, path, headers)
+ res = self._get_url(template, server, path, headers)
csum = None
if checksum is not None:
@@ -193,8 +193,22 @@ class DockerSource(Source):
debug("FAIL %s\n" % str(e))
raise
- def _get_url(self,server, path, headers):
- url = "https://" + server + path
+ def _get_url(self, template, server, path, headers):
+ if template.protocol is None:
+ protocol = "https"
+ else:
+ protocol = template.protocol
+
+ if server is None:
+ if template.hostname is None:
+ server = "index.docker.io"
+ else:
+ if template.port is not None:
+ server = template.hostname + ":" + template.port
+ else:
+ server = template.hostname
+
+ url = urlparse.urlunparse((protocol, server, path, None, None, None))
debug("Fetching %s..." % url)
req = urllib2.Request(url=url)
@@ -204,16 +218,18 @@ class DockerSource(Source):
req.add_header(h, headers[h])
#www Auth header starts
- if self.www_auth_username is not None:
- base64string = base64.encodestring('%s:%s' % (self.www_auth_username, self.www_auth_password)).replace('\n', '')
+ if template.username and template.password:
+ base64string = base64.encodestring(
+ '%s:%s' % (template.username,
+ template.password)).replace('\n', '')
req.add_header("Authorization", "Basic %s" % base64string)
#www Auth header finish
return urllib2.urlopen(req)
- def _get_json(self,server, path, headers):
+ def _get_json(self, template, server, path, headers):
try:
- res = self._get_url(server, path, headers)
+ res = self._get_url(template, server, path, headers)
data = json.loads(res.read())
debug("OK\n")
return (data, res)
@@ -221,11 +237,11 @@ class DockerSource(Source):
debug("FAIL %s\n" % str(e))
raise
- def create_template(self, templatename, templatedir, connect=None, format=None):
+ def create_template(self, template, templatedir, connect=None, format=None):
if format is None:
format = self.default_disk_format
self._check_disk_format(format)
- imagelist = self._get_image_list(templatename,templatedir)
+ imagelist = self._get_image_list(template, templatedir)
imagelist.reverse()
parentImage = None
@@ -252,7 +268,7 @@ class DockerSource(Source):
if not format in supportedFormats:
raise ValueError(["Unsupported image format %s" % format])
- def _get_image_list(self,templatename,destdir):
+ def _get_image_list(self, template, destdir):
imageparent = {}
imagenames = {}
imagedirs = os.listdir(destdir)
@@ -265,13 +281,13 @@ class DockerSource(Source):
jsonfile = destdir + "/" + imagetagid + "/template.json"
if os.path.exists(jsonfile):
with open(jsonfile,"r") as f:
- template = json.load(f)
- parent = template.get("parent",None)
+ data = json.load(f)
+ parent = data.get("parent",None)
if parent:
imageparent[imagetagid] = parent
- if not templatename in imagenames:
- raise ValueError(["Image %s does not exist locally" %templatename])
- imagetagid = imagenames[templatename]
+ if not template.path in imagenames:
+ raise ValueError(["Image %s does not exist locally" % template.path])
+ imagetagid = imagenames[template.path]
imagelist = []
while imagetagid != None:
imagelist.append(imagetagid)
@@ -310,7 +326,7 @@ class DockerSource(Source):
cmd = cmd + params
subprocess.call(cmd)
- def delete_template(self, templatename, templatedir):
+ def delete_template(self, template, templatedir):
imageusage = {}
imageparent = {}
imagenames = {}
@@ -324,9 +340,9 @@ class DockerSource(Source):
jsonfile = templatedir + "/" + imagetagid + "/template.json"
if os.path.exists(jsonfile):
with open(jsonfile,"r") as f:
- template = json.load(f)
+ data = json.load(f)
- parent = template.get("parent",None)
+ parent = data.get("parent",None)
if parent:
if parent not in imageusage:
imageusage[parent] = []
@@ -334,10 +350,10 @@ class DockerSource(Source):
imageparent[imagetagid] = parent
- if not templatename in imagenames:
- raise ValueError(["Image %s does not exist locally" %templatename])
+ if not template.path in imagenames:
+ raise ValueError(["Image %s does not exist locally" % template.path])
- imagetagid = imagenames[templatename]
+ imagetagid = imagenames[template.path]
while imagetagid != None:
debug("Remove %s\n" % imagetagid)
parent = imageparent.get(imagetagid,None)
@@ -360,15 +376,15 @@ class DockerSource(Source):
parent = None
imagetagid = parent
- def _get_template_data(self, templatename, templatedir):
- imageList = self._get_image_list(templatename, templatedir)
+ def _get_template_data(self, template, templatedir):
+ imageList = self._get_image_list(template, templatedir)
toplayer = imageList[0]
diskfile = templatedir + "/" + toplayer + "/template.qcow2"
configfile = templatedir + "/" + toplayer + "/template.json"
return configfile, diskfile
- def get_disk(self,templatename, templatedir, imagedir, sandboxname):
- configfile, diskfile = self._get_template_data(templatename, templatedir)
+ def get_disk(self, template, templatedir, imagedir, sandboxname):
+ configfile, diskfile = self._get_template_data(template, templatedir)
tempfile = imagedir + "/" + sandboxname + ".qcow2"
if not os.path.exists(imagedir):
os.makedirs(imagedir)
@@ -379,8 +395,8 @@ class DockerSource(Source):
subprocess.call(cmd)
return tempfile
- def get_command(self, templatename, templatedir, userargs):
- configfile, diskfile = self._get_template_data(templatename, templatedir)
+ def get_command(self, template, templatedir, userargs):
+ configfile, diskfile = self._get_template_data(template, templatedir)
configParser = DockerConfParser(configfile)
cmd = configParser.getCommand()
entrypoint = configParser.getEntrypoint()
@@ -393,8 +409,8 @@ class DockerSource(Source):
else:
return entrypoint + cmd
- def get_env(self, templatename, templatedir):
- configfile, diskfile = self._get_template_data(templatename, templatedir)
+ def get_env(self, template, templatedir):
+ configfile, diskfile = self._get_template_data(template, templatedir)
configParser = DockerConfParser(configfile)
return configParser.getEnvs()
diff --git a/libvirt-sandbox/image/sources/Source.py b/libvirt-sandbox/image/sources/Source.py
index 8a21f90..597a7fb 100644
--- a/libvirt-sandbox/image/sources/Source.py
+++ b/libvirt-sandbox/image/sources/Source.py
@@ -2,6 +2,7 @@
# -*- coding: utf-8 -*-
#
# Copyright (C) 2015 Universitat Politècnica de Catalunya.
+# Copyright (C) 2015 Red Hat, Inc
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -33,14 +34,10 @@ class Source():
pass
@abstractmethod
- def download_template(self, templatename, templatedir,
- registry=None, username=None, password=None):
+ def download_template(self, template, templatedir):
"""
- :param templatename: name of the template image to download
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to store the template
- :param registry: optional hostname of image registry server
- :param username: optional username to authenticate against registry server
- :param password: optional password to authenticate against registry server
Download a template from the registry, storing it in the local
filesystem
@@ -48,10 +45,10 @@ class Source():
pass
@abstractmethod
- def create_template(self, templatename, templatedir,
+ def create_template(self, template, templatedir,
connect=None, format=None):
"""
- :param templatename: name of the template image to create
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to store the template
:param connect: libvirt connection URI
:param format: disk image format
@@ -63,9 +60,9 @@ class Source():
pass
@abstractmethod
- def delete_template(self, templatename, templatedir):
+ def delete_template(self, template, templatedir):
"""
- :param templatename: name of the template image to delete
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path from which to delete template
Delete all local files associated with the template
@@ -73,9 +70,9 @@ class Source():
pass
@abstractmethod
- def get_command(self, templatename, templatedir, userargs):
+ def get_command(self, template, templatedir, userargs):
"""
- :param templatename: name of the template image to query
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which templates are stored
:param userargs: user specified arguments to run
@@ -85,9 +82,9 @@ class Source():
pass
@abstractmethod
- def get_disk(self,templatename, templatedir, imagedir, sandboxname):
+ def get_disk(self, template, templatedir, imagedir, sandboxname):
"""
- :param templatename: name of the template image to download
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to find template
:param imagedir: local directory in which to storage disk image
@@ -97,9 +94,9 @@ class Source():
pass
@abstractmethod
- def get_env(self,templatename, templatedir):
+ def get_env(self, template, templatedir):
"""
- :param templatename: name of the template image to download
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to find template
Get the dict of environment variables to set
diff --git a/libvirt-sandbox/image/template.py b/libvirt-sandbox/image/template.py
new file mode 100644
index 0000000..0ad767b
--- /dev/null
+++ b/libvirt-sandbox/image/template.py
@@ -0,0 +1,110 @@
+#
+# -*- coding: utf-8 -*-
+# Authors: Daniel P. Berrange <berrange(a)redhat.com>
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+
+import urlparse
+import importlib
+
+class Template(object):
+
+ def __init__(self,
+ source, protocol,
+ hostname, port,
+ username, password,
+ path, params):
+ """
+ :param source: template source name
+ :param protocol: network transport protocol or None
+ :param hostname: registry hostname or None
+ :param port: registry port or None
+ :param username: username or None
+ :param password: password or None
+ :param path: template path identifier
+ :param params: template parameters
+
+ docker:///ubuntu
+
+ docker+https://index.docker.io/ubuntu?tag=latest
+ """
+
+ self.source = source
+ self.protocol = protocol
+ self.hostname = hostname
+ self.port = port
+ self.username = username
+ self.password = password
+ self.path = path
+ self.params = params
+ if self.params is None:
+ self.params = {}
+
+ def get_source_impl(self):
+ mod = importlib.import_module(
+ "libvirt_sandbox.image.sources." +
+ self.source.capitalize() + "Source")
+ classname = self.source.capitalize() + "Source"
+ classimpl = getattr(mod, classname)
+ return classimpl()
+
+ def __repr__(self):
+ if self.protocol is not None:
+ scheme = self.source + "+" + self.protocol
+ else:
+ scheme = self.source
+ if self.hostname:
+ if self.port:
+ netloc = self.hostname + ":" + self.port
+ else:
+ netloc = self.hostname
+
+ if self.username:
+ if self.password:
+ auth = self.username + ":" + self.password
+ else:
+ auth = self.username
+ netloc = auth + "@" + netloc
+ else:
+ netloc = None
+
+ query = "&".join([key + "=" + self.params[key] for key in self.params.keys()])
+ return urlparse.urlunparse((scheme, netloc, self.path, None, query, None))
+
+ @classmethod
+ def from_uri(klass, uri):
+ o = urlparse.urlparse(uri)
+
+ idx = o.scheme.find("+")
+ if idx == -1:
+ source = o.scheme
+ protocol = None
+ else:
+ source = o.scheme[0:idx]
+ protocol = o.schema[idx + 1:]
+
+ query = {}
+ if o.query is not None and o.query != "":
+ for param in o.query.split("&"):
+ (key, val) = param.split("=")
+ query[key] = val
+ return klass(source, protocol,
+ o.hostname, o.port,
+ o.username, o.password,
+ o.path, query)
+
--
2.4.3
9 years, 3 months
- 2
- 8
[libvirt] [PATCH sandbox] docker: don't assume X-Docker-Token is set
by Daniel P. Berrange
The Red Hat docker registry (registry.access.redhat.com) does
not set any X-Docker-Token HTTP header in its responses. Change
the code so it only passes around this header if it is actually
present.
Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com>
---
libvirt-sandbox/image/sources/DockerSource.py | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/libvirt-sandbox/image/sources/DockerSource.py b/libvirt-sandbox/image/sources/DockerSource.py
index f367c8f..78b2a53 100644
--- a/libvirt-sandbox/image/sources/DockerSource.py
+++ b/libvirt-sandbox/image/sources/DockerSource.py
@@ -83,10 +83,14 @@ class DockerSource(Source):
checksums = {}
for layer in data:
pass
+
+ headers = {}
+ if token is not None:
+ headers["Authorization"] = "Token" + token
(data, res) = self._get_json(template,
registryendpoint,
"/v1/repositories" + template.path + "/tags",
- { "Authorization": "Token " + token })
+ headers)
cookie = res.info().getheader('Set-Cookie')
@@ -98,7 +102,7 @@ class DockerSource(Source):
(data, res) = self._get_json(template,
registryendpoint,
"/v1/images/" + imagetagid + "/ancestry",
- { "Authorization": "Token "+ token })
+ headers)
if data[0] != imagetagid:
raise ValueError(["Expected first layer id '%s' to match image id '%s'",
@@ -121,7 +125,7 @@ class DockerSource(Source):
res = self._save_data(template,
registryendpoint,
"/v1/images/" + layerid + "/json",
- { "Authorization": "Token " + token },
+ headers,
jsonfile)
createdFiles.append(jsonfile)
@@ -134,7 +138,7 @@ class DockerSource(Source):
self._save_data(template,
registryendpoint,
"/v1/images/" + layerid + "/layer",
- { "Authorization": "Token "+token },
+ headers,
datafile, datacsum, layersize)
createdFiles.append(datafile)
--
2.4.3
9 years, 3 months
- 2
- 4
[libvirt] [libvirt-python][PATCH] generator: fix build fail with old xml lib
by Luyao Huang
https://bugzilla.redhat.com/show_bug.cgi?id=1222795#c6
if build libvirt-python with some old xml lib (python-pyxml),
build will fail and error like this:
File "generator.py", line 139, in start
if "string" in attrs:
File "/usr/local/lib/python2.7/site-packages/_xmlplus/sax/xmlreader.py" \
, line 316, in __getitem__
return self._attrs[name]
KeyError: 0
This is an old issue and have been mentioned in commit 3ae0a76d.
There is no __contains__ in class AttributesImpl, python will use
__getitem__ in this place, so we will get error.
Let's use 'YYY in XXX.keys()' to avoid this issue.
Signed-off-by: Luyao Huang <lhuang(a)redhat.com>
---
generator.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/generator.py b/generator.py
index 2fc838c..d9ae17e 100755
--- a/generator.py
+++ b/generator.py
@@ -136,7 +136,7 @@ class docParser(xml.sax.handler.ContentHandler):
elif attrs['file'] == "libvirt-qemu":
qemu_enum(attrs['type'],attrs['name'],attrs['value'])
elif tag == "macro":
- if "string" in attrs:
+ if "string" in attrs.keys():
params.append((attrs['name'], attrs['string']))
def end(self, tag):
--
1.8.3.1
9 years, 3 months
- 3
- 2
[libvirt] [PATCH 0/4] Implement mockup capabilities cache in QEMU tests
by Ján Tomko
v2: https://www.redhat.com/archives/libvir-list/2015-September/msg00374.html
v3:
* split out the movement of existing initialization code to qemuTestDriverInit
from the cache addition
* qemuTestDriverInit now adds an empty capability cache, so most tests are unchanged
* renamed 'cleanup' to 'error' in qemuTestDriverInit, since it's only called on error paths
* fixed indentation issues to pass syntax-check
* removed mode changes to 755
Pavel Fedin (4):
tests: split out common qemu driver initialization
Implement infrastracture for mocking up QEMU capabilities cache
Use mockup cache
Removed unneeded check
src/qemu/qemu_capabilities.c | 16 ++++-----
src/qemu/qemu_capspriv.h | 36 ++++++++++++++++++++
src/qemu/qemu_domain.c | 10 ++----
tests/domainsnapshotxml2xmltest.c | 10 ++----
tests/qemuagenttest.c | 11 +++---
tests/qemuargv2xmltest.c | 12 ++-----
tests/qemuhotplugtest.c | 32 +++++++++---------
tests/qemuxml2argvtest.c | 17 +++++-----
tests/qemuxml2xmltest.c | 8 ++---
tests/qemuxmlnstest.c | 17 +++++-----
tests/testutilsqemu.c | 70 +++++++++++++++++++++++++++++++++++++++
tests/testutilsqemu.h | 7 ++++
12 files changed, 170 insertions(+), 76 deletions(-)
create mode 100644 src/qemu/qemu_capspriv.h
--
2.4.6
9 years, 3 months
- 3
- 14
[libvirt] [PATCH] test driver: don't unlock pool after freeing it
by David Mansfield
The attached patch (taken from my modified Fedora 22 source rpm,
1.2.13.1-2.fc22, sorry), fixes a case where, in the test driver, memory
is accessed after it's freed.
Patch applies to latest git with:
Hunk #1 succeeded at 4395 (offset -469 lines).
The illegal access was found using valgrind.
The function testStoragePoolUndefine() calls virStoragePoolObjRemove()
(which seems to free the memory for that pool structure) and then in the
cleanup stanza calls virStoragePoolObjUnlock() which tampers with the
freed structure.
--
Thanks,
David Mansfield
9 years, 3 months
- 2
- 3
[libvirt] [PATCH sandbox] virt-sandbox-image: switch to use URI to identify templates
by Daniel P. Berrange
Currently the CLI syntax is somewhat docker specific requiring
inclusion of --registry arg to identify the docker download
server. Other app containers have a notion of download server,
but don't separate it from the template name.
This patch removes that docker-ism by changing to use a URI
for identifying the template image. So instead of
virt-sandbox-image download \
--source docker --registry index.docker.io
--username dan --password 123456 ubuntu:15.04
You can use
virt-sandbox-image download docker://dan@123456:index.docker.io/ubuntu?tag=15.04
The only mandatory part is the source prefix and image name, so
that can shorten to just
virt-sandbox-image download docker:///ubuntu
to pull down the latest ubuntu image, from the default registry
using no authentication.
---
libvirt-sandbox/image/cli.py | 73 +++++--------
libvirt-sandbox/image/sources/DockerSource.py | 146 ++++++++++++++------------
libvirt-sandbox/image/sources/Source.py | 33 +++---
libvirt-sandbox/image/template.py | 109 +++++++++++++++++++
4 files changed, 232 insertions(+), 129 deletions(-)
create mode 100644 libvirt-sandbox/image/template.py
diff --git a/libvirt-sandbox/image/cli.py b/libvirt-sandbox/image/cli.py
index c8b5ba5..dd109a8 100755
--- a/libvirt-sandbox/image/cli.py
+++ b/libvirt-sandbox/image/cli.py
@@ -3,7 +3,7 @@
# Authors: Daniel P. Berrange <berrange(a)redhat.com>
# Eren Yagdiran <erenyagdiran(a)gmail.com>
#
-# Copyright (C) 2013 Red Hat, Inc.
+# Copyright (C) 2013-2015 Red Hat, Inc.
# Copyright (C) 2015 Universitat Politècnica de Catalunya.
#
# This program is free software; you can redistribute it and/or modify
@@ -34,6 +34,8 @@ import subprocess
import random
import string
+from libvirt_sandbox.image import template
+
if os.geteuid() == 0:
default_template_dir = "/var/lib/libvirt/templates"
default_image_dir = "/var/lib/libvirt/images"
@@ -44,15 +46,6 @@ else:
debug = False
verbose = False
-import importlib
-def dynamic_source_loader(name):
- name = name[0].upper() + name[1:]
- modname = "libvirt_sandbox.image.sources." + name + "Source"
- mod = importlib.import_module(modname)
- classname = name + "Source"
- classimpl = getattr(mod, classname)
- return classimpl()
-
gettext.bindtextdomain("libvirt-sandbox", "/usr/share/locale")
gettext.textdomain("libvirt-sandbox")
try:
@@ -73,11 +66,10 @@ def info(msg):
def download(args):
try:
- dynamic_source_loader(args.source).download_template(templatename=args.template,
- templatedir=args.template_dir,
- registry=args.registry,
- username=args.username,
- password=args.password)
+ tmpl = template.Template.from_uri(args.template)
+ source = tmpl.get_source_impl()
+ source.download_template(template=tmpl,
+ templatedir=args.template_dir)
except IOError,e:
print "Source %s cannot be found in given path" %args.source
except Exception,e:
@@ -85,17 +77,21 @@ def download(args):
def delete(args):
try:
- dynamic_source_loader(args.source).delete_template(templatename=args.template,
- templatedir=args.template_dir)
+ tmpl = template.Template.from_uri(args.template)
+ source = tmpl.get_source_impl()
+ source.delete_template(template=tmpl,
+ templatedir=args.template_dir)
except Exception,e:
print "Delete Error %s", str(e)
def create(args):
try:
- dynamic_source_loader(args.source).create_template(templatename=args.template,
- templatedir=args.template_dir,
- connect=args.connect,
- format=args.format)
+ tmpl = template.Template.from_uri(args.template)
+ source = tmpl.get_source_impl()
+ source.create_template(template=tmpl,
+ templatedir=args.template_dir,
+ connect=args.connect,
+ format=args.format)
except Exception,e:
print "Create Error %s" % str(e)
@@ -104,19 +100,22 @@ def run(args):
global image_dir
if args.connect is not None:
check_connect(args.connect)
- source = dynamic_source_loader(args.source)
+
+ tmpl = template.Template.from_uri(args.template)
+ source = tmpl.get_source_impl()
+
name = args.name
if name is None:
randomid = ''.join(random.choice(string.lowercase) for i in range(10))
- name = args.template + ":" + randomid
+ name = tmpl.path[1:] + ":" + randomid
- diskfile = source.get_disk(templatename=args.template,
+ diskfile = source.get_disk(template=tmpl,
templatedir=args.template_dir,
imagedir=args.image_dir,
sandboxname=name)
format = "qcow2"
- commandToRun = source.get_command(args.template, args.template_dir, args.args)
+ commandToRun = source.get_command(tmpl, args.template_dir, args.args)
if len(commandToRun) == 0:
commandToRun = ["/bin/sh"]
cmd = ['virt-sandbox', '--name', name]
@@ -130,7 +129,7 @@ def run(args):
params.append('-N')
params.append(networkArgs)
- allEnvs = source.get_env(args.template, args.template_dir)
+ allEnvs = source.get_env(tmpl, args.template_dir)
envArgs = args.env
if envArgs is not None:
allEnvs = allEnvs + envArgs
@@ -143,7 +142,7 @@ def run(args):
else:
pass
- allVolumes = source.get_volumes(args.template, args.template_dir)
+ allVolumes = source.get_volumes(tmpl, args.template_dir)
volumeArgs = args.volume
if volumeArgs is not None:
allVolumes = allVolumes + volumeArgs
@@ -172,7 +171,7 @@ def run(args):
def requires_template(parser):
parser.add_argument("template",
- help=_("name of the template"))
+ help=_("URI of the template"))
def requires_name(parser):
parser.add_argument("-n","--name",
@@ -184,23 +183,10 @@ def check_connect(connectstr):
raise ValueError("URI '%s' is not supported by virt-sandbox-image" % connectstr)
return True
-def requires_source(parser):
- parser.add_argument("-s","--source",
- default="docker",
- help=_("name of the template"))
-
def requires_connect(parser):
parser.add_argument("-c","--connect",
help=_("Connect string for libvirt"))
-def requires_auth_conn(parser):
- parser.add_argument("-r","--registry",
- help=_("Url of the custom registry"))
- parser.add_argument("-u","--username",
- help=_("Username for the custom registry"))
- parser.add_argument("-p","--password",
- help=_("Password for the custom registry"))
-
def requires_template_dir(parser):
global default_template_dir
parser.add_argument("-t","--template-dir",
@@ -217,8 +203,6 @@ def gen_download_args(subparser):
parser = subparser.add_parser("download",
help=_("Download template data"))
requires_template(parser)
- requires_source(parser)
- requires_auth_conn(parser)
requires_template_dir(parser)
parser.set_defaults(func=download)
@@ -226,7 +210,6 @@ def gen_delete_args(subparser):
parser = subparser.add_parser("delete",
help=_("Delete template data"))
requires_template(parser)
- requires_source(parser)
requires_template_dir(parser)
parser.set_defaults(func=delete)
@@ -234,7 +217,6 @@ def gen_create_args(subparser):
parser = subparser.add_parser("create",
help=_("Create image from template data"))
requires_template(parser)
- requires_source(parser)
requires_connect(parser)
requires_template_dir(parser)
parser.add_argument("-f","--format",
@@ -247,7 +229,6 @@ def gen_run_args(subparser):
help=_("Run an already built image"))
requires_name(parser)
requires_template(parser)
- requires_source(parser)
requires_connect(parser)
requires_template_dir(parser)
requires_image_dir(parser)
diff --git a/libvirt-sandbox/image/sources/DockerSource.py b/libvirt-sandbox/image/sources/DockerSource.py
index ff950ce..f367c8f 100644
--- a/libvirt-sandbox/image/sources/DockerSource.py
+++ b/libvirt-sandbox/image/sources/DockerSource.py
@@ -2,6 +2,7 @@
# -*- coding: utf-8 -*-
#
# Copyright (C) 2015 Universitat Politècnica de Catalunya.
+# Copyright (C) 2015 Red Hat, Inc
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -29,6 +30,8 @@ import os
import subprocess
import shutil
import collections
+import urlparse
+
class DockerConfParser():
@@ -55,12 +58,6 @@ class DockerConfParser():
class DockerSource(Source):
- www_auth_username = None
- www_auth_password = None
-
- def __init__(self):
- self.default_index_server = "index.docker.io"
-
def _check_cert_validate(self):
major = sys.version_info.major
SSL_WARNING = "SSL certificates couldn't be validated by default. You need to have 2.7.9/3.4.3 or higher"
@@ -70,43 +67,38 @@ class DockerSource(Source):
if (major == 2 and sys.hexversion < py2_7_9_hexversion) or (major == 3 and sys.hexversion < py3_4_3_hexversion):
sys.stderr.write(SSL_WARNING)
- def download_template(self, templatename, templatedir,
- registry=None, username=None, password=None):
- if registry is None:
- registry = self.default_index_server
-
- if username is not None:
- self.www_auth_username = username
- self.www_auth_password = password
-
+ def download_template(self, template, templatedir):
self._check_cert_validate()
- tag = "latest"
- offset = templatename.find(':')
- if offset != -1:
- tag = templatename[offset + 1:]
- templatename = templatename[0:offset]
+
try:
- (data, res) = self._get_json(registry, "/v1/repositories/" + templatename + "/images",
- {"X-Docker-Token": "true"})
+ (data, res) = self._get_json(template,
+ None,
+ "/v1/repositories" + template.path + "/images",
+ {"X-Docker-Token": "true"})
except urllib2.HTTPError, e:
- raise ValueError(["Image '%s' does not exist" % templatename])
+ raise ValueError(["Image '%s' does not exist" % template])
registryendpoint = res.info().getheader('X-Docker-Endpoints')
token = res.info().getheader('X-Docker-Token')
checksums = {}
for layer in data:
pass
- (data, res) = self._get_json(registryendpoint, "/v1/repositories/" + templatename + "/tags",
- { "Authorization": "Token " + token })
+ (data, res) = self._get_json(template,
+ registryendpoint,
+ "/v1/repositories" + template.path + "/tags",
+ { "Authorization": "Token " + token })
cookie = res.info().getheader('Set-Cookie')
+ tag = template.params.get("tag", "latest")
if not tag in data:
- raise ValueError(["Tag '%s' does not exist for image '%s'" % (tag, templatename)])
+ raise ValueError(["Tag '%s' does not exist for image '%s'" % (tag, template)])
imagetagid = data[tag]
- (data, res) = self._get_json(registryendpoint, "/v1/images/" + imagetagid + "/ancestry",
- { "Authorization": "Token "+token })
+ (data, res) = self._get_json(template,
+ registryendpoint,
+ "/v1/images/" + imagetagid + "/ancestry",
+ { "Authorization": "Token "+ token })
if data[0] != imagetagid:
raise ValueError(["Expected first layer id '%s' to match image id '%s'",
@@ -126,8 +118,11 @@ class DockerSource(Source):
datafile = layerdir + "/template.tar.gz"
if not os.path.exists(jsonfile) or not os.path.exists(datafile):
- res = self._save_data(registryendpoint, "/v1/images/" + layerid + "/json",
- { "Authorization": "Token " + token }, jsonfile)
+ res = self._save_data(template,
+ registryendpoint,
+ "/v1/images/" + layerid + "/json",
+ { "Authorization": "Token " + token },
+ jsonfile)
createdFiles.append(jsonfile)
layersize = int(res.info().getheader("Content-Length"))
@@ -136,12 +131,15 @@ class DockerSource(Source):
if layerid in checksums:
datacsum = checksums[layerid]
- self._save_data(registryendpoint, "/v1/images/" + layerid + "/layer",
- { "Authorization": "Token "+token }, datafile, datacsum, layersize)
+ self._save_data(template,
+ registryendpoint,
+ "/v1/images/" + layerid + "/layer",
+ { "Authorization": "Token "+token },
+ datafile, datacsum, layersize)
createdFiles.append(datafile)
index = {
- "name": templatename,
+ "name": template.path,
}
indexfile = templatedir + "/" + imagetagid + "/index.json"
@@ -160,9 +158,11 @@ class DockerSource(Source):
shutil.rmtree(d)
except:
pass
- def _save_data(self,server, path, headers, dest, checksum=None, datalen=None):
+
+ def _save_data(self, template, server, path, headers,
+ dest, checksum=None, datalen=None):
try:
- res = self._get_url(server, path, headers)
+ res = self._get_url(template, server, path, headers)
csum = None
if checksum is not None:
@@ -201,8 +201,22 @@ class DockerSource(Source):
debug("FAIL %s\n" % str(e))
raise
- def _get_url(self,server, path, headers):
- url = "https://" + server + path
+ def _get_url(self, template, server, path, headers):
+ if template.protocol is None:
+ protocol = "https"
+ else:
+ protocol = template.protocol
+
+ if server is None:
+ if template.hostname is None:
+ server = "index.docker.io"
+ else:
+ if template.port is not None:
+ server = template.hostname + ":" + template.port
+ else:
+ server = template.hostname
+
+ url = urlparse.urlunparse((protocol, server, path, None, None, None))
debug("Fetching %s..." % url)
req = urllib2.Request(url=url)
@@ -212,16 +226,18 @@ class DockerSource(Source):
req.add_header(h, headers[h])
#www Auth header starts
- if self.www_auth_username is not None:
- base64string = base64.encodestring('%s:%s' % (self.www_auth_username, self.www_auth_password)).replace('\n', '')
+ if template.username and template.password:
+ base64string = base64.encodestring(
+ '%s:%s' % (template.username,
+ template.password)).replace('\n', '')
req.add_header("Authorization", "Basic %s" % base64string)
#www Auth header finish
return urllib2.urlopen(req)
- def _get_json(self,server, path, headers):
+ def _get_json(self, template, server, path, headers):
try:
- res = self._get_url(server, path, headers)
+ res = self._get_url(template, server, path, headers)
data = json.loads(res.read())
debug("OK\n")
return (data, res)
@@ -229,11 +245,11 @@ class DockerSource(Source):
debug("FAIL %s\n" % str(e))
raise
- def create_template(self, templatename, templatedir, connect=None, format=None):
+ def create_template(self, template, templatedir, connect=None, format=None):
if format is None:
format = self.default_disk_format
self._check_disk_format(format)
- imagelist = self._get_image_list(templatename,templatedir)
+ imagelist = self._get_image_list(template, templatedir)
imagelist.reverse()
parentImage = None
@@ -260,7 +276,7 @@ class DockerSource(Source):
if not format in supportedFormats:
raise ValueError(["Unsupported image format %s" % format])
- def _get_image_list(self,templatename,destdir):
+ def _get_image_list(self, template, destdir):
imageparent = {}
imagenames = {}
imagedirs = os.listdir(destdir)
@@ -273,13 +289,13 @@ class DockerSource(Source):
jsonfile = destdir + "/" + imagetagid + "/template.json"
if os.path.exists(jsonfile):
with open(jsonfile,"r") as f:
- template = json.load(f)
- parent = template.get("parent",None)
+ data = json.load(f)
+ parent = data.get("parent",None)
if parent:
imageparent[imagetagid] = parent
- if not templatename in imagenames:
- raise ValueError(["Image %s does not exist locally" %templatename])
- imagetagid = imagenames[templatename]
+ if not template.path in imagenames:
+ raise ValueError(["Image %s does not exist locally" % template.path])
+ imagetagid = imagenames[template.path]
imagelist = []
while imagetagid != None:
imagelist.append(imagetagid)
@@ -318,7 +334,7 @@ class DockerSource(Source):
cmd = cmd + params
subprocess.call(cmd)
- def delete_template(self, templatename, templatedir):
+ def delete_template(self, template, templatedir):
imageusage = {}
imageparent = {}
imagenames = {}
@@ -332,9 +348,9 @@ class DockerSource(Source):
jsonfile = templatedir + "/" + imagetagid + "/template.json"
if os.path.exists(jsonfile):
with open(jsonfile,"r") as f:
- template = json.load(f)
+ data = json.load(f)
- parent = template.get("parent",None)
+ parent = data.get("parent",None)
if parent:
if parent not in imageusage:
imageusage[parent] = []
@@ -342,10 +358,10 @@ class DockerSource(Source):
imageparent[imagetagid] = parent
- if not templatename in imagenames:
- raise ValueError(["Image %s does not exist locally" %templatename])
+ if not template.path in imagenames:
+ raise ValueError(["Image %s does not exist locally" % template.path])
- imagetagid = imagenames[templatename]
+ imagetagid = imagenames[template.path]
while imagetagid != None:
debug("Remove %s\n" % imagetagid)
parent = imageparent.get(imagetagid,None)
@@ -368,15 +384,15 @@ class DockerSource(Source):
parent = None
imagetagid = parent
- def _get_template_data(self, templatename, templatedir):
- imageList = self._get_image_list(templatename, templatedir)
+ def _get_template_data(self, template, templatedir):
+ imageList = self._get_image_list(template, templatedir)
toplayer = imageList[0]
diskfile = templatedir + "/" + toplayer + "/template.qcow2"
configfile = templatedir + "/" + toplayer + "/template.json"
return configfile, diskfile
- def get_disk(self,templatename, templatedir, imagedir, sandboxname):
- configfile, diskfile = self._get_template_data(templatename, templatedir)
+ def get_disk(self, template, templatedir, imagedir, sandboxname):
+ configfile, diskfile = self._get_template_data(template, templatedir)
tempfile = imagedir + "/" + sandboxname + ".qcow2"
if not os.path.exists(imagedir):
os.makedirs(imagedir)
@@ -387,8 +403,8 @@ class DockerSource(Source):
subprocess.call(cmd)
return tempfile
- def get_command(self, templatename, templatedir, userargs):
- configfile, diskfile = self._get_template_data(templatename, templatedir)
+ def get_command(self, template, templatedir, userargs):
+ configfile, diskfile = self._get_template_data(template, templatedir)
configParser = DockerConfParser(configfile)
cmd = configParser.getCommand()
entrypoint = configParser.getEntrypoint()
@@ -401,13 +417,13 @@ class DockerSource(Source):
else:
return entrypoint + cmd
- def get_env(self, templatename, templatedir):
- configfile, diskfile = self._get_template_data(templatename, templatedir)
+ def get_env(self, template, templatedir):
+ configfile, diskfile = self._get_template_data(template, templatedir)
configParser = DockerConfParser(configfile)
return configParser.getEnvs()
- def get_volumes(self, templatename, templatedir):
- configfile, diskfile = self._get_template_data(templatename, templatedir)
+ def get_volumes(self, template, templatedir):
+ configfile, diskfile = self._get_template_data(template, templatedir)
configParser = DockerConfParser(configfile)
return configParser.getVolumes()
diff --git a/libvirt-sandbox/image/sources/Source.py b/libvirt-sandbox/image/sources/Source.py
index f1dd3e7..bb69682 100644
--- a/libvirt-sandbox/image/sources/Source.py
+++ b/libvirt-sandbox/image/sources/Source.py
@@ -2,6 +2,7 @@
# -*- coding: utf-8 -*-
#
# Copyright (C) 2015 Universitat Politècnica de Catalunya.
+# Copyright (C) 2015 Red Hat, Inc
#
# This library is free software; you can redistribute it and/or
# modify it under the terms of the GNU Lesser General Public
@@ -33,14 +34,10 @@ class Source():
pass
@abstractmethod
- def download_template(self, templatename, templatedir,
- registry=None, username=None, password=None):
+ def download_template(self, template, templatedir):
"""
- :param templatename: name of the template image to download
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to store the template
- :param registry: optional hostname of image registry server
- :param username: optional username to authenticate against registry server
- :param password: optional password to authenticate against registry server
Download a template from the registry, storing it in the local
filesystem
@@ -48,10 +45,10 @@ class Source():
pass
@abstractmethod
- def create_template(self, templatename, templatedir,
+ def create_template(self, template, templatedir,
connect=None, format=None):
"""
- :param templatename: name of the template image to create
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to store the template
:param connect: libvirt connection URI
:param format: disk image format
@@ -63,9 +60,9 @@ class Source():
pass
@abstractmethod
- def delete_template(self, templatename, templatedir):
+ def delete_template(self, template, templatedir):
"""
- :param templatename: name of the template image to delete
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path from which to delete template
Delete all local files associated with the template
@@ -73,9 +70,9 @@ class Source():
pass
@abstractmethod
- def get_command(self, templatename, templatedir, userargs):
+ def get_command(self, template, templatedir, userargs):
"""
- :param templatename: name of the template image to query
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which templates are stored
:param userargs: user specified arguments to run
@@ -85,9 +82,9 @@ class Source():
pass
@abstractmethod
- def get_disk(self,templatename, templatedir, imagedir, sandboxname):
+ def get_disk(self, template, templatedir, imagedir, sandboxname):
"""
- :param templatename: name of the template image to download
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to find template
:param imagedir: local directory in which to storage disk image
@@ -97,9 +94,9 @@ class Source():
pass
@abstractmethod
- def get_env(self,templatename, templatedir):
+ def get_env(self, template, templatedir):
"""
- :param templatename: name of the template image to download
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to find template
Get the dict of environment variables to set
@@ -107,9 +104,9 @@ class Source():
pass
@abstractmethod
- def get_volumes(self,templatename, templatedir):
+ def get_volumes(self, template, templatedir):
"""
- :param templatename: name of the template image to download
+ :param template: libvirt_sandbox.template.Template object
:param templatedir: local directory path in which to find template
Get the list of volumes associated with the template
diff --git a/libvirt-sandbox/image/template.py b/libvirt-sandbox/image/template.py
new file mode 100644
index 0000000..9021242
--- /dev/null
+++ b/libvirt-sandbox/image/template.py
@@ -0,0 +1,109 @@
+#
+# -*- coding: utf-8 -*-
+# Authors: Daniel P. Berrange <berrange(a)redhat.com>
+#
+# Copyright (C) 2015 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+
+import urlparse
+import importlib
+
+class Template(object):
+
+ def __init__(self,
+ source, protocol,
+ hostname, port,
+ username, password,
+ path, params):
+ """
+ :param source: template source name
+ :param protocol: network transport protocol or None
+ :param hostname: registry hostname or None
+ :param port: registry port or None
+ :param username: username or None
+ :param password: password or None
+ :param path: template path identifier
+ :param params: template parameters
+
+ docker:///ubuntu
+
+ docker+https://index.docker.io/ubuntu?tag=latest
+ """
+
+ self.source = source
+ self.protocol = protocol
+ self.hostname = hostname
+ self.port = port
+ self.username = username
+ self.password = password
+ self.path = path
+ self.params = params
+ if self.params is None:
+ self.params = {}
+
+ def get_source_impl(self):
+ mod = importlib.import_module(
+ "libvirt_sandbox.image.sources." +
+ self.source.capitalize() + "Source")
+ classname = self.source.capitalize() + "Source"
+ classimpl = getattr(mod, classname)
+ return classimpl()
+
+ def __repr__(self):
+ if self.protocol is not None:
+ scheme = self.source + "+" + self.protocol
+ else:
+ scheme = self.source
+ if self.hostname:
+ if self.port:
+ netloc = self.hostname + ":" + self.port
+ else:
+ netloc = self.hostname
+
+ if self.username:
+ if self.password:
+ auth = self.username + ":" + self.password
+ else:
+ auth = self.username
+ netloc = auth + "@" + netloc
+ else:
+ netloc = None
+
+ query = "&".join([key + "=" + self.params[key] for key in self.params.keys()])
+ return urlparse.urlunparse((scheme, netloc, self.path, None, query, None))
+
+ @classmethod
+ def from_uri(klass, uri):
+ o = urlparse.urlparse(uri)
+
+ idx = o.scheme.find("+")
+ if idx == -1:
+ source = o.scheme
+ protocol = None
+ else:
+ source = o.scheme[0:idx]
+ protocol = o.schema[idx + 1:]
+
+ query = {}
+ for param in o.query.split("&"):
+ (key, val) = param.split("=")
+ query[key] = val
+ return klass(source, protocol,
+ o.hostname, o.port,
+ o.username, o.password,
+ o.path, query)
+
--
2.4.3
9 years, 3 months
- 1
- 0