[libvirt] [PATCH 0/9] Glue domain and storage
by Osier Yang
This is the 4th part to implement NPIV migration support [1].
Part 1:
https://www.redhat.com/archives/libvir-list/2013-January/msg00859.html
Part 2: (Already ACKed by Michal)
https://www.redhat.com/archives/libvir-list/2013-January/msg00859.html
Part 3:
https://www.redhat.com/archives/libvir-list/2013-January/msg01012.html
To achieve the NPIV migration support, Part 2, Part 3 and this set are
necessary, Part 1 is optional. However, this set can be reviewd independantly.
============
This introduces new XMLs to specify the disk source with storage like
<disk type='volume' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source pool="$pool_name" volume='$vol_name'/>
<seclabel relabel='no'/>
</source>
<target dev='vdb' bus='virtio'/>
</disk>
And before domain starting, the source represented by storage is
translated into the real underlying source.
Diff with RFC:
* The XMLs are more simpler - only using pool name and volume
name to specify disk source.
* Support network pool (rbd, and sheepdog)
* Support startupPolicy for volume type disk
* Support seclabels for volume type disk
* Fix bugs on disk source formating
Osier Yang (9):
rng: Add definition for network disk source
Introduce new XMLs to specify disk source using libvirt storage
New files to glue domain and storage together
Implement translateDiskSourcePool
qemu: Translate the source when starting domain
Support startupPolicy for 'volume' disk
conf: Fix bugs on disk source formating
Support seclabels for volume type disk
Support network pool for volume disk
docs/formatdomain.html.in | 49 +++--
docs/schemas/domaincommon.rng | 111 +++++---
src/Makefile.am | 3 +-
src/conf/domain_conf.c | 278 ++++++++++++++------
src/conf/domain_conf.h | 9 +
src/conf/domain_storage.c | 44 +++
src/conf/domain_storage.h | 37 +++
src/libvirt_private.syms | 5 +
src/qemu/qemu_driver.c | 7 +
src/storage/storage_driver.c | 144 ++++++++++
.../qemuxml2argv-disk-source-pool.xml | 33 +++
tests/qemuxml2xmltest.c | 1 +
12 files changed, 573 insertions(+), 148 deletions(-)
create mode 100644 src/conf/domain_storage.c
create mode 100644 src/conf/domain_storage.h
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-disk-source-pool.xml
[1] https://www.redhat.com/archives/libvir-list/2012-November/msg00826.html
Osier
11 years, 9 months
[libvirt] [PATCH 00/17] virsh: Rework string option retrieval
by Peter Krempa
This patchset fixes retrieval of string arguments in virsh to report sane
errors in all the places it's used. A new string option retrieval function is
added that supports error reporting and changes appropriate places to use it.
Along with that this series cleans up a few places with old code and bad style.
*** BLURB HERE ***
Peter Krempa (17):
virsh-snapshot: Fix coding style and remove dead code
virsh-volume: Rename helper function makeCloneXML to vshMakeCloneXML
virsh-pool: Rename helper function buildPoolXML to vshBuildPoolXML
virsh: Add helper to request string arguments with error reporting
virsh-pool: Update pool commands to use vshCommandOptStringReq
virsh-volume: Update volume commands to use vshCommandOptStringReq
virsh-domain-monitor: Update domain commands to use
vshCommandOptStringReq
virsh-domain: Update domain commands to use vshCommandOptStringReq
virsh-host: Update host commands to use vshCommandOptStringReq
virsh-network: Update network commands to use vshCommandOptStringReq
virsh-interface: Update interface commands to use
vshCommandOptStringReq
virsh-nodedev: Update node device commands to use
vshCommandOptStringReq
virsh-nodedev: Refactor error paths, error messages and whitespace
virsh-snapshot: Refactor cmdSnapshotDumpXML
virsh-snapshot: Update snapshot commands to use vshCommandOptStringReq
virsh-nwfilter: Update nwfilter commands to use vshCommandOptStringReq
virsh-secret: Refactor error paths
tools/virsh-domain-monitor.c | 44 +++----
tools/virsh-domain.c | 299 +++++++++++++++++--------------------------
tools/virsh-host.c | 9 +-
tools/virsh-interface.c | 8 +-
tools/virsh-network.c | 19 +--
tools/virsh-nodedev.c | 44 ++++---
tools/virsh-nwfilter.c | 4 +-
tools/virsh-pool.c | 56 ++++----
tools/virsh-secret.c | 30 +++--
tools/virsh-snapshot.c | 55 +++-----
tools/virsh-volume.c | 73 +++++------
tools/virsh.c | 51 ++++++++
tools/virsh.h | 4 +
13 files changed, 320 insertions(+), 376 deletions(-)
--
1.8.1.1
11 years, 9 months
[libvirt] RFC: Admin interface for connection monitoring
by Martin Kletzander
Hello everyone.
Trying to create new interface for the libvirt daemon, mainly for
connection monitoring, I came across few things I'd like to
propose/discuss. This interface will be used to manage active
connections of the daemon (list/kill) and other until-now impossible
things (e.g. changing logging filters on-the-fly).
Connection to this interface will be:
- available even when no normal/priority workers are available,
- distinguished (or completely separated) from other daemon
connections,
- (should be) more restricted than other connections.
Having the connection available with all the worker threads being
blocked/deadlocked creates the possibility of for example listing the
connections even when all APIs block. Such manipulations mustn't be
possible over the standard connections and that's why the connections
should be distinguished. The connection restriction is hard to check,
but the permissions for the socket should be less or equal to the rw
socket with having no authentication only when rw socket has no
authentication.
My idea is following:
- new 'admin_protocol.x' with specification of new program,
- default connection limitation to this interface to only root-owned
socket with 700 (authentication optional, see above), so only root on
the host can utilize it,
- configurable number of admin workers, but with 1 as a default,
- splitting usable parts of virsh and reusing them to create
virsh-admin binary.
More controversial ideas I'd like to hear somebody else's opinion (even
more than for the other things):
- create a new separate library (libvirt-admin.so) to be used in the
client, this library will have it's own virAdminConnect{,Auth}Open(),
probably without an URI parameter (or with, but disabled for now, in
case we want to enable remote connections later)
- it is IMHO pointless to add a driver type for this, but it should
definitely be configurable at compile time as well as in config
(admin_workers = 0, maybe),
- keep the connection handling on the current code and instead of
duplicating much of it (so there will be a new service, much like the
read-only socket, but for admin API only),
- create a new pool of workers just for admin connections (similarly to
the priority workers pool), with the dispatching being recognized by
the protocol it is created in (remote, qemu, lxc protocols will go to
normal/priority workers, whether admin protocol will be dispatched to
the admin workers) and of course impossible from other connections
than the admin one, which leads me to the next point;
- we can either add next flag for admin connection (similarly to the
readonly one) or make it future-friendly and change the readonly bool
into a virConnectionType enum (for example).
I must admit, I'm probably doing something wrong or we have old
documentation. Commands like 'make -C src rpcgen' don't work and I
don't understand why admin_protocol-structs isn't automatically
generated, but I've made it work for now.
Have a nice day,
Martin
11 years, 9 months
[libvirt] How to connect to console of domain on PowerPC?
by Yin Olivia-R63875
Hi,
I tried to use libvirt to run KVM/QEMU on Freescale PowerPC platforms.
So far there's only one serial device (spapr-vty) defined in QEMU to work as console for IBM PSeries platform.
There's no serial device support in QEMU for Freescale PowerPC (ePAPR).
libvirt/src/qemu/qemu_command.c
/* This function generates the correct '-device' string for character
* devices of each architecture.
*/
char *
qemuBuildChrDeviceStr(virDomainChrDefPtr serial,
virBitmapPtr qemuCaps,
char *os_arch,
char *machine)
{
virBuffer cmd = VIR_BUFFER_INITIALIZER;
if (STREQ(os_arch, "ppc64") && STREQ(machine, "pseries")) {
if (serial->deviceType == VIR_DOMAIN_CHR_DEVICE_TYPE_SERIAL &&
serial->source.type == VIR_DOMAIN_CHR_TYPE_PTY &&
serial->info.type == VIR_DOMAIN_DEVICE_ADDRESS_TYPE_SPAPRVIO) {
virBufferAsprintf(&cmd, "spapr-vty,chardev=char%s",
serial->info.alias);
if (qemuBuildDeviceAddressStr(&cmd, &serial->info, qemuCaps) < 0)
goto error;
}
} else
virBufferAsprintf(&cmd, "isa-serial,chardev=char%s,id=%s",
serial->info.alias, serial->info.alias);
if (virBufferError(&cmd)) {
virReportOOMError();
goto error;
}
return virBufferContentAndReset(&cmd);
error:
virBufferFreeAndReset(&cmd);
return NULL;
}
We usually connect guest with telnet.
For instance,
/usr/bin/qemu-system-ppc -name demo -M ppce500v2 -enable-kvm -m 256 -nographic -kernel /media/ram/uImage -initrd /media/ram/ramdisk -append "root=/dev/ram rw console=ttyS0,115200" -serial tcp::4445,server
Then to run 'telnet 10.193.20.xxx 4445' could connect the guest.
The temporary workaround is not add '-device' string after '-serial' option.
diff -Nur libvirt-0.10.1.orig/src/qemu/qemu_command.c libvirt-0.10.1/src/qemu/qemu_command.c
--- libvirt-0.10.1.orig/src/qemu/qemu_command.c 2012-08-30 15:35:18.000000000 +0530
+++ libvirt-0.10.1/src/qemu/qemu_command.c 2012-10-05 17:19:32.060368755 +0530
@@ -5501,13 +5501,15 @@
virCommandAddArg(cmd, devstr);
VIR_FREE(devstr);
- virCommandAddArg(cmd, "-device");
- if (!(devstr = qemuBuildChrDeviceStr(serial, qemuCaps,
+ if (!STREQ(def->os.arch, "ppc")) {
+ virCommandAddArg(cmd, "-device");
+ if (!(devstr = qemuBuildChrDeviceStr(serial,
+ qemuCaps,
def->os.arch,
def->os.machine)))
- goto error;
- virCommandAddArg(cmd, devstr);
- VIR_FREE(devstr);
+ goto error;
+ virCommandAddArg(cmd, devstr);
+ VIR_FREE(devstr);
+ }
} else {
virCommandAddArg(cmd, "-serial");
if (!(devstr = qemuBuildChrArgStr(&serial->source, NULL)))
Applying the above patch to libvirt, all the other domain control commands could work except 'virsh console domain'.
# cat >demo.args <<EOF
> /usr/bin/qemu-system-ppc -name demo -M ppce500v2 -enable-kvm -m 256
> -nographic -kernel /media/ram/uImage -initrd /media/ram/ramdisk
> -append "root=/dev/ram rw console=ttyS0,115200" -serial
> tcp::4445,server -net nic EOF
# vi demo.args
/usr/bin/qemu-system-ppc -name demo -M ppce500v2 -enable-kvm -m 256 -nographic -kernel /media/ram/uImage -initrd /media/ram/ramdisk -append "root=/dev/ram rw console=ttyS0,115200" -serial tcp::4445,server -net nic
# virsh domxml-from-native qemu-argv demo.args >demo.xml # vi demo.xml <domain type='kvm'>
<name>demo</name>
<uuid>985d7154-83c8-0763-cbac-ecd159eee8a6</uuid>
<memory unit='KiB'>262144</memory>
<currentMemory unit='KiB'>262144</currentMemory>
<vcpu placement='static'>1</vcpu>
<os>
<type arch='ppc' machine='ppce500v2'>hvm</type>
<kernel>/media/ram/uImage</kernel>
<initrd>/media/ram/ramdisk</initrd>
<cmdline>root=/dev/ram rw console=ttyS0,115200</cmdline>
</os>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-ppc</emulator>
<serial type='tcp'>
<source mode='bind' host='' service='4445'/>
<protocol type='raw'/>
<target port='0'/>
</serial>
<console type='tcp'>
<source mode='bind' host='' service='4445'/>
<protocol type='raw'/>
<target type='serial' port='0'/>
</console>
<memballoon model='virtio'/>
</devices>
</domain>
# virsh -c qemu:///system define demo.xml # virsh -c qemu:///system start demo
But it seemed that can't connect to the console.
# virsh -c qemu:///system console demo
Connected to domain test
Escape character is ^]
error: internal error character device (null) is not using a PTY
I tried also use '-serial pty' option,
/usr/bin/qemu-system-ppc -name test -M ppce500v2 -enable-kvm -m 256 -nographic -kernel /media/ram/uImage -initrd /media/ram/ramdisk -append "root=/dev/ram rw console=ttyS0,115200" -serial pty
Then there's no other output after the below message:
# virsh -c qemu:///system console demo
Connected to domain test
Escape character is ^]
It seemed not libvirt group issue.
I also tried the LXC.
It could connect to the console if <init>/bin/sh/</init> instead of <init>/sbin/init</init>
Best Regards,
Olivia
11 years, 9 months
[libvirt] [PATCH] rpc: Fix crash on error paths of message dispatching
by Peter Krempa
When reading and dispatching of a message failed the message was freed
but wasn't removed from the message queue.
After that when the connection was about to be closed the pointer for
the message was still present in the queue and it was passed to
virNetMessageFree which tried to call the callback function from an
uninitialized pointer.
This patch removes the message from the queue before it's freed.
* rpc/virnetserverclient.c: virNetServerClientDispatchRead:
- avoid use after free of RPC messages
---
src/rpc/virnetserverclient.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/src/rpc/virnetserverclient.c b/src/rpc/virnetserverclient.c
index af0560e..446e1e9 100644
--- a/src/rpc/virnetserverclient.c
+++ b/src/rpc/virnetserverclient.c
@@ -987,6 +987,7 @@ readmore:
/* Decode the header so we can use it for routing decisions */
if (virNetMessageDecodeHeader(msg) < 0) {
+ virNetMessageQueueServe(&client->rx);
virNetMessageFree(msg);
client->wantClose = true;
return;
@@ -996,6 +997,7 @@ readmore:
* file descriptors */
if (msg->header.type == VIR_NET_CALL_WITH_FDS &&
virNetMessageDecodeNumFDs(msg) < 0) {
+ virNetMessageQueueServe(&client->rx);
virNetMessageFree(msg);
client->wantClose = true;
return; /* Error */
@@ -1005,6 +1007,7 @@ readmore:
for (i = msg->donefds ; i < msg->nfds ; i++) {
int rv;
if ((rv = virNetSocketRecvFD(client->sock, &(msg->fds[i]))) < 0) {
+ virNetMessageQueueServe(&client->rx);
virNetMessageFree(msg);
client->wantClose = true;
return;
--
1.8.1.1
11 years, 9 months
[libvirt] [Libvirt]How to get libvirt version after git clone?
by harryxiyou
Hi all,
After i clone libvirt git repo, how i could get this libvirt version exactly?
$ git clone git://libvirt.org/libvirt.git
$ git branch -a
* master
remotes/origin/HEAD -> origin/master
remotes/origin/master
remotes/origin/v0.10.2-maint
remotes/origin/v0.9.11-maint
remotes/origin/v0.9.6-maint
I can just see up stuffs but exact current version. Could anyone give me
some suggestions? Thanks in advance.
--
Thanks
Harry Wei
11 years, 9 months
[libvirt] [Libvirt]Snapshot operations in Libvirt
by harryxiyou
Hi all,
I see our libvirt snapshot XML form like following, which this
XML is to create a disk snapshot of just vda on a qemu domain
with two disks.
<domainsnapshot>
<description>Snapshot of OS install and updates</description>
<disks>
<disk name='/path/to/old'>
<source file='/path/to/new'/>
</disk>
<disk name='vdb' snapshot='no'/>
</disks>
</domainsnapshot>
My understanding about up XML is like this: it creates a snapshot
for '/path/to/old', which '/path/to/old' would be the read-only backing
file to the new active file '/path/to/new'. I also have some questions
about current libvirt snapshot operations like following.
1, Is my up understanding right? Or maybe you have any other suggestions.
2, I wonder whether libvirt *NOW* support start a snapshot and then
mount this new storage block, which is based on the snapshot.
Please give me some real examples (like Sheepdog volumes) if you
can.
--
Thanks
Harry Wei
11 years, 9 months
[libvirt] [PATCH] hyperv: Avoid Coverity warning about resource leak in hypervOpen
by Matthias Bolte
Coverity tagged the esxhypervPrivate pointer a potentially leaked
because of the conditional free call. But this is a false positive,
there is not actual leak.
Avoid the conditial code to make this more obvious to Coverity.
---
This patch is meant as a replacement for this patch
https://www.redhat.com/archives/libvir-list/2013-January/msg00542.html
src/hyperv/hyperv_driver.c | 7 ++-----
1 file changed, 2 insertions(+), 5 deletions(-)
diff --git a/src/hyperv/hyperv_driver.c b/src/hyperv/hyperv_driver.c
index 601a85a..0d2eaae 100644
--- a/src/hyperv/hyperv_driver.c
+++ b/src/hyperv/hyperv_driver.c
@@ -200,14 +200,11 @@ hypervOpen(virConnectPtr conn, virConnectAuthPtr auth, unsigned int flags)
}
conn->privateData = priv;
-
+ priv = NULL;
result = VIR_DRV_OPEN_SUCCESS;
cleanup:
- if (result == VIR_DRV_OPEN_ERROR) {
- hypervFreePrivate(&priv);
- }
-
+ hypervFreePrivate(&priv);
VIR_FREE(username);
VIR_FREE(password);
hypervFreeObject(priv, (hypervObject *)computerSystem);
--
1.7.9.5
11 years, 9 months
[libvirt] [PATCH] esx: Avoid Coverity warning about resource leak in esxOpen
by Matthias Bolte
Commit 4445e16bfa8056980ac643fabf17186f9e685925 changed the signature
of esxConnectToHost and esxConnectToVCenter by replacing the esxPrivate
pointer with virConnectPtr. The esxPrivate pointer was then retrieved
again from virConnectPtr's privateData. This resulted in a NULL pointer
dereference, because the privateData pointer was not yet initialized at
the point where esxConnectToHost and esxConnectToVCenter are called.
This was fixed in commit b126715a48cd0cbe32ec6468c267cd8cf2961c55 that
moved the initialization of privateData before the problematic calls.
Coverity tagged the esxPrivate pointer a potentially leaked because of
the conditional free call. But this is a false positive, there is not
actual leak.
Avoid this warning from Coverity by making the call to esxFreePrivate
unconditional and changing esxConnectToHost and esxConnectToVCenter back
to take a esxPrivate pointer directly. This allows to assign esxPrivate
to the virConnectPtr's privateData pointer as one of the last steps in
esxOpen making it more obvious that it is not initialized during the
earlier steps of esxOpen.
---
This patch is meant as a replacement this patch:
https://www.redhat.com/archives/libvir-list/2013-January/msg00530.html
src/esx/esx_driver.c | 23 ++++++++++-------------
1 file changed, 10 insertions(+), 13 deletions(-)
diff --git a/src/esx/esx_driver.c b/src/esx/esx_driver.c
index 1366c81..dad10a1 100644
--- a/src/esx/esx_driver.c
+++ b/src/esx/esx_driver.c
@@ -650,7 +650,8 @@ esxCapsInit(esxPrivate *priv)
static int
-esxConnectToHost(virConnectPtr conn,
+esxConnectToHost(esxPrivate *priv,
+ virConnectPtr conn,
virConnectAuthPtr auth,
char **vCenterIpAddress)
{
@@ -663,7 +664,6 @@ esxConnectToHost(virConnectPtr conn,
esxVI_String *propertyNameList = NULL;
esxVI_ObjectContent *hostSystem = NULL;
esxVI_Boolean inMaintenanceMode = esxVI_Boolean_Undefined;
- esxPrivate *priv = conn->privateData;
esxVI_ProductVersion expectedProductVersion = STRCASEEQ(conn->uri->scheme, "esx")
? esxVI_ProductVersion_ESX
: esxVI_ProductVersion_GSX;
@@ -785,7 +785,8 @@ esxConnectToHost(virConnectPtr conn,
static int
-esxConnectToVCenter(virConnectPtr conn,
+esxConnectToVCenter(esxPrivate *priv,
+ virConnectPtr conn,
virConnectAuthPtr auth,
const char *hostname,
const char *hostSystemIpAddress)
@@ -796,7 +797,6 @@ esxConnectToVCenter(virConnectPtr conn,
char *unescapedPassword = NULL;
char *password = NULL;
char *url = NULL;
- esxPrivate *priv = conn->privateData;
if (hostSystemIpAddress == NULL &&
(priv->parsedUri->path == NULL || STREQ(priv->parsedUri->path, "/"))) {
@@ -1008,8 +1008,6 @@ esxOpen(virConnectPtr conn, virConnectAuthPtr auth,
priv->supportsLongMode = esxVI_Boolean_Undefined;
priv->usedCpuTimeCounterId = -1;
- conn->privateData = priv;
-
/*
* Set the port dependent on the transport protocol if no port is
* specified. This allows us to rely on the port parameter being
@@ -1036,7 +1034,7 @@ esxOpen(virConnectPtr conn, virConnectAuthPtr auth,
if (STRCASEEQ(conn->uri->scheme, "esx") ||
STRCASEEQ(conn->uri->scheme, "gsx")) {
/* Connect to host */
- if (esxConnectToHost(conn, auth,
+ if (esxConnectToHost(priv, conn, auth,
&potentialVCenterIpAddress) < 0) {
goto cleanup;
}
@@ -1075,7 +1073,7 @@ esxOpen(virConnectPtr conn, virConnectAuthPtr auth,
}
}
- if (esxConnectToVCenter(conn, auth,
+ if (esxConnectToVCenter(priv, conn, auth,
vCenterIpAddress,
priv->host->ipAddress) < 0) {
goto cleanup;
@@ -1085,7 +1083,7 @@ esxOpen(virConnectPtr conn, virConnectAuthPtr auth,
priv->primary = priv->host;
} else { /* VPX */
/* Connect to vCenter */
- if (esxConnectToVCenter(conn, auth,
+ if (esxConnectToVCenter(priv, conn, auth,
conn->uri->server,
NULL) < 0) {
goto cleanup;
@@ -1101,13 +1099,12 @@ esxOpen(virConnectPtr conn, virConnectAuthPtr auth,
goto cleanup;
}
+ conn->privateData = priv;
+ priv = NULL;
result = VIR_DRV_OPEN_SUCCESS;
cleanup:
- if (result == VIR_DRV_OPEN_ERROR) {
- esxFreePrivate(&priv);
- }
-
+ esxFreePrivate(&priv);
VIR_FREE(potentialVCenterIpAddress);
return result;
--
1.7.9.5
11 years, 9 months
[libvirt] [Sheepdog][Libvirt]Test boot VMs from sheepdog volumes in Libvirt
by harryxiyou
Hi all,
We can get "Boot VMs from sheepdog volumes in Libvirt" way
from https://github.com/collie/sheepdog/wiki/Libvirt like following.
a, prepare a file containing an XML domain description
$ cat > sheepdog.xml
<domain type='qemu'>
<name>testvm</name>
<memory>1048576</memory>
<os>
<type arch='x86_64'>hvm</type>
</os>
<devices>
<disk type='network'>
<source protocol="sheepdog" name="testvdi"/>
<target dev='hda' bus='ide'/>
</disk>
<graphics type='vnc' port='-1' autoport='yes'/>
</devices>
</domain>
b, boot from testvdi with virsh
$ virsh create sheepdog.xml
c, connect to a VNC console of the running VM
$ vncviewer localhost
I have some following questions about up three steps.
1, Before a or b(up boot step) step, should we have to install our OS
into 'testvdi'?
That is to say, we have to install OS from os.iso file into 'testvdi',
which should use
'qemu-system-x86_64' command to install OS, right?
2, Before this test, shoul we have to build Sheepdog, QEMU, Libvirt environment?
3, This is the storage online management in Libvirt, right?
4, This test just test the codes Morita take a patch named "add
network disk support"
for Libvirt, which is located here:
http://libvirt.org/git/?p=libvirt.git;a=commit;h=036ad5052b43fe9f0d197e89...
and also test Sheepdog driver for QEMU, right?
5, If i installed a 32bits OS to testvdi, should i modify
<os>
<type arch='x86_64'>hvm</type>
</os>
to be
<os>
<type arch='x86'>hvm</type>
</os>
, right?
6, The step c (up boot step), which is
$ vncviewer localhost
Actually, i am not clear about this 'localhost'. Why should we link
localhost but other ip addresses?
Could anyone please give me some suggestions? Thanks in advance ;-)
--
Thanks
Harry Wei
11 years, 9 months