- Devel - libvirt List Archives

[libvirt] [PATCH v2 0/2] Rework reference counting in QEMU
by Martin Kletzander 23 Dec '14

23 Dec '14

Our reference locking for objects is great and powerful thing. The problem is that it was either not followed through completely or the design was not complete, but it doesn't matter now. There are few bugs in the code due to the reference counting and the daemon is lacking some performance in specific scenarios. This "series" tried to fix that using the following idea: - Each API working with a domain object that has to get it from the list will have its *own* reference, not borrowed one from the list. - When adding a domain into the list, the reference counter is increased (this is the reference that is there just for being in the list) and when being removed, it is decreased. No special-casing of "if this is was the last reference" and other funny stuff. - When job is created, there is no need to increase the reference counter as there are at least two references for the domain: 1) The API that created the job has one, so if it's not async it will be kept until the API ends and at that point the job won't exist any more. 2) The domain list has one and even though I said nobody needs to rely on that, async APIs probably will do that, but there's an excuse for that. In order to remove the domain from the list, you need a job and that won't succeed unless the async one ended. So we're good in this case as well. After searching through the code for all things that needed to be removed and fixing everything I could possibly think of, I tried a few things on my setup and it looks like it works. However, I haven't tried *every single API*, but I hope that's understandable. On the other hand, I asked Pavel to try running virt-test with these patches applied, hopefully we'll get an idea about how reliable this "series" is. I used "series" (with quotes) on purpose, because first patch just adds two new wrappers for slightly modified function and the second one changes the whole qemu driver at once. Unfortunately the second patch couldn't be broken up to more parts due to the nature of the fix. Martin Kletzander (2): conf: Rework virDomainObjListFindByUUID to allow more concurrent APIs qemu: completely rework reference counting src/conf/domain_conf.c | 29 +- src/conf/domain_conf.h | 2 + src/libvirt_private.syms | 1 + src/qemu/THREADS.txt | 40 ++- src/qemu/qemu_domain.c | 49 ++-- src/qemu/qemu_domain.h | 12 +- src/qemu/qemu_driver.c | 708 ++++++++++++++++------------------------------ src/qemu/qemu_migration.c | 111 +++----- src/qemu/qemu_migration.h | 10 +- src/qemu/qemu_process.c | 77 ++--- 10 files changed, 400 insertions(+), 639 deletions(-) -- 2.2.0

3 9

Re: [libvirt] [openstack-dev] [nova] - 'nova reboot' causes console-log truncated
by Surojit Pathak 23 Dec '14

23 Dec '14

On 11/14/14 2:02 AM, Daniel P. Berrange wrote: > On Thu, Nov 13, 2014 at 01:55:06PM -0800, Surojit Pathak wrote: >> Hi all, >> >> [Issue observed] >> If we issue 'nova reboot <server>', we get to have the console output of the >> latest bootup of the server only. The console output of the previous boot >> for the same server vanishes due to truncation[1]. If we do reboot from >> within the VM instance [ #sudo reboot ], or reboot the instance with 'virsh >> reboot <instance>' the behavior is not the same, where the console.log keeps >> increasing, with the new output being appended. >> This loss of history makes some debugging scenario difficult due to lack of >> information being available. >> >> Please point me to any solution/blueprint for this issue, if already >> planned. Otherwise, please comment on my analysis and proposals as solution, >> below - >> >> [Analysis] >> Nova's libvirt driver on compute node tries to do a graceful restart of the >> server instance, by attempting a soft_reboot first. If soft_reboot fails, it >> attempts a hard_reboot. As part of soft_reboot, it brings down the instance >> by calling shutdown(), and then calls createWithFlags() to bring this up. >> Because of this, qemu-kvm process for the instance gets terminated and new >> process is launched. In QEMU, the chardev file is opened with O_TRUNC, and >> thus we lose the previous content of the console.log file. >> On the other-hand, during 'virsh reboot <instance>', the same qemu-kvm >> process continues, and libvirt actually does a qemuDomainSetFakeReboot(). >> Thus the same file continues capturing the new console output as a >> continuation into the same file. > Nova and libvirt have support for issuing a graceful reboot via the QEMU > guest agent. So if you make sure that is installed, and tell Nova to use > it, then Nova won't have to stop & recreate the QEMU process and thus > won't have the problem of overwriting the logs. Hi Daniel, Having GA to do graceful restart is nice option. But if it were to just preserve the same console file, even 'virsh reboot' achieves the purpose. As I explained in my original analysis, Nova seems to have not taken the path, as it does not want to have a false positive, where the GA does not respond or 'virDomain.reboot' fails later and the domain is not really restarted. [ CC-ed vish, author of nova <http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/>/virt <http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt/>/libvirt <http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt/libvir…>/driver.py <http://tripsgrips.corp.gq1.yahoo.com:8080/source/xref/nova/nova/virt/libvir…> ] IMHO, QEMU should preserve the console-log file for a given domain, if it exists, by not opening with O_TRUNC option, instead opening with O_APPEND. I would like to draw a comparison of a real computer to which we might be connected over serial console, and the box gets powered down and up with external button press, and we do not lose the console history, if connected. And that's what is the experience console-log intends to provide. If you think, this is agreeable, please let me know, I will send the patch to qemu-devel@. -- Regards, SURO

2 3

Re: [libvirt] [PATCH 2/2] Add tests to xmconfigtest
by Jim Fehlig 23 Dec '14

23 Dec '14

Chun Yan Liu wrote: > >>>> On 12/23/2014 at 09:42 AM, in message <5498C888.6000903(a)suse.com>, Jim Fehlig >>>> > <jfehlig(a)suse.com> wrote: > >> Chunyan Liu wrote: >> >> Hi Chunyan, >> >> Thanks for the fix, and the test! But I have a few questions regarding >> the latter... >> >> >>> Add tests to testing HVM default features (pae, acpi, apic) >>> conversion from xm config to libvirt xml and from libvirt >>> xml to xm config. >>> >>> Signed-off-by: Chunyan Liu <cyliu(a)suse.com> >>> --- >>> .../xmconfigdata/test-fullvirt-default-feature.cfg | 23 +++++++++++ >>> .../test-fullvirt-default-feature.cfg.out | 26 ++++++++++++ >>> .../xmconfigdata/test-fullvirt-default-feature.xml | 48 >>> >> ++++++++++++++++++++++ >> >>> tests/xmconfigtest.c | 9 +++- >>> 4 files changed, 105 insertions(+), 1 deletion(-) >>> create mode 100644 tests/xmconfigdata/test-fullvirt-default-feature.cfg >>> create mode 100644 tests/xmconfigdata/test-fullvirt-default-feature.cfg.out >>> create mode 100644 tests/xmconfigdata/test-fullvirt-default-feature.xml >>> >>> diff --git a/tests/xmconfigdata/test-fullvirt-default-feature.cfg >>> >> b/tests/xmconfigdata/test-fullvirt-default-feature.cfg >> >>> new file mode 100644 >>> index 0000000..5ce234f >>> --- /dev/null >>> +++ b/tests/xmconfigdata/test-fullvirt-default-feature.cfg >>> >>> >> >> Why is this file needed? >> > " > Here we are testing default value conversion. That is: > if there is no apci/pae/apic specified in xm config, after conversion, > libvirt xml should include: > <features> > <apic/> > <acpi/> > <pae/> > </features> > Ah, ok. Agreed that this test is missing. > So, from xm config -> xml, the cfg file should be like this. > >> >> >>> @@ -0,0 +1,23 @@ >>> +name = "XenGuest2" >>> +uuid = "c7a5fdb2-cdaf-9455-926a-d65c16db1809 >>> +maxmem = 579 >>> +memory = 394 >>> +vcpus = 1 >>> +builder = "hvm" >>> +kernel = "/usr/lib/xen/boot/hvmloader" >>> +boot = "d" >>> +hpet = 1 >>> +localtime = 0 >>> +on_poweroff = "destroy" >>> +on_reboot = "restart" >>> +on_crash = "restart" >>> +device_model = "/usr/lib/xen/bin/qemu-dm" >>> +sdl = 0 >>> +vnc = 1 >>> +vncunused = 1 >>> +vnclisten = "127.0.0.1" >>> +vncpasswd = "123poi" >>> +vif = [ >>> >> "mac=00:16:3e:66:92:9c,bridge=xenbr1,script=vif-bridge,model=e1000,type=ioem >> u" ] >> >>> +parallel = "none" >>> +serial = "none" >>> +disk = [ "phy:/dev/HostVG/XenGuest2,hda,w", >>> >> "file:/root/boot.iso,hdc:cdrom,r" ] >> >>> diff --git a/tests/xmconfigdata/test-fullvirt-default-feature.cfg.out >>> >> b/tests/xmconfigdata/test-fullvirt-default-feature.cfg.out >> >>> new file mode 100644 >>> index 0000000..97a9827 >>> --- /dev/null >>> +++ b/tests/xmconfigdata/test-fullvirt-default-feature.cfg.out >>> >>> >> >> IMO, this file should be renamed to 'test-fullvirt-default-feature.cfg'. >> > > And from xml -> xm config, if in xml there is: > <features> > <apic/> > <acpi/> > <pae/> > </features> > Then after conversion, in xm config out file there will be explicitly: > acpi=1 > apic=1 > pae=1 > > So, thlis is a little different from test-fullvirt-default-feature.cfg. > This is actually tested in all of the other test-fullvirt-* tests. I don't think we need an explicit test for it. > >> >> >>> @@ -0,0 +1,26 @@ >>> +name = "XenGuest2" >>> +uuid = "c7a5fdb2-cdaf-9455-926a-d65c16db1809" >>> +maxmem = 579 >>> +memory = 394 >>> +vcpus = 1 >>> +builder = "hvm" >>> +kernel = "/usr/lib/xen/boot/hvmloader" >>> +boot = "d" >>> +pae = 1 >>> +acpi = 1 >>> +apic = 1 >>> +hpet = 1 >>> +localtime = 0 >>> +on_poweroff = "destroy" >>> +on_reboot = "restart" >>> +on_crash = "restart" >>> +device_model = "/usr/lib/xen/bin/qemu-dm" >>> +sdl = 0 >>> +vnc = 1 >>> +vncunused = 1 >>> +vnclisten = "127.0.0.1" >>> +vncpasswd = "123poi" >>> +vif = [ >>> >> "mac=00:16:3e:66:92:9c,bridge=xenbr1,script=vif-bridge,model=e1000,type=ioem >> u" ] >> >>> +parallel = "none" >>> +serial = "none" >>> +disk = [ "phy:/dev/HostVG/XenGuest2,hda,w", >>> >> "file:/root/boot.iso,hdc:cdrom,r" ] >> >>> diff --git a/tests/xmconfigdata/test-fullvirt-default-feature.xml >>> >> b/tests/xmconfigdata/test-fullvirt-default-feature.xml >> >>> new file mode 100644 >>> index 0000000..57a6531 >>> --- /dev/null >>> +++ b/tests/xmconfigdata/test-fullvirt-default-feature.xml >>> @@ -0,0 +1,48 @@ >>> +<domain type='xen'> >>> + <name>XenGuest2</name> >>> + <uuid>c7a5fdb2-cdaf-9455-926a-d65c16db1809</uuid> >>> + <memory unit='KiB'>592896</memory> >>> + <currentMemory unit='KiB'>403456</currentMemory> >>> + <vcpu placement='static'>1</vcpu> >>> + <os> >>> + <type arch='i686' machine='xenfv'>hvm</type> >>> + <loader type='rom'>/usr/lib/xen/boot/hvmloader</loader> >>> + <boot dev='cdrom'/> >>> + </os> >>> + <features> >>> + <acpi/> >>> + <apic/> >>> + <pae/> >>> + </features> >>> + <clock offset='utc' adjustment='reset'> >>> + <timer name='hpet' present='yes'/> >>> + </clock> >>> + <on_poweroff>destroy</on_poweroff> >>> + <on_reboot>restart</on_reboot> >>> + <on_crash>restart</on_crash> >>> + <devices> >>> + <emulator>/usr/lib/xen/bin/qemu-dm</emulator> >>> + <disk type='block' device='disk'> >>> + <driver name='phy'/> >>> + <source dev='/dev/HostVG/XenGuest2'/> >>> + <target dev='hda' bus='ide'/> >>> + </disk> >>> + <disk type='file' device='cdrom'> >>> + <driver name='file'/> >>> + <source file='/root/boot.iso'/> >>> + <target dev='hdc' bus='ide'/> >>> + <readonly/> >>> + </disk> >>> + <interface type='bridge'> >>> + <mac address='00:16:3e:66:92:9c'/> >>> + <source bridge='xenbr1'/> >>> + <script path='vif-bridge'/> >>> + <model type='e1000'/> >>> + </interface> >>> + <input type='mouse' bus='ps2'/> >>> + <input type='keyboard' bus='ps2'/> >>> + <graphics type='vnc' port='-1' autoport='yes' listen='127.0.0.1' >>> >> passwd='123poi'> >> >>> + <listen type='address' address='127.0.0.1'/> >>> + </graphics> >>> + </devices> >>> +</domain> >>> diff --git a/tests/xmconfigtest.c b/tests/xmconfigtest.c >>> index 0c6f803..b0b7b30 100644 >>> --- a/tests/xmconfigtest.c >>> +++ b/tests/xmconfigtest.c >>> @@ -176,21 +176,26 @@ testCompareHelper(const void *data) >>> const struct testInfo *info = data; >>> char *xml = NULL; >>> char *cfg = NULL; >>> + char *cfgout = NULL; >>> >>> if (virAsprintf(&xml, "%s/xmconfigdata/test-%s.xml", >>> abs_srcdir, info->name) < 0 || >>> virAsprintf(&cfg, "%s/xmconfigdata/test-%s.cfg", >>> + abs_srcdir, info->name) < 0 || >>> + virAsprintf(&cfgout, "%s/xmconfigdata/test-%s.cfg.out", >>> abs_srcdir, info->name) < 0) >>> goto cleanup; >>> >>> if (info->mode == 0) >>> - result = testCompareParseXML(cfg, xml, info->version); >>> + result = testCompareParseXML(virFileExists(cfgout) ? cfgout : cfg, >>> + xml, info->version); >>> else >>> result = testCompareFormatXML(cfg, xml, info->version); >>> >>> cleanup: >>> VIR_FREE(xml); >>> VIR_FREE(cfg); >>> + VIR_FREE(cfgout); >>> >>> >> >> With the change suggested above, this hunk can be removed from >> xmconfigtest.c. 'make check' passes for me with these changes. >> > > 'make check' could pass, since it explicitly specify acpi|pae|apic=1 in > xm config, and explicitly include those features in xml. But our interested > case is not tested, what we are trying to test is: if user doesn't specify > acpi|pae|apic, xml should by default include those features. > Yep, understood. Unlike the existing tests, in this case we only want to test xm -> xml conversion. I think a better solution would be to introduce DO_TEST_PARSE and DO_TEST_FORMAT macros, calling those in DO_TEST and individually when only needing to test one conversion. Regards, Jim

2 1

[libvirt] [PATCH 0/2] fix xen HVM pae|apic|acpi features parser
by Chunyan Liu 23 Dec '14

23 Dec '14

According to xm.config manual, HVM pae|apic|acpi feature default is 1 (enabled). But in conversion from xm config to libvirt xml, if xm config doesn't contain pae|apic|acpi, it sets default value to 0, this causes some problems in HVM guest. Update parser codes to set HVM pae|apic|acpi default value to 1 to match xm config convension. Add tests data to test it. Chunyan Liu (2): xenconfig: set HVM pae/apic/acpi/ default to 1 Add tests to xmconfigtest src/xenconfig/xen_common.c | 6 +-- .../xmconfigdata/test-fullvirt-default-feature.cfg | 23 +++++++++++ .../test-fullvirt-default-feature.cfg.out | 26 ++++++++++++ .../xmconfigdata/test-fullvirt-default-feature.xml | 48 ++++++++++++++++++++++ tests/xmconfigtest.c | 9 +++- 5 files changed, 108 insertions(+), 4 deletions(-) create mode 100644 tests/xmconfigdata/test-fullvirt-default-feature.cfg create mode 100644 tests/xmconfigdata/test-fullvirt-default-feature.cfg.out create mode 100644 tests/xmconfigdata/test-fullvirt-default-feature.xml -- 1.8.4.5

4 5

[libvirt] [PATCH v2] test: fix nwfilter tests following changes in virfirewall.c
by Stefan Berger 22 Dec '14

22 Dec '14

Some of the nwfilter tests are now failing since --concurrent shows up in the ebtables command. To avoid this, implement a function preventing the probing for lock support in the eb/iptables tools and use it in the tests. Signed-off-by: Stefan Berger <stefanb(a)linux.vnet.ibm.com> --- src/libvirt_private.syms | 1 + src/util/virfirewall.c | 9 +++++++++ src/util/virfirewall.h | 2 ++ tests/nwfilterebiptablestest.c | 3 +++ tests/nwfilterxml2firewalltest.c | 2 ++ 5 files changed, 17 insertions(+) diff --git a/src/libvirt_private.syms b/src/libvirt_private.syms index 2647d36..22d9116 100644 --- a/src/libvirt_private.syms +++ b/src/libvirt_private.syms @@ -1362,6 +1362,7 @@ virFirewallRuleAddArgList; virFirewallRuleAddArgSet; virFirewallRuleGetArgCount; virFirewallSetBackend; +virFirewallSetLockOverride; virFirewallStartRollback; virFirewallStartTransaction; diff --git a/src/util/virfirewall.c b/src/util/virfirewall.c index 8496062..b536912 100644 --- a/src/util/virfirewall.c +++ b/src/util/virfirewall.c @@ -107,6 +107,13 @@ VIR_ONCE_GLOBAL_INIT(virFirewall) static bool iptablesUseLock; static bool ip6tablesUseLock; static bool ebtablesUseLock; +static bool lockOverride; /* true to avoid lock probes */ + +void +virFirewallSetLockOverride(bool avoid) +{ + lockOverride = avoid; +} static void virFirewallCheckUpdateLock(bool *lockflag, @@ -135,6 +142,8 @@ virFirewallCheckUpdateLocking(void) const char *ebtablesArgs[] = { EBTABLES_PATH, "--concurrent", "-L", NULL, }; + if (lockOverride) + return; virFirewallCheckUpdateLock(&iptablesUseLock, iptablesArgs); virFirewallCheckUpdateLock(&ip6tablesUseLock, diff --git a/src/util/virfirewall.h b/src/util/virfirewall.h index 1129219..dbf3975 100644 --- a/src/util/virfirewall.h +++ b/src/util/virfirewall.h @@ -106,4 +106,6 @@ void virFirewallStartRollback(virFirewallPtr firewall, int virFirewallApply(virFirewallPtr firewall); +void virFirewallSetLockOverride(bool avoid); + #endif /* __VIR_FIREWALL_H__ */ diff --git a/tests/nwfilterebiptablestest.c b/tests/nwfilterebiptablestest.c index e04bc21..e1330ef 100644 --- a/tests/nwfilterebiptablestest.c +++ b/tests/nwfilterebiptablestest.c @@ -24,6 +24,7 @@ #include "testutils.h" #include "nwfilter/nwfilter_ebiptables_driver.h" #include "virbuffer.h" +#include "virfirewall.h" #define __VIR_FIREWALL_PRIV_H_ALLOW__ #include "virfirewallpriv.h" @@ -522,6 +523,8 @@ mymain(void) { int ret = 0; + virFirewallSetLockOverride(true); + if (virFirewallSetBackend(VIR_FIREWALL_BACKEND_DIRECT) < 0) { ret = -1; goto cleanup; diff --git a/tests/nwfilterxml2firewalltest.c b/tests/nwfilterxml2firewalltest.c index 01527f4..167ad42 100644 --- a/tests/nwfilterxml2firewalltest.c +++ b/tests/nwfilterxml2firewalltest.c @@ -474,6 +474,8 @@ mymain(void) ret = -1; \ } while (0) + virFirewallSetLockOverride(true); + if (virFirewallSetBackend(VIR_FIREWALL_BACKEND_DIRECT) < 0) { ret = -1; goto cleanup; -- 1.9.3

3 7

[libvirt] Zanata xml config
by Carlos Munoz 22 Dec '14

22 Dec '14

Hi, The migration of the libvirt project translations to Zanata is now complete. Translators can start working now at https://fedora.zanata.org/project/view/libvirt To use Zanata, you need to 1. Register in https://fedora.zanata.org 2. If you are not the maintainer of your project yet, contact the admin to add you as maintainer. 3. Follow the instruction http://zanata.org/help/cli/cli-configuration/ 4. Place the attached zanata.xml in the root directory of repository 5. (Optional) integrate "zanata push" and "zanata pull" to your build scripts. If you have any questions , feel free to contact the Zanata dev team. Regards, -- Carlos A. Munoz Software Engineering Supervisor Engineering - Internationalization Red Hat

1 0

[libvirt] [PATCH] Update my email address in AUTHORS.in
by Claudio Bley 21 Dec '14

21 Dec '14

--- pushed under the trivial rule. AUTHORS.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AUTHORS.in b/AUTHORS.in index 3c783f7..329924f 100644 --- a/AUTHORS.in +++ b/AUTHORS.in @@ -10,7 +10,7 @@ The primary maintainers and people with commit access rights: Alex Jia <ajia(a)redhat.com> Cédric Bosdonnat <cbosdonnat(a)suse.com> Christophe Fergeau <cfergeau(a)redhat.com> -Claudio Bley <cbley(a)av-test.de> +Claudio Bley <claudio.bley(a)gmail.com> Cole Robinson <crobinso(a)redhat.com> Daniel Berrange <berrange(a)redhat.com> Daniel Veillard <veillard(a)redhat.com> -- 2.2.1

2 2

[libvirt] [PATCH] maint: update .mailmap
by Claudio Bley 21 Dec '14

21 Dec '14

Add an email alias after updating my email address in commit 738a2ae. --- Pushed under the trivial rule. .mailmap | 1 + 1 file changed, 1 insertion(+) diff --git a/.mailmap b/.mailmap index 826c63c..84a6dbe 100644 --- a/.mailmap +++ b/.mailmap @@ -5,6 +5,7 @@ <bozzolan(a)gmail.com> <redshift(a)gmx.com> <charles_duffy(a)messageone.com> <charles(a)dyfis.net> +<claudio.bley(a)gmail.com> <cbley(a)av-test.de> <dfj(a)redhat.com> <dfj(a)dfj.bne.redhat.com> <eblake(a)redhat.com> <ebb9(a)byu.net> <gdolley(a)arpnetworks.com> <gdolley(a)ucla.edu> -- 2.2.1

1 0

[libvirt] [PATCH] Fix typo s/interpetation/interpretation/
by Claudio Bley 20 Dec '14

20 Dec '14

Fix the typo in struct virSecurityModel's comment for its doi field. --- Pushed under the trivial rule. include/libvirt/libvirt-host.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/libvirt/libvirt-host.h b/include/libvirt/libvirt-host.h index 0704672..f760a55 100644 --- a/include/libvirt/libvirt-host.h +++ b/include/libvirt/libvirt-host.h @@ -148,7 +148,7 @@ typedef struct _virSecurityModel virSecurityModel; struct _virSecurityModel { char model[VIR_SECURITY_MODEL_BUFLEN]; /* security model string */ - char doi[VIR_SECURITY_DOI_BUFLEN]; /* domain of interpetation */ + char doi[VIR_SECURITY_DOI_BUFLEN]; /* domain of interpretation */ }; /** -- 2.2.1

1 0

[libvirt] [PATCH 2/2] lxc: Don't crash on NULL ifname_guest_actual
by Guido Günther 20 Dec '14

20 Dec '14

Reported and patch provided by Bastian Blank at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=769600 --- src/lxc/lxc_container.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lxc/lxc_container.c b/src/lxc/lxc_container.c index 1b9e2f2..f986f71 100644 --- a/src/lxc/lxc_container.c +++ b/src/lxc/lxc_container.c @@ -472,7 +472,7 @@ lxcContainerGetNetDef(virDomainDefPtr vmDef, const char *devName) for (i = 0; i < vmDef->nnets; i++) { netDef = vmDef->nets[i]; - if (STREQ(netDef->ifname_guest_actual, devName)) + if (netDef->ifname_guest_actual && STREQ(netDef->ifname_guest_actual, devName)) return netDef; } -- 2.1.3

2 2