February 2014 - Devel - Libvirt List Archives

by Michal Privoznik

Tier four. Michal Privoznik (3): network_conf: Expose virNetworkDefFormatInternal network: Introduce network hooks network: Taint networks that are using hook script docs/hooks.html.in | 98 ++++++++++++++++++++++++++++++++------ src/conf/network_conf.c | 64 ++++++++++++++++++++++--- src/conf/network_conf.h | 20 ++++++++ src/libvirt_private.syms | 4 ++ src/lxc/lxc_driver.c | 4 +- src/lxc/lxc_process.c | 6 +-- src/network/bridge_driver.c | 111 ++++++++++++++++++++++++++++++++++++++++++-- src/network/bridge_driver.h | 19 ++++---- src/qemu/qemu_command.c | 2 +- src/qemu/qemu_hotplug.c | 14 +++--- src/qemu/qemu_process.c | 4 +- src/util/virhook.c | 13 +++++- src/util/virhook.h | 11 +++++ 13 files changed, 321 insertions(+), 49 deletions(-) -- 1.8.5.3

10 years, 9 months

2
10
0 / 0

[libvirt] [PATCH v9] bhyve: add a basic driver

by Roman Bogorodskiy

Changes from v8: - Implement domainUndefine and two more functions it depends on: domainIsActive and domainIsPersistent Changes from v7: - Squashed in ACL support - Check for disk and bus type for bhyve and disk type for bhyveload - Handle case when URI == NULL in ConnectOpen - Call bhyveload only after we've built bhyve command to avoid unneeded load/reload for wrong domain configuration - Cleanup unload calls on errors - Minor style fixes Changes from v6: - Fix typo: s/LIBIVRT_DRIVER_RESULT_BHYVE/LIBVIRT_DRIVER_RESULT_BHYVE/ - Report domain state in 'dominfo' - Add a patch which implements ACL support Now both 'make check' and 'make syntax-check' pass. Changes from v5: - Obtain version using uname(3) - Cleanup driver global objects in StateCleanup instead of ConnectClose Changes from v4: - Set acpi and apic flags based on domain definition - Add more detailed description about -H and -P flags of bhyve to justify theirs usage Roman Bogorodskiy (1): bhyve: add a basic driver configure.ac | 7 + daemon/libvirtd.c | 9 + include/libvirt/virterror.h | 1 + m4/virt-driver-bhyve.m4 | 57 ++++ po/POTFILES.in | 3 + src/Makefile.am | 31 ++ src/bhyve/bhyve_command.c | 314 ++++++++++++++++++++ src/bhyve/bhyve_command.h | 41 +++ src/bhyve/bhyve_driver.c | 707 ++++++++++++++++++++++++++++++++++++++++++++ src/bhyve/bhyve_driver.h | 28 ++ src/bhyve/bhyve_process.c | 227 ++++++++++++++ src/bhyve/bhyve_process.h | 36 +++ src/bhyve/bhyve_utils.h | 49 +++ src/conf/domain_conf.c | 3 +- src/conf/domain_conf.h | 1 + src/driver.h | 1 + src/libvirt.c | 3 + src/util/virerror.c | 1 + 18 files changed, 1518 insertions(+), 1 deletion(-) create mode 100644 m4/virt-driver-bhyve.m4 create mode 100644 src/bhyve/bhyve_command.c create mode 100644 src/bhyve/bhyve_command.h create mode 100644 src/bhyve/bhyve_driver.c create mode 100644 src/bhyve/bhyve_driver.h create mode 100644 src/bhyve/bhyve_process.c create mode 100644 src/bhyve/bhyve_process.h create mode 100644 src/bhyve/bhyve_utils.h -- 1.8.4.3

10 years, 9 months

3
6
0 / 0

[libvirt] [PATCH REPOST] Introduce new OOM testing support

by Daniel P. Berrange

The previous OOM testing support would re-run the entire "main" method each iteration, failing a different malloc each time. When a test suite has 'n' allocations, the number of repeats requires is (n * (n + 1) ) / 2. This gets very large, very quickly. This new OOM testing support instead integrates at the virtTestRun level, so each individual test case gets repeated, instead of the entire test suite. This means the values of 'n' are orders of magnitude smaller. The simple usage is $ VIR_TEST_OOM=1 ./qemuxml2argvtest ... 29) QEMU XML-2-ARGV clock-utc ... OK Test OOM for nalloc=36 .................................... OK 30) QEMU XML-2-ARGV clock-localtime ... OK Test OOM for nalloc=36 .................................... OK 31) QEMU XML-2-ARGV clock-france ... OK Test OOM for nalloc=38 ...................................... OK ... the second lines reports how many mallocs have to be failed, and thus how many repeats of the test will be run. If it crashes, then running under valgrind will often show the problem $ VIR_TEST_OOM=1 ../run valgrind ./qemuxml2argvtest When debugging problems it is also helpful to select an individual test case $ VIR_TEST_RANGE=30 VIR_TEST_OOM=1 ../run valgrind ./qemuxml2argvtest When things get really tricky, it is possible to request that just specific allocs are failed. eg to fail allocs 5 -> 12, use $ VIR_TEST_RANGE=30 VIR_TEST_OOM=1:5-12 ../run valgrind ./qemuxml2argvtest In the worse case, you might want to know the stack trace of the alloc which was failed then VIR_TEST_OOM_TRACE can be set. If it is set to 1 then it will only print if it thinks a mistake happened. This is often not reliable, so setting it to 2 will make it print the stack trace for every alloc that is failed. $ VIR_TEST_OOM_TRACE=2 VIR_TEST_RANGE=30 VIR_TEST_OOM=1:5-5 ../run valgrind ./qemuxml2argvtest 30) QEMU XML-2-ARGV clock-localtime ... OK Test OOM for nalloc=36 !virAllocN /home/berrange/src/virt/libvirt/src/util/viralloc.c:180 virHashCreateFull /home/berrange/src/virt/libvirt/src/util/virhash.c:144 virDomainDefParseXML /home/berrange/src/virt/libvirt/src/conf/domain_conf.c:11745 virDomainDefParseNode /home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12646 virDomainDefParse /home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12590 testCompareXMLToArgvFiles /home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:106 virtTestRun /home/berrange/src/virt/libvirt/tests/testutils.c:250 mymain /home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:418 (discriminator 2) virtTestMain /home/berrange/src/virt/libvirt/tests/testutils.c:750 ?? ??:0 _start ??:? FAILED Signed-off-by: Daniel P. Berrange <berrange(a)redhat.com> --- cfg.mk | 2 +- docs/internals/oomtesting.html.in | 213 ++++++++++++++++++++++++++++++++++++++ docs/sitemap.html.in | 4 + tests/qemuargv2xmltest.c | 14 +-- tests/qemuxml2argvtest.c | 12 ++- tests/qemuxmlnstest.c | 18 ++-- tests/testutils.c | 195 +++++++++++++++++++++++++++++++++- tests/testutils.h | 2 + 8 files changed, 438 insertions(+), 22 deletions(-) create mode 100644 docs/internals/oomtesting.html.in diff --git a/cfg.mk b/cfg.mk index 207dfeb..bbc966b 100644 --- a/cfg.mk +++ b/cfg.mk @@ -948,7 +948,7 @@ exclude_file_name_regexp--sc_libvirt_unmarked_diagnostics = \ exclude_file_name_regexp--sc_po_check = ^(docs/|src/rpc/gendispatch\.pl$$) exclude_file_name_regexp--sc_prohibit_VIR_ERR_NO_MEMORY = \ - ^(include/libvirt/virterror\.h|daemon/dispatch\.c|src/util/virerror\.c)$$ + ^(include/libvirt/virterror\.h|daemon/dispatch\.c|src/util/virerror\.c|docs/internals/oomtesting\.html\.in)$$ exclude_file_name_regexp--sc_prohibit_access_xok = ^src/util/virutil\.c$$ diff --git a/docs/internals/oomtesting.html.in b/docs/internals/oomtesting.html.in new file mode 100644 index 0000000..c5edacf --- /dev/null +++ b/docs/internals/oomtesting.html.in @@ -0,0 +1,213 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> +<html xmlns="http://www.w3.org/1999/xhtml"> + <body> + <h1>Out of memory testing</h1> + + <ul id="toc"></ul> + + + <p> + This page describes how to use the test suite todo out of memory + testing. + </p> + + <h2>Building with OOM testing</h2> + + <p> + Since OOM testing requires hooking into the malloc APIs, it is + not enabled by default. The flag <code>--enable-test-oom</code> + must be given to <code>configure</code>. When this is done the + libvirt allocation APIs will have some hooks enabled. + </p> + + <pre> +$ ./configure --enable-test-oom +</pre> + + + <h2><a name="basicoom">Basic OOM testing support</a></h2> + + <p> + The first step in validating OOM usage is to run a test suite + with full OOM testing enabled. This is done by setting the + <code>VIR_TEST_OOM=1</code> environment variable. The way this + works is that it runs the test once normally to "prime" any + static memory allocations. Then it runs it once more counting + the total number of memory allocations. Then it runs it in a + loop failing a different memory allocation each time. For every + memory allocation failure triggered, it expects the test case + to return an error. OOM testing is quite slow requiring each + test case to be executed O(n) times, where 'n' is the total + number of memory allocations. This results in a total number + of memory allocations of '(n * (n + 1) ) / 2' + </p> + + <pre> +$ VIR_TEST_OOM=1 ./qemuxml2argvtest + 1) QEMU XML-2-ARGV minimal ... OK + Test OOM for nalloc=42 .......................................... OK + 2) QEMU XML-2-ARGV minimal-s390 ... OK + Test OOM for nalloc=28 ............................ OK + 3) QEMU XML-2-ARGV machine-aliases1 ... OK + Test OOM for nalloc=38 ...................................... OK + 4) QEMU XML-2-ARGV machine-aliases2 ... OK + Test OOM for nalloc=38 ...................................... OK + 5) QEMU XML-2-ARGV machine-core-on ... OK + Test OOM for nalloc=37 ..................................... OK +...snip... +</pre> + + <p> + In this output, the first line shows the normal execution and + the test number, and the second line shows the total number + of memory allocations from that test case. + </p> + + <h3><a name="valgrind">Tracking failures with valgrind</a></h3> + + <p> + The test suite should obviously *not* crash during OOM testing. + If it does crash, then to assist in tracking down the problem + it is worth using valgrind and only running a single test case. + For example, supposing test case 5 crashed. Then re-run the + test with + </p> + + <pre> +$ VIR_TEST_OOM=1 VIR_TEST_RANGE=5 ../run valgrind ./qemuxml2argvtest +...snip... + 5) QEMU XML-2-ARGV machine-core-on ... OK + Test OOM for nalloc=37 ..................................... OK +...snip... + </pre> + + <p> + Valgrind should report the cause of the crash - for example a + double free or use of uninitialized memory or NULL pointer + access. + </p> + + <h3><a name="stacktraces">Tracking failures with stack traces</a></h3> + + <p> + With some really difficult bugs valgrind is not sufficient to + identify the cause. In this case, it is useful to identify the + precise allocation which was failed, to allow the code path + to the error to be traced. The <code>VIR_TEST_OOM</code> + env variable can be given a range of memory allocations to + test. So if a test case has 150 allocations, it can be told + to only test allocation numbers 7-10. The <code>VIR_TEST_OOM_TRACE</code> + variable can be used to print out stack traces. + </p> + + <pre> +$ VIR_TEST_OOM_TRACE=2 VIR_TEST_OOM=1:7-10 VIR_TEST_RANGE=5 \ + ../run valgrind ./qemuxml2argvtest + 5) QEMU XML-2-ARGV machine-core-on ... OK + Test OOM for nalloc=37 !virAllocN +/home/berrange/src/virt/libvirt/src/util/viralloc.c:180 +virDomainDefParseXML +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:11786 (discriminator 1) +virDomainDefParseNode +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12677 +virDomainDefParse +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12621 +testCompareXMLToArgvFiles +/home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:107 +virtTestRun +/home/berrange/src/virt/libvirt/tests/testutils.c:266 +mymain +/home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:388 (discriminator 2) +virtTestMain +/home/berrange/src/virt/libvirt/tests/testutils.c:791 +__libc_start_main +??:? +_start +??:? +!virAlloc +/home/berrange/src/virt/libvirt/src/util/viralloc.c:133 +virDomainDiskDefParseXML +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:4790 +virDomainDefParseXML +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:11797 +virDomainDefParseNode +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12677 +virDomainDefParse +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12621 +testCompareXMLToArgvFiles +/home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:107 +virtTestRun +/home/berrange/src/virt/libvirt/tests/testutils.c:266 +mymain +/home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:388 (discriminator 2) +virtTestMain +/home/berrange/src/virt/libvirt/tests/testutils.c:791 +__libc_start_main +??:? +_start +??:? +!virAllocN +/home/berrange/src/virt/libvirt/src/util/viralloc.c:180 +virXPathNodeSet +/home/berrange/src/virt/libvirt/src/util/virxml.c:609 +virDomainDefParseXML +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:11805 +virDomainDefParseNode +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12677 +virDomainDefParse +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12621 +testCompareXMLToArgvFiles +/home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:107 +virtTestRun +/home/berrange/src/virt/libvirt/tests/testutils.c:266 +mymain +/home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:388 (discriminator 2) +virtTestMain +/home/berrange/src/virt/libvirt/tests/testutils.c:791 +__libc_start_main +??:? +_start +??:? +!virAllocN +/home/berrange/src/virt/libvirt/src/util/viralloc.c:180 +virDomainDefParseXML +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:11808 (discriminator 1) +virDomainDefParseNode +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12677 +virDomainDefParse +/home/berrange/src/virt/libvirt/src/conf/domain_conf.c:12621 +testCompareXMLToArgvFiles +/home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:107 +virtTestRun +/home/berrange/src/virt/libvirt/tests/testutils.c:266 +mymain +/home/berrange/src/virt/libvirt/tests/qemuxml2argvtest.c:388 (discriminator 2) +virtTestMain +/home/berrange/src/virt/libvirt/tests/testutils.c:791 +__libc_start_main +??:? +_start +??:? + </pre> + + <h3><a name="noncrash">Non-crash related problems</a></h3> + + <p> + Not all memory allocation bugs result in code crashing. Sometimes + the code will be silently ignoring the allocation failure, resulting + in incorrect data being produced. For example the XML parser may + mistakenly treat an allocation failure as indicating that an XML + attribute was not set in the input document. It is hard to identify + these problems from the test suite automatically. For this, the + test suites should be run with <code>VIR_TEST_DEBUG=1</code> set + and then stderr analysed for any unexpected data. For example, + the XML conversion may show an embedded "(null)" literal, or the + test suite might complain about missing elements / attributes + in the actual vs expected data. These are all signs of bugs in + OOM handling. In the future the OOM tests will be enhanced to + validate that an error VIR_ERR_NO_MEMORY is returned for each + allocation failed, rather than some other error. + </p> + </body> +</html> diff --git a/docs/sitemap.html.in b/docs/sitemap.html.in index 08c2fbd..7d0610b 100644 --- a/docs/sitemap.html.in +++ b/docs/sitemap.html.in @@ -342,6 +342,10 @@ <a href="internals/locking.html">Lock managers</a> <span>Use lock managers to protect disk content</span> </li> + <li> + <a href="internals/oomtesting.html">Out of memory testing</a> + <span>Simulating OOM conditions in the test suite</span> + </li> </ul> </li> <li> diff --git a/tests/qemuargv2xmltest.c b/tests/qemuargv2xmltest.c index 1b13fcc..e75c85d 100644 --- a/tests/qemuargv2xmltest.c +++ b/tests/qemuargv2xmltest.c @@ -42,7 +42,6 @@ static int testCompareXMLToArgvFiles(const char *xml, char *cmd = NULL; int ret = -1; virDomainDefPtr vmdef = NULL; - char *log; if (virtTestLoadFile(cmdfile, &cmd) < 0) goto fail; @@ -53,13 +52,16 @@ static int testCompareXMLToArgvFiles(const char *xml, cmd, NULL, NULL, NULL))) goto fail; - if ((log = virtTestLogContentAndReset()) == NULL) - goto fail; - if ((*log != '\0') != expect_warning) { + if (!virtTestOOMActive()) { + char *log; + if ((log = virtTestLogContentAndReset()) == NULL) + goto fail; + if ((*log != '\0') != expect_warning) { + VIR_FREE(log); + goto fail; + } VIR_FREE(log); - goto fail; } - VIR_FREE(log); if (!virDomainDefCheckABIStability(vmdef, vmdef)) { fprintf(stderr, "ABI stability check failed on %s", xml); diff --git a/tests/qemuxml2argvtest.c b/tests/qemuxml2argvtest.c index 7a5b50d..26d63d3 100644 --- a/tests/qemuxml2argvtest.c +++ b/tests/qemuxml2argvtest.c @@ -284,7 +284,8 @@ static int testCompareXMLToArgvFiles(const char *xml, if (!(vmdef = virDomainDefParseFile(xml, driver.caps, driver.xmlopt, QEMU_EXPECTED_VIRT_TYPES, VIR_DOMAIN_XML_INACTIVE))) { - if (flags & FLAG_EXPECT_PARSE_ERROR) + if (!virtTestOOMActive() && + (flags & FLAG_EXPECT_PARSE_ERROR)) goto ok; goto out; } @@ -357,7 +358,8 @@ static int testCompareXMLToArgvFiles(const char *xml, migrateFrom, migrateFd, NULL, VIR_NETDEV_VPORT_PROFILE_OP_NO_OP, &testCallbacks))) { - if (flags & FLAG_EXPECT_FAILURE) { + if (!virtTestOOMActive() && + (flags & FLAG_EXPECT_FAILURE)) { ret = 0; if (virTestGetDebug() > 1) fprintf(stderr, "Got expected error: %s\n", @@ -371,7 +373,8 @@ static int testCompareXMLToArgvFiles(const char *xml, goto out; } - if (!!virGetLastError() != !!(flags & FLAG_EXPECT_ERROR)) { + if (!virtTestOOMActive() && + (!!virGetLastError() != !!(flags & FLAG_EXPECT_ERROR))) { if (virTestGetDebug() && (log = virtTestLogContentAndReset())) fprintf(stderr, "\n%s", log); goto out; @@ -392,7 +395,8 @@ static int testCompareXMLToArgvFiles(const char *xml, } ok: - if (flags & FLAG_EXPECT_ERROR) { + if (!virtTestOOMActive() && + (flags & FLAG_EXPECT_ERROR)) { /* need to suppress the errors */ virResetLastError(); } diff --git a/tests/qemuxmlnstest.c b/tests/qemuxmlnstest.c index 9426bf7..20a5ccd 100644 --- a/tests/qemuxmlnstest.c +++ b/tests/qemuxmlnstest.c @@ -122,15 +122,17 @@ static int testCompareXMLToArgvFiles(const char *xml, &testCallbacks))) goto fail; - if (!!virGetLastError() != expectError) { - if (virTestGetDebug() && (log = virtTestLogContentAndReset())) - fprintf(stderr, "\n%s", log); - goto fail; - } + if (!virtTestOOMActive()) { + if (!!virGetLastError() != expectError) { + if (virTestGetDebug() && (log = virtTestLogContentAndReset())) + fprintf(stderr, "\n%s", log); + goto fail; + } - if (expectError) { - /* need to suppress the errors */ - virResetLastError(); + if (expectError) { + /* need to suppress the errors */ + virResetLastError(); + } } if (!(actualargv = virCommandToString(cmd))) diff --git a/tests/testutils.c b/tests/testutils.c index 32fe374..52d07eb 100644 --- a/tests/testutils.c +++ b/tests/testutils.c @@ -47,6 +47,13 @@ #include "virprocess.h" #include "virstring.h" +#ifdef TEST_OOM +# ifdef TEST_OOM_TRACE +# include <dlfcn.h> +# include <execinfo.h> +# endif +#endif + #ifdef HAVE_PATHS_H # include <paths.h> #endif @@ -64,12 +71,37 @@ static unsigned int testDebug = -1; static unsigned int testVerbose = -1; static unsigned int testExpensive = -1; +#ifdef TEST_OOM +static unsigned int testOOM = 0; +static unsigned int testOOMStart = -1; +static unsigned int testOOMEnd = -1; +static unsigned int testOOMTrace = 0; +# ifdef TEST_OOM_TRACE +void *testAllocStack[30]; +int ntestAllocStack; +# endif +#endif +static bool testOOMActive = false; + static size_t testCounter = 0; static size_t testStart = 0; static size_t testEnd = 0; char *progname; +bool virtTestOOMActive(void) +{ + return testOOMActive; +} + +#ifdef TEST_OOM_TRACE +static void virTestAllocHook(int nalloc ATTRIBUTE_UNUSED, + void *opaque ATTRIBUTE_UNUSED) +{ + ntestAllocStack = backtrace(testAllocStack, ARRAY_CARDINALITY(testAllocStack)); +} +#endif + void virtTestResult(const char *name, int ret, const char *msg, ...) { va_list vargs; @@ -108,6 +140,35 @@ void virtTestResult(const char *name, int ret, const char *msg, ...) va_end(vargs); } +#ifdef TEST_OOM_TRACE +static void +virTestShowTrace(void) +{ + size_t j; + for (j = 2; j < ntestAllocStack; j++) { + Dl_info info; + char *cmd; + + dladdr(testAllocStack[j], &info); + if (info.dli_fname && + strstr(info.dli_fname, ".so")) { + if (virAsprintf(&cmd, "addr2line -f -e %s %p", + info.dli_fname, + ((void*)((unsigned long long)testAllocStack[j] + - (unsigned long long)info.dli_fbase))) < 0) + continue; + } else { + if (virAsprintf(&cmd, "addr2line -f -e %s %p", + (char*)(info.dli_fname ? info.dli_fname : "<unknown>"), + testAllocStack[j]) < 0) + continue; + } + ignore_value(system(cmd)); + VIR_FREE(cmd); + } +} +#endif + /* * Runs test * @@ -154,7 +215,7 @@ virtTestRun(const char *title, !((testCounter-1) % 40)) { fprintf(stderr, " %-3zu\n", (testCounter-1)); fprintf(stderr, " "); - } + } if (ret == 0) fprintf(stderr, "."); else if (ret == EXIT_AM_SKIP) @@ -163,6 +224,77 @@ virtTestRun(const char *title, fprintf(stderr, "!"); } +#ifdef TEST_OOM + if (testOOM && ret != EXIT_AM_SKIP) { + int nalloc; + int oomret; + int start, end; + size_t i; + virResetLastError(); + virAllocTestInit(); +# ifdef TEST_OOM_TRACE + virAllocTestHook(virTestAllocHook, NULL); +# endif + oomret = body(data); + nalloc = virAllocTestCount(); + fprintf(stderr, " Test OOM for nalloc=%d ", nalloc); + if (testOOMStart == -1 || + testOOMEnd == -1) { + start = 0; + end = nalloc; + } else { + start = testOOMStart; + end = testOOMEnd + 1; + } + testOOMActive = true; + for (i = start; i < end; i++) { + bool missingFail = false; +# ifdef TEST_OOM_TRACE + memset(testAllocStack, 0, ARRAY_CARDINALITY(testAllocStack)); + ntestAllocStack = 0; +# endif + virAllocTestOOM(i + 1, 1); + oomret = body(data); + + /* fprintf() disabled because XML parsing APIs don't allow + * distinguish between element / attribute not present + * in the XML (which is non-fatal), vs OOM / malformed + * which should be fatal. Thus error reporting for + * optionally present XML is mostly broken. + */ + if (oomret == 0) { + missingFail = true; +# if 0 + fprintf(stderr, " alloc %zu failed but no err status\n", i + 1); +# endif + } else { + virErrorPtr lerr = virGetLastError(); + if (!lerr) { +# if 0 + fprintf(stderr, " alloc %zu failed but no error report\n", i + 1); +# endif + missingFail = true; + } + } + if ((missingFail && testOOMTrace) || (testOOMTrace > 1)) { + fprintf(stderr, "%s", "!"); +# ifdef TEST_OOM_TRACE + virTestShowTrace(); +# endif + ret = -1; + } else { + fprintf(stderr, "%s", "."); + } + } + testOOMActive = false; + if (ret == 0) + fprintf(stderr, " OK\n"); + else + fprintf(stderr, " FAILED\n"); + virAllocTestInit(); + } +#endif /* TEST_OOM */ + return ret; } @@ -191,7 +323,6 @@ virtTestLoadFile(const char *file, char **buf) tmplen = buflen = st.st_size + 1; if (VIR_ALLOC_N(*buf, buflen) < 0) { - fprintf(stderr, "%s: larger than available memory (> %d)\n", file, buflen); VIR_FORCE_FCLOSE(fp); return -1; } @@ -467,7 +598,8 @@ virtTestLogOutput(virLogSource source ATTRIBUTE_UNUSED, { struct virtTestLogData *log = data; virCheckFlags(VIR_LOG_STACK_TRACE,); - virBufferAsprintf(&log->buf, "%s: %s", timestamp, str); + if (!testOOMActive) + virBufferAsprintf(&log->buf, "%s: %s", timestamp, str); } static void @@ -535,6 +667,9 @@ int virtTestMain(int argc, { int ret; char *testRange = NULL; +#ifdef TEST_OOM + char *oomstr; +#endif if (!virFileExists(abs_srcdir)) return EXIT_AM_HARDFAIL; @@ -590,6 +725,60 @@ int virtTestMain(int argc, } } +#ifdef TEST_OOM + if ((oomstr = getenv("VIR_TEST_OOM")) != NULL) { + char *next; + if (testDebug == -1) + testDebug = 1; + testOOM = 1; + if (oomstr[0] != '\0' && + oomstr[1] == ':') { + if (virStrToLong_ui(oomstr + 2, &next, 10, &testOOMStart) < 0) { + fprintf(stderr, "Cannot parse range %s\n", oomstr); + return EXIT_FAILURE; + } + if (*next == '\0') { + testOOMEnd = testOOMStart; + } else { + if (*next != '-') { + fprintf(stderr, "Cannot parse range %s\n", oomstr); + return EXIT_FAILURE; + } + if (virStrToLong_ui(next+1, NULL, 10, &testOOMEnd) < 0) { + fprintf(stderr, "Cannot parse range %s\n", oomstr); + return EXIT_FAILURE; + } + } + } else { + testOOMStart = -1; + testOOMEnd = -1; + } + } + +# ifdef TEST_OOM_TRACE + if ((oomstr = getenv("VIR_TEST_OOM_TRACE")) != NULL) { + if (virStrToLong_ui(oomstr, NULL, 10, &testOOMTrace) < 0) { + fprintf(stderr, "Cannot parse oom trace %s\n", oomstr); + return EXIT_FAILURE; + } + } +# else + if (getenv("VIR_TEST_OOM_TRACE")) { + fprintf(stderr, "%s", "OOM test tracing not enabled in this build\n"); + return EXIT_FAILURE; + } +# endif +#else /* TEST_OOM */ + if (getenv("VIR_TEST_OOM")) { + fprintf(stderr, "%s", "OOM testing not enabled in this build\n"); + return EXIT_FAILURE; + } + if (getenv("VIR_TEST_OOM_TRACE")) { + fprintf(stderr, "%s", "OOM test tracing not enabled in this build\n"); + return EXIT_FAILURE; + } +#endif /* TEST_OOM */ + ret = (func)(); virResetLastError(); diff --git a/tests/testutils.h b/tests/testutils.h index 674d3df..29eb9d9 100644 --- a/tests/testutils.h +++ b/tests/testutils.h @@ -44,6 +44,8 @@ extern char *progname; # error Fix Makefile.am # endif +bool virtTestOOMActive(void); + void virtTestResult(const char *name, int ret, const char *msg, ...) ATTRIBUTE_FMT_PRINTF(3,4); int virtTestRun(const char *title, -- 1.8.5.3

10 years, 9 months

2
1
0 / 0

[libvirt] Schedule for upcoming release of 1.2.2

by Daniel Veillard

Since February ends up on the 28, I think it makes sense to enter the freeze for 1.2.2 next week-end if we want to have it by the end of the month, so please try to get features pushed this week, well if you want them in the next release, thanks, Daniel -- Daniel Veillard | Open Source and Standards, Red Hat veillard(a)redhat.com | libxml Gnome XML XSLT toolkit http://xmlsoft.org/ http://veillard.com/ | virtualization library http://libvirt.org/

10 years, 9 months

4
5
0 / 0

[libvirt] [PATCH 0/8 v3] Avoid unsafe usage of /proc/$PID/root in LXC driver

by Daniel P. Berrange

An update of https://www.redhat.com/archives/libvir-list/2014-February/msg00401.html Eric had previously suggested that we could have our fork helper pass by a pre-opened file descriptor and then label that, so we can move the labelling code out of the subprocess. While I believe that is technically possible, it plays havoc with the security driver APIs. We'd basically have to create an entire new set of APIs that take an FD instead of path / object in virSecurityManager. I don't think that's a net win in terms of complexity added. Considering what we're running in the fork helper - We're Linux only so can rely on sane glibc malloc behaviour - We're already using security manager calls in the QEMU driver fork helper - We've protected the logging + error APIs in virFork So I think we'll get away with bending the POSIX rules on async safe functions in this instance. Changes in this version - Avoid use of virCgroupAllowDeviceByPath where we already have the major/minor number - Avoid overwriting errors set by virCgroupAllow* - Use return value check of < 0 instead of != 0 - Other misc typos Eric mentioned Daniel P. Berrange (7): Add virFileMakeParentPath helper function Add helper for running code in separate namespaces Avoid unsafe use of /proc/$PID/root in LXC disk hotplug Avoid unsafe use of /proc/$PID/root in LXC USB hotplug Avoid unsafe use of /proc/$PID/root in LXC block hostdev hotplug Avoid unsafe use of /proc/$PID/root in LXC chardev hostdev hotplug Avoid unsafe use of /proc/$PID/root in LXC hotunplug code Eric Blake (1): Avoid unsafe use of /proc/$PID/root in LXC shutdown/reboot code src/libvirt_private.syms | 2 + src/lxc/lxc_driver.c | 507 ++++++++++++++++++++++++----------------------- src/util/virfile.c | 27 +++ src/util/virfile.h | 1 + src/util/virinitctl.c | 26 +-- src/util/virinitctl.h | 5 +- src/util/virprocess.c | 107 ++++++++++ src/util/virprocess.h | 11 + 8 files changed, 418 insertions(+), 268 deletions(-) -- 1.8.5.3

10 years, 9 months

2
16
0 / 0

[libvirt] libvirtd ssl configuration

by Гусев Павел

Hi! I found little semantics bug: [13:53:40] root@dedicated-04:~ # LC_ALL=C libvirtd -h libvirtd: invalid option -- 'h' Usage: libvirtd [options] Options: -v | --verbose Verbose messages. -d | --daemon Run as a daemon & write PID file. -l | --listen Listen for TCP/IP connections. -t | --timeout <secs> Exit after timeout period. -f | --config <file> Configuration file. | --version Display version information. -p | --pid-file <file> Change name of PID file. libvirt management daemon: Default paths: Configuration file (unless overridden by -f): /etc/libvirt/libvirtd.conf Sockets: /var/run/libvirt/libvirt-sock /var/run/libvirt/libvirt-sock-ro TLS: CA certificate: /etc/pki/CA/caert.pem Server certificate: /etc/pki/libvirt/servercert.pem Server private key: /etc/pki/libvirt/private/serverkey.pem PID file (unless overridden by -p): /var/run/libvirtd.pid I think that caert.pem should be cacert.pem =) Tnx. С уважением, Гусев Павел Руководитель отдела системного администрирования QSOFT | Ведущий web-интегратор офис 7(495) 771-7363 #110 | моб. 7(926) 850-1108 pgusev(a)qsoft.ru Москва, Авангардная улица, 3 | qsoft.ru San Francisco, 222 Columbus Ave | qsoftus.com

10 years, 9 months

2
2
0 / 0

[libvirt] [PATCH 0/2] Taint domains altered by hook scripts

by Michal Privoznik

Currently, we only allow XML changing in qemu driver and nowhere else. Michal Privoznik (2): virDomainTaintFlags: Introduce VIR_DOMAIN_TAINT_HOOK qemu: Implement VIR_DOMAIN_TAINT_HOOK src/conf/domain_conf.c | 3 ++- src/conf/domain_conf.h | 1 + src/qemu/qemu_domain.c | 4 ++++ src/qemu/qemu_domain.h | 2 ++ src/qemu/qemu_migration.c | 10 ++++++++++ 5 files changed, 19 insertions(+), 1 deletion(-) -- 1.8.5.2

10 years, 9 months

2
6
0 / 0

[libvirt] Exposing and calculating CPU APIC IDs (was Re: [Qemu-devel] [RFC 1/3] target-i386: moving registers of vmstate from cpu_exec_init() to x86_cpu_realizefn())

by Eduardo Habkost

On Wed, Jan 15, 2014 at 03:37:04PM +0100, Igor Mammedov wrote: > On Wed, 15 Jan 2014 20:24:01 +0800 > Chen Fan <chen.fan.fnst(a)cn.fujitsu.com> wrote: > > On Tue, 2014-01-14 at 11:40 +0100, Igor Mammedov wrote: > > > On Tue, 14 Jan 2014 17:27:20 +0800 > > > Chen Fan <chen.fan.fnst(a)cn.fujitsu.com> wrote: > > > > > > > the intend of this patch is to register cpu vmstates with apic id instead of cpu > > > > index, due to the property setting of apic_id is behind the cpu initialization. so > > > > we move the registers of cpu vmstate from cpu_exec_init() to x86_cpu_realizefn() to > > > > let the set apicid as the parameter. > > > > > > > > Signed-off-by: Chen Fan <chen.fan.fnst(a)cn.fujitsu.com> > > > > --- > > > > exec.c | 5 +++++ > > > > target-i386/cpu.c | 9 +++++++++ > > > > 2 files changed, 14 insertions(+) > > > > > > > > diff --git a/exec.c b/exec.c > > > > index 7e49e8e..9be5855 100644 > > > > --- a/exec.c > > > > +++ b/exec.c > > > > @@ -438,7 +438,9 @@ CPUState *qemu_get_cpu(int index) > > > > void cpu_exec_init(CPUArchState *env) > > > > { > > > > CPUState *cpu = ENV_GET_CPU(env); > > > > +#if !defined(TARGET_I386) > > > > CPUClass *cc = CPU_GET_CLASS(cpu); > > > > +#endif > > > > CPUState *some_cpu; > > > > int cpu_index; > > > > > > > > @@ -460,6 +462,8 @@ void cpu_exec_init(CPUArchState *env) > > > > #if defined(CONFIG_USER_ONLY) > > > > cpu_list_unlock(); > > > > #endif > > > > + > > > > +#if !defined(TARGET_I386) > > > > if (qdev_get_vmsd(DEVICE(cpu)) == NULL) { > > > > vmstate_register(NULL, cpu_index, &vmstate_cpu_common, cpu); > > > > } > > > > @@ -472,6 +476,7 @@ void cpu_exec_init(CPUArchState *env) > > > > if (cc->vmsd != NULL) { > > > > vmstate_register(NULL, cpu_index, cc->vmsd, cpu); > > > > } > > > > +#endif /* !TARGET_I386 */ > > > > } > > > > > > > > #if defined(TARGET_HAS_ICE) > > > > diff --git a/target-i386/cpu.c b/target-i386/cpu.c > > > > index 967529a..dada6f6 100644 > > > > --- a/target-i386/cpu.c > > > > +++ b/target-i386/cpu.c > > > > @@ -2552,6 +2552,7 @@ static void x86_cpu_apic_realize(X86CPU *cpu, Error **errp) > > > > static void x86_cpu_realizefn(DeviceState *dev, Error **errp) > > > > { > > > > CPUState *cs = CPU(dev); > > > > + CPUClass *cc = CPU_GET_CLASS(cs); > > > > X86CPU *cpu = X86_CPU(dev); > > > > X86CPUClass *xcc = X86_CPU_GET_CLASS(dev); > > > > CPUX86State *env = &cpu->env; > > > > @@ -2615,6 +2616,14 @@ static void x86_cpu_realizefn(DeviceState *dev, Error **errp) > > > > cpu_reset(cs); > > > > > > > > xcc->parent_realize(dev, &local_err); > > > > + > > > > + if (qdev_get_vmsd(DEVICE(cs)) == NULL) { > > > > + vmstate_register(NULL, env->cpuid_apic_id, &vmstate_cpu_common, cs); > > > > + } > > > > + > > > > + if (cc->vmsd != NULL) { > > > > + vmstate_register(NULL, env->cpuid_apic_id, cc->vmsd, cs); > > > > + } > > > how about doing it in common CPUclass.realize() > > > you can use get_arch_id() for getting CPU id, it returns cpu_index by default > > > and apic_id for target-i386. > > > > Thanks for your kind suggestion, does this mean we can directly move > > vmstate_register to cpu_common_realizefn()? > yep, that is a gist of it. > > There is more to the issue with discontinuous CPUs, a lot of code still > use cpu_index and the way it's allocated is not compatible with discontinuous > CPUs, so this issue should be fixed as well. > > Also you propose to use a raw apic id with CLI/user interface. > I recall there were objections to it since APIC ID contains topology > information and it's not trivial for user to get it right. > The last idea that was discussed to fix it was not expose APIC ID to > user but rather introduce QOM hierarchy like: > /machine/node/N/socket/X/core/Y/thread/Z > and use it in user interface as a means to specify an arbitrary CPU > and let QEMU calculate APIC ID based on this path. > > But nobody took on implementing it yet. We're taking so long to get a decent interface implemented, that part of me is considering exposing the APIC ID directly like suggested before, and requiring libvirt to calculate topology-aware APIC IDs[1] to properly implement CPU hotplug (and possibly for other tasks). Another part of me is hoping that the libvirt developers ask us to please not do that, so I can use it as argument against exposing the APIC IDs directly the next time we discuss this. :) [1] See target-i386/topology.h for references -- Eduardo

10 years, 9 months

4
10
0 / 0

[libvirt] [PATCH] Revert "storage: Introduce internal pool support"

by Peter Krempa

The internal pools were an idea in one of the first iterations of the gluster series that deemed unused. Somehow still this got pushed. Remove it as the internal flag isn't needed. This reverts commit 362da8209d760fc1acd3a1c8df5b04aa676492eb. --- src/conf/storage_conf.c | 3 --- src/conf/storage_conf.h | 1 - src/storage/storage_driver.c | 12 ++++-------- 3 files changed, 4 insertions(+), 12 deletions(-) diff --git a/src/conf/storage_conf.c b/src/conf/storage_conf.c index eaa9325..e4232e9 100644 --- a/src/conf/storage_conf.c +++ b/src/conf/storage_conf.c @@ -2172,9 +2172,6 @@ static bool virStoragePoolMatch(virStoragePoolObjPtr poolobj, unsigned int flags) { - if (poolobj->internal) - return false; - /* filter by active state */ if (MATCH(VIR_CONNECT_LIST_STORAGE_POOLS_FILTERS_ACTIVE) && !((MATCH(VIR_CONNECT_LIST_STORAGE_POOLS_ACTIVE) && diff --git a/src/conf/storage_conf.h b/src/conf/storage_conf.h index cada861..251b968 100644 --- a/src/conf/storage_conf.h +++ b/src/conf/storage_conf.h @@ -332,7 +332,6 @@ struct _virStoragePoolObj { int active; int autostart; unsigned int asyncjobs; - bool internal; virStoragePoolDefPtr def; virStoragePoolDefPtr newDef; diff --git a/src/storage/storage_driver.c b/src/storage/storage_driver.c index 64451c5..e0ebdb0 100644 --- a/src/storage/storage_driver.c +++ b/src/storage/storage_driver.c @@ -376,8 +376,7 @@ storageConnectNumOfStoragePools(virConnectPtr conn) { virStoragePoolObjPtr obj = driver->pools.objs[i]; virStoragePoolObjLock(obj); if (virConnectNumOfStoragePoolsCheckACL(conn, obj->def) && - virStoragePoolObjIsActive(obj) && - !obj->internal) + virStoragePoolObjIsActive(obj)) nactive++; virStoragePoolObjUnlock(obj); } @@ -402,8 +401,7 @@ storageConnectListStoragePools(virConnectPtr conn, virStoragePoolObjPtr obj = driver->pools.objs[i]; virStoragePoolObjLock(obj); if (virConnectListStoragePoolsCheckACL(conn, obj->def) && - virStoragePoolObjIsActive(obj) && - !obj->internal) { + virStoragePoolObjIsActive(obj)) { if (VIR_STRDUP(names[got], obj->def->name) < 0) { virStoragePoolObjUnlock(obj); goto cleanup; @@ -437,8 +435,7 @@ storageConnectNumOfDefinedStoragePools(virConnectPtr conn) { virStoragePoolObjPtr obj = driver->pools.objs[i]; virStoragePoolObjLock(obj); if (virConnectNumOfDefinedStoragePoolsCheckACL(conn, obj->def) && - !virStoragePoolObjIsActive(obj) && - !obj->internal) + !virStoragePoolObjIsActive(obj)) nactive++; virStoragePoolObjUnlock(obj); } @@ -463,8 +460,7 @@ storageConnectListDefinedStoragePools(virConnectPtr conn, virStoragePoolObjPtr obj = driver->pools.objs[i]; virStoragePoolObjLock(obj); if (virConnectListDefinedStoragePoolsCheckACL(conn, obj->def) && - !virStoragePoolObjIsActive(obj) && - !obj->internal) { + !virStoragePoolObjIsActive(obj)) { if (VIR_STRDUP(names[got], obj->def->name) < 0) { virStoragePoolObjUnlock(obj); goto cleanup; -- 1.8.5.3

10 years, 9 months

2
2
0 / 0

[libvirt] [PATCH v8 0/7] Support keyboard device

by Li Zhang

From: Li Zhang <zhlcindy(a)linux.vnet.ibm.com> This patchset is to add keyboard input device. For PPC64, it doesn't support a default keyboard device when the graphic is enabled. Libvirt supports QEMU command line as "-device VGA" which won't create any keyboard device for it. So it requires libvirt to add a default USB keyboard device for it. This patchset is to add keyboard input device and a default USB keyboard for PPC64. The related discussion in QEMU community: http://lists.nongnu.org/archive/html/qemu-devel/2013-11/msg01734.html Li Zhang (7): conf: Add a keyboard input device type conf: Add one interface to add default input devices conf: Remove the implicit PS2 mouse for non-X86 platforms and add an implicit PS2 keyboard device for X86 platforms. qemu_cap: Add USB keyboard capability qemu: format qemu command line for USB keyboard xen: format xen command line for USB keyboard Add a default USB keyboard and USB mouse for PPC64 v8 -> v7: * Add USB keyboard output XML format when VIR_DOMAIN_XML_MIGRATABLE isn't set. (Daniel P. && Jan Tomko) * Fix test cases because of XML changes. v7 -> v6: * Corret indention errors in code. (Jan Tomko) * Correct make check error in xen driver. (Jan Tomko) * Split 5/6 to 2 parts, qemu and xen. (Jan Tomko) v6 -> v5: * Change kbd to keyboard, and correct -usbdevice parameter. * Remove implicit PS/2 keyboard output in XML in case older libvirtd can read it. (Jan Tomko) * Remove implicit keyboard for XEN to make it safer. v5 -> v4: * Add PS/2 KBD definition in XML file. (Daniel.P.Berrange) * Allow PS/2 KBD devices and set KBD default bus as PS2 for X86, USB for non-x86. (Daniel.P.Berrange) * Add an implicit PS/2 KBD as PS/2 mouse. (Danniel.P.Berrange) v4 -> v3: * Don't remove PS2 mouse device for other virtualization drivers (Jan Tomko). v3 -> v2: * Handle the KBD device type in xen and QEMU driver. (Daniel.P.Berrange) * Remove PS2 mouse device for non-X86 platforms. * Move virDomainDefMaybeAddInput to a new patch. (Jan Tomko) * Replace VIR_REALLOC_N with VIR_APPEND_ELEMENT. (Jan Tomoko) * Fix several typos. (Jan Tomoko) * Add a virReportError when QEMU_CAPS_DEVICE_USB_KBD can't be gotten. (Jan Tomoko) v2 -> v1: * change ifs to switch clause. * reconstruct the patches docs/schemas/domaincommon.rng | 1 + src/conf/domain_conf.c | 116 ++++++++++++++------- src/conf/domain_conf.h | 5 + src/libvirt_private.syms | 1 + src/qemu/qemu_capabilities.c | 3 + src/qemu/qemu_capabilities.h | 1 + src/qemu/qemu_command.c | 41 ++++++-- src/qemu/qemu_domain.c | 23 +++- src/util/virarch.h | 2 + src/xenxs/xen_sxpr.c | 27 +++-- src/xenxs/xen_xm.c | 30 ++++-- tests/qemucapabilitiesdata/caps_1.2.2-1.caps | 1 + tests/qemucapabilitiesdata/caps_1.3.1-1.caps | 1 + tests/qemucapabilitiesdata/caps_1.4.2-1.caps | 1 + tests/qemucapabilitiesdata/caps_1.5.3-1.caps | 1 + tests/qemucapabilitiesdata/caps_1.6.0-1.caps | 1 + tests/qemucapabilitiesdata/caps_1.6.50-1.caps | 1 + tests/qemuhelptest.c | 8 ++ ...qemuhotplug-console-compat-2+console-virtio.xml | 1 + .../qemuxml2argv-console-compat-2.xml | 1 + .../qemuxml2argv-graphics-listen-network.xml | 1 + .../qemuxml2argv-graphics-listen-network2.xml | 1 + .../qemuxml2argv-graphics-sdl-fullscreen.xml | 1 + .../qemuxml2argvdata/qemuxml2argv-graphics-sdl.xml | 1 + .../qemuxml2argv-graphics-spice-compression.xml | 1 + .../qemuxml2argv-graphics-spice-qxl-vga.xml | 1 + .../qemuxml2argv-graphics-spice-timeout.xml | 2 + .../qemuxml2argv-graphics-spice.xml | 1 + .../qemuxml2argv-graphics-vnc-policy.xml | 1 + .../qemuxml2argv-graphics-vnc-sasl.xml | 1 + .../qemuxml2argv-graphics-vnc-socket.xml | 1 + .../qemuxml2argv-graphics-vnc-tls.xml | 1 + .../qemuxml2argv-graphics-vnc-websocket.xml | 1 + .../qemuxml2argvdata/qemuxml2argv-graphics-vnc.xml | 1 + tests/qemuxml2argvdata/qemuxml2argv-input-xen.xml | 1 + .../qemuxml2argv-net-bandwidth.xml | 1 + tests/qemuxml2argvdata/qemuxml2argv-pci-bridge.xml | 1 + .../qemuxml2argvdata/qemuxml2argv-pseries-disk.xml | 3 +- .../qemuxml2argv-pseries-usb-kbd.args | 9 ++ .../qemuxml2argv-pseries-usb-kbd.xml | 19 ++++ .../qemuxml2argv-serial-spiceport.xml | 1 + tests/qemuxml2argvtest.c | 3 + .../qemuxml2xmlout-graphics-listen-network2.xml | 1 + .../qemuxml2xmlout-graphics-spice-timeout.xml | 1 + tests/vmx2xmldata/vmx2xml-graphics-vnc.xml | 1 + 45 files changed, 256 insertions(+), 66 deletions(-) create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-pseries-usb-kbd.args create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-pseries-usb-kbd.xml -- 1.8.2.1

10 years, 9 months

2
11
0 / 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

Devel February 2014