[libvirt] [PATCH V2 0/4]] qemu: report actual vcpu state in virVcpuInfo
by Viktor Mihajlovski
Currently, the virVcpuInfo returned by virDomainGetVcpus() will always
report a state of VIR_VCPU_RUNNING for each defined domain vcpu even if
the vcpu is currently in the halted state.
As the monitor interface is in fact reporting the accurate state, it is
rather easy to transport this information with the existing API.
This is done by
- adding a new state of VIR_VCPU_HALTED
- extending the monitor to pass back the halted state for the vcpus
- adding a new field to the private domain vcpu object reflecting the
halted state for the vcpu
- modifying the driver code to report the vcpu state based on the halted
indicator
- extending virsh vcpuinfo to also display the halted state
The vcpu state is however not recorded in the internal XML format, since
the state can change asynchronously (without notification).
V2 is a rebase on top of Peter Krempa's CPU hotplug modernization.
Viktor Mihajlovski (4):
domain: Add new VCPU state "halted"
qemu: Add monitor support for CPU halted state
qemu: Add domain support for VCPU halted state
qemu: Ensure reported VCPU state is current in driver API
include/libvirt/libvirt-domain.h | 1 +
src/qemu/qemu_domain.c | 69 ++++++++++++++++++++++++++++++++++++++++
src/qemu/qemu_domain.h | 5 +++
src/qemu/qemu_driver.c | 23 ++++++++++++--
src/qemu/qemu_monitor.c | 6 +++-
src/qemu/qemu_monitor.h | 5 +++
src/qemu/qemu_monitor_json.c | 3 ++
src/qemu/qemu_monitor_text.c | 8 ++++-
tests/qemumonitorjsontest.c | 8 ++---
tools/virsh-domain.c | 3 +-
10 files changed, 122 insertions(+), 9 deletions(-)
--
1.9.1
8 years, 2 months
[libvirt] [PATCH 0/6] qemu: fix UDP chardev hotplug and make qemumonitorjsontest less useless
by Peter Krempa
Most of the time qemumonitorjson test isn't really testing much. Add support
for actually testing chardev hotplug and fix it for the UDP case.
Peter Krempa (6):
conf: Sanitize formatting of UDP chardev source
tests: qemu: Add support for testing aguments on monitor verbatim
tests: qemumonitorjson: Don't do multiple tests in one virTestRun
tests: qemumonitorjsontest: Do some actual testing in
qemuMonitorJSONTestAttachChardev
qemu: monitor: Simplify construction of chardev backends
qemu: monitor: Properly configure backend for UDP chardevs
src/conf/domain_conf.c | 42 ++++----
src/qemu/qemu_monitor_json.c | 51 +++++-----
tests/qemumonitorjsontest.c | 222 ++++++++++++++++++++++++++++++++-----------
tests/qemumonitortestutils.c | 112 ++++++++++++++++++++++
tests/qemumonitortestutils.h | 6 ++
5 files changed, 326 insertions(+), 107 deletions(-)
--
2.10.0
8 years, 2 months
[libvirt] [PATCH 00/12] Add length (duration) params for iotune throttling
by John Ferlan
https://bugzilla.redhat.com/show_bug.cgi?id=1349898
Do a little housekeeping and minor adjustments to existing code, then
add the various "-length" options for the <iotune> code.
John Ferlan (12):
docs: Fix typo in libvirt-domain.h parameter description
include: Update description for <iotune> max params
tests: Add blkdeviotune-max xml2xmltest
qemu: Convert from shorthand to longer throttling names
qemu: Adjust how supportMaxOptions is used.
include: Add new definitions for duration for bps/iops throttling
caps: Add new capability for the bps/iops throttling length
qemu: Add length for bps/iops throttling parameters to driver
conf: Add a formatting macro for all the blkiotune values
conf: Adjust the PARSE_IOTUNE macro
conf: Add support for blkiotune "_length" options
qemu: Add the length options to the iotune command line
docs/formatdomain.html.in | 40 +++++-
docs/schemas/domaincommon.rng | 38 ++++++
include/libvirt/libvirt-domain.h | 124 ++++++++++++++++--
src/conf/domain_conf.c | 142 +++++++++++----------
src/conf/domain_conf.h | 6 +
src/qemu/qemu_capabilities.c | 2 +
src/qemu/qemu_capabilities.h | 1 +
src/qemu/qemu_command.c | 96 ++++++--------
src/qemu/qemu_driver.c | 106 ++++++++++++++-
src/qemu/qemu_monitor.c | 7 +-
src/qemu/qemu_monitor.h | 3 +-
src/qemu/qemu_monitor_json.c | 72 ++++++-----
src/qemu/qemu_monitor_json.h | 3 +-
.../caps_2.6.0-gicv2.aarch64.xml | 1 +
.../caps_2.6.0-gicv3.aarch64.xml | 1 +
tests/qemucapabilitiesdata/caps_2.6.0.ppc64le.xml | 1 +
tests/qemucapabilitiesdata/caps_2.6.0.x86_64.xml | 1 +
tests/qemucapabilitiesdata/caps_2.7.0.x86_64.xml | 1 +
tests/qemumonitorjsontest.c | 17 ++-
.../qemuxml2argv-blkdeviotune-max-length.args | 34 +++++
.../qemuxml2argv-blkdeviotune-max-length.xml | 65 ++++++++++
.../qemuxml2argv-blkdeviotune-max.args | 10 +-
.../qemuxml2argv-blkdeviotune-max.xml | 14 +-
.../qemuxml2argv-blkdeviotune.args | 5 +-
tests/qemuxml2argvtest.c | 4 +
.../qemuxml2xmlout-blkdeviotune-max-length.xml | 1 +
.../qemuxml2xmlout-blkdeviotune-max.xml | 1 +
tests/qemuxml2xmltest.c | 2 +
28 files changed, 616 insertions(+), 182 deletions(-)
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-blkdeviotune-max-length.args
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-blkdeviotune-max-length.xml
create mode 120000 tests/qemuxml2xmloutdata/qemuxml2xmlout-blkdeviotune-max-length.xml
create mode 120000 tests/qemuxml2xmloutdata/qemuxml2xmlout-blkdeviotune-max.xml
--
2.7.4
8 years, 2 months
[libvirt] [PATCH RESEND v2 0/5] Implementation of QEMU vhost-scsi
by Eric Farman
[Resending after the release of 2.2.0; no changes other than a rebase
to current master and the associated tweaking to the capabilities patch]
This patch series provides a libvirt implementation of the vhost-scsi
interface in QEMU. As near as I can see, this was discussed upstream in
July 2014[1], and ended in a desire to replace a vhost-scsi controller
in favor of a hostdev element instead[2].
There is no capability check in this series for vhost-scsi in the underlying
QEMU. Using a recent QEMU built with --disable-vhost-scsi fails with "not a
valid device model name."
Host Filesystem Example:
# ls /sys/kernel/config/target/vhost/
discovery_auth naa.5001405df3e54061 version
# ls /sys/kernel/config/target/vhost/naa.5001405df3e54061/tpgt_1/lun/
lun_0
QEMU Example (snippet):
-device vhost-scsi-ccw,wwpn=naa.5001405df3e54061,devno=fe.0.1000
Libvirt Example (snippet):
<hostdev mode='subsystem' type='scsi_host'>
<source protocol='vhost' wwpn='naa.5001405df3e54061'/>
<address type='ccw' cssid='0xfe' ssid='0x0' devno='0x1000'/>
</hostdev>
Guest Viewpoint:
# lsscsi
[1:0:1:0] disk LIO-ORG disk0 4.0 /dev/sda
# dmesg | grep 1:
[ 6.065735] scsi host1: Virtio SCSI HBA
[ 6.093892] scsi 1:0:1:0: Direct-Access LIO-ORG disk0 4.0 PQ: 0 ANSI: 5
[ 6.313615] sd 1:0:1:0: Attached scsi generic sg0 type 0
[ 6.314981] sd 1:0:1:0: [sda] 29360128 512-byte logical blocks: (15.0 GB/14.0 GiB)
[ 6.317290] sd 1:0:1:0: [sda] Write Protect is off
[ 6.317566] sd 1:0:1:0: [sda] Mode Sense: 43 00 10 08
[ 6.317853] sd 1:0:1:0: [sda] Write cache: enabled, read cache: enabled, supports DPO and FUA
[ 6.352722] sd 1:0:1:0: [sda] Attached SCSI disk
Changelog:
v2->v2.1:
- Rebased to current master (6 September)
v1->v2: https://www.redhat.com/archives/libvir-list/2016-August/msg01028.html
- Rebase
- Applies to current master (20 August)
- Added a capability check for QEMU 2.7
- Fixed the qemuxml2argv tests as the -smp options had changed
- Reworked ccwaddrs parameter in virDomainCCWAddressAssign call
- Comments
- Squashed documentation, XML schema, XML parsing, and infrastructure
patches into a single patch
- Switched from "hostdev type='scsi'" to "hostdev type='scsi_host'";
this removes the refactoring patches since we're not shoe-horning
a new SCSI protocol into the existing code
- Reworked the handling of the fd's such that we send them to qemu
after any possible errors could occur and cause us to back out
- s/qemuBuildSCSIVhostHostdevDevStr/qemuBuildHostHostdevDevStr/
- Added virBufferCheckError, and an error message for vhostfdSize > 1,
in qemuBuildHostHostdevDevStr
- Added qemuBuildDeviceAddressStr in qemuBuildHostHostdevDevStr, thus
superceding the last patch in the v1 series
- Other
- Simplified the vhostfd logic to just be a single int, rather than
an alloc'd array (left the vhostfdSize described above as an
identifier for if QEMU ever supports multiple vhostfds)
- Replaced "qemuMonitorAddDevice" with "qemuMonitorAddDeviceWithFd"
in hotplug routine
v1: https://www.redhat.com/archives/libvir-list/2016-July/msg01004.html
[1] http://www.redhat.com/archives/libvir-list/2014-July/msg01235.html
[2] http://www.redhat.com/archives/libvir-list/2014-July/msg01390.html
Eric Farman (5):
Introduce a "scsi_host" hostdev type
qemu: Introduce vhost-scsi capability
qemu: Add vhost-scsi string for -device parameter
qemu: Allow hotplug of vhost-scsi device
tests: Introduce basic vhost-scsi test
docs/formatdomain.html.in | 24 ++++
docs/schemas/domaincommon.rng | 23 ++++
src/conf/domain_audit.c | 2 +
src/conf/domain_conf.c | 62 ++++++++-
src/conf/domain_conf.h | 17 +++
src/libvirt_private.syms | 1 +
src/qemu/qemu_capabilities.c | 3 +
src/qemu/qemu_capabilities.h | 3 +
src/qemu/qemu_command.c | 80 +++++++++++
src/qemu/qemu_command.h | 6 +
src/qemu/qemu_domain_address.c | 10 ++
src/qemu/qemu_hotplug.c | 149 +++++++++++++++++++++
src/security/security_dac.c | 2 +
src/util/virscsi.c | 26 ++++
src/util/virscsi.h | 1 +
tests/domaincapsschemadata/full.xml | 1 +
tests/qemucapabilitiesdata/caps_1.5.3.x86_64.xml | 1 +
tests/qemucapabilitiesdata/caps_1.6.0.x86_64.xml | 1 +
tests/qemucapabilitiesdata/caps_1.7.0.x86_64.xml | 1 +
tests/qemucapabilitiesdata/caps_2.1.1.x86_64.xml | 1 +
tests/qemucapabilitiesdata/caps_2.4.0.x86_64.xml | 1 +
tests/qemucapabilitiesdata/caps_2.5.0.x86_64.xml | 1 +
.../caps_2.6.0-gicv2.aarch64.xml | 1 +
.../caps_2.6.0-gicv3.aarch64.xml | 1 +
tests/qemucapabilitiesdata/caps_2.6.0.ppc64le.xml | 1 +
tests/qemucapabilitiesdata/caps_2.6.0.x86_64.xml | 1 +
tests/qemucapabilitiesdata/caps_2.7.0.x86_64.xml | 1 +
.../qemuxml2argv-hostdev-scsi-vhost-scsi.args | 24 ++++
.../qemuxml2argv-hostdev-scsi-vhost-scsi.xml | 33 +++++
tests/qemuxml2argvmock.c | 12 ++
tests/qemuxml2argvtest.c | 3 +
31 files changed, 491 insertions(+), 2 deletions(-)
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-hostdev-scsi-vhost-scsi.args
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-hostdev-scsi-vhost-scsi.xml
--
1.9.1
8 years, 2 months
[libvirt] [PATCH 0/3] Cleanup/unify RBD backend context setup
by John Ferlan
While working on a way to create a luks volume on an RBD backend - I figured
I'd clean up the initialization of the environment a bit. Rather than a lot
of copy/paste code - use a common initializer.
John Ferlan (3):
rbd: Change virStorageBackendRBDCloseRADOSConn to be static void
rbd: Change from static to alloc contexts
rbd: Move the encryption check in build
src/storage/storage_backend_rbd.c | 150 +++++++++++++++++++-------------------
1 file changed, 73 insertions(+), 77 deletions(-)
--
2.7.4
8 years, 2 months
[libvirt] [PATCH 0/2] Add support for preallocated fd memory
by Jaroslav Safka
Hi the xml element to be extended with additional children is the memoryBacking element.
We would like to introduce 3 new elements source,access and allocation
<memoryBacking>
<source type="file|anonymous" path='/path/to/qemu/'/>
<access Mode="shared|private"/>
<allocation mode="immediate|ondemand"/>
</memoryBacking>
If allocation is immediate then -mem-prealloc should be added to the qemu commanline.
If source is file then
-object memory-backend-file,id=mem,size=1024M,mem-path=/var/lib/libvirt/qemu -numa node,memdev=mem
Should be added to the qemu commandline
If access is shared then the share=on parameter should be added to the memory-backend-file e.g.
-object memory-backend-file,id=mem,size=1024M,mem-path=/var/lib/libvirt/qemu,share=on
Jaroslav Safka (2):
Add support for preallocated fd memory
Add support for preallocated fd memory
docs/schemas/domaincommon.rng | 37 +++++
src/conf/domain_conf.c | 149 +++++++++++++++-----
src/conf/domain_conf.h | 34 +++++
src/qemu/qemu_command.c | 156 ++++++++++++++++-----
src/qemu/qemu_command.h | 4 +
.../qemuxml2argv-fd-memory-no-numa-topology.args | 34 +++++
.../qemuxml2argv-fd-memory-no-numa-topology.xml | 96 +++++++++++++
.../qemuxml2argv-fd-memory-numa-topology.args | 34 +++++
.../qemuxml2argv-fd-memory-numa-topology.xml | 99 +++++++++++++
.../qemuxml2argv-memorybacking-set.xml | 32 +++++
.../qemuxml2argv-memorybacking-unset.xml | 32 +++++
tests/qemuxml2argvtest.c | 24 ++++
.../qemuxml2xmlout-memorybacking-set.xml | 40 ++++++
.../qemuxml2xmlout-memorybacking-unset.xml | 40 ++++++
tests/qemuxml2xmltest.c | 3 +
15 files changed, 748 insertions(+), 66 deletions(-)
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-fd-memory-no-numa-topology.args
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-fd-memory-no-numa-topology.xml
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-fd-memory-numa-topology.args
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-fd-memory-numa-topology.xml
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-memorybacking-set.xml
create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-memorybacking-unset.xml
create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-memorybacking-set.xml
create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-memorybacking-unset.xml
--
2.5.5
--------------------------------------------------------------
Intel Research and Development Ireland Limited
Registered in Ireland
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
Registered Number: 308263
This e-mail and any attachments may contain confidential material for the sole
use of the intended recipient(s). Any review or distribution by others is
strictly prohibited. If you are not the intended recipient, please contact the
sender and delete all copies.
8 years, 2 months
[libvirt] Entering freeze for libvirt-2.3.0
by Daniel Veillard
Since nobody complained about my earlier message with the release plan,
I tagged libvirt-2.3.0 candidate release 1 in git and pushed signed tarball
and rpms to the usual place:
ftp://libvirt.org/libvirt/
As usual my limited testing is really not sufficient so please give it a try,
I enjoy the view of a completely green https://ci.centos.org/view/libvirt-project/ :-)
but that doesn't test portability to other platforms for example !
Then I will try to push rc2 on Thursday, that way the final release can happen
during the week-end or on Monday if all goes well,
thanks for testing it !
Daniel
--
Daniel Veillard | Open Source and Standards, Red Hat
veillard(a)redhat.com | libxml Gnome XML XSLT toolkit http://xmlsoft.org/
http://veillard.com/ | virtualization library http://libvirt.org/
8 years, 2 months
[libvirt] [PATCH 0/3] introduce pull backups
by Nikolay Shirokovskiy
Hi, everyone.
Please take a look at pull backup API and its qemu implementation. There is no any
documentation inside yet so I put it here.
Backup is described in xml like next:
<domainbackup>
<address type="ip" host="0.0.0.0" port="7000"/>
<disks>
<disk name='sda'>
<source file="/root/backup.hdd"/>
</disk>
<disk name='sdb' present='no'/>
</disks>
</domainbackup>
<disks>, <disk>, <source> elements and 'present' attribute are optional. The
export address is bare minimum to export all non read only disks with
inserted media. This xml description is close to what we have for snapshots.
Implementation depends on experimental qemu command 'x-blockdev-del' and
'blockdev-add' which is still described as work-in-progress even it has
no 'x-' prefix.
I failed to find the information whether client reading disks being backed up
is able to detect all failures in the process or not so some efforts
are done to track back up blockjobs failures and report them on pull
backup stop. At the same time other clean up errors on stop are ignored
so that client can know the essence was the backup operation successful
or not.
There is a work in progress to provide dirty bitmap information thru
nbd to be able to make incremental backups in pull backup scheme.
Thus the design should be future proof in this aspect. I see next
changes to xml description:
<domainbackup>
<address type="ip" host="0.0.0.0" port="7000"/>
<branch name="backup1"/>
<disks>
<disk name='sda' type='incremental'>
<source file="/root/backup.hdd"/>
<bitmap granularity='32Kib'/>
</disk>
</disks>
</domainbackup>
Branch name essentially becames dirty backup name in terms of qemu.
This option makes possible to have independent branches of
incremental backups with possibly different schedule and granularity.
Stop operation in case of incremental backups should be specified
further to finish/cancel the operation - we have flags for that.
Nikolay Shirokovskiy (3):
qemu: store guest visible disk size from qemu monitor block info
qemu: special error code in case of no job on cancel block job
introduce pull backup
examples/object-events/event-test.c | 3 +
include/libvirt/libvirt-domain-backup.h | 45 +++
include/libvirt/libvirt-domain.h | 3 +
include/libvirt/libvirt.h | 1 +
include/libvirt/virterror.h | 1 +
po/POTFILES.in | 2 +
src/Makefile.am | 3 +
src/access/viraccessperm.c | 3 +-
src/access/viraccessperm.h | 6 +
src/conf/backup_conf.c | 295 ++++++++++++++
src/conf/backup_conf.h | 85 ++++
src/conf/domain_conf.c | 2 +-
src/driver-hypervisor.h | 11 +
src/libvirt-domain-backup.c | 86 ++++
src/libvirt_private.syms | 6 +
src/libvirt_public.syms | 2 +
src/qemu/qemu_blockjob.c | 2 +
src/qemu/qemu_conf.h | 1 +
src/qemu/qemu_domain.h | 5 +
src/qemu/qemu_driver.c | 684 ++++++++++++++++++++++++++++++++
src/qemu/qemu_monitor.c | 37 ++
src/qemu/qemu_monitor.h | 12 +
src/qemu/qemu_monitor_json.c | 133 ++++++-
src/qemu/qemu_monitor_json.h | 16 +
src/remote/remote_driver.c | 2 +
src/remote/remote_protocol.x | 33 +-
src/util/virerror.c | 1 +
tests/qemumonitorjsontest.c | 36 ++
tools/Makefile.am | 1 +
tools/virsh-backup.c | 150 +++++++
tools/virsh-backup.h | 28 ++
tools/virsh-domain.c | 3 +-
tools/virsh.c | 2 +
tools/virsh.h | 1 +
34 files changed, 1693 insertions(+), 8 deletions(-)
create mode 100644 include/libvirt/libvirt-domain-backup.h
create mode 100644 src/conf/backup_conf.c
create mode 100644 src/conf/backup_conf.h
create mode 100644 src/libvirt-domain-backup.c
create mode 100644 tools/virsh-backup.c
create mode 100644 tools/virsh-backup.h
--
1.8.3.1
8 years, 2 months
[libvirt] [PATCH v2 1/2] NSS: Add explicit check to not report expired lease
by Nehal J Wani
The NSS module shouldn't rely on custom leases database to not have
entries for leases which have expired.
Change-Id: Ic3e043003d33ded0da74696a1d27ed4967ddbfb8
---
tools/nss/libvirt_nss.c | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)
diff --git a/tools/nss/libvirt_nss.c b/tools/nss/libvirt_nss.c
index 54c4a2a..cb3edf5 100644
--- a/tools/nss/libvirt_nss.c
+++ b/tools/nss/libvirt_nss.c
@@ -42,6 +42,7 @@
#include "virlease.h"
#include "viralloc.h"
#include "virfile.h"
+#include "virtime.h"
#include "virerror.h"
#include "virstring.h"
#include "virsocketaddr.h"
@@ -114,6 +115,8 @@ findLease(const char *name,
ssize_t i, nleases;
leaseAddress *tmpAddress = NULL;
size_t ntmpAddress = 0;
+ time_t currtime;
+ long long expirytime;
*address = NULL;
*naddress = 0;
@@ -161,6 +164,11 @@ findLease(const char *name,
nleases = virJSONValueArraySize(leases_array);
DEBUG("Read %zd leases", nleases);
+ if ((currtime = time(NULL)) == (time_t) - 1) {
+ ERROR("Failed to get current system time");
+ goto cleanup;
+ }
+
for (i = 0; i < nleases; i++) {
virJSONValuePtr lease;
const char *lease_name;
@@ -181,6 +189,16 @@ findLease(const char *name,
if (STRNEQ_NULLABLE(name, lease_name))
continue;
+ if (virJSONValueObjectGetNumberLong(lease, "expiry-time", &expirytime) < 0) {
+ /* A lease cannot be present without expiry-time */
+ ERROR("expiry-time field missing for %s", name);
+ goto cleanup;
+ }
+
+ /* Do not report expired lease */
+ if (expirytime < (long long) currtime)
+ continue;
+
DEBUG("Found record for %s", lease_name);
*found = true;
--
2.4.11
8 years, 2 months
[libvirt] [PATCH v2 0/5] Combine various query-block json call paths
by John Ferlan
v1: http://www.redhat.com/archives/libvir-list/2016-September/msg01446.html
NOTE: Patch 1 already ACK'd
Patches 2-4 adjusted slightly to not create/use qemuMonitorJSONQueryBlockArgs
instead opting to pass all the args (also shortened helper names slightly).
Patch 5 adjusted to receive all args in qemuMonitorJSONQueryBlock and then
call one of two callbacks based on whether the table or search is being used.
I did have a version that used ATTRIBUTE_UNUSED and one generic callback, but
I thought that was uglier.
John Ferlan (5):
qemu: Create common code for JSON "query-block" call
qemu: Split out filling of JSONGetBlockInfo data
qemu: Split out filling of JSONBlockStats data
qemu: Split out filling of JSONDiskNameLookup data
qemu: Combine the various ways to call query-block
src/qemu/qemu_monitor_json.c | 356 +++++++++++++++++++++++++------------------
1 file changed, 205 insertions(+), 151 deletions(-)
For "statistical purposes" after each patch the I checked --shortstat:
Patch 1: 34 insertions, 29 deletions
Patch 2: 93 insertions, 68 deletions
Patch 3: 122 insertions, 78 deletions
Patch 4: 158 insertions, 86 deletions
Patch 5: 205 insertions, 151 deletions
Of the insertions, there's 53 lines of function header comments and
46 lines of function declarations...
--
2.7.4
8 years, 2 months
