[libvirt PATCH 00/29] Refactor scripts in tests/cputestdata

Hotplugng disk not adding backing...

[libvirt PATCH 0/2] Schema fixes...

Tim Wiederhake

Tuesday, 15 December 2020 Tue, 15 Dec '20

10:24 a.m.

This series refactors the various scripts found in tests/cputestdata and adds support for CORE_CAPABILITY MSR, as found on e.g. SnowRidge. Acquiring test data on a new system is a two step process. "cpu-gather.sh" gathers information on the target machine and has as few dependencies as possible. "cpu-parse.sh" processes this information but requires access to a libvirt source tree and has more dependencies, e.g. "xmltodict". This series merges three of the four involved scripts (cpu-gather.sh, cpu-parse.sh and cpu-reformat.py) into a single python3 script. python3 already was a dependency for cpu-gather.sh and care has been taken to not depend on modules that are not installed by default [1]. Merging the fourth script, cpu-cpuid.py, will come in a seperate series. Patches 1 to 14 transform cpu-gather into a python script, preserving the format of the output (except for consistent "\n" line endings; previously the tool would output a mix of "\n" and "\r\n"). Patches 15 to 23 merge cpu-parse into the script. In this process, the format of the intermediary data is changed to json. Patches 24 to 29 add support for "all in one" operation and extracting IA32_CORE_CAPABILITY_MSR, which can be found on e.g. SnowRidge CPUs. Old usage: ./cpu-gather.sh | ./cpu-parse.sh New: ./cpu-gather.py [--gather] | ./cpu-gather.py --parse Alternative on single machine: ./cpu-gather.py --gather --parse [1] https://docs.python.org/3/py-modindex.html Tim Wiederhake (29): cpu-cpuid: Shorten overly long line cpu-gather: Create python wrapper for shell script cpu-gather: Move model_name to new script cpu-gather: Allow overwriting model name cpu-gather: Move cpuid call to new script cpu-gather: Allow overwriting cpuid binary location cpu-gather: Move msr decoding to new script cpu-gather: Move qemu detection to new script cpu-gather: Move static model expansion to new script cpu-gather: Move static model extraction to new script cpu-gather: Move simple model extraction to new script cpu-gather: Move full model extraction to new script cpu-gather: Merge model gathering logic cpu-gather: Delete old script cpu-gather: Separate data input and output cpu-parse: Wrap with python script cpu-gather: Transport data as json cpu-parse: Move model name detection to new script cpu-parse: Move file name generation to new script cpu-parse: Move xml output to new script cpu-parse: Move json output to new script cpu-parse: Move call to cpu-cpuid.py to new script cpu-parse: Delete old script cpu-gather: Ignore empty responses from qemu cpu-gather: Ignore shutdown messages from qemu cpu-gather: Parse cpuid leaves early cpu-gather: Allow gathering and parsing data in one step. cpu-gather: Prepare gather_msr for reading multiple msr cpu-gather: Add IA32_CORE_CAPABILITY_MSR tests/cputestdata/cpu-cpuid.py | 5 +- tests/cputestdata/cpu-gather.py | 365 ++++++++++++++++++++++++++++++ tests/cputestdata/cpu-gather.sh | 103 --------- tests/cputestdata/cpu-parse.sh | 65 ------ tests/cputestdata/cpu-reformat.py | 9 - 5 files changed, 368 insertions(+), 179 deletions(-) create mode 100755 tests/cputestdata/cpu-gather.py delete mode 100755 tests/cputestdata/cpu-gather.sh delete mode 100755 tests/cputestdata/cpu-parse.sh delete mode 100755 tests/cputestdata/cpu-reformat.py -- 2.26.2

Show replies by date

Tim Wiederhake

Tuesday, 15 December Tue, 15 Dec

10:24 a.m.

New subject: [libvirt PATCH 01/29] cpu-cpuid: Shorten overly long line

flake8 was complaining. Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-cpuid.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/tests/cputestdata/cpu-cpuid.py b/tests/cputestdata/cpu-cpuid.py index 37d00e3c97..dac43debb6 100755 --- a/tests/cputestdata/cpu-cpuid.py +++ b/tests/cputestdata/cpu-cpuid.py @@ -158,8 +158,9 @@ def parseMap(): cpuMap = {} for feature in data["cpus"]["feature"]: for fType in ["cpuid", "msr"]: - if fType in feature: - cpuMap[feature["@name"]] = parseMapFeature(fType, feature[fType]) + if fType not in feature: + continue + cpuMap[feature["@name"]] = parseMapFeature(fType, feature[fType]) return cpuMap -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 02/29] cpu-gather: Create python wrapper for shell script

This changes the invocation from ./cpu-gather.sh | ./cpu-parse.sh to ./cpu-gather.py | ./cpu-parse.sh Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 13 +++++++++++++ tests/cputestdata/cpu-gather.sh | 5 +++++ 2 files changed, 18 insertions(+) create mode 100755 tests/cputestdata/cpu-gather.py diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py new file mode 100755 index 0000000000..f7030eb48b --- /dev/null +++ b/tests/cputestdata/cpu-gather.py @@ -0,0 +1,13 @@ +#!/usr/bin/env python3 + +import os +import subprocess + + +def main(): + os.environ["CPU_GATHER_PY"] = "true" + subprocess.check_call("./cpu-gather.sh") + + +if __name__ == "__main__": + main() diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index 7574324d1c..cd65d74da5 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -4,6 +4,11 @@ # distro does not provide such package, you can find the sources or binary # packages at https://www.etallen.com/cpuid.html +if [ -z "${CPU_GATHER_PY}" ]; then + echo >&2 "Do not call this script directly. Use 'cpu-gather.py' instead." + exit 1 +fi + grep 'model name' /proc/cpuinfo | head -n1 cpuid -1r -- 2.26.2

Michal Prívozník

Friday, 18 December Fri, 18 Dec

12:08 p.m.

New subject: [libvirt PATCH 02/29] cpu-gather: Create python wrapper for shell script

On 12/15/20 5:24 PM, Tim Wiederhake wrote:

...

I think cpu-parse.sh which shows the example of invocation too should be updated as well: diff --git i/tests/cputestdata/cpu-parse.sh w/tests/cputestdata/cpu-parse.sh index 7501c57cba..2981e9193c 100755 --- i/tests/cputestdata/cpu-parse.sh +++ w/tests/cputestdata/cpu-parse.sh @@ -1,7 +1,7 @@ #!/bin/bash # Usage: -# ./cpu-gather.sh | ./cpu-parse.sh +# ./cpu-gather.py | ./cpu-parse.sh data=`cat` Michal

Tim Wiederhake

Tuesday, 15 December Tue, 15 Dec

10:24 a.m.

New subject: [libvirt PATCH 03/29] cpu-gather: Move model_name to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 13 +++++++++++++ tests/cputestdata/cpu-gather.sh | 2 -- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index f7030eb48b..1b02df6ec7 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -4,7 +4,20 @@ import os import subprocess +def gather_name(): + with open("/proc/cpuinfo", "rt") as f: + for line in f.readlines(): + if line.startswith("model name"): + return line.split(":", 2)[1].strip() + + exit("Error: '/proc/cpuinfo' does not contain a model name.") + + def main(): + name = gather_name() + print("model name\t: {}".format(name)) + + print(end="", flush=True) os.environ["CPU_GATHER_PY"] = "true" subprocess.check_call("./cpu-gather.sh") diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index cd65d74da5..b671f223a5 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -9,8 +9,6 @@ if [ -z "${CPU_GATHER_PY}" ]; then exit 1 fi -grep 'model name' /proc/cpuinfo | head -n1 - cpuid -1r echo -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 04/29] cpu-gather: Allow overwriting model name

Some hardware, e.g. exotic platforms or pre-production hardware, may report wrong or random data for the cpu model name. As the name of the created files is derived from that name, this may lead to issues. Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 19 ++++++++++++++++--- 1 file changed, 16 insertions(+), 3 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 1b02df6ec7..4e8c72e4f4 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -1,20 +1,33 @@ #!/usr/bin/env python3 +import argparse import os import subprocess -def gather_name(): +def gather_name(args): + if args.name: + return args.name + with open("/proc/cpuinfo", "rt") as f: for line in f.readlines(): if line.startswith("model name"): return line.split(":", 2)[1].strip() - exit("Error: '/proc/cpuinfo' does not contain a model name.") + exit("Error: '/proc/cpuinfo' does not contain a model name.\n" + "Use '--model' to set a model name.") def main(): - name = gather_name() + parser = argparse.ArgumentParser(description="Gather cpu test data") + parser.add_argument( + "--name", + help="CPU model name. " + "If unset, model name is read from '/proc/cpuinfo'.") + + args = parser.parse_args() + + name = gather_name(args) print("model name\t: {}".format(name)) print(end="", flush=True) -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 05/29] cpu-gather: Move cpuid call to new script

Turn the comment on how to aquire cpuid into a runtime error message. Use "http" instead of "https" in the URL, as the latter is broken. Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 25 +++++++++++++++++++++++++ tests/cputestdata/cpu-gather.sh | 7 ------- 2 files changed, 25 insertions(+), 7 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 4e8c72e4f4..97655399c8 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -18,6 +18,25 @@ def gather_name(args): "Use '--model' to set a model name.") +def gather_cpuid_leaves(): + try: + output = subprocess.check_output( + ["cpuid", "-1r"], + universal_newlines=True) + except FileNotFoundError as e: + exit("Error: '{}' not found.\n'cpuid' can be usually found in a " + "package named identically. If your distro does not provide such " + "package, you can find the sources or binary packages at " + "'http://www.etallen.com/cpuid.html'.".format(e.filename)) + + for line in output.split("\n"): + if not line: + continue + if line == "CPU:": + continue + yield line.strip() + + def main(): parser = argparse.ArgumentParser(description="Gather cpu test data") parser.add_argument( @@ -30,6 +49,12 @@ def main(): name = gather_name(args) print("model name\t: {}".format(name)) + leaves = gather_cpuid_leaves() + print("CPU:") + for leave in leaves: + print(" {}".format(leave)) + print() + print(end="", flush=True) os.environ["CPU_GATHER_PY"] = "true" subprocess.check_call("./cpu-gather.sh") diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index b671f223a5..f84215e777 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -1,17 +1,10 @@ #!/bin/bash -# -# The cpuid tool can be usually found in a package called "cpuid". If your -# distro does not provide such package, you can find the sources or binary -# packages at https://www.etallen.com/cpuid.html if [ -z "${CPU_GATHER_PY}" ]; then echo >&2 "Do not call this script directly. Use 'cpu-gather.py' instead." exit 1 fi -cpuid -1r -echo - python3 <<EOF from struct import pack, unpack from fcntl import ioctl -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 06/29] cpu-gather: Allow overwriting cpuid binary location

This is useful if cpuid was compiled from source in a non-standard location. Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 97655399c8..75cf290a28 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -18,10 +18,11 @@ def gather_name(args): "Use '--model' to set a model name.") -def gather_cpuid_leaves(): +def gather_cpuid_leaves(args): + cpuid = args.path_to_cpuid or "cpuid" try: output = subprocess.check_output( - ["cpuid", "-1r"], + [cpuid, "-1r"], universal_newlines=True) except FileNotFoundError as e: exit("Error: '{}' not found.\n'cpuid' can be usually found in a " @@ -43,13 +44,18 @@ def main(): "--name", help="CPU model name. " "If unset, model name is read from '/proc/cpuinfo'.") + parser.add_argument( + "--path-to-cpuid", + metavar="PATH", + help="Path to 'cpuid' utility. " + "If unset, the first executable 'cpuid' in $PATH is used.") args = parser.parse_args() name = gather_name(args) print("model name\t: {}".format(name)) - leaves = gather_cpuid_leaves() + leaves = gather_cpuid_leaves(args) print("CPU:") for leave in leaves: print(" {}".format(leave)) -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 07/29] cpu-gather: Move msr decoding to new script

Fixes the leaking file descriptors. Does not silently ignore errors (e.g. permission denied on /dev/cpu/0/msr if run as non-root) and always attempt to read from /dev/kvm if /dev/cpu/0/msr failed. 'gather_msr()' returns a dictionary of values, as a later patch will add more registers to be interrogated. Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 34 ++++++++++++++++++++++++++++ tests/cputestdata/cpu-gather.sh | 40 --------------------------------- 2 files changed, 34 insertions(+), 40 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 75cf290a28..db2348b460 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -1,8 +1,11 @@ #!/usr/bin/env python3 import argparse +import fcntl import os +import struct import subprocess +import sys def gather_name(args): @@ -38,6 +41,31 @@ def gather_cpuid_leaves(args): yield line.strip() +def gather_msr(): + IA32_ARCH_CAPABILITIES_MSR = 0x10a + KVM_GET_MSRS = 0xc008ae88 + + try: + with open("/dev/cpu/0/msr", "rb") as f: + f.seek(IA32_ARCH_CAPABILITIES_MSR) + buf = f.read(8) + msr = struct.unpack("=Q", buf)[0] + return "", {IA32_ARCH_CAPABILITIES_MSR: msr} + except IOError as e: + print("Warning: {}".format(e), file=sys.stderr) + + try: + bufIn = struct.pack("=LLLLQ", 1, 0, IA32_ARCH_CAPABILITIES_MSR, 0, 0) + with open("/dev/kvm", "rb") as f: + bufOut = fcntl.ioctl(f, KVM_GET_MSRS, bufIn) + msr = struct.unpack("=LLLLQ", bufOut)[4] + return " via KVM", {IA32_ARCH_CAPABILITIES_MSR: msr} + except IOError as e: + print("Warning: {}".format(e), file=sys.stderr) + + return None, {} + + def main(): parser = argparse.ArgumentParser(description="Gather cpu test data") parser.add_argument( @@ -61,6 +89,12 @@ def main(): print(" {}".format(leave)) print() + via, msr = gather_msr() + if via is not None: + print("MSR{}:".format(via)) + for key, value in sorted(msr.items()): + print(" 0x{:x}: 0x{:016x}\n".format(int(key), value)) + print(end="", flush=True) os.environ["CPU_GATHER_PY"] = "true" subprocess.check_call("./cpu-gather.sh") diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index f84215e777..427b81a64b 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -5,46 +5,6 @@ if [ -z "${CPU_GATHER_PY}" ]; then exit 1 fi -python3 <<EOF -from struct import pack, unpack -from fcntl import ioctl -import sys, errno - -IA32_ARCH_CAPABILITIES_MSR = 0x10a -KVM_GET_MSRS = 0xc008ae88 - -def print_msr(msr, via=None): - if via is None: - print("MSR:") - else: - print("MSR via %s:" % via) - print(" 0x%x: 0x%016x" % (IA32_ARCH_CAPABILITIES_MSR, msr)) - print() - -try: - fd = open("/dev/cpu/0/msr", "rb") - fd.seek(IA32_ARCH_CAPABILITIES_MSR) - buf = fd.read(8) - msr = unpack("=Q", buf)[0] - - print_msr(msr) - sys.exit(0) -except IOError as e: - # The MSR is not supported on the host - if e.errno == errno.EIO: - sys.exit(0) - -try: - fd = open("/dev/kvm", "r") - bufIn = pack("=LLLLQ", 1, 0, IA32_ARCH_CAPABILITIES_MSR, 0, 0) - bufOut = ioctl(fd, KVM_GET_MSRS, bufIn) - msr = unpack("=LLLLQ", bufOut)[4] - - print_msr(msr, via="KVM") -except IOError as e: - pass -EOF - qemu=qemu-system-x86_64 for cmd in /usr/bin/$qemu /usr/bin/qemu-kvm /usr/libexec/qemu-kvm; do if [[ -x $cmd ]]; then -- 2.26.2

Michal Prívozník

Friday, 18 December Fri, 18 Dec

12:08 p.m.

New subject: [libvirt PATCH 07/29] cpu-gather: Move msr decoding to new script

On 12/15/20 5:24 PM, Tim Wiederhake wrote:

...

Previously, if /dev/cpu/0/msr was not present, then no warning was written. Now I get this warning, fortunately on stderr so cpu-parse.sh is not affected. I'm not saying it's a bad thing, these scripts are expected to be called by advanced users (if not developers) and this warning may shed more light. Just want to point out it was noticed ;-)

...

+ + try: + bufIn = struct.pack("=LLLLQ", 1, 0, IA32_ARCH_CAPABILITIES_MSR, 0, 0) + with open("/dev/kvm", "rb") as f: + bufOut = fcntl.ioctl(f, KVM_GET_MSRS, bufIn) + msr = struct.unpack("=LLLLQ", bufOut)[4] + return " via KVM", {IA32_ARCH_CAPABILITIES_MSR: msr} + except IOError as e: + print("Warning: {}".format(e), file=sys.stderr) + + return None, {} + +

Michal

Tim Wiederhake

Tuesday, 15 December Tue, 15 Dec

10:24 a.m.

New subject: [libvirt PATCH 08/29] cpu-gather: Move qemu detection to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 17 +++++++++++++++++ tests/cputestdata/cpu-gather.sh | 8 -------- 2 files changed, 17 insertions(+), 8 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index db2348b460..7f9479f78d 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -77,9 +77,25 @@ def main(): metavar="PATH", help="Path to 'cpuid' utility. " "If unset, the first executable 'cpuid' in $PATH is used.") + parser.add_argument( + "--path-to-qemu", + metavar="PATH", + help="Path to qemu. " + "If unset, will try '/usr/bin/qemu-system-x86_64', " + "'/usr/bin/qemu-kvm', and '/usr/libexec/qemu-kvm'.") args = parser.parse_args() + if not args.path_to_qemu: + args.path_to_qemu = "qemu-system-x86_64" + search = [ + "/usr/bin/qemu-system-x86_64", + "/usr/bin/qemu-kvm", + "/usr/libexec/qemu-kvm"] + for f in search: + if os.path.isfile(f): + args.path_to_qemu = f + name = gather_name(args) print("model name\t: {}".format(name)) @@ -97,6 +113,7 @@ def main(): print(end="", flush=True) os.environ["CPU_GATHER_PY"] = "true" + os.environ["qemu"] = args.path_to_qemu subprocess.check_call("./cpu-gather.sh") diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index 427b81a64b..4b4ac1a47c 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -5,14 +5,6 @@ if [ -z "${CPU_GATHER_PY}" ]; then exit 1 fi -qemu=qemu-system-x86_64 -for cmd in /usr/bin/$qemu /usr/bin/qemu-kvm /usr/libexec/qemu-kvm; do - if [[ -x $cmd ]]; then - qemu=$cmd - break - fi -done - qom_get() { path='/machine/unattached/device[0]' -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 09/29] cpu-gather: Move static model expansion to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 45 +++++++++++++++++++++++++++++++++ tests/cputestdata/cpu-gather.sh | 7 ----- 2 files changed, 45 insertions(+), 7 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 7f9479f78d..e4a82fc949 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -2,6 +2,7 @@ import argparse import fcntl +import json import os import struct import subprocess @@ -66,6 +67,47 @@ def gather_msr(): return None, {} +def call_qemu(qemu, qmp_cmds): + cmd = [ + qemu, + "-machine", "accel=kvm", + "-cpu", "host", + "-nodefaults", + "-nographic", + "-qmp", "stdio"] + + stdin = list() + stdin.append("{\"execute\": \"qmp_capabilities\"}") + stdin.extend([json.dumps(o) for o in qmp_cmds]) + stdin.append("{\"execute\": \"quit\"}") + + try: + output = subprocess.check_output( + cmd, + universal_newlines=True, + input="\n".join(stdin)) + except subprocess.CalledProcessError: + exit("Error: Non-zero exit code from '{}'.".format(qemu)) + except FileNotFoundError: + exit("Error: File not found: '{}'.".format(qemu)) + + return output + + +def gather_static_model(args): + output = call_qemu(args.path_to_qemu, [ + { + "execute": "query-cpu-model-expansion", + "arguments": + { + "type": "static", + "model": {"name": "host"} + }, + "id": "model-expansion" + }]) + return output + + def main(): parser = argparse.ArgumentParser(description="Gather cpu test data") parser.add_argument( @@ -111,9 +153,12 @@ def main(): for key, value in sorted(msr.items()): print(" 0x{:x}: 0x{:016x}\n".format(int(key), value)) + static_model = gather_static_model(args) + print(end="", flush=True) os.environ["CPU_GATHER_PY"] = "true" os.environ["qemu"] = args.path_to_qemu + os.environ["model"] = static_model subprocess.check_call("./cpu-gather.sh") diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index 4b4ac1a47c..af0f40b61c 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -21,13 +21,6 @@ model_expansion() '{"type":"'"$mode"'","model":'"$model"'},"id":"model-expansion"}' } -model=$( - $qemu -machine accel=kvm -cpu host -nodefaults -nographic -qmp stdio <<EOF -{"execute":"qmp_capabilities"} -$(model_expansion static '{"name":"host"}') -{"execute":"quit"} -EOF -) model=$( echo "$model" | \ sed -ne 's/^{"return": {"model": {$.*{.*}$}}, .*/{\1}/p' -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 10/29] cpu-gather: Move static model extraction to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 13 ++++++++++--- tests/cputestdata/cpu-gather.sh | 5 ----- 2 files changed, 10 insertions(+), 8 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index e4a82fc949..0c8d8e93d0 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -91,7 +91,9 @@ def call_qemu(qemu, qmp_cmds): except FileNotFoundError: exit("Error: File not found: '{}'.".format(qemu)) - return output + for line in output.split("\n"): + if line: + yield json.loads(line) def gather_static_model(args): @@ -105,7 +107,12 @@ def gather_static_model(args): }, "id": "model-expansion" }]) - return output + + for o in output: + if o.get("id") == "model-expansion": + return o["return"]["model"] + + return None def main(): @@ -158,7 +165,7 @@ def main(): print(end="", flush=True) os.environ["CPU_GATHER_PY"] = "true" os.environ["qemu"] = args.path_to_qemu - os.environ["model"] = static_model + os.environ["model"] = json.dumps(static_model) if static_model else "" subprocess.check_call("./cpu-gather.sh") diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index af0f40b61c..726f013908 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -21,11 +21,6 @@ model_expansion() '{"type":"'"$mode"'","model":'"$model"'},"id":"model-expansion"}' } -model=$( - echo "$model" | \ - sed -ne 's/^{"return": {"model": {$.*{.*}$}}, .*/{\1}/p' -) - $qemu -machine accel=kvm -cpu host -nodefaults -nographic -qmp stdio <<EOF {"execute":"qmp_capabilities"} $( -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 11/29] cpu-gather: Move simple model extraction to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 60 +++++++++++++++++++++++++++++++++ tests/cputestdata/cpu-gather.sh | 13 ------- 2 files changed, 60 insertions(+), 13 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 0c8d8e93d0..5c03d226b6 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -115,6 +115,63 @@ def gather_static_model(args): return None +def gather_full_model(args, static_model): + if static_model: + return [] + else: + return call_qemu(args.path_to_qemu, [ + { + "execute": "qom-get", + "arguments": + { + "path": "/machine/unattached/device[0]", + "property": "feature-words" + }, + "id": "feature-words" + }, + { + "execute": "qom-get", + "arguments": + { + "path": "/machine/unattached/device[0]", + "property": "family" + }, + "id": "family" + }, + { + "execute": "qom-get", + "arguments": + { + "path": "/machine/unattached/device[0]", + "property": "model" + }, + "id": "model" + }, + { + "execute": "qom-get", + "arguments": + { + "path": "/machine/unattached/device[0]", + "property": "stepping" + }, + "id": "stepping" + }, + { + "execute": "qom-get", + "arguments": + { + "path": "/machine/unattached/device[0]", + "property": "model-id" + }, + "id": "model-id" + }, + { + "execute": "query-cpu-definitions", + "id": "definitions" + } + ]) + + def main(): parser = argparse.ArgumentParser(description="Gather cpu test data") parser.add_argument( @@ -161,6 +218,9 @@ def main(): print(" 0x{:x}: 0x{:016x}\n".format(int(key), value)) static_model = gather_static_model(args) + model = gather_full_model(args, static_model) + for o in model: + print(json.dumps(o)) print(end="", flush=True) os.environ["CPU_GATHER_PY"] = "true" diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index 726f013908..05faf14a96 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -5,13 +5,6 @@ if [ -z "${CPU_GATHER_PY}" ]; then exit 1 fi -qom_get() -{ - path='/machine/unattached/device[0]' - echo '{"execute":"qom-get","arguments":{"path":"'$path'",' \ - '"property":"'$1'"},"id":"'$1'"}' -} - model_expansion() { mode=$1 @@ -26,12 +19,6 @@ $qemu -machine accel=kvm -cpu host -nodefaults -nographic -qmp stdio <<EOF $( if [ "x$model" != x ]; then model_expansion full "$model" - else - qom_get feature-words - qom_get family - qom_get model - qom_get stepping - qom_get model-id fi ) {"execute":"query-cpu-definitions","id":"definitions"} -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 12/29] cpu-gather: Move full model extraction to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 16 +++++++++++++++- tests/cputestdata/cpu-gather.sh | 20 -------------------- 2 files changed, 15 insertions(+), 21 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 5c03d226b6..ea1d42f1ec 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -117,7 +117,21 @@ def gather_static_model(args): def gather_full_model(args, static_model): if static_model: - return [] + return call_qemu(args.path_to_qemu, [ + { + "execute": "query-cpu-model-expansion", + "arguments": + { + "type": "full", + "model": static_model + }, + "id": "model-expansion" + }, + { + "execute": "query-cpu-definitions", + "id": "definitions" + } + ]) else: return call_qemu(args.path_to_qemu, [ { diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh index 05faf14a96..5cef26c8e6 100755 --- a/tests/cputestdata/cpu-gather.sh +++ b/tests/cputestdata/cpu-gather.sh @@ -4,23 +4,3 @@ if [ -z "${CPU_GATHER_PY}" ]; then echo >&2 "Do not call this script directly. Use 'cpu-gather.py' instead." exit 1 fi - -model_expansion() -{ - mode=$1 - model=$2 - - echo '{"execute":"query-cpu-model-expansion","arguments":' \ - '{"type":"'"$mode"'","model":'"$model"'},"id":"model-expansion"}' -} - -$qemu -machine accel=kvm -cpu host -nodefaults -nographic -qmp stdio <<EOF -{"execute":"qmp_capabilities"} -$( - if [ "x$model" != x ]; then - model_expansion full "$model" - fi -) -{"execute":"query-cpu-definitions","id":"definitions"} -{"execute":"quit"} -EOF -- 2.26.2

Tim Wiederhake

10:24 a.m.

New subject: [libvirt PATCH 13/29] cpu-gather: Merge model gathering logic

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 13 ++++--------- 1 file changed, 4 insertions(+), 9 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index ea1d42f1ec..c639c433d2 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -96,7 +96,7 @@ def call_qemu(qemu, qmp_cmds): yield json.loads(line) -def gather_static_model(args): +def gather_model(args): output = call_qemu(args.path_to_qemu, [ { "execute": "query-cpu-model-expansion", @@ -108,14 +108,11 @@ def gather_static_model(args): "id": "model-expansion" }]) + static_model = None for o in output: if o.get("id") == "model-expansion": - return o["return"]["model"] + static_model = o["return"]["model"] - return None - - -def gather_full_model(args, static_model): if static_model: return call_qemu(args.path_to_qemu, [ { @@ -231,15 +228,13 @@ def main(): for key, value in sorted(msr.items()): print(" 0x{:x}: 0x{:016x}\n".format(int(key), value)) - static_model = gather_static_model(args) - model = gather_full_model(args, static_model) + model = gather_model(args) for o in model: print(json.dumps(o)) print(end="", flush=True) os.environ["CPU_GATHER_PY"] = "true" os.environ["qemu"] = args.path_to_qemu - os.environ["model"] = json.dumps(static_model) if static_model else "" subprocess.check_call("./cpu-gather.sh") -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 14/29] cpu-gather: Delete old script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 5 ----- tests/cputestdata/cpu-gather.sh | 6 ------ 2 files changed, 11 deletions(-) delete mode 100755 tests/cputestdata/cpu-gather.sh diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index c639c433d2..9c609aa6e9 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -232,11 +232,6 @@ def main(): for o in model: print(json.dumps(o)) - print(end="", flush=True) - os.environ["CPU_GATHER_PY"] = "true" - os.environ["qemu"] = args.path_to_qemu - subprocess.check_call("./cpu-gather.sh") - if __name__ == "__main__": main() diff --git a/tests/cputestdata/cpu-gather.sh b/tests/cputestdata/cpu-gather.sh deleted file mode 100755 index 5cef26c8e6..0000000000 --- a/tests/cputestdata/cpu-gather.sh +++ /dev/null @@ -1,6 +0,0 @@ -#!/bin/bash - -if [ -z "${CPU_GATHER_PY}" ]; then - echo >&2 "Do not call this script directly. Use 'cpu-gather.py' instead." - exit 1 -fi -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 15/29] cpu-gather: Separate data input and output

This is a preparatory step to replace the output format with something more readable. Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 50 +++++++++++++++++++++------------ 1 file changed, 32 insertions(+), 18 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 9c609aa6e9..005d1019b6 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -183,6 +183,36 @@ def gather_model(args): ]) +def gather(args): + result = dict() + result["name"] = gather_name(args) + result["leaves"] = list(gather_cpuid_leaves(args)) + result["via"], result["msr"] = gather_msr() + result["model"] = list(gather_model(args)) + return result + + +def output_to_text(data): + output = list() + + output.append("model name\t: {}".format(data["name"])) + + output.append("CPU:") + for leave in data["leaves"]: + output.append(" {}".format(leave)) + output.append("") + + if data["via"] is not None: + output.append("MSR{}:".format(data["via"])) + for key, value in sorted(data["msr"].items()): + output.append(" 0x{:x}: 0x{:016x}\n".format(int(key), value)) + + for o in data["model"]: + output.append(json.dumps(o)) + + return "\n".join(output) + + def main(): parser = argparse.ArgumentParser(description="Gather cpu test data") parser.add_argument( @@ -213,24 +243,8 @@ def main(): if os.path.isfile(f): args.path_to_qemu = f - name = gather_name(args) - print("model name\t: {}".format(name)) - - leaves = gather_cpuid_leaves(args) - print("CPU:") - for leave in leaves: - print(" {}".format(leave)) - print() - - via, msr = gather_msr() - if via is not None: - print("MSR{}:".format(via)) - for key, value in sorted(msr.items()): - print(" 0x{:x}: 0x{:016x}\n".format(int(key), value)) - - model = gather_model(args) - for o in model: - print(json.dumps(o)) + data = gather(args) + print(output_to_text(data)) if __name__ == "__main__": -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 16/29] cpu-parse: Wrap with python script

This changes the invocation from ./cpu-gather.py | ./cpu-parse.sh to ./cpu-gather.py [--gather] | ./cpu-gather.py --parse Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 29 +++++++++++++++++++++++++++-- tests/cputestdata/cpu-parse.sh | 6 ++++-- 2 files changed, 31 insertions(+), 4 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 005d1019b6..83f175d342 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -192,6 +192,15 @@ def gather(args): return result +def parse(args): + os.environ["CPU_GATHER_PY"] = "true" + output = subprocess.check_output( + "./cpu-parse.sh", + stderr=subprocess.STDOUT, + universal_newlines=True) + print(output) + + def output_to_text(data): output = list() @@ -231,8 +240,21 @@ def main(): "If unset, will try '/usr/bin/qemu-system-x86_64', " "'/usr/bin/qemu-kvm', and '/usr/libexec/qemu-kvm'.") + mode = parser.add_mutually_exclusive_group() + mode.add_argument( + "--gather", + action="store_true", + help="Acquire data on target system. This is the default.") + mode.add_argument( + "--parse", + action="store_true", + help="Parse data for libvirt use.") + args = parser.parse_args() + if not args.gather and not args.parse: + args.gather = True + if not args.path_to_qemu: args.path_to_qemu = "qemu-system-x86_64" search = [ @@ -243,8 +265,11 @@ def main(): if os.path.isfile(f): args.path_to_qemu = f - data = gather(args) - print(output_to_text(data)) + if args.gather: + data = gather(args) + print(output_to_text(data)) + else: + parse(args) if __name__ == "__main__": diff --git a/tests/cputestdata/cpu-parse.sh b/tests/cputestdata/cpu-parse.sh index 7501c57cba..fa4344b6ad 100755 --- a/tests/cputestdata/cpu-parse.sh +++ b/tests/cputestdata/cpu-parse.sh @@ -1,7 +1,9 @@ #!/bin/bash -# Usage: -# ./cpu-gather.sh | ./cpu-parse.sh +if [ -z "${CPU_GATHER_PY}" ]; then + echo >&2 "Do not call this script directly. Use 'cpu-gather.py' instead." + exit 1 +fi data=`cat` -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 17/29] cpu-gather: Transport data as json

More reliable, easier to parse, easier to edit. Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 83f175d342..7c51a47353 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -193,9 +193,12 @@ def gather(args): def parse(args): + data = json.load(sys.stdin) + os.environ["CPU_GATHER_PY"] = "true" output = subprocess.check_output( "./cpu-parse.sh", + input=output_to_text(data), stderr=subprocess.STDOUT, universal_newlines=True) print(output) @@ -267,7 +270,7 @@ def main(): if args.gather: data = gather(args) - print(output_to_text(data)) + json.dump(data, sys.stdout, indent=2) else: parse(args) -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 18/29] cpu-parse: Move model name detection to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 1 + tests/cputestdata/cpu-parse.sh | 2 -- 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 7c51a47353..0b1019456c 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -196,6 +196,7 @@ def parse(args): data = json.load(sys.stdin) os.environ["CPU_GATHER_PY"] = "true" + os.environ["model"] = data["name"] output = subprocess.check_output( "./cpu-parse.sh", input=output_to_text(data), diff --git a/tests/cputestdata/cpu-parse.sh b/tests/cputestdata/cpu-parse.sh index fa4344b6ad..5b8e57e427 100755 --- a/tests/cputestdata/cpu-parse.sh +++ b/tests/cputestdata/cpu-parse.sh @@ -7,8 +7,6 @@ fi data=`cat` -model=`sed -ne '/^model name[ ]*:/ {s/^[^:]*: $.*$/\1/p; q}' <<<"$data"` - fname=`sed -e 's/^ *//; s/ *$//; s/[ -]\+ \+/ /g; -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 19/29] cpu-parse: Move file name generation to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 18 ++++++++++++++++++ tests/cputestdata/cpu-parse.sh | 13 ------------- 2 files changed, 18 insertions(+), 13 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 0b1019456c..1a15cc1ff0 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -4,6 +4,7 @@ import argparse import fcntl import json import os +import re import struct import subprocess import sys @@ -192,11 +193,28 @@ def gather(args): return result +def parse_filename(data): + filename = data["name"].strip() + filename = re.sub("[ -]+ +", " ", filename) + filename = re.sub("\$([Rr]|[Tt][Mm])\$", "", filename) + filename = re.sub(".*(Intel|AMD) ", "", filename) + filename = re.sub(" (Duo|Quad|II X[0-9]+)", " ", filename) + filename = re.sub(" (CPU|Processor)", "", filename) + filename = re.sub(" @.*", "", filename) + filename = re.sub(" APU .*", "", filename) + filename = re.sub(" SE$", "", filename) + filename = re.sub(" ", "-", filename) + return "x86_64-cpuid-{}".format(filename) + + def parse(args): data = json.load(sys.stdin) + filename = parse_filename(data) + os.environ["CPU_GATHER_PY"] = "true" os.environ["model"] = data["name"] + os.environ["fname"] = filename output = subprocess.check_output( "./cpu-parse.sh", input=output_to_text(data), diff --git a/tests/cputestdata/cpu-parse.sh b/tests/cputestdata/cpu-parse.sh index 5b8e57e427..2a41897538 100755 --- a/tests/cputestdata/cpu-parse.sh +++ b/tests/cputestdata/cpu-parse.sh @@ -7,19 +7,6 @@ fi data=`cat` -fname=`sed -e 's/^ *//; - s/ *$//; - s/[ -]\+ \+/ /g; - s/($[Rr]\|[Tt][Mm]$)//g; - s/.*$Intel\|AMD$ //; - s/ $Duo\|Quad\|II X[0-9]\+$ / /; - s/ $CPU\|Processor$\>//; - s/ @.*//; - s/ APU .*//; - s/ SE$//; - s/ /-/g' <<<"$model"` -fname="x86_64-cpuid-$fname" - xml() { hex='$0x[0-9a-f]\+$' -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 20/29] cpu-parse: Move xml output to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 40 +++++++++++++++++++++++++++++++++ tests/cputestdata/cpu-parse.sh | 18 --------------- 2 files changed, 40 insertions(+), 18 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 1a15cc1ff0..bc5c7dbb15 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -207,10 +207,50 @@ def parse_filename(data): return "x86_64-cpuid-{}".format(filename) +def output_xml(data, filename): + leave_pattern = re.compile( + "^\\s*" + "(0x[0-9a-f]+)\\s*" + "(0x[0-9a-f]+):\\s*" + "eax=(0x[0-9a-f]+)\\s*" + "ebx=(0x[0-9a-f]+)\\s*" + "ecx=(0x[0-9a-f]+)\\s*" + "edx=(0x[0-9a-f]+)\\s*$") + + leave_template = \ + " <cpuid" \ + " eax_in='{}'" \ + " ecx_in='{}'" \ + " eax='{}'" \ + " ebx='{}'" \ + " ecx='{}'" \ + " edx='{}'" \ + "/>\n" + + msr_template = " <msr index='0x{:x}' edx='0x{:08x}' eax='0x{:08x}'/>\n" + + print(filename) + with open(filename, "wt") as f: + f.write("\n".format(data["name"])) + f.write("<cpudata arch='x86'>\n") + for line in data["leaves"]: + match = leave_pattern.match(line) + f.write(leave_template.format(*match.groups())) + for key, value in sorted(data["msr"].items()): + f.write(msr_template.format( + int(key), + 0xffffffff & (value >> 32), + 0xffffffff & (value >> 0))) + f.write("</cpudata>\n") + + def parse(args): data = json.load(sys.stdin) filename = parse_filename(data) + filename_xml = "{}.xml".format(filename) + + output_xml(data, filename_xml) os.environ["CPU_GATHER_PY"] = "true" os.environ["model"] = data["name"] diff --git a/tests/cputestdata/cpu-parse.sh b/tests/cputestdata/cpu-parse.sh index 2a41897538..84d37d0df4 100755 --- a/tests/cputestdata/cpu-parse.sh +++ b/tests/cputestdata/cpu-parse.sh @@ -7,21 +7,6 @@ fi data=`cat` -xml() -{ - hex='$0x[0-9a-f]\+$' - matchCPUID="$hex $hex: eax=$hex ebx=$hex ecx=$hex edx=$hex" - substCPUID="<cpuid eax_in='\\1' ecx_in='\\2' eax='\\3' ebx='\\4' ecx='\\5' edx='\\6'\\/>" - - matchMSR="$hex: $hex$.......[0-9a-f]$" - substMSR="<msr index='\\1' edx='\\2' eax='0x\\3'\\/>" - - echo "" - echo "<cpudata arch='x86'>" - sed -ne "s/^ *$matchCPUID$/ $substCPUID/p; s/^ *$matchMSR$/ $substMSR/p" - echo "</cpudata>" -} - json() { first=true @@ -36,9 +21,6 @@ json() done } -xml <<<"$data" >$fname.xml -echo $fname.xml - json <<<"$data" >$fname.json if [[ -s $fname.json ]]; then echo $fname.json -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 21/29] cpu-parse: Move json output to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 30 ++++++++++++++++++++++++++++++ tests/cputestdata/cpu-parse.sh | 16 ---------------- tests/cputestdata/cpu-reformat.py | 9 --------- 3 files changed, 30 insertions(+), 25 deletions(-) delete mode 100755 tests/cputestdata/cpu-reformat.py diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index bc5c7dbb15..2c3f39ab35 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -244,13 +244,43 @@ def output_xml(data, filename): f.write("</cpudata>\n") +def output_json(data, filename): + replies = list() + for reply in data["model"]: + if "QMP" in reply: + continue + if "timestamp" in reply: + continue + if "return" in reply and not reply["return"]: + continue + replies.append(reply) + + if not replies: + return + + if "model-expansion" not in [reply.get("id") for reply in replies]: + exit( + "Error: Missing query-cpu-model-expansion reply in " + "{}".format(filename)) + + print(filename) + with open(filename, "wt") as f: + for reply in replies: + if reply is not replies[0]: + f.write("\n") + json.dump(reply, f, indent=2) + f.write("\n") + + def parse(args): data = json.load(sys.stdin) filename = parse_filename(data) filename_xml = "{}.xml".format(filename) + filename_json = "{}.json".format(filename) output_xml(data, filename_xml) + output_json(data, filename_json) os.environ["CPU_GATHER_PY"] = "true" os.environ["model"] = data["name"] diff --git a/tests/cputestdata/cpu-parse.sh b/tests/cputestdata/cpu-parse.sh index 84d37d0df4..aeb6e4e07f 100755 --- a/tests/cputestdata/cpu-parse.sh +++ b/tests/cputestdata/cpu-parse.sh @@ -7,23 +7,7 @@ fi data=`cat` -json() -{ - first=true - sed -ne '/{"QMP".*/d; - /{"return": {}}/d; - /{"timestamp":.*/d; - /^{/p' <<<"$data" | \ - while read; do - $first || echo - first=false - $(dirname $0)/cpu-reformat.py <<<"$REPLY" - done -} - -json <<<"$data" >$fname.json if [[ -s $fname.json ]]; then - echo $fname.json if ! grep -q model-expansion $fname.json; then echo "Missing query-cpu-model-expansion reply in $name.json" >&2 exit 1 diff --git a/tests/cputestdata/cpu-reformat.py b/tests/cputestdata/cpu-reformat.py deleted file mode 100755 index fcc6b8ab41..0000000000 --- a/tests/cputestdata/cpu-reformat.py +++ /dev/null @@ -1,9 +0,0 @@ -#!/usr/bin/env python3 - -import sys -import json - -dec = json.JSONDecoder() -data, pos = dec.raw_decode(sys.stdin.read()) -json.dump(data, sys.stdout, indent=2, separators=(',', ': ')) -print("") -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 22/29] cpu-parse: Move call to cpu-cpuid.py to new script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 10 ++++++++++ tests/cputestdata/cpu-parse.sh | 10 ---------- 2 files changed, 10 insertions(+), 10 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 2c3f39ab35..373f179c8d 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -282,6 +282,16 @@ def parse(args): output_xml(data, filename_xml) output_json(data, filename_json) + if not os.path.isfile(filename_json): + return + if os.path.getsize(filename_json) == 0: + return + + output = subprocess.check_output( + ["./cpu-cpuid.py", "diff", filename_json], + universal_newlines=True) + print(output) + os.environ["CPU_GATHER_PY"] = "true" os.environ["model"] = data["name"] os.environ["fname"] = filename diff --git a/tests/cputestdata/cpu-parse.sh b/tests/cputestdata/cpu-parse.sh index aeb6e4e07f..eea15ded2e 100755 --- a/tests/cputestdata/cpu-parse.sh +++ b/tests/cputestdata/cpu-parse.sh @@ -6,13 +6,3 @@ if [ -z "${CPU_GATHER_PY}" ]; then fi data=`cat` - -if [[ -s $fname.json ]]; then - if ! grep -q model-expansion $fname.json; then - echo "Missing query-cpu-model-expansion reply in $name.json" >&2 - exit 1 - fi - $(dirname $0)/cpu-cpuid.py diff $fname.json -else - rm $fname.json -fi -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 23/29] cpu-parse: Delete old script

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 31 ------------------------------- tests/cputestdata/cpu-parse.sh | 8 -------- 2 files changed, 39 deletions(-) delete mode 100755 tests/cputestdata/cpu-parse.sh diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 373f179c8d..b6acae9eb7 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -292,37 +292,6 @@ def parse(args): universal_newlines=True) print(output) - os.environ["CPU_GATHER_PY"] = "true" - os.environ["model"] = data["name"] - os.environ["fname"] = filename - output = subprocess.check_output( - "./cpu-parse.sh", - input=output_to_text(data), - stderr=subprocess.STDOUT, - universal_newlines=True) - print(output) - - -def output_to_text(data): - output = list() - - output.append("model name\t: {}".format(data["name"])) - - output.append("CPU:") - for leave in data["leaves"]: - output.append(" {}".format(leave)) - output.append("") - - if data["via"] is not None: - output.append("MSR{}:".format(data["via"])) - for key, value in sorted(data["msr"].items()): - output.append(" 0x{:x}: 0x{:016x}\n".format(int(key), value)) - - for o in data["model"]: - output.append(json.dumps(o)) - - return "\n".join(output) - def main(): parser = argparse.ArgumentParser(description="Gather cpu test data") diff --git a/tests/cputestdata/cpu-parse.sh b/tests/cputestdata/cpu-parse.sh deleted file mode 100755 index eea15ded2e..0000000000 --- a/tests/cputestdata/cpu-parse.sh +++ /dev/null @@ -1,8 +0,0 @@ -#!/bin/bash - -if [ -z "${CPU_GATHER_PY}" ]; then - echo >&2 "Do not call this script directly. Use 'cpu-gather.py' instead." - exit 1 -fi - -data=`cat` -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 24/29] cpu-gather: Ignore empty responses from qemu

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index b6acae9eb7..10dbeb87cf 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -93,8 +93,12 @@ def call_qemu(qemu, qmp_cmds): exit("Error: File not found: '{}'.".format(qemu)) for line in output.split("\n"): - if line: - yield json.loads(line) + if not line: + continue + response = json.loads(line) + if "return" in response and not response["return"]: + continue + yield response def gather_model(args): -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 25/29] cpu-gather: Ignore shutdown messages from qemu

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 10dbeb87cf..49c72df320 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -98,6 +98,8 @@ def call_qemu(qemu, qmp_cmds): response = json.loads(line) if "return" in response and not response["return"]: continue + if response.get("event") == "SHUTDOWN": + continue yield response -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 26/29] cpu-gather: Parse cpuid leaves early

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 48 ++++++++++++++++++--------------- 1 file changed, 26 insertions(+), 22 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 49c72df320..fe660c486e 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -24,6 +24,15 @@ def gather_name(args): def gather_cpuid_leaves(args): + leave_pattern = re.compile( + "^\\s*" + "(0x[0-9a-f]+)\\s*" + "(0x[0-9a-f]+):\\s*" + "eax=(0x[0-9a-f]+)\\s*" + "ebx=(0x[0-9a-f]+)\\s*" + "ecx=(0x[0-9a-f]+)\\s*" + "edx=(0x[0-9a-f]+)\\s*$") + cpuid = args.path_to_cpuid or "cpuid" try: output = subprocess.check_output( @@ -36,11 +45,16 @@ def gather_cpuid_leaves(args): "'http://www.etallen.com/cpuid.html'.".format(e.filename)) for line in output.split("\n"): - if not line: + match = leave_pattern.match(line) + if not match: continue - if line == "CPU:": - continue - yield line.strip() + yield { + "eax_in": int(match.group(1), 0), + "ecx_in": int(match.group(2), 0), + "eax": int(match.group(3), 0), + "ebx": int(match.group(4), 0), + "ecx": int(match.group(5), 0), + "edx": int(match.group(6), 0)} def gather_msr(): @@ -214,23 +228,14 @@ def parse_filename(data): def output_xml(data, filename): - leave_pattern = re.compile( - "^\\s*" - "(0x[0-9a-f]+)\\s*" - "(0x[0-9a-f]+):\\s*" - "eax=(0x[0-9a-f]+)\\s*" - "ebx=(0x[0-9a-f]+)\\s*" - "ecx=(0x[0-9a-f]+)\\s*" - "edx=(0x[0-9a-f]+)\\s*$") - leave_template = \ " <cpuid" \ - " eax_in='{}'" \ - " ecx_in='{}'" \ - " eax='{}'" \ - " ebx='{}'" \ - " ecx='{}'" \ - " edx='{}'" \ + " eax_in='0x{0[eax_in]:08x}'" \ + " ecx_in='0x{0[ecx_in]:02x}'" \ + " eax='0x{0[eax]:08x}'" \ + " ebx='0x{0[ebx]:08x}'" \ + " ecx='0x{0[ecx]:08x}'" \ + " edx='0x{0[edx]:08x}'" \ "/>\n" msr_template = " <msr index='0x{:x}' edx='0x{:08x}' eax='0x{:08x}'/>\n" @@ -239,9 +244,8 @@ def output_xml(data, filename): with open(filename, "wt") as f: f.write("\n".format(data["name"])) f.write("<cpudata arch='x86'>\n") - for line in data["leaves"]: - match = leave_pattern.match(line) - f.write(leave_template.format(*match.groups())) + for leave in data["leaves"]: + f.write(leave_template.format(leave)) for key, value in sorted(data["msr"].items()): f.write(msr_template.format( int(key), -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 27/29] cpu-gather: Allow gathering and parsing data in one step.

Make ./cpu-gather.py --gather --parse an alias of ./cpu-gather.py [--gather] | ./cpu-gather.py --parse Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index fe660c486e..46997c8a48 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -282,9 +282,7 @@ def output_json(data, filename): f.write("\n") -def parse(args): - data = json.load(sys.stdin) - +def parse(data): filename = parse_filename(data) filename_xml = "{}.xml".format(filename) filename_json = "{}.json".format(filename) @@ -320,16 +318,16 @@ def main(): help="Path to qemu. " "If unset, will try '/usr/bin/qemu-system-x86_64', " "'/usr/bin/qemu-kvm', and '/usr/libexec/qemu-kvm'.") - - mode = parser.add_mutually_exclusive_group() - mode.add_argument( + parser.add_argument( "--gather", action="store_true", - help="Acquire data on target system. This is the default.") - mode.add_argument( + help="Acquire data on target system. This is the default. " + "If '--parse' is not set, outputs data on stdout.") + parser.add_argument( "--parse", action="store_true", - help="Parse data for libvirt use.") + help="Parse data for libvirt use. " + "If '--gather' is not set, expects input on stdin.") args = parser.parse_args() @@ -348,9 +346,13 @@ def main(): if args.gather: data = gather(args) - json.dump(data, sys.stdout, indent=2) - else: - parse(args) + if not args.parse: + json.dump(data, sys.stdout, indent=2) + + if args.parse: + if not args.gather: + data = json.load(sys.stdin) + parse(data) if __name__ == "__main__": -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 28/29] cpu-gather: Prepare gather_msr for reading multiple msr

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 46997c8a48..9c9eec6d93 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -58,24 +58,29 @@ def gather_cpuid_leaves(args): def gather_msr(): - IA32_ARCH_CAPABILITIES_MSR = 0x10a + msrs = dict() + addresses = [ + 0x10a, # IA32_ARCH_CAPABILITIES_MSR + ] KVM_GET_MSRS = 0xc008ae88 try: with open("/dev/cpu/0/msr", "rb") as f: - f.seek(IA32_ARCH_CAPABILITIES_MSR) - buf = f.read(8) - msr = struct.unpack("=Q", buf)[0] - return "", {IA32_ARCH_CAPABILITIES_MSR: msr} + for addr in addresses: + f.seek(addr) + buf = f.read(8) + msrs[addr] = struct.unpack("=Q", buf)[0] + return "", msrs except IOError as e: print("Warning: {}".format(e), file=sys.stderr) try: - bufIn = struct.pack("=LLLLQ", 1, 0, IA32_ARCH_CAPABILITIES_MSR, 0, 0) with open("/dev/kvm", "rb") as f: - bufOut = fcntl.ioctl(f, KVM_GET_MSRS, bufIn) - msr = struct.unpack("=LLLLQ", bufOut)[4] - return " via KVM", {IA32_ARCH_CAPABILITIES_MSR: msr} + for addr in addresses: + bufIn = struct.pack("=LLLLQ", 1, 0, addr, 0, 0) + bufOut = fcntl.ioctl(f, KVM_GET_MSRS, bufIn) + msrs[addr] = struct.unpack("=LLLLQ", bufOut)[4] + return " via KVM", msrs except IOError as e: print("Warning: {}".format(e), file=sys.stderr) -- 2.26.2

Tim Wiederhake

10:25 a.m.

New subject: [libvirt PATCH 29/29] cpu-gather: Add IA32_CORE_CAPABILITY_MSR

Signed-off-by: Tim Wiederhake <twiederh(a)redhat.com> --- tests/cputestdata/cpu-gather.py | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/cputestdata/cpu-gather.py b/tests/cputestdata/cpu-gather.py index 9c9eec6d93..a01c615504 100755 --- a/tests/cputestdata/cpu-gather.py +++ b/tests/cputestdata/cpu-gather.py @@ -61,6 +61,7 @@ def gather_msr(): msrs = dict() addresses = [ 0x10a, # IA32_ARCH_CAPABILITIES_MSR + 0xcf, # IA32_CORE_CAPABILITY_MSR ] KVM_GET_MSRS = 0xc008ae88 -- 2.26.2

Michal Prívozník

Friday, 18 December Fri, 18 Dec

12:08 p.m.

On 12/15/20 5:24 PM, Tim Wiederhake wrote:

...

Reviewed-by: Michal Privoznik <mprivozn(a)redhat.com> and will push shortly. Michal

1465

days inactive

1468

days old

devel@lists.libvirt.org

Manage subscription

32 comments

2 participants

tags (0)

participants (2)

Michal Prívozník
Tim Wiederhake

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

[libvirt PATCH 00/29] Refactor scripts in tests/cputestdata