[libvirt] [PATCH v2 2/6] tools: introduce a data driven impl of virt-host-validate

The current virt-host-validate command has a bunch of checks defined in the source code which are thus only extensible by the upstream project, or downstream code modification. The checks are implemented by a fairly simple set of rules, mostly matching the contents of files, or output from commands, against some expected state or regex. This lends itself very well to having the checks defined in data, in the form of rules which can be processed by a generic engine. This patch introduces an implementation of virt-host-validate which does exactly that, using rules defined by a set of YAML files. Parsing XML/JSON/YAML/etc from C is incredibly tedious, which has long discouraged this conversion. The size & fagility of the parsing code in C would be too high to justify the benefits of a rewrite. This new impl is thus written in Go to take advantage of its YAML encoding feature that lets you simply annotate struct fields to define an YAML parser. YAML was chosen instead of XML because the file format is more user readable, than XML, and the Go parsing code is even simpler than with XML. JSON would have been another valid option instead of YAML. JSON is said to be optimized for the simplest machine parsing at the cost of human readability. YAML is said to be optimized for human readability at the cost of more complex parsers. Interestingly YAML is a true superset of JSON so although we used a YAML parser, our code can in fact load JSON files anyway. Comparing the JSON & YAML files for some example rules, the YAML did indeed appear more readable. The new impl also has a few new features in its CLI interface. It is possible display a list of all facts that are set, instead of just the subset which have reports associated with them. For example, by default $ virt-host-validate Checking cgroup memory controller present...PASS Checking cgroup memory controller mounted...PASS Checking cgroup cpu controller present...PASS Checking cgroup cpu controller mounted...PASS Checking cgroup cpuacct controller present...PASS Checking cgroup cpuacct controller mounted...PASS ...snip... But it can be run with $ virt-host-validate -q -f Set fact 'libvirt.driver.qemu' = 'true' Set fact 'libvirt.driver.lxc' = 'true' Set fact 'libvirt.driver.parallels' = 'true' Set fact 'cpu.arch' = 'x86_64' Set fact 'os.kernel' = 'Linux' Set fact 'os.release' = '5.1.16-300.fc30.x86_64' Set fact 'os.version' = '#1 SMP Wed Jul 3 15:06:51 UTC 2019' Set fact 'os.cgroup.controller.cpuset' = 'true' Set fact 'os.cgroup.controller.cpu' = 'true' Set fact 'os.cgroup.controller.cpuacct' = 'true' ...snip... This is quite useful for generating reports to include on bug reports / support requests, as it provides much more detailed information than is included in the summary messages. The main thing lost with this new impl is support for translation, which covers two sources: - Reports associated with the facts defined in the YAML. It is unclear how we would best deal with this. Merge translations from the .po file into the .yaml files, and then ensure we pick the appropriate one to display. - Error messages when things go wrong in the code itself. This would need to use some gettext like translation system for golang. I have not investigated the options here yet This impl uses three 3rd party dependancies - github.com/spf13/pflag - replacement command line arg parsing. Go's standard CLI arg parsing uses single dash for long options and does not support short options as a distinct concept. This is a widely used drop-in replacement that does the traditional long/short opt processing. - golang.org/x/sys - this is actually distributed with Golang but is considered an optional part of the runtime, since its APIs are not platform portable. This is needed to check file access permissions and to extract uname data. - github.com/ghodss/yaml - this provides the YAML parsing API. The observant reviewer will notice the facts.go file contains 'json:' attribute annotations. This is because the YAML parser actually leverages the JSON parser internally. Signed-off-by: Daniel P. Berrangé <berrange(a)redhat.com&gt; --- tools/host-validate/go.mod | 10 + tools/host-validate/go.sum | 9 + tools/host-validate/main.go | 98 +++++ tools/host-validate/pkg/engine.go | 481 +++++++++++++++++++++ tools/host-validate/pkg/facts.go | 585 ++++++++++++++++++++++++++ tools/host-validate/pkg/facts_test.go | 50 +++ 6 files changed, 1233 insertions(+) create mode 100644 tools/host-validate/go.mod create mode 100644 tools/host-validate/go.sum create mode 100644 tools/host-validate/main.go create mode 100644 tools/host-validate/pkg/engine.go create mode 100644 tools/host-validate/pkg/facts.go create mode 100644 tools/host-validate/pkg/facts_test.go diff --git a/tools/host-validate/go.mod b/tools/host-validate/go.mod new file mode 100644 index 0000000000..fe4d56441d --- /dev/null +++ b/tools/host-validate/go.mod @@ -0,0 +1,10 @@ +module libvirt.org/host-validate + +go 1.11 + +require ( + github.com/ghodss/yaml v1.0.0 + github.com/spf13/pflag v1.0.3 + golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a + gopkg.in/yaml.v2 v2.2.2 // indirect +) diff --git a/tools/host-validate/go.sum b/tools/host-validate/go.sum new file mode 100644 index 0000000000..8f74f6a45f --- /dev/null +++ b/tools/host-validate/go.sum @@ -0,0 +1,9 @@ +github.com/ghodss/yaml v1.0.0 h1:wQHKEahhL6wmXdzwWG11gIVCkOv05bNOh+Rxn0yngAk= +github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04= +github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg= +github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4= +golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a h1:aYOabOQFp6Vj6W1F80affTUvO9UxmJRx8K0gsfABByQ= +golang.org/x/sys v0.0.0-20190813064441-fde4db37ae7a/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= +gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI= diff --git a/tools/host-validate/main.go b/tools/host-validate/main.go new file mode 100644 index 0000000000..f6dae86558 --- /dev/null +++ b/tools/host-validate/main.go @@ -0,0 +1,98 @@ +/* + * This file is part of the libvirt project + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>;. + * + * Copyright (C) 2019 Red Hat, Inc. + * + */ + +package main + +import ( + "flag" + "fmt" + "github.com/spf13/pflag" + "io/ioutil" + vl "libvirt.org/host-validate/pkg" + "os" + "path/filepath" + "strings" +) + +func main() { + var showfacts bool + var quiet bool + var rulesdir string + + pflag.BoolVarP(&showfacts, "show-facts", "f", false, "Show raw fact names and values") + pflag.BoolVarP(&quiet, "quiet", "q", false, "Don't report on fact checks") + pflag.StringVarP(&rulesdir, "rules-dir", "r", "/usr/share/libvirt/host-validate", "Directory to load validation rules from") + + pflag.CommandLine.AddGoFlagSet(flag.CommandLine) + pflag.Parse() + // Convince glog that we really have parsed CLI + flag.CommandLine.Parse([]string{}) + + if len(pflag.Args()) > 1 { + fmt.Printf("syntax: %s [OPTIONS] [DRIVER]\n", os.Args[0]) + os.Exit(1) + } + + driver := "" + if len(pflag.Args()) == 1 { + driver = pflag.Args()[0] + } + + files, err := ioutil.ReadDir(rulesdir) + if err != nil { + fmt.Printf("Unable to load rules from '%s': %s\n", rulesdir, err) + os.Exit(1) + } + var lists []vl.FactList + for _, file := range files { + path := filepath.Join(rulesdir, file.Name()) + if !strings.HasSuffix(path, ".yaml") { + continue + } + facts, err := vl.NewFactList(path) + if err != nil { + fmt.Printf("Unable to load facts '%s': %s\n", path, err) + os.Exit(1) + } + lists = append(lists, *facts) + } + + var output vl.EngineOutput + if !quiet { + output |= vl.ENGINE_OUTPUT_REPORTS + } + if showfacts { + output |= vl.ENGINE_OUTPUT_FACTS + } + + engine := vl.NewEngine(output, driver) + + failed, err := engine.Validate(vl.MergeFactLists(lists)) + if err != nil { + fmt.Printf("Unable to validate facts: %s\n", err) + os.Exit(1) + } + if failed != 0 { + os.Exit(2) + } + + os.Exit(0) +} diff --git a/tools/host-validate/pkg/engine.go b/tools/host-validate/pkg/engine.go new file mode 100644 index 0000000000..61676cdce8 --- /dev/null +++ b/tools/host-validate/pkg/engine.go @@ -0,0 +1,481 @@ +/* + * This file is part of the libvirt project + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>;. + * + * Copyright (C) 2019 Red Hat, Inc. + * + */ + +package pkg + +// This defines the logic that executes the checks defined +// for each fact that is loaded. It is responsible for printing +// out messages as the facts are processed too. + +import ( + "fmt" + "golang.org/x/sys/unix" + "io/ioutil" + "os" + "os/exec" + "regexp" + "runtime" + "strings" +) + +type Engine struct { + Facts map[string]string + Errors uint + Output EngineOutput + Driver string +} + +type EngineOutput int + +const ( + // Print the raw key, value pairs for each fact set + ENGINE_OUTPUT_FACTS = EngineOutput(1 << 0) + + // Print the human targetted reports for facts set + ENGINE_OUTPUT_REPORTS = EngineOutput(1 << 1) +) + +// Create an engine able to process a list of facts +func NewEngine(output EngineOutput, driver string) *Engine { + engine := &Engine{} + + engine.Output = output + engine.Facts = make(map[string]string) + engine.Driver = driver + + return engine +} + +// Set the value associated with a fact +func (engine *Engine) SetFact(name, value string) { + engine.Facts[name] = value + if (engine.Output & ENGINE_OUTPUT_FACTS) != 0 { + fmt.Printf("Set fact '%s' = '%s'\n", name, value) + } +} + +func (engine *Engine) EvalExpression(expr *Expression) (bool, error) { + if expr.Any != nil { + for _, subexpr := range expr.Any.Expressions { + ok, err := engine.EvalExpression(&subexpr) + if err != nil { + return false, err + } + if ok { + return true, nil + } + } + return false, nil + } else if expr.All != nil { + for _, subexpr := range expr.All.Expressions { + ok, err := engine.EvalExpression(&subexpr) + if err != nil { + return false, err + } + if !ok { + return false, nil + } + } + return true, nil + } else if expr.Fact != nil { + val, ok := engine.Facts[expr.Fact.Name] + if !ok { + return false, nil + } + if expr.Fact.Match == "regex" { + match, err := regexp.Match(expr.Fact.Value, []byte(val)) + if err != nil { + return false, err + } + return match, nil + } else if expr.Fact.Match == "exists" { + return true, nil + } else { + return val == expr.Fact.Value, nil + } + } else { + return false, fmt.Errorf("Expected expression any or all or fact") + } +} + +// Report a fact that failed to have the desired value +func (engine *Engine) Fail(fact *Fact) { + engine.Errors++ + if fact.Report == nil { + return + } + if (engine.Output & ENGINE_OUTPUT_REPORTS) != 0 { + hint := "" + if fact.Hint != nil { + hint = " (" + fact.Hint.Message + ")" + } + if fact.Report.Level == "note" { + fmt.Printf("\033[34mNOTE\033[0m%s\n", hint) + } else if fact.Report.Level == "warn" { + fmt.Printf("\033[33mWARN\033[0m%s\n", hint) + } else { + fmt.Printf("\033[31mFAIL\033[0m%s\n", hint) + } + } +} + +// Report a fact that has the desired value +func (engine *Engine) Pass(fact *Fact) { + if fact.Report == nil { + return + } + if (engine.Output & ENGINE_OUTPUT_REPORTS) != 0 { + fmt.Printf("\033[32mPASS\033[0m\n") + } +} + +func utsString(v []byte) string { + n := 0 + for i, _ := range v { + if v[i] == 0 { + break + } + n++ + } + return string(v[0:n]) +} + +// Populate the engine with values for a built-in fact +func (engine *Engine) SetValueBuiltIn(fact *Fact) error { + var uts unix.Utsname + err := unix.Uname(&uts) + if err != nil { + return err + } + + if fact.Name == "os.kernel" { + engine.SetFact(fact.Name, utsString(uts.Sysname[0:])) + } else if fact.Name == "os.release" { + engine.SetFact(fact.Name, utsString(uts.Release[0:])) + } else if fact.Name == "os.version" { + engine.SetFact(fact.Name, utsString(uts.Version[0:])) + } else if fact.Name == "cpu.arch" { + engine.SetFact(fact.Name, utsString(uts.Machine[0:])) + } else if fact.Name == "libvirt.driver" { + if engine.Driver != "" { + engine.SetFact(fact.Name+"."+engine.Driver, "true") + } else { + if runtime.GOOS == "linux" { + engine.SetFact(fact.Name+".qemu", "true") + engine.SetFact(fact.Name+".lxc", "true") + engine.SetFact(fact.Name+".parallels", "true") + } else if runtime.GOOS == "freebsd" { + engine.SetFact(fact.Name+".bhyve", "true") + } + } + } else { + return fmt.Errorf("Unknown built-in fact '%s'", fact.Name) + } + + return nil +} + +func (engine *Engine) SetValueBool(fact *Fact) error { + ok, err := engine.EvalExpression(fact.Value.Bool) + if err != nil { + return err + } + got := "true" + want := "true" + if !ok { + got = "false" + } + if fact.Report != nil && fact.Report.Pass != "" { + want = fact.Report.Pass + } + engine.SetFact(fact.Name, got) + if got == want { + engine.Pass(fact) + } else { + engine.Fail(fact) + } + return nil +} + +func unescape(val string) (string, error) { + escapes := map[rune]string{ + 'a': "\x07", + 'b': "\x08", + 'e': "\x1b", + 'f': "\x0c", + 'n': "\x0a", + 'r': "\x0d", + 't': "\x09", + 'v': "\x0b", + '\\': "\x5c", + '0': "\x00", + } + var ret string + escape := false + for _, c := range val { + if c == '\\' { + escape = true + } else if escape { + unesc, ok := escapes[c] + if !ok { + return "", fmt.Errorf("Unknown escape '\\%c'", c) + } + ret += string(unesc) + escape = false + } else { + ret += string(c) + } + } + return ret, nil +} + +func (engine *Engine) SetValueParse(fact *Fact, parse *Parse, context string, val string) error { + if parse == nil { + engine.SetFact(context, val) + return nil + } + if parse.Whitespace == "trim" { + val = strings.TrimSpace(val) + } + if parse.Scalar != nil { + if parse.Scalar.Regex != "" { + re, err := regexp.Compile(parse.Scalar.Regex) + if err != nil { + return err + } + matches := re.FindStringSubmatch(val) + if parse.Scalar.Match >= uint(len(matches)) { + val = "" + } else { + val = matches[parse.Scalar.Match] + } + } + engine.SetFact(context, val) + } else if parse.List != nil { + if val == "" { + return nil + } + sep, err := unescape(parse.List.Separator) + if err != nil { + return err + } + bits := strings.Split(val, sep) + count := uint(0) + for i, bit := range bits { + if i < int(parse.List.SkipHead) { + continue + } + if i >= (len(bits) - int(parse.List.SkipTail)) { + continue + } + subcontext := fmt.Sprintf("%s.%d", context, i) + err := engine.SetValueParse(fact, parse.List.Parse, subcontext, bit) + if err != nil { + return err + } + count++ + if count >= parse.List.Limit { + break + } + } + } else if parse.Set != nil { + if val == "" { + return nil + } + sep, err := unescape(parse.Set.Separator) + if err != nil { + return err + } + bits := strings.Split(val, sep) + for i, bit := range bits { + if i < int(parse.Set.SkipHead) { + continue + } + if i >= (len(bits) - int(parse.Set.SkipTail)) { + continue + } + if parse.Set.Regex != "" { + re, err := regexp.Compile(parse.Set.Regex) + if err != nil { + return err + } + matches := re.FindStringSubmatch(bit) + if parse.Set.Match >= uint(len(matches)) { + bit = "" + } else { + bit = matches[parse.Set.Match] + } + } + subcontext := fmt.Sprintf("%s.%s", context, bit) + engine.SetFact(subcontext, "true") + } + } else if parse.Dict != nil { + sep, err := unescape(parse.Dict.Separator) + if err != nil { + return err + } + dlm, err := unescape(parse.Dict.Delimiter) + if err != nil { + return err + } + bits := strings.Split(val, sep) + for _, bit := range bits { + pair := strings.SplitN(bit, dlm, 2) + if len(pair) != 2 { + //return fmt.Errorf("Cannot split %s value '%s' on '%s'", fact.Name, pair, parse.Dict.Delimiter) + continue + } + key := strings.TrimSpace(pair[0]) + subcontext := fmt.Sprintf("%s.%s", context, key) + err := engine.SetValueParse(fact, parse.Dict.Parse, subcontext, pair[1]) + if err != nil { + return err + } + } + } else { + return fmt.Errorf("Expecting scalar or list or dict to parse") + } + + return nil +} + +func (engine *Engine) SetValueString(fact *Fact) error { + val, ok := engine.Facts[fact.Value.String.Fact] + if !ok { + return fmt.Errorf("Fact %s not present", fact.Value.String.Fact) + } + + return engine.SetValueParse(fact, fact.Value.String.Parse, fact.Name, string(val)) +} + +func (engine *Engine) SetValueFile(fact *Fact) error { + data, err := ioutil.ReadFile(fact.Value.File.Path) + if err != nil { + if os.IsNotExist(err) && fact.Value.File.IgnoreMissing { + return nil + } + return err + } + + return engine.SetValueParse(fact, fact.Value.File.Parse, fact.Name, string(data)) +} + +func (engine *Engine) SetValueDirEnt(fact *Fact) error { + files, err := ioutil.ReadDir(fact.Value.DirEnt.Path) + if err != nil { + if os.IsNotExist(err) && fact.Value.DirEnt.IgnoreMissing { + return nil + } + return err + } + for _, file := range files { + engine.SetFact(fmt.Sprintf("%s.%s", fact.Name, file.Name()), "true") + } + return nil +} + +func (engine *Engine) SetValueCommand(fact *Fact) error { + var args []string + for _, arg := range fact.Value.Command.Args { + args = append(args, arg) + } + cmd := exec.Command(fact.Value.Command.Name, args...) + out, err := cmd.Output() + if err != nil { + return err + } + + return engine.SetValueParse(fact, fact.Value.Command.Parse, fact.Name, string(out)) +} + +func (engine *Engine) SetValueAccess(fact *Fact) error { + var flags uint32 + if fact.Value.Access.Check == "exists" { + flags = 0 + } else if fact.Value.Access.Check == "readable" { + flags = unix.R_OK + } else if fact.Value.Access.Check == "writable" { + flags = unix.W_OK + } else if fact.Value.Access.Check == "executable" { + flags = unix.X_OK + } else { + return fmt.Errorf("No access check type specified for %s", + fact.Value.Access.Path) + } + err := unix.Access(fact.Value.Access.Path, flags) + if err != nil { + engine.SetFact(fact.Name, "false") + engine.Fail(fact) + } else { + engine.SetFact(fact.Name, "true") + engine.Pass(fact) + } + return nil +} + +func (engine *Engine) ValidateFact(fact *Fact) error { + if fact.Filter != nil { + ok, err := engine.EvalExpression(fact.Filter) + if err != nil { + return err + } + if !ok { + return nil + } + } + if fact.Report != nil && (engine.Output&ENGINE_OUTPUT_REPORTS) != 0 { + fmt.Printf("Checking %s...", fact.Report.Message) + } + + if fact.Value.BuiltIn != nil { + return engine.SetValueBuiltIn(fact) + } else if fact.Value.Bool != nil { + return engine.SetValueBool(fact) + } else if fact.Value.String != nil { + return engine.SetValueString(fact) + } else if fact.Value.File != nil { + return engine.SetValueFile(fact) + } else if fact.Value.DirEnt != nil { + return engine.SetValueDirEnt(fact) + } else if fact.Value.Command != nil { + return engine.SetValueCommand(fact) + } else if fact.Value.Access != nil { + return engine.SetValueAccess(fact) + } else { + return fmt.Errorf("No information provided for value in fact %s", fact.Name) + } +} + +// Validate all facts in the list, returning a count of +// any non-fatal errors encountered. +func (engine *Engine) Validate(facts FactList) (uint, error) { + err := facts.Sort() + if err != nil { + return 0, err + } + for _, fact := range facts.Facts { + err = engine.ValidateFact(fact) + if err != nil { + return 0, err + } + } + return engine.Errors, nil +} diff --git a/tools/host-validate/pkg/facts.go b/tools/host-validate/pkg/facts.go new file mode 100644 index 0000000000..4bb935cc50 --- /dev/null +++ b/tools/host-validate/pkg/facts.go @@ -0,0 +1,585 @@ +/* + * This file is part of the libvirt project + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>;. + * + * Copyright (C) 2019 Red Hat, Inc. + * + */ + +package pkg + +// This file defines the data structures that are being +// parsed from the YAML files. Note that the YAML parser +// internally uses the Golang JSON parser, hence the 'json:' +// comments against struct fields. + +import ( + "fmt" + "github.com/ghodss/yaml" + "io/ioutil" + "strings" +) + +// A list of all the facts we are going to validate +type FactList struct { + Facts []*Fact `json:"facts"` +} + +// A fact is a description of a single piece of information +// we wish to check. Conceptually a fact is simply a plain +// key, value pair where both parts are strings. +// +// Every fact has a name which is a dot separated list of +// strings, eg 'cpu.family.arm'. By convention the dots +// are forming an explicit hierarchy, so a common prefix +// on names is used to group related facts. +// +// Optionally a report can be associated with a fact +// This is a freeform string intended to be read by +// humans, eg 'hardware virt possible' +// +// If a report is given, there can also be an optional +// hint given, which is used when a fact fails to match +// some desired condition. This is another freeform string +// intended to be read by humans, eg +// 'enable cpuset cgroup controller in Kconfig' +// +// The optional filter is an expression that can be used +// to skip the processing of this fact when certain +// conditions are not met. eg, a filter might skip +// the checking of cgroups when the os kernel is not "linux" +// +// Finally there is a mandatory value. This defines how +// to extract the value for setting the fact. +// +// +// +type Fact struct { + Name string `json:"name"` + Report *Report `json:"report,omitempty"` + Hint *Report `json:"hint,omitempty"` + Filter *Expression `json:"filter,omitempty"` + Value Value `json:"value"` +} + +// A report is a message intended to be targetted at humans +// +// The message can be an arbitrary string, informing them +// of some relevant information +// +// The level is one of 'warn' or 'note' or 'error', with +// 'error' being assumed if no value is given +type Report struct { + Message string `json:"message"` + Level string `json:"level,omitempty"` + Pass string `json:"pass,omitempty"` +} + +// An expression is used to evaluate some complex conditions +// +// Expressions can be simple, comparing a single fact to +// some desired match. +// +// Expressions can be compound, requiring any or all of a +// list of sub-expressions to evaluate to true. +type Expression struct { + Any *ExpressionCompound `json:"any,omitempty"` + All *ExpressionCompound `json:"all,omitempty"` + Fact *ExpressionFact `json:"fact,omitempty"` +} + +// A compound expression is simply a list of expressions +// to be evaluated +type ExpressionCompound struct { + Expressions []Expression `json:"expressions,omitempty"` +} + +// A fact expression defines a rule that compares the +// value associated with the fact, to some desired +// match. +// +// The name gives the name of the fact to check +// +// The semantics of value vary according to the match +// type +// +// If the match type is 'regex', then the value must +// match against the declared regular expression. +// +// If the match type is 'exists', the value is ignored +// and the fact must simply exist. +// +// If the match type is not set, then a plain string +// equality comparison is done +type ExpressionFact struct { + Name string `json:"name"` + Value string `json:"value,omitempty"` + Match string `json:"match,omitempty"` +} + +// A value defines the various data sources for +// setting a fact's value. Only one of the data +// sources is permitted to be non-nil for each +// fact +// +// A builtin value is one of the standard facts +// defined in code. +// +// A bool value is one set to 'true' or 'false' +// depending on the results of evaluating an +// expression. It is user error to construct +// an expression which is self-referential +// +// A string value is one set by parsing the +// the value of another fact. A typical use +// case would be to split a string based on +// a whitespace separator +// +// A file value is one set by parsing the +// contents of a file on disk +// +// A dirent value results in creation of +// many facts, one for each directory entry +// seen +// +// An access value is one that sets a value +// to 'true' or 'false' depending on the +// permissions of a file +// +// A command value is one set by parsing +// the output of a command's stdout +type Value struct { + BuiltIn *ValueBuiltIn `json:"builtin,omitempty"` + Bool *Expression `json:"bool,omitempty"` + String *ValueString `json:"string,omitempty"` + File *ValueFile `json:"file,omitempty"` + DirEnt *ValueDirEnt `json:"dirent,omitempty"` + Access *ValueAccess `json:"access,omitempty"` + Command *ValueCommand `json:"command,omitempty"` +} + +// Valid built-in fact names are +// - os.{kernel,release,version} and cpu.arch, +// set from uname() syscall results +// - libvirt.driver set from a command line arg +type ValueBuiltIn struct { +} + +// Sets a value from a command. +// +// The name is the binary command name, either +// unqualified and resolved against $PATH, or +// or fully qualified +// +// A command can be given an arbitray set of +// arguments when run +// +// By default the entire contents of stdout +// will be set as the fact value. +// +// It is possible to instead parse the stdout +// data to extract interesting pieces of information +// from it +type ValueCommand struct { + Name string `json:"name"` + Args []string `json:"args,omitempty"` + Parse *Parse `json:"parse,omitempty"` +} + +// Sets a value from a file contents +// +// The path is the fully qualified filename path +// +// By default the entire contents of the file +// will be set as the fact value. +// +// It is possible to instead parse the file +// data to extract interesting pieces of information +// from it +type ValueFile struct { + Path string `json:"path"` + Parse *Parse `json:"parse,omitempty"` + IgnoreMissing bool `json:"ignoreMissing,omitempty"` +} + +// Sets a value from another fact +// +// The fact is the name of the other fact +// +// By default the entire contents of the other fact +// will be set as the fact value. +// +// More usually though the other fact value will be +// parsed to extract interesting pieces of information +// from it +type ValueString struct { + Fact string `json:"fact"` + Parse *Parse `json:"parse,omitempty"` +} + +// Sets a value from a list of directory entries +// +// The path is the fully qualified path of the directory +// +// By default an error will be raised if the directory +// does not exist. Typically a filter rule would be +// desired to skip processing of the fact in cases +// where it is known the directory may not exist. +// +// If filters are not practical though, missing directory +// can be made non-fatal +type ValueDirEnt struct { + Path string `json:"path"` + IgnoreMissing bool `json:"ignoreMissing,omitempty"` +} + +// Sets a value from the access permissions of a file +// +// The path is the fully qualified path of the file +// +// The check can be one of the strings 'readable', +// 'writable' or 'executable'. +type ValueAccess struct { + Path string `json:"path"` + Check string `json:"check"` +} + +// The parse object defines a set of rules for +// parsing strings to extract interesting +// pieces of data +// +// The optional whitespace attribute can be set to +// 'trim' to cause leading & trailing whitespace to +// be removed before further processing +// +// To extract a single data item from the string, +// the scalar parsing rule can be used +// +// To extract an ordered list of data items from +// the string, the list parsing rule can be used +// +// To extract an unordered list of data items, +// with duplicates excluded, the set parsing rule +// can be used +// +// To extract a list of key, value pairs from the +// string, the dict parsing rule can be used +type Parse struct { + Whitespace string `json:"whitespace,omitempty"` + Scalar *ParseScalar `json:"scalar,omitempty"` + List *ParseList `json:"list,omitempty"` + Set *ParseSet `json:"set,omitempty"` + Dict *ParseDict `json:"dict,omitempty"` +} + +// Parsing a string to extract a single data item +// using a regular expression. +// +// The regular expression should contain at least +// one capturing group. The match attribute indicates +// which capturing group will be used to set the +// fact value. +type ParseScalar struct { + Regex string `json:"regex,omitempty"` + Match uint `json:"match,omitempty"` +} + +// Parsing a string to extract an ordered list of +// data items +// +// The separator declares the boundary on which +// the string will be split +// +// The skip head attribute should be non-zero if +// any leading elements in the list are to be +// discarded. This is typically useful if the +// list contains a header/label as the first +// entry +// +// The skip tail attribute should be non-zero if +// any trailing elements in the list are to be +// discarded +// +// The limit attribute sets an upper bound on +// the number of elements that will be kept in +// the list. It is applied after discarding any +// leading or trailing elements. +// +// Each element in the list is then itself parsed +type ParseList struct { + Separator string `json:"separator,omitempty"` + SkipHead uint `json:"skiphead,omitempty"` + SkipTail uint `json:"skiptail,omitempty"` + Limit uint `json:"limit,omitempty"` + Parse *Parse `json:"parse,omitempty"` +} + +// Parsing a string to extract an unordered list of +// data items with duplicates removed +// +// The separator declares the boundary on which +// the string will be split +// +// The skip head attribute should be non-zero if +// any leading elements in the list are to be +// discarded. This is typically useful if the +// list contains a header/label as the first +// entry +// +// The skip tail attribute should be non-zero if +// any trailing elements in the list are to be +// discarded +// +// Each element is then parsed using a regular +// expression +// +// The regular expression should contain at least +// one capturing group. The match attribute indicates +// which capturing group will be used to set the +// fact value. +type ParseSet struct { + Separator string `json:"separator,omitempty"` + SkipHead uint `json:"skiphead,omitempty"` + SkipTail uint `json:"skiptail,omitempty"` + Regex string `json:"regex,omitempty"` + Match uint `json:"match,omitempty"` +} + +// Parsing a string to extract an unordered list of +// data items with duplicates removed +// +// The separator declares the boundary on which +// the string will be split to acquire the list +// of pairs +// +// The delimiter declares the boundary to separate +// the key from the value +// +// The value is then further parsed with the declared +// rules +type ParseDict struct { + Separator string `json:"separator,omitempty"` + Delimiter string `json:"delimiter,omitempty"` + Parse *Parse `json:"parse,omitempty"` +} + +// Helper for parsing a string containing an YAML +// doc defining a list of facts +func (f *FactList) Unmarshal(doc string) error { + return yaml.Unmarshal([]byte(doc), f) +} + +// Create a new fact list, loading from the +// specified plain file +func NewFactList(filename string) (*FactList, error) { + yamlstr, err := ioutil.ReadFile(filename) + if err != nil { + return nil, err + } + + facts := &FactList{} + err = facts.Unmarshal(string(yamlstr)) + if err != nil { + return nil, err + } + + return facts, nil +} + +// Used to ensure that no fact has a name which is a sub-string of +// another fact. +func validateNames(names map[string]*Fact) error { + for name, _ := range names { + bits := strings.Split(name, ".") + subname := "" + for _, bit := range bits[0 : len(bits)-1] { + if subname == "" { + subname = bit + } else { + subname = subname + "." + bit + } + _, ok := names[subname] + + if ok { + return fmt.Errorf("Fact name '%s' has fact '%s' as a substring", + name, subname) + } + } + } + + return nil +} + +// Identify the name of the corresponding fact that +// is referenced, by chopping off suffixes until a +// match is found +func findFactReference(names map[string]*Fact, name string) (string, error) { + bits := strings.Split(name, ".") + subname := "" + for _, bit := range bits { + if subname == "" { + subname = bit + } else { + subname = subname + "." + bit + } + _, ok := names[subname] + if ok { + return subname, nil + } + } + + return "", fmt.Errorf("Cannot find fact providing %s", name) +} + +// Build up a list of dependant facts referenced by an expression +func addDepsExpr(deps *map[string][]string, names map[string]*Fact, fact *Fact, expr *Expression) error { + if expr.Any != nil { + for _, sub := range expr.Any.Expressions { + err := addDepsExpr(deps, names, fact, &sub) + if err != nil { + return err + } + } + } else if expr.All != nil { + for _, sub := range expr.All.Expressions { + err := addDepsExpr(deps, names, fact, &sub) + if err != nil { + return err + } + } + } else if expr.Fact != nil { + ref, err := findFactReference(names, expr.Fact.Name) + if err != nil { + return err + } + entries, _ := (*deps)[fact.Name] + entries = append(entries, ref) + (*deps)[fact.Name] = entries + } + return nil +} + +// Build up a list of dependancies between facts +func addDeps(deps *map[string][]string, names map[string]*Fact, fact *Fact) error { + if fact.Filter != nil { + err := addDepsExpr(deps, names, fact, fact.Filter) + if err != nil { + return err + } + } + if fact.Value.Bool != nil { + err := addDepsExpr(deps, names, fact, fact.Value.Bool) + if err != nil { + return err + } + } + if fact.Value.String != nil { + ref, err := findFactReference(names, fact.Value.String.Fact) + if err != nil { + return err + } + entries, _ := (*deps)[fact.Name] + entries = append(entries, ref) + (*deps)[fact.Name] = entries + } + return nil +} + +// Perform a topological sort on facts so that they can be +// processed in the order required to satisfy dependancies +// between facts +func (facts *FactList) Sort() error { + deps := make(map[string][]string) + names := make(map[string]*Fact) + + var remaining []string + for _, fact := range facts.Facts { + deps[fact.Name] = []string{} + names[fact.Name] = fact + remaining = append(remaining, fact.Name) + } + + err := validateNames(names) + if err != nil { + return err + } + + for _, fact := range facts.Facts { + err = addDeps(&deps, names, fact) + if err != nil { + return err + } + } + + var sorted []string + done := make(map[string]bool) + for len(remaining) > 0 { + prev_done := len(done) + var skipped []string + for _, fact := range remaining { + using, ok := deps[fact] + if !ok { + done[fact] = true + sorted = append(sorted, fact) + } else { + unsolved := false + for _, entry := range using { + _, ok := done[entry] + if !ok { + unsolved = true + break + } + } + if !unsolved { + sorted = append(sorted, fact) + done[fact] = true + } else { + skipped = append(skipped, fact) + } + } + } + + if len(done) == prev_done { + return fmt.Errorf("Cycle detected in facts") + } + + remaining = skipped + } + + var newfacts []*Fact + for _, name := range sorted { + newfacts = append(newfacts, names[name]) + } + + facts.Facts = newfacts + + return nil +} + +// Create a new fact list that contains all the facts +// from the passed in list of fact lists +func MergeFactLists(lists []FactList) FactList { + var allfacts []*Fact + for _, list := range lists { + for _, fact := range list.Facts { + allfacts = append(allfacts, fact) + } + } + + facts := FactList{} + facts.Facts = allfacts + return facts +} diff --git a/tools/host-validate/pkg/facts_test.go b/tools/host-validate/pkg/facts_test.go new file mode 100644 index 0000000000..850ba50bfb --- /dev/null +++ b/tools/host-validate/pkg/facts_test.go @@ -0,0 +1,50 @@ +/* + * This file is part of the libvirt project + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with this library. If not, see + * <http://www.gnu.org/licenses/>;. + * + * Copyright (C) 2019 Red Hat, Inc. + * + */ + +package pkg + +import ( + "fmt" + "io/ioutil" + "path" + "strings" + "testing" +) + +func testYAMLFile(t *testing.T, filename string) { + _, err := NewFactList(filename) + if err != nil { + t.Fatal(fmt.Errorf("Cannot parse %s: %s", filename, err)) + } +} + +func TestParse(t *testing.T) { + dir := path.Join("..", "rules") + files, err := ioutil.ReadDir(dir) + if err != nil { + t.Fatal(fmt.Errorf("Cannot read %s: %s", dir, err)) + } + for _, file := range files { + if strings.HasSuffix(file.Name(), ".yaml") { + testYAMLFile(t, path.Join(dir, file.Name())) + } + } +} -- 2.21.0