Lucas Meneghel Rodrigues | 3 Mar 15:44 2010
Picon

Re: [KVM-AUTOTEST PATCH v4] KVM test: A memory efficient kvm_config implementation

On Tue, 2010-03-02 at 19:30 +0200, Michael Goldish wrote:
> This patch:
> 
> - Makes kvm_config use less memory during parsing, by storing config data
>   compactly in arrays during parsing, and generating the final dicts only when
>   requested.
>   On my machine this results in 5-10 times less memory being used (depending on
>   the size of the final generated list).
>   This allows the test configuration to keep expanding without having the
>   parser run out of memory.
> 
> - Adds config.fork_and_parse(), a function that parses a config file/string in
>   a forked process and then terminates the process.  This works around Python's
>   policy of keeping allocated memory to itself even after the objects occupying
>   the memory have been destroyed.  If the process that does the parsing is the
>   same one that runs the tests, less memory will be available to the VMs during
>   testing.
> 
> - Makes parsing 4-5 times faster as a result of the new internal representation.
> 
> Overall, kvm_config's memory usage should now be negligible in most cases.
> 
> Changes from v3:
> - Use the homemade 'configreader' class instead of regular files in parse()
>   and parse_variants() (readline() and/or seek() are very slow).
> - Use a regex cache dict (regex_cache).
> - Use a string cache dict in addition to the list (object_cache_indices).
> - Some changes to fork_and_parse() (disable buffering).
> 
> Changes from v2:
> - Merged _get_next_line() and _get_next_line_indent().
> - Made _array_get_name() faster.
> 
> Changes from v1:
> - Added config.get_generator() which is similar to get_list() but returns a
>   dict generator instead of a list.  This should save some more memory and will
>   make tests start sooner.
> - Use get_generator() in control.
> - Call waitpid() at the end of fork_and_parse().

As the generated patch is kinda fragmented for posting comments inline,
I am going to throw just a block of minor comments after I have reviewed
the code:

Observations:

* When a file is missing, it's more appropriate to raise a IOError than
an Exception, so we must change that. Also, it's important to follow the
coding standards for raising exceptions.
• I was wondering whether make fork_and_parse a public interface for the
config object was the right decision, maybe all calls to parse_file
should be done in a fork_and_parse fashion? I guess I got your point in
making it a public interface and separate it from parse_file, but isn't
that kinda confusing for the users (I mean, people writing control files
for kvm autotest)?
• About buffering on fork_and_parse: The performance penalties in
disabling buffering varies, with caches dropped it was something like
3-5%, after 'warming up' it was something like 8-11%, so it's small
stuff. But we can favour speed in this case so the final version won't
disable buffering.

Compliments:

• The configreader class was a very interesting move, simple, clean and
fast. Congrats!
• The output of the config system is good for debugging purposes, so
we'll stick with it.
• Thank you very much for your work, now we have faster parsing, that
consumes a lot less memory, so smaller boxes will benefit a *lot* from
that.

What I am going to do:

• I will re-send the version with the tiny changes I made so it gets
recorded on patchwork, and soon after I'll apply it upstream. I think
from this point on we might have only minor tweaks to make.

> 
> Signed-off-by: Michael Goldish <mgoldish <at> redhat.com>
> ---
>  client/tests/kvm/control          |   30 +-
>  client/tests/kvm/control.parallel |   21 +-
>  client/tests/kvm/kvm_config.py    |  832 ++++++++++++++++++++++---------------
>  3 files changed, 535 insertions(+), 348 deletions(-)
> 
> diff --git a/client/tests/kvm/control b/client/tests/kvm/control
> index 163286e..15c4539 100644
> --- a/client/tests/kvm/control
> +++ b/client/tests/kvm/control
>  <at>  <at>  -30,34 +30,38  <at>  <at>  import kvm_utils, kvm_config
>  # set English environment (command output might be localized, need to be safe)
>  os.environ['LANG'] = 'en_US.UTF-8'
>  
> -build_cfg_path = os.path.join(kvm_test_dir, "build.cfg")
> -build_cfg = kvm_config.config(build_cfg_path)
> -# Make any desired changes to the build configuration here. For example:
> -#build_cfg.parse_string("""
> +str = """
> +# This string will be parsed after build.cfg.  Make any desired changes to the
> +# build configuration here.  For example:
>  #release_tag = 84
> -#""")
> -if not kvm_utils.run_tests(build_cfg.get_list(), job):
> +"""
> +build_cfg = kvm_config.config()
> +build_cfg_path = os.path.join(kvm_test_dir, "build.cfg")
> +build_cfg.fork_and_parse(build_cfg_path, str)
> +if not kvm_utils.run_tests(build_cfg.get_generator(), job):
>      logging.error("KVM build step failed, exiting.")
>      sys.exit(1)
>  
> -tests_cfg_path = os.path.join(kvm_test_dir, "tests.cfg")
> -tests_cfg = kvm_config.config(tests_cfg_path)
> -# Make any desired changes to the test configuration here. For example:
> -#tests_cfg.parse_string("""
> +str = """
> +# This string will be parsed after tests.cfg.  Make any desired changes to the
> +# test configuration here.  For example:
>  #display = sdl
>  #install|setup: timeout_multiplier = 3
> -#""")
> +"""
> +tests_cfg = kvm_config.config()
> +tests_cfg_path = os.path.join(kvm_test_dir, "tests.cfg")
> +tests_cfg.fork_and_parse(tests_cfg_path, str)
>  
>  pools_cfg_path = os.path.join(kvm_test_dir, "address_pools.cfg")
>  tests_cfg.parse_file(pools_cfg_path)
>  hostname = os.uname()[1].split(".")[0]
> -if tests_cfg.filter("^" + hostname):
> +if tests_cfg.count("^" + hostname):
>      tests_cfg.parse_string("only ^%s" % hostname)
>  else:
>      tests_cfg.parse_string("only ^default_host")
>  
>  # Run the tests
> -kvm_utils.run_tests(tests_cfg.get_list(), job)
> +kvm_utils.run_tests(tests_cfg.get_generator(), job)
>  
>  # Generate a nice HTML report inside the job's results dir
>  kvm_utils.create_report(kvm_test_dir, job.resultdir)
> diff --git a/client/tests/kvm/control.parallel b/client/tests/kvm/control.parallel
> index 343f694..07bc6e5 100644
> --- a/client/tests/kvm/control.parallel
> +++ b/client/tests/kvm/control.parallel
>  <at>  <at>  -160,19 +160,22  <at>  <at>  if not params.get("mode") == "noinstall":
>  # ----------------------------------------------------------
>  import kvm_config
>  
> -filename = os.path.join(pwd, "kvm_tests.cfg")
> -cfg = kvm_config.config(filename)
> -
> -# If desirable, make changes to the test configuration here.  For example:
> -# cfg.parse_string("install|setup: timeout_multiplier = 2")
> -# cfg.parse_string("only fc8_quick")
> -# cfg.parse_string("display = sdl")
> +str = """
> +# This string will be parsed after tests.cfg.  Make any desired changes to the
> +# test configuration here.  For example:
> +#install|setup: timeout_multiplier = 3
> +#only fc8_quick
> +#display = sdl
> +"""
> +cfg = kvm_config.config()
> +filename = os.path.join(pwd, "tests.cfg")
> +cfg.fork_and_parse(filename, str)
>  
> -filename = os.path.join(pwd, "kvm_address_pools.cfg")
> +filename = os.path.join(pwd, "address_pools.cfg")
>  if os.path.exists(filename):
>      cfg.parse_file(filename)
>      hostname = os.uname()[1].split(".")[0]
> -    if cfg.filter("^" + hostname):
> +    if cfg.count("^" + hostname):
>          cfg.parse_string("only ^%s" % hostname)
>      else:
>          cfg.parse_string("only ^default_host")
> diff --git a/client/tests/kvm/kvm_config.py b/client/tests/kvm/kvm_config.py
> index 798ef56..7ff28e4 100755
> --- a/client/tests/kvm/kvm_config.py
> +++ b/client/tests/kvm/kvm_config.py
>  <at>  <at>  -2,10 +2,10  <at>  <at> 
>  """
>  KVM configuration file utility functions.
>  
> - <at> copyright: Red Hat 2008-2009
> + <at> copyright: Red Hat 2008-2010
>  """
>  
> -import logging, re, os, sys, StringIO, optparse
> +import logging, re, os, sys, optparse, array, traceback, cPickle
>  import common
>  from autotest_lib.client.common_lib import error
>  from autotest_lib.client.common_lib import logging_config, logging_manager
>  <at>  <at>  -21,490 +21,670  <at>  <at>  class config:
>      """
>      Parse an input file or string that follows the KVM Test Config File format
>      and generate a list of dicts that will be later used as configuration
> -    parameters by the the KVM tests.
> +    parameters by the KVM tests.
>  
>       <at> see: http://www.linux-kvm.org/page/KVM-Autotest/Test_Config_File
>      """
>  
> -    def __init__(self, filename=None, debug=False):
> +    def __init__(self, filename=None, debug=True):
>          """
> -        Initialize the list and optionally parse filename.
> +        Initialize the list and optionally parse a file.
>  
>           <at> param filename: Path of the file that will be taken.
> -         <at> param debug: Whether to turn debugging output.
> +         <at> param debug: Whether to turn on debugging output.
>          """
> -        self.list = [{"name": "", "shortname": "", "depend": []}]
> -        self.debug = debug
> +        self.list = [array.array("H", [4, 4, 4, 4])]
> +        self.object_cache = []
> +        self.object_cache_indices = {}
> +        self.regex_cache = {}
>          self.filename = filename
> +        self.debug = debug
>          if filename:
>              self.parse_file(filename)
>  
> 
>      def parse_file(self, filename):
>          """
> -        Parse filename, return the resulting list and store it in .list. If
> -        filename does not exist, raise an exception.
> +        Parse file.  If it doesn't exist, raise an exception.
>  
>           <at> param filename: Path of the configuration file.
>          """
>          if not os.path.exists(filename):
>              raise Exception, "File %s not found" % filename
>          self.filename = filename
> -        file = open(filename, "r")
> -        self.list = self.parse(file, self.list)
> -        file.close()
> -        return self.list
> +        str = open(filename).read()
> +        self.list = self.parse(configreader(str), self.list)
>  
> 
>      def parse_string(self, str):
>          """
> -        Parse a string, return the resulting list and store it in .list.
> +        Parse a string.
>  
> -         <at> param str: String that will be parsed.
> +         <at> param str: String to parse.
>          """
> -        file = StringIO.StringIO(str)
> -        self.list = self.parse(file, self.list)
> -        file.close()
> -        return self.list
> +        self.list = self.parse(configreader(str), self.list)
>  
> 
> -    def get_list(self):
> -        """
> -        Return the list of dictionaries. This should probably be called after
> -        parsing something.
> +    def fork_and_parse(self, filename=None, str=None):
>          """
> -        return self.list
> +        Parse a file and/or a string in a separate process to save memory.
>  
> +        Python likes to keep memory to itself even after the objects occupying
> +        it have been destroyed.  If during a call to parse_file() or
> +        parse_string() a lot of memory is used, it can only be freed by
> +        terminating the process.  This function works around the problem by
> +        doing the parsing in a forked process and then terminating it, freeing
> +        any unneeded memory.
>  
> -    def match(self, filter, dict):
> -        """
> -        Return True if dict matches filter.
> +        Note: if an exception is raised during parsing, its information will be
> +        printed, and the resulting list will be empty.  The exception will not
> +        be raised in the process calling this function.
>  
> -         <at> param filter: A regular expression that defines the filter.
> -         <at> param dict: Dictionary that will be inspected.
> +         <at> param filename: Path of file to parse (optional).
> +         <at> param str: String to parse (optional).
>          """
> -        filter = re.compile(r"(\.|^)(%s)(\.|$)" % filter)
> -        return bool(filter.search(dict["name"]))
> -
> -
> -    def filter(self, filter, list=None):
> +        r, w = os.pipe()
> +        r, w = os.fdopen(r, "r", 0), os.fdopen(w, "w", 0)
> +        pid = os.fork()
> +        if not pid:
> +            # Child process
> +            r.close()
> +            try:
> +                if filename:
> +                    self.parse_file(filename)
> +                if str:
> +                    self.parse_string(str)
> +            except:
> +                traceback.print_exc()
> +                self.list = []
> +            # Convert the arrays to strings before pickling because at least
> +            # some Python versions can't pickle/unpickle arrays
> +            l = [a.tostring() for a in self.list]
> +            cPickle.dump((l, self.object_cache), w, -1)
> +            w.close()
> +            os._exit(0)
> +        else:
> +            # Parent process
> +            w.close()
> +            (l, self.object_cache) = cPickle.load(r)
> +            r.close()
> +            os.waitpid(pid, 0)
> +            self.list = []
> +            for s in l:
> +                a = array.array("H")
> +                a.fromstring(s)
> +                self.list.append(a)
> +
> +
> +    def get_generator(self):
>          """
> -        Filter a list of dicts.
> +        Generate dictionaries from the code parsed so far.  This should
> +        probably be called after parsing something.
>  
> -         <at> param filter: A regular expression that will be used as a filter.
> -         <at> param list: A list of dictionaries that will be filtered.
> +         <at> return: A dict generator.
>          """
> -        if list is None:
> -            list = self.list
> -        return [dict for dict in list if self.match(filter, dict)]
> +        for a in self.list:
> +            name, shortname, depend, content = _array_get_all(a, self.object_cache)
> +            dict = {"name": name, "shortname": shortname, "depend": depend}
> +            self._apply_content_to_dict(dict, content)
> +            yield dict
>  
> 
> -    def split_and_strip(self, str, sep="="):
> +    def get_list(self):
>          """
> -        Split str and strip quotes from the resulting parts.
> +        Generate a list of dictionaries from the code parsed so far.
> +        This should probably be called after parsing something.
>  
> -         <at> param str: String that will be processed
> -         <at> param sep: Separator that will be used to split the string
> +         <at> return: A list of dicts.
>          """
> -        temp = str.split(sep, 1)
> -        for i in range(len(temp)):
> -            temp[i] = temp[i].strip()
> -            if re.findall("^\".*\"$", temp[i]):
> -                temp[i] = temp[i].strip("\"")
> -            elif re.findall("^\'.*\'$", temp[i]):
> -                temp[i] = temp[i].strip("\'")
> -        return temp
> -
> +        return list(self.get_generator())
>  
> -    def get_next_line(self, file):
> -        """
> -        Get the next non-empty, non-comment line in a file like object.
>  
> -         <at> param file: File like object
> -         <at> return: If no line is available, return None.
> +    def count(self, filter=".*"):
>          """
> -        while True:
> -            line = file.readline()
> -            if line == "": return None
> -            stripped_line = line.strip()
> -            if len(stripped_line) > 0 \
> -                    and not stripped_line.startswith('#') \
> -                    and not stripped_line.startswith('//'):
> -                return line
> -
> +        Return the number of dictionaries whose names match filter.
>  
> -    def get_next_line_indent(self, file):
> +         <at> param filter: A regular expression string.
>          """
> -        Return the indent level of the next non-empty, non-comment line in file.
> -
> -         <at> param file: File like object.
> -         <at> return: If no line is available, return -1.
> -        """
> -        pos = file.tell()
> -        line = self.get_next_line(file)
> -        if not line:
> -            file.seek(pos)
> -            return -1
> -        line = line.expandtabs()
> -        indent = 0
> -        while line[indent] == ' ':
> -            indent += 1
> -        file.seek(pos)
> -        return indent
> -
> -
> -    def add_name(self, str, name, append=False):
> -        """
> -        Add name to str with a separator dot and return the result.
> -
> -         <at> param str: String that will be processed
> -         <at> param name: name that will be appended to the string.
> -         <at> return: If append is True, append name to str.
> -                Otherwise, pre-pend name to str.
> -        """
> -        if str == "":
> -            return name
> -        # Append?
> -        elif append:
> -            return str + "." + name
> -        # Prepend?
> -        else:
> -            return name + "." + str
> +        exp = self._get_filter_regex(filter)
> +        count = 0
> +        for a in self.list:
> +            name = _array_get_name(a, self.object_cache)
> +            if exp.search(name):
> +                count += 1
> +        return count
>  
> 
> -    def parse_variants(self, file, list, subvariants=False, prev_indent=-1):
> +    def parse_variants(self, cr, list, subvariants=False, prev_indent=-1):
>          """
> -        Read and parse lines from file like object until a line with an indent
> -        level lower than or equal to prev_indent is encountered.
> +        Read and parse lines from a configreader object until a line with an
> +        indent level lower than or equal to prev_indent is encountered.
>  
> -         <at> brief: Parse a 'variants' or 'subvariants' block from a file-like
> -        object.
> -         <at> param file: File-like object that will be parsed
> -         <at> param list: List of dicts to operate on
> +         <at> brief: Parse a 'variants' or 'subvariants' block from a configreader
> +            object.
> +         <at> param cr: configreader object to be parsed.
> +         <at> param list: List of arrays to operate on.
>           <at> param subvariants: If True, parse in 'subvariants' mode;
> -        otherwise parse in 'variants' mode
> -         <at> param prev_indent: The indent level of the "parent" block
> -         <at> return: The resulting list of dicts.
> +            otherwise parse in 'variants' mode.
> +         <at> param prev_indent: The indent level of the "parent" block.
> +         <at> return: The resulting list of arrays.
>          """
>          new_list = []
>  
>          while True:
> -            indent = self.get_next_line_indent(file)
> +            pos = cr.tell()
> +            (indented_line, line, indent) = cr.get_next_line()
>              if indent <= prev_indent:
> +                cr.seek(pos)
>                  break
> -            indented_line = self.get_next_line(file).rstrip()
> -            line = indented_line.strip()
>  
>              # Get name and dependencies
> -            temp = line.strip("- ").split(":")
> -            name = temp[0]
> -            if len(temp) == 1:
> -                dep_list = []
> -            else:
> -                dep_list = temp[1].split()
> +            (name, depend) = map(str.strip, line.lstrip("- ").split(":"))
>  
>              # See if name should be added to the 'shortname' field
> -            add_to_shortname = True
> -            if name.startswith(" <at> "):
> -                name = name.strip(" <at> ")
> -                add_to_shortname = False
> -
> -            # Make a deep copy of list
> -            temp_list = []
> -            for dict in list:
> -                new_dict = dict.copy()
> -                new_dict["depend"] = dict["depend"][:]
> -                temp_list.append(new_dict)
> +            add_to_shortname = not name.startswith(" <at> ")
> +            name = name.lstrip(" <at> ")
> +
> +            # Store name and dependencies in cache and get their indices
> +            n = self._store_str(name)
> +            d = self._store_str(depend)
> +
> +            # Make a copy of list
> +            temp_list = [a[:] for a in list]
>  
>              if subvariants:
>                  # If we're parsing 'subvariants', first modify the list
> -                self.__modify_list_subvariants(temp_list, name, dep_list,
> -                                               add_to_shortname)
> -                temp_list = self.parse(file, temp_list,
> -                        restricted=True, prev_indent=indent)
> +                if add_to_shortname:
> +                    for a in temp_list:
> +                        _array_append_to_name_shortname_depend(a, n, d)
> +                else:
> +                    for a in temp_list:
> +                        _array_append_to_name_depend(a, n, d)
> +                temp_list = self.parse(cr, temp_list, restricted=True,
> +                                       prev_indent=indent)
>              else:
>                  # If we're parsing 'variants', parse before modifying the list
>                  if self.debug:
> -                    self.__debug_print(indented_line,
> -                                       "Entering variant '%s' "
> -                                       "(variant inherits %d dicts)" %
> -                                       (name, len(list)))
> -                temp_list = self.parse(file, temp_list,
> -                        restricted=False, prev_indent=indent)
> -                self.__modify_list_variants(temp_list, name, dep_list,
> -                                            add_to_shortname)
> +                    _debug_print(indented_line,
> +                                 "Entering variant '%s' "
> +                                 "(variant inherits %d dicts)" %
> +                                 (name, len(list)))
> +                temp_list = self.parse(cr, temp_list, restricted=False,
> +                                       prev_indent=indent)
> +                if add_to_shortname:
> +                    for a in temp_list:
> +                        _array_prepend_to_name_shortname_depend(a, n, d)
> +                else:
> +                    for a in temp_list:
> +                        _array_prepend_to_name_depend(a, n, d)
>  
>              new_list += temp_list
>  
>          return new_list
>  
> 
> -    def parse(self, file, list, restricted=False, prev_indent=-1):
> +    def parse(self, cr, list, restricted=False, prev_indent=-1):
>          """
> -        Read and parse lines from file until a line with an indent level lower
> -        than or equal to prev_indent is encountered.
> -
> -         <at> brief: Parse a file-like object.
> -         <at> param file: A file-like object
> -         <at> param list: A list of dicts to operate on (list is modified in
> -        place and should not be used after the call)
> -         <at> param restricted: if True, operate in restricted mode
> -        (prohibit 'variants')
> -         <at> param prev_indent: the indent level of the "parent" block
> -         <at> return: Return the resulting list of dicts.
> +        Read and parse lines from a configreader object until a line with an
> +        indent level lower than or equal to prev_indent is encountered.
> +
> +         <at> brief: Parse a configreader object.
> +         <at> param cr: A configreader object.
> +         <at> param list: A list of arrays to operate on (list is modified in
> +            place and should not be used after the call).
> +         <at> param restricted: If True, operate in restricted mode
> +            (prohibit 'variants').
> +         <at> param prev_indent: The indent level of the "parent" block.
> +         <at> return: The resulting list of arrays.
>           <at> note: List is destroyed and should not be used after the call.
> -        Only the returned list should be used.
> +            Only the returned list should be used.
>          """
> +        current_block = ""
> +
>          while True:
> -            indent = self.get_next_line_indent(file)
> +            pos = cr.tell()
> +            (indented_line, line, indent) = cr.get_next_line()
>              if indent <= prev_indent:
> +                cr.seek(pos)
> +                self._append_content_to_arrays(list, current_block)
>                  break
> -            indented_line = self.get_next_line(file).rstrip()
> -            line = indented_line.strip()
> -            words = line.split()
>  
>              len_list = len(list)
>  
> -            # Look for a known operator in the line
> -            operators = ["?+=", "?<=", "?=", "+=", "<=", "="]
> -            op_found = None
> -            op_pos = len(line)
> -            for op in operators:
> -                pos = line.find(op)
> -                if pos >= 0 and pos < op_pos:
> -                    op_found = op
> -                    op_pos = pos
> -
> -            # Found an operator?
> -            if op_found:
> +            # Parse assignment operators (keep lines in temporary buffer)
> +            if "=" in line:
>                  if self.debug and not restricted:
> -                    self.__debug_print(indented_line,
> -                                       "Parsing operator (%d dicts in current "
> -                                       "context)" % len_list)
> -                (left, value) = self.split_and_strip(line, op_found)
> -                filters_and_key = self.split_and_strip(left, ":")
> -                filters = filters_and_key[:-1]
> -                key = filters_and_key[-1]
> -                filtered_list = list
> -                for filter in filters:
> -                    filtered_list = self.filter(filter, filtered_list)
> -                # Apply the operation to the filtered list
> -                if op_found == "=":
> -                    for dict in filtered_list:
> -                        dict[key] = value
> -                elif op_found == "+=":
> -                    for dict in filtered_list:
> -                        dict[key] = dict.get(key, "") + value
> -                elif op_found == "<=":
> -                    for dict in filtered_list:
> -                        dict[key] = value + dict.get(key, "")
> -                elif op_found.startswith("?"):
> -                    exp = re.compile("^(%s)$" % key)
> -                    if op_found == "?=":
> -                        for dict in filtered_list:
> -                            for key in dict.keys():
> -                                if exp.match(key):
> -                                    dict[key] = value
> -                    elif op_found == "?+=":
> -                        for dict in filtered_list:
> -                            for key in dict.keys():
> -                                if exp.match(key):
> -                                    dict[key] = dict.get(key, "") + value
> -                    elif op_found == "?<=":
> -                        for dict in filtered_list:
> -                            for key in dict.keys():
> -                                if exp.match(key):
> -                                    dict[key] = value + dict.get(key, "")
> +                    _debug_print(indented_line,
> +                                 "Parsing operator (%d dicts in current "
> +                                 "context)" % len_list)
> +                current_block += line + "\n"
> +                continue
> +
> +            # Flush the temporary buffer
> +            self._append_content_to_arrays(list, current_block)
> +            current_block = ""
> +
> +            words = line.split()
>  
>              # Parse 'no' and 'only' statements
> -            elif words[0] == "no" or words[0] == "only":
> +            if words[0] == "no" or words[0] == "only":
>                  if len(words) <= 1:
>                      continue
> -                filters = words[1:]
> +                filters = map(self._get_filter_regex, words[1:])
>                  filtered_list = []
>                  if words[0] == "no":
> -                    for dict in list:
> +                    for a in list:
> +                        name = _array_get_name(a, self.object_cache)
>                          for filter in filters:
> -                            if self.match(filter, dict):
> +                            if filter.search(name):
>                                  break
>                          else:
> -                            filtered_list.append(dict)
> +                            filtered_list.append(a)
>                  if words[0] == "only":
> -                    for dict in list:
> +                    for a in list:
> +                        name = _array_get_name(a, self.object_cache)
>                          for filter in filters:
> -                            if self.match(filter, dict):
> -                                filtered_list.append(dict)
> +                            if filter.search(name):
> +                                filtered_list.append(a)
>                                  break
>                  list = filtered_list
>                  if self.debug and not restricted:
> -                    self.__debug_print(indented_line,
> -                                       "Parsing no/only (%d dicts in current "
> -                                       "context, %d remain)" %
> -                                       (len_list, len(list)))
> +                    _debug_print(indented_line,
> +                                 "Parsing no/only (%d dicts in current "
> +                                 "context, %d remain)" %
> +                                 (len_list, len(list)))
> +                continue
>  
>              # Parse 'variants'
> -            elif line == "variants:":
> +            if line == "variants:":
>                  # 'variants' not allowed in restricted mode
>                  # (inside an exception or inside subvariants)
>                  if restricted:
>                      e_msg = "Using variants in this context is not allowed"
>                      raise error.AutotestError(e_msg)
>                  if self.debug and not restricted:
> -                    self.__debug_print(indented_line,
> -                                       "Entering variants block (%d dicts in "
> -                                       "current context)" % len_list)
> -                list = self.parse_variants(file, list, subvariants=False,
> +                    _debug_print(indented_line,
> +                                 "Entering variants block (%d dicts in "
> +                                 "current context)" % len_list)
> +                list = self.parse_variants(cr, list, subvariants=False,
>                                             prev_indent=indent)
> +                continue
>  
>              # Parse 'subvariants' (the block is parsed for each dict
>              # separately)
> -            elif line == "subvariants:":
> +            if line == "subvariants:":
>                  if self.debug and not restricted:
> -                    self.__debug_print(indented_line,
> -                                       "Entering subvariants block (%d dicts in "
> -                                       "current context)" % len_list)
> +                    _debug_print(indented_line,
> +                                 "Entering subvariants block (%d dicts in "
> +                                 "current context)" % len_list)
>                  new_list = []
> -                # Remember current file position
> -                pos = file.tell()
> +                # Remember current position
> +                pos = cr.tell()
>                  # Read the lines in any case
> -                self.parse_variants(file, [], subvariants=True,
> +                self.parse_variants(cr, [], subvariants=True,
>                                      prev_indent=indent)
>                  # Iterate over the list...
> -                for index in range(len(list)):
> -                    # Revert to initial file position in this 'subvariants'
> -                    # block
> -                    file.seek(pos)
> +                for index in xrange(len(list)):
> +                    # Revert to initial position in this 'subvariants' block
> +                    cr.seek(pos)
>                      # Everything inside 'subvariants' should be parsed in
>                      # restricted mode
> -                    new_list += self.parse_variants(file, list[index:index+1],
> +                    new_list += self.parse_variants(cr, list[index:index+1],
>                                                      subvariants=True,
>                                                      prev_indent=indent)
>                  list = new_list
> +                continue
>  
>              # Parse 'include' statements
> -            elif words[0] == "include":
> +            if words[0] == "include":
>                  if len(words) <= 1:
>                      continue
>                  if self.debug and not restricted:
> -                    self.__debug_print(indented_line,
> -                                       "Entering file %s" % words[1])
> +                    _debug_print(indented_line, "Entering file %s" % words[1])
>                  if self.filename:
>                      filename = os.path.join(os.path.dirname(self.filename),
>                                              words[1])
>                      if os.path.exists(filename):
> -                        new_file = open(filename, "r")
> -                        list = self.parse(new_file, list, restricted)
> -                        new_file.close()
> +                        str = open(filename).read()
> +                        list = self.parse(configreader(str), list, restricted)
>                          if self.debug and not restricted:
> -                            self.__debug_print("", "Leaving file %s" % words[1])
> +                            _debug_print("", "Leaving file %s" % words[1])
>                      else:
>                          logging.warning("Cannot include %s -- file not found",
>                                          filename)
>                  else:
>                      logging.warning("Cannot include %s because no file is "
>                                      "currently open", words[1])
> +                continue
>  
>              # Parse multi-line exceptions
>              # (the block is parsed for each dict separately)
> -            elif line.endswith(":"):
> +            if line.endswith(":"):
>                  if self.debug and not restricted:
> -                    self.__debug_print(indented_line,
> -                                       "Entering multi-line exception block "
> -                                       "(%d dicts in current context outside "
> -                                       "exception)" % len_list)
> -                line = line.strip(":")
> +                    _debug_print(indented_line,
> +                                 "Entering multi-line exception block "
> +                                 "(%d dicts in current context outside "
> +                                 "exception)" % len_list)
> +                line = line[:-1]
>                  new_list = []
> -                # Remember current file position
> -                pos = file.tell()
> +                # Remember current position
> +                pos = cr.tell()
>                  # Read the lines in any case
> -                self.parse(file, [], restricted=True, prev_indent=indent)
> +                self.parse(cr, [], restricted=True, prev_indent=indent)
>                  # Iterate over the list...
> -                for index in range(len(list)):
> -                    if self.match(line, list[index]):
> -                        # Revert to initial file position in this
> -                        # exception block
> -                        file.seek(pos)
> +                exp = self._get_filter_regex(line)
> +                for index in xrange(len(list)):
> +                    name = _array_get_name(list[index], self.object_cache)
> +                    if exp.search(name):
> +                        # Revert to initial position in this exception block
> +                        cr.seek(pos)
>                          # Everything inside an exception should be parsed in
>                          # restricted mode
> -                        new_list += self.parse(file, list[index:index+1],
> +                        new_list += self.parse(cr, list[index:index+1],
>                                                 restricted=True,
>                                                 prev_indent=indent)
>                      else:
> -                        new_list += list[index:index+1]
> +                        new_list.append(list[index])
>                  list = new_list
> +                continue
>  
>          return list
>  
> 
> -    def __debug_print(self, str1, str2=""):
> +    def _get_filter_regex(self, filter):
>          """
> -        Nicely print two strings and an arrow.
> +        Return a regex object corresponding to a given filter string.
>  
> -         <at> param str1: First string
> -         <at> param str2: Second string
> +        All regular expressions given to the parser are passed through this
> +        function first.  Its purpose is to make them more specific and better
> +        suited to match dictionary names: it forces simple expressions to match
> +        only between dots or at the beginning or end of a string.  For example,
> +        the filter 'foo' will match 'foo.bar' but not 'foobar'.
>          """
> -        if str2:
> -            str = "%-50s ---> %s" % (str1, str2)
> -        else:
> -            str = str1
> -        logging.debug(str)
> -
> -
> -    def __modify_list_variants(self, list, name, dep_list, add_to_shortname):
> -        """
> -        Make some modifications to list, as part of parsing a 'variants' block.
> -
> -         <at> param list: List to be processed
> -         <at> param name: Name to be prepended to the dictionary's 'name' key
> -         <at> param dep_list: List of dependencies to be added to the dictionary's
> -                'depend' key
> -         <at> param add_to_shortname: Boolean indicating whether name should be
> -                prepended to the dictionary's 'shortname' key as well
> -        """
> -        for dict in list:
> -            # Prepend name to the dict's 'name' field
> -            dict["name"] = self.add_name(dict["name"], name)
> -            # Prepend name to the dict's 'shortname' field
> -            if add_to_shortname:
> -                dict["shortname"] = self.add_name(dict["shortname"], name)
> -            # Prepend name to each of the dict's dependencies
> -            for i in range(len(dict["depend"])):
> -                dict["depend"][i] = self.add_name(dict["depend"][i], name)
> -            # Add new dependencies
> -            dict["depend"] += dep_list
> -
> -
> -    def __modify_list_subvariants(self, list, name, dep_list, add_to_shortname):
> -        """
> -        Make some modifications to list, as part of parsing a 'subvariants'
> -        block.
> -
> -         <at> param list: List to be processed
> -         <at> param name: Name to be appended to the dictionary's 'name' key
> -         <at> param dep_list: List of dependencies to be added to the dictionary's
> -                'depend' key
> -         <at> param add_to_shortname: Boolean indicating whether name should be
> -                appended to the dictionary's 'shortname' as well
> -        """
> -        for dict in list:
> -            # Add new dependencies
> -            for dep in dep_list:
> -                dep_name = self.add_name(dict["name"], dep, append=True)
> -                dict["depend"].append(dep_name)
> -            # Append name to the dict's 'name' field
> -            dict["name"] = self.add_name(dict["name"], name, append=True)
> -            # Append name to the dict's 'shortname' field
> -            if add_to_shortname:
> -                dict["shortname"] = self.add_name(dict["shortname"], name,
> -                                                  append=True)
> +        try:
> +            return self.regex_cache[filter]
> +        except KeyError:
> +            exp = re.compile(r"(\.|^)(%s)(\.|$)" % filter)
> +            self.regex_cache[filter] = exp
> +            return exp
> +
> +
> +    def _store_str(self, str):
> +        """
> +        Store str in the internal object cache, if it isn't already there, and
> +        return its identifying index.
> +
> +         <at> param str: String to store.
> +         <at> return: The index of str in the object cache.
> +        """
> +        try:
> +            return self.object_cache_indices[str]
> +        except KeyError:
> +            self.object_cache.append(str)
> +            index = len(self.object_cache) - 1
> +            self.object_cache_indices[str] = index
> +            return index
> +
> +
> +    def _append_content_to_arrays(self, list, content):
> +        """
> +        Append content (config code containing assignment operations) to a list
> +        of arrays.
> +
> +         <at> param list: List of arrays to operate on.
> +         <at> param content: String containing assignment operations.
> +        """
> +        if content:
> +            str_index = self._store_str(content)
> +            for a in list:
> +                _array_append_to_content(a, str_index)
> +
> +
> +    def _apply_content_to_dict(self, dict, content):
> +        """
> +        Apply the operations in content (config code containing assignment
> +        operations) to a dict.
> +
> +         <at> param dict: Dictionary to operate on.  Must have 'name' key.
> +         <at> param content: String containing assignment operations.
> +        """
> +        for line in content.splitlines():
> +            op_found = None
> +            op_pos = len(line)
> +            for op in ops:
> +                pos = line.find(op)
> +                if pos >= 0 and pos < op_pos:
> +                    op_found = op
> +                    op_pos = pos
> +            if not op_found:
> +                continue
> +            (left, value) = map(str.strip, line.split(op_found, 1))
> +            if value and ((value[0] == '"' and value[-1] == '"') or
> +                          (value[0] == "'" and value[-1] == "'")):
> +                value = value[1:-1]
> +            filters_and_key = map(str.strip, left.split(":"))
> +            filters = filters_and_key[:-1]
> +            key = filters_and_key[-1]
> +            for filter in filters:
> +                exp = self._get_filter_regex(filter)
> +                if not exp.search(dict["name"]):
> +                    break
> +            else:
> +                ops[op_found](dict, key, value)
> +
> +
> +# Assignment operators
> +
> +def _op_set(dict, key, value):
> +    dict[key] = value
> +
> +
> +def _op_append(dict, key, value):
> +    dict[key] = dict.get(key, "") + value
> +
> +
> +def _op_prepend(dict, key, value):
> +    dict[key] = value + dict.get(key, "")
> +
> +
> +def _op_regex_set(dict, exp, value):
> +    exp = re.compile("^(%s)$" % exp)
> +    for key in dict:
> +        if exp.match(key):
> +            dict[key] = value
> +
> +
> +def _op_regex_append(dict, exp, value):
> +    exp = re.compile("^(%s)$" % exp)
> +    for key in dict:
> +        if exp.match(key):
> +            dict[key] += value
> +
> +
> +def _op_regex_prepend(dict, exp, value):
> +    exp = re.compile("^(%s)$" % exp)
> +    for key in dict:
> +        if exp.match(key):
> +            dict[key] = value + dict[key]
> +
> +
> +ops = {
> +    "=": _op_set,
> +    "+=": _op_append,
> +    "<=": _op_prepend,
> +    "?=": _op_regex_set,
> +    "?+=": _op_regex_append,
> +    "?<=": _op_regex_prepend,
> +}
> +
> +
> +# Misc functions
> +
> +def _debug_print(str1, str2=""):
> +    """
> +    Nicely print two strings and an arrow.
> +
> +     <at> param str1: First string.
> +     <at> param str2: Second string.
> +    """
> +    if str2:
> +        str = "%-50s ---> %s" % (str1, str2)
> +    else:
> +        str = str1
> +    logging.debug(str)
> +
> +
> +# configreader
> +
> +class configreader:
> +    """
> +    Preprocess an input string and provide file-like services.
> +    This is intended as a replacement for the file and StringIO classes,
> +    whose readline() and/or seek() methods seem to be slow.
> +    """
> +
> +    def __init__(self, str):
> +        """
> +        Initialize the reader.
> +
> +         <at> param str: The string to parse.
> +        """
> +        self.line_index = 0
> +        self.lines = []
> +        for line in str.splitlines():
> +            line = line.rstrip().expandtabs()
> +            stripped_line = line.strip()
> +            indent = len(line) - len(stripped_line)
> +            if (not stripped_line
> +                or stripped_line.startswith("#")
> +                or stripped_line.startswith("//")):
> +                continue
> +            self.lines.append((line, stripped_line, indent))
> +
> +
> +    def get_next_line(self):
> +        """
> +        Get the next non-empty, non-comment line in the string.
> +
> +         <at> param file: File like object.
> +         <at> return: (line, stripped_line, indent), where indent is the line's
> +            indent level or -1 if no line is available.
> +        """
> +        try:
> +            if self.line_index < len(self.lines):
> +                return self.lines[self.line_index]
> +            else:
> +                return (None, None, -1)
> +        finally:
> +            self.line_index += 1
> +
> +
> +    def tell(self):
> +        """
> +        Return the current line index.
> +        """
> +        return self.line_index
> +
> +
> +    def seek(self, index):
> +        """
> +        Set the current line index.
> +        """
> +        self.line_index = index
> +
> +
> +# Array structure:
> +# ----------------
> +# The first 4 elements contain the indices of the 4 segments.
> +# a[0] -- Index of beginning of 'name' segment (always 4).
> +# a[1] -- Index of beginning of 'shortname' segment.
> +# a[2] -- Index of beginning of 'depend' segment.
> +# a[3] -- Index of beginning of 'content' segment.
> +# The next elements in the array comprise the aforementioned segments:
> +# The 'name' segment begins with a[a[0]] and ends with a[a[1]-1].
> +# The 'shortname' segment begins with a[a[1]] and ends with a[a[2]-1].
> +# The 'depend' segment begins with a[a[2]] and ends with a[a[3]-1].
> +# The 'content' segment begins with a[a[3]] and ends at the end of the array.
> +
> +# The following functions append/prepend to various segments of an array.
> +
> +def _array_append_to_name_shortname_depend(a, name, depend):
> +    a.insert(a[1], name)
> +    a.insert(a[2] + 1, name)
> +    a.insert(a[3] + 2, depend)
> +    a[1] += 1
> +    a[2] += 2
> +    a[3] += 3
> +
> +
> +def _array_prepend_to_name_shortname_depend(a, name, depend):
> +    a[1] += 1
> +    a[2] += 2
> +    a[3] += 3
> +    a.insert(a[0], name)
> +    a.insert(a[1], name)
> +    a.insert(a[2], depend)
> +
> +
> +def _array_append_to_name_depend(a, name, depend):
> +    a.insert(a[1], name)
> +    a.insert(a[3] + 1, depend)
> +    a[1] += 1
> +    a[2] += 1
> +    a[3] += 2
> +
> +
> +def _array_prepend_to_name_depend(a, name, depend):
> +    a[1] += 1
> +    a[2] += 1
> +    a[3] += 2
> +    a.insert(a[0], name)
> +    a.insert(a[2], depend)
> +
> +
> +def _array_append_to_content(a, content):
> +    a.append(content)
> +
> +
> +def _array_get_name(a, object_cache):
> +    """
> +    Return the name of a dictionary represented by a given array.
> +
> +     <at> param a: Array representing a dictionary.
> +     <at> param object_cache: A list of strings referenced by elements in the array.
> +    """
> +    return ".".join([object_cache[i] for i in a[a[0]:a[1]]])
> +
> +
> +def _array_get_all(a, object_cache):
> +    """
> +    Return a 4-tuple containing all the data stored in a given array, in a
> +    format that is easy to turn into an actual dictionary.
> +
> +     <at> param a: Array representing a dictionary.
> +     <at> param object_cache: A list of strings referenced by elements in the array.
> +     <at> return: A 4-tuple: (name, shortname, depend, content), in which all
> +        members are strings except depend which is a list of strings.
> +    """
> +    name = ".".join([object_cache[i] for i in a[a[0]:a[1]]])
> +    shortname = ".".join([object_cache[i] for i in a[a[1]:a[2]]])
> +    content = "".join([object_cache[i] for i in a[a[3]:]])
> +    depend = []
> +    prefix = ""
> +    for n, d in zip(a[a[0]:a[1]], a[a[2]:a[3]]):
> +        for dep in object_cache[d].split():
> +            depend.append(prefix + dep)
> +        prefix += object_cache[n] + "."
> +    return name, shortname, depend, content
> +
>  
> 
>  if __name__ == "__main__":
>      parser = optparse.OptionParser()
>      parser.add_option('-f', '--file', dest="filename", action='store',
>                        help='path to a config file that will be parsed. '
> -                           'If not specified, will parse kvm_tests.cfg '
> -                           'located inside the kvm test dir.')
> +                           'If not specified, will parse tests.cfg located '
> +                           'inside the kvm test dir.')
>      parser.add_option('--verbose', dest="debug", action='store_true',
>                        help='include debug messages in console output')
>  
>  <at>  <at>  -518,9 +698,9  <at>  <at>  if __name__ == "__main__":
>      # Here we configure the stand alone program to use the autotest
>      # logging system.
>      logging_manager.configure_logging(KvmLoggingConfig(), verbose=debug)
> -    list = config(filename, debug=debug).get_list()
> +    dicts = config(filename, debug=debug).get_generator()
>      i = 0
> -    for dict in list:
> +    for dict in dicts:
>          logging.info("Dictionary #%d:", i)
>          keys = dict.keys()
>          keys.sort()

_______________________________________________
Autotest mailing list
Autotest <at> test.kernel.org
http://test.kernel.org/cgi-bin/mailman/listinfo/autotest

Gmane