Re: [PATCH 1/1] sstate.bbclass: Improve sstate_installpkg performance

On Thu, 2012-05-10 at 18:13 -0500, Mark Hatle wrote:
> In a pathological case, lots of files to process, the sstate_installpkg
> performance was very poor.  It interated over each file and ran 3
> individual sed commands per file.  Changing this to keep iterating
> but running only a single command took about 1/3 time time.
> 
> However, when looking at the corresponding sstate_hardcode_path
> function, it was clear we could optimize this further.
> 
> Using the same encoding logic to specify only the minimumal sed
> operation necessary, and using xargs to avoid the os.system call the
> install step was able to be performed in 13% of the original time.
> 
> Example timing numbers for perl:
> 
> 3m7s original code
> 1m20s single sed, but interating
> 0m26s using xargs and limited sed
> 
> Signed-off-by: Mark Hatle <mark.hatle@...>
> ---
>  meta/classes/sstate.bbclass |   28 +++++++++++++++++++++-------
>  1 files changed, 21 insertions(+), 7 deletions(-)
> 
> diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
> index a8c98e5..ad7d121 100644
> --- a/meta/classes/sstate.bbclass
> +++ b/meta/classes/sstate.bbclass
>  <at>  <at>  -174,18 +174,29  <at>  <at>  def sstate_installpkg(ss, d):
>      bb.build.exec_func('sstate_unpack_package', d)
>  
>      # Fixup hardcoded paths
> +    #
> +    # Note: The logic below must match the reverse logic in
> +    # sstate_hardcode_path(d)
> +
>      fixmefn =  sstateinst + "fixmepath"
>      if os.path.isfile(fixmefn):
>          staging = d.getVar('STAGING_DIR', True)
>          staging_target = d.getVar('STAGING_DIR_TARGET', True)
>          staging_host = d.getVar('STAGING_DIR_HOST', True)
> -        fixmefd = open(fixmefn, "r")
> -        fixmefiles = fixmefd.readlines()
> -        fixmefd.close()
> -        for file in fixmefiles:
> -            os.system("sed -i -e s:FIXMESTAGINGDIRTARGET:%s:g %s" % (staging_target, sstateinst + file))
> -            os.system("sed -i -e s:FIXMESTAGINGDIRHOST:%s:g %s" % (staging_host, sstateinst + file))
> -            os.system("sed -i -e s:FIXMESTAGINGDIR:%s:g %s" % (staging, sstateinst + file))
> +
> +	if bb.data.inherits_class('native', d) or bb.data.inherits_class('nativesdk', d) or
bb.data.inherits_class('crosssdk', d) or bb.data.inherits_class('cross-canadian', d):
> +		sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIR:%s:g'" % (staging)
> +	elif bb.data.inherits_class('cross', d):
> +		sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRTARGET:%s:g; s:FIXMESTAGINGDIR:%s:g'" %
(staging_target, staging)
> +	else:
> +		sstate_sed_cmd = "sed -i -e 's:FIXMESTAGINGDIRHOST:%s:g'" % (staging_host)
> +
> +	# Add sstateinst to each filename in fixmepath, use xargs to efficiently call sed
> +	sstate_hardcode_cmd = "sed -e 's:^:%s:g' %s | xargs %s" % (sstateinst, fixmefn, sstate_sed_cmd)
> +
> +	print "Replacing fixme paths in sstate package: %s" % (sstate_hardcode_cmd)
> +	os.system(sstate_hardcode_cmd)
> +
>          # Need to remove this or we'd copy it into the target directory and may 
>          # conflict with another writer
>          os.remove(fixmefn)
>  <at>  <at>  -300,6 +311,9  <at>  <at>  python sstate_cleanall() {
>  def sstate_hardcode_path(d):
>  	# Need to remove hardcoded paths and fix these when we install the
>  	# staging packages.
> +	#
> +	# Note: the logic in this function needs to match the reverse logic
> +	# in sstate_installpkg(ss, d)
>  
>  	staging = d.getVar('STAGING_DIR', True)
>  	staging_target = d.getVar('STAGING_DIR_TARGET', True)

I was thinking this looked familiar, it was the other half of the
problem we originally solved:

http://git.yoctoproject.org/cgit.cgi/poky/commit/meta/classes/sstate.bbclass?id=db94ad4cf32d8ce3e97a8287d1c89a58d008a142

:)

Cheers,

Richard