Logging and bc. [Was: Re: r3353]

Ag. D. Hatzimanikas a.hatzim at gmail.com
Sat May 19 04:27:37 PDT 2007

On Fri, May 18, at 07:21 M.Canales.es wrote:
> El Jueves, 17 de Mayo de 2007 21:49, Ag. D. Hatzimanikas escribió:
> > Sometimes packages during installation, leave in the root system files that
> > are _older_ than the timestamp (mostly docs or configs), in a result
> > find can not track them.
> I know that that was an issue with some BLFS packages, but not sure about LFS 
> packages. It's a matter to test it.

I did a quick test with:
find / -xdev ! -path "/root/*" ! -path "/tmp/*" ! -newer \
/usr/include/asm/param.h -type f -printf "%AY/%Am/%Ad\t%p\n" |sort |less

And here is the report:

2005/01/27      /usr/share/xml/docbook/xml-dtd-4.4/docbook.cat
2005/01/27      /usr/share/xml/docbook/xml-dtd-4.4/ent/README
2005/01/27      /usr/share/xml/docbook/xml-dtd-4.4/soextblx.dtd
2006/10/26      /usr/share/xml/docbook/xml-dtd-4.5/docbook.cat
2006/10/26      /usr/share/xml/docbook/xml-dtd-4.5/ent/README
2006/10/26      /usr/share/xml/docbook/xml-dtd-4.5/soextblx.dtd

And then I remembered that I installed docbook by hand without the timestamp 
There are some more, mostly X fonts and some icons from mplayer, that I
put them with a single cp from a different partition.

The strange thing is that, if you look at the kernel-headers build, you can see 
that the first header that get installed in the system (and the first installed
file in a fresh LFS build actually) is the param.h, although find reports
(with %AS) that the older header is the /usr/include/asm/unistd.h.

There are also some kernel headers log-ed in the above find command (which it 
shouldn't), but I didn't find the way to force find to look only for older headers
than param.h or unistd.h for that matter.

> > This can be avoided, if during unpacking, either the '-m' or '--touch' tar
> > option is being used, so tar will not extract file's modified time,
> > or else from within the sources, execute the following
> > find . -exec touch {} \;
> >
> > But *this* is overkill, since tar can do the same thing with no cost.
> The tar option looks more obvious, but I think that to be useful the package's 
> sources files touch need be done after creating the timestamp-marker file.


> And we must be sure that that will not mess some package build. For example, 
> what could happen if some foo.{in,am} file ends having a newer timestamp than 
> foo?

I don't have enough knowledge to comment about your last sentence, but in
practice, for more than 6 month usage (about 10 builds) of this method I
didn't have a failure of this kind.
In fact I didn't have a single failure of any kind for quite sometime now, except 
an incompatibility with a snapshot of gcc-4.2 and openssl but this is another issue.

Manuel and George,

Dan posted a patch in february [1] aiming to replace bc with perl, which I agree.
Why we have to use an external application, when perl can do the math?
He is using printf and two variables to calculate the seconds, while - a long time
now - I use almost a method like Jhalfs to create the sbu report, so I am posting a
snippet of this patch I played a bit yesterday night, as I am more
familiar and it looks more simple.

If you are interested I can do the rest. I stopped in one point as you will 
see, that I needed a log file created by jhalfs to chech the pattern, 
which I don't have.
<confess> I don't use jhalfs</confess>.

I also replaced the 3 commands in the row (grep-head-sed) with a single
sed substitution.
>From a quick test over 150 package-logs the pattern seems to operate reliably.

There is no gain in time, (single sed and perl), in fact is sssliglty slower, but 
will help us to get rid of the bc dependency.
A quick grep for bc shows that is being used only during this operation.

But see by yourself.

Index: common/create-sbu_du-report.sh
--- common/create-sbu_du-report.sh	(revision 3355)
+++ common/create-sbu_du-report.sh	(working copy)
@@ -49,9 +49,9 @@
 # Match the first timed log to extract the SBU unit value from it
 BASELOG=`grep -l "^real\>" $LOGSDIR/* | head -n1`
 echo -e "\n\nUsing $BASELOG to obtain the SBU unit value." >> "$REPORT"
-BASEMINUTES=`grep "^real\>" $BASELOG | cut -f2 | sed -e 's/m.*//'`
-BASESECONDS=`grep "^real\>" $BASELOG | cut -f2 | sed -e 's/.*m//;s/s//'`
-SBU_UNIT=`echo "scale=3; $BASEMINUTES * 60 + $BASESECONDS" | bc`
+BASEMINUTES=`sed -n 's/^real[ \t]*\([[:digit:]]*\)m.*$/\1/p' $BASELOG`
+BASESECONDS=`sed -n 's/^real[ \t].*m\([[:digit:]]*\)\..*$/\1/p' $BASELOG`
+SBU_UNIT=`perl -e '$sbu = '$BASEMINUTES' * '60' + '$BASESECONDS'; print ("$sbu\n");'`
 echo -e "The SBU unit value is equal to $SBU_UNIT seconds.\n" >> "$REPORT"
 # Set the first value to 0 for grand totals calculation
@@ -65,11 +65,11 @@
   # Build time
   BUILDTIME=`grep "^real\>" $log | cut -f2`
   # Build time in seconds
-  MINUTES=`grep "^real\>" $log | cut -f2 | sed -e 's/m.*//'`
-  SECS=`grep "^real\>" $log | cut -f2 | sed -e 's/.*m//;s/s//'`
-  TIME=`echo "scale=3; $MINUTES * 60 + $SECS" | bc`
+  MINUTES=`sed -n 's/^real[ \t]*\([[:digit:]]*\)m.*$/\1/p'`
+  SECS=`sed -n 's/^real[ \t].*m\([[:digit:]]*\)\..*$/\1/p'`
+  TIME=`perl -e '$time = '$BASEMINUTES' * '60' + '$BASESECONDS'; print ("$time\n");'`
   # Calculate build time in SBU
-  SBU=`echo "scale=3; $TIME / $SBU_UNIT" | bc`
+  SBU=`perl -e '$sbu = '$TIME' / '$SBU_UNIT'; print ("$sbu\n");'`
   # Append SBU value to SBU2 for grand total
   SBU2="$SBU2 + $SBU"

1. http://linuxfromscratch.org/pipermail/alfs-discuss/2007-February/009114.html

More information about the alfs-discuss mailing list