this is the kind of jump that has me scratching my head. could the interleave of CMD2 and CEP2 be that efficient when fdupes and i7 go at it together?

this weekend i went from running fdupes on 2 of my boxen to the 5 which support it, all with

while fdupes -Lrqn /var/lib/boinc-client ; do sleep 600;done & disown


Statistics Date Total Run Time
(y:d:h:m:s) Points Generated Results Returned
11/27/10 0:025:14:30:59 74,515 96
11/26/10 0:016:00:13:13 70,115 80
11/25/10 0:013:20:53:36 49,686 82
11/24/10 0:015:03:46:29 48,962 105

device lineup
1 pb17 ppc1ghz -macosx
2 dell 11z netbook 1.2ghz c2m -windows 7 home 64
1 desknote athlon 2100+ windows xp
2 p4 3.0ghz HT fc12 + fdupes
4 c2q 6600 Kubu 10.10 +btrfs + gzip + fdupes
2 athlon x2 2.0ghz Kubu 10.10 +btrfs+compress + fdupes boot from 8gb thumbdrive
2 2.5ghz c2d penryn mbp Kubu 10.10 +btrfs+compress + 128gb-ssd + fdupes
12 i980x 4ghz + 64gb-ssd +btrfs+compress + fdupes
26 cores total.

CEP2+CMD5 for the HTs, CEP2-only for the DUAL+quad, all others default

I'm not sure how to rationalize 25 days up from 16 computing time, especially considering there were stretches of hours where fuseblows caused manual restarting.

thanks sekerob

im bogged out in release mode for my dayjob however a c++ STL rewrite of fdupes looks like a near goal for me if the obsession takes hold enough.

things i would investigate besides fdupes/filesystem normalization is using the llvm jit+linker as the last mile compiler/linker to give clang a shot at hardware optimization and better x64 distinction.

llvm is a program graph optimizer. not a compiler, per se, but a compiler optimization toolkit.

the same logic holds for code segments that holds for disk buffers, the fewer segments of code linked seperately, the more complete the compiler can normalize the graph of execution. i've seen scientific and AI binaries with 100's of thousands of dynamic link segments. llvm post-linker optimization would merge this out 9 time out of 10.

if i can get my hands on some source code makefiles i can give a whack at alternate jit-run and static llvm binaries

for tunability i have a feeling nothing will beat the i7 variety of quad cores for a time to come. good luck with your build :)

my most recent machine was work driven, i spent a millesecond comparing a liquid cooled overclock setup vs. a 980x shoestring build and the OEM performance numbers on the 980x outweighed any complex cooling equation i could think of countering with.

sqqqasm

In the end took the fdupes out as it was doing something nasty to the HFCC work units one after the other failing via the inconclusive > pending validation > error route. Clean logs and still the validator dismissing them. After removing fdupes, the subsequent HFCC validated again, so it's a thumbs down for this and probably also FAAH which uses the same science application. Not tried HPF2, so cant advise on that.

Anyway, last few days spend time dissecting fstab further and blowing it so bad at times that booting back into Linux became impossible or a permanent hang, not getting passed the mounting point of the ntfs and USB drives... but there is the Live CD and now got recovery down to a fine art and 6 minutes tops to create root user and restore the backup fstab file. What was learned and applied creating a stable environment which is slightly faster in booting and post sign in auto-mounting of the various blck devices is:

1. Adding a nobootwait to for instance the NTFS drive mounts which substantially shortened the time before the sign on screen appears.

UUID=B8B06265B06229D8 /media/HDD ntfs-3g defaults,local=en_CA.UTF-8,nobootwait 0 0

2. Adding noatime and nodiratime to the ext4 drive mounts who's purpose it is to disable the constant updating of the access stamp on files and directories slowing disk activity and with 6600+ files per CEP2 task it will make a bit of a difference with its constant disk accessing. For those doing pure crunching it does not matter, but those using the journaling features of Zeitgeist might see read only visits to files not being shown in the activity journal, though testing this on the latest 0.5.01 did still log them. A minor inconvenience.

UUID=fef2286b-94ae-4df5-991f-b8ac4846b108 / ext4 defaults,errors=remount-ro,noatime,nodiratime 0 1

Note that the above are mounts created by the mkfs process. Other mount line might look like

/dev/sdb1 /media/BOINC_Data ext4 defaults 0 0

Some reading material: https://wiki.archlinux.org/index.php/Fstab#atime_options which also touches on tmpfs which I fiddled with early in the Linux days that started for me May 1, 2010 :P

Don't look at me when you mess up... you were at the keyboard :-)