The XCOMP/2 home page

What is XCOMP/2

Welcome to the home of XCOMP/2, a recursive file compare utility. XCOMP/2 is for comparing files what is XCOPY is for copying files, something I was definitely missing in the OS/2 product. A port for WIN32 is also available (for those who are sometimes forced to use Windows, but you need OS/2 to get access to that port!).

You may also want to take a look into the ISOCOMP/2 tool, which lets you compare a CD-ROM's contents with the ISO image (or RSJ track image) file it has been burnt with, or you may create an ISO image out of a CD-ROM.

Well, above explanation should be self explanatory. There are basically 2 variations for the commandline:

• The first which allows you to compare the contents of specified files at the Source path with the equivalent named files at the target path. Additionally, you might specify e.g. the /CHK option to also calculate the CRC32 and MD5 Checksum for the Source files.
• The second allows you to specify only the files at the Source path. This it is not checked, that commandline only makes sense together with the /CHK option, in that case the CRC32 and MD5 checksums calculated from each file is written into the Checksum file specified, or compared with the Checksums recorded there earlier (if the Checksum file already exists).

One note about the line containing the Throughput measurements. In that example I was comparing some Lotus Notes databases on a backup CD-RW with the orignal on a harddisk. You can see that the Source drive, a CD-ROM drive, delivers a not too high throughput (it seems that that this 8-16 speed CD-ROM drive isn't too comfortable reading CD-RWs), while the Target drive, a Ultra-Wide SCSI drive delivers excellent performance (ok, actually it's the HPFS386 cache ;-) giving an acceptable Total performance.

The performance given for Source and Destination is just the raw performance of the OS/2 DosRead() API without accounting for overhead like doing the file searched and the comparison itself. The Total performance does include all overhead, that is the timer is started when the first file is searched and stopped when all files have been compared. Thus, don't take the performance specified too serious!

If the Source and Destination drives are different physical drives you may reduce the run-time by specifying the option /MP. Using that option causes 2 threads to be used for reading the Source and Destination files simultaneously instead of doing that sequentially with a single thread.
This option is most useful when reading from 2 drives of similar speed e.g. a CD-ROM and a CD-RW drive (because the 2 slow accesses will be done in parallel instead of sequential), it's less useful when having drives with a great difference in speed (because reading the fast drive takes almost no time compared to reading the slow drive, thus the slow drive alone affects the overall performance).

Use the option /!C if you do not want the comparison to stop waiting for the user pressing a key for every miscompare. This option may only be useful when you use the /LOG option or redirect the output into a file, as otherwise the lines stating the miscompare scroll away too fast.

Use the option /!F if you do not want the comparison to stop whenever a file in the source location can not be found in the target location. This option may only be useful when you use the /LOG option or redirect the output into a file, as otherwise the lines stating the miscompare scroll away too fast.

You may also notice that you may have a different buffer size. This is normal, because XCOMP/2 takes the amount of physical memory into account to be efficient on one hand and to limit swapping on the other hand. Using the option /TINY XCOMP/2 will use 2 64kB buffers instead of a percentage of the RAM installed. This might be useful if you are running a memory constraint system.

Finally, to ensure that you can notice the progress, a percentage indicator will be displayed while comparing files that are larger than the given buffer size.

If you just want to see if all files exists without actually comparing them (thus saving much time for the price of possibly overlooking corrupt data), then invoke XCOMP/2 with the /SIMULATE option.

With the /CHK option you can specify the fully qualified path of the Checksum file containing the CRC32 and MD5 checksums for all Source files found. If the Checksum file does not exist already, it will be created, otherwise the Checksums within that file recoreded earlier will be compared to what has been calculated for the Source files just now. The option /!C also affects problems detected for the /CHK option.

Warning! It is expected that the files in the Checksum file do correspond to the qualified source files, that is, XCOMP/2 will not try to synchronize if one or more files have been added or deleted! I thus recommend to use the Checksum file only against files that are static, e.g. you use XCOMP/2 to calculate the Checksum file, and then burn the Source files and the Checksum file onto a CD-ROM, later on you can verify that the Source files on the CD-ROM are still valid by running XCOMP/2 against them and the Checksum file. You no longer would need to have a second copy of the files to know if they are still valid, however if XCOMP/2's Checksum calculation tell you that the Checksums no longer match, you just know the files got corrupted but you still may need to obtain an uncorrupted copy of the files!

When comparing using CD, CD-R or CR-RW media, you may get alarmed by miscompares which disappear when running the same (or limited to a subdirectory only) comparison again. I'm not sure what causes that, but I suspect that misreads of the drive (e.g. weak bits) occured. Reading the same files again may just cause different read results due to the drive's head repositioning in a different way.

Examples

As said, my primary motivation to write XCOMP/2 was to compare what has been burned onto a CD-R or CD-RW with the original data, as I do not trust write operations that have no explicit verification, e.g even if it takes some time I do verify the data when copying something to a floppy disk media.

And from my personal experience I'm right being sceptical here. I already have encountered a few CD-Rs and CD-RWs that contained currupted data. I have seen effects that best can be described as weak bits, which used to be a copy-protection mechanism for games on floppy disks. Weak bits are bits read from a media sometimes incorrectly and sometimes correctly, and that rate may even be different between different CD-ROM drives.

The technical reasons seem to be (and looking around in the Internet confirms that):

• CD-Rs and CD-RWs just contain large amounts of data, so even if the probability of a single bit failing is very low, the high number of bits on a CD may cause some faults.
• CDs were initially developed to contain audio data where it does not make much difference if a few bits are read incorrectly, the human ear would not notice the difference. For data CDs even a single bit error will cause a problem, even if much more capacity on a CD is spent for error detection and correction.
• CDs were initially rated as single speed, today's recorders can burn a the tenfold or higher speed. That of course decreases the time to burn a single bit accordingly, that is the time to position the laser and turn the laser power on to heat up the substrate gets shorter and shorter. It is easy to imagine that this can increase the failure rate, and even if the CD-R or CD-RW is rated for a high speed you may get better results buring with a lower speed.
• Different CD-ROM drives have different characteristics and may produce different result, or may even be incompatible with some burners. Combine that with increasing writing speeds, and again we have the problem that the time to read a bit gets shorter and shorter.
• When using CD-RWs instead of CD-Rs the failure rate may increase because of the different design of such medias, especially regarding the reflection. Combine that with the fact that different materials require the writing laser to operate at different power levels, and you can just hope that the media correctly tells the burner which power level to use.
• CD-ROMs initially were specified to contain 74 minutes or 650 MBs of data, to increase the capacity the limits of those specifications were reached (e.g. for 80 minutes or 700 MB CDs) or even exceeded (e.g. for 90 minutes or 800 MB CDs).
• Last but not least CD-Rs and CD-RWs are getting cheaper and cheaper, and from that cost pressure it's not hard to imagine that this will put some pressure on manufacturing for being as cheap as possible.
• At least for CD-Rs it seems that the important reflection layer that is fundamental for the CD-ROM drive being able to read the CD is the same layer that covers the top side of a CD (the side where one writes the CD title onto). If that layer is scratched even slightly, you have some worthless plastic waste in your hands (and in contrast to the bottom layer that one can't be repaired as far as I know).

Handling ISO image files (and RSJ track image files)

XCOMP/2 allows you to do a comparison at the file level, that is you need one or more source files that will be compared to one or more target files. Sometimes however, you don't have the source files have burnt onto a CD-ROM, but only the corresponding ISO image (or RSJ track image).

An ISO image is just the datastream in a single file, that makes up the contents of a CD-ROM session from the beginning to the end. Again, you want to be sure that the contents of the ISO image file you have created a CD-ROM from exactly equals wioth what has been written onto the CD-ROM.

There's a solution for that problem too, just use the ISOCOMP/2 tool, which lets you compare a CD-ROM's contents with the ISO image (or RSJ track image) file it has been burnt with.

What's new

The following items have changed between V3.10 and its predecessor 3.00:

• Fixed a problem running out of stack space when recursing into a too deeply nested directory structure. Stack space increased from 64kB to 512kB which is enough now to recurse into the deepest directory structure possible on HPFS. Thanks to Peter Skye for reporting that bug.
• Fixed that XCOMP/2 incorrectly showed SYS3 errors on FAT drives that WIN9x has accessed (and written VFAT long filenames). Thanks to Peter Skye for reporting that inconsistency (inconsistency, because it looks like WIN9x somehow messes up the VFAT in a way OS/2 APIs return strange results).
• Added a check that XCOMP/2 tries to exclude the Checksum file from the path specified for the comparism and Checksum calculation, which is very useful if the Checksum file is within that path (because the Checksum for the Checksum file itself for the current run never will be what has been recorded in the previous one, and that causes XCOMP/2 to pause and getting out of sync with errors). Note! You probably can specify the paths for the files for the comparison or Checksum calculation and the path for the Checksum file in such a way that XCOMP/2 will not always be able to figure out that the Checksum file is in the path specified for the comparison or Checksum calculation.
• Added the option /P to pause XCOMP/2 on exit.
• Added a dump of possible memory leaks when compiled in Debug mode.
• Added the option /!S to request XCOMP/2 not to recursively descent into subdirectories. Note that Linux's MD5SUM does ignore subdirectories.
• Added that the Checksum file is written or read in the format of MD5SUM, if the extension specified for the Checksum file is .MD5. Note that MD5SUM in contrast to XCOMP/2 (unless the option /!S is specified, ignores subdirectories.
• Added support for an arbitrary filename sequence of the Checksum file contents in relation to the filename sequence in the filesystem. In other words, the sequence of the filenames in the Checksum file do no longer need to be exactly the sequence the files are returned from the filesystem.
  This is especially usefuly when you burn the contents of e.g. a HPFS drive onto a CD-ROM, because the HPFS filesystem always returns the files in the alphabetically ascending sequence of the filename, whereas the CD-ROM's CDFS returns the filenames in the sequence they were burnt onto the CD-ROM which may be any arbitrary sequence.
• Changed that percentage progress indicator does not rewrite if percentage has not changed.
• Changed that the Checksum file does no longer contain full but relative paths.
• Changed that the filenames in the Checksum file are compared case insensitive, because that's the default under OS/2 (OS/2 is case-preserving, but not case-sensitive). Note, that Linux in contrast is case-sensitive.

The following items have changed between V3.00 and its predecessor 2.50:

• Fixed that running XCOMP/2 against many large files and/or very long caused an integer overflow by switching to 64-bit integer calculations.
• Fixed with a workaround, that the OS/2 profiling APIs sometimes return and earlier end than starting time.
• Added the option /CHK which adds Checksum calculation and verification support to XCOMP/2.
• Added the option /!B which silences XCOMP/2 when a severe error accessing a file at the source or target location occurs. This may be useful if you use the /LOG option or output redirection to run XCOMP/2 unattendedly, or you expect lots of miscompares that would cause lots of annoying beeps.
• Added special checking for errors caused by defective source or target medias (e.g. causing CRC or Sector Not Found errors), and informing the user that the media may be suspect.
  The rational behing that is, that I have experienced that especially high speed CD-RWs (10x writing factor) tend to have weak bits or hard bit failures. Considering the mechanical, optical and timing requirements when writing CD-RWs and CD-Rs at high speeds I'm not really surprised that sometimes a few bits toggle. Another reason that every self-burnt CD should be tested!
• Added the ISOCOMP/2 ISO image (and RSJ track image) comparison utility.
• Changed that XCOMP/2 disables the System Error handler, but handles them locally (that is no System Error popup window will show up).

The following items have changed between V2.50 and its predecessor 2.40:

• Added a workaround for a possible TVFS bug (TVFS does sometimes return files during enumeration of all directories within a (sub)directory, may be related to the mix of TVLINKing files and directories mixed).
• Added the option /SIMULATE which does all XCOMP/2 would do, but does not actually compare the files (that is, XCOMP/2 recurses into all subdirectories, opens all files to check for their existance, checks their sizes and closes them again without acutally comparing their contents). This may be useful to ensure that all files exist, can be opened and are of the same size, neglecting any possible differences. Of course, this is much faster than acutally comparing the files.
• Added that XCOMP/2 will pause if the source file can't be opened due to being locked by a process (e.g. SWAPPER.DAT), unless option /!F is specified.
• Added a port to the WIN32 environment (as I wanted to confirm that OS/2 is the superior platform also for an development environment ;-).
• Fixed that error message processing sometimes contained redundant code, removed redundancy.

The following items have changed between V2.40 and its predecessor 2.30:

• Added the options /!C (which relaces the option /NOPAUSE), and /!F
• Added the option /LOG[:[drive:\|\\server\][path\]filename] to allow XCOMP/2 to write a logfile so that you can run it unattended while still ensuring that you will not miss any problem. If you specify a filename with that option, XCOMP/2 will log all problems into that file, otherwise it will log them into the file specified by the XCOMP environment variable or if that is not available into the file XComp.Log in the directory XComp.Exe was launched from.
• Added the option /LINE to enhance problem determination. You likely will only need that option during problem determination.
• Changed that XCOMP/2 will now by default pause not only when a compared files does not match but also when a file available at the source location has no equivalent in the target location. Thanks to Timur Tabi for that suggestion.
• Fixed that using "." in the source location when the current directory is the root directory failed with an error message.

• Added that XCOMP/2 will beep when it has terminated, this will alert you when XCOMP/2 is running in the background.
• Added that XCOMP/2 will now display the directories it traverses into in addition to the file it compares in those directories.
• Fixed that if XCOMP/2 gets an error accessing a directory to compare files or to find subdirectories within (e.g. due to access right or file system problems), it will now display the directory in error and either continue or terminate (depending on the type of the error). Thanks to Wilbur E. Hong for that suggestion!

• Fixed that if an error occurs during opening of the files no filename was displayed but just the error message preventing the user to see which file(s) failed.

• Added documentation for the /TINY commandline option which uses 2 64kB buffers for the file comparisons.
• Added to display the retry counter when retrying a comparison due to a miscompare.

• Fixed a severe bug that caused to ignore miscompares within files of the same length (was introduced when I added a retry count to handle weak bit errors when reading CD-Rs in older CD-ROM drives).
• Fixed that the executable contains the correct BLDLEVEL signature.
• Added a message showing the total files compared.

• The commandline option /NOPAUSE has been added to prevent the default behaviour of V2.00 to stop further comparison when an error is detected waiting for the user to press a key. This option may be useful if you log XCOMP's output into a file (using file redirection).
  Up to V1.50 the default option was not to stop when an error was detected, which was modelled after XCOPY's behaviour, but proofed to be a bad idea when comparing much files (e.g. CD-R's or CD-RW's) as the non matching files easily scrolled away.
• Checking of some return codes has been improved.
• Added a consistency check to be sure the correct parts of the files are compared.
• The major improvement is a more intelligent interpretation of the commandline arguments regarding the source and target paths. Please see the examples for more details.
• Fixed a race condition when using the option /MP that caused to show miscompares when there were none.

CRC32 and MD5 checksum logic

The calculation of CRC32 and MD5 checksums is based on what that seems to be the standard algorithm used on the Internet. As I'm certainly not an expert in cryptology, and thus can't tell how good the quality of the CRC32 and MD5 checksums is (AFAIK it's rather good) , I used the sample codes I found on the Internet after a little searching as a starting point.

The CRC32 implementation I dereived from some freely available sample codes, the MD implementation is derived from the RSA Data Security, Inc. MD5 Message-Digest Algorithm sample implementation, which is also freely available if above identification is included on materials based on it.

For MD5SUM compatible Checksum files you may need to specify the option /!S additionally, as MD5SUM does not seem to be able to recurse into subdirectories (at least I couldn't figure it out).

If you write the Checksum file into the path of the files you requested the comparison or Checksum calculation to be done, you would modify the Checksum file in that path while XCOMP/2 has already started its Checksum calculations (and that modification would of course result into different Checksum file contents). In other words, you would remove the floor under XCOMP/2's feet!
Thus XCOMP/2 tries to detect the Checksum file in the filesystem and ignore it for the Checksum processing.

XCOMP/2 has to handle that situations, and thus you have to observe:

• the sequence of the files in the path does not need to be the same between the initial writing of the Checksum file and any comparism with the Checksum file later. For e.g. HPFS drives this is automatically the case, because that filesystem always sorts the files in the same sort order, however depending on the character set the sorting between a HPFS media and e.g. a CD-ROM media may be different.
• XCOMP/2 will exclude the Checksum file itself (for the comparison or Checksum calculation). This is necessary, because when writing the Checksum file its contents permanentely change and thus also its Checksum, while the Checksum written for the Checksum file itself was only valid during one of that changes.
• XCOMP/2 will run case-insensitive, as this is the default under OS/2. Under Linux however, MD5SUM is likely case-sensitive.

XCOMP/2 download

You are welcome to download XCOMP/2 V3.10, which includes the OS/2 version and WIN32 port of XCOMP/2 and ISOCOMP/2 (inclusive its source written with VisualAge C++) from this site. You should also be able to find it on sites connected with a higher bandwidth like e.g.Hobbes.

XCOMP/2 source code

XCOMP/2 includes the complete source code to recompile it. Just run Protect xcomp2v310 Source.zie to obtain the source code archive Source.zip out of the encrypted file Source.zie.

Note: You have to have access to an OS/2 PC to decrypt the encrypted source code archive, because I provide the decryption executable only for OS/2. These tools are for OS/2 users who are forced to used Windows sometimes, but not for users that uncritically help to extend the power of the evil's empire monopoly!

Then just unzip Source.zip, preferably at the path you want to compile XCOMP/2 from, both for OS/2 and WIN32 source code and the WIN32 executables.

To compile XCOMP/2 for OS/2 you need:

• IBM VisualAge C++ V3.6 CSD2 for OS/2 (or equivalent)
• IBM OS/2 Warp Toolkit V4.5 (or equivalent)
• just type NMAKE ["NODEBUG=1"]

To compile XCOMP/2 for WIN32 you need:

• IBM VisualAge C++ V3.6 FP2 for Windows NT (or equivalent)
• MS SDK included in IBM VisualAge C++ V3.6 (or equivalent) just type cd WIN32 followed by NMAKE ["NODEBUG=1"]