Solving Complex Puzzles (encryption)
( A study of Cool Edit'96)
By
Indian_Trail

Cool edit from Syntrillium offers alot of fun for serious students of reverse engineering. The program it self is also quite useful (for a change). Incase you're unfamilliar with it, Cool Edit is an audio app wich allows you to record signals from your line_in input on your soundcard. Very nice these days to convert Vinyl records to the CD format. Cool edit is probably most useful at making soundeffects for a lets say computer gamez. There you have it.

WE ARE NOT going to sample music, we are going to see how smart the author is when it comes to protection schemes :) Cool Edit is shareware so you won't have any problems downloading it from any good shareware site on the web, or from Syntrilliums homepage. Once you have it, run it and you'll discover that you are bound to choose two of the functions available in Cool Edit. If you register it, you can use all functions at the same time, very nice indeed. Speaking about registration, it has a serial# protection scheme, wich we already are familliar with. Lets see if we can find an echo of the correct serialnumber in memory. This is as said before done by bpr (breakpoint on range) on the adresses that you fake serial# dwells.

A bpx on GetDlgItemTextA will get you here!!!

* Reference To: USER32.GetDlgItemTextA, Ord:00EDh
                                  |
:00429669 8B3D7CF84900            mov edi, dword ptr [0049F87C]
:0042966F 6A2F                    push 0000002F
:00429671 50                      push eax
:00429672 68F4030000              push 000003F4
:00429677 56                      push esi
:00429678 FFD7                    call edi
:0042967A 8D44241C                lea eax, dword ptr [esp+1C]
:0042967E 6A0E                    push 0000000E
:00429680 50                      push eax
:00429681 33DB                    xor ebx, ebx
:00429683 68F3030000              push 000003F3
:00429688 56                      push esi
:00429689 FFD7                    call edi
:0042968B 385C241C                cmp byte ptr [esp+1C], bl
:0042968F 0F84FB000000            je 00429790
:00429695 8B0DE0574900            mov ecx, dword ptr [004957E0]

If you investigate what manipulations and comparisment that is done to your fake serial# you'll see that it's only read one time. If you try to find an echo of a correct serial number, you fail big time, there's none. It seems that the programmers has finally learnt something about protection, you'll see later that they still have alot to learn. Anyway since we can't fish a working serial from memory, we just have to make a close examination of this particular scheme. A good starting point will be that only place that Cool Edit reads our fake serial#.



:00429757 33C0                    xor eax, eax
:00429759 33D2                    xor edx, edx
:0042975B 0FBFF9                  movsx edi, cx
:0042975E 6641                    inc cx
:00429760 8A443C14                mov al, byte ptr [esp+14+edi]
:00429764 8A543C1C                mov dl, byte ptr [esp+1C+edi]
:00429768 8D1CBF                  lea ebx, dword ptr [edi+4*edi]
:0042976B 03C2                    add eax, edx
:0042976D 8D149B                  lea edx, dword ptr [ebx+4*ebx]
:00429770 03D7                    add edx, edi
:00429772 BB1A000000              mov ebx, 0000001A
:00429777 8D8410F7000000          lea eax, dword ptr [eax+edx+000000F7]
:0042977E 99                      cdq
:0042977F F7FB                    idiv ebx
:00429781 80C241                  add dl, 41
:00429784 6683F908                cmp cx, 0008
:00429788 88543C2C                mov byte ptr [esp+2C+edi], dl
:0042978C 7CC9                    jl 00429757 
Here is the only location where our serial# is used by Cool Edit. The above code use three memory locations:

    ESP+14 ------> ?????????
    ESP+1C ------> Our serial#
    ESP+2C ------> Out put from the routine

So, it use some values from esp+14 and executes some math function (basic) and stores the outcome at esp+2c. If you look at esp+2c when this routine is done, you'll find a string containing eight letters. The above loop is run eight times due to cmp cx, 0008. What now? Follow the new creation of your word at esp+2c and you'll end up here:

:004290FA 8B942408010000          mov edx, dword ptr [esp+00000108]
:00429101 0FBE0410                movsx eax, byte ptr [eax+edx]
:00429105 83E841                  sub eax, 00000041
:00429108 6683F908                cmp cx, 0008
:0042910C 89442450                mov dword ptr [esp+50], eax
:00429110 DA442450                fiadd dword ptr [esp+50]
:00429114 7CD5                    jl 004290EB
Just below this code is alot of tests on a flag esp+10.

:0042912D 817C2410BFD11800        cmp dword ptr [esp+10], 0018D1BF
:00429135 0F8491010000            je 004292CC
:0042913B 817C2410CBC7FFE5        cmp dword ptr [esp+10], E5FFC7CB
:00429143 0F8483010000            je 004292CC
:00429149 837C241001              cmp dword ptr [esp+10], 00000001
:0042914E 0F8478010000            je 004292CC
:00429154 817C241065423D34        cmp dword ptr [esp+10], 343D4265
:0042915C 0F846A010000            je 004292CC
:00429162 817C2410D7A64FC4        cmp dword ptr [esp+10], C44FA6D7
:0042916A 0F845C010000            je 004292CC
:00429170 817C241078B15301        cmp dword ptr [esp+10], 0153B178
:00429178 0F844E010000            je 004292CC
:0042917E 817C241098840100        cmp dword ptr [esp+10], 00018498
:00429186 0F8440010000            je 004292CC
:0042918C 817C2410B1E40BB0        cmp dword ptr [esp+10], B00BE4B1
:00429194 0F8432010000            je 004292CC
:0042919A 817C24109C3D3574        cmp dword ptr [esp+10], 74353D9C
:004291A2 0F8424010000            je 004292CC
:004291A8 817C2410296C065E        cmp dword ptr [esp+10], 5E066C29
:004291B0 0F8416010000            je 004292CC
:004291B6 817C24103F203424        cmp dword ptr [esp+10], 2434203F
:004291BE 0F8408010000            je 004292CC

If esp+10 is equal to any of the above values, we'll get the messagebox telling us that we have entered an invalid number. The test that fails us is the compare to see if esp+10 is 01. Change the flag esp+10 and move on. You'll end up with some calls to lstrcmpi. There are six or seven calls to compare our string with some other strings. If a match is found we return from the routine with eax set to 0003 or 0001 or 0002, depending on wich string that matched our string.

Now a lazy cracker would probably patch the code so the correct string is compared with the correct string and everything would be fine, would it?

We now have a hum of whats going on, but there's still alot of missing pieces. We havent seen what's done with our name, neither do we know if the values at esp+14 are constant and where does the correct strings come from? That's what we must find out.

Lets start with studing how our string is created.

Lets call routine at 00429757 for input string creation routine.

What do we know about it's role in the scheme? Well, it's the last link in the chain to create our string. After this routine our string is ready and nothing is done to change it. It use the values from our serial# and some values at esp+14.

Values at Esp+14

Breakpointing or tracing will show you that just after the Call to GteDlgItemTextA a word is copied to esp+14, the word is "infobari". If you continue you'll end up at 4296D0, where the letters of "infobari" is xored with each char in our name. Nothing more happens until the input string routine, wich is the final link in the chain. Here's 4296d0:

Esp+3c is where our name is stored.

:004296D0 0FBFC3                  movsx eax, bx
:004296D3 0FBFCB                  movsx ecx, bx
:004296D6 6643                    inc bx
:004296D8 99                      cdq
:004296D9 33C2                    xor eax, edx
:004296DB 2BC2                    sub eax, edx
:004296DD 83E007                  and eax, 00000007
:004296E0 33C2                    xor eax, edx
:004296E2 2BC2                    sub eax, edx
:004296E4 0FBFF8                  movsx edi, ax
:004296E7 8D4104                  lea eax, dword ptr [ecx+04] 
:004296EA 99                      cdq
:004296EB 33C2                    xor eax, edx
:004296ED 2BC2                    sub eax, edx
:004296EF 83E007                  and eax, 00000007
:004296F2 33C2                    xor eax, edx
:004296F4 2BC2                    sub eax, edx
:004296F6 8A440414                mov al, byte ptr [esp+eax+14]
:004296FA 32440C3C                xor al, byte ptr [esp+ecx+3C]
:004296FE 8D4C243C                lea ecx, dword ptr [esp+3C]
:00429702 00443C14                add byte ptr [esp+edi+14], al
:00429706 51                      push ecx
:00429707 FFD5                    call ebp
:00429709 0FBFCB                  movsx ecx, bx
:0042970C 3BC1                    cmp eax, ecx
:0042970E 7FC0                    jg 004296D0

What happen here:

It has a counter that count # times the loop has run, and an index to "infobari". The index is calculated by count+4 AND:ed with 7. It then xors the byte in "infobari" pointed to by index with a letter in our name indexed by count, that is the first time the first letter in our name and the second time the second letter and so on. After the xor it adds the sum to "infobari" where index points to. The loop is run until count is equal to the length of our name. Lets translate this into C code:

unsigned char str[]="infobari"; char name[]="your name";
char tmp; int len, index,count;
len=strlen(name);
count=0;index=0;

do
{
index=count+4; index=index&7;
tmp=str[index]^name[count];
str[index]+=tmp;
count++;
}while(count < len );


Let's translate the input string routine as well using the same variables as above.

number[]="12345678"; //My fake serial #
do
{
str[count]+=number[count];
index=(26*count)+str[count]+247;
index=index%26;
count++;
}while(count < 8);

Please notice that the routines in ASM is uncommented because we are on an intermediate lesson and you should be able to translate or at least understand my translation to C. If you don't understand it, sit down and think about it and set up math formulas for what happens and abbreviate. Like x=4t+3t+5t is abbreviated to x=12t.

Now we have recreated the part of the protection scheme that manipulates our inputs. Lets move on to see what this flag esp+10 is: You'll easily find out that the flag is set at 428ECB:

:00428EBD 837C245800              cmp dword ptr [esp+58], 00000000
:00428EC2 7507                    jne 00428ECB
:00428EC4 837C245C00              cmp dword ptr [esp+5C], 00000000
:00428EC9 7508                    jne 00428ED3
:00428ECB C744241001000000        mov [esp+10], 00000001
It's apparently depending on two other flags esp+58 and esp+5c. We need to dig deeper. Bpmb on these two flags and
:00428C55 80F920                  cmp cl, 20
:00428C58 7504                    jne 00428C5E
:00428C5A FF44245C                inc [esp+5C]


:00428DE6 8A8C0480000000          mov cl, byte ptr [esp+eax+00000080]
:00428DED 80F921                  cmp cl, 21
:00428DF0 7405                    je 00428DF7
:00428DF2 80F95F                  cmp cl, 5F
:00428DF5 7508                    jne 00428DFF
:00428DF7 C744245801000000        mov [esp+58], 00000001
:00428DFF 80F930                  cmp cl, 30


So esp+58 is set at 428DF7 and esp+5C is set at 428C5A.
Esp+58 must be non zero and esp+5c must be zero. 20h = space and 21="!" and 5F="_".
This means that our name must hold a space but not "!" or "_".

Now we'l investigate how the correct string are created. Notice that I use the term correct string even though only one or two of these strings will result in a succesful registration. Start from the lpstrcmpi and see what adress the correct string stored at then use winice breakpoint features or trace the code until something is written to that adress. You'll see that it uses lstrcpyA to copy seven strings to memory starting from code adress 428B90.

The strings are therefore defined by the programmer, so I suggest we change the name from "correct strings" to "programmer defined strings" wich we abbreviate to PDS. The PDS are:


    ESP+14 ----> HOJDIVAD    PDS1
    ESP+20 ----> HOJDIVAD    PDS2
    ESP+38 ----> LOVELOVE    PDS3
    ESP+44 ----> PEACEONE    PDS4
    ESP+2C ----> ARTHLOVE    PDS5
    ESP+6C ----> DIGITALW    PDS6
    ESP+60 ----> GODLUVSU    PDS7


Now lets concentrate on what is done with these strings.
At 428CCD6 the PDS letters are substracted with 65 (41h)


:00428CD6 806C041441              sub byte ptr [esp+eax+14], 41
:00428CDB 806C042041              sub byte ptr [esp+eax+20], 41
:00428CE0 806C043841              sub byte ptr [esp+eax+38], 41
:00428CE5 806C044441              sub byte ptr [esp+eax+44], 41
:00428CEA 806C042C41              sub byte ptr [esp+eax+2C], 41
:00428CEF 806C046C41              sub byte ptr [esp+eax+6C], 41
:00428CF4 806C046041              sub byte ptr [esp+eax+60], 41
:00428CF9 6683F908                cmp cx, 0008
:00428CFD 7CD2                    jl 00428CD1

Our name is turned into capital letters and all spaces are stripped, lets call that string for Cname (capital name). At 428EF3 the PDS are altered. Each PDS has it's own unique formula as shown below:

PDS2[index] = PDS2[index]*13+14
PDS3[index] = PDS3[index]*17+23
PDS4[index] = PDS4[index]*31+29
PDS5[index] = PDS5[index]*37+17
PDS6[index] = PDS6[index]*23+37
PDS7[index] = PDS7[index]*PDS7[index]-117


Lets input a variable X = length of Cname. The above is done in a loop wich is executed X times. If index > 7 then index is set to zero. Index ís kept when the loop is done and used in the next altering.

The PDS[index] are then substracted by 12+(each ascii value Cname) in a loop wich executes X times and if index = 7 then index is set to zero. Index is kept also here to be used in the next routine:

Each letter in the PDS[index] are xored with each of the letter in Cname. Wich is also done in a loop wich (as you already guessed) executes X times and if index = 7 then index is set to zero.

The finally manipulation with the PDS are:

PDS[index] = PDS[index]+( altered letters of infobari pointed to by index)
PDS[index] = PDS[index]%26    % = modulus ie the rest of integer division
PDS[index] = PDS[index]+65


The above is done 8 times and index is incremented throughout the loop.

Well there you have it, the complete protection scheme of Cool edit '96. Well perhaps not the complete Scheme there are a few more things. Remember that esp+10 had to be nonezero, and at the location of the compare there was some more tests of esp+10 resulting in an invalid reg number. If you want to you could study and see what those checks are for. Apart from that I have given you the complete scheme.

Well now that we know how the scheme works how do we collect a working serialnumber from it?

I say we go for the programming, I choosed C/C++ you can however use what kind of language you want.

Here is the protection scheme in C/C++.

-------------------C/C++----------------- #include
#include
#include
main()
{
unsigned char str[]="ifoobari"; unsigned char userpw[8];
unsigned char strtmp[8]; char name[]="Indian";
char number[]="12345600";char capname[40]; unsigned char correct[]="ARTHLOVE";
unsigned char copy[9];
int len,index,count=0; unsigned char tmp;
len=strlen(name);
cout.setf(ios::hex);

do
{
index=count+4;
index=index&7;
tmp=str[index]^name[count];
str[count&7]+=tmp;
count++;
}while(count

strcpy(copy,str);
count=0;

do
{
str[count]+=number[count];
index=(26*count)+str[count]+247;
index=index%26;
index+=65;
userpw[count]= index;
count++;
}while(count<8);


userpw[8]='\0';
//     Make Upper Case Letters of input name
     index=0;
     for (count=0;count < len; count++)
     {
     if (name[count]!=' ')
       {
             capname[index]=name[count]&223;
       index++;
       }
     }
     capname[index]='\0';


for (count=0;count < 8; count ++){
correct[count]-=65;

}
tmp=12;


for (count=0;count < len; count ++){
correct[count]=(17*correct[count])+23;
tmp+=capname[count];
}


index=count;
for (count=0; count < len; count++) {
index=index&7;
correct[index]-=tmp;
index++;
}


for (count=0; count < len; count++){
index=index&7;
correct[index]=correct[index]^capname[count];
index++;
}


for (count=0;count<8;count++)
{
tmp=(correct[count]+copy[count]);
tmp=tmp%26;
tmp+=65;
cout << tmp; // output the correct letters
}
return 0;
}
-----------------------

Now I've given you the source for the Protection Scheme, all you have to do is to reverse it. Make it output the correct number you shall input and you have a keygenerator.

If you don't understand C, pick it up as fast as you can. Not because you're kewl or anything if you know C but because you'll find most programming examples in C/C++.

Why haven't I given you the source for the keygenerator.

  1. You should be able to do some parts your self.
  2. I hate microsoft and the only hope in the software scene is the shareware programmers. Therefore I am concerned about them and I do care about GOOD shareware programmers.
  3. My opinion is that if you use a program to make money, you should pay for that program. But if you use a program for a hobby, you should be able to do so without paying for it UNLESS you WANNA support it.
Well See you in the DOS4GW tutorial (to be released soon I hope.)

/Indian Trail
(Saddle all the horses far on the indian trail, 'til it's time to change the key and jump into a different scale) 1