GUIDE TO (mostly) HARMLESS HACKING Vol. 3 No. 11 Through the looking glass: Finding evidence of your cracker ____________________________________________________________ by Chris Kuethe University of Alberta Department of Mathematical Sciences ckuethe@ualberta.ca You've subscribed to Bugtraq and The Happy Hacker list, bought yourself a copy of _The_Happy_Hacker_, and read _The_Cuckoo's_Egg_ a few times. It's been a very merry Christmas, with the arrival of a cable modem and a load of cash for you, so you run out and go shopping to start your own hacker lab. A week or so later, you notice that one of your machines is being an especially slow slug (Hi Sherwood!) and you've got no disk space. Guess what - you got cracked, and now it's time to clean up the mess. The only way to be sure you get it right is to restore from a clean backup - usually install media and canonical source - but let's see what the h4x0r left for us to study. In late October of this year, we experienced a rash of attacks on some workstations here at the University of Alberta's Department of Mathematical Sciences. Most of our faculty machines run RedHat 5.1 (there's a good platform to learn how to try to secure...) since it's cheap and easy to install. Workstations are often dual-boot with Windows 95, but we'll be phasing that out as we get Citrix WinFrame installed. This paper is an analysis of the compromise of one professor's machine. One fine day I was informed that we'd just had another break-in, and it was time for me to show my bosses some magic. But like a skilled card shark who's forced to use an unmarked deck, my advantage of being at the console had been tainted. Our cracker had used a decent rootkit and almost covered her tracks. In general, a rootkit is a collection of utilities a cracker will install in order to keep her root access. Things like versions of ps, ls, passwd, sh, and other fairly essential utilities will be replaced with versions containing back doors. In this way, the cracker can control how much evidence she leaves behind. Ls gets replaced so that the cracker's files don't show up, and ps is done so that her processes are not displayed either. In general, after a crack, you can bet something very important on a sniffer having been installed. Packet sniffers - programs that record network traffic - are not part of a rootkit per se, but they are nearly as loved by hackers as a buggered copy of ls. Who wouldn't want to try snarf other user's passwords? In nearly all cases, you can trust the copy of ls on the cracked box to lie like a rug. Don't bet on finding any suspicious files with it, and don't trust the file sizes or dates it reports; there's a reason why a rootkit binary is generally bigger than the real one, but we'll get there in a moment. In order to find anything interesting, you'll have to use find. Find is a clever version of 'ls -RalF | grep | grep | ... | grep '. It has a powerful matching syntax to allow precise specification of where to look and what to look for. I wasn't being picky - anything whose name began with a dot was worth looking at. The command: find / -name ".*" -ls Sandwiched in the middle of a ton of useless temporary files and the usual '.thingrc' files (settings like MS-DOS's .ini) we found '/etc/rc.d/init.d/...'. Yes, with 3 dots. One dot by itself isn't suspicious, nor are two. Play around with DOS for about two seconds and you'll see why: '.' means "this directory" and '..' means "one directory up." They exist in every directory and are necessary for the proper operation of the file system. But '...' ? That has no special reason to exist. Well, it was getting late, and I was fried after a day of class and my contacts were drying up, so I listed /etc/rc.d/init.d/ to check for this object. Nada. Just the usual System V / RedHat 5.1 init files. To see who was lying, changed my directory into /tmp/foo, the echoed the current date into a file called '...' and tried ls on it. '...' was not found. I'd found the first rootkit binary: a copy of ls written to not show the name '...' . Now that we knew that '...' was not part of a canonical distribution, I moved into to it and had a look. There were only two files; linsniffer and tcp.log. I viewed tcp.log with more and made a list of the staff who would get some unhappy news. Ps didn't show the sniffer running, but ps should not be trusted in this case, so I had to check another way. We were running in tcsh, an enhanced C-syntax shell which supports asynchronous (background) job execution. I typed './linsniffer &' which told tcsh to run the program called linsniffer in this directory, and background it. Tcsh said that was job #1, with process ID 2640. Time for another ps - and no linsniffer. Well, that wasn't too shocking. Either ps was hacked or linsniffer changed its name to something else. The kicker: 'ps 2640' reported that there were no processes available. Good enough. Ps got cracked. This was the second rootkit binary. Kill the sniffer. Now we check the obvious: /etc/passwd. There were no strange entries and all the logins worked. That is, the passwords were unchanged. In fact the only weird thing was that the file had been modified earlier in the day. An invocation of last showed us that 'bomb' had logged in for a short time around 2:35 am. That time would prove to be significant. Ain't nobody here but us chickens, and none of us is called bomb... I went and got my crack-detection disk - a locked floppy with binaries I trust - and mounted the RedHat CD. I used my clean ls and found that the real ls was about 28K, while the rootkit one was over 130K! Would anyone like to explain to me what all those extra bytes are supposed to be? File has the answer: ELF 32-bit LSB executable, Intel 80386, version 1, dynamically linked, not stripped. Aha! So when she compiled it, our script kiddie forgot to strip the file. That means that gcc left all its debugging info in the file. Indeed, stripping the program brings it down to 36K, which is about reasonable for the extra functionality (hiding certain files) that was added. Remember how I mentioned that the increased file size is important? This is where we find out why. First, new "features" have been added. Second, the binaries have verbose symbol tables, to aid debugging without having to include full debug code. And third, many script kiddies like to compile things with debugging enabled, thinking that it'll give them more debug-mode backdoors. Certainly our 'kiddie was naive enough to think so. Her copy of ls had a full symbol table, and source and was compiled from /home/users/c/chlorine/fileutils-3.13/ls.c - which is useful info. We can fetch canonical distributions and compare those against what's installed to get another clue into what she may have damaged. I naively headed for the log files, which were, of course, pure as the driven snow, but I still did find out something useful: this box seemed to have TCP wrappers installed. OK, those must have failed somehow since she got in to our system. On RH51, the TCP wrappers live in /usr/sbin/in.* so what's this in.sockd doing in /sbin? Being Naughty, that's what. I munged in.sockd through strings, and found some very interesting strings indeed. I quote: You are being logged , FUCK OFF , /bin/sh , Password: , backon . I doubt that this is part of an official RedHat release. I quickly checked the other TCP wrappers, and found that RedHat's in.rshd is 11K, and the one on the HD was 200K. OK, 2 bogus wrappers. It seems that, looking at the file dates, this cracked wrapper came out the day after RH51 was released. Spooky, huh? I noticed that these binaries, though dynamically linked, used libc5, not libc6 which we have. Sure, libc5 exists, but nothing, and I mean nothing at all uses it. Pure background compatibility code. After checking the other suspect binaries, they too used libc5. That's where strings and grep (or a pager) gets used. Now I'm getting bored of looking by hand, so lets narrow our search a little using find. Try everything in October of this year... I doubt our cracker was the patient sort - look at her mistakes so far - so she probably didn't get on before the beginning of the month. I don't claim to be a master of the find syntax, so I did this: find / -xdev -ls | grep "Oct" | grep -v "19[89][0-7]" > octfiles.txt In English: start from the root, and don't check on other drives, print out all the file names. Pass this through a grep which filters everything except for "Oct" and then another grep to filter out years that I don't care about. Sure, the 80's produced some good music (Depeche Mode) and good code (UN*X / BSD) but this is not the time to study history. One of the files reported by the find was /sbin/in.sockd. Interestingly enough, ps said that there was one unnamed process with a low (76) process id owned by uid=0, gid=26904. That group is unknown on campus here - whose is it? And how did this file get run so early so as to get that low a PID? In.sockd has that uid/gid pair... funky. It has to get called from the init scripts since this process appears on startup, with a consistently low PID. Grepping the rc.sysinit file for in.sockd, the last 2 lines of the file are this: #Start Socket Deamon exec in.sockd Yeah, sure... That's not part of the normal install. And Deamon is spelled wrong. Should a spell checker be included as an crack-detector? Well, RedHat isn't famous for poor docs and tons of typos, but it is possible to add words to a dictionary. So our cracker tried to install a backdoor and tried to disguise it by stuffing it in with some related programs. This adds credibility to my theory that our cracker has so far confined her skills to net searches for premade exploits. The second daemon that was contaminated was rshd. About 10 times as big as the standard copy, it can't be up to anything but trouble. What does rsh mean here? RemoteSHell or RootShell? Your guess is as good as mine. So far what we've found are compromised versions of ls, ps, rshd, in.sockd, and the party's just beginning. I suggest that once you're finished reading this, you do a web search for rootkit and see how many you can scrounge up and defeat. You have to know what to look for in order to be able to remove it. While the log files had been all but wiped clean, the console still had some errors printed on it, quite a few after 0235h. One of these was a refusal to serve root access to / via nfs at 0246h. That coincided perfectly with the last access time to the NFS manpage. So our script kiddie found something neat, and she tried to mount this computer via NFS, but she didn't set it up properly. All crackers, I'd say, make mistakes. If they did everything perfectly we'd never notice them and there would be no problems. But it's the problems that arise from their flaws that cause us any amount of grief. So read your manuals. The more thoroughly you know your system, the more likely you are to notice abnormalities. One of the useful things (for stopping a cracker) about NFS is that if the server goes down, all the NFS clients with directories still mounted will hang. You'll have to 120-cycle the machine to get it back. Hmmm. This presents an interesting tool opportunity: write a script to detect an NFS hack, and if a remote machine gets in, ifconfig that interface off. Granted, that presents a possible denial-of-service if authorized users get cut off. But it's useful if you don't want your workstation getting compromised. At this point I gave up. I learned what I'd set out to do - how to find the crap left behind by a cracker. Since the owner of this system had all her files on (removed) removable media there was no danger of them being in any way compromised. The ~janedoe directory was mounted off a jaz disk which she took home at night, so I just dropped the CD into her drive and reinstalled. This is why you always keep user files on a separate partition, why you always keep backups and why it's a good plan to write down where to get the sources for things you downloaded, if you can't keep the original archives. Now that we've accumulated enough evidence and we're merely spirited sluggers pulverizing and equine cadaver, it's time to consider the appropriate response. Similar to Carolyn's you-can-get-punched and you-can-go-to-jail warnings, I would suggest that a vicious hack back is not appropriate. In Canada, the RCMP does actually have their collective head out of the sand. I am not a lawyer, so don't do anything based on these words except find a lawyer of your own. With that out of the way, suffice it to say that we're big on property protection here. Aside from finding a lawyer of your own, my advice here is for you to call the national police, whoever they are. People like the RCMP, FBI, BKA and KGB probably don't mind a friendly phone call, especially if you're calling to see how you can become a better law-abiding citizen. Chances are, you'll get some really good tips, or at least some handy references. And of course you'll know someone who'll help you prosecute. My communication with RCMP's Commercial Crimes unit (that includes theft of computing and/or network services) can be summarized as follows: E-mail has no expectation of privacy. You wish email was a secret, but wake up and realize that it's riskier than a postcard. As a sysadmin, you can do _ANYTHING_ you want with your computer - since it's your responsibility either because you own it or because you are its appointed custodian - so long as you warn the users first. So I can monitor each and every byte all of my users send or receive, since they've been warned verbally, electronically and in writing, of my intent to do so. My browse of the FBI's website shows similar things. But that was only browsing. Don't run afoul of provincial or state laws regulating the interception of electronic communication either. NOTE: While I have attempted to make this reconstruction of events as accurate as possible, there's always a chance I might have misread a log entry, or have misinterpreted something. Further, this article is solely my opinion, and should not be read as the official position of my employer. Appendix A: Programs you want to have in a crack-detection kit =========== find, ps, ls, cp, rm, mv gdb, nm, strings, file, strip (gnu)tar, gzip, grep less / more vi / pico tcsh / bash / csh / sh mount These should all be statistically linked. Items separated by commas mean "all of these," items separated by forward slashes mean "pick your favorite." Appendix B: References =========== WinFrame www.citrix.com RedHat 5.1 www.redhat.com www.rootshell.com www.netspace.org/lsv-archive/bugtraq.html About the filesystem McKusik, M.K. , Joy, W.N. , Leffler, S.J. , Fabry, R.S. "A Fast File System for UNIX" Unix System Manager's Manual Computer Systems Reseach Group, Berkeley. SMM-14. April 1986 LEA and Computer Crime www.rcmp-grc.gc.ca/html/cpu-cri.htm www.fbi.gov/programs/compcrim.htm -- Chris Kuethe: System Administrator - U of A Math Dept http://www.[math.]ualberta.ca/~ckuethe/ pager: 403.917.6448 office: CAB553, x1704 cell: 903.9475 wargames@edmc.net ckuethe@ualberta.ca ckuethe@math.ualberta.ca ckuethe@gecko.math.ualberta.ca __________________________________________________________________ To subscribe to the Happy Hacker Digest, email hacker@techbroker.com with the message "subscribe hh." Unsubscribe by adding an "Un" in front of the word "subscribe". (Tough .... huh.) This is a list devoted to *legal* hacking! If you plan to use any information in this Digest or at our Web site to commit crime, go away! Foo on you! Happy Hacker is a 501 (c) (3) tax deductible organization in the United States operating under Shepherd's Fold Ministries. Yes! This is all a plot to save your immortal souls! For Windows questions, email keydet89@yahoo.com or editor@techbroker.com For Unix questions, contactunixeditor@techbroker.com For Macs, write Strider Happy Hacker Grand Pooh-bah: Carolyn Meinel ________________________________________________________________________