A General Introduction to

SunOS Systems Administration

General Introduction

The purpose of this chapter is to cover some of the basic components of the UNIX operating system and to ensure that terms are well understood before starting the bulk of the book. It describes the tasks an administrator performs, the operating system, memory, processes, disks,file systems, files, directories, users and peripheral attachments.

Administrative Tasks

An administrator is expected to perform many functions to keep the system running and ensure normal operation for the users. Sometimes this tends to be like herding cats¹. To maintain a working system there are some common functions that need to be performed:

• Configure new machines
• Configure and maintain a local network
• Handle user queries and complaints
• Maintain network continuity and uptime
• Create and maintain a good backup and recovery policy
• Create and maintain a security policy
• Install and maintain applications
• Monitor system usage and tune for performance

This document covers most of the common day-to-day actions performed by an administrator. It does not aim to make the reader a Systems Administrator directly, but rather to provide guidance on the basics of administration.

BSD, SVR3, SVR4 and SunOS

There are a few flavours of UNIX available and most Sun Microsystems equipment will soon be running a new version called Solaris 2. This is based on a merge of Berkeley Software Distribution 4.3 (or BSD for short), System V Release 3 (the V is pronounced five not vee), Xenix (pronounced zee-nix or zen-nix) and SunOS. The result is a sort of co-standard called System V Release 4 or SVR4 for short (the V in the latter case is pronounced vee!). The systems covered in this document directly relate to SunOS 4.1.3 (hereinafter referred to as SunOS). SunOS is sometimes referred to as Solaris 1. This document does not cover Solaris 2.

I have also restricted the content to SPARC based machines, although the bulk of the document will be relevant to the older machines such as Sun3 and the like.

Although it would be useful to cover applications such as WordPerfect, FrameMaker or the like, there are simply too many and most have specific quirks and thus these are not dealt with here.

A companion document entitled A Guideline to a Standard Operating Environment is available as a seperate document and covers some of the aspects of system layout that have proved useful to me over the years.

The Operating System

SunOS is a multi-user, multi-tasking operating system. It was originally based on Berkeley UNIX 4.2 BSD, but contains enhancements from BSD 4.3. It can support many users via terminals although the most common use is that of a single workstation being used by one person with a windowing system such as Massachusetts Institute of Technologies X11R5 or Sun Microsystems OpenWindows. It does not deal with external hardware interrupts in a real time fashion. This makes it unsuitable, without special software, to run industrial control applications in critical situations. There are variants of UNIX such as QNX which provide this facility.

Single and Multi-user

The operating system can exist in one of two states when running. The first is a kind of supervisory state for administrative uses called single-user mode. The second is the normal state of operation called multi-user. Single user mode is used mainly for things such as backups and so on and has a minimal set of programs running.

Kernel

The main program that the operating system is termed the kernel. From this point on, this term will be used. The word kernel is used to refer to the entire suite of underlying processes that control the activity of the machine. It is accessible through three main processes: init, swapper, and pagedaemon.

The SunOS Memory System

Every computer system uses memory. SunOS uses a memory system that encompasses all of physical memory plus an area on disk called swap. The memory system is termed virtual memory. Unlike MS-DOS which uses a small amount of memory and a pseudo-memory area, SunOS uses a single flat structure that defines the entire addressable memory space. This space can be increased by adding either physical memory or increasing swap areas. MS-DOS only allows programs in the lower 640 Kilobytes of memory and SunOS operates in a slightly similar fashion although programs run in the amount of physical memory available. This may range from 32 Megabytes on a small workstation to more than 256 Megabytes on a large server.

Processes and Programs

SunOS can run many programs pseudo-simultaneously using a single processor. Some later machines have multiple processors but SunOS can only take limited advantage of these. At any time dozens of programs, called processes, can be in memory or stored on swap. Pseudo-simultaneously simply means that although many programs are classified as running, at any instant only one process is actually executing instructions in the CPU. The system keeps a series of process queues to ensure that system time is parcelled out evenly. Fundamentally a process can be either running, sleeping, idle, I/O wait, swapped or zombie.

It is important to understand the distinction between a process and a program. A program is a set of instructions for executing a specific task. A process is an instance of that program. At any time, the kernel may be managing many instances of a program and each of these instances are processes.

The kernel keeps track of the processes by assigning them to queues depending on their current state. When a process is sleeping for example, and an event occurs which requires that process to wake up and deal with the event, then the kernel moves the process to the runnable queue where a special occurence called a context switch (discussed below) will bring it into action to deal with the event.

Process Table

The kernel also keeps track of processes via a special table called the process table. This table lists all processes being handled by the kernel. It can be accessed via a user command to see the status of the processes. A process is considered alive while it has a process table entry and has code executing. Under SunOS the table has a finite size determined by an algorithm defined in a file used when reconfiguring the kernel. This, and other values, will be discussed in the Kernel Configuration chapter.

Signals

The kernel communicates with the processes in a variety of ways but one important method is accessible to users and administrators directly. This communication takes the form of signals. When a process returns from a context switch and is about to start running again, it looks in it's signal queue to see if any signals are due for processing. It actions them according to it's programmed mechanism at this time.

These signals can be system signals such as SIGTERM which causes the process to terminate (provided the programmer has not altered the default processing) or SIGKILL which causes the process to terminate immediately.

Signals are used by programmers to take note of certain events. An example is the special process that controls terminal logins. This process reads a configuration file when it starts. If changes are made to this file, then sending a signal called SIGHUP to the process causes it to re-read the configuration file.

The fact that a process must have returned from a context switch to examine it's signal queue can cause a curious situation. If a process has called a function that attempts to read from a source that is currently out of action, such as a remote drive being inaccessible, then the function call is disk bound and will never return from the context switch. This means that it will never examine it's signal queue so no amount of sending SIGKILL signals to it will cause the process to terminate.

Zombies

Another curious situation can occur with processes. This is when a parent process has in created a child process and the parent gets into a state where it never checks the return status of the child. (If the parent dies its parent takes over the child).

During the termination of the child process, the child tries to return an exit code to the parent. Since the parent has hung, the child sits in a strange state where it has no code, no files open, no existence in fact other than an entry in the process table which can be thought of as its gravestone. This state is the zombie state. It has absolutely no effect on the system in any way other than taking up a process table slot. It also can't be killed since it is no longer a process and hence cannot return the exit code.

Process Segments and Demand Paging

Processes are composed of three segments; a text segment (the actual executable code), an initialised data segment (the preset data such as prompts and so on) and un-initialised data (data that has not been filled in yet). When a program is run, the kernel reads in chunks (called pages of the program as it needs it. This is called demand paging.

Page Faults

When the process is running and needs data from another page that is not in memory, the act of trying to access this data causes what is called a page fault. There is nothing the wrong, it's just a term. The kernel registers this, goes and gets the page, installs it in memory and tells the program to continue. In this fashion only the pages that are required to action the parts of the program that a user uses are loaded and all the extraneous pages that are hardly ever used never take up valuable memory.

When a page fault occurs, the kernel reads in the required page either from swap or the program image on the file system. This allows a process to transparently request only the pages it needs and ignore those that get called rarely.

Pagedaemon

There is a process called the pagedaemon which is triggered into life when the amount of available memory drops to a set of thresholds and this process scans physical memory looking for pages that have not been accessed since it last looked. If it finds his such a page it marks it as free in physical memory and allows that page of memory to be either used by another process or reclaimed later on. This means that a process that hardly ever uses a page of code will find that page will have to be read back in as the pagedaemon has snaffled it.

The pagedaemon achieves this by treating pages in physical memory as if they were connected in a clock face style loop. There are two pointers called hands which sweep around this loop of pages. As the first hand touches a page, it tags it. The next hand sweeps along behind the first hand at a fixed distance called the handspread. As the second hand reaches a page, it checks to see if the tag is still present from the first hand. If it is, then the page is eligible for being swapped out or just by plain freed up. As a process executes, it reads pages. The act of reading a page causes the tag in that page to be reset. This is means that the second hand ignores that page since it has just been used during the handspread time.

As the load on the machine increases, the rate at which the hands move increases. This rate of motion is called the scan rate.

Shared Libraries

Programs are defined as statically or dynamically linked. This depends on it's use of shared libraries. Shared libraries are groups of functions that are frequently called by processes. An example is the printf() function which is often used to send data tothe screen. Since it is wasteful to have each program retain a copy of the printf() function in memory, this function is loaded into memory once when it is needed from a shared library. Programs which are dynamically linked use this single copy when they require it. A statically linked program has it's own private copy.

This seems wasteful but is very important for system programs which for a number of reasons, cannot rely on the commonly used shared libraries.

Context Switches

The kernel runs through the list of processes running and parcels out time for each of them sequentially in a round-robin fashion. As each process runs, it makes function calls. Some of these function calls cause the kernel to decide that the process should be put on hold for a moment. If so it stops executing instructions for the process, saves the state of the process and activates the next process in the run queue. This is called a context switch. The process that was stopped will eventually get to run again without Never being aware that it was in a sort of coma for a fraction of a second.

Core Dumps

The kernel normally runs in a supervisory mode while user processes run in a less privileged mode. This enables the kernel to watch over user processes and take appropriate action when required. This may mean blocking the process if it attempts to perform some illegal action. In some cases it may mean destroying the process and producing what is called a core dump. This hails back to the days of core memory, but today is simply an image of the process just before it was killed by the kernel.

There are tools for examining this core dump to determine what the process was doing.

When a process calls a function that requires it to wait for some I/O to be performed, it is considered to be I/O blocked . The blocking may only occur for a fraction of time, but causes a context switch.

Re-Entrancy

At any moment there may be several instances of a program running. The kernel allows users to share the text segment of a program but not the per-process data. Thus if several may users are running the standard program for command line input called the C-Shell, the kernel simply uses the same instructions (text segment) and allocates separate data areas on a per-process basis. Code that allows this is called re-entrant code. MS-DOS is primarily non-reentrant which means that only one user can be using the machine, code or operating system functions at a time (this includes TSR's and drivers). This is because there are special fixed memory locations that are directly accessible to a user program. If more than one program were running, it would be possible for both of them to try to set or read the same value with disastrous consequences. UNIX uses a per-process data segment and a kernel interface to avoid this problem.

Since multiple instances of a program will have identical text and different initialised and un-initialised segments, the kernel needs only to have one copy of the pages required for the text segment. For this scheme to operate, the text segment must be capable of running anywhere in memory (this is called Position Independent Code or PIC) and not have any specific locations that are globally accessible for modification. A program that conforms to this is termed re-entrant. This means that at any time, multiple users can be using the same code without interfering with each other in any way.

Paging

While the system is running, physical memory pages are used or freed up. Sometimes a situation occurs when the number of physical pages is insufficient to hold all the processes code and data. When the kernel detects this it moves some pages that have not been used recently into the raw disk area known as swap. Then, when they are required again, it can retrieve them. The swap area is much faster than using a file system. This act of moving pages back and forth from the swap area is called paging and occurs quite naturally on any system. Up to a point, it does not impose a strain on the machine.

Swapping

When the amount of memory drops to the MINFREE threshold, the system starts to an perform swapping where entire program images are moved back and forth to the swap area. When the amount of free memory drops to the DESFREE threshold, the system starts what is called desperation swapping. In this case, processes that have not been active for a time will get swapped first. Since the most likely processes in this category are command shells and the like, this degrades the system performance dramatically.

Thrashing

When a significant number of processes are in memory, the kernel has to keep moving a pages in and out rapidly to parcel time effectively. This can sometimes lead to a state where more time is spent moving pages about than actually executing the instructions.

The result is a large number of processes all frantically trying to use pages that are being swapped in and out at a maniacal pace. This is called thrashing. If a system gets to this state, the only real solution is either to reduce the number or size of programs or buy more memory!

Disks and File Systems

Root and Usr

A standard system has at least one disk broken up into a minimum of three partitionsTwo of these partitions hold a hierarchical structure of directories and files called a file system. The standard partitioning requires the existence of a root file system, a (pronounced user or you-ess-are) file system and an area for swapping. All other structures are optional. A disk can have up to eight partitions and each partition can be accessed either directly (outside the normal file system handling routines) as what is known as a raw partition (used mainly for swap but is often used by database products such as Ingres or Oracle for high speed data access), or via the file system handling routines as what is called a block structured device.

Mounting Filesystems

Unlike MS-DOS which uses the concept of drives called C:, D: and so on, UNIX only has one view of the data structure. When the system boots, it loads the root file system and attaches the usr file system. All remaining file systems are attached to these loaded file systems. File systems are attached in a process called mounting. The process is one of associating a directory (called a mountpoint) on an existing file system with the of the directory tree that comprises the new file system. From the time a file system is mounted onto the directory, that file system becomes part of a bigger file system. File systems can be mounted in this fashion across a network in a manner dealt with in the Network File System document..

A mountpoint is simply a directory that is used to mount a filesystem. A file system must be mounted on a directory to be accessed by all normal user methods. The act of a mounting a filesystem to a directory/mountpoint is called mounting and is controlled by the mount(8) program. File systems can be multiply mounted on different mountpoints but only via NFS (or something similar). They can be dis-mounted with the umount(8) program (notice the u-mount not un-mount). A filesystem has no implied mountpoint associated with it.

The root file system holds all the system files and the mountpoint for the usr partition to be attached to. The usr file system consists of the programs and libraries needed to run the basic system. It may contain other programs such as applications.

Files and Directories

Everything is a file from the operating system perspective. Terminals, drives, memory, directories and so on. It is quite feasible to open physical memory as a file and read it given sufficient privilege. In fact under SVR4, even processes themselves are visible as a sort of file in a special directory. All files and directories are owned by somebody, belong to some group of users and have permissions associated with them. This allows security to be exerted at the individual file level.

There are several file types. These reflect the need to distinguish between various objects. The basic file types are regular, directories, character special, block special, socket, symbolic link and named pipes. A regular file can be either a text file, a program or a set of data such as a database or it's index. A directory is a regular file that is treated as a table of entries describing other files. Character special and Block special files are what are called hooks into the driver routines either in the kernel or into a loadable device driver. A socket is a special hook used for inter-process communication either within the machine or between machines. Symbolic links are discussed later, but are basically names that point to other files. A pipe is a method of communication between processes and a named pipe is a pipe that has a filename associated with it.

God and Users

There are basically two classes of users on the system. God and everybody else. There is considerable debate raging about this, but to use the system now, we must accept that one user has complete, unlimited power over the system. This user is called the superuser or root user.

To get access to a UNIX system, the user has to log in. This requires a name to login to the system with and a password. This information is held in a system file detailed in the System Files chapter. Once logged in correctly, that user has power only over files that have permissions set to allow that user control.

Switching User

At any time, a user may switch user. This is done with the su(1v) command and changes them into the user they desire. To do this they must know the new users login name and password. Since they can change to another user, they can also change to the superuser providing they know the root password. This is obviously dangerous and so it is prudent to reduce the number of users who can change to the superuser and have the root password be relatively obscure.

The issue of passwords is a thorny one. SunOS uses the Data Encryption Standard (DES) which is a one-way system for password encryption. Although it is suspected that the National Security Agency (NSA) in the U.S.A. is able to crack passwords directly, it is practically impossible to be given an encrypted password and extract the original word. However, there are several freely available dictionary matching password cracking packages on the market and given an average machine they can crack 30-40% of the passwords in a reasonable time frame. The reason is that users choose passwords with some meaning to them. Car registration, names, and so on. I have found that passwords based on the concatenation of unlikely words tend to fool the dictionary based crackers. For example: beerbabble, bloggsding, writeonly.

When a user logs in they have an environment setup for them through special scripts. They have a home directory and belong to certain groups. This setup is solely the responsibility of the administrator.

The su(1v) command can take two forms. The first simply changes the name and adopts the basic environment variables associated with the new user. The second effectively acts as if the user had directly logged in as that user. Any user wishing to switch user to root, should use the full pathname of the su command: /bin/su , rather than the short name for security purposes.

The superuser or root can access any resource on the system. It is perfectly possible for a root user to blow away the entire system with eight keystrokes. (I did this accidentally on a running system in the early eighties)

The Machine

A computer system consists of a main unit and attached peripherals. These peripherals may include terminals, printers, plotters, tape units and disk drives. (Networking is dealt with separately.)

SBus

The main unit consists of a motherboard with on-board memory, power supply, keyboard, mouse and attached SBus cards. The SBus cards are such things as frame buffers for graphics screens, Prestoserve cards to improve network data flow for file systems and serial and parallel card controllers.

SCSI

The motherboard has an internal Simple Computer System Interface (SCSI, pronounced scuzzy) bus and at least one external SCSI port. This is SCSI bus esp0. If there are additional SBus cards or ports then these are numbered from esp1 up.

Target ID

Devices attached to a SCSI bus have a special identification called a Target ID. These target ID's must be unique and be in the range 0 to 6. Only seven devices can be attached to a single SCSI port (there is an ID 7, but this is the SCSI controller and doesn't count). It is suggested that asynchronous devices (such as tape units and older disk drives) be placed on a different path from synchronous devices (such as newer disk drives) when multiple SCSI ports are present.

Devices are chained together starting from the port on the main unit. There is a maximum length associated with a SCSI chain. This is roughly six metres. There is a new standard called Differential SCSI which allows a maximum SCSI chain length of three metres. When attaching units together assume each unit has roughly 500 mm of cable internally. More precise figures can be obtained from the manufacturer, but if such detail is required, it may be time either to shorten the cables or purchase a second SCSI card!

The cables have one of three styles of connector. These are:

SCSI-1 Also known as Thick or D50. This is a D shaped connector with 50 round pins arranged in three rows in a D shape.

SCSI-2 Also known as Thin or Honda. This small connector has 50 fine flat pins arranged in two rows.

CEN Also known as Centronics. This looks like a PC printer cable connector, only larger.

Termination

All SCSI chains must be terminated at both ends. Since one end of the SCSI chain is connected to the main system unit, that end does not have to be worried about as the termination has already been taken care of by the manufacturer. The end that connects out of the main system unit does need explicit termination. This is done with a special connector called a terminator.

The terminator can be either internal or external. An internal terminator is usually provided by a resistor pack or jumper (sometimes called TERMPWR) placed on the controller for the device. Most drives tend to have this terminator provided. External terminators come in two flavours; passive and active. An active terminator (sometimes called a regulated terminator) can sometimes be identified by the presence of a green LED although Sun's terminators have a LED and some are passive. It is better to ensure you simply purchase Active terminators. Active terminators are used on fast SCSI busses as they provide better impedance matching for the bus. Synchronous devices like late model drives can tolerate passive terminators, but all devices prefer active ones.

A SCSI bus has one termination inside the CPU unit and one at the end of the chain. They can sometimes tolerate three terminators, where the last device has an internal terminator and one external, but not four. Experience suggests that three terminators load the bus to the extent that it doesn't function. This tolerance is a function of the bus characteristics. If there are four terminations then the controller can be seen by a probing the SCSI chain, but will not respond to the kernel. This sometimes happens when a drive housing has an extra drive inserted where the existing and new drive have internal termination. Given all this, it is simply better to ensure that you have exactly two terminators!

A word of caution: It is not wise to use the 50 pin connectors on fast SCSI chains.

When shipped, the kernel is coded for a standard set of target ID's. These are:

0 disk (sd3)

1 disk (sd1)

2 disk (sd2)

3 disk (sd0)

4 tape unit (st0)

5 tape unit (st1)

6 CDROM unit (sr0)

Whether these devices are external or internal is irrelevant. These targets are not fixed and can be altered by customizing the kernel although the use of more than four disks on a SCSI chain may lead to performance degradation.