back

documents

Learning to Administrate Linux by Example

Examples for RedHat-based systems

Creation: 2006/01/04

^{Simple note:

Commands prepended with $ can be done as a regular user. Those with # need to be done as root, or via sudo.
While this document was written for RedHat-based systems, a great many portions of it will carry-over to other distributions, as well as carrying over to BSD-based systems.

Key
RED areas are those that are incomplete or have not been double-checked yet. Use with caution or send updates/information to david.p@hostdime.com.
GREEN areas are recommended responses to explain the previously-addressed error to a client.}

Contents

Configuring
Maintenance
Services
Network
Load Issues

Crashing Machines

Out of Memory
Kernel Panic
Frozen Console
Disk Crashes

Tools
- man
- fsck
- rm
- top
- vi
- ssh
- iptables

Configuring

/etc/fstab - Disk Mounting

Although it is a file, and not a device or tool, fstab does have it's own man-page

Where to mount a partition
What filesystem a partition uses
How many partitions/disks you're using
Who may mount different partitions
Who owns the information on a partition when it is mounted

Back to document index

Network

DHCP:
```
# network-config
```
Select DHCP. You should be good to go.

Static:

# network-config

De-Select DHCP. Enter your IP, gateway, nameserver, & !!EDIT!!

Check that this did the trick:

$ ping -c 5 google.com

(or any other generic site that allows ping requests/replies). If you get a response, something like:

PING google.com (64.233.167.99): 56 data bytes
64 bytes from 64.233.167.99: icmp_seq=0 ttl=242 time=42.258ms
64 bytes from 64.233.167.99: icmp_seq=0 ttl=242 time=42.258ms
64 bytes from 64.233.167.99: icmp_seq=0 ttl=242 time=42.258ms
64 bytes from 64.233.167.99: icmp_seq=0 ttl=242 time=42.258ms
64 bytes from 64.233.167.99: icmp_seq=0 ttl=242 time=42.258ms
--- google.com ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
rount-trip min/avg/max/stddev = 42.258/42.258/42.258/0 ms

Then you're good to go! If not, then something else isn't right.

Back to document index

BIOS

Settings
I have noticed after some time that we have a lot of different BIOS on the motherboards here; however, almost all of them are rather simple to figure out if you are willing to take the time to cycle through each menu/screen once or twice. The following are some common settings that should be on pretty much all of our servers, little things that we do/don't need:
- Power-On Resume: ON Somewhere in power-options, there should be an option of Power-On Resume or Power-Interrupt Choice or something similar. This is what the computer will do after power has been stopped. You have the options of "off" "prior/previous" or "on" - meaning that after the machine has lost power, should it come back on, go to the prior/previous state, or stay off. In the case that we ever did have power fail (bad power-strip, fuse, or otherwise), we do want all machines to power themselves back on so that we don't have to run around click a bunch of on-buttons.
- USB: DISABLED We use PS/2 keyboards when we do have to console a machine, and have no need for these machines having the USB ports running/enabled. Disable them.
- SERIAL: DISABLED We also do not use serial connections here, disable these.
- Shared Video Memory Allocation (Size): 4MB/8MB We most often use integrated (on-motherboard) video cards. Since these share off of the RAM, having them set higher, means more RAM is allocated (set aside) for use by the video-card. As none of our servers have monitors attached 99.9% of the time, setting aside RAM for graphical purposes is rather silly. Some of these come set anywhere from 64MB to 128MB, which is just an additional 64/128MB that could be getting used by the system and isn't. Please set these to the smallest value it will allow.
- Floppy Disk: DISABLED Most of the computers here do not have a floppy drive installed or attached. Most bios still seem to set this default to a 3-1/4" disk-drive. You can disable this as it is unnecessary.
- Quick-Boot: ENABLED/ON Quick-boot allows the machine to boot without a full memory scan and other checks at each boot. We want our machines down as little as possible, so enabling a slightly-faster boot never hurts.
- AC'97 On-Board Sound: DISABLED/OFF This is the integrated sound-card. As we have no speakers or headphones hooked up to any of our systems there's no need to have this enabled.
- ONBOARD LAN 10/100 ETHERNET: ENABLED/ON Any combination of the prior string should be on, as 99%+ of the time we use the onboard 10/100 card for all networking purposes.

Back to document index

Boot-Sequence/Order
All of the machines here should have the same boot-sequence set in their BIOS:
1. DISABLED
2. Hard-Disk 1
3. DISABLED
We do not need CDROM/Floppy/USB-FDD or any other option ever enabled. The reason why Hard-Disk 1 is our second option and the first is DISABLED is so that when/if we ever need to boot to PXE (LAN-ROM), we can simply change the first option to LAN/ROM and leave the rest intact.

Back to document index

Maintenance

Installing a New Kernel

On machines currently using a 2.4 series kernel, only update with a 2.4!
On machines currently using a 2.6 series kernel, only update with a 2.6!

Grab the latest kernel RPM:
- Kernel 2.4 series: http://isoredirect.centos.org/centos/3/updates/i386/RPMS/
- Kernel 2.6 series: http://isoredirect.centos.org/centos/4/updates/i386/RPMS/
Do a search (^f in most modern browsers) for "kernel-smp" and in almost all instances use the most recent that says the right architecture.

Save kernel RPM to server:

# wget http://isodirect.centos.org/centos/#/updates/i386/RPMS/kernel-smp-####.rpm

Extract and install rpm:
```
# rpm -ivh /path/to/kernel-smp-####.rpm
```
Edit /boot/grub/grub.conf to reflect the new kernel. ^{Actually, you are just checking to make sure that the .rpm installed it correctly, as the entry should be there)}Add a new boot block (leaving the old still-intact) with the new KERNEL-ID:
```
title CentOS (KERNEL-ID)
	root (hd0,2)
	kernel /boot/vmlinuz-KERNEL-ID ro root=/dev/hda5 rhgb quiet
	initrd /boot/vmlinuz-KERNEL-ID.img
```
Add this as the top entry. Otherwise, you will need to also change the line "default=0" to reflect which entry it is (top is 0, from there on down add numerically).
Run kudzu, the Red Hat hardware-detector/configuration-tool:
```
# kudzu
```
Reboot into new kernel (system should automatically go into new kernel since we made it default via /boot/grub/grub.conf):
```
# reboot
```
Now that machine is back up, make sure we're in the new kernel environment:
```
$ uname -a
```

Back to document index

Going into Single-User Mode

Power-on the machine.
After the BIOS screens have run, you will be prompted by the GRUB boot interface. There may be multiple lines; however, the default (one that will boot), will be highlighted. Press the escape ^esc key to stop the boot-countdown.
Now that the boot-countdown is halted, select the kernel you wish to boot into (or, more often, leave the default one highlighted), and press "e" to edit the boot command.
You will be greeted by a new edit screen. Choose (using your up/down arrow keys), the line that ends with something similar to:
```
root=LABEL=/
```
or perhaps
```
root=/dev/hda5
```
To this line (at the end of it), add the text:
```
single
```
and press enter, this will boot you into single-user mode.

Back to document index

Resetting root Password

go into single-user mode

# passwd
Changing password for user root.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.

Back to document index

Changing root Password Temporarily

Boot into single-user
Making a backup-copy of the shadow file:
```
# cp /etc/shadow /etc/shadow.back
```

Create your own password:

# passwd
Changing password for user root.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.

Boot into runlevel 3 or do whatever else you needed to do:
```
telinit 3
```
Move the backup copy of /etc/shadow back in place (thus restoring their root password):
```
# mv /etc/shadow.back /etc/shadow
mv: overwrite `/etc/shadow'? y
```
Exit/quit/whatever else

Performing a File-System Check

Information about fsck

First, make sure that the filesystem you are going to fsck is not mounted. Perhaps you need to fsck /dev/hdb1, you would do:
```
$ mount | grep /dev/hdb1
```
If you get a response line, perhaps something like:
```
/dev/hdb1 on /backup type ext3 (rw,usrquota,noatime)
```
Then you need to unmount this partition before you proceed. If get no response, skip to step 3.
Unmount the partition:
```
# umount /dev/hdb1
```
Do the fsck:
```
# fsck -y /dev/hdb1
```
The prior command may run for... seconds, minutes, hours, days... you name it. This all depends on how much work has to be done to the filesystem. Sometimes it will exist happily; othertimes, it may tell you to run fsck again. If it does, run it again. Otherwise, reboot (or remount) and enjoy.
Remount the partition:
```
# mount /dev/hdb1
```

Back to document index

Going into low-level /bin/bash

Power-on the machine.
After the BIOS screens have run, you will be prompted by the GRUB boot interface. There may be multiple lines; however, the default (one that will boot), will be highlighted. Press the escape ^esc key to stop the boot-countdown.
Now that the boot-countdown is halted, select the kernel you wish to boot into (or, more often, leave the default one highlighted), and press "e" to edit the boot command.
You will be greeted by a new edit screen. Choose (using your up/down arrow keys), the line that ends with something similar to:
```
root=LABEL=/
```
or perhaps
```
root=/dev/hda5
```
To this line (at the end of it), add the text:
```
init=/bin/bash
```
and press enter, this will boot you into /bin/bash

Back to document index

/bin/bash fsck'ing

Filesystem Error:
Press ctrl+D for maintenance or enter to continue

boot it into /bin/bash

Boot into /bin/bash
Do an fsck

Back to document index

Services

httpd - Apache, Web-Server

The httpd service/process can be called upon to be started, stopped, or restarted by:
```
# service httpd ___
```
Where ___ is start/stop/restart.
You can easily check the configuration file of httpd for errors by doing:
```
# service httpd configtest
```
If you get any response other than:
```
Syntax OK
```
then start fixing!

Back to document index

exim - eMail

# service exim ___

Back to document index

named - Bind/DNS

# service named ___

Back to document index

mysql - Database

# service mysql ___

Back to document index

chkservd - Check Service Daemon

# service chkservd ___

Back to document index

Network

Network Connection/Finding IP:

Check to make sure the ethernet cable is plugged in.
If ethernet cable is plugged in, login to console and type:
```
# mii-tool
```
Correct Response should contain "link ok" such as:
```
eth0: negotiated 100baseTx-FD, link ok
```
or
```
eth0: no autonegotiation, 10baseT-HD, link ok
```
Bad Response:
```
eth0: no link
```
If you got the Correct Response, continue; if you got the Bad Response, check your cables/router/hub.
Bring up the local loopback connection by typing
```
# ifup lo
```
Now bring the ethernet connection up, by typing
```
# ifup eth0
```
- If the connection was not up, you should see: (on a DHCP network)
```
Determining IP information for eth0... done.
```
  This means you are now connected and have an IP.
- If eth0 is already up, you may either get no response, or something similar to:
```
Determining IP information for eth0...dhclient(####) is already running - exiting.
	failed.
```
  Depending on whether you are on a DHCP or Static IP.
Either of these responses means you have an IP address that you can find by typing:
```
$ ifconfig eth0 | grep "inet "
```
This should give you a response of:
```
inet addr:##.#.#.### Bcast:##.#.#.### Mask:###.###.###.#
```
Where the #'s directly after inet addr is your IP address.

Back to document index

Block an IP with iptables

# iptables -I INPUT -j DROP -s THE-IP-HERE

I

INPUT

j

s

DROP

# iptables -nL | grep THE-IP-HERE

# vi /etc/sysconfig/iptables-config

# Additional iptables modules (nat helper)
# Default: -empty-
#IPTABLES_MODULES="ip_nat_ftp"

# Save current firewall rules on stop.
# Value: yes|no,  default: no
#IPTABLES_SAVE_ON_STOP="no"

# Save current firewall rules on restart.
# Value: yes|no,  default: no
#IPTABLES_SAVE_ON_RESTART="no"

# Save (and restore) rule counter.
# Value: yes|no,  default: no
#IPTABLES_SAVE_COUNTER="no"

# Numeric status output
# Value: yes|no,  default: no
#IPTABLES_STATUS_NUMERIC="no"

# Additional iptables modules (nat helper)
# Default: -empty-
#IPTABLES_MODULES="ip_nat_ftp"

# Save current firewall rules on stop.
# Value: yes|no,  default: no
IPTABLES_SAVE_ON_STOP="yes"

# Save current firewall rules on restart.
# Value: yes|no,  default: no
IPTABLES_SAVE_ON_RESTART="yes"

# Save (and restore) rule counter.
# Value: yes|no,  default: no
IPTABLES_SAVE_COUNTER="yes"

# Numeric status output
# Value: yes|no,  default: no
#IPTABLES_STATUS_NUMERIC="no"

Back to document index

Unblocking an IP with iptables

iptables -nL --line-numbers | grep THE-IP-HERE

L

n

line-numbers

### DROP	all	--	##.##.###.###	0.0.0.0/0

# iptables -D INPUT ###

D

INPUT

line-number

# iptables -nL | grep THE-IP-HERE

Back to document index

Blocking Abusive IPs

(Redirects from "Load Issues from DOS/DDOS")

A quick way to see if anyone is hitting your server is the following command-line/script:
```
$ netstat -an | awk '{ print $5 }' | cut -d ":" -f 1 | sort | uniq -c | sort -n | grep -v [a-zA-Z]
```
Now that you have the output from this, are any of the numbers in the first column greater than 30? 50? 100? Unless your server is used specifically by one or two main IPs (inner-office, etc), then most often any IP address with more than 50 (sometimes less) connections is causing your server to build a higher load or even locking out true users. So, assuming that there is a connection with more than 30 connections, it is probably in your best interest to block them.
If there are many connections in excess of 50 or so, rather than manually blocking each, check out my DDOS script. This will add blocks for anyone with 50-399 connections to the server (and automatically check for who has that many).

Back to document index

Load Issues

If your machine is not as responsive as usual, best thing to do is check to see what the CPU Load looks like. This can be done in a multitude of ways, but perhaps the simplest is to type:

$ w

This will give you an unformatted look at your machine's load averages. For every CPU that you have, normal may bring 2-3 load per CPU. Thus, if you have a dual-core machine with a 2 load, this is discussed as if it were a single-core machine with a 1 load. Machines can handle loadtimes into the thousands without crashing; however, sluggishness is normally very noticable when you begin to hit 5 or more per CPU. Here we discuss a few reasons why load gets high, and a few simple ways to deal with it.
Back to document index

Load Issues from DOS/DDOS

Redirect to Blocking Abusive IPs

Back to document index

Load Issues from Specific Processes

kill PID

kill -9 PID

ps aux | grep X

ps aux | grep perl

which

_{This happens most often it seems with forums and users with intensive mysql usage}

# ls -l /proc/PID/cwd

lrwxrwxrwx  1  httpd httpd 0 Jan 1  12:00  PID/cwd -> /home/USERNAME/public_html/forum

_{Similarly, for a perl process you may notice it is commonly run by nobody, and httpd is generally run as nobody, www, apache, or perhaps httpd.}

Back to document index

Load Issues from Specific Users

ps aux | awk '{ print $1 }' | sort | uniq -c | sort -n

sort

c

uniq

Load Issues from Specific Processes

ps aux | grep fred

ps aux | grep USER | awk '{ print $2 }' | xargs kill

for pid in `ps aux | grep USER | awk '{ print $2 }'`; do kill $pid; done

limits.conf

/etc/security/

# vi /etc/security/limits.conf

^{(see more on vi)}

username/groupname   type-of-limit   what-to-limit   limit-value

Thus, if we have user USERNAME who is running too many processes, we may add the following:
```
USERNAME   hard   nproc   20
```
This will limit the user USERNAME to a hard value of his number of processes to 20 or less. You may also use soft in place of hard to allow occasionally bursts above this number.
The other type of limit we will frequently use is for accounts that do not run too many processes, but run one or two processes that put a high load on the machine, for them we will use:
```
USERNAME   hard   cpu   1
```
Again, we can exchange hard for soft, and this limits their max cpu time in minutes to 1. The limits.conf is a powerful file, but that is all we'll touch on it for now.

Back to document index

Load Issues from exim

exim -bpc

exiqgrep -iz | wc -l

exiqgrep -iz | xargs exim -Mrm

Message MESSAGE-ID has been removed

Line Mismatch MESSAGE-ID

rm -fv /var/spool/exim/input/*/MESSAGE-ID*

^{(see more on rm)}

Back to document index

Load Issues from httpd/Apache

blocking abusive IPs

Check how many httpd processes are running:
```
$ ps aux | grep http | wc -l
```
If this results in something less than 100, no worries, and httpd is probably not overloading your machine. If it's more than 100, continue...
Check out the MaxClients from the httpd.conf:
```
$ grep MaxClient /usr/local/apache/conf/httpd.conf
```
Sometimes a customer/client will get overly hopefully and believe that their machine can handle 500+ clients... Generally, it can't. Setting this to a more reasonable 100-200 (or a nice median of 150) is better.
```
# vi /usr/local/apache/conf/httpd.conf
```
Then, use "/" to search:
```
/MaxClient
```
then "i" to edit, and change the value.
Now reload apache:
```
# service httpd restart
```

Back to document index

Crashing Machines

Out of Memory

When a machine crashes due to being out of memory, this is because it was attempting to write something to short-term memory (RAM or swap) and no longer had any place to put this information (because both RAM and SWAP are fully used). Most likely this is due to having a machine running a poorly written script (or abusive script/program) or perhaps just being configured such that programs such as Apache can try to handle too many connections, etc. Solving this is a matter of configuring the machine with more RAM/SWAP, or checking your programs for resource usage and your users for any odd/new scripts that may've taken a ton of memory. Back to document index

Kernel Panic

Back to document index

Frozen Console

Back to document index

Disk Crashes

Back to document index

Tools

man

man man

man

Back to document index

fsck

f

s

c

k

```
fsck -y /dev/partition
```
Using "y" will answer "yes" automatically to any questions asked while fsck is running.

Performing a File-System Check

Back to document index

rm

r

m

does not

permanently

permanently

gone, forever

```
rm -f
```
Using "f" forces removal of a file, without prompting you with a choice to remove it or not.
```
rm -i
```
Using "i" will have rm inquire interactively for each file whether you wish to remove it or not.
```
rm -r
```
Using "r" will remove files recursively.
```
rm -v
```
Using "v" will make the output verbose, showing each file removal to output.

last

$ rm -if file
$
$ rm -fi file
rm: remove regular file `file'?

Back to document index

top

top

Use "q" to exit top at any time.
Pressing "z" on more modern versions of top will colorize the output making it a little easier to decipher. "z" is not a start-up call though, and it will not work by
```
top -z
```
```
top -i
```
Using "i" toggles whether or not idle-processes are shown. "i" can be called from inside top after it has been started.
```
top -d #
```
Using "d" toggles the interval in seconds (#) of how often top should update the display. "d" can be called from inside top after it has been started. A time of "0" is instant, realtime, and may require ^c to cancel/quit. ^{Some systems also use "s" for this call.}
```
top -u USER
```
Using "u" toggles which user is displayed. By default, all users are displayed. "u" can be called from inside top after it has been started.
Pressing "k" will prompt you for a PID to kill. On some systems you will then be prompted for a signal code to send to the program to stop it. The most popular are [15] & [9].

Back to document index

vi

vi

vi /path/filename

vi temp.file

i

i

access these by pressing [esc] then ":" then the command

q!

wq

/TEXT

##

not

s/ORIGINAL/NEW/g

%s/ORIGINAL/NEW/g

Back to document index

ssh

-v

-p #

Back to document index

iptables

Block an IP with iptables
Unblocking an IP with iptables

Back to document index