Sunday, October 3, 2010

FTP and VSFTPD in LINUX


Introduction
The File Transfer Protocol (FTP) is used as one of the most common means of copying files between servers over the Internet. Most web based download sites use the built in FTP capabilities of web browsers and therefore most server oriented operating systems usually include an FTP server application as part of the software suite. Linux is no exception.
This chapter will show you how to convert your Linux box into an FTP server using the default Very Secure FTP Daemon (VSFTPD) package included in Fedora.
FTP Overview
FTP relies on a pair of TCP ports to get the job done. It operates in two connection channels as I'll explain:
FTP Control Channel, TCP Port 21: All commands you send and the ftp server's responses to those commands will go over the control connection, but any data sent back (such as "ls" directory lists or actual file data in either direction) will go over the data connection.
FTP Data Channel, TCP Port 20: This port is used for all subsequent data transfers between the client and server.
In addition to these channels, there are several varieties of FTP.
Types of FTP
From a networking perspective, the two main types of FTP are active and passive. In active FTP, the FTP server initiates a data transfer connection back to the client. For passive FTP, the connection is initiated from the FTP client. These are illustrated in Figure 15-1.
Figure 15-1 Active And Passive FTP Illustrated

From a user management perspective there are also two types of FTP: regular FTP in which files are transferred using the username and password of a regular user FTP server, and anonymous FTP in which general access is provided to the FTP server using a well known universal login method.
Take a closer look at each type.
Active FTP
The sequence of events for active FTP is:
1.      Your client connects to the FTP server by establishing an FTP control connection to port 21 of the server. Your commands such as 'ls' and 'get' are sent over this connection.
2.      Whenever the client requests data over the control connection, the server initiates data transfer connections back to the client. The source port of these data transfer connections is always port 20 on the server, and the destination port is a high port (greater than 1024) on the client.
3.      Thus the ls listing that you asked for comes back over the port 20 to high port connection, not the port 21 control connection.
FTP active mode therefore transfers data in a counter intuitive way to the TCP standard, as it selects port 20 as it's source port (not a random high port that's greater than 1024) and connects back to the client on a random high port that has been pre-negotiated on the port 21 control connection.
Active FTP may fail in cases where the client is protected from the Internet via many to one NAT (masquerading). This is because the firewall will not know which of the many servers behind it should receive the return connection.
Passive FTP
Passive FTP works differently:
1.      Your client connects to the FTP server by establishing an FTP control connection to port 21 of the server. Your commands such as ls and get are sent over that connection.
2.      Whenever the client requests data over the control connection, the client initiates the data transfer connections to the server. The source port of these data transfer connections is always a high port on the client with a destination port of a high port on the server.
Passive FTP should be viewed as the server never making an active attempt to connect to the client for FTP data transfers. Because client always initiates the required connections, passive FTP works better for clients protected by a firewall.
As Windows defaults to active FTP, and Linux defaults to passive, you'll probably have to accommodate both forms when deciding upon a security policy for your FTP server.
Regular FTP
By default, the VSFTPD package allows regular Linux users to copy files to and from their home directories with an FTP client using their Linux usernames and passwords as their login credentials.
VSFTPD also has the option of allowing this type of access to only a group of Linux users, enabling you to restrict the addition of new files to your system to authorized personnel.
The disadvantage of regular FTP is that it isn't suitable for general download distribution of software as everyone either has to get a unique Linux user account or has to use a shared username and password. Anonymous FTP allows you to avoid this difficulty.
Anonymous FTP
Anonymous FTP is the choice of Web sites that need to exchange files with numerous unknown remote users. Common uses include downloading software updates and MP3s and uploading diagnostic information for a technical support engineers' attention. Unlike regular FTP where you login with a preconfigured Linux username and password, anonymous FTP requires only a username of anonymous and your email address for the password. Once logged in to a VSFTPD server, you automatically have access to only the default anonymous FTP directory (/var/ftp in the case of VSFTPD) and all its subdirectories.
Using anonymous FTP as a remote user is fairly straight forward. VSFTPD can be configured to support user-based and or anonymous FTP in its configuration file which you'll see later.
How To Download And Install VSFTPD
Most Linux software products are available in a precompiled package format. Downloading and installing packages isn't hard. It is best to use the latest version of VSFTPD.
When searching for the file, remember that the VSFTPD packages' filename usually starts with the word vsftpd followed by a version number, as in vsftpd-1.2.1-5.i386.rpm for Redhat/Fedora or vsftpd_2.0.4-0ubuntu4_i386.deb for Ubuntu.
How To Get VSFTPD Started
With Fedora, Redhat, Ubunbtu and Debian You can start, stop, or restart VSFTPD after booting by using these commands:
[root@bigboy tmp]# /etc/init.d/vsftpd start
[root@bigboy tmp]# /etc/init.d/vsftpd stop
[root@bigboy tmp]# /etc/init.d/vsftpd restart
With Redhat / Fedora you can configure VSFTPD to start at boot you can use the chkconfig command.
[root@bigboy tmp]# chkconfig vsftpd on
With Ubuntu / Debian the sysv-rc-conf command can be used like this:
root@u-bigboy:/tmp# sysv-rc-conf on

Testing the Status of VSFTPD
You can always test whether the VSFTPD process is running by using the netstat -a command which lists all the TCP and UDP ports on which the server is listening for traffic. This example shows the expected output.
[root@bigboy root]# netstat -a | grep ftp
tcp        0        0        *:ftp         *:*        LISTEN
[root@bigboy root]#
If VSFTPD wasn't running, there would be no output at all.
The vsftpd.conf File
VSFTPD only reads the contents of its vsftpd.conf configuration file only when it starts, so you'll have to restart VSFTPD each time you edit the file in order for the changes to take effect. The file may be located in either the /etc or the /etc/vsftpd directories depending on your Linux distribution.
This file uses a number of default settings you need to know about.
  • VSFTPD runs as an anonymous FTP server. Unless you want any remote user to log into to your default FTP directory using a username of anonymous and a password that's the same as their email address, I would suggest turning this off. The configuration file's anonymous_enable directive can be set to no to disable this feature. You'll also need to simultaneously enable local users to be able to log in by removing the comment symbol (#) before the local_enable instruction.
  • If you enable anonymous FTP with VSFTPD, remember to define the root directory that visitors will visit. This is done with the anon_root directive.
anon_root=/data/directory
  • VSFTPD allows only anonymous FTP downloads to remote users, not uploads from them. This can be changed by modifying the anon_upload_enable directive shown later.
  • VSFTPD doesn't allow anonymous users to create directories on your FTP server. You can change this by modifying the anon_mkdir_write_enable directive.
  • VSFTPD logs FTP access to the /var/log/vsftpd.log log file. You can change this by modifying the xferlog_file directive.
  • By default VSFTPD expects files for anonymous FTP to be placed in the /var/ftp directory. You can change this by modifying the anon_root directive. There is always the risk with anonymous FTP that users will discover a way to write files to your anonymous FTP directory. You run the risk of filling up your /var partition if you use the default setting. It is best to make the anonymous FTP directory reside in its own dedicated partition.
The configuration file is fairly straight forward as you can see in the snippet below where we enable anonymous FTP and individual accounts simultaneously.
# Allow anonymous FTP?
anonymous_enable=YES
...
# The directory which vsftpd will try to change
# into after an anonymous login. (Default = /var/ftp)
anon_root=/data/directory
...
# Uncomment this to allow local users to log in.
local_enable=YES
...
# Uncomment this to enable any form of FTP write command.
# (Needed even if you want local users to be able to upload files)
write_enable=YES
...
# Uncomment to allow the anonymous FTP user to upload files. This only
# has an effect if global write enable is activated. Also, you will
# obviously need to create a directory writable by the FTP user.
#anon_upload_enable=YES
...
# Uncomment this if you want the anonymous FTP user to be able to create
# new directories.
#anon_mkdir_write_enable=YES
...
# Activate logging of uploads/downloads.
xferlog_enable=YES
...
# You may override where the log file goes if you like.
# The default is shown below.
xferlog_file=/var/log/vsftpd.log
...
To activate or deactivate a feature, remove or add the # at the beginning of the appropriate line.
Other vsftpd.conf Options
There are many other options you can add to this file:
  • Limiting the maximum number of client connections (max_clients)
  • Limiting the number of connections by source IP address (max_per_ip)
  • The maximum rate of data transfer per anonymous login. (anon_max_rate)
  • The maximum rate of data transfer per non-anonymous login. (local_max_rate)
Descriptions on this and more can be found in the vsftpd.conf man pages.

FTP Security Issues
FTP has a number of security drawbacks, but you can overcome them in some cases. You can restrict an individual Linux user's access to non-anonymous FTP, and you can change the configuration to not display the FTP server's software version information, but unfortunately, though very convenient, FTP logins and data transfers are not encrypted.
The /etc/vsftpd.ftpusers File
For added security, you may restrict FTP access to certain users by adding them to the list of users in the /etc/vsftpd.ftpusers file. The VSFTPD package creates this file with a number of entries for privileged users that normally shouldn't have FTP access. As FTP doesn't encrypt passwords, thereby increasing the risk of data or passwords being compromised, it is a good idea to let these entries remain and add new entries for additional security.
Anonymous Upload
If you want remote users to write data to your FTP server, then you should create a write-only directory within /var/ftp/pub. This will allow your users to upload but not access other files uploaded by other users. The commands you need are:
[root@bigboy tmp]# mkdir /var/ftp/pub/upload
[root@bigboy tmp]# chmod 722 /var/ftp/pub/upload
FTP Greeting Banner
Change the default greeting banner in the vsftpd.conf file to make it harder for malicious users to determine the type of system you have. The directive in this file is.
ftpd_banner= New Banner Here
Using SCP As Secure Alternative To FTP
One of the disadvantages of FTP is that it does not encrypt your username and password. This could make your user account vulnerable to an unauthorized attack from a person eavesdropping on the network connection. Secure Copy (SCP) and Secure FTP (SFTP) provide encryption and could be considered as an alternative to FTP for trusted users. SCP does not support anonymous services, however, a feature that FTP does support.
Troubleshooting FTP
You should always test your FTP installation by attempting to use an FTP client to log in to your FTP server to transfer sample files.
The most common sources of day-to-day failures are incorrect usernames and passwords.
Initial setup failures could be caused by firewalls along the path between the client and server blocking some or all types of FTP traffic. Typical symptoms of this are either connection timeouts or the ability to use the ls command to view the contents of a directory without the ability to either upload or download files. Follow the firewall rule guidelines to help overcome this problem.
Tutorial
FTP has many uses, one of which is allowing numerous unknown users to download files. You have to be careful, because you run the risk of accidentally allowing unknown persons to upload files to your server. This sort of unintended activity can quickly fill up your hard drive with illegal software, images, and music for the world to download, which in turn can clog your server's Internet access and drive up your bandwidth charges.
FTP Users with Only Read Access to a Shared Directory
In this example, anonymous FTP is not desired, but a group of trusted users need to have read only access to a directory for downloading files. Here are the steps:
1) Disable anonymous FTP. Comment out the anonymous_enable line in the vsftpd.conf file like this:
# Allow anonymous FTP?
anonymous_enable=NO
2) Enable individual logins by making sure you have the local_enable line uncommented in the vsftpd.conf file like this:
# Uncomment this to allow local users to log in.
local_enable=YES
3) Start VSFTP.
[root@bigboy tmp]# service vsftpd start
4) Create a user group and shared directory. In this case, use /home/ftp-users and a user group name of ftp-users for the remote users
[root@bigboy tmp]# groupadd ftp-users
[root@bigboy tmp]# mkdir /home/ftp-docs
5) Make the directory accessible to the ftp-users group.
[root@bigboy tmp]# chmod 750 /home/ftp-docs
[root@bigboy tmp]# chown root:ftp-users /home/ftp-docs
6) Add users, and make their default directory /home/ftp-docs
[root@bigboy tmp]# useradd -g ftp-users -d /home/ftp-docs user1
[root@bigboy tmp]# useradd -g ftp-users -d /home/ftp-docs user2
[root@bigboy tmp]# useradd -g ftp-users -d /home/ftp-docs user3
[root@bigboy tmp]# useradd -g ftp-users -d /home/ftp-docs user4
[root@bigboy tmp]# passwd user1
[root@bigboy tmp]# passwd user2
[root@bigboy tmp]# passwd user3
[root@bigboy tmp]# passwd user4
7) Copy files to be downloaded by your users into the /home/ftp-docs directory
8) Change the permissions of the files in the /home/ftp-docs directory for read only access by the group
[root@bigboy tmp]# chown root:ftp-users /home/ftp-docs/*
[root@bigboy tmp]# chmod 740 /home/ftp-docs/*
Users should now be able to log in via FTP to the server using their new usernames and passwords. If you absolutely don't want any FTP users to be able to write to any directory, then you should set the write_enable line in your vsftpd.conf file to no:
write_enable = NO
Remember, you must restart VSFTPD for the configuration file changes to take effect.
Sample Login Session To Test Functionality
Here is a simple test procedure you can use to make sure everything is working correctly:
1) Check for the presence of a test file on the ftp client server.
[root@smallfry tmp]# ll
total 1
-rw-r--r-- 1 root root 0 Jan 4 09:08 testfile
[root@smallfry tmp]#
2) Connect to bigboy via FTP
[root@smallfry tmp]# ftp 192.168.1.100
Connected to 192.168.1.100 (192.168.1.100)
220 ready, dude (vsFTPd 1.1.0: beat me, break me)
Name (192.168.1.100:root): user1
331 Please specify the password.
Password:
230 Login successful. Have fun.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp>
As expected, we can't do an upload transfer of testfile to bigboy.
ftp> put testfile
local: testfile remote: testfile
227 Entering Passive Mode (192,168,1,100,181,210)
553 Could not create file.
ftp>
But we can view and download a copy of the VSFTPD RPM located on the FTP server bigboy.
ftp> ls
227 Entering Passive Mode (192,168,1,100,35,173)
150 Here comes the directory listing.
-rwxr----- 1 0 502 76288 Jan 04 17:06 vsftpd-1.1.0-1.i386.rpm
226 Directory send OK.
ftp> get vsftpd-1.1.0-1.i386.rpm vsftpd-1.1.0-1.i386.rpm.tmp
local: vsftpd-1.1.0-1.i386.rpm.tmp remote: vsftpd-1.1.0-1.i386.rpm
227 Entering Passive Mode (192,168,1,100,44,156)
150 Opening BINARY mode data connection for vsftpd-1.1.0-1.i386.rpm (76288 bytes).
226 File send OK.
76288 bytes received in 0.499 secs (1.5e+02 Kbytes/sec)
ftp> exit
221 Goodbye.
[root@smallfry tmp]#
As expected, anonymous FTP fails.
[root@smallfry tmp]# ftp 192.168.1.100
Connected to 192.168.1.100 (192.168.1.100)
220 ready, dude (vsFTPd 1.1.0: beat me, break me)
Name (192.168.1.100:root): anonymous
331 Please specify the password.
Password:
530 Login incorrect.
Login failed.
ftp> quit
221 Goodbye.
[root@smallfry tmp]#
Now that testing is complete, you can make this a regular part of your FTP server's operation.
Conclusion
FTP is a very useful software application that can have enormous benefit to a Web site or to collaborative computing in which files need to be shared between business partners. Although insecure, it is universally accessible, because FTP clients are a part of all operating systems and Web browsers. If data encryption security is of great importance to you, then you should probably consider SCP as a possible alternative.

RAID in LINUX


Introduction
The main goals of using redundant arrays of inexpensive disks (RAID) are to improve disk data performance and provide data redundancy.
RAID can be handled either by the operating system software or it may be implemented via a purpose built RAID disk controller card without having to configure the operating system at all. This chapter will explain how to configure the software RAID schemes supported by RedHat/Fedora Linux.
For the sake of simplicity, the chapter focuses on using RAID for partitions that include neither the /boot or the root (/) filesystems.
RAID Types
Whether hardware- or software-based, RAID can be configured using a variety of standards. Take a look at the most popular.
Linear Mode RAID
In the Linear RAID, the RAID controller views the RAID set as a chain of disks. Data is written to the next device in the chain only after the previous one is filled.
The aim of Linear RAID is to accommodate large filesystems spread over multiple devices with no data redundancy. A drive failure will corrupt your data.
Linear mode RAID is not supported by Fedora Linux.
RAID 0
With RAID 0, the RAID controller tries to evenly distribute data across all disks in the RAID set.
Envision a disk as if it were a plate, and think of the data as a cake. You have four cakes- chocolate, vanilla, cherry and strawberry-and four plates. The initialization process of RAID 0, divides the cakes and distributes the slices across all the plates. The RAID 0 drivers make the operating system feel that the cakes are intact and placed on one large plate. For example, four 9GB hard disks configured in a RAID 0 set are seen by the operating system to be one 36GB disk.
Like Linear RAID, RAID 0 aims to accommodate large filesystems spread over multiple devices with no data redundancy. The advantage of RAID 0 is data access speed. A file that is spread over four disks can be read four times as fast. You should also be aware that RAID 0 is often called striping.
RAID 0 can accommodate disks of unequal sizes. When RAID runs out of striping space on the smallest device, it then continues the striping using the available space on the remaining drives. When this occurs, the data access speed is lower for this portion of data, because the total number of RAID drives available is reduced. For this reason, RAID 0 is best used with drives of equal size.
RAID 0 is supported by Fedora Linux. Figure 26.1 illustrates the data allocation process in RAID 0.
RAID 1
With RAID 1, data is cloned on a duplicate disk. This RAID method is therefore frequently called disk mirroring. Think of telling two people the same story so that if one forgets some of the details you can ask the other one to remind you.
When one of the disks in the RAID set fails, the other one continues to function. When the failed disk is replaced, the data is automatically cloned to the new disk from the surviving disk. RAID 1 also offers the possibility of using a hot standby spare disk that will be automatically cloned in the event of a disk failure on any of the primary RAID devices.
RAID 1 offers data redundancy, without the speed advantages of RAID 0. A disadvantage of software-based RAID 1 is that the server has to send data twice to be written to each of the mirror disks. This can saturate data busses and CPU use. With a hardware-based solution, the server CPU sends the data to the RAID disk controller once, and the disk controller then duplicates the data to the mirror disks. This makes RAID-capable disk controllers the preferred solution when implementing RAID 1.
A limitation of RAID 1 is that the total RAID size in gigabytes is equal to that of the smallest disk in the RAID set. Unlike RAID 0, the extra space on the larger device isn't used.
RAID 1 is supported by Fedora Linux. Figure 26.1 illustrates the data allocation process in RAID 1.
Figure 26-1 RAID 0 And RAID 1 Operation
RAID 4
RAID 4 operates likes RAID 0 but inserts a special error-correcting or parity chunk on an additional disk dedicated to this purpose.
RAID 4 requires at least three disks in the RAID set and can survive the loss of a single drive only. When this occurs, the data in it can be recreated on the fly with the aid of the information on the RAID set's parity disk. When the failed disk is replaced, it is repopulated with the lost data with the help of the parity disk's information.
RAID 4 combines the high speed provided of RAID 0 with the redundancy of RAID 1. Its major disadvantage is that the data is striped, but the parity information is not. In other words, any data written to any section of the data portion of the RAID set must be followed by an update of the parity disk. The parity disk can therefore act as a bottleneck. For this reason, RAID 4 isn't used very frequently.
RAID 4 is not supported by Fedora Linux.
RAID 5
RAID 5 improves on RAID 4 by striping the parity data between all the disks in the RAID set. This avoids the parity disk bottleneck, while maintaining many of the speed features of RAID 0 and the redundancy of RAID 1. Like RAID 4, RAID 5 can survive the loss of a single disk only.
RAID 5 is supported by Fedora Linux. Figure 26.2 illustrates the data allocation process in RAID 5.
Linux RAID 5 requires a minimum of three disks or partitions.
Figure 26-2 RAID 5 Operation
Before You Start
Specially built hardware-based RAID disk controllers are available for both IDE and SCSI drives. They usually have their own BIOS, so you can configure them right after your system's the power on self test (POST). Hardware-based RAID is transparent to your operating system; the hardware does all the work.
If hardware RAID isn't available, then you should be aware of these basic guidelines to follow when setting up software RAID.
IDE Drives
To save costs, many small business systems will probably use IDE disks, but they do have some limitations.
  • The total length of an IDE cable can be only a few feet long, which generally limits IDE drives to small home systems.
  • IDE drives do not hot swap. You cannot replace them while your system is running.
  • Only two devices can be attached per controller.
  • The performance of the IDE bus can be degraded by the presence of a second device on the cable.
  • The failure of one drive on an IDE bus often causes the malfunctioning of the second device. This can be fatal if you have two IDE drives of the same RAID set attached to the same cable.
For these reasons, I recommend you use only one IDE drive per controller when using RAID, especially in a corporate environment. In a home or SOHO setting, IDE-based software RAID may be adequate.
Serial ATA Drives
Serial ATA type drives are rapidly replacing IDE, or Ultra ATA, drives as the preferred entry level disk storage option because of a number of advantages:
  • The drive data cable can be as long as 1 meter in length versus IDE's 18 inches.
  • Serial ATA has better error checking than IDE.
  • There is only one drive per cable which makes hot swapping, or the capability to replace components while the system is still running, possible without the fear of affecting other devices on the data cable.
  • There are no jumpers to set on Serial ATA drives to make it a master or slave which makes them simpler to configure.
  • IDE drives have a 133Mbytes/s data rate whereas the Serial ATA specification starts at 150 Mbytes/sec with a goal of reaching 600 Mbytes/s over the expected ten year life of the specification.
If you can't afford more expensive and faster SCSI drives, Serial ATA would be the preferred device for software and hardware RAID
SCSI Drives
SCSI hard disks have a number of features that make them more attractive for RAID use than either IDE or Serial ATA drives.
  • SCSI controllers are more tolerant of disk failures. The failure of a single drive is less likely to disrupt the remaining drives on the bus.
  • SCSI cables can be up to 25 meters long, making them suitable for data center applications.
  • Much more than two devices may be connected to a SCSI cable bus. It can accommodate 7 (single-ended SCSI) or 15 (all other SCSI types) devices.
  • Some models of SCSI devices support "hot swapping" which allows you to replace them while the system is running.
  • SCSI currently supports data rates of up to 640 Mbytes/s making them highly desirable for installations where rapid data access is imperative.
SCSI drives tend to be more expensive than IDE drives, however, which may make them less attractive for home use.
Should I Use Software RAID Partitions Or Entire Disks?
It is generally a not a good idea to share RAID-configured partitions with non-RAID partitions. The reason for this is obvious: A disk failure could still incapacitate a system.
If you decide to use RAID, all the partitions on each RAID disk should be part of a RAID set. Many people simplify this problem by filling each disk of a RAID set with only one partition.
Backup Your System First
Software RAID creates the equivalent of a single RAID virtual disk drive made up of all the underlying regular partitions used to create it. You have to format this new RAID device before your Linux system can store files on it. Formatting, however, causes all the old data on the underlying RAID partitions to be lost. It is best to backup the data on these and any other partitions on the disk drive on which you want implement RAID. A mistake could unintentionally corrupt valid data.
Configure RAID In Single User Mode
As you will be modifying the disk structure of your system, you should also consider configuring RAID while your system is running in single-user mode from the VGA console. This makes sure that most applications and networking are shutdown and that no other users can access the system, reducing the risk of data corruption during the exercise.
[root@bigboy tmp]# init 1
Once finished, issue the exit command, and your system will boot in the default runlevel provided in the /etc/inittab file.
Configuring Software RAID
Configuring RAID using Fedora Linux requires a number of steps that need to be followed carefully. In the tutorial example, you'll be configuring RAID 5 using a system with three pre-partitioned hard disks. The partitions to be used are:
/dev/hde1
/dev/hdf2
/dev/hdg1
Be sure to adapt the various stages outlined below to your particular environment.
RAID Partitioning
You first need to identify two or more partitions, each on a separate disk. If you are doing RAID 0 or RAID 5, the partitions should be of approximately the same size, as in this scenario. RAID limits the extent of data access on each partition to an area no larger than that of the smallest partition in the RAID set.
Determining Available Partitions
First use the fdisk -l command to view all the mounted and unmounted filesystems available on your system. You may then also want to use the df -k command, which shows only mounted filesystems but has the big advantage of giving you the mount points too.
These two commands should help you to easily identify the partitions you want to use. Here is some sample output of these commands.
[root@bigboy tmp]# fdisk -l

Disk /dev/hda: 12.0 GB, 12072517632 bytes
255 heads, 63 sectors/track, 1467 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hda1   *         1        13    104391   83  Linux
/dev/hda2            14       144   1052257+  83  Linux
/dev/hda3           145       209    522112+  82  Linux swap
/dev/hda4           210      1467  10104885    5  Extended
/dev/hda5           210       655   3582463+  83  Linux
...
...
/dev/hda15         1455      1467    104391   83  Linux
[root@bigboy tmp]#

[root@bigboy tmp]# df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda2              1035692    163916    819164  17% /
/dev/hda1               101086      8357     87510   9% /boot
/dev/hda15              101086      4127     91740   5% /data1
...
...
...
/dev/hda7              5336664    464228   4601344  10% /var
[root@bigboy tmp]#
Unmount the Partitions
You don't want anyone else accessing these partitions while you are creating the RAID set, so you need to make sure they are unmounted.
[root@bigboy tmp]# umount /dev/hde1
[root@bigboy tmp]# umount /dev/hdf2
[root@bigboy tmp]# umount /dev/hdg1

Prepare The Partitions With FDISK
You have to change each partition in the RAID set to be of type FD (Linux raid autodetect), and you can do this with fdisk. Here is an example using /dev/hde1.
[root@bigboy tmp]# fdisk /dev/hde
The number of cylinders for this disk is set to 8355.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
  (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help):
Use FDISK Help
Now use the fdisk m command to get some help:
Command (m for help): m
  ...
  ...
  p   print the partition table
  q   quit without saving changes
  s   create a new empty Sun disklabel
  t   change a partition's system id
  ...
  ...
Command (m for help):
Set The ID Type
Partition /dev/hde1 is the first partition on disk /dev/hde. Modify its type using the t command, and specify the partition number and type code. You also should use the L command to get a full listing of ID types in case you forget. In this case, RAID uses type fd, it may be different for your version of Linux.
Command (m for help): t
Partition number (1-5): 1
Hex code (type L to list codes): L


...
...
...
16  Hidden FAT16    61   SpeedStor       f2  DOS secondary
17  Hidden HPFS/NTF 63  GNU HURD or Sys fd  Linux raid auto
18  AST SmartSleep  64  Novell Netware  fe  LANstep
1b  Hidden Win95 FA 65  Novell Netware  ff  BBT
Hex code (type L to list codes): fd
Changed system type of partition 1 to fd (Linux raid autodetect)

Command (m for help):
Make Sure The Change Occurred
Use the p command to get the new proposed partition table:
Command (m for help): p

Disk /dev/hde: 4311 MB, 4311982080 bytes
16 heads, 63 sectors/track, 8355 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes

   Device Boot    Start       End    Blocks   Id  System
/dev/hde1             1      4088   2060320+  fd  Linux raid autodetect
/dev/hde2          4089      5713    819000   83  Linux
/dev/hde4          6608      8355    880992    5  Extended
/dev/hde5          6608      7500    450040+  83  Linux
/dev/hde6          7501      8355    430888+  83  Linux

Command (m for help):

Save The Changes
Use the w command to permanently save the changes to disk /dev/hde:
Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

[root@bigboy tmp]#
The error above will occur if any of the other partitions on the disk is mounted.
Repeat For The Other Partitions
For the sake of brevity, I won't show the process for the other partitions. It's enough to know that the steps for changing the IDs for /dev/hdf2 and /dev/hdg1 are very similar.

Preparing the RAID Set
Now that the partitions have been prepared, we have to merge them into a new RAID partition that we'll then have to format and mount. Here's how it's done.
Create the RAID Set
You use the mdadm command with the --create option to create the RAID set. In this example we use the --level option to specify RAID 5, and the --raid-devices option to define the number of partitions to use.
[root@bigboy tmp]# mdadm --create --verbose /dev/md0 --level=5 \
   --raid-devices=3 /dev/hde1 /dev/hdf2 /dev/hdg1

mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 64K
mdadm: /dev/hde1 appears to contain an ext2fs file system
    size=48160K  mtime=Sat Jan 27 23:11:39 2007
mdadm: /dev/hdf2 appears to contain an ext2fs file system
    size=48160K  mtime=Sat Jan 27 23:11:39 2007
mdadm: /dev/hdg1 appears to contain an ext2fs file system
    size=48160K  mtime=Sat Jan 27 23:11:39 2007
mdadm: size set to 48064K
Continue creating array? y
mdadm: array /dev/md0 started.
[root@bigboy tmp]#
Confirm RAID Is Correctly Inititalized
The /proc/mdstat file provides the current status of all RAID devices. Confirm that the initialization is finished by inspecting the file and making sure that there are no initialization related messages. If there are, then wait until there are none.
[root@bigboy tmp]# cat /proc/mdstat
Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 hdg1[2] hde1[1] hdf2[0]
      4120448 blocks level 5, 32k chunk, algorithm 3 [3/3] [UUU]

unused devices:
[root@bigboy tmp]#
Notice that the new RAID device is called /dev/md0. This information will be required for the next step.
Format The New RAID Set
Your new RAID partition now has to be formatted. The mkfs.ext3 command is used to do this.
[root@bigboy tmp]# mkfs.ext3 /dev/md0
mke2fs 1.39 (29-May-2006)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
36144 inodes, 144192 blocks
7209 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
18 block groups
8192 blocks per group, 8192 fragments per group
2008 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729

Writing inode tables: done                           
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@bigboy tmp]#
Create the mdadm.conf Configuration File
Your system doesn't automatically remember all the component partitions of your RAID set. This information has to be kept in the mdadm.conf file. The formatting can be tricky, but fortunately the output of the mdadm --detail --scan --verbose command provides you with it. Here we see the output sent to the screen.
[root@bigboy tmp]# mdadm --detail --scan --verbose
ARRAY /dev/md0 level=raid5 num-devices=4
UUID=77b695c4:32e5dd46:63dd7d16:17696e09
   devices=/dev/hde1,/dev/hdf2,/dev/hdg1
[root@bigboy tmp]#

Here we export the screen output to create the configuration file.
[root@bigboy tmp]# mdadm --detail --scan --verbose > /etc/mdadm.conf

Create A Mount Point For The RAID Set
The next step is to create a mount point for /dev/md0. In this case we'll create one called /mnt/raid
[root@bigboy mnt]# mkdir /mnt/raid
Edit The /etc/fstab File
The /etc/fstab file lists all the partitions that need to mount when the system boots. Add an Entry for the RAID set, the /dev/md0 device.
/dev/md0      /mnt/raid     ext3    defaults    1 2
Do not use labels in the /etc/fstab file for RAID devices; just use the real device name, such as /dev/md0. In older Linux versions, the /etc/rc.d/rc.sysinit script would check the /etc/fstab file for device entries that matched RAID set names listed in the now unused /etc/raidtab configuration file. The script would not automatically start the RAID set driver for the RAID set if it didn't find a match. Device mounting would then occur later on in the boot process. Mounting a RAID device that doesn't have a loaded driver can corrupt your data and produce this error.
Starting up RAID devices: md0(skipped)
Checking filesystems
/raiddata: Superblock has a bad ext3 journal(inode8)
CLEARED.
***journal has been deleted - file system is now ext 2 only***

/raiddata: The filesystem size (according to the superblock) is 2688072 blocks.
The physical size of the device is 8960245 blocks.
Either the superblock or the partition table is likely to be corrupt!
/boot: clean, 41/26104 files, 12755/104391 blocks

/raiddata: UNEXPECTED INCONSISTENCY; Run fsck manually (ie without -a or -p options).
If you are not familiar with the /etc/fstab file use the man fstab command to get a comprehensive explanation of each data column it contains.
The /dev/hde1, /dev/hdf2, and /dev/hdg1 partitions were replaced by the combined /dev/md0 partition. You therefore don't want the old partitions to be mounted again. Make sure that all references to them in this file are commented with a # at the beginning of the line or deleted entirely.
#/dev/hde1       /data1        ext3    defaults        1 2
#/dev/hdf2       /data2        ext3    defaults        1 2
#/dev/hdg1       /data3        ext3    defaults        1 2
Mount The New RAID Set
Use the mount command to mount the RAID set. You have your choice of methods:
  • The mount command's -a flag causes Linux to mount all the devices in the /etc/fstab file that have automounting enabled (default) and that are also not already mounted.
[root@bigboy tmp]# mount -a
  • You can also mount the device manually.
[root@bigboy tmp]# mount /dev/md0 /mnt/raid

Check The Status Of The New RAID
The /proc/mdstat file provides the current status of all the devices.
[root@bigboy tmp]# raidstart /dev/md0
[root@bigboy tmp]# cat /proc/mdstat
Personalities : [raid5]
read_ahead 1024 sectors
md0 : active raid5 hdg1[2] hde1[1] hdf2[0]
      4120448 blocks level 5, 32k chunk, algorithm 3 [3/3] [UUU]

unused devices:
[root@bigboy tmp]#
Conclusion
Linux software RAID provides redundancy across partitions and hard disks, but it tends to be slower and less reliable than RAID provided by a hardware-based RAID disk controller.
Hardware RAID configuration is usually done via the system BIOS when the server boots up, and once configured, it is absolutely transparent to Linux. Unlike software RAID, hardware RAID requires entire disks to be dedicated to the purpose and when combined with the fact that it usually requires faster SCSI hard disks and an additional controller card, it tends to be expensive.
Remember to take these factors into consideration when determining the right solution for your needs and research the topic thoroughly before proceeding. Weighing cost versus reliability is always a difficult choice in systems administration.

LVM in Linux ( RHEL CENTOS UBUNTU DEBIAN )

Make four partitions of size 100M, 50M, 75M and 150M with partition ID 8e. 8e is the ID of LVM type partition.

Suppose the newly created partitions are:  /dev/hda10, /dev/hda11, /dev/hda12, /dev/hda13.

Now create Physical Volume:
#pvcreate  /dev/hda10
#pvcreate  /dev/hda11
#pvcreate  /dev/hda12
#pvcreate /dev/hda13

To check status of pv:
#pvscan
#pvdisplay

Now create volume group:
#vgcreate  vg01  /dev/hda10  /dev/hda11
#vgcreate vg02  /dev/hda12  /dev/hda13

To check status of vg:
#vgscan
#vgdisplay

Now create logical volume:
#lvcreate   --name   lv1  --size  110M vg01
#lvcreate   --name   lv2  --size  170M vg02

To check status of lv:
#lvscan
#lvdisplay

Now need to format these newly created logical volume with ext3 filesystem
#mkfs.ext3   /dev/vg01/lv1
#mkfs.ext3  /dev/vg02/lv2

Now mount the logical volumes to folder:
#mkdir   /raidtest1
#mkdir  /raidtest2
# mount   /dev/vg01/lv1  /raidtest1
#mount   /dev/vg02/lv2  /raidtest2
#df –h

Now we will extend or reduce our volume size: (Unmount first)
For extending:
#umount  /raidtest1
#lvextend  -L +15M /dev/vg01/lv1
#e2fsck  -f  /dev/vg01/lv1
#resize2fs /dev/vg01/lv1
For reducing:
#umount  /raidtest2
#dumpefs  /dev/vg02/lv2 | grep –I ‘Block size’
#e2fsck  -f  /dev/vg02/lv2
#resize2fs /dev/vg02/lv2  143360
#lvreduce  -L  -140M  /dev/vg02/lv2

Other commands(need to delete associated lv first to do something on vg, need to delete associated vg fisrt to do something on pv):
#lvrename   vg01   lv1    lv3
#lvrename   vg02   lv2    lv4
#lvremove   /dev/vg01/lv3
#lvremove  /dev/vg02/lv4
#vgrename  vg01   vg03
#vgrename  vg02   vg04
#vgremove  vg03
#vgremove  vg04
#pvremove  /dev/hda10
#pvremove  /dev/hda11
#pvremove  /dev/hda12
#pvremove  /dev/hda14

Thursday, January 21, 2010

RHCE EXAM PART 3 (IP FORWARDING & LINUX ROUTER)

You will be asked to make your machine able to forward IP and act like a router.
To do this:

# cat /proc/sys/net/ipv4/ip_forward
If the returned value is 0 that means IP forwarding is disabled.

Then
# echo 1 > /proc/sys/net/ipv4/ip_forward
This will make IP forwarding active temporarily

To make it permanent
# vim /etc/sysctl.conf
net.ipv4.ip_forward=1

And done.