vSphere 6.0 – What’s New in VMware Fault Tolerance (FT)

VMware Fault Tolerance (FT) is being one of my favorite feature but because of its vCPU limitation,  it was not helping to protect the Mission Critical applications. With vSphere 6.0, VMware broken the limitation lock of Fault Tolerance. FT VM now supports  up to 4 vCPUs and 64 GB of RAM (which was 1 vCPU and 64 GB RAM in vSphere 5.5). With this vSMP support, now FT can be used to protect your mission Critical applications. Along with the vSMP FT Support, there are lot more features has been added in FT with vSphere 6.0.

Benefits of Fault Tolerance:

• Continuous Availability with zero downtime and zero data loss.
• NO TCP connections loss during failover
• Fault Tolerance is completely transparent to Guest OS.
• FT doesn't depend on Guest OS and application
• Instantaneous Failover from primary VM to Secondary VM in case of ESXi host failure.

What's New in vSphere 6.0 Fault Tolerance:

• FT Support up to 4 vCPUs and 64 GB RAM
• Fast Check-pointing, a new Scalable technology is introduced to keep primary and secondary in Sync by replacing "Record-Replay"
• vSphere 6.0, Supports vMotion of both Primary and Secondary Virtual Machine
• With vSphere 6.0, you will be able to  backup your virtual machines. FT Supports for vStorage API for Data Protection (VADP) and it also supports all leading VADP solutions in Market like symantec, EMC, HP, etc.
• With vSphere 6.0, FT supports  all virtual Disk Type like EZT, Thick or Thin Provisioned disks. It supports only Eager Zeroed Thick with vSphere  5.5 and earlier versions.
• Snapshot of FT configured Virtual Machines are supported with vSphere 6.0
• New version of FT keeps the Separate copies of VM files like .VMX, .VMDK files to protect primary VM from both Host and Storage failures. You are allowed to keep both primary and secondary VM files on different datastore.

Difference between vSphere 5.5 and vSphere 6.0 FT:

 

vSphere 6.0 – New Configuration Maximums

VMware vSphere6.0 introduced with lot of new enhancements and feature addition as compared with the previous versions of vSphere.

New Configuration Maximums of vSphere 6.0:

• vSphere 6.0 clusters now supports 64 Nodes and 8000 VM's  ( which was 32 Nodes and 4000 VM's in vSphere 5.5)
• vCenter Server Appliance (vCSA 6.0) supports up to 1000 Hosts and 10,000 virtual machines with embedded vPostgres database.
• ESXi 6.0 host now supports up to 480 physical CPUs and 12TB RAM ( Which was 320 CPUs and 4 TB in vSphere 5.5)
• ESXi 6.0 Host supports 1000 VMs and 32 Serial ports ( which was 512 VM per Host in vSphere 5.5)
• vSphere 6.0 VM's will support up to 128 vCPUs and 4TB vRAM (Which was 64 vCPUs and 1TB of memory in vSphere 5.5)
• vSphere 6.0 continuous to support 64 TB Datastores as same as in vSphere 5.5
• Increased support for virtual graphics including Nvidia vGPU
• Support for New Operating Systems like FreeBSD 10.0 and Asianux 4 SP3
• Fault Tolerance (FT) in vSphere 6.0 now supports upto 4 vCPUs ( which was only 1 vCPU in vSphere 5.5)

I would like to provide a Quick Comparison table between the configuration maximums of vSphere 5.5 and vSphere 6.0

Reset root password in SLES

To reset the root password we need to enter into single user mode. Entering into single user mode in SuSe Enterprise Linux is different compare to Red Hat Linux.

Following is the procedure to reset root password and entering into single user mode into SLES.

For SLES 11.x:

Go to the boot loader screen, type below command in the boot instruction option:

init=/bin/bash

Press enter to boot. This will boot the "SUSE Linux Enterprise Server 11.x" in the single user mode. In this mode there is no file system which is mounted, so we will use mount command to remount the file system in read & write mode.

(none):/#mount -o remount,rw /

(none):/#mount -r remount,rw /usr  (if the /usr file system is on the separate partition)

Now try to run commands like resetting the root password.

(none):/#passwd root

To reboot the machine type "exit" two times.

The kdump Crash Recovery Service

When the kdump crash dumping mechanism  is enabled, the system is booted from the context of another kernel. This second kernel reserves a small amount of memory and its only purpose is to capture the core dump image in case the system crashes.

Being able to analyze the core dump significantly helps to determine the exact cause of the system failure, and it is therefore strongly recommended to have this feature enabled.

Installing the kdump Service:

In order to use the kdump service on your system, make sure we have the kexec-tools package installed. To do so, type the following at a shell prompt as root.

Configuring the memory usage:

Memory reserved for the kdump kernel is always reserved during system boot, which means that the amount of memory is specified in the system's boot loader configuration. To configure the amount of memory to be reserved for thee kdump kernel, edit the /boot/grub/grub.conf file and add crashkernel=<size>M or crashkernel=auto to the list of kernel options as shown in following /boot/grub/grub.conf file. The crashkernel=auto option only reserves the memory if the physical memory of the system is equal to or greater than:

• 2 GB on 32-bit and 64-bit x86 architectures;
• 2 GB on PowerPC if the page size is 4KB, or 8GB otherwise;
• 4 GB on IBM S/390

When the kdump crash recovery is enabled, the minimum memory requirements increase by the amount of memory reserved for it. This value is determined by the user, and defaults to 128MB plus 64MB for each TB of physical memory (that is, a total of 192 MB for a system with 1 TB of physical memory). The memory can be attempted up to the maximum of 896 MB if required. This is recommended especially in large environments, for example in systems with a large number of Logical Unit Numbers (LUNs).

Configuring the Target Type:

When a kernel crash is captured, the core dump can be either stored as a file in a local file system, written directly to a device, or sent over a network using the NFS (Network File System) or SSH (Secure Shell) protocol. Only one of these options can be set at the moment, and the default option is to store the vmcore file in the /var/crash directory of the local file system. To change this, as root , open the /etc/kdump.conf configuration file in a text editor and edit the options.

To change the local directory in which the core dump is to be saved, remove the hash sign (#) from the beginning of the #path /var/crash line, and replace the value with a desired directory path.

Optionally, if you want to write the file  to a different partition, follow the same procedure  with the #ext4 /dev/sda3 line  as well, and change both the file system type and the device ( a device name, a filesystem label,  and UUID are all supported) accordingly.

To write the dump directly to a device, remove the hash sign (#) from the beginning of the #raw /dev/sda5 line, and replace the value with a desired device name.

To write the dump to a remote machine using the NFS protocol, remove the has sign (#) from the beginning of the #net my.server.com:/export/tmp line, and replace the value with a valid host name and directory path.

To store the dump to a remote machine using SSH protocol, remove the hash sign (#) from the beginning of the #net user@my.server.com line, and replace the value  with a valid  user name and host name.

The vmcore.flat file must be converted, when transferring a core file to a remote target over SSH, the core file needs to be serialized for the tarnsfer. This creates  a vmcore.flat file in the /var/crash directory on the target system, which is unreadable  by the crash utility. To convert vmcore.flat to a dump file that is readable by crash, run the following command as root on the target system:

Configuring the core collector:

To reduce the size of the vmcore dump file, kdump allows you to specify an external application (that is, a core collector) to compress the data, and optionally leave out all irrelevant information. Currently, the only fully supported  core collector is makedumpfile.

To enable the core collector, as root, open the /etc/kdump.conf configuration file in a text editor, remove the hash sign (#) from the beginning of the #core_collector makedumpfile -c --message-level 1 -d 31 line, and edit the command-line options as described below.

To enable the dump file compression, add the -c parameter.

To remove certain pages from dump, add the -d value parameter, where value is a sum of values of pages you want to omit as described in following table. To remove both zero and free pages.

Supported filtering levels:

Changing the Default Action:

By default, when kdump fails, to create a core dump, the root file system is mounted and /sbin/init is run. To change this behavior, as root, open the /etc/kdump.conf configuration file in a text editor, remove the hash  sign (#) from the beginning of the #default shell line, and replace the value with a desired action as described in following table.

Supported Actions:

Enabling the services:

To start the kdump daemon at boot time,  type the following at a shell prompt as root.

To start the service in the current session, using the following command as root:

Testing the configuration:

To test the configuration, reboot the system with kdump enabled, and make sure that the service is running

Then type the following commands at a shell prompt.

This will force the Linux kernel to crash, and the address-YYYY-MM-DD-HH:MM:SS/vmcore file will be copied to the location which we have selected in the configuration.

Reset root password in RHEL7

Resetting root password in RHEL7 is slightly different than the older versions.

Reboot and edit grub2:

Reboot the system and press any key to stop the auto-boot from the default kernel. Press "e" after selecting the kernel line.

Append rd.break to kernel:

When we are in edit mode, go to the line starting with linux16 and append rd.break to the end of the line.

Reboot the system:

Press CTLR+x after appending the rd.break to the kernel. This will reboot the system into emergency mode.

Remount sysroot:

First we will remount the sysroot file system in read write mode and then use chroot to got into a chroot jail.

Reset root password:

Finally type password command in the command line and set the new password for root user. You might get some warnings like "password fails dictionary check" if your password is week. You may safely ignore the warning and set the password you want.

SELinux relabeling:

Make sure that all unlabeled files (including shadow file) gets relabeled during booting. Touching the hidden file autoleabel instructs SELinux to relabel the files changed outside of its regular context, like the file /etc/shadow.

Sync:

To flush all cache to disk, type the command.

type twice the exit command to leave the chroot environment and logout. The system will apply some SELinux contexts and reboot. You can now login using your newly set root password. 

Run Levels in RHEL7

Systemd has replaced sysVinit as the default service manager in RHEL7. Some of the sysVinit commands have been symlinked to their RHEL7 counterparts. However this will eventually be deprecated in favor of the standard commands in the feature.

SysVinit V/s systemd runlevels:

Here is a comparison between SysVinit runlevels V/s systemd targets.

SysVinit/Runlevel	Systemd Target	Function
0	Runlevel0.target, poweroff.target	System Halt/Shutdown
1,s,Single	Runlevel1.target, rescue.target	Single-user mode
2,4	Runlevel2.target, runlevel4.target, multi-user.target	User Defined/Site-Specific runlevels. By default, identical to 3.
3	Runlevel3.target, multi-user.target	Multi-User, Non-graphical mode, text console only
5	Runlevel5.target, graphical.target	Multi-user, graphicalmode
6	Runlevel6.target, reboot.target	Reboot
Emergency	Emergency.target	Emergency mode

Changing runlevels with systemd:

The runlevel target can be changed by using the systemctl isolate command:

# systemctl isolate multi-user.target

To view what targets are available you can issue the list-units option with the type target.

# systemctl list-units --type=target

Run level 3 is  emulated by multi-user.target. This is done by symbolic link and can be used interchangeably.

# systemctl isolate multi-user.target # systemctl isolate runlevel3.target # ls -l /usr/lib/systemd/system/runlevel3.target lrwxrwxrwx 1 root root 17 Oct 18 11:41 /usr/lib/systemd/system/runlevel3.target -> multi-user.target

Run level 5 is emulated by graphical.target. This is also done by symbolic link and can be used interchangeably.

# systemctl isolate graphical.target # systemctl isolate runlevel5.target # ls -l /usr/lib/systemd/system/runlevel5.target lrwxrwxrwx 1 root root 16 Oct 18 11:41 /usr/lib/systemd/system/runlevel5.target -> graphical.target

Changing the default runlevel:

The default runlevel can be changed by using the set-default option.

To get the currently set default, you can use the get-default option.

The default runlevel in systemd can also be set using the below method ( not recommended though).

The default target can also be set in the kernel line during boot by adding the following option: 

Run Levels in RHEL6

The SysV init runlevel system provides a standard process for controlling which programs init launches or halts when initializing a runlevel.

The configuration files for SysV init are located in the /etc/rc.d/ directory. Within this directory, are the rc, rc.local, rc.sysinit and optionally, the rc.serial scripts as well as the following directories.

The init.d/ directory contains the script used by the /sbin/init command when controlling services. Each of the numbered directories represent the six default run levels configured by default under RHEL.

Runlevels are a state, or mode, defined by the services listed in the SysV /etc/rc.d/rc<x>.d/ directory, where <x> is the number of the runlevel.

The following runlevels are defined by default for RHEL.

Runlevel	Description
0	Halt
1	Single-user text mode
2	Not Used ( user-definable)
3	Full multi-user text mode
4	Not used (user-definable)
5	Full multi-user graphical mode
6	Reboot

Identify current run level:

Use the 'runlevel" command to get the previous and current runlevel of the system.

Changing the current level:

Use the following command to change the current runlevel.

#init <new_runlevel>

Changing the default runlevel:

Use any text editor to open the file /etc/inittab. Edit the last line of this file and change the number after id: to the desired runlevel.

Soft Links & Hard Links

What are links in Linux?


A Link in Linux is a pointer to a file. Like Pointers in any programming languages, links in Linux are pointers pointing to a file or a directory. Creating Links is a Kind of shortcuts to access a file. The two different type of links in UNIX are.


• Soft Links (or) Symbolic Links
• Hard Links


Difference Between Soft links and hard links:



Soft Links	Hard Links
Soft Links Can be created across file systems	Had Links can be created only within the File System
Soft Link has a different inode number than the original file	Had links have the same inode number as the original File
On deleting the original file, soft links cannot be accessed	On deleting the original file, had linked file can still be accessed
Soft links need extra memory to store the original file name as its data	Had links do not need any extra data memory to save since  it uses links
Source file need not exist for soft link file creation	Source file should exist
Can be created on a file or on a directory	Can be created only on files, not on directories
Access to the file is slower due to the overhead to access  file	Access to the file is faster compared to soft link



How do we create Soft Links and Hard Links? And How do we access them?


We have a file named "file1" with the following contents:


$ cat file1
Welcome


To create a hard link of file1:


$ ln file1 file2


To create  a soft link of file1:


$ ln -s file1 file3


Once the links are created, the linked files contain the same content as of the original file. See below


$ cat file2
welcome
$ cat file3
Welcome


Note: Soft links can be created on non-existent files as well.


•  "file2" and "file3" being the linked files, can I say which is soft link & which is hard link?


Yes. When we do the listing of the files with "ls -li" option:


$ ls -li
total 20
9962464 -rw-r--r-- 2 guru users 8 Mar  9  file1
9962464 -rw-r--r-- 2 guru users 8 Mar  9  file2
9962471 lrwxrwxrwx 1 guru users 5 Mar  9  file3 -> file1


If you notice "file3" shows "->" towards "file1". This indicates file3 is a soft-link to file1. In case of file2, if we notice the inode numbers of file1 and file2, they are the same. Same indoe number indicates file1 and file2 are hard links. Also note, the link count of these files is 2.


• File1 and file2 are hard links? Can we say which is the original file and which is the hard linked file?:


No. We can not say which is the original file and which one is hard-linked file. Once a hard-linked is created, it is like 2 files pointing to the same location. In fact, once a hard link is created on a file, using the term "original file" is actually incorrect.




• How is it possible that while deleting the original file, still the hard linked file is accessible and the soft linked file is not?


Let us look into the inode structure of the hard and soft links to understand in details. 













Fig1: Hard link representation ( Files pointing to indoe, in turn pointing to data location)


File1 and File2 are hard links, As we know, they both have the same inode number ( 9962464) and hence they both point to the same indoe structure. One of the parameters in the inode tells the location of the file contents, and hence the contents are accessed by both the files.


Now, say we try to delete the file "File2". When the file is deleted, only the link connecting of File2 to the indoe structure  gets disconnected. However, the inode and the file contents and the link of FIle1  are still in tact, and hence the file is still accessible.


However, the same is not the case with the soft link. In case of Soft Link.

















Fg2: Soft Link Representation (Files pointing to inodes, in turn pointing to data location)


File1 and File3 both have different inode numbers. These different inode numbers point to different data locations. File1 location points to the contents of the file. However, if you see the content location of File3, it contains "File1" which is nothing but the original file from which the soft link is formed. Now, when "File3" is accessed, through its inode metadata, it gets the name "File1" and searches for "File1" and reaches to the content of "File1" Hence once the "File1" is deleted, "File3" becomes dangling meaning it points to something which does not exist. And so, the soft link becomes inaccessible.


For these same reasons, we can create a soft link on an non-existent file, however the same is not possible for hard links.


• Soft links can span across File systems whereas hard links can be created only within a file system. Why so?


Hard links refers to its file using the inode number. Inode numbers are specific within a file system. This means a file "f1" in file system "X" can have the same inode numbers as of a file  "f2" in file system "Y". This is very much possible. For this reason, had links cannot be shared across file systems.


However, the same is not true for soft links, Soft links refer to the files using the file name as seen in the earlier case. Hence a soft links can be accessed across the file systems.


• Does soft links and hard links work only at the file level? Does it work at the directory level as well?


Hard links works only at the file level. However, soft links works at the directory level as well. This particular use of soft links, to create links at the directory level, serves us a shortcut for many frequently used directories.