Chapter 18: Essential Troubleshooting Skills

Sommaire

Booting Steps

To fix boot issues, it is essential to have a good understanding of the boot procedure. If an issue occurs during boot, you need to be able to judge in which phase of the boot procedure the issue occurs so that you can select the appropriate tool to fix the issue.

1. Performing POST: The machine is powered on. From the system firmware, which can be the modern Universal Extended Firmware Interface (UEFI) or the classical Basic Input Output System (BIOS), the Power-On Self-Test (POST) is executed, and the hardware that is required to start the system is initialized.

2. Selecting the bootable device: Either from the UEFI boot firmware or from the BIOS, a bootable device is located.

3. Loading the boot loader: From the bootable device, a boot loader is located. On RHEL, this is usually GRUB 2.

4. Loading the kernel: The boot loader may present a boot menu to the user or can be configured to automatically start a default operating system. To load Linux, the kernel is loaded together with the initramfs. The initramfs contains kernel modules for all hardware that is required to boot, as well as the initial scripts required to proceed to the next stage of booting. On RHEL 8, the initramfs contains a complete operational system (which may be used for troubleshooting purposes).

5. Starting /sbin/init: Once the kernel is loaded into memory, the first of all processes is loaded, but still from the initramfs. This is the /sbin/init process, which on RHEL is linked to Systemd. The udev daemon is loaded as well to take care of further hardware initialization. All this is still happening from the initramfs image.

6. Processing initrd.target: The Systemd process executes all units from the initrd.target, which prepares a minimal operating environment, where the root file system on disk is mounted on the /sysroot directory. At this point, enough is loaded to pass to the system installation that was written to the hard drive.

7. Switching to the root file system: The system switches to the root file system that is on disk and at this point can load the Systemd process from disk as well.

8. Running the default target: Systemd looks for the default target to execute and runs all of its units. In this process, a login screen is presented, and the user can authenticate. Note that the login prompt can be prompted before all Systemd unit files have been loaded successfully. So, seeing a login prompt does not necessarily mean that your server is fully operational yet.

System Boot

Passing Kernel Boot Arguments

Accessing the Boot Prompt

When your server boots, you briefly see the GRUB 2 menu. Look fast because it will only last for a few seconds. From this boot menu you can type e to enter a mode where you can edit commands, or c to enter a full GRUB command prompt . To pass boot options to a starting kernel, use e .

After passing an e to the GRUB boot menu, you’ll see the interface that is in Figure 19.2 . From this interface, scroll down to locate the section that begins with linux16 /vmlinuz followed by a lot of arguments. This is the line that tells GRUB how to start a kernel, and by default it looks like this:

linux16 /vmlinuz-0-rescue-5dea58df1a3b4cb5947ddb6c78a6773f root=UUID=432d640e-3339-45fa-a66d-89da9c869550 ro rd.lvm.lv=centos/ swap vconsole.font=latarcyrheb-sun16 rd.lvm.lv=centos/root crashkernel=auto vconsole.keymap=us rhgb quiet

To start, it is a good idea to remove the rhgb and quiet parts from this line; these arguments hide boot messages for you, and typically you do want to see what is happening while booting. In the next section you learn about some troubleshooting options that you can enter from the GRUB boot prompt.

After entering the boot options you want to use, press Ctrl+X to start the kernel with these options. Notice that these options are used one time only and are not persistent. To make them persistent you must modify the contents of the /etc/ default/grub configuration file and use grub2-mkconfig -o /boot/grub2/grub.cfg to apply the modification.

Reset Root Password

Resetting the Root Password

1. On system boot, press e when the GRUB 2 boot menu is shown.

2. Enter rd.break as boot argument to the line that loads the kernel and press Ctrl+X to boot with this option.

3. You’ll now be dropped at the end of the boot stage where initramfs is loaded, just before a mount of the root file system on the directory /.

4. Type mount -o remount,rw /sysroot to get read/write access to the system image.

5. At this point, make the contents of the /sysimage directory your new root directory by typing chroot /sysroot .

6. Now you can enter passwd and set the new password for the user root.

7. Because at this very early boot stage SELinux has not been activated yet, the context type on /etc/shadow will be messed up. If you reboot at this point, no one will be able to log in. So you must make sure that the context type is set correctly. To do this, at this point you should load the SELinux policy by using load_policy -i .

8. Now you can manually set the correct context type to /etc/shadow. To do this, type chcon -t shadow_t /etc/shadow .

9. Reboot. You can now log in with the changed password for user root. systemctl reboot

Grub2 Protection

PROTECTING GRUB 2 WITH A PASSWORD

• Password is required for modifying menu entries but not for booting existing menu entries;
• Password is required for modifying menu entries and for booting one, several, or all menu entries.

Configuring GRUB 2 to Require a Password only for Modifying Entries

To require password authentication for modifying GRUB 2 entries, follow these steps: Run the grub2-setpassword command as root:

Following this procedure creates a /boot/grub2/user.cfg file that contains the hash of the password. The user for this password, root, is defined in the /boot/grub2/grub.cfg file. With this change, modifying a boot entry during booting requires you to specify the root user name and your password.

Configuring GRUB 2 to Require a Password for Modifying and Booting Entries (RHEL 7)

menuentry 'Red Hat Enterprise Linux Server (3.10.0-327.18.2.rt56.223.el7_2.x86_64) 7.2 (Maipo)' --class red --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-327.el7.x86_64-advanced-c109825c-de2f-4340-a0ef-4f47d19fe4bf' {
        load_video
        set gfxpayload=keep

Manual changes to the /boot/grub2/grub.cfg persist when new kernel versions are installed, but are lost when re-generating grub.cfg using the grub2-mkconfig command. Therefore, to retain password protection, use the above procedure after every use of grub2-mkconfig.

If you delete the --unrestricted parameter from every menu entry in the /boot/grub2/grub.cfg file, all newly installed kernels will have menu entry created without --unrestricted and hence automatically inherit the password protection. Passwords Set Before Updating to Red Hat Enterprise Linux 7.2

The grub2-setpassword tool was added in Red Hat Enterprise Linux 7.2 and is now the standard method of setting GRUB 2 passwords. This is in contrast to previous versions of Red Hat Enterprise Linux, where boot entries needed to be manually specified in the /etc/grub.d/40_custom file, and super users - in the /etc/grub.d/01_users file.

Booting entries without the --unrestricted parameter requires the root password. However, GRUB 2 also enables creating additional non-root users that can boot such entries without providing a password. Modifying the entries still requires the root password. For information on creating such users, see the GRUB 2 Manual.

Rescue Disk

Rescue Disk

If you are lucky when you encounter trouble, you’ll still be able to boot from hard disk. If you are a bit less lucky, you’ll just see a blinking cursor on a system that does not boot at all. If that happens, you need a rescue disk. The default rescue image for Red Hat Enterprise Linux is on the installation disk. When booting from the installation disk, you’ll see a Troubleshooting menu item. Select this item to get access to the options you need to repair your machine.