HOWTO: Prepare ASM Disks for RHEL55 on Silo’ed x86
Author: Antony
Date: 21-December-2010
(1). Terminology
· “ASM” stands for Automated Storage Management, and may be used to manage Linux block devices directly.
· “ASMLib refers to the Linux specific library provided by Oracle for providing ASM with read/write access to disk devices. Disk device name persistence can be a difficulty for Linux. For example, change the storage configuration and a disk that originally appeared as /dev/sdg might appear as /dev/sdh following a reboot. ASMLib is meant to provide disk name persistence for ASM disks.
· “Silo’ed” is a Lloyds term indicating that there is no hypervisor (e.g. VMware) and RHEL is installed directly on x86 hardware.
(2). Synopsis
The document describes the steps for preparing RHEL5 on silo’ed x86 for ASM with DM-MPIO for multipathing.
The 2.6 kernel device file naming scheme known as udev dynamically creates device file names at boot time. However, this can give rise to possibility of changes to device file names, for example, a device that may have once been named /dev/sdg, may be renamed to /dev/sdh following a reboot.
The /dev/dm-* devices are used internally by device-mapper-multipath and are non-persistent across reboot. The /dev/mpath* devices may not be available during early stages of boot. However, /dev/mapper/ devices can be persistent and are created sufficiently early during the boot process – The recommendation seems to be to only use these to access and interact with multipath devices.
Oracle strongly recommends using ASMLib instead of udev for effective handling of multipath disks for ASM. When using ASMLib, you should not need to worry about device naming and this should ensure that ASM sees the correct disk devices no matter what the /dev/ names are.
(3). Install ASMLib
Oracle’s ASMLib downloads can be reached at the following URL: -
Select operating system version (e.g. Red Hat Enterprise Linux 5 AS).
Download RPMs for library and tools, oracleasm-support and oracleasmlib for the relevant architecture (e.g. x86).
Download RPM for Driver, oracleasm for relevant architecture and kernel (e.g. x86 and 2.6.18-194.3.1.el5).
Install these RPMs on the Oracle nodes, similar to the following example: -
root# rpm –ivh oracleasm-support-2.1.3-1.el5.x86_64.rpm oracleasmlib-2.0.4-1.el5.x86_64.rpm oracleasm-2.6.18-194.3.1.el5-2.0.5-1.el5.x86_64.rpm
The oracleasm package is specific for the Linux kernel version and release, and it needs to be recompiled or downloaded from the Oracle repository for each kernel update.
(4). Partition Disks for ASM
ASM tends to be configured with two main storage areas: -
· Data Disk Group for Data files, Control files and Redo logs.
· Flash Disk Group for Archive logs, Backup files and Flashback logs.
As a rule-of-thumb, Oracle might suggest setting the Flash Disk Group to twice the size of Data Disk Group, but it very much depends on Oracle database requirement, and you should be guided by the Oracle Designer on this.
Disk partitions may either be LVM managed or ASM managed, not both. The disk headers differ, and LVM does not recognise ASM disk and ASM does not recognise LVM disk.
WARNING: Take special care not to erroneously configure an LVM disk to ASM or vice versa, otherwise this could result in data corruption.
For RHEL5, it is recommended to use ASM with partitioned block devices.
root# parted /dev/mapper/<mpath#> mklabel msdos unit s mkpart primary "1 -1"
root# partprobe
Substitute <mpath#> for the mpath device name/number (e.g. mpath2).
Perform these steps for each disk to be configured for ASM.
This above parted command creates a single partition with the full capacity of the disk volume (i.e. from start to end cylinders).
Use of parted (as opposed to fdisk) is recommended for RHEL5 to future proof the transition from MBR (Master Boot Record) to GPT (GUID Partition Table) and support for addressing in excess of 2TB.
Look under the /dev/mapper directory, and you should see that block devices have been created for each new partition. For example, if you partitioned /dev/mapper/mpath2, then /dev/mapper/mpath2p1 should have been created.
(5). Configure ASMLib
Make the following amendment to the /etc/sysconfig/oracleasm to enable ASM to scans all disks it is allowed to discover.
ASMLib should be configured to use only multipath devices. This can be accomplished by setting the ASMLib scanning order in the /etc/sysconfig/oracleasm configuration file (which is a link to oracleasm-_dev_oracleasm in the same directory).
ORACLEASM_SCANORDER="dm-*"
Configure on-boot properties for ASMLib: -
root# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]
root# chkconfig --list oracleasm
oracleasm 0:off 1:off 2:on 3:on 4:on 5:on 6:off
root# service oracleasm status
Checking if ASM is loaded: yes
Checking if /dev/oracleasm is mounted: yes
root# df -ha /dev/oracleasm
Filesystem Size Used Avail Use% Mounted on
oracleasmfs 0 0 0 - /dev/oracleasm
(6). Define ASM Volumes
Run the following command to list all mpath devices and associated UUID: -
root# multipath –l
Identify the mpath devices(s) designated for ASM (i.e. the ones partitioned earlier in this document), and parse the mpath device and also the 3 digit part of the UUID which is specific to the disk: -
root# multipath -l | grep <mpath#> | sed -e 's/(//' -e 's/)//' | awk '{print $1,substr($2,27,3);}'
Substitute <mpath#> for the mpath device name/number (e.g. mpath2).
Use the following name format as the label ASM disks: -
VOL_<3-digits-of-uuid>
Substitute <3-digits-of-uuid> for 3 digit part of the UUID which is specific to the disk.
With RHEL5, you can create the ASMLib disks on the /dev/mapper/<partition> devices.
Define ASM volume, similar to the following example: -
root# service oracleasm createdisk VOL_816 /dev/mapper/mpath4p1
root# service oracleasm querydisk -d VOL_816
Substitute the 3 digits and block device name of the partition.
The label created on the partitioned device allows ASMLib daemon to scan and auto-detect ASM devices. It should simplify device management because auto-detection works around the issue with non-persistent device names. In addition, there is no need to configure permissions on the ASM devices because the ASM instance accesses them via the ASMLib kernel module.
Invoke ASMLib to scan the disks: -
root# service oracleasm scandisks
List defined ASM volumes: -
root# service oracleasm listdisks
root# ls –altr /dev/oracleasm/disks
The theory is that the createdisk command makes the disk persistent for Oracle – For example, reinstall a system and lose the udev configuration, but ASMLib labels on ASM disks remain - ASMLib should just requires invocation of the scandisks command to rediscover. The writer discovered that the ASM volume header can be read from a disk partition using the following command, and this could be used to provide a means of verification: -
root# dd if=/dev/mapper/<mpathx> bs=512 count=2 | strings -an 2
Substitute <mpath#> for the mpath device name/number (e.g. mpath2).
(7). Next Step …
Please handover to the Oracle Builder to perform the Oracle binaries install and complete ASM configuration.
(8). Risks, Issues & Further Considerations
At the time of writing this document, the ASM configuration for RHEL5 was only experimental at Lloyds. The Linux Rapid Deployment Team was allowed very little time to research and develop this before putting this to use for a project (e.g. Genesis SIT environment, in this case, which is an early adopter of Linux).
This section seeks to identify aspects of potential risk requiring further investigation, which must either be resolved or managed.
Needless to say, all of these aspects should be verified and tested during the course of Linux Rapid Deployment project.
It is logical to mention these aspects so that there is a clear record and to avoid these from being overlooked.
i. Problematic Disk Discovery by ASM?
After Oracle was installed, there was a problem with ASM discovering the disks due to the way the discovery string was set. As a temporary workaround, Wayne Douglas set asm_diskstring to /dev/oracleasm/disks/VOL* (as per SQL command alter system set asm_diskstring …), instead of ORCL:*. This should only be a regarded as a temporary workaround, because this bypasses the ASMLib API. Wayne Douglas intends to raise an SR with Oracle to resolve this discovery issue.
ii. Add ASMLib packages to Custom Channel on RHN Satellite
With a view to automating the provisioning process for Oracle Roles, consideration should be given to resigning the ASMLib RPMs with GPG Key and rhnpush’ing these to a Custom Channel.
iii. Are all multipath devices scaned by ASMLib (not just ones for configured for ASM)?
To read disk devices, it follows that ASMLib must grant ASM effective access rights for the purpose (i.e. via the ASMLib kernel module). OS administrators may not want oracle processes to access system disks just so that ASM knows to ignore them. Does this introduce any risk of Oracle accidentally writing over a system volume? Should we even bother to try to mitigate against this (e.g. possibly using ORACLEASM_SCANEXCLUDE in /etc/sysconfig/oracleasm).
iv. Can disk device name persistence be maintained?
The RHEL5 2.6 kernel device file naming scheme known as udev dynamically creates device file names at boot time. However, this can give rise to possibility of changes to device file names, for example, a device that may have once been named /dev/sdg say, may be renamed as /dev/sdh following a reboot.
With the user_friendly_names parameter set to yes in /etc/multipath.conf, multipath should automatically maintain the contents of /var/lib/multipath/bindings, and this is understood to define the mapping between UUID and mpath device name.
Also, when using ASMLib, you should not need to worry about device naming and this should ensure that ASM sees the correct devices no matter what /dev/ names.
The only way to be certain is to test. A good start would be to perform multiple node restarts and reboots to ensure that device names persist.
v. Could this configuration be adopted in the future for a shared disk cluster (e.g. RAC)?
The answer should be yes, but this would require research, development and testing to ensure that disk device discovery is both persistent and consistent across all nodes participating in a RAC cluster.
No comments:
Post a Comment