Oracle RAC 11G ACFS 集群文件系统配置

#Oracle #RAC #11G #ACFS #集群文件系统配置

1 背景知识

本文主要介绍如何在 Oracle 11G 环境下添加ACFS 集群文件系统

2. 问题需求

在DSG 虚拟机服务器上一套centos 6.4 64位系统上oracle 11.2.0.4 rac测试环境，发现acfs和advm无法使用，ASMCA图形界面的acfs和volume选项卡为灰色！这是由于acfs不支持centos系统导致的，下面就介绍下如何在配置centos上使用acfs集群文件系统！

3. 问题描述

\[grid@raca ~\]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.4.0 Production on Mon Nov 9 15:49:08 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

With the Real Application Clusters and Automatic Storage Management options

SQL>  

SQL> alter diskgroup data set attribute 'COMPATIBLE.ASM'='11.2';

Diskgroup altered.

SQL> alter diskgroup data add volume acfsvol size 1G;

alter diskgroup data add volume acfsvol size 1G

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15477: cannot communicate with the volume driver

启动acfs 功能发现ORACLE 并不支持CENTOS上acfs功能

\[root@raca ~\]# /u01/gapp/grid/11g/bin/acfsload start -s

ACFS-9459: ADVM/ACFS is not supported on this OS version: 'centos-release-6.4'

4. 修改osds_acfslib.pm脚本代码

5. rac2节点做同样的修改

| cd /u01/gapp/grid/11g/lib/cp -p osds_acfslib.pm osds_acfslib.pm.bak vi osds_acfslib.pm |

 if ((defined($release)) && \# Redhat or OEL if defined

 (($release =~ /^redhat-release/) || \# straight RH

 ($release =~ /^enterprise-release/) || \# Oracle Enterprise Linux

 ($release =~ /^oraclelinux-release/))) \# Oracle Linux

 {

将上面的代码片段修改如下:

 if ((defined($release)) &&                    \# Redhat or OEL if defined

 (($release =~ /^redhat-release/) || \# straight RH

 ($release =~ /^enterprise-release/) || \# Oracle Enterprise Linux

 ($release =~ /^centos-release/) || \# CentOS hack

   ($release =~ /^oraclelinux-release/))) \# Oracle Linux

 {

6. 安装acfs，配置acfs和advm模块启动自动加载，rac2节点做同样的配置

/u01/app/11.2.0/grid/bin/acfsload start -s

ACFS-9129: ADVM/ACFS not installed

1 1.1 安装acsfs和advm 模块

注意：lsb_relase 命令如果没有的话就使用yum install lsb 安装上。没有这个命令会报错。

\[root@racb bin\]# ./acfsload start -s

ACFS-9129: ADVM/ACFS not installed

\[root@racb bin\]# acfsroot install

ACFS-9300: ADVM/ACFS distribution files found.

ACFS-9307: Installing requested ADVM/ACFS software.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9321: Creating udev for ADVM/ACFS.

ACFS-9323: Creating module dependencies - this may take some time.

ACFS-9154: Loading 'oracleoks.ko' driver.

ACFS-9154: Loading 'oracleadvm.ko' driver.

ACFS-9154: Loading 'oracleacfs.ko' driver.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9156: Detecting control device '/dev/asm/.asm\_ctl\_spec'.

ACFS-9156: Detecting control device '/dev/ofsctl'.

ACFS-9309: ADVM/ACFS installation correctness verified.

2 1.2 启动ACFS

\[root@racb bin\]# acfsload start     

ACFS-9391: Checking for existing ADVM/ACFS installation.

ACFS-9392: Validating ADVM/ACFS installation files for operating system.

ACFS-9393: Verifying ASM Administrator setup.

ACFS-9308: Loading installed ADVM/ACFS drivers.

ACFS-9327: Verifying ADVM/ACFS devices.

ACFS-9156: Detecting control device '/dev/asm/.asm\_ctl\_spec'.

ACFS-9156: Detecting control device '/dev/ofsctl'.

ACFS-9322: completed

3 1.3 检查安装是否成功

\[root@racb bin\]# lsmod | grep oracle

oracleacfs 1990406 0

oracleadvm 250040 0

oracleoks 427672 2 oracleacfs,oracleadvm

4 1.4 自启动脚本

\[root@racb init.d\]# cat /etc/init.d/acfsload

#!/bin/sh

#chkconfig: 2345 30 21

\# description : Load Oracle ASM volume driver on system startup

ORACLE_HOME=/u01/gapp/grid/11g/

export ORACLE_HOME

$ORACLE_HOME/bin/acfsload start -s

5 1.5 增加启动脚本到chkconfig 里

\[root@racb init.d\]# chmod 755 /etc/init.d/acfsload

\[root@racb init.d\]# chkconfig --add acfsload

\[root@racb init.d\]# chkconfig | grep acfsload

acfsload 0:off 1:off 2:on 3:on 4:on 5:on 6:off

7. 添加ora.registry.acfs到ocr配置中

1 5.1 注册服务模板文件

\[root@raca ~\]# /u01/gapp/grid/11g/bin/crsctl add type ora.registry.acfs.type \

\> -basetype ora.local_resource.type \

\> -file /u01/gapp/grid/11g/crs/template/registry.acfs.type

2 5.2 注册服务类型

\[root@raca init.d\]# /u01/gapp/grid/11g/bin/crsctl add resource ora.registry.acfs \

\> -attr ACL=\\'owner:root:rwx,pgrp:oinstall:r-x,other::r--\\' \

\> -type ora.registry.acfs.type -f

3 5.3 检查是否注册成功

su - grid -c crs_stat | grep acfs

NAME=ora.registry.acfs

TYPE=ora.registry.acfs.type

4 5.4 确认注册

 \[root@rac1 ~\]# /sbin/acfsutil registry

8. 创建卷并格式化成acfs集群文件系统并挂载

1 6.1 创建新的磁盘组

alter diskgroup data add volume acfsvol size 1G;

Diskgroup altered.

创建磁盘组例子：

```sql
SQL> CREATE DISKGROUP dg4 external redundancy DISK 'ORCL:FDISK_DEV_SDF3' ATTRIBUTE 'compatible.asm' = '11.2',

'compatible.rdbms' = '11.2','compatible.advm'='11.2';

Diskgroup created.


## 8.2. 6.1 创建共享的磁盘目录（两个节点都执行）

```bash
\[root@raca init.d\]# mkdir /sharedisk

\[root@racb init.d\]# mkdir /sharedisk

2 6.2 创建卷并格式化

创建asm卷例子：

SQL> alter diskgroup dg4 add volume acfsvol size 200M;

Diskgroup altered.

\[root@b1 asm\]# ls /dev/asm  
acfsvol-303

格式化：

\[root@raca init.d\]# mkfs -t acfs -n SOFT /dev/asm/soft-207 

mkfs.acfs: version = 11.2.0.4.0

mkfs.acfs: on-disk version = 39.0

mkfs.acfs: volume = /dev/asm/soft-207

mkfs.acfs: volume size = 21474836480

mkfs.acfs: Format complete.

3 6.3 注册ACFS到ocr配置中

\[root@raca init.d\]# acfsutil registry -a -f /dev/asm/soft-207 /sharedisk/

acfsutil registry: mount point /sharedisk successfully added to Oracle Registry

4 6.4 挂载ACFS集群文件系统

\[root@raca init.d\]# mount.acfs -o all

5 6.5 检查挂载是否成功

\[root@raca init.d\]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda3      46G 19G 25G 43% /

tmpfs 1.9G 567M 1.4G 30% /dev/shm

/dev/sda1 2.0G 83M 1.8G 5% /boot

/dev/sr0 4.1G 4.1G 0 100% /mnt/cd

/dev/sdc1 30G 1.7G 27G 6% /dsg

/dev/asm/soft-207 20G 117M 20G 1% /sharedisk

\[root@racb init.d\]# df -h

Filesystem Size Used Avail Use% Mounted on

/dev/sda3 46G 19G 25G 43% /

tmpfs 1.9G 567M 1.4G 30% /dev/shm

/dev/sda1 2.0G 83M 1.8G 5% /boot

/dev/sr0 4.1G 4.1G 0 100% /mnt/cd

/dev/sdc1 30G 1.7G 27G 6% /dsg

/dev/asm/soft-207 20G 117M 20G 1% /sharedisk

6 6.6 ACFS 属性检查

\[root@racb init.d\]# acfsutil info fs

/sharedisk

 ACFS Version: 11.2.0.4.0

  flags: MountPoint,Available

 mount time: Mon Nov 9 15:44:21 2015

 volumes: 1

 total size: 21474836480

 total free: 21352267776

 primary volume: /dev/asm/soft-207

 label: SOFT

 flags:           Primary,Available,ADVM

 on-disk version: 39.0

 allocation unit: 4096

 major, minor: 252, 105985

 size: 21474836480

 free: 21352267776

 ADVM diskgroup SOFT

 ADVM resize increment: 33554432

 ADVM redundancy: unprotected

 ADVM stripe columns: 4

 ADVM stripe width: 131072

 number of snapshots: 0

 snapshot space usage: 0

 replication status: DISABLED