Oracle 11g集群Rac安装 Centos7

Oracle 11g RAC 双节点

一、规划

1.1、系统规划

磁盘名称	磁盘大小	作用
/dev/sda	20G	OS 操作系统

1.2、网络规划

主机名称	rac1	rac2
public ip	172.16.130.200	172.16.130.210
private ip	10.0.80.200	10.0.80.210
vip	172.16.130.201	172.16.130.211
scan ip	172.16.130.220	172.16.130.220

1.3、磁盘规划

采用Free NAS 的 iSCSI 块共享存储

磁盘名称	磁盘大小	ASM磁盘名称	作用
/dev/sdb	2G	asm-ocr1	OCR/Voting File
/dev/sdc	2G	asm-ocr2	OCR/Voting File
/dev/sdd	2G	asm-ocr3	OCR/Voting File
/dev/sde	20G	asm-data	Data Files

二、系统环境及参数设置

2.1、YUM仓库配置（双节点）

mv /etc/yum.repos.d/* /tmp/
echo "[local_yum]" >> /etc/yum.repos.d/henry.repo
echo "name = henry_repo" >> /etc/yum.repos.d/henry.repo
echo "baseurl = file:///mnt/" >> /etc/yum.repos.d/henry.repo
echo "enabled = 1" >> /etc/yum.repos.d/henry.repo
echo "gpgcheck = 0" >> /etc/yum.repos.d/henry.repo
mount /dev/cdrom /mnt/

yum -y install autoconf automake binutils binutils-devel bison cpp dos2unix ftp gcc gcc-c++ lrzsz python-devel compat-libcap1 compat-libstdc* libXpm-*.i686 libXext libXext.i686 libX11 libX11.i686 libxcb libxcb.i686 libaio* libaio-devel libaio-devel.i686 ksh libXp libaio-devel numactl numactl-devel make sysstat unixODBC unixODBC-devel elfutils-libelf-devel redhat-lsb-core unzip libXrender libXrender-devel nfs-utils smartmontools tigervnc* libvirt* e2fsprogs.x86_64 libXau* libXi* libXtst* libstdc* glibc* libgcc* ksh gcc-c++*

2.2、编辑HOSTS文件（双节点）

vim /etc/hosts
# 添加内容
#scanip
172.16.130.220 racscan
#rac1
172.16.130.200 rac1
10.0.80.200 rac1pri
172.16.130.201 rac1vip
#rac2
172.16.130.210 rac2
10.0.80.210 rac2pri
172.16.130.211 rac2vip

2.3、创建用户和组（双节点）

/usr/sbin/groupadd -g 5001 oinstall
/usr/sbin/groupadd -g 5002 dba
/usr/sbin/groupadd -g 5003 oper
/usr/sbin/groupadd -g 5004 asmadmin
/usr/sbin/groupadd -g 5005 asmoper
/usr/sbin/groupadd -g 5006 asmdba
/usr/sbin/useradd -g oinstall -G dba,asmdba,oper -d /home/oracle -m oracle
/usr/sbin/useradd -g oinstall -G asmadmin,asmdba,asmoper,dba -d /home/grid -m grid
# 修改用户 oracle 及 grid 密码
echo "oracle" | passwd --stdin oracle
echo "grid" | passwd --stdin grid

2.4、创建文件夹及授权（双节点）

mkdir -p /u01/app/{grid,11g/grid,oracle/product/11g/db_01,oraInventory}
chown -R grid:oinstall /u01/app/grid
chown -R grid:oinstall /u01/app/11g
chown -R oracle:oinstall /u01/app/oracle
chown -R grid:oinstall /u01/app/oraInventory
chmod -R 775 /u01

2.5、用户资源限制配置（双节点）

vim /etc/security/limits.conf
# 添加内容
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft stack 10240
oracle hard stack 32768
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
grid soft stack 10240
grid hard stack 32768
* soft memlock unlimited
* hard memlock unlimited
root soft stack 10240
root hard stack 32768

2.6 、将资源限制加入验证模块（双节点）

echo "session required pam_limits.so" >> /etc/pam.d/login

2.7、修改内核参数（双节点）

vim /etc/sysctl.conf
# 编辑文件添加以下内容
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
kernel.panic_on_oops = 1
kernel.shmmax = 7730941132
kernel.shmall = 1887436
kernel.shmmni = 4096
# 执行命令刷新生效
sysctl -p

2.8、关闭透明大页（双节点）

vim /etc/rc.local
# 添加内容
if test -f /sys/kernel/mm/transparent_hugepage/enabled; then
echo never > /sys/kernel/mm/transparent_hugepage/enabled
fi
if test -f /sys/kernel/mm/transparent_hugepage/defrag; then
echo never > /sys/kernel/mm/transparent_hugepage/defrag
fi

2.9、关闭防火墙（双节点）

systemctl stop firewalld.service
systemctl disable firewalld.service

2.10、关闭SELinux（双节点）

sed -i '/^SELINUX=.*/ s//SELINUX=disabled/' /etc/selinux/config

vim /etc/sysconfig/selinux
# 修改内容
SELINUX=disabled

2.11 关闭其他不需要的服务（双节点-可选）

systemctl stop avahi-dnsconfd
systemctl stop avahi-daemon
systemctl stop cups
systemctl stop postfix
systemctl stop smartd
systemctl disable avahi-dnsconfd
systemctl disable avahi-daemon
systemctl disable cups
systemctl disable postfix
systemctl disable smartd

三、编辑用户环境变量

3.1、GRID用户环境变量

3.1.1、节点1

su - grid
vim .bash_profile
# 添加内容
PS1="[`whoami`@`hostname`:"'$PWD]$'
export PS1
umask 022
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_HOSTNAME=rac1
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
ORACLE_HOME=/u01/app/11g/grid; export ORACLE_HOME
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
PATH=.:$PATH:$HOME/bin:$ORACLE_HOME/bin; export PATH
THREADS_FLAG=native; export THREADS_FLAG
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi

3.1.2、节点2

su - grid
vim .bash_profile
# 添加内容
PS1="[`whoami`@`hostname`:"'$PWD]$'
export PS1
umask 022
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_HOSTNAME=rac2
ORACLE_SID=+ASM2; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
ORACLE_HOME=/u01/app/11g/grid; export ORACLE_HOME
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
PATH=.:$PATH:$HOME/bin:$ORACLE_HOME/bin; export PATH
THREADS_FLAG=native; export THREADS_FLAG
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi

3.2、ORACLE用户环境变量

3.2.1、节点1

su - oracle
vim .bash_profile
# 添加内容
PS1="[`whoami`@`hostname`:"'$PWD]$'
export PS1
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_HOSTNAME=rac1
export ORACLE_UNQNAME=SALESERP
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11g/db_01; export ORACLE_HOME
ORACLE_SID=SALES1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
NLS_LANG=AMERICAN_AMERICA.ZHS16GBK;export NLS_LANG
PATH=.:$PATH:$HOME/bin:$ORACLE_BASE/product/11g/db_01/bin:$ORACLE_HOME/bin; export PATH
THREADS_FLAG=native; export THREADS_FLAG
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi

3.2.2、节点2

su - oracle
vim .bash_profile
# 添加内容
PS1="[`whoami`@`hostname`:"'$PWD]$'
export PS1
export TMP=/tmp
export TMPDIR=$TMP
export ORACLE_HOSTNAME=rac2
export ORACLE_UNQNAME=SALESERP
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11g/db_01; export ORACLE_HOME
ORACLE_SID=SALES2; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="yyyy-mm-dd HH24:MI:SS"; export NLS_DATE_FORMAT
NLS_LANG=AMERICAN_AMERICA.ZHS16GBK;export NLS_LANG
PATH=.:$PATH:$HOME/bin:$ORACLE_BASE/product/11g/db_01/bin:$ORACLE_HOME/bin; export PATH
THREADS_FLAG=native; export THREADS_FLAG
if [ $USER = "oracle" ] || [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi

四、SSH 互信

4.1、GRID用户互信

4.1.1、节点1

su - grid
rm -rf ~/.ssh
mkdir ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa

4.1.2 节点2

su - grid
rm -rf ~/.ssh
mkdir ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa

4.1.3 节点1追加互信并拷贝至节点2

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh rac2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
scp /home/grid/.ssh/authorized_keys rac2:~/.ssh/authorized_keys

4.1.4 节点1验证互信（无需密码）

ssh rac1 date
ssh rac2 date
ssh rac1pri date
ssh rac2pri date

4.1.5 节点2验证互信（无需密码）

ssh rac1 date
ssh rac2 date
ssh rac1pri date
ssh rac2pri date

4.2 ORACLE用户互信

4.2.1 节点1

su - oracle
rm -rf ~/.ssh
mkdir ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa

4.2.2 节点2

su - oracle
rm -rf ~/.ssh
mkdir ~/.ssh
ssh-keygen -t rsa
ssh-keygen -t dsa

4.2.3 节点1追加互信并拷贝至节点2

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
ssh rac2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
ssh rac2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
scp /home/oracle/.ssh/authorized_keys rac2:~/.ssh/authorized_keys

4.2.4 节点1验证互信（无需密码）

ssh rac1 date
ssh rac2 date
ssh rac1pri date
ssh rac2pri date

4.2.5 节点2验证互信（无需密码）

ssh rac1 date
ssh rac2 date
ssh rac1pri date
ssh rac2pri date

五、存储划分

本次测试采用Free NAS 的iSCSI块共享服务

5.1、挂载iSCSI

配置好iscsi后，通过iscsi发现并挂载

#发现iscsi target
[root@rac1 ~]# iscsiadm -m discovery -t st -p 10.0.80.190
10.0.80.190:3260,-1 iqn.2015-10.org.freenas.ctl:freenas-iscsi
#登录iscsi target
[root@rac1 ~]# iscsiadm -m node -T iqn.2015-10.org.freenas.ctl:freenas-iscsi -l
Logging in to [iface: default, target: iqn.2015-10.org.freenas.ctl:freenas-iscsi, portal: 10.0.80.190,3260] (multiple)
Login to [iface: default, target: iqn.2015-10.org.freenas.ctl:freenas-iscsi, portal: 10.0.80.190,3260] successful.

5.2、将盘添加到udev规则和用户组

#将重命名的盘添加udev规则和所属用户、用户组
for i in b c d e;
do
echo "KERNEL==\"sd*\", ENV{DEVTYPE}==\"disk\", SUBSYSTEM==\"block\", PROGRAM==\"/usr/lib/udev/scsi_id -g -u -d \$devnode\",RESULT==\"`/usr/lib/udev/scsi_id -g -u -d /dev/sd$i`\", RUN+=\"/bin/sh -c 'mknod /dev/asm-diskb b \$major \$minor; chown grid:asmadmin /dev/asm-diskb; chmod 0660 /dev/asm-diskb'\"" >> /etc/udev/rules.d/99-oracle-asmdevices.rules
done
#使配置生效
[root@rac1 ~]# partprobe /dev/sd[b-e]
[root@rac2 ~]# udevadm control --reload
[root@rac2 ~]# udevadm trigger

六、GRID集群软件安装

1、上传集群软件包

将p13390677_112040_Linux-x86-64_3of7.zip上传到rac1上边

解压p13390677_112040_Linux-x86-64_3of7.zip到/u01/app/11g/grid文件夹中

进入文件夹/u01/app/11g/grid

2、安装新集群

主节点安装/u01/app/11g/grid/cv/rpm/cvuqdisk-1.0.9-1.rpm，安装后将/u01/app/11g/grid/cv/rpm/cvuqdisk-1.0.9-1.rpm包传到其他节点安装

运行命令：./runInstaller依次选择以下选项

Skip software updates;
#Next>
Install and Configure Oracle Grid Infrastructure for a Cluster
#Next>
Advanced Installation
#Next>
选择语言
#Next>
Cluster Name:ora-rac        #集群名称
SCAN Name:racscan           #hosts中的SCAN IP 的解析名
SCAN Port:1521
取消 Configure GNS
#Next>
Public HostName             Virtual HostName
rac1                        rac1vip         #本机信息
rac2                        rac2vip         #在这里添加其他节点的信息
#Next>
Interface Name              Subnet              Use for
ens33                       172.16.130.0        Public
ens34                       10.0.80.0           Private
#Next>
Oracle Automatic Storage Management（Oracle ASM）
#Next>
Disk Group Name DATA
Redundancy Normal
AU Size 4MB
ADD Disks  #点击change Discovery Path  更改位asm磁盘位置 /dev/asm*
选择创建磁盘组的磁盘，三个2G的磁盘
#Next>
Use same passwords for there accounts;
输入密码
#Next>
Do not use Intelligent Platform Management Interface (IPMI)
#Next>
不做变动
#Next>
不做变动
#Next>
不做变动
#Next>
不做变动
#Next>
I Gnore All打上勾
#Next>
#Install
按要求在两个节点使用root用户执行脚本`/u01/app/oraInventory/orainstRoot.sh`和`/u01/app/11g/grid/root.sh`