- Design
- Setting up OpenFiler(SAN)
- Installing RAC1
- Installing RAC2
- DNS Records
- Setting up GRID infrastructure>
- Installing the database
- Control
- Settings
- OCR and Voting
- Services
- Errors
- Best Practices

Applications to be used during setup
Oracle
http://www.oracle.com/technetwork/database/enterprise-edition/downloads/112010-linx8664soft-100572.html
linux.x64_11gR2_database
linux.x64_11gR2_grid
On Linux operating systems (distros)
https://edelivery.oracle.com/EPD/Search/handle_go
Along with the operating system, the version of oracle that is running should be certified by Oracle
https://metalink.oracle.com (requires registering a user ID & password, on behalf of your organization) -> can be found in the Certifications category/tab
This isn't a requirement if the operating system being used isn't certified. However, using a system that hasn't been certified can result in continuity problems as well as ineligibility for official Oracle support. In a production enironment this is highly discouraged !!
A Turkish bank had made this mistake at one point, resulting in serious problems.
SAN (open filer)
http://www.openfiler.com/community/download/
Network Settings
NAS
NIC-1 -> custom-vmnet1 (hostonly)
NIC-2 -> bridge
RAC1
NIC-1 -> custom-vmnet1 (hostonly)
NIC-2 -> bridge
RAC2
NIC-1 -> custom-vmnet1 (hostonly)
NIC-2 -> bridge
NAS
[root@nas ~]# ifconfigeth0 Link encap:Ethernet HWaddr 00:0C:29:6C:7F:AE
inet addr:10.0.0.10 Bcast:10.0.0.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe6c:7fae/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:24594 errors:0 dropped:0 overruns:0 frame:0
TX packets:36880 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:5044034 (4.8 MiB) TX bytes:49076445 (46.8 MiB)
eth1 Link encap:Ethernet HWaddr 00:0C:29:6C:7F:B8
inet addr:192.168.137.101 Bcast:192.168.137.255 Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe6c:7fb8/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:84 errors:0 dropped:0 overruns:0 frame:0
TX packets:14 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:7608 (7.4 KiB) TX bytes:843 (843.0 b)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:8 errors:0 dropped:0 overruns:0 frame:0
TX packets:8 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:560 (560.0 b) TX bytes:560 (560.0 b)
RAC1 (Before setting up Oracle)
[root@rac1 ~]# ifconfig
eth0 Link encap:Ethernet HWaddr 00:0C:29:CB:1B:0F
inet addr:10.0.0.20 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:92903 errors:0 dropped:0 overruns:0 frame:0
TX packets:71071 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:111026826 (105.8 MiB) TX bytes:23477544 (22.3 MiB)
eth1 Link encap:Ethernet HWaddr 00:0C:29:CB:1B:19
inet addr:192.168.137.20 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:278 errors:0 dropped:0 overruns:0 frame:0
TX packets:282 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:57151 (55.8 KiB) TX bytes:57122 (55.7 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:27765 errors:0 dropped:0 overruns:0 frame:0
TX packets:27765 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9532941 (9.0 MiB) TX bytes:9532941 (9.0 MiB)
After setting up Oracle
eth0 Link encap:Ethernet HWaddr 00:0C:29:CB:1B:0Finet addr:10.0.0.20 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:92903 errors:0 dropped:0 overruns:0 frame:0
TX packets:71071 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:111026826 (105.8 MiB) TX bytes:23477544 (22.3 MiB)
eth1 Link encap:Ethernet HWaddr 00:0C:29:CB:1B:19
inet addr:192.168.137.20 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:278 errors:0 dropped:0 overruns:0 frame:0
TX packets:282 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:57151 (55.8 KiB) TX bytes:57122 (55.7 KiB)
eth1:2 Link encap:Ethernet HWaddr 00:0C:29:CB:1B:19
inet addr:192.168.137.52 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1:3 Link encap:Ethernet HWaddr 00:0C:29:CB:1B:19
inet addr:192.168.137.50 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1:4 Link encap:Ethernet HWaddr 00:0C:29:CB:1B:19
inet addr:192.168.137.40 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:27765 errors:0 dropped:0 overruns:0 frame:0
TX packets:27765 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9532941 (9.0 MiB) TX bytes:9532941 (9.0 MiB)
RAC2 (Before setting up Oracle)
[root@rac2 ~]# ifconfigeth0 Link encap:Ethernet HWaddr 00:0C:29:FA:26:13
inet addr:10.0.0.21 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:137807 errors:0 dropped:0 overruns:0 frame:0
TX packets:113477 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:149604744 (142.6 MiB) TX bytes:33850337 (32.2 MiB)
eth1 Link encap:Ethernet HWaddr 00:0C:29:FA:26:1D
inet addr:192.168.137.21 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:569 errors:0 dropped:0 overruns:0 frame:0
TX packets:619 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:133589 (130.4 KiB) TX bytes:135330 (132.1 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:18384 errors:0 dropped:0 overruns:0 frame:0
TX packets:18384 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9657346 (9.2 MiB) TX bytes:9657346 (9.2 MiB)
After setting up Oracle
[root@rac2 ~]# ifconfigeth0 Link encap:Ethernet HWaddr 00:0C:29:FA:26:13
inet addr:10.0.0.21 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:137807 errors:0 dropped:0 overruns:0 frame:0
TX packets:113477 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:149604744 (142.6 MiB) TX bytes:33850337 (32.2 MiB)
eth1 Link encap:Ethernet HWaddr 00:0C:29:FA:26:1D
inet addr:192.168.137.21 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:569 errors:0 dropped:0 overruns:0 frame:0
TX packets:619 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:133589 (130.4 KiB) TX bytes:135330 (132.1 KiB)
eth1:1 Link encap:Ethernet HWaddr 00:0C:29:FA:26:1D
inet addr:192.168.137.41 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
eth1:2 Link encap:Ethernet HWaddr 00:0C:29:FA:26:1D
inet addr:192.168.137.51 Bcast:192.168.137.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:18384 errors:0 dropped:0 overruns:0 frame:0
TX packets:18384 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9657346 (9.2 MiB) TX bytes:9657346 (9.2 MiB)
Setting up OpenFiler(SAN)
http://www.openfiler.com/community/download/
https://10.0.0.10:446 accessible using the following log-on credentials: user : openfiler password : password
Configuring the disks so that they can be accessed by both the RAC1 and RAC2 servers (3x100Gb & 1x50Gb)

Beginning setup by booting Enterprise-R5-U6-Server-x86_64-dvd.iso
Packets to be loaded (without loading sound, office, video, etc. tools)
Desktop Environments | Gnome Desktop Environment |
Applications | Editors Graphical Internet Text-based Internet |
Development | Development Libraries Development Tools Legacy Software Development |
Servers | Server Configuration Tools |
Base System | Administration Tools Base Java Legacy Software Support System Tools X Window System |
It's good practice to make sure that the setup was successfully completed and that all services are running before taking the necessary security precautions.
The more services/programs/processes that are running, the longer it can take to find the cause of any problems.
(Example: The firewall could be blocking client access to the database, looking for the cause this inaccessibility within Oracle's own services would be a waste of time.)
(Approx 60Gb of disk space was used with this particular setup however it is recommended reserving at least 150-200Gb for a production environment as 50-60Gb dump files can be created).
Once the setup is finished and the system has been restarted, open the system as a root user and prepare for the RAC setup step by step.
Step 1
In VMWare workstation, right-click on the RAC1 virtual machine and select install vmware tools.
The VM tool CD will get automatically mounted. Open the CD's directory, within the /media directory.
sh vmware-config-tools.pl
VMWare gets loaded after hitting enter on all the questions.
Step 2
Type vmware-toolbox in the terminal, press enter and tick the ntp checkbox in the window.
Copy the grid and database setup in the /setup folder.
chown -R oracle:oinstall /setup
chmod -R +x /setup
Step 3
Loading the necessary RPMs
rpm -Uvh binutils-2.*
rpm -Uvh compat-libstdc++-33*
rpm -Uvh elfutils-libelf-0.*
rpm -Uvh elfutils-libelf-devel-*
rpm -Uvh gcc-4.*
rpm -Uvh gcc-c++-4.*
rpm -Uvh glibc-2.*
rpm -Uvh glibc-common-2.*
rpm -Uvh glibc-devel-2.*
rpm -Uvh glibc-headers-2.*
rpm -Uvh ksh-2*
rpm -Uvh libaio-0.*
rpm -Uvh libaio-devel-0.*
rpm -Uvh libgcc-4.*
rpm -Uvh libstdc++-4.*
rpm -Uvh libstdc++-devel-4.*
rpm -Uvh make-3.*
rpm -Uvh sysstat-7.*
rpm -Uvh unixODBC-2.*
rpm -Uvh unixODBC-devel-2.*
Loading the RPMs needed for the ASM these are available within the first 2 file CDs. The 3rd CD can be downloaded from this link: http://download.oracle.com/otn_software/asmlib/oracleasmlib-2.0.4-1.el5.x86_64.rpm
rpm -Uvh oracleasm-2.6.18-238.el5-2.0.5-1.el5.x86_64.rpm
rpm -Uvh oracleasm-support-2.1.4-1.el5.x86_64.rpm
rpm -Uvh oracleasmlib-2.0.4-1.el5.x86_64.rpm
In the /grid/rpm directory
rpm -Uvh cvuqdisk-1.0.7-1.rpm
Step 4
Procedures for users
groupadd -g 1000 oinstall
groupadd -g 1200 dba
useradd -u 1100 -g oinstall -G dba oracle
passwd oracle
mkdir -p /u01/app/11.2.0/grid
mkdir -p /u01/app/oracle/product/11.2.0/db_1
chown -R oracle:oinstall /u01
chmod -R 775 /u01/
Step 5
Network Settings
The contents of /etc/hosts
127.0.0.1 localhost.localdomain localhost
# Public
192.168.137.20 rac1.sysdba.org rac1
192.168.137.21 rac2.sysdba.org rac2
# Virtual
192.168.137.40 rac1-vip.sysdba.org rac1-vip
192.168.137.41 rac2-vip.sysdba.org rac2-vip
# Private
10.0.0.20 rac1-priv.sysdba.org rac1-priv
10.0.0.21 rac2-priv.sysdba.org rac2-priv
10.0.0.10 nas.sysdba.org nas
Contents of /etc/resolve.conf
nameserver 192.168.137.100
Step 6
System settings:
Add the following lines to /etc/sysctl.conf, the same line numbers automatically get updated.
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 1054504960
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=262144
10 net.core.rmem_max=4194304
11 net.core.wmem_default=262144
12 net.core.wmem_max=1048586
/sbin/sysctl -p
/etc/security/limits.conf
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
/etc/pam.d/login
session required pam_limits.so
As the Oracle Cluster Time Synchronization Service (ctssd) wşll be used, Linux's ntp service will be stopped.
# service ntpd stop
# chkconfig ntpd off
# mv /etc/ntp.conf /etc/ntp.conf.org
# rm /var/run/ntpd.pid
Step 7
Add the following lines to Oracle's user profile (/home/oracle/.bash_profile)
# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=rac1.sysdba.org; export ORACLE_HOSTNAME
ORACLE_UNQNAME=rac; export ORACLE_UNQNAME
ORACLE_BASE=/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/db_1; export ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=$ORACLE_HOME/bin:$PATH; export PATH
PATH=/usr/sbin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
Note: If Oracle's grid has been setup in /oragrid within /oracle, add the following lines into the oracle user and the root user profiles.
For the oracle user
ORACLE_HOSTNAME=rac1.sysdba.org; export ORACLE_HOSTNAME
ORACLE_UNQNAME=RAC; export ORACLE_UNQNAME
ORACLE_BASE=/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/db; export ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
For the root user (who's automatically also a grid owner)
ORAGRID_BASE=/oragrid
ORAGRID_HOME=$ORAGRID_BASE/product/11.2.0/grid; export ORAGRID_HOME
PATH=$ORAGRID_HOME/bin:$PATH; export PATH
Step 8
Checking whether iscsi (Internet SCSI) has been loaded
rpm -qa | grep iscsi-initiator-utils
if it hasn't been loaded mount the CD
mount -r /dev/cdrom /media/cdrom
cd /media/cdrom/Server....
rpm -Uvh iscsi-initiator-utils.... iscsi is loaded and started
It's necessary to add LUN to the openfiler's iscsi and map the volumes.
Following this, in the "iscsi target" "Network ACL" tab give permission to "internal_network".
iscsiadm -m discovery -t sendtargets -p 10.0.0.10
iscsiadm -m node -T iqn.2006-01.com.openfiler:tsn.de0b812f5012 -p 10.0.0.10 -l
The following lines should appear during the operating system's restart sequence:
The disks should automatically be visible to the os after restarting (before it has been formatted).
Step 9
(The procedures shown here can also be accomplished using the asmca gui tool (ASM configuration assistant)
Configuring the disks (using the sequence of "n", "p", "1", "w")
[root@rac1 Server]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 12446.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
(e.g., DOS FDISK, OS/2 FDISK)
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-12446, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-12446, default 12446):
Using default value 12446
Command (m for help): p
Disk /dev/sdb: 102.3 GB, 102374572032 bytes
255 heads, 63 sectors/track, 12446 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 12446 99972463+ 83 Linux
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.
Carrying out the same procedures in both the sdc and sdd disks
[root@rac1 Server]# fdisk -l çıktısının aşağıdaki gibi olması lazım.
Disk /dev/sda: 64.4 GB, 64424509440 bytes
255 heads, 63 sectors/track, 7832 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 7832 62806117+ 8e Linux LVM
Disk /dev/dm-0: 60.0 GB, 60095987712 bytes
255 heads, 63 sectors/track, 7306 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1: 4194 MB, 4194304000 bytes
255 heads, 63 sectors/track, 509 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-1 doesn't contain a valid partition table
Disk /dev/sdb: 102.3 GB, 102374572032 bytes
255 heads, 63 sectors/track, 12446 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 12446 99972463+ 83 Linux
Disk /dev/sdc: 102.3 GB, 102374572032 bytes
255 heads, 63 sectors/track, 12446 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 12446 99972463+ 83 Linux
Disk /dev/sdd: 102.3 GB, 102374572032 bytes
255 heads, 63 sectors/track, 12446 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 12446 99972463+ 83 Linux
Disk /dev/sde: 51.1 GB, 51170508800 bytes
64 heads, 32 sectors/track, 48800 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sde doesn't contain a valid partition table
Step 10
Setting the shared disks of the ASM environment and loading the kernel modules
oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle
Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]:
Writing Oracle ASM library driver configuration: done
Loading the kernel modules
oracleasm init
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm
To create the ASM disks configure the paths of the disks which were created using iSCSI
#oracleasm createdisk DISK1 /dev/sdb1
Writing disk header: done
Instantiating disk: done
#oracleasm createdisk DISK2 /dev/sdc1
Writing disk header: done
Instantiating disk: done
#oracleasm createdisk DISK3 /dev/sdd1
Writing disk header: done
Instantiating disk: done
Scanning the disks
#oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
Listing the disks
#oracleasm listdisks
DISK1
DISK2
DISK3
RAC1 has now been setup, moving on to RAC2.
After setting up RAC1, RAC2 is easier. Actually, all that needs to be done is to clone RAC1 and change a few settings.
Step 1.
Shutdown RAC1's server using the [b][i]shutdown -h now[/i][/b] command.
Copy RAC1's folders to RAC2.
Step 2.
Using a text editor open rac1.vmx in the RAC2 folder and change the displayName = "RAC1" to displayName = "RAC2".
Step 3.
Disconnect the ethernet card connection in the VMWare control panel.
Open system and change hostname, ip addresses and profile information.
Tick the Connected checkbox in the VMWare's Network Card settings and restart the server.
Check to see if the disks are visible and that the TCP IP has been configuredcorrectly.
ping -c 2 nas
ping -c 2 rac1
ping -c 2 rac2
ping -c 2 rac1-priv
ping -c 2 rac2-priv
[root@rac2 ~]# fdisk -l
Disk /dev/sda: 64.4 GB, 64424509440 bytes
255 heads, 63 sectors/track, 7832 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 7832 62806117+ 8e Linux LVM
Disk /dev/dm-0: 60.0 GB, 60095987712 bytes
255 heads, 63 sectors/track, 7306 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-0 doesn't contain a valid partition table
Disk /dev/dm-1: 4194 MB, 4194304000 bytes
255 heads, 63 sectors/track, 509 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/dm-1 doesn't contain a valid partition table
Disk /dev/sdb: 102.3 GB, 102374572032 bytes
255 heads, 63 sectors/track, 12446 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 12446 99972463+ 83 Linux
Disk /dev/sdc: 102.3 GB, 102374572032 bytes
255 heads, 63 sectors/track, 12446 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 12446 99972463+ 83 Linux
Disk /dev/sdd: 102.3 GB, 102374572032 bytes
255 heads, 63 sectors/track, 12446 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 12446 99972463+ 83 Linux
Disk /dev/sde: 51.1 GB, 51170508800 bytes
64 heads, 32 sectors/track, 48800 cylinders
Units = cylinders of 2048 * 512 = 1048576 bytes
Disk /dev/sde doesn't contain a valid partition table
You have new mail in /var/spool/mail/root
Step 4.
In the /home/oracle.bash_profile file.
# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=rac2.sysdba.org; export ORACLE_HOSTNAME
ORACLE_UNQNAME=rac; export ORACLE_UNQNAME
ORACLE_BASE=/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/db_1; export ORACLE_HOME
ORACLE_SID=rac2; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
PATH=/usr/sbin:$PATH; export PATH
PATH=$ORACLE_HOME/bin:$PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
DNS Records can be viewed in the images below.
Before setting up SCAN (Single Client Access Name) IPs they have to respond to pings to show that they are active, this is something that is done automatically during setup.
As an Oracle user start the setup using the /setup/database/runInstaller script.
*******************************************************
Test whether the Real Application Cluster has been properly setup by using the following applications /u01/app/11.2.0/grid/bin .
srvctl config scan
srvctl config scan_listener
srvctl config database -d instance_ismi
srvctl status database -d instance_ismi
crsctl check crs
crsctl stat res -t
crsctl stat res -t init
crs_stat -t
srvctl status asm
srvctl config asm -n node_name
crsctl status serverpool
crsctl status resource
crsctl check cluster -all
crsctl query css votedisk
cluvfy comp clocksync -n all -verbose
cluvfy comp clocksync
Logs
tail -1000f /u01/11.2.0/grid/log/$HOSTNAME/ohasd/ohasd.log
tail -1000f /u01/11.2.0/grid/log/$HOSTNAME/alert$HOSTNAME.log
tail -1000f /u01/11.2.0/grid/log/$HOSTNAME/agent/ohasd/oraagent_grid/oraagent_grid.log
tail -1000f /u01/11.2.0/grid/log/$HOSTNAME/crsd/crsd.log
tail -1000f /u01/11.2.0/grid/log/$HOSTNAME/agent/ohasd/orarootagent_root/orarootagent_root.log
tail -1000f /u01/11.2.0/grid/log/$HOSTNAME/cssd/ocssd.log
tail -2000f /u01/11.2.0/grid/log/$HOSTNAME/gpnpd/gpnpd.log
cd /u01/app/oracle/diag/rdbms/rac/rac1/trace
SELECT inst_name FROM v$active_instances;
select INSTANCE_NAME from GV$ASM_CLIENT;
[root@rac2 bin]# ./crsctl status resource -w "TYPE co 'ora'" -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.LISTENER.lsnr
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.asm
ONLINE ONLINE rac1 Started
ONLINE ONLINE rac2 Started
ora.eons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.gsd
OFFLINE OFFLINE rac1
OFFLINE OFFLINE rac2
ora.net1.network
ONLINE ONLINE rac1
ONLINE ONLINE rac2
ora.ons
ONLINE ONLINE rac1
ONLINE ONLINE rac2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE rac2
ora.LISTENER_SCAN2.lsnr
1 ONLINE ONLINE rac1
ora.LISTENER_SCAN3.lsnr
1 ONLINE ONLINE rac1
ora.oc4j
1 OFFLINE OFFLINE
ora.rac.db
1 ONLINE ONLINE rac1 Open
2 ONLINE ONLINE rac2 Open
ora.rac1.vip
1 ONLINE ONLINE rac1
ora.rac2.vip
1 ONLINE ONLINE rac2
ora.scan1.vip
1 ONLINE ONLINE rac2
ora.scan2.vip
1 ONLINE ONLINE rac1
ora.scan3.vip
1 ONLINE ONLINE rac1
Connecting
Operating system: Win7 x64
Oracle client: 11.2.0.1.0
tnsnames.ora
RAC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac-scan.sysdba.org)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = rac.sysdba.org)
)
)
C:Usersmehmet>sqlplus sys/PASSWORD@rac as sysdba
The error output
SQL*Plus: Release 11.2.0.1.0 Production on Fri Jul 15 15:50:14 2011
Copyright (c) 1982, 2010, Oracle. All rights reserved.
ERROR:
ORA-12537: TNS:connection closed
Enter user-name:
ERROR:
ORA-12560: TNS:protocol adapter error
Enter user-name:
ERROR:
ORA-12560: TNS:protocol adapter error
SP2-0157: unable to CONNECT to ORACLE after 3 attempts, exiting SQL*Plus
Solution 1.
The hosts file in C:WindowsSystem32driversetc.
192.168.137.20 rac1.sysdba.org rac1
192.168.137.21 rac2.sysdba.org rac2
In the tcp/ip section of the public ethernet card properties:
advanced ->DNS-> Append these DNS suffixes
Enter the domain name, in this example its sysdba.org
C:Usersmehmet>sqlplus sys/password@rac as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Fri Jul 15 16:10:18 2011
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
Solution 2.
Upgrading to 11.2.0.2 by installing a patch.
Source: ORA-12545 or ORA-12537 While Connecting to RAC through SCAN name [ID 970619.1]
*******************************************************
RAM Settings
alter system set memory_target=20G scope=spfile sid='*';
alter system set sga_max_size=0 scope=spfile sid='*';
alter system set sga_target=0 scope=spfile sid='*';
alter system set pga_aggregate_target=0 scope=spfile sid='*';
*******************************************************
The voting disks are in the OCR_VOTE asm disk group, these will be moved to the OCR_VOT disk group.
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 116eee3911a94fd4bfdc32072f5178c5 (/dev/ASM/OCR_VOTE01) [OCR_VOTE]
2. ONLINE 2e7b89d8e3bf4ffcbf079de552f7e828 (/dev/ASM/OCR_VOTE02) [OCR_VOTE]
3. ONLINE 8e0516586f604f04bf73f2ee9be47925 (/dev/ASM/OCR_VOTE03) [OCR_VOTE]
Located 3 voting disk(s).
ASMCMD> lsdg
State Type Rebal Sector Block AU Total_MB Free_MB Req_mir_free_MB Usable_file_MB Offline_disks Voting_files Name
MOUNTED NORMAL N 512 4096 1048576 30717 23631 10239 6696 0 N DATA/
MOUNTED NORMAL N 512 4096 1048576 20478 10512 0 5256 0 N FRA/
MOUNTED NORMAL N 512 4096 1048576 3069 2916 1023 946 0 N OCR_VOT/
MOUNTED NORMAL N 512 4096 1048576 30717 29791 10239 9776 0 Y OCR_VOTE/
crsctl replace votedisk +OCR_VOT
[root@dba01 bin]# crsctl replace votedisk +OCR_VOT
Failed to create voting files on disk group OCR_VOT.
Change to configuration failed, but was successfully rolled back.
CRS-4000: Command Replace failed, or completed with errors.
There wasn't any useful information in the grid alert logs, however reviewing the asm log:
tail -1000f /u01/app/oracle/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log |more
NOTE: Creating voting files in diskgroup OCR_VOT
Wed Jul 03 10:49:13 2013
NOTE: Voting File refresh pending for group 4/0x7bd855db (OCR_VOT)
NOTE: Attempting voting file creation in diskgroup OCR_VOT
ERROR: Voting file allocation failed for group OCR_VOT
Errors in file /u01/app/oracle/diag/asm/+asm/+ASM1/trace/+ASM1_ora_11164.trc:
The error message in the trc file:
ORA-15221: ASM operation requires compatible.asm of 11.2.0.0.0 or higher
Checking the OCR_VOT version:
set linesize 100
select inst_id, name, state, type, free_mb, substr(compatibility,1,10) compatibility from gv$asm_diskgroup where name='OCR_VOT' order by name,inst_id;
INST_ID NAME STATE TYPE FREE_MB COMPATIBIL
---------- ------------------------------ ----------- ------ ---------- ----------
1 OCR_VOT MOUNTED NORMAL 2790 10.1.0.0.0
2 OCR_VOT MOUNTED NORMAL 2790 10.1.0.0.0
Its still on version 10.1.0.0.0, so needs to be updated so that its the same version as the other disk groups.
As a grid user:
ALTER DISKGROUP OCR_VOT SET ATTRIBUTE 'compatible.asm' = '11.2';
To check
select inst_id, name, state, type, free_mb, substr(compatibility,1,10) compatibility from gv$asm_diskgroup where name='OCR_VOT' order by name,inst_id;
INST_ID NAME STATE TYPE FREE_MB COMPATIBIL
---------- ------------------------------ ----------- ------ ---------- ----------
1 OCR_VOT MOUNTED NORMAL 2784 11.2.0.0.0
2 OCR_VOT MOUNTED NORMAL 2784 11.2.0.0.0
Testing again
Successful addition of voting disk 32b87db627a24f8bbfd2ce9df33a2c85.
Successful addition of voting disk 45f34d59c03b4f4dbfb924a06451f105.
Successful addition of voting disk 97d43d268c334f83bf8e757c0d869606.
Successful deletion of voting disk 116eee3911a94fd4bfdc32072f5178c5.
Successful deletion of voting disk 2e7b89d8e3bf4ffcbf079de552f7e828.
Successful deletion of voting disk 8e0516586f604f04bf73f2ee9be47925.
Successfully replaced voting disk group with +OCR_VOT.
CRS-4266: Voting file(s) successfully replaced
Checking
[root@dba01 bin]# crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 32b87db627a24f8bbfd2ce9df33a2c85 (/dev/ASM/OCR_VOTE04) [OCR_VOT]
2. ONLINE 45f34d59c03b4f4dbfb924a06451f105 (/dev/ASM/OCR_VOTE05) [OCR_VOT]
3. ONLINE 97d43d268c334f83bf8e757c0d869606 (/dev/ASM/OCR_VOTE06) [OCR_VOT]
This shows that the voting disks are now in the OCR_VOT disk group.
The OCR is present in OCR_VOTE, DATA and FRA. Lets create them in the OCR_VOT group and delete the ones from the OCR_VOTE group.
[root@dba01 bin]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3268
Available space (kbytes) : 258852
ID : 538730918
Device/File Name : +OCR_VOTE
Device/File integrity check succeeded
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File Name : +fra
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@dba01 bin]# cat /etc/oracle/ocr.loc
#Device/file getting replaced by device +fra
ocrconfig_loc =+OCR_VOTE
ocrmirrorconfig_loc =+DATA
ocrconfig_loc3 =+fra
[root@dba01 bin]# cat /etc/oracle/ocr.loc
#Device/file getting replaced by device +OCR_VOT
ocrconfig_loc =+OCR_VOTE
ocrmirrorconfig_loc =+DATA
ocrconfig_loc3 =+fra
ocrconfig_loc4 =+OCR_VOT
local_only =false
[root@dba01 bin]# cat /etc/oracle/ocr.loc
#Device/file +OCR_VOTE getting replaced by device +DATA
ocrconfig_loc =+DATA
ocrmirrorconfig_loc =+fra
ocrconfig_loc3 =+OCR_VOT
[root@dba01 ~]# ocrcheck -config
Oracle Cluster Registry configuration is :
Device/File Name : +DATA
Device/File Name : +fra
Device/File Name : +OCR_VOT
[grid@dba01 ~]$ cluvfy comp ocr -n all
Verifying OCR integrity
Checking OCR integrity...
Checking the absence of a non-clustered configuration...
All nodes free of non-clustered, local-only configurations
ASM Running check passed. ASM is running on all specified nodes
Checking OCR config file "/etc/oracle/ocr.loc"...
OCR config file "/etc/oracle/ocr.loc" check successful
Disk group for ocr location "+DATA" available on all the nodes
Disk group for ocr location "+fra" available on all the nodes
Disk group for ocr location "+OCR_VOT" available on all the nodes
NOTE:
This check does not verify the integrity of the OCR contents. Execute 'ocrcheck' as a privileged user to verify the contents of OCR.
OCR integrity check passed
Verification of OCR integrity was successful.
********************************
<tab=Services>
srvctl add service
-d my_db_name
-s my_service_name
-r my_active_instance
-a my_passive_instance
-P BASIC
-e SELECT
-y AUTOMATIC
-z 180
-w 5
Adding the service as Oracle_home's owner or "oracle"
node_2 is used as a backup in case node_1 has problems (instance name: fugu, node_1=fugu1, node_2=fugu2)
rvctl start service -d fugu -s fugu_service
To retrieve information
srvctl config service -d fugu -s fugu_service
Updating the client's tnsnames.ora
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = raca-scan.sysdba.org)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = fugu_service)
)
)
C:WindowsSystem32>sqlplus <a href="mailto:sys/oracle@fugu_ser as">sys/oracle@fugu_ser as</a> sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Paz Mar 3 16:46:55 2013
Copyright (c) 1982, 2010, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP
Data Mining and Real Application Testing options
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_file_name_convert string
db_name string fugu
db_unique_name string fugu
global_names boolean FALSE
instance_name string fugu1
lock_name_space string
log_file_name_convert string
processor_group_name string
service_names string fugu_service
Viewing the fugu service in the listener
lsnrctl status
Service "+ASM" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "fugu" has 1 instance(s).
Instance "fugu1", status READY, has 1 handler(s) for this service...
Service "fuguXDB" has 1 instance(s).
Instance "fugu1", status READY, has 1 handler(s) for this service...
Service "fugu_service" has 1 instance(s).
Instance "fugu1", status READY, has 1 handler(s) for this service...
Service "nija" has 1 instance(s).
Instance "nija1", status READY, has 1 handler(s) for this service...
Service "nijaXDB" has 1 instance(s).
Instance "nija1", status READY, has 1 handler(s) for this service...
The command completed successfully
************************
ora.dba02.vip FAILED OVER
crsctl status resource ora.dba02.vip
crsctl stop resource ora.dba02.vip
crsctl start resource ora.dba02.vip
1-OS Watcher Black Box User Guide (Doc ID 301137.1)
2-RAC and Oracle Clusterware Best Practices and Starter Kit (Platform Independent) (Doc ID 810394.1)
3-RACcheck - RAC Configuration Audit Tool [ID 1268927.1]
4-Top 11 Things to do NOW to Stabilize your RAC Cluster Environment [ID 1344678.1] To BottomTo Bottom
5-RAC and Oracle Clusterware Best Practices and Starter Kit (Solaris) (Doc ID 811280.1)
6-Grid Infrastructure Redundant Interconnect and ora.cluster_interconnect.haip (Doc ID 1210883.1)
7-The Basics of IPv4 Subnet and Oracle Clusterware (Doc ID 1386709.1)
8-Frequently Node Reboots, There Is Nothing In The Logs (Doc ID 437420.1)
9-CRS Diagnostic Data Gathering: A Summary of Common tools and their Usage (Doc ID 783456.1)
***********************************************************************************
5 Comments
BernadineStu
I see your site needs some fresh & unique content. Writing manually is time
consuming, there is tool for this task. Just search in gogle
for: Fejlando’s tips
Mehmet Dervişoğlu
Thanks a lot when I have time I will try
oil Change duarte
Hello I am so happy I found your site, I really found you by error, while I
was researching on Google for something else, Anyhow I am here now and would just like to
say many thanks for a incredible post and a all round exciting blog (I also love the
theme/design), I don’t have time to browse it all at the minute but
I have book-marked it and also included your RSS feeds, so
when I have time I will be back to read much more, Please do keep up the fantastic jo.
Mehmet Dervişoğlu
I am glad it helped you 🙂