Archive for April, 2010

MAA challenge lab, part #4

Friday, April 30th, 2010

No posts on my blog for long time, need to change that. So i was trying to get MAA (Maximum Availability Architecture by Oracle) lab again in shape for writing Master of Science thesis…

Somewhere near Janurary/Februrary this year:

  • Primary VM RAC nodes prac1, prac2, prac3 are working again, but database db3 is not (unable to archive to db3dg on srac1,srac2). Main root cause was that experiments in April of 2009 with log_archive_min_succeed_dest=2 setting caused losing sync betweeen primary and standby
  • Standby VM RAC nodes srac1, srac2 VMs are up again, but without db3dg database and ASM instances

So some actions were performed (Feburary/March):

  • ASM +DATA1 recreated on both new srac1 & srac2 VMs (on dedicated VG/LV: vg01/db3dg on physical synapse server, on dedicated 15k RPM SCSI drive[4th in system] named sdb [sda is RAID5 for OS/XEN/VMs]).
  • Fixed some XEN configuration files for VMs with “w!” (disabled shared locks mode) & losetup bug in /etc/xen/scripts/block.
  • Standby rebuilded (from RMAN duplicate backup + FAL gap fetching), so finally i’ve working MAA again :)

Problematic thing is that after failover/switchover due to differences in primary (64-bit) and standby (32-bit) i have to invalidate & recompile all PL/SQL packages (very time consuming on old hardware! and seems that Broker is unable to handle that case):

So last week:

  • prac7, prac8 32-bit VMs were launched.
  • ‘ve switched to using FreeNX server from typical X11 SSH forwarding (much more interactive on slow OpenVPN!).
  • Also i’ve upgraded Grid Control (OMS) to 10.2.0.5 (next step will be to 11g), OMS database repository to 11.1.0.6 (fro 10.1.0.4).
  • Next i deployed 32-bit clusterware (11.1) on those notes, played a little bit with OCR corruptions [metalink note ID 399482.1] after hitting mysterious listener outages (OCR corruptions were not the case for it, it was permission issue on single directory – doh!)
  • Created clustered ASM, and created 32-bit RAC database named “db4.lab1″.
  • I’ve also deployed OEM/GC agents on prac7, prac8 (10.2.0.5 versions) directly from GC (awesome for mass deployments!).
  • Yesterday I’ve exported using expdp 1 schema from db3.lab1 database (64-bit RAC) and imported it into “db4.lab1″…

Planned soon:

  • Upgrading GC to 11g and upgrading agents to 11g too.
  • Building MAA for “db4.lab1″, plan is to create DataGuard for it on srac1, srac2 VMs (32-biit too, they already host DataGuard for “db3.lab1″). But this one is going to use DataGuard Broker to get FailStart Failovers working (2 node primary RAC with 2 node standby RAC)
  • Extending primary RAC to prac9 (to be created), so to have 3 node primary RAC for “db4.lab1″ protected by DataGuard broker with 2 node standby RAC
  • Fun with Snapshot standby and Real Application Testing
  • Detailed measurements of failover times with SYNC DataGuard and Broker – orajdbcstat (my little utility for testing JDBC/FCF failover times, perhaps i’ll uplift it for UCP from 11.2g)