Archive for the ‘NetApp’ Category

WAFL performance VS sequential reads: part III, FC LUN performance from AIX vs read_realloc

Thursday, July 7th, 2011

Continuing my series about LUN fragmentation in WALF (part1, part2) I wanted to give a try to read_realloc option. Mostly the same storage system (still DataOnTap 7.3.2) and settings were used with an exception that i’ve even used more powerful AIX with POWER7 CPU cores system (shouldn’t matter as i was completely not constrained with CPU power during the benchmarks).

The methodology was simple:
1) Do a lot of random and sequential writes to JFS2 on top of that LUN on FlexVol having read_realloc=off to fragment the LUN as much as possible
2) Start a loop of random large reads (simulating let’s say many random full table scans of up to 1MB in Oracle data file) on top of that JFS2 filesystem (BTW: this time without CIO)

The full measurement took something about 18 hours (yes, storage was hammered 18 hours all the time with just reads). Results are pretty interesting:

… it basically shows how Netapp storage can self-optimize and adapt based on the workload. Now i’m just wondering how it works under the hood (i.e. what is the trade-off or worst possible scenario for this kind of setting). What was even more interesting is that reallocate measure indicated after the whole cycle that the LUN was really nicely de-fragmented:

Thu Jul  7 03:32:20 EDT [X: wafl.reallocate.check.value:info]: Allocation measurement check on '/vol/fc_fill_aggr_sm/lun' is 3.

Posted In: AIX, NetApp, Performance, SAN | 1 Comment »

WAFL performance VS sequential reads: part II, FC LUN defragmentation

Monday, June 27th, 2011

After partI – where i’ve been simulating typical Oracle workload (generating 70:30 read to write ratio on FC LUN) and creating snapshots – i’ve wanted to try different performance tests. In order to achieve the same performance characteristics, so i’ve deleted all my snapshots, so my FlexVol ended up again in 40% utilization:

X> snap list full_aggr_test
Volume full_aggr_test
working...

No snapshots exist.
X>
X> df -g full_aggr_test
Filesystem               total       used      avail capacity  Mounted on
/vol/full_aggr_test/       50GB       20GB       29GB      40%  /vol/full_aggr_test/
/vol/full_aggr_test/.snapshot        0GB        0GB        0GB     ---%  /vol/full_aggr_test/.snapshot
X>

Later i’ve executed Orion stress test, in a identical way like in partI on the same enviorniment. As you can see still the LUN is fragmented because any kind of sequential read is going to be impacted (maximum read observed ~17MB/s):

root@Y:# grep Maximum orion*
orion_20110627_1116_summary.txt:Maximum Large MBPS=17.07 @ Small=0 and Large=9
orion_20110627_1116_summary.txt:Maximum Small IOPS=683 @ Small=24 and Large=0
root@Y:#

So in order to fight with this performance issue one can establish the root cause:

X> reallocate measure /vol/full_aggr_test
Reallocation scan will be started on '/vol/full_aggr_test'.
Monitor the system log for results.
X>

System log will reveal this:

Mon Jun 27 07:35:31 EDT [X: wafl.scan.start:info]: Starting WAFL layout measurement on volume full_aggr_test.
Mon Jun 27 07:35:32 EDT [X: wafl.reallocate.check.highAdvise:info]: Allocation check on '/vol/full_aggr_test' is 8, hotspot 0 (threshold 4), consider running reallocate.

This seems to be identical to running measure on the LUN:

X> reallocate measure  /vol/full_aggr_test/lun01
Reallocation scan will be started on '/vol/full_aggr_test/lun01'.
Monitor the system log for results.
X>

Log will show this:

Mon Jun 27 07:45:21 EDT [X: wafl.scan.start:info]: Starting WAFL layout measurement on volume full_aggr_test.
Mon Jun 27 07:45:21 EDT [X: wafl.reallocate.check.highAdvise:info]: Allocation check on '/vol/full_aggr_test/lun01' is 8, hotspot 0 (threshold 4), consider running reallocate.

So in both cases we were recommended to defragment the LUN, but keep in mind that this is a rather resource hungry operation, as it might involve reading and rewriting the full contents of the data!

X> reallocate start -f -p /vol/full_aggr_test/lun01
Reallocation scan will be started on '/vol/full_aggr_test/lun01'.
Monitor the system log for results.
X>

Log will show that the operation has started …

Mon Jun 27 07:46:23 EDT [X: wafl.br.revert.slow:info]: The aggregate 'sm_aggr1' contains blocks that require redirection; 'revert_to' might take longer than expected.
Mon Jun 27 07:46:23 EDT [X: wafl.scan.start:info]: Starting file reallocating on volume full_aggr_test.

As you can see it is rather low CPU activity however , physical utilization of the disks is reported as high (don’t be fooled by low write activity – this is function of time, it does perform a lot of writes later):

 CPU   NFS  CIFS  HTTP   Total    Net kB/s   Disk kB/s     Tape kB/s Cache Cache  CP   CP Disk    FCP iSCSI   FCP  kB/s iSCSI  kB/s
                                  in   out   read  write  read write   age   hit time  ty util                 in   out    in   out
 10%     0     0     0     157     0     0  22372  19320     0     0    53s  94%  58%  :   97%    156     0   589   175     0     0
 10%     1     0     0     108     0     0  24884      0     0     0    53s  94%   0%  -   92%    106     0   256   585     0     0
  9%     0     0     0     101     0     0  25284     24     0     0    53s  94%   0%  -   93%    100     0   421   260     0     0
 12%     0     0     0     627    20    25  25620      8     0     0    53s  94%   0%  -   92%    511     0   297   132     0     0
 11%     0     0     0     792     0     0  22832      0     0     0    53s  94%   0%  -   90%    652     0   670   461     0     0
  6%     1     0     0      81     1     1  25232     24     0     0    53s  99%   0%  -   92%     78     0   233   253     0     0

One can monitor the progress by using “status” command and in fact observe

X> reallocate status -v /vol/full_aggr_test/lun01
Reallocation scans are on
/vol/full_aggr_test/lun01:
        State: Reallocating: Block 1347456 of 5242880 (25%), updated 1346434
        Flags: doing_force,measure_only,repeat,keep_vvbn
    Threshold: 4
     Schedule: n/a
     Interval: 1 day
 Optimization: 8
  Measure Log: n/a
X>
[..]
X> reallocate status -v /vol/full_aggr_test/lun01
Reallocation scans are on
/vol/full_aggr_test/lun01:
        State: Idle
        Flags: measure_only,repeat
    Threshold: 4
     Schedule: n/a
     Interval: 1 day
 Optimization: 8
  Measure Log: n/a

X> sysstat -x 1
 CPU   NFS  CIFS  HTTP   Total    Net kB/s   Disk kB/s     Tape kB/s Cache Cache  CP   CP Disk    FCP iSCSI   FCP  kB/s iSCSI  kB/s
                                  in   out   read  write  read write   age   hit time  ty util                 in   out    in   out
 53%     1     0     0     678     1     1  29428   1556     0     0     1   72%   9%  :   11%    573     0   311 21077     0     0
 34%     0     0     0     443     0     0  22028     32     0     0     1   78%   0%  -    5%    442     0  1068 20121     0     0
 40%     0     0     0     172     0     0  16360      0     0     0     1   77%   0%  -    4%    171     0   367 14450     0     0
 CTRL+C
X>

Later results indicate that indeed sequential reads are back to their top value (~42MB/s) and this was our starting point on fresh FlexVol inside LUN in partI…

root@Y:# grep Maximum orion*
orion_20110627_1208_summary.txt:Maximum Large MBPS=42.73 @ Small=0 and Large=9
orion_20110627_1208_summary.txt:Maximum Small IOPS=645 @ Small=25 and Large=0
root@Y:#

In the next series i’ll try to investiage the various AIX JFS2/CIO behaviours and to some degree the performance characteristics of Netapp storage and it’s options (e.g. read_realloc option). Stay tuned…

Posted In: NetApp, Oracle, SAN | 2 Comments »

WAFL performance VS sequential reads: part I, FC LUN performance from AIX vs FlexVol utilization

Friday, June 24th, 2011

Some time ago i’ve started thinking about getting more serious about performance research on AIX6.1, PowerHA and Oracle stack on top of Netapp storage. One of the first things that i wanted to measure was how the Netapp’s WALF handles FlexVolume utilization in correlation to space usage. In theory long sequential reads (as like in Oracle Datawarehouses in which i’m interested) could be affected because of the fragmentation introduced by WAFL (Write *Anywhere* File System).

Some specs first:

  • DataONTap version 7.3.2, single Netapp controller 3160 was tested (but in cluster).
  • The test was done using Oracle’s Orion 11.1.0.7 storage benchmarking tool on top of AIX (orion_aix_ppc64 -run advanced -num_disks 5 -cache_size 2048 -write 30 -duration 20 -type rand -matrix basic) – as you can see the read vs write ratio was 70% to 30%, but only long reads were presented (i was not interested in the performance of 8kB reads/writes, just 1MB long reads)
  • AIX was connected via VFCs to two separate VIOS, which each were connected using 2 FC 8 GBps links each (but running in 4 GBps mode due to the Brocade SAN switches not supporting 8 GBps)
  • Netapp controller was having 4 GBps ports
  • AIX was using 6.1′s internal MPIO (round-robin) for the tested LUN
  • AIX hdisk for LUN was set to the default value of 12 (as per Netapp Host Attachement Kit)
  • AIX JFS2 filesystem was mounted with Concurrent I/O to prevent AIX from caching and read-aheads (still AIX had 3GB of RAM allocated but VMM should not use it)
  • Netapp storage controller was having 4 processors, 8GB RAM and 2GB for NVRAM (as indicated by sysconfig output, of course as this is cluster, only 1GB was available)
  • LUN size was 20GB, on top of 50GB FlexVol on RAID-DP aggregate with 5x FC 15k RPM disks
X> vol options full_aggr_test
nosnap=off, nosnapdir=off, minra=off, no_atime_update=off, nvfail=off,
ignore_inconsistent=off, snapmirrored=off, create_ucode=on,
convert_ucode=off, maxdirsize=83804, schedsnapname=ordinal,
fs_size_fixed=off, compression=off, guarantee=volume, svo_enable=off,
svo_checksum=off, svo_allow_rman=off, svo_reject_errors=off,
no_i2p=off, fractional_reserve=100, extent=off, try_first=volume_grow,
read_realloc=off, snapshot_clone_dependency=off
X> df -Ag aggr_used
Aggregate                total       used      avail capacity
aggr_used               1102GB      668GB      433GB      61%
aggr_used/.snapshot         0GB        0GB        0GB     ---%
X>
X> snap list -A aggr_used
Aggregate aggr_used
working...

No snapshots exist.
X> snap sched -A aggr_used
Aggregate aggr_used: 0 0 0
X>

WALF aggregate was idle during that test and configured and running as follows:

Aggregate aggr_used (online, raid_dp) (block checksums)
  Plex /aggr_used/plex0 (online, normal, active)
    RAID group /aggr_used/plex0/rg0 (normal)

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------
      dparity   2d.18   2d    1   2   FC:B   -  FCAL 15000 418000/856064000  420156/860480768
      parity    1d.19   1d    1   3   FC:A   -  FCAL 15000 418000/856064000  420156/860480768
      data      2d.21   2d    1   5   FC:B   -  FCAL 15000 418000/856064000  420156/860480768
      data      1d.22   1d    1   6   FC:A   -  FCAL 15000 418000/856064000  420156/860480768
      data      2d.23   2d    1   7   FC:B   -  FCAL 15000 418000/856064000  420156/860480768

[..]

X> aggr status aggr_used -v
           Aggr State           Status            Options
       aggr_used online          raid_dp, aggr     nosnap=off, raidtype=raid_dp,
                                                  raidsize=16,
                                                  ignore_inconsistent=off,
                                                  snapmirrored=off,
                                                  resyncsnaptime=60,
                                                  fs_size_fixed=off,
                                                  snapshot_autodelete=on,
                                                  lost_write_protect=on
[..]

X> df -g full_aggr_test
Filesystem               total       used      avail capacity  Mounted on
/vol/full_aggr_test/       50GB       20GB       29GB      40%  /vol/full_aggr_test/
/vol/full_aggr_test/.snapshot        0GB        0GB        0GB     ---%  /vol/full_aggr_test/.snapshot
X>
X> snap list full_aggr_test
Volume full_aggr_test
working...

No snapshots exist.
X>

From AIX point of view, filesystem was configured as follows (notice those big files for Orion’s use):

root@Y:# df -m .
Filesystem    MB blocks      Free %Used    Iused %Iused Mounted on
/dev/fslv00    19456.00   2099.70   90%        7     1% /fullaggr
root@Y:# du -sm *
10000.01        bigfile
7353.00 bigfile2
0.00    lost+found
0.00    orion.lun
root@Y:#

Results:

Methodology:
For each next attempt a snapshot was created to grow the space used inside the FlexVol until it was full (40%..100%). There was single Orion execution after each snapshot was created.The Y-axis represents maximum bandwidth observed for sequential 1MB reads (as reported by Orion). The Z-axis (depth) ranging from 1 to 10 reprents the number of concurrent/paralell reads being done by Orion (to let’s say simulate multiple full table scans happening on the same LUN). As it is visible from the graph when FlexVol utilization is close or equal to 100% a nearly more than double performance impact can be observed (40-45 MB/s vs 10-15MB/s). The sane FlexVol utilization minimium seems to somewhere max at 70% to avoid problems with fragmentation. AIX system was mostly coming with default settings without any more advanced optimizations (that was done on purpose except Concurrent I/O).

Posted In: AIX, NetApp, Performance, SAN | 5 Comments »

PowerHA failure scenario when dealing with SAN-booted LPARs – part I

Sunday, January 9th, 2011

Together with Jedrzej we’ve exposed rather interesting weaknees in IBM PowerHA 5.5 solution (in the old days it was called HACMP). Normally you would assume that in case major cataclysm such as *complete* storage disappear on the active node, PowerHA or AIX has internal mechanism to prevent downtime by switching the services to the next active node (as defined in PowerHA policies/confguration). This is starting to be really interesting when we start talking about SAN BOOT-ed AIX LPARs. As everybody knows any form of assumption is bad (this is starting to be my mantra), and as we show it here avoid this pitfall requires requires SOME PLANNING AND CONFIGURATION to avoid ending in long downtime….

The enviorniment was consisting of:

  • Two LPARs (jkwha001d = active; jkwha002d = passive) running on separate Power frames.
  • Each LPAR on each frame was protected by separate pair of VIOS (to provide redundancy for NPIV-enabled FC fabrics and enable to have Shared Ethernet Failovers).
  • AIX 6.1 TL3
  • Oracle 10gR2
  • PowerHA 5.5SP2
  • Netapp DataOntap 7.3 storage (Netapp cluster, 2 controllers) – yes, AIX LPARs are also SAN BOOT-ed from this storage
  • 1 network-based (IPv4) heartbeat
  • 1 disk-based (on FC LUN) heartbeat
  • Properly configured & running scripts for starting and stopping Oracle
  • No application monitors configured

We’ve performed two experiments:

  1. Simulating total storage losing only on jkwha001d (active one, hosting Oracle) by unmapping Virtual Fabre Channel (NPIV) host adapters (vadapters) on both VIOS protecting jkwha001d.
  2. Simulating total storage losing only on jkwha002d (active one, hosting Oracle — after failover) by forcing disconnection of jkwha002d Fibre Channel initiators (WWNs) on Netapp cluster.

both of which simulated the same thing… Virtual LPAR running without storage at all. More on result in next post…

-Jakub.

Posted In: AIX, cluster, NetApp, PowerHA | Comments Off

Testing NetApp’s SyncMirror whole plex failure (on simulator)

Thursday, May 20th, 2010

Together with Lukasz Borek we’ve tested the SyncMirror reliability under some stress…. In short basic NetApp deployment just uses RAID-DP (Double Parity, so you are might 2 disks fail before you loose data). If your data is critical, they you can actually configure SyncMirror which just mirrors aggregate (on top of which are volumes) to two plexes, each consisting of RAID group(s). This behaves like RAID-1 which is using two RAID-DP (you have double ammount of disks protecting you). Simple demonstration (just before this we’ve removed all spare disks for demonstration purposes)

So we have our aggregate aggr_mirr:

filerA> aggr status -r
Aggregate aggr_mirr (online, raid_dp, mirrored) (block checksums)
  Plex /aggr_mirr/plex0 (online, normal, active, pool0)
    RAID group /aggr_mirr/plex0/rg0 (normal)

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------
      dparity   v5.19   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
      parity    v5.20   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
      data      v5.24   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448

  Plex /aggr_mirr/plex1 (online, normal, active, pool0)
    RAID group /aggr_mirr/plex1/rg0 (normal)

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------
      dparity   v5.25   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
      parity    v5.26   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
      data      v5.27   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448

[..]

Pool1 spare disks (empty)

Pool0 spare disks (empty)

Broken disks

RAID Disk       Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
---------       ------  ------------- ---- ---- ---- ----- --------------    --------------
admin removed   v4.16   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.17   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.20   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.21   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.22   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.24   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.25   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.26   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.27   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.28   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.29   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v4.32   v4    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v5.28   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v5.29   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
admin removed   v5.32   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448

We want to fail all of the disks (5.19, 5.20, 5.24) that are part of /aggr_mirr/plex0/rg0 (that simulates loosing whole plex, we wanted to test that aggr_mirr is going to be able survive crash of one of the plexes):

filerA> disk fail v5.19
*** You are about to prefail the following file system disk, ***
*** which will eventually result in it being failed ***
  Disk /aggr_mirr/plex0/rg0/v5.19

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------
      dparity   v5.19   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
***
Really prefail disk v5.19?  y

WARNING! There is no spare disk available to which to copy.
Are you sure you want to continue with disk fail (y/n)? y
disk fail: The following disk was prefailed: v5.19
Disk v5.19 has been prefailed.  Its contents will be copied to a
replacement disk, and the prefailed disk will be failed out.
filerA>
Tue Apr 27 18:03:52 GMT [raid.rg.diskcopy.cant.start:warning]: /aggr_mirr/plex0/rg0: unable to start disk copy for v5.19: No block checksum disk of required type and size is available, targeting Pool0
filerA>
filerA> disk fail v5.20
*** You are about to prefail the following file system disk, ***
*** which will eventually result in it being failed ***
  Disk /aggr_mirr/plex0/rg0/v5.20

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------
      parity    v5.20   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
***
Really prefail disk v5.20? yes

WARNING! There is no spare disk available to which to copy.
Are you sure you want to continue with disk fail (y/n)? y
disk fail: The following disk was prefailed: v5.20
Disk v5.20 has been prefailed.  Its contents will be copied to a
replacement disk, and the prefailed disk will be failed out.
filerA>
filerA>
filerA> disk fail v5.24
*** You are about to prefail the following file system disk, ***
*** which will eventually result in it being failed ***
  Disk /aggr_mirr/plex0/rg0/v5.24

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------
      data      v5.24   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
***
Really prefail disk v5.24? y

WARNING! There is no spare disk available to which to copy.
Are you sure you want to continue with disk fail (y/n)? y
disk fail: The following disk was prefailed: v5.24
Disk v5.24 has been prefailed.  Its contents will be copied to a
replacement disk, and the prefailed disk will be failed out.
filerA>
filerA> disk simpull  v5.19
Tue Apr 27 18:06:08 GMT [raid.config.filesystem.disk.missing:info]: File system Disk /aggr_mirr/plex0/rg0/v5.19 Shelf ? Bay ? [NETAPP   VD-1000MB-FZ-520 0042] S/N [16402503] is missing.
filerA>
Tue Apr 27 18:06:08 GMT [raid.rg.recons.missing:notice]: RAID group /aggr_mirr/plex0/rg0 is missing 1 disk(s).
Tue Apr 27 18:06:08 GMT [raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr_mirr/plex0/rg0: No block checksum disk of required type and size is available, targeting Pool1
filerA>
filerA> disk simpull  v5.20
Tue Apr 27 18:06:16 GMT [raid.disk.missing:info]: Disk /aggr_mirr/plex0/rg0/v5.20 Shelf ? Bay ? [NETAPP   VD-1000MB-FZ-520 0042] S/N [16402504] is missing from the system
Tue Apr 27 18:06:16 GMT [raid.config.filesystem.disk.missing:info]: File system Disk /aggr_mirr/plex0/rg0/v5.20 Shelf ? Bay ? [NETAPP   VD-1000MB-FZ-520 0042] S/N [16402504] is missing.
filerA>
Tue Apr 27 18:06:17 GMT [raid.rg.recons.missing:notice]: RAID group /aggr_mirr/plex0/rg0 is missing 2 disk(s).
Tue Apr 27 18:06:17 GMT [raid.rg.recons.cantStart:warning]: The reconstruction cannot start in RAID group /aggr_mirr/plex0/rg0: No block checksum disk of required type and size is available, targeting Pool1
filerA>
filerA> disk simpull  v5.24
Tue Apr 27 18:06:20 GMT [raid.config.filesystem.disk.missing:info]: File system Disk /aggr_mirr/plex0/rg0/v5.24 Shelf ? Bay ? [NETAPP   VD-1000MB-FZ-520 0042] S/N [16402507] is missing.
Tue Apr 27 18:06:20 GMT [raid.vol.mirror.degraded:error]: Aggregate aggr_mirr is mirrored and one plex has failed. It is no longer protected by mirroring.

So we’ve destroyed whole plex! Great success! ;) In real world this would be more like whole shelf failure. NetApp controller of course tried also to call home, because this is pretty serious:

Tue Apr 27 18:06:20 GMT [callhome.syncm.plex:CRITICAL]: Call home for SYNCMIRROR PLEX FAILED

Let’s verify again:

filerA> aggr status -r
Aggregate aggr_mirr (online, raid_dp, mirror degraded) (block checksums)
  Plex /aggr_mirr/plex0 (offline, failed, inactive, pool0)
    RAID group /aggr_mirr/plex0/rg0 (partial)

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------
      dparity   FAILED          N/A                        1020/2089984
      parity    FAILED          N/A                        1020/2089984
      data      FAILED          N/A                        1020/2089984
      Raid group is missing 3 disks.

  Plex /aggr_mirr/plex1 (online, normal, active, pool0)
    RAID group /aggr_mirr/plex1/rg0 (normal)

      RAID Disk Device  HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)
      --------- ------  ------------- ---- ---- ---- ----- --------------    --------------
      dparity   v1.25   v1    ?   ?   FC:A   0  FCAL  N/A  1020/2089984      1027/2104448
      parity    v5.26   v5    ?   ?   FC:B   0  FCAL  N/A  1020/2089984      1027/2104448
      data      v1.27   v1    ?   ?   FC:A   0  FCAL  N/A  1020/2089984      1027/2104448

[..]

During the whole PLEX destruction process, we were continually writing to NFS volume on this aggr_mirr, in synchronus mode (to avoid caching at OS level and to always hit the [NV]RAM – of course no NVRAM in simulator….), as you can see from below output, no single I/O error at all from NFS client perspective:

2097152 bytes (2.1 MB) copied, 0.498529 seconds, 4.2 MB/s
Thu Apr 29 16:10:32 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.51502 seconds, 4.1 MB/s
Thu Apr 29 16:10:33 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.514629 seconds, 4.1 MB/s
Thu Apr 29 16:10:35 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.419478 seconds, 5.0 MB/s
Thu Apr 29 16:10:36 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.411578 seconds, 5.1 MB/s
Thu Apr 29 16:10:38 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.444393 seconds, 4.7 MB/s
Thu Apr 29 16:10:39 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.43042 seconds, 4.9 MB/s
Thu Apr 29 16:10:41 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.417388 seconds, 5.0 MB/s
Thu Apr 29 16:10:42 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.504876 seconds, 4.2 MB/s
Thu Apr 29 16:10:44 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.425559 seconds, 4.9 MB/s
Thu Apr 29 16:10:45 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.453916 seconds, 4.6 MB/s
Thu Apr 29 16:10:46 CEST 2010
2+0 records in
2+0 records out
2097152 bytes (2.1 MB) copied, 0.417439 seconds, 5.0 MB/s

Posted In: NetApp, SAN | 2 Comments »