客户的一套11.2.0.1 RAC系统采用ASM diskgroup 存放ocr和votedisk，该REG diskgroup中的某个LUN disk由于硬件的原因损坏了，导致冗余的votedisk表决磁盘有一个处于OFFLINE状态，客户希望能删除该OFFLINE的votedisk并新增一个可用的。 在删除该votedisk文件时出现了CRS-4258的错误，错误如下:
crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- --------- 1. OFFLINE 5b3380d6367e4f94bf19e9db5f2f684e () [] 2. ONLINE 6802e6d139354fb3bf95725dd01a02fd (/dev/ocr2) [REG] 3. ONLINE a433d51ebd2d4facbfc8e95b017f5393 (/dev/asm-disk1) [REG] 4. ONLINE 3784d344bffa4f6ebff21c4dd3c873bd (/dev/asm-disk2) [REG]Located 4 voting disk(s).crsctl delete css votedisk 5b3380d6367e4f94bf19e9db5f2f684eCRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASM
居然无法移除ASM存储上的voting files，太搞笑了。 客户在MOS上找到了CRS-4258相关问题的Note:
CRS-4258: Addition and Deletion of Voting Files are not Allowed Because the Voting Files are on ASM in 11gR2 [ID 1060146.1]Applies to:Oracle Server - Enterprise Edition - Version: 11.2.0.1 to 11.2.0.1 - Release: 11.2 to 11.2Information in this document applies to any platform.SymptomsCRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASM in 11gR2.ChangesStale voting files are seen after accidently dropping one of ASM disks belonging to the ASM diskgroup where voting files are stored.And CRS-4258 occurs when trying to delete the stale voting files using crsctl delete css votedisk FUID.[root@grid]# crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 5b91aad0a2184f3dbfa8f970e8ae4d49 (/dev/oracleasm/disks/ASM10) [PLAY]2. ONLINE 53b1b40b73164f9ebf3f498f6d460187 (/dev/oracleasm/disks/ASM9) [PLAY]3. OFFLINE 82dfd04b96f14f6dbf36f5a62b118f61 () [][root@grid]# crsctl delete css votedisk 82dfd04b96f14f6dbf36f5a62b118f61CRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASMCause1. Seeing stale voting files is due to bug 9024611.2. "delete" command is not available , only "replace" command is available when voting files are stored on ASM diskgroup. Please see Oracle Clusterware Administration and Deployment Guide11g Release 2 (11.2)Solution1. This issue is permanently fixed in 11.2.0.2.0.2. Apply patch 9024611. Please contact Oracle support if this patch is not available on your platform.3. If CSS has stale voting files even after applying patch 9024611, do the following workaround -WORKAROUND:Do something to trigger ASM to try to relocate the voting file.e.g) $ crsctl replace votedisk +asm_disk_group --- Put available ASM diskgroup $ crsctl query css votedisk --- Check if voting files are all online on the new ASM diskgroup $ crsctl replace votedisk +PLAY -- Put the original ASM diskgroup where voting files were 4. If the workaround above cannot be followed for any reason then you can request the patch for unpublished bug 9409327 for your platform.ReferencesBUG:9294664 - NOT ABLE TO REMOVE THE VOTEDISK WHICH IS OFFILNEHdr: 9294664 11.2.0.1 PCW 11.2.0.1 ADMUTL PRODID-5 PORTID-226 9024611Abstract: NOT ABLE TO REMOVE THE VOTEDISK WHICH IS OFFILNEPROBLEM:--------crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 9f7f4f7f798d4f69bfe31653894421a2 (ORCL:GRID1) [GRID]2. OFFLINE a9b785a59c3c4f67bf15babc67ffb79a () []3. OFFLINE 29988f37fa794f12bfea3f3672c99609 () []4. ONLINE a8b3a040195c4f54bfce8ef21bd4fa07 (ORCL:GRID3) [GRID]5. ONLINE a1e4fbd9df6f4f67bf8fc12fe9780721 (ORCL:GRID2) [GRID]Located 5 voting disk(s).[root@sdc-drrac01 grid]# crsctl delete css votedisk a9b785a59c3c4f67bf15babc67ffb79aCRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASMDIAGNOSTIC ANALYSIS:--------------------Ct is performing some voting disk failover scenarios in which he has removed the 2 votedisk which were on ASM buy drop disk using asmlib and after that recreating the disk again and start the cluster in exclusive mode and start the ASM and mount the diskgourp So that rebalancing has been done but after that crsctl query css votedisk## STATE File Universal Id File Name Disk group-- ----- ----------------- --------- ---------1. ONLINE 9f7f4f7f798d4f69bfe31653894421a2 (ORCL:GRID1) [GRID]2. OFFLINE a9b785a59c3c4f67bf15babc67ffb79a () []3. OFFLINE 29988f37fa794f12bfea3f3672c99609 () []4. ONLINE a8b3a040195c4f54bfce8ef21bd4fa07 (ORCL:GRID3) [GRID]5. ONLINE a1e4fbd9df6f4f67bf8fc12fe9780721 (ORCL:GRID2) [GRID]Located 5 voting disk(s).and not able to drop the vote disk which is offiline WORKAROUND:-----------n/aRELATED BUGS:-------------as per bug 9024611 tried the workaround:but while running crsctl css votedisk delete we got syntax error and found that there is no command with crsctl css ...
这个Note说明不能移除ASM存储内voting files的问题在11.2.0.2.0上已经解决了，也可以通过安装one-off patch 9024611来修复。 但是实际在11.2.0.2上测试可以发现仍旧无法删除ASM上的voting files:
root@rh2 ~]# crsctl query crs releaseversionOracle High Availability Services release version on the local node is [11.2.0.2.0][root@rh2 ~]# crsctl query crs activeversionOracle Clusterware active version on the cluster is [11.2.0.2.0][grid@rh2 ~]$ /s01/grid/OPatch/opatch lsinventoryInvoking OPatch 11.2.0.1.1Oracle Interim Patch Installer version 11.2.0.1.1Copyright (c) 2009, Oracle Corporation. All rights reserved.Oracle Home : /s01/gridCentral Inventory : /s01/app/oraInventory from : /etc/oraInst.locOPatch version : 11.2.0.1.1OUI version : 11.2.0.2.0OUI location : /s01/grid/ouiLog file location : /s01/grid/cfgtoollogs/opatch/opatch2011-08-04_18-50-34PM.logPatch history file: /s01/grid/cfgtoollogs/opatch/opatch_history.txtLsinventory Output file location : /s01/grid/cfgtoollogs/opatch/lsinv/lsinventory2011-08-04_18-50-34PM.txt--------------------------------------------------------------------------------Installed Top-level Products (1): Oracle Grid Infrastructure 11.2.0.2.0There are 1 products installed in this Oracle Home.There are no Interim patches installed in this Oracle Home.Rac system comprising of multiple nodes Local node = rh2 Remote node = rh3--------------------------------------------------------------------------------OPatch succeeded.[root@rh2 ~]# crsctl delete css votedisk a433d51ebd2d4facbfc8e95b017f5393CRS-4258: Addition and deletion of voting files are not allowed because the voting files are on ASM
又是一个伪修复的Bug....!! 无法,寄希望与replace能解决问题，结果发现：
crsctl replace votedisk +DATAFailed to create voting files on disk group DATA.Change to configuration failed, but was successfully rolled back.CRS-4000: Command Replace failed, or completed with errors.
方法四中指出的unpublished bug 9409327(Patch 9409327: OFFLINE VF ENTRY REMAINS AFTER PATCH FOR BUG 9024611)，目前仅在IBM AIX on POWER Systems (64-bit)的11.2.0.1上有对应的补丁。