Oracle 19c CRS重启案例记录

对文件系统进行较大写入操作(30g文件写入),导致RAC一个节点被驱逐重启。

MOS info:
Bug 30472602 CRS: oracssdagent is rebooting this node because the file system did not respond

This note gives a brief overview of bug 30472602.
The content was last updated on: 15-JAN-2020
Click here for details of each of the sections below.
Affects:

Product (Component) Oracle Server (PCW)
Range of versions believed to be affected Versions BELOW 20.1
Versions confirmed as being affected
19.4.0
Platforms affected Generic (all / most platforms affected)
Fixed:

The fix for 30472602 is first included in
20.1.0
19.6.0.0.200114 (Jan 2020) OCW Release Update Revision(OCW RU)

Interim patches may be available for earlier versions - click here to check.
Symptoms:

Related To:

Error May Occur
CRS-8011 / CRS-8013
Cluster Ready Services / Parallel Server Management
Description

OCSSD agent reboots the node after seeing that CRS Home file system is not accessible. Some FS operation (like cp) may cause the slowness in accessing the FS.
After applying the fix, it will use directIO instead for Home file system monitoring.

Rediscovery Information:

The OCSSD agent reboots the node with Home file System not accessible.

2019-11-02 11:09:23.810 [OHASD(52413)]CRS-8011: reboot advisory message from host: cfcldx0016, component: cssagent, with time stamp: L-2019-11-02-10:57:43.659
2019-11-02 11:09:23.811 [OHASD(52413)]CRS-8013: reboot advisory message text: oracssdagent is rebooting this node because the file system for path /u01/app/19.4.0.0/grid_fadbbp190716/log/cfcldx0016 did not respond in 60290 milliseconds.

Workaround

Here is the procedure to disable file system monitoring.

1) Set the monitor attribute to false.
crsctl modify resource ora.cssd -attr "CHECKFS_MONITOR=false" -init
2) Kill cssdagent process. Following is an example.
# ps -eaf | grep cssdagent
mkallana 25467 1 0 Nov01 ? 00:09:44 /ade/mkallana_193dbru2/oracle/bin/cssdagent
# kill -9 25467
3) The cssdagent process will get automatically restarted and monitoring would be disabled in the new instance of cssdagent.
Please check the resource attribute value to ensure that it is disabled. Eg. crsctl stat res ora.cssd -init -p | grep CHECKFS_MONITOR
CHECKFS_MONITOR=false

Important Note: Please try it on a test system before doing it on the production system

Leave Comment