Last month my phone rang at 2:14 AM. Not an alarm. An actual call from the on-call number, which on our team usually means one thing: something in the cluster fell over. It was one of our 2-node RAC databases. Node 2 had gone down hard, the application team was already seeing connection errors, and the person who called me said the one sentence every DBA dreads: "It just rebooted itself, nobody touched anything."
That's a RAC node eviction. And if you've been an Oracle DBA for more than a few months, you've either seen one already or you will soon. The frustrating part isn't fixing it. Most of the time, the fix is simple. The frustrating part is that when you're half asleep, and the phone is buzzing, it's very easy to open the wrong log first, chase the wrong clue, and burn 40 minutes before you even understand what actually happened.
So this post is not a theory dump on RAC internals. It's the actual sequence I follow, in the order I follow it, when a node gets evicted. Which files I open first, what I'm looking for in each one, and how I narrow down whether it's a network problem, a storage problem, or the node just choking on load. I'll use that 2 AM incident as the running example.
First: 30 seconds on who's talking to whom
Before jumping into logs, it helps to have this picture in your head, because every troubleshooting step below maps to one of these connections.
The part that matters most for eviction troubleshooting is the bottom half of this picture. Every node in a RAC cluster proves it's "alive" to the other nodes in two independent ways:
1. A network heartbeat over the private interconnect. Basically, the nodes pinging each other constantly.
2. A disk heartbeat. Each node writes to the voting disk at a fixed interval, and the others can see that write.
CSSD (Cluster Synchronization Services) is the process watching both. If a node misses too many of either kind of heartbeat (the threshold is controlled by the misscount parameter, 30 seconds by default in most versions), CSSD on the healthy node(s) declares the quiet node dead and, to avoid a split-brain situation where two nodes think they separately own the same data, the cluster forces that node to reboot. That's the eviction. It's not a bug; it's the cluster doing its job and protecting your data. Which is exactly why panicking and just bouncing the instance without checking the logs first usually doesn't help.
Step 1: Don't Touch the Database Alert Log Yet
This is the mistake I see most often, including from myself a few years back. The instance is down, so the instinct is to go straight to alert_<SID>.log. You'll find something like this:
Errors in file .../ORCL2_lmon_12345.trc: ORA-29740: evicted by instance number 1, group incarnation 7 LMON (ospid: 12345): terminating the instance due to error 29740 Instance terminated by LMON, pid = 12345
And that's it. That's all the DB alert log will ever tell you. The instance was told to die by the clusterware. It's a symptom, not a cause. The actual explanation lives one layer down, in the Grid Infrastructure logs. This is the order I go through, and honestly writing it down as a diagram is the only reason I stopped skipping steps under pressure:
A quick note on paths, since these move around release to release and people always ask me for them: from 11.2 onwards everything sits under
$GRID_HOME/log/<hostname>/, with a subfolder per daemon (cssd, crsd, ohasd, etc). If you're on 19c and can't find ocssd.log by browsing, adrci or just find $GRID_HOME/log/$(hostname -s) -iname "*.log" -mmin -180 will get you there fast.Step 2: Read ocssd.log for the Actual Heartbeat Message
This is where the real story is. In our 2 AM case, ocssd.log on the surviving node (racnode1) had this, trimmed down:
[CSSD]CLSSNM00008: node racnode2 (2) at 90% heartbeat fatal, removal in 2.910 seconds [CSSD]CLSSNM00008: node racnode2 (2) missed(6) network heartbeats [CSSD]CLSSNM00008: node racnode2 (2) is impending reconfig, flag 918030, misstime 27090 [CSSD]clssnmSendingThread: sending remove message for racnode2, number 2 [CSSD]clssscExit: CSSD signal 11 received Removal of node racnode2 (2) from cluster complete
"missed network heartbeats" is the key phrase. If it had said something about the voting disk or a "disk timeout," I'd have gone hunting in storage instead. This one line is basically a fork in the road, and it's the reason I now keep this little decision tree taped (mentally, at least) next to the incident checklist:
In our case, it was squarely the left branch. Network. So the next stop wasn't ASM; it was the network team's territory, except I checked it myself first because waiting for a ticket to get picked up at 2 AM is not a plan.
Step 3: Chase the Network Heartbeat Miss
A few commands I actually ran, in this order:
# confirm which interface is carrying the private interconnect $ oifcfg getif eth0 10.10.20.0 global public eth1 192.168.10.0 global cluster_interconnect # from the surviving node, hammer the private IP of the node that got evicted $ ping -c 20 192.168.10.12 # check the interface for errors/drops, not just "is it up" $ ifconfig eth1 $ ethtool -S eth1 | egrep -i "error|drop|discard" # and don't forget the switch side if you have access $ netstat -s | grep -i retrans
The ping came back fine a few minutes later. Which is normal and also unhelpful, because by the time you're checking, the blip is usually over. What actually gave it away was ethtool -S on eth1 showing a jump in rx_missed_errors right around 02:11, which lined up almost to the second with the "missed(6) network heartbeats" entry in ocssd.log. We pulled in the network team the next morning, and it turned out to be a flaky SFP module on that host's NIC that had been intermittently dropping packets for a couple of weeks without crossing any monitoring threshold. Nobody had noticed because it never dropped enough packets to trip the switch's own alerting. Just enough, once, to blow past the interconnect's 30-second misscount window.
The point of walking through this isn't the specific root cause. Yours will probably be different. The point is that the log told us which of three very different investigations to start (network vs storage vs OS load), and that saved a good hour of looking in the wrong place.
What It Looks Like When It's Storage, Not Network
Just so this isn't a one-scenario post. If ocssd.log instead shows something like:
[CSSD]clssnmvDiskCheck: (0x7f2a4c003) voting file /dev/oracleasm/disks/VOTE01 not written for 210610 ms [CSSD]clssnmvDiskCheck: Aborting, 1 of 3 configured voting disks available, need 2
Then you stop looking at the network entirely and go straight to storage: check the ASM alert log for the same time window, run iostat -x 2 (or check your storage array's own latency dashboard) for I/O latency spikes, and if you're on multipath, check whether a path failover was in progress (multipath -ll, or the DM-Multipath logs). Voting disk write timeouts are almost always storage latency, not the disk being literally unreachable. The array is just too slow to answer within the CSS timeout, often during a SAN-side maintenance window or a sudden IO storm from something unrelated sharing the same array.
And the third case: no heartbeat message at all in ocssd.log, node just stopped responding. That's usually the node itself. CPU pegged, memory swapping, or occasionally a massive GC-style pause from something running on the host (yes, even on a "DB-only" server, someone eventually runs an ad-hoc script that eats all the CPU). For that one, OS Watcher or Cluster Health Monitor (oclumon dumpnodeview if it's still running, or CHM's own repository if OSW isn't installed) is what actually saved me the one time I had a genuinely hung node. vmstat from OSW showed run queue length going from single digits to 400+ in about ninety seconds.
Step 4: After You've Found the Cause, Close the Loop
Once the node came back up on its own (self-healing after a clean reboot, which is the usual case) I still ran through this before calling it done:
1. crsctl check crs and crsctl status resource -t on the recovered node. Make sure every resource is actually ONLINE and not just "starting."
2. crsctl query css votedisk. Confirm all voting disks are visible again.
3. Checked the alert log of the recovered instance for a clean startup, no repeat ORA-29740 within the next hour.
4. Opened a ticket with the network team with the exact timestamp and the ethtool output, instead of a vague "cluster had an issue last night."
That last point matters more than people give it credit for. "The node got evicted" is not a root cause and won't get fixed. "eth1 on racnode2 showed rx_missed_errors spiking at 02:11:04, correlating with a 6-heartbeat miss in ocssd.log" gets you a network engineer pulling an SFP module the same day.
Checklist Version: For When You're Getting Paged
1. Alert log just confirms the instance died. Don't stop there.
2. CRS alert log tells you who decided to evict and roughly when.
3. ocssd.log tells you WHY. Network heartbeat, disk heartbeat, or neither.
4. Network miss. Check the interconnect NIC/switch first, with timestamps.
5. Disk miss. Check storage/ASM latency around the same window.
6. Neither. The node was hung. Pull OSW/CHM data for that exact time.
7. Fix the actual cause. Don't just restart things and hope.
8. Write down the timestamp-to-cause mapping before you forget it by morning.
RAC troubleshooting has a reputation for being intimidating, mostly because there are four or five log files involved instead of one. Once you know the order to read them in, and you know that ocssd.log is really the file doing the talking, it stops being scary and becomes a fairly mechanical process. Which is exactly what you want at 2 AM.
Keep learning... Have a great day!!!
Thank you,
Amit Pawar
Email: amitpawar.dba@gmail.com
WhatsApp No: +91-8454841011
