Recovery from complete loss of all online redo log files using RMAN

STEP 1: Check database details.

SQL> select instance_name,version from v$instance;

INSTANCE_NAME    VERSION

---------------- -----------------

TEST           11.2.0.3.0

SQL> select member from v$Logfile;

MEMBER

-------------------------------

/u02/ORACLE/TEST/redo03.log

/u02/ORACLE/TEST/redo02.log

/u02/ORACLE/TEST/redo01.log

STEP 2: Delete log files from os level to create scenario.

TEST:/u02/ORACLE/TEST>rm redo*.log

STEP 3: find last archive log sequence.

If one or all of the online redo logfiles are delete then the database hangs and in the alert log file we can see the following error message:

Tue Jan 30 00:47:19 2015

ARC1: Failed to archive thread 1 sequence 93 (0)

Tue Jan 30 00:47:24 2007

Errors in file /opt/oracle/admin/TEST/bdump/TEST_arc0_32722.trc:

ORA-00313: open failed for members of log group 2 of thread 1

ORA-00312: online log 2 thread 1: '/u02/ORACLE/TEST/redo02.log'

ORA-27037: unable to obtain file status

Linux-x86_64 Error: 2: No such file or directory

The file is missing at the operating system level.

Using RMAN we can recover from this error by restoring the database from the backup and recovering to the last available archived redo logfile.

From the error message in the log file we can get the last archived file in our case it is sequence 92 as the error shows that it fails to archive the log file sequence 93.

SQL> select * from v$Log;

  GROUP#   THREAD#  SEQUENCE#   BYTES    MEMBERS ARC   STATUS    FIRST_CHANGE#   FIRST_TIM

---------- ---------- ---------- ---------- ---------- --- ---------------- ------------- ---

    1          1         95   52428800          1 NO  CURRENT     3203078        30-JAN-15

    2          1         93   52428800          1 NO  INACTIVE    3202983        30-JAN-15

    3          1         94   52428800          1 NO  INACTIVE    3203074        30-JAN-15

At the operating system also we can find the last archived logfile:

TEST:/u02/ORACLE/TEST/arch> ls –lrt

total 54824

-rw-r-----    1 oracle   dba        714240 Jan 29 16:02 arch_1_90_613129285.dbf

-rw-r-----    1 oracle   dba      46281216 Jan 30 00:37 arch_1_91_613129285.dbf

-rw-r-----    1 oracle   dba         11264 Jan 30 00:41 arch_1_92_613129285.dbf

STEP 4: Shut down the database and startup in Mount stage.

SQL> shutdown immediate;

Database closed.

Database dismounted.

ORACLE instance shut down.

SQL> startup mount;

ORACLE instance started.

Total System Global Area  167772160 bytes

Fixed Size                  2069680 bytes

Variable Size              92277584 bytes

Database Buffers           67108864 bytes

Redo Buffers                6316032 bytes

Database mounted.

STEP 5: Connect to rman to Restore and Recover database.

TEST:/u02/ORACLE/TEST>rman target /

Recovery Manager: Release 10.2.0.2.0 - Production on Tue Jan 30 00:53:21 2015

Copyright (c) 1982, 2005, Oracle.  All rights reserved.

connected to target database: TEST (DBID=1493612009, not open)

RMAN> run {

2> set until sequence 93;

3> restore database;

4> recover database;

5>  alter database open resetlogs;

6> }

executing command: SET until clause

Starting restore at 30-JAN-15

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=156 devtype=DISK

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: sid=155 devtype=SBT_TAPE

channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.2.4.0

channel ORA_SBT_TAPE_1: starting datafile backupset restore

channel ORA_SBT_TAPE_1: specifying datafile(s) to restore from backup set

restoring datafile 00001 to /u02/ORACLE/TEST/system01.dbf

restoring datafile 00002 to /u02/ORACLE/TEST/undotbs01.dbf

restoring datafile 00003 to /u02/ORACLE/TEST/sysaux01.dbf

restoring datafile 00004 to /u02/ORACLE/TEST/users01.dbf

restoring datafile 00005 to /u02/ORACLE/TEST/users02.dbf

restoring datafile 00006 to /u02/ORACLE/TEST/users03.dbf

restoring datafile 00007 to /u02/ORACLE/TEST/users05.dbf

restoring datafile 00008 to /u02/ORACLE/TEST/users06.dbf

restoring datafile 00009 to /u02/ORACLE/TEST/users07.dbf

restoring datafile 00010 to /u02/ORACLE/TEST/users04.dbf

restoring datafile 00011 to /u02/ORACLE/TEST/drtbs1.dbf

restoring datafile 00012 to /u02/ORACLE/TEST/drtbs2.dbf

restoring datafile 00013 to /tmp/undonew.dbf

channel ORA_SBT_TAPE_1: reading from backup piece 5mi8ornj_1_1

channel ORA_SBT_TAPE_1: restored backup piece 1

piece handle=5mi8ornj_1_1 tag=TAG20070130T004019

channel ORA_SBT_TAPE_1: restore complete, elapsed time: 00:01:06

Finished restore at 30-JAN-15

Starting recover at 30-JAN-15

using channel ORA_DISK_1

using channel ORA_SBT_TAPE_1

starting media recovery

archive log thread 1 sequence 92 is already on disk as file /u02/ORACLE/TEST/arch/arch_1_92_613129285.dbf

archive log filename=/u02/ORACLE/TEST/arch/arch_1_92_613129285.dbf thread=1 sequence=92

media recovery complete, elapsed time: 00:00:01

Finished recover at 30-JAN-15

database opened

RMAN>exit

STEP 6: Check all the files are created physically.

The recovery process creates the online redo logfiles at the operating system level also.

TEST:/u02/ORACLE/TEST>ls -lrt redo*

-rw-r-----    1 oracle   dba      52429312 Jan 30 01:00 redo03.log

-rw-r-----    1 oracle   dba      52429312 Jan 30 01:00 redo02.log

-rw-r-----    1 oracle   dba      52429312 Jan 30 01:00 redo01.log

Since we have done an incomplete recover with open resetlogs, we should take a fresh complete backup of the database.

NOTE: Please make sure you remove all the old archived logfiles from the archived area.

I hope this article helped you. Your suggestions/feedback are most welcome.

Keep learning... Have a great day!!!

Database Point-In-Time Recovery

 STEP 1: Note the current database clock time

SQL> select to_char(sysdate,'DD-MON-YY:HH24:MI:SS') from dual;

TO_CHAR(SYSDATE,'DD-MON-

------------------------

20-FEB-15:17:45:55

STEP 2: Drop a table

We will now drop the table MYOBJECTS and then perform an RMAN point-in-time recovery to restore the database to a point in time just before the table was dropped. Note the number of rows in the table.

SQL> select count(*) from myobjects;

  COUNT(*)

----------

     13420

SQL> drop table myobjects;

Table dropped.

STEP 3: Shutdown and then mount the database in preparation for restore and recovery.

SQL> conn / as sysdba

Connected.

SQL> shutdown abort

ORACLE instance shut down.

SQL> startup mount;

[oracle@NVMBD1BZY150D00 CLONEDB]$ rman target / catalog rman/rman@rmanp

Recovery Manager: Release 11.2.0.3.0 - Production on Fri Feb 20 17:49:27 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: CLONEDB (DBID=958582256, not open)

connected to recovery catalog database

Note – the incremental backup since available is restored before applying the archive log files. The archive log files that will be applied are those that have been generated after the Level 1 incremental backup.

Since we are restoring the database to a point in the past, we will have to open the database with the RESETLOGS option.

RMAN> run {

2> set until time "to_date('20-FEB-15:17:45:55','dd-MON-yy hh24:mi:ss')";

3> restore database;

4> recover database;

5> }

executing command: SET until clause

Starting restore at 20-FEB-15

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=176 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

channel ORA_DISK_1: restoring datafile 00001 to /data/oracle/app/oracle/oradata/CLONEDB/system.dbf

channel ORA_DISK_1: restoring datafile 00002 to /data/oracle/app/oracle/oradata/CLONEDB/user04.dbf

channel ORA_DISK_1: restoring datafile 00003 to /data/oracle/app/oracle/oradata/CLONEDB/sysaux.dbf

channel ORA_DISK_1: restoring datafile 00004 to /data/oracle/app/oracle/oradata/CLONEDB/undo.dbf

channel ORA_DISK_1: restoring datafile 00005 to /data/oracle/app/oracle/oradata/CLONEDB/test1_tmp.dbf

channel ORA_DISK_1: restoring datafile 00006 to /data/oracle/app/oracle/oradata/CLONEDB/test_corrupt.dbf

channel ORA_DISK_1: restoring datafile 00007 to /data/oracle/app/oracle/oradata/CLONEDB/users_01.dbf

channel ORA_DISK_1: reading from backup piece /data1/backup/clonedb/bkp.38pvodgm_1_1

channel ORA_DISK_1: piece handle=/data1/backup/clonedb/bkp.38pvodgm_1_1 tag=LEVEL 0

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:01:25

Finished restore at 20-FEB-15

Starting recover at 20-FEB-15

using channel ORA_DISK_1

channel ORA_DISK_1: starting incremental datafile backup set restore

channel ORA_DISK_1: specifying datafile(s) to restore from backup set

destination for restore of datafile 00001: /data/oracle/app/oracle/oradata/CLONEDB/system.dbf

destination for restore of datafile 00002: /data/oracle/app/oracle/oradata/CLONEDB/user04.dbf

destination for restore of datafile 00003: /data/oracle/app/oracle/oradata/CLONEDB/sysaux.dbf

destination for restore of datafile 00004: /data/oracle/app/oracle/oradata/CLONEDB/undo.dbf

destination for restore of datafile 00005: /data/oracle/app/oracle/oradata/CLONEDB/test1_tmp.dbf

destination for restore of datafile 00006: /data/oracle/app/oracle/oradata/CLONEDB/test_corrupt.dbf

destination for restore of datafile 00007: /data/oracle/app/oracle/oradata/CLONEDB/users_01.dbf

channel ORA_DISK_1: reading from backup piece /data1/backup/clonedb/bkp.3cpvog7a_1_1

channel ORA_DISK_1: piece handle=/data1/backup/clonedb/bkp.3cpvog7a_1_1 tag=LEVEL 1 BACKUP

channel ORA_DISK_1: restored backup piece 1

channel ORA_DISK_1: restore complete, elapsed time: 00:00:01

starting media recovery

archived log for thread 1 with sequence 6 is already on disk as file /data1/clonedb/archive/1_6_872166459.dbf

archived log for thread 1 with sequence 7 is already on disk as file /data1/clonedb/archive/1_7_872166459.dbf

archived log for thread 1 with sequence 8 is already on disk as file /data1/clonedb/archive/1_8_872166459.dbf

archived log for thread 1 with sequence 9 is already on disk as file /data1/clonedb/archive/1_9_872166459.dbf

archived log for thread 1 with sequence 10 is already on disk as file /data1/clonedb/archive/1_10_872166459.dbf

archived log for thread 1 with sequence 11 is already on disk as file /data1/clonedb/archive/1_11_872166459.dbf

archived log for thread 1 with sequence 12 is already on disk as file /data1/clonedb/archive/1_12_872166459.dbf

archived log file name=/data1/clonedb/archive/1_6_872166459.dbf thread=1 sequence=6

archived log file name=/data1/clonedb/archive/1_7_872166459.dbf thread=1 sequence=7

archived log file name=/data1/clonedb/archive/1_8_872166459.dbf thread=1 sequence=8

archived log file name=/data1/clonedb/archive/1_9_872166459.dbf thread=1 sequence=9

archived log file name=/data1/clonedb/archive/1_10_872166459.dbf thread=1 sequence=10

media recovery complete, elapsed time: 00:00:01

Finished recover at 20-FEB-15

STEP 4: Start the database with resetlogs option.

RMAN> sql 'alter database open resetlogs';

sql statement: alter database open resetlogs

STEP 5: Connect to database and confirm that table is present.

We will now connect to the database to confirm that our table which we had earlier dropped has been restored as well as the record count in the table matches that taken before the table drop.

[oracle@NVMBD1BZY150D00 CLONEDB]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Feb 20 17:59:52 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> select count(*) from myobjects;

  COUNT(*)

----------

     13420

STEP 6: Reset the database in recovery catalog.

Because we have done a RESETLOGS we need to reset the database in the recovery catalog.

[oracle@NVMBD1BZY150D00 CLONEDB]$ rman target / catalog rman/rman@rmanp

RMAN> reset database;

new incarnation of database registered in recovery catalog

starting full resync of recovery catalog

full resync complete

STEP 7: Standard Practice take full fresh backup of database.

We will now take a fresh level 0 incremental backup

RMAN> backup incremental level 0 database;

I hope this article helped you. Your suggestions/feedback are most welcome.

Keep learning... Have a great day!!!

RMAN-06059: expected archived log not found, loss of archived log compromises recoverability

While RMAN full backup, you may encountered “RMAN-06059: expected archived log not found, loss of archived log compromises recoverability”

Error log:

[oracle@NVMBD1BZY150D00 ~]$ rman target / cmdfile=bkp_inc_lev0.rcv

Recovery Manager: Release 11.2.0.3.0 - Production on Fri Feb 20 12:31:30 2015

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: CLONEDB (DBID=958582256)

RMAN> run

2> {

3> backup incremental level 0 database tag "level 0 "  plus archivelog delete input tag "ARCHIVE BACKUP";

4> }

5>

6>

Starting backup at 20-FEB-15

current log archived

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: SID=176 device type=DISK

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03002: failure of backup plus archivelog command at 02/20/2015 12:31:31

RMAN-06059: expected archived log not found, loss of archived log compromises recoverability

ORA-19625: error identifying file /data1/clonedb/archive/1_50_872092231.dbf

ORA-27037: unable to obtain file status

Linux-x86_64 Error: 2: No such file or directory

Additional information: 3

Recovery Manager complete.

Cause:

This can happen for a variety of reasons; the file has been manually moved or deleted,
the archive log destination has recently been changed,
the file has been compressed, etc

Solution:

Your options are either to restore the missing file(s), or to perform a crosscheck.
To perform a crosscheck, run the following command from within RMAN:

RMAN> CROSSCHECK ARCHIVELOG ALL;

Above command will give information of expired RMAN repository after verification. (i.e. in catalog or controlfile)

Above will marked archives as Expired who are not available physically and who are not required for any kind of recovery.

RMAN> DELETE EXPIRED ARCHIVELOG ALL;

Above command will delete all archive logs who are marked as expired while crosscheck.

I hope this article helped you. Your suggestions/feedback are most welcome.

Keep learning... Have a great day!!!

RECOVER LOSS OF ALL CONTROLFILES (NO CATALOG)

In this session will be discussing scenario of recovering database from loss of all controlfile using rman.

STEP1: CREATE A SCENARIO FOR TEST PURPOSE.

[oracle@NVMBD1BZY150D00 clonedb]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Feb 19 15:50:38 2015

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to:

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production

With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> conn oracle/oracle

Connected.

SQL>CREATE TABLE myobjects AS SELECT * FROM DBA_OBJECTS;

SQL> select table_name from user_tables;

TABLE_NAME

------------------------------

MYOBJECTS

SQL> insert into myobjects select * from myobjects;

919664 rows created.

SQL> commit;

Commit complete.

SQL> select count(*) from myobjects;

  COUNT(*)

----------

   1839328   >>>> need to check this record count after recovery

SQL> archive log list

Database log mode              Archive Mode

Automatic archival             Enabled

Archive destination            USE_DB_RECOVERY_FILE_DEST

Oldest online log sequence     5

Next log sequence to archive   7

Current log sequence           7

Note - current log sequence is 7 - not archived but contains the last committed changes that we made

Note - archive logs will not be found in $ARCV area, but in the flashback location

STEP2: STIMULATE A FAILURE

SQL> select name from v$controlfile;

NAME

-------------------------------------------------------------------------------

/u01/ORACLE/testdb/control01.ctl

/u01/ORACLE/testdb/control02.ctl

/u01/ORACLE/testdb/control03.ctl

SQL> !rm /u01/ORACLE/testdb/*.ctl

SQL> alter tablespace users online;

alter tablespace users online

*

ERROR at line 1:

ORA-00603: ORACLE server session terminated by fatal error.

STEP3: SHUT DONW THE DATABASE AND STARTUP IN NOMOUNT STAGE TO RECOVER CONTROL FILE.

SQL> shutdown abort

ORACLE instance shut down.

SQL> startup nomount;

ORACLE instance started.

Total System Global Area  893386752 bytes

Fixed Size                  2076816 bytes

Variable Size             432017264 bytes

Database Buffers          452984832 bytes

Redo Buffers                6307840 bytes

Since we are not using a RMAN catalog we need to set the DBID

RMAN> set dbid=2415549446;

executing command: SET DBID

STEP4: RESTORE THE CONTROLFILE.

RMAN> run {

2> restore controlfile from autobackup;

3> }

Starting restore at 18-SEP-14

using target database control file instead of recovery catalog

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=156 devtype=DISK

recovery area destination: /u01/ORACLE/flash_recovery_area

database name (or database unique name) used for search: TESTDB

channel ORA_DISK_1: autobackup found in the recovery area

channel ORA_DISK_1: autobackup found: /u01/ORACLE/flash_recovery_area/TESTDB/autobackup/2007_09_18/o1_mf_s_633601094_3gynd74g_.bkp

channel ORA_DISK_1: control file restore from autobackup complete

output filename=/u01/ORACLE/testdb/control01.ctl

output filename=/u01/ORACLE/testdb/control02.ctl

output filename=/u01/ORACLE/testdb/control03.ctl

Finished restore at 18-SEP-07

STEP5: MOUNT AND RECOVER THE DATABASE;

RMAN> alter database mount;

database mounted

released channel: ORA_DISK_1

RMAN> recover database;

Starting recover at 18-SEP-07

Starting implicit crosscheck backup at 18-SEP-07

allocated channel: ORA_DISK_1

channel ORA_DISK_1: sid=156 devtype=DISK

Crosschecked 1 objects

Finished implicit crosscheck backup at 18-SEP-07

Starting implicit crosscheck copy at 18-SEP-07

using channel ORA_DISK_1

Finished implicit crosscheck copy at 18-SEP-07

searching for all files in the recovery area

cataloging files...

cataloging done

List of Cataloged Files

=======================

File Name: /u01/ORACLE/flash_recovery_area/TESTDB/autobackup/2007_09_18/o1_mf_s_633601094_3gynd74g_.bkp

using channel ORA_DISK_1

allocated channel: ORA_SBT_TAPE_1

channel ORA_SBT_TAPE_1: sid=155 devtype=SBT_TAPE

channel ORA_SBT_TAPE_1: Data Protection for Oracle: version 5.2.4.0

datafile 4 not processed because file is offline

starting media recovery

archive log thread 1 sequence 6 is already on disk as file /u01/ORACLE/flash_recovery_area/TESTDB/archivelog/2007_09_18/o1_mf_1_6_3gyn7vnk_.arc

archive log thread 1 sequence 7 is already on disk as file /u01/ORACLE/testdb/redo03.log

archive log filename=/u01/ORACLE/flash_recovery_area/TESTDB/archivelog/2007_09_18/o1_mf_1_6_3gyn7vnk_.arc thread=1 sequence=6

archive log filename=/u01/ORACLE/testdb/redo03.log thread=1 sequence=7  >>>> current redo log with committed but unarchived changes applied

media recovery complete, elapsed time: 00:00:09

Finished recover at 18-SEP-07

STEP6: OPEN DATABASE WITH RESETLOGS OPTION.

SQL> alter database open resetlogs;

Database altered.

SQL> conn ORACLE/ORACLE

Connected.

SQL> select count(*) from myobjects;

  COUNT(*)

----------

   1839328

I hope this article helped you. Your suggestions/feedback are most welcome.

Keep learning... Have a great day!!!