Friday 30 August 2024

Step by Step Oracle RAC Standby Database Switchover

1. On Primary:

a. Verify there is no log file gap between the primary and the standby database

SELECT status, gap_status 
FROM v$archive_dest_status 
WHERE dest_id = 2;

STATUS    GAP_STATUS
--------- ------------------------
VALID     NO GAP

b. Verify that the primary can be switched to the standby role.

SET lines 4000
SELECT name, log_mode, controlfile_type, open_mode, database_role, switchover_status 
FROM v$database;

NAME      LOG_MODE     CONTROL OPEN_MODE            DATABASE_ROLE    SWITCHOVER_STATUS
--------- ------------ ------- -------------------- ---------------- --------------------
ORADB     ARCHIVELOG   CURRENT READ WRITE           PRIMARY          TO STANDBY

c. Check Temp files match count in Primary and Standby

COL name FOR a45
SELECT ts#, name, status 
FROM v$tempfile;

col name for a45
    select ts#,name,ts#,status  from v$tempfile;
 
 
       TS# NAME                                                 TS# STATUS
---------- --------------------------------------------- ---------- -------
         3 /u01/app/oracle/oradata/ORADB/temp01.dbf               3 ONLINE
         3 /u01/app/oracle/oradata/ORADB/pdbseed/temp012          3 ONLINE
           020-09-03_00-49-51-498-AM.dbf
 
         3 /u01/app/oracle/oradata/ORADB/pdb1/temp01.dbf          3 ONLINE


d. Clear Potential Blocking Parameters & Jobs

SHOW PARAMETER job_queue_processes
ALTER SYSTEM SET job_queue_processes=0 SCOPE=BOTH;

show parameter job_queue_processes         
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes                  integer     40
 
ALTER SYSTEM SET job_queue_processes=0 SCOPE=BOTH;
System altered.



COL owner FOR a30
COL job_name FOR a40
COL start_date FOR a30
COL end_date FOR a30        
SELECT owner, job_name, start_date, end_date, enabled 
FROM dba_scheduler_jobs 
WHERE enabled='TRUE' AND owner NOT IN 'SYS'

OWNER                JOB_NAME                         START_DATE                     END_DATE                       ENABL
-------------------- -------------------------------- ------------------------------ ------------------------------ -----
ORACLE_OCM           MGMT_CONFIG_JOB                  04-SEP-20 12.53.14.000000 AM +                                TRUE
                                                      04:00
											   
ORACLE_OCM           MGMT_STATS_CONFIG_JOB            04-SEP-20 12.53.14.000000 AM +                                TRUE
                                                      04:00
											   
EXECUTE DBMS_SCHEDULER.DISABLE('ORACLE_OCM.MGMT_CONFIG_JOB');

PL/SQL procedure successfully completed.

EXECUTE DBMS_SCHEDULER.DISABLE('ORACLE_OCM.MGMT_STATS_CONFIG_JOB');

PL/SQL procedure successfully completed.

e. Create Guaranteed Restore Point on Primary and Standby and make sure they are dropped post successful switchover

SHOW PARAMETER db_recovery_file_dest

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest                string      /u01/app/oracle/fra
db_recovery_file_dest_size           big integer 22732M

SQL> show parameter db_recovery_file_dest_size
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 22732M
 
 
SQL> alter system set db_recovery_file_dest_size=5G;
System altered.
 
SQL> show parameter db_recovery_file_dest_size
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest_size           big integer 5G

SQL> create restore point switchover_restore_point  guarantee flashback database;
 
Restore point created.

select NAME,TIME,GUARANTEE_FLASHBACK_DATABASE,storage_size/1024/1024/1024 from v$restore_point;
 
NAME                           TIME                                     GUA STORAGE_SIZE/1024/1024/1024
------------------------------ ---------------------------------------- --- ---------------------------
SWITCHOVER_RESTORE_POINT       08-FEB-24 10.04.30.000000000 AM          YES                    .1953125


2. On Primary:

a. Switch the Primary to the Standby Role:

ALTER DATABASE COMMIT TO SWITCHOVER TO PHYSICAL STANDBY WITH SESSION SHUTDOWN;

Database altered.

b. Verify the standby has received the end-of-redo (EOR) log(s) in standby alert log

Alert log sample message :
PR00 (PID:5275): Resetting standby activation ID 2803353007 (0xa717c5af)
2020-09-04T01:32:44.494716+04:00
Media Recovery End-Of-Redo indicator encountered
2020-09-04T01:32:44.494784+04:00
Media Recovery Continuing
PR00 (PID:5275): Media Recovery Waiting for T-1.S-32
2020-09-04T01:32:45.535769+04:00
.... (PID:5531): The Time Management Interface (TMI) is being enabled for role transition
.... (PID:5531): information.  This will result in messages beingoutput to the alert log
.... (PID:5531): file with the prefix 'TMI: '.  This is being enabled to make the timing of
.... (PID:5531): the various stages of the role transition available for diagnostic purposes.
.... (PID:5531): This output will end when the role transition is complete.
SWITCHOVER: received request 'ALTER DATABASE COMMIT TO SWITCHOVER  TO PRIMARY' from primary database.
2020-09-04T01:32:45.536371+04:00
ALTER DATABASE SWITCHOVER TO PRIMARY (oradb_s2)
Maximum wait for role transition is 15 minutes.
TMI: kcv_commit_to_so_to_primary wait for MRP to finish BEGIN 2020-09-04 01:32:45.536801
Switchover: Media recovery is still active
 rmi (PID:5531): Role Change: Canceling MRP - no more redo to apply
2020-09-04T01:32:45.538289+04:00
PR00 (PID:5275): MRP0: Background Media Recovery cancelled with status 16037

c. Shutdown and restart the database in standby mode

[oracle@srv1 ~]$ sqlplus / as sysdba
 
SQL*Plus: Release 19.0.0.0.0 - Production on Thu Feb 8 10:10:15 2024
Version 19.3.0.0.0
 
Copyright (c) 1982, 2019, Oracle.  All rights reserved.
 
Connected to an idle instance.
 
SQL> startup nomount
ORACLE instance started.
 
Total System Global Area 1073738888 bytes
Fixed Size                  9143432 bytes
Variable Size             276824064 bytes
Database Buffers          780140544 bytes
Redo Buffers                7630848 bytes
SQL> alter database mount standby database
  2  ;
 
Database altered.


3. On Standby:

a. Check the MRP process state

Select process,status,sequence# from v$managed_standby;
PROCESS   STATUS        SEQUENCE#
--------- ------------ ----------
ARCH      CLOSING              86
DGRD      ALLOCATED             0
DGRD      ALLOCATED             0
ARCH      CLOSING              94
ARCH      CONNECTED             0
ARCH      CONNECTED             0
MRP0      WAIT_FOR_LOG         95


b. Switch the Standby to Primary

SQL> Alter database commit to switchover to primary with session shutdown;
 
Database altered.
 
SQL> Alter database open;
 
Database altered.
SQL> Select name, open_mode, database_role from v$database;
 
NAME      OPEN_MODE            DATABASE_ROLE
--------- -------------------- ----------------
ORADB     READ WRITE           PRIMARY

4. Start Redo Apply on the new physical standby database

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE DISCONNECT FROM SESSION;
 
Database altered.

5. Enable log shipping on new primary

ALTER SYSTEM SET log_archive_dest_state_2=ENABLE SCOPE=BOTH;

System altered.

6. Set job_queue_processes parameter on new standby database

SHOW PARAMETER job_queue_processes    
ALTER SYSTEM SET job_queue_processes=40 SCOPE=BOTH;

7. Start secondary new primary instance

In case of standalone database no action needed, as we already have database open. Start all the instances in RAC environment which were stopped before start of the activity.
srvctl status database -d prod -v
srvctl start instance -d prod -i prod2
srvctl status database -d prod -v

8. Start secondary new standby instance

In case of standalone database no action needed, as we already have database open in mount mode. Start all the instances in RAC environment which were stopped before start of the activity.
srvctl status database -d stby -v
srvctl start instance -d stby -i stby2
srvctl status database -d stby -v

9. Schedule backups on new primary and create DB links if they exist on primary new

START all cronjobs and DBMS_SCHEDULER jobs

EXECUTE DBMS_SCHEDULER.ENABLE('ORACLE_OCM.MGMT_CONFIG_JOB');

PL/SQL procedure successfully completed.

EXECUTE DBMS_SCHEDULER.ENABLE('ORACLE_OCM.MGMT_STATS_CONFIG_JOB');

PL/SQL procedure successfully completed.

SQL> SELECT OWNER, JOB_NAME, START_DATE, END_DATE, ENABLED FROM DBA_SCHEDULER_JOBS WHERE ENABLED='TRUE' AND OWNER NOT IN 'SYS';
 
OWNER      JOB_NAME                       START_DATE                               END_DATE                       ENABL
---------- ------------------------------ ---------------------------------------- ------------------------------ -----
ORACLE_OCM MGMT_CONFIG_JOB                04-SEP-20 12.53.14.000000 AM +04:00                                     TRUE
ORACLE_OCM MGMT_STATS_CONFIG_JOB          04-SEP-20 12.53.14.000000 AM +04:00                                     TRUE
 

Note: Post switchover, ensure sufficient Temporary tablespace is available on the new primary. If not, add it.

On New Standby:

10. Drop the restore point created earlier

SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;
 
Database altered.
 
SQL> select NAME,TIME,GUARANTEE_FLASHBACK_DATABASE,storage_size/1024/1024/1024 from v$restore_point;
NAME
--------------------------------------------------------------------------------
TIME                                                                        GUA
--------------------------------------------------------------------------- ---
STORAGE_SIZE/1024/1024/1024
---------------------------
SWITCHOVER_RESTORE_POINT
08-FEB-24 10.04.30.000000000 AM                                             YES
                   .1953125
  
SQL> DROP RESTORE POINT switchover_restore_point;
 
Restore point dropped.
 
SQL> ALTER DATABASE RECOVER MANAGED STANDBY DATABASE USING CURRENT LOGFILE DISCONNECT from session parallel 6;
Database altered.

Friday 19 July 2024

OLR Internals


In Oracle 11g Release 2, Oracle introduced the Oracle Local Registry (OLR), a new feature that complements the Oracle Cluster Registry (OCR) within the Grid Infrastructure. The OLR is an essential component stored locally on each node in the Grid Infrastructure home directory, specifically at 'grid_home/cdata/<hostname>.olr', with its location documented in '/etc/oracle/olr.loc'.
Each node maintains its unique OLR file within the Grid Infrastructure software home. The OLR contains critical security contexts required by the Oracle High Availability Service during the early stages of Clusterware startup. The data within the OLR is vital for the Oracle High Availability Services daemon (OHASD) to initialize, encompassing information about GPnP wallets, Clusterware configurations, and version details. This information, alongside the Grid Plug and Play (GPnP) configuration file, is crucial for locating the voting disks. If these disks are stored in ASM, the GPnP profile’s discovery string will be used by the cluster synchronization daemon for their retrieval.

In this post, I'll delve into the purpose of the OLR, its necessity, and the type of data it contains. To fully understand these aspects, we need to examine the contents of an OLR dump:


ocrdump -local -stdout
[SYSTEM]
[SYSTEM.version]
[SYSTEM.version.activeversion]
[SYSTEM.version.hostnames.rac1]
[SYSTEM.version.hostnames.rac2]
[SYSTEM.ORA_CRS_HOME]
[SYSTEM.evm]
[SYSTEM.GPnP.profiles]
[SYSTEM.CRSADMIN]
[SYSTEM.CRSUSER]
[SYSTEM.CRSD]
[SYSTEM.CRSD.SERVERPOOLS]
[SYSTEM.CRSD.SERVERS]
[SYSTEM.CRSD.SERVERS.rac1.STATE]
[SYSTEM.CRSD.TYPES]
[SYSTEM.CRSD.TYPES.ora!local_resource!type.START_DEPENDENCIES]
[SYSTEM.CRSD.TYPES.ora!local_resource!type.START_DEPENDENCIES.CONFIG]
[SYSTEM.CRSD.TYPES.ora!local_resource!type.STOP_DEPENDENCIES]
[SYSTEM.CRSD.TYPES.ora!local_resource!type.STOP_DEPENDENCIES.CONFIG]
[SYSTEM.CRSD.TYPES.ora!network!type]
[SYSTEM.CRSD.TYPES.ora!cluster_resource!type.AUTO_START]
[SYSTEM.CRSD.TYPES.ora!database!type.INSTANCE_FAILOVER]
[SYSTEM.CRSD.TYPES.ora!database!type.ORACLE_HOME]
[SYSTEM.CRSD.RESOURCES.ora!net1!network]
[SYSTEM.CRSD.RESOURCES.ora!gsd]
[SYSTEM.CRSD.RESOURCES.ora!LISTENER_SCAN1!lsnr.INTERNAL]
[SYSTEM.CRSD.RESOURCES.ora!LISTENER_SCAN2!lsnr]
[SYSTEM.CRSD.RESOURCES.ora!oc4j]
[SYSTEM.CRSD.RESOURCES.ora!rac2!vip.INTERNAL]
[SYSTEM.CRSD.RESOURCES.ora!LISTENER!lsnr.INTERNAL]
[SYSTEM.CRSD.RESOURCES.ora!orcl!db.INTERNAL]
[DATABASE.NODEAPPS.rac2]
[DATABASE.VIP_RANGE]
[DATABASE.ASM]
[DATABASE.ASM.rac1.+asm1.VERSION]
[DATABASE.ASM.rac2.+asm2.ORACLE_HOME]
[CRS]
[CRS.CUR]
[CRS.HIS]
[CRS.SEC]
[CRS.STAGE]
[CRS.STAGE.node1]

I've tried to format the output for clarity. The OLR contains extensive information, including ORA_CRS_HOME, Clusterware versions, configurations, local host versions, active versions, GPnP details, OCR latest backup times and locations, node names, and the status of node resources—whether they need to start or not—and the start & stop dependencies of these resources. These dependencies are classified as either weak (optional) or hard (mandatory).

Understanding the purpose of the OLR is crucial. Although the OCR needs to be accessible by Clusterware to determine which resources to start on a node, Oracle 11gR2 allows the OCR to be stored in ASM. This poses a challenge because ASM itself is a resource that must be started. Here, the OLR comes into play. Being a locally available file on the operating system, the OLR can be read by any process with the appropriate privileges, without any dependencies.

The High Availability Services stack comprises daemons that communicate with their counterparts on other nodes. Once the High Availability Services stack is operational, the cluster node can join the cluster and use shared components like the OCR. Part of the startup sequence for the High Availability Services stack is stored in the GPnP profile, but it also relies on information from the OLR.

A pertinent question is why OCR is still needed if we have OLR. Comparing the OLR and OCR reveals that the OLR contains fewer keys. For example, an 'ocrdump' might show 704 keys for the OCR versus 526 keys for the OLR. Most keys in the OLR pertain to the OHASD process, while those in the OCR are related to CRSD. This indicates the necessity of the OLR (along with the GPnP profile) for starting the High Availability Services stack.

I hope this explanation helps you understand the OLR, its purpose, its content, and why it is essential.


I hope this article helped you. Your suggestions/feedback are most welcome.

Keep learning... Have a great day!!!


Thank you,
Amit Pawar
Email: amitpawar.dba@gmail.com
WhatsApp No: +91-8454841011


Saturday 2 December 2023

Transport Good Execution Plan(Hash Plan) from one database to another database

1. Create Empty Tuning Set


BEGIN
  DBMS_SQLTUNE.CREATE_SQLSET (
    sqlset_name  => 'SQLT_WKLD_STS', 
    description  => 'STS to store SQL from the private SQL area' 
);
END;

2. Load the SQL Information for the SQL ID into this Tuning Set

DECLARE c_sqlarea_cursor DBMS_SQLTUNE.SQLSET_CURSOR; BEGIN OPEN c_sqlarea_cursor FOR SELECT VALUE(p) FROM TABLE( DBMS_SQLTUNE.SELECT_CURSOR_CACHE( ' sql_id = ''2t8gdjy0pbs57'' AND plan_hash_value = ''618955923'' ') ) p; -- load the tuning set DBMS_SQLTUNE.LOAD_SQLSET ( sqlset_name => 'SQLT_WKLD_STS' , populate_cursor => c_sqlarea_cursor ); END; /

3. Display the content of Sql Tuning set

SELECT SQL_ID, PARSING_SCHEMA_NAME AS "SCH", SQL_TEXT, ELAPSED_TIME AS "ELAPSED", BUFFER_GETS FROM TABLE( DBMS_SQLTUNE.SELECT_SQLSET( 'SQLT_WKLD_STS' ) ); SQL_ID SCH SQL_TEXT ELAPSED BUFFER_GETS ------------- ------------------------- -------------------------------------------------------------------------------- ---------- ----------- 2t8gdjy0pbs57 TESTUSER INSERT /*+ APPEND */ INTO W_L_T.PROJECT as SELECT….. 24844903 2577896


SELECT sql_id, parsing_schema_name as "SCH", sql_text,buffer_gets as "B_GETS",disk_reads, ROUND(disk_reads/buffer_gets*100,2) "%_DISK" FROM TABLE( DBMS_SQLTUNE.SELECT_SQLSET( 'SQLT_WKLD_STS', '(disk_reads/buffer_gets) >= 0.50' ) ); SQL_ID SCH SQL_TEXT B_GETS DISK_READS %_DISK ------------- ------------------------- -------------------------------------------------------------------------------- ---------- ---------- ---------- 2t8gdjy0pbs57 TESTUSER INSERT /*+ APPEND */ INTO W_L_T.PROJECT as SELECT….. 2577896 2435837 94.49

4. Create a Staging table to hold the exported SQL Tuning set

BEGIN DBMS_SQLTUNE.CREATE_STGTAB_SQLSET (table_name => 'MY_TUNING_SET', schema_name => 'PDBADMIN'); END; /

5. Load the SQL Tuning Set information to the Staging Table
BEGIN DBMS_SQLTUNE.PACK_STGTAB_SQLSET ( sqlset_name => 'SQLT_WKLD_STS', sqlset_owner => 'PDBADMIN', staging_table_name => 'MY_TUNING_SET', staging_schema_owner => 'PDBADMIN' ); END; /


6. Export The table

nohup expdp userid="USERNAME/PASSWORD@lnx01.oraclevcn.com:1521/PDB1S001.oraclevcn.com" parfile/u01/dumps/mytunigset_table.par &

7. Import of the Staging table

nohup impdp userid="USERNAME/PASSWORD@lnx02.oraclevcn.com:1521/PDB1P001.oraclevcn.com" parfile/u01/dumps/mytunigset_table.par &

8. Unpack the SQL Tuning set from the staging table to the destination server

BEGIN DBMS_SQLTUNE.UNPACK_STGTAB_SQLSET ( sqlset_name => '%' , sqlset_owner => 'PDBADMIN' , replace => true , staging_table_name => 'MY_TUNING_SET' , staging_schema_owner => 'PDBADMIN'); END; /

9. Load the plan from SQL Tuning Set to SQL Plan Baseline

VARIABLE v_plan_cnt NUMBER EXECUTE :v_plan_cnt := DBMS_SPM.LOAD_PLANS_FROM_SQLSET( - sqlset_name => 'SQLT_WKLD_STS', - sqlset_owner => 'PDBADMIN', - basic_filter => 'sql_id = ''2t8gdjy0pbs57'' AND plan_hash_value = 618955923' );

10. Flush the two SQLs from the shared pool, so that the optimizer will pick the new plan

select 'exec DBMS_SHARED_POOL.PURGE ('''||ADDRESS||', '||HASH_VALUE||''', ''C'');' from V$SQLAREA where SQL_ID in ('2t8gdjy0pbs57'); exec SYS.DBMS_SHARED_POOL.PURGE ('0000001006B396C8, 2113289046', 'C');

11. To verify the Baseline created in the database

select sql_handle, plan_name, enabled, accepted, fixed from dba_sql_plan_baselines;

12. Drop SQL tuning Set from Source

BEGIN DBMS_SQLTUNE.drop_sqlset (sqlset_name => 'SQLT_WKLD_STS'); END;








I hope this article helped you. Your suggestions/feedback are most welcome.

Keep learning... Have a great day!!!


Thank you,
Amit Pawar
Email: amitpawar.dba@gmail.com
WhatsApp No: +91-8454841011