Presentation - Oracle

Oracle Active Data Guard 

Database Rolling Maintenance 

Larry M. Carpenter 

Distinguished Product Manager - Oracle 

Kalyan Kumar Kallepally 

Senior Database Administrator - Wellcome Trust Sanger Institute 

Joydip Kundu 

Director, Software Development - Oracle 

1 

Copyright © 2012, Oracle and/or its affiliates. All rights reserved 

Confidential – Oracle Restricted.

Agenda 

• A look at Rolling Upgrades today 

• A customer’s viewpoint 

• Simple Database Upgrades for the future 

• In Closing 

2 



Database Rolling Upgrades 

Database A 

Install new Oracle version in separate 

homes on A & B, set guaranteed PROD 

restore point (GRP) on A 

Temporarily convert B to use SQL 

Apply, upgrade and sync 

PROD 

synchronize 

synchronize 

My Oracle Support Note 949322.1 

Database B 

Physical 

STBY 

Logical 

STBY 

Switchover, flashback A to GRP, 

mount in new/upgraded home 

Physical 

STBY 

SWITCHOVER 

PROD 

release n 

release n+1 

Upgrade A via redo stream 

and synchronize 

Physical 

STBY 

synchronize 

PROD 

3 Copyright © 2011, Oracle and/or its affiliates. All rights reserved.

Agenda 





4 



Introducing 

Kalyan Kallepally 

Senior Database Administrator 

5 



Upgrade Databases With Minimal 

downtime 

Oracle Open World - 2012 

by 

Kalyan Kallepally 

Senior Database Administrator 

Wellcome Trust Sanger Institute 

Cambridge, UK 

kalyan.kallepally@sanger.ac.uk

Who am I 

• Working as Senior Database Administrator at 

– Wellcome Trust Sanger Institute 

• 10+ years of experience in Oracle (8-11gr2) 

• 5+ years of experience in MySQL, SQL Server 

• Recently started blogging 

– www.dbakalyan.wordpress.com 

– You can reach me on dba_kkk@yahoo.co.uk

Wellcome Trust Sanger Institute 

• One of the leading genomics centres in the world 

• Located in Hinxton, south of Cambridge, UK 

• Founded in 1993 by the Wellcome Trust and the UK Medical Research Council 

(MRC) 

• Primarily funded by the Wellcome Trust 

• The institute is named after Dr Fred Sanger – The double Nobel prize-winning 

biochemist 

• Responsible for the completion of 1/3 of the human genome

• Current computational 

capacity of 16k cores: 

– 12k cores on blade format 

– 4k cores in racked servers 

• Current usable storage 

capacity 16 Petabytes: 

– Estimated growth 3 

Petabytes/year 

• Data shared globally across 

the world with many 

scientific research institutes

• Primary database (300G) 

– ASM 

Database Layout 

– 2 Physical standbys (local and remote) 

• Local standby is within same site as of the primary 

but placed in a different quadrant of the datacentre 

• DR standby is 10 miles away from the Primary with 

dedicated private 10Gb/s WAN to DR datacentre

Upgrade Process 

• Upgraded database from 11.1.0.6 to 11.2.0.3 

using Transient Logical standby 

– Down time reduced by 98% 

• Upgrade process is split into 2 phases: 

– Phase 1 (Upgrade Primary Database) 

– Phase 2 (Upgrade Standbys & Switchback)

Phase - 1 

Upgrade Primary Database

Read Write Clients 

11.1.0.6 

Read Only Clients 

Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 

Physical Standby 

Database 

ASM 

Production 

Database 

Backup 

ASM 

Oracle Home (11.1.0.6) 

Broker 

Sync or Async 

Redo shipping 


11.1.0.6 

Redo 

apply 

DR Site 


Database 

ASM 

Redo Logs 

Oracle Home (11.1.0.6)

Upgrade Pre-requisites 

• Disable Data Guard Broker 

• Protection level must be either Maximum 

availability or Maximum performance 

• Ensure Flashback is enabled on all databases in 

the Data Guard configuration 

• Create guaranteed restore points 

• Parameter “compatible” must be set to same on 

all the databases


11.1.0.6 


Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 

Primary 

Database 

Backup 

ASM 


• Disable Broker 

• Enable Flashback 

• Create Guaranteed Restore points 

Broker 

Sync or Async 

Redo shipping 


11.1.0.6 

Redo Logs 

Redo 

apply 

DR Site 


Database 

ASM 



11.1.0.6 


Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 

Primary 

Database 

Backup 

ASM 


• Convert the DR physical standby to 

Transient Logical Standby 

• Check if Logical standby is in sync 

Broker 

Sync or Async 

Redo shipping 


11.1.0.6 

Redo Logs 

Redo 

apply 

SQL 

apply 

Transform Redo to SQL 

DR Site 

Physical Logical Standby 

Database 

ASM 



11.1.0.6 


Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 

Primary 

Database 

Backup 

ASM 


• Install new version of Oracle 

database Software 

• Upgrade the LSTBY using DBUA from 

the new ORACLE HOME 

• After successful upgrade check if 

LSTBY is in sync again 

Broker 

Sync or Async 

Redo shipping 


11.1.0.6 11.2.0.3 

Redo Logs 

SQL 

apply 


Oracle Home (11.2.0.3) (11.1.0.6) 

DR Site 

Logical Standby 

Database 

ASM


11.1.0.6 


Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 

Primary 

Database 

Backup 

ASM 


SQL 

apply 

Transform 

Redo to SQL 

• Plan for switch over to LSTBY 

• Stop the redo transmission to PSTBY 

• Up to this point absolutely there is no 

impact on the availability of the 

primary database. 

• Switch over 

• Re-point the clients, services 

Broker 

Sync or Async 

Redo shipping 


11.2.0.3 

Redo Logs 

Online 

Redo Logs 

SQL 

apply 



DR Site 


Database 

ASM

Phase - 2 

Upgrade Standbys & Switch Back


11.1.0.6 


Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 


Database 

Backup 

ASM 


SQL 

apply 

Transform 

Redo to SQL 

• LSTBY (previous Primary) is in a 

Transient state 

• It is Running lower version than primary 

• Cannot understand/apply changes 

received from the higher version of 

primary database 

• Cannot provide disaster recovery 

Broker 

Sync or Async 

Redo shipping 


11.2.0.3 

Online 

Redo Logs 


DR Site 

Primary 

Database 

ASM


11.1.0.6 


Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 

Physical Logical Standby 

Database 

Backup 

ASM 


Redo 

SQL 

apply 

apply 

Transform 

Redo to SQL 

• Flashback LSTBY to guaranteed restore 

point and shut down 

• Install new oracle version 

• Mount database from new home and 

convert LSTBY to PSTBY 

• When Redo Apply starts, it registers a new 

incarnation from the primary database 

Broker 

Sync or Async 

Redo shipping 


11.2.0.3 

Online 

Redo Logs 


DR Site 

Primary 

Database 

ASM


11.1.0.6 11.2.0.3 


Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 


Database 

Backup 

ASM 


Redo 

apply 

Online Redo Logs 

Redo Logs 

• Wait for physical standby to be upgraded 

via the redo stream 

• At this point you have a second 

interruption to the database availability 

• Switch Back Primary to its Original 

location 

• Re-point the clients accordingly 

Broker 

Sync or Async 

Redo shipping 


11.2.0.3 

Redo Online Logs 

Redo Logs 

Redo 

apply 


DR Site 

Primary 

Database 

ASM


11.2.0.3 


Sync or Async 

Redo shipping 

11.1.0.6 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 

Primary 

Database 

Backup 

ASM 


Online 

Redo Logs 

Broker 

Sync or Async 

Redo shipping 


11.2.0.3 

• Install new database home on other 

Redo 

apply 

standbys 

Redo Logs 

• Mount PSTBY from new home 

• Flashback database to guaranteed 

restore point Oracle Home (11.2.0.3) 


DR Site 


Database 

ASM


11.2.0.3 


Sync or Async 

Redo shipping 

11.1.0.6 11.2.0.3 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 

Primary 

Database 

Backup 

ASM 


Online 

Redo Logs 

• Enable the redo transmission to 

local PSTBY to catch up to new 

incarnation 

• PSTBY will be upgraded via redo 

stream 

• Enable Broker 

Broker 

Sync or Async 

Redo shipping 


11.2.0.3 

Redo Logs 

Redo 

apply 


DR Site 


Database 

ASM


11.2.0.3 


Sync or Async 

Redo shipping 

11.2.0.3 

Redo Logs 

Redo 

apply 

Local Site 


Database 

ASM 

Primary 

Database 

Backup 

ASM 


Online 

Redo Logs 

Broker 

Sync or Async 

Redo shipping 


11.2.0.3 

• Drop guaranteed restore points 

Redo 

apply 

• Rolling upgrade has Finished and 

Redo Logs 

databases are back in the original 

positions 

• With very minimal interruptions Oracle Home (11.2.0.3) 


DR Site 


Database 

ASM

Pros & Cons 

Regular /Conventional Upgrade 

• Upgrade using DBUA/CLI 

(50mins) 

• Upgrade using datapump - 

Complete Down time required 

for exports, scp & imports 

(approx 3 hrs) 

– Manually recreate standbys 

after the upgrade 

• Errors at any stage can 

increase down time 

Rolling Upgrade using Transient LSTBY 

• Down time is required for only 

switchover (reduced by 98% - 

5mins) 

• Standbys are automatically 

upgraded 

• Fail safe options at each step 

• Down time is independent of 

database size

Caveats 

• Need to upgrade grid/ASM before the database upgrade 

• Need to manually update the OCR (srvctl) registries 

• If Upgrading from 11gR1 to 11gR2 then move the database 

spfile from $OH/dbs to ASM - if not, DBUA will fail 

• If the “compatible” parameter is changed after LSTBY upgrade 

- database cannot be rolled back 

• To change the “compatible” parameter after the upgrade 

- another downtime is needed

References 

• http://www.oracle.com/technetwork/database/features/availability/maawp-11g-transientlogicalrollingu-1-131927.pdf 

• http://www.oracle.com/technetwork/database/features/availability/maawp-11g-upgrades-made-easy-131972.pdf

Over to Larry 

Thank you

Just How Many Steps Is That? 

1. Disable the Broker configuration 

2. Stop the Broker at the Primary 

3. Stop the Broker at the Standbys 

4. Configure the Redo transport parameters 

on the target standby 

5. Configure the Redo transport parameters 

on bystander standbys 

6. Set the LOG_FILE_NAME_CONVERT 

parameter to clear logfiles 

7. Create a guaranteed restore point on the 

primary database 

8. Create a guaranteed restore point on the 

standby database 

9. Cancel Managed Recovery 

10. Execute DBMS_LOGSTDBY.BUILD at 

primary 

11. Convert the standby to a Transient 

Logical standby using KEEP IDENTITY 

12. Open the Transient Logical standby 

13. Start SQL Apply 

14. Turn off automatic archive log delete 

15. Synchronize 

30 



But Wait! There’s More! 

16. Stop SQL Apply 

17. Create a 2 nd guaranteed restore point 

18. Stop Redo Transport 

19. Upgrade the Transient Logical standby 

20. Enable Redo Transport 

21. Restart SQL Apply 


23. Start Switchover at Primary 

24. Complete Switchover at standby 

25. Open new Primary 

26. Stop redo transport 

27. Flashback the original Primary 

28. Mount original Primary in new home 

29. Convert it to a physical standby 

30. Shutdown 

31. Mount new Physical Standby 

32. Start Redo Apply 

33. Enable redo transport 


35. Start switchover on new primary 

31 



And finally 

36. Complete switchover at standby 

37. Open new Primary 

38. Restart old Primary as a standby 

39. Start Managed Recovery 

40. Drop 3 guaranteed restore points 

41. Enable the Broker Configuration 

32 



Agenda 





33 



DBMS_ROLLING: Concepts 

• Rolling changes can be applied on the whole Data Guard configuration 

– Three stages (Specification – Compilation – Execution) 

– Execution has three stages: Start, Switchover, Finish 

• Two key groups 

– Leading Group 

• Changes are applied first (before Switchover) 

• Has a Master Database (the future primary database) 

– Trailing Group 

• Changes are applied last (after Switchover) 

• Has a Master Database (the original primary) 

34 



DBMS_ROLLING: Concepts 

• Leading Group 

– Need to identify the Master Database (LGM) during Specification 

– LGM starts as a Physical Standby, converted into a Logical Standby 

(START), and then becomes the primary database (SWITCHOVER) 

– Other databases in the Leading Group protect the LGM 

– LGM responsibility transferrable on failure 

• Trailing Group 

– Contains the original primary database (Trailing Group Master) 

– Other databases in the Trailing Group protect the TGM 

– TGM responsibility transferrable on failure 

35 



Data Guard Simple Rolling Upgrades 

Simple, Reliable, Repeatable Database Rolling Upgrades 

Start Switchover Finish 

36 



Goal 

Simple, Reliable, Repeatable 

• Early problem detection 

• Dedicated interface: DBMS_ROLLING PL/SQL package 

• Centralized, simplified, and uniform execution 

• Fault tolerance 

• Configuration rollback 

• Centralized monitoring: DBA_ROLLING views 

37 



Upgrading The Database Software 

• The Data Guard Simple Rolling Upgrades procedure will 

be usable for database version upgrades starting with 

the first patchset of Oracle Database12c. 

– To upgrade from previous Oracle Database11g releases to 

Oracle Database12c and to move from the first release of Oracle 

Database 12c to the first patchset you will need to use the 

classic Transient Logical Standby upgrade procedure. 

• The procedure can be used for other database 

maintenance tasks with the first release 

38 



What Kind Of Changes Can Be Made? 

First Release Examples 

• Add Partitioning to Non-Partitioned Tables 

• Change Basicfile LOBs to Securefile LOBs 

• Change XML-CLOB to Binary XML 

• Alter table to be OLTP-compressed 

39 



New Data Type Support 

Eliminating Barriers to Utilizing Database Rolling Upgrade 

• Advanced Data Type Support 

– XML OR 

– Binary XML 

– Securefile Deduplication 

– Objects/Collections (includes SDO_GEOMETRY, DICOM) 

– New Support for Oracle-Supplied Features 

• DBFS 

• AQ 

• Oracle Text 

• XDB 

40 



Process – Planning & Setup Phase 

• Generate an upgrade plan 

– Call DBMS_ROLLING.INIT_PLAN 

• Generates an upgrade plan with a configuration specific set of instructions to 

guide the administrator through the upgrade process 

– Call DBMS_ROLLING.SET_PARAMETER 

• Modify parameters of the rolling upgrade 

• Prepare your changes to the database 

41 



Process - Execution Phase 

• Call DBMS_ROLLING.START_PLAN 

– Configures primary and standby databases participating in the upgrade 

• Make changes to the standby database 

• Call DBMS_ROLLING.SWITCHOVER 

– Swaps roles between current primary and new primary with the changes, switchover is only 

downtime required 

• Call DBMS_ROLLING.FINISH_PLAN 

– Completes upgrade of the old primary and bystanders and resynchronizes with the new primary 

42 



A Sample Walkthrough - Planning Phase 

Current Configuration 

• Primary database in Boston 

• Physical standby database in 

Cambridge (SYNC) 

• 2 nd Physical standby database in 

Chicago (ASYNC) 

• Broker Controlled 

DGMGRL> SHOW CONFIGURATION 

Configuration – OOW2012 

Protection Mode: MaxAvailability 

Databases: 

bos - Primary database 

cam - Physical standby database 

chi - Physical standby database 

Fast-Start Failover: DISABLED 

Configuration Status: 

SUCCESS 

DGMGRL> 

43 




Start the Planning Process 

• Connect to the Primary 

• Execute the INIT_PLAN 

procedure 

• Sets up default parameters 

– 2 for each database 

– 25 general parameters 

SQL> exec dbms_rolling.init_plan(‘cam’); 

PL/SQL procedure successfully completed. 

SQL> select name, curval from 

2 dba_rolling_parameters where scope is null; 

NAME 

CURVAL 

------------------------ ------------------------ 

ACTIVE_SESSIONS_TIMEOUT 3600 

ACTIVE_SESSIONS_WAIT 0 

BACKUP_CTLFILE 

rolling_change_backup.f 

DICTIONARY_LOAD_TIMEOUT 3600 

DICTIONARY_LOAD_WAIT 0 

EVENT_RECORDS 10000 

FAILOVER 0 

GRP_PREFIX 

DBMSRU_ 

… 

UPDATED_TGS_WAIT 1 

25 rows selected. 

44 




Reconfigure Redo Transport 

• Create Redo Transport 

parameters at target database 

– In our example, on CAM to point 

to BOS and CHI 

• Required to complete the process 

after the switchover as the Broker 

is disabled. 

SQL> alter system set log_archive_dest_5='service="BOS", 

SYNC reopen=30 db_unique_name="BOS" 

valid_for=(PRIMARY_ROLE, ONLINE_LOGFILE)'; 

System altered. 

SQL> alter system set log_archive_dest_6='service="CHI", 

ASYNC reopen=30 db_unique_name="CHI" 

valid_for=(PRIMARY_ROLE, ONLINE_LOGFILE)'; 

System altered. 

SQL> 

45 




Examining Parameters of the Plan 

• Example “INVOLVEMENT” 

• Each database protects the 

Primary when in the Standby role 

– BOS Primary 

• CAM and CHI protect BOS 

– CAM Primary 

• BOS and CHI protect CAM 

– CHI Primary 

• BOS and CAM protect CHI 

SQL> select scope, name, curval 

2 from dba_rolling_parameters 

3 where scope is not null order by scope; 

SCO NAME 

CURVAL 

--- --------------- --------------- 

bos INVOLVEMENT FULL 

bos PROTECTS NONE 

cam INVOLVEMENT FULL 

cam PROTECTS PRIMARY 

chi INVOLVEMENT FULL 

chi PROTECTS PRIMARY 

6 rows selected. 

SQL> 

46 




Build the Plan when ready 

• Validates plan parameters 

• Creates an upgrade plan 

– Successful build required in order 

to perform a rolling upgrade 

• Parameter changes require that 

the procedure is called again 

• Complete plan can be reviewed in 

DBA_ROLLING_PLAN 

SQL> select revision, status, phase 

2 from dba_rolling_status; 

REVISION STATUS PHASE 

---------- ------------ -------------- 

0 READY BUILD PENDING 

SQL> exec dbms_rolling.build_plan(); 





---------- ------------ -------------- 

1 READY START PENDING 

47 



A Sample Walkthrough - Start Phase 

Start the Process 

• Must be executed on the primary 

database 

– Formally starts the rolling upgrade 

operation. 

– When complete, the configuration 

possesses a fully configured 

transient logical standby 

database. 

• Still no outage 

SQL> exec dbms_rolling.start_plan(); 


SQL> select dbun, role, engine_status, 

2 update_progress from dba_rolling_databases; 

DBUN ROLE ENGINE_STATUS UPDATE_PROG 

---------- --------- -------------- ----------- 

bos PRIMARY NOT APPLICABLE NOT STARTED 

cam LOGICAL RUNNING NOT STARTED 

chi PHYSICAL RUNNING NOT STARTED 




---------- ------------ -------------- 

1 READY SWITCH PENDING 

48 



A Sample Walkthrough - Start Phase 

Perform the changes 

• Make your changes to the database 

– For example, move a table from LOBs to Securefile LOBs 

• Stop SQL Apply 

• Make the changes to the Transient Logical Standby 

• Restart SQL Apply 

49 



A Sample Walkthrough - Switchover Phase 

Switchover! 

• Prior to the switchover you can 

move read clients to the transient 

logical standby to avoid an outage 

• Update users will be stalled and 

eventually must reconnect to the 

new primary using standard client 

failover best practices. 

• Outage should be no more than 

90-120 seconds for the database 

SQL> exec dbms_rolling.switchover(); 





---------- --------- -------------- ----------- 

bos LOGICAL STOPPED NOT STARTED 

cam PRIMARY NOT APPLICABLE NOT STARTED 

chi PHYSICAL STOPPED NOT STARTED 




---------- ------------ -------------- 

1 READY FINISH PENDING 

50 



A Sample Walkthrough - Finish Phase 

Finish the change 

• Manually restart and mount the all 

other databses 

• Execute the FINISH Procedure to 

finalize the rolling upgrade 

operation. 

– Configures the former primary as 

a physical standby, and configures 

remaining physical standbys to 

recover the change redo from the 

new primary. 

SQL> exec dbms_rolling.finish_plan(); 





---------- --------- -------------- ----------- 

bos PHYSICAL RUNNING FINISHED 

cam PRIMARY NOT APPLICABLE FINISHED 

chi PHYSICAL RUNNING FINISHED 




---------- ------------ -------------- 

1 READY DONE 

51 



A Sample Walkthrough - Finish Phase 

Re-enable the Broker and switchover 

• Enable the Broker 

– It reconfigures itself to the new 

Primary database ‘CAM’. 

• Execute a Broker switchover 

– If required to return to the original 

configuration. 

DGMGRL> enable configuration; 

Enabled. 

DGMGRL> show configuration 

Configuration – OOW2012 

Protection Mode: MaxAvailability 

Databases: 

cam - Primary database 

bos - Physical standby database 

chi - Physical standby database 

Fast-Start Failover: DISABLED 

Configuration Status: 

SUCCESS 

DGMGRL> switchover to bos; 

Performing switchover NOW, please wait... 

… 

Switchover succeeded, new primary is "bos" 

52 



Agenda 





53 



DBMS_ROLLING: Key Features 

• Specify – Compile – Execute Protocol 

– Catch configuration errors at Compilation step 

– Run-time errors are detected during Execution 

• State is kept in the database 

– Provides robustness 

• Run-time steps are constant 

– Regardless of how many databases are involved 

• Handles failure at the original primary database 

• Allows Data Protection for the upgraded primary right from the start 

54 



Behind the Scenes 

Current Structure 

New Structure 

Sets guaranteed restore point 

(GRP) on A and B 

Database A 

PROD 

Synchronize – redo apply 

Database B 

Physical 

STBY 

Converts B to transient logical 

PROD 

Synchronize – sql apply 

Logical 

STBY 

Switchover 

Physical 

STBY 

SWITCHOVER 

PROD 

Flashback A to GRP, upgrade 

via redo stream 

Physical 

STBY 

Synchronize – redo apply 

PROD 

55 



Now How Many Steps? 

• Prepare for maintenance 

– Disable the Broker, INIT_PLAN, BUILD_PLAN, Configure Redo transport 

• START_PLAN 

• Perform Maintenance 

• SWITCHOVER 

• Mount original Primary 

• FINISH_PLAN 

– Enable the Broker, Switchover 

56 



Resources 

• OTN HA Portal: 

http://www.oracle.com/goto/availability 

• Maximum Availability Architecture (MAA): 

http://www.oracle.com/goto/maa 

• MAA Blogs: 

http://blogs.oracle.com/maa 

• Exadata on OTN: 

http://www.oracle.com/technetwork/database/exadata/index.html 

• Oracle HA Customer Success Stories on OTN: 

http://www.oracle.com/technetwork/database/features/ha-casestudies- 

098033.html

Graphic Section Divider 

58

Presentation - Oracle

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?