Avamar for SAP - EMC Community Network
Avamar for SAP - EMC Community Network
Avamar for SAP - EMC Community Network
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong><br />
with <strong>EMC</strong> <strong>Avamar</strong><br />
A Detailed Review<br />
<strong>EMC</strong> In<strong>for</strong>mation Infrastructure Solutions<br />
Abstract<br />
This white paper describes how the backup and restore operation of <strong>SAP</strong> systems benefit from client-based<br />
deduplication technology using <strong>EMC</strong> ® <strong>Avamar</strong> ® . The white paper also addresses how customers can use the <strong>EMC</strong><br />
NetWorker ® PowerSnap module to reduce the per<strong>for</strong>mance impact on the <strong>SAP</strong> production server during backup.<br />
July 2010
Copyright © 2010 <strong>EMC</strong> Corporation. All rights reserved.<br />
<strong>EMC</strong> believes the in<strong>for</strong>mation in this publication is accurate as of its publication date. The in<strong>for</strong>mation is subject to<br />
change without notice.<br />
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” <strong>EMC</strong> CORPORATION MAKES NO<br />
REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS<br />
PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR<br />
FITNESS FOR A PARTICULAR PURPOSE.<br />
Use, copying, and distribution of any <strong>EMC</strong> software described in this publication requires an applicable software<br />
license.<br />
For the most up-to-date listing of <strong>EMC</strong> product names, see <strong>EMC</strong> Corporation Trademarks on <strong>EMC</strong>.com<br />
All other trademarks used herein are the property of their respective owners.<br />
Part number: H7052<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
2
Table of Contents<br />
Executive summary ........................................................................................................................... 5<br />
Business case ............................................................................................................................... 5<br />
Product solution............................................................................................................................. 5<br />
Key results ..................................................................................................................................... 5<br />
Introduction........................................................................................................................................ 6<br />
Overview ....................................................................................................................................... 6<br />
Purpose ......................................................................................................................................... 6<br />
Scope ............................................................................................................................................ 6<br />
Audience ....................................................................................................................................... 6<br />
Terminology ................................................................................................................................... 6<br />
Key components ............................................................................................................................... 8<br />
Introduction .................................................................................................................................... 8<br />
<strong>EMC</strong> Symmetrix VMAX ................................................................................................................. 8<br />
<strong>EMC</strong> <strong>Avamar</strong> ................................................................................................................................. 8<br />
<strong>EMC</strong> NetWorker PowerSnap Module <strong>for</strong> Symmetrix .................................................................... 8<br />
<strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle ............................................................................... 8<br />
VMware vSphere ........................................................................................................................... 9<br />
IDES <strong>SAP</strong> ERP ............................................................................................................................. 9<br />
Data Protection Advisor ................................................................................................................ 9<br />
Physical architecture ....................................................................................................................... 10<br />
Overview ..................................................................................................................................... 10<br />
Physical environment .................................................................................................................. 10<br />
Hardware resources .................................................................................................................... 11<br />
Software resources ..................................................................................................................... 11<br />
Configuration and design ................................................................................................................ 12<br />
Introduction .................................................................................................................................. 12<br />
Key considerations ...................................................................................................................... 12<br />
LUN design .................................................................................................................................. 13<br />
<strong>SAP</strong> application server configuration .......................................................................................... 14<br />
PowerSnap specific configuration ............................................................................................... 15<br />
Data load configuration ............................................................................................................... 16<br />
Testing and validation ..................................................................................................................... 17<br />
Introduction .................................................................................................................................. 17<br />
Scenarios overview ..................................................................................................................... 17<br />
Test group 1: Baseline with BCT off............................................................................................ 18<br />
Test group 2: <strong>Avamar</strong> on with BCT off ........................................................................................ 18<br />
Test group 3: Baseline with BCT on............................................................................................ 20<br />
Test group 4: <strong>Avamar</strong> on with BCT on ........................................................................................ 21<br />
Results validation ........................................................................................................................ 22<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
3
<strong>Avamar</strong> compared to baseline .................................................................................................... 22<br />
<strong>Avamar</strong> compared to BCT ........................................................................................................... 24<br />
Conclusion....................................................................................................................................... 28<br />
Introduction .................................................................................................................................. 28<br />
Client-based deduplication by <strong>Avamar</strong>........................................................................................ 28<br />
<strong>Avamar</strong> and Oracle BCT ............................................................................................................. 28<br />
Automation of snapshot-based backup using PowerSnap ......................................................... 28<br />
Centralized management with NetWorker integration ................................................................ 28<br />
References ...................................................................................................................................... 29<br />
White papers ............................................................................................................................... 29<br />
Product documentation ............................................................................................................... 29<br />
Other documentation ................................................................................................................... 29<br />
Additional in<strong>for</strong>mation ...................................................................................................................... 30<br />
Introduction .................................................................................................................................. 30<br />
Baseline - BR*Tools configuration file <strong>for</strong> RMAN ........................................................................ 30<br />
Baseline - BR*Tools configuration file <strong>for</strong> PowerSnap ................................................................ 31<br />
Baseline - NM<strong>SAP</strong> configuration file <strong>for</strong> RMAN .......................................................................... 32<br />
Baseline - NM<strong>SAP</strong> configuration file <strong>for</strong> PowerSnap .................................................................. 33<br />
<strong>Avamar</strong> - BR*Tools configuration file <strong>for</strong> RMAN ......................................................................... 33<br />
<strong>Avamar</strong> - BR*Tools configuration file <strong>for</strong> PowerSnap ................................................................. 35<br />
<strong>Avamar</strong> - NM<strong>SAP</strong> configuration file <strong>for</strong> RMAN ............................................................................ 36<br />
<strong>Avamar</strong> - NM<strong>SAP</strong> configuration file <strong>for</strong> PowerSnap .................................................................... 37<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
4
Executive summary<br />
Business case<br />
<strong>SAP</strong> data growth can be as much as 60 percent per year. In many cases there are<br />
multiple <strong>SAP</strong> solutions with production and non-production data growing<br />
continuously. This situation is driving <strong>SAP</strong> customers to revisit their backup and<br />
recovery strategies as traditional backup systems are being pushed to capacity.<br />
The <strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong> – Enabled by NetWorker Module <strong>for</strong> <strong>SAP</strong><br />
with <strong>Avamar</strong> solution is designed to enable <strong>SAP</strong> customers to achieve new levels of<br />
data reduction and provide fast, secure backup <strong>for</strong> their production and nonproduction<br />
<strong>SAP</strong> environments. In the process, <strong>SAP</strong> backup time and growth of<br />
secondary storage are reduced, along with decreased network utilization. In case of<br />
data corruption, users can recover the system in minimal time by using full backup.<br />
Product<br />
solution<br />
This solution demonstrates the benefits of client-based deduplication in an<br />
<strong>SAP</strong>/Oracle environment on the <strong>EMC</strong> ® Symmetrix ® plat<strong>for</strong>m as well as the benefits<br />
of integrating <strong>EMC</strong> PowerSnap and the <strong>EMC</strong> NetWorker ® Module <strong>for</strong> <strong>SAP</strong> with<br />
Oracle (NM<strong>SAP</strong>) using the <strong>EMC</strong> <strong>Avamar</strong> ® plug-in and NetWorker.<br />
The solution builds a replica of a common customer environment and validates the<br />
configuration <strong>for</strong> both per<strong>for</strong>mance and functionality of client-based deduplication<br />
enabled by <strong>Avamar</strong>.<br />
NetWorker is used as the backup management tool to per<strong>for</strong>m the multi-stream<br />
backup to <strong>Avamar</strong>. Through interaction with the PowerSnap module, NetWorker<br />
controls the process of snapshot-based backup and eliminates the per<strong>for</strong>mance<br />
impact of the backup job on the production server being backed up.<br />
<strong>EMC</strong> Data Protection Advisor (DPA) collects relevant backup in<strong>for</strong>mation. In this<br />
solution, we will show how you can use this analysis tool <strong>for</strong> data collection, data<br />
recording, and backup metrics management.<br />
Key results<br />
This white paper demonstrates the following benefits <strong>for</strong> the solution:<br />
• <strong>Avamar</strong> reduces the network utilization of the backup jobs by 57 percent while<br />
achieving a deduplication ratio of 10:1.<br />
• NetWorker PowerSnap enables the client-based deduplication without additional<br />
CPU utilization on the production host.<br />
• Multi-streaming with NM<strong>SAP</strong> enabled by NetWorker allows the backup job to finish<br />
over two and a half times faster than if per<strong>for</strong>med with a single stream.<br />
• Restoration from <strong>Avamar</strong> taking advantage of multi-streaming with NM<strong>SAP</strong><br />
enabled by NetWorker is six times faster than Recovery Manager (RMAN) with<br />
BCT.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
5
Introduction<br />
Overview<br />
This white paper clarifies the benefits of client-based deduplication and mirror-based<br />
backups <strong>for</strong> <strong>SAP</strong> landscapes. It validates the feasibility of implementing a<br />
deduplication approach in a typical <strong>SAP</strong> environment.<br />
The reference architecture presented in this white paper will help you prepare a<br />
backup and recovery plan and topology <strong>for</strong> <strong>SAP</strong> systems.<br />
Purpose<br />
The purpose of the white paper is to:<br />
• Provide an understanding of client-based deduplication <strong>for</strong> backup of <strong>SAP</strong> systems<br />
• Supply an overview of the snapshot-based backup method and explain the reason<br />
<strong>for</strong> using this backup method<br />
• Demonstrate the integration capability of NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle<br />
(NM<strong>SAP</strong>), PowerSnap, and <strong>Avamar</strong> with central control by NetWorker<br />
Scope<br />
This white paper documents the software and hardware resources involved in this<br />
solution as well as the corresponding configuration <strong>for</strong> building the test environment.<br />
The white paper also addresses the solution design, the test scenario pattern, and<br />
the conclusion based on test results and analysis.<br />
It is assumed the reader has knowledge and experience using BR*Tools <strong>for</strong> backup<br />
and recovery of <strong>SAP</strong> with Oracle databases. Detailed explanations of these concepts<br />
are not in the scope of this paper.<br />
Audience<br />
This white paper is intended <strong>for</strong> customers, partners, and <strong>EMC</strong> employees, including<br />
IT planners, <strong>SAP</strong> architects, Basis administrators, and any other people involved in<br />
evaluating, acquiring, managing, operating, or designing an <strong>SAP</strong> landscape<br />
infrastructure.<br />
Terminology<br />
This section defines terms used in this document.<br />
Term<br />
DEV<br />
QAS<br />
PRD<br />
RMAN<br />
BCT<br />
Definition<br />
<strong>SAP</strong> development system<br />
<strong>SAP</strong> quality assurance system, also referred to as test system<br />
<strong>SAP</strong> production system<br />
Recovery Manager, an Oracle built-in tool <strong>for</strong> backup and<br />
restore, which per<strong>for</strong>ms backup at the database level<br />
Block Change Tracking, an Oracle feature that reduces backup<br />
times on incremental backups by tracking changed blocks since<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
6
the last backup on a dedicated file<br />
BR*Tools<br />
Proxy Host<br />
BACKINT<br />
Baseline<br />
backup<br />
LSMW<br />
PIT recovery<br />
Solution<br />
Enabler<br />
Toolset that the <strong>SAP</strong> <strong>for</strong> Oracle database administration (DBA)<br />
tasks provide<br />
Also known as Data Mover, the secondary host, from which<br />
PowerSnap implements snapshot-based backup<br />
The external backup library called by BR*Tools to per<strong>for</strong>m <strong>SAP</strong><br />
system backup at the file system level. It is <strong>SAP</strong> certified.<br />
Backup to SATA volumes without using the deduplication<br />
technology<br />
Legacy System Migration Workbench (LSMW), an <strong>SAP</strong> specific<br />
tool written in Advance Business Application Programming<br />
(ABAP) to load data into the system<br />
Point-in-time recovery<br />
<strong>EMC</strong> shared libraries <strong>for</strong> use by Solutions Enabler applications,<br />
and the Symmetrix Command Line Interface (SYMCLI) <strong>for</strong> use by<br />
storage administrators and systems engineers.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
7
Key components<br />
Introduction<br />
This section briefly describes the key solution components. For details on all of the<br />
components that make up the architecture, see the “Physical architecture” section in<br />
this paper.<br />
<strong>EMC</strong><br />
Symmetrix<br />
VMAX<br />
<strong>EMC</strong> Symmetrix VMAX provides high-end storage <strong>for</strong> the virtual data center.<br />
Symmetrix VMAX scales up to 2 PB of usable protected capacity and consolidates<br />
more workloads with a much smaller footprint than alternative arrays.<br />
The innovative <strong>EMC</strong> Symmetrix Virtual Matrix Architecture of VMAX seamlessly<br />
scales per<strong>for</strong>mance, capacity, and connectivity on demand to meet all application<br />
requirements. Symmetrix VMAX can be deployed with Flash drives, and Fibre<br />
Channel and Serial Advanced Technology Attachment (SATA) drives, with tiering<br />
fully automated with FAST. It supports virtualized and physical servers, including<br />
open systems, mainframe, and IBM i hosts.<br />
<strong>EMC</strong> <strong>Avamar</strong><br />
<strong>EMC</strong> <strong>Avamar</strong> backup and recovery solutions utilize patented global data<br />
deduplication technology to identify redundant data at the source, minimizing backup<br />
data be<strong>for</strong>e it is sent over the LAN/WAN. With <strong>Avamar</strong>, you can achieve new levels<br />
of data reduction and enable fast, secure backup <strong>for</strong> your VMware environments,<br />
remote offices, LAN/NAS servers, and desktop/laptop systems. In the process, you<br />
will reduce backup time, growth of secondary storage, and network utilization.<br />
<strong>Avamar</strong> deduplicated backups function like full backups and can be recovered in just<br />
one step. You do not need to restore full backups and subsequent incremental<br />
backups. In addition, <strong>Avamar</strong> verifies backup data recoverability, so there are no<br />
surprises, and encrypts data <strong>for</strong> secure electronic backups.<br />
<strong>EMC</strong> NetWorker<br />
PowerSnap<br />
Module <strong>for</strong><br />
Symmetrix<br />
The PowerSnap Module enhances NetWorker by allowing snapshot-based data<br />
protection and availability during backups <strong>for</strong> the Symmetrix storage system. The<br />
backup is virtually instantaneous, regardless of the volume size. The PowerSnap<br />
Module software enables frequent backups with little impact on the network or<br />
application server, and provides backups of large databases or data stored in file<br />
systems.<br />
<strong>EMC</strong> NetWorker<br />
Module <strong>for</strong> <strong>SAP</strong><br />
with Oracle<br />
<strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle (NM<strong>SAP</strong>) ensures that the missioncritical<br />
data is available by delivering fast, online backup and recovery <strong>for</strong> <strong>SAP</strong>. With<br />
NM<strong>SAP</strong>, <strong>SAP</strong> administrators can centrally manage backup operations and<br />
schedules.<br />
Featuring data deduplication support through its embedded <strong>Avamar</strong> client, NM<strong>SAP</strong><br />
reduces the impact of data protection operations on production servers and<br />
networks, and significantly reduces the amount of backup storage required.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
8
VMware<br />
vSphere<br />
VMware vSphere is the foundation <strong>for</strong> a dynamic, self-optimizing IT infrastructure. It<br />
is a robust, production-proven virtualization layer that abstracts processor, memory,<br />
storage, and networking resources into multiple virtual machines. VMware vSphere<br />
allows enterprises to dramatically reduce hardware and operating costs by sharing<br />
resources across a virtual environment.<br />
IDES <strong>SAP</strong> ERP <strong>SAP</strong> ERP IDES is the "Internet Demonstration and Evaluation System" in the R/3<br />
System, and represents a model company. It consists of an international group with<br />
subsidiaries in several countries. IDES contains application data <strong>for</strong> various business<br />
scenarios you can run in the <strong>SAP</strong> system. The business processes in the IDES<br />
system are designed to reflect real-life business requirements, and have access to<br />
many realistic characteristics. IDES uses easy-to-follow business scenarios to show<br />
you the comprehensive functions of the R/3 System. The focal point of IDES,<br />
however, is not the functionality itself, but the business processes and their<br />
integration.<br />
Data Protection<br />
Advisor<br />
Data Protection Advisor (DPA) is an advanced backup reporting and analysis<br />
application that gives its users complete visibility into everything happening within<br />
their backup environment. DPA gathers data from backup servers, backup clients,<br />
databases, network switches, SAN switches, and tape libraries. DPA correlates<br />
in<strong>for</strong>mation from each of these areas and provides advanced reporting,<br />
troubleshooting, per<strong>for</strong>mance management, and capacity planning.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
9
Physical architecture<br />
Overview<br />
The following section identifies and briefly describes the technology and components<br />
used in the environment.<br />
Physical<br />
environment<br />
The following diagram illustrates the overall physical architecture of the environment:<br />
The <strong>SAP</strong> DEV and QAS systems, NetWorker server, and DPA server are all built on<br />
the virtual machine managed by the VMware ESX server. NetWorker controls the<br />
backup and restoration processes, and the DPA server captures the data results.<br />
The <strong>SAP</strong> PRD system and the proxy host reside on the physical machines, which<br />
are connected with the ESX server through a LAN <strong>for</strong> communication purposes<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
10
etween NM<strong>SAP</strong>, PowerSnap and the NetWorker server. <strong>EMC</strong> Symmetrix VMAX is<br />
attached to all application servers through the SAN. In addition, the NetWorker<br />
defines <strong>Avamar</strong> as the deduplication node in device configuration.<br />
Hardware<br />
resources<br />
The hardware used to validate the solution is listed in the following table.<br />
Equipment Quantity Configuration<br />
<strong>EMC</strong> VMAX SE 1 52 x 300 GB 15k FC disks<br />
52 x 1 TB 7.2k SATA disks<br />
VMware ESX server 1 Dell PowerEdge R900 16 Core<br />
<strong>SAP</strong> host 4 • DEV: VMware Virtual Machine 2 CPUs, 8 GB RAM<br />
• QAS: VMware Virtual Machine 2 CPUs, 8 GB RAM<br />
• PRD: Dell PowerEdge 6850 16 Core<br />
• Proxy host: Dell PowerEdge R710 16 Core<br />
<strong>EMC</strong> NetWorker server 1 VMware Virtual Machine 2 CPUs, 8 GB RAM<br />
<strong>EMC</strong> DPA server 1 VMware Virtual Machine 2 CPUs, 8 GB RAM<br />
SAN switch 1 Cisco Catalyst 9509<br />
<strong>EMC</strong> <strong>Avamar</strong> Appliance 1 <strong>EMC</strong> <strong>Avamar</strong> Data Store Grid (six nodes) Gen2<br />
Software<br />
resources<br />
The software used to validate the solution is listed in the following table.<br />
Software<br />
Version<br />
Linux Red Hat Enterprise<br />
5.3 AS5 U2 64 bit<br />
Oracle Database 10.2.0.4<br />
<strong>SAP</strong> Application<br />
IDES ERP 6.0 EhP4<br />
<strong>EMC</strong> NetWorker 7.6<br />
<strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle 4.0<br />
<strong>EMC</strong> NetWorker PowerSnap Module <strong>for</strong><br />
Symmetrix DMX /VMAX<br />
2.5<br />
<strong>EMC</strong> Solutions Enabler 7.0.1.0<br />
VMware vSphere Server 4.0 Update 1<br />
<strong>EMC</strong> <strong>Avamar</strong> software 4.1<br />
<strong>EMC</strong> Data Protection Advisor 5.5<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
11
Configuration and design<br />
Introduction<br />
This section provides the configuration in<strong>for</strong>mation <strong>for</strong> this solution, including:<br />
• Disk allocation pattern<br />
• Relevant <strong>SAP</strong> system settings<br />
• Corresponding NetWorker components<br />
• Key considerations or prerequisites <strong>for</strong> the proposed test scenarios<br />
The <strong>SAP</strong> systems presented in this solution follow the principle of the standard<br />
three-system landscape known as DEV, QAS, and PRD.<br />
Key<br />
considerations<br />
<strong>EMC</strong> considered the following when per<strong>for</strong>ming the test scenarios in this solution:<br />
• An Advanced File Type Device is created on SATA disks on VMAX as the<br />
destination <strong>for</strong> baseline backup.<br />
• RMAN is configured as the default backup tool in BR*Tools <strong>for</strong> backing up<br />
database files to backup destination. The database control file, the BR*Tools<br />
profiles, and logs are still backed up through BACKINT.<br />
• BACKINT is specified as the default method <strong>for</strong> backing up all files with<br />
PowerSnap, including database data files, the control file, the BR*Tools profiles,<br />
and logs.<br />
• You do not need to install a separate <strong>Avamar</strong> client. <strong>Avamar</strong> works as a<br />
deduplication node in the NetWorker server and NetWorker manages all backup<br />
streams to <strong>Avamar</strong>.<br />
• By default, <strong>SAP</strong> per<strong>for</strong>ms a cumulative backup when running an incremental<br />
backup job.<br />
• Backup <strong>for</strong> archive logs is not configured <strong>for</strong> the development and testing systems.<br />
The production system is separately configured to run every two hours without<br />
connecting to <strong>Avamar</strong> or PowerSnap.<br />
A baseline backup is established to compare the results with that of the client-based<br />
deduplication backup. To facilitate database size changes, <strong>EMC</strong> used Legacy<br />
System Migration Workbench (LSMW) <strong>for</strong> data population to simulate a realtime data<br />
increase in the customer environment. In this solution, the comparison between<br />
baseline and deduplication backup is only conducted on the production system. <strong>EMC</strong><br />
also used the development and testing systems in this solution to validate the<br />
feasibility of client-based deduplication on virtualized plat<strong>for</strong>ms.<br />
When doing backup in a traditional environment that demands optimized<br />
per<strong>for</strong>mance, you should per<strong>for</strong>m an incremental backup with BCT to reduce the<br />
backup duration and network traffic. There<strong>for</strong>e it is useful in this solution to measure<br />
the per<strong>for</strong>mance difference between BCT and <strong>Avamar</strong>.<br />
Taking the per<strong>for</strong>mance impact on the backup client into consideration, <strong>EMC</strong><br />
developed a series of approaches <strong>for</strong> backing up a client through a mirrored volume.<br />
PowerSnap works together with NetWorker to interact with Solutions Enabler to<br />
accomplish this task. <strong>EMC</strong> recommends that you configure BACKINT as the default<br />
backup method <strong>for</strong> all files when integrating PowerSnap with NM<strong>SAP</strong> and<br />
NetWorker.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
12
LUN design<br />
The following table describes the storage design pattern <strong>for</strong> the <strong>SAP</strong> DEV and QAS<br />
systems built on virtual machines.<br />
RAID<br />
type<br />
Disk<br />
type<br />
LUN size Purpose Filesystem<br />
10 VMFS 64 GB Linux root and<br />
swap<br />
10 RDM 32 GB Original redo log /oracle//origlogA<br />
/oracle//origlogB<br />
10 RDM 32 GB Mirrored redo log /oracle//mirrlogA<br />
/oracle//mirrlogA<br />
5 RDM 64 GB Oracle archive log /oracle//oraarch<br />
/<br />
5 RDM 64 GB x<br />
4<br />
5 RDM 64 GB x<br />
4<br />
5 RDM 64 GB x<br />
4<br />
5 RDM 64 GB x<br />
4<br />
Database data files<br />
Database data files<br />
Database data files<br />
Database data files<br />
/oracle//sapdata1<br />
/oracle//sapdata2<br />
/oracle//sapdata3<br />
/oracle//sapdata4<br />
5 RDM 32 GB <strong>SAP</strong> BIN /sapmnt/<br />
/usr/sap/<br />
/usr/sap/trans<br />
5 RDM 32 GB Oracle BIN /oracle/client<br />
/oracle/stage<br />
/oracle//102_64<br />
The following table describes the storage design pattern <strong>for</strong> the <strong>SAP</strong> PRD system<br />
built on physical machines.<br />
RAID type LUN size Purpose Filesystem<br />
10 64 GB Linux root and swap /<br />
10 32 GB Original redo log /oracle//origlogA<br />
/oracle//origlogB<br />
10 32 GB Mirrored redo log /oracle//mirrlogA<br />
/oracle//mirrlogA<br />
5 64 GB Oracle archive log /oracle//oraarch<br />
5 64 GB x 4 Database data files /oracle//sapdata1<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
13
5 64 GB x 4 Database data files /oracle//sapdata2<br />
5 64 GB x 4 Database data files /oracle//sapdata3<br />
5 64 GB x 4 Database data files /oracle//sapdata4<br />
5 32 GB <strong>SAP</strong> BIN /sapmnt/<br />
/usr/sap/<br />
/usr/sap/trans<br />
5 32 GB Oracle BIN /oracle/client<br />
/oracle/stage<br />
/oracle//102_64<br />
<strong>SAP</strong><br />
application<br />
server<br />
configuration<br />
The solution is validated with the following configuration specified to <strong>SAP</strong> systems,<br />
where PowerSnap is disabled. Configuration of PRD and the proxy host specific to<br />
PowerSnap is outlined separately.<br />
DEV<br />
Profile characteristic<br />
Database size<br />
Value<br />
271 GB<br />
Installed agent<br />
BR*Tools - Backup device type<br />
BR*Tools - Backup type<br />
BR*Tools - Backup mode<br />
NM<strong>SAP</strong><br />
rman_util<br />
online_cons<br />
full<br />
QAS<br />
Profile characteristic<br />
Database size<br />
Installed agent<br />
BR*Tools - Backup device type<br />
BR*Tools - Backup type<br />
BR*Tools - Backup mode<br />
Value<br />
270 GB<br />
NM<strong>SAP</strong><br />
rman_util<br />
online_cons<br />
full<br />
PRD<br />
Profile characteristic<br />
Database size<br />
Installed agent<br />
Value<br />
278 GB<br />
NM<strong>SAP</strong>, PowerSnap (valid in PowerSnap<br />
enabled backup)<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
14
BR*Tools - Backup device type<br />
BR*Tools - Backup type<br />
BR*Tools - Backup mode<br />
rman_util<br />
online_cons<br />
full<br />
Note The meanings of BR*Tools parameters are explained below.<br />
• rman_util<br />
RMAN backup in combination with a backup library and the backup tool of another<br />
manufacturer. BACKINT provides an interface to the external backup tool and is<br />
also used to back up profiles, log files, and the control file.<br />
• online_cons<br />
The backup is per<strong>for</strong>med when the database is open. Both the database files and<br />
the offline redo log files generated during the backup are copied to the same<br />
volume. As a result, you have a logically consistent dataset available.<br />
• Full<br />
Full database backup at level 0. This backup can be part of an incremental backup<br />
strategy.<br />
For more in<strong>for</strong>mation about the explanation of a BR*Tools configuration, refer to <strong>SAP</strong><br />
online help.<br />
Refer to the “Additional in<strong>for</strong>mation” section <strong>for</strong> more in<strong>for</strong>mation about the<br />
configuration <strong>for</strong> the PRD system.<br />
PowerSnap<br />
specific<br />
configuration<br />
The following tables list the configuration characteristics of PRD and the proxy host<br />
using PowerSnap.<br />
PRD<br />
Profile characteristic<br />
Database size<br />
Value<br />
278 GB<br />
Installed agent<br />
BR*Tools - Backup device type<br />
BR*Tools - Backup type<br />
BR*Tools - Backup mode<br />
NM<strong>SAP</strong>, PowerSnap<br />
util_file_online<br />
online_cons<br />
all_data<br />
Note The new value of BR*Tools parameter is explained below.<br />
• util_file_online<br />
Use external backup programs <strong>for</strong> file-by-file backup. Use this parameter value <strong>for</strong><br />
an online backup, if it is supported by the external backup program. This<br />
dynamically sets and ends the backup status of the tablespaces to be saved and<br />
thus greatly reduces the volume of offline redo log files during the backup.<br />
• all_data<br />
Back up the files of all tablespaces except pure index tablespaces or empty<br />
tablespaces.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
15
Proxy host<br />
Profile characteristic<br />
Installed agent<br />
Synchronized volumes<br />
Snapshot type<br />
PowerSnap backup type<br />
Value<br />
PowerSnap<br />
Database data files<br />
BCV<br />
Immediate live backup (serverless snapshot<br />
backup)<br />
Note<br />
• This solution uses BCV emulation because PowerSnap requires this mode. You<br />
can define BCV as the value of the snapshot type <strong>for</strong> the proxy host configuration.<br />
In addition, you can use BCVs <strong>for</strong> local replication and mount them to the proxy<br />
host in this solution.<br />
• The database control file, the BR*Tools profile, and log files are always backed up<br />
directly from the production host.<br />
Immediate Live Backup refers to a process where BCVs are split and immediately<br />
backed up by the PowerSnap module to conventional storage. The BCVs are<br />
automatically synchronized with the source volumes.<br />
For more in<strong>for</strong>mation about the backup/restore definition enabled by NM<strong>SAP</strong> with<br />
PowerSnap, refer to the <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle 4.0<br />
Administration Guide.<br />
Refer to the “Additional in<strong>for</strong>mation” section <strong>for</strong> more in<strong>for</strong>mation about the<br />
configuration <strong>for</strong> the PRD system.<br />
Data load<br />
configuration<br />
Data load simulates the data volume increase and changes generated by day-to-day<br />
business operations in the <strong>SAP</strong> systems. In order to get an acceptable and relatively<br />
stable result, we per<strong>for</strong>med three data loads, as shown in each scenario.<br />
The load pattern is outlined in the table below.<br />
Profile characteristic<br />
<strong>SAP</strong> module where the data is generated<br />
Object to be generated<br />
Value<br />
SD<br />
Sales order<br />
How many records generated each time 100,000<br />
Data volume increase in database by each<br />
load<br />
Approximately 2.7 GB (10% data<br />
increase of total database)<br />
Note<br />
Sales orders were generated randomly to avoid generating records that<br />
would artificially tend to deduplicate well. This random process retrieves data<br />
from over 20,000 possible customers and 10,000 possible items in the<br />
master data and combines them to generate random orders.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
16
Testing and validation<br />
Introduction<br />
This section provides detailed in<strong>for</strong>mation on the test scenarios’ result validation.<br />
The test scenarios were grouped and carried out based on different combinations of<br />
key components’ functionality. The key components refer to client-based<br />
deduplication enabled by <strong>Avamar</strong> and Oracle incremental backup optimized by BCT.<br />
The following statements apply to all tested scenarios:<br />
• The backup stream is directed to the Advance File Type Device (baseline backup)<br />
when <strong>Avamar</strong> is disabled.<br />
• The backup stream is always directed to the <strong>Avamar</strong> deduplication node through<br />
NetWorker when <strong>Avamar</strong> is enabled.<br />
• A full backup is conducted with BCT disabled.<br />
• An incremental backup is conducted with BCT enabled.<br />
• All resulting data is captured by DPA.<br />
Unless otherwise specified, PowerSnap is disabled <strong>for</strong> the PRD system during<br />
backup.<br />
All <strong>SAP</strong> systems are copied using storage cloning <strong>for</strong> rapid refresh and consistency<br />
in the context of all test scenarios.<br />
Scenarios<br />
overview<br />
The test per<strong>for</strong>med on the target system covers both backup and restore operations.<br />
Backup is divided into two stages: Stage 1 means an initial backup, and stage 2<br />
refers to the subsequent full/incremental backup upon completion of each data load.<br />
Data load applies to the production system in particular.<br />
<strong>EMC</strong> processed four groups of tests with different configurations of <strong>Avamar</strong> and BCT<br />
in order to meet the following objectives:<br />
• Measure the achieved deduplication ratio as well as any advantages brought by<br />
<strong>Avamar</strong><br />
• Measure and compare the advantages and disadvantages of using Oracle BCT<br />
with <strong>Avamar</strong> <strong>for</strong> backups<br />
• Observe how PowerSnap improves the backup process <strong>for</strong> the production system<br />
• Compare the efficiency between multi-stream backups and single-stream backups<br />
• Validate <strong>Avamar</strong> support <strong>for</strong> virtualized <strong>SAP</strong> systems<br />
The following table summarizes the different scenarios that were tested in this<br />
solution.<br />
Factor<br />
<strong>Avamar</strong> off<br />
BCT off<br />
Backup stage 1<br />
(Initial)<br />
Backup stage 2<br />
(3 data loads)<br />
Restore<br />
DEV QAS PRD DEV QAS PRD DEV QAS PRD<br />
F F F N/A N/A N/A C C C<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
17
<strong>Avamar</strong> on<br />
BCT off<br />
<strong>Avamar</strong> off<br />
BCT on<br />
<strong>Avamar</strong> on<br />
BCT on<br />
F F F* N/A N/A F/F/F* N/A N/A N/A<br />
N/A N/A F N/A N/A I/I/I N/A N/A N/A<br />
F F F N/A N/A I/I/I N/A N/A C P<br />
Note<br />
F=Full Backup, I=Incremental Backup, C=Complete Restore, P=PIT Restore.<br />
*Backup will be per<strong>for</strong>med twice <strong>for</strong> PRD: one is conducted with RMAN; and<br />
the other is conducted with PowerSnap.<br />
Test group 1:<br />
Baseline with<br />
BCT off<br />
Procedure<br />
Step Action<br />
1 Per<strong>for</strong>med one-time backup <strong>for</strong> DEV, QAS, and PRD.<br />
2 Wiped out the database of the above system and restored from the<br />
existing backup.<br />
Result<br />
SID Scenario CPU<br />
utilization<br />
(%)<br />
<strong>Network</strong><br />
throughput<br />
(MB/s)<br />
Data<br />
transfer<br />
(GB<br />
Backup<br />
source<br />
size<br />
(GB)<br />
Duration<br />
(min)<br />
PRD Backup 4.37 78.55 218.76 215 47.53<br />
SID Scenario Restore target size (GB) Duration (min)<br />
PRD Restore 215 79.1<br />
Note<br />
The above results were captured directly from DPA.<br />
The baseline measurement recorded is used <strong>for</strong> comparison with the following test.<br />
Test group 2:<br />
<strong>Avamar</strong> on with<br />
BCT off<br />
Procedure<br />
Step Action<br />
1 Per<strong>for</strong>med backup <strong>for</strong> DEV and QAS to <strong>Avamar</strong>.<br />
2 Per<strong>for</strong>med backup <strong>for</strong> PRD, which contains two phases:<br />
a. Initial backup (full backup)<br />
b. Subsequent backup (full backup), each with data load with the<br />
predefined pattern<br />
The second phase was repeated three times to achieve a stable and<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
18
easonable result.<br />
Switched the backup stream from multiple to single to see how it<br />
affected the backup speed.<br />
3 a. Switched backup method to BACKINT and enabled<br />
PowerSnap.<br />
b. Executed the same procedure as step 2.<br />
c. Wiped out the database and restore from the latest backup.<br />
Result<br />
The following tables show the key metrics regarding backup process <strong>for</strong> PRD.<br />
• Without PowerSnap<br />
Scenario<br />
CPU<br />
utilization<br />
(%)<br />
<strong>Network</strong><br />
throughput<br />
(MB/s)<br />
Data<br />
transferred<br />
(GB)<br />
Backup<br />
source<br />
size<br />
(GB)<br />
Duration<br />
(min)<br />
Initial 25.13 33.62 77.46 215 39.32<br />
1st data load 26.78 12.38 24.25 218 33.43<br />
2nd data<br />
load<br />
27.82 11.06 20.88 222 32.22<br />
3rd data load 27.45 11.73 22.75 225 33.10<br />
Note<br />
The above results were captured directly from DPA.<br />
Deduplication occurs when the <strong>SAP</strong> system has been initially backed up and has<br />
achieved a 64 percent deduplication rate (roughly, 3:1 ratio). When the data volume<br />
increases, the average deduplication ratio can rise up to 10:1. Compared to the<br />
baseline backup, the network utilization drops by 57 percent, and the CPU utilization<br />
increases by 21 percent.<br />
• Without PowerSnap – Single-stream mode<br />
CPU<br />
utilization<br />
(%)<br />
<strong>Network</strong><br />
throughput<br />
(MB/s)<br />
Data<br />
transferred<br />
(GB)<br />
Backup<br />
source size<br />
(GB)<br />
Duration<br />
(min)<br />
10.33 5.21 26.21 228 85.87<br />
Note<br />
The above results were captured directly from DPA.<br />
Working in single-stream mode does not fully utilize the network bandwidth and CPU<br />
recourses, resulting in longer backup duration over the multi-stream mode. The multistream<br />
backup finishes in less than half the time of the single-stream job.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
19
• With PowerSnap<br />
Scenario<br />
CPU<br />
utilization<br />
(%) on<br />
PRD<br />
CPU<br />
utilization<br />
(%) on<br />
proxy<br />
host<br />
<strong>Network</strong><br />
throughput<br />
(MB/s)<br />
Data<br />
transfer<br />
(GB)<br />
Backup<br />
source<br />
size<br />
(GB)<br />
Duration<br />
(min)<br />
Initial 0.28 18.91 28.56 75.05 321 44.85<br />
1st data<br />
load<br />
2nd data<br />
load<br />
3rd data<br />
load<br />
0.19 23.55 1.15 2.03 321 30.15<br />
0.18 23.59 1.11 1.96 321 30.20<br />
0.25 24.63 1.23 2.17 321 30.13<br />
SID Scenario Restore target size (GB) Duration (min)<br />
PRD Restore 321 49.57<br />
Note<br />
The above results were captured directly from DPA.<br />
The deduplication ratio of initial backup in PowerSnap mode is around 4:1. It<br />
increases sharply to 100:1 on average, with each data load.<br />
In the context of the PowerSnap backup, the proxy host takes over the workload of<br />
both the CPU and the network from the production host. The CPU utilization on the<br />
PRD host stays under 1 percent throughout the backup job (as expected).<br />
Backup using an external library (BACKINT) takes full advantage of client-based<br />
deduplication by transferring data through file-system-level copy. The amount of<br />
transferred data is almost identical to the size data volume increases in the database<br />
along with data load. The backup source size remains the same as there is sufficient<br />
free space in the database to accommodate the appended data.<br />
The restoration with BACKINT in PowerSnap mode is 1.6 times faster than the<br />
baseline.<br />
Test group 3:<br />
Baseline with<br />
BCT on<br />
Procedure<br />
Step Action<br />
1 Per<strong>for</strong>med initial backup <strong>for</strong> PRD with BCT turned on.<br />
2 Per<strong>for</strong>med data load with the same pattern as step 2 in the previous<br />
group. The subsequent backup with each data load was run in the<br />
incremental mode.<br />
3 Repeated the above step three times.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
20
Result<br />
Scenario<br />
CPU<br />
utilization<br />
(%)<br />
<strong>Network</strong><br />
throughput<br />
(MB/s)<br />
Data<br />
transfer<br />
(GB)<br />
Backup<br />
target<br />
size (GB)<br />
Duration<br />
(min)<br />
Initial 3.32 60.54 217.38 215 61.28<br />
1 st data load 0.41 36.32 4.75 4.16 2.23<br />
2 nd data load 1.19 43.05 7.95 7.4 3.15<br />
3 rd data load 1.97 44.27 10.82 10 4.17<br />
Note<br />
The above results were captured directly from DPA.<br />
Data transferred is slightly greater than the backup source size, because data<br />
transmission over the network has overhead on the carrier in the TCP/IP layer.<br />
Incremental backup enhanced by BCT shows outstanding per<strong>for</strong>mance in terms of<br />
the backup length. BCT reduces the backup duration because it only backs up the<br />
changed data since the last backup.<br />
Test group 4:<br />
<strong>Avamar</strong> on with<br />
BCT on<br />
Procedure<br />
Step Action<br />
1 Per<strong>for</strong>med backup <strong>for</strong> DEV and QAS to <strong>Avamar</strong> with BCT turned on.<br />
2 Per<strong>for</strong>med initial backup <strong>for</strong> PRD to <strong>Avamar</strong> with BCT turned on.<br />
3 Per<strong>for</strong>med data load with the same pattern as step 2 in the previous<br />
group. The subsequent backup with each data load was run in the<br />
incremental mode.<br />
4 Repeated the above step three times.<br />
5 Wiped out the PRD database and restored it from the most recent full<br />
backup plus incremental backup.<br />
6 Per<strong>for</strong>med a PIT restoration based on the previous step to revert the <strong>SAP</strong><br />
system to an earlier status.<br />
Result<br />
Scenario<br />
CPU<br />
utilization<br />
(%)<br />
<strong>Network</strong><br />
throughput<br />
(MB/s)<br />
Data<br />
transfer<br />
(GB)<br />
Backup<br />
target size<br />
(GB)<br />
Duration<br />
(min)<br />
Initial 22.56 29.58 75.48 215 43.55<br />
1 st data load 11.95 10.16 1.68 4.19 2.82<br />
2 nd data load 16.08 16.78 3.17 7.41 3.22<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
21
3 rd data load 16.43 19.78 4.75 10 4.1<br />
Note<br />
The above results were captured directly from DPA.<br />
Incorporating <strong>Avamar</strong> with BCT results in a 3:1 deduplication ratio <strong>for</strong> the initial<br />
backup and 2:1 on average <strong>for</strong> the subsequent incremental backup.<br />
The backup source size is similar to the one as documented in group 3. The only<br />
difference is that <strong>Avamar</strong> deduplicates the data be<strong>for</strong>e transferring it, which reflects<br />
the actual increased size of the database. This test shows that using NM<strong>SAP</strong> with<br />
the <strong>Avamar</strong> plug-in in conjunction with Oracle BCT enables faster incremental<br />
backups (15 times faster than the <strong>Avamar</strong>-enhanced full backup) with a reduced<br />
network utilization and storage footprint.<br />
SID Scenario Restore target size (GB) Duration (min)<br />
PRD Complete Restore 223 313.95<br />
PRD PIT Restore 215 300.27<br />
Note<br />
The above results were captured directly from DPA.<br />
The table shows that the restoration with BCT from <strong>Avamar</strong> is far slower than the<br />
baseline. There<strong>for</strong>e, this test scenario is not practical because it is too timeconsuming<br />
to recover the <strong>SAP</strong> production system.<br />
Results<br />
validation<br />
The following sections consolidate the test results and provide an in-depth analysis<br />
across test scenarios.<br />
<strong>Avamar</strong><br />
compared to<br />
baseline<br />
The <strong>SAP</strong> system backup can benefit from <strong>Avamar</strong> in terms of reducing the backup<br />
storage footprint as well as improving per<strong>for</strong>mance.<br />
The following section compares the configuration of baseline backup with that of an<br />
<strong>Avamar</strong> backup (BACKINT with <strong>Avamar</strong>).<br />
Backup storage footprint<br />
The delta size between two full backups is relatively small compared to the database<br />
size. For backup cycles, the conventional backup storage consumption increases<br />
linearly. When <strong>Avamar</strong> is introduced, the backup storage consumption increases<br />
with the same proportion as that of the delta size between two full backups <strong>for</strong> <strong>SAP</strong><br />
databases which decreases the backup storage footprint significantly.<br />
Thus, <strong>for</strong> very large (up to TB level) <strong>SAP</strong> databases, using <strong>Avamar</strong> as the backup<br />
destination facilitates the backup storage management with high deduplication.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
22
Per<strong>for</strong>mance<br />
As the graph below shows, the <strong>Avamar</strong>-enabled full backup completes in 20 percent<br />
less time while using 60 percent less bandwidth. The side effect is that <strong>Avamar</strong><br />
consumes more CPU resources at runtime because <strong>Avamar</strong> always runs a<br />
comparison of the differences between the data yet to be backed up and the data<br />
that has already been backed up.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
23
<strong>Avamar</strong><br />
compared to<br />
BCT<br />
Starting from Oracle 10g, Block Change Tracking (BCT) is provided to speed up<br />
database incremental backup. When BCT is active, Oracle uses the Block Change<br />
Tracking File (BCTF) book to list the data blocks that have been changed since the<br />
last RMAN level 0 backup. RMAN then uses the in<strong>for</strong>mation from the BCTF in<br />
incremental backups to read and back up only those blocks that have been changed.<br />
For detailed in<strong>for</strong>mation, refer to <strong>SAP</strong> Note 964619.<br />
The following comparison is between the configuration of BACKINT with <strong>Avamar</strong>,<br />
BCT without <strong>Avamar</strong>, and BCT with <strong>Avamar</strong>.<br />
Backup Storage Footprint<br />
The total data transferred to <strong>Avamar</strong> with initial and all three subsequent backups is<br />
around 81 GB, 66 percent less than the data conducted by incremental backup jobs<br />
using BCT only. When BCT works together with <strong>Avamar</strong>, it requires almost the same<br />
space as BACKINT with <strong>Avamar</strong>.<br />
Per<strong>for</strong>mance<br />
For initial backup (level 0) to the same <strong>SAP</strong> system, BCT without <strong>Avamar</strong> leads to<br />
higher network traffic and a longer backup duration over the pure <strong>Avamar</strong>-based<br />
backup, but requires less CPU resources. Incorporating BCT and <strong>Avamar</strong> brings<br />
similar results as BACKINT with <strong>Avamar</strong>.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
24
• Initial Full Backup<br />
Note<br />
The CPU utilization in the scenario of BACKINT backup with <strong>Avamar</strong> refers<br />
to the one that happens on the proxy host.<br />
• Subsequent Incremental Backup<br />
In subsequent incremental backup, BACKINT with <strong>Avamar</strong> saves much more<br />
network bandwidth, but BCT requires much less time to complete the backup. BCT<br />
with <strong>Avamar</strong> offers better per<strong>for</strong>mance than BCT only in terms of both average<br />
network bandwidth and CPU utilization.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
25
Note<br />
The CPU utilization in the scenario of BACKINT backup with <strong>Avamar</strong> refers<br />
to the one that happens on the proxy host. The charts presented above<br />
show the average measurement results of three data loads.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
26
As the following chart illustrates, <strong>Avamar</strong> with BACKINT restores the same <strong>SAP</strong><br />
system 6.3 times faster than BCT from <strong>Avamar</strong>.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
27
Conclusion<br />
Introduction<br />
This section provides a comprehensive summary to the product benefits and<br />
limitations through comparison of the functions of the key components.<br />
Client-based<br />
deduplication<br />
by <strong>Avamar</strong><br />
<strong>Avamar</strong> reduces the backup storage footprint significantly by eliminating duplicated<br />
data from the backup set. <strong>Avamar</strong> only transfers new or changed data, based on<br />
preselected settings, since the last backup.<br />
Conventional backup through Ethernet requires high network bandwidth to ensure a<br />
short backup window. It is ideal to have a dedicated subnet <strong>for</strong> the backup<br />
environment. When sharing network with other applications, both the backup job and<br />
the application efficiency will be affected. By using <strong>Avamar</strong>, administrators can<br />
minimize the utilized network bandwidth attributable to backup jobs and share the<br />
network with multiple applications instead. However, backup with <strong>Avamar</strong> causes<br />
higher CPU utilization on the client.<br />
<strong>Avamar</strong> and<br />
Oracle BCT<br />
<strong>Avamar</strong> offers optimized backup per<strong>for</strong>mance regarding network utilization and<br />
minimizes the backup storage space required <strong>for</strong> backup contents. However, Oracle<br />
BCT offers the ability to complete incremental backups in the least time. When BCT<br />
and <strong>Avamar</strong> are combined, customers can achieve not only data deduplication and<br />
average-level network bandwidth, but also a short backup window <strong>for</strong> incremental<br />
backup. However, the restoration time is not acceptable.<br />
Automation of<br />
snapshotbased<br />
backup<br />
using<br />
PowerSnap<br />
PowerSnap allows automation of the snapshot/mirror-based backup though<br />
NetWorker. In this solution, PowerSnap with the mirror-split mode per<strong>for</strong>ms backup<br />
to <strong>Avamar</strong> to eliminate the impact of CPU utilization on the production host when<br />
<strong>Avamar</strong> is introduced. Backing up <strong>SAP</strong> systems from the proxy client minimizes the<br />
per<strong>for</strong>mance impact on the <strong>SAP</strong> production server in terms of CPU, network, and<br />
disk I/O.<br />
Centralized<br />
management<br />
with NetWorker<br />
integration<br />
As a central-management backup solution, NetWorker offers seamless integration<br />
with <strong>SAP</strong>, <strong>Avamar</strong>, and PowerSnap. NM<strong>SAP</strong> in conjunction with NetWorker allows<br />
users to per<strong>for</strong>m a backup to <strong>Avamar</strong> in the multi-stream mode, which is 2.6 times<br />
faster than in a single-stream mode.<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
28
References<br />
White papers<br />
For additional in<strong>for</strong>mation, see the white papers listed below.<br />
• Backing Up Large <strong>SAP</strong> Oracle Databases Using <strong>EMC</strong> <strong>Avamar</strong> and Oracle RMAN -<br />
A Detailed Review<br />
• Backup of <strong>SAP</strong> Oracle Databases in <strong>EMC</strong> Symmetrix Environments with <strong>EMC</strong><br />
NetWorker PowerSnap - Applied Technology<br />
Product<br />
documentation<br />
For additional in<strong>for</strong>mation, see the product documents listed below.<br />
• <strong>EMC</strong> NetWorker Release 7.6 Installation Guide<br />
• <strong>EMC</strong> NetWorker Release 7.6 Administration Guide<br />
• <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle Release 4.0 Installation Guide<br />
• <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle Release 4.0 Administration Guide<br />
• <strong>EMC</strong> NetWorker PowerSnap Module <strong>for</strong> Symmetrix DMX Version 2.5 Installation<br />
Guide<br />
• <strong>EMC</strong> NetWorker PowerSnap Module <strong>for</strong> Symmetrix DMX Version 2.5<br />
Administration Guide<br />
• <strong>EMC</strong> <strong>Avamar</strong> 4.1 Server Software Installation Manual<br />
• <strong>EMC</strong> <strong>Avamar</strong> Backup Clients 4.1 Installation Guide and Reference Manual<br />
• <strong>EMC</strong> <strong>Avamar</strong> 4.1 System Administration Manual<br />
• Master Guide <strong>for</strong> <strong>SAP</strong> ERP 6.0<br />
Other<br />
documentation<br />
For additional in<strong>for</strong>mation, see the document listed below.<br />
• <strong>SAP</strong> Note 964619 - RMAN: Incremental backups with Block Change Tracking<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
29
Additional in<strong>for</strong>mation<br />
Introduction<br />
This section describes the sample scripts used in this white paper.<br />
Baseline -<br />
BR*Tools<br />
configuration<br />
file <strong>for</strong> RMAN<br />
initPRDrman.sap<br />
************************************************************************************************<br />
backup_mode = full<br />
restore_mode = full<br />
backup_type = online_cons<br />
backup_dev_type = rman_util<br />
backup_root_dir = /oracle/PRD/sapbackup<br />
stage_root_dir = /oracle/PRD/sapbackup<br />
compress = no<br />
archive_function = save_delete<br />
archive_copy_dir = /oracle/PRD/sapbackup<br />
archive_stage_dir = /oracle/PRD/sapbackup<br />
saveset_members = 1<br />
rman_channels = 4<br />
rman_parms = "ENV=(NSR_DEBUG_LEVEL=9, NSR_SERVER=networker,<br />
NSR_CLIENT=sapprd, NSR_DATA_VOLUME_POOL=UC334PRDF,<br />
NSR_NWPATH=/usr/sbin/)"<br />
util_par_file = /oracle/PRD/102_64/dbs/initPRDrman.utl<br />
************************************************************************************************<br />
initPRDrman.utl<br />
************************************************************************************************<br />
NSR_NWPATH = /usr/sbin/<br />
parallelism = 4<br />
ss_group_by_fs = TRUE<br />
expiration = 3 Months<br />
pool = UC334PRDF<br />
server = networker<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
30
client = sapprd<br />
ssNameFormat = new<br />
NSR_SAVESET_RETENTION = 3 Months<br />
NSR_SAVESET_BROWSE = 3 Months<br />
************************************************************************************************<br />
Baseline -<br />
BR*Tools<br />
configuration<br />
file <strong>for</strong><br />
PowerSnap<br />
initPRD.sap<br />
************************************************************************************************<br />
backup_mode = full<br />
restore_mode = full<br />
backup_type = online_cons<br />
backup_dev_type = util_file_online<br />
backup_root_dir = /oracle/PRD/sapbackup<br />
stage_root_dir = /oracle/PRD/sapbackup<br />
compress = no<br />
archive_function = save_delete<br />
archive_copy_dir = /oracle/PRD/sapbackup<br />
archive_stage_dir = /oracle/PRD/sapbackup<br />
util_par_file = /oracle/PRD/102_64/dbs/initPRD.utl<br />
************************************************************************************************<br />
initPRD.utl<br />
************************************************************************************************<br />
NSR_NWPATH = /usr/sbin/<br />
parallelism = 4<br />
ss_group_by_fs = TRUE<br />
expiration = 3 Months<br />
pool = UC334PRDF<br />
server = networker<br />
client = sapprd<br />
ssNameFormat = new<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
31
NSR_SAVESET_RETENTION = 3 Months<br />
NSR_SAVESET_BROWSE = 3 Months<br />
ps_backup_mode = TRUE<br />
ps_archive_mode = FALSE<br />
ps_restore_mode = TRUE<br />
ps_inquire_mode = TRUE<br />
ps_exclude_backup_bi_run_nums = 2<br />
ps_exclude_archive_bi_run_nums = 1;2<br />
ps_exclude_backup_paths =<br />
/oracle/PRD/102_64/dbs/*;/oracle/PRD/sapreorg/*;/oracle/PRD/sa<br />
pbackup/*<br />
ps_opaque_pfilename = /oracle/PRD/102_64/dbs/nsrsapps.cfg<br />
************************************************************************************************<br />
nsrsapps.cfg<br />
************************************************************************************************<br />
NSR_DATA_MOVER = sapprx<br />
SYMM_SNAP_REMOTE = FALSE<br />
SYMM_SNAP_TECH = BCV<br />
NSR_PS_DEBUG_LEVEL = 9<br />
SYMM_ON_DELETE = RETAIN_RESOURCE<br />
NSR_SNAP_TYPE = symm-dmx<br />
NSR_PS_SAVE_PARALLELISM = 4<br />
************************************************************************************************<br />
Baseline -<br />
NM<strong>SAP</strong><br />
configuration<br />
file <strong>for</strong> RMAN<br />
nsrsapsvrman.cfg<br />
************************************************************************************************<br />
BR_EXEC = brbackup -p initPRDrman.sap<br />
ORACLE_HOME = /oracle/PRD/102_64<br />
ORACLE_SID = PRD<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
32
NLS_LANG = AMERICAN_AMERICA.US7ASCII<br />
<strong>SAP</strong>DATA_HOME = /oracle/PRD<br />
<strong>SAP</strong>_BIN = /sapmnt/PRD/exe<br />
LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib<br />
OS_USR_PASSWD=xxxxxxxxxxxxxxxx<br />
************************************************************************************************<br />
Baseline -<br />
NM<strong>SAP</strong><br />
configuration<br />
file <strong>for</strong><br />
PowerSnap<br />
nsrsapsv.cfg<br />
************************************************************************************************<br />
BR_EXEC = brbackup -p initPRD.sap -d util_file_online -r<br />
/oracle/PRD/102_64/dbs/initPRD.utl -t online_cons -m all_data<br />
-k no -l E<br />
ORACLE_HOME = /oracle/PRD/102_64<br />
ORACLE_SID = PRD<br />
NLS_LANG = AMERICAN_AMERICA.US7ASCII<br />
<strong>SAP</strong>DATA_HOME = /oracle/PRD<br />
<strong>SAP</strong>_BIN = /sapmnt/PRD/exe<br />
LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib<br />
OS_USR_PASSWD=xxxxxxxxxxxxxxxx<br />
************************************************************************************************<br />
<strong>Avamar</strong> -<br />
BR*Tools<br />
configuration<br />
file <strong>for</strong> RMAN<br />
initPRDrmanavt.sap<br />
************************************************************************************************<br />
backup_mode = full<br />
restore_mode = full<br />
backup_type = online_cons<br />
backup_dev_type = rman_util<br />
backup_root_dir = /oracle/PRD/sapbackup<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
33
stage_root_dir = /oracle/PRD/sapbackup<br />
compress = no<br />
archive_function = save_delete<br />
archive_copy_dir = /oracle/PRD/sapbackup<br />
archive_stage_dir = /oracle/PRD/sapbackup<br />
saveset_members = 1<br />
rman_channels = 4<br />
rman_send=(<br />
"'NSR_ENV=(NSR_DEDUP_BACKUP=TRUE,<br />
NSR_DEDUP_NODE=avamar01-u)'",<br />
"channel sbt_1 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_1)'",<br />
"channel sbt_2 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_2)'",<br />
"channel sbt_3 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_3)'",<br />
"channel sbt_4 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_4)'"<br />
)<br />
rman_parms = "ENV=(NSR_DEBUG_LEVEL=9, NSR_SERVER=networker,<br />
NSR_CLIENT=sapprd, NSR_DATA_VOLUME_POOL=UC334PRDFAVT,<br />
NSR_NWPATH=/usr/sbin/)"<br />
util_par_file = /oracle/PRD/102_64/dbs/initPRDrmanavt.utl<br />
************************************************************************************************<br />
initPRDrmanavt.utl<br />
************************************************************************************************<br />
NSR_NWPATH = /usr/sbin/<br />
parallelism = 4<br />
ss_group_by_fs = TRUE<br />
expiration = 3 Months<br />
pool = UC334PRDFAVT<br />
server = networker<br />
client = sapprd<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
34
ssNameFormat = new<br />
NSR_SAVESET_RETENTION = 3 Months<br />
NSR_SAVESET_BROWSE = 3 Months<br />
************************************************************************************************<br />
<strong>Avamar</strong> -<br />
BR*Tools<br />
configuration<br />
file <strong>for</strong><br />
PowerSnap<br />
initPRD.sap<br />
************************************************************************************************<br />
backup_mode = full<br />
restore_mode = full<br />
backup_type = online_cons<br />
backup_dev_type = util_file_online<br />
backup_root_dir = /oracle/PRD/sapbackup<br />
stage_root_dir = /oracle/PRD/sapbackup<br />
compress = no<br />
archive_function = save_delete<br />
archive_copy_dir = /oracle/PRD/sapbackup<br />
archive_stage_dir = /oracle/PRD/sapbackup<br />
util_par_file = /oracle/PRD/102_64/dbs/initPRD.utl<br />
************************************************************************************************<br />
initPRDavt.utl<br />
************************************************************************************************<br />
NSR_NWPATH = /usr/sbin/<br />
parallelism = 4<br />
ss_group_by_fs = TRUE<br />
expiration = 3 Months<br />
pool = UC334PRDFAVT<br />
server = networker<br />
client = sapprd<br />
ssNameFormat = new<br />
NSR_SAVESET_RETENTION = 3 Months<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
35
NSR_SAVESET_BROWSE = 3 Months<br />
ps_backup_mode = TRUE<br />
ps_archive_mode = FALSE<br />
ps_restore_mode = TRUE<br />
ps_inquire_mode = TRUE<br />
ps_exclude_backup_bi_run_nums = 2<br />
ps_exclude_archive_bi_run_nums = 1;2<br />
ps_exclude_backup_paths =<br />
/oracle/PRD/102_64/dbs/*;/oracle/PRD/sapreorg/*;/oracle/PRD/sa<br />
pbackup/*<br />
ps_opaque_pfilename = /oracle/PRD/102_64/dbs/nsrsapps.cfg<br />
************************************************************************************************<br />
nsrsapps.cfg<br />
************************************************************************************************<br />
NSR_DATA_MOVER = sapprx<br />
SYMM_SNAP_REMOTE = FALSE<br />
SYMM_SNAP_TECH = BCV<br />
NSR_PS_DEBUG_LEVEL = 9<br />
SYMM_ON_DELETE = RETAIN_RESOURCE<br />
NSR_SNAP_TYPE = symm-dmx<br />
NSR_PS_SAVE_PARALLELISM = 4<br />
<strong>Avamar</strong> -<br />
NM<strong>SAP</strong><br />
configuration<br />
file <strong>for</strong> RMAN<br />
nsrsapsvrmanavt.cfg<br />
************************************************************************************************<br />
BR_EXEC = brbackup -p initPRDrmanavt.sap<br />
ORACLE_HOME = /oracle/PRD/102_64<br />
ORACLE_SID = PRD<br />
NLS_LANG = AMERICAN_AMERICA.US7ASCII<br />
<strong>SAP</strong>DATA_HOME = /oracle/PRD<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
36
<strong>SAP</strong>_BIN = /sapmnt/PRD/exe<br />
LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib<br />
OS_USR_PASSWD=xxxxxxxxxxxxxxxx<br />
************************************************************************************************<br />
<strong>Avamar</strong> -<br />
NM<strong>SAP</strong><br />
configuration<br />
file <strong>for</strong><br />
PowerSnap<br />
nsrsapsvavt.cfg<br />
************************************************************************************************<br />
BR_EXEC = brbackup -p initPRD.sap -d util_file_online -r<br />
/oracle/PRD/102_64/dbs/initPRDavt.utl -t online_cons -m<br />
all_data -k no -l E<br />
ORACLE_HOME = /oracle/PRD/102_64<br />
ORACLE_SID = PRD<br />
NLS_LANG = AMERICAN_AMERICA.US7ASCII<br />
<strong>SAP</strong>DATA_HOME = /oracle/PRD<br />
<strong>SAP</strong>_BIN = /sapmnt/PRD/exe<br />
LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib<br />
OS_USR_PASSWD=xxxxxxxxxxxxxxxx<br />
************************************************************************************************<br />
<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />
Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />
37