20.04.2014 Views

Avamar for SAP - EMC Community Network

Avamar for SAP - EMC Community Network

Avamar for SAP - EMC Community Network

SHOW MORE
SHOW LESS

Create successful ePaper yourself

Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong><br />

with <strong>EMC</strong> <strong>Avamar</strong><br />

A Detailed Review<br />

<strong>EMC</strong> In<strong>for</strong>mation Infrastructure Solutions<br />

Abstract<br />

This white paper describes how the backup and restore operation of <strong>SAP</strong> systems benefit from client-based<br />

deduplication technology using <strong>EMC</strong> ® <strong>Avamar</strong> ® . The white paper also addresses how customers can use the <strong>EMC</strong><br />

NetWorker ® PowerSnap module to reduce the per<strong>for</strong>mance impact on the <strong>SAP</strong> production server during backup.<br />

July 2010


Copyright © 2010 <strong>EMC</strong> Corporation. All rights reserved.<br />

<strong>EMC</strong> believes the in<strong>for</strong>mation in this publication is accurate as of its publication date. The in<strong>for</strong>mation is subject to<br />

change without notice.<br />

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” <strong>EMC</strong> CORPORATION MAKES NO<br />

REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS<br />

PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR<br />

FITNESS FOR A PARTICULAR PURPOSE.<br />

Use, copying, and distribution of any <strong>EMC</strong> software described in this publication requires an applicable software<br />

license.<br />

For the most up-to-date listing of <strong>EMC</strong> product names, see <strong>EMC</strong> Corporation Trademarks on <strong>EMC</strong>.com<br />

All other trademarks used herein are the property of their respective owners.<br />

Part number: H7052<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

2


Table of Contents<br />

Executive summary ........................................................................................................................... 5<br />

Business case ............................................................................................................................... 5<br />

Product solution............................................................................................................................. 5<br />

Key results ..................................................................................................................................... 5<br />

Introduction........................................................................................................................................ 6<br />

Overview ....................................................................................................................................... 6<br />

Purpose ......................................................................................................................................... 6<br />

Scope ............................................................................................................................................ 6<br />

Audience ....................................................................................................................................... 6<br />

Terminology ................................................................................................................................... 6<br />

Key components ............................................................................................................................... 8<br />

Introduction .................................................................................................................................... 8<br />

<strong>EMC</strong> Symmetrix VMAX ................................................................................................................. 8<br />

<strong>EMC</strong> <strong>Avamar</strong> ................................................................................................................................. 8<br />

<strong>EMC</strong> NetWorker PowerSnap Module <strong>for</strong> Symmetrix .................................................................... 8<br />

<strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle ............................................................................... 8<br />

VMware vSphere ........................................................................................................................... 9<br />

IDES <strong>SAP</strong> ERP ............................................................................................................................. 9<br />

Data Protection Advisor ................................................................................................................ 9<br />

Physical architecture ....................................................................................................................... 10<br />

Overview ..................................................................................................................................... 10<br />

Physical environment .................................................................................................................. 10<br />

Hardware resources .................................................................................................................... 11<br />

Software resources ..................................................................................................................... 11<br />

Configuration and design ................................................................................................................ 12<br />

Introduction .................................................................................................................................. 12<br />

Key considerations ...................................................................................................................... 12<br />

LUN design .................................................................................................................................. 13<br />

<strong>SAP</strong> application server configuration .......................................................................................... 14<br />

PowerSnap specific configuration ............................................................................................... 15<br />

Data load configuration ............................................................................................................... 16<br />

Testing and validation ..................................................................................................................... 17<br />

Introduction .................................................................................................................................. 17<br />

Scenarios overview ..................................................................................................................... 17<br />

Test group 1: Baseline with BCT off............................................................................................ 18<br />

Test group 2: <strong>Avamar</strong> on with BCT off ........................................................................................ 18<br />

Test group 3: Baseline with BCT on............................................................................................ 20<br />

Test group 4: <strong>Avamar</strong> on with BCT on ........................................................................................ 21<br />

Results validation ........................................................................................................................ 22<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

3


<strong>Avamar</strong> compared to baseline .................................................................................................... 22<br />

<strong>Avamar</strong> compared to BCT ........................................................................................................... 24<br />

Conclusion....................................................................................................................................... 28<br />

Introduction .................................................................................................................................. 28<br />

Client-based deduplication by <strong>Avamar</strong>........................................................................................ 28<br />

<strong>Avamar</strong> and Oracle BCT ............................................................................................................. 28<br />

Automation of snapshot-based backup using PowerSnap ......................................................... 28<br />

Centralized management with NetWorker integration ................................................................ 28<br />

References ...................................................................................................................................... 29<br />

White papers ............................................................................................................................... 29<br />

Product documentation ............................................................................................................... 29<br />

Other documentation ................................................................................................................... 29<br />

Additional in<strong>for</strong>mation ...................................................................................................................... 30<br />

Introduction .................................................................................................................................. 30<br />

Baseline - BR*Tools configuration file <strong>for</strong> RMAN ........................................................................ 30<br />

Baseline - BR*Tools configuration file <strong>for</strong> PowerSnap ................................................................ 31<br />

Baseline - NM<strong>SAP</strong> configuration file <strong>for</strong> RMAN .......................................................................... 32<br />

Baseline - NM<strong>SAP</strong> configuration file <strong>for</strong> PowerSnap .................................................................. 33<br />

<strong>Avamar</strong> - BR*Tools configuration file <strong>for</strong> RMAN ......................................................................... 33<br />

<strong>Avamar</strong> - BR*Tools configuration file <strong>for</strong> PowerSnap ................................................................. 35<br />

<strong>Avamar</strong> - NM<strong>SAP</strong> configuration file <strong>for</strong> RMAN ............................................................................ 36<br />

<strong>Avamar</strong> - NM<strong>SAP</strong> configuration file <strong>for</strong> PowerSnap .................................................................... 37<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

4


Executive summary<br />

Business case<br />

<strong>SAP</strong> data growth can be as much as 60 percent per year. In many cases there are<br />

multiple <strong>SAP</strong> solutions with production and non-production data growing<br />

continuously. This situation is driving <strong>SAP</strong> customers to revisit their backup and<br />

recovery strategies as traditional backup systems are being pushed to capacity.<br />

The <strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong> – Enabled by NetWorker Module <strong>for</strong> <strong>SAP</strong><br />

with <strong>Avamar</strong> solution is designed to enable <strong>SAP</strong> customers to achieve new levels of<br />

data reduction and provide fast, secure backup <strong>for</strong> their production and nonproduction<br />

<strong>SAP</strong> environments. In the process, <strong>SAP</strong> backup time and growth of<br />

secondary storage are reduced, along with decreased network utilization. In case of<br />

data corruption, users can recover the system in minimal time by using full backup.<br />

Product<br />

solution<br />

This solution demonstrates the benefits of client-based deduplication in an<br />

<strong>SAP</strong>/Oracle environment on the <strong>EMC</strong> ® Symmetrix ® plat<strong>for</strong>m as well as the benefits<br />

of integrating <strong>EMC</strong> PowerSnap and the <strong>EMC</strong> NetWorker ® Module <strong>for</strong> <strong>SAP</strong> with<br />

Oracle (NM<strong>SAP</strong>) using the <strong>EMC</strong> <strong>Avamar</strong> ® plug-in and NetWorker.<br />

The solution builds a replica of a common customer environment and validates the<br />

configuration <strong>for</strong> both per<strong>for</strong>mance and functionality of client-based deduplication<br />

enabled by <strong>Avamar</strong>.<br />

NetWorker is used as the backup management tool to per<strong>for</strong>m the multi-stream<br />

backup to <strong>Avamar</strong>. Through interaction with the PowerSnap module, NetWorker<br />

controls the process of snapshot-based backup and eliminates the per<strong>for</strong>mance<br />

impact of the backup job on the production server being backed up.<br />

<strong>EMC</strong> Data Protection Advisor (DPA) collects relevant backup in<strong>for</strong>mation. In this<br />

solution, we will show how you can use this analysis tool <strong>for</strong> data collection, data<br />

recording, and backup metrics management.<br />

Key results<br />

This white paper demonstrates the following benefits <strong>for</strong> the solution:<br />

• <strong>Avamar</strong> reduces the network utilization of the backup jobs by 57 percent while<br />

achieving a deduplication ratio of 10:1.<br />

• NetWorker PowerSnap enables the client-based deduplication without additional<br />

CPU utilization on the production host.<br />

• Multi-streaming with NM<strong>SAP</strong> enabled by NetWorker allows the backup job to finish<br />

over two and a half times faster than if per<strong>for</strong>med with a single stream.<br />

• Restoration from <strong>Avamar</strong> taking advantage of multi-streaming with NM<strong>SAP</strong><br />

enabled by NetWorker is six times faster than Recovery Manager (RMAN) with<br />

BCT.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

5


Introduction<br />

Overview<br />

This white paper clarifies the benefits of client-based deduplication and mirror-based<br />

backups <strong>for</strong> <strong>SAP</strong> landscapes. It validates the feasibility of implementing a<br />

deduplication approach in a typical <strong>SAP</strong> environment.<br />

The reference architecture presented in this white paper will help you prepare a<br />

backup and recovery plan and topology <strong>for</strong> <strong>SAP</strong> systems.<br />

Purpose<br />

The purpose of the white paper is to:<br />

• Provide an understanding of client-based deduplication <strong>for</strong> backup of <strong>SAP</strong> systems<br />

• Supply an overview of the snapshot-based backup method and explain the reason<br />

<strong>for</strong> using this backup method<br />

• Demonstrate the integration capability of NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle<br />

(NM<strong>SAP</strong>), PowerSnap, and <strong>Avamar</strong> with central control by NetWorker<br />

Scope<br />

This white paper documents the software and hardware resources involved in this<br />

solution as well as the corresponding configuration <strong>for</strong> building the test environment.<br />

The white paper also addresses the solution design, the test scenario pattern, and<br />

the conclusion based on test results and analysis.<br />

It is assumed the reader has knowledge and experience using BR*Tools <strong>for</strong> backup<br />

and recovery of <strong>SAP</strong> with Oracle databases. Detailed explanations of these concepts<br />

are not in the scope of this paper.<br />

Audience<br />

This white paper is intended <strong>for</strong> customers, partners, and <strong>EMC</strong> employees, including<br />

IT planners, <strong>SAP</strong> architects, Basis administrators, and any other people involved in<br />

evaluating, acquiring, managing, operating, or designing an <strong>SAP</strong> landscape<br />

infrastructure.<br />

Terminology<br />

This section defines terms used in this document.<br />

Term<br />

DEV<br />

QAS<br />

PRD<br />

RMAN<br />

BCT<br />

Definition<br />

<strong>SAP</strong> development system<br />

<strong>SAP</strong> quality assurance system, also referred to as test system<br />

<strong>SAP</strong> production system<br />

Recovery Manager, an Oracle built-in tool <strong>for</strong> backup and<br />

restore, which per<strong>for</strong>ms backup at the database level<br />

Block Change Tracking, an Oracle feature that reduces backup<br />

times on incremental backups by tracking changed blocks since<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

6


the last backup on a dedicated file<br />

BR*Tools<br />

Proxy Host<br />

BACKINT<br />

Baseline<br />

backup<br />

LSMW<br />

PIT recovery<br />

Solution<br />

Enabler<br />

Toolset that the <strong>SAP</strong> <strong>for</strong> Oracle database administration (DBA)<br />

tasks provide<br />

Also known as Data Mover, the secondary host, from which<br />

PowerSnap implements snapshot-based backup<br />

The external backup library called by BR*Tools to per<strong>for</strong>m <strong>SAP</strong><br />

system backup at the file system level. It is <strong>SAP</strong> certified.<br />

Backup to SATA volumes without using the deduplication<br />

technology<br />

Legacy System Migration Workbench (LSMW), an <strong>SAP</strong> specific<br />

tool written in Advance Business Application Programming<br />

(ABAP) to load data into the system<br />

Point-in-time recovery<br />

<strong>EMC</strong> shared libraries <strong>for</strong> use by Solutions Enabler applications,<br />

and the Symmetrix Command Line Interface (SYMCLI) <strong>for</strong> use by<br />

storage administrators and systems engineers.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

7


Key components<br />

Introduction<br />

This section briefly describes the key solution components. For details on all of the<br />

components that make up the architecture, see the “Physical architecture” section in<br />

this paper.<br />

<strong>EMC</strong><br />

Symmetrix<br />

VMAX<br />

<strong>EMC</strong> Symmetrix VMAX provides high-end storage <strong>for</strong> the virtual data center.<br />

Symmetrix VMAX scales up to 2 PB of usable protected capacity and consolidates<br />

more workloads with a much smaller footprint than alternative arrays.<br />

The innovative <strong>EMC</strong> Symmetrix Virtual Matrix Architecture of VMAX seamlessly<br />

scales per<strong>for</strong>mance, capacity, and connectivity on demand to meet all application<br />

requirements. Symmetrix VMAX can be deployed with Flash drives, and Fibre<br />

Channel and Serial Advanced Technology Attachment (SATA) drives, with tiering<br />

fully automated with FAST. It supports virtualized and physical servers, including<br />

open systems, mainframe, and IBM i hosts.<br />

<strong>EMC</strong> <strong>Avamar</strong><br />

<strong>EMC</strong> <strong>Avamar</strong> backup and recovery solutions utilize patented global data<br />

deduplication technology to identify redundant data at the source, minimizing backup<br />

data be<strong>for</strong>e it is sent over the LAN/WAN. With <strong>Avamar</strong>, you can achieve new levels<br />

of data reduction and enable fast, secure backup <strong>for</strong> your VMware environments,<br />

remote offices, LAN/NAS servers, and desktop/laptop systems. In the process, you<br />

will reduce backup time, growth of secondary storage, and network utilization.<br />

<strong>Avamar</strong> deduplicated backups function like full backups and can be recovered in just<br />

one step. You do not need to restore full backups and subsequent incremental<br />

backups. In addition, <strong>Avamar</strong> verifies backup data recoverability, so there are no<br />

surprises, and encrypts data <strong>for</strong> secure electronic backups.<br />

<strong>EMC</strong> NetWorker<br />

PowerSnap<br />

Module <strong>for</strong><br />

Symmetrix<br />

The PowerSnap Module enhances NetWorker by allowing snapshot-based data<br />

protection and availability during backups <strong>for</strong> the Symmetrix storage system. The<br />

backup is virtually instantaneous, regardless of the volume size. The PowerSnap<br />

Module software enables frequent backups with little impact on the network or<br />

application server, and provides backups of large databases or data stored in file<br />

systems.<br />

<strong>EMC</strong> NetWorker<br />

Module <strong>for</strong> <strong>SAP</strong><br />

with Oracle<br />

<strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle (NM<strong>SAP</strong>) ensures that the missioncritical<br />

data is available by delivering fast, online backup and recovery <strong>for</strong> <strong>SAP</strong>. With<br />

NM<strong>SAP</strong>, <strong>SAP</strong> administrators can centrally manage backup operations and<br />

schedules.<br />

Featuring data deduplication support through its embedded <strong>Avamar</strong> client, NM<strong>SAP</strong><br />

reduces the impact of data protection operations on production servers and<br />

networks, and significantly reduces the amount of backup storage required.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

8


VMware<br />

vSphere<br />

VMware vSphere is the foundation <strong>for</strong> a dynamic, self-optimizing IT infrastructure. It<br />

is a robust, production-proven virtualization layer that abstracts processor, memory,<br />

storage, and networking resources into multiple virtual machines. VMware vSphere<br />

allows enterprises to dramatically reduce hardware and operating costs by sharing<br />

resources across a virtual environment.<br />

IDES <strong>SAP</strong> ERP <strong>SAP</strong> ERP IDES is the "Internet Demonstration and Evaluation System" in the R/3<br />

System, and represents a model company. It consists of an international group with<br />

subsidiaries in several countries. IDES contains application data <strong>for</strong> various business<br />

scenarios you can run in the <strong>SAP</strong> system. The business processes in the IDES<br />

system are designed to reflect real-life business requirements, and have access to<br />

many realistic characteristics. IDES uses easy-to-follow business scenarios to show<br />

you the comprehensive functions of the R/3 System. The focal point of IDES,<br />

however, is not the functionality itself, but the business processes and their<br />

integration.<br />

Data Protection<br />

Advisor<br />

Data Protection Advisor (DPA) is an advanced backup reporting and analysis<br />

application that gives its users complete visibility into everything happening within<br />

their backup environment. DPA gathers data from backup servers, backup clients,<br />

databases, network switches, SAN switches, and tape libraries. DPA correlates<br />

in<strong>for</strong>mation from each of these areas and provides advanced reporting,<br />

troubleshooting, per<strong>for</strong>mance management, and capacity planning.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

9


Physical architecture<br />

Overview<br />

The following section identifies and briefly describes the technology and components<br />

used in the environment.<br />

Physical<br />

environment<br />

The following diagram illustrates the overall physical architecture of the environment:<br />

The <strong>SAP</strong> DEV and QAS systems, NetWorker server, and DPA server are all built on<br />

the virtual machine managed by the VMware ESX server. NetWorker controls the<br />

backup and restoration processes, and the DPA server captures the data results.<br />

The <strong>SAP</strong> PRD system and the proxy host reside on the physical machines, which<br />

are connected with the ESX server through a LAN <strong>for</strong> communication purposes<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

10


etween NM<strong>SAP</strong>, PowerSnap and the NetWorker server. <strong>EMC</strong> Symmetrix VMAX is<br />

attached to all application servers through the SAN. In addition, the NetWorker<br />

defines <strong>Avamar</strong> as the deduplication node in device configuration.<br />

Hardware<br />

resources<br />

The hardware used to validate the solution is listed in the following table.<br />

Equipment Quantity Configuration<br />

<strong>EMC</strong> VMAX SE 1 52 x 300 GB 15k FC disks<br />

52 x 1 TB 7.2k SATA disks<br />

VMware ESX server 1 Dell PowerEdge R900 16 Core<br />

<strong>SAP</strong> host 4 • DEV: VMware Virtual Machine 2 CPUs, 8 GB RAM<br />

• QAS: VMware Virtual Machine 2 CPUs, 8 GB RAM<br />

• PRD: Dell PowerEdge 6850 16 Core<br />

• Proxy host: Dell PowerEdge R710 16 Core<br />

<strong>EMC</strong> NetWorker server 1 VMware Virtual Machine 2 CPUs, 8 GB RAM<br />

<strong>EMC</strong> DPA server 1 VMware Virtual Machine 2 CPUs, 8 GB RAM<br />

SAN switch 1 Cisco Catalyst 9509<br />

<strong>EMC</strong> <strong>Avamar</strong> Appliance 1 <strong>EMC</strong> <strong>Avamar</strong> Data Store Grid (six nodes) Gen2<br />

Software<br />

resources<br />

The software used to validate the solution is listed in the following table.<br />

Software<br />

Version<br />

Linux Red Hat Enterprise<br />

5.3 AS5 U2 64 bit<br />

Oracle Database 10.2.0.4<br />

<strong>SAP</strong> Application<br />

IDES ERP 6.0 EhP4<br />

<strong>EMC</strong> NetWorker 7.6<br />

<strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle 4.0<br />

<strong>EMC</strong> NetWorker PowerSnap Module <strong>for</strong><br />

Symmetrix DMX /VMAX<br />

2.5<br />

<strong>EMC</strong> Solutions Enabler 7.0.1.0<br />

VMware vSphere Server 4.0 Update 1<br />

<strong>EMC</strong> <strong>Avamar</strong> software 4.1<br />

<strong>EMC</strong> Data Protection Advisor 5.5<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

11


Configuration and design<br />

Introduction<br />

This section provides the configuration in<strong>for</strong>mation <strong>for</strong> this solution, including:<br />

• Disk allocation pattern<br />

• Relevant <strong>SAP</strong> system settings<br />

• Corresponding NetWorker components<br />

• Key considerations or prerequisites <strong>for</strong> the proposed test scenarios<br />

The <strong>SAP</strong> systems presented in this solution follow the principle of the standard<br />

three-system landscape known as DEV, QAS, and PRD.<br />

Key<br />

considerations<br />

<strong>EMC</strong> considered the following when per<strong>for</strong>ming the test scenarios in this solution:<br />

• An Advanced File Type Device is created on SATA disks on VMAX as the<br />

destination <strong>for</strong> baseline backup.<br />

• RMAN is configured as the default backup tool in BR*Tools <strong>for</strong> backing up<br />

database files to backup destination. The database control file, the BR*Tools<br />

profiles, and logs are still backed up through BACKINT.<br />

• BACKINT is specified as the default method <strong>for</strong> backing up all files with<br />

PowerSnap, including database data files, the control file, the BR*Tools profiles,<br />

and logs.<br />

• You do not need to install a separate <strong>Avamar</strong> client. <strong>Avamar</strong> works as a<br />

deduplication node in the NetWorker server and NetWorker manages all backup<br />

streams to <strong>Avamar</strong>.<br />

• By default, <strong>SAP</strong> per<strong>for</strong>ms a cumulative backup when running an incremental<br />

backup job.<br />

• Backup <strong>for</strong> archive logs is not configured <strong>for</strong> the development and testing systems.<br />

The production system is separately configured to run every two hours without<br />

connecting to <strong>Avamar</strong> or PowerSnap.<br />

A baseline backup is established to compare the results with that of the client-based<br />

deduplication backup. To facilitate database size changes, <strong>EMC</strong> used Legacy<br />

System Migration Workbench (LSMW) <strong>for</strong> data population to simulate a realtime data<br />

increase in the customer environment. In this solution, the comparison between<br />

baseline and deduplication backup is only conducted on the production system. <strong>EMC</strong><br />

also used the development and testing systems in this solution to validate the<br />

feasibility of client-based deduplication on virtualized plat<strong>for</strong>ms.<br />

When doing backup in a traditional environment that demands optimized<br />

per<strong>for</strong>mance, you should per<strong>for</strong>m an incremental backup with BCT to reduce the<br />

backup duration and network traffic. There<strong>for</strong>e it is useful in this solution to measure<br />

the per<strong>for</strong>mance difference between BCT and <strong>Avamar</strong>.<br />

Taking the per<strong>for</strong>mance impact on the backup client into consideration, <strong>EMC</strong><br />

developed a series of approaches <strong>for</strong> backing up a client through a mirrored volume.<br />

PowerSnap works together with NetWorker to interact with Solutions Enabler to<br />

accomplish this task. <strong>EMC</strong> recommends that you configure BACKINT as the default<br />

backup method <strong>for</strong> all files when integrating PowerSnap with NM<strong>SAP</strong> and<br />

NetWorker.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

12


LUN design<br />

The following table describes the storage design pattern <strong>for</strong> the <strong>SAP</strong> DEV and QAS<br />

systems built on virtual machines.<br />

RAID<br />

type<br />

Disk<br />

type<br />

LUN size Purpose Filesystem<br />

10 VMFS 64 GB Linux root and<br />

swap<br />

10 RDM 32 GB Original redo log /oracle//origlogA<br />

/oracle//origlogB<br />

10 RDM 32 GB Mirrored redo log /oracle//mirrlogA<br />

/oracle//mirrlogA<br />

5 RDM 64 GB Oracle archive log /oracle//oraarch<br />

/<br />

5 RDM 64 GB x<br />

4<br />

5 RDM 64 GB x<br />

4<br />

5 RDM 64 GB x<br />

4<br />

5 RDM 64 GB x<br />

4<br />

Database data files<br />

Database data files<br />

Database data files<br />

Database data files<br />

/oracle//sapdata1<br />

/oracle//sapdata2<br />

/oracle//sapdata3<br />

/oracle//sapdata4<br />

5 RDM 32 GB <strong>SAP</strong> BIN /sapmnt/<br />

/usr/sap/<br />

/usr/sap/trans<br />

5 RDM 32 GB Oracle BIN /oracle/client<br />

/oracle/stage<br />

/oracle//102_64<br />

The following table describes the storage design pattern <strong>for</strong> the <strong>SAP</strong> PRD system<br />

built on physical machines.<br />

RAID type LUN size Purpose Filesystem<br />

10 64 GB Linux root and swap /<br />

10 32 GB Original redo log /oracle//origlogA<br />

/oracle//origlogB<br />

10 32 GB Mirrored redo log /oracle//mirrlogA<br />

/oracle//mirrlogA<br />

5 64 GB Oracle archive log /oracle//oraarch<br />

5 64 GB x 4 Database data files /oracle//sapdata1<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

13


5 64 GB x 4 Database data files /oracle//sapdata2<br />

5 64 GB x 4 Database data files /oracle//sapdata3<br />

5 64 GB x 4 Database data files /oracle//sapdata4<br />

5 32 GB <strong>SAP</strong> BIN /sapmnt/<br />

/usr/sap/<br />

/usr/sap/trans<br />

5 32 GB Oracle BIN /oracle/client<br />

/oracle/stage<br />

/oracle//102_64<br />

<strong>SAP</strong><br />

application<br />

server<br />

configuration<br />

The solution is validated with the following configuration specified to <strong>SAP</strong> systems,<br />

where PowerSnap is disabled. Configuration of PRD and the proxy host specific to<br />

PowerSnap is outlined separately.<br />

DEV<br />

Profile characteristic<br />

Database size<br />

Value<br />

271 GB<br />

Installed agent<br />

BR*Tools - Backup device type<br />

BR*Tools - Backup type<br />

BR*Tools - Backup mode<br />

NM<strong>SAP</strong><br />

rman_util<br />

online_cons<br />

full<br />

QAS<br />

Profile characteristic<br />

Database size<br />

Installed agent<br />

BR*Tools - Backup device type<br />

BR*Tools - Backup type<br />

BR*Tools - Backup mode<br />

Value<br />

270 GB<br />

NM<strong>SAP</strong><br />

rman_util<br />

online_cons<br />

full<br />

PRD<br />

Profile characteristic<br />

Database size<br />

Installed agent<br />

Value<br />

278 GB<br />

NM<strong>SAP</strong>, PowerSnap (valid in PowerSnap<br />

enabled backup)<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

14


BR*Tools - Backup device type<br />

BR*Tools - Backup type<br />

BR*Tools - Backup mode<br />

rman_util<br />

online_cons<br />

full<br />

Note The meanings of BR*Tools parameters are explained below.<br />

• rman_util<br />

RMAN backup in combination with a backup library and the backup tool of another<br />

manufacturer. BACKINT provides an interface to the external backup tool and is<br />

also used to back up profiles, log files, and the control file.<br />

• online_cons<br />

The backup is per<strong>for</strong>med when the database is open. Both the database files and<br />

the offline redo log files generated during the backup are copied to the same<br />

volume. As a result, you have a logically consistent dataset available.<br />

• Full<br />

Full database backup at level 0. This backup can be part of an incremental backup<br />

strategy.<br />

For more in<strong>for</strong>mation about the explanation of a BR*Tools configuration, refer to <strong>SAP</strong><br />

online help.<br />

Refer to the “Additional in<strong>for</strong>mation” section <strong>for</strong> more in<strong>for</strong>mation about the<br />

configuration <strong>for</strong> the PRD system.<br />

PowerSnap<br />

specific<br />

configuration<br />

The following tables list the configuration characteristics of PRD and the proxy host<br />

using PowerSnap.<br />

PRD<br />

Profile characteristic<br />

Database size<br />

Value<br />

278 GB<br />

Installed agent<br />

BR*Tools - Backup device type<br />

BR*Tools - Backup type<br />

BR*Tools - Backup mode<br />

NM<strong>SAP</strong>, PowerSnap<br />

util_file_online<br />

online_cons<br />

all_data<br />

Note The new value of BR*Tools parameter is explained below.<br />

• util_file_online<br />

Use external backup programs <strong>for</strong> file-by-file backup. Use this parameter value <strong>for</strong><br />

an online backup, if it is supported by the external backup program. This<br />

dynamically sets and ends the backup status of the tablespaces to be saved and<br />

thus greatly reduces the volume of offline redo log files during the backup.<br />

• all_data<br />

Back up the files of all tablespaces except pure index tablespaces or empty<br />

tablespaces.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

15


Proxy host<br />

Profile characteristic<br />

Installed agent<br />

Synchronized volumes<br />

Snapshot type<br />

PowerSnap backup type<br />

Value<br />

PowerSnap<br />

Database data files<br />

BCV<br />

Immediate live backup (serverless snapshot<br />

backup)<br />

Note<br />

• This solution uses BCV emulation because PowerSnap requires this mode. You<br />

can define BCV as the value of the snapshot type <strong>for</strong> the proxy host configuration.<br />

In addition, you can use BCVs <strong>for</strong> local replication and mount them to the proxy<br />

host in this solution.<br />

• The database control file, the BR*Tools profile, and log files are always backed up<br />

directly from the production host.<br />

Immediate Live Backup refers to a process where BCVs are split and immediately<br />

backed up by the PowerSnap module to conventional storage. The BCVs are<br />

automatically synchronized with the source volumes.<br />

For more in<strong>for</strong>mation about the backup/restore definition enabled by NM<strong>SAP</strong> with<br />

PowerSnap, refer to the <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle 4.0<br />

Administration Guide.<br />

Refer to the “Additional in<strong>for</strong>mation” section <strong>for</strong> more in<strong>for</strong>mation about the<br />

configuration <strong>for</strong> the PRD system.<br />

Data load<br />

configuration<br />

Data load simulates the data volume increase and changes generated by day-to-day<br />

business operations in the <strong>SAP</strong> systems. In order to get an acceptable and relatively<br />

stable result, we per<strong>for</strong>med three data loads, as shown in each scenario.<br />

The load pattern is outlined in the table below.<br />

Profile characteristic<br />

<strong>SAP</strong> module where the data is generated<br />

Object to be generated<br />

Value<br />

SD<br />

Sales order<br />

How many records generated each time 100,000<br />

Data volume increase in database by each<br />

load<br />

Approximately 2.7 GB (10% data<br />

increase of total database)<br />

Note<br />

Sales orders were generated randomly to avoid generating records that<br />

would artificially tend to deduplicate well. This random process retrieves data<br />

from over 20,000 possible customers and 10,000 possible items in the<br />

master data and combines them to generate random orders.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

16


Testing and validation<br />

Introduction<br />

This section provides detailed in<strong>for</strong>mation on the test scenarios’ result validation.<br />

The test scenarios were grouped and carried out based on different combinations of<br />

key components’ functionality. The key components refer to client-based<br />

deduplication enabled by <strong>Avamar</strong> and Oracle incremental backup optimized by BCT.<br />

The following statements apply to all tested scenarios:<br />

• The backup stream is directed to the Advance File Type Device (baseline backup)<br />

when <strong>Avamar</strong> is disabled.<br />

• The backup stream is always directed to the <strong>Avamar</strong> deduplication node through<br />

NetWorker when <strong>Avamar</strong> is enabled.<br />

• A full backup is conducted with BCT disabled.<br />

• An incremental backup is conducted with BCT enabled.<br />

• All resulting data is captured by DPA.<br />

Unless otherwise specified, PowerSnap is disabled <strong>for</strong> the PRD system during<br />

backup.<br />

All <strong>SAP</strong> systems are copied using storage cloning <strong>for</strong> rapid refresh and consistency<br />

in the context of all test scenarios.<br />

Scenarios<br />

overview<br />

The test per<strong>for</strong>med on the target system covers both backup and restore operations.<br />

Backup is divided into two stages: Stage 1 means an initial backup, and stage 2<br />

refers to the subsequent full/incremental backup upon completion of each data load.<br />

Data load applies to the production system in particular.<br />

<strong>EMC</strong> processed four groups of tests with different configurations of <strong>Avamar</strong> and BCT<br />

in order to meet the following objectives:<br />

• Measure the achieved deduplication ratio as well as any advantages brought by<br />

<strong>Avamar</strong><br />

• Measure and compare the advantages and disadvantages of using Oracle BCT<br />

with <strong>Avamar</strong> <strong>for</strong> backups<br />

• Observe how PowerSnap improves the backup process <strong>for</strong> the production system<br />

• Compare the efficiency between multi-stream backups and single-stream backups<br />

• Validate <strong>Avamar</strong> support <strong>for</strong> virtualized <strong>SAP</strong> systems<br />

The following table summarizes the different scenarios that were tested in this<br />

solution.<br />

Factor<br />

<strong>Avamar</strong> off<br />

BCT off<br />

Backup stage 1<br />

(Initial)<br />

Backup stage 2<br />

(3 data loads)<br />

Restore<br />

DEV QAS PRD DEV QAS PRD DEV QAS PRD<br />

F F F N/A N/A N/A C C C<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

17


<strong>Avamar</strong> on<br />

BCT off<br />

<strong>Avamar</strong> off<br />

BCT on<br />

<strong>Avamar</strong> on<br />

BCT on<br />

F F F* N/A N/A F/F/F* N/A N/A N/A<br />

N/A N/A F N/A N/A I/I/I N/A N/A N/A<br />

F F F N/A N/A I/I/I N/A N/A C P<br />

Note<br />

F=Full Backup, I=Incremental Backup, C=Complete Restore, P=PIT Restore.<br />

*Backup will be per<strong>for</strong>med twice <strong>for</strong> PRD: one is conducted with RMAN; and<br />

the other is conducted with PowerSnap.<br />

Test group 1:<br />

Baseline with<br />

BCT off<br />

Procedure<br />

Step Action<br />

1 Per<strong>for</strong>med one-time backup <strong>for</strong> DEV, QAS, and PRD.<br />

2 Wiped out the database of the above system and restored from the<br />

existing backup.<br />

Result<br />

SID Scenario CPU<br />

utilization<br />

(%)<br />

<strong>Network</strong><br />

throughput<br />

(MB/s)<br />

Data<br />

transfer<br />

(GB<br />

Backup<br />

source<br />

size<br />

(GB)<br />

Duration<br />

(min)<br />

PRD Backup 4.37 78.55 218.76 215 47.53<br />

SID Scenario Restore target size (GB) Duration (min)<br />

PRD Restore 215 79.1<br />

Note<br />

The above results were captured directly from DPA.<br />

The baseline measurement recorded is used <strong>for</strong> comparison with the following test.<br />

Test group 2:<br />

<strong>Avamar</strong> on with<br />

BCT off<br />

Procedure<br />

Step Action<br />

1 Per<strong>for</strong>med backup <strong>for</strong> DEV and QAS to <strong>Avamar</strong>.<br />

2 Per<strong>for</strong>med backup <strong>for</strong> PRD, which contains two phases:<br />

a. Initial backup (full backup)<br />

b. Subsequent backup (full backup), each with data load with the<br />

predefined pattern<br />

The second phase was repeated three times to achieve a stable and<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

18


easonable result.<br />

Switched the backup stream from multiple to single to see how it<br />

affected the backup speed.<br />

3 a. Switched backup method to BACKINT and enabled<br />

PowerSnap.<br />

b. Executed the same procedure as step 2.<br />

c. Wiped out the database and restore from the latest backup.<br />

Result<br />

The following tables show the key metrics regarding backup process <strong>for</strong> PRD.<br />

• Without PowerSnap<br />

Scenario<br />

CPU<br />

utilization<br />

(%)<br />

<strong>Network</strong><br />

throughput<br />

(MB/s)<br />

Data<br />

transferred<br />

(GB)<br />

Backup<br />

source<br />

size<br />

(GB)<br />

Duration<br />

(min)<br />

Initial 25.13 33.62 77.46 215 39.32<br />

1st data load 26.78 12.38 24.25 218 33.43<br />

2nd data<br />

load<br />

27.82 11.06 20.88 222 32.22<br />

3rd data load 27.45 11.73 22.75 225 33.10<br />

Note<br />

The above results were captured directly from DPA.<br />

Deduplication occurs when the <strong>SAP</strong> system has been initially backed up and has<br />

achieved a 64 percent deduplication rate (roughly, 3:1 ratio). When the data volume<br />

increases, the average deduplication ratio can rise up to 10:1. Compared to the<br />

baseline backup, the network utilization drops by 57 percent, and the CPU utilization<br />

increases by 21 percent.<br />

• Without PowerSnap – Single-stream mode<br />

CPU<br />

utilization<br />

(%)<br />

<strong>Network</strong><br />

throughput<br />

(MB/s)<br />

Data<br />

transferred<br />

(GB)<br />

Backup<br />

source size<br />

(GB)<br />

Duration<br />

(min)<br />

10.33 5.21 26.21 228 85.87<br />

Note<br />

The above results were captured directly from DPA.<br />

Working in single-stream mode does not fully utilize the network bandwidth and CPU<br />

recourses, resulting in longer backup duration over the multi-stream mode. The multistream<br />

backup finishes in less than half the time of the single-stream job.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

19


• With PowerSnap<br />

Scenario<br />

CPU<br />

utilization<br />

(%) on<br />

PRD<br />

CPU<br />

utilization<br />

(%) on<br />

proxy<br />

host<br />

<strong>Network</strong><br />

throughput<br />

(MB/s)<br />

Data<br />

transfer<br />

(GB)<br />

Backup<br />

source<br />

size<br />

(GB)<br />

Duration<br />

(min)<br />

Initial 0.28 18.91 28.56 75.05 321 44.85<br />

1st data<br />

load<br />

2nd data<br />

load<br />

3rd data<br />

load<br />

0.19 23.55 1.15 2.03 321 30.15<br />

0.18 23.59 1.11 1.96 321 30.20<br />

0.25 24.63 1.23 2.17 321 30.13<br />

SID Scenario Restore target size (GB) Duration (min)<br />

PRD Restore 321 49.57<br />

Note<br />

The above results were captured directly from DPA.<br />

The deduplication ratio of initial backup in PowerSnap mode is around 4:1. It<br />

increases sharply to 100:1 on average, with each data load.<br />

In the context of the PowerSnap backup, the proxy host takes over the workload of<br />

both the CPU and the network from the production host. The CPU utilization on the<br />

PRD host stays under 1 percent throughout the backup job (as expected).<br />

Backup using an external library (BACKINT) takes full advantage of client-based<br />

deduplication by transferring data through file-system-level copy. The amount of<br />

transferred data is almost identical to the size data volume increases in the database<br />

along with data load. The backup source size remains the same as there is sufficient<br />

free space in the database to accommodate the appended data.<br />

The restoration with BACKINT in PowerSnap mode is 1.6 times faster than the<br />

baseline.<br />

Test group 3:<br />

Baseline with<br />

BCT on<br />

Procedure<br />

Step Action<br />

1 Per<strong>for</strong>med initial backup <strong>for</strong> PRD with BCT turned on.<br />

2 Per<strong>for</strong>med data load with the same pattern as step 2 in the previous<br />

group. The subsequent backup with each data load was run in the<br />

incremental mode.<br />

3 Repeated the above step three times.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

20


Result<br />

Scenario<br />

CPU<br />

utilization<br />

(%)<br />

<strong>Network</strong><br />

throughput<br />

(MB/s)<br />

Data<br />

transfer<br />

(GB)<br />

Backup<br />

target<br />

size (GB)<br />

Duration<br />

(min)<br />

Initial 3.32 60.54 217.38 215 61.28<br />

1 st data load 0.41 36.32 4.75 4.16 2.23<br />

2 nd data load 1.19 43.05 7.95 7.4 3.15<br />

3 rd data load 1.97 44.27 10.82 10 4.17<br />

Note<br />

The above results were captured directly from DPA.<br />

Data transferred is slightly greater than the backup source size, because data<br />

transmission over the network has overhead on the carrier in the TCP/IP layer.<br />

Incremental backup enhanced by BCT shows outstanding per<strong>for</strong>mance in terms of<br />

the backup length. BCT reduces the backup duration because it only backs up the<br />

changed data since the last backup.<br />

Test group 4:<br />

<strong>Avamar</strong> on with<br />

BCT on<br />

Procedure<br />

Step Action<br />

1 Per<strong>for</strong>med backup <strong>for</strong> DEV and QAS to <strong>Avamar</strong> with BCT turned on.<br />

2 Per<strong>for</strong>med initial backup <strong>for</strong> PRD to <strong>Avamar</strong> with BCT turned on.<br />

3 Per<strong>for</strong>med data load with the same pattern as step 2 in the previous<br />

group. The subsequent backup with each data load was run in the<br />

incremental mode.<br />

4 Repeated the above step three times.<br />

5 Wiped out the PRD database and restored it from the most recent full<br />

backup plus incremental backup.<br />

6 Per<strong>for</strong>med a PIT restoration based on the previous step to revert the <strong>SAP</strong><br />

system to an earlier status.<br />

Result<br />

Scenario<br />

CPU<br />

utilization<br />

(%)<br />

<strong>Network</strong><br />

throughput<br />

(MB/s)<br />

Data<br />

transfer<br />

(GB)<br />

Backup<br />

target size<br />

(GB)<br />

Duration<br />

(min)<br />

Initial 22.56 29.58 75.48 215 43.55<br />

1 st data load 11.95 10.16 1.68 4.19 2.82<br />

2 nd data load 16.08 16.78 3.17 7.41 3.22<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

21


3 rd data load 16.43 19.78 4.75 10 4.1<br />

Note<br />

The above results were captured directly from DPA.<br />

Incorporating <strong>Avamar</strong> with BCT results in a 3:1 deduplication ratio <strong>for</strong> the initial<br />

backup and 2:1 on average <strong>for</strong> the subsequent incremental backup.<br />

The backup source size is similar to the one as documented in group 3. The only<br />

difference is that <strong>Avamar</strong> deduplicates the data be<strong>for</strong>e transferring it, which reflects<br />

the actual increased size of the database. This test shows that using NM<strong>SAP</strong> with<br />

the <strong>Avamar</strong> plug-in in conjunction with Oracle BCT enables faster incremental<br />

backups (15 times faster than the <strong>Avamar</strong>-enhanced full backup) with a reduced<br />

network utilization and storage footprint.<br />

SID Scenario Restore target size (GB) Duration (min)<br />

PRD Complete Restore 223 313.95<br />

PRD PIT Restore 215 300.27<br />

Note<br />

The above results were captured directly from DPA.<br />

The table shows that the restoration with BCT from <strong>Avamar</strong> is far slower than the<br />

baseline. There<strong>for</strong>e, this test scenario is not practical because it is too timeconsuming<br />

to recover the <strong>SAP</strong> production system.<br />

Results<br />

validation<br />

The following sections consolidate the test results and provide an in-depth analysis<br />

across test scenarios.<br />

<strong>Avamar</strong><br />

compared to<br />

baseline<br />

The <strong>SAP</strong> system backup can benefit from <strong>Avamar</strong> in terms of reducing the backup<br />

storage footprint as well as improving per<strong>for</strong>mance.<br />

The following section compares the configuration of baseline backup with that of an<br />

<strong>Avamar</strong> backup (BACKINT with <strong>Avamar</strong>).<br />

Backup storage footprint<br />

The delta size between two full backups is relatively small compared to the database<br />

size. For backup cycles, the conventional backup storage consumption increases<br />

linearly. When <strong>Avamar</strong> is introduced, the backup storage consumption increases<br />

with the same proportion as that of the delta size between two full backups <strong>for</strong> <strong>SAP</strong><br />

databases which decreases the backup storage footprint significantly.<br />

Thus, <strong>for</strong> very large (up to TB level) <strong>SAP</strong> databases, using <strong>Avamar</strong> as the backup<br />

destination facilitates the backup storage management with high deduplication.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

22


Per<strong>for</strong>mance<br />

As the graph below shows, the <strong>Avamar</strong>-enabled full backup completes in 20 percent<br />

less time while using 60 percent less bandwidth. The side effect is that <strong>Avamar</strong><br />

consumes more CPU resources at runtime because <strong>Avamar</strong> always runs a<br />

comparison of the differences between the data yet to be backed up and the data<br />

that has already been backed up.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

23


<strong>Avamar</strong><br />

compared to<br />

BCT<br />

Starting from Oracle 10g, Block Change Tracking (BCT) is provided to speed up<br />

database incremental backup. When BCT is active, Oracle uses the Block Change<br />

Tracking File (BCTF) book to list the data blocks that have been changed since the<br />

last RMAN level 0 backup. RMAN then uses the in<strong>for</strong>mation from the BCTF in<br />

incremental backups to read and back up only those blocks that have been changed.<br />

For detailed in<strong>for</strong>mation, refer to <strong>SAP</strong> Note 964619.<br />

The following comparison is between the configuration of BACKINT with <strong>Avamar</strong>,<br />

BCT without <strong>Avamar</strong>, and BCT with <strong>Avamar</strong>.<br />

Backup Storage Footprint<br />

The total data transferred to <strong>Avamar</strong> with initial and all three subsequent backups is<br />

around 81 GB, 66 percent less than the data conducted by incremental backup jobs<br />

using BCT only. When BCT works together with <strong>Avamar</strong>, it requires almost the same<br />

space as BACKINT with <strong>Avamar</strong>.<br />

Per<strong>for</strong>mance<br />

For initial backup (level 0) to the same <strong>SAP</strong> system, BCT without <strong>Avamar</strong> leads to<br />

higher network traffic and a longer backup duration over the pure <strong>Avamar</strong>-based<br />

backup, but requires less CPU resources. Incorporating BCT and <strong>Avamar</strong> brings<br />

similar results as BACKINT with <strong>Avamar</strong>.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

24


• Initial Full Backup<br />

Note<br />

The CPU utilization in the scenario of BACKINT backup with <strong>Avamar</strong> refers<br />

to the one that happens on the proxy host.<br />

• Subsequent Incremental Backup<br />

In subsequent incremental backup, BACKINT with <strong>Avamar</strong> saves much more<br />

network bandwidth, but BCT requires much less time to complete the backup. BCT<br />

with <strong>Avamar</strong> offers better per<strong>for</strong>mance than BCT only in terms of both average<br />

network bandwidth and CPU utilization.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

25


Note<br />

The CPU utilization in the scenario of BACKINT backup with <strong>Avamar</strong> refers<br />

to the one that happens on the proxy host. The charts presented above<br />

show the average measurement results of three data loads.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

26


As the following chart illustrates, <strong>Avamar</strong> with BACKINT restores the same <strong>SAP</strong><br />

system 6.3 times faster than BCT from <strong>Avamar</strong>.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

27


Conclusion<br />

Introduction<br />

This section provides a comprehensive summary to the product benefits and<br />

limitations through comparison of the functions of the key components.<br />

Client-based<br />

deduplication<br />

by <strong>Avamar</strong><br />

<strong>Avamar</strong> reduces the backup storage footprint significantly by eliminating duplicated<br />

data from the backup set. <strong>Avamar</strong> only transfers new or changed data, based on<br />

preselected settings, since the last backup.<br />

Conventional backup through Ethernet requires high network bandwidth to ensure a<br />

short backup window. It is ideal to have a dedicated subnet <strong>for</strong> the backup<br />

environment. When sharing network with other applications, both the backup job and<br />

the application efficiency will be affected. By using <strong>Avamar</strong>, administrators can<br />

minimize the utilized network bandwidth attributable to backup jobs and share the<br />

network with multiple applications instead. However, backup with <strong>Avamar</strong> causes<br />

higher CPU utilization on the client.<br />

<strong>Avamar</strong> and<br />

Oracle BCT<br />

<strong>Avamar</strong> offers optimized backup per<strong>for</strong>mance regarding network utilization and<br />

minimizes the backup storage space required <strong>for</strong> backup contents. However, Oracle<br />

BCT offers the ability to complete incremental backups in the least time. When BCT<br />

and <strong>Avamar</strong> are combined, customers can achieve not only data deduplication and<br />

average-level network bandwidth, but also a short backup window <strong>for</strong> incremental<br />

backup. However, the restoration time is not acceptable.<br />

Automation of<br />

snapshotbased<br />

backup<br />

using<br />

PowerSnap<br />

PowerSnap allows automation of the snapshot/mirror-based backup though<br />

NetWorker. In this solution, PowerSnap with the mirror-split mode per<strong>for</strong>ms backup<br />

to <strong>Avamar</strong> to eliminate the impact of CPU utilization on the production host when<br />

<strong>Avamar</strong> is introduced. Backing up <strong>SAP</strong> systems from the proxy client minimizes the<br />

per<strong>for</strong>mance impact on the <strong>SAP</strong> production server in terms of CPU, network, and<br />

disk I/O.<br />

Centralized<br />

management<br />

with NetWorker<br />

integration<br />

As a central-management backup solution, NetWorker offers seamless integration<br />

with <strong>SAP</strong>, <strong>Avamar</strong>, and PowerSnap. NM<strong>SAP</strong> in conjunction with NetWorker allows<br />

users to per<strong>for</strong>m a backup to <strong>Avamar</strong> in the multi-stream mode, which is 2.6 times<br />

faster than in a single-stream mode.<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

28


References<br />

White papers<br />

For additional in<strong>for</strong>mation, see the white papers listed below.<br />

• Backing Up Large <strong>SAP</strong> Oracle Databases Using <strong>EMC</strong> <strong>Avamar</strong> and Oracle RMAN -<br />

A Detailed Review<br />

• Backup of <strong>SAP</strong> Oracle Databases in <strong>EMC</strong> Symmetrix Environments with <strong>EMC</strong><br />

NetWorker PowerSnap - Applied Technology<br />

Product<br />

documentation<br />

For additional in<strong>for</strong>mation, see the product documents listed below.<br />

• <strong>EMC</strong> NetWorker Release 7.6 Installation Guide<br />

• <strong>EMC</strong> NetWorker Release 7.6 Administration Guide<br />

• <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle Release 4.0 Installation Guide<br />

• <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with Oracle Release 4.0 Administration Guide<br />

• <strong>EMC</strong> NetWorker PowerSnap Module <strong>for</strong> Symmetrix DMX Version 2.5 Installation<br />

Guide<br />

• <strong>EMC</strong> NetWorker PowerSnap Module <strong>for</strong> Symmetrix DMX Version 2.5<br />

Administration Guide<br />

• <strong>EMC</strong> <strong>Avamar</strong> 4.1 Server Software Installation Manual<br />

• <strong>EMC</strong> <strong>Avamar</strong> Backup Clients 4.1 Installation Guide and Reference Manual<br />

• <strong>EMC</strong> <strong>Avamar</strong> 4.1 System Administration Manual<br />

• Master Guide <strong>for</strong> <strong>SAP</strong> ERP 6.0<br />

Other<br />

documentation<br />

For additional in<strong>for</strong>mation, see the document listed below.<br />

• <strong>SAP</strong> Note 964619 - RMAN: Incremental backups with Block Change Tracking<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

29


Additional in<strong>for</strong>mation<br />

Introduction<br />

This section describes the sample scripts used in this white paper.<br />

Baseline -<br />

BR*Tools<br />

configuration<br />

file <strong>for</strong> RMAN<br />

initPRDrman.sap<br />

************************************************************************************************<br />

backup_mode = full<br />

restore_mode = full<br />

backup_type = online_cons<br />

backup_dev_type = rman_util<br />

backup_root_dir = /oracle/PRD/sapbackup<br />

stage_root_dir = /oracle/PRD/sapbackup<br />

compress = no<br />

archive_function = save_delete<br />

archive_copy_dir = /oracle/PRD/sapbackup<br />

archive_stage_dir = /oracle/PRD/sapbackup<br />

saveset_members = 1<br />

rman_channels = 4<br />

rman_parms = "ENV=(NSR_DEBUG_LEVEL=9, NSR_SERVER=networker,<br />

NSR_CLIENT=sapprd, NSR_DATA_VOLUME_POOL=UC334PRDF,<br />

NSR_NWPATH=/usr/sbin/)"<br />

util_par_file = /oracle/PRD/102_64/dbs/initPRDrman.utl<br />

************************************************************************************************<br />

initPRDrman.utl<br />

************************************************************************************************<br />

NSR_NWPATH = /usr/sbin/<br />

parallelism = 4<br />

ss_group_by_fs = TRUE<br />

expiration = 3 Months<br />

pool = UC334PRDF<br />

server = networker<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

30


client = sapprd<br />

ssNameFormat = new<br />

NSR_SAVESET_RETENTION = 3 Months<br />

NSR_SAVESET_BROWSE = 3 Months<br />

************************************************************************************************<br />

Baseline -<br />

BR*Tools<br />

configuration<br />

file <strong>for</strong><br />

PowerSnap<br />

initPRD.sap<br />

************************************************************************************************<br />

backup_mode = full<br />

restore_mode = full<br />

backup_type = online_cons<br />

backup_dev_type = util_file_online<br />

backup_root_dir = /oracle/PRD/sapbackup<br />

stage_root_dir = /oracle/PRD/sapbackup<br />

compress = no<br />

archive_function = save_delete<br />

archive_copy_dir = /oracle/PRD/sapbackup<br />

archive_stage_dir = /oracle/PRD/sapbackup<br />

util_par_file = /oracle/PRD/102_64/dbs/initPRD.utl<br />

************************************************************************************************<br />

initPRD.utl<br />

************************************************************************************************<br />

NSR_NWPATH = /usr/sbin/<br />

parallelism = 4<br />

ss_group_by_fs = TRUE<br />

expiration = 3 Months<br />

pool = UC334PRDF<br />

server = networker<br />

client = sapprd<br />

ssNameFormat = new<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

31


NSR_SAVESET_RETENTION = 3 Months<br />

NSR_SAVESET_BROWSE = 3 Months<br />

ps_backup_mode = TRUE<br />

ps_archive_mode = FALSE<br />

ps_restore_mode = TRUE<br />

ps_inquire_mode = TRUE<br />

ps_exclude_backup_bi_run_nums = 2<br />

ps_exclude_archive_bi_run_nums = 1;2<br />

ps_exclude_backup_paths =<br />

/oracle/PRD/102_64/dbs/*;/oracle/PRD/sapreorg/*;/oracle/PRD/sa<br />

pbackup/*<br />

ps_opaque_pfilename = /oracle/PRD/102_64/dbs/nsrsapps.cfg<br />

************************************************************************************************<br />

nsrsapps.cfg<br />

************************************************************************************************<br />

NSR_DATA_MOVER = sapprx<br />

SYMM_SNAP_REMOTE = FALSE<br />

SYMM_SNAP_TECH = BCV<br />

NSR_PS_DEBUG_LEVEL = 9<br />

SYMM_ON_DELETE = RETAIN_RESOURCE<br />

NSR_SNAP_TYPE = symm-dmx<br />

NSR_PS_SAVE_PARALLELISM = 4<br />

************************************************************************************************<br />

Baseline -<br />

NM<strong>SAP</strong><br />

configuration<br />

file <strong>for</strong> RMAN<br />

nsrsapsvrman.cfg<br />

************************************************************************************************<br />

BR_EXEC = brbackup -p initPRDrman.sap<br />

ORACLE_HOME = /oracle/PRD/102_64<br />

ORACLE_SID = PRD<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

32


NLS_LANG = AMERICAN_AMERICA.US7ASCII<br />

<strong>SAP</strong>DATA_HOME = /oracle/PRD<br />

<strong>SAP</strong>_BIN = /sapmnt/PRD/exe<br />

LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib<br />

OS_USR_PASSWD=xxxxxxxxxxxxxxxx<br />

************************************************************************************************<br />

Baseline -<br />

NM<strong>SAP</strong><br />

configuration<br />

file <strong>for</strong><br />

PowerSnap<br />

nsrsapsv.cfg<br />

************************************************************************************************<br />

BR_EXEC = brbackup -p initPRD.sap -d util_file_online -r<br />

/oracle/PRD/102_64/dbs/initPRD.utl -t online_cons -m all_data<br />

-k no -l E<br />

ORACLE_HOME = /oracle/PRD/102_64<br />

ORACLE_SID = PRD<br />

NLS_LANG = AMERICAN_AMERICA.US7ASCII<br />

<strong>SAP</strong>DATA_HOME = /oracle/PRD<br />

<strong>SAP</strong>_BIN = /sapmnt/PRD/exe<br />

LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib<br />

OS_USR_PASSWD=xxxxxxxxxxxxxxxx<br />

************************************************************************************************<br />

<strong>Avamar</strong> -<br />

BR*Tools<br />

configuration<br />

file <strong>for</strong> RMAN<br />

initPRDrmanavt.sap<br />

************************************************************************************************<br />

backup_mode = full<br />

restore_mode = full<br />

backup_type = online_cons<br />

backup_dev_type = rman_util<br />

backup_root_dir = /oracle/PRD/sapbackup<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

33


stage_root_dir = /oracle/PRD/sapbackup<br />

compress = no<br />

archive_function = save_delete<br />

archive_copy_dir = /oracle/PRD/sapbackup<br />

archive_stage_dir = /oracle/PRD/sapbackup<br />

saveset_members = 1<br />

rman_channels = 4<br />

rman_send=(<br />

"'NSR_ENV=(NSR_DEDUP_BACKUP=TRUE,<br />

NSR_DEDUP_NODE=avamar01-u)'",<br />

"channel sbt_1 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_1)'",<br />

"channel sbt_2 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_2)'",<br />

"channel sbt_3 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_3)'",<br />

"channel sbt_4 'NSR_ENV=(NSR_DEDUP_CACHE_TAG=prdavt_4)'"<br />

)<br />

rman_parms = "ENV=(NSR_DEBUG_LEVEL=9, NSR_SERVER=networker,<br />

NSR_CLIENT=sapprd, NSR_DATA_VOLUME_POOL=UC334PRDFAVT,<br />

NSR_NWPATH=/usr/sbin/)"<br />

util_par_file = /oracle/PRD/102_64/dbs/initPRDrmanavt.utl<br />

************************************************************************************************<br />

initPRDrmanavt.utl<br />

************************************************************************************************<br />

NSR_NWPATH = /usr/sbin/<br />

parallelism = 4<br />

ss_group_by_fs = TRUE<br />

expiration = 3 Months<br />

pool = UC334PRDFAVT<br />

server = networker<br />

client = sapprd<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

34


ssNameFormat = new<br />

NSR_SAVESET_RETENTION = 3 Months<br />

NSR_SAVESET_BROWSE = 3 Months<br />

************************************************************************************************<br />

<strong>Avamar</strong> -<br />

BR*Tools<br />

configuration<br />

file <strong>for</strong><br />

PowerSnap<br />

initPRD.sap<br />

************************************************************************************************<br />

backup_mode = full<br />

restore_mode = full<br />

backup_type = online_cons<br />

backup_dev_type = util_file_online<br />

backup_root_dir = /oracle/PRD/sapbackup<br />

stage_root_dir = /oracle/PRD/sapbackup<br />

compress = no<br />

archive_function = save_delete<br />

archive_copy_dir = /oracle/PRD/sapbackup<br />

archive_stage_dir = /oracle/PRD/sapbackup<br />

util_par_file = /oracle/PRD/102_64/dbs/initPRD.utl<br />

************************************************************************************************<br />

initPRDavt.utl<br />

************************************************************************************************<br />

NSR_NWPATH = /usr/sbin/<br />

parallelism = 4<br />

ss_group_by_fs = TRUE<br />

expiration = 3 Months<br />

pool = UC334PRDFAVT<br />

server = networker<br />

client = sapprd<br />

ssNameFormat = new<br />

NSR_SAVESET_RETENTION = 3 Months<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

35


NSR_SAVESET_BROWSE = 3 Months<br />

ps_backup_mode = TRUE<br />

ps_archive_mode = FALSE<br />

ps_restore_mode = TRUE<br />

ps_inquire_mode = TRUE<br />

ps_exclude_backup_bi_run_nums = 2<br />

ps_exclude_archive_bi_run_nums = 1;2<br />

ps_exclude_backup_paths =<br />

/oracle/PRD/102_64/dbs/*;/oracle/PRD/sapreorg/*;/oracle/PRD/sa<br />

pbackup/*<br />

ps_opaque_pfilename = /oracle/PRD/102_64/dbs/nsrsapps.cfg<br />

************************************************************************************************<br />

nsrsapps.cfg<br />

************************************************************************************************<br />

NSR_DATA_MOVER = sapprx<br />

SYMM_SNAP_REMOTE = FALSE<br />

SYMM_SNAP_TECH = BCV<br />

NSR_PS_DEBUG_LEVEL = 9<br />

SYMM_ON_DELETE = RETAIN_RESOURCE<br />

NSR_SNAP_TYPE = symm-dmx<br />

NSR_PS_SAVE_PARALLELISM = 4<br />

<strong>Avamar</strong> -<br />

NM<strong>SAP</strong><br />

configuration<br />

file <strong>for</strong> RMAN<br />

nsrsapsvrmanavt.cfg<br />

************************************************************************************************<br />

BR_EXEC = brbackup -p initPRDrmanavt.sap<br />

ORACLE_HOME = /oracle/PRD/102_64<br />

ORACLE_SID = PRD<br />

NLS_LANG = AMERICAN_AMERICA.US7ASCII<br />

<strong>SAP</strong>DATA_HOME = /oracle/PRD<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

36


<strong>SAP</strong>_BIN = /sapmnt/PRD/exe<br />

LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib<br />

OS_USR_PASSWD=xxxxxxxxxxxxxxxx<br />

************************************************************************************************<br />

<strong>Avamar</strong> -<br />

NM<strong>SAP</strong><br />

configuration<br />

file <strong>for</strong><br />

PowerSnap<br />

nsrsapsvavt.cfg<br />

************************************************************************************************<br />

BR_EXEC = brbackup -p initPRD.sap -d util_file_online -r<br />

/oracle/PRD/102_64/dbs/initPRDavt.utl -t online_cons -m<br />

all_data -k no -l E<br />

ORACLE_HOME = /oracle/PRD/102_64<br />

ORACLE_SID = PRD<br />

NLS_LANG = AMERICAN_AMERICA.US7ASCII<br />

<strong>SAP</strong>DATA_HOME = /oracle/PRD<br />

<strong>SAP</strong>_BIN = /sapmnt/PRD/exe<br />

LD_LIBRARY_PATH = /oracle/PRD/102_64/lib:/usr/lib/nsr:/usr/lib<br />

OS_USR_PASSWD=xxxxxxxxxxxxxxxx<br />

************************************************************************************************<br />

<strong>EMC</strong> Backup and Recovery <strong>for</strong> <strong>SAP</strong><br />

Enabled by <strong>EMC</strong> NetWorker Module <strong>for</strong> <strong>SAP</strong> with <strong>EMC</strong> <strong>Avamar</strong>—A Detailed Review<br />

37

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!