NetApp SnapVault

Netapp SnapVault is a heterogeneous disk-to-disk backup solution for Netapp filers and heterogeneous OS systems (Windows, Linux , Solaris, HPUX and AIX). Basically, Snapvault uses netapp snapshot technology to take point-in-time snapshot and store them as online backups. In event of data loss or corruption on a filer, the backup data can be restored from the SnapVault filer with less downtime. It has significant advantages over traditional tape backups, such as:

• Reduce backup windows versus traditional tape-based backup
• Media cost savings
• No backup/recovery failures due to media errors
• Simple and Fast recovery of corrupted or destroyed data

Snapvault consists of major two entities –  snapvault clients and a snapvault storage server. A snapvault client (Netapp filers and unix/windows servers) is the system whose data should be backed-up.  The SnapVault server is a Netapp filer – which gets the data from clients and backs up data. For Server to Netapp Snapvault, we need to install Open System Snapvault client software provided by Netapp, on the servers. Using the snapvault agent software, the Snapvault server can pull and backup data on to the backup qtrees. SnapVault protects data on a client system by maintaining a number of read-only versions (snapshots) of that data on a SnapVault filer. The replicated data on the snapvault server system can be accessed via NFS or CIFS. The client systems can restore entire directories or single files directly from the snapvault filer.  Snapvault requires primary and secondary license.

NetApp Snapmirror And Its Advantages

NetApp® SnapMirror® software has been the preferred technology for replication and disaster recovery in a wide variety of NetApp storage environments for years because of its proven efficiency, simplicity, and modest cost when compared with other DR solutions. Over the years, NetApp has continued to enhance SnapMirror with new features and capabilities to make the product fit an even broader range of requirements and to use network bandwidth even more efficiently.

The use of SnapMirror technology offers significant advantages:

Efficient
Block-level updates reduce network bandwidth and time requirements. Starting with Data ONTAP® 7.3.2, volume SnapMirror also offers native network compression to further reduce bandwidth costs.

Flexible
Data can be replicated between dissimilar NetApp storage systems. One-to-one, one-to-many, many-to-one, or many-to-many replication topologies are supported with async mode.

More productive
When you use SnapMirror in combination with NetApp FlexClone®, you can use the data stored in your DR environment for dev/test, data mining, or other purposes.

Consistent. Through integration with the NetApp SnapManager® suite, application data can be replicated while making sure of full consistency for quick recovery.

Safe
Your DR plan can be tested without affecting production and ongoing replication so you can test more frequently to make sure there aren’t any surprises should disaster strike. To protect against application data corruption, your DR site can keep multiple Snapshot® copies on hand and quickly and easily restore to a point in time before the data corruption occurred.

There are two operating modes for SnapMirror: volume and qtree. Volume SnapMirror is generally the preferred mode. Because of its relative popularity, much of our development effort, including integration with the SnapManager suite of products, has focused on volume SnapMirror. As a result, volume SnapMirror offers greater flexibility and efficiency. This chapter of Back to Basics explores how volume SnapMirror technology is implemented, the most common use cases, best practices for implementing SnapMirror, and more.

Avamar Checkpoints

Avamar Checkpoints
Checkpoints are system-wide backups taken for the express purpose of assisting with disaster recovery. Checkpoints are typically scheduled twice daily and validated once daily (during the maintenance window). You also can create and validate additional server checkpoints on an on-demand basis.

Checkpoint validation, which is also called an Avamar Hash Filesystem check (HFS check), is an internal operation that validates the integrity of a specific checkpoint. Once a checkpoint has passed an HFS check, it can be considered reliable enough to be used for a system rollback.

The actual process that performs HFS checks is hfscheck; it is similar to the UNIX fsck command.

You can schedule HFS checks by using Avamar Administrator. You also can manually initiate an HFS check by running avmaint hfscheck directly from a command shell.

An HFS check might take several hours depending on the amount of data on the Avamar server. For this reason, each validation operation can be individually configured to perform all checks (full validation) or perform a partial "rolling" check which fully validates all new and modified stripes, then partially checks a subset of unmodified stripes.

Initiating an HFS check requires significant amounts of system resources. To reduce contention with normal server operation, an HFS check can be throttled. Additionally, during this time, the server is placed in read-only mode. Once the check has been initiated, normal server access is resumed. You can also optionally suspend command dispatches during this time, although this is not typically done.  If HFS check detects errors in one or more stripes, it automatically attempts to repair them.

Avamar - Clients Agents Plug-ins

Avamar clients
Avamar provides client software for various computing platforms. Each client comprises a client agent and one or more plug-ins.

Agents
Avamar agents are platform-specific software processes that run on the client and communicate with the Management Console Server (MCS) and any plug-ins installed on that client.

Plug-ins
The following topics provide details on the two types of Avamar plug-ins.

File system plug-ins
File system plug-ins are used to browse, back up, and restore files or directories on a specific client file system. Avamar currently provides plug-ins for the following operating systems:

• Free BSD
• HP-UX
• IBM AIX
• Linux
• Mac OS X
• Microsoft Windows
• Microsoft Windows Volume Shadow Copy Service (VSS)
• SCO OpenServer
• SCO UnixWare
• Oracle Solaris
• Novell NetWare
• VMware

Application plug-ins
Application plug-ins support backup and restore of databases or other special applications. Avamar currently provides plug-ins for the following applications:

• IBM DB2
• Lotus Domino
• Microsoft Exchange
• Microsoft Hyper-V
• Microsoft Office SharePoint Server (MOSS)
• Microsoft SQL Server
• NDMP for NAS devices, including EMC Celerra IP storage systems and Network Appliance filers
• Oracle
• SAP with Oracle
• Sybase ASE

Avamar Encryption

Avamar Encryption

To provide enhanced security during client/server data transfers, Avamar supports two levels of “in-flight” encryption: Medium and High. The exact encryption technology and bit strength used for any given client/server connection depends on a number of factors, including the client platform and Avamar server version.

The default encryption method is used for client/server data transfers (None, Medium, or High) when you create and edit groups.  The group encryption method can also be overriden for a specific client on the Client Properties tab of the Edit Client dialog box, for a specific backup on the On Demand Backup Options dialog box, or for a specific restore on the Restore Options dialog box.

To enable encryption of data in transit, the Avamar server data nodes each require a unique public/private key pair and a signed X.509 certificate that is associated with the public key.

When the Avamar server is installed, a public/private key pair and a self-signed certificate are generated automatically in the /data01/home/admin directory on each Avamar server storage node and in the /usr/local/avamar/etc directory on the utility node. However, because self-signing is not recommended in production environments, you should generate and install a key and signed certificate from either a commercial or private CA.

Each individual Avamar server can also be configured to encrypt data stored on the server “at rest.” The decision to encrypt all data stored in an Avamar server is typically a one-time decision that is made when the server is initially deployed at a customer site.

Avamar NDMP Accelerator Node for NAS

Avamar NDMP Accelerator Node

The limited scalability of traditional single system NAS platforms has been well documented as an issue.  Less well documented are the issues that can arise when the backup and recovery process can't keep up with the growth.

That is where the Avamar NDMP Accelerator node come in,  it support multiple storage devices and up to 8 simultaneous streams, performing real-time data dedupe.  Of course, there needs to be sufficient RAM and configuration on the accelerator. It supports VNX, Celerra, and NAS devices running Data ONTAP.  It supports NDMP version 4 over 100/1000 Mb/sec Ethernet and will work over a LAN or WAN.

The NDMP backups contain all storage device accounting information and ACLs.  Only volumes are visible, and include/exclude lists are not supported.  When combined with VNX or Celerra, the NDMP accelerator performs incremental backup at the volume level only and defaults to full backup of sub-root directories.  Best practice is to not backup more than 10 million files in a single backup job, which can be overridden in the config but jobs may fail for lack of memory with too many files.

When backing up NetApp Filers, the NDMP accelerator is backward compatible to Avamar 4.1 servers, but version 5.0 or newer is recommended.  The appliance will backup and restore volumes, qtrees and directories, and SnapVault snapshots must be backed up in full.

The NDMP accelerator will perform file level restores, as well as single directories and sub-directories, but data backed up on other Avamar client types are not restorable to a NAS - if it came from NDMP, it stays on NDMP and vice-versa.  There is no web restore with external authentication - users must have local authentication, and ACLs are fully supported, but there is no control for overwrite options.

Both LAN and WAN location of the NDMP device are supported, but it is recommended to have the accelerator local to the device being protected.   8 GB of memory is required in the appliance for multiple streams, and the recommendation is to have 4GB of memory per stream (8 streams max).

Multi-volume accounts will attempt to backup all volumes simultaneously, opening up to the possibility that there would be too many simultaneous streams.  It is best practice to combine all multi-volume accounts into one account and have a single multi-volume backup using the avsetupndmp script.

EMC NetWorker

EMC NetWorker

EMC NetWorker helps you protect your data by simplifying and centralizing backup and recovery operations. With its record-breaking performance, NetWorker is the ideal backup software for small offices as well as large data centers.

With the appropriate NetWorker modules, you can gain protection for popular database, messaging, content, and enterprise resource planning (ERP) applications. You can also choose from a menu of options to add specialized functionality.

Below are the lists of benefits of EMC NetWorker:

Open architecture

• Ensure reliable backup and recovery across local area network (LAN), wide area network (WAN), and storage area network (SAN) environments.

Heterogeneous support

• Simplify management with one solution for UNIX, Microsoft® Windows, Linux, NetWare, OpenVMS, Macintosh, and VMware virtualized systems.

“Hot” backup and recovery

• Back up enterprise applications and file systems while they are in production.

Centralized backup and recovery

• Gain complete control of your data across a variety of environments.

Global data deduplication

• Speed backups, reduce bandwidth consumption, and lower storage requirements by eliminating duplicate data at the source and target.

Parallelism and multiplexing

• Optimize use of your drives and libraries for rapid data protection in large environments.

Advanced staging and cloning

• Create multiple copies of backup data to reduce risk and ensure business continuity.

Open tape format

• Port tapes between different NetWorker servers and operating systems and migrate backups between platforms

NetApp Most Commonly Used Commands

NetApp Most Commonly Used Commands

man (man pages)
Browses through man(ual) pages of command documentation

sysconfig (-a, -r, -c, -t, -m)
Shows information about filer (hardware, disks, aggregates & RAID groups, ...)

options
Queries or changes values for various “registry” options setup. Walks through initial setup questions: filer name, IP addresses, etc., but does not erase any data

cifs setup
Walks through CIFS setup questions: domain/workgroup membership etc.

sysstat -x -s 1
Prints out all-round performance statistics

license
Adds/removes/prints licenses on filer

version (-b)
Prints out Data ONTAP & Diagnostics/Firmware version numbers

rdfile
Reads a text file and prints contents to console (Unix “cat”)

wrfile
Reads from console and sends output to text file (Unix “Cat >”)

snap (restore)
Performs snapshot operations, eg. restore from snapshot snapvault

snapmirror
Manipulates/controls SnapMirror/SnapVault/OSSV operations from the command-line

cf
Controls clustering, eg. enabling/disabling, forcing takeover & giveback

aggr
Creates/expands/destroys/manipulates aggregates, eg. change options

vol
Create/resizes/destroys/manipulates volumes, eg. change options

df
Shows free disk space (volumes, aggregates, also inodes)

qtree
Creates/manipulates qtrees (=special directories)

vif
Creates/destroys/manipulates virtual network interfaces (eg.team interfaces for failover or load-balancing)

ifconfig
Sets network IP configuration (put in /etc/rc to survive reboots)

ifstat
Shows network interface statistics

netdiag
Performs basic network diagnostic testing

ndmpd
ndmpcopy
Manipulates NDMP settings, or use ndmpcopy to copy files via NDMP

priv set
priv set advanced
priv set diag

Goes into advanced/diagnostics mode

Avamar Virtual Edition (AVE) and VM Backup Solutions

Avamar Virtual Edition for VMware also streamlines deployment. But instead of a hardware appliance packaging, the Avamar software is delivered as a virtual appliance that is able to run in a VMware virtual machine and leverage shared server and storage resources. Avamar Virtual Edition has the added benefit of introducing cost-effective virtual-to-virtual (V2V) or virtual-to-physical (V2P) disaster recovery through its replication feature.

Avamar’s de-duplication capability enhances other supported backup approaches for virtual server environments, including VMware Consolidated Backup, a backup agent in the ESX Service Console and a backup agent in the guest operating system. Data de-duplication reduces the traditional backup load on the virtual machine’s shared physical resources—CPU, memory, NIC and disk storage.

VMware Consolidated Backup (VCB) performs a snapshot of the production virtual disk (containing the operating system, applications and data for a virtual machine). The VCB snapshot is copied to a proxy server where the backup application can back it up. The virtual disk image (.vmdk extension file) appears as a new file every day, which traditional backup applications will back up every time. The VCB approach removes the backup processing burden from the ESX Server host by offloading it
to the VCB proxy server.  It also enables full-image backups of running virtual machines and file-level backups for Windows virtual machines, and eliminates the need to manage backup agents in each virtual machine for most scenarios.  Installing an Avamar agent on the VCB proxy server enables subfile de-duplication of files and the virtual disk (.vmdk) images.

ESX Server Service Console-Based Backup requires an Avamar software agent installed on the service console of each ESX Server host that needs protection. Using this method, de-duplication occurs at the ESX Service Console and the administrator can choose to back up the virtual machine either online or offline. Advantages of this method include less management overhead since backup agents are not needed inside the virtual machines; no backup proxy server or SAN required since backup occurs on the ESX Server host; and recovery is performed directly to the ESX Server host.
ƒ
Guest Level Backup requires installing a backup agent inside each virtual machine guest to communicate with the backup application (same general approach used to backup a physical server). This method enables full, incremental and application-specific backup, but it is very burdensome on the host system’s shared resources. Client-based de-duplication such as Avamar’s helps significantly. Installing an Avamar agent within each virtual machine guest enables de-duplication within and across
virtual machines to significantly reduce the backup strain on shared physical resources and applications, as well as the amount of data stored to disk.

Avamar and VMware

Avamar software quickly and efficiently protects VMware Infrastructure environments by reducing the size of backup data within and across virtual machines - using agents in the virtual machines or on the VMware vStorage APIs for Data Protection proxy server.

For virtual machine backups, Avamar eliminates traditional backup bottlenecks caused by the large amount of redundant data that must pass through the same set of shared resources - the physical server's CPU, Ethernet adapter, memory, and disk storage.

Avamar reduces the traditional backup load - up to 200 percent weekly - to as little as 2 percent over the same day period, dramatically reducing backup times and resource utilization.

Key Avamar benefits include:

• Up to 10x faster daily full backups
• Up to 500:1 reduction in required daily network bandwidth
• Up to 50:1 reduction in required global backup storage media
• Encryption of backup data in flight and at rest
• Fault tolerance across Avamar nodes and elimination of single points of failure using the patented RAIN technology
• Scalable grid architecture
• Daily server integrity and data recoverability checks
• Simple one-step recovery
• Flexible deployment options, including EMC Avamar Data Store and EMC Avamar Virtual Edition for VMware (a virtual appliance)
• Improved physical server consolidation rations

Avamar Integration with Data Domain

Starting from Avamar release v6.0 onwards, not only comes with more new features, Avamar provides support for integration with Data Domain.

On the hardware side, this release offers higher density nodes with greater performance. Nodes are now offered in 4 different capacities: 1.3 TB, 2.6 TB, 3.9 TB, and 7.8 TB. The standard Utility and Accelerator nodes continue to be available and serve the same function in Avamar v6.0 as they did in previous releases. What is more, is the performance of the node relative to important internal system maintenance and upkeep scales with capacity. The 7.8 TB node offers twice the performance of the 3.9 TB node. For some important tasks like garbage collection and file system checks this means there is no longer any good reason not to use the highest density nodes even for data with exceptionally high change rates.

Even better, prices have declined while capacities have gone up-in some cases the cost per TB has improved by more than 20%. And the power consumption per TB has also dropped considerably-by as much as 65% in the case of the 7.8 TB nodes. All this equals bigger grids which cost less to acquire, less to power and cool in the data centre, and use less of your valuable raised floor square footage to store data.

Under the covers there are lots of significant performance improvements to the new nodes to make sure that things work better than before: EMC has leveraged the Intel multi-core architecture to speed up startup and checking of data stripes. Memory utilization is improved, and overall backup performance increases.

One of the biggest changes that many customers willnotice will be the changes to the networking architecture. The lines between internal and external networks are no longer blurred. There is a network that is clearly an internal management and administration network (which is redundant). And customers will connect an Avamar grid to their external network to provide for backup bandwidth and redundancy if desired. (As an aside, for those of you that want Cisco everywhere in your network, it is now possible to connect your Avamar Data Store nodes directly to your Cisco network. You can maintain the homogeneity of your IP network and continue to manage and support only Cisco devices.)

On the software side, the improvements are every bit as significant:

• Support for EMC Secure Remote Support (ESRS)
• A new installer/updater service for client and server code upgrades and patches that is customer accessible
• An Avamar client manager that enables management of large groups of clients with a single pane of glass, and allows wizard driven task execution against groups of clients. (Another aside: it also allows moving a client or a group of clients from one Avamar grid to another. For those of you managing large numbers of desktop and laptop clients, or desktop and laptop clients for a globally distributed workforce, this will be a particularly welcome addition that will make administration a lot easier.)
•  Overall, there have been a least a dozen significant enhancements (and many minor updates) to the desktop and laptopbackup experience with the Avamar DT/LT client that will make Avamar easier, faster, more flexible, and easier to support for these environments. From an end user perspective, the self-service recovery window has been improved, which is a welcome upgrade. From an administrative point of view, the administration of large numbers of clients has been made easier too.
• Improvements have been made across a wide range of general clients, and specific modules, including the backup of Oracle systems, Oracle RAC systems, NDMP backups, backups for Iomega ix12 systems (a very cool feature which essentially puts an Avamar client on the Iomega storage system for powerful remote backup of these systems), VMware Image Backup enhancements, Windows 2008 server backups, SQL server systems, Exchange backup, and a host of others. I will return to the topic of Windows applications backup and VMware backup in the near future, because there is enough good stuff there to warrant a separate discussion.

And, to leave the best for last: Avamar is now offering integration with Data Domain systems.

With release 6.0 of the Avamar software, EMC will now support Data Domain systems as a backup target. This combines the performance and scale of the Data Domain systems with the ease of use and simplicity of the Avamar software. Now any application, of any size, can be protected with Avamar. And the scale of Avamar grows very significantly-with the addition of up to 285 TB of capacity per Data Domain system (and up to 4 systems supported per Avamar server).

In the initial release, EMC is supporting the backup of Oracle, SQL, Exchange, SharePoint, and VMware Images to Data Domain systemswith Avamar software. The Avamar GUI will manage the backup, restore, and replication of data for these systems, as well as monitoring and reporting on the Data Domain systems that are the backup targets. Avamar clients get an embedded DD BOOST v2.3.1, which is responsible for actually sending the backup data from the client to the target Data Domain system. Now, even very largedatabases and database with very high daily change rates can be easily integrated into Avamar.

Overview of Avamar

Overview of Avamar - Avamar Node Types and Processes
There are several different types of Nodes within an Avamar Server.

Utility Node - The identity of the Avamar Server which provides the bulk of the internal Avamar server processes such as: Avamar Administrator (used to manage the Avamar Server from a gui), cron jobs, DNS, NTP, external authentication, web access, MCS and EMS.

Storage Node - This node run a process called gsan.  This service communicates with the avtar command on the individual backup clients.  Avtar communicates with a storage nodes gsan process, then that storage node spreads the data across the available data nodes.

Spare Node - An active node that  is present in a multi-node RAIN (Redundant Array of Independent Nodes) grid.  The interesting thing about the Spare Node is that it is NOT a hot spare.  There is a procedure in place that needs to be followed when a failed Storage Node occurs.  In fact, the Avamar Server doesn’t even care if the Spare Node is powered on, so if you are a “Green” conscience company, feel free to leave the Spare Node powered down.  EMC Best Practice is to leave the Spare Node up and active though.

Avamar Backups clients can be installed with 2 different types of client plug-ins: File System and Databases.  The Avamar client uses 2 processes in the backup and restore process: Avtar and Avagent.  The Avagent process listens for backup/restore work orders from the MCS service on the Utility Node using port 28002 and executes the avtar command that handles the backup/restore processes and communicates with the gsan service on the storage nodes.  This process is depicted in the included picture above.

MCS (Management Console Server) provides centralized management including scheduling of backups, restore of backups, monitoring and reporting.

EMS (Enterprise Manager Server) provides web based management for multiple Avamar Servers as well as monitoring and configuration for Avamar Replication.

Web Access provides access to documentation, backup plug-ins as well as remote File System  restore access to end users.