RE: RMAN Performance Maladies

Thanks for the detailed reply.  

Just to reiterate or elaborate on my configuration:  

We are NOT backing up to TAPE.  

We are backing up to a SAN disk array (RAID5). This is what is recommended as a compromise between performance and efficiency by our disk storage management team.  

We are using 4 channels, not 8, as someone had inferred.  

Alas, as is confirmed here, the statistics in the views seem to indicate we're getting about the advertised throughput - and Oracle confirms that it is "acceptable". But it puts a heavy load on our server, which in turn causes application performance issues for the 4 databases running on it while RMAN is backing up this big one. Very disappointing considering we're moving from a vendor solution (BMC SqlBackTrack) which did the backup in @30 minutes without so much as a peep.  

....Keep the good responses coming - I am reading them all!!!      

From: Naqi Mirza [mailto:naqimirza_at_yahoo.com] Sent: Monday, February 12, 2007 1:15 PM
To: Michael Fontana; oracle-l_at_freelists.org Subject: Re: RMAN Performance Maladies

Hi,
I know one site that has their 4TB database backup in almost 18 hours, they are on 9.2, using veritas netbackup 5. What are you using by the way, I seemed to have missed that in the post? You also haven't mentioned whether your backups are to disk or tape? If tape then which tape technology. If its LTO and i remember correctly, LTO3 is the latest and each drive has a maximum rate of 80mb/s, So you may want to be the math for that and work out what you can achieve - at least with all things being equal. Which is really a question in itself - are all drives streaming with data and if so to what extent?. See from what I have come to understand about rman - you will only be able to write data to tape at least as fast as you can read it, if you're not reading fast enough you won't be able to write much either. Now, there are two dynamic performance views you can query to obtain this information. Depending o n how the backup is being done-v$backup_io and v$backup_aysnc_io (hope thats correct, do check the oracle reference manual). Have a look at the column - effective_bytes_per_second with respect to the column type (values here of INPUT - is what you are reading at, and OUTPUT is what you are writing at - either to disk or tape). See if you are reading at decent speed , then check to see if you are writing at least at this rate - you can also check with your backup software to see the throughput that is being achieved. The problem I witnessed at a certain site - never got resolved, the site was also using RAID 5 - I suspected this as the culprit - however incompetence on my part, I could never really prove it. A few more things which you can check - since i did the same. Is see whether a a file copy of a particular datafile results in the same rate being achieved by your backup software. The issue we had was that netbackup would show the total throughput of the backup as 50mb/s no matter what was done - channel increase, maxopenfiles, filesperset, etc. The sysadmin at the site claimed that rman was the problem and that o/s backups would typically give them 100mb/s. I removed rman from the equation, and had netbackup backup a datafile in backup mode - guess what it still showed a throughput of 50mb/s - so it wasn't rman and this is what I believe, alas, to this day.
Do check iostat , during your backup runs - does it reveal any io contention that could be hindering the backup. Hows your cpu usage during backups - rman is 'apparently' heavy on the cpu. One more thing to add to the equation here, I know another site, which i used as a comparison to the 18 hour site. Comparing the effective_bytes_per_Second there showed a tremendous difference, they were reading alot more data and therefore writing more too, thus , there 1.5TB database would complete its backup in about 2-3hours (cannot recall now exactly which).
Well, I have digressed a bit here, hoping perhaps someone might be able to give me a clue or two too. Hope what I have written gives you a start - checkout the otn.oracle.com site - there are a few papers on there that help you troubleshoot rman performance problems.

Naqi

• Original Message ---- From: Michael Fontana <MFontana_at_verio.net> To: oracle-l_at_freelists.org Sent: Monday, 12 February, 2007 9:15:22 PM Subject: RMAN Performance Maladies

We have just implemented RMAN against our largest Siebel implementation (don't laugh - but it's less then 1/2 terrabyte). We did so because now that we have upgraded to Oracle 10gR2 (10.2.0.2) we can take advantage of flash recovery features and full backup compression similar to which we achived with our old backup product (BMC Sqlbacktrack). We customarily take full backups nightly, as per our SLA, and at this point, incrementals are not an option. However, the actual runtime to complete a full database backup with RMAN compared to SqlBackTrack has gone from 30 minutes to over 4 1/2 hours!

I have seen anecdotal reports of long runtimes for full RMAN backups on this list, but I've seen an overwhelming number of guffaws from the more elite and experienced poster indicating that this must be user error or some other database performance problem that any competent DBA should easily be able to solve without posting to this list. However, even after opening an SR with Oracle support, who seems to believe our runtime is in the expected range, we still believe something can be done to overcome the huge IO spike and slow performance we are witnessing during the backup, and we are not that experienced with RMAN from a performance and tuning perspective, and would appreciated anyone's insights or experiences.

My question is - has anyone actually run RMAN against a large (> 200g) database right out of the box without tuning?

Has anyone spent a great deal of time tuning?

We are running 4 channels, have an 8 cpu Sunsparc solaris with 16g of memory, and have experimented with recommended parameter settings for maxsetsize and maxfilesperset without success. Any information or assistance or comment would be appreciated, thanks in advance!

--

http://www.freelists.org/webpage/oracle-l

Inbox full of unwanted email?
<http://us.rd.yahoo.com/mail/uk/taglines/default/nowyoucan/spam_1gb/*htt p://us.rd.yahoo.com/evt=40565/*http://uk.docs.yahoo.com/nowyoucan.html> Get leading protection and 1GB storage with All New Yahoo! Mail.

--

http://www.freelists.org/webpage/oracle-l Received on Mon Feb 12 2007 - 15:17:23 CST