EMC (SRDF) and Sybase

Discussion:

(too old to reply)

unknown

2008-05-15 07:40:29 UTC

We are getting ready for a receive server handover for a new
system. We will be providing DBA support for the new
system. Our user / system owner has been given a quote by
an outside vendor that they should implement a Sybase / SUN
cluster / EMC (SRDF) solution.

It would look something like this...

Primary Datacenter:
- SUN Cluster Server (SUN box)
- Sybase ASE (version 15)
- EMC storage
- EMC SRDF replication

DR Site / Datacenter (cold standby):
- Sun Cluster Server (SUN box)
- EMC storage
- EMC SRDF replicated copy.
NOTE: This is a cold standby solution. EMC disks are
mounted upon failover (where replicated copies of Sybase ASE
is located and raw devices are located)

IMPORTANT NOTE: The acceptable system downtime is 2 hours.

We think that this is an overkill solution and does not meet
all DR and/or recovery needs. For example...

1) Does not address database corruption.
2) Does not protect against human error... (Joe Smith
deletes important binary files from the "sybase" directory /
installation or raw device path locations are altered or
deleted).
3) Abrupt shutdown of Sybase dataserver causes database
corruption.

We are recommending using Sybase Replication Server instead
of a "Cluster Server / EMC (SRDF)" solution.

Benefits of a Sybase Replication Server solution...

1) Provides a "Warm-Standby", which is immediately
available.
2) If there is database corruption, it will not be
replicated to "Warm-Standby" environment.
3) If there is dataserver catostrophic failure, the
"Warm-Standby" is available immediately.

Not to mention... the user / system owner SLA gives us a 2
hour recovery window. We think that Sybase Replication
Server is an appropriate solution to meet the SLA. Where
the "Cluster / EMC (SRDF)" solution is overkill and does not
address all recovery needs.

What are your thoughts?

Sincere thanks for sending your feedback.

unknown

2008-05-15 12:26:34 UTC

Permalink

I prefer SRDF.

Another option would be to use "Mirror Activator" layered on top of SRDF.
It mirrors only the transaction logs and reads the transactions to rebuild
the data on the secondary system. We did not use it because our devices had
data from one database and log from another database.But it appears to be a
good product.

SRDF is maintenance free style replication. We had both. SRDF was bullet
proof.
--
Cory Sane
[Member of TeamSybase]
Certified Sybase Associate DBA for ASE 15.0
not a Sybase Inc. employee

Post by unknown
We are getting ready for a receive server handover for a new
system. We will be providing DBA support for the new
system. Our user / system owner has been given a quote by
an outside vendor that they should implement a Sybase / SUN
cluster / EMC (SRDF) solution.
It would look something like this...
- SUN Cluster Server (SUN box)
- Sybase ASE (version 15)
- EMC storage
- EMC SRDF replication
- Sun Cluster Server (SUN box)
- EMC storage
- EMC SRDF replicated copy.
NOTE: This is a cold standby solution. EMC disks are
mounted upon failover (where replicated copies of Sybase ASE
is located and raw devices are located)
IMPORTANT NOTE: The acceptable system downtime is 2 hours.
We think that this is an overkill solution and does not meet
all DR and/or recovery needs. For example...
1) Does not address database corruption.
2) Does not protect against human error... (Joe Smith
deletes important binary files from the "sybase" directory /
installation or raw device path locations are altered or
deleted).
3) Abrupt shutdown of Sybase dataserver causes database
corruption.
We are recommending using Sybase Replication Server instead
of a "Cluster Server / EMC (SRDF)" solution.
Benefits of a Sybase Replication Server solution...
1) Provides a "Warm-Standby", which is immediately
available.
2) If there is database corruption, it will not be
replicated to "Warm-Standby" environment.
3) If there is dataserver catostrophic failure, the
"Warm-Standby" is available immediately.
Not to mention... the user / system owner SLA gives us a 2
hour recovery window. We think that Sybase Replication
Server is an appropriate solution to meet the SLA. Where
the "Cluster / EMC (SRDF)" solution is overkill and does not
address all recovery needs.
What are your thoughts?
Sincere thanks for sending your feedback.

unknown

2008-05-16 06:58:58 UTC

Permalink

Cory,

Thank you for your reponse and feedback.

Unfortunately, SRDF doesn't address the following...

- Database Corruption (data integrity)
- Dataserver binary file corruption (from catostrophic
dataserver failure, admin accidentally deleting binary
files)
- Warm-Standby (because SRDF is a Cold Standby solution)
- Up to the point of failure Recovery (if there is database
coruption or dataserver binary file coruption)

In addition, becuase SRDF is a Cold Standby solution, you
cannot point your DSS users to the DR server for workload
balancing.

What are your thoughts?

Thank you.

Post by unknown
I prefer SRDF.
Another option would be to use "Mirror Activator" layered
on top of SRDF. It mirrors only the transaction logs and
reads the transactions to rebuild the data on the
secondary system. We did not use it because our devices
had data from one database and log from another
database.But it appears to be a good product.
SRDF is maintenance free style replication. We had both.
SRDF was bullet proof.
--
Cory Sane
[Member of TeamSybase]
Certified Sybase Associate DBA for ASE 15.0
not a Sybase Inc. employee

Post by unknown
getting ready for a receive server handover for a new
system. We will be providing DBA support for the new
system. Our user / system owner has been given a quote

by an outside vendor that they should implement a Sybase /

Post by unknown
SUN cluster / EMC (SRDF) solution.
It would look something like this...
- SUN Cluster Server (SUN box)
- Sybase ASE (version 15)
- EMC storage
- EMC SRDF replication
- Sun Cluster Server (SUN box)
- EMC storage
- EMC SRDF replicated copy.
NOTE: This is a cold standby solution. EMC disks are
mounted upon failover (where replicated copies of Sybase
ASE is located and raw devices are located)
IMPORTANT NOTE: The acceptable system downtime is 2

hours. >

Post by unknown
We think that this is an overkill solution and does not
meet all DR and/or recovery needs. For example...
1) Does not address database corruption.
2) Does not protect against human error... (Joe Smith
deletes important binary files from the "sybase"
directory / installation or raw device path locations
are altered or deleted).
3) Abrupt shutdown of Sybase dataserver causes database
corruption.
We are recommending using Sybase Replication Server
instead of a "Cluster Server / EMC (SRDF)" solution.
Benefits of a Sybase Replication Server solution...
1) Provides a "Warm-Standby", which is immediately
available.
2) If there is database corruption, it will not be
replicated to "Warm-Standby" environment.
3) If there is dataserver catostrophic failure, the
"Warm-Standby" is available immediately.
Not to mention... the user / system owner SLA gives us a
2 hour recovery window. We think that Sybase
Replication Server is an appropriate solution to meet
the SLA. Where the "Cluster / EMC (SRDF)" solution is
overkill and does not address all recovery needs.
What are your thoughts?
Sincere thanks for sending your feedback.

mpeppler@peppler.org

2008-05-16 07:19:38 UTC

Permalink

Post by unknown
Cory,
Thank you for your reponse and feedback.
Unfortunately, SRDF doesn't address the following...
- Database Corruption (data integrity)
- Dataserver binary file corruption (from catostrophic
dataserver failure, admin accidentally deleting binary
files)
- Warm-Standby (because SRDF is a Cold Standby solution)
- Up to the point of failure Recovery (if there is database
coruption or dataserver binary file coruption)
In addition, becuase SRDF is a Cold Standby solution, you
cannot point your DSS users to the DR server for workload
balancing.
What are your thoughts?

Add normal database dumps, and transaction log dumps on a regular
basis (every 15 minutes, for example). If the log dumps and db dumps
are also on drives handled by SRDF then you can recover reasonably
well from a catastrophic failure.

A database reload on modern hardware doesn't take that long (I've seen
15-20 minutes for a 50GB database on AIX with an EMC SAN). Obviously
if you have very large databases then you would probably have to
figure out a different solution.

Beyond that you would need to go to HA solutions, I think.

Michael

MikeG

2008-05-16 08:18:46 UTC

Permalink

Post by unknown
In addition, becuase SRDF is a Cold Standby solution, you
cannot point your DSS users to the DR server for workload
balancing.

With Mirror Activator you can.

You say two hours to recover, but how much data can you lose, if any ?

One worrying thing I found while digging around on the net.
http://storage.ittoolbox.com/groups/technical-functional/emc-l/a-smoking-hole-in-the-ground-with-srdfs-1650088

Reading through that thread I got the idea that to get a guaranteed restart
R2 in DR, I had to split the mirror prior to the data centre accident,
strange I thought ?

So I checked the SRDF documentation and its at section 6.8.2 of the Sybase
on EMC Solutions Guide,
I quote "Some data integrity issues caused by the rolling disaster cannot be
resolved through normal restart processing and may require full database
recovery using appropriate backups, journals and logs."

Best make sure you still do full backups and tran log dumps, plus put the
dumps file system in the SRDF consistency group (what a bandwidth drain) ?

Post by unknown
getting ready for a receive server handover for a new
system. We will be providing DBA support for the new
system. Our user / system owner has been given a quote

by an outside vendor that they should implement a Sybase /

hours. >

unknown

2008-05-16 11:41:59 UTC

Permalink

Mary L.,

SRDF is meant to solve the problem of host or site failure. I've done SRDF
with a return-to-operation (RTO) of 2 hours. It works

Database corruption is a different problem. Backups can only solve this and
an application with a return-to-operation (RTO) of 2 hours should be bullet
proof at the user security level and the application code level.

Sybase program library tampering: This is the reason why a site has "change
management" - No one touchs production without approval.

It is common for me to keep 2 program libraries on one machine. Normally
they are two different patch versions like 12.5.4esd#4 and 12.5.4esd#7. If
something did truly happen to 12.5.4esd#7 I could fall back to esd#4 - but
in 9 years with a good change management process in place the fallback has
never happened due to tampering.
--
Cory Sane
[Member of TeamSybase]
Certified Sybase Associate DBA for ASE 15.0
not a Sybase Inc. employee

Post by unknown
getting ready for a receive server handover for a new
system. We will be providing DBA support for the new
system. Our user / system owner has been given a quote

by an outside vendor that they should implement a Sybase /

hours. >

A. M.

2008-05-21 04:52:58 UTC

Permalink

Post by unknown
Unfortunately, SRDF doesn't address the following...
- Database Corruption (data integrity)
- Dataserver binary file corruption (from catostrophic
dataserver failure, admin accidentally deleting binary
files)

You can't actually damage binary files on Unix
systems that way. Basically, something would
have to be writing to them and that doesn't
happen when you're just running them. Things
like deliberate writes or libraries and other
key files being deleted would be more likely.

Post by unknown
- Warm-Standby (because SRDF is a Cold Standby solution)
- Up to the point of failure Recovery (if there is database
coruption or dataserver binary file coruption)
In addition, becuase SRDF is a Cold Standby solution, you
cannot point your DSS users to the DR server for workload
balancing.

Correct. Basically, there are two forms of
corruption - logical and physical. You can
protect against the latter by simple disk
mirroring but that doesn't prevent issues
like file deletion. You'd need to lock the
filesystem down for that and maintain good
security.

Preventing logical corruption requires a
good logical data model with a good security
design. And the db developers should write
programs that do logical checks (i.e. balances
are all correct for financials etc.). You also
need to guard against bugs caused by the software
itself by running regular dbcc checks.

-am © MMVIII

Mark A. Parsons

2008-05-17 00:33:19 UTC

Permalink

If SRDF is running in synchronous mode you may experience performance degradation on the primary while waiting for the
remote IO to complete.

If SRDF is running in asynchronous mode, you lose the primary site, and the various disks are not in sync ... you could
end up with corruption in your remote database devices (sometimes bad enough that you won't be able to a) bring up the
dataserver or b) bring up a database) ... ymmv

A reliable network for NFS-mounted drives would make the ol' dump-n-load method quite effective for a good bit less than
SRDF. Granted, dump-n-load does not address database corruption ... but it's a heck of a lot cheaper/easier to setup
and maintain than SRDF.

Post by unknown
We are recommending using Sybase Replication Server instead
of a "Cluster Server / EMC (SRDF)" solution.
Benefits of a Sybase Replication Server solution...
1) Provides a "Warm-Standby", which is immediately
available.
2) If there is database corruption, it will not be
replicated to "Warm-Standby" environment.
3) If there is dataserver catostrophic failure, the
"Warm-Standby" is available immediately.

Repserver requires an additional skill set to setup and maintain.

There's been a few comments about using Mirror Activator. Mirror Activator is basically a stand-alone replication agent
which relies on physical mirroring (eg, SRDF) of the log device. Since Mirror Activator is a replication agent it must
have a repserver to talk to. A Mirror Activator solution requires that you have the EMC and Repserver products in place
... probably an overkill when repserver would normally suffice.

-------------------

It looks like you've got a good grasp of the pro's and con's ... and some additional input from the newsgroup.

At this point I'd go back to the pro-EMC folks with your comments/concerns/findings and see if/how they can address said
issues.

A. M.

2008-05-21 04:44:33 UTC

Permalink

Did they say why or justify any of their recommendations
in any way? If not, question everything. I'm especially
curious about the cluster recommendation.

Post by unknown
NOTE: This is a cold standby solution. EMC disks are
mounted upon failover (where replicated copies of Sybase ASE
is located and raw devices are located)
IMPORTANT NOTE: The acceptable system downtime is 2 hours.

Why when with warm standby you can have it sooner?
If they are recommending clustering, it suggests
a large system. So a 2 hour downtime sounds very,
very odd to say the least.

Post by unknown
We think that this is an overkill solution and does not meet
all DR and/or recovery needs. For example...

My thoughts exactly.

Post by unknown
1) Does not address database corruption.

Correct, but that's a different issue.

Post by unknown
2) Does not protect against human error... (Joe Smith
deletes important binary files from the "sybase" directory /
installation or raw device path locations are altered or
deleted).

Also correct but its also another issue.

Post by unknown
3) Abrupt shutdown of Sybase dataserver causes database
corruption.

No, it wouldn't. Pulling the plug on ASE only
means that the logs will have open transactions.
ASE can easily recover from this. Pulling the
power on the system may have other repercussions,
however.

Basically, physical mirroring copies everything on
the source to the target regardless. It doesn't
matter what the data actually is nor does it care.
You can't prevent user stupidity with it.

Post by unknown
We are recommending using Sybase Replication Server instead
of a "Cluster Server / EMC (SRDF)" solution.

I would probably recommend that too but I don't
know all the facts here.

Post by unknown
Benefits of a Sybase Replication Server solution...
1) Provides a "Warm-Standby", which is immediately
available.

Correct.

Post by unknown
2) If there is database corruption, it will not be
replicated to "Warm-Standby" environment.

That depends on what you mean by "corruption".
User stupidity is still present. If someone,
say a third party vendor's hack, comes in and
truncates a table and you're replicating
table truncation, then you have a problem
similar to someone deleting files.

Post by unknown
3) If there is dataserver catostrophic failure, the
"Warm-Standby" is available immediately.
Not to mention... the user / system owner SLA gives us a 2
hour recovery window. We think that Sybase Replication
Server is an appropriate solution to meet the SLA. Where
the "Cluster / EMC (SRDF)" solution is overkill and does not
address all recovery needs.
What are your thoughts?

I agree. Couple it with OpenSwitch and your
users may not even notice. Sybase Cluster Server
(not the Sun Cluster Server) might also be viable
if you do have a large site.

-am © MMVIII