AWS RDS Detecting failover
Date: Mon, 2 Oct 2017 19:48:34 +0000
Message-ID: <BY2PR0101MB08876E20487D06E59C20519FD07D0_at_BY2PR0101MB0887.prod.exchangelabs.com>
Hi all,
In my testing, when a client is connected to a multi-az RDS instance and I force failover, that client doesn't 'see' it. If it makes any DB request after or during the failover it ends up timing out - eventually. Unfortunately this timeout is controlled by the tcp keepalive setting which defaults to 2 hours. Not very helpful when the actual failover can be complete in a couple of minutes.
I'm wondering what other RDS users are doing to handle this scenario.
I've tried tinkering with sqlnet.ora params but couldn't find any that would allow a connected client to detect the failover.
We may have many connected clients - both on-prem and from AWS - including our on-prem DB servers using DB links. I'm reluctant to muck with OS-level tcp keepalive params, and in some cases that may not even be possible (e.g. from another RDS instance).
My current solution involves using socat (http://www.dest-unreach.org/socat/) as a proxy. I can easily adjust the tcp keepalive parameters with this and depending on the values I set for those parameters I can detect failover almost immediately.
However it means either running socat on every client, or having a dedicated containter/ec2 running socat - which I then have to monitor. If that container/ec2 fails but the DB doesn't all in-flight connections are lost.
I'm thinking there has to be a better way. I've contacted AWS support but they suggested mucking with the tcp keepalive settings on all clients. Or alternatively using SNS and Lambdas to notify/kill connected clients. The latter wasn't ideal because the Lambda didn't get fired until well after the failover (8-10 mins), and I have a mixture of AWS and on-prem clients, so the notification part is messy.
Thanks for any suggestions.
Steve
This email is intended solely for the use of the addressee and may contain information that is confidential, proprietary, or both. If you receive this email in error please immediately notify the sender and delete the email.
-- http://www.freelists.org/webpage/oracle-lReceived on Mon Oct 02 2017 - 21:48:34 CEST