zydronium/google-nomulus
1
0
Fork 0
mirror of https://github.com/google/nomulus.git synced 2025-05-13 07:57:13 +02:00
Code Issues Projects Releases Packages Wiki Activity

Improve logs in the GCP proxy

Browse source 
Tweaked a few logging levels to not spam error level logs. Also make it easy to debug issues in case relay retry fails.

[1] Put non-fatal exceptions that should be logged at warning in their explicit sets. Also always use the root cause to determine if an exception is non-fatal, because sometimes the actual causes are wrapped inside other exceptions.

[2] Record the cause of a relay failure, and record if a relay retry is successful. This way we can look at the log and figure out if a relay is eventually successful.

[3] Add a log when the frontend connection from the client is terminated.

[4] Alway close the relay channel when a relay has failed, which, depend on if the channel is frontend or backend, will reconnect and trigger a retry.

[5] Lastly changed failure test to use assertThrows instead of fail.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208649916
This commit is contained in:
jianglai 2018-08-14 08:22:31 -07:00
parent b552c1d115
commit 0e64015cdf
10 changed files with 154 additions and 95 deletions
Download patch file Download diff file Expand all files Collapse all files

42
java/google/registry/proxy/handler/RelayHandler.java
View file

@ -66,39 +66,51 @@ public class RelayHandler<I> extends SimpleChannelInboundHandler<I> {
logger.atSevere().log("Relay channel not specified for channel: %s", channel);
ChannelFuture unusedFuture = channel.close();
} else {
writeToRelayChannel(channel, relayChannel, msg);
writeToRelayChannel(channel, relayChannel, msg, false);
}
}
public static void writeToRelayChannel(Channel channel, Channel relayChannel, Object msg) {
public static void writeToRelayChannel(
Channel channel, Channel relayChannel, Object msg, boolean retry) {
ChannelFuture unusedFuture =
relayChannel
.writeAndFlush(msg)
.addListener(
future -> {
if (!future.isSuccess()) {
logger.atWarning().log(
"Relay failed: %s --> %s\nINBOUND: %s\nOUTBOUND: %s",
// TODO (jianglai): do not log the message once retry behavior is confirmed.
logger.atWarning().withCause(future.cause()).log(
"Relay failed: %s --> %s\nINBOUND: %s\nOUTBOUND: %s\nMESSAGE: %s",
channel.attr(PROTOCOL_KEY).get().name(),
relayChannel.attr(PROTOCOL_KEY).get().name(),
channel,
relayChannel);
relayChannel,
msg);
// If we cannot write to the relay channel and the originating channel has
// a relay buffer (i. e. we tried to relay the frontend to the backend), store
// the message in the buffer for retry later. Otherwise, we are relaying from
// the backend to the frontend, and this relay failure cannot be recovered
// from, we should just kill the relay (frontend) channel, which in turn will
// kill the backend channel. We should not kill any backend channel while the
// the frontend channel is open, because that will just trigger a reconnect.
// It is fine to just save the message object itself, not a clone of it,
// because if the relay is not successful, its content is not read, therefore
// its buffer is not cleared.
// the message in the buffer for retry later. The relay channel (backend) should
// be killed (if it is not already dead, usually the relay is unsuccessful
// because the connection is closed), and a new backend channel will re-connect
// as long as the frontend channel is open. Otherwise, we are relaying from the
// backend to the frontend, and this relay failure cannot be recovered from: we
// should just kill the relay (frontend) channel, which in turn will kill the
// backend channel. It is fine to just save the message object itself, not a
// clone of it, because if the relay is not successful, its content is not read,
// therefore its buffer is not cleared.
Queue<Object> relayBuffer = channel.attr(RELAY_BUFFER_KEY).get();
if (relayBuffer != null) {
channel.attr(RELAY_BUFFER_KEY).get().add(msg);
} else {
ChannelFuture unusedFuture2 = relayChannel.close();
}
ChannelFuture unusedFuture2 = relayChannel.close();
} else if (retry) {
// TODO (jianglai): do not log the message once retry behavior is confirmed.
logger.atInfo().log(
"Relay retry succeeded: %s --> %s\nINBOUND: %s\nOUTBOUND: %s\nsMESSAGE: %s",
channel.attr(PROTOCOL_KEY).get().name(),
relayChannel.attr(PROTOCOL_KEY).get().name(),
channel,
relayChannel,
msg);
}
});
}