Commit graph

8 commits

Author SHA1 Message Date
jianglai
c5c0051f5e Ensure that no reference counted objects leak memory
The objects stored in the relay buffer may leak memory when they are no longer used. Alway remember to release their reference count in all cases.

Also save the relay channel and its name in BackendMetricsHandler when the handler is registered. This is because when retrying a relay, the write is sent as soon as the channel is connected, and the channelActive function is not called yet.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208757730
2018-08-20 14:06:24 -04:00
jianglai
4965478cce Correctly retry relay of reference counted objects
It turns out in the edge case where a write occurs at the same moment that the
relay connection is terminated, the current retry mechanism is not sufficient
because it stores reference coutned objects whose internal buffers are already
freed.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208738065
2018-08-20 14:03:18 -04:00
jianglai
2e2898e17c Fix WHOIS issues
[1] Web whois should redirect to www.registry.google. whois.registry.google also points to the proxy IP, so redirecting to whois.registry.google just makes it loop. Also allow HEAD in web whois request in case that is used in monitoring.

[2] Separately, there's a bug introduced in [] where exception handling of inbound messages is moved to HttpsRelayServiceHandler. However the quota handlers are installed behind the HttpServiceServiceHandler in the channel pipeline, therefore the exception thrown in quota handlers never got processed. This results in hung connection when quota exceeded.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208651011
2018-08-20 14:00:08 -04:00
jianglai
0e64015cdf Improve logs in the GCP proxy
Tweaked a few logging levels to not spam error level logs. Also make it easy to debug issues in case relay retry fails.

[1] Put non-fatal exceptions that should be logged at warning in their explicit sets. Also always use the root cause to determine if an exception is non-fatal, because sometimes the actual causes are wrapped inside other exceptions.

[2] Record the cause of a relay failure, and record if a relay retry is successful. This way we can look at the log and figure out if a relay is eventually successful.

[3] Add a log when the frontend connection from the client is terminated.

[4] Alway close the relay channel when a relay has failed, which, depend on if the channel is frontend or backend, will reconnect and trigger a retry.

[5] Lastly changed failure test to use assertThrows instead of fail.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=208649916
2018-08-20 13:58:30 -04:00
jianglai
4ff77fb370 Automatic reconnect to GAE when the connection is dropped
The connection to GAE is not persistent and can drop. Reconnect when that happens, as long as the connection from the client is still active.

We need to consider the fact that while a reconnection is happening, the client may be sending requests that was relayed to the old connection, which is not going through. In that case these requests are queued and will be retried when the new connection is available.

Since we are no longer tying the lifecycles of the two connections, we cannot automatically terminate one when another is terminated. Also we need to explicitly control how WHOIS connection is terminated, not depending on the HTTP connection header.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=207335498
2018-08-10 13:46:48 -04:00
jianglai
05f166918f Migrating to fluent logging (green)
This is a 'green' Flogger migration CL. Green CLs are intended to be as
safe as possible and should be easy to review and submit.

No changes should be necessary to the code itself prior to submission,
but small changes to BUILD files may be required.

Changes within files are completely independent of each other, so this CL
can be safely split up for review using tools such as Rosie.

For more information, see []
Base CL: 197331037

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197466715
2018-05-30 12:18:54 -04:00
jianglai
68b24f0a54 Migrate to internal FormattingLogger in GCP proxy in preparation of migration to Flogger
-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=197199265
2018-05-30 12:18:54 -04:00
jianglai
7e42ee48a4 Open source GCP proxy
Dagger updated to 2.13, along with all its dependencies.

Also allows us to have multiple config files for different environment (prod, sandbox, alpha, local, etc) and specify which one to use on the command line with a --env flag. Therefore the same binary can be used in all environments.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=176551289
2017-11-21 19:19:03 -05:00