Sync summary queries can hang

There seems to be situations where PDB sync summary query transactions can remain open and PDB sync stops logging and hangs. When this happens sync will stop until PDB is restarted. It's also possible that running SELECT pg_cancel_backend(pid); on the query will restore sync, but this is less certain to work then a full PDB restart. 

We recently saw this issue when a PDB was in the middle of pulling reports from a replica and the replica was upgraded. link to related slack msgs

Order of events:

• primary started its report sync at: 2020-10-13T02:26:55.984Z
• replica received a shutdown signal at: 2020-10-13T02:27:23.876Z
• replica saw the errors in the following gist during shutdown: shutdown-error-gist

After the replica was shutdown and upgraded the sync on the primary never logged again and there was an open sync summary query observed in pg_stat_activity which stayed open idle in transaction waiting on ClientRead.

We recently added a thread interrupter for sync in PDB-4909 but it seems like there are still edge cases that this work didn't cover.

It's possible that adding a statement_timeout to sync queries would help avoid this issue.

We'll work towards reproducing this error in the coming days will update the ticket with what we find.