Uploaded image for project: 'Puppet Communications Protocol'
  1. Puppet Communications Protocol
  2. PCP-627

pxp-agent should handle duplicate non-blocking transactions idempotently

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: pxp-agent 1.5.0
    • Component/s: pxp-agent
    • Labels:
      None
    • Template:
    • Acceptance Criteria:
      Hide

      At-least-once message delivery should result in idempotent behavior on pxp-agent (within a reasonable time-frame, i.e. the spool-dir-purge-ttl).

      Show
      At-least-once message delivery should result in idempotent behavior on pxp-agent (within a reasonable time-frame, i.e. the spool-dir-purge-ttl ).
    • Team:
      Dumpling
    • Story Points:
      3
    • Sprint:
      FF 2017-01-11, FF 2017-01-25, FF 2017-02-08
    • Release Notes:
      New Feature
    • Release Notes Summary:
      Hide
      pxp-agent will now respond to a PXP non-blocking requests that uses a duplicate transaction ID by sending a provisional response, rather than an error message; status requests can then be sent as normal to check on the status of the original request that was duplicated. It will also detect duplicate IDs that are stored on-disk, rather than only those in-memory (so it will no longer "forget" when the process is restarted).
      Show
      pxp-agent will now respond to a PXP non-blocking requests that uses a duplicate transaction ID by sending a provisional response, rather than an error message; status requests can then be sent as normal to check on the status of the original request that was duplicated. It will also detect duplicate IDs that are stored on-disk, rather than only those in-memory (so it will no longer "forget" when the process is restarted).

      Description

      Networking is inherently lossy. A request sent to pxp-agent results in a response to the original sender; if that response is lost, the sender may attempt to re-send the request (using the same transaction id). Blocking requests are meant to be quick and repeatable, so we can safely re-run them. Non-blocking requests are expected to be on-going; we should handle repeated requests idempotently, meaning the action should not be re-executed.

      Also, if an agent appears to be connected to more than one broker, the easiest way to ensure delivery is send a message via both brokers. This scenario can happen when an agent disconnects from one broker without socket shutdown and connects to the other, before the timeout has occurred. We choose to deal with multiple potential connections via transaction idempotency rather than negotiation between brokers to determine which is correct (because negotiation is a trickier distributed systems problem).

      pxp-agent currently maintains a metadata store of previously completed non-blocking requests, so it can respond to status requests. Use the existing metadata store to detect when a non-blocking request is received and send an rpc_provisional_response that should be identical to if we were starting a task, but don't start one. The initiator is then expected to send a status request.

      This should be a minor addition to https://github.com/puppetlabs/pxp-agent/blob/1.3.0/lib/src/request_processor.cc#L372-L375 to check the ResultsStorage for the transaction, and return a similar error as when the request is already running. In either case the subsequent action from the controller should be to request the status of that transaction.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              michael.smith Michael Smith
              Reporter:
              michael.smith Michael Smith
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support