Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-880

Our shutdown routine throws exceptions

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: PDB 2.2.0
    • Fix Version/s: PDB 3.0.0
    • Component/s: None
    • Labels:
      None
    • Template:
    • Story Points:
      2
    • Sprint:
      PuppetDB 2014-11-05

      Description

      Whenever PuppetDB shuts down, we seem to throw exceptions. This looks like it is a shutdown ordering issue, or at least there are sub-components we are not shutting down correctly:

      2014-09-16 20:31:21,469 INFO  [c.p.p.c.services] Shutting down message broker.
      2014-09-16 20:31:21,479 WARN  [o.a.a.b.T.Service] Failed to remove session ID:ubuntu-1004-64-1-60347-1410899477295-5:1:1
      org.apache.activemq.broker.BrokerStoppedException: Broker has been stopped: org.apache.activemq.broker.BrokerService$3@25778bf0
              at org.apache.activemq.broker.ErrorBroker.removeSession(ErrorBroker.java:99) ~[puppetdb.jar:na]
              at org.apache.activemq.broker.MutableBrokerFilter.removeSession(MutableBrokerFilter.java:175) ~[puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:646) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:750) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1068) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996) [puppetdb.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
              at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
      2014-09-16 20:31:21,485 WARN  [o.a.a.b.TransportConnection] Failed to remove consumer: ID:ubuntu-1004-64-1-60347-1410899477295-5:1:-1:1. Reason: org.apache.activemq.broker.BrokerStoppedException: Broker has been stopped: org.apache.activemq.broker.BrokerService$3@25778bf0
      org.apache.activemq.broker.BrokerStoppedException: Broker has been stopped: org.apache.activemq.broker.BrokerService$3@25778bf0
              at org.apache.activemq.broker.ErrorBroker.removeConsumer(ErrorBroker.java:155) ~[puppetdb.jar:na]
              at org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:123) ~[puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:596) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:633) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:750) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1068) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996) [puppetdb.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
              at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
      2014-09-16 20:31:21,485 WARN  [o.a.a.b.T.Service] Failed to remove session ID:ubuntu-1004-64-1-60347-1410899477295-5:1:-1
      org.apache.activemq.broker.BrokerStoppedException: Broker has been stopped: org.apache.activemq.broker.BrokerService$3@25778bf0
              at org.apache.activemq.broker.ErrorBroker.removeSession(ErrorBroker.java:99) ~[puppetdb.jar:na]
              at org.apache.activemq.broker.MutableBrokerFilter.removeSession(MutableBrokerFilter.java:175) ~[puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:646) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:750) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection.doStop(TransportConnection.java:1068) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransportConnection$4.run(TransportConnection.java:996) [puppetdb.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_51]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_51]
              at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
      2014-09-16 20:31:21,486 WARN  [o.a.a.b.T.Service] Failed to remove connection ConnectionInfo {commandId = 1, responseRequired = true, connectionId = ID:ubuntu-1004-64-1-60347-1410899477295-5:1, clientId = ID:ubuntu-1004-64-1-60347-1410899477295-4:1, clientIp = vm://localhost#0, userName = null, password = *****, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true, faultTolerant = false, failoverReconnect = false}, reason: org.apache.activemq.broker.BrokerStoppedException: Broker has been stopped: org.apache.activemq.broker.BrokerService$3@25778bf0
      2014-09-16 20:31:21,522 ERROR [o.a.a.s.k.KahaDBStore] Failed to reset batching
      java.lang.IllegalStateException: PageFile is not loaded
              at org.apache.kahadb.page.PageFile.assertLoaded(PageFile.java:800) ~[puppetdb.jar:na]
              at org.apache.kahadb.page.PageFile.tx(PageFile.java:294) ~[puppetdb.jar:na]
              at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.resetBatching(KahaDBStore.java:575) ~[puppetdb.jar:na]
              at org.apache.activemq.store.ProxyMessageStore.resetBatching(ProxyMessageStore.java:111) [puppetdb.jar:na]
              at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.resetBatch(QueueStorePrefetch.java:85) [puppetdb.jar:na]
              at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.stop(AbstractStoreCursor.java:72) [puppetdb.jar:na]
              at org.apache.activemq.broker.region.cursors.StoreQueueCursor.stop(StoreQueueCursor.java:84) [puppetdb.jar:na]
              at org.apache.activemq.broker.region.Queue.stop(Queue.java:909) [puppetdb.jar:na]
              at org.apache.activemq.broker.region.AbstractRegion.stop(AbstractRegion.java:118) [puppetdb.jar:na]
              at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41) [puppetdb.jar:na]
              at org.apache.activemq.broker.region.RegionBroker.doStop(RegionBroker.java:713) [puppetdb.jar:na]
              at org.apache.activemq.broker.jmx.ManagedRegionBroker.doStop(ManagedRegionBroker.java:122) [puppetdb.jar:na]
              at org.apache.activemq.broker.region.RegionBroker.stop(RegionBroker.java:201) [puppetdb.jar:na]
              at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161) [puppetdb.jar:na]
              at org.apache.activemq.broker.scheduler.SchedulerBroker.stop(SchedulerBroker.java:104) [puppetdb.jar:na]
              at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161) [puppetdb.jar:na]
              at org.apache.activemq.broker.BrokerFilter.stop(BrokerFilter.java:161) [puppetdb.jar:na]
              at org.apache.activemq.broker.TransactionBroker.stop(TransactionBroker.java:193) [puppetdb.jar:na]
              at org.apache.activemq.broker.BrokerService$3.stop(BrokerService.java:1961) [puppetdb.jar:na]
              at org.apache.activemq.util.ServiceStopper.stop(ServiceStopper.java:41) [puppetdb.jar:na]
              at org.apache.activemq.broker.BrokerService.stop(BrokerService.java:624) [puppetdb.jar:na]
              at com.puppetlabs.mq$stop_broker_BANG_.invoke(mq.clj:125) [na:na]
              at com.puppetlabs.puppetdb.cli.services$shutdown_mq_broker.invoke(services.clj:222) [na:na]
              at com.puppetlabs.puppetdb.cli.services$stop_puppetdb.invoke(services.clj:228) [na:na]
              at com.puppetlabs.puppetdb.cli.services$reify__18993$f_positional__19002$fnk19059_positional__19060$fn__19061.invoke(services.clj:354) [na:na]
              at com.puppetlabs.puppetdb.cli.services$reify__18993$fn$reify__19107.stop(services.clj:354) [na:na]
              at puppetlabs.trapperkeeper.services$eval16113$fn__16140$G__16101__16143.invoke(services.clj:12) [na:na]
              at puppetlabs.trapperkeeper.services$eval16113$fn__16140$G__16100__16147.invoke(services.clj:12) [na:na]
              at puppetlabs.trapperkeeper.internal$run_lifecycle_fn_BANG_.invoke(internal.clj:143) [na:na]
              at puppetlabs.trapperkeeper.internal$shutdown_BANG_$fn__17163.invoke(internal.clj:287) [na:na]
              at puppetlabs.trapperkeeper.internal$shutdown_BANG_.invoke(internal.clj:286) [na:na]
              at puppetlabs.trapperkeeper.internal$initialize_shutdown_service_BANG_$fn__17173.invoke(internal.clj:302) [na:na]
              at clojure.lang.AFn.run(AFn.java:22) [puppetdb.jar:na]
              at java.lang.Thread.run(Thread.java:744) [na:1.7.0_51]
      2014-09-16 20:31:21,529 INFO  [p.t.s.w.jetty9-service] Shutting down web server.
      2014-09-16 20:31:21,529 INFO  [p.t.s.w.jetty9-core] Shutting down web server.
      

        Attachments

          Activity

            jsd-sla-details-panel

              People

              • Assignee:
                wyatt Wyatt Alt
                Reporter:
                ken Ken Barber
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: