Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-577

PuppetDB refuses to start broker if KahaDB lock file exists

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: PDB 1.6.3
    • Fix Version/s: PDB 2.1.0
    • Component/s: None
    • Labels:
    • Environment:

      CentOS 6.5 x64 with OpenJDK 1.7

    • Template:
    • Story Points:
      1
    • Sprint:
      20140618 to 20140702

      Description

      I upgraded from 1.6.2 to 1.6.3 after shutting PuppetDB down. The daemon didn't come up all the way, as clients couldn't connect. Log file indicates:

      2014-04-10 09:05:19,464 INFO  [main] [cli.services] PuppetDB version 1.6.3
      2014-04-10 09:05:19,645 INFO  [main] [cli.services] Starting broker
      

      It sat there for about 15 minutes before I had to kill -9 the java process. Before I did this, any client attempting to connect got the error:

      2014-04-10 09:06:06,273 WARN  [qtp601609992-46] [server.AbstractHttpConnection] /v3/commands
      org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing;
       nested exception is javax.jms.JMSException: Could not create Transport. Reason: java.io.IOException: Bro
      ker named 'localhost' does not exist.
              at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:316)
              at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)
              at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)
              at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:543)
              at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:653)
              at sun.reflect.GeneratedMethodAccessor673.invoke(Unknown Source)
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
              at java.lang.reflect.Method.invoke(Method.java:606)
              at clojure.lang.Reflector.invokeMatchingMethod(Reflector.java:93)
              at clojure.lang.Reflector.invokeInstanceMethod(Reflector.java:28)
              at clamq.jms$jms_producer$reify__4656.publish(jms.clj:29)
              at clamq.jms$jms_producer$reify__4656.publish(jms.clj:30)
              at clamq.protocol.producer$eval4608$fn__4609$G__4600__4617.invoke(producer.clj:3)
              at clamq.protocol.producer$eval4608$fn__4609$G__4599__4626.invoke(producer.clj:3)
              at clojure.lang.AFn.applyToHelper(AFn.java:167)
              at clojure.lang.AFn.applyTo(AFn.java:151)
              at clojure.core$apply.invoke(core.clj:619)
              at com.puppetlabs.mq$connect_and_publish_BANG_.doInvoke(mq.clj:148)
              at clojure.lang.RestFn.invoke(RestFn.java:439)
              at com.puppetlabs.puppetdb.command$enqueue_raw_command_BANG_$fn__5462.invoke(command.clj:255)
              at com.puppetlabs.puppetdb.command$enqueue_raw_command_BANG_.invoke(command.clj:254)
              at com.puppetlabs.puppetdb.http.v1.command$enqueue_command.invoke(command.clj:22)
              at com.puppetlabs.middleware$verify_accepts_content_type$fn__5916.invoke(middleware.clj:104)
              at com.puppetlabs.middleware$verify_checksum$fn__5930.invoke(middleware.clj:161)
              at com.puppetlabs.middleware$validate_query_params$fn__5921.invoke(middleware.clj:131)
              at com.puppetlabs.puppetdb.http.v1.command$command_app.invoke(command.clj:27)
              at com.puppetlabs.puppetdb.http.v3$v3_app$fn__7989.invoke(v3.clj:18)
              at net.cgrand.moustache$alter_request$fn__6169.invoke(moustache.clj:54)
              at com.puppetlabs.puppetdb.http.v3$v3_app.invoke(v3.clj:18)
              at com.puppetlabs.puppetdb.http.server$routes$fn__9513.invoke(server.clj:41)
              at net.cgrand.moustache$alter_request$fn__6169.invoke(moustache.clj:54)
              at com.puppetlabs.puppetdb.http.server$routes.invoke(server.clj:41)
              at ring.middleware.resource$wrap_resource$fn__9480.invoke(resource.clj:14)
              at ring.middleware.params$wrap_params$fn__6111.invoke(params.clj:55)
              at com.puppetlabs.middleware$wrap_with_authorization$fn__5895.invoke(middleware.clj:37)
              at com.puppetlabs.middleware$wrap_with_certificate_cn$fn__5899.invoke(middleware.clj:52)
              at com.puppetlabs.middleware$wrap_with_default_body$fn__5903.invoke(middleware.clj:59)
              at com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__5937$fn__5946.invoke(middleware.clj:183)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__5874$fn__5875$fn__5876.invoke(metrics.clj:14)
              at com.puppetlabs.utils.metrics.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown Source)
              at com.yammer.metrics.core.Timer.time(Timer.java:91)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__5874$fn__5875.invoke(metrics.clj:14)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__5874$fn__5875$fn__5876.invoke(metrics.clj:14)
              at com.puppetlabs.utils.metrics.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown Source)
              at com.yammer.metrics.core.Timer.time(Timer.java:91)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__5874$fn__5875.invoke(metrics.clj:14)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_.invoke(metrics.clj:17)
              at com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__5937.invoke(middleware.clj:182)
              at com.puppetlabs.middleware$wrap_with_globals$fn__5907.invoke(middleware.clj:70)
              at com.puppetlabs.middleware$wrap_with_debug_logging$fn__5891.invoke(middleware.clj:24)
              at ring.adapter.jetty$proxy_handler$fn__5794.invoke(jetty.clj:18)
              at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$0.handle(Unknown Source)
              at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:267)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
              at org.eclipse.jetty.server.Server.handle(Server.java:349)
              at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
              at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
              at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:857)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
              at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
              at java.lang.Thread.run(Thread.java:744)
      Caused by: javax.jms.JMSException: Could not create Transport. Reason: java.io.IOException: Broker named 'localhost' does not exist.
              at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:35)
              at org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:254)
              at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:267)
              at org.apache.activemq.ActiveMQConnectionFactory.createActiveMQConnection(ActiveMQConnectionFactory.java:239)
              at org.apache.activemq.ActiveMQConnectionFactory.createConnection(ActiveMQConnectionFactory.java:185)
              at org.springframework.jms.connection.SingleConnectionFactory.doCreateConnection(SingleConnectionFactory.java:342)
              at org.springframework.jms.connection.SingleConnectionFactory.initConnection(SingleConnectionFactory.java:288)
              at org.springframework.jms.connection.SingleConnectionFactory.createConnection(SingleConnectionFactory.java:225)
              at org.springframework.jms.support.JmsAccessor.createConnection(JmsAccessor.java:184)
              at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:456)
              ... 63 more
      Caused by: java.io.IOException: Broker named 'localhost' does not exist.
              at org.apache.activemq.transport.vm.VMTransportFactory.doCompositeConnect(VMTransportFactory.java:116)
              at org.apache.activemq.transport.vm.VMTransportFactory.doConnect(VMTransportFactory.java:54)
              at org.apache.activemq.transport.TransportFactory.doConnect(TransportFactory.java:51)
              at org.apache.activemq.transport.TransportFactory.connect(TransportFactory.java:80)
              at org.apache.activemq.ActiveMQConnectionFactory.createTransport(ActiveMQConnectionFactory.java:252)
              ... 71 more
      

      Ken Barber mentioned that it was probably the KahaDB lock file still existing that caused the broker to not start up, and sure enough, there was a lock file at /var/lib/puppetdb/mq/localhost/KahaDB/lock:

      # ls -ltr /var/lib/puppetdb/mq/localhost/KahaDB/
      total 26132
      -rw-r--r-- 1 puppetdb puppetdb        0 Feb 13 09:26 lock
      -rw-r--r-- 1 puppetdb puppetdb 33030144 Apr 10 09:03 db-9912.log
      -rw-r--r-- 1 puppetdb puppetdb    28720 Apr 10 09:03 db.redo
      -rw-r--r-- 1 puppetdb puppetdb    32768 Apr 10 09:03 db.data
      

      It wasn't removed after PuppetDB was shut down. Manually removing it and then starting up PuppetDB resolved the issue.

        Attachments

          Activity

            People

            • Assignee:
              wyatt Wyatt Alt
              Reporter:
              stephenprice Stephen Price
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Zendesk Support