Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-707

The first request to PuppetDB after DB backend restart fails

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: PDB 1.6.3
    • Fix Version/s: PDB 2.2.1
    • Component/s: None
    • Labels:
      None
    • Template:
    • Story Points:
      3
    • Sprint:
      20140813 to 20140827, 20140827 to 20140910

      Description

      The first request sent to PuppetDB after its backend DB has been restarted fails. Any subsequent requests are processed normally.

      Steps to reproduce

      service pe-postgresql restart
      curl -i --cacert /opt/puppet/share/puppet-dashboard/certs/pe-internal-dashboard.ca_cert.pem --cert /opt/puppet/share/puppet-dashboard/certs/pe-internal-dashboard.cert.pem --key /opt/puppet/share/puppet-dashboard/certs/pe-internal-dashboard.private_key.pem 'https://<puppetdb host>:8081/v3/event-counts?query=%5B%22and%22%2C%5B%22and%22%2C%5B%22%3C%22%2C%22report-receive-time%22%2C%222014-06-13T13%3A24%3A05Z%22%5D%5D%2C%5B%22not%22%2C%5B%22%3D%22%2C%22containing-class%22%2Cnull%5D%5D%5D&distinct-resources=true&distinct-start-time=2014-06-13T12%3A54%3A05Z&distinct-end-time=2014-06-13T13%3A24%3A05Z&summarize-by=containing-class&count-by=certname&counts-filter=%5B%22%3E%22%2C%22failures%22%2C0%5D&limit=1000&offset=0&order-by=%5B%7B%22field%22%3A%22containing-class%22%2C%22order%22%3A%22ASC%22%7D%5D&include-total=true'
      

      where <puppetdb host> is the host where PuppetDB is running

      Additional information

      The output of the curl command looks like this:

      HTTP/1.1 500 Server Error
      Date: Fri, 13 Jun 2014 13:41:17 GMT
      Content-Length: 0
      Connection: close
      Server: Jetty(7.x.y-SNAPSHOT)
      

      The /var/log/pe-puppetdb/pe-puppetdb.log log file contains the following information:

      2014-06-13 06:41:17,177 ERROR [qtp820981039-41] [bonecp.ConnectionHandle] Database access problem. Killing off all remaining connect
      ions in the connection pool. SQL State = 57P01
      2014-06-13 06:41:17,180 WARN  [qtp820981039-41] [server.AbstractHttpConnection] /v3/event-counts?query=%5B%22and%22%2C%5B%22and%22%2
      C%5B%22%3C%22%2C%22report-receive-time%22%2C%222014-06-13T13%3A24%3A05Z%22%5D%5D%2C%5B%22not%22%2C%5B%22%3D%22%2C%22containing-class
      %22%2Cnull%5D%5D%5D&distinct-resources=true&distinct-start-time=2014-06-13T12%3A54%3A05Z&distinct-end-time=2014-06-13T13%3A24%3A05Z&
      summarize-by=containing-class&count-by=certname&counts-filter=%5B%22%3E%22%2C%22failures%22%2C0%5D&limit=1000&offset=0&order-by=%5B%
      7B%22field%22%3A%22containing-class%22%2C%22order%22%3A%22ASC%22%7D%5D&include-total=true
      org.postgresql.util.PSQLException: This connection has been closed.
              at org.postgresql.jdbc2.AbstractJdbc2Connection.checkClosed(AbstractJdbc2Connection.java:822)
              at org.postgresql.jdbc2.AbstractJdbc2Connection.setAutoCommit(AbstractJdbc2Connection.java:769)
              at com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1063)
              at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:222)
              at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__2278.invoke(jdbc.clj:228)
              at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186)
              at com.puppetlabs.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:225)
              at com.puppetlabs.puppetdb.http.event_counts$produce_body.invoke(event_counts.clj:28)
              at com.puppetlabs.puppetdb.http.event_counts$routes$fn__7581$fn__7585$fn__7590.invoke(event_counts.clj:46)
              at com.puppetlabs.puppetdb.http.event_counts$routes$fn__7581$fn__7585.invoke(event_counts.clj:43)
              at com.puppetlabs.puppetdb.http.event_counts$routes$fn__7581.invoke(event_counts.clj:43)
              at com.puppetlabs.middleware$verify_accepts_content_type$fn__6056.invoke(middleware.clj:104)
              at com.puppetlabs.middleware$validate_query_params$fn__6061.invoke(middleware.clj:131)
              at com.puppetlabs.middleware$wrap_with_paging_options$fn__6051.invoke(middleware.clj:84)
              at com.puppetlabs.puppetdb.http.v3$v3_app$fn__7835.invoke(v3.clj:60)
              at net.cgrand.moustache$alter_request$fn__6453.invoke(moustache.clj:54)
              at com.puppetlabs.puppetdb.http.v3$v3_app.invoke(v3.clj:20)
              at com.puppetlabs.puppetdb.http.server$routes$fn__9305.invoke(server.clj:41)
              at net.cgrand.moustache$alter_request$fn__6453.invoke(moustache.clj:54)
              at com.puppetlabs.puppetdb.http.server$routes.invoke(server.clj:41)
              at ring.middleware.resource$wrap_resource$fn__9272.invoke(resource.clj:17)
              at ring.middleware.params$wrap_params$fn__6395.invoke(params.clj:55)
              at com.puppetlabs.middleware$wrap_with_authorization$fn__6035.invoke(middleware.clj:37)
              at com.puppetlabs.middleware$wrap_with_certificate_cn$fn__6039.invoke(middleware.clj:52)
              at com.puppetlabs.middleware$wrap_with_default_body$fn__6043.invoke(middleware.clj:59)
              at com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__6077$fn__6086.invoke(middleware.clj:183)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__6014$fn__6015$fn__6016.invoke(metrics.clj:14)
              at com.puppetlabs.utils.metrics.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown Source)
              at com.yammer.metrics.core.Timer.time(Timer.java:91)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__6014$fn__6015.invoke(metrics.clj:14)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__6014$fn__6015$fn__6016.invoke(metrics.clj:14)
              at com.puppetlabs.utils.metrics.proxy$java.lang.Object$Callable$f8c5758f.call(Unknown Source)
              at com.yammer.metrics.core.Timer.time(Timer.java:91)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_$fn__6014$fn__6015.invoke(metrics.clj:14)
              at com.puppetlabs.utils.metrics$multitime_BANG__STAR_.invoke(metrics.clj:17)
              at com.puppetlabs.middleware$wrap_with_metrics_STAR_$fn__6077.invoke(middleware.clj:182)
              at com.puppetlabs.middleware$wrap_with_globals$fn__6047.invoke(middleware.clj:70)
              at com.puppetlabs.middleware$wrap_with_debug_logging$fn__6031.invoke(middleware.clj:24)
              at ring.adapter.jetty$proxy_handler$fn__5782.invoke(jetty.clj:18)
              at ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$0.handle(Unknown Source)
              at org.eclipse.jetty.server.handler.GzipHandler.handle(GzipHandler.java:267)
              at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
              at org.eclipse.jetty.server.Server.handle(Server.java:349)
              at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
              at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:884)
              at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:938)
              at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:634)
              at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
              at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:76)
              at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:191)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:609)
              at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:45)
              at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
              at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
              at java.lang.Thread.run(Thread.java:744)
      

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  ken Ken Barber
                  Reporter:
                  michal.ruzicka Michal Ruzicka
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: