Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-904

Adding a fact with a large value makes the PG DB enter recovery mode / crash

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: PDB 2.2.0
    • Fix Version/s: PDB 2.2.1
    • Component/s: None
    • Labels:
      None
    • Environment:

      PostgreSQL 9.3.5 from PGDG: postgresql93-server-9.3.5-1PGDG.rhel6.x86_64
      Centos 6

    • Template:
    • Story Points:
      2
    • Sprint:
      PuppetDB 2014-10-22

      Description

      Adding a fact with a large value makes the PG DB enter recovery mode / crash.

      Fact value is ~32KB.

      postmaster[16195]: segfault at 7fff7957dfe8 ip 00007feb2f10c93c sp 00007fff7957dff0 error 6 in pg_trgm.so[7feb2f109000+8000]
      postmaster[16211]: segfault at 7fff7957dfe8 ip 00007feb2f10c93c sp 00007fff7957dff0 error 6 in pg_trgm.so[7feb2f109000+8000]
      postmaster[16230]: segfault at 7fff7957dfe8 ip 00007feb2f10c93c sp 00007fff7957dff0 error 6 in pg_trgm.so[7feb2f109000+8000]
      postmaster[16246]: segfault at 7fff7957dfe8 ip 00007feb2f10c93c sp 00007fff7957dff0 error 6 in pg_trgm.so[7feb2f109000+8000]
      postmaster[16268]: segfault at 7fff7957dfe8 ip 00007feb2f10c93c sp 00007fff7957dff0 error 6 in pg_trgm.so[7feb2f109000+8000]
      

      2014-10-01 04:59:26,080 ERROR [c.p.p.command] [e19e8eaa-ad20-43ad-97a2-9b28d91f646b] [replace facts] Retrying after attempt 4, due to: org.postgresql.util.PSQLException: ERROR: index row requires 8440 bytes, maximum size is 8191
      org.postgresql.util.PSQLException: ERROR: index row requires 8440 bytes, maximum size is 8191
              at org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2157) ~[puppetdb.jar:na]
              at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1886) ~[puppetdb.jar:na]
              at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:255) ~[puppetdb.jar:na]
              at org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement.java:555) ~[puppetdb.jar:na]
              at org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2Statement.java:417) ~[puppetdb.jar:na]
              at org.postgresql.jdbc2.AbstractJdbc2Statement.executeUpdate(AbstractJdbc2Statement.java:363) ~[puppetdb.jar:na]
              at com.jolbox.bonecp.PreparedStatementHandle.executeUpdate(PreparedStatementHandle.java:203) ~[puppetdb.jar:na]
              at clojure.java.jdbc.internal$do_prepared_return_keys_STAR_$fn__6343.invoke(internal.clj:337) ~[na:na]
              at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:223) ~[na:na]
              at clojure.java.jdbc.internal$do_prepared_return_keys_STAR_.invoke(internal.clj:337) ~[na:na]
              at clojure.lang.AFn.applyToHelper(AFn.java:156) ~[puppetdb.jar:na]
              at clojure.lang.AFn.applyTo(AFn.java:144) ~[puppetdb.jar:na]
              at clojure.core$apply.invoke(core.clj:626) ~[puppetdb.jar:na]
              at clojure.java.jdbc$insert_values.doInvoke(jdbc.clj:263) ~[na:na]
              at clojure.lang.RestFn.invoke(RestFn.java:442) ~[puppetdb.jar:na]
              at clojure.java.jdbc$insert_records$ins_v__6411.invoke(jdbc.clj:278) ~[na:na]
              at clojure.core$map$fn__4245.invoke(core.clj:2557) ~[puppetdb.jar:na]
              at clojure.lang.LazySeq.sval(LazySeq.java:40) ~[puppetdb.jar:na]
              at clojure.lang.LazySeq.seq(LazySeq.java:49) ~[puppetdb.jar:na]
              at clojure.lang.RT.seq(RT.java:484) ~[puppetdb.jar:na]
              at clojure.core$seq.invoke(core.clj:133) ~[puppetdb.jar:na]
              at clojure.core$dorun.invoke(core.clj:2855) ~[puppetdb.jar:na]
              at clojure.core$doall.invoke(core.clj:2871) ~[puppetdb.jar:na]
              at clojure.java.jdbc$insert_records.doInvoke(jdbc.clj:279) ~[na:na]
              at clojure.lang.RestFn.applyTo(RestFn.java:139) ~[puppetdb.jar:na]
              at clojure.core$apply.invoke(core.clj:626) ~[puppetdb.jar:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9557$fact_value_new_ids__9558$fn__9559.invoke(storage.clj:866) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9557$fact_value_new_ids__9558.invoke(storage.clj:857) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9581$fact_values_to_ids__9582$fn__9583.invoke(storage.clj:894) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9581$fact_values_to_ids__9582.invoke(storage.clj:872) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9607$new_fact_value_ids_STAR___9608$fn__9609.invoke(storage.clj:916) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9607$new_fact_value_ids_STAR___9608.invoke(storage.clj:897) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9631$new_fact_value_ids__9632$fn__9633.invoke(storage.clj:926) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9631$new_fact_value_ids__9632.invoke(storage.clj:919) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9787$update_facts_BANG___9788$fn__9792.invoke(storage.clj:1003) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9787$update_facts_BANG___9788.invoke(storage.clj:996) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9981$replace_facts_BANG___9982$fn__9986$fn__9988.invoke(storage.clj:1204) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na]
              at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9981$replace_facts_BANG___9982$fn__9986.invoke(storage.clj:1201) ~[na:na]
              at com.puppetlabs.puppetdb.scf.storage$eval9981$replace_facts_BANG___9982.invoke(storage.clj:1194) ~[na:na]
              at com.puppetlabs.puppetdb.command$eval11280$fn__11283$fn__11296.invoke(command.clj:381) ~[na:na]
              at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__6496$fn__6497.invoke(jdbc.clj:223) ~[na:na]
              at clojure.java.jdbc.internal$transaction_STAR_.invoke(internal.clj:204) ~[na:na]
              at com.puppetlabs.jdbc$with_transacted_connection_fn$fn__6496.invoke(jdbc.clj:222) ~[na:na]
              at clojure.java.jdbc.internal$with_connection_STAR_.invoke(internal.clj:186) ~[na:na]
              at com.puppetlabs.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:219) ~[na:na]
              at com.puppetlabs.puppetdb.command$eval11280$fn__11283.invoke(command.clj:379) ~[na:na]
              at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na]
              at com.puppetlabs.puppetdb.command$eval11265$fn__11267.invoke(command.clj:364) ~[na:na]
              at clojure.lang.MultiFn.invoke(MultiFn.java:231) ~[puppetdb.jar:na]
              at com.puppetlabs.puppetdb.command$produce_message_handler$fn__11452.invoke(command.clj:647) ~[na:na]
              at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__11401$fn__11405.invoke(command.clj:554) ~[na:na]
              at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na]
              at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]
              at com.puppetlabs.puppetdb.command$wrap_with_discard$fn__11401.invoke(command.clj:553) ~[na:na]
              at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__11386$fn__11387.invoke(command.clj:507) ~[na:na]
              at com.puppetlabs.puppetdb.command.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source) ~[na:na]
              at com.yammer.metrics.core.Timer.time(Timer.java:91) ~[puppetdb.jar:na]
              at com.puppetlabs.puppetdb.command$wrap_with_exception_handling$fn__11386.invoke(command.clj:506) ~[na:na]
              at com.puppetlabs.puppetdb.command$wrap_with_command_parser$fn__11396.invoke(command.clj:529) [na:na]
              at com.puppetlabs.puppetdb.command$wrap_with_meter$fn__11376.invoke(command.clj:467) [na:na]
              at com.puppetlabs.puppetdb.command$wrap_with_thread_name$fn__11410.invoke(command.clj:569) [na:na]
              at com.puppetlabs.mq$create_message_listener$reify__10557.onMessage(mq.clj:270) [na:na]
              at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) [puppetdb.jar:na]
              at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) [puppetdb.jar:na]
              at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) [puppetdb.jar:na]
              at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) [puppetdb.jar:na]
              at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) [puppetdb.jar:na]
              at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) [puppetdb.jar:na]
              at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) [puppetdb.jar:na]
              at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) [puppetdb.jar:na]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
      

      2014-10-01 10:01:41,639 ERROR [c.j.b.PoolWatchThread] Error in trying to obtain a connection. Retrying in 7000ms
      org.postgresql.util.PSQLException: FATAL: the database system is in recovery mode
              at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:398) ~[puppetdb.jar:na]
              at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:173) ~[puppetdb.jar:na]
              at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:64) ~[puppetdb.jar:na]
              at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:136) ~[puppetdb.jar:na]
              at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:29) ~[puppetdb.jar:na]
              at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:21) ~[puppetdb.jar:na]
              at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:31) ~[puppetdb.jar:na]
              at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24) ~[puppetdb.jar:na]
              at org.postgresql.Driver.makeConnection(Driver.java:393) ~[puppetdb.jar:na]
              at org.postgresql.Driver.connect(Driver.java:267) ~[puppetdb.jar:na]
              at java.sql.DriverManager.getConnection(DriverManager.java:571) ~[na:1.7.0_67]
              at java.sql.DriverManager.getConnection(DriverManager.java:215) ~[na:1.7.0_67]
              at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:256) ~[puppetdb.jar:na]
              at com.jolbox.bonecp.ConnectionHandle.obtainInternalConnection(ConnectionHandle.java:211) ~[puppetdb.jar:na]
              at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:170) ~[puppetdb.jar:na]
              at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:101) [puppetdb.jar:na]
              at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82) [puppetdb.jar:na]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_67]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_67]
              at java.lang.Thread.run(Thread.java:745) [na:1.7.0_67]
      

        Attachments

          Activity

            jsd-sla-details-panel

              People

              • Assignee:
                ken Ken Barber
                Reporter:
                seitzbg Bryan Seitz
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: