Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-3357

PuppetDB fails to update catalogs when rich data types are enabled

    Details

    • Type: Bug
    • Status: Open
    • Priority: Normal
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Template:
    • Team:
      PuppetDB
    • QA Risk Assessment:
      Needs Assessment

      Description

      When a rich data type is added to a resource, the resulting catalog contains resources with ext_parameters keys. PuppetDB tries to work these new keys into an UPDATE operation on the catalog_resources table, which fails during statement preparation with a 'No hstore extension installed' error. No errors occur if the resource is initially stored with sensitive data.

      The root cause for this behavior is the same as PDB-3322, which concerns the Sensitive data type.

      Reproduction Case

      Install PE 2017.1.0 on CentOS 7.

      • Add a file resource to the default node in /etc/puppetlabs/code/environments/production/manifests/site.pp that uses the binary_file function to manage content with binary data:

      file{'/tmp/rich_data_test':
        ensure  => present,
        content => binary_file('/bin/true'),
      }
      

      • Run puppet agent -t to compile a catalog and enter the resource into PuppetDB.
      • Enable rich data and reload Puppet Server:

      puppet config set --section main rich_data true
      systemctl reload pe-puppetserver
      

      • Run puppet agent -t.

      Outcome

      The agent run completes successfully, but /var/log/puppetlabs/puppetdb/puppetdb.log shows an error during catalog storage:

      2017-03-15 19:33:12,583 ERROR [p.p.command] [8] [replace catalog] Retrying after attempt 0 for pe-201710-master.puppetdebug.vlan, due to: org.postgresql.util.PSQLException: No hstore extension installed.
      org.postgresql.util.PSQLException: No hstore extension installed.
      	at org.postgresql.jdbc.PgPreparedStatement.setMap(PgPreparedStatement.java:553)
      	at org.postgresql.jdbc.PgPreparedStatement.setObject(PgPreparedStatement.java:1036)
      	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.setObject(HikariProxyPreparedStatement.java)
      	at clojure.java.jdbc$eval21518$fn__21519.invokePrim(jdbc.clj:341)
      	at clojure.java.jdbc$eval21518$fn__21519.invoke(jdbc.clj)
      	at clojure.java.jdbc$eval21497$fn__21498$G__21488__21507.invoke(jdbc.clj:328)
      	at clojure.java.jdbc$set_parameters$fn__21594.invoke(jdbc.clj:478)
      	at clojure.core$map_indexed$mapi__7050$fn__7051.invoke(core.clj:7024)
      	at clojure.lang.LazySeq.sval(LazySeq.java:40)
      	at clojure.lang.LazySeq.seq(LazySeq.java:49)
      	at clojure.lang.RT.seq(RT.java:521)
      	at clojure.core$seq__4357.invokeStatic(core.clj:137)
      	at clojure.core$dorun.invokeStatic(core.clj:3024)
      	at clojure.core$dorun.invoke(core.clj:3024)
      	at clojure.java.jdbc$set_parameters.invokeStatic(jdbc.clj:477)
      	at clojure.java.jdbc$set_parameters.invoke(jdbc.clj:474)
      	at clojure.java.jdbc$db_do_execute_prepared_statement.invokeStatic(jdbc.clj:760)
      	at clojure.java.jdbc$db_do_execute_prepared_statement.invoke(jdbc.clj:748)
      	at clojure.java.jdbc$db_do_prepared.invokeStatic(jdbc.clj:786)
      	at clojure.java.jdbc$db_do_prepared.doInvoke(jdbc.clj:770)
      	at clojure.lang.RestFn.invoke(RestFn.java:464)
      	at clojure.java.jdbc$execute_BANG_$execute_helper__21696.invoke(jdbc.clj:891)
      	at clojure.java.jdbc$execute_BANG_.invokeStatic(jdbc.clj:894)
      	at clojure.java.jdbc$execute_BANG_.doInvoke(jdbc.clj:875)
      	at clojure.lang.RestFn.invoke(RestFn.java:464)
      	at clojure.java.jdbc$update_BANG_.invokeStatic(jdbc.clj:1077)
      	at clojure.java.jdbc$update_BANG_.doInvoke(jdbc.clj:1066)
      	at clojure.lang.RestFn.invoke(RestFn.java:470)
      	at clojure.lang.AFn.applyToHelper(AFn.java:165)
      	at clojure.lang.RestFn.applyTo(RestFn.java:132)
      	at clojure.core$apply.invokeStatic(core.clj:654)
      	at clojure.core$apply.doInvoke(core.clj:641)
      	at clojure.lang.RestFn.invoke(RestFn.java:533)
      	at puppetlabs.puppetdb.jdbc$update_BANG_.invokeStatic(jdbc.clj:72)
      	at puppetlabs.puppetdb.jdbc$update_BANG_.doInvoke(jdbc.clj:66)
      	at clojure.lang.RestFn.invoke(RestFn.java:445)
      	at puppetlabs.puppetdb.scf.storage$eval28865$update_catalog_resources_BANG___28870$fn__28871$fn__28872.invoke(storage.clj:583)
      	at puppetlabs.puppetdb.utils$eval6351$diff_fn__6356$fn__6357.invoke(utils.clj:67)
      	at puppetlabs.puppetdb.utils$eval6351$diff_fn__6356.invoke(utils.clj:56)
      	at puppetlabs.puppetdb.scf.storage$eval28905$add_resources_BANG___28910$fn__28911$fn__28912.invoke(storage.clj:603)
      	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:620)
      	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
      	at clojure.lang.RestFn.invoke(RestFn.java:425)
      	at puppetlabs.puppetdb.scf.storage$eval28905$add_resources_BANG___28910$fn__28911.invoke(storage.clj:601)
      	at puppetlabs.puppetdb.scf.storage$eval28905$add_resources_BANG___28910.invoke(storage.clj:594)
      	at puppetlabs.puppetdb.scf.storage$eval29106$update_catalog_associations_BANG___29111$fn__29115$fn__29117.invoke(storage.clj:713)
      	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.scf.storage$eval29106$update_catalog_associations_BANG___29111$fn__29115.invoke(storage.clj:712)
      	at puppetlabs.puppetdb.scf.storage$eval29106$update_catalog_associations_BANG___29111.invoke(storage.clj:707)
      	at puppetlabs.puppetdb.scf.storage$eval29144$replace_existing_catalog__29149$fn__29150$fn__29151.invoke(storage.clj:731)
      	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.scf.storage$eval29144$replace_existing_catalog__29149$fn__29150.invoke(storage.clj:729)
      	at puppetlabs.puppetdb.scf.storage$eval29144$replace_existing_catalog__29149.invoke(storage.clj:717)
      	at puppetlabs.puppetdb.scf.storage$eval29205$replace_catalog_BANG___29214$fn__29220$fn__29222$fn__29223.invoke(storage.clj:773)
      	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:620)
      	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
      	at clojure.lang.RestFn.invoke(RestFn.java:425)
      	at puppetlabs.puppetdb.scf.storage$eval29205$replace_catalog_BANG___29214$fn__29220$fn__29222.invoke(storage.clj:753)
      	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.scf.storage$eval29205$replace_catalog_BANG___29214$fn__29220.invoke(storage.clj:752)
      	at puppetlabs.puppetdb.scf.storage$eval29205$replace_catalog_BANG___29214.invoke(storage.clj:745)
      	at puppetlabs.puppetdb.command$replace_catalog_STAR_$fn__32369.invoke(command.clj:279)
      	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn__22193$fn__22194.invoke(jdbc.clj:308)
      	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:595)
      	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
      	at clojure.lang.RestFn.invoke(RestFn.java:464)
      	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn__22193.invoke(jdbc.clj:307)
      	at puppetlabs.puppetdb.jdbc$eval22167$retry_sql_STAR___22172$fn__22173$fn__22174.invoke(jdbc.clj:285)
      	at puppetlabs.puppetdb.jdbc$eval22167$retry_sql_STAR___22172$fn__22173.invoke(jdbc.clj:284)
      	at puppetlabs.puppetdb.jdbc$eval22167$retry_sql_STAR___22172.invoke(jdbc.clj:275)
      	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invokeStatic(jdbc.clj:305)
      	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:300)
      	at puppetlabs.puppetdb.command$replace_catalog_STAR_.invokeStatic(command.clj:277)
      	at puppetlabs.puppetdb.command$replace_catalog_STAR_.invoke(command.clj:274)
      	at puppetlabs.puppetdb.command$replace_catalog.invokeStatic(command.clj:288)
      	at puppetlabs.puppetdb.command$replace_catalog.invoke(command.clj:283)
      	at puppetlabs.puppetdb.command$process_command_BANG_.invokeStatic(command.clj:395)
      	at puppetlabs.puppetdb.command$process_command_BANG_.invoke(command.clj:389)
      	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_$fn__32525.invoke(command.clj:448)
      	at puppetlabs.puppetdb.command$call_with_quick_retry$fn__32518.invoke(command.clj:432)
      	at puppetlabs.puppetdb.command$call_with_quick_retry.invokeStatic(command.clj:431)
      	at puppetlabs.puppetdb.command$call_with_quick_retry.invoke(command.clj:429)
      	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_.invokeStatic(command.clj:446)
      	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_.invoke(command.clj:444)
      	at puppetlabs.puppetdb.command$process_cmdref$fn__32534.invoke(command.clj:508)
      	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__30356$fn__30357$fn__30358.invoke(metrics.clj:14)
      	at puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__30356$fn__30357.invoke(metrics.clj:14)
      	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__30356$fn__30357$fn__30358.invoke(metrics.clj:14)
      	at puppetlabs.puppetdb.utils.metrics.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_$fn__30356$fn__30357.invoke(metrics.clj:14)
      	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_.invokeStatic(metrics.clj:17)
      	at puppetlabs.puppetdb.utils.metrics$multitime_BANG__STAR_.invoke(metrics.clj:6)
      	at puppetlabs.puppetdb.command$process_cmdref.invokeStatic(command.clj:504)
      	at puppetlabs.puppetdb.command$process_cmdref.invoke(command.clj:490)
      	at puppetlabs.puppetdb.command$message_handler$fn__32542.invoke(command.clj:554)
      	at puppetlabs.puppetdb.threadpool$dochan$fn__32301$fn__32302.invoke(threadpool.clj:117)
      	at puppetlabs.puppetdb.threadpool$call_on_threadpool$fn__32296.invoke(threadpool.clj:95)
      	at clojure.lang.AFn.run(AFn.java:22)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Expected Outcome

      PuppetDB accepts catalogs updates when rich data has been enabled.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                chuck Charlie Sharpsteen
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Zendesk Support