Uploaded image for project: 'PuppetDB'
  1. PuppetDB
  2. PDB-3322

PuppetDB fails to update catalogs when sensitive parameters are added

    Details

    • Template:
    • Team:
      Systems Engineering
    • Story Points:
      2
    • Sprint:
      PuppetDB 2017-04-05
    • Release Notes:
      New Feature
    • Release Notes Summary:
      Sensitive parameters, which are created in puppet code using Sensitive.new, are now redacted before being sent to PuppetDB.
    • QA Risk Assessment:
      Needs Assessment

      Description

      When sensitive data is added to a resource, the resulting catalog contains a sensitive_parameters key. PuppetDB tries to work this new key into an UPDATE operation on the catalog_resources table, which fails during statement preparation with a 'Can't infer the SQL type to use for an instance of clojure.lang.PersistentVector' error. No errors occur if the resource is initially stored with sensitive data.

      Reproduction Case

      Install PE 2016.4.3.

      • Add a user resource to the default node in /etc/puppetlabs/code/environments/production/manifests/site.pp:

        user {'AzureDiamond':
          ensure   => present,
          password => 'hunter2',
        }
        

      • Run puppet agent -t to compile a catalog and enter the resource into PuppetDB.
      • Update the resource to mark the password as sensitive:

        user {'AzureDiamond':
          ensure   => present,
          password => Sensitive.new('hunter2'),
        }
        

      • Run puppet agent -t.

      Outcome

      The second agent run completes successfully, but /var/log/puppetlabs/puppetdb/puppetdb.log shows an error during catalog storage:

      2017-03-07 15:29:15,702 ERROR [p.p.mq-listener] [57cc2d5b-5334-47a1-8426-c4f1b78db715] [replace catalog] Retrying after attempt 0 for pe-201651-master.puppetdebug.vlan, due to: org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of clojure.lang.PersistentVector. Use setObject() with an explicit Types value to specify the type to use.
      org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of clojure.lang.PersistentVector. Use setObject() with an explicit Types value to specify the type to use.
      	at org.postgresql.jdbc.PgPreparedStatement.setObject(PgPreparedStatement.java:1039)
      	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.setObject(HikariProxyPreparedStatement.java)
      	at clojure.java.jdbc$eval21273$fn__21274.invokePrim(jdbc.clj:341)
      	at clojure.java.jdbc$eval21273$fn__21274.invoke(jdbc.clj)
      	at clojure.java.jdbc$eval21252$fn__21253$G__21243__21262.invoke(jdbc.clj:328)
      	at clojure.java.jdbc$set_parameters$fn__21349.invoke(jdbc.clj:478)
      	at clojure.core$map_indexed$mapi__7050$fn__7051.invoke(core.clj:7024)
      	at clojure.lang.LazySeq.sval(LazySeq.java:40)
      	at clojure.lang.LazySeq.seq(LazySeq.java:49)
      	at clojure.lang.RT.seq(RT.java:521)
      	at clojure.core$seq__4357.invokeStatic(core.clj:137)
      	at clojure.core$dorun.invokeStatic(core.clj:3024)
      	at clojure.core$dorun.invoke(core.clj:3024)
      	at clojure.java.jdbc$set_parameters.invokeStatic(jdbc.clj:477)
      	at clojure.java.jdbc$set_parameters.invoke(jdbc.clj:474)
      	at clojure.java.jdbc$db_do_execute_prepared_statement.invokeStatic(jdbc.clj:760)
      	at clojure.java.jdbc$db_do_execute_prepared_statement.invoke(jdbc.clj:748)
      	at clojure.java.jdbc$db_do_prepared.invokeStatic(jdbc.clj:786)
      	at clojure.java.jdbc$db_do_prepared.doInvoke(jdbc.clj:770)
      	at clojure.lang.RestFn.invoke(RestFn.java:464)
      	at clojure.java.jdbc$execute_BANG_$execute_helper__21451.invoke(jdbc.clj:891)
      	at clojure.java.jdbc$execute_BANG_.invokeStatic(jdbc.clj:894)
      	at clojure.java.jdbc$execute_BANG_.doInvoke(jdbc.clj:875)
      	at clojure.lang.RestFn.invoke(RestFn.java:464)
      	at clojure.java.jdbc$update_BANG_.invokeStatic(jdbc.clj:1077)
      	at clojure.java.jdbc$update_BANG_.doInvoke(jdbc.clj:1066)
      	at clojure.lang.RestFn.invoke(RestFn.java:470)
      	at clojure.lang.AFn.applyToHelper(AFn.java:165)
      	at clojure.lang.RestFn.applyTo(RestFn.java:132)
      	at clojure.core$apply.invokeStatic(core.clj:654)
      	at clojure.core$apply.doInvoke(core.clj:641)
      	at clojure.lang.RestFn.invoke(RestFn.java:533)
      	at puppetlabs.puppetdb.jdbc$update_BANG_.invokeStatic(jdbc.clj:72)
      	at puppetlabs.puppetdb.jdbc$update_BANG_.doInvoke(jdbc.clj:66)
      	at clojure.lang.RestFn.invoke(RestFn.java:445)
      	at puppetlabs.puppetdb.scf.storage$eval28606$update_catalog_resources_BANG___28611$fn__28612$fn__28613.invoke(storage.clj:583)
      	at puppetlabs.puppetdb.utils$eval6348$diff_fn__6353$fn__6354.invoke(utils.clj:65)
      	at puppetlabs.puppetdb.utils$eval6348$diff_fn__6353.invoke(utils.clj:54)
      	at puppetlabs.puppetdb.scf.storage$eval28646$add_resources_BANG___28651$fn__28652$fn__28653.invoke(storage.clj:603)
      	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:620)
      	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
      	at clojure.lang.RestFn.invoke(RestFn.java:425)
      	at puppetlabs.puppetdb.scf.storage$eval28646$add_resources_BANG___28651$fn__28652.invoke(storage.clj:601)
      	at puppetlabs.puppetdb.scf.storage$eval28646$add_resources_BANG___28651.invoke(storage.clj:594)
      	at puppetlabs.puppetdb.scf.storage$eval28847$update_catalog_associations_BANG___28852$fn__28856$fn__28858.invoke(storage.clj:713)
      	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.scf.storage$eval28847$update_catalog_associations_BANG___28852$fn__28856.invoke(storage.clj:712)
      	at puppetlabs.puppetdb.scf.storage$eval28847$update_catalog_associations_BANG___28852.invoke(storage.clj:707)
      	at puppetlabs.puppetdb.scf.storage$eval28885$replace_existing_catalog__28890$fn__28891$fn__28892.invoke(storage.clj:731)
      	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.scf.storage$eval28885$replace_existing_catalog__28890$fn__28891.invoke(storage.clj:729)
      	at puppetlabs.puppetdb.scf.storage$eval28885$replace_existing_catalog__28890.invoke(storage.clj:717)
      	at puppetlabs.puppetdb.scf.storage$eval28946$replace_catalog_BANG___28955$fn__28961$fn__28963$fn__28964.invoke(storage.clj:773)
      	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:620)
      	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
      	at clojure.lang.RestFn.invoke(RestFn.java:425)
      	at puppetlabs.puppetdb.scf.storage$eval28946$replace_catalog_BANG___28955$fn__28961$fn__28963.invoke(storage.clj:753)
      	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.scf.storage$eval28946$replace_catalog_BANG___28955$fn__28961.invoke(storage.clj:752)
      	at puppetlabs.puppetdb.scf.storage$eval28946$replace_catalog_BANG___28955.invoke(storage.clj:745)
      	at puppetlabs.puppetdb.command$replace_catalog_STAR_$fn__42268.invoke(command.clj:259)
      	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn__21948$fn__21949.invoke(jdbc.clj:308)
      	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:595)
      	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
      	at clojure.lang.RestFn.invoke(RestFn.java:464)
      	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn__21948.invoke(jdbc.clj:307)
      	at puppetlabs.puppetdb.jdbc$eval21922$retry_sql_STAR___21927$fn__21928$fn__21929.invoke(jdbc.clj:285)
      	at puppetlabs.puppetdb.jdbc$eval21922$retry_sql_STAR___21927$fn__21928.invoke(jdbc.clj:284)
      	at puppetlabs.puppetdb.jdbc$eval21922$retry_sql_STAR___21927.invoke(jdbc.clj:275)
      	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invokeStatic(jdbc.clj:305)
      	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:300)
      	at puppetlabs.puppetdb.command$replace_catalog_STAR_.invokeStatic(command.clj:257)
      	at puppetlabs.puppetdb.command$replace_catalog_STAR_.invoke(command.clj:253)
      	at puppetlabs.puppetdb.command$replace_catalog.invokeStatic(command.clj:268)
      	at puppetlabs.puppetdb.command$replace_catalog.invoke(command.clj:262)
      	at puppetlabs.puppetdb.command$process_command_BANG_.invokeStatic(command.clj:373)
      	at puppetlabs.puppetdb.command$process_command_BANG_.invoke(command.clj:368)
      	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_$fn__42465.invoke(command.clj:435)
      	at puppetlabs.puppetdb.command$call_with_quick_retry$fn__42460.invoke(command.clj:421)
      	at puppetlabs.puppetdb.command$call_with_quick_retry.invokeStatic(command.clj:420)
      	at puppetlabs.puppetdb.command$call_with_quick_retry.invoke(command.clj:418)
      	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_.invokeStatic(command.clj:433)
      	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_.invoke(command.clj:431)
      	at puppetlabs.puppetdb.command$reify__42469$service_fnk__11430__auto___positional$reify__42480$fn__42483.invoke(command.clj:471)
      	at puppetlabs.puppetdb.mq_listener$reify__42717$service_fnk__11430__auto___positional$reify__42727.process_message(mq_listener.clj:399)
      	at puppetlabs.puppetdb.mq_listener$reify__42717$service_fnk__11430__auto___positional$reify__42727$process_msg__42728.invoke(mq_listener.clj:367)
      	at puppetlabs.puppetdb.mq_listener$wrap_with_discard$fn__42545$fn__42547.invoke(mq_listener.clj:228)
      	at puppetlabs.puppetdb.mq_listener.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.mq_listener$wrap_with_discard$fn__42545.invoke(mq_listener.clj:227)
      	at puppetlabs.puppetdb.mq_listener$wrap_with_exception_handling$fn__42533$fn__42535.invoke(mq_listener.clj:182)
      	at puppetlabs.puppetdb.mq_listener.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
      	at com.codahale.metrics.Timer.time(Timer.java:101)
      	at puppetlabs.puppetdb.mq_listener$wrap_with_exception_handling$fn__42533.invoke(mq_listener.clj:181)
      	at puppetlabs.puppetdb.mq_listener$wrap_with_command_parser$fn__42541.invoke(mq_listener.clj:204)
      	at puppetlabs.puppetdb.mq_listener$wrap_with_meter$fn__42525.invoke(mq_listener.clj:142)
      	at puppetlabs.puppetdb.mq_listener$wrap_with_thread_name$fn__42552.invoke(mq_listener.clj:243)
      	at puppetlabs.puppetdb.mq_listener$start_receiver$reify__42710.onMessage(mq_listener.clj:347)
      	at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1401)
      	at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
      	at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
      	at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
      	at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      Dropping a tracepoint into clojure.java.jdbc$eval21273$fn__21274.invokePrim(jdbc.clj:341) shows that the following UPDATE statement is being prepared and that the error occurs when parameter 1 is being set to ["password"]:

      [#object[com.zaxxer.hikari.pool.HikariProxyPreparedStatement 0x242441f1 "HikariProxyPreparedStatement@606355953 wrapping UPDATE catalog_resources SET sensitive_parameters = ? WHERE certname_id = ? and type = ? and title = ?"] 1 ["password"]]
      

      Expected Outcome

      PuppetDB accepts catalog updates where resource parameters have been marked as sensitive.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  rob.browning Rob Browning
                  Reporter:
                  chuck Charlie Sharpsteen
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  13 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: