[PDB-3322] PuppetDB fails to update catalogs when sensitive parameters are added Created: 2017/03/07  Updated: 2017/05/30  Resolved: 2017/04/04

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: PDB 4.2.3
Fix Version/s: PDB 4.2.3.2, PDB 4.4.0

Type: Bug Priority: Major
Reporter: Charlie Sharpsteen Assignee: Rob Browning
Resolution: Fixed Votes: 0
Labels: davis-triage, docs_reviewed
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Blocks
Duplicate
is duplicated by PDB-1387 Support data anonymization at the sto... Closed
Relates
relates to PDB-3357 PuppetDB fails to update catalogs whe... Closed
relates to DOCUMENT-684 Sensitive data type page should call ... Resolved
Template:
Team: Systems Engineering
Story Points: 2
Sprint: PuppetDB 2017-04-05
Release Notes: New Feature
Release Notes Summary: Sensitive parameters, which are created in puppet code using Sensitive.new, are now redacted before being sent to PuppetDB.
QA Risk Assessment: Needs Assessment

 Description   

When sensitive data is added to a resource, the resulting catalog contains a sensitive_parameters key. PuppetDB tries to work this new key into an UPDATE operation on the catalog_resources table, which fails during statement preparation with a 'Can't infer the SQL type to use for an instance of clojure.lang.PersistentVector' error. No errors occur if the resource is initially stored with sensitive data.

Reproduction Case

Install PE 2016.4.3.

  • Add a user resource to the default node in /etc/puppetlabs/code/environments/production/manifests/site.pp:

    user {'AzureDiamond':
      ensure   => present,
      password => 'hunter2',
    }
    

  • Run puppet agent -t to compile a catalog and enter the resource into PuppetDB.
  • Update the resource to mark the password as sensitive:

    user {'AzureDiamond':
      ensure   => present,
      password => Sensitive.new('hunter2'),
    }
    

  • Run puppet agent -t.

Outcome

The second agent run completes successfully, but /var/log/puppetlabs/puppetdb/puppetdb.log shows an error during catalog storage:

2017-03-07 15:29:15,702 ERROR [p.p.mq-listener] [57cc2d5b-5334-47a1-8426-c4f1b78db715] [replace catalog] Retrying after attempt 0 for pe-201651-master.puppetdebug.vlan, due to: org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of clojure.lang.PersistentVector. Use setObject() with an explicit Types value to specify the type to use.
org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of clojure.lang.PersistentVector. Use setObject() with an explicit Types value to specify the type to use.
	at org.postgresql.jdbc.PgPreparedStatement.setObject(PgPreparedStatement.java:1039)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.setObject(HikariProxyPreparedStatement.java)
	at clojure.java.jdbc$eval21273$fn__21274.invokePrim(jdbc.clj:341)
	at clojure.java.jdbc$eval21273$fn__21274.invoke(jdbc.clj)
	at clojure.java.jdbc$eval21252$fn__21253$G__21243__21262.invoke(jdbc.clj:328)
	at clojure.java.jdbc$set_parameters$fn__21349.invoke(jdbc.clj:478)
	at clojure.core$map_indexed$mapi__7050$fn__7051.invoke(core.clj:7024)
	at clojure.lang.LazySeq.sval(LazySeq.java:40)
	at clojure.lang.LazySeq.seq(LazySeq.java:49)
	at clojure.lang.RT.seq(RT.java:521)
	at clojure.core$seq__4357.invokeStatic(core.clj:137)
	at clojure.core$dorun.invokeStatic(core.clj:3024)
	at clojure.core$dorun.invoke(core.clj:3024)
	at clojure.java.jdbc$set_parameters.invokeStatic(jdbc.clj:477)
	at clojure.java.jdbc$set_parameters.invoke(jdbc.clj:474)
	at clojure.java.jdbc$db_do_execute_prepared_statement.invokeStatic(jdbc.clj:760)
	at clojure.java.jdbc$db_do_execute_prepared_statement.invoke(jdbc.clj:748)
	at clojure.java.jdbc$db_do_prepared.invokeStatic(jdbc.clj:786)
	at clojure.java.jdbc$db_do_prepared.doInvoke(jdbc.clj:770)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at clojure.java.jdbc$execute_BANG_$execute_helper__21451.invoke(jdbc.clj:891)
	at clojure.java.jdbc$execute_BANG_.invokeStatic(jdbc.clj:894)
	at clojure.java.jdbc$execute_BANG_.doInvoke(jdbc.clj:875)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at clojure.java.jdbc$update_BANG_.invokeStatic(jdbc.clj:1077)
	at clojure.java.jdbc$update_BANG_.doInvoke(jdbc.clj:1066)
	at clojure.lang.RestFn.invoke(RestFn.java:470)
	at clojure.lang.AFn.applyToHelper(AFn.java:165)
	at clojure.lang.RestFn.applyTo(RestFn.java:132)
	at clojure.core$apply.invokeStatic(core.clj:654)
	at clojure.core$apply.doInvoke(core.clj:641)
	at clojure.lang.RestFn.invoke(RestFn.java:533)
	at puppetlabs.puppetdb.jdbc$update_BANG_.invokeStatic(jdbc.clj:72)
	at puppetlabs.puppetdb.jdbc$update_BANG_.doInvoke(jdbc.clj:66)
	at clojure.lang.RestFn.invoke(RestFn.java:445)
	at puppetlabs.puppetdb.scf.storage$eval28606$update_catalog_resources_BANG___28611$fn__28612$fn__28613.invoke(storage.clj:583)
	at puppetlabs.puppetdb.utils$eval6348$diff_fn__6353$fn__6354.invoke(utils.clj:65)
	at puppetlabs.puppetdb.utils$eval6348$diff_fn__6353.invoke(utils.clj:54)
	at puppetlabs.puppetdb.scf.storage$eval28646$add_resources_BANG___28651$fn__28652$fn__28653.invoke(storage.clj:603)
	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:620)
	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at puppetlabs.puppetdb.scf.storage$eval28646$add_resources_BANG___28651$fn__28652.invoke(storage.clj:601)
	at puppetlabs.puppetdb.scf.storage$eval28646$add_resources_BANG___28651.invoke(storage.clj:594)
	at puppetlabs.puppetdb.scf.storage$eval28847$update_catalog_associations_BANG___28852$fn__28856$fn__28858.invoke(storage.clj:713)
	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.scf.storage$eval28847$update_catalog_associations_BANG___28852$fn__28856.invoke(storage.clj:712)
	at puppetlabs.puppetdb.scf.storage$eval28847$update_catalog_associations_BANG___28852.invoke(storage.clj:707)
	at puppetlabs.puppetdb.scf.storage$eval28885$replace_existing_catalog__28890$fn__28891$fn__28892.invoke(storage.clj:731)
	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.scf.storage$eval28885$replace_existing_catalog__28890$fn__28891.invoke(storage.clj:729)
	at puppetlabs.puppetdb.scf.storage$eval28885$replace_existing_catalog__28890.invoke(storage.clj:717)
	at puppetlabs.puppetdb.scf.storage$eval28946$replace_catalog_BANG___28955$fn__28961$fn__28963$fn__28964.invoke(storage.clj:773)
	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:620)
	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
	at clojure.lang.RestFn.invoke(RestFn.java:425)
	at puppetlabs.puppetdb.scf.storage$eval28946$replace_catalog_BANG___28955$fn__28961$fn__28963.invoke(storage.clj:753)
	at puppetlabs.puppetdb.scf.storage.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.scf.storage$eval28946$replace_catalog_BANG___28955$fn__28961.invoke(storage.clj:752)
	at puppetlabs.puppetdb.scf.storage$eval28946$replace_catalog_BANG___28955.invoke(storage.clj:745)
	at puppetlabs.puppetdb.command$replace_catalog_STAR_$fn__42268.invoke(command.clj:259)
	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn__21948$fn__21949.invoke(jdbc.clj:308)
	at clojure.java.jdbc$db_transaction_STAR_.invokeStatic(jdbc.clj:595)
	at clojure.java.jdbc$db_transaction_STAR_.doInvoke(jdbc.clj:568)
	at clojure.lang.RestFn.invoke(RestFn.java:464)
	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn$fn__21948.invoke(jdbc.clj:307)
	at puppetlabs.puppetdb.jdbc$eval21922$retry_sql_STAR___21927$fn__21928$fn__21929.invoke(jdbc.clj:285)
	at puppetlabs.puppetdb.jdbc$eval21922$retry_sql_STAR___21927$fn__21928.invoke(jdbc.clj:284)
	at puppetlabs.puppetdb.jdbc$eval21922$retry_sql_STAR___21927.invoke(jdbc.clj:275)
	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invokeStatic(jdbc.clj:305)
	at puppetlabs.puppetdb.jdbc$with_transacted_connection_fn.invoke(jdbc.clj:300)
	at puppetlabs.puppetdb.command$replace_catalog_STAR_.invokeStatic(command.clj:257)
	at puppetlabs.puppetdb.command$replace_catalog_STAR_.invoke(command.clj:253)
	at puppetlabs.puppetdb.command$replace_catalog.invokeStatic(command.clj:268)
	at puppetlabs.puppetdb.command$replace_catalog.invoke(command.clj:262)
	at puppetlabs.puppetdb.command$process_command_BANG_.invokeStatic(command.clj:373)
	at puppetlabs.puppetdb.command$process_command_BANG_.invoke(command.clj:368)
	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_$fn__42465.invoke(command.clj:435)
	at puppetlabs.puppetdb.command$call_with_quick_retry$fn__42460.invoke(command.clj:421)
	at puppetlabs.puppetdb.command$call_with_quick_retry.invokeStatic(command.clj:420)
	at puppetlabs.puppetdb.command$call_with_quick_retry.invoke(command.clj:418)
	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_.invokeStatic(command.clj:433)
	at puppetlabs.puppetdb.command$process_command_and_respond_BANG_.invoke(command.clj:431)
	at puppetlabs.puppetdb.command$reify__42469$service_fnk__11430__auto___positional$reify__42480$fn__42483.invoke(command.clj:471)
	at puppetlabs.puppetdb.mq_listener$reify__42717$service_fnk__11430__auto___positional$reify__42727.process_message(mq_listener.clj:399)
	at puppetlabs.puppetdb.mq_listener$reify__42717$service_fnk__11430__auto___positional$reify__42727$process_msg__42728.invoke(mq_listener.clj:367)
	at puppetlabs.puppetdb.mq_listener$wrap_with_discard$fn__42545$fn__42547.invoke(mq_listener.clj:228)
	at puppetlabs.puppetdb.mq_listener.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.mq_listener$wrap_with_discard$fn__42545.invoke(mq_listener.clj:227)
	at puppetlabs.puppetdb.mq_listener$wrap_with_exception_handling$fn__42533$fn__42535.invoke(mq_listener.clj:182)
	at puppetlabs.puppetdb.mq_listener.proxy$java.lang.Object$Callable$7da976d4.call(Unknown Source)
	at com.codahale.metrics.Timer.time(Timer.java:101)
	at puppetlabs.puppetdb.mq_listener$wrap_with_exception_handling$fn__42533.invoke(mq_listener.clj:181)
	at puppetlabs.puppetdb.mq_listener$wrap_with_command_parser$fn__42541.invoke(mq_listener.clj:204)
	at puppetlabs.puppetdb.mq_listener$wrap_with_meter$fn__42525.invoke(mq_listener.clj:142)
	at puppetlabs.puppetdb.mq_listener$wrap_with_thread_name$fn__42552.invoke(mq_listener.clj:243)
	at puppetlabs.puppetdb.mq_listener$start_receiver$reify__42710.onMessage(mq_listener.clj:347)
	at org.apache.activemq.ActiveMQMessageConsumer.dispatch(ActiveMQMessageConsumer.java:1401)
	at org.apache.activemq.ActiveMQSessionExecutor.dispatch(ActiveMQSessionExecutor.java:131)
	at org.apache.activemq.ActiveMQSessionExecutor.iterate(ActiveMQSessionExecutor.java:202)
	at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
	at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Dropping a tracepoint into clojure.java.jdbc$eval21273$fn__21274.invokePrim(jdbc.clj:341) shows that the following UPDATE statement is being prepared and that the error occurs when parameter 1 is being set to ["password"]:

[#object[com.zaxxer.hikari.pool.HikariProxyPreparedStatement 0x242441f1 "HikariProxyPreparedStatement@606355953 wrapping UPDATE catalog_resources SET sensitive_parameters = ? WHERE certname_id = ? and type = ? and title = ?"] 1 ["password"]]

Expected Outcome

PuppetDB accepts catalog updates where resource parameters have been marked as sensitive.



 Comments   
Comment by Michael Smith [ 2017/03/31 ]

This will specifically avoid adding Sensitive types to PuppetDB?

Comment by Matt Spaulding [ 2017/05/30 ]

I just installed PE 2017.2.1 which includes PDB 4.4.0 and I am still encountering this problem. I get the exact same stack trace as described in this ticket. Can it be reopened?

Comment by Nick Walker [ 2017/05/30 ]

Matt Spaulding I replied in our support ticket. The command queue will retain messages that were sent to PuppetDB from before this fix was in place. After enough time, new catalogs of the appropriate format should overwrite problematic catalogs in the command queue. If all your agents have checked in and your command queue depth is 0 then you should clear the command queue with the following instructions.

service pe-puppetdb stop; 
mv /opt/puppetlabs/server/data/puppetdb/stockpile /tmp
service pe-puppetdb start;

Generated at Mon Sep 28 16:04:09 PDT 2020 using Jira 8.5.2#805002-sha1:a66f9354b9e12ac788984e5d84669c903a370049.