[PDB-2034] schema error on replace facts insert-facts-pv-pairs! Created: 2015/10/06  Updated: 2015/10/29  Resolved: 2015/10/19

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: PDB 2.3.7, PDB 3.1.0
Fix Version/s: PDB 3.2.0

Type: Bug Priority: Normal
Reporter: Wyatt Alt Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Story Points: 3
Sprint: PuppetDB 2015-10-21
QA Contact: Kurt Wall

 Description   

There's a user in IRC whose replace-facts commands are failing with
https://gist.github.com/Tombar/31c5ddae25f99bf33294

note the [nil 6887]

I still haven't figured out how this is happening, but it seems like realize-paths! must be returning a nil.



 Comments   
Comment by Martin Loy [ 2015/10/06 ]

Hello wyatt,

I keep digging until i finally find out which was the offending facter, from the facter code and output it looks like the facter itself is totally wrong, I hope this help you troubleshoot the issue at your side.

root@deployer:/etc/facter/facts.d# cat /etc/facter/facts.d/rundeck_facts.rb
#!/usr/bin/env ruby
# Fact: rundeck_version
#
# Purpose: Return facts for the running version of rundeck.
#
 
if File.exists?('/etc/debian_version')
  version = %["dpkg-query -W -f 'rundeck_version=${Version}\n' rundeck"]
  puts version
elsif File.exists?('/etc/redhat-release') or File.exists?('/etc/centos-release')
  version = %["rpm -q --qf 'rundeck_version=%{VERSION}\n' rundeck"]
  puts version

the facter output:

root@deployer:/etc/facter/facts.d# ./rundeck_facts.rb
"dpkg-query -W -f 'rundeck_version=${Version}
' rundeck"

Comment by Wyatt Alt [ 2015/10/08 ]

Martin Loy

Thanks for tracking that down – I was not thinking that the problem would turn out to be with a specific fact, but that seems to be the case. PuppetDB seems to fail to process facts with names that begin with an escaped quote, which yours does. I can reproduce the error with this custom fact:

Facter.add("\"foo") do setcode do "hello world" end end

I'm about to start looking into what's causing the problem (and I'll update the ticket with that info), but I think your issues will go away if that fact is fixed.

Wyatt

Comment by Martin Loy [ 2015/10/08 ]

Wyatt Alt i've solve my issue by removing the fact from the catalog and already report the bug to the fine folks of puppet-community https://github.com/puppet-community/puppet-rundeck/issues/132

Hope my troubles end up helping you make a better puppetdb!

Regards

M

Comment by Josh Behrends [ 2015/10/09 ]

The exact same issue effected us too! Except for us this issue ended up blocking all puppet runs to both of our puppet servers! This is should be marked as a critical issue.

We're using puppetserver v2.1.1 and puppetdb v3.1.0

Before finding this ticket here is what my troubleshooting found:
It started with my puppetdb machine running out of disk space, and dying. The culprit was the KahaDB folder which was full of .log files.. So I removed the contents of "/opt/puppetlabs/server/data/puppetdb/mq/localhost/" and started puppetdb.
Everything was fine for about 24hrs, and then it happened again. So I adjusted down the store-usage and temp-usage down to 10gb/5gb, removed the KahaDB folders and started everything back up again.

Then I got user complaints that puppet wasn't running. I found that puppetdb was up and logging nothing. Puppetserver was logging reqeusts for catalogs from clients, and then just doing nothing..

Poked around in the JMX interface for puppetdb and found the activemq broker's storage usage percent was at 100% used. There was also about 30 messages in the DLQ. I did a purge operation via JMX on the DLQ and all of a sudden puppetdb started logging entries, and puppetserver started working once again.

Once I found the node that had this SAME fact referenced above I did a test puppet run. I found that every time puppet ran on that host puppetserver would submit the facts to puppetdb and then puppetdb would log the same stack trace described in this ticket and drop a message in the DLQ. At this point the internal activemq store usage would start to climb until it hit it's limit which wedged puppetdb, and also seemed to wedge BOTH of my puppetservers.

Hope this brain dump helps fix this issue.

Comment by Wyatt Alt [ 2015/10/11 ]

Josh Behrends yes, thanks for chiming in.

EDIT: I missed that you are apparently seeing this with the same fact Martin mentioned above. If I understand that right, is there a specific module this is coming from?

EDIT2: Sorry, I'm behind here. I see that you've issued a PR against puppet-community that fixes the issue on that end. I assume that means you're out of the woods for now?

Original message:
To bring you up to speed with the status of this, I've found four cases so far where you'll encounter this issue, each with a distinct cause but apparently similar:

  • an escape-quoted fact name
  • a fact name that begins with an escaped quote (this case spurred the ticket)
  • a fact name containing the string "#~" somewhere other than the end
  • a fact name ending with the string "#~"

I've got a fix that covers the first three cases, and I'm hoping to nail the fourth down tomorrow. If you'd be willing to send me one of the failed replace-facts commands in your dead letter directory (/opt/puppetlabs/server/data/puppetdb/mq/localhost/discarded/replace-facts) I can either confirm that it's a known case, or identify another cause.

My email is wyatt@puppetlabs.com – if you're willing, I'd recommend sending it there rather than attaching it to this public ticket.

Wyatt

Comment by Josh Behrends [ 2015/10/11 ]

Heh, yup we are now in a good state now. Just commented here to make sure you guys knew the symptoms I found, and that this was happening in puppetdb v3.1

Comment by Wyatt Alt [ 2015/10/11 ]

Ok good to hear, and thanks again. We'll get the fix in in the next few days and put it out in the next release.

Generated at Mon Sep 23 08:43:56 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.