[PDB-2034] schema error on replace facts insert-facts-pv-pairs! Created: 2015/10/06 Updated: 2015/10/29 Resolved: 2015/10/19
|Affects Version/s:||PDB 2.3.7, PDB 3.1.0|
|Fix Version/s:||PDB 3.2.0|
|Remaining Estimate:||Not Specified|
|Time Spent:||Not Specified|
|Original Estimate:||Not Specified|
There's a user in IRC whose replace-facts commands are failing with
note the [nil 6887]
I still haven't figured out how this is happening, but it seems like realize-paths! must be returning a nil.
|Comment by Martin Loy [ 2015/10/06 ]|
I keep digging until i finally find out which was the offending facter, from the facter code and output it looks like the facter itself is totally wrong, I hope this help you troubleshoot the issue at your side.
the facter output:
|Comment by Wyatt Alt [ 2015/10/08 ]|
Thanks for tracking that down – I was not thinking that the problem would turn out to be with a specific fact, but that seems to be the case. PuppetDB seems to fail to process facts with names that begin with an escaped quote, which yours does. I can reproduce the error with this custom fact:
I'm about to start looking into what's causing the problem (and I'll update the ticket with that info), but I think your issues will go away if that fact is fixed.
|Comment by Martin Loy [ 2015/10/08 ]|
Wyatt Alt i've solve my issue by removing the fact from the catalog and already report the bug to the fine folks of puppet-community https://github.com/puppet-community/puppet-rundeck/issues/132
Hope my troubles end up helping you make a better puppetdb!
|Comment by Josh Behrends [ 2015/10/09 ]|
The exact same issue effected us too! Except for us this issue ended up blocking all puppet runs to both of our puppet servers! This is should be marked as a critical issue.
We're using puppetserver v2.1.1 and puppetdb v3.1.0
Before finding this ticket here is what my troubleshooting found:
Then I got user complaints that puppet wasn't running. I found that puppetdb was up and logging nothing. Puppetserver was logging reqeusts for catalogs from clients, and then just doing nothing..
Poked around in the JMX interface for puppetdb and found the activemq broker's storage usage percent was at 100% used. There was also about 30 messages in the DLQ. I did a purge operation via JMX on the DLQ and all of a sudden puppetdb started logging entries, and puppetserver started working once again.
Once I found the node that had this SAME fact referenced above I did a test puppet run. I found that every time puppet ran on that host puppetserver would submit the facts to puppetdb and then puppetdb would log the same stack trace described in this ticket and drop a message in the DLQ. At this point the internal activemq store usage would start to climb until it hit it's limit which wedged puppetdb, and also seemed to wedge BOTH of my puppetservers.
Hope this brain dump helps fix this issue.
|Comment by Wyatt Alt [ 2015/10/11 ]|
Josh Behrends yes, thanks for chiming in.
EDIT: I missed that you are apparently seeing this with the same fact Martin mentioned above. If I understand that right, is there a specific module this is coming from?
EDIT2: Sorry, I'm behind here. I see that you've issued a PR against puppet-community that fixes the issue on that end. I assume that means you're out of the woods for now?
I've got a fix that covers the first three cases, and I'm hoping to nail the fourth down tomorrow. If you'd be willing to send me one of the failed replace-facts commands in your dead letter directory (/opt/puppetlabs/server/data/puppetdb/mq/localhost/discarded/replace-facts) I can either confirm that it's a known case, or identify another cause.
My email is email@example.com – if you're willing, I'd recommend sending it there rather than attaching it to this public ticket.
|Comment by Josh Behrends [ 2015/10/11 ]|
Heh, yup we are now in a good state now. Just commented here to make sure you guys knew the symptoms I found, and that this was happening in puppetdb v3.1
|Comment by Wyatt Alt [ 2015/10/11 ]|
Ok good to hear, and thanks again. We'll get the fix in in the next few days and put it out in the next release.