[PDB-2003] facts GC can exceed jdbc max message size if too many paths are deleted Created: 2015/09/28  Updated: 2015/10/28  Resolved: 2015/10/06

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: PDB 2.3.8
Fix Version/s: PDB 2.3.8, PDB 3.2.0

Type: Bug Priority: Normal
Reporter: Wyatt Alt Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File maestro_factset     File replace-facts.tar.bz2    
Template:
Story Points: 3
Sprint: PuppetDB 2015-10-07
QA Contact: Kurt Wall

 Description   

This was reported on the puppet-users list here:
https://groups.google.com/forum/#!topic/puppet-users/o_4K-UcuTQQ

I don't know the specific context in which the user is encountering it, but I'm able to reproduce this issue by creating and storing a very large array-valued structured fact with map-valued elements, then updating the fact by inserting an element at the front of the array to force recomputation of all the paths. This problem was encountered on 2.3.7, but it affects master/3.x as well.

We'll need to coordinate with the user to figure out what's going on with his particular setup. If this relates to a large array-valued structured fact, reformulating the top-level structure as a map would likely fix the issue. Ultimately, if we want to support arbitrarily large structured facts we'll want to expose jdbc's max message size as a tunable. Alternatively we should document our limits.



 Comments   
Comment by Matt Jarvis [ 2015/09/30 ]

Can you give me answers to the following:

  • has PuppetDB been running fine prior to this issue or have you recently adopted it?

Yes, although we may not have noticed the error for a while. We've been running it in production for around 3 years.

  • does it seem possible that you have no structured facts in your database?

We're using puppet 3.7.4, and don't actually create that many custom facts, so it may be possible if they aren't created by default.

  • can you give me the first 10 rows of this query?
    select count,factset_id from facts group by factset_id order by count desc;

count | factset_id
------+-----------
13532 | 99
13324 | 72
417 | 153
345 | 130
341 | 29
329 | 160
327 | 163
319 | 170
318 | 41
316 | 151

  • can you get the certname of the failing node using
    select certname from factsets where id=<value of $26355>;
    and send me the output of
    curl -X GET http://localhost:8080/v4/factsets -d 'query=["=","certname","<your certname>"]'

Will attach

  • once you have the certname, is there anything special about that node that you're aware of?

Nothing majorly - it's a 64bit ARM compute node in OpenStack

  • can you send me the compressed contents of the failed replace-facts commands in your dead letter directory? These will be located at
    /opt/puppetlabs/server/data/puppetdb/mq/discarded/replace-facts
    if you're on PC1 and
    /var/lib/puppetdb/mq/discarded/replace-facts
    if you aren't, assuming you're using the default pathing.

Will also attach

Comment by Matt Jarvis [ 2015/09/30 ]

curl output attached

Comment by Matt Jarvis [ 2015/09/30 ]

Attached dead letter directory contents

Comment by Matt Jarvis [ 2015/10/01 ]

I noticed the status of this has changed - any clues ?

Comment by Wyatt Alt [ 2015/10/14 ]

Matt Jarvis Sorry, I missed your last comment somehow. We released a fix for this today in PDB 2.3.8.

Comment by Wyatt Alt [ 2015/10/14 ]

Matt Jarvis The root issue was that your facts update was creating a prepared statement with more parameters than the maximum 16 bit integer, which is the limit imposed by the Postgres JDBC driver. We've addressed this by handling these updates in batches when more than 6000 values are updated at once.

Generated at Mon Dec 16 05:35:42 PST 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.