[PDB-2256] Regression/Performance Issue introduced by PDB-135 Created: 2015/12/13  Updated: 2016/01/11  Resolved: 2015/12/21

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: PDB 3.2.0, PDB 3.2.1, PDB 3.2.2
Fix Version/s: PDB 3.2.3

Type: Bug Priority: Normal
Reporter: Olivier Hanesse Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Story Points: 3
Sprint: PuppetDB 2015-12-30

 Description   

Hello,

I got a regression/performance issue using puppetdb-termini, version > 3.2

PR #1640 has introduced a warning message when using invalid UTF-8 sequence.

The issue is located here :

https://github.com/puppetlabs/puppetdb/blob/master/puppet/lib/puppet/util/puppetdb/char_encoding.rb#L38

This function just takes forever to complete when using a "big" catalog.

Step to reproduce :

It's pretty easy, you just need to trigger this function (so with a catalog with some non UTF-8 caracters), and a "big" catalog. For example :

 
node default{
 
$loops = range("0", "500")
each($loops) |$iteration| {
  file {"/tmp/test-$iteration":
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    content => template('test/test.erb'),
  }
}
 
}
 
with test.erb, including some non UTF-8 caracters like a Kerberos Keytab file : 
 
<%data = File.open('/etc/krb5.keytab').read-%><%= data -%>

On my computer, performance of this function is :

With range 10 : 0s
With range 100 : 8s
With range 500 : 3min5s
With range > 10000, still waiting to complete

As you can see, this isn't linear. Catalog compilation isn't affected by this (always <1s).

On a production environment, this bug leads to a puppetserver having all workers 'blocked' on this function, with a 100% cpu consumption.

I think this issue, is specific to Jruby. Using Ruby interpreter, I couldn't reproduce it.

Looking at some jruby issues, I've seen this one : https://github.com/jruby/jruby/issues/504
This could be linked, but I amn't sure.

Regards

Olivier



 Comments   
Comment by Ryan Senior [ 2015/12/14 ]

This is some great investigation Olivier Hanesse. We have a new iteration that starts on Wednesday, this will be included then (if not sooner). Thanks for all of the detail!

Chris Price - did you see anything like this with JRuby working on file sync?

Comment by Chris Price [ 2015/12/14 ]

Ryan Senior no, but I do think it would probably be worthwhile to write a simple reproducer, all in ruby, that can be run from the CLI to compare MRI/JRuby. We have seen some cases where e.g. the Puppet code lexer was slower in JRuby, and that's the easiest way to tease out where the issue might be.

Comment by Alexey Lapitsky [ 2015/12/17 ]

Hi,

Here's a PR with a fix and a benchmark: https://github.com/puppetlabs/puppetdb/pull/1789

Generated at Tue Jun 25 08:32:06 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.