Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-8467

Starting to get broken pipe (HTTP 500) on certain manifest - gone after restarting puppetserver

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: PUP 4.10.11, PUP 5.3.6, PUP 5.5.0
    • Component/s: None
    • Labels:
      None
    • Template:
      PUP Bug Template
    • Team:
      Froyo
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Warnings from internal poor coding practices should no longer be logged to the process' stderr (resolving an issue where log managers could cause a broken pipe).
    • QA Risk Assessment:
      Needs Assessment

      Description

      After a while a certain manifest started firing HTTP 500 on the puppetserver side, though nothing had changed.

      clients were aborting catalog compilation with:

      Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Method call, Broken pipe - <STDERR> at /srv/puppet/code/environments/production/modules/amodule/manifests/init.pp:16:35 on node node1.example.com
      

      Server logs were showing the following:

      2017-12-08 18:29:02,277 ERROR [qtp526724218-38738] [puppetserver] Puppet Evaluation Error: Error while evaluating a Method call, Broken pipe - <STDERR> at /srv/puppet/code/environments/production/modules/amodule/manifests/init.pp:16:35 on node node1.example.com
      org/jruby/RubyIO.java:1457:in `write'
      org/jruby/RubyIO.java:1428:in `write'
      org/jruby/RubyKernel.java:1000:in `eval'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/ruby_function_instantiator.rb:22:in `create'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/module_loaders.rb:166:in `find'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:151:in `internal_load'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:32:in `load_typed'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:143:in `internal_load'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:32:in `load_typed'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:143:in `internal_load'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:32:in `load_typed'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:143:in `internal_load'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:32:in `load_typed'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:143:in `internal_load'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/base_loader.rb:32:in `load_typed'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/loader.rb:53:in `load'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:304:in `call_function'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:951:in `call_function_with_block'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:944:in `eval_CallMethodExpression'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:90:in `visit_this_1'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:81:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:358:in `eval_AssignmentExpression'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:90:in `visit_this_1'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:81:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:646:in `block in eval_BlockExpression'
      org/jruby/RubyArray.java:1734:in `each'
      org/jruby/RubyEnumerable.java:936:in `inject'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:646:in `eval_BlockExpression'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:90:in `visit_this_1'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:81:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:72:in `block in evaluate'
      org/jruby/RubyKernel.java:1120:in `catch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:71:in `block in evaluate'
      org/jruby/RubyKernel.java:1120:in `catch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:70:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:136:in `evaluate_code'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:79:in `block in evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:71:in `evaluate'
      org/jruby/RubyArray.java:1734:in `each'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:384:in `evaluate_classes'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/include.rb:16:in `include'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:60:in `invoke'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:43:in `block in dispatch'
      org/jruby/RubyKernel.java:1120:in `catch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:42:in `dispatch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:46:in `block in call'
      org/jruby/RubyKernel.java:1120:in `catch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `call'
      /srv/puppet/code/environments/production/modules/site/configsets/manifests/types/immerx_10.pp:6:in `<eval>'
      org/jruby/RubyKernel.java:1000:in `eval'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:307:in `block in call_function'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:305:in `call_function'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:949:in `call_function_with_block'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:918:in `eval_CallNamedFunctionExpression'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:90:in `visit_this_1'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:81:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:646:in `block in eval_BlockExpression'
      org/jruby/RubyArray.java:1734:in `each'
      org/jruby/RubyEnumerable.java:936:in `inject'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:646:in `eval_BlockExpression'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:90:in `visit_this_1'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:81:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:72:in `block in evaluate'
      org/jruby/RubyKernel.java:1120:in `catch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:71:in `block in evaluate'
      org/jruby/RubyKernel.java:1120:in `catch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:70:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:136:in `evaluate_code'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:79:in `block in evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:71:in `evaluate'
      org/jruby/RubyArray.java:1734:in `each'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:384:in `evaluate_classes'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/loader/../../../puppet/functions/include.rb:16:in `include'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:60:in `invoke'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:43:in `block in dispatch'
      org/jruby/RubyKernel.java:1120:in `catch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:42:in `dispatch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:46:in `block in call'
      org/jruby/RubyKernel.java:1120:in `catch'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:45:in `call'
      /srv/puppet/code/environments/production/manifests/site.pp:121:in `<eval>'
      org/jruby/RubyKernel.java:1000:in `eval'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:307:in `block in call_function'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:305:in `call_function'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:949:in `call_function_with_block'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:918:in `eval_CallNamedFunctionExpression'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:90:in `visit_this_1'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:81:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:646:in `block in eval_BlockExpression'
      org/jruby/RubyArray.java:1734:in `each'
      org/jruby/RubyEnumerable.java:936:in `inject'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:646:in `eval_BlockExpression'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:90:in `visit_this_1'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:81:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:133:in `block in evaluate_code'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/scope.rb:953:in `with_guarded_scope'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:131:in `evaluate_code'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:79:in `block in evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:71:in `evaluate'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:505:in `evaluate_ast_node'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:168:in `block in compile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:168:in `block in compile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:153:in `compile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:33:in `compile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:297:in `block in compile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:295:in `block in compile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:224:in `block in benchmark'
      uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/benchmark.rb:308:in `realtime'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:293:in `compile'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:195:in `find'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:121:in `do_find'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `block in call'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:263:in `override'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
      

      Restarting the puppetserver made the problem go away.

      All hosts using that module were affected.

      The offending line in the manifest looks like:

         $dump_sources = $hosts.map |$h| { "https://${acustomfcuntion('somevalue',$h)}.example.com" }
      

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              duritong Peter Meier
              Reporter:
              duritong Peter Meier
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support