[HI-569] Hiera filecache class should not store nil when an exception is thrown Created: 2017/04/17  Updated: 2017/05/06

Status: Open
Project: Hiera
Component/s: None
Affects Version/s: HI 3.2.2
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: Thomas Kishel Assignee: Unassigned
Resolution: Unresolved Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Team: Puppet Developer Experience
QA Risk Assessment: Needs Assessment

 Description   

The error is caused by:

https://github.com/puppetlabs/hiera/blob/3.2.2/lib/hiera/backend/yaml_backend.rb#L22

Steps to reproduce:

1. Install PE 2016.4.3 with a simple hiera_include('classes') inside of node default in site.pp.
2. Configure common.yaml with classes: []
3. Change permissions on common.yaml from 644 to 640.
4. Perform two puppet agent runs.
5. Notice the cryptic error message on the second run.

Expected behavior:

The helpful error message returned in the first agent run should always be returned.

Actual behavior:

The helpful error message is only returned once and is buried by subsequent agent reports. The cryptic error message is returned on all subsequent agent runs and makes troubleshooting extremely time consuming and costly.

Potential fix:

The filecache class should not store nil when an exception is thrown. Doing so results in the cryptic error message on all subsequent runs.

I suspect the root cause of the bug is here: https://github.com/puppetlabs/hiera/blob/3.2.2/lib/hiera/filecache.rb#L53

There is no handler for when when the passed block throws an exception which results in

@cache[path][:data] 

being set to nil.

Storing nil causes the cryptic error message on all subsequent runs.

[root@abc123 ~]# puppet agent -t --no-noop 
Info: Using configured environment 'test' 
Info: Retrieving pluginfacts 
Info: Retrieving plugin 
Info: Loading facts 
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Permission denied - /opt/puppet-data/hiera/common.yaml at /etc/puppetlabs/code/environments/test/manifests/site.pp on node abc123.domain.com 
Warning: Not using cache on failed catalog 
Error: Could not retrieve catalog; skipping run
 
[root@abc123 ~]# puppet agent -t --no-noop 
Info: Using configured environment 'test' 
Info: Retrieving pluginfacts 
Info: Retrieving plugin 
Info: Loading facts 
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, undefined method `empty?' for nil:NilClass at /etc/puppetlabs/code/environments/SBX/manifests/site.pp on node abc123.domain.com 
Warning: Not using cache on failed catalog 
Error: Could not retrieve catalog; skipping run 



 Comments   
Comment by Henrik Lindberg [ 2017/04/22 ]

Ping Thomas Hallgren

Comment by Thomas Hallgren [ 2017/04/24 ]

A stack trace from the server log would be very helpful to isolate this problem.

If there's a chance to try an upgrade to PE 2017.1.x/Puppet Agent 1.9.x, then that might solve the problem. Especially if the suspicion is that it stems from the Hiera Filecache class (it's no longer in use).

Comment by Thomas Kishel [ 2017/04/24 ]

Jeff has moved on, but asked for this to be filed.
When reproducing, I used 000, as 640 wasn't restrictive enough.

[root@pe-201643-master production]# chmod 000 hieradata/common.yaml
 
[root@pe-201643-master production]# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Permission denied - /etc/puppetlabs/code/environments/production/hieradata/common.yaml at /etc/puppetlabs/code/environments/production/manifests/site.pp:5:3 on node pe-201643-master.puppetdebug.vlan
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
 
[root@pe-201643-master production]# puppet agent -t
Info: Using configured environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item classes in any Hiera data file and no default supplied at /etc/puppetlabs/code/environments/production/manifests/site.pp:5:3 on node pe-201643-master.puppetdebug.vlan
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
 
[root@pe-201643-master production]# cat /var/log/puppetlabs/puppetserver/puppetserver.log 
2017-04-24 22:30:27,889 INFO  [main] [o.e.j.u.log] Logging initialized @22803ms
2017-04-24 22:30:35,091 INFO  [async-dispatch-2] [p.t.s.w.jetty9-service] Initializing web server(s).
2017-04-24 22:30:35,212 INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for pe-puppet-profiler service
2017-04-24 22:30:35,214 INFO  [async-dispatch-2] [p.s.j.jruby-puppet-service] Initializing the JRuby service
2017-04-24 22:30:35,240 INFO  [async-dispatch-2] [p.s.j.jruby-pool-manager-service] Initializing the JRuby service
2017-04-24 22:30:35,299 INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for pe-jruby-metrics service
2017-04-24 22:30:35,300 INFO  [async-dispatch-2] [p.s.v.versioned-code-service] No code-id-command set for versioned-code-service. Code-id will be nil.
2017-04-24 22:30:35,300 INFO  [async-dispatch-2] [p.s.v.versioned-code-service] No code-content-command set for versioned-code-service. Attempting to fetch code content will fail.
2017-04-24 22:30:35,307 INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-internal] Creating JRubyInstance with id 1.
2017-04-24 22:30:56,255 WARN  [clojure-agent-send-pool-0] [puppetserver] Puppet Setting 'always_cache_features' is deprecated and has been replaced by 'always_retry_plugins'.(at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/defaults.rb:319:in `handle')
2017-04-24 22:30:56,293 INFO  [clojure-agent-send-pool-0] [puppetserver] Puppet Puppet settings initialized; run mode: master
2017-04-24 22:30:58,050 INFO  [clojure-agent-send-pool-0] [p.s.j.i.jruby-agents] Finished creating JRubyInstance 1 of 1
2017-04-24 22:30:58,062 INFO  [async-dispatch-2] [p.s.c.puppet-server-config-core] Not overriding webserver settings with values from core Puppet
2017-04-24 22:30:58,124 INFO  [async-dispatch-2] [p.p.certificate-authority] CA already initialized for SSL
2017-04-24 22:30:58,143 INFO  [async-dispatch-2] [p.s.c.certificate-authority-service] CA Service adding a ring handler
2017-04-24 22:30:58,374 INFO  [async-dispatch-2] [p.p.certificate-authority] Master already initialized for SSL
2017-04-24 22:30:58,374 INFO  [async-dispatch-2] [p.e.s.m.master-service] Master Service adding a ring handler
2017-04-24 22:30:58,375 INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for pe-master service
2017-04-24 22:30:58,453 WARN  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Empty contextPath
2017-04-24 22:30:58,488 INFO  [async-dispatch-2] [p.s.p.puppet-admin-service] Starting Puppet Admin web app
2017-04-24 22:30:58,492 INFO  [async-dispatch-2] [p.t.s.w.jetty9-service] Starting web server(s).
2017-04-24 22:30:58,498 INFO  [pool-2-thread-1] [p.e.s.a.analytics-core] Not checking for updates - opt out file /etc/puppetlabs/puppetserver/opt-out exists.
2017-04-24 22:30:58,498 INFO  [pool-2-thread-1] [p.e.s.a.analytics-core] Not sending analytics data - (reason: /etc/puppetlabs/analytics-opt-out exists).
2017-04-24 22:30:58,810 INFO  [async-dispatch-2] [p.t.s.w.jetty9-core] Starting web server.
2017-04-24 22:30:58,828 INFO  [async-dispatch-2] [o.e.j.s.Server] jetty-9.2.z-SNAPSHOT
2017-04-24 22:30:58,908 INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.ServletContextHandler@18b5ea96{/packages,file:/opt/puppetlabs/server/data/packages/public/,AVAILABLE}
2017-04-24 22:30:58,984 INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@623419ad{/puppet-ca,null,AVAILABLE}
2017-04-24 22:30:58,984 INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@42e52bfe{/puppet,null,AVAILABLE}
2017-04-24 22:30:58,984 INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@23ab7311{/,null,AVAILABLE}
2017-04-24 22:30:58,985 INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@6110efb8{/puppet-admin-api,null,AVAILABLE}
2017-04-24 22:30:59,137 INFO  [async-dispatch-2] [o.e.j.s.ServerConnector] Started ServerConnector@1118d779{SSL-HTTP/1.1}{0.0.0.0:8140}
2017-04-24 22:30:59,138 INFO  [async-dispatch-2] [o.e.j.s.Server] Started @54053ms
2017-04-24 22:30:59,140 INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status callback function for status-service service
2017-04-24 22:30:59,140 INFO  [async-dispatch-2] [p.t.s.s.status-service] Registering status service HTTP API at /status
2017-04-24 22:30:59,176 INFO  [async-dispatch-2] [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@1207f4bc{/status,null,AVAILABLE}
2017-04-24 22:30:59,258 INFO  [async-dispatch-2] [p.e.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
2017-04-24 22:30:59,259 INFO  [async-dispatch-2] [p.e.s.l.pe-legacy-routes-service] The legacy routing service has successfully started and is now ready to handle requests
2017-04-24 22:30:59,297 INFO  [async-dispatch-2] [p.e.s.a.analytics-service] Puppet Server Analytics has successfully started and will run in the background
2017-04-24 22:35:44,765 INFO  [qtp20587571-64] [puppetserver] Puppet Caching node for pe-201643-master.puppetdebug.vlan
2017-04-24 22:35:45,341 INFO  [qtp20587571-63] [puppetserver] mount[pe_packages] allowing * access
2017-04-24 22:35:45,366 INFO  [qtp20587571-63] [puppetserver] mount[pe_modules] allowing * access
2017-04-24 22:35:53,669 INFO  [qtp20587571-66] [puppetserver] Puppet 'replace_facts' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID a7d55cce-81e8-47dd-9ee5-fd428d9832ad
2017-04-24 22:35:54,486 INFO  [qtp20587571-66] [puppetserver] Puppet Caching node for pe-201643-master.puppetdebug.vlan
2017-04-24 22:36:12,720 INFO  [qtp20587571-66] [puppetserver] Puppet Compiled catalog for pe-201643-master.puppetdebug.vlan in environment production in 18.02 seconds
2017-04-24 22:36:12,747 INFO  [qtp20587571-66] [puppetserver] Puppet Caching catalog for pe-201643-master.puppetdebug.vlan
2017-04-24 22:36:14,611 INFO  [qtp20587571-66] [puppetserver] Puppet 'replace_catalog' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID bd5955e1-0a4f-469f-add6-840ec4bf0bfa
2017-04-24 22:36:45,429 INFO  [qtp20587571-62] [puppetserver] Puppet 'store_report' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID 843ae00f-c6df-45c4-bef8-3bc30afdcb18
2017-04-24 22:36:51,896 INFO  [qtp20587571-65] [puppetserver] Puppet Caching node for pe-201643-master.puppetdebug.vlan
2017-04-24 22:36:53,522 INFO  [qtp20587571-64] [puppetserver] Puppet 'replace_facts' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID 62cee680-d654-4602-901a-b761f09b3fcf
2017-04-24 22:36:54,248 INFO  [qtp20587571-64] [puppetserver] Puppet Caching node for pe-201643-master.puppetdebug.vlan
2017-04-24 22:37:03,624 INFO  [qtp20587571-64] [puppetserver] Puppet Compiled catalog for pe-201643-master.puppetdebug.vlan in environment production in 9.19 seconds
2017-04-24 22:37:03,624 INFO  [qtp20587571-64] [puppetserver] Puppet Caching catalog for pe-201643-master.puppetdebug.vlan
2017-04-24 22:37:05,232 INFO  [qtp20587571-64] [puppetserver] Puppet 'replace_catalog' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID 1e236d40-1d88-4bd7-a003-00dee950854f
2017-04-24 22:37:21,838 INFO  [qtp20587571-62] [puppetserver] Puppet 'store_report' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID 0e48bf7c-817b-4374-bc96-3cb132991ded
2017-04-24 22:37:55,410 INFO  [qtp20587571-65] [puppetserver] Puppet Caching node for pe-201643-master.puppetdebug.vlan
2017-04-24 22:37:56,390 INFO  [qtp20587571-64] [puppetserver] Puppet 'replace_facts' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID d9ee4526-7950-44d0-bba3-132644d182b5
2017-04-24 22:37:57,056 INFO  [qtp20587571-64] [puppetserver] Puppet Caching node for pe-201643-master.puppetdebug.vlan
2017-04-24 22:37:57,675 ERROR [qtp20587571-64] [puppetserver] Puppet Evaluation Error: Error while evaluating a Function Call, Permission denied - /etc/puppetlabs/code/environments/production/hieradata/common.yaml at /etc/puppetlabs/code/environments/production/manifests/site.pp:5:3 on node pe-201643-master.puppetdebug.vlan
org/jruby/RubyIO.java:3804:in `read'
org/jruby/RubyIO.java:3987:in `read'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/filecache.rb:52:in `read_file'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/yaml_backend.rb:18:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:124:in `datasourcefiles'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:95:in `datasources'
org/jruby/RubyArray.java:2414:in `map'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:90:in `datasources'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:119:in `datasourcefiles'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/yaml_backend.rb:17:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:274:in `lookup'
org/jruby/RubyKernel.java:1242:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:272:in `lookup'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:267:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera.rb:116:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera_puppet.rb:11:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/puppet_function.rb:72:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/puppet_function.rb:46:in `hiera_no_default'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:45:in `invoke'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:35:in `dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:44:in `call'
/etc/puppetlabs/code/environments/production/manifests/site.pp:5:in `call_function'
org/jruby/RubyKernel.java:1079:in `eval'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:276:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:274:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:917:in `call_function_with_block'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:886:in `eval_CallNamedFunctionExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:622:in `eval_BlockExpression'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:622:in `eval_BlockExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:182:in `evaluate_code'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:555:in `evaluate_ast_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:178:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:178:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:162:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:33:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:266:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:264:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/benchmark.rb:295:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:222:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:262:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:53:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:121:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'
Puppet$$Server$$Master_134244152.gen:13:in `handleRequest'
request_handler_core.clj:273:in `invoke'
jruby_request.clj:46:in `invoke'
jruby_request.clj:31:in `invoke'
request_handler_service.clj:34:in `handle_request'
request_handler.clj:3:in `invoke'
request_handler.clj:3:in `invoke'
core.clj:2493:in `invoke'
ring_middleware.clj:284:in `invoke'
core.clj:168:in `invoke'
core.clj:211:in `invoke'
core.clj:45:in `invoke'
core.clj:343:in `invoke'
core.clj:51:in `invoke'
ringutils.clj:86:in `invoke'
master_core.clj:428:in `invoke'
ring.clj:21:in `invoke'
ring.clj:12:in `invoke'
comidi.clj:249:in `invoke'
http.clj:152:in `invoke'
http.clj:152:in `invoke'
http.clj:148:in `invoke'
comidi.clj:336:in `invoke'
jetty9_core.clj:424:in `invoke'
normalized_uri_helpers.clj:80:in `invoke'
2017-04-24 22:37:57,675 ERROR [qtp20587571-64] [puppetserver] Puppet Evaluation Error: Error while evaluating a Function Call, Permission denied - /etc/puppetlabs/code/environments/production/hieradata/common.yaml at /etc/puppetlabs/code/environments/production/manifests/site.pp:5:3 on node pe-201643-master.puppetdebug.vlan
2017-04-24 22:37:57,676 ERROR [qtp20587571-64] [puppetserver] Puppet Server Error: Evaluation Error: Error while evaluating a Function Call, Permission denied - /etc/puppetlabs/code/environments/production/hieradata/common.yaml at /etc/puppetlabs/code/environments/production/manifests/site.pp:5:3 on node pe-201643-master.puppetdebug.vlan
org/jruby/RubyIO.java:3804:in `read'
org/jruby/RubyIO.java:3987:in `read'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/filecache.rb:52:in `read_file'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/yaml_backend.rb:18:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:124:in `datasourcefiles'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:95:in `datasources'
org/jruby/RubyArray.java:2414:in `map'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:90:in `datasources'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:119:in `datasourcefiles'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend/yaml_backend.rb:17:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:274:in `lookup'
org/jruby/RubyKernel.java:1242:in `catch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:272:in `lookup'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/backend.rb:267:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera.rb:116:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera_puppet.rb:11:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/puppet_function.rb:72:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/puppet_function.rb:46:in `hiera_no_default'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:45:in `invoke'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:35:in `dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:44:in `call'
/etc/puppetlabs/code/environments/production/manifests/site.pp:5:in `call_function'
org/jruby/RubyKernel.java:1079:in `eval'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:276:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:274:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:917:in `call_function_with_block'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:886:in `eval_CallNamedFunctionExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:622:in `eval_BlockExpression'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:622:in `eval_BlockExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:182:in `evaluate_code'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:555:in `evaluate_ast_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:178:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:178:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:162:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:33:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:266:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:264:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/benchmark.rb:295:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:222:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:262:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:53:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:121:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'
Puppet$$Server$$Master_134244152.gen:13:in `handleRequest'
request_handler_core.clj:273:in `invoke'
jruby_request.clj:46:in `invoke'
jruby_request.clj:31:in `invoke'
request_handler_service.clj:34:in `handle_request'
request_handler.clj:3:in `invoke'
request_handler.clj:3:in `invoke'
core.clj:2493:in `invoke'
ring_middleware.clj:284:in `invoke'
core.clj:168:in `invoke'
core.clj:211:in `invoke'
core.clj:45:in `invoke'
core.clj:343:in `invoke'
core.clj:51:in `invoke'
ringutils.clj:86:in `invoke'
master_core.clj:428:in `invoke'
ring.clj:21:in `invoke'
ring.clj:12:in `invoke'
comidi.clj:249:in `invoke'
http.clj:152:in `invoke'
http.clj:152:in `invoke'
http.clj:148:in `invoke'
comidi.clj:336:in `invoke'
jetty9_core.clj:424:in `invoke'
normalized_uri_helpers.clj:80:in `invoke'
2017-04-24 22:37:57,912 INFO  [qtp20587571-63] [puppetserver] Puppet 'store_report' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID 7d067901-c3c2-4c9d-ae6e-1d82d8c4f4a0
2017-04-24 22:38:42,382 INFO  [qtp20587571-62] [puppetserver] Puppet Caching node for pe-201643-master.puppetdebug.vlan
2017-04-24 22:38:43,179 INFO  [qtp20587571-67] [puppetserver] Puppet 'replace_facts' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID df1a3b55-a3c0-45b2-ae98-12001fe66678
2017-04-24 22:38:43,636 INFO  [qtp20587571-67] [puppetserver] Puppet Caching node for pe-201643-master.puppetdebug.vlan
2017-04-24 22:38:44,075 ERROR [qtp20587571-67] [puppetserver] Puppet Evaluation Error: Error while evaluating a Function Call, Could not find data item classes in any Hiera data file and no default supplied at /etc/puppetlabs/code/environments/production/manifests/site.pp:5:3 on node pe-201643-master.puppetdebug.vlan
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera_puppet.rb:14:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/puppet_function.rb:72:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/puppet_function.rb:46:in `hiera_no_default'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:45:in `invoke'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:35:in `dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:44:in `call'
/etc/puppetlabs/code/environments/production/manifests/site.pp:5:in `call_function'
org/jruby/RubyKernel.java:1079:in `eval'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:276:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:274:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:917:in `call_function_with_block'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:886:in `eval_CallNamedFunctionExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:622:in `eval_BlockExpression'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:622:in `eval_BlockExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:182:in `evaluate_code'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:555:in `evaluate_ast_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:178:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:178:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:162:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:33:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:266:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:264:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/benchmark.rb:295:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:222:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:262:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:53:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:121:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'
Puppet$$Server$$Master_134244152.gen:13:in `handleRequest'
request_handler_core.clj:273:in `invoke'
jruby_request.clj:46:in `invoke'
jruby_request.clj:31:in `invoke'
request_handler_service.clj:34:in `handle_request'
request_handler.clj:3:in `invoke'
request_handler.clj:3:in `invoke'
core.clj:2493:in `invoke'
ring_middleware.clj:284:in `invoke'
core.clj:168:in `invoke'
core.clj:211:in `invoke'
core.clj:45:in `invoke'
core.clj:343:in `invoke'
core.clj:51:in `invoke'
ringutils.clj:86:in `invoke'
master_core.clj:428:in `invoke'
ring.clj:21:in `invoke'
ring.clj:12:in `invoke'
comidi.clj:249:in `invoke'
http.clj:152:in `invoke'
http.clj:152:in `invoke'
http.clj:148:in `invoke'
comidi.clj:336:in `invoke'
jetty9_core.clj:424:in `invoke'
normalized_uri_helpers.clj:80:in `invoke'
2017-04-24 22:38:44,075 ERROR [qtp20587571-67] [puppetserver] Puppet Evaluation Error: Error while evaluating a Function Call, Could not find data item classes in any Hiera data file and no default supplied at /etc/puppetlabs/code/environments/production/manifests/site.pp:5:3 on node pe-201643-master.puppetdebug.vlan
2017-04-24 22:38:44,076 ERROR [qtp20587571-67] [puppetserver] Puppet Server Error: Evaluation Error: Error while evaluating a Function Call, Could not find data item classes in any Hiera data file and no default supplied at /etc/puppetlabs/code/environments/production/manifests/site.pp:5:3 on node pe-201643-master.puppetdebug.vlan
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera_puppet.rb:14:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/puppet_function.rb:72:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/hiera/puppet_function.rb:46:in `hiera_no_default'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatch.rb:45:in `invoke'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/dispatcher.rb:35:in `dispatch'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/functions/function.rb:44:in `call'
/etc/puppetlabs/code/environments/production/manifests/site.pp:5:in `call_function'
org/jruby/RubyKernel.java:1079:in `eval'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:276:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:274:in `call_function'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:917:in `call_function_with_block'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:886:in `eval_CallNamedFunctionExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:622:in `eval_BlockExpression'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:622:in `eval_BlockExpression'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:69:in `visit_this_1'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:32:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:182:in `evaluate_code'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:555:in `evaluate_ast_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:178:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:178:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:162:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:33:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:266:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:264:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/META-INF/jruby.home/lib/ruby/1.9/benchmark.rb:295:in `realtime'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:222:in `benchmark'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:262:in `compile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:53:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:121:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:48:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:241:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:47:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `process'
org/jruby/RubyArray.java:1613:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:87:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:60:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:58:in `process'
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/master.rb:42:in `handleRequest'
Puppet$$Server$$Master_134244152.gen:13:in `handleRequest'
request_handler_core.clj:273:in `invoke'
jruby_request.clj:46:in `invoke'
jruby_request.clj:31:in `invoke'
request_handler_service.clj:34:in `handle_request'
request_handler.clj:3:in `invoke'
request_handler.clj:3:in `invoke'
core.clj:2493:in `invoke'
ring_middleware.clj:284:in `invoke'
core.clj:168:in `invoke'
core.clj:211:in `invoke'
core.clj:45:in `invoke'
core.clj:343:in `invoke'
core.clj:51:in `invoke'
ringutils.clj:86:in `invoke'
master_core.clj:428:in `invoke'
ring.clj:21:in `invoke'
ring.clj:12:in `invoke'
comidi.clj:249:in `invoke'
http.clj:152:in `invoke'
http.clj:152:in `invoke'
http.clj:148:in `invoke'
comidi.clj:336:in `invoke'
jetty9_core.clj:424:in `invoke'
normalized_uri_helpers.clj:80:in `invoke'
2017-04-24 22:38:44,241 INFO  [qtp20587571-65] [puppetserver] Puppet 'store_report' command for pe-201643-master.puppetdebug.vlan submitted to PuppetDB with UUID 95d0b35d-3ea9-4024-a89b-ca5c8df98a51

Generated at Sat Feb 22 06:15:36 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.