[PUP-9587] Serial execution with puppet device and ciscopuppet fails - Puppet Jira Server

XML

Word

Printable

Details

Type: Bug
Status: Closed
Priority: Major
Resolution: Fixed
Affects Version/s: PUP 6.0.5, PUP 6.0.7
Fix Version/s: PUP 5.5.16, PUP 6.0.8, PUP 6.4.1, PUP 6.4.3, PUP 6.5.0
Component/s: None
Labels:
- resolved-issue-added

Environment:

Hide

PE 2019.0.2

[root@h0wm6jin2afatag ~]# puppet module list

/etc/puppetlabs/code/environments/production/modules

├── puppetlabs-ciscopuppet (v2.0.0)

├── puppetlabs-concat (v5.3.0)

├── puppetlabs-device_manager (v2.7.0)

├── puppetlabs-hocon (v1.0.1)

├── puppetlabs-netdev_stdlib (v0.18.0)

├── puppetlabs-puppetserver_gem (v1.1.0)

├── puppetlabs-resource_api (v1.0.0)

├── puppetlabs-stdlib (v5.2.0)

└── puppetlabs-translate (v1.2.0)

/etc/puppetlabs/code/modules (no modules installed)

/opt/puppetlabs/puppet/modules

├── puppetlabs-facter_task (v0.4.0)

├── puppetlabs-package (v0.4.1)

├── puppetlabs-pe_accounts (v2016.5.0)

├── puppetlabs-pe_bootstrap (v0.3.0)

├── puppetlabs-pe_concat (v1.1.1)

├── puppetlabs-pe_hocon (v2019.0.0)

├── puppetlabs-pe_infrastructure (v2018.1.0)

├── puppetlabs-pe_inifile (v1.1.3)

├── puppetlabs-pe_install (v2018.1.0)

├── puppetlabs-pe_java_ks (v2016.4.0)

├── puppetlabs-pe_nginx (v2017.1.0)

├── puppetlabs-pe_postgresql (v2016.5.0)

├── puppetlabs-pe_puppet_authorization (v2016.2.0)

├── puppetlabs-pe_r10k (v2016.2.0)

├── puppetlabs-pe_razor (v1.1.0)

├── puppetlabs-pe_repo (v2018.1.0)

├── puppetlabs-pe_staging (v0.3.3)

├── puppetlabs-pe_support_script (v2.10.0)

├── puppetlabs-puppet_conf (v0.3.0)

├── puppetlabs-puppet_enterprise (v2018.1.0)

└── puppetlabs-service (v0.5.0)

node h0wm6jin2afatag.delivery.puppetlabs.net {

  include ciscopuppet::proxy

  device_manager { 'nexus1':

    type => 'cisco_nexus',

    credentials => {

      address => 'mvtc9ra62ieh96u.delivery.puppetlabs.net',

      username => 'admin',

      password => 'admin',

      transport => 'http',

  device_manager { 'nexus2':

    type => 'cisco_nexus',

    credentials => {

      address => 'fz33nfg6krsy1zt.delivery.puppetlabs.net',

      username => 'admin',

      password => 'admin',

      transport => 'http',

[root@h0wm6jin2afatag ~]# /opt/puppetlabs/puppet/bin/gem list

*** LOCAL GEMS ***

bigdecimal (default: 1.3.4)

cisco_node_utils (2.0.0)

cmath (default: 1.0.0)

colored (1.2)

cri (2.15.1)

csv (default: 1.0.0)

date (default: 1.0.0)

deep_merge (1.0.1)

did_you_mean (1.2.0)

etc (default: 1.0.0)

ethon (0.9.0)

facter (3.12.3, 2.5.1)

faraday (0.9.1)

faraday_middleware (0.9.1)

faraday_middleware-multi_json (0.0.6)

fast_gettext (1.1.2)

fcntl (default: 1.0.0)

ffi (1.9.10)

fiddle (default: 1.0.0)

fileutils (default: 1.0.2)

gettext (3.2.2)

gettext-setup (0.28)

google-protobuf (3.7.0 x86_64-linux)

googleapis-common-protos-types (1.0.3)

grpc (1.14.2 x86_64-linux)

hiera (3.4.6)

hiera-eyaml (2.1.0)

highline (1.6.21)

hocon (1.2.5)

httpclient (2.8.3)

io-console (default: 0.4.6)

ipaddr (default: 1.2.0)

json (default: 2.1.0)

locale (2.1.2)

log4r (1.1.10)

minitar (0.6.1)

minitest (5.10.3)

multi_json (1.13.1, 1.11.0)

multipart-post (2.0.0)

net-ssh (4.2.0)

net-telnet (0.1.1)

net_http_unix (0.2.2)

openssl (default: 2.1.2)

pe_backup_tools (1.0.2.11.gf6a3fe6)

power_assert (1.1.1)

psych (default: 3.0.2)

puppet (6.0.5)

puppet-resource_api (1.6.3)

puppet_forge (2.2.8)

puppetserver-ca (1.2.1)

r10k (3.0.3)

rake (12.3.0)

rdoc (default: 6.0.1)

rugged (0.25.1.1)

scanf (default: 1.0.0)

sdbm (default: 1.0.0)

semantic_puppet (1.0.2, 1.0.1)

stringio (default: 0.0.1)

strscan (default: 1.0.0)

test-unit (3.2.7)

text (1.3.1)

trollop (2.1.2)

typhoeus (1.0.2)

webrick (default: 1.4.2)

xmlrpc (0.3.0)

zlib (default: 1.0.0)

[root@h0wm6jin2afatag ~]# /opt/puppetlabs/server/bin/puppetserver gem list

*** LOCAL GEMS ***

deep_merge (1.0.1)

did_you_mean (default: 1.0.1)

facter (2.5.1)

fast_gettext (1.1.2)

gettext (3.2.2)

hiera-eyaml (2.1.0)

highline (1.6.21)

hocon (1.2.5)

jar-dependencies (default: 0.3.12)

jruby-openssl (0.9.21 java)

jruby-readline (1.2.2 java)

json (1.8.3 java)

locale (2.1.2)

minitest (default: 5.4.1)

multi_json (1.13.1)

net-telnet (default: 0.1.1)

power_assert (default: 0.2.3)

psych (2.2.4 java)

puppet-resource_api (1.6.3)

puppetserver-ca (1.2.1)

rake (default: 10.4.2)

rdoc (default: 4.2.0)

semantic_puppet (1.0.2)

test-unit (default: 3.1.1)

text (1.3.1)

trollop (2.9.9, 2.1.2)

Show

PE 2019.0.2 [root@h0wm6jin2afatag ~]# puppet module list /etc/puppetlabs/code/environments/production/modules ├── puppetlabs-ciscopuppet (v2.0.0) ├── puppetlabs-concat (v5.3.0) ├── puppetlabs-device_manager (v2.7.0) ├── puppetlabs-hocon (v1.0.1) ├── puppetlabs-netdev_stdlib (v0.18.0) ├── puppetlabs-puppetserver_gem (v1.1.0) ├── puppetlabs-resource_api (v1.0.0) ├── puppetlabs-stdlib (v5.2.0) └── puppetlabs-translate (v1.2.0) /etc/puppetlabs/code/modules (no modules installed) /opt/puppetlabs/puppet/modules ├── puppetlabs-facter_task (v0.4.0) ├── puppetlabs-package (v0.4.1) ├── puppetlabs-pe_accounts (v2016.5.0) ├── puppetlabs-pe_bootstrap (v0.3.0) ├── puppetlabs-pe_concat (v1.1.1) ├── puppetlabs-pe_hocon (v2019.0.0) ├── puppetlabs-pe_infrastructure (v2018.1.0) ├── puppetlabs-pe_inifile (v1.1.3) ├── puppetlabs-pe_install (v2018.1.0) ├── puppetlabs-pe_java_ks (v2016.4.0) ├── puppetlabs-pe_nginx (v2017.1.0) ├── puppetlabs-pe_postgresql (v2016.5.0) ├── puppetlabs-pe_puppet_authorization (v2016.2.0) ├── puppetlabs-pe_r10k (v2016.2.0) ├── puppetlabs-pe_razor (v1.1.0) ├── puppetlabs-pe_repo (v2018.1.0) ├── puppetlabs-pe_staging (v0.3.3) ├── puppetlabs-pe_support_script (v2.10.0) ├── puppetlabs-puppet_conf (v0.3.0) ├── puppetlabs-puppet_enterprise (v2018.1.0) └── puppetlabs-service (v0.5.0) node h0wm6jin2afatag.delivery.puppetlabs.net { include ciscopuppet::proxy device_manager { 'nexus1': type => 'cisco_nexus', credentials => { address => 'mvtc9ra62ieh96u.delivery.puppetlabs.net', username => 'admin', password => 'admin', transport => 'http', } } device_manager { 'nexus2': type => 'cisco_nexus', credentials => { address => 'fz33nfg6krsy1zt.delivery.puppetlabs.net', username => 'admin', password => 'admin', transport => 'http', } } } [root@h0wm6jin2afatag ~]# /opt/puppetlabs/puppet/bin/gem list *** LOCAL GEMS *** bigdecimal (default: 1.3.4) cisco_node_utils (2.0.0) cmath (default: 1.0.0) colored (1.2) cri (2.15.1) csv (default: 1.0.0) date (default: 1.0.0) deep_merge (1.0.1) did_you_mean (1.2.0) etc (default: 1.0.0) ethon (0.9.0) facter (3.12.3, 2.5.1) faraday (0.9.1) faraday_middleware (0.9.1) faraday_middleware-multi_json (0.0.6) fast_gettext (1.1.2) fcntl (default: 1.0.0) ffi (1.9.10) fiddle (default: 1.0.0) fileutils (default: 1.0.2) gettext (3.2.2) gettext-setup (0.28) google-protobuf (3.7.0 x86_64-linux) googleapis-common-protos-types (1.0.3) grpc (1.14.2 x86_64-linux) hiera (3.4.6) hiera-eyaml (2.1.0) highline (1.6.21) hocon (1.2.5) httpclient (2.8.3) io-console (default: 0.4.6) ipaddr (default: 1.2.0) json (default: 2.1.0) locale (2.1.2) log4r (1.1.10) minitar (0.6.1) minitest (5.10.3) multi_json (1.13.1, 1.11.0) multipart-post (2.0.0) net-ssh (4.2.0) net-telnet (0.1.1) net_http_unix (0.2.2) openssl (default: 2.1.2) pe_backup_tools (1.0.2.11.gf6a3fe6) power_assert (1.1.1) psych (default: 3.0.2) puppet (6.0.5) puppet-resource_api (1.6.3) puppet_forge (2.2.8) puppetserver-ca (1.2.1) r10k (3.0.3) rake (12.3.0) rdoc (default: 6.0.1) rugged (0.25.1.1) scanf (default: 1.0.0) sdbm (default: 1.0.0) semantic_puppet (1.0.2, 1.0.1) stringio (default: 0.0.1) strscan (default: 1.0.0) test-unit (3.2.7) text (1.3.1) trollop (2.1.2) typhoeus (1.0.2) webrick (default: 1.4.2) xmlrpc (0.3.0) zlib (default: 1.0.0) [root@h0wm6jin2afatag ~]# /opt/puppetlabs/server/bin/puppetserver gem list *** LOCAL GEMS *** deep_merge (1.0.1) did_you_mean (default: 1.0.1) facter (2.5.1) fast_gettext (1.1.2) gettext (3.2.2) hiera-eyaml (2.1.0) highline (1.6.21) hocon (1.2.5) jar-dependencies (default: 0.3.12) jruby-openssl (0.9.21 java) jruby-readline (1.2.2 java) json (1.8.3 java) locale (2.1.2) minitest (default: 5.4.1) multi_json (1.13.1) net-telnet (default: 0.1.1) power_assert (default: 0.2.3) psych (2.2.4 java) puppet-resource_api (1.6.3) puppetserver-ca (1.2.1) rake (default: 10.4.2) rdoc (default: 4.2.0) semantic_puppet (1.0.2) test-unit (default: 3.1.1) text (1.3.1) trollop (2.9.9, 2.1.2)

Team:
Network Automation
Method Found:
Needs Assessment

Release Notes:
Bug Fix
Release Notes Summary:

Hide
puppet device would not manage multiple network devices in a single run. This was a regression introduced in 6.0.5 due to ~~PUP-8766~~.

Show
puppet device would not manage multiple network devices in a single run. This was a regression introduced in 6.0.5 due to PUP-8766 .

QA Risk Assessment:
Needs Assessment

Description

When executing puppet device with multiple instances of ciscopuppet nodes beyond the first fail to execute:

[root@h0wm6jin2afatag ~]# puppet device -v --trace

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Retrieving locales

Info: starting applying configuration to nexus1 at file:///etc/puppetlabs/puppet/devices/nexus1.conf

Info: Using configured environment 'production'

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Retrieving locales

/opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/cisco_node_utils-2.0.0/lib/cisco_node_utils/node.rb:154: warning: constant ::Fixnum is deprecated

Info: Caching catalog for nexus1

Info: Applying configuration version '1553200309'

Notice: Applied catalog in 0.12 seconds

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Retrieving locales

Info: starting applying configuration to nexus2 at file:///etc/puppetlabs/puppet/devices/nexus2.conf

Warning: Unable to fetch my node definition, but the agent run will continue:

Warning: Error 403 on SERVER: Forbidden request: /puppet/v3/node/nexus2 (method :get). Please see the server logs for details.

Info: Retrieving pluginfacts

Info: Retrieving plugin

Info: Retrieving locales

/opt/puppetlabs/puppet/lib/ruby/gems/2.5.0/gems/cisco_node_utils-2.0.0/lib/cisco_node_utils/node.rb:154: warning: constant ::Fixnum is deprecated

Error: Could not retrieve catalog from remote server: Error 403 on SERVER: Forbidden request: /puppet/v3/catalog/nexus2 (method :post). Please see the server logs for details.

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:241:in `is_http_200?'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:120:in `find'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:464:in `block in retrieve_new_catalog'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:518:in `block in thinmark'

/opt/puppetlabs/puppet/lib/ruby/2.5.0/benchmark.rb:308:in `realtime'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:517:in `thinmark'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:463:in `retrieve_new_catalog'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:75:in `retrieve_catalog'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:167:in `prepare_and_retrieve_catalog'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:341:in `run_internal'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:239:in `block in run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:252:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:217:in `run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:327:in `block (2 levels) in main'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:256:in `each'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:256:in `collect'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:256:in `block in main'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:252:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:241:in `main'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:390:in `run_command'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:671:in `exit_on_fail'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:135:in `run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'

/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

Error: Could not retrieve catalog; skipping run

Error: Could not send report: Error 403 on SERVER: Forbidden request: /puppet/v3/report/nexus2 (method :put). Please see the server logs for details.

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:241:in `is_http_200?'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:190:in `save'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:285:in `save'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:416:in `send_report'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:388:in `run_internal'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:239:in `block in run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:252:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/configurer.rb:217:in `run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:327:in `block (2 levels) in main'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:256:in `each'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:256:in `collect'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:256:in `block in main'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:252:in `override'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/device.rb:241:in `main'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:390:in `run_command'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `block in run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:671:in `exit_on_fail'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:382:in `run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:135:in `run'

/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:73:in `execute'

/opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

Server Log

2019-03-21T20:31:49.135Z INFO  [qtp2129953650-177] [puppetserver] Puppet 'replace_facts' command for nexus1 submitted to PuppetDB with UUID 15cc8e0d-5e6c-4ee8-9c6f-2e1c0129763e

2019-03-21T20:31:49.301Z WARN  [qtp2129953650-177] [puppetserver] Puppet Host is missing hostname and/or domain: nexus1

2019-03-21T20:31:49.305Z INFO  [qtp2129953650-177] [puppetserver] Puppet Compiled catalog for nexus1 in environment production in 0.07 seconds

2019-03-21T20:31:49.305Z INFO  [qtp2129953650-177] [puppetserver] Puppet Caching catalog for nexus1

2019-03-21T20:31:49.342Z INFO  [qtp2129953650-177] [puppetserver] Puppet 'replace_catalog' command for nexus1 submitted to PuppetDB with UUID a0568882-d669-440d-b4ec-16379250973b

2019-03-21T20:31:49.588Z INFO  [qtp2129953650-181] [puppetserver] Puppet 'store_report' command for nexus1 submitted to PuppetDB with UUID 06465d95-6019-4f82-8f76-1ea37d7d04b8

2019-03-21T20:31:51.111Z ERROR [qtp2129953650-176] [p.t.a.rules] Forbidden request: nexus1(10.16.114.187) access to /puppet/v3/node/nexus2 (method :get) (authenticated: true) denied by rule 'puppetlabs node'.

2019-03-21T20:31:55.961Z ERROR [qtp2129953650-176] [p.t.a.rules] Forbidden request: nexus1(10.16.114.187) access to /puppet/v3/catalog/nexus2 (method :post) (authenticated: true) denied by rule 'puppetlabs catalog'.

2019-03-21T20:31:55.996Z ERROR [qtp2129953650-181] [p.t.a.rules] Forbidden request: nexus1(10.16.114.187) access to /puppet/v3/report/nexus2 (method :put) (authenticated: true) denied by rule 'puppetlabs report'

Attachments

Issue Links

relates to

PUP-9579 device application is broken in 6.4

Closed

Activity

People

Assignee:: David Schmitt

Reporter:: Rick Sherman

Votes:: 1 Vote for this issue

Watchers:: 6 Start watching this issue

Dates

Created:: 2019/03/21 1:57 PM

Updated:: 2019/07/12 1:44 AM

Resolved:: 2019/04/05 11:30 AM