Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-10772

fresh puppet agent installation fails when msgpack is enabled on puppetserver

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: PUP 7.0.0, PUP 6.20.0
    • Component/s: None
    • Labels:
    • Template:
      PUP Bug Template
    • Team:
      Coremunity
    • Sprint:
      Platform Core KANBAN
    • Method Found:
      Needs Assessment
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Hide
      The agent claimed to accept catalogs serialized using the "rich_data_msgpack" format, but if the "msgpack" gem wasn't installed, then the agent would fail to deserialize the catalog and fail the run. Now the agent only claims to support that format when the "msgpack" gem is installed in the puppet agent's vendored ruby (such as when using the "puppet_gem" package provider).
      Show
      The agent claimed to accept catalogs serialized using the "rich_data_msgpack" format, but if the "msgpack" gem wasn't installed, then the agent would fail to deserialize the catalog and fail the run. Now the agent only claims to support that format when the "msgpack" gem is installed in the puppet agent's vendored ruby (such as when using the "puppet_gem" package provider).
    • QA Risk Assessment:
      Needs Assessment

      Description

      Hi!
      (I'm not sure if this is a bug in puppet agent or puppetserver)

      I'm running the latest puppetserver

      root ~ # puppetserver -v
      puppetserver version: 6.14.1
      root ~ # 
      

      the msgpack gem is installed:

      root ~ # puppetserver gem list msgpack
       
      *** LOCAL GEMS ***
       
      msgpack (1.3.3 java)
      root ~ # 
      

      preferred_serialization_format isn't set on the server:

      [main]
          ca_server = *
          certname = *
          codedir = /etc/puppetlabs/code
          environmentpath = /etc/puppetlabs/code/environments
          hiera_config = $confdir/hiera.yaml
          hostprivkey = $privatekeydir/$certname.pem { mode = 640 }
          logdir = /var/log/puppetlabs/puppet
          pluginfactsource = puppet:///pluginfacts
          pluginsource = puppet:///plugins
          privatekeydir = $ssldir/private_keys { group = service }
          reports = puppetdb,foreman
          rundir = /var/run/puppetlabs
          server = *
          show_diff = true
          ssldir = /etc/puppetlabs/puppet/ssl
          vardir = /opt/puppetlabs/puppet/cache
       
      [agent]
          classfile = $statedir/classes.txt
          default_schedules = false
          environment = snow
          localconfig = $vardir/localconfig
          masterport = 8140
          noop = false
          report = true
          runinterval = 7200
          splay = true
          splaylimit = 7200
          usecacheonfailure = true
       
      [master]
          autosign = /etc/puppetlabs/puppet/autosign.conf { mode = 0664 }
          ca = true
          certname = *
          external_nodes = /etc/puppetlabs/puppet/node.rb
          logdir = /var/log/puppetlabs/puppetserver
          node_terminus = exec
          parser = current
          rundir = /var/run/puppetlabs/puppetserver
          ssldir = /etc/puppetlabs/puppet/ssl
          storeconfigs = true
          strict_variables = true
          trusted_external_command = /etc/puppetlabs/puppet/trusted-external-commands/*.rb
          vardir = /opt/puppetlabs/server/data/puppetserver
      

      We rolled out the msgpack gem to existing agents:

        if fact('aio_agent_version') {
          package { 'msgpack':
            ensure   => 'present',
            provider => 'puppet_gem',
          }
        }
      

      because I forgot to install it on the servers as well, we did that with multiple days delay:

          package { 'msgpack-puppetserver':
            ensure   => 'present',
            name     => 'msgpack',
            provider => 'puppetserver_gem',
            notify   => Service['puppetserver'],
          }
      

      this means by the time it got installed on the puppetservers, all existing agents already had the gem installed. those agents produce the following debug output:

      Debug: catalog supports formats: rich_data_json json rich_data_msgpack msgpack pson yaml dot
      ...
      Debug: file_metadata supports formats: json msgpack pson yaml
      Debug: Closing connection for https://*:8140
      Debug: Creating new connection for https://*:8140
      Debug: Starting connection for https://*:8140
      Debug: Using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256
      Debug: HTTP GET https://*:8140/puppet/v3/file_metadatas/plugins?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=md5&source_permissions=ignore&environment=snow returned 200 OK
      Debug: Caching connection for https://*:8140
      ...
      Debug: file_metadata supports formats: json msgpack pson yaml
      Debug: Using cached connection for https://*:8140
      Debug: HTTP GET https://*:8140/puppet/v3/file_metadatas/locales?recurse=true&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&ignore=%2A.pot&ignore=config.yaml&links=follow&checksum_type=md5&source_permissions=ignore&environment=snow returned 200 OK
      Debug: Caching connection for https://*:8140
      Debug: Finishing transaction 40300740
      

      thats from a up2date box:

      root ~ # puppet --version
      6.19.1
      root ~ # /opt/puppetlabs/puppet/bin/gem list msgpack
       
      *** LOCAL GEMS ***
       
      msgpack (1.3.1)
      root ~ # 
      

      now a new setup. I installed the puppet6-el7-release rpm and afterwards puppet-agent. the config file is empty:

      root ~ # cat /etc/puppetlabs/puppet/puppet.conf 
      # This file can be used to override the default puppet settings.
      # See the following links for more details on what settings are available:
      # - https://puppet.com/docs/puppet/latest/config_important_settings.html
      # - https://puppet.com/docs/puppet/latest/config_about_settings.html
      # - https://puppet.com/docs/puppet/latest/config_file_main.html
      # - https://puppet.com/docs/puppet/latest/configuration.html
      root ~ #
      

      this nodes logs:

      Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
      Debug: catalog supports formats: rich_data_json json rich_data_msgpack pson yaml dot
      Debug: Using cached connection for https://:8140
      Debug: HTTP POST https://*:8140/puppet/v3/catalog/*?environment=production returned 200 OK
      Debug: Caching connection for https://*:8140
      Error: Could not retrieve catalog from remote server: Failed to deserialize Puppet::Resource::Catalog from rich_data_msgpack: uninitialized constant MessagePack
      Wrapped exception:
      uninitialized constant MessagePack
      Warning: Not using cache on failed catalog
      Error: Could not retrieve catalog; skipping run
      Debug: Resolving service 'report' using Puppet::HTTP::Resolver::SRV
      Debug: Searching for SRV records for domain: *
      Debug: Found 0 SRV records for: _x-puppet-report._tcp.*
      Debug: Searching for SRV records for domain: *
      Debug: Using cached record for _x-puppet._tcp.*
      Debug: Yielding next server of *:8140
      Debug: Using cached connection for https://*:8140
      Debug: Caching connection for https://*:8140
      Debug: Resolved service 'report' to https://*:8140/puppet/v3
      Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
      Debug: report supports formats: json pson yaml
      Debug: Using cached connection for https://*:8140
      Debug: HTTP PUT https://*:8140/puppet/v3/report/*?environment=production returned 200 OK
      Debug: Caching connection for https://*:8140
      Debug: Closing connection for https://*:8140
      

      installed gems on the new agent:

      root ~ # /opt/puppetlabs/puppet/bin/gem list
       
      *** LOCAL GEMS ***
       
      bigdecimal (default: 1.3.4)
      cmath (default: 1.0.0)
      concurrent-ruby (1.1.5)
      csv (default: 1.0.0)
      date (default: 1.0.0)
      deep_merge (1.0.1)
      did_you_mean (1.2.0)
      etc (default: 1.0.0)
      facter (3.14.14)
      facter-ng (4.0.43)
      fast_gettext (1.1.2)
      fcntl (default: 1.0.0)
      ffi (1.13.1)
      fiddle (default: 1.0.0)
      fileutils (default: 1.0.2)
      gettext (3.2.2)
      hiera (3.6.0)
      hiera-eyaml (3.2.0)
      highline (1.6.21)
      hocon (1.3.1)
      httpclient (2.8.3)
      io-console (default: 0.4.6)
      ipaddr (default: 1.2.0)
      json (default: 2.1.0)
      locale (2.1.3)
      minitest (5.10.3)
      multi_json (1.14.1)
      net-ssh (4.2.0)
      net-telnet (0.1.1)
      openssl (default: 2.1.2)
      optimist (3.0.1)
      power_assert (1.1.1)
      psych (default: 3.0.2)
      puppet (6.19.1)
      puppet-resource_api (1.8.13)
      rake (12.3.3)
      rdoc (default: 6.0.1.1)
      scanf (default: 1.0.0)
      sdbm (default: 1.0.0)
      semantic_puppet (1.0.2)
      stringio (default: 0.0.1)
      strscan (default: 1.0.0)
      sys-filesystem (1.3.2)
      test-unit (3.2.7)
      text (1.3.1)
      thor (1.0.1)
      webrick (default: 1.4.2)
      xmlrpc (0.3.0)
      zlib (default: 1.0.0)
      root ~ #
      

      I got a few questions here:

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              josh Josh Cooper
              Reporter:
              bastelfreak Tim Meusel
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support