Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-10775

Puppet agent fails with `Failed to deserialize Puppet::Resource::Catalog`

    XMLWordPrintable

Details

    • Bug
    • Status: Accepted
    • Normal
    • Resolution: Unresolved
    • None
    • None
    • None
    • None
    • Needs Assessment
    • Needs Assessment

    Description

      Hey josh,

      I did further debugging for https://tickets.puppetlabs.com/browse/PUP-10772. while doing so I discovered that I've 5 Ubuntu 16.04 machines, all of them currently show the
      `Failed to deserialize Puppet::Resource::Catalog` error message that I show later

      my client:

      root~ # facter -p os
      pup{
        architecture => "amd64",
        distro => {
          codename => "xenial",
          description => "Ubuntu 16.04.7 LTS",
          id => "Ubuntu",
          release => {
            full => "16.04",
            major => "16.04"
          }
        },
        family => "Debian",
        hardware => "x86_64",
        name => "Ubuntu",
        release => {
          full => "16.04",
          major => "16.04"
        },
        selinux => {
          enabled => false
        }
      }
      root~ # puppet --version
      6.19.1
      root ~ # 
      

      preferred_serialization_format is not set:

      root ~ # cat /etc/puppetlabs/puppet/puppet.conf
       
      [main]
          certname = *
          codedir = /etc/puppetlabs/code
          hostprivkey = $privatekeydir/$certname.pem { mode = 640 }
          logdir = /var/log/puppetlabs/puppet
          pluginfactsource = puppet:///pluginfacts
          pluginsource = puppet:///plugins
          privatekeydir = $ssldir/private_keys { group = service }
          rundir = /var/run/puppetlabs
          server = *
          show_diff = true
          ssldir = /etc/puppetlabs/puppet/ssl
          vardir = /opt/puppetlabs/puppet/cache
       
      [agent]
          classfile = $statedir/classes.txt
          default_schedules = false
          environment = production
          localconfig = $vardir/localconfig
          masterport = 8140
          noop = false
          report = true
          runinterval = 7200
          splay = true
          splaylimit = 7200
          usecacheonfailure = true
      root ~ # 
      

      but msgpack is installed:

      root ~ # /opt/puppetlabs/puppet/bin/gem list msgpack
       
      *** LOCAL GEMS ***
       
      msgpack (1.3.3, 1.3.1)
      root ~ # 
      

      during a puppetrun to a 6.14.1 puppetserver with msgpack installed I see the following messages:

      starting SSL for *:8140...
      SSL established
      <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+json, application/json, application/vnd.puppet.rich+msgpack, application/x-msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40356\r\n\r\n"
      ...
      Info: Unable to serialize catalog to json, retrying with pson
      ...
      

      the agent run works fine. Now the same agent connects to a puppetserver without msgpack installed:

      ...
      starting SSL for *:8140...
      SSL established
      <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+json, application/json, application/vnd.puppet.rich+msgpack, application/x-msgpack, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40357\r\n\r\n"
      ...
      -> "Content-Type: text/pson;charset=iso-8859-1\r\n"
      -> "Transfer-Encoding: chunked\r\n"
      -> "Connection: keep-alive\r\n"
      -> "X-Puppet-Version: 6.19.1\r\n"
      -> "Vary: Accept-Encoding, User-Agent\r\n"
      -> "Content-Encoding: gzip\r\n"
      ...
      Error: Could not retrieve catalog from remote server: Failed to deserialize Puppet::Resource::Catalog from pson: expected value in object at '"ELF\u0002\u0001\u0'!
      Wrapped exception:
      expected value in object at '"ELF\u0002\u0001\u0'!
      Warning: Not using cache on failed catalog
      Error: Could not retrieve catalog; skipping run
      

      now it fails? Sadly I don't know since when this is the case. I suspect someone added a new file resource with a binary that's read in with file(). But this is really just a guess. Our codebase is pretty huge and digging through all of the modules takes some time. I cannot reproduce this on other agents with a different operating system. Sadly the code on those Ubuntu 16.04 boxes does only work on this specific OS. I don't know if this Ubuntu specific or indeed some puppet code.

      Third run. msgpack installed again on the puppetserver. also enabled preferred_serialization_format = msgpack on the agent:

      starting SSL for *:8140...
      SSL established
      <- "POST /puppet/v3/catalog/*?environment=production HTTP/1.1\r\nX-Puppet-Version: 6.19.1\r\nUser-Agent: Puppet/6.19.1 Ruby/2.5.8-p224 (x86_64-linux)\r\nAccept: application/vnd.puppet.rich+msgpack, application/x-msgpack, application/vnd.puppet.rich+json, application/json, text/pson\r\nContent-Type: application/x-www-form-urlencoded\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: *:8140\r\nContent-Length: 40356\r\n\r\n"
      ...
      -> "Date: Thu, 05 Nov 2020 17:14:27 GMT\r\n"
      -> "Content-Type: application/vnd.puppet.rich+msgpack;charset=iso-8859-1\r\n"
      -> "Transfer-Encoding: chunked\r\n"
      -> "Connection: keep-alive\r\n"
      -> "X-Puppet-Version: 6.19.1\r\n"
      -> "Vary: Accept-Encoding, User-Agent\r\n"
      -> "Content-Encoding: gzip\r\n"
      ...
      Info: Unable to serialize catalog to json, retrying with pson
      ....
      

      the agent run works as well with this setup. but I think the info message is wrong. While it might try pson, afterwards msgpack is used to decode the payload? Or the msgpack serialized to msgpack?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              bastelfreak Tim Meusel
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:

                Zendesk Support