Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-7554

Puppet runs fail when hiera.yaml version 3 detected in a module

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: PUP 4.10.1
    • Fix Version/s: PUP 4.10.2, PUP 5.0.0
    • Component/s: None
    • Environment:

      PE 2016.4.5 with Hiera.yaml 3 files in modules

    • Template:
    • Epic Link:
    • Team:
      Agent
    • Sub-team:
    • Story Points:
      1
    • Sprint:
      Agent 2017-05-31, Agent 2017-06-14
    • CS Priority:
      Major
    • CS Impact:
      Customers will experience failures with the existing code set upon an upgrade to the latest LTS release.
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Hide
      With the introduction of Hiera 5 there were errors if a module or environment root contained a hiera.yaml in the hiera 3 format. These files were never used earlier, but now those files are part of the hiera 5 configuration. The issue is fixed by ignoring the files if they are version 3 and by logging a warning when encountered. Best practice is to migrate to hiera 5, and otherwise to move those files out of the way.
      Show
      With the introduction of Hiera 5 there were errors if a module or environment root contained a hiera.yaml in the hiera 3 format. These files were never used earlier, but now those files are part of the hiera 5 configuration. The issue is fixed by ignoring the files if they are version 3 and by logging a warning when encountered. Best practice is to migrate to hiera 5, and otherwise to move those files out of the way.
    • QA Risk Assessment:
      Automate
    • QA Risk Assessment Reason:
      covered by unit tests

      Description

      This is a regression when upgrading from the 2016.4.x LTS to 2016.4.5. Puppet version 4.10.1 runs fail when an hiera.yaml version 3 file is found in the module. For a number of reasons, customers may have hiera.yaml files in modules and the environment. There should be a warning about the version and it should be ignores instead of causing a failure.

      Impact
      Customers that upgrade to the latest LTS which have hiera.yaml 3 files in their modules will have failures. The upgrade should allow for customers to migrate into a hiera 5 type environment without failures.

        Attachments

          Issue Links

            Activity

            Hide
            geoff.nichols Geoff Nichols added a comment -
            Show
            geoff.nichols Geoff Nichols added a comment - PR #5887 was merged to puppet#stable at https://github.com/puppetlabs/puppet/commit/93ac4dca91b510cc6f76a83111308f813ec111cf .
            Hide
            kenn Kenn Hussey added a comment -

            Henrik Lindberg please provide release notes for this issue, if applicable.

            Show
            kenn Kenn Hussey added a comment - Henrik Lindberg please provide release notes for this issue, if applicable.
            Hide
            erict Eric Thompson added a comment -

            Thomas Hallgren, Ethan Brown would the spec tests benefit from the addition of something like:

            index a69f96c..5578471 100644
            --- a/spec/unit/functions/lookup_spec.rb
            +++ b/spec/unit/functions/lookup_spec.rb
            @@ -2084,6 +2084,9 @@ describe "The lookup function" do
                         expect(lookup('mod_a::a')).to eql('value mod_a::a (from environment)')
                         expect(warnings).to include(/hiera.yaml version 3 found at module root was ignored/)
                       end
            +          it 'does not raise an error' do
            +            expect { lookup('mod_a::a') }.to_not raise_error(Puppet::Error)
            +          end
                     end
             
                     context "but a metadata.json with 'module_data_provider=hiera'" do
            

            that passes for me and ensures we don't error in addition to the warning.

            Show
            erict Eric Thompson added a comment - Thomas Hallgren , Ethan Brown would the spec tests benefit from the addition of something like: index a69f96c..5578471 100644 --- a/spec/unit/functions/lookup_spec.rb +++ b/spec/unit/functions/lookup_spec.rb @@ -2084,6 +2084,9 @@ describe "The lookup function" do expect(lookup('mod_a::a')).to eql('value mod_a::a (from environment)') expect(warnings).to include(/hiera.yaml version 3 found at module root was ignored/) end + it 'does not raise an error' do + expect { lookup('mod_a::a') }.to_not raise_error(Puppet::Error) + end end   context "but a metadata.json with 'module_data_provider=hiera'" do that passes for me and ensures we don't error in addition to the warning.
            Hide
            erict Eric Thompson added a comment - - edited

            validated against puppet-agent build: e334e8d1e02858d51509997a4d354e93e7d94981

            [root@n8mphtq05lcfzkx production]# cat /opt/puppetlabs/puppet/VERSION
            1.10.1.37.ge334e8d
            

            containing puppet sha: puppet 4.10.1-90-g33b929c
            with:

            [root@n8mphtq05lcfzkx somemodule]# puppet agent -t --server $(hostname -f)
            Info: Using configured environment 'production'
            Info: Retrieving pluginfacts
            Info: Retrieving plugin
            Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Function lookup() did not find a value for the name 'somemodule::somekey' on node n8mphtq05lcfzkx.delivery.puppetlabs.net
            Warning: Not using cache on failed catalog
            Error: Could not retrieve catalog; skipping run
             
            # [expected]
             
            [root@n8mphtq05lcfzkx somemodule]# puppet lookup somemodule::somekey
            Warning: /etc/puppetlabs/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
               (in /etc/puppetlabs/puppet/hiera.yaml)
            Warning: hiera.yaml version 3 found at the environment root was ignored
               (in /etc/puppetlabs/code/environments/production/hiera.yaml)
            Warning: hiera.yaml version 3 found at module root was ignored
               (in /etc/puppetlabs/code/environments/production/modules/somemodule/hiera.yaml)
             
            # [expected]
             
            [root@n8mphtq05lcfzkx production]# cat modules/somemodule/hiera.yaml modules/somemodule/manifests/init.pp manifests/site.pp
            ---
            :backends:
              - yaml
            :hierarchy:
              - "nodes/%{::trusted.certname}"
              - common
             
            :yaml:
            # datadir is empty here, so hiera uses its defaults:
            # - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix
            # - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows
            # When specifying a datadir, make sure the directory exists.
              :datadir:
            class somemodule {
              notify{"somemodule_${lookup('somemodule::somekey')}":}
            }
            include somemodule
            [root@n8mphtq05lcfzkx production]# cat modules/somemodule/hieradata/common.yaml
            ---
            somemodule::somekey: somevalue
            

            Show
            erict Eric Thompson added a comment - - edited validated against puppet-agent build: e334e8d1e02858d51509997a4d354e93e7d94981 [root@n8mphtq05lcfzkx production]# cat /opt/puppetlabs/puppet/VERSION 1.10.1.37.ge334e8d containing puppet sha: puppet 4.10.1-90-g33b929c with: [root@n8mphtq05lcfzkx somemodule]# puppet agent -t --server $(hostname -f) Info: Using configured environment 'production' Info: Retrieving pluginfacts Info: Retrieving plugin Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Function lookup() did not find a value for the name 'somemodule::somekey' on node n8mphtq05lcfzkx.delivery.puppetlabs.net Warning: Not using cache on failed catalog Error: Could not retrieve catalog; skipping run   # [expected]   [root@n8mphtq05lcfzkx somemodule]# puppet lookup somemodule::somekey Warning: /etc/puppetlabs/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 (in /etc/puppetlabs/puppet/hiera.yaml) Warning: hiera.yaml version 3 found at the environment root was ignored (in /etc/puppetlabs/code/environments/production/hiera.yaml) Warning: hiera.yaml version 3 found at module root was ignored (in /etc/puppetlabs/code/environments/production/modules/somemodule/hiera.yaml)   # [expected]   [root@n8mphtq05lcfzkx production]# cat modules/somemodule/hiera.yaml modules/somemodule/manifests/init.pp manifests/site.pp --- :backends: - yaml :hierarchy: - "nodes/%{::trusted.certname}" - common   :yaml: # datadir is empty here, so hiera uses its defaults: # - /etc/puppetlabs/code/environments/%{environment}/hieradata on *nix # - %CommonAppData%\PuppetLabs\code\environments\%{environment}\hieradata on Windows # When specifying a datadir, make sure the directory exists. :datadir: class somemodule { notify{"somemodule_${lookup('somemodule::somekey')}":} } include somemodule [root@n8mphtq05lcfzkx production]# cat modules/somemodule/hieradata/common.yaml --- somemodule::somekey: somevalue
            Hide
            erict Eric Thompson added a comment - - edited

            i could not get the warning to show up in the agent log no matter what i did. --strict, --debug, --verbose did not affect this.

            is the warning supposed to be master/server only?

            [root@n8mphtq05lcfzkx production]# grep ignored /var/log/puppetlabs/puppetserver/puppetserver.log
            2017-05-31 15:52:55,912 WARN  [qtp1533593850-56] [puppetserver] Puppet hiera.yaml version 3 found at the environment root was ignored
            2017-05-31 15:55:18,137 WARN  [qtp1533593850-59] [puppetserver] Puppet hiera.yaml version 3 found at module root was ignored
            

            Show
            erict Eric Thompson added a comment - - edited i could not get the warning to show up in the agent log no matter what i did. --strict, --debug, --verbose did not affect this. is the warning supposed to be master/server only? [root@n8mphtq05lcfzkx production]# grep ignored /var/log/puppetlabs/puppetserver/puppetserver.log 2017-05-31 15:52:55,912 WARN [qtp1533593850-56] [puppetserver] Puppet hiera.yaml version 3 found at the environment root was ignored 2017-05-31 15:55:18,137 WARN [qtp1533593850-59] [puppetserver] Puppet hiera.yaml version 3 found at module root was ignored
            Hide
            erict Eric Thompson added a comment -

            John Duarte FYI there is a test suite for hiera5 in the PDE testrail group.
            we really need to re-org tests by their project, IMO.

            i'd love to move the hiera5 and language test-cases back out of PDE into wherever the rest of the puppet tests should go.
            i have a feeling the case linked to this ticket is a dupe

            Show
            erict Eric Thompson added a comment - John Duarte FYI there is a test suite for hiera5 in the PDE testrail group. we really need to re-org tests by their project, IMO. i'd love to move the hiera5 and language test-cases back out of PDE into wherever the rest of the puppet tests should go. i have a feeling the case linked to this ticket is a dupe
            Hide
            thomas.hallgren Thomas Hallgren added a comment -

            Eric Thompson, afaik, all unit tests implicitly tests for no error so adding a test like that doesn't validate anything.

            Show
            thomas.hallgren Thomas Hallgren added a comment - Eric Thompson , afaik, all unit tests implicitly tests for no error so adding a test like that doesn't validate anything.
            Hide
            thomas.hallgren Thomas Hallgren added a comment -

            Eric Thompson, regarding the warning. It will only show up in the log on the master, and it will only show up once so what you see is to be expected.

            Show
            thomas.hallgren Thomas Hallgren added a comment - Eric Thompson , regarding the warning. It will only show up in the log on the master, and it will only show up once so what you see is to be expected.
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            Moving this back to "Ready for Test" as it behaves as expected.

            Show
            henrik.lindberg Henrik Lindberg added a comment - Moving this back to "Ready for Test" as it behaves as expected.
            Hide
            josh Josh Cooper added a comment -
            Show
            josh Josh Cooper added a comment - Merged to stable in https://github.com/puppetlabs/puppet/commit/93ac4dc

              People

              • Assignee:
                Unassigned
                Reporter:
                jarret.lavallee Jarret Lavallee
              • Votes:
                0 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support