Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-7409

Knockout prefix does not remove an entire key:value entry from a hash

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Done
    • Affects Version/s: PUP 4.9.4
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Environment:

      Test Vagrant environment using PE2017.1.1

    • Template:
    • Team:
      Puppet Developer Experience
    • Sprint:
      Forge - To Accept
    • QA Risk Assessment:
      Needs Assessment

      Description

      While doing some testing on the behaviour of deep_merge using Hiera, it was noted that there was some unexpected behaviour when using a knockout_prefix in the lookup options.

      The common.yaml files were defined as follows:

      [root@pe-201711-master production]# cat data/common.yaml
      ---
        profile::test::hashdeep:
          --key1: 'a'
          key2: '--f'
          --key3: '--g'
          key5: 'val5'
          array1:
            - --a
            - b
            - e
            - f
      [root@pe-201711-master production]# cat modules/profile/data/common.yaml
      ---
        profile::test::hashdeep:
          key1: 'a'
          key2: 'b'
          key3: 'c'
          key4: 'd'
          array1:
            - a
            - b
            - c
            - d
        lookup_options:
          profile::test::hashdeep:
            merge:
              strategy: deep
              knockout_prefix: --
              sort_merge_arrays: true
      [root@pe-201711-master production]#
      

      The expected behaviour was that either using puppet lookup or using a notify statement, key1 and key3 would not be included in the returned data, and key2 would return an empty, undefined value.

      However, it appears that using the knockout_prefix on a key name simply prepends the prefix to the key name in the output, rather than removing the key and value completely. Using the knockout_prefix on the value for a given key works as expected, with the value being removed and rest to an undefined, empty value.

      [root@pe-201711-master production]# puppet lookup profile::test::hashdeep --explain
      Warning: /etc/puppetlabs/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5
         (in /etc/puppetlabs/puppet/hiera.yaml)
      Warning: Undefined variable 'pp_application';
         (file & line not available)
      Warning: Undefined variable 'pp_role';
         (file & line not available)
      Warning: Undefined variable 'pp_environment';
         (file & line not available)
      Warning: Undefined variable 'pp_datacenter';
         (file & line not available)
      Warning: Undefined variable 'pp_zone';
         (file & line not available)
      Warning: Undefined variable '::pp_application';
         (file & line not available)
      Warning: Undefined variable '::pp_role';
         (file & line not available)
      Warning: Undefined variable '::pp_environment';
         (file & line not available)
      Warning: Undefined variable '::pp_datacenter';
         (file & line not available)
      Warning: Undefined variable '::pp_zone';
         (file & line not available)
      Searching for "lookup_options"
        Module "profile" Data Provider (hiera configuration version 5)
          Using configuration "/etc/puppetlabs/code/environments/production/modules/profile/hiera.yaml"
          Hierarchy entry "Module Hiera Data"
            Merge strategy hash
              Path "/etc/puppetlabs/code/environments/production/modules/profile/data/CentOS-7.yaml"
                Original path: "%{facts.os.name}-%{facts.os.release.major}.yaml"
                Path not found
              Path "/etc/puppetlabs/code/environments/production/modules/profile/data/CentOS.yaml"
                Original path: "%{facts.os.name}.yaml"
                Path not found
              Path "/etc/puppetlabs/code/environments/production/modules/profile/data/RedHat.yaml"
                Original path: "%{facts.os.family}.yaml"
                Path not found
              Path "/etc/puppetlabs/code/environments/production/modules/profile/data/common.yaml"
                Original path: "common.yaml"
                Found key: "lookup_options" value: {
                  "profile::test::hashdeep" => {
                    "merge" => {
                      "strategy" => "deep",
                      "knockout_prefix" => "--",
                      "sort_merge_arrays" => true
                    }
                  }
                }
              Path "/etc/puppetlabs/code/environments/production/modules/profile/data/profile.yaml"
                Original path: "profile.yaml"
                Path not found
              Merged result: {
                "profile::test::hashdeep" => {
                  "merge" => {
                    "strategy" => "deep",
                    "knockout_prefix" => "--",
                    "sort_merge_arrays" => true
                  }
                }
              }
      Using merge options from "lookup_options" hash
      Searching for "profile::test::hashdeep"
        Merge strategy deep
          Options: {
            "knockout_prefix" => "--",
            "sort_merge_arrays" => true
          }
          Global Data Provider (hiera configuration version 3)
            Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
            Hierarchy entry "yaml"
              Merge strategy deep
                Options: {
                  "knockout_prefix" => "--",
                  "sort_merge_arrays" => true
                }
                Path "/etc/puppetlabs/code/environments/production/hieradata/nodes/pe-201711-master.puppetdebug.vlan.yaml"
                  Original path: "nodes/%{::trusted.certname}"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/hieradata/common.yaml"
                  Original path: "common"
                  No such key: "profile::test::hashdeep"
          Environment Data Provider (hiera configuration version 5)
            Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
            Hierarchy entry "Global yaml hierarchy"
              Merge strategy deep
                Options: {
                  "knockout_prefix" => "--",
                  "sort_merge_arrays" => true
                }
                Path "/etc/puppetlabs/code/environments/production/data/nodes/pe-201711-master.puppetdebug.vlan.yaml"
                  Original path: "nodes/%{::trusted.certname}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/data/role/--.yaml"
                  Original path: "role/%{::pp_application}-%{::pp_role}-%{::pp_environment}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/data/role/-.yaml"
                  Original path: "role/%{::pp_application}-%{::pp_role}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/data/role/-.yaml"
                  Original path: "role/%{::pp_application}-%{::pp_environment}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/data/role/.yaml"
                  Original path: "role/%{::pp_application}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/data/location/-.yaml"
                  Original path: "location/%{::pp_datacenter}-%{::pp_zone}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/data/location/.yaml"
                  Original path: "location/%{::pp_datacenter}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/data/location/.yaml"
                  Original path: "location/%{::pp_zone}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/data/common.yaml"
                  Original path: "common.yaml"
                  Found key: "profile::test::hashdeep" value: {
                    "--key1" => "a",
                    "key2" => "--f",
                    "--key3" => "--g",
                    "key5" => "val5",
                    "array1" => [
                      "b",
                      "e",
                      "f"
                    ]
                  }
                Merged result: {
                  "--key1" => "a",
                  "key2" => "--f",
                  "--key3" => "--g",
                  "key5" => "val5",
                  "array1" => [
                    "b",
                    "e",
                    "f"
                  ]
                }
          Module "profile" Data Provider (hiera configuration version 5)
            Using configuration "/etc/puppetlabs/code/environments/production/modules/profile/hiera.yaml"
            Hierarchy entry "Module Hiera Data"
              Merge strategy deep
                Options: {
                  "knockout_prefix" => "--",
                  "sort_merge_arrays" => true
                }
                Path "/etc/puppetlabs/code/environments/production/modules/profile/data/CentOS-7.yaml"
                  Original path: "%{facts.os.name}-%{facts.os.release.major}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/modules/profile/data/CentOS.yaml"
                  Original path: "%{facts.os.name}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/modules/profile/data/RedHat.yaml"
                  Original path: "%{facts.os.family}.yaml"
                  Path not found
                Path "/etc/puppetlabs/code/environments/production/modules/profile/data/common.yaml"
                  Original path: "common.yaml"
                  Found key: "profile::test::hashdeep" value: {
                    "key1" => "a",
                    "key2" => "b",
                    "key3" => "c",
                    "key4" => "d",
                    "array1" => [
                      "a",
                      "b",
                      "c",
                      "d"
                    ]
                  }
                Path "/etc/puppetlabs/code/environments/production/modules/profile/data/profile.yaml"
                  Original path: "profile.yaml"
                  Path not found
                Merged result: {
                  "key1" => "a",
                  "key2" => "b",
                  "key3" => "c",
                  "key4" => "d",
                  "array1" => [
                    "a",
                    "b",
                    "c",
                    "d"
                  ]
                }
          Merged result: {
            "key1" => "a",
            "key2" => "",
            "key3" => "c",
            "key4" => "d",
            "array1" => [
              "b",
              "c",
              "d",
              "e",
              "f"
            ],
            "--key1" => "a",
            "--key3" => "",
            "key5" => "val5"
          }
      [root@pe-201711-master production]#
      

      Is this a bug or is it by design, and if it is by design, how could an entire key:value pair be removed from the output using a knockout_prefix value?

        Attachments

          Activity

            People

            • Assignee:
              thomas.hallgren Thomas Hallgren
              Reporter:
              stefan.pijnappels Stefan Pijnappels
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Zendesk Support