Uploaded image for project: 'Hiera'
  1. Hiera
  2. HI-127

Allow escaping %{...} in hiera data to avoid replacement (add literal function)

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HI 2.0.0
    • Component/s: DOCS
    • Labels:
    • Template:
    • Story Points:
      1
    • Sprint:
      Week 2014-4-02 to 2014-4-09

      Description

      UPDATE

      Add a literal function to hiera to allow characters to be included without special interpretation.

      Example: literal('%')

      ORIGINAL

      I have some data that contains Apache variables (such as %

      {SERVER_NAME}, %{QUERY_STRING}, etc )
      in a Hiera file (Yaml backend). However, when I retrieve the value, the string is replaced with
      empty value. For instance, for:

      myserver: http://%{SERVER_NAME}

      I got the value "http//"

      I found out that during the lookup in the YAML backend, the answer is parsed and Hiera
      tries to resolve the variable or replace it by an empty string if the variable is not found.

      We should have a way to escape this from any String or disable the variable resolution
      completely in the order to get the correct value.

      What about escaping with "\" character?
      So, if we have this data in Hiera file:

      myserver: http://%

      {SERVER_NAME}

      We get the correct string:

      myserver: http://%{SERVER_NAME}

        Attachments

          Issue Links

            Activity

            Hide
            guiwoda Guido Contreras Woda added a comment - - edited

            There's a simple workaround for this, if you really need to do it and you'd rather not change your manifest / template:

            foo: "%%{THIS_DOESNT_EXIST}{BAR}"
            

            Will output

            foo: %{BAR}
            

            As expected.

            Show
            guiwoda Guido Contreras Woda added a comment - - edited There's a simple workaround for this, if you really need to do it and you'd rather not change your manifest / template: foo: "%%{THIS_DOESNT_EXIST}{BAR}" Will output foo: %{BAR} As expected.
            Hide
            Savar Simon added a comment -

            this workaround doesn't work with hiera 1.3 I think.. looks like hiera 1.3 is doing multiple times a check on the string and will replace all variables and also all new "generated" variables afterwards as well.. at least in my case hiera 1.1 failed (exception), 1.2 worked and 1.3 gives me empty strings..

            Show
            Savar Simon added a comment - this workaround doesn't work with hiera 1.3 I think.. looks like hiera 1.3 is doing multiple times a check on the string and will replace all variables and also all new "generated" variables afterwards as well.. at least in my case hiera 1.1 failed (exception), 1.2 worked and 1.3 gives me empty strings..
            Hide
            chuck Charlie Sharpsteen added a comment -

            After reviewing this issue and the linked Redmine ticket, I can think of three possible implementations:

            Option 1: Backslash escaping

            Form:

            \%{some::interpolation}
            

            Pros:

            • Very familiar. Closely mirrors how Puppet variable interpolation is escaped.

            Cons:

            • Using a backslash as an escape character also requires implementing support for escaping backslashes. This may require turning the Hiera interpolator into a full-blown parser.
            • Would have a heavy impact on any existing data that contains Windows paths.

            Option 2: Doubled Sigil Escaping

            Form:

            %%{some::interpolation}
            

            Pros:

            • Less invasive than backslash escaping.

            Cons:

            • Introduces another syntactical difference between Hiera interpolation and Puppet interpolation.
            • Still not entirely backwards compatible.

            Option 3: Interpolation Function

            Form:

            %{escape('some::interpolation')}
            

            Pros:

            • Uses the new "interpolation function" support introduced in Hiera 1.3.0. Backwards compatible.

            Cons:

            • Verbose syntax.
            • Can't handle all cases. In particular, the regular expression used to parse interpolation functions means we cannot escape any string containing single or double quotation marks such as other interpolation functions:

            # This won't work:
            %{escape('scope("some::interpolation")')}
            


            Option 3 seems like the only choice that can be implemented without waiting for a 2.0 major version break. I have a working implementation, but would love to hear suggestions before throwing it up as a pull request.

            Show
            chuck Charlie Sharpsteen added a comment - After reviewing this issue and the linked Redmine ticket, I can think of three possible implementations: Option 1: Backslash escaping Form: \%{some::interpolation} Pros: Very familiar. Closely mirrors how Puppet variable interpolation is escaped. Cons: Using a backslash as an escape character also requires implementing support for escaping backslashes. This may require turning the Hiera interpolator into a full-blown parser. Would have a heavy impact on any existing data that contains Windows paths. Option 2: Doubled Sigil Escaping Form: %%{some::interpolation} Pros: Less invasive than backslash escaping. Cons: Introduces another syntactical difference between Hiera interpolation and Puppet interpolation. Still not entirely backwards compatible. Option 3: Interpolation Function Form: %{escape('some::interpolation')} Pros: Uses the new "interpolation function" support introduced in Hiera 1.3.0. Backwards compatible. Cons: Verbose syntax. Can't handle all cases. In particular, the regular expression used to parse interpolation functions means we cannot escape any string containing single or double quotation marks such as other interpolation functions: # This won't work: %{escape('scope("some::interpolation")')} Option 3 seems like the only choice that can be implemented without waiting for a 2.0 major version break. I have a working implementation , but would love to hear suggestions before throwing it up as a pull request.
            Hide
            Savar Simon added a comment -
            • I think a backslash is also already used by yaml in normal "" strings.. so I would avoid that
            • Interpolation function does not look nice and if this cannot handle everything ..
            • I would like to have %%.. but I understand, that this will break stuff from others..
            Show
            Savar Simon added a comment - I think a backslash is also already used by yaml in normal "" strings.. so I would avoid that Interpolation function does not look nice and if this cannot handle everything .. I would like to have %%.. but I understand, that this will break stuff from others..
            Hide
            andy Andrew Parker added a comment -

            Charlie Sharpsteen, I want to make sure that I'm understanding your Option 3 correctly. With that support, in order to do what is asked for in the description you'd need to write:

            myserver: "http://%{escape('%')}{SERVER_NAME}"
            

            I think a better name than escape would be literal, if I'm understanding the semantics of this correctly.

            You are right, though, that it is pretty verbose. Maybe there is a simpler way. What we really need is for a way of getting a literal % that is followed by an opening curly brace. This means that we really just need a syntax to produce a % since that syntax alone (as long as it doesn't end with a %) would be enough to avoid the interpolation. So, with that in mind, what about something much simpler:

            myserver: "http://%{%}{SERVER_NAME}"
            

            In this case an interpolation of just % produces a literal % value. This is mostly backwards compatible since it would only shadow a lookup for a variable named %, which I really doubt exists.

            Show
            andy Andrew Parker added a comment - Charlie Sharpsteen , I want to make sure that I'm understanding your Option 3 correctly. With that support, in order to do what is asked for in the description you'd need to write: myserver: "http://%{escape('%')}{SERVER_NAME}" I think a better name than escape would be literal , if I'm understanding the semantics of this correctly. You are right, though, that it is pretty verbose. Maybe there is a simpler way. What we really need is for a way of getting a literal % that is followed by an opening curly brace. This means that we really just need a syntax to produce a % since that syntax alone (as long as it doesn't end with a % ) would be enough to avoid the interpolation. So, with that in mind, what about something much simpler: myserver: "http://%{%}{SERVER_NAME}" In this case an interpolation of just % produces a literal % value. This is mostly backwards compatible since it would only shadow a lookup for a variable named % , which I really doubt exists.
            Hide
            chuck Charlie Sharpsteen added a comment -

            Andrew Parker Actually, my implementation looks like this:

            myserver: "http://%{escape('SERVER_NAME')}"
            

            That interpolation would expand to:

            myserver: "http://%{SERVER_NAME}"
            

            Your example goes in a different direction — one that is is more flexible than what I had implemented.

            Show
            chuck Charlie Sharpsteen added a comment - Andrew Parker Actually, my implementation looks like this: myserver: "http://%{escape('SERVER_NAME')}" That interpolation would expand to: myserver: "http://%{SERVER_NAME}" Your example goes in a different direction — one that is is more flexible than what I had implemented.
            Hide
            chrisportman Chris Portman added a comment -

            Hi Andrew Parker,

            In your last comment on PR #184 (https://github.com/puppetlabs/hiera/pull/184), are you saying that %

            {%} {Something literal}

            already works?

            Thanks,
            Chris

            Show
            chrisportman Chris Portman added a comment - Hi Andrew Parker , In your last comment on PR #184 ( https://github.com/puppetlabs/hiera/pull/184 ), are you saying that % {%} {Something literal} already works? Thanks, Chris
            Hide
            chuck Charlie Sharpsteen added a comment -

            Well... sort of.

            For example, given the following Hiera data:

            ---
            some_key: 'This string contains an %{%}{interpolation}'
            

            One could define the key "%" to return "%" and thus turn "%

            {%}" into an escape:

            # hiera some_key %=%
            This string contains an %{interpolation}
            



            However, this is mostly a hack and would be all sorts of messy to use from Puppet. The idea would be to update the Hiera interpolation code to treat "%{%}

            " as an escape which would be pretty much backwards compatible as you would have to be doing something crazy to define "%" as a key for Hiera interpolation.

            I'll look into re-organizing my escape function to achieve this.

            Show
            chuck Charlie Sharpsteen added a comment - Well... sort of. For example, given the following Hiera data: --- some_key: 'This string contains an %{%}{interpolation}' One could define the key "%" to return "%" and thus turn "% {%}" into an escape: # hiera some_key %=% This string contains an %{interpolation} However, this is mostly a hack and would be all sorts of messy to use from Puppet. The idea would be to update the Hiera interpolation code to treat "%{%} " as an escape which would be pretty much backwards compatible as you would have to be doing something crazy to define "%" as a key for Hiera interpolation. I'll look into re-organizing my escape function to achieve this.
            Hide
            chuck Charlie Sharpsteen added a comment -
            Show
            chuck Charlie Sharpsteen added a comment - Pull request submitted: https://github.com/puppetlabs/hiera/pull/185
            Hide
            chuck Charlie Sharpsteen added a comment -

            Also worth noting is that the prior workaround of:

            foo: "%%{THIS_DOESNT_EXIST}{BAR}"
            

            Now works again under Hiera 1.3.2 due to way HI-115 was resolved.

            Show
            chuck Charlie Sharpsteen added a comment - Also worth noting is that the prior workaround of: foo: "%%{THIS_DOESNT_EXIST}{BAR}" Now works again under Hiera 1.3.2 due to way HI-115 was resolved.
            Hide
            chuck Charlie Sharpsteen added a comment -

            And finally, I just noticed that a "%" sign followed by an empty interpolation works:

            ---
            some_key: 'This string contains an %%{}{interpolation}'
            

            That makes me wonder if we should just add some unit tests that enforce that behavior for "%%{}" and call this ticket solved.

            Show
            chuck Charlie Sharpsteen added a comment - And finally, I just noticed that a "%" sign followed by an empty interpolation works: --- some_key: 'This string contains an %%{}{interpolation}' That makes me wonder if we should just add some unit tests that enforce that behavior for "%%{}" and call this ticket solved.
            Hide
            chuck Charlie Sharpsteen added a comment -

            After discussion during the 3/26/2014 PR triage, it was decide to:

            • Proceed with the 'literal' function.
            • Drop the "% {%}

              " shorthand.

            • Leave "%%{}" as an undocumented and untested behavior.

            PR 185 updated and merged into master in commit 73da4ee.

            Show
            chuck Charlie Sharpsteen added a comment - After discussion during the 3/26/2014 PR triage, it was decide to: Proceed with the 'literal' function. Drop the "% {%} " shorthand. Leave "%%{}" as an undocumented and untested behavior. PR 185 updated and merged into master in commit 73da4ee .
            Hide
            kurt.wall Kurt Wall added a comment -

            Verified, I think, in puppet-3.5.0-0.1rc3.181.el6 (SHA=d659254a3d95c165ab0b2a104e03f9229079d9ce)

            # global.yaml
            ---
            key_1: 'Shorthand version: %{%}{sekrit}'
            key_2: 'Undocumented/untested version: %%{sekrit}'
            key_3: '% followed by empty interpolation: %%{}{sekrit}'
            key_4: %{literal('%')}{sekrit}
             
            # for n in 1 2 3 4; do hiera key_${n}; done
            Shorthand version: {sekrit}
            Undocumented/untested version: %
            % followed by empty interpolation: %{sekrit}
            %{sekrit}
            

            Show
            kurt.wall Kurt Wall added a comment - Verified, I think, in puppet-3.5.0-0.1rc3.181.el6 (SHA=d659254a3d95c165ab0b2a104e03f9229079d9ce) # global.yaml --- key_1: 'Shorthand version: %{%}{sekrit}' key_2: 'Undocumented/untested version: %%{sekrit}' key_3: '% followed by empty interpolation: %%{}{sekrit}' key_4: %{literal('%')}{sekrit}   # for n in 1 2 3 4; do hiera key_${n}; done Shorthand version: {sekrit} Undocumented/untested version: % % followed by empty interpolation: %{sekrit} %{sekrit}
            Hide
            chuck Charlie Sharpsteen added a comment -

            Looks good. Kurt Wall: Thanks for running through all the proposals discussed in this ticket and showing what the final behaviors are

            Show
            chuck Charlie Sharpsteen added a comment - Looks good. Kurt Wall : Thanks for running through all the proposals discussed in this ticket and showing what the final behaviors are
            Hide
            rnelson0@gmail.com Rob Nelson added a comment -

            Correcting resolved tickets to closed as needed (triage-a-thon).

            Show
            rnelson0@gmail.com Rob Nelson added a comment - Correcting resolved tickets to closed as needed (triage-a-thon).
            Hide
            rnelson0@gmail.com Rob Nelson added a comment -

            Unclosing 1.4.0 tickets.

            Show
            rnelson0@gmail.com Rob Nelson added a comment - Unclosing 1.4.0 tickets.
            Hide
            nanliu Nan Liu added a comment -

            This feature should be added to the documentation.

            Show
            nanliu Nan Liu added a comment - This feature should be added to the documentation.
            Hide
            chuck Charlie Sharpsteen added a comment -

            Nan Liu: literal('%') will be added to the documentation once Hiera 1.4.0 ships. As noted above, %%{} works by coincidence and has no test coverage, so it won't be documented as an expected behavior.

            Show
            chuck Charlie Sharpsteen added a comment - Nan Liu : literal('%') will be added to the documentation once Hiera 1.4.0 ships. As noted above, %%{} works by coincidence and has no test coverage, so it won't be documented as an expected behavior.
            Hide
            abegosum Aaron M. Bond added a comment -

            On puppet 3.6.2 with Hiera 1.3.4, the recommended escape solution of literal('%') isn't working for me with a string member of an array in my hiera configuration.

            Here's a sample of my hiera yaml:

            global.yaml

            additional_config:
                  - 'RewriteEngine on'
                  - 'RewriteCond %{HTTP_HOST} ^origin- [NC]'
            

            This yields the predictable:

              RewriteCond  ^origin- [NC]
            

            Trying the fix literal('%') doesn't seem to work as expected. See the following yaml and result (note that I've used double quotes for the string to avoid escape problems with the single quotes in the literal call):

            global.yaml

                additional_config:
                  - 'RewriteEngine on'
                  - "RewriteCond %{literal('%')}{HTTP_HOST} ^origin- [NC]"
            

            This yields:

              RewriteCond {HTTP_HOST} ^origin- [NC]
            

            For rigor, I also tried to manually escape the single quotes in the yaml:

            global.yaml

                additional_config:
                  - 'RewriteEngine on'
                  - 'RewriteCond %{literal(''%'')}{HTTP_HOST} ^origin- [NC]'
            

            This yields the same result as above.

            Finally, sadly, the workaround of splitting the interpolation with an empty interpolation DOES work:

            global.yaml

                additional_config:
                  - 'RewriteEngine on'
                  - 'RewriteCond %%{}{HTTP_HOST} ^origin- [NC]'
            

            Yields:

              RewriteCond %{HTTP_HOST} ^origin- [NC]
            

            So, just curious- the above comment from Charlie seems to indicate that this works in Hiera 1.4. The target version is 2.0 on the ticket. 1) Was this an actual change or just an addition to the testing and documentation for a newer version? If the latter, does anyone know why the prescribed solution isn't working for me? And 2) Is this change (documentation OR code) going to be included in 1.4 or 2.0?

            Thanks all, and sorry for the lack of brevity.

            Show
            abegosum Aaron M. Bond added a comment - On puppet 3.6.2 with Hiera 1.3.4, the recommended escape solution of literal('%') isn't working for me with a string member of an array in my hiera configuration. Here's a sample of my hiera yaml: global.yaml additional_config: - 'RewriteEngine on' - 'RewriteCond %{HTTP_HOST} ^origin- [NC]' This yields the predictable: RewriteCond ^origin- [NC] Trying the fix literal('%') doesn't seem to work as expected. See the following yaml and result (note that I've used double quotes for the string to avoid escape problems with the single quotes in the literal call): global.yaml additional_config: - 'RewriteEngine on' - "RewriteCond %{literal('%')}{HTTP_HOST} ^origin- [NC]" This yields: RewriteCond {HTTP_HOST} ^origin- [NC] For rigor, I also tried to manually escape the single quotes in the yaml: global.yaml additional_config: - 'RewriteEngine on' - 'RewriteCond %{literal(''%'')}{HTTP_HOST} ^origin- [NC]' This yields the same result as above. Finally, sadly, the workaround of splitting the interpolation with an empty interpolation DOES work: global.yaml additional_config: - 'RewriteEngine on' - 'RewriteCond %%{}{HTTP_HOST} ^origin- [NC]' Yields: RewriteCond %{HTTP_HOST} ^origin- [NC] So, just curious- the above comment from Charlie seems to indicate that this works in Hiera 1.4. The target version is 2.0 on the ticket. 1) Was this an actual change or just an addition to the testing and documentation for a newer version? If the latter, does anyone know why the prescribed solution isn't working for me? And 2) Is this change (documentation OR code) going to be included in 1.4 or 2.0? Thanks all, and sorry for the lack of brevity.
            Hide
            abegosum Aaron M. Bond added a comment - - edited

            Addendum- looking at the commits, 1.3.4 should at least have support for

            %{literal('%')}

            due to the fact that the commit mentioned here (https://github.com/puppetlabs/hiera/commit/73da4ee) occurred before the 1.3.4 tag. Not sure why my syntax isn't working.

            Again, I may be missing something obvious here- I'm relatively new to hiera.

            Show
            abegosum Aaron M. Bond added a comment - - edited Addendum- looking at the commits, 1.3.4 should at least have support for %{literal('%')} due to the fact that the commit mentioned here ( https://github.com/puppetlabs/hiera/commit/73da4ee ) occurred before the 1.3.4 tag. Not sure why my syntax isn't working. Again, I may be missing something obvious here- I'm relatively new to hiera.
            Hide
            chuck Charlie Sharpsteen added a comment -

            Aaron M. Bond: Commit 73da4ee was made to the master branch, 1.3.4 was cut from stable and does not include this feature. The literal function will be released in the next Hiera feature release.

            For 1.3.4, the %%{} workaround is currently the only viable solution that I know of.

            Show
            chuck Charlie Sharpsteen added a comment - Aaron M. Bond : Commit 73da4ee was made to the master branch, 1.3.4 was cut from stable and does not include this feature. The literal function will be released in the next Hiera feature release. For 1.3.4, the %%{} workaround is currently the only viable solution that I know of.
            Hide
            chuck Charlie Sharpsteen added a comment -

            Also, Hiera 1.4 has been dropped in favor of shipping a Hiera 2.0.

            Show
            chuck Charlie Sharpsteen added a comment - Also, Hiera 1.4 has been dropped in favor of shipping a Hiera 2.0.
            Hide
            brettswift Brett Swift added a comment -

            I get an error when this literal function appears twice in the same hierarchy tier.

            I have this:

             - "cluster/%{literal('%')}{::cluster}/%{literal('%')}{role}"
            

            and the error I get is:

            Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Function Call, Error from DataBinding 'hiera' while looking up 'puppet::data_hierarchy': Detected in [literal('%')] at ~~(class that is doing the lookup~~

            If i take either the first literal function or the last literal function out it fails.

            Show
            brettswift Brett Swift added a comment - I get an error when this literal function appears twice in the same hierarchy tier. I have this: - "cluster/%{literal('%')}{::cluster}/%{literal('%')}{role}" and the error I get is: Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Evaluation Error: Error while evaluating a Function Call, Error from DataBinding 'hiera' while looking up 'puppet::data_hierarchy': Detected in [literal('%')] at ~~ (class that is doing the lookup ~~ If i take either the first literal function or the last literal function out it fails.
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            Brett Swift Please open a new ticket with the problem you found, and describe details about version used etc. This ticket is already closed.

            Show
            henrik.lindberg Henrik Lindberg added a comment - Brett Swift Please open a new ticket with the problem you found, and describe details about version used etc. This ticket is already closed.
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            Steve Traylen you assigned yourself to this closed ticket. Was that by accident?

            Show
            henrik.lindberg Henrik Lindberg added a comment - Steve Traylen you assigned yourself to this closed ticket. Was that by accident?

              People

              • Assignee:
                Unassigned
                Reporter:
                redmine.exporter redmine.exporter
              • Votes:
                4 Vote for this issue
                Watchers:
                23 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support