Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-7579

Tags do not support UTF-8 characters

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: PUP 5.0.0
    • Component/s: None
    • Template:
    • Acceptance Criteria:
      Hide

      Solves issue with concat module described in MODULES-4684.

      Show
      Solves issue with concat module described in MODULES-4684.
    • Epic Link:
    • Team:
      Agent
    • Story Points:
      2
    • Sprint:
      Agent 2017-06-14
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Hide
      Puppet will now accept unicode tags, useful if you want to run a subset of resources in a catalog using a tag name that isn't US-ASCII. This also allows the concat::fragment defines from the concat module to write to file paths that are not US-ASCII. Marking as a bug fix since puppet accepts unicode for other inputs like resource name/titles, but not tags. We'll also need to update the public docs that specify the valid tag regex.
      Show
      Puppet will now accept unicode tags, useful if you want to run a subset of resources in a catalog using a tag name that isn't US-ASCII. This also allows the concat::fragment defines from the concat module to write to file paths that are not US-ASCII. Marking as a bug fix since puppet accepts unicode for other inputs like resource name/titles, but not tags. We'll also need to update the public docs that specify the valid tag regex.
    • QA Risk Assessment:
      No Action
    • QA Risk Assessment Reason:
      Check in includes tests for the change

      Description

      The Regex for tag validation (at https://github.com/puppetlabs/puppet/blob/master/lib/puppet/util/tagging.rb#L4) is strictly ASCII:

      ValidTagRegex = /\A[0-9A-Za-z_][0-9A-Za-z_:.-]*\Z/

      This prevents the usage of tags in the following case:

      puppet apply -e 'notify { "hi": tag => "_メインディレクトリ_醸造所" }'

      With a stack like:

      Error: Invalid tag '_メインディレクトリ_醸造所' at /root/foo.pp:1 on node h3a12zjzc6608rh.delivery.puppetlabs.net
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/tagging.rb:27:in `block in tag'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/tagging.rb:13:in `each'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/tagging.rb:13:in `tag'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:185:in `set_parameter'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:385:in `block in extract_parameters'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:381:in `each'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:381:in `extract_parameters'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:368:in `initialize'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:130:in `initialize'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_resource_support.rb:38:in `new'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_resource_support.rb:38:in `block in create_resources'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_resource_support.rb:37:in `map'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_resource_support.rb:37:in `create_resources'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:321:in `create_resources'
      		 
       
      		 
      ▽
      		 
      require 'puppet/util/tag_set'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:842:in `block in eval_ResourceExpression'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:839:in `map'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:839:in `eval_ResourceExpression'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:48:in `block in visit_this'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `each'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `visit_this'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:71:in `visit_this_1'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
      		 
      /root/foo.pp:in `stack'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/puppet_stack.rb:30:in `eval'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/puppet_stack.rb:30:in `stack'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:715:in `eval_Program'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:48:in `block in visit_this'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `each'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `visit_this'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:71:in `visit_this_1'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:132:in `evaluate'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:184:in `evaluate_code'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:635:in `evaluate_main'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:174:in `block (2 levels) in compile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:174:in `block in compile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:294:in `override'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:162:in `compile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:33:in `compile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:268:in `block (2 levels) in compile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:266:in `block in compile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:224:in `block in benchmark'
      		 
      /opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:264:in `compile'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:55:in `find'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:256:in `block in main'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:294:in `override'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:225:in `main'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:358:in `block in run'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:542:in `exit_on_fail'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:358:in `run'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
      		 
      /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
      		 
      /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

      A more lenient and Unicode friendly approach is to use the Unicode friendly word regex like:

      ValidTagRegex = /\A[[:word:]][[[:word:]]:.-]*\Z/

      This appears necessary to fix the concat module issue reported at MODULES-4684

      The open question around this is whether or not tags are restricted in this fashion due to some important underlying reason.

      Thomas Hallgren spent some time optimizing regular expressions for tags in https://github.com/puppetlabs/puppet/commit/0badfce72bafc767bf3099718c63324f51108996 and the original check (based on words) goes back 10 years to https://github.com/puppetlabs/puppet/commit/f98be4a7198326b26f1072c401b3e337f340db40#diff-ac9d5450f67274b038705194387c3907R31

        Attachments

          Issue Links

          blocks

          Bug - A problem which impairs or prevents the functions of the product. MODULES-4864 vcsrepo ignores "force" for git repos

          • Major - Major loss of function.
          • Resolved
          relates to

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. PUP-7536 Improve Tag Support

          • Normal - Regular issue, some loss of functionality under specific circumstances.
          • Open

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              ethan Ethan Brown
              QA Contact:
              Eric Delaney
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Zendesk Support