Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-7579

Tags do not support UTF-8 characters

          Details

          • Type: Bug
          • Status: Closed
          • Priority: Normal
          • Resolution: Fixed
          • Affects Version/s: None
          • Fix Version/s: PUP 5.0.0
          • Component/s: None
          • Template:
          • Acceptance Criteria:
            Hide

            Solves issue with concat module described in MODULES-4684.

            Show
            Solves issue with concat module described in MODULES-4684.
          • Epic Link:
          • Team:
            Agent
          • Story Points:
            2
          • Sprint:
            Agent 2017-06-14
          • Release Notes:
            Bug Fix
          • Release Notes Summary:
            Hide
            Puppet will now accept unicode tags, useful if you want to run a subset of resources in a catalog using a tag name that isn't US-ASCII. This also allows the concat::fragment defines from the concat module to write to file paths that are not US-ASCII. Marking as a bug fix since puppet accepts unicode for other inputs like resource name/titles, but not tags. We'll also need to update the public docs that specify the valid tag regex.
            Show
            Puppet will now accept unicode tags, useful if you want to run a subset of resources in a catalog using a tag name that isn't US-ASCII. This also allows the concat::fragment defines from the concat module to write to file paths that are not US-ASCII. Marking as a bug fix since puppet accepts unicode for other inputs like resource name/titles, but not tags. We'll also need to update the public docs that specify the valid tag regex.
          • QA Risk Assessment:
            No Action
          • QA Risk Assessment Reason:
            Check in includes tests for the change

            Description

            The Regex for tag validation (at https://github.com/puppetlabs/puppet/blob/master/lib/puppet/util/tagging.rb#L4) is strictly ASCII:

            ValidTagRegex = /\A[0-9A-Za-z_][0-9A-Za-z_:.-]*\Z/

            This prevents the usage of tags in the following case:

            puppet apply -e 'notify { "hi": tag => "_メインディレクトリ_醸造所" }'

            With a stack like:

            Error: Invalid tag '_メインディレクトリ_醸造所' at /root/foo.pp:1 on node h3a12zjzc6608rh.delivery.puppetlabs.net
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/errors.rb:106:in `fail'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/tagging.rb:27:in `block in tag'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/tagging.rb:13:in `each'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/tagging.rb:13:in `tag'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:185:in `set_parameter'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:385:in `block in extract_parameters'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:381:in `each'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:381:in `extract_parameters'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource.rb:368:in `initialize'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:130:in `initialize'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_resource_support.rb:38:in `new'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_resource_support.rb:38:in `block in create_resources'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_resource_support.rb:37:in `map'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_resource_support.rb:37:in `create_resources'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/runtime3_support.rb:321:in `create_resources'
            		 
             
            		 
            ▽
            		 
            require 'puppet/util/tag_set'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:842:in `block in eval_ResourceExpression'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:839:in `map'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:839:in `eval_ResourceExpression'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:48:in `block in visit_this'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `each'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `visit_this'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:71:in `visit_this_1'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
            		 
            /root/foo.pp:in `stack'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/puppet_stack.rb:30:in `eval'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/puppet_stack.rb:30:in `stack'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:715:in `eval_Program'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:48:in `block in visit_this'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `each'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:42:in `visit_this'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/visitor.rb:71:in `visit_this_1'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/evaluator/evaluator_impl.rb:82:in `evaluate'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/pops/parser/evaluating_parser.rb:63:in `evaluate'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast/pops_bridge.rb:132:in `evaluate'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/ast.rb:31:in `safeevaluate'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/resource/type.rb:184:in `evaluate_code'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:81:in `block in evaluate'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/resource.rb:73:in `evaluate'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:635:in `evaluate_main'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:174:in `block (2 levels) in compile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:174:in `block in compile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:294:in `override'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:162:in `compile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/parser/compiler.rb:33:in `compile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:268:in `block (2 levels) in compile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:266:in `block in compile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:224:in `block in benchmark'
            		 
            /opt/puppetlabs/puppet/lib/ruby/2.1.0/benchmark.rb:294:in `realtime'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:223:in `benchmark'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:264:in `compile'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:55:in `find'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:194:in `find'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:256:in `block in main'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:65:in `override'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:294:in `override'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:225:in `main'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application/apply.rb:170:in `run_command'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:358:in `block in run'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util.rb:542:in `exit_on_fail'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/application.rb:358:in `run'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:132:in `run'
            		 
            /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/command_line.rb:72:in `execute'
            		 
            /opt/puppetlabs/puppet/bin/puppet:5:in `<main>'

            A more lenient and Unicode friendly approach is to use the Unicode friendly word regex like:

            ValidTagRegex = /\A[[:word:]][[[:word:]]:.-]*\Z/

            This appears necessary to fix the concat module issue reported at MODULES-4684

            The open question around this is whether or not tags are restricted in this fashion due to some important underlying reason.

            Thomas Hallgren spent some time optimizing regular expressions for tags in https://github.com/puppetlabs/puppet/commit/0badfce72bafc767bf3099718c63324f51108996 and the original check (based on words) goes back 10 years to https://github.com/puppetlabs/puppet/commit/f98be4a7198326b26f1072c401b3e337f340db40#diff-ac9d5450f67274b038705194387c3907R31

              Attachments

                Issue Links

                blocks

                Bug - A problem which impairs or prevents the functions of the product. MODULES-4864 vcsrepo ignores "force" for git repos

                • Major - Major loss of function.
                • Resolved
                relates to

                Epic - Created by JIRA Software - do not edit or delete. Issue type for a big user story that needs to be broken down. PUP-7536 Improve Tag Support

                • Normal - Regular issue, some loss of functionality under specific circumstances.
                • Open

                  Activity

                    People

                    • Assignee:
                      Unassigned
                      Reporter:
                      ethan Ethan Brown
                      QA Contact:
                      Eric Delaney
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      3 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved:

                        Zendesk Support