Uploaded image for project: 'Hiera'
  1. Hiera
  2. HI-519

Hiera Filecache doesn't support encoding / doesn't load JSON backend files as UTF-8


    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Fixed
    • Affects Version/s: HI 3.2.0
    • Fix Version/s: HI 3.3.1
    • Component/s: None
    • Labels:
    • Template:
    • Acceptance Criteria:

      Create a UTF8 JSON file that contains Unicode characters and read it with Hiera

      Create a UTF8 JSON file that contains Unicode characters and read it with Hiera
    • Team:
    • Story Points:
    • Sprint:
      AP 2017-01-25
    • Release Notes:
      Bug Fix
    • Release Notes Summary:
      Hiera did not read data files into cache using utf-8 which could lead to encoding problems occurring much later. Now the data files (yaml, json) are read with utf-8 encoding.
    • QA Risk Assessment:
      No Action
    • QA Risk Assessment Reason:
      Covered in spec; low risk of data loss or downtime


      When using a YAML backend, files are automatically loaded as UTF-8 by default, thanks to Ruby code at https://github.com/ruby/ruby/blob/trunk/ext/psych/lib/psych.rb#L474-L476

      However, when loading JSON files, Puppet simply calls Ruby's File.read without specifying an encoding. On Windows this will cause problems trying to read UTF8 files, as Ruby will attempt to instead use the default external encoding.

      See code at https://github.com/puppetlabs/hiera/blob/4e85a733cbce09634a59005cda6ae169648c67ab/lib/hiera/filecache.rb#L50-L61

      The fix should be to specify :encoding => 'bom|utf8' as an option to File.read.

      If there are other consumers of the FileCache class, then the read_file method should probably take an additional options parameter so that callers may override the default behavior.

      It might also be useful to scan the Hiera code for other such calls to File where the encoding has not been explicitly specified.

      Additional Acceptance Criteria copied from HI-520:

      • The Hiera yaml_load_file delegates to Rubys loading of YAML files which specifies :encoding => 'bom|utf8' when loading files. This isn't documented well, and we don't have any protection against regressions in the ability of Hiera to load YAML files (for instance, if another YAML parser was selected). Some additional specs should be added to cover this situation.


          Issue Links



              • Assignee:
                ethan Ethan Brown
              • Votes:
                0 Vote for this issue
                9 Start watching this issue


                • Created:

                  Zendesk Support