Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-6432

Puppet::Util::Execution.execute cannot handle Unicode output on Windows

    Details

    • Type: Bug
    • Status: Ready for Engineering
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: PUP 4.5.2
    • Fix Version/s: PUP 5.y
    • Component/s: Windows
    • Labels:
    • Template:
    • Acceptance Criteria:
      Hide
      • Executing a simple cmd.exe based exec should properly return Unicode characters
      • It may be necessary to take special steps in using a console that properly supports Unicode (cmd.exe hosted in ConEmu will behave properly for instance)
      • Ensure that shell behavior is not broken on Linux / other operating systems (where LC_ALL may be used)
      Show
      Executing a simple cmd.exe based exec should properly return Unicode characters It may be necessary to take special steps in using a console that properly supports Unicode (cmd.exe hosted in ConEmu will behave properly for instance) Ensure that shell behavior is not broken on Linux / other operating systems (where LC_ALL may be used)
    • Epic Link:
    • Team:
      Coremunity
    • Story Points:
      5

      Description

      When shelling out to call external applications, Puppet does not ensure that stdin / stdout / stderr streams are capable of handling Unicode.

      On Ruby 2.1.9 I was able to reproduce this pretty easily. A fix should include specifying encodings to the file streams that get ultimately passed to the CreateProcessW Windows API call.

      Glenn Sarti has produced some tests demonstrating the behavior:

        describe "#execute (Windows) STDOUT", :if => Puppet.features.microsoft_windows? do
          let(:utf8text) do
            "utf8testfile" + [195, 164].pack('c*').force_encoding(Encoding::UTF_8)
          end
          let(:temputf8filename) do
            script_containing(utf8text, :windows => "@ECHO OFF\nECHO #{utf8text}\nEXIT 100")
          end
       
          it "should execute with non-english characters in command line" do
            result = Puppet::Util::Execution.execute("cmd /c \"#{temputf8filename}\"", :failonfail => false)
            expect(temputf8filename.encoding.name).to eq('UTF-8')
            expect(result.exitstatus).to eq(100)
            expect(result).to eq(utf8text)
          end
        end
      

      Current results:

      DL is deprecated, please use Fiddle
      Run options:
        include {:locations=>{"./spec/integration/util/execution_spec.rb"=>[43]}}
        exclude {:broken=>true, :benchmark=>true}
       
      Puppet::Util::Execution
        #execute (Windows) STDOUT
          should execute with non-english characters in command line (FAILED - 1)
       
      Failures:
       
        1) Puppet::Util::Execution#execute (Windows) STDOUT should execute with non-en
      glish characters in command line
           Failure/Error: expect(result).to eq(utf8text)
       
             expected: "utf8testfile\u00E4"
                  got: "utf8testfileä\n"
       
             (compared using ==)
       
             Diff:
             @@ -1,2 +1,2 @@
             -utf8testfileä
             +utf8testfileä
           # ./spec/integration/util/execution_spec.rb:48:in `block (3 levels) in <top
       (required)>'
       
      Finished in 1.08 seconds (files took 1.95 seconds to load)
      1 example, 1 failure
      

      Files are created in Puppet::Util::Execution.execute at https://github.com/puppetlabs/puppet/blob/master/lib/puppet/util/execution.rb#L187-L189 and in multiple locations in Uniquefile at https://github.com/puppetlabs/puppet/blob/master/lib/puppet/file_system/uniquefile.rb#L39-L53

      I have been able to solve the problem, but the current solution is unsatisfactory, as it requires changes to the file modes and the global Windows codepage to be set with chcp 65001 to Unicode.

        Attachments

          Issue Links

            Activity

              jsd-sla-details-panel

                People

                • Assignee:
                  Unassigned
                  Reporter:
                  ethan Ethan Brown
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  6 Start watching this issue

                  Dates

                  • Created:
                    Updated: