Uploaded image for project: 'Beaker'
  1. Beaker
  2. BKR-652

Beaker: Internal SSH library does not properly handle Unicode file names when SCP'ing files

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Template:
    • Epic Link:
    • Team:
      Quality Engineering

      Description

      In an effort to test some of the behavior of writing Unicode file names with Beaker, I discovered that the implementation of the SSH library Beaker relies on has some bad behavior with how it builds up the SCP command.

      I have validated that using command line tools to send the file to a regular Windows host with Cygwin work successfully:

      [puppetlabs-dsc] curl -d --url vmpooler.delivery.puppetlabs.net/vm/win-2012r2-x86_64
      {
        "ok": true,
        "win-2012r2-x86_64": {
          "hostname": "ajd859bkz5m7s6q",
          "ok": true
        },
        "domain": "delivery.puppetlabs.net"
      }%                                                                                                                
       
      [puppetlabs-dsc] scp /Users/Iristyle/source/puppetlabs-dsc/Gemfile  Administrator@ajd859bkz5m7s6q.delivery.puppetlabs.net:/cygdrive/c/Windows/Temp/ㅲㅂㅄㅉㅑㅧㅩㅙㅘㅀㅅㅶㅅ
      The authenticity of host 'ajd859bkz5m7s6q.delivery.puppetlabs.net (10.32.125.138)' can't be established.
      ECDSA key fingerprint is SHA256:ltZ3Nvso36chchDBcD6guGGqqRdA2Wun1/r3np2WDCg.
      Are you sure you want to continue connecting (yes/no)? yes
      Warning: Permanently added 'ajd859bkz5m7s6q.delivery.puppetlabs.net,10.32.125.138' (ECDSA) to the list of known hosts.
      Gemfile                                                                         100% 5414     5.3KB/s   00:00
      

      I have also validated that a Windows host with Bitvise SSH + Git Bash is accepting unicode file paths correctly when sending files over the command line:

      [puppetlabs-dsc] scp /Users/Iristyle/source/puppetlabs-dsc/Gemfile  Administrator@j7qrsglshltemmr.delivery.puppetlabs.net:/Windows/Temp/ㅲㅂㅄㅉㅑㅧㅩㅙㅘㅀㅅㅶㅅ
      The authenticity of host 'j7qrsglshltemmr.delivery.puppetlabs.net (10.32.125.101)' can't be established.
      ECDSA key fingerprint is SHA256:1p6UzvWHD4XcDdae8ND2yIQAezMiIClYW97pi3uop9w.
      Are you sure you want to continue connecting (yes/no)? yes
      Warning: Permanently added 'j7qrsglshltemmr.delivery.puppetlabs.net,10.32.125.101' (ECDSA) to the list of known hosts.
      Gemfile                                                                         100% 5414     5.3KB/s   00:00
      

      However, within a Beaker session, the same file path fails to copy with the following stack trace:

      localhost $ scp /var/folders/1b/rzv96bhd5h9bwtdnz7g0xmlm0000gn/T/beaker20151210-10712-sae1z win-2008r2-x86_64-agent:/cygdrive/c/ㅲㅂㅄㅉㅑㅧㅩㅙㅘㅀㅅㅶㅅ {:ignore => }
      RuntimeError: scp: /cygdrive/c/\ㅲ\ㅂ\ㅄ\ㅉ\ㅑ\ㅧ\ㅩ\ㅙ\ㅘ\ㅀ\ㅅ\ㅶ\ㅅ: Error opening file: No such file or directory.
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-scp-1.2.1/lib/net/scp.rb:398
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-scp-1.2.1/lib/net/scp.rb:369
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/channel.rb:311
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/channel.rb:311
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:223
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:223
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:223
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:206
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/session.rb:170
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/connection/channel.rb:269
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-scp-1.2.1/lib/net/scp.rb:284
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/ssh_connection.rb:281
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/host.rb:366
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/dsl/helpers/host_helpers.rb:192
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/shared/host_manager.rb:115
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/shared/host_manager.rb:104
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/shared/host_manager.rb:103
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/shared/host_manager.rb:103
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/dsl/patterns.rb:35
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/dsl/helpers/host_helpers.rb:187
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/dsl/helpers/host_helpers.rb:258
      /usr/local/Cellar/rbenv/0.4.0/versions/1.9.3-p551/lib/ruby/1.9.1/tempfile.rb:320
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/dsl/helpers/host_helpers.rb:252
      /Users/Iristyle/source/puppetlabs-dsc/tests/acceptance/tests/basic_dsc_resources/file/file_valid_unicode_source.rb:38
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/test_case.rb:128
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/test_case.rb:128
      /usr/local/Cellar/rbenv/0.4.0/versions/1.9.3-p551/lib/ruby/1.9.1/benchmark.rb:295
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/test_case.rb:125
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/test_suite.rb:311
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/test_suite.rb:308
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/test_suite.rb:308
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/test_suite.rb:357
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/cli.rb:157
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/lib/beaker/cli.rb:99
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bundler/gems/beaker-e648991bc5ff/bin/beaker:6
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bin/beaker:23
      /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/bin/beaker:23
      

      I was able to debug the code and I see that there are a couple of bad things that the SCP library code is doing:

      In a debugging session, I unmangled the string, ensured it was still UTF-8 and observed it failing when the attempt was made to append UTF-8 data to an ASCII-8BIT string.

      From: /Users/Iristyle/source/puppetlabs-dsc/.bundle/gems/ruby/1.9.1/gems/net-ssh-2.9.2/lib/net/ssh/buffer.rb @ line 284 Net::SSH::Buffer#write:

      283: def write(*data)
      => 284: data.each

      Unknown macro: { |datum| @content << datum }

      285: self
      286: end

      [44] pry(#<Net::SSH::Buffer>)> data
      => ["scp -t /windows/temp/b626ce81-a0f2-4c41-813c-381d082dce92/ㅲㅂㅄㅉㅑㅧㅩㅙㅘㅀㅅㅶㅅ"]
      [45] pry(#<Net::SSH::Buffer>)> @content
      => "b\x00\x00\b\xDD\x00\x00\x00\x04exec\x01\x00\x00\x00a"
      [46] pry(#<Net::SSH::Buffer>)> @content.encoding
      => #<Encoding:ASCII-8BIT>
      [47] pry(#<Net::SSH::Buffer>)> data[0].encoding
      => #<Encoding:UTF-8>

      To fix this will probably require patches to net-scp and/or net-ssh.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                ethan Ethan Brown
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support