Uploaded image for project: 'Puppet'
  1. Puppet
  2. PUP-6296

Computing checksum slow on big files

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Normal
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Environment:

      Merged a PR in kahless
      Using puppet agent version
      Puppet: 4.4.2 in production
      Facter: 3.1.6
      Ruby: 2.1.8

    • Template:
    • Acceptance Criteria:
      Hide

      the checksum show be quick - seconds instead of minutes

      Show
      the checksum show be quick - seconds instead of minutes

      Description

      Part of this puppetlabs-modules PR (https://github.com/puppetlabs/puppetlabs-modules/pull/5837) there is a purge resource and it will compute the checksum. It seems to be extremely slow for big files

      output of puppet agent -t

      Info: /Stage[main]/Apache::Purge/File[/var/log/apache2]: Recursively backing up to filebucket
      Info: Computing checksum on file /var/log/apache2/jenkins_proxy_jenkins-enterprise.delivery.puppetlabs.net_access.log
      Info: /Stage[main]/Apache::Purge/File[/var/log/apache2]: Filebucketed /var/log/apache2/jenkins_proxy_jenkins-enterprise.delivery.puppetlabs.net_access.log to puppet with sum 2c4aa061e17de20c37e96bbfc9a35099
      

      The second line Starting with 'Computing checksum' was running for about 10-15 minutes... what is our implementation of that MD5 checksum?
      The file is about 2GB of apache access logs.

      I'm not sure what else you need to reproduce the issue, but let me know.

      Info: Stage[main]: Unscheduling all events on Stage[main]
      Notice: Applied catalog in 2239.88 seconds
      

      When the other runs with small files were about 400 seconds max

      See https://peconsole.ops.puppetlabs.net/#/cm/report/e10b04f4b6248f3d7175182ba985440da939a934/logs

        Attachments

          Activity

            People

            Assignee:
            samuel Samuel Beaulieu
            Reporter:
            samuel Samuel Beaulieu
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Zendesk Support