Uploaded image for project: 'Puppet Server'
  1. Puppet Server
  2. SERVER-2662

Puppet Server CA always creates type 1 authority key identifiers



    • Template:
      PUP Bug Template
    • Team:
    • Story Points:
    • Method Found:
      Customer Feedback
    • Zendesk Ticket IDs:
    • Zendesk Ticket Count:
    • QA Risk Assessment:
      Needs Assessment


      Multiple types of key identifier exist, but Puppet server always uses type 1.
      This behavior makes the CA unable to properly use intermediate CA certificates
      issued with type 2 identifiers — such as those created by Vault PKI and
      AWS ACM.

      During validation of items signed by a CA, such as Certificates or CRLs,
      the Authority Key Identifier field of the signed item is matched against the
      Subject Key Identifier field of the CA cert. These matches create the links
      in a chain along which validation will proceed to verify signatures, dates,
      authorized uses, etc. If the key identifiers fail to match, most software
      will throw errors like "path building failed" or "issuer cannot be found".

      RFC 5280 defines two types of key identifier:

      • Type 1: A 160 bit SHA-1 hash of the subjectPublicKey of a certificate.
      • Type 2: The four-bit string 0100 (which smells like a version number that
        will increment in the future) followed by 60 bits of the SHA-1 hash of
        the subjectPublicKey of a certificate.

      Puppet Server CA always computes a type 1 key identifier from the subjectPublicKey
      of its CA certificate when populating the Authority Key identifier field of any
      certificate or CRL that it signs. This is incorrect behavior when the CA
      operates as an intermediate and will result in validation failures if the
      intermediate CA certificate was not issued with a type 1 identifier. Instead of
      computing a type 1 identifier for each operation, the CA should copy the value
      of the Subject Key Identifier field that was signed into its certificate. This
      would make the CA compatible with type 1, type 2, or other identifier types
      standardized in the future.

      Reproduction Case

      • Install Puppet Server 6 on CentOS 7:

      yum install -y http://yum.puppetlabs.com/puppet6-release-el-7.noarch.rpm
      yum install -y puppetserver

      • Install Vault and configure it to run in dev mode as a service:

      source /etc/profile.d/puppet-agent.sh
      puppet module install puppet/archive --version 4.3.0
      puppet apply <<EOF
      # Quick and very dirty Vault installation.
      # Don't ever use this anywhere near production ;)
      package { 'unzip':
        ensure => 'present',
      archive { '/tmp/vault_1.2.3_linux_amd64.zip':
        source       => 'https://releases.hashicorp.com/vault/1.2.3/vault_1.2.3_linux_amd64.zip',
        extract      => true,
        extract_path => '/usr/local/sbin',
        creates      => '/usr/local/sbin/vault',
        require      => Package['unzip']
      file { 'vault-dev.service':
        path    => '/etc/systemd/system/vault-dev.service',
        content => @(EOC),
        Description=Vault development server
        ExecStart=/usr/local/sbin/vault server -dev -dev-listen-address=
        | EOC
      exec { 'refresh systemd':
        command     => '/usr/bin/systemctl daemon-reload',
        refreshonly => true,
        subscribe   => File['vault-dev.service'],
      service { 'vault-dev':
        ensure    => running,
        subscribe => Exec['refresh systemd'],

      • Configure Vault PKI with a root certificate:

      export VAULT_ADDR=''
      # Ref: https://www.vaultproject.io/docs/secrets/pki/index.html
      vault secrets enable pki
      vault secrets tune -max-lease-ttl=8760h pki
      vault write pki/root/generate/internal \
        common_name=root-ca.puppet.test \

      • Configure Puppet Server with an intermediate CA issued by Vault

      mkdir intermediate_ca && cd intermediate_ca
      cat <<EOF > openssl.cnf
      [ req ]
      prompt = no
      distinguished_name = req_distinguished_name
      req_extensions = v3_req
      [ req_distinguished_name ]
      commonName = PuppetCA
      [ v3_req ]
      subjectAltName = DNS:intermediate-ca.puppet.test
      /opt/puppetlabs/puppet/bin/openssl req -config openssl.cnf \
        -nodes -new -newkey rsa:4096 \
        -keyout puppet_ca_key.pem \
        -out puppet_ca_csr.pem
      export VAULT_ADDR=''
      vault write pki/root/sign-intermediate \
        csr=@puppet_ca_csr.pem \
        format=pem_bundle \
        ttl=4380h > ca-bundle.pem
      vault read /pki/cert/crl > crls.pem
      puppetserver ca import --cert-bundle ca-bundle.pem --crl-chain crls.pem --private-key puppet_ca_key.pem

      • Start Puppet Server and run the agent to verify everything works:

      puppet config set server $(hostname -f)
      systemctl start puppetserver
      puppet agent -t

      • Print the Authority Key Identifier for the CRL and the Subject Key Identifier for the intermediate CA:

      /opt/puppetlabs/puppet/bin/openssl crl \
        -in $(puppet config print hostcrl) \
        -text -noout | grep --after 1 'Authority Key'
      /opt/puppetlabs/puppet/bin/openssl x509 \
        -in $(puppet config print localcacert) \
        -text -noout | grep --after 1 'Subject Key'

      • Generate and revoke a test certificate:

      puppetserver ca generate --certname foo.puppet.test
      puppetserver ca clean --certname foo.puppet.test

      • Check that agent still works and the Authority Key Identifier has
        not changed:

      puppet agent -t
      /opt/puppetlabs/puppet/bin/openssl crl \
        -in $(puppet config print hostcrl) \
        -text -noout | grep --after 1 'Authority Key'


      Puppet fails with a missing CRL error:

      [root@sx5txzicd4q58j6 intermediate_ca]# puppet agent -t
      Error: The CRL issued by 'CN=PuppetCA' is missing
      Error: Could not run: The CRL issued by 'CN=PuppetCA' is missing

      The Authority Key Identifier has changed from what it was before the run,
      which matched the Subject Key Identifier of the intermediate CA cert:

      [root@sx5txzicd4q58j6 intermediate_ca]# /opt/puppetlabs/puppet/bin/openssl crl   -in $(puppet config print hostcrl)   -text -noout | grep -A1 'Authority Key'
                  X509v3 Authority Key Identifier:
      [root@sx5txzicd4q58j6 intermediate_ca]# /opt/puppetlabs/puppet/bin/openssl x509 -in $(puppet config print localcacert) -text -noout |grep -A1 'Subject Key'
                  X509v3 Subject Key Identifier:


      [root@sx5txzicd4q58j6 intermediate_ca]# /opt/puppetlabs/puppet/bin/openssl crl   -in $(puppet config print hostcrl)   -text -noout | grep -A1 'Authority Key'
                  X509v3 Authority Key Identifier:

      Expected Outcome

      Agent runs continue to work after revocation and the Authority Key Identifier
      in the updated CRL matches the "PuppetCA" intermediate certificate.

      Additional Notes

      The CRL makes an easy demonstration that uses only one node. However, any external agent node added to the server will fail with validation errors:

      [root@k9hjxz663xl96e1 ~]# puppet agent -t
      Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
      Info: Creating a new SSL certificate request for k9hjxz663xl96e1.delivery.puppetlabs.net
      Info: Certificate Request fingerprint (SHA256): 26:D6:2B:33:1F:41:2F:6B:95:81:90:65:7C:A7:6E:99:9C:4F:0F:98:C0:AD:1C:61:53:80:04:81:56:BD:38:8D
      Info: Downloaded certificate for k9hjxz663xl96e1.delivery.puppetlabs.net from dq4a9q8qcg1d68w.delivery.puppetlabs.net
      Warning: The issuer 'CN=PuppetCA' of certificate 'CN=k9hjxz663xl96e1.delivery.puppetlabs.net' cannot be found locally
      Warning: Unable to fetch my node definition, but the agent run will continue:
      Warning: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown
      Info: Retrieving pluginfacts
      Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Failed to generate additional resources using 'eval_generate': SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown
      Error: /File[/opt/puppetlabs/puppet/cache/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet:///pluginfacts: SSL_connect returned=1 errno=0 state=error: sslv3 alert certificate unknown

      Copy the subject key id into the authority key id extension when signing certs and generating CRLs.




            Unassigned Unassigned
            chuck Charlie Sharpsteen
            3 Vote for this issue
            12 Start watching this issue



                Zendesk Support