[PUP-2310] Puppet client does not update and does consult the crl during authentication Created: 2014/04/21  Updated: 2019/09/20  Resolved: 2019/05/13

Status: Resolved
Project: Puppet
Component/s: None
Affects Version/s: None
Fix Version/s: PUP 6.5.0

Type: Bug Priority: Normal
Reporter: redmine.exporter Assignee: Josh Cooper
Resolution: Fixed Votes: 8
Labels: redmine, resolved-issue-added, resolved_issue_added
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by PUP-2103 Puppet client does not update and doe... Closed
is duplicated by PUP-8563 The puppet agent should have a 'local... Closed
is duplicated by PUP-9152 Agents should re-download CRLs when t... Closed
Relates
relates to PUP-894 Too easy to hit "CRL not yet valid fo... Closed
relates to SERVER-1181 check if it's valid to sync cacrl to ... Closed
relates to PUP-8913 Agents should re-download CRLs when t... Closed
relates to PUP-9152 Agents should re-download CRLs when t... Closed
Template: PUP Bug Template
Epic Link: Simplify agent SSL initialization
Team: Coremunity
Sprint: Platform Core KANBAN
CS Priority: Reviewed
Release Notes: Enhancement
Release Notes Summary: By default, puppet agents download their CRL from the CA once, but never refresh it. In 6.5.0 is is now possible to specify the `crl_refresh_interval` puppet setting. If specified as a duration, such as 8h, 7d, etc, then the agent will refresh its CRL whenever it next runs and the elapsed time since the CRL was last refreshed exceeds the duration.

In general, the duration should be greater than the `runinterval`. Setting it to an equal or lesser value will cause the CRL to be refreshed on every agent run.

If the agent downloads a new CRL, then it will use the new CRL for all subsequent network requests. If the refresh request fails or if the CRL is unchanged on the CA, then the agent run will continue using the local CRL it already has.
QA Contact: Eric Thompson

 Description   

I my tests puppet client never updates it's /var/lib/puppet/ssl/ca/ca_crl.pem from the master
even if I delete it - it is not fetched from master then client runs.

Another issue is that puppet client does not consult the crl - after revoking cert of node dev2.internal on master - and manually copying /var/lib/puppet/ssl/ca/

{ca_crl.pem,inventory.txt}

to client mon1a.internal and restarting the client to make sure it can pickup the crl changes - I was still able to trigger client puppet run on mon1a.internal from dev2.internal.

It looks like puppet - client does not take the crl into consideration then authenticating.

The relevant config on mon1a.internal is
<pre>

  1. allow all authenticated nodes to trigger puppet run
    path /run
    method save
    auth yes
    allow *
    </pre>
    this ACL comes first in the auth.conf file

And this is the command I used to triger puppet run from dev2.internal
<pre>
curl --cert /var/lib/puppet/ssl/certs/dev2.internal.pem --key /var/lib/puppet/ssl/private_keys/dev2.internal.pem --cacert /var/lib/puppet/ssl/certH "Content-Type: text/pson" -d "{}" https://mon1a.internal:8139/production/run/dev2.internal
</pre>
Could these problems be taken care of?

Thanks
Alex

Ed- description from #9205 which is closely related included

We came across this in a weird way. Last night we reissued the CA certificate, which had expired. We then reissued the puppetmaster and puppetca certificate (which we had to do for RHEL4 and RHEL5 but all other systems were happy without this step). We then noticed on RHEL4 and RHEL5 that they were still complaining about cert validation, but ONLY for getting plugins and sending the report (it got a catalog and was able to get files for modules, etc, just fine). We did an strace and found this was the only times it was trying to get a CRL (and was failing). Why is this the only time the CRL was in play?



 Comments   
Comment by Shane Madden [ 2014/09/10 ]

Josh Cooper Seems like the duplication relationship is the other way around? PUP-2103 has a lower number, but this is the original exported issue from 2011 (with the extra related and duplicate links for context).

I'm a little surprised that there's been no movement on this in 3 years since it was first reported, since...

  • being able to effectively revoke a compromised certificate is a gap in Puppet's security model in any environment with more than one master, unless someone implements CRL updating by hand (which they won't notice they need to do in many cases, since it downloads automatically the first time), and
  • this seems relatively simple to implement in a sane way without involving the complexity of going down the OCSP road; expose the CRL in a puppet:// url on a master, and have a default resource (similar to pluginsync) which fetches that CRL file.

So, any chance that this gets addressed any time soon?

Comment by Josh Cooper [ 2014/09/29 ]

Shane Madden Yep, thanks, I missed that the "earliest" ticket was in fact "later" in jira. As far as when, it's something we're aware of, don't know when it will be addressed.

Comment by Shane Madden [ 2014/09/29 ]

Great, thanks Josh Cooper, glad it's still on the radar!

Comment by Anchor [ 2014/10/27 ]

I've submitted https://github.com/puppetlabs/puppet/pull/3247, which is a fix for https://projects.puppetlabs.com/issues/16842 (which was marked as a duplicate of this issue).

That pull request defines CertificateRevocationList#expiration to return the CRL's next_update timestamp, so that the indirector cache knows to invalidate it once next_update is passed. It does not address the broader issue of when the CRL should be consulted, nor does it fix the reported problem with the agent not fetching a missing CRL (I haven't tried to reproduce the latter problem).

Comment by Josh Cooper [ 2015/03/16 ]

Providing some context about why PR #3247 was closed. Puppet hardcodes the CRL next_update time to be 5 years in the future, see https://github.com/puppetlabs/puppet/blob/3.7.4/lib/puppet/ssl/certificate_revocation_list.rb#L88. With this PR, the puppet agent would not consider its cache to be stale for 5 years, leaving us in effectively the same position we are in today.

If you patched puppet to use a shorter next_update time, or made it configurable, and the agent's CRL expired, then in the common case where no agent certs had been revoked, the agent would download the same CRL it had, discover the CRL is still expired, and hopefully not get into an infinite loop.

So somehow the CRL on the master needs to be updated periodically (to bump the next_update time). AFAIK, the only way to do that from the puppet CLI is to revoke a cert. So you could generate a temp cert, and then revoke it, but that's a bit lame. Ideally, the certificate_revocation application would have an update command that just bumps the next_update time and resigns the CRL.

Also, the CA cert's not_after time is determined by puppet's ca_ttl setting which also defaults to 5 years, see https://github.com/puppetlabs/puppet/blob/3.7.4/lib/puppet/defaults.rb#L962-L967. So we'd need to make sure the CA doesn't expire before the CRL.

Long term we are looking at adding a cert-related CLI to puppetserver and adding OCSP support. /cc Chris Price, Jeremy Barlow

Comment by John De Stefano [ 2016/09/22 ]

We started using Puppet in production five years ago this month. Guess what happened? Our client CRLs have expired, so they have stopped checking in.

It would be great if someone could have a look at this? It was last commented on 1.5 years ago, and it's kind of important – not just for us, but anyone whose Puppet infrastructure is approaching five years of age.

Thanks!

Comment by Christopher Wood [ 2018/02/06 ]

Noting these while I'm looking at my internal CRL ticket again:

https://ask.puppet.com/question/3843/multiple-puppet-masters-with-single-ca-server/

https://gist.github.com/derektamsen/f986216ed10bdc66e650

Comment by Maggie Dreyer [ 2018/10/02 ]

We attempted to make this change and had to revert it due to issues with PE's HA capability, see PUP-9152 (specifically this comment. We hope to be able to re-implement the changes in that ticket as part of making improvements to the HA story for the CA.

Comment by Kris Bosland [ 2019/05/10 ]

Merged into master at 31966dd.

Comment by Jorie Tappa [ 2019/05/13 ]

Passed CI. Please add release notes if applicable!

Generated at Fri Jan 24 23:26:48 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.