[SERVER-528] Partial state error if run puppet agent on master before first puppetserver service start Created: 2015/04/01  Updated: 2016/10/26  Resolved: 2016/10/26

Status: Resolved
Project: Puppet Server
Component/s: None
Affects Version/s: None
Fix Version/s: SERVER 2.7.0

Type: Bug Priority: Normal
Reporter: Jeremy Barlow Assignee: Jeremy Barlow
Resolution: Fixed Votes: 9
Labels: maintenance
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by PUP-5523 Incorrect user and group set on /etc/... Closed
is duplicated by SERVER-899 Missing cert when puppet-agent started Closed
Relates
relates to PUP-4194 Puppet's logdir permissions prevent p... Closed
relates to SERVER-446 Ubuntu puppetserver pacakge out-of-bo... Closed
relates to SERVER-1233 Puppet Server fails with CA 'partial ... Closed
relates to SERVER-569 puppetserver won't start if a cert re... Closed
relates to SERVER-891 A half-completed SSL configuration br... Closed
relates to SERVER-1212 puppetserver left in broken state if ... Closed
Template:
Epic Link: SSL UX Improvements
Team: Systems Engineering
Story Points: 2
Sprint: SE 2016-10-19, SE 2016-11-02
Release Notes: Bug Fix
Release Notes Summary: In prior releases, if an agent was run before the Puppet Server service was first started, a private key and public key would be created for the agent but the Puppet Server service would subsequently fail to start with an error message like the following:

java.lang.IllegalStateException: Cannot initialize master with partial state; need all files or none.
Found:
/var/lib/puppet/ssl/private_keys/master.pem
Missing:
/var/lib/puppet/ssl/certs/master.pem

For the fix in this release, Puppet Server will use the pre-generated public and private key to generate a certificate for the master and will startup properly.
QA Contact: Erik Dasher

 Description   

The following sequence of steps would cause the puppetserver service to fail on initial startup:

1) Install puppetserver package.

2) On the master, run "puppet agent -t".

This generates a private key but not the associated cert.

3) Start the puppetserver service.

The following error appears in the puppetserver.log output and the service fails to start:

2015-04-01 09:42:00,736 INFO  [p.s.p.puppet-admin-service] Starting Puppet Admin web app
2015-04-01 09:42:00,754 ERROR [p.t.internal] Error during service init!!!
java.lang.IllegalStateException: Cannot initialize master with partial state; need all files or none.
Found:
/var/lib/puppet/ssl/private_keys/jb-centos7.localdomain.pem
Missing:
/var/lib/puppet/ssl/certs/jb-centos7.localdomain.pem

The Ruby master/CA would go ahead and generate the cert and boot to ready in this situation. We should revisit whether or not it would make sense to have the puppetserver master/CA go ahead and generate a cert automatically from a master private key if no cert is found at service startup.



 Comments   
Comment by Jeremy Barlow [ 2015/04/01 ]

Eric Sorenson - what are your thoughts on this one? Would it make sense to you to have Puppet Server follow the Ruby master / CA route and generate the certificate - to be more self-healing? Or do you think that the current Puppet Server behavior is acceptable and understandable for the user to remediate the problem - e.g., just delete the private key or run the "puppet cert" tools locally to get the cert generated so that Puppet Server can boot up properly? fyi, another user ran into a very similar - if not the same - problem in SERVER-446.

Comment by Jeremy Barlow [ 2015/04/27 ]

fyi, we've had a couple of other instances of this ticket reported from different users. Linked tickets are PUP-4194 and SERVER-569.

Comment by Chris Price [ 2015/04/29 ]

moving this to 'open' so it'll come up in triage again.

Comment by Kevin Corcoran [ 2015/07/22 ]

Jeremy Barlow's idea to have Puppet Server's CA automatically generate a certificate when a private key exists sounds like a reasonable improvement to me.

Comment by Marco Rodrigues [ 2016/03/22 ]

Any valid workaround for this issue ?

Comment by Chris Price [ 2016/03/22 ]

Marco Rodrigues You might have a look at SERVER-1233. We'd be interested in hearing the details of what happened in your situation; e.g. what distro are you running, did you encounter an out-of-memory error on the initial server startup.

The best workaround for now, if this is happening to you in a brand new installation, is to stop the puppet service, remove any keys/certs that were generated in the /etc/puppetlabs/puppet/ssl directory, and then start up Puppet Server. After Puppet Server starts successfully, it should have generated valid certs for the CA and the master, and subsequent agent runs should work fine.

We'll try to prioritize a better long-term solution for this in an upcoming release.

Comment by Marco Rodrigues [ 2016/03/22 ]

@Chris Price, I did install using rpm and I don't saw any out of memory error. My problem I think was to run puppet agent and signed with the default puppet server SSL and now I got that issue when I try to use my own certificates.

Environment: RHEL 7.2

puppet-agent-1.4.0-1.el7.x86_64
puppetserver-2.3.0-1.el7.noarch
puppetlabs-release-pc1-1.0.0-1.el7.noarch

I have an issue that probably is related.

I've modified the webserver.conf with: ssl-key : /etc/puppetlabs/puppet/ssl/private_keys/machine_name.key

The hostprivkey keep getting the .pem file instead of the key file.

  1. puppet master --configprint hostprivkey
    /etc/puppetlabs/puppet/ssl/private_keys/machine_name.pem
  1. puppet master --configprint hostcert
    /etc/puppetlabs/puppet/ssl/certs/machine_name.pem
  1. puppet master --configprint localcacert
    /etc/puppetlabs/puppet/ssl/certs/ca.pem

I've already removed the ssl directory and regenerated using the following doc: https://docs.puppetlabs.com/puppet/4.3/reference/ssl_regenerate_certificates.html#step-1-clear-and-regenerate-certs-on-your-puppet-master

Comment by Chris Price [ 2016/03/22 ]

Marco Rodrigues If you are trying to use your own certs rather than ones generated by Puppet Server and/or the puppet agent, you might need to add some configuration to Puppet Server's webserver.conf file. Does this doc help?:

https://docs.puppetlabs.com/puppetserver/latest/external_ca_configuration.html#web-server-configuration

Comment by Marco Rodrigues [ 2016/03/23 ]

Chris Price I already have that configuration, but it doesn't allow me to point to a .key. I found out this bug report due to that problem assuming it's the same issue. I think it's because I did run puppet-agent before configuring the CA.

Comment by Marco Rodrigues [ 2016/03/23 ]

Chris Price, I found out the issue and now it's working. But it seems the parameter ssl-key on the webserver.conf doesn't like when I specify a file .key instead of .pem. Now it's working with .pem instead of .key.

Comment by Chris Price [ 2016/03/28 ]

Marco Rodrigues does the .key file have a different format? Or are you saying that simply changing the file extension fixed it?

Comment by Marco Rodrigues [ 2016/03/29 ]

Chris Price Yes, changing the .key to .pem did the trick. I'm having others issues as puppet agent doesn't arrive to identify it fully (I spend around 4 or 5 days without success), but probably that's for another ticket or private msg if you allow me to.

Comment by Chris Price [ 2016/03/29 ]

Interesting, I wouldn't expect the file extension to be an issue. We should look into that on our end.

Please feel free to create another ticket for your agent issue, and feel free to ping me on it, but we'd like to keep the conversation public because it might be useful for future users that are troubleshooting an issue and searching for information.

Comment by Justin Honold [ 2016/06/13 ]

This behavior impacts me in a situation where I'm using AWS CloudFormation to stand up infrastructure in the following way: minimal OS image -> minimal cfn-init to get Puppet going -> Puppet for everything else. Including Puppet setting up Puppet itself. Major chicken/egg thing with the agent coming before the server, as the agent (with an apply) is what sets up the server. Manual steps here are no fun, the idea is total automation.

Comment by Jeremy Barlow [ 2016/10/26 ]

Merged to puppetserver#master at 68ee12. Overnight runs in CI were green so I'm going to mark this resolved.

Generated at Sun Jun 17 18:31:19 PDT 2018 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.