[PDB-656] puppetdb 2.0.0 throws exception after yum update Created: 2014/05/06  Updated: 2016/04/27  Resolved: 2014/05/07

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: PDB 2.0.0
Fix Version/s: PDB 2.0.0

Type: Bug Priority: Normal
Reporter: Aaron Zauner Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: puppetdb
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

CentOS 6.5


Issue Links:
Duplicate
is duplicated by PDB-661 puppetdb 2.0 not starting after upgrade Closed
Template:

 Description   

After doing a simple `yum update` on the puppet server to fix the EPEL issue puppetdb was as well upgraded. PuppetDB won't start and throws an exception because of a missing log4j.properties file (which should be depreciated, as far as I can tell from the git sources). Puppetlabs Devel repositories active.

I'm currently trying to fix this issue by downgrading to PuppetDB 1.6.

Is this a known issue? I could not find a bug report for this yet.

[root@puppet puppetdb]# puppetdb foreground
ch.qos.logback.core.joran.spi.JoranException: Could not open [/etc/puppetdb/log4j.properties].
 at ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:80)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:68)
    sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethodAccessorImpl.java:-2)
    sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke (Method.java:606)
    clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
    clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
    puppetlabs.trapperkeeper.logging$configure_logger_via_file_BANG_.invoke (logging.clj:81)
    puppetlabs.trapperkeeper.logging$configure_logging_BANG_.invoke (logging.clj:90)
    puppetlabs.trapperkeeper.config$initialize_logging_BANG_.invoke (config.clj:103)
    puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke (core.clj:111)
    puppetlabs.trapperkeeper.core$run.invoke (core.clj:145)
    puppetlabs.trapperkeeper.core$main.doInvoke (core.clj:160)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    com.puppetlabs.puppetdb.cli.services$_main.doInvoke (services.clj:365)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invoke (core.clj:624)
    com.puppetlabs.puppetdb.core$run_command.invoke (core.clj:87)
    com.puppetlabs.puppetdb.core$_main.doInvoke (core.clj:95)
    clojure.lang.RestFn.invoke (RestFn.java:436)
    clojure.lang.Var.invoke (Var.java:388)
    clojure.lang.AFn.applyToHelper (AFn.java:160)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invoke (core.clj:624)
    clojure.main$main_opt.invoke (main.clj:315)
    clojure.main$main.doInvoke (main.clj:420)
    clojure.lang.RestFn.invoke (RestFn.java:482)
    clojure.lang.Var.invoke (Var.java:401)
    clojure.lang.AFn.applyToHelper (AFn.java:171)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)
Caused by: java.io.FileNotFoundException: /etc/puppetdb/log4j.properties (No such file or directory)
 at java.io.FileInputStream.open (FileInputStream.java:-2)
    java.io.FileInputStream.<init> (FileInputStream.java:146)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:75)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:68)
    sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethodAccessorImpl.java:-2)
    sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke (Method.java:606)
    clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
    clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
    puppetlabs.trapperkeeper.logging$configure_logger_via_file_BANG_.invoke (logging.clj:81)
    puppetlabs.trapperkeeper.logging$configure_logging_BANG_.invoke (logging.clj:90)
    puppetlabs.trapperkeeper.config$initialize_logging_BANG_.invoke (config.clj:103)
    puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke (core.clj:111)
    puppetlabs.trapperkeeper.core$run.invoke (core.clj:145)
    puppetlabs.trapperkeeper.core$main.doInvoke (core.clj:160)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    com.puppetlabs.puppetdb.cli.services$_main.doInvoke (services.clj:365)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invoke (core.clj:624)
    com.puppetlabs.puppetdb.core$run_command.invoke (core.clj:87)
    com.puppetlabs.puppetdb.core$_main.doInvoke (core.clj:95)
    clojure.lang.RestFn.invoke (RestFn.java:436)
    clojure.lang.Var.invoke (Var.java:388)
    clojure.lang.AFn.applyToHelper (AFn.java:160)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invoke (core.clj:624)
    clojure.main$main_opt.invoke (main.clj:315)
    clojure.main$main.doInvoke (main.clj:420)
    clojure.lang.RestFn.invoke (RestFn.java:482)
    clojure.lang.Var.invoke (Var.java:401)
    clojure.lang.AFn.applyToHelper (AFn.java:171)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)
[root@puppet puppetdb]# rpm -qa | grep puppetdb
puppetdb-terminus-2.0.0-0.1rc1.el6.noarch
puppetdb-2.0.0-0.1rc1.el6.noarch
[root@puppet puppetdb]# 

This error can also appear like this:

ch.qos.logback.core.joran.spi.JoranException: Problem parsing XML document. See previously reported errors.
 at ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents (SaxEventRecorder.java:67)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:134)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:99)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:76)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:68)
    sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethodAccessorImpl.java:-2)
    sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke (Method.java:606)
    clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
    clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
    puppetlabs.trapperkeeper.logging$configure_logger_via_file_BANG_.invoke (logging.clj:81)
    puppetlabs.trapperkeeper.logging$configure_logging_BANG_.invoke (logging.clj:90)
    puppetlabs.trapperkeeper.config$initialize_logging_BANG_.invoke (config.clj:103)
    puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke (core.clj:111)
    puppetlabs.trapperkeeper.core$run.invoke (core.clj:145)
    puppetlabs.trapperkeeper.core$main.doInvoke (core.clj:160)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    com.puppetlabs.puppetdb.cli.services$_main.doInvoke (services.clj:365)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invoke (core.clj:624)
    com.puppetlabs.puppetdb.core$run_command.invoke (core.clj:87)
    com.puppetlabs.puppetdb.core$_main.doInvoke (core.clj:95)
    clojure.lang.RestFn.invoke (RestFn.java:436)
    clojure.lang.Var.invoke (Var.java:388)
    clojure.lang.AFn.applyToHelper (AFn.java:160)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invoke (core.clj:624)
    clojure.main$main_opt.invoke (main.clj:315)
    clojure.main$main.doInvoke (main.clj:420)
    clojure.lang.RestFn.invoke (RestFn.java:482)
    clojure.lang.Var.invoke (Var.java:401)
    clojure.lang.AFn.applyToHelper (AFn.java:171)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)
Caused by: org.xml.sax.SAXParseException: Content is not allowed in prolog.
 at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse (AbstractSAXParser.java:1239)
    com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse (SAXParserImpl.java:649)
    com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.parse (SAXParserImpl.java:333)
    ch.qos.logback.core.joran.event.SaxEventRecorder.recordEvents (SaxEventRecorder.java:61)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:134)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:99)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:76)
    ch.qos.logback.core.joran.GenericConfigurator.doConfigure (GenericConfigurator.java:68)
    sun.reflect.NativeMethodAccessorImpl.invoke0 (NativeMethodAccessorImpl.java:-2)
    sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57)
    sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
    java.lang.reflect.Method.invoke (Method.java:606)
    clojure.lang.Reflector.invokeMatchingMethod (Reflector.java:93)
    clojure.lang.Reflector.invokeInstanceMethod (Reflector.java:28)
    puppetlabs.trapperkeeper.logging$configure_logger_via_file_BANG_.invoke (logging.clj:81)
    puppetlabs.trapperkeeper.logging$configure_logging_BANG_.invoke (logging.clj:90)
    puppetlabs.trapperkeeper.config$initialize_logging_BANG_.invoke (config.clj:103)
    puppetlabs.trapperkeeper.core$boot_with_cli_data.invoke (core.clj:111)
    puppetlabs.trapperkeeper.core$run.invoke (core.clj:145)
    puppetlabs.trapperkeeper.core$main.doInvoke (core.clj:160)
    clojure.lang.RestFn.applyTo (RestFn.java:137)
    clojure.core$apply.invoke (core.clj:624)
    com.puppetlabs.puppetdb.cli.services$_main.doInvoke (services.clj:365)
    clojure.lang.RestFn.invoke (RestFn.java:421)
    clojure.lang.Var.invoke (Var.java:383)
    clojure.lang.AFn.applyToHelper (AFn.java:156)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invoke (core.clj:624)
    com.puppetlabs.puppetdb.core$run_command.invoke (core.clj:87)
    com.puppetlabs.puppetdb.core$_main.doInvoke (core.clj:95)
    clojure.lang.RestFn.invoke (RestFn.java:436)
    clojure.lang.Var.invoke (Var.java:388)
    clojure.lang.AFn.applyToHelper (AFn.java:160)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.core$apply.invoke (core.clj:624)
    clojure.main$main_opt.invoke (main.clj:315)
    clojure.main$main.doInvoke (main.clj:420)
    clojure.lang.RestFn.invoke (RestFn.java:482)
    clojure.lang.Var.invoke (Var.java:401)
    clojure.lang.AFn.applyToHelper (AFn.java:171)
    clojure.lang.Var.applyTo (Var.java:700)
    clojure.main.main (main.java:37)



 Comments   
Comment by Ken Barber [ 2014/05/06 ]

Lets talk about the immediate problem first ...

Change your config in /etc/puppetdb/conf.d/config.ini so that the line for logging now reflects the new logback configuration file:

logging-config = /etc/puppetdb/logback.xml

And then restart.

Comment by Aaron Zauner [ 2014/05/06 ]

This in fact seems to work.

Shouldn't maybe there be a notice to people upgrading to 2.0 about issues like that?

Thanks for your help.

Comment by Ken Barber [ 2014/05/06 ]

Aaron Zauner so the problem stems from ... if /etc/puppetdb/conf.d/config.ini has been modified from its original in the past, then RPM will no longer just blat new configuration into place when upgrading a package. Instead it drops a config.ini.rpmnew (can you check this exists?) with the better proposed configuration. In this manner its cowardly refusing to just replace configuration, because your local edits may be important to you.

On debian however, it prompts you instead of just dropping an rpmnew, which gives you the opportunity to diff with your existing version and make a decision if the vendored version is better or not. Some would argue this is better, but either way - RPM/YUM has nothing like that AFAIK.

So the reality is, this way of YUM cowardly not updating configuration in this way is pretty standard and any sysadmin should be watching for those things during upgrades. Its unfortunate that we can't be louder at RPM install time, this especially applies to cases where users are upgrading packages automatically without user intervention (or the user seeing any mention of rpmnew files being created). Perhaps there is a better way that I don't know about.

Having said that we can change the release notes for PuppetDB 2.0 to warn people ... which at least helps the users who read the release notes before upgrading a major version. Some users won't read them, and they'll raise bugs like this one (or hopefully search first and find this one).

Comment by Aaron Zauner [ 2014/05/06 ]

Thanks for the explaination. I've completely missed the *.rpmnew files.

Comment by Ken Barber [ 2014/05/07 ]

Updated announcement for now to warn people to look out for this problem.

Generated at Wed Jun 26 23:58:50 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.