[PDB-1962] Failed to start Apache ActiveMQ Created: 2015/09/11  Updated: 2018/09/18  Resolved: 2015/09/11

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: None
Fix Version/s: PDB 3.2.0

Type: Bug Priority: Blocker
Reporter: Sergey Assignee: Ken Barber
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: HTML File puppetdb_debug    
Template:

 Description   

Trying to start puppetdb but encounter an error:

ERROR [o.a.a.b.BrokerService] Failed to start Apache ActiveMQ ([localhost, null], {})
java.lang.RuntimeException: org.apache.activemq.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).

Tried to remove Kaha.db and logfiles but it didn't help

Please any help guys



 Comments   
Comment by Ken Barber [ 2015/09/11 ]

We'll need more details:

  • Provide the full debugging output up until this line, and after it, not just the line itself. If there is a stack trace of some form include that also. puppetdb foreground --debug is one way to attempt to run it in the foreground with debugging.
  • Can you provide the version of PuppetDB that you are using? And puppet as well, that might be helpful.
  • What distro and version?
  • Also, any of the configuration files in /etc/puppetlabs/puppetdb/conf.d would be helpful for us to look at.
  • Does this happen as soon as you start? Can you access the dashboard on http://localhost:8080/ via a browser after this occurs?
  • When you say you removed KahaDB, what process did you follow to do that?
Comment by Sergey [ 2015/09/11 ]

1. debug output attached
2. puppetdb version: 3.1.0
3. CentOS Linux release 7.0.1406 (Core)
4.

/etc/puppetlabs/puppetdb/conf.d/config.ini

# See README.md for more thorough explanations of each section and
# option.
 
[global]
# Store mq/db data in a custom directory
vardir = /opt/puppetlabs/server/data/puppetdb
 
# Use an external logback config file
logging-config = /etc/puppetlabs/puppetdb/logback.xml
 
[command-processing]
# How many command-processing threads to use, defaults to (CPUs / 2)
# threads = 4
 
# Maximum amount of disk space (in MB) to allow for ActiveMQ persistent message storage
# store-usage = 102400
 
# Maximum amount of disk space (in MB) to allow for ActiveMQ temporary message storage
# temp-usage = 51200

/etc/puppetlabs/puppetdb/conf.d/database.ini

[database]
# For the embedded DB: org.hsqldb.jdbcDriver
# For PostgreSQL: org.postgresql.Driver
# Defaults to embedded DB
classname = org.hsqldb.jdbcDriver
 
# For the embedded DB: hsqldb
# For PostgreSQL: postgresql
# Defaults to embedded DB
subprotocol = hsqldb
 
# For the embedded DB: file:/path/to/database;hsqldb.tx=mvcc;sql.syntax_pgs=true
# For PostgreSQL: //host:port/databaseName
# Defaults to embedded DB located in <vardir>/db
subname = file:/opt/puppetlabs/server/data/puppetdb/db/db;hsqldb.tx=mvcc;sql.syntax_pgs=true
 
# Connect as a specific user
# username = foobar
 
# Use a specific password
# password = foobar
 
# How often (in minutes) to compact the database
# gc-interval = 60
 
# Number of seconds before any SQL query is considered 'slow'; offending
# queries will not be interrupted, but will be logged at the WARN log level.
log-slow-statements = 10

/etc/puppetlabs/puppetdb/conf.d/jetty.ini

[jetty]
# IP address or hostname to listen for clear-text HTTP. To avoid resolution
# issues, IP addresses are recommended over hostnames.
# Default is `localhost`.
 host = 0.0.0.0
 
# Port to listen on for clear-text HTTP.
port = 8080
 
# The following are SSL specific settings. They can be configured
# automatically with the tool `puppetdb ssl-setup`, which is normally
# ran during package installation.
 
# IP address to listen on for HTTPS connections. Hostnames can also be used
# but are not recommended to avoid DNS resolution issues. To listen on all
# interfaces, use `0.0.0.0`.
ssl-host = 0.0.0.0
 
# The port to listen on for HTTPS connections
ssl-port = 8081
 
# Private key path
ssl-key = /etc/puppetlabs/puppetdb/ssl/private.pem
 
# Public certificate path
ssl-cert = /etc/puppetlabs/puppetdb/ssl/public.pem
 
# Certificate authority path
ssl-ca-cert = /etc/puppetlabs/puppetdb/ssl/ca.pem
 
# Access logging configuration path. To turn off access logging
# comment out the line with `access-log-config=...`
access-log-config = /etc/puppetlabs/puppetdb/request-logging.xml

5. Puppetdb fails right away after start and no connections to port coudl be established. What we have done before was an LVM extension - just to note.
6. I followed this instructions https://docs.puppetlabs.com/puppetdb/latest/trouble_kahadb_corruption.html

Thanks ahead,
Sergey

Comment by Ken Barber [ 2015/09/11 ]

Sergey I think the paths in the KahaDB document are old (we'll need to update this). The basedir should be: /opt/puppetlabs/server/data/puppetdb for removal of KahaDB (so /opt/puppetlabs/server/data/puppetdb/mq/localhost is the path to remove the KahaDB dir).

The exception looks like it's trying to consume a corrupt entry on the job scheduler (so a command that was previously tried and failed):

java.lang.RuntimeException: org.apache.activemq.protobuf.InvalidProtocolBufferException: Protocol message contained an invalid tag (zero).
 at org.apache.activemq.broker.BrokerService.getJobSchedulerStore (BrokerService.java:1897)
    org.apache.activemq.broker.BrokerService.getSystemUsage (BrokerService.java:1100)
    org.apache.activemq.broker.BrokerService.getProducerSystemUsage (BrokerService.java:1168)
    org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter (BrokerService.java:645)
    org.apache.activemq.broker.BrokerService.startPersistenceAdapter (BrokerService.java:640)
    org.apache.activemq.broker.BrokerService.start (BrokerService.java:605)
    puppetlabs.puppetdb.mq$start_broker_BANG_.invoke (mq.clj:120)

So my feelings are that there is still some bad KahaDB content here, make sure your mq/localhost directory in that new path is truly clear before restarting.

Comment by Ken Barber [ 2015/09/11 ]

Sergey also, you extended your LVM volume, is this because your disk filled up and things started to fail? KahaDB isn't great when the disk fills up, it's unfortunate but true. We've been waiting for LevelDB to become stable and considering switching to that instead if it brings more stability under failure.

I generally suggest to people to use the limiting size controls for MQ, so we stop processing if we hit an internal limit, instead of waiting for a disk to fill up. That avoids corruption at least, and it means you can limit the MQ usage to avoid filling the disk up in the first place. Docs are here:

http://docs.puppetlabs.com/puppetdb/3.1/configure.html#store-usage

Comment by Sergey [ 2015/09/11 ]

Hi Kenneth,

that helped me

rm -rf /opt/puppetlabs/server/data/puppetdb/mq/localhost/*

I was just flushing Kahadb folder neglecting sibling subfolder. Seems now we are fine.
Thanks for prompt help!

Comment by Ken Barber [ 2015/09/11 ]

Cool, I've updated the docs: https://github.com/puppetlabs/puppetdb/commit/06d5a83f325207a5a022adfe27072e0578323070

Should be live in 15 minutes or so.

Generated at Mon Sep 23 13:18:21 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.