Details
-
Bug
-
Status: Resolved
-
Normal
-
Resolution: Done
-
PDB 5.2.15
-
None
Description
Strange bug:
After I upgraded puppetdb from 5.2.12-1stretch to 5.2.15-1stretch, I can no longer access the metrics endpoint.
All requests to /metrics/v1/mbeans/* get at "404 not found" response while requests to e.g. /pdb/query/v4/ work just fine (200 OK).
Prior to Jun 30, my system (Debian 9) was running puppetdb 5.2.12-1stretch and everything was working fine.
On Jun 30, an 'apt-get upgrade' upgraded puppet-agent, puppet5-release, puppetdb, puppetserver and puppetdb-termini:
dpkg.log.3.gz:2020-06-30 08:34:48 upgrade puppet-agent:amd64 5.5.18-1stretch 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:48 status half-configured puppet-agent:amd64 5.5.18-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:48 status unpacked puppet-agent:amd64 5.5.18-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:48 status half-installed puppet-agent:amd64 5.5.18-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:50 status half-installed puppet-agent:amd64 5.5.18-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 configure puppet-agent:amd64 5.5.20-1stretch <none>
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status unpacked puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status half-configured puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:34:51 status installed puppet-agent:amd64 5.5.20-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 upgrade puppet5-release:all 5.0.0-7stretch 5.0.0-11stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status half-configured puppet5-release:all 5.0.0-7stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status unpacked puppet5-release:all 5.0.0-7stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status half-installed puppet5-release:all 5.0.0-7stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status half-installed puppet5-release:all 5.0.0-7stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status unpacked puppet5-release:all 5.0.0-11stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status unpacked puppet5-release:all 5.0.0-11stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 upgrade puppetdb:all 5.2.12-1stretch 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status half-configured puppetdb:all 5.2.12-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status unpacked puppetdb:all 5.2.12-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status half-installed puppetdb:all 5.2.12-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status half-installed puppetdb:all 5.2.12-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 upgrade puppetserver:all 5.3.11-1stretch 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status half-configured puppetserver:all 5.3.11-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status unpacked puppetserver:all 5.3.11-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:33 status half-installed puppetserver:all 5.3.11-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status half-installed puppetserver:all 5.3.11-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 upgrade puppetdb-termini:all 5.2.12-1stretch 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status half-configured puppetdb-termini:all 5.2.12-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status unpacked puppetdb-termini:all 5.2.12-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status half-installed puppetdb-termini:all 5.2.12-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status half-installed puppetdb-termini:all 5.2.12-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status unpacked puppetdb-termini:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:34 status unpacked puppetdb-termini:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 configure puppetdb:all 5.2.15-1stretch <none>
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status unpacked puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:36:40 status half-configured puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:20 status installed puppetdb:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:20 configure puppetdb-termini:all 5.2.15-1stretch <none>
|
dpkg.log.3.gz:2020-06-30 08:37:20 status unpacked puppetdb-termini:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:20 status half-configured puppetdb-termini:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:20 status installed puppetdb-termini:all 5.2.15-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 configure puppetserver:all 5.3.13-1stretch <none>
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status unpacked puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:21 status half-configured puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:51 status installed puppetserver:all 5.3.13-1stretch
|
dpkg.log.3.gz:2020-06-30 08:37:51 configure puppet5-release:all 5.0.0-11stretch <none>
|
dpkg.log.3.gz:2020-06-30 08:37:51 status unpacked puppet5-release:all 5.0.0-11stretch
|
dpkg.log.3.gz:2020-06-30 08:37:51 status unpacked puppet5-release:all 5.0.0-11stretch
|
dpkg.log.3.gz:2020-06-30 08:37:51 status unpacked puppet5-release:all 5.0.0-11stretch
|
dpkg.log.3.gz:2020-06-30 08:37:51 status half-configured puppet5-release:all 5.0.0-11stretch
|
dpkg.log.3.gz:2020-06-30 08:37:51 status installed puppet5-release:all 5.0.0-11stretch
|
The machine is now running puppetdb v5.2.15. Now the metrics endpoint stopped working.
Here I have grep:ed a request for the metric "pct-resource-dupes" before and after:
puppetdb-access-2020-06-29.0.log.gz:127.0.0.1 - - [29/Jun/2020:17:24:14 +0200] "GET /metrics/v1/mbeans/puppetlabs.puppetdb.population:name=pct-resource-dupes HTTP/1.1" 200 28 "https://puppet-explorer.foo.bar.se/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0" 2
|
puppetdb-access-2020-06-30.0.log.gz:127.0.0.1 - - [30/Jun/2020:12:56:10 +0200] "GET /metrics/v1/mbeans/puppetlabs.puppetdb.population:name=pct-resource-dupes HTTP/1.1" 404 304 "https://puppet-explorer.foo.bar.se/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36" 4
|
On the 29th I got a '200' response while since the 30th I only get '404':s.
I can't find any configuration to explain why it stopped working.
I can't find any logs to explain why it stopped working.
The only change I know I had to do after the upgrade:
In the puppet-agent 5.5.20 version, there was a big change (a bugfix) as to how puppet uses the http_proxy ENV variables as well as the no_proxy configuration.
I don't see how a change in my proxy settings would break only the metrics endpoint and nothing else.
Please let me know if you need more details.
A tcpdump on the loopback interface:
|
12:48:21.691571 IP 127.0.0.1.57864 > 127.0.0.1.8080: Flags [P.], seq 627:1245, ack 487, win 350, options [nop,nop,TS val 83538331 ecr 83538329], length 618: HTTP: GET /metrics/v1/mbeans/puppetlabs.puppetdb.p
|
0x0000: 4500 029e 72d8 4000 4006 c77f 7f00 0001 E...r.@.@.......
|
0x0010: 7f00 0001 e208 1f90 208a f1e4 0441 4e10 .............AN.
|
0x0020: 8018 015e 0093 0000 0101 080a 04fa b19b ...^............
|
0x0030: 04fa b199 4745 5420 2f6d 6574 7269 6373 ....GET./metrics
|
0x0040: 2f76 312f 6d62 6561 6e73 2f70 7570 7065 /v1/mbeans/puppe
|
0x0050: 746c 6162 732e 7075 7070 6574 6462 2e70 tlabs.puppetdb.p
|
0x0060: 6f70 756c 6174 696f 6e3a 6e61 6d65 3d6e opulation:name=n
|
0x0070: 756d 2d72 6573 6f75 7263 6573 2048 5454 um-resources.HTT
|
0x0080: 502f 312e 310d 0a48 6f73 743a 206c 6f63 P/1.1..Host:.loc
|
0x0090: 616c 686f 7374 3a38 3038 300d 0a55 7365 alhost:8080..Use
|
0x00a0: 722d 4167 656e 743a 204d 6f7a 696c 6c61 r-Agent:.Mozilla
|
0x00b0: 2f35 2e30 2028 5769 6e64 6f77 7320 4e54 /5.0.(Windows.NT
|
0x00c0: 2036 2e31 3b20 5769 6e36 343b 2078 3634 .6.1;.Win64;.x64
|
0x00d0: 3b20 7276 3a38 302e 3029 2047 6563 6b6f ;.rv:80.0).Gecko
|
0x00e0: 2f32 3031 3030 3130 3120 4669 7265 666f /20100101.Firefo
|
0x00f0: 782f 3830 2e30 0d0a 4163 6365 7074 3a20 x/80.0..Accept:.
|
0x0100: 6170 706c 6963 6174 696f 6e2f 6a73 6f6e application/json
|
0x0110: 2c20 7465 7874 2f70 6c61 696e 2c20 2a2f ,.text/plain,.*/
|
0x0120: 2a0d 0a41 6363 6570 742d 4c61 6e67 7561 *..Accept-Langua
|
0x0130: 6765 3a20 656e 2d47 422c 656e 3b71 3d30 ge:.en-GB,en;q=0
|
0x0140: 2e38 2c65 6e2d 5553 3b71 3d30 2e36 2c73 .8,en-US;q=0.6,s
|
0x0150: 762d 5345 3b71 3d30 2e34 2c73 763b 713d v-SE;q=0.4,sv;q=
|
0x0160: 302e 320d 0a41 6363 6570 742d 456e 636f 0.2..Accept-Enco
|
0x0170: 6469 6e67 3a20 677a 6970 2c20 6465 666c ding:.gzip,.defl
|
0x0180: 6174 652c 2062 720d 0a52 6566 6572 6572 ate,.br..Referer
|
0x0190: 3a20 6874 7470 733a 2f2f 7075 7070 6574 :.https://puppet
|
0x01a0: 2d65 7870 6c6f 7265 722e 7075 7070 6574 -explorer.......
|
.....
|
12:48:21.691869 IP 127.0.0.1.8080 > 127.0.0.1.57864: Flags [P.], seq 487:964, ack 1245, win 361, options [nop,nop,TS val 83538331 ecr 83538331], length 477: HTTP: HTTP/1.1 404 Not Found
|
0x0000: 4500 0211 1db7 4000 4006 1d2e 7f00 0001 E.....@.@.......
|
0x0010: 7f00 0001 1f90 e208 0441 4e10 208a f44e .........AN....N
|
0x0020: 8018 0169 0006 0000 0101 080a 04fa b19b ...i............
|
0x0030: 04fa b19b 4854 5450 2f31 2e31 2034 3034 ....HTTP/1.1.404
|
0x0040: 204e 6f74 2046 6f75 6e64 0d0a 4461 7465 .Not.Found......
|
.....
|