[CPR-421] ubuntu trusty update failure Created: 2017/05/09  Updated: 2017/12/28  Resolved: 2017/12/28

Status: Closed
Project: Community Package Repository
Component/s: None
Affects Version/s: None
Fix Version/s: 2017/08/02

Type: Improvement Priority: Blocker
Reporter: Tom Hey Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Ubuntu 14.04 running in AWS region west-2


Template:
QA Risk Assessment: Needs Assessment

 Description   

This ticket is pretty much as duplicate of CPR-419, but only effects some CDNs. I've started this as a blocker as it prevents puppet installing on some AWS regions - which is quite a big deal.

The apt-update command fails unless I manually point apt.puppetlabs.com at a different CDN

root@ip-172-30-2-30:~# apt-get update
...
Reading package lists... Done
W: GPG error: http://apt.puppetlabs.com trusty Release: The following signatures were invalid: BADSIG 7F438280EF8D349F Puppet, Inc. Release Key (Puppet, Inc. Release Key) <release@puppet.com>

Is there any way to force an update to all of your CDN servers?

Test that demonstrates the problem/ fix, starting with an AWS instance in us-west-2c:

ubuntu@ip-172-30-2-30:~$ sudo -s
root@ip-172-30-2-30:~# apt-get update
Ign http://us-west-2.ec2.archive.ubuntu.com trusty InRelease
....
Get:28 http://security.ubuntu.com trusty-security/universe Translation-en [91.4 kB]
Fetched 11.9 MB in 4s (2,499 kB/s)                                             
Reading package lists... Done

root@ip-172-30-2-30:~# wget https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb
--2017-05-09 11:20:24--  https://apt.puppetlabs.com/puppetlabs-release-pc1-trusty.deb
Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 52.84.21.35, 52.84.21.66, 52.84.21.107, ...
Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|52.84.21.35|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 13652 (13K) [application/x-debian-package]
Saving to: ‘puppetlabs-release-pc1-trusty.deb’
 
100%[===================================================================================================================================================================>] 13,652      --.-K/s   in 0s      
 
2017-05-09 11:20:24 (456 MB/s) - ‘puppetlabs-release-pc1-trusty.deb’ saved [13652/13652]
 
root@ip-172-30-2-30:~# dpkg -i puppetlabs-release-pc1-trusty.deb 
Selecting previously unselected package puppetlabs-release-pc1.
(Reading database ... 51307 files and directories currently installed.)
Preparing to unpack puppetlabs-release-pc1-trusty.deb ...
Unpacking puppetlabs-release-pc1 (1.1.0-2trusty) ...
Setting up puppetlabs-release-pc1 (1.1.0-2trusty) ...

Apt-update now process a BADSIG error:

root@ip-172-30-2-30:~# apt-get update
Ign http://us-west-2.ec2.archive.ubuntu.com trusty InRelease
....
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/main Sources        
Get:1 http://apt.puppetlabs.com trusty Release.gpg [841 B]                     
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/restricted Sources  
....
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/restricted amd64 Packages
Get:2 http://apt.puppetlabs.com trusty Release [54.2 kB]                       
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-updates/universe amd64 Packages
....
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/universe amd64 Packages
Ign http://apt.puppetlabs.com trusty Release                                   
Hit http://us-west-2.ec2.archive.ubuntu.com trusty-backports/multiverse amd64 Packages
....
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Get:3 http://apt.puppetlabs.com trusty/PC1 amd64 Packages [26.4 kB]
Ign http://apt.puppetlabs.com trusty/PC1 Translation-en_US         
Ign http://apt.puppetlabs.com trusty/PC1 Translation-en
Fetched 81.4 kB in 2s (35.9 kB/s)
Reading package lists... Done
W: GPG error: http://apt.puppetlabs.com trusty Release: The following signatures were invalid: BADSIG 7F438280EF8D349F Puppet, Inc. Release Key (Puppet, Inc. Release Key) <release@puppet.com>

Checking the CDN server:

root@ip-172-30-2-30:~# nslookup apt.puppetlabs.com
Server:		172.30.0.2
Address:	172.30.0.2#53
 
Non-authoritative answer:
apt.puppetlabs.com	canonical name = d5lz8ppryy0af.cloudfront.net.
Name:	d5lz8ppryy0af.cloudfront.net
Address: 52.84.21.224
Name:	d5lz8ppryy0af.cloudfront.net
Address: 52.84.21.226
Name:	d5lz8ppryy0af.cloudfront.net
Address: 52.84.21.229
Name:	d5lz8ppryy0af.cloudfront.net
Address: 52.84.21.35
Name:	d5lz8ppryy0af.cloudfront.net
Address: 52.84.21.66
Name:	d5lz8ppryy0af.cloudfront.net
Address: 52.84.21.107
Name:	d5lz8ppryy0af.cloudfront.net
Address: 52.84.21.122
Name:	d5lz8ppryy0af.cloudfront.net
Address: 52.84.21.127
 
root@ip-172-30-2-30:~# ping apt.puppetlabs.com
PING d5lz8ppryy0af.cloudfront.net (52.84.21.226) 56(84) bytes of data.
64 bytes from server-52-84-21-226.sea32.r.cloudfront.net (52.84.21.226): icmp_seq=1 ttl=239 time=7.35 ms
64 bytes from server-52-84-21-226.sea32.r.cloudfront.net (52.84.21.226): icmp_seq=2 ttl=239 time=7.41 ms
 
--- d5lz8ppryy0af.cloudfront.net ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 7.352/7.384/7.416/0.032 ms

Pointing at a different server, from a working host in the UK:

root@ip-172-30-2-30:~# vi /etc/hosts
root@ip-172-30-2-30:~# cat /etc/hosts
127.0.0.1 localhost
 
216.137.63.14 apt.puppetlabs.com                   << Added this
 
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

Succeeding apt-get update:

root@ip-172-30-2-30:~# apt-get update
root@ip-172-30-2-30:~# apt-get update
Ign http://us-west-2.ec2.archive.ubuntu.com trusty InRelease
....
Hit http://security.ubuntu.com trusty-security InRelease                       
Ign http://apt.puppetlabs.com trusty InRelease                               
Hit http://security.ubuntu.com trusty-security/main Sources               
Get:1 http://apt.puppetlabs.com trusty Release.gpg [836 B]         
Hit http://security.ubuntu.com trusty-security/universe Sources   
Hit http://apt.puppetlabs.com trusty Release                      
Hit http://security.ubuntu.com trusty-security/main amd64 Packages 
Hit http://apt.puppetlabs.com trusty/PC1 amd64 Packages
Hit http://security.ubuntu.com trusty-security/universe amd64 Packages
....
Hit http://security.ubuntu.com trusty-security/universe Translation-en
Ign http://apt.puppetlabs.com trusty/PC1 Translation-en_US         
Ign http://apt.puppetlabs.com trusty/PC1 Translation-en
Fetched 836 B in 2s (348 B/s)
Reading package lists... Done
 
root@ip-172-30-2-30:~# ping apt.puppetlabs.com
PING apt.puppetlabs.com (216.137.63.14) 56(84) bytes of data.
64 bytes from apt.puppetlabs.com (216.137.63.14): icmp_seq=1 ttl=238 time=155 ms
64 bytes from apt.puppetlabs.com (216.137.63.14): icmp_seq=2 ttl=238 time=155 ms
64 bytes from apt.puppetlabs.com (216.137.63.14): icmp_seq=3 ttl=238 time=155 ms



 Comments   
Comment by Jordi [ 2017/05/09 ]

Is anyone looking into this? I know it has been reported already but having the same issues ...

sudo apt-get update
............                     
Get:1 http://apt.puppetlabs.com trusty Release.gpg [841 B]                     
Hit http://eu-west-1.ec2.archive.ubuntu.com trusty-updates/main Sources        
Hit http://apt.newrelic.com newrelic Release.gpg                               
Hit http://apt.puppetlabs.com trusty Release                                   
Hit http://eu-west-1.ec2.archive.ubuntu.com trusty-updates/restricted Sources  
Hit http://eu-west-1.ec2.archive.ubuntu.com trusty-updates/universe Sources    
Hit http://eu-west-1.ec2.archive.ubuntu.com trusty-updates/multiverse Sources  
............                         
Err http://apt.puppetlabs.com trusty Release                       

W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://apt.puppetlabs.com trusty Release: The following signatures were invalid: BADSIG 7F438280EF8D349F Puppet, Inc. Release Key (Puppet, Inc. Release Key) <release@puppet.com>
W: Failed to fetch http://apt.puppetlabs.com/dists/trusty/Release  
W: Some index files failed to download. They have been ignored, or old ones used instead.

; <<>> DiG 9.9.5-3ubuntu0.14-Ubuntu <<>> apt.puppetlabs.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62772
;; flags: qr rd ra; QUERY: 1, ANSWER: 9, AUTHORITY: 0, ADDITIONAL: 1
 
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;apt.puppetlabs.com.		IN	A
 
;; ANSWER SECTION:
apt.puppetlabs.com.	1	IN	CNAME	d5lz8ppryy0af.cloudfront.net.
d5lz8ppryy0af.cloudfront.net. 44 IN	A	54.192.28.28
d5lz8ppryy0af.cloudfront.net. 44 IN	A	54.192.28.58
d5lz8ppryy0af.cloudfront.net. 44 IN	A	54.192.28.83
d5lz8ppryy0af.cloudfront.net. 44 IN	A	54.192.28.103
d5lz8ppryy0af.cloudfront.net. 44 IN	A	54.192.28.123
d5lz8ppryy0af.cloudfront.net. 44 IN	A	54.192.28.156
d5lz8ppryy0af.cloudfront.net. 44 IN	A	54.192.28.167
d5lz8ppryy0af.cloudfront.net. 44 IN	A	54.192.28.178
 
;; Query time: 1 msec
;; SERVER: 172.16.0.23#53(172.16.0.23)
;; WHEN: Tue May 09 14:37:38 UTC 2017
;; MSG SIZE  rcvd: 217

Thanks.

Comment by Morgan Rhodes [ 2017/05/09 ]

We're working on this right now, should be working again shortly.

Comment by Morgan Rhodes [ 2017/05/09 ]

This should be working now, let us know if you have any more issues!

Comment by Jordi [ 2017/05/09 ]

Morgan Rhodes It works fine for me.

 wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb -O /tmp/puppetlabs-release-trusty.deb
--2017-05-09 16:56:39--  https://apt.puppetlabs.com/puppetlabs-release-trusty.deb
Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 54.192.28.167, 54.192.28.178, 54.192.28.28, ...
Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|54.192.28.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16944 (17K) [application/x-debian-package]
Saving to: ‘/tmp/puppetlabs-release-trusty.deb’
 
100%[=============================================================================================================================================================================================>] 16,944      --.-K/s   in 0s      
 
2017-05-09 16:56:39 (183 MB/s) - ‘/tmp/puppetlabs-release-trusty.deb’ saved [16944/16944]

sudo dpkg -i /tmp/puppetlabs-release-trusty.deb
Selecting previously unselected package puppetlabs-release.
(Reading database ... 98506 files and directories currently installed.)
Preparing to unpack .../puppetlabs-release-trusty.deb ...
Unpacking puppetlabs-release (1.1-1) ...
Setting up puppetlabs-release (1.1-1) ...

sudo apt-get update
..................................
Ign http://apt.puppetlabs.com trusty InRelease
Get:1 http://apt.puppetlabs.com trusty Release.gpg [841 B]                     
Get:2 http://apt.puppetlabs.com trusty Release [54.2 kB]  
..................................

sudo apt-get install -y puppet
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  efibootmgr grub-efi-amd64-bin grub-pc-bin libefivar0 mokutil sbsigntool
  secureboot-db shim
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
  augeas-lenses debconf-utils facter hiera libaugeas-ruby libaugeas0
  puppet-common ruby-augeas ruby-json ruby-shadow virt-what
Suggested packages:
  augeas-doc augeas-tools puppet-el vim-puppet ruby-selinux libselinux-ruby1.8
  librrd-ruby1.9.1 librrd-ruby1.8
The following NEW packages will be installed:
  augeas-lenses debconf-utils facter hiera libaugeas-ruby libaugeas0 puppet
  puppet-common ruby-augeas ruby-json ruby-shadow virt-what
0 upgraded, 12 newly installed, 0 to remove and 7 not upgraded.
Need to get 1,874 kB of archives.
After this operation, 11.0 MB of additional disk space will be used.
Get:1 http://apt.puppetlabs.com/ trusty/main facter all 2.4.6-1puppetlabs1 [73.3 kB]
Get:2 http://apt.puppetlabs.com/ trusty/main hiera all 1.3.4-1puppetlabs1 [12.0 kB]
Get:3 http://apt.puppetlabs.com/ trusty/main puppet-common all 3.8.7-1puppetlabs1 [1,269 kB]
..................................

Thanks a mil.

J.

Comment by Tom Hey [ 2017/05/09 ]

Work for me as well Thanks for fixing this quickly

Generated at Thu Jan 23 01:42:17 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.