[CPR-401] Inconsistent information returned by apt.puppetlabs.com Created: 2017/01/11  Updated: 2017/08/02  Resolved: 2017/01/25

Status: Closed
Project: Community Package Repository
Component/s: Download Infrastructure
Affects Version/s: None
Fix Version/s: 2017/08/02

Type: Bug Priority: Normal
Reporter: Tim Bishop Assignee: Morgan Rhodes
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File kestrel.png     PNG File willow.png    
Template:
QA Risk Assessment: Needs Assessment

 Description   

I have two servers in the same location, both of which resolve apt.puppetlabs.com the same way:

kestrel % host apt.puppetlabs.com
apt.puppetlabs.com is an alias for downloads.puppetlabs.com.
downloads.puppetlabs.com is an alias for burji3.puppetlabs.com.
burji3.puppetlabs.com has address 192.155.89.90
burji3.puppetlabs.com has IPv6 address 2600:3c03::f03c:91ff:fedb:6b1d

willow % host apt.puppetlabs.com
apt.puppetlabs.com is an alias for downloads.puppetlabs.com.
downloads.puppetlabs.com is an alias for burji3.puppetlabs.com.
burji3.puppetlabs.com has address 192.155.89.90
burji3.puppetlabs.com has IPv6 address 2600:3c03::f03c:91ff:fedb:6b1d

However, they're returning different data for at least the amd64 xenial packages:

kestrel % curl -s https://apt.puppetlabs.com/dists/xenial/PC1/binary-amd64/Packages | grep '1\.8\.2'
kestrel %

(so no output)

willow % curl -s https://apt.puppetlabs.com/dists/xenial/PC1/binary-amd64/Packages | grep '1\.8\.2'
Version: 1.8.2-1xenial
Filename: pool/xenial/PC1/p/puppet-agent/puppet-agent_1.8.2-1xenial_amd64.deb
willow %

So the first server isn't getting the latest packages because the package list is out of date.

I can't see how this is my fault, thus the ticket. My best guess is that burji3.puppetlabs.com is a load balancer and has multiple servers behind it, one of which isn't updating properly.



 Comments   
Comment by Morgan Rhodes [ 2017/01/25 ]

Tim Bishop we've been unable to reproduce this and everything looks right on our end, are you still seeing this?

Comment by Tim Bishop [ 2017/01/25 ]

Yes, still seeing the problem. In fact, it can pretty obviously be seen just by looking at the index page. Compare:

The first one shows older files.

Unless there's something really peculiar going on at my end, it must be that there's more than one server at your's. Is that correct? Are you load balancing? Maybe stick a file at the top level of each server saying which machine it is, then I can tell you which one I'm getting content from.

Comment by Morgan Rhodes [ 2017/01/25 ]

Tim Bishop I'd suggest you check for odd entries in /etc/hosts or something weird with DNS caching on your end. The server seeing older files seems to be hitting our old download server. These sync occasionally (and I'll start one now), but to get the latest software you'll want to make sure you're hitting our current server.

Comment by Tim Bishop [ 2017/01/25 ]

It's literally just started working, so whatever you did affected it. I'll double check DNS, etc.

Comment by Morgan Rhodes [ 2017/01/25 ]

I just synced the old server. Definitely check DNS though.

Comment by Tim Bishop [ 2017/01/25 ]

It was my fault ultimately, not your's. A check of curl in verbose mode showed it was connecting to a different IP, so once I'd fixed it everything is fine. Totally sorry for wasting your time!

Comment by Morgan Rhodes [ 2017/01/25 ]

No worries! Glad it's resolved now!

Generated at Mon Jan 27 11:39:59 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.