[PUP-2172] Don't use an http proxy when connecting to localhost Created: 2014/04/07  Updated: 2019/09/16  Resolved: 2019/08/14

Status: Resolved
Project: Puppet
Component/s: None
Affects Version/s: PUP 3.4.3, PUP 5.4.0
Fix Version/s: PUP 5.5.17, PUP 6.4.4, PUP 6.8.0

Type: Bug Priority: Normal
Reporter: Kylo Ginsberg Assignee: Josh Cooper
Resolution: Fixed Votes: 1
Labels: resolved-issue-added
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PUP-9316 implement no_proxy in puppet.conf Resolved
Template: PUP Bug Template
Epic Link: Agent HTTP
Team: Coremunity
Story Points: 3
Sprint: Platform Core KANBAN
Release Notes: Bug Fix
Release Notes Summary: If an HTTP proxy is configured either in puppet settings or the HTTP_PROXY_* environment variables, then puppet will not use the proxy when connecting to localhost or 127.0.0.1. This behavior can be modified by changing the no_proxy puppet setting or the NO_PROXY environment variable.
QA Contact: Erik Dasher

 Description   

We had a PE site which had set the http_proxy_{port,host} settings for PMT usage, which had the unintended consequence that all puppet master http connections went through the proxy, including those to the dashboard for node lookups. When something (not yet identified) changed in the network/proxy configuration, these lookups failed causing all agent runs to fail.

This raises the question about whether it ever makes sense for the master to use a proxy for connections to localhost.

We've considered two approaches here:
1) add a new setting for http proxy exclusions. This could be set to, e.g. localhost;*.internal.lan, etc.
2) never using the proxy settings when talking to localhost

There may be some other approaches that would work as well.



 Comments   
Comment by Josh Cooper [ 2014/04/07 ]

One fun fact is that the PMT uses two different code paths when making network connections. One path uses open-uri, which relies on environment specific proxy settings. The other path uses puppet's networking code. I think the former is used when searching, the latter when downloading, though I may have those backwards. So we need to make sure that this PR accounts for both scenarios.

This is based off of my recollection from the open source PMT circa 3.3 in the puppet repo. Things may have changed in pe-puppet or in PMT itself that this is no longer an issue.

Comment by Andrew Parker [ 2014/04/09 ]

I think adding a configurable list of excludes makes sense. There are a couple things to keep in mind:

  • The proxy exclusion needs to be reevaluated whenever a redirect is followed (I think this should trivially happen from the existing code)
  • As Josh points out, we aren't consistent everywhere. We should do a check of the code and make sure everything is consistently making HTTP requests
  • Right now some parts respect the http_proxy (or HTTP_PROXY) environment variable, but not everything. We need to combine those code paths so that everything respects the same proxy settings in the same way.
Comment by Doug Rosser [ 2017/05/16 ]

Thank you for filing this issue. We agree it is likely an improvement, but due to other issues demanding precedence, we don’t anticipate being able to address this any time soon. If you are interested in submitting a patch to the repository for this project at https://github.com/puppetlabs, please open a pull request and re-open this ticket. Pending that, we are closing this as “Won’t Fix.” We may revisit it at a later time, and if so will re-open this ticket.

Comment by James Ralston [ 2018/03/13 ]

I'm re-opening this ticket, because I believe the severity of the problem hasn't been fully appreciated.

Any security-competent site that requires outbound HTTP[S] traffic to use a network proxy will have configured the proxy to deny all requests for internal resources. (Otherwise, internal users can bypass internal network ACLs by routing requests through the proxy.)

This means that any application that is proxy-aware must implement an exception list for the proxy. Without this exception list, the application is essentially offering its users two choices:

  • You can access foreign resources, but not local resources.
  • You can access local resources, but not foreign resources.

We see this in the applications that rely on the curl-style (http_proxy, https_proxy, and no_proxy) environment variables. E.g.:

$ set | grep proxy
http_proxy=http://proxy.example.org:8080
https_proxy=http://proxy.example.org:8080
no_proxy=localhost,.example.org

The no_proxy setting is not optional: without this, applications will not be able to access local resources.

As a real-world example of just how critical the no_proxy list is, consider this example:

package { 'r10k':
  ensure   => present,
  provider => puppet_gem,
}

This cannot be implemented with sites that use a proxy.

If I do not configure the agent to use the proxy, then the agent will fail to install the gem:

Error: Execution of '/opt/puppetlabs/puppet/bin/gem install --no-rdoc --no-ri r10k' returned 2: ERROR:  Could not find a valid gem 'r10k' (>= 0), here is why:
          Unable to download data from https://rubygems.org/ - Errno::ENETUNREACH: Failed to open TCP connection to api.rubygems.org:443 (Network is unreachable - connect(2) for "api.rubygems.org" port 443) (https://api.rubygems.org/specs.4.8.gz)

But if I configure the agent to use the proxy, the agent will attempt to use the proxy to connect to the Puppet server, which fails:

Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: SSL_connect returned=1 errno=0 state=error: certificate verify failed: [unable to get local issuer certificate for /CN=puppetserver.example.org]

To put it as bluntly as possible: without the ability to add exception lists, Puppet's proxy handling is fundamentally broken, and for Puppet users who have to use a (properly-configured) proxy server, Puppet's broken proxy handling breaks core features of Puppet.

(The only reason why Puppet has been usable for us until now is because up to this point, we have haven't needed to set proxy settings for Puppet, because we have only used package resources to install RPM packages, and we mirror all yum repositories locally. But we have customers who want to use Puppet package resources with other providers, and mirroring all of rubygems.org, pypi.python.org, et. al. locally just to work around Puppet's broken proxy handling is a really tough sell.)

Comment by Josh Cooper [ 2019/08/07 ]

Tickets PUP-9942 and PUP-9316 allow a proxy exception list to be specified via NO_PROXY environment variable or Puppet[:no_proxy] puppet setting respectively, which I think addresses most of the concerns in this ticket.

Ticket PUP-8027 covers the gem provider issue.

The remaining issue is whether puppet should not use a proxy for some hosts by default. Skipping localhost is easy enough, but I don't think we should skip hosts in the same domain by default. That could be a breaking change for some, and be highly dependent on the network environment. For this ticket, I'm thinking we should just skip localhost by default.

Comment by Josh Cooper [ 2019/08/12 ]

Merged to 5.5.x in https://github.com/puppetlabs/puppet/commit/e8d183a7774677331632dbbb6b83a51ca49b07d9

Comment by Alexandru Popa [ 2019/08/14 ]

Josh Cooper can you please add release notes if needed. Also I think this ticket can be closed because the commit is on master branch and the CI is green.

Thanks,

Alex

Comment by Josh Cooper [ 2019/08/14 ]

Passed CI in c956ad95fcd

Generated at Thu Jan 23 01:03:33 PST 2020 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.