[PUP-731] Masters cannot reliably distinguish between multiple versions of a type/function/plugin used in different environments Created: 2013/11/13  Updated: 2016/12/12  Resolved: 2016/09/13

Status: Closed
Project: Puppet
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Normal
Reporter: redmine.exporter Assignee: Andrew Parker
Resolution: Fixed Votes: 38
Labels: customer, redmine
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Relates
relates to PUP-6999 Revise Environments Limitations page ... Closed
relates to SERVER-94 Environment Isolation Closed
relates to PUP-1033 Updated functions are not reloaded wi... Closed
relates to PUP-2151 Have master and agent use different l... Closed
Template:

 Description   

Quoted from a previous ticket, #4409

The key problem is that a master can’t reliably distinguish between two versions of the same native type that are used in different environments (or between an environment which uses a native type and an environment which doesn’t). This is due to the fact that native types are defined using ruby code, which the master loads into Ruby classes via “require”. Since there can only be one Ruby class with a given name at a time, this prevents the master from being able to have two different versions of the same type in two different environments. This makes life difficult for people who are trying to use a “test” environment to try out a new version of a native type on a limited set of nodes before deploying it to all nodes.

A secondary problem is that the location where the master looks for the definition of native types is not the same as the location of plug-in files that the master distributes to agents. This leads to confusion even for people who are not using a “test” environment, because it means that they have to put their type definitions in two places, one where they can be picked up by the master and one where they can be sent as plug-ins to agents.



 Comments   
Comment by Patrick Hemmer [ 2014/02/07 ]

I've been trying to load a hiera backend via a puppet module, and after reading numerous bug reports have ended up here.
The problem I'm running into is that the hiera backend is defined in `modules/hiera-gpg/lib/hiera/backend/gpg_backend.rb`. This path is not added to the ruby $LOAD_PATH, and so trying to use the backend fails with `Notice: hiera(): Cannot load backend gpg: cannot load such file – hiera/backend/gpg_backend`.

I do not use environments at all as I think they are a very ugly solution to what they are meant to solve. But this bug still affects me as the module doesn't load.

So I'm posting this comment to provide a workaround for anyone else coming up with this issue.
My solution is to add some code at the very beginning of `site.pp` to modify `$LOAD_PATH`. Yes it's a hack, but it works.

$load_path = inline_template('<%
scope.lookupvar("::settings::modulepath").split(":").each do |mp|
  Dir.glob(mp + "/*/lib").sort.each do |mlp|
    $LOAD_PATH.unshift(mlp)
    $LOAD_PATH.uniq!
  end
end
%>')

It basically goes through every module in your `modulepath` setting and adds the lib dirs (that exist) to `$LOAD_PATH`.

Comment by Steven Willis [ 2014/02/18 ]

Hey Patrick Hemmer,

I'm wondering if your problem isn't exactly related to this issue. It could be that the hiera-gpg ruby gem was installed into a location that puppet doesn't normally look. I think there's a difference between puppet's modulepath and ruby's (and ruby gem's) $LOAD_PATH. Maybe you installed hiera-gpg into puppet's modulepath rather than where gems should go?

Not to buzz market myself, but I wrote a similar module that installs as a puppet module rather than a ruby gem. That way, you can install it with: puppet module install and be sure it's in puppet's modulepath. You can read about it here: http://forge.puppetlabs.com/compete/hiera_yamlgpg.

Also, I noticed this issue has been marked as 'Needs Information'. What information does it need? The comments on the original bug report outlined the issue pretty well. I could probably help give some more information if necessary.

Comment by Patrick Hemmer [ 2014/02/18 ]

Steven Willis That's exactly where the gem is going, and yes there is a difference between modulepath and $LOAD_PATH, which is exactly why I had to put in my "hack".
This situation is covered by this issue as explained in the second paragraph of the issue description: "A secondary problem is that the location where the master looks for the definition of native types is not the same as the location of plug-in files that the master distributes to agents."

Comment by Steven Willis [ 2014/02/18 ]

Patrick Hemmer, I was trying to say that I think puppet's modulepath is probably not the place for ruby gems. That's what I'm interpreting your modules/ directory to be in your example. I think what you want to do is install hiera-gpg into the standard ruby gem location rather than puppet's modulepath.

For example, we're running puppet enterprise, so our gem install path is /opt/puppet/lib/ruby/gems/1.9.1. Under this directory for us is gpgme-2.0.2 and others. This is where I think you'd want to install hiera-gpg. Our puppet modulepath for the master is: /etc/puppetlabs/puppet/modules:/opt/puppet/share/puppet/modules:/etc/puppetlabs/puppet/env/production/modules. I don't believe you'd want to install gems under these paths (again, that's how I was interpreting what's happening in our case, if I'm wrong, please excuse me).

As you can see from the last entry in our modulepath, we'd like to be able to use per environment modules, but that doesn't work great (hence this ticket).

Comment by Patrick Hemmer [ 2014/02/18 ]

Supporting putting gems in the modulepath is mandatory.

Consider the scenario where you deploy a repo that has hiera encrypted data, and now puppet needs the hiera-gpg gem to work. So before you deploy the repo, you have to install hiera-gpg. Ok, so you make a commit to your puppet repo to install the hiera-gpg package first, and then you make another commit to start using it. Cumbersome, but manageable.
But now lets say you have multiple independent environments (development, test, staging, production), and you merge your development code into staging. Both those commits are going to go at the same time, and now you've broken puppet since the repo requires hiera-gpg, but hiera-gpg isn't installed.
Also consider the scenario where you're not using puppet masters, but using puppet standalone instead. When puppet is run for the first time it'll fail for the same reason as before (hiera-gpg not installed, but repo needs it). So now you have to start installing these packages by hand (or as part of whatever installs puppet). While this is doable, I think it should be avoided if not necessary, which by putting the gem in the repo itself, makes it unnecessary.

So the solution is to deploy the repo along with the dependencies needed to make the repo work. This is what tools like bundler were created for. And in effect, my little "hack" is one of the things bundler does.

Comment by Steven Willis [ 2014/02/18 ]

Supporting putting gems in the modulepath is mandatory.

I don't believe that's supported right now, nor intended to be supported. If that's something you need, then I think you should open a different issue, since I don't think it's really related to this issue.

As to your case about needing to have a puppet module or ruby gem installed prior to use, that does kind of stink and we run into issues with that in lots of different areas. Sometimes the best you can do is just know that after 2 runs of puppet agent, it'll work. For us, since we use a master, we just need to make sure the right puppet modules and ruby gems exist on the master in the right place. It doesn't really matter for the agents since they don't run the code (except for facts of course).

Comment by Felix Frank [ 2014/02/19 ]

+1 about the gem topic being a thread of its own.

As far as I'm concerned, the meat of this here ticket is summerized in this comment: http://projects.puppetlabs.com/issues/12173#note-8

Comment by Andrew Parker [ 2014/02/28 ]

Yes, I think the gem topic is different from this.

I've changed the status to "Designing" since this is a problem that we've known about for a long time and we've had a lot of ideas about how it needs to proceed. Felix's comment about note-8 in 12173 being the most complete and accurate writeup is right. The current plan to move this forward is to work toward running puppet using JRuby and then we can ensure that each environment is fully separate.

Comment by Axel Bock [ 2014/04/03 ]

Hi Guys, I would really appreciate a solution here. I am hitting this issue HARD right now and it is really a pain. Waiting for JRuby sounds ... very long to me. Any chance of having this earlier?

Comment by Felix Frank [ 2014/04/03 ]

How is it hitting you? Perhaps we can suggest a workaround for the time being.

Barring support for JRuby, there are just immense technical barriers that keep us from implementing a fix.

Comment by Axel Bock [ 2014/04/03 ]

I'm writing a function for puppet to use, and I am getting an error I cannot reproduce outside of puppet. The current workaround is ...

  • change the function on teh puppet master
  • run puppet agent --noop on the puppet master to sync
  • restart the puppet master (or httpd in that case)
  • run puppet on the client.
    If I knew how I could reproduce exactly the ruby runtime under which puppet runs could cut puppet out of it and work with ruby scripts, which would be massively more productive. But I don't, and I'm fairly new to ruby so my improvise-fu is limited :/
Comment by Felix Frank [ 2014/04/03 ]

Ah, functions. That's a lot easier than types, actually. However, there will likely be no fix in Puppet 3.x (correct me if I'm wrong, Andy) since the function API will change in Puppet 4 and this problem will resolve itself.

In the meantime, you can take measures to ensure that your function code is reloaded in a timely fashion.

I have a patch that enables a brute force puppet master option, see https://github.com/ffrank/puppet/tree/ticket/master/17210-reload-functions

An less intrusive alternative is to configure your Passenger to have each puppet master process handle only one catalog request, then terminate. Apache will keep launching new masters for you. Note that this will carry a performance penalty. A compromise would be to do this only in a vhost on an alternate port, use that port for your function debugging, and once you're satisfied, do restart apache to make the current state visible to all master processes.

Comment by Axel Bock [ 2014/04/03 ]

would that passenger change also solve the issue that the client gets the function which is under /var/lib/puppet/... on the master? because currently this is the case. but that sounds like a relief, actually, I think I'm gonna try that. any docs which point out the relevant passenger flags?

Comment by Felix Frank [ 2014/04/03 ]

http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerMaxRequests

I'm actually unsure wether the client will indeed prefer /var/lib/puppet/lib as a download source over the actual module location(s). My feeling is that is should not, and if it does, that this would be a bug in and of itself (not related to the problem here). If you continue to face this particular problem, please file a new ticket or find an existing one that fits the effect you are observing. Thanks!

Comment by John Julien [ 2014/04/03 ]

I saw the issue with the master serving out of /var/lib. Trying to upgrade from Puppet 2.x to Puppet 3.x on my master, the 3.x master was serving functions that ship from 2.x because he was checking into our global puppet vip which was still 2.x. So he was using the pluginsync version of his client functions to compile catalogs.

The ensure_resources.rb function was what threw the error and alerted us to this. symbolizehash! was removed from the Puppet Core in 3.x but is called out in create_resources.rb in 2.x.

I figured this was the same bug until I read the last comment by Felix Frank

Comment by Axel Bock [ 2014/04/03 ]

Felix Frank yup, it very much seems that the client is served from /var/lib/puppet/lib/... . I'll verify this and act accordingly. Thanks for your help!

Comment by Andrew Parker [ 2014/04/03 ]

One of the issues here is that both the master and the agent share the same libdir location and that the agent's default plugindest is the libdir and that puppet adds libdir to the ruby LOAD_PATH and that the ruby LOAD_PATH is consulted when loading plugins. What this causes is that the environment of the agent that runs on the master will leak into the other environments . You should be able to get around this by changing either the libdir or the plugindest in the agent section of your puppet.conf.

If this solves a large portion of the problem that you are seeing, then we can make a simple fix during 3.x, I think, by just changing the default libdir to be $vardir/agent/lib and $vardir/master/lib.

Comment by John Julien [ 2014/04/03 ]

I modified puppet.conf to use $vardir/agent/lib and $vardir/master/lib, restarted pe-httpd. The result was a 500 error from clients. I created $vardir/master/lib and $vardir/agent/lib and my client runs were then successful.

While both $vardir/

{master,agent}

/lib were empty my client runs were successful against the master. After running the puppet agent against a 2.x master, $vardir/agent/lib was populated. The client runs against this 3.x master continue to be successful.

Everything seems to be working with the 3.x master compiling catalogs using 3.x shipped functions, regardless of the fact that it checks into a 2.x master. One thing that seemed odd to me is none of the core puppet functions are being included in pluginsync now for clients hitting this 3.x master. Was that a 3.x change perhaps?

This is going to make our upgrade so much easier, thank you!

Comment by Andrew Parker [ 2014/04/03 ]

John Julien, good to hear that it worked. I'm not certain about the change that you are asking about wrt core puppet functions. It might be that we now exclude functions from pluginsync since they are only needed on the master.

Comment by Felix Frank [ 2014/04/04 ]

I concur. It's my suspicion that what John Julien observed earlier was a bug triggered by a peculiar master config. The master is not supposed to include the puppet core in the plugins that get synced to agents.

Comment by Jon McKenzie [ 2014/08/25 ]

As an interim step, would it at least be possible to notify (e.g. via debug messages) that distributed lib has been loaded more than once, or to specify the path of the lib that was loaded? We ran into a similar issue in our environment, and the error messages (which essentially boiled down to 'invalid parameter' messages) were extremely unhelpful in determining the ultimate root cause.

Comment by Nick Moriarty [ 2015/01/20 ]

I'd like to ask for some clarification around Puppetmaster's behaviour with this sort of problem, as we want to run with directory environments but wish to know a little more about how and when this issue occurs (with a view to working around it).

We run Puppetmaster 3.7 on Apache/Passenger. Currently, all environments are served by one passenger pool, so the Puppetmasters will no doubt be prone to this issue, as systems in different environments will be randomly assigned to free workers.

What I'm not sure of, and hoping that someone may be able to clarify - in the instance that there are multiple directory environments present at startup, will a given Puppetmaster process behave correctly as long as it is only serving requests for a single environment (e.g. because of a reverse proxy segregating requests), or will it load (some?) plugins from each environment at startup, and therefore always be in an inconsistent state?

If it's only "polluting" Ruby constants (and hence potentially hitting this problem) when it's actually compiling catalogs for hosts in multiple environments (and not loading things at startup), I believe I could potentially work around this problem by configuring Passenger to use separate worker pools per environment (as the URIs appear to all begin with the environment name, so it should be able to easily separate the requests). Is this likely to work with any form of reliable segregation between environments?

A point I'm particularly interested in is how Puppetmaster behaves if an ENC overrides a node's environment - does it stop there and let the node re-request its catalogue for the right environment, or does it 'switch' into the other environment? We may wish to use ENC-supplied environments in the near future, and I realised that the workaround I'm talking about could be prone to issues if puppetmaster processes ever perform a switch like this.

Finally, is there an expected behaviour with regards to whether a Puppetmaster process will see one version or another (e.g. whichever was loaded first / most recently), or is it even possible to end up with neither (e.g. if the versions interfere or end up merged in some manner)?

I guess answers to the above might be helpful to others in deciding how to work with environments.

Many thanks.

Comment by Felix Frank [ 2015/01/20 ]

Hi,

as this discussion does not pertain to a possible resolution, it would
be better to take it to the puppet-users mailing list, actually. But now
that we are here, see comments inline.

This should work as far as I can tell. The most common failure is
updating Ruby code that has been changed during a master process's
lifetime, but sure, inconsistencies might also occur.

So long as a given process is confined to one environment, it will only
suffer stale type/function definitions, no mixups.

The whole transaction is restarted, with the agent issuing a new catalog
request. After all, the set of available facts might differ among
environments.

I'm not certain just when custom types are loaded. For custom functions,
those are picked up when needed. You could very well end up with
different cached functions that are from different points in time, with
neither function changing internally until the process terminates.

Comment by Henrik Lindberg [ 2015/01/20 ]

Just to be clear, the long term solution is covered by SERVER-94 since no matter how much isolation we add to the immediately loaded "puppet objects" (loading them in a way that they do not step on each other), any logic that they in turn load that is not written to exist in multiple versions at the same time will never work without the sandboxing that the SERVER-94 epic is about.

Comment by Jo Rhett [ 2015/01/20 ]

SERVER-94 is an empty epic with no description and no issues. Hardly useful to refer us to that.

Comment by Henrik Lindberg [ 2015/01/20 ]

Sorry about it not being more detailed, It is still the epic that will contain details (and detailed tickers) about the sandboxing feature that will be developed for Puppet Server.

Comment by Eric Sorenson [ 2016/09/13 ]

The Type part of this is fixed as part of Puppet 4.6's pcore type implementation.
The Functions part of it is fixed in the Puppet 4 Function API.
The remaining bits are tracked in the SERVER-96 epic.

Comment by Zee Alexander [ 2016/09/13 ]

Not Eric Sorenson orly? If I write a function in Ruby, using the most up-to-date method calls and whatnot, it will successfully be segregated by environment with multiple versions of the same Ruby function?

Comment by Henrik Lindberg [ 2016/09/13 ]

Zee Alexander yes, if you use the 4.x function API, and 4.6.2 or later. In earlier versions there are problems with functions written in the puppet language under some conditions (addressed in 4.6.2).

Comment by Chris Price [ 2016/09/13 ]

Henrik Lindberg isn't it also the case that in Puppet 3, ruby functions would be env-isolated as long as you didn't do anything fancy like define a ruby class or method inside your puppet function definition? (I know that's not a complete or acceptable solution, but since we seem to be enumerating the actual state of the issue here...)

Comment by Zee Alexander [ 2016/09/13 ]

Henrik Lindberg so, what's the current state of issues with environment isolation and ruby extensions to Puppet?

In architect we're still warning people about this in general, so I'd like to correct the record if possible.

Comment by Henrik Lindberg [ 2016/09/14 ]

I believe SERVER-94 is the ticket for the remaining work on the Puppet Server side.

The state of Ruby code isolation in general in Puppet > 4.6.2 is:

  • 4.x functions are safe provided the rules documented in the 4.x function API are followed (for example, do not bind to Ruby constants)
  • 3.x functions are safe, but not if they have external helper code (same rules as for 4.x apply, no binding to constants etc.)
  • Resource types are safe provided that the generate types command has been run (manually or automatically by Puppet Server). Here, if user changes implementation, the command must be executed again. The result is that no ruby code is loaded for the resource types
  • Helper code - helper Ruby code that is not inside of functions is not safe. It must be handled as an extension to puppet (i.e. the same version used in all environments, just like any other gem used by puppet).

We know we need to work on the documentation and messaging around env isolation support.

Comment by Jo Rhett [ 2016/12/12 ]

Really should update https://docs.puppet.com/puppet/latest/environments_limitations.html to indicate which versions the big warning at the top applies to, now that this is fixed.

Comment by Henrik Lindberg [ 2016/12/12 ]

Ping Jean Bond - see Jo Rhett's comment above. (This is "environment isolation").

Comment by Jean Bond [ 2016/12/12 ]

I'm not going to lie, that's one of my favorite docs headings ever. Also, making a ticket for the docs work now, thanks!

Generated at Thu Dec 12 02:34:41 PST 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.