[PUP-1073] Support common package name in two different providers Created: 2013/12/16  Updated: 2017/05/17  Resolved: 2015/02/17

Status: Closed
Project: Puppet
Component/s: Docs, Types and Providers
Affects Version/s: PUP 2.7.25, PUP 3.3.1
Fix Version/s: PUP 4.0.0

Type: Bug Priority: Normal
Reporter: redmine.exporter Assignee: Michael Smith
Resolution: Fixed Votes: 16
Labels: redmine
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by PUP-4276 different package types share namespace Closed
is duplicated by PUP-2313 Package resources should use $title r... Closed
Relates
relates to PUP-4184 Puppet resource package behaves incon... Closed
relates to PUP-6516 support the same service name in reso... Accepted
relates to PUP-1287 Manage both 32 and 64-bit packages wi... Closed
relates to PUP-3721 Puppet resource should always emit th... Ready for Engineering
relates to SERVER-360 Fix createrepo issue with ruby puppet... Closed
Template:
Epic Link: 4.0 Important Fixes
Story Points: 3
Sprint: Platform Client 2014-11-26, Platform Client 2014-12-17, Platform Client 2015-01-07, Platform Client 2015-01-21
QA Contact: Kurt Wall

 Description   

I have a common package name, that's in two different package managers
(one with yum the other with gem)

        package { "remove-mysql":
                name     => "mysql",
                provider => "yum",
                ensure   => absent,
        }

---------------------------------------------------------

        package { "gem-mysql":
                name    => "mysql",
                ensure   => "2.7",
                provider => gem,
        }

---------------------------------------------------------

I get this error.

Jul  3 08:43:34 puppetd[11872]: Could not retrieve catalog:
Puppet::Parser::AST::Resource failed with error ArgumentError: Cannot
alias Package[gem-mysql] to mysql; resource Package[mysql] already
exists at /etc/puppet/modules/ruby-mysql/manifests/init.pp:11 on node 


The solution to this was to generate a uniqueness key based on [name, provider], as mentioned by Andrew Parker. That allows the example above to work, but doesn't resolve when multiple providers manage the same package. Several changes were needed:

  1. Set paramclass(:provider).isnamevar on the Package type, so the :provider parameter is recognized as part of the namevar.
  2. Overloaded the title_patterns method on the Package type, so the title could be parsed into namevars (it only sets :name, not :provider).
  3. The catalog assumed that isomorphic resources only needed to be aliased if the title and name didn't match. An alias is automatically created for a resource's title; if titles don't match but the namevar does they should still refer to the same thing. However, name is not equivalent to namevar for a composite namevar, so the test for whether to create an alias uses the uniqueness_key (an array containing all namevars). The impact of this change is to correctly identify when a resource with a composite namevar has already been created, as in

    package { "add-mysql":
      name   => "mysql",
      ensure => present,
    }
     
    package { "mysql":
      ensure => present,
    }
    

  4. The change to catalog means that creating aliases now depends on the uniqueness_key. That implementation in resource type could end up returning an array without the name specified if the :name parameter wasn't specifically specified. This change was made so conflicts between implicit and explicit default provider are detected at catalog compilation instead of trying to apply the manifest.

Changes (1) and (2) are required for any provider that uses a composite namevar.

Impact of this change:

  • Querying resources by name on the command-line when multiple packages from different providers are specified only returns the package with the default provider (PUP-3721).
  • Referring to Package resources by Package[$namevar] in Puppet code doesn't work for a composite namevar, so it no longer works for Package. Querying by Package[$title] still works. This is a breaking change in Puppet 4.


 Comments   
Comment by Pedro Côrte-Real [ 2013/12/16 ]

Is the solution suggested in the previous ticket valid? I mean having a $pkgname variable that doesn't have to be unique instead of the current use of $name. Last time I had a quick look at the code it seemed to be quite simple to implement and fully backwards compatible.

Comment by Trevor Vaughan [ 2014/02/20 ]

This seems to be something that has been asked for for around 6 or so years.

Any chance of getting some traction on this? Gem conflicts with local package names are absolutely deadly.

Comment by Drew Blessing [ 2014/03/10 ]

This issue is being discussed on the puppet-dev list at https://groups.google.com/forum/#!msg/puppet-dev/LatVZFUkwEM/Y-EyhpLHRHsJ. So far I think the idea Pedro Côrte-Real suggested both here and in the discussion is the most valid solution. I'd love to hear about the feasibility from someone at Puppet Labs.

Comment by Hunter (Hunner) Haugen [ 2014/03/11 ]

I agree with this solution also.

If the holdup is finding a solution to "allow any resource to be declared with non-unique namevars" then we may be thinking too hard about a solution that doesn't fit the problem. The package resource is kind of special in this regard.

Comment by Richard Raseley [ 2014/04/17 ]

+1 for Pedro's suggestion.

Comment by Jason Antman [ 2014/04/18 ]

I'll also throw in my support for Pedro's suggestion. I see the idealistic side of wanting this to work for all types and be fixed in the RAL, but on the other hand, I've heard a lot of complaints/questions/confusion about this issue. Almost all of them involved the Package type. I think even fixing it in a sub-optimal way for the Package type is a big win for end users.

Comment by Richard Pijnenburg [ 2014/08/06 ]

Any update on this?
I'm running against this issue with my Elasticsearch module.

Comment by Stefanos Harhalakis [ 2014/09/24 ]

For the fun of it, here's another case of the same problem:

tidy{'etcmail1':
  path => '/etc/mail',
  age => '30d',
  matches => 'sendmail.mc.[0-9]*
  recurse => 1,
}
tidy{'etcmail2':
  path => '/etc/mail',
  age => '30d',
  matches => 'sendmail.cf.[0-9]*
  recurse => 1,
}

The above causes this:

Cannot alias Tidy[xxx] to ["/etc/mail"] at yyy.pp:134; resource ["Tidy", "/etc/mail"] already declared at zzz.pp:107

Obviously the example is an oversimplification of an actual case, but the use case is valid.

Comment by Andrew Parker [ 2014/10/06 ]

Here is one more option that I just came up with: remove automatic aliasing of the uniqueness_key (the namevar). This functionality means that a resource such as:

package { "yum-mysql": name => mysql, provider => yum }

Can be referenced both as Package["yum-mysql"] as well as Package["mysql"]. By removing that automatic aliasing, then both the package example given above and the tidy example start to work.

Comment by Andrew Parker [ 2014/10/14 ]

After a long, drawn out battle of emails, the thread on puppet-dev has reached a conclusion about how to tackle this problem.

Charlie Sharpsteen discovered that we can use the provider + title_patterns to generate a uniqueness_key that allows for the behavior that is needed. He spiked a patch that we can use as the basis for the work.

This change resolves example #1 by using the combination of [name, provider] to enforce uniqueness, instead of the current [name]. The following clash would no longer be possible:

    # Composite key: ['foo', 'gem']
    package { 'foo-gem': name => 'foo', ensure => '1.0', provider => 'gem' }
 
    # Same composite key: ['foo', 'gem']
    package { 'gem-foo': name => 'foo', ensure => '2.0', provider => 'gem' }

Example #2 is also resolved:

    # Composite key: ['httpd-server', nil]
    package { 'httpd-server': ensure => '2.4.0' }
 
    # Same composite key: ['httpd-server', nil]
    package { 'web-server': name => 'httpd-server', ensure => '2.0.12' }

The possibility for conflict still exists between providers that happen to manage the same pool of packages and between implicit and explicit use of the default provider. For example, the following will result in competing resources on RedHat:

    # Composite key: ['httpd-server', nil]
    package { 'httpd-server': ensure => installed }
 
    # Composite key: ['httpd-server', 'yum']
    package { 'httpd': ensure => absent, name => 'httpd-server', provider => 'yum' }
 
    # Composite key: ['httpd-server', 'rpm']
    package { 'webserver': ensure => '2.4.0', name => 'httpd-server', provider => 'rpm' }

So, a composite key does not provide an airtight guarantee of uniqueness but is better than dropping isomorphism. We may be able to improve this situation by turning missing composite key values into smart defaults when the agent prepares a catalog for application.

Comment by Eric Sorenson [ 2014/11/04 ]

Andrew Parker Charlie Sharpsteen The composite namevar proposal seems like it got pretty widespread approval on the thread and could substantially help users who are being impacted by this bug. Can we continue to move this forward?

Comment by Andrew Parker [ 2014/11/04 ]

Eric Sorenson, at the moment this is still targeted at PUP 4.0.0. Assuming that it doesn't get dropped in priority and has to get bumped from 4.0.0 to a later release, then we can tackle it. Right now my understanding is that we need to do all of the code removals, the JSON transition, and any other breaks before we'll get to this. However, maybe the client team has some cycles to tackle this before then (Kylo Ginsberg)? This isn't a backwards incompatible change, so other than it being a problem for longer, we can also release it in 4.1.0 if need be.

Comment by Michael Smith [ 2014/12/02 ]

FR Notes:
Ensure packages with the same name from different providers can be installed, and verify they're functional.
Check that name collisions from the same provider are still prevented.
Spot check that name collisions for other resource types are still prevented.
Review error messages.

Comment by Kurt Wall [ 2015/01/16 ]

Verified in master at SHA= 5f223761454cc4995db41491349156ad18bf1d67. Packages with the same name from different providers can be installed and are functional:

# x.pp
package { "remove_it":
  name     => "vim",
  provider => "yum",
  ensure   => "absent"
}
 
package { "gem_vim":
  name     => "vim",
  provider => "gem",
  ensure   => "0.0.1"
}

Before the commit:

# puppet apply  x.pp
Error: Puppet::Parser::AST::Resource failed with error ArgumentError: Cannot alias Package[gem_vim] to ["vim"] at /root/puppet/x.pp:11; resource ["Package", "vim"] already declared at /root/puppet/x.pp:5 at /root/puppet/x.pp:11 on node lpin2u373ickfal.delivery.puppetlabs.net
Wrapped exception:
Cannot alias Package[gem_vim] to ["vim"] at /root/puppet/x.pp:11; resource ["Package", "vim"] already declared at /root/puppet/x.pp:5
Error: Puppet::Parser::AST::Resource failed with error ArgumentError: Cannot alias Package[gem_vim] to ["vim"] at /root/puppet/x.pp:11; resource ["Package", "vim"] already declared at /root/puppet/x.pp:5 at /root/puppet/x.pp:11 on node lpin2u373ickfal.delivery.puppetlabs.net

After the commit:

# git log -1
commit 5f223761454cc4995db41491349156ad18bf1d67
Merge: 6df5fd9 0e1d929
Author: Henrik Lindberg <henrik.lindberg@cloudsmith.com>
Date:   Fri Jan 16 17:25:19 2015 +0100
 
    Merge branch 'stable'
 
# bundle exec puppet apply x.pp
Already evaluated rest at /root/puppet/.bundle/ruby/gems/facter-2.3.0/lib/facter/ec2.rb:21, reevaluating anyways
Already evaluated rest at /root/puppet/.bundle/ruby/gems/facter-2.3.0/lib/facter/ec2.rb:38, reevaluating anyways
Already evaluated rest at /root/puppet/.bundle/ruby/gems/facter-2.3.0/lib/facter/gce.rb:4, reevaluating anyways
Notice: Compiled catalog for lpin2u373ickfal.delivery.puppetlabs.net in environment production in 0.84 seconds
Notice: /Stage[main]/Main/Package[gem_vim]/ensure: created
Notice: Applied catalog in 3.73 seconds

The vim gem is installed:

[root@lpin2u373ickfal puppet]# gem list vim
 
*** LOCAL GEMS ***
 
vim (0.0.1)

Name collisions from the same provider produce an error:

# y.pp
package { "remove_it":
  name     => "vim-X11",
  provider => "yum",
  ensure   => "absent",
}
 
package { "gem_vim":
  name     => "vim-X11",
  provider => "yum",
  ensure   => "latest"
}
 
# bundle exec puppet apply y.pp
Already evaluated rest at /root/puppet/.bundle/ruby/gems/facter-2.3.0/lib/facter/ec2.rb:21, reevaluating anyways
Already evaluated rest at /root/puppet/.bundle/ruby/gems/facter-2.3.0/lib/facter/ec2.rb:38, reevaluating anyways
Already evaluated rest at /root/puppet/.bundle/ruby/gems/facter-2.3.0/lib/facter/gce.rb:4, reevaluating anyways
Error: Evaluation Error: Error while evaluating a Resource Statement, Cannot alias Package[gem_vim] to ["vim-X11", "yum"] at /root/puppet/y.pp:7; resource ["Package", "vim-X11", "yum"] already declared at /root/puppet/y.pp:1 at /root/puppet/y.pp:7:1 on node lpin2u373ickfal.delivery.puppetlabs.net
Error: Evaluation Error: Error while evaluating a Resource Statement, Cannot alias Package[gem_vim] to ["vim-X11", "yum"] at /root/puppet/y.pp:7; resource ["Package", "vim-X11", "yum"] already declared at /root/puppet/y.pp:1 at /root/puppet/y.pp:7:1 on node lpin2u373ickfal.delivery.puppetlabs.net

Comment by Kurt Wall [ 2015/01/16 ]

Other resource types don't permit duplicate names:

# z.pp
notify { "vim_gem":
  name => 'vim',
}
notify { "gem_vim":
  name => 'vim',
}
 
# bundle exec puppet apply z.pp
Error: Evaluation Error: Error while evaluating a Resource Statement, Cannot alias Notify[gem_vim] to ["vim"] at /root/puppet/z.pp:4; resource ["Notify", "vim"] already declared at /root/puppet/z.pp:1 at /root/puppet/z.pp:4:1 on node lpin2u373ickfal.delivery.puppetlabs.net
Error: Evaluation Error: Error while evaluating a Resource Statement, Cannot alias Notify[gem_vim] to ["vim"] at /root/puppet/z.pp:4; resource ["Notify", "vim"] already declared at /root/puppet/z.pp:1 at /root/puppet/z.pp:4:1 on node lpin2u373ickfal.delivery.puppetlabs.net

Comment by Kurt Wall [ 2015/01/16 ]

Resolved per previous comments.

Comment by Kylo Ginsberg [ 2015/01/20 ]

Eric Sorenson I know you had some interest in this one. Please mark it resolved or comment as you see fit. Thanks!

Comment by Michael Smith [ 2015/02/17 ]

This does appear to be the first resource type with a composite namevar in Puppet core. Package[$title] works fine, but Package[$namevar] no longer works. It would have to look something like Package[$namevar, $provider], and that needs to be addressed in the parser first.

Comment by Henrik Lindberg [ 2015/02/17 ]

We can not change Package[a, b] to be a reference to a single resource since that means an array of two resources [Package[a], Package[b]]. That would be a major change and impact users manifests in a bad way. We must continue to make references in a unique way (i.e. Package['whatever']) can never be allowed to resolve to more that one resource).

The described functionality that Package["yum-foo"] and Package["foo"] would be references to the same thing assumes that there are no other packages that are also resolvable as Package["foo"] - and this must be an illegal reference in that case. Further, this means that the type system cannot determine if two references are the same without resolving against the actual resource type - that is a really really bad idea (as in "don't do that"). Alternatively we must introduce syntax in the title that makes it possible to parse the title and determine the generality (i.e "yum-foo" can be seen as a specialization of just "foo", it is a sibling to "gem-foo"). This however mean that it cannot be up to each type to determine how such references are expressed.

Comment by Henrik Lindberg [ 2015/02/17 ]

So, if someone wanted to make a reference it would need to be done like Package["$provider-$namevar"] for the Package type, knowing that it maps that way to title.

Comment by Michael Smith [ 2015/02/17 ]

Supporting Package["$provider-$namevar"] seems dangerous as it can collide with existing names. Does Package[["$provider"], ["$namevar"]] already have a meaning?

Comment by Henrik Lindberg [ 2015/02/17 ]

Yes, whatever is specified is evaluated, turned into an array and flattened - then each entry becomes one reference. It is common to do things like Package[$a, $b] where $a and $b are arrays of titles.

Comment by Henrik Lindberg [ 2015/02/17 ]

Why not just let the user set whatever unique title they like (that is the reference and it must be unique), and then set name and provider either automatically by parsing the title, or manually setting them to whatever is needed. That way, title is specified, it must be unique - internally name+provider must be unique.

Comment by Nicholas Fagerlund [ 2015/02/17 ]

From an explainability standpoint, I think it's perfectly reasonable to say that resource types with composite namevars just can't take advantage of automatic reference aliasing, and you need to use the title to refer to them. Auto aliasing always seemed like kind of a misfeature to me in the first place, tbh; too magical.

Comment by Nicholas Fagerlund [ 2015/02/17 ]

Michael Smith Yo, unless there's anything else you were meaning to mention here, I consider that an awesome description update and a finished ticket. I'll start filing docs tickets for all the work this represents, and you can re-close this one once you're pretty sure you've finished the brain dump.

Comment by Daniel Helgenberger [ 2015/06/19 ]

I am on Puppet 3.8.1 and still facing this issue. It says closed; did I miss something?

Comment by Charlie Sharpsteen [ 2015/06/19 ]

Daniel Helgenberger: This "fix version" for this ticket is 4.0.0 which is the first version that allows multiple providers to use the same name to identify packages. Puppet 3.8.1 still operates under the constraint that package names must be globally unique across all providers.

Comment by Daniel Helgenberger [ 2015/06/19 ]

Ok thanks for the clarification - if I just read better I would have seen PUP 4.0.0. I think there's only a slim change for a backport?

Comment by Charlie Sharpsteen [ 2015/06/19 ]

I think there's only a slim change for a backport?

Yes. As noted in the description, adding support for multiple providers breaks certain types of resource references. That makes this a breaking change and thus unsuitable for backporting to 3.x.

Comment by Eric Sorenson [ 2015/06/19 ]

Sorry, won't backport into 3.x - it changes the behaviour by adding
significance to package resources' naming convention. That's why it had to go
into 4.0 in the first place.

Generated at Wed Oct 16 02:07:35 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.