Uploaded image for project: 'Hiera'
  1. Hiera
  2. HI-46

Hiera should support alternate environments

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: UX
    • Labels:
    • Template:
    • CS Priority:
      Reviewed

      Description

      Currently hiera supports one `hiera.yaml` file hardcoded to be in the same location as `puppet.conf` (which is the `config` puppet directive.

      Having separate `hiera.yaml`'s per puppet environment would go along with having separate `site.pp`'s, modules, etc. per environment.

      UPDATE

      This feature is released in Puppet 4.4 / PE 2016.1 - See the full documentation here: https://docs.puppet.com/puppet/latest/reference/lookup_quick.html

      R.I.Pienaar has a blog series about it too: https://www.devco.net/archives/2016/03/13/the-puppet-4-lookup-function.php

      Please see PUP-4474 for the work on "data in modules" which also covers "data in environments". TL;DR - yes it will be possible to have hiera data per environment, as well as hiera data in a module, and all of that in combination with the existing global hiera.

      Use this ticket to comment on use-cases if you like, as the PUP-4474 and related tickets are about the implementation and thus are far more detailed/technical.

        Attachments

          Issue Links

            Activity

            Hide
            ghoneycutt garrett honeycutt added a comment -

            Separate hiera.yaml files are necessary because different environments may need different hierarchies/backends specified.

            Show
            ghoneycutt garrett honeycutt added a comment - Separate hiera.yaml files are necessary because different environments may need different hierarchies/backends specified.
            Hide
            bucksfan1991 Jeff Borders added a comment -

            We need a per environment router. Pronto.

            Show
            bucksfan1991 Jeff Borders added a comment - We need a per environment router. Pronto.
            Hide
            nick.walker Nick Walker added a comment -

            Even if you specify hiera_config to use the $environment variable it is only loaded once upon startup so you end up with $envrionment = production.

            hiera_config = $confdir/environments/$environment/hiera.yaml
            

            Ends up being equivalent to:

            hiera_config = $confdir/environments/production/hiera.yaml
            

            Show
            nick.walker Nick Walker added a comment - Even if you specify hiera_config to use the $environment variable it is only loaded once upon startup so you end up with $envrionment = production. hiera_config = $confdir/environments/$environment/hiera.yaml Ends up being equivalent to: hiera_config = $confdir/environments/production/hiera.yaml
            Hide
            fei Fei Guo added a comment -

            In the environment support research with Puppet Test Pilots, we have several users requesting this feature "hiera.yaml be aware of environments"

            Show
            fei Fei Guo added a comment - In the environment support research with Puppet Test Pilots, we have several users requesting this feature "hiera.yaml be aware of environments"
            Hide
            JiriHorky Jiri Horky added a comment -

            This is a major problem with increasing priority as the hiera gets more and more inevitable. The puppet's environments are great but to really be able to test complex changes, this bug must be resolved.

            Show
            JiriHorky Jiri Horky added a comment - This is a major problem with increasing priority as the hiera gets more and more inevitable. The puppet's environments are great but to really be able to test complex changes, this bug must be resolved.
            Hide
            zubac@avast.com Michal Zubac added a comment -

            solved by prefixing each hierarchy item in hiera.yaml with %{::environment}/. having separate hiera subtrees checked out for each branch/environment.

            Show
            zubac@avast.com Michal Zubac added a comment - solved by prefixing each hierarchy item in hiera.yaml with %{::environment}/ . having separate hiera subtrees checked out for each branch/environment.
            Hide
            zubac@avast.com Michal Zubac added a comment -

            problem with having separate hierachy definition for each branch/environment is still open though.

            Show
            zubac@avast.com Michal Zubac added a comment - problem with having separate hierachy definition for each branch/environment is still open though.
            Hide
            ghoneycutt garrett honeycutt added a comment -

            Michal Zubac It is not solved by prefixing each level of the hierarchy with an environment. hiera_hash() and hiera_array() will merge data from the entire hierarchy which allows data from one environment to leak into another.

            Show
            ghoneycutt garrett honeycutt added a comment - Michal Zubac It is not solved by prefixing each level of the hierarchy with an environment. hiera_hash() and hiera_array() will merge data from the entire hierarchy which allows data from one environment to leak into another.
            Hide
            ripienaar R.I.Pienaar added a comment -

            those functions only take data from the hierarchy that matches the node. So it will work fine

            Show
            ripienaar R.I.Pienaar added a comment - those functions only take data from the hierarchy that matches the node. So it will work fine
            Hide
            ghoneycutt garrett honeycutt added a comment - - edited

            R.I.Pienaar If you knew the other environment names, couldn't you just specify your environment as another one and access their data?

            Also, how does it solve the issue of different environments using multiple backends?

            Show
            ghoneycutt garrett honeycutt added a comment - - edited R.I.Pienaar If you knew the other environment names, couldn't you just specify your environment as another one and access their data? Also, how does it solve the issue of different environments using multiple backends?
            Hide
            jorhett Jo Rhett added a comment -

            Garret, I'm not sure that preventing people from poking under the hood is necessary. I'm aware of one site that does this extensively and considers it a feature. That's a YMMV issue

            I totally agree with you that separate hiera.yaml files are useful for configuring different backends and hierarchy paths. However, one could argue that this perhaps marks the distinction where a separate Puppet server is necessary. I personally don't think it's a problem to mark the hiera hierarchy as the boundary distinction between distinct servers. Given how trivial it is to run N puppet servers on the same box sharing some common module paths, this is perhaps a documentation request more than a technical problem.

            All IMHO naturally, and I have no clue how this would impact PE product. I suspect its only a few man-days worth of effort to autobuild distinct servers from the control panel. (I've built such a thing in roughly half a day for a customer, although I was able to re-use their existing web panel and tune it specifically to their needs so my scope was less)

            Show
            jorhett Jo Rhett added a comment - Garret, I'm not sure that preventing people from poking under the hood is necessary. I'm aware of one site that does this extensively and considers it a feature. That's a YMMV issue I totally agree with you that separate hiera.yaml files are useful for configuring different backends and hierarchy paths. However, one could argue that this perhaps marks the distinction where a separate Puppet server is necessary. I personally don't think it's a problem to mark the hiera hierarchy as the boundary distinction between distinct servers. Given how trivial it is to run N puppet servers on the same box sharing some common module paths, this is perhaps a documentation request more than a technical problem. All IMHO naturally, and I have no clue how this would impact PE product. I suspect its only a few man-days worth of effort to autobuild distinct servers from the control panel. (I've built such a thing in roughly half a day for a customer, although I was able to re-use their existing web panel and tune it specifically to their needs so my scope was less)
            Hide
            ghoneycutt garrett honeycutt added a comment -

            Jo Rhett Your work around of using multiple puppet masters is very interesting, but still a work around for what this ticket aims to solve. If Puppet could use different Hiera configurations, separate puppet masters would not be needed.

            Show
            ghoneycutt garrett honeycutt added a comment - Jo Rhett Your work around of using multiple puppet masters is very interesting, but still a work around for what this ticket aims to solve. If Puppet could use different Hiera configurations, separate puppet masters would not be needed.
            Hide
            jorhett Jo Rhett added a comment -

            I understand, but if it was my project I would question if there was a demonstrable need that justified the effort. From what I have seen of these internals, you're looking at a pretty significant refactor. IMHO and I'll shut up as my opinion is expressed

            Show
            jorhett Jo Rhett added a comment - I understand, but if it was my project I would question if there was a demonstrable need that justified the effort. From what I have seen of these internals, you're looking at a pretty significant refactor. IMHO and I'll shut up as my opinion is expressed
            Hide
            james.sweeny James Sweeny added a comment - - edited

            This is covered a bit in the original Redmine ticket, but the real importance of having environment specific hiera.yaml files comes down to how you release changes to the hierarchy.

            It's a very common pattern to use multiple environments (very often tied to a VCS branch) to test changes before they get merged into a mainline/moved to production. With hiera.yaml being the same across all environments, it is extremely inconvenient to make a hierarchy change in dev or a feature branch before merging or deploying to production servers. This seriously cripples many organizations' ability to test changes to the code base, since hierarchy changes need to be done all at once.

            The argument for this feature is that hiera.yaml should not be any different in terms of how it can be tested from the hieradata itself (which can reference $environment) or the puppet modules and manifests. The only way to do this testing without allowing multiple environment-specific versions of hiera.yaml is to have a separate master for each application landscape and manually move hiera.yaml changes among them. This is, of course, extremely impractical if you're trying to test short-lived feature branches.

            Show
            james.sweeny James Sweeny added a comment - - edited This is covered a bit in the original Redmine ticket, but the real importance of having environment specific hiera.yaml files comes down to how you release changes to the hierarchy. It's a very common pattern to use multiple environments (very often tied to a VCS branch) to test changes before they get merged into a mainline/moved to production. With hiera.yaml being the same across all environments, it is extremely inconvenient to make a hierarchy change in dev or a feature branch before merging or deploying to production servers. This seriously cripples many organizations' ability to test changes to the code base, since hierarchy changes need to be done all at once. The argument for this feature is that hiera.yaml should not be any different in terms of how it can be tested from the hieradata itself (which can reference $environment) or the puppet modules and manifests. The only way to do this testing without allowing multiple environment-specific versions of hiera.yaml is to have a separate master for each application landscape and manually move hiera.yaml changes among them. This is, of course, extremely impractical if you're trying to test short-lived feature branches.
            Hide
            jorhett Jo Rhett added a comment -

            I agree with the essence of what you are saying but "impossible" it is not. Create another server with a distinct hiera.yaml file and reference the same manifest and module directories. It's like 3 minutes of work if you are hacking fast, and less than an hour's work to build scripts that will build such environments on demand.

            So I agree with you – but back off from "impossible" to "requires a non-standard config" which is a completely different level of need.

            Show
            jorhett Jo Rhett added a comment - I agree with the essence of what you are saying but "impossible" it is not. Create another server with a distinct hiera.yaml file and reference the same manifest and module directories. It's like 3 minutes of work if you are hacking fast, and less than an hour's work to build scripts that will build such environments on demand. So I agree with you – but back off from "impossible" to "requires a non-standard config" which is a completely different level of need.
            Hide
            jorhett Jo Rhett added a comment -

            There is really nothing "extreme" about the difficulty. I have this working at several sites. If you do it without a script it's like 5 commands on the command line total. The implementation of multiple servers with shared paths is trivial to implement and easy to understand. It's not "extreme" except in the sense of "extremely elegant"

            I am reversing myself-- I don't agree with this in principle. This works for me today. This is a documentation problem at worst.

            Show
            jorhett Jo Rhett added a comment - There is really nothing "extreme" about the difficulty. I have this working at several sites. If you do it without a script it's like 5 commands on the command line total. The implementation of multiple servers with shared paths is trivial to implement and easy to understand. It's not "extreme" except in the sense of "extremely elegant" I am reversing myself-- I don't agree with this in principle. This works for me today. This is a documentation problem at worst.
            Hide
            fiddyspence Chris Spence added a comment -

            I rather think that the whole point of the ticket is so you don't have to create another distinct puppet master to be able to traverse a different hierarchy in hiera.

            Show
            fiddyspence Chris Spence added a comment - I rather think that the whole point of the ticket is so you don't have to create another distinct puppet master to be able to traverse a different hierarchy in hiera.
            Hide
            lee Lee Lowder added a comment -

            While multiple masters are easy to setup - for both open source and Enterprise users, each master will consume at least one Enterprise license, and potentially 3 licenses if a split install is used.

            So using multiple masters is one possible workaround for some cases, it is not always a practical or viable one.

            There is also the cost that comes into effect. While spinning up a new VM is usually relatively cheap, it is not always an insignificant cost in terms of time, effort, hardware and licensing.

            Show
            lee Lee Lowder added a comment - While multiple masters are easy to setup - for both open source and Enterprise users, each master will consume at least one Enterprise license, and potentially 3 licenses if a split install is used. So using multiple masters is one possible workaround for some cases, it is not always a practical or viable one. There is also the cost that comes into effect. While spinning up a new VM is usually relatively cheap, it is not always an insignificant cost in terms of time, effort, hardware and licensing.
            Hide
            smegthelight Richard Clark added a comment - - edited

            This is a feature ticket right ? I want this for supporting different backends. Having a separate hiera.yaml underneath the environmentpath would prevent me from having to have multiple masters (my current work around). It is a simplification of my setup, and is in line with the separate environment work being done.

            Show
            smegthelight Richard Clark added a comment - - edited This is a feature ticket right ? I want this for supporting different backends. Having a separate hiera.yaml underneath the environmentpath would prevent me from having to have multiple masters (my current work around). It is a simplification of my setup, and is in line with the separate environment work being done.
            Hide
            ben.ford Ben Ford added a comment -

            Is there anything about environments that couldn't be easily scripted out to multiple Puppet masters? The fact that I could script a workaround doesn't mean that I should have to.

            Assuming that hiera.yaml is interpreted for each environment is a very common mistake. If we choose not to support the commonly expected functionality, we should have a damn good reason why not. Having a workaround for it is not a good reason.

            Show
            ben.ford Ben Ford added a comment - Is there anything about environments that couldn't be easily scripted out to multiple Puppet masters? The fact that I could script a workaround doesn't mean that I should have to . Assuming that hiera.yaml is interpreted for each environment is a very common mistake. If we choose not to support the commonly expected functionality, we should have a damn good reason why not. Having a workaround for it is not a good reason.
            Hide
            jorhett Jo Rhett added a comment - - edited

            "While spinning up a new VM is usually relatively cheap"

            Nobody said new VM. I've run ~20 puppet masters from the same host, sans any virtualization. This is what I mean by "trivial" – a few commands at the shell, viola you have a new puppetmaster with an identical everything except a different hiera.yaml.

            The point of my argument is that this is a relatively unimportant problem with a trivial workaround. There are MAYBE 50 sites feeling pain with this problem, and I'll bet that most of them have already built their own workaround.

            There are many SERIOUS problems with Puppet that have been waiting YEARS to get addressed. If we have some free hours, let's solve real problems that big environments are having to work around every day? Look at my bug list, there's hard issues on there waiting for someone to address.

            How about nailing any of the half dozen bugs that make tagmail useless, including the recent decision to send 400 deprecation messages a minute via tagmail? Or how about the deadlocks in the puppetdb result forwarding which drag the puppetmaster down and make it implausible for large sites?

            Show
            jorhett Jo Rhett added a comment - - edited "While spinning up a new VM is usually relatively cheap" Nobody said new VM. I've run ~20 puppet masters from the same host, sans any virtualization. This is what I mean by "trivial" – a few commands at the shell, viola you have a new puppetmaster with an identical everything except a different hiera.yaml. The point of my argument is that this is a relatively unimportant problem with a trivial workaround. There are MAYBE 50 sites feeling pain with this problem, and I'll bet that most of them have already built their own workaround. There are many SERIOUS problems with Puppet that have been waiting YEARS to get addressed. If we have some free hours, let's solve real problems that big environments are having to work around every day? Look at my bug list, there's hard issues on there waiting for someone to address. How about nailing any of the half dozen bugs that make tagmail useless, including the recent decision to send 400 deprecation messages a minute via tagmail? Or how about the deadlocks in the puppetdb result forwarding which drag the puppetmaster down and make it implausible for large sites?
            Hide
            jorhett Jo Rhett added a comment -

            "Is there anything about environments that couldn't be easily scripted out to multiple Puppet masters?"

            Fallback module paths. That was easy. I can list dozens. If you're going to pick a rhetorical question make it one with technical merit

            Show
            jorhett Jo Rhett added a comment - "Is there anything about environments that couldn't be easily scripted out to multiple Puppet masters?" Fallback module paths. That was easy. I can list dozens. If you're going to pick a rhetorical question make it one with technical merit
            Hide
            eric Eric Shamow added a comment -

            Jo Rhett the debate about whether or not environment use is appropriate may be valid, but based on what data do you base your count about the number of sites where this is a serious issue?

            Pervasiveness isn't a "back of the napkin" measure. This affects more users than you realize, and restrains sensible best practices patterns to the limitations of the tool rather than the appropriate design.

            Show
            eric Eric Shamow added a comment - Jo Rhett the debate about whether or not environment use is appropriate may be valid, but based on what data do you base your count about the number of sites where this is a serious issue? Pervasiveness isn't a "back of the napkin" measure. This affects more users than you realize, and restrains sensible best practices patterns to the limitations of the tool rather than the appropriate design.
            Hide
            smegthelight Richard Clark added a comment -

            31 votes, 40 watcher. If I did my query correct, this is the highest voted ticket in Jira. That in itself is pretty sad, but It is either being gamed (weird?), or it affects more sites than any other issue.

            Show
            smegthelight Richard Clark added a comment - 31 votes, 40 watcher. If I did my query correct, this is the highest voted ticket in Jira. That in itself is pretty sad, but It is either being gamed (weird?), or it affects more sites than any other issue.
            Hide
            jorhett Jo Rhett added a comment -

            Is this really a request for a war of votes? That having entire IT departments of big companies log in and express their annoyance through clicks means more than having the most engaged person on the team attempt to reason wtih you?

            How about the three multi-national companies I'm working with who are not renewing their PE licensing because they're reinvesting that money instead to build out a team to maintain an internal branch because their constant and repeated attempts to get critical bugs fixed achieves nothing? And yes they have expressed that exact sentiment to their salespeople.

            Every one of us has patches sitting unanswered, uncategorized that are critical problems that affect hundreds of companies. Through prioritizations like this, Puppet Labs does a fantastic job of ensuring that nobody submits code back to the project because it has been shown to be a waste of time and effort. We can't even get the problems considered.

            Show
            jorhett Jo Rhett added a comment - Is this really a request for a war of votes? That having entire IT departments of big companies log in and express their annoyance through clicks means more than having the most engaged person on the team attempt to reason wtih you? How about the three multi-national companies I'm working with who are not renewing their PE licensing because they're reinvesting that money instead to build out a team to maintain an internal branch because their constant and repeated attempts to get critical bugs fixed achieves nothing? And yes they have expressed that exact sentiment to their salespeople. Every one of us has patches sitting unanswered, uncategorized that are critical problems that affect hundreds of companies. Through prioritizations like this, Puppet Labs does a fantastic job of ensuring that nobody submits code back to the project because it has been shown to be a waste of time and effort. We can't even get the problems considered.
            Hide
            eric Eric Shamow added a comment -

            Jo Rhett I didn't ask for a war of clicks, I asked for data indicating that the problem is not as pervasive as you believe it to be. You have presented me with anecdotal information.

            I'd appreciate if you'd reach out privately if you have information about customers who are not renewing for this reason so that we can close the loop.

            Meanwhile with regard to the importance of this ticket - as somebody who has spent a lot of time in the field, the perspective from there is deeply distorted. Field/consulting work provides a perspective on the needs of a self-selected and atypical group of users to begin with. Extending that to field/consulting data collected by one individual and not a team means that what you've got is a some data that has to fit into an overall heat map.

            In short: not all users use the features you do and you won't care about the features they do. That doesn't mean their needs are less important than yours or vice versa, and it also doesn't mean that the loudest voice gets to shout down a feature request that's critical to our users. What's more, features are not selected for work solely based on pervasiveness OR severity. It's a combination of those questions in addition to helping particular and focused market segments succeed with the product. That means sometimes code you just don't care about it going to get some attention, because a nontrivial number of other people DO care about it.

            Informal polling of our Pro Services team has found the exact opposite of the data you suggest. I suspect the truth is somewhere in the middle.

            Show
            eric Eric Shamow added a comment - Jo Rhett I didn't ask for a war of clicks, I asked for data indicating that the problem is not as pervasive as you believe it to be. You have presented me with anecdotal information. I'd appreciate if you'd reach out privately if you have information about customers who are not renewing for this reason so that we can close the loop. Meanwhile with regard to the importance of this ticket - as somebody who has spent a lot of time in the field, the perspective from there is deeply distorted. Field/consulting work provides a perspective on the needs of a self-selected and atypical group of users to begin with. Extending that to field/consulting data collected by one individual and not a team means that what you've got is a some data that has to fit into an overall heat map. In short: not all users use the features you do and you won't care about the features they do. That doesn't mean their needs are less important than yours or vice versa, and it also doesn't mean that the loudest voice gets to shout down a feature request that's critical to our users. What's more, features are not selected for work solely based on pervasiveness OR severity. It's a combination of those questions in addition to helping particular and focused market segments succeed with the product. That means sometimes code you just don't care about it going to get some attention, because a nontrivial number of other people DO care about it. Informal polling of our Pro Services team has found the exact opposite of the data you suggest. I suspect the truth is somewhere in the middle.
            Hide
            mlehner616 MIke Lehner added a comment -

            Eric Shamow This sentiment is appreciated by us small guys who bought Puppet Enterprise because it's ease of deployment and manageability, not because of it's massive scalability and ability to get bugs resolved within hours/days. I am a single admin with 4-5 critical roles and I need the deployment, development workflow to be as efficient and streamlined as possible.

            Jo Rhett Consider yourself fortunate to have an entire team(s) to maintain an internal branch of puppet. This bug actually hinders a best practice workflow and for small teams like mine for example where I AM the team, a small overhead of deploying multiple puppetmasters turns out to be a large overhead. I need the development and deployment workflow to be as streamlined and efficient as possible. This bug is actually a wrench in the cogs to accomplishing this. Multiple puppetmasters introduces complexity and with complexity comes more complexity. Nodes now need to not only be configured to point to a separate environment they must then be pointed to a separate puppetmaster. More moving parts here. I believe your understating the introduction of complexity here. If puppet's goal is to simplify the life of the sysadmin let's not code for the exception but to the standard. Now if the standard is multi-national conglomerates I suppose I'm out of luck in that case. I have worked with MANY software companies and I personally have found Puppet one of the most receptive to suggestions on the road map. I certainly hope that remains the case and the product doesn't massively increase in complexity simply biggest the largest bidder has complex requirements.

            Show
            mlehner616 MIke Lehner added a comment - Eric Shamow This sentiment is appreciated by us small guys who bought Puppet Enterprise because it's ease of deployment and manageability, not because of it's massive scalability and ability to get bugs resolved within hours/days. I am a single admin with 4-5 critical roles and I need the deployment, development workflow to be as efficient and streamlined as possible. Jo Rhett Consider yourself fortunate to have an entire team(s) to maintain an internal branch of puppet. This bug actually hinders a best practice workflow and for small teams like mine for example where I AM the team, a small overhead of deploying multiple puppetmasters turns out to be a large overhead. I need the development and deployment workflow to be as streamlined and efficient as possible. This bug is actually a wrench in the cogs to accomplishing this. Multiple puppetmasters introduces complexity and with complexity comes more complexity. Nodes now need to not only be configured to point to a separate environment they must then be pointed to a separate puppetmaster. More moving parts here. I believe your understating the introduction of complexity here. If puppet's goal is to simplify the life of the sysadmin let's not code for the exception but to the standard. Now if the standard is multi-national conglomerates I suppose I'm out of luck in that case. I have worked with MANY software companies and I personally have found Puppet one of the most receptive to suggestions on the road map. I certainly hope that remains the case and the product doesn't massively increase in complexity simply biggest the largest bidder has complex requirements.
            Hide
            alexharv074 Alex Harvey added a comment -

            Add me to the 'this is a big problem that needs to be fixed' camp.

            Show
            alexharv074 Alex Harvey added a comment - Add me to the 'this is a big problem that needs to be fixed' camp.
            Hide
            squiggly Andre Hurano added a comment -

            Its sooooo stupid this hasn't been fixed yet. It's probably like a 2 line change somewhere.

            And yes, it is a huge problem. HUGE. It's the weak link in managing a multi-team puppet infrastructure.

            Show
            squiggly Andre Hurano added a comment - Its sooooo stupid this hasn't been fixed yet. It's probably like a 2 line change somewhere. And yes, it is a huge problem. HUGE. It's the weak link in managing a multi-team puppet infrastructure.
            Hide
            crayfishx Craig Dunn added a comment -

            I dont understand the reluctance to do this - it doesn't seem to be a major architectural change and the level of interest in this ticket suggests there is enough demand. My take on it is, puppet directory environments are a great way to do several things, including;

            • Have the master serve different versions/releases of Puppet code
            • Have the master serve different "traditional" environments (dev, QA, staging)
            • Allow for multiple users to share the same Puppet infrastructure by autonomising the modulepath and manifest
            • Allow for puppet code of varying levels of maturity to be tested before production release

            In all of these examples, and more, I see valid use cases for wanting or needing to change the hierarchy search order or hiera backends. I am currently working on a cloud solution that leverages r10k and puppet environments to enable projects to have the maximum flexibility with their requirements but maintain some level of centralization and code sharing - this issue is the only thing I'm having to come up with work arounds for.

            Allowing environments to self contain their modules and their site.pp but not hiera.yaml just seems like another example of environments nearly being a great feature (lets not forget about resource types and environments).

            Now 40 votes and 50 watchers now...

            Show
            crayfishx Craig Dunn added a comment - I dont understand the reluctance to do this - it doesn't seem to be a major architectural change and the level of interest in this ticket suggests there is enough demand. My take on it is, puppet directory environments are a great way to do several things, including; Have the master serve different versions/releases of Puppet code Have the master serve different "traditional" environments (dev, QA, staging) Allow for multiple users to share the same Puppet infrastructure by autonomising the modulepath and manifest Allow for puppet code of varying levels of maturity to be tested before production release In all of these examples, and more, I see valid use cases for wanting or needing to change the hierarchy search order or hiera backends. I am currently working on a cloud solution that leverages r10k and puppet environments to enable projects to have the maximum flexibility with their requirements but maintain some level of centralization and code sharing - this issue is the only thing I'm having to come up with work arounds for. Allowing environments to self contain their modules and their site.pp but not hiera.yaml just seems like another example of environments nearly being a great feature (lets not forget about resource types and environments). Now 40 votes and 50 watchers now...
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            Please understand that the support for directory based environments was no easy thing to implement and we were plagued by all sorts of unpleasant surprises along the way. What seemed like a trivial thing to implement caused us months of work, and we are not sure we chased out all of the demons yet. Throwing yet another complex thing like hiera into the mix until the environment support works as it should would have been a very very bad idea. (By complex here, it is more the implications of how you need to be able to configure things, how they combine, etc. than the actual hiera technology).

            At the moment, the environment handling supports both legacy and directory based environments. This blocks us from performing clean up of the code until the legacy environment support has been removed. Environments also come into play on the agent and environments are reused across agent and master code when running puppet apply. The amount of code, and assumptions about how things work that permeate the code base (and the thousands of tests) was a quite substantial task to tackle, and we did reach a point were we were just about ready to reverse the work and not add the directory environment support at all as it started to threaten getting the new language features released in Puppet 4.0.

            During this time, also having a discussion about additional features on top of the directory environments was simply something we had to put off.

            And no, it is not a two line change somewhere. The startup order of puppet, settings, applications, faces, environments, indirections, etc. is incredibly complex. A two line change may in fact buy you an even dozen can of worms individually labeled with things like performance, caching, startup order, command line overrides, data binding API, etc. etc.

            I know the next question is WHEN? I honestly cannot answer that, but I can say that this will neither happen in 3.7 nor 4.0 (that will follow short after 3.7 as it is intended to be a cleanup and removal release and a switch to the "future parser/evaluator"). The team is fully booked for this work (and on that note, I should probably mention that, yes we are hiring).

            Show
            henrik.lindberg Henrik Lindberg added a comment - Please understand that the support for directory based environments was no easy thing to implement and we were plagued by all sorts of unpleasant surprises along the way. What seemed like a trivial thing to implement caused us months of work, and we are not sure we chased out all of the demons yet. Throwing yet another complex thing like hiera into the mix until the environment support works as it should would have been a very very bad idea. (By complex here, it is more the implications of how you need to be able to configure things, how they combine, etc. than the actual hiera technology). At the moment, the environment handling supports both legacy and directory based environments. This blocks us from performing clean up of the code until the legacy environment support has been removed. Environments also come into play on the agent and environments are reused across agent and master code when running puppet apply. The amount of code, and assumptions about how things work that permeate the code base (and the thousands of tests) was a quite substantial task to tackle, and we did reach a point were we were just about ready to reverse the work and not add the directory environment support at all as it started to threaten getting the new language features released in Puppet 4.0. During this time, also having a discussion about additional features on top of the directory environments was simply something we had to put off. And no, it is not a two line change somewhere. The startup order of puppet, settings, applications, faces, environments, indirections, etc. is incredibly complex. A two line change may in fact buy you an even dozen can of worms individually labeled with things like performance, caching, startup order, command line overrides, data binding API, etc. etc. I know the next question is WHEN? I honestly cannot answer that, but I can say that this will neither happen in 3.7 nor 4.0 (that will follow short after 3.7 as it is intended to be a cleanup and removal release and a switch to the "future parser/evaluator"). The team is fully booked for this work (and on that note, I should probably mention that, yes we are hiring).
            Hide
            ghoneycutt garrett honeycutt added a comment -

            Puppet Labs, could you please provide a status update on this request and where it fits into your road map?

            Thank you!

            Show
            ghoneycutt garrett honeycutt added a comment - Puppet Labs, could you please provide a status update on this request and where it fits into your road map? Thank you!
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            Current status is that 4.0 was pushed out to end of the year, and the current plan includes providing a good way to plug in the "hiera data in modules"-module. We know that handling one hiera per environment is high on the wish list but I am uncertain if it will make it into 4.0. We have infrastructure and cleanup things that stand in the way - the system is creating way too many instances of Environment at the moment and this needs to be fixed in order to tie more things to the environment (like a hiera) - it may be that the work on wiring "hiera-data-in-modules" solves part of this problem and we may be able to support one hiera per environment because of that, but that work is not completed yet - so no promises; it may require a lot more work. Or, put differently, we know that is we do this the naive way we will create a performance problem.

            In 4.0 - we are also removing the non directory environments - the old environments are a blocker to allowing hiera per environment.

            garrett honeycutt while that was perhaps not the answer you wanted, it is the state of things - and it is not in any way forgotten. If it does not make it into 4.0, it can be done in a 4.x release as we should then have the foundation to base it on.

            Show
            henrik.lindberg Henrik Lindberg added a comment - Current status is that 4.0 was pushed out to end of the year, and the current plan includes providing a good way to plug in the "hiera data in modules"-module. We know that handling one hiera per environment is high on the wish list but I am uncertain if it will make it into 4.0. We have infrastructure and cleanup things that stand in the way - the system is creating way too many instances of Environment at the moment and this needs to be fixed in order to tie more things to the environment (like a hiera) - it may be that the work on wiring "hiera-data-in-modules" solves part of this problem and we may be able to support one hiera per environment because of that, but that work is not completed yet - so no promises; it may require a lot more work. Or, put differently, we know that is we do this the naive way we will create a performance problem. In 4.0 - we are also removing the non directory environments - the old environments are a blocker to allowing hiera per environment. garrett honeycutt while that was perhaps not the answer you wanted, it is the state of things - and it is not in any way forgotten. If it does not make it into 4.0, it can be done in a 4.x release as we should then have the foundation to base it on.
            Hide
            brettswift Brett Swift added a comment -

            I thought I'd mention this is a dependency of an r10k feature as well:
            https://github.com/puppetlabs/r10k/issues/195

            Show
            brettswift Brett Swift added a comment - I thought I'd mention this is a dependency of an r10k feature as well: https://github.com/puppetlabs/r10k/issues/195
            Hide
            ghoneycutt garrett honeycutt added a comment -

            As 4.0 draws near, is this going to make it in?

            Show
            ghoneycutt garrett honeycutt added a comment - As 4.0 draws near, is this going to make it in?
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            PUP-1640 is in Puppet 4.0.0 - that is the mechanism that allows modules, and the environment to specify which technology to use to lookup data. It comes with a default implementation based on calling functions. Support for other technologies to be supplied in modules. I know R.I.Pienaar is working on an implementation for Hiera.

            Hiera is currently a singleton service which makes it difficult to make it support modules and environments. The hiera code base needs to be refactored to allow multiple hierarchies in the same runtime. Doing this major change is an open issue.

            Show
            henrik.lindberg Henrik Lindberg added a comment - PUP-1640 is in Puppet 4.0.0 - that is the mechanism that allows modules, and the environment to specify which technology to use to lookup data. It comes with a default implementation based on calling functions. Support for other technologies to be supplied in modules. I know R.I.Pienaar is working on an implementation for Hiera. Hiera is currently a singleton service which makes it difficult to make it support modules and environments. The hiera code base needs to be refactored to allow multiple hierarchies in the same runtime. Doing this major change is an open issue.
            Hide
            crazydave David Chute added a comment -

            Is this really a bug in the enterprise pricing model?

            As far as I can tell if you want to make guaranteed safe changes to the hiera backend (or indeed your enc, if you have one) then IMO you are best off having a separate puppetmaster per environment.
            This gives you a place to test changes to hiera (and enc) before rolling out onto higher environments, then eventually live.

            With a few soft links and multiple puppermaster service definitions you can probably easily have this in one environment, but why would you do that?
            Do you really want the new guy applying a change to your dev hiera/modules/enc on the same server as you host your live code.
            And how do you test OS patch updates to the puppetmaster server itself without multiple puppetmasters?

            We are aiming for the same release process for puppetmaster and related modules and config as we have for any other application in our organisation.
            So we can take a tagged version from SVN, build it into a tar file, and promote the same tar file through the environments.
            Then looking in SVN, hiera is then the only place that has environment specific information.

            Do organisations put all environments onto one box because of costs associated to the Enterprise pricing model?

            I appreciate this seems to go against the vision of Puppet 4.0, but I personally think the vision is either flawed or I have misunderstood it.

            Show
            crazydave David Chute added a comment - Is this really a bug in the enterprise pricing model? As far as I can tell if you want to make guaranteed safe changes to the hiera backend (or indeed your enc, if you have one) then IMO you are best off having a separate puppetmaster per environment. This gives you a place to test changes to hiera (and enc) before rolling out onto higher environments, then eventually live. With a few soft links and multiple puppermaster service definitions you can probably easily have this in one environment, but why would you do that? Do you really want the new guy applying a change to your dev hiera/modules/enc on the same server as you host your live code. And how do you test OS patch updates to the puppetmaster server itself without multiple puppetmasters? We are aiming for the same release process for puppetmaster and related modules and config as we have for any other application in our organisation. So we can take a tagged version from SVN, build it into a tar file, and promote the same tar file through the environments. Then looking in SVN, hiera is then the only place that has environment specific information. Do organisations put all environments onto one box because of costs associated to the Enterprise pricing model? I appreciate this seems to go against the vision of Puppet 4.0, but I personally think the vision is either flawed or I have misunderstood it.
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            David Chute You bring up very good points - the problems you point out (together with a lot of other problems / considerations) lead to the "agnostic approach" in PUP-1640 which is the start of the implementation where modules and environments can have their own set of data (in other words 'have its own hiera with its own hierarchy'). In the very first implementation we cannot make this work with the existing hiera since it only supports a single hierarchy, possibly one that points into environment specific locations. That is what people are using today; this way, they do not have to give everyone access to the top level (non environment specific data - such as storing data for a hiera based light ENC), instead users only edit the data files in the environment, but the are referenced from the overall config.

            With the new solution the top level "global" hiera does not have to deal with environments (and modules; which it cannot handle well at all today).

            We expect to release hiera based data in modules and environments as well - but it will not be ready in time for 4.0.0.

            Show
            henrik.lindberg Henrik Lindberg added a comment - David Chute You bring up very good points - the problems you point out (together with a lot of other problems / considerations) lead to the "agnostic approach" in PUP-1640 which is the start of the implementation where modules and environments can have their own set of data (in other words 'have its own hiera with its own hierarchy'). In the very first implementation we cannot make this work with the existing hiera since it only supports a single hierarchy, possibly one that points into environment specific locations. That is what people are using today; this way, they do not have to give everyone access to the top level (non environment specific data - such as storing data for a hiera based light ENC), instead users only edit the data files in the environment, but the are referenced from the overall config. With the new solution the top level "global" hiera does not have to deal with environments (and modules; which it cannot handle well at all today). We expect to release hiera based data in modules and environments as well - but it will not be ready in time for 4.0.0.
            Hide
            crazydave David Chute added a comment -

            Henrik Lindberg I'm not sure I understand the use-case for PUP-1640, other than I can see why a module developer wants a tidy way to specify defaults for values otherwise loaded from hiera. But to me that seems separate from the environment issue.

            But in any case, with a puppetmaster per environment you do not need to support multiple concurrent environment data in hiera.
            You simply have multiple hiera sub-directories, one for each environment, and 'switch on' the needed sub-directory with a soft link.

            Directories;

            ...
            /etc/puppet/hiera/dev1
            /etc/puppet/hiera/dev2
            /etc/puppet/hiera/ci
            /etc/puppet/hiera/pre-prod
            /etc/puppet/hiera/prod
            ...
            

            Hiera config;

            ...
              :datadir: /etc/puppet/hiera/active-environment
            ...
            

            Soft link;

            ln -s /etc/puppet/hiera/pre-prod  /etc/puppet/hiera/active-environment
            

            Some might consider this with a puppetmaster per environment a workaround for this ticket. But I think it should be the standard way to deploy puppet.

            Show
            crazydave David Chute added a comment - Henrik Lindberg I'm not sure I understand the use-case for PUP-1640 , other than I can see why a module developer wants a tidy way to specify defaults for values otherwise loaded from hiera. But to me that seems separate from the environment issue. But in any case, with a puppetmaster per environment you do not need to support multiple concurrent environment data in hiera. You simply have multiple hiera sub-directories, one for each environment, and 'switch on' the needed sub-directory with a soft link. Directories; ... /etc/puppet/hiera/dev1 /etc/puppet/hiera/dev2 /etc/puppet/hiera/ci /etc/puppet/hiera/pre-prod /etc/puppet/hiera/prod ... Hiera config; ... :datadir: /etc/puppet/hiera/active-environment ... Soft link; ln -s /etc/puppet/hiera/pre-prod /etc/puppet/hiera/active-environment Some might consider this with a puppetmaster per environment a workaround for this ticket. But I think it should be the standard way to deploy puppet.
            Hide
            ghoneycutt garrett honeycutt added a comment -

            Henrik Lindberg Is this ticket tightly coupled with data in modules? I thought that was a totally separate issue from Puppet supporting different Hiera setups per environment.

            Show
            ghoneycutt garrett honeycutt added a comment - Henrik Lindberg Is this ticket tightly coupled with data in modules? I thought that was a totally separate issue from Puppet supporting different Hiera setups per environment.
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            garrett honeycutt - They are tightly coupled, since "data in modules" is essentially the same problem as "data in environment". The rock bottom problem in hiera is that it is a singleton implementation; it can only have one hierarchy (albeit with magic levels in the hierarchy to support "data in a module" as done in the experimental module that R.I wrote)), and it caches data from backends in a global manner. This means it will leak memory when used with multiple and changing environments.

            We want global, environment, and module data lookup to be separate (handled by separate instances of the technology that provides the data). That is what PUP-1640 provides (David Chute). We will provide (or work with R.I; to provide) an implementation for hiera as well.

            Show
            henrik.lindberg Henrik Lindberg added a comment - garrett honeycutt - They are tightly coupled, since "data in modules" is essentially the same problem as "data in environment". The rock bottom problem in hiera is that it is a singleton implementation; it can only have one hierarchy (albeit with magic levels in the hierarchy to support "data in a module" as done in the experimental module that R.I wrote)), and it caches data from backends in a global manner. This means it will leak memory when used with multiple and changing environments. We want global, environment, and module data lookup to be separate (handled by separate instances of the technology that provides the data). That is what PUP-1640 provides ( David Chute ). We will provide (or work with R.I; to provide) an implementation for hiera as well.
            Hide
            nathan Nathan Valentine added a comment - - edited

            @David, I think I'm understanding your point... If not, tell me to "Shut up."

            Its quite common for folks to populate a Puppet environment with code + data + data routing logic from a topic branch of their control repo for testing purposes. Puppet environments become, in effect, a way to model stages in the delivery and testing pipelines. Thus the request for Hiera-router-per-environment as you need to all three pieces to have some level of confidence that breakage will not ensue when changes hit production.

            Show
            nathan Nathan Valentine added a comment - - edited @David, I think I'm understanding your point... If not, tell me to "Shut up." Its quite common for folks to populate a Puppet environment with code + data + data routing logic from a topic branch of their control repo for testing purposes. Puppet environments become, in effect, a way to model stages in the delivery and testing pipelines. Thus the request for Hiera-router-per-environment as you need to all three pieces to have some level of confidence that breakage will not ensue when changes hit production.
            Hide
            crazydave David Chute added a comment -

            Yes, I think you have got my point.

            So IMO this 'feature' is a trap. We should be asking why people want this feature and addressing that in a better way.

            This is a feature asking for a shared component between multiple environments. That component is (at the very least) the part that does the environment switching (as you put it, the routing logic).
            How will you test an update to that switch/data-routing-logic when it is part of both your dev and your live infrastructure?

            Show
            crazydave David Chute added a comment - Yes, I think you have got my point. So IMO this 'feature' is a trap. We should be asking why people want this feature and addressing that in a better way. This is a feature asking for a shared component between multiple environments. That component is (at the very least) the part that does the environment switching (as you put it, the routing logic). How will you test an update to that switch/data-routing-logic when it is part of both your dev and your live infrastructure?
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            I agree that the current pattern of having an overall hierarchy that includes levels that refer to data directories inside of the current environment is an anti-pattern. (Even worse if it is looking in multiple environments). The new solution makes data in modules and environments self contained - it does not matter what is in the top level hierarchy, or even if it is used at all (well, you can always use it to override what is configured in any environment; which is possibly a good feature for patching and mocking in a test environment, but use should be kept to a minimum).

            Show
            henrik.lindberg Henrik Lindberg added a comment - I agree that the current pattern of having an overall hierarchy that includes levels that refer to data directories inside of the current environment is an anti-pattern. (Even worse if it is looking in multiple environments). The new solution makes data in modules and environments self contained - it does not matter what is in the top level hierarchy, or even if it is used at all (well, you can always use it to override what is configured in any environment; which is possibly a good feature for patching and mocking in a test environment, but use should be kept to a minimum).
            Hide
            nathan Nathan Valentine added a comment -

            @david, I think I may now see the disconnect...

            The ticket was filed requesting some way to allow a per-environment hiera.yaml which is really just to say per-environment data routing logic. That is necessary to test both code and data through a promotion/testing pipeline. The technique is not 100% foolproof as there's almost always going to be some data which is particular to 'production' which it does not make sense to include in the Hiera data as static (YAML/JSON/etc). Repo commit hashes/tags/branch names is an example where this gets complicated. Most folks whom I've spoken with have this kind of data sourced from either a CMDB, service discovery layer, or even via a dynamic Hiera backend (hiera-http) but having per-environment Hiera settings and testing the static Hiera data does nearly close the gap.

            I don't recall anyone who I've spoken with about this ticket in particular, nor about the larger issue the ticket addresses, having requested that ability to query Hiera data across environments unless you count queries to external data stores (see above comments re CMDB, etc) where the data is not tightly bound to any particular environment.

            Henrik Lindberg, I'm glad to hear there are changes to the data binding layer coming which might obsolete the request in this ticket. Will take a look ASAP.

            Show
            nathan Nathan Valentine added a comment - @david, I think I may now see the disconnect... The ticket was filed requesting some way to allow a per-environment hiera.yaml which is really just to say per-environment data routing logic. That is necessary to test both code and data through a promotion/testing pipeline. The technique is not 100% foolproof as there's almost always going to be some data which is particular to 'production' which it does not make sense to include in the Hiera data as static (YAML/JSON/etc). Repo commit hashes/tags/branch names is an example where this gets complicated. Most folks whom I've spoken with have this kind of data sourced from either a CMDB, service discovery layer, or even via a dynamic Hiera backend (hiera-http) but having per-environment Hiera settings and testing the static Hiera data does nearly close the gap. I don't recall anyone who I've spoken with about this ticket in particular, nor about the larger issue the ticket addresses, having requested that ability to query Hiera data across environments unless you count queries to external data stores (see above comments re CMDB, etc) where the data is not tightly bound to any particular environment. Henrik Lindberg , I'm glad to hear there are changes to the data binding layer coming which might obsolete the request in this ticket. Will take a look ASAP.
            Hide
            ghoneycutt garrett honeycutt added a comment -

            My use case is for entirely different backends and hierarchies for different environments as opposed to building a dev->test->prod type of pipeline.

            Show
            ghoneycutt garrett honeycutt added a comment - My use case is for entirely different backends and hierarchies for different environments as opposed to building a dev->test->prod type of pipeline.
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            This ticket is turning out to be a good discussion about use cases and requirements. (I like that).

            To recap and talk about the ideas going forward. PUP-1640 adds the mechanism with which it is possible to provide data that is environment specific, and that is specific to a module. PUP-3948 makes the lookup function the consolidated lookup mechanism (it looks up in global hiera, in data from the environment, and in a module, and it can do the types of merges and tricks that the combined functions hiera, hiera_array, and hiera_hash can do (and a little bit more). The function does this without having a direct dependency on hiera - only on the current Data Binding API, and the new Data Provider API.

            In practice this means that much data that is now in a single hierarchy for all environments can migrate to be environment specific (or indeed migrate to modules if they are defaults for modules). Then the global data will be used for "site specific data" or "environment overrides/amends". This should help flowing a version of a configuration from a dev env to a test env to a production env on the same or different masters.

            The new approach allows having a separate hierarchy in the environment - it does not even have to be hiera based (if/when/where that makes more sense).

            After 4.0.0 is out, I want us to refactor hiera so it functions under the Data Provider API (i.e. that it no longer is a singleton (== strictly global).

            Show
            henrik.lindberg Henrik Lindberg added a comment - This ticket is turning out to be a good discussion about use cases and requirements. (I like that). To recap and talk about the ideas going forward. PUP-1640 adds the mechanism with which it is possible to provide data that is environment specific, and that is specific to a module. PUP-3948 makes the lookup function the consolidated lookup mechanism (it looks up in global hiera, in data from the environment, and in a module, and it can do the types of merges and tricks that the combined functions hiera , hiera_array , and hiera_hash can do (and a little bit more). The function does this without having a direct dependency on hiera - only on the current Data Binding API, and the new Data Provider API. In practice this means that much data that is now in a single hierarchy for all environments can migrate to be environment specific (or indeed migrate to modules if they are defaults for modules). Then the global data will be used for "site specific data" or "environment overrides/amends". This should help flowing a version of a configuration from a dev env to a test env to a production env on the same or different masters. The new approach allows having a separate hierarchy in the environment - it does not even have to be hiera based (if/when/where that makes more sense). After 4.0.0 is out, I want us to refactor hiera so it functions under the Data Provider API (i.e. that it no longer is a singleton (== strictly global).
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            I am curious about the use case "query across all environments" - what is the purpose of that? As I see it that seems to imply asking for "what is the value of parameter x::y::z for node n if it were in any of the currently known environments" - such a query cannot be answered without evaluating the puppet logic since it may lookup with different strategies (not all parameters get their value with priority lookup).

            If we take the node out of the equation - it is clearly possible to just iterate over all the known environments and lookup a key; it does not however then know which module to use for default values - it could guess module based on the fully qualified name, which would be correct most of the time (i.e. unless someone instantiates something that is outside of a module's name space).

            Show
            henrik.lindberg Henrik Lindberg added a comment - I am curious about the use case "query across all environments" - what is the purpose of that? As I see it that seems to imply asking for "what is the value of parameter x::y::z for node n if it were in any of the currently known environments" - such a query cannot be answered without evaluating the puppet logic since it may lookup with different strategies (not all parameters get their value with priority lookup). If we take the node out of the equation - it is clearly possible to just iterate over all the known environments and lookup a key; it does not however then know which module to use for default values - it could guess module based on the fully qualified name, which would be correct most of the time (i.e. unless someone instantiates something that is outside of a module's name space).
            Hide
            peiriannydd Trevor Vaughan added a comment -

            I'm not a fan of the 'query across all envrionments' use case. I think that hiera data should match the way environments currently work.

            In other words, look in my environment, then look in $basemodulepath (or the equivalent for hiera).

            By default, what I would expect as a user would be for it to look in the environment, then go up to the default puppet/hiera.yaml for direction.

            Show
            peiriannydd Trevor Vaughan added a comment - I'm not a fan of the 'query across all envrionments' use case. I think that hiera data should match the way environments currently work. In other words, look in my environment, then look in $basemodulepath (or the equivalent for hiera). By default, what I would expect as a user would be for it to look in the environment, then go up to the default puppet/hiera.yaml for direction.
            Hide
            kmueller Karl Mueller added a comment -

            I would be against "query over environments" as well. I think it's unclear and unsafe.

            For our particular use case, we've extracted most of the hiera files out to %

            {environment}

            /xxx/yyy type entries, so we are now doing most things "per environment". Defaults for all environments are just xxx/yyy and are overridden, but it's not ideal.

            One particular issue, which I think which is unsolved, is the inability to have the hiera.yaml be specific to an environment, thereby using and validating it. (or, for example, changing defaults for all environment) Globally sharing the hiera.yaml file, even with per-environment locations, is not ideal.

            Show
            kmueller Karl Mueller added a comment - I would be against "query over environments" as well. I think it's unclear and unsafe. For our particular use case, we've extracted most of the hiera files out to % {environment} /xxx/yyy type entries, so we are now doing most things "per environment". Defaults for all environments are just xxx/yyy and are overridden, but it's not ideal. One particular issue, which I think which is unsolved, is the inability to have the hiera.yaml be specific to an environment, thereby using and validating it. (or, for example, changing defaults for all environment) Globally sharing the hiera.yaml file, even with per-environment locations, is not ideal.
            Hide
            henrik.lindberg Henrik Lindberg added a comment -

            The intended implementation of "data in modules" will allow different hierarchies in different modules and different environments, as well as at the global-across-all-environments level. The intended implementation has an explicit hierarchy of global wins over environment, and environment wins over module. See PUP-4474.

            I don't see a good reason to implement a "cross environment data query".

            Show
            henrik.lindberg Henrik Lindberg added a comment - The intended implementation of "data in modules" will allow different hierarchies in different modules and different environments, as well as at the global-across-all-environments level. The intended implementation has an explicit hierarchy of global wins over environment, and environment wins over module. See PUP-4474 . I don't see a good reason to implement a "cross environment data query".
            Hide
            senax Frank Ederveen added a comment -

            Hi, it seems to me that there might be two different requests/issues here.

            1) different hiera.yaml for each environment
            2) one hiera, looking for data in different environments

            In my company we use environments as 'puppet+hiera' versions. For example 1_2_3, 1_3_1.
            Once a version/environment has been released it is never changed anymore. In our ENC; each server has an version/environment which is managed in puppet.conf.

            This is pretty much a requirement for companies which require very strict change control; for example financial services.

            For this scenario, having one shared hiera.yaml is not so good. Changes to backends or the hierarchy will affect already released environments.

            Thanks,
            Frank

            Show
            senax Frank Ederveen added a comment - Hi, it seems to me that there might be two different requests/issues here. 1) different hiera.yaml for each environment 2) one hiera, looking for data in different environments In my company we use environments as 'puppet+hiera' versions. For example 1_2_3, 1_3_1. Once a version/environment has been released it is never changed anymore. In our ENC; each server has an version/environment which is managed in puppet.conf. This is pretty much a requirement for companies which require very strict change control; for example financial services. For this scenario, having one shared hiera.yaml is not so good. Changes to backends or the hierarchy will affect already released environments. Thanks, Frank
            Hide
            ripienaar R.I.Pienaar added a comment -

            Frank Ederveen there is now a environment level hiera with a per environment hiera.yaml with their own settings and so forth.

            This is unfortunately not older hiera but rather the new lookup system so you more or less only have 2 backends to choose from today, but that appears to be the way forward and whatever backends people like needs porting.

            Show
            ripienaar R.I.Pienaar added a comment - Frank Ederveen there is now a environment level hiera with a per environment hiera.yaml with their own settings and so forth. This is unfortunately not older hiera but rather the new lookup system so you more or less only have 2 backends to choose from today, but that appears to be the way forward and whatever backends people like needs porting.

              People

              • Assignee:
                Unassigned
                Reporter:
                redmine.exporter redmine.exporter
              • Votes:
                63 Vote for this issue
                Watchers:
                84 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Zendesk Support