[PUP-3524] Switch from YAML (et al) to JSON Created: 2014/10/23  Updated: 2018/03/07  Resolved: 2017/02/23

Status: Closed
Project: Puppet
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Epic Priority: Normal
Reporter: Andrew Parker Assignee: Unassigned
Resolution: Won't Fix Votes: 1
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
relates to PUP-3852 Switch from PSON to JSON as default s... Closed
relates to PUP-3706 console format should pretty print ne... Closed
Epic Name: JSON All The Things!


I'd like to move us away from PSON and onto a standard format. YAML is out of the question because it is either slow and unsafe (all of the YAML vulnerabilities) or extremely slow and safe (safe_yaml). MessagePack might be nice. It is pretty well specified, has a fairly large number of libraries written for it, but it doesn't do much to help us solve the wild west of encoding in puppet. In MessagePack there aren't really any enforcements of string encodings and everything is treated as an array of bytes.

In order to keep consistency across various puppet projects we'll be going with JSON. JSON requires that everything is valid UTF-8, which gives us a nice deliberateness to handling data. JSON is pretty fast (not as fast as MessagePack) and there are a lot of libraries if it turns out that the built in json isn't fast enough (puppet-server could use jrjackson, for instance).

Comment by Henrik Lindberg [ 2014/10/23 ]

One note that I also sent out on puppet dev.

I would like us to add a Binary datatype upfront instead of doing the base64 encoding in the puppet code. Instead, it is the serialization formats responsibility to transform it into a form that can be transported. A JSON in text form can then do the base64 encoding. A MsgPack / JSON can instead use the binary directly.

Even if our first cut of this always performs a base64 encoding the user logic does not have to change.

Thus, instead of calling base64(content) and setting the encoding in the File resource, a Binary is created directly with a binary(encoding, content) function.

BTW: MsgPack is essentially binary-JSON with compression. We can easily support a msgpack/json format in addition to text/json. Where the text/json for file content is either a String or Hash, where the hash is

{ encoding => base64encodedString }

and the Msgpack is either String or a Hash with

{ encoding => MsgPackBinary }

Without any other changes, the only place we support the Binary is in File's content so that serialization/deserialization needs to be special. When we have typed parameters, the serialization is instead driven by the type.

Comment by Andrew Parker [ 2014/10/27 ]

I'm leaving out things like config files that puppet doesn't produce. This means things like the certificate extensions on an agent aren't on this list of items. I did create a ticket for the exec ENC, but it isn't something that really has to be done and we can decide to just close it "won't fix" with no detriment to other tickets.

Comment by Chris Price [ 2015/01/27 ]

After discussion with Kylo Ginsberg (and a bit with Henrik Lindberg as well, we decided to break this into two chunks of work. The ticket was formerly titled "Switch from YAML and PSON to JSON". We're going to break out the PSON/networking layer subtasks into a separate epic/story (PUP-3852), and move the related tickets over there. The remaining stuff in this epic should be mostly about files we store on disk, which are most (all?) stored as YAML.

The plan for now is to have the Puppet Server team own the PSON epic in the run-up to Puppet 5.0, and the Client/Language teams would own the remaining tickets.

Comment by Moses Mendoza [ 2016/12/05 ]

Ping Chris Price Henrik Lindberg is this effort still on the docket for Puppet 5? Still as high-value as we originally considered it?

Comment by Henrik Lindberg [ 2016/12/06 ]

Still high value - we are not ready to get this done for 5.0.0 even if we made progress; there is a Pcore implementation for JVM now that supports Pcore serialization over Json and MsgPack. Think we can make that an opt-in during the 5.x series. Changing version to 5.y

Generated at Thu Apr 09 08:06:13 PDT 2020 using Jira 8.5.2#805002-sha1:a66f9354b9e12ac788984e5d84669c903a370049.