[PDB-1465] PR (1372): (maint) Detangle query-eng database and http streaming - ajroetker Created: 2015/04/30  Updated: 2015/07/10  Resolved: 2015/05/14

Status: Closed
Project: PuppetDB
Component/s: None
Affects Version/s: None
Fix Version/s: PDB 3.0.0

Type: Task Priority: Normal
Reporter: gepetto-bot Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: github
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Template:
Story Points: 1
Sprint: PuppetDB 2015-05-06, PuppetDB 2015-05-20

 Description   

(maint) Detangle query-eng database and http streaming

Pull Request Description


This commit addresses the "callback hell" of the query_eng.clj. Prior to
this commit the query-eng stream-query-result function had an output-fn
arguement whose interface was as follows, "output-fn takes a function
(f) as an argument and applies that funtion to another function (g),
(g) must realize the values of a lazy sequence of database rows."
This was primarily do to the fact the we need to stream results from the
database as well as stream results of http, which had to be done in a
separate thread. Another layer of complexity was that the http streaming
that we did also needed access to the count-query vector only available
int the stream-query result vector. All this complexity makes it very
confusing developing callbacks for querying puppetdb in process
directly, such as from PE extensions. This commit moves the http
streaming up a functional layer into produce-streaming body, passing
stream-query-result a promise to retrieve the count-query.


(webhooks-id: f057e4fbbbf4ec29d8053021a9487c24)



 Comments   
Comment by gepetto-bot [ 2015/04/30 ]

ajroetker commented:

@wkalt @mullr @senior it looks scary but its actually less scary!! Basically stream-query-results has been fixed such that the callback that it takes is what we apply directly to the result-sets we stream from the database, so no more complicated callbacks and wrappers are necessary in the sync repo yada yada yada.

Comment by gepetto-bot [ 2015/04/30 ]

pljenkinsro commented:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1133/

Comment by gepetto-bot [ 2015/05/01 ]

pljenkinsro commented:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1138/

Comment by gepetto-bot [ 2015/05/05 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1153/

Comment by gepetto-bot [ 2015/05/05 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1163/

Comment by gepetto-bot [ 2015/05/06 ]

ajroetker commented:

@pljenkinsro retest this please

Comment by gepetto-bot [ 2015/05/06 ]

pljenkinsro commented:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1172/

Comment by gepetto-bot [ 2015/05/06 ]

senior commented:

I'm taking a look at this

Comment by gepetto-bot [ 2015/05/06 ]

senior commented:

I tested the patch, works fine for me. Mostly minor comments, but I think we need to revisit the count-query stuff.

Comment by gepetto-bot [ 2015/05/06 ]

pljenkinsro commented:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1178/

Comment by gepetto-bot [ 2015/05/08 ]

ajroetker commented:

Refactored but relies on @wkalt's patch to remove late project

Comment by gepetto-bot [ 2015/05/12 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1203/

Comment by gepetto-bot [ 2015/05/12 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1202/

Comment by gepetto-bot [ 2015/05/12 ]

ajroetker commented:

@pljenkinsro retest this please

Comment by gepetto-bot [ 2015/05/12 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1204/

Comment by gepetto-bot [ 2015/05/12 ]

ajroetker commented:

@pljenkinsro retest this please

Comment by gepetto-bot [ 2015/05/12 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1205/

Comment by gepetto-bot [ 2015/05/12 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1206/

Comment by gepetto-bot [ 2015/05/12 ]

ajroetker commented:

@pljenkinsro retest this please

Comment by gepetto-bot [ 2015/05/12 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1207/

Comment by gepetto-bot [ 2015/05/13 ]

ajroetker commented:

@pljenkinsro retest this please

Comment by gepetto-bot [ 2015/05/13 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1210/

Comment by gepetto-bot [ 2015/05/13 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1214/

Comment by gepetto-bot [ 2015/05/13 ]

ajroetker commented:

@pljenkinsro retest this please

Comment by gepetto-bot [ 2015/05/13 ]

pljenkinsro commented:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1215/

Comment by gepetto-bot [ 2015/05/13 ]

ajroetker commented:

@senior that was indeed the problem, looks good now!

Comment by gepetto-bot [ 2015/05/14 ]

ajroetker commented:

@senior fixed for your comment

Comment by gepetto-bot [ 2015/05/14 ]

pljenkinsro commented:

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1219/

Comment by gepetto-bot [ 2015/05/14 ]

ajroetker commented:

@pljenkinsro retest this please

Comment by gepetto-bot [ 2015/05/14 ]

pljenkinsro commented:

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://jenkins.puppetlabs.com/job/platform_puppetdb_intn-sys_pr/1220/

Comment by gepetto-bot [ 2015/05/14 ]

Pull request (maint) Detangle query-eng database and http streaming has been closed.

Generated at Sat Aug 24 21:43:49 PDT 2019 using JIRA 7.7.1#77002-sha1:e75ca93d5574d9409c0630b81c894d9065296414.