* Re: [Bitcoin-development] getmemorypool BIP process
@ 2012-03-03 15:44 Luke-Jr
2012-03-04 0:18 ` Stefan Thomas
0 siblings, 1 reply; 10+ messages in thread
From: Luke-Jr @ 2012-03-03 15:44 UTC (permalink / raw)
To: bitcoin-development
On Saturday, March 03, 2012 10:05:58 AM Gavin Andresen wrote:
> > HTTP and JSON-RPC are a client-server model; there is no way for the
> > server to make calls to the client. It's not practical to expect clients
> > to run their own JSON-RPC server - many cannot listen on WAN ports at
> > all.
>
> You're doing that thing where either you say something before you've
> done adequate research, or you're being needlessly pedantic; I just
> copied and pasted this from section 2.1 of the JSON-RPC spec:
I wasn't aware anyone was considering JSON-RPC over anything other than HTTP.
I've updated the draft to include long polling, and remove some assumptions of
using HTTP for transport.
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [Bitcoin-development] getmemorypool BIP process 2012-03-03 15:44 [Bitcoin-development] getmemorypool BIP process Luke-Jr @ 2012-03-04 0:18 ` Stefan Thomas 0 siblings, 0 replies; 10+ messages in thread From: Stefan Thomas @ 2012-03-04 0:18 UTC (permalink / raw) To: bitcoin-development > I've updated the draft to include long polling, and remove some assumptions of > using HTTP for transport. Looks good to me. On 3/3/2012 4:44 PM, Luke-Jr wrote: > On Saturday, March 03, 2012 10:05:58 AM Gavin Andresen wrote: >>> HTTP and JSON-RPC are a client-server model; there is no way for the >>> server to make calls to the client. It's not practical to expect clients >>> to run their own JSON-RPC server - many cannot listen on WAN ports at >>> all. >> You're doing that thing where either you say something before you've >> done adequate research, or you're being needlessly pedantic; I just >> copied and pasted this from section 2.1 of the JSON-RPC spec: > I wasn't aware anyone was considering JSON-RPC over anything other than HTTP. > > I've updated the draft to include long polling, and remove some assumptions of > using HTTP for transport. > > ------------------------------------------------------------------------------ > Virtualization& Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > ^ permalink raw reply [flat|nested] 10+ messages in thread
* [Bitcoin-development] getmemorypool BIP process
@ 2012-02-28 22:06 Luke-Jr
2012-03-03 14:23 ` Stefan Thomas
2012-03-03 23:51 ` Geir Harald Hansen
0 siblings, 2 replies; 10+ messages in thread
From: Luke-Jr @ 2012-02-28 22:06 UTC (permalink / raw)
To: bitcoin-development
Please review and comment/critique:
https://en.bitcoin.it/wiki/BIP_DRAFT:_getmemorypool
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [Bitcoin-development] getmemorypool BIP process 2012-02-28 22:06 Luke-Jr @ 2012-03-03 14:23 ` Stefan Thomas 2012-03-03 15:00 ` Luke-Jr 2012-03-03 23:51 ` Geir Harald Hansen 1 sibling, 1 reply; 10+ messages in thread From: Stefan Thomas @ 2012-03-03 14:23 UTC (permalink / raw) To: bitcoin-development From what I understand the BIP uses a polling model, e.g. a miner would use getmemorypool to request new work from a pool in intervals. Would it make sense to specify a version of the API supporting long polling? For BitcoinJS specifically, since we also support JSON-RPC over TCP, I'd even be interested in a genuine "push" API. Something like C->S "listenmemorypool", S->C "updatememorypool" (continually). On 2/28/2012 11:06 PM, Luke-Jr wrote: > Please review and comment/critique: > https://en.bitcoin.it/wiki/BIP_DRAFT:_getmemorypool > > ------------------------------------------------------------------------------ > Keep Your Developer Skills Current with LearnDevNow! > The most comprehensive online learning library for Microsoft developers > is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, > Metro Style Apps, more. Free future releases when you subscribe now! > http://p.sf.net/sfu/learndevnow-d2d > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] getmemorypool BIP process 2012-03-03 14:23 ` Stefan Thomas @ 2012-03-03 15:00 ` Luke-Jr 2012-03-03 17:08 ` Michael Grønager 0 siblings, 1 reply; 10+ messages in thread From: Luke-Jr @ 2012-03-03 15:00 UTC (permalink / raw) To: bitcoin-development On Saturday, March 03, 2012 9:23:08 AM Stefan Thomas wrote: > From what I understand the BIP uses a polling model, e.g. a miner would > use getmemorypool to request new work from a pool in intervals. Would it > make sense to specify a version of the API supporting long polling? You mean explicitly mention the getwork longpoll protocol as part of the getmemorypool BIP? Probably a good idea. > For BitcoinJS specifically, since we also support JSON-RPC over TCP, I'd > even be interested in a genuine "push" API. Something like C->S > "listenmemorypool", S->C "updatememorypool" (continually). HTTP and JSON-RPC are a client-server model; there is no way for the server to make calls to the client. It's not practical to expect clients to run their own JSON-RPC server - many cannot listen on WAN ports at all. ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] getmemorypool BIP process 2012-03-03 15:00 ` Luke-Jr @ 2012-03-03 17:08 ` Michael Grønager 2012-03-04 0:18 ` Stefan Thomas 0 siblings, 1 reply; 10+ messages in thread From: Michael Grønager @ 2012-03-03 17:08 UTC (permalink / raw) To: Luke-Jr; +Cc: bitcoin-development > > HTTP and JSON-RPC are a client-server model; there is no way for the server to > make calls to the client. It's not practical to expect clients to run their > own JSON-RPC server - many cannot listen on WAN ports at all. Well, I think what Stefan had in mind was http keep-alive combined with an event system. So similar to the way a web chat application work, just for json-rpc. BitcoinJS already uses this for realtime updating a webwallet. Libcoin is also prepared for this with a quite advanced, non-blocking, http server so I second Stefan that an update function could indeed be of relevance. Btw - question to Stefan as the JavaScript guru - what do you consider the standard/defacto-standard/right/best-practice way of doing S->C json-rpc, what (javascript) library do you use for this? Cheers, Michael > > ------------------------------------------------------------------------------ > Virtualization & Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] getmemorypool BIP process 2012-03-03 17:08 ` Michael Grønager @ 2012-03-04 0:18 ` Stefan Thomas 0 siblings, 0 replies; 10+ messages in thread From: Stefan Thomas @ 2012-03-04 0:18 UTC (permalink / raw) To: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 4051 bytes --] > Btw - question to Stefan as the JavaScript guru - what do you consider the standard/defacto-standard/right/best-practice way of doing S->C json-rpc, what (javascript) library do you use for this? As for an explicitly standard way, there is none. The JSON-RPC 1.0 spec <http://json-rpc.org/wiki/specification> says "The specifications do not require a certain transport protocol. The use of TCP/IP socket streams is encouraged. The serialized request and response objects are sent to the peers through the byte streams. " The JSON-RPC 2.0 spec <http://jsonrpc.org/specification> goes out of its way to say "It is transport agnostic in that the concepts can be used within the same process, over sockets, over http, or in many various message passing environments." The de-facto standard for bidirectional JSON-RPC is plain TCP sockets. BitcoinJS currently implements this - we detect whether an incoming connection is HTTP or raw JSON-RPC based on the first character. (HTTP must start with an uppercase letter, raw JSON-RPC must start with an opening curly bracket.) There are two things to watch out for with JSON-RPC over plain TCP: 1. Plain TCP sockets (unlike HTTP) have no standardized authentication mechanism, so I added an extra RPC call auth("username", "password"). 2. The TCP packets may or may not correspond to JSON-RPC messages. You can either use a streaming JSON parser (yajl in ANSI C, Jackson in Java, etc.), or you can just count (non-string-literal) curly braces to detect when a complete message has arrived. Many JSON-RPC libraries come with TCP socket support out of the box: http://json-rpc.org/wiki/implementations We're planning to add more features to our JSON-RPC API in the future, such as: - JSON-RPC over TLS sockets - Challenge-response authentication - TLS client handshake (certificate authentication) As for HTTP Keep-Alive: It works, but I don't think it's very widely supported among client libraries and HTTP isn't really made for this type of thing, so my gut instinct would be to avoid it. That said, it doesn't hurt to offer the option. On 3/3/2012 6:08 PM, Michael Grønager wrote: >> HTTP and JSON-RPC are a client-server model; there is no way for the server to >> make calls to the client. It's not practical to expect clients to run their >> own JSON-RPC server - many cannot listen on WAN ports at all. > Well, I think what Stefan had in mind was http keep-alive combined with an event system. So similar to the way a web chat application work, just for json-rpc. BitcoinJS already uses this for realtime updating a webwallet. Libcoin is also prepared for this with a quite advanced, non-blocking, http server so I second Stefan that an update function could indeed be of relevance. > > Btw - question to Stefan as the JavaScript guru - what do you consider the standard/defacto-standard/right/best-practice way of doing S->C json-rpc, what (javascript) library do you use for this? > > Cheers, > > Michael > > >> ------------------------------------------------------------------------------ >> Virtualization& Cloud Management Using Capacity Planning >> Cloud computing makes use of virtualization - but cloud computing >> also focuses on allowing computing to be delivered as a service. >> http://www.accelacomm.com/jaw/sfnl/114/51521223/ >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > ------------------------------------------------------------------------------ > Virtualization& Cloud Management Using Capacity Planning > Cloud computing makes use of virtualization - but cloud computing > also focuses on allowing computing to be delivered as a service. > http://www.accelacomm.com/jaw/sfnl/114/51521223/ > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > [-- Attachment #2: Type: text/html, Size: 5548 bytes --] ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] getmemorypool BIP process 2012-02-28 22:06 Luke-Jr 2012-03-03 14:23 ` Stefan Thomas @ 2012-03-03 23:51 ` Geir Harald Hansen 2012-03-04 1:04 ` Luke-Jr 1 sibling, 1 reply; 10+ messages in thread From: Geir Harald Hansen @ 2012-03-03 23:51 UTC (permalink / raw) To: bitcoin-development On 28.02.2012 23:06, Luke-Jr wrote: > Please review and comment/critique: > https://en.bitcoin.it/wiki/BIP_DRAFT:_getmemorypool Looking forward to implementing this in my pool backend and miner. A few comments: "transactions add or remove transactions (both of the above; default if "transactions" omitted) " In the above, you may want to specify that the "transactions" referred to here is the one in the first table (JSON-RPC response object) and not the mutations. For a moment I thought free tx editing was the default. Long polling as currently implemented in pools has a race condition. Does the miner reconnect first or does another block change happen first? "Double" block changes are common with merged mining and I'm doing all sorts of tricks in my pool backend to reduce this problem. How about another entry "longpollid" in long poll responses. The last seen longpollid should be included by the client in future long poll requests. This enables the server to see if the client has missed any block changes. The ID could perhaps be submitted in an HTTP header (X-LongPollID?) if we wish to keep the JSON-RPC params empty, or params could hold an object with a key "longpollid". Could be a string or number, like "workid". Another useful value in the getmemorypool response would be "height", so the miner can include the correct height in the coinbase. I would like that in bitcoind as well. One JSON-RPC call instead of two, and no race condition between getmemorypool and getblocknumber. It should be explained how target vs. fulltarget works. Perhaps some things should be optional for a client to implement? I think "noncerange" is of limited use and there's a good chance of getting the endianness wrong. Regards, Geir Harald Hansen ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] getmemorypool BIP process 2012-03-03 23:51 ` Geir Harald Hansen @ 2012-03-04 1:04 ` Luke-Jr 2012-03-04 17:49 ` Geir Harald Hansen 0 siblings, 1 reply; 10+ messages in thread From: Luke-Jr @ 2012-03-04 1:04 UTC (permalink / raw) To: bitcoin-development On Saturday, March 03, 2012 6:51:34 PM Geir Harald Hansen wrote: > Long polling as currently implemented in pools has a race condition. > Does the miner reconnect first or does another block change happen > first? "Double" block changes are common with merged mining and I'm > doing all sorts of tricks in my pool backend to reduce this problem. How would you suggest addressing this? I presume if a share solves blocks on multiple chains, you just longpoll once when that's successful? > How about another entry "longpollid" in long poll responses. The last > seen longpollid should be included by the client in future long poll > requests. This enables the server to see if the client has missed any > block changes. The ID could perhaps be submitted in an HTTP header > (X-LongPollID?) if we wish to keep the JSON-RPC params empty, or params > could hold an object with a key "longpollid". Could be a string or > number, like "workid". Hmm, the problem is that adding any parameters to getmemorypool itself breaks compatibility with bitcoind 0.5, and using HTTP headers makes it HTTP-specific again. Any ideas? > Another useful value in the getmemorypool response would be "height", so > the miner can include the correct height in the coinbase. I would like > that in bitcoind as well. One JSON-RPC call instead of two, and no race > condition between getmemorypool and getblocknumber. Good catch. Should this be required (since it might be necessary for future Bitcoin blocks), or just "should" for compatibility? > It should be explained how target vs. fulltarget works. What is unclear about this? > Perhaps some things should be optional for a client to implement? Doing this safely needs some way for clients to communicate capabilities to the server, which has the problem of passing parameters to getmemorypool. > I think "noncerange" is of limited use and there's a good chance of getting > the endianness wrong. There is no mining hardware to date that exhausts even half the nonce space, so I'd really prefer to see this as a required feature on the miner side. On the other hand, it's merely an extension for getwork, so I can see the problem so long as we're using getwork proxies. Luke ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] getmemorypool BIP process 2012-03-04 1:04 ` Luke-Jr @ 2012-03-04 17:49 ` Geir Harald Hansen 0 siblings, 0 replies; 10+ messages in thread From: Geir Harald Hansen @ 2012-03-04 17:49 UTC (permalink / raw) To: Luke-Jr; +Cc: bitcoin-development On 04.03.2012 02:04, Luke-Jr wrote: > On Saturday, March 03, 2012 6:51:34 PM Geir Harald Hansen wrote: >> Long polling as currently implemented in pools has a race condition. >> Does the miner reconnect first or does another block change happen >> first? "Double" block changes are common with merged mining and I'm >> doing all sorts of tricks in my pool backend to reduce this problem. > > How would you suggest addressing this? I presume if a share solves blocks on > multiple chains, you just longpoll once when that's successful? If the client doesn't tell you which block change it saw last, you can't avoid this problem completely. >> How about another entry "longpollid" in long poll responses. The last >> seen longpollid should be included by the client in future long poll >> requests. This enables the server to see if the client has missed any >> block changes. The ID could perhaps be submitted in an HTTP header >> (X-LongPollID?) if we wish to keep the JSON-RPC params empty, or params >> could hold an object with a key "longpollid". Could be a string or >> number, like "workid". > > Hmm, the problem is that adding any parameters to getmemorypool itself breaks > compatibility with bitcoind 0.5, and using HTTP headers makes it HTTP-specific > again. Any ideas? I think long polling and reconnecting to LP is an HTTP-specific problem anyway? So it may be best to go with an HTTP header. See below for an idea for a capabilities reporting system, though, which would allow putting this data into JSON. >> Another useful value in the getmemorypool response would be "height", so >> the miner can include the correct height in the coinbase. I would like >> that in bitcoind as well. One JSON-RPC call instead of two, and no race >> condition between getmemorypool and getblocknumber. > > Good catch. Should this be required (since it might be necessary for future > Bitcoin blocks), or just "should" for compatibility? I think "should" makes sense. >> It should be explained how target vs. fulltarget works. > > What is unclear about this? Reading through it again it is mostly clear except for one thing, how to submit "share/merkle" style. It doesn't say exactly how to submit this data. It says merkle tree, but a merkle branch should suffice. Also, I believe the coinbase must always be the first transaction of a block? So we won't need the transaction's index in the merkle tree (it's always index zero). This would then be just an array of sha256 hashes making up the merkle branch for the coinbase. I'd prefer if block header, coinbase tx and merkle branch were submitted in 3 pieces. When merged mining and submitting a block generating proof to the aux backend you put an auxiliary proof of work between the block header and transactions. So the header and transactions are not necessarily adjacent. Also, there's no merkle branch in a regular bitcoin block. >> Perhaps some things should be optional for a client to implement? > > Doing this safely needs some way for clients to communicate capabilities to > the server, which has the problem of passing parameters to getmemorypool. Let's add an optional "servercapabilities" key on the server side, the value being a space-delimited list of what's supported by the server. If that list contains "clientcapabilities" then the client may, instead of sending an empty request, send an object where one of the keys is "clientcapabilities", and the value is a space-delimited list of extensions. Those names are a bit long, maybe "caps" is better. Ok, this is not entirely clean. For the first request the client won't know what the server supports. For subsequent requests I guess always assume the last caps the server reported. >> I think "noncerange" is of limited use and there's a good chance of getting >> the endianness wrong. > > There is no mining hardware to date that exhausts even half the nonce space, > so I'd really prefer to see this as a required feature on the miner side. On > the other hand, it's merely an extension for getwork, so I can see the problem > so long as we're using getwork proxies. Yes, you can end up getting half a nonce range from an upstream server, and not being able to make use of it because your downstream clients don't support it. A single GPU can make full use of the noncerange if you allow the timestamp to slip a few seconds into the past. I believe most miners do that today. Also worth including are the "hostlist" and "switchto" getwork extensions. I think those can fit right into the JSON response, to keep with being transport-agnostic, but otherwise look like they do for getwork. Regards, Geir H. Hansen ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2012-03-04 17:49 UTC | newest] Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2012-03-03 15:44 [Bitcoin-development] getmemorypool BIP process Luke-Jr 2012-03-04 0:18 ` Stefan Thomas -- strict thread matches above, loose matches on Subject: below -- 2012-02-28 22:06 Luke-Jr 2012-03-03 14:23 ` Stefan Thomas 2012-03-03 15:00 ` Luke-Jr 2012-03-03 17:08 ` Michael Grønager 2012-03-04 0:18 ` Stefan Thomas 2012-03-03 23:51 ` Geir Harald Hansen 2012-03-04 1:04 ` Luke-Jr 2012-03-04 17:49 ` Geir Harald Hansen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox