* [Bitcoin-development] Implementing batch processing for -blocknotify
@ 2013-05-31 11:56 Rune Kjær Svendsen
2013-05-31 12:10 ` Michael Hendricks
` (3 more replies)
0 siblings, 4 replies; 10+ messages in thread
From: Rune Kjær Svendsen @ 2013-05-31 11:56 UTC (permalink / raw)
To: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 2991 bytes --]
Hello dear list
I have an application that wants to keep up with new blocks as they come
in. For that I can use the -blocknotify option with bitcoind, which will
execute my application for each new block.
The problem is that my app isn't necessarily quick enough to finish its
work before a new block comes in and the app is executed again. This means
the that bitcoind might keep executing my application even though the
previous instance hasn't finished, and that's fairly inefficient
resource-wise, as many instances of the application will be running
simultaneously.
I've discussed this with wumpus on bitcoin-dev, and we figured out a
solution that might be better. It could replace -blocknotify or we could
put it in a new function called -batchblocknotify
The idea is that when bitcoind is executed with the -batchblocknotify
option, and it's missing a lot of blocks, upon the first incoming block,
the command specified by -batchblocknotify is executed, and if additional
blocks come in while this command is still running, we add the block hashes
to a list instead of executing the command again. When the previous command
finishes, we execute it again and pass two parameters to it: 1. the first
block hash in the list of queued blocks, and 2. the number of blocks that
have come in while the last command was executing.
This prevents bitcoind from "fork bombing" the system, and allows the
command to handle incoming blocks in batches.
Would this make sense as an approach?
I've been looking at the code and I'm not sure how to implement it.
As far as I can see, I need to pass an object - whose state is retained
between calls - to the thread function (runCommand) that runs the command,
which contains a variable that keeps track of whether a previously executed
command is still running, and that contains a list of block hashes that
haven't been processed. And I'm not sure how to do this.
The runCommand thread is started in SetBestChain() in
main.cpp. SetBestChain() is executed by ConnectBestBlock() in main.cpp.
ConnectBestBlock() is executed by CBlock::AddToBlockIndex() in main.cpp.
CBlock::AddToBlockIndex() is executed by CBlock::AcceptBlock() in main.cpp.
CBlock::AcceptBlock() is executed by ProcessBlock() in main.cpp.
ProcessBlock() is executed by ProcessMessage() in main.cpp. And so on, and
so forth.
What's the right way to create an object that can be passed to the
runCommand thread, whose state is retained, so it can hold information
about whether the -batchblocknotify command is still executing, and contain
a list of blocks that are waiting to be passed to the -batchblocknotify
command?
I assume I shouldn't add a new parameter to the ProcessMessage() function,
which passes it to ProcessBlock(), which passes it to AcceptBlock() which
passes it to AddToBlockIndex()... and so on.
Would it be appropriate to store this object inside the CValidationState
class that is passed to SetBestChain()?
I'm not quite so how to go about this.
/Rune
[-- Attachment #2: Type: text/html, Size: 3449 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 11:56 [Bitcoin-development] Implementing batch processing for -blocknotify Rune Kjær Svendsen
@ 2013-05-31 12:10 ` Michael Hendricks
2013-05-31 12:37 ` Rune Kjær Svendsen
2013-05-31 12:54 ` Andy Parkins
` (2 subsequent siblings)
3 siblings, 1 reply; 10+ messages in thread
From: Michael Hendricks @ 2013-05-31 12:10 UTC (permalink / raw)
To: Rune Kjær Svendsen; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 722 bytes --]
On Fri, May 31, 2013 at 5:56 AM, Rune Kjær Svendsen <runesvend@gmail•com>wrote:
> I have an application that wants to keep up with new blocks as they come
> in. For that I can use the -blocknotify option with bitcoind, which will
> execute my application for each new block.
>
> The problem is that my app isn't necessarily quick enough to finish its
> work before a new block comes in and the app is executed again.
>
In a similar circumstance, I changed my -blocknotify script to quickly
append necessary information to a queue and immediately exit. A separate
script runs at all times monitoring this queue for work and performs the
labor intensive calculations.
I hope that helps.
--
Michael
[-- Attachment #2: Type: text/html, Size: 1132 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 12:10 ` Michael Hendricks
@ 2013-05-31 12:37 ` Rune Kjær Svendsen
2013-05-31 19:25 ` Jeff Garzik
0 siblings, 1 reply; 10+ messages in thread
From: Rune Kjær Svendsen @ 2013-05-31 12:37 UTC (permalink / raw)
To: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1773 bytes --]
On Fri, May 31, 2013 at 2:10 PM, Michael Hendricks <michael@ndrix•org>wrote:
> On Fri, May 31, 2013 at 5:56 AM, Rune Kjær Svendsen <runesvend@gmail•com>wrote:
>
>> I have an application that wants to keep up with new blocks as they come
>> in. For that I can use the -blocknotify option with bitcoind, which will
>> execute my application for each new block.
>>
>> The problem is that my app isn't necessarily quick enough to finish its
>> work before a new block comes in and the app is executed again.
>>
>
> In a similar circumstance, I changed my -blocknotify script to quickly
> append necessary information to a queue and immediately exit. A separate
> script runs at all times monitoring this queue for work and performs the
> labor intensive calculations.
>
I've thought about this as well. It just seems somewhat clunky to me. I'd
really prefer having bitcoind put out messages in batches, if it's doable,
that is.
I'd run into a lot of concurrency issues, as far as I can see, where I
can't be sure that the queue isn't written to while, for example, it is
opened by the program that needs to process the queue items.
What if a disk operation takes a long time to finish, and a two queue
operations want to add to the queue simultaneously? This really brings
forward all the horrors of concurrent programming.
On Fri, May 31, 2013 at 2:17 PM, Jeremy Spilman <jeremy@taplink•co> wrote:
> Would it work to just block the bitcoind thread until your process exits?
>
I don't think that's optimal, no. That would slow down synchronization
drastically.
It would be really nimble for bitcoind to be able to synchronize at full
speed, and only send out events when necessary, batching together
previously queued items.
[-- Attachment #2: Type: text/html, Size: 3100 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 11:56 [Bitcoin-development] Implementing batch processing for -blocknotify Rune Kjær Svendsen
2013-05-31 12:10 ` Michael Hendricks
@ 2013-05-31 12:54 ` Andy Parkins
2013-05-31 13:05 ` Jeff Garzik
2013-05-31 22:20 ` Chris Double
3 siblings, 0 replies; 10+ messages in thread
From: Andy Parkins @ 2013-05-31 12:54 UTC (permalink / raw)
To: bitcoin-development
On Friday 31 May 2013 12:56:43 Rune Kjær Svendsen wrote:
> I have an application that wants to keep up with new blocks as they come
> in. For that I can use the -blocknotify option with bitcoind, which will
> execute my application for each new block.
>
> The problem is that my app isn't necessarily quick enough to finish its
> work before a new block comes in and the app is executed again. This means
> the that bitcoind might keep executing my application even though the
> previous instance hasn't finished, and that's fairly inefficient
> resource-wise, as many instances of the application will be running
> simultaneously.
Have your program try to create a unix-domain socket when it starts. If it
can't create it (because one already exists at that path), then connect to it.
You then have two modes:
- Creator of socket, listens to socket for more incoming data, and adds it to
some sort of internal block queue.
- Client to socket, pushes output of -blocknotify to socket and exits
Your concurrency problems go away because only one process is ever actually
doing something with the data.
Should be fairly straight forward. The client is simple. The server is two
threads, one listening on the socket and then briefly locking and updating a
queue, and one thread briefly locking and removing from the queue.
Andy
--
Dr Andy Parkins
andyparkins@gmail•com
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 11:56 [Bitcoin-development] Implementing batch processing for -blocknotify Rune Kjær Svendsen
2013-05-31 12:10 ` Michael Hendricks
2013-05-31 12:54 ` Andy Parkins
@ 2013-05-31 13:05 ` Jeff Garzik
2013-05-31 22:20 ` Chris Double
3 siblings, 0 replies; 10+ messages in thread
From: Jeff Garzik @ 2013-05-31 13:05 UTC (permalink / raw)
To: Rune Kjær Svendsen; +Cc: bitcoin-development
On Fri, May 31, 2013 at 7:56 AM, Rune Kjær Svendsen <runesvend@gmail•com> wrote:
> Hello dear list
>
> I have an application that wants to keep up with new blocks as they come in.
> For that I can use the -blocknotify option with bitcoind, which will execute
> my application for each new block.
>
> The problem is that my app isn't necessarily quick enough to finish its work
> before a new block comes in and the app is executed again. This means the
> that bitcoind might keep executing my application even though the previous
> instance hasn't finished, and that's fairly inefficient resource-wise, as
> many instances of the application will be running simultaneously.
>
> I've discussed this with wumpus on bitcoin-dev, and we figured out a
> solution that might be better. It could replace -blocknotify or we could put
> it in a new function called -batchblocknotify
Similar to other suggestions in this thread,
If your -blocknotify execution is too slow, then create a solution
that simply queues work. There is no need to add this code to
bitcoind itself.
Another solution is modifying pynode to directly listen to a trusted
node (bitcoind), and monitor blocks as they arrive and are announced.
That does not fix the problem of slow block processing on your side,
but is another way to implement -blocknotify-like behavior.
--
Jeff Garzik
Senior Software Engineer and open source evangelist
BitPay, Inc. https://bitpay.com/
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 12:37 ` Rune Kjær Svendsen
@ 2013-05-31 19:25 ` Jeff Garzik
0 siblings, 0 replies; 10+ messages in thread
From: Jeff Garzik @ 2013-05-31 19:25 UTC (permalink / raw)
To: Rune Kjær Svendsen; +Cc: Bitcoin Dev
On Fri, May 31, 2013 at 8:37 AM, Rune Kjær Svendsen <runesvend@gmail•com> wrote:
> I've thought about this as well. It just seems somewhat clunky to me. I'd
> really prefer having bitcoind put out messages in batches, if it's doable,
> that is.
>
> I'd run into a lot of concurrency issues, as far as I can see, where I can't
> be sure that the queue isn't written to while, for example, it is opened by
> the program that needs to process the queue items.
>
> What if a disk operation takes a long time to finish, and a two queue
> operations want to add to the queue simultaneously? This really brings
> forward all the horrors of concurrent programming.
This is not a compelling need to update bitcoind for this.
The vast majority of systems are currently capable of processing a
block, before another block arrives.
As for parallel processing, your "what if" has been a solved problem
for decade(s) now.
--
Jeff Garzik
Senior Software Engineer and open source evangelist
BitPay, Inc. https://bitpay.com/
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 11:56 [Bitcoin-development] Implementing batch processing for -blocknotify Rune Kjær Svendsen
` (2 preceding siblings ...)
2013-05-31 13:05 ` Jeff Garzik
@ 2013-05-31 22:20 ` Chris Double
2013-05-31 23:29 ` Wladimir
3 siblings, 1 reply; 10+ messages in thread
From: Chris Double @ 2013-05-31 22:20 UTC (permalink / raw)
To: Rune Kjær Svendsen; +Cc: bitcoin-development
On Fri, May 31, 2013 at 11:56 PM, Rune Kjær Svendsen
<runesvend@gmail•com> wrote:
> I'm not quite so how to go about this.
As others have said, queuing outside of bitcoind is a better approach.
I use zeromq for this situation. blocknotify runs a program which uses
zeromq's pub/sub to queue and the application subscribes to this to
get notified of the data.
--
http://www.bluishcoder.co.nz
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 22:20 ` Chris Double
@ 2013-05-31 23:29 ` Wladimir
2013-05-31 23:47 ` Chris Double
0 siblings, 1 reply; 10+ messages in thread
From: Wladimir @ 2013-05-31 23:29 UTC (permalink / raw)
To: Chris Double; +Cc: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1401 bytes --]
Chris,
Using zmq is a great fit for high-speed notifications such as this. Have
you seen the pull request to integrate zmq directly into bitcoind, so that
you don't even need -blocknotify?
https://github.com/bitcoin/bitcoin/pull/2415
If not: we could use some testing there!
Wladimir
On Sat, Jun 1, 2013 at 12:20 AM, Chris Double <chris.double@double•co.nz>wrote:
> On Fri, May 31, 2013 at 11:56 PM, Rune Kjær Svendsen
> <runesvend@gmail•com> wrote:
> > I'm not quite so how to go about this.
>
> As others have said, queuing outside of bitcoind is a better approach.
> I use zeromq for this situation. blocknotify runs a program which uses
> zeromq's pub/sub to queue and the application subscribes to this to
> get notified of the data.
>
> --
> http://www.bluishcoder.co.nz
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite
> It's a free troubleshooting tool designed for production
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://p.sf.net/sfu/appdyn_d2d_ap2
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 2337 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 23:29 ` Wladimir
@ 2013-05-31 23:47 ` Chris Double
2013-06-01 13:12 ` Rune Kjær Svendsen
0 siblings, 1 reply; 10+ messages in thread
From: Chris Double @ 2013-05-31 23:47 UTC (permalink / raw)
To: Bitcoin Dev
On Sat, Jun 1, 2013 at 11:29 AM, Wladimir <laanwj@gmail•com> wrote:
> Using zmq is a great fit for high-speed notifications such as this. Have you
> seen the pull request to integrate zmq directly into bitcoind, so that you
> don't even need -blocknotify?
>
> https://github.com/bitcoin/bitcoin/pull/2415
>
> If not: we could use some testing there!
I hadn't seen this, thanks! I've integrated zmq myself for block
notifications in bitcoin and the alt coins I use it on my bitparking
merge mining pool. I would love to see something official. I'll try
out the patch in the pull request.
Chris.
--
http://www.bluishcoder.co.nz
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Implementing batch processing for -blocknotify
2013-05-31 23:47 ` Chris Double
@ 2013-06-01 13:12 ` Rune Kjær Svendsen
0 siblings, 0 replies; 10+ messages in thread
From: Rune Kjær Svendsen @ 2013-06-01 13:12 UTC (permalink / raw)
To: Bitcoin Dev
[-- Attachment #1: Type: text/plain, Size: 1467 bytes --]
Thanks for the input all! You make a lot of sense :). I will go with the
Unix socket method, I think. It's about time I learn about those.
/Rune
On Sat, Jun 1, 2013 at 1:47 AM, Chris Double <chris.double@double•co.nz>wrote:
> On Sat, Jun 1, 2013 at 11:29 AM, Wladimir <laanwj@gmail•com> wrote:
> > Using zmq is a great fit for high-speed notifications such as this. Have
> you
> > seen the pull request to integrate zmq directly into bitcoind, so that
> you
> > don't even need -blocknotify?
> >
> > https://github.com/bitcoin/bitcoin/pull/2415
> >
> > If not: we could use some testing there!
>
> I hadn't seen this, thanks! I've integrated zmq myself for block
> notifications in bitcoin and the alt coins I use it on my bitparking
> merge mining pool. I would love to see something official. I'll try
> out the patch in the pull request.
>
> Chris.
> --
> http://www.bluishcoder.co.nz
>
>
> ------------------------------------------------------------------------------
> Get 100% visibility into Java/.NET code with AppDynamics Lite
> It's a free troubleshooting tool designed for production
> Get down to code-level detail for bottlenecks, with <2% overhead.
> Download for free and get started troubleshooting in minutes.
> http://p.sf.net/sfu/appdyn_d2d_ap2
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 2455 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2013-06-01 13:12 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2013-05-31 11:56 [Bitcoin-development] Implementing batch processing for -blocknotify Rune Kjær Svendsen
2013-05-31 12:10 ` Michael Hendricks
2013-05-31 12:37 ` Rune Kjær Svendsen
2013-05-31 19:25 ` Jeff Garzik
2013-05-31 12:54 ` Andy Parkins
2013-05-31 13:05 ` Jeff Garzik
2013-05-31 22:20 ` Chris Double
2013-05-31 23:29 ` Wladimir
2013-05-31 23:47 ` Chris Double
2013-06-01 13:12 ` Rune Kjær Svendsen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox