* [Bitcoin-development] Bumping up against flood control limits again?
@ 2011-06-14 16:44 Mike Hearn
2011-06-14 17:07 ` Dave Smith
2011-06-14 17:14 ` Jeff Garzik
0 siblings, 2 replies; 3+ messages in thread
From: Mike Hearn @ 2011-06-14 16:44 UTC (permalink / raw)
To: bitcoin-development
Block sizes have started to get quite large once again. Whilst testing
chain download today I was disconnected due to going over the 10mb
flood control limit. Infuriatingly, I can't reproduce this reliably.
But at 500 blocks an average of 20kb per block will cause this. As we
can see from the block explorer, the average is probably quite close
to that.
The flood control seems like a pretty serious scalability limitation.
I can see a few solutions. One is to raise the limit again. Another is
to raise the limit and simultaneously lower the batch size. 500 blocks
in one message means very large messages no matter how big the flood
control limit is. Going down to 100 or even 50 would hurt chain
download speed quite a bit in high latency environments, but chain
download is already a serious bottleneck.
Thoughts?
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Bitcoin-development] Bumping up against flood control limits again?
2011-06-14 16:44 [Bitcoin-development] Bumping up against flood control limits again? Mike Hearn
@ 2011-06-14 17:07 ` Dave Smith
2011-06-14 17:14 ` Jeff Garzik
1 sibling, 0 replies; 3+ messages in thread
From: Dave Smith @ 2011-06-14 17:07 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
Does it make more sense to have the flood limit based on size in bytes
versus # of blocks?
(I'm a n00b, so pardon my ignorance). :)
D.
On Tue, Jun 14, 2011 at 10:44 AM, Mike Hearn <mike@plan99•net> wrote:
> Block sizes have started to get quite large once again. Whilst testing
> chain download today I was disconnected due to going over the 10mb
> flood control limit. Infuriatingly, I can't reproduce this reliably.
> But at 500 blocks an average of 20kb per block will cause this. As we
> can see from the block explorer, the average is probably quite close
> to that.
>
> The flood control seems like a pretty serious scalability limitation.
> I can see a few solutions. One is to raise the limit again. Another is
> to raise the limit and simultaneously lower the batch size. 500 blocks
> in one message means very large messages no matter how big the flood
> control limit is. Going down to 100 or even 50 would hurt chain
> download speed quite a bit in high latency environments, but chain
> download is already a serious bottleneck.
>
> Thoughts?
>
> ------------------------------------------------------------------------------
> EditLive Enterprise is the world's most technically advanced content
> authoring tool. Experience the power of Track Changes, Inline Image
> Editing and ensure content is compliant with Accessibility Checking.
> http://p.sf.net/sfu/ephox-dev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [Bitcoin-development] Bumping up against flood control limits again?
2011-06-14 16:44 [Bitcoin-development] Bumping up against flood control limits again? Mike Hearn
2011-06-14 17:07 ` Dave Smith
@ 2011-06-14 17:14 ` Jeff Garzik
1 sibling, 0 replies; 3+ messages in thread
From: Jeff Garzik @ 2011-06-14 17:14 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
On Tue, Jun 14, 2011 at 12:44 PM, Mike Hearn <mike@plan99•net> wrote:
> Block sizes have started to get quite large once again. Whilst testing
> chain download today I was disconnected due to going over the 10mb
> flood control limit. Infuriatingly, I can't reproduce this reliably.
> But at 500 blocks an average of 20kb per block will cause this. As we
> can see from the block explorer, the average is probably quite close
> to that.
>
> The flood control seems like a pretty serious scalability limitation.
> I can see a few solutions. One is to raise the limit again. Another is
> to raise the limit and simultaneously lower the batch size. 500 blocks
> in one message means very large messages no matter how big the flood
> control limit is. Going down to 100 or even 50 would hurt chain
> download speed quite a bit in high latency environments, but chain
> download is already a serious bottleneck.
The main goal was not flood control but preventing an internal buffer
memory explosion. We already have the block chain on disk, so in
theory, if we can -eliminate- the outgoing network buffer and simply
use a pointer into the block chain file, we can send as much data as
we want.
HTTP servers certainly don't buffer huge amounts in memory; they would
keel over if so. HTTP servers have been working on the reverse, in
fact: moving the data-pushing over to sendfile(2) syscall and similar
optimizations.
This is an unfortunate relic of how bitcoin P2P code is written. If
the remote side has reduced their TCP window to zero, bitcoin will
still buffer so that it may continue processing other P2P traffic from
other nodes. That makes sense in the case of tiny, 31-byte address
messages -- one must handle the case of a half-sent message before
write(2) refuses additional data -- but not huge block chain download
messages.
The P2P code just wasn't written for huge amounts of streaming data,
and needs some serious thinking... I agree 100% that it is an issue
we will start bumping into, if we haven't already.
--
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti•com
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2011-06-14 17:14 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-06-14 16:44 [Bitcoin-development] Bumping up against flood control limits again? Mike Hearn
2011-06-14 17:07 ` Dave Smith
2011-06-14 17:14 ` Jeff Garzik
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox