From: Jeff Garzik <jgarzik@exmulti•com>
To: Mike Hearn <mike@plan99•net>
Cc: bitcoin-development@lists•sourceforge.net
Subject: Re: [Bitcoin-development] Bumping up against flood control limits again?
Date: Tue, 14 Jun 2011 13:14:21 -0400 [thread overview]
Message-ID: <BANLkTik78Lokrm_09RW5EBkq2mJC3kMzxA@mail.gmail.com> (raw)
In-Reply-To: <BANLkTinuEayfr5S9NEcW+cTLpv2r8iwRgDXYniOAHgQ43zwGqQ@mail.gmail.com>
On Tue, Jun 14, 2011 at 12:44 PM, Mike Hearn <mike@plan99•net> wrote:
> Block sizes have started to get quite large once again. Whilst testing
> chain download today I was disconnected due to going over the 10mb
> flood control limit. Infuriatingly, I can't reproduce this reliably.
> But at 500 blocks an average of 20kb per block will cause this. As we
> can see from the block explorer, the average is probably quite close
> to that.
>
> The flood control seems like a pretty serious scalability limitation.
> I can see a few solutions. One is to raise the limit again. Another is
> to raise the limit and simultaneously lower the batch size. 500 blocks
> in one message means very large messages no matter how big the flood
> control limit is. Going down to 100 or even 50 would hurt chain
> download speed quite a bit in high latency environments, but chain
> download is already a serious bottleneck.
The main goal was not flood control but preventing an internal buffer
memory explosion. We already have the block chain on disk, so in
theory, if we can -eliminate- the outgoing network buffer and simply
use a pointer into the block chain file, we can send as much data as
we want.
HTTP servers certainly don't buffer huge amounts in memory; they would
keel over if so. HTTP servers have been working on the reverse, in
fact: moving the data-pushing over to sendfile(2) syscall and similar
optimizations.
This is an unfortunate relic of how bitcoin P2P code is written. If
the remote side has reduced their TCP window to zero, bitcoin will
still buffer so that it may continue processing other P2P traffic from
other nodes. That makes sense in the case of tiny, 31-byte address
messages -- one must handle the case of a half-sent message before
write(2) refuses additional data -- but not huge block chain download
messages.
The P2P code just wasn't written for huge amounts of streaming data,
and needs some serious thinking... I agree 100% that it is an issue
we will start bumping into, if we haven't already.
--
Jeff Garzik
exMULTI, Inc.
jgarzik@exmulti•com
prev parent reply other threads:[~2011-06-14 17:14 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2011-06-14 16:44 Mike Hearn
2011-06-14 17:07 ` Dave Smith
2011-06-14 17:14 ` Jeff Garzik [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=BANLkTik78Lokrm_09RW5EBkq2mJC3kMzxA@mail.gmail.com \
--to=jgarzik@exmulti$(echo .)com \
--cc=bitcoin-development@lists$(echo .)sourceforge.net \
--cc=mike@plan99$(echo .)net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox