* [Bitcoin-development] Building a node crawler to map network
@ 2011-09-06 7:42 Steve
2011-09-06 8:29 ` Steve
2011-09-06 14:36 ` Rick Wesson
0 siblings, 2 replies; 10+ messages in thread
From: Steve @ 2011-09-06 7:42 UTC (permalink / raw)
To: bitcoin-development
Hi All,
I started messing around today with building a node crawler to try and
map out the bitcoin network and hopefully provide some useful
statistics. It's very basic so far using a mutilated bitcoinj to
connect (due me being java developer and not having a clue with c/c++).
If it's worthwhile I'll hack bitcoinj some more to run on top Netty to
take advantage of it's NIO architecture (netty's been shown to handle
1/2 million concurrent connections so would be ideal for the purpose).
Hoping to a get a bit of input into what would be useful as well as
strategy for getting max possible connections without distorted data. I
seem to recall Gavin talking about the need for some kind of network
health monitoring so I assume there's a need for something like this...
Firstly at the moment basically I'm just storing version message and the
results of getaddr for each node that I can connect to. Is there any
other useful info that can be extracted from a node that's worth collecting?
Second and main issue is how to connect. From my first very basic
probing it seems the very vast majority of nodes don't accept incoming
connections no doubt due to lack of upnp. So it seems the active crawl
approach is not really ideal for the purpose. Even if it was used the
resultant data would be hopelessly distorted.
A honeypot approach would probably be better if there was some way to
make a node 'attractive' to other nodes to connect to. That way it
could capture non-listening nodes as well. If there is some way to
influence other nodes to connect to the crawler node that solves the
problem. If there isn't which I suspect is the case then perhaps
another approach is to build an easy to deploy crawler node that many
volunteers could run and that could then upload collected data to a
central repository.
While I'm asking questions I'll add one more regarding the getaddr
message. It seems most nodes return about 1000 addresses in response to
this message. Obviously most of these nodes haven't actually talked to
all 1000 on the list so where does this list come from? Is it mixture
of addresses obtained from other nodes somehow sorted by timestamp?
Does it include some nodes discovered by IRC/DNS? Or are those only used
to find the first nodes to connect to?
Thanks for any input... Hopefully I can build something that's useful
for the network...
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 7:42 [Bitcoin-development] Building a node crawler to map network Steve
@ 2011-09-06 8:29 ` Steve
2011-09-06 8:36 ` Christian Decker
2011-09-06 14:36 ` Rick Wesson
1 sibling, 1 reply; 10+ messages in thread
From: Steve @ 2011-09-06 8:29 UTC (permalink / raw)
To: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 1288 bytes --]
>
> While I'm asking questions I'll add one more regarding the getaddr
> message.
Talking to myself here. I just sent this message then found this
brilliant set of articles in the Dev & Tech forum which answers the
question very nicely: *https://bitcointalk.org/index.php?topic=41722.0
<https://bitcointalk.org/index.php?board=6.0>
*
Anyway just as an FYI I've been running v0.0.0.0.0.0.0.0.1 for about an
hour. It's only running 10 concurrent connections due to girlfriend
complaining she couldn't watch youtube but here's some early results.
New nodes: 19319 // node address discovered but no contact attempt made yet
Contacted nodes: 754
Uncontactable nodes: 3253
Limbo nodes: 9 //not as exciting as it sounds, just nodes with connect
in progress
Total nodes: 23335 // about 5000 from initial IRC discover, the rest are
from getaddr
Versions: {
300=1,
31900=7,
31902=1,
32000=2,
32001=7,
32002=22,
32100=100,
32200=24,
32300=277,
32400=317,
32500=2}
Fails: {
ConnectException: Connection refused=377,
IOException: Socket is disconnected=87,
SocketException: Network is unreachable=2,
ProtocolException: Error deserializing message =1,
NoRouteToHostException: No route to host=115,
SocketException: Connection reset=149,
SocketTimeoutException: connect timed out=2521}
[-- Attachment #2: Type: text/html, Size: 2019 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 8:29 ` Steve
@ 2011-09-06 8:36 ` Christian Decker
2011-09-06 12:49 ` Mike Hearn
0 siblings, 1 reply; 10+ messages in thread
From: Christian Decker @ 2011-09-06 8:36 UTC (permalink / raw)
To: shadders.del; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 2451 bytes --]
Hi Steve,
before attempting to hack BitcoinJ to use NIO you might want to take a look
at BitDroid (https://github.com/cdecker/BitDroid-Network), which is my
attempt to build an easily extensible network client (no crypto stuff so
far) on top of NIO and a simple publish-subscribe architecture. I build a
crawler like yours with just a single class that subscribes to events
published and closes and opens connections to crawl.
HTH,
Christian
On Tue, Sep 6, 2011 at 10:29 AM, Steve <shadders.del@gmail•com> wrote:
> **
>
>
> While I'm asking questions I'll add one more regarding the getaddr message.
>
>
>
> Talking to myself here. I just sent this message then found this brilliant
> set of articles in the Dev & Tech forum which answers the question very
> nicely: *https://bitcointalk.org/index.php?topic=41722.0<https://bitcointalk.org/index.php?board=6.0>
> *
> Anyway just as an FYI I've been running v0.0.0.0.0.0.0.0.1 for about an
> hour. It's only running 10 concurrent connections due to girlfriend
> complaining she couldn't watch youtube but here's some early results.
>
> New nodes: 19319 // node address discovered but no contact attempt made yet
> Contacted nodes: 754
> Uncontactable nodes: 3253
> Limbo nodes: 9 //not as exciting as it sounds, just nodes with connect in
> progress
> Total nodes: 23335 // about 5000 from initial IRC discover, the rest are
> from getaddr
>
> Versions: {
> 300=1,
> 31900=7,
> 31902=1,
> 32000=2,
> 32001=7,
> 32002=22,
> 32100=100,
> 32200=24,
> 32300=277,
> 32400=317,
> 32500=2}
>
> Fails: {
> ConnectException: Connection refused=377,
> IOException: Socket is disconnected=87,
> SocketException: Network is unreachable=2,
> ProtocolException: Error deserializing message =1,
> NoRouteToHostException: No route to host=115,
> SocketException: Connection reset=149,
> SocketTimeoutException: connect timed out=2521}
>
>
>
>
> ------------------------------------------------------------------------------
> Special Offer -- Download ArcSight Logger for FREE!
> Finally, a world-class log management solution at an even better
> price-free! And you'll get a free "Love Thy Logs" t-shirt when you
> download Logger. Secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsisghtdev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
[-- Attachment #2: Type: text/html, Size: 3565 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 8:36 ` Christian Decker
@ 2011-09-06 12:49 ` Mike Hearn
2011-09-06 13:27 ` Steve
0 siblings, 1 reply; 10+ messages in thread
From: Mike Hearn @ 2011-09-06 12:49 UTC (permalink / raw)
To: Christian Decker; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 239 bytes --]
Actually Steve, take a look at the bitcoinj mailing list today. Somebody has
already built this and has it running. It's accumulating data at the moment,
they'll announce it more widely soon. But I think there's no need to
duplicate work.
[-- Attachment #2: Type: text/html, Size: 255 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 12:49 ` Mike Hearn
@ 2011-09-06 13:27 ` Steve
2011-09-06 13:31 ` Mike Hearn
0 siblings, 1 reply; 10+ messages in thread
From: Steve @ 2011-09-06 13:27 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
Hi Mike,
I've looked but can't find a post like you're talking about. Can you
point me to it?
If so then bollocks... I'm looking for something useful to do atm.
PoolServerJ is in a holding pattern atm as I've stabilisied all the bugs
I know about and am waiting for several pools to finish testing and move
into production so I'm twiddling thumbs trying to figure out how to
spend my time.
On 06/09/11 22:49, Mike Hearn wrote:
> Actually Steve, take a look at the bitcoinj mailing list today.
> Somebody has already built this and has it running. It's accumulating
> data at the moment, they'll announce it more widely soon. But I think
> there's no need to duplicate work.
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 13:27 ` Steve
@ 2011-09-06 13:31 ` Mike Hearn
2011-09-06 14:17 ` Steve
0 siblings, 1 reply; 10+ messages in thread
From: Mike Hearn @ 2011-09-06 13:31 UTC (permalink / raw)
To: shadders.del; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 1183 bytes --]
>
> I've looked but can't find a post like you're talking about. Can you point
> me to it?
>
https://groups.google.com/forum/?pli=1#!topic/bitcoinj/LSlZdUWcCdk
> If so then bollocks... I'm looking for something useful to do atm.
> PoolServerJ is in a holding pattern atm as I've stabilisied all the bugs I
> know about and am waiting for several pools to finish testing and move into
> production so I'm twiddling thumbs trying to figure out how to spend my
> time.
>
Patches to BitCoinJ are always welcome :-)
If you'd rather do your own thing, you could experiment with writing a proxy
that sits in front of bitcoind and multiplexes connections. Gavin is
concerned about socket exhaustion as users move to lightweight clients.
Multiplexing proxies are a battle-tested technique for reducing the strain
of this type of thing. BitCoinJ uses thread-per-connection so wouldn't do a
good job of that right now, but allowing it to use a mix of async io and
multi-threading would be a nice improvement. It'd need some changes to
bitcoind as well for a really good effort, to allow for IPs to be forwarded.
I'm happy to discuss it more with you over on the bitcoinj list if wanted.
[-- Attachment #2: Type: text/html, Size: 1686 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 13:31 ` Mike Hearn
@ 2011-09-06 14:17 ` Steve
2011-09-06 14:52 ` Mike Hearn
0 siblings, 1 reply; 10+ messages in thread
From: Steve @ 2011-09-06 14:17 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 2345 bytes --]
Hi Mike,
I expect I'll be submitting patches for bitcoinj sometime in the future
but I'm not really across it yet to the point where I'd be confident
submitting patches right now...
This proxy sound like a good match for what I've been up to lately
though so long as it wouldn't involve direct changes to bitcoind on my
part. My c/c++ skills are non-existent.
However I have been building a pool protocol using protobufs and netty
for non-blocking IO and I'd imagine the kind of multiplexing proxy
you're talking about could be easily implemented using netty.
I'm not really understanding the use case though. I believe most
bitcoind's have a default max connections of 8. Is the goal to increase
this without fundamentally altering the bitcoind concurrency model? Or
is it to provide capactity for a more hub/client oriented network? If
the latter then presumably this is functionality that should ideally be
native to the client in the long term in the form of NIO?
On 06/09/11 23:31, Mike Hearn wrote:
>
> I've looked but can't find a post like you're talking about. Can
> you point me to it?
>
> https://groups.google.com/forum/?pli=1#!topic/bitcoinj/LSlZdUWcCdk
> <https://groups.google.com/forum/?pli=1#%21topic/bitcoinj/LSlZdUWcCdk>
>
> If so then bollocks... I'm looking for something useful to do atm.
> PoolServerJ is in a holding pattern atm as I've stabilisied all
> the bugs I know about and am waiting for several pools to finish
> testing and move into production so I'm twiddling thumbs trying to
> figure out how to spend my time.
>
>
> Patches to BitCoinJ are always welcome :-)
>
> If you'd rather do your own thing, you could experiment with writing a
> proxy that sits in front of bitcoind and multiplexes connections.
> Gavin is concerned about socket exhaustion as users move to
> lightweight clients. Multiplexing proxies are a battle-tested
> technique for reducing the strain of this type of thing. BitCoinJ uses
> thread-per-connection so wouldn't do a good job of that right now, but
> allowing it to use a mix of async io and multi-threading would be a
> nice improvement. It'd need some changes to bitcoind as well for a
> really good effort, to allow for IPs to be forwarded. I'm happy to
> discuss it more with you over on the bitcoinj list if wanted.
[-- Attachment #2: Type: text/html, Size: 3612 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 7:42 [Bitcoin-development] Building a node crawler to map network Steve
2011-09-06 8:29 ` Steve
@ 2011-09-06 14:36 ` Rick Wesson
1 sibling, 0 replies; 10+ messages in thread
From: Rick Wesson @ 2011-09-06 14:36 UTC (permalink / raw)
To: shadders.del; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 3330 bytes --]
I've got minna patches for nio based on bitcoinj. I've enumerated the
network a few times and am working on a DNS seed service as well as some
weather reports.
Happy to start a branch when the committers are ready.
-rick
On Tue, Sep 6, 2011 at 12:42 AM, Steve <shadders.del@gmail•com> wrote:
> Hi All,
>
> I started messing around today with building a node crawler to try and
> map out the bitcoin network and hopefully provide some useful
> statistics. It's very basic so far using a mutilated bitcoinj to
> connect (due me being java developer and not having a clue with c/c++).
> If it's worthwhile I'll hack bitcoinj some more to run on top Netty to
> take advantage of it's NIO architecture (netty's been shown to handle
> 1/2 million concurrent connections so would be ideal for the purpose).
>
> Hoping to a get a bit of input into what would be useful as well as
> strategy for getting max possible connections without distorted data. I
> seem to recall Gavin talking about the need for some kind of network
> health monitoring so I assume there's a need for something like this...
>
> Firstly at the moment basically I'm just storing version message and the
> results of getaddr for each node that I can connect to. Is there any
> other useful info that can be extracted from a node that's worth
> collecting?
>
> Second and main issue is how to connect. From my first very basic
> probing it seems the very vast majority of nodes don't accept incoming
> connections no doubt due to lack of upnp. So it seems the active crawl
> approach is not really ideal for the purpose. Even if it was used the
> resultant data would be hopelessly distorted.
>
> A honeypot approach would probably be better if there was some way to
> make a node 'attractive' to other nodes to connect to. That way it
> could capture non-listening nodes as well. If there is some way to
> influence other nodes to connect to the crawler node that solves the
> problem. If there isn't which I suspect is the case then perhaps
> another approach is to build an easy to deploy crawler node that many
> volunteers could run and that could then upload collected data to a
> central repository.
>
> While I'm asking questions I'll add one more regarding the getaddr
> message. It seems most nodes return about 1000 addresses in response to
> this message. Obviously most of these nodes haven't actually talked to
> all 1000 on the list so where does this list come from? Is it mixture
> of addresses obtained from other nodes somehow sorted by timestamp?
> Does it include some nodes discovered by IRC/DNS? Or are those only used
> to find the first nodes to connect to?
>
> Thanks for any input... Hopefully I can build something that's useful
> for the network...
>
>
> ------------------------------------------------------------------------------
> Special Offer -- Download ArcSight Logger for FREE!
> Finally, a world-class log management solution at an even better
> price-free! And you'll get a free "Love Thy Logs" t-shirt when you
> download Logger. Secure your free ArcSight Logger TODAY!
> http://p.sf.net/sfu/arcsisghtdev2dev
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 4105 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 14:17 ` Steve
@ 2011-09-06 14:52 ` Mike Hearn
2011-09-06 15:25 ` Steve
0 siblings, 1 reply; 10+ messages in thread
From: Mike Hearn @ 2011-09-06 14:52 UTC (permalink / raw)
To: shadders.del; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 2549 bytes --]
On Tue, Sep 6, 2011 at 4:17 PM, Steve <shadders.del@gmail•com> wrote:
> **
> I'm not really understanding the use case though. I believe most
> bitcoind's have a default max connections of 8. Is the goal to increase
> this without fundamentally altering the bitcoind concurrency model?
>
bitcoind already uses asynchronous IO. That's not the problem.
The issue came up in a conversation about scalability. If Bitcoins
popularity continues to grow, users are very likely to migrate away from
running full verifying nodes to lightweight clients, either a different mode
of the Satoshi client or different implementations like the Android Wallet
or MultiBit.
Lightweight clients cannot verify thus should not relay. And they'll be run
by users who just want to send/receive coins from time to time, so don't
leave the programs running 24/7. The result could be running out of sockets
(like we have had problems with recently). It's especially true because
lightweight clients cannot check transactions for themselves. If they want
to show transactions appearing immediately (and they do), they have to use
"heard from lots of nodes" as a proxy for validity. So lightweight clients
are likely to be socket intensive.
We could solve this by just hoping that lots of people run full nodes. The
problem is that a full node is quite an intensive thing already, it uses
lots of CPU and disk seeks, and will just get more expensive in future. And
as transaction traffic increases, that leaves less CPU time available to
service thousands of connected clients. The ROI of bringing up a new node
decreases at the same time as the userbase increases.
One traditional approach to solving this is frontend proxies. Jabber.com/org
used this technique many years ago, and Google has also used it to scale up the
lockservice<http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf>
(see
section 3.1). It's effective because often maintaining connections to
thousands of clients doesn't involve much brainwork, just shifting bytes
around. This is especially true of Bitcoin. So if somebody is running a full
node already they could increase their client capacity by just bringing up a
frontend proxy and having it handle things like outbound tx
broadcasts/deduping inbound broadcasts, connection setup, relaying recently
found blocks etc. A well written proxy could probably support tens of
thousands of simultaneous clients which frees up the bitcoinds time for
verification and wallet manipulation.
[-- Attachment #2: Type: text/html, Size: 3033 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [Bitcoin-development] Building a node crawler to map network
2011-09-06 14:52 ` Mike Hearn
@ 2011-09-06 15:25 ` Steve
0 siblings, 0 replies; 10+ messages in thread
From: Steve @ 2011-09-06 15:25 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 3120 bytes --]
Thanks for the overview Mike. I just bailed up Gavin on IRC and between
that convo and what you've just written I'm starting to picture a plan
in my head... This sounds right up my alley, I wish I didn't have to go
to bed right now as I've got a ton of ideas buzzing around I'd like to
get started on right now. But I'll be onto it as soon as I've got a
free moment...
On 07/09/11 00:52, Mike Hearn wrote:
> On Tue, Sep 6, 2011 at 4:17 PM, Steve <shadders.del@gmail•com
> <mailto:shadders.del@gmail•com>> wrote:
>
> I'm not really understanding the use case though. I believe most
> bitcoind's have a default max connections of 8. Is the goal to
> increase this without fundamentally altering the bitcoind
> concurrency model?
>
>
> bitcoind already uses asynchronous IO. That's not the problem.
>
> The issue came up in a conversation about scalability. If Bitcoins
> popularity continues to grow, users are very likely to migrate away
> from running full verifying nodes to lightweight clients, either a
> different mode of the Satoshi client or different implementations like
> the Android Wallet or MultiBit.
>
> Lightweight clients cannot verify thus should not relay. And they'll
> be run by users who just want to send/receive coins from time to time,
> so don't leave the programs running 24/7. The result could be running
> out of sockets (like we have had problems with recently). It's
> especially true because lightweight clients cannot check transactions
> for themselves. If they want to show transactions appearing
> immediately (and they do), they have to use "heard from lots of nodes"
> as a proxy for validity. So lightweight clients are likely to be
> socket intensive.
>
> We could solve this by just hoping that lots of people run full nodes.
> The problem is that a full node is quite an intensive thing already,
> it uses lots of CPU and disk seeks, and will just get more expensive
> in future. And as transaction traffic increases, that leaves less CPU
> time available to service thousands of connected clients. The ROI of
> bringing up a new node decreases at the same time as the userbase
> increases.
>
> One traditional approach to solving this is frontend proxies.
> Jabber.com/org used this technique many years ago, and Google has also
> used it to scale up the lockservice
> <http://static.googleusercontent.com/external_content/untrusted_dlcp/labs.google.com/en/us/papers/chubby-osdi06.pdf> (see
> section 3.1). It's effective because often maintaining connections to
> thousands of clients doesn't involve much brainwork, just shifting
> bytes around. This is especially true of Bitcoin. So if somebody is
> running a full node already they could increase their client capacity
> by just bringing up a frontend proxy and having it handle things like
> outbound tx broadcasts/deduping inbound broadcasts, connection setup,
> relaying recently found blocks etc. A well written proxy could
> probably support tens of thousands of simultaneous clients which frees
> up the bitcoinds time for verification and wallet manipulation.
[-- Attachment #2: Type: text/html, Size: 4453 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2011-09-06 15:25 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2011-09-06 7:42 [Bitcoin-development] Building a node crawler to map network Steve
2011-09-06 8:29 ` Steve
2011-09-06 8:36 ` Christian Decker
2011-09-06 12:49 ` Mike Hearn
2011-09-06 13:27 ` Steve
2011-09-06 13:31 ` Mike Hearn
2011-09-06 14:17 ` Steve
2011-09-06 14:52 ` Mike Hearn
2011-09-06 15:25 ` Steve
2011-09-06 14:36 ` Rick Wesson
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox