* [Bitcoin-development] Announcing the Statoshi fork
@ 2014-05-07 19:12 Jameson Lopp
2014-05-07 19:46 ` Wladimir
` (2 more replies)
0 siblings, 3 replies; 19+ messages in thread
From: Jameson Lopp @ 2014-05-07 19:12 UTC (permalink / raw)
To: bitcoin-development
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful.
http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/
https://jlopp.github.io/statoshi/
Feedback is appreciated!
- - Jameson
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTaoWSAAoJEIch3FSFNiDclLoH/0CXPTum6B2cfoNsacihHuS9
9wt50sOgghttS3J/kloP315ijY7p2HSmhvqL2G/DWYh5Vx0f6gTaUAokQ8H6x4EV
3/pdZG+9a6eegpCtgr+IgphgPSEufzct/Mp7pKTAbH0G61toOM5ZfIgdL2X/2tpx
4TjOmjhZRHuglzsM9934EjezIsR7l2vaRQB0r1LPGSgWmDSKTTb2uK7xvD1zg0tz
QXb0hl7A6rw1xZwmw3i+PujshJCbjVh8QrFT55GYi05yYdBsS6BAG46F5D8Uvn+M
FnCBLdRfWrTzQXIxoLrtBmM1JXOJKdMhmG0p2mwzwXEGR7MR2suS/+Bb7iHJTpA=
=iNNR
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 19:12 [Bitcoin-development] Announcing the Statoshi fork Jameson Lopp
@ 2014-05-07 19:46 ` Wladimir
2014-05-07 19:57 ` Jameson Lopp
2014-05-07 19:50 ` Mike Hearn
2014-05-07 19:55 ` Pavol Rusnak
2 siblings, 1 reply; 19+ messages in thread
From: Wladimir @ 2014-05-07 19:46 UTC (permalink / raw)
To: Jameson Lopp; +Cc: bitcoin-development
On Wed, May 7, 2014 at 9:12 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful.
>
> http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/
>
> https://jlopp.github.io/statoshi/
>
> Feedback is appreciated!
Ooh nice graphs!
We were coincidentally talking about showing stats from a node on a
local web site on the #bitcoin-dev IRC a few days ago.
At some point, if we're going to offer Bitcoin Core node-only
installers, it'd be nice to include something like this so the user
can keep an eye on their node(s).
Wladimir
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 19:12 [Bitcoin-development] Announcing the Statoshi fork Jameson Lopp
2014-05-07 19:46 ` Wladimir
@ 2014-05-07 19:50 ` Mike Hearn
2014-05-07 20:00 ` Jameson Lopp
2014-05-07 19:55 ` Pavol Rusnak
2 siblings, 1 reply; 19+ messages in thread
From: Mike Hearn @ 2014-05-07 19:50 UTC (permalink / raw)
To: Jameson Lopp; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 695 bytes --]
Really nice! We definitely need to put together a team who really cares
about the operations side of the network and this is a fantastic start.
It'd be nice if you didn't assume knowledge of what statsd is out of the
box. Given the name I'd assumed it was a small UNIX daemon but it seems
it's actually a Javascript thingy?
It looks like putting together a monitored bitcoind setup can be quite a
lot of work. I wonder if there are ways to simplify it. For example, would
it make sense for someone to run a community statsd and graphite instance,
so we can get aggregate statistics across many nodes and the node operators
don't have to set everything up themselves? Does that make any sense?
[-- Attachment #2: Type: text/html, Size: 804 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 19:12 [Bitcoin-development] Announcing the Statoshi fork Jameson Lopp
2014-05-07 19:46 ` Wladimir
2014-05-07 19:50 ` Mike Hearn
@ 2014-05-07 19:55 ` Pavol Rusnak
2 siblings, 0 replies; 19+ messages in thread
From: Pavol Rusnak @ 2014-05-07 19:55 UTC (permalink / raw)
To: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 350 bytes --]
On 05/07/2014 09:12 PM, Jameson Lopp wrote:
> I've created a Bitcoin Core fork that outputs statistics to StatsD.
How complex is the patchset? Would it be possible to merge it into the
mainline and enable compilation of this feature conditionally by some
build-time option?
--
Best Regards / S pozdravom,
Pavol Rusnak <stick@gk2•sk>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 19:46 ` Wladimir
@ 2014-05-07 19:57 ` Jameson Lopp
2014-05-07 20:18 ` Wladimir
0 siblings, 1 reply; 19+ messages in thread
From: Jameson Lopp @ 2014-05-07 19:57 UTC (permalink / raw)
To: Wladimir; +Cc: bitcoin-development
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
The next logical step may be for us to offer a public instance of these graphs; I'd be happy to work with you to set one up.
I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer?
- - Jameson
On 05/07/2014 03:46 PM, Wladimir wrote:
> On Wed, May 7, 2014 at 9:12 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful.
>>
>> http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/
>>
>> https://jlopp.github.io/statoshi/
>>
>> Feedback is appreciated!
>
> Ooh nice graphs!
>
> We were coincidentally talking about showing stats from a node on a
> local web site on the #bitcoin-dev IRC a few days ago.
>
> At some point, if we're going to offer Bitcoin Core node-only
> installers, it'd be nice to include something like this so the user
> can keep an eye on their node(s).
>
> Wladimir
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTapAsAAoJEIch3FSFNiDcyMEIAIds0yo9zeWcqNqGZ+UNltoH
hNt8NhYOgL/6WNeLVYdRmCrrdNn/KMSLcAZmOQ0U+W/qL3xh1RB59o3BcBnW05Yr
ZxY5ajKKq+oz70ShMNUkVnzFSStMhH9fKnolrF0mgSx4CU9e0YTx/LBc/u9ulypO
QNZydiiegwvTFjMxHItgU5xo/wzySazmyxN9x3Gls98vDfSjE3Rt/DTqAwHleD3t
SlIu4RU2iPpAW/6MgfWqAw+CrbZ2NNKp7a7+0gsUlbdDP1h6WEvoae5sUzRmvLB3
rHMmRoRvTl4Hl1bG7CKyM4D3piBkpDf/nMqnAAFNYkocS5xVpHM1WrTMDAmkSLk=
=/TPp
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 19:50 ` Mike Hearn
@ 2014-05-07 20:00 ` Jameson Lopp
2014-05-07 20:12 ` Mike Hearn
0 siblings, 1 reply; 19+ messages in thread
From: Jameson Lopp @ 2014-05-07 20:00 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
I completely agree that this setup is far too difficult to reasonably expect anyone to implement it. You're correct that we could run a single StatsD daemon and have quite a few nodes sending statistics to it - this is really what StatsD was designed for - sampling small amounts of stats from high volume systems. There would be an issue of trust, however - StatsD was also only really designed to be run inside of highly secure infrastructure where you trust all of the machines that are talking to it.
- - Jameson
On 05/07/2014 03:50 PM, Mike Hearn wrote:
> Really nice! We definitely need to put together a team who really cares
> about the operations side of the network and this is a fantastic start.
>
> It'd be nice if you didn't assume knowledge of what statsd is out of the
> box. Given the name I'd assumed it was a small UNIX daemon but it seems
> it's actually a Javascript thingy?
>
> It looks like putting together a monitored bitcoind setup can be quite a
> lot of work. I wonder if there are ways to simplify it. For example, would
> it make sense for someone to run a community statsd and graphite instance,
> so we can get aggregate statistics across many nodes and the node operators
> don't have to set everything up themselves? Does that make any sense?
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTapDsAAoJEIch3FSFNiDcqNMIAKbliLXVJ+CovCB+etqwCYfD
oTs0uxEIZvnGSzqn6i3z4tgzF7kbnpVThiHpG2MO2BytasGzfr+FsDLFpvkkvx0Q
yffa1J5uB5QhY/POwg6GCN4DEmy8tbd+I5rSTKg1m/JuXZV2B4TFgr6GF+t+gBNT
O2TTIngcl77XbGDF2XKOZSAHaIFx2KjdZNg5lfInVxDeA1dDzlU/vmEB37fx80EH
IzD7FTllLp5qhlwYxaQTVxXmbjy+pXbtArYsVSRgoscgJI3DDKpxQj3uzjxouEHv
27QkhuS2dzFioQhLPcnz/PtcCkE1l0aDNf6n7OiJ3lGygQ0+AKZHpyuvFujpSY8=
=QCYI
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 20:00 ` Jameson Lopp
@ 2014-05-07 20:12 ` Mike Hearn
0 siblings, 0 replies; 19+ messages in thread
From: Mike Hearn @ 2014-05-07 20:12 UTC (permalink / raw)
To: Jameson Lopp; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 422 bytes --]
It looks like the packet format statsd expects is rather simple - it should
be easy to experiment with.
Perhaps a good next step would be to improve your patch so that someone can
get a feed of the stats packets via TCP by e.g. ssh tunnelling to their
host. Once it's easy to get a feed of simple stats packets, people can
easily experiment with different kinds of aggregation and monitoring
software from their desktop.
[-- Attachment #2: Type: text/html, Size: 615 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 19:57 ` Jameson Lopp
@ 2014-05-07 20:18 ` Wladimir
2014-05-07 20:25 ` Mike Hearn
` (2 more replies)
0 siblings, 3 replies; 19+ messages in thread
From: Wladimir @ 2014-05-07 20:18 UTC (permalink / raw)
To: Jameson Lopp; +Cc: bitcoin-development
On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer?
Depends on just how much stuff it is. The idea is primarily to have an
installer for running a (wallet-less) node as an OS background
service.
Having some statistics available would be worth some extra download
size, otherwise it would be pretty much invisible.
We'd already decided that we would need something like Python for the
stats service. Implementing things like web services in C++ is just
not realistic given the time constraints and the great already-written
code that is out there. As an optional tool it should be external, not
part of bitcoind itself.
I suppose the chart rendering happens client-side? In that case the
web service just has to collect and provide the data, and serve static
html/js files.
Wladimir
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 20:18 ` Wladimir
@ 2014-05-07 20:25 ` Mike Hearn
2014-05-07 20:31 ` Charlie 'Charles' Shrem
2014-05-07 21:07 ` Wladimir
2014-05-07 20:28 ` Nelson Castillo
2014-05-07 20:35 ` Jameson Lopp
2 siblings, 2 replies; 19+ messages in thread
From: Mike Hearn @ 2014-05-07 20:25 UTC (permalink / raw)
To: Wladimir; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 3705 bytes --]
I think there a few different possible ways to go here.
One is to try and simplify the setup of all the components so it all gets
installed together. That might be feasible in some quite restricted setups
but the installation instructions for Graphite look kind of terrifying.
Another is to export stats over regular TCP and make them public so
literally anyone can listen to the stats feed for any node. Then people who
dig stats and graphs could work on stats aggregators that give global
network visibility independently, effectively crawling the p2p network for
data. It'd have the advantage of having zero setup for the node operators
and not require much in the way of resources.
For what it's worth, although the environment is a bit different inside
Google the latter approach is used. Monitoring servers locate servers of
interest via a discovery service, connect to them and start streaming stats
data into a database service that can then be queried later to get graphs.
The stats are also run through various rules to obtain alerts about
problematic conditions. For example, if a subset of the network splits it
might be hard to notice that if the node operators aren't paying attention
and Matt's fork alert/emailing code isn't set up. But if there was a site
crawling nodes and aggregating chain heights by version, that could trigger
an alert to people who *are* paying attention.
I know from practical experience that monitoring and analysis tends to
appeal more to certain types of people than others. So I quite like the
"let anyone monitor" approach. However, it may not be appropriate in a P2P
network, I did not think about it much.
Obviously I'm assuming none of the stats expose privacy sensitive data.
On Wed, May 7, 2014 at 10:18 PM, Wladimir <laanwj@gmail•com> wrote:
> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com>
> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > I agree that it would be awesome to offer these types of stats with the
> installer; unfortunately the route I've taken has dependencies on several
> other other pieces of software to do all the heavy lifting of stats
> aggregation and chart rendering. I'm assuming that you would not want to
> build any of that processing into Bitcoin Core itself; would you be opposed
> to packaging other software along with the installer?
>
> Depends on just how much stuff it is. The idea is primarily to have an
> installer for running a (wallet-less) node as an OS background
> service.
>
> Having some statistics available would be worth some extra download
> size, otherwise it would be pretty much invisible.
>
> We'd already decided that we would need something like Python for the
> stats service. Implementing things like web services in C++ is just
> not realistic given the time constraints and the great already-written
> code that is out there. As an optional tool it should be external, not
> part of bitcoind itself.
>
> I suppose the chart rendering happens client-side? In that case the
> web service just has to collect and provide the data, and serve static
> html/js files.
>
> Wladimir
>
>
> ------------------------------------------------------------------------------
> Is your legacy SCM system holding you back? Join Perforce May 7 to find
> out:
> • 3 signs your SCM is hindering your productivity
> • Requirements for releasing software faster
> • Expert tips and advice for migrating your SCM now
> http://p.sf.net/sfu/perforce
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 4756 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 20:18 ` Wladimir
2014-05-07 20:25 ` Mike Hearn
@ 2014-05-07 20:28 ` Nelson Castillo
2014-05-08 11:22 ` Wladimir
2014-05-07 20:35 ` Jameson Lopp
2 siblings, 1 reply; 19+ messages in thread
From: Nelson Castillo @ 2014-05-07 20:28 UTC (permalink / raw)
To: Wladimir; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 2071 bytes --]
On Wed, May 7, 2014 at 3:18 PM, Wladimir <laanwj@gmail•com> wrote:
> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com>
> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > I agree that it would be awesome to offer these types of stats with the
> installer; unfortunately the route I've taken has dependencies on several
> other other pieces of software to do all the heavy lifting of stats
> aggregation and chart rendering. I'm assuming that you would not want to
> build any of that processing into Bitcoin Core itself; would you be opposed
> to packaging other software along with the installer?
>
> Depends on just how much stuff it is. The idea is primarily to have an
> installer for running a (wallet-less) node as an OS background
> service.
>
> Having some statistics available would be worth some extra download
> size, otherwise it would be pretty much invisible.
>
> We'd already decided that we would need something like Python for the
> stats service. Implementing things like web services in C++ is just
> not realistic given the time constraints and the great already-written
> code that is out there. As an optional tool it should be external, not
> part of bitcoind itself.
>
> I suppose the chart rendering happens client-side? In that case the
> web service just has to collect and provide the data, and serve static
> html/js files.
>
Is SNMP an option? That way you do not need to implement clients and there
are many tools written.
> Wladimir
>
>
> ------------------------------------------------------------------------------
> Is your legacy SCM system holding you back? Join Perforce May 7 to find
> out:
> • 3 signs your SCM is hindering your productivity
> • Requirements for releasing software faster
> • Expert tips and advice for migrating your SCM now
> http://p.sf.net/sfu/perforce
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 3065 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 20:25 ` Mike Hearn
@ 2014-05-07 20:31 ` Charlie 'Charles' Shrem
2014-05-07 21:07 ` Wladimir
1 sibling, 0 replies; 19+ messages in thread
From: Charlie 'Charles' Shrem @ 2014-05-07 20:31 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 4680 bytes --]
I'm going to install this now on my full node, looks really cool!
This is my node page: http://199.58.210.124/
Thanks,
Charlie
CharlieShrem.com | *Please **encrypt messages with my PGP key
<http://charlieshrem.com/contact/>*
On Wed, May 7, 2014 at 4:25 PM, Mike Hearn <mike@plan99•net> wrote:
> I think there a few different possible ways to go here.
>
> One is to try and simplify the setup of all the components so it all gets
> installed together. That might be feasible in some quite restricted setups
> but the installation instructions for Graphite look kind of terrifying.
>
> Another is to export stats over regular TCP and make them public so
> literally anyone can listen to the stats feed for any node. Then people who
> dig stats and graphs could work on stats aggregators that give global
> network visibility independently, effectively crawling the p2p network for
> data. It'd have the advantage of having zero setup for the node operators
> and not require much in the way of resources.
>
> For what it's worth, although the environment is a bit different inside
> Google the latter approach is used. Monitoring servers locate servers of
> interest via a discovery service, connect to them and start streaming stats
> data into a database service that can then be queried later to get graphs.
>
> The stats are also run through various rules to obtain alerts about
> problematic conditions. For example, if a subset of the network splits it
> might be hard to notice that if the node operators aren't paying attention
> and Matt's fork alert/emailing code isn't set up. But if there was a site
> crawling nodes and aggregating chain heights by version, that could trigger
> an alert to people who *are* paying attention.
>
> I know from practical experience that monitoring and analysis tends to
> appeal more to certain types of people than others. So I quite like the
> "let anyone monitor" approach. However, it may not be appropriate in a P2P
> network, I did not think about it much.
>
> Obviously I'm assuming none of the stats expose privacy sensitive data.
>
>
>
> On Wed, May 7, 2014 at 10:18 PM, Wladimir <laanwj@gmail•com> wrote:
>
>> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com>
>> wrote:
>> > -----BEGIN PGP SIGNED MESSAGE-----
>> > Hash: SHA1
>> >
>> > I agree that it would be awesome to offer these types of stats with the
>> installer; unfortunately the route I've taken has dependencies on several
>> other other pieces of software to do all the heavy lifting of stats
>> aggregation and chart rendering. I'm assuming that you would not want to
>> build any of that processing into Bitcoin Core itself; would you be opposed
>> to packaging other software along with the installer?
>>
>> Depends on just how much stuff it is. The idea is primarily to have an
>> installer for running a (wallet-less) node as an OS background
>> service.
>>
>> Having some statistics available would be worth some extra download
>> size, otherwise it would be pretty much invisible.
>>
>> We'd already decided that we would need something like Python for the
>> stats service. Implementing things like web services in C++ is just
>> not realistic given the time constraints and the great already-written
>> code that is out there. As an optional tool it should be external, not
>> part of bitcoind itself.
>>
>> I suppose the chart rendering happens client-side? In that case the
>> web service just has to collect and provide the data, and serve static
>> html/js files.
>>
>> Wladimir
>>
>>
>> ------------------------------------------------------------------------------
>> Is your legacy SCM system holding you back? Join Perforce May 7 to find
>> out:
>> • 3 signs your SCM is hindering your productivity
>> • Requirements for releasing software faster
>> • Expert tips and advice for migrating your SCM now
>> http://p.sf.net/sfu/perforce
>> _______________________________________________
>> Bitcoin-development mailing list
>> Bitcoin-development@lists•sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>>
>
>
>
> ------------------------------------------------------------------------------
> Is your legacy SCM system holding you back? Join Perforce May 7 to find
> out:
> • 3 signs your SCM is hindering your productivity
> • Requirements for releasing software faster
> • Expert tips and advice for migrating your SCM now
> http://p.sf.net/sfu/perforce
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
>
[-- Attachment #2: Type: text/html, Size: 7904 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 20:18 ` Wladimir
2014-05-07 20:25 ` Mike Hearn
2014-05-07 20:28 ` Nelson Castillo
@ 2014-05-07 20:35 ` Jameson Lopp
2014-05-07 21:04 ` Charlie 'Charles' Shrem
2 siblings, 1 reply; 19+ messages in thread
From: Jameson Lopp @ 2014-05-07 20:35 UTC (permalink / raw)
To: Wladimir; +Cc: bitcoin-development
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
The charts are generated on-demand by Graphite, which is a Django app.
I will note that one reason I chose StatsD is because it sends the stats via UDP rather than TCP, which is a non-blocking operation. I didn't want the sending of stats to affect the node's performance.
- - Jameson
On 05/07/2014 04:18 PM, Wladimir wrote:
> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer?
>
> Depends on just how much stuff it is. The idea is primarily to have an
> installer for running a (wallet-less) node as an OS background
> service.
>
> Having some statistics available would be worth some extra download
> size, otherwise it would be pretty much invisible.
>
> We'd already decided that we would need something like Python for the
> stats service. Implementing things like web services in C++ is just
> not realistic given the time constraints and the great already-written
> code that is out there. As an optional tool it should be external, not
> part of bitcoind itself.
>
> I suppose the chart rendering happens client-side? In that case the
> web service just has to collect and provide the data, and serve static
> html/js files.
>
> Wladimir
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTapj5AAoJEIch3FSFNiDcofAIALHi7XgQi8pf75btujaeBsX3
nniRD0yZIkoAvPlvFLiKQGE8TH+VR8Sb9fQACzmajYx1yjD0gN4xvkJXbI+pkeP5
L8ZryhqxL5qCh/OI4+fkWlsp5Nwx89QvUepdXXdc/AQGQJIEMceiZOLDcjbk29Yb
vCsyJL5yhzM9BM0cImuvrOBPtF3/L6DbgHP8OLD2LHRl4paJ1UDtfYCx3HVO9wp8
ZWq1oCaFyoYmUyx8GTUzbLjh9sOgaq43GKYec/kQSLmFxhhMF0dGNDMiwD/xz1i7
LIswjlEKHZYOWWL3SMQg3pLlOTzGH4mHg++BAyrtzZ5CHlc1rjsPSk2d2Df/8Zc=
=GFu9
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 20:35 ` Jameson Lopp
@ 2014-05-07 21:04 ` Charlie 'Charles' Shrem
0 siblings, 0 replies; 19+ messages in thread
From: Charlie 'Charles' Shrem @ 2014-05-07 21:04 UTC (permalink / raw)
To: Jameson Lopp; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 3271 bytes --]
For those who are interested, here is my 9 step way to install a full node.
I tried to make it as universal as possible.
http://charlieshrem.com/node/
Thanks,
Charlie
CharlieShrem.com | *Please **encrypt messages with my PGP key
<http://charlieshrem.com/contact/>*
On Wed, May 7, 2014 at 4:35 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> The charts are generated on-demand by Graphite, which is a Django app.
>
> I will note that one reason I chose StatsD is because it sends the stats
> via UDP rather than TCP, which is a non-blocking operation. I didn't want
> the sending of stats to affect the node's performance.
>
> - - Jameson
>
> On 05/07/2014 04:18 PM, Wladimir wrote:
> > On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com>
> wrote:
> >> -----BEGIN PGP SIGNED MESSAGE-----
> >> Hash: SHA1
> >>
> >> I agree that it would be awesome to offer these types of stats with the
> installer; unfortunately the route I've taken has dependencies on several
> other other pieces of software to do all the heavy lifting of stats
> aggregation and chart rendering. I'm assuming that you would not want to
> build any of that processing into Bitcoin Core itself; would you be opposed
> to packaging other software along with the installer?
> >
> > Depends on just how much stuff it is. The idea is primarily to have an
> > installer for running a (wallet-less) node as an OS background
> > service.
> >
> > Having some statistics available would be worth some extra download
> > size, otherwise it would be pretty much invisible.
> >
> > We'd already decided that we would need something like Python for the
> > stats service. Implementing things like web services in C++ is just
> > not realistic given the time constraints and the great already-written
> > code that is out there. As an optional tool it should be external, not
> > part of bitcoind itself.
> >
> > I suppose the chart rendering happens client-side? In that case the
> > web service just has to collect and provide the data, and serve static
> > html/js files.
> >
> > Wladimir
> >
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.14 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJTapj5AAoJEIch3FSFNiDcofAIALHi7XgQi8pf75btujaeBsX3
> nniRD0yZIkoAvPlvFLiKQGE8TH+VR8Sb9fQACzmajYx1yjD0gN4xvkJXbI+pkeP5
> L8ZryhqxL5qCh/OI4+fkWlsp5Nwx89QvUepdXXdc/AQGQJIEMceiZOLDcjbk29Yb
> vCsyJL5yhzM9BM0cImuvrOBPtF3/L6DbgHP8OLD2LHRl4paJ1UDtfYCx3HVO9wp8
> ZWq1oCaFyoYmUyx8GTUzbLjh9sOgaq43GKYec/kQSLmFxhhMF0dGNDMiwD/xz1i7
> LIswjlEKHZYOWWL3SMQg3pLlOTzGH4mHg++BAyrtzZ5CHlc1rjsPSk2d2Df/8Zc=
> =GFu9
> -----END PGP SIGNATURE-----
>
>
> ------------------------------------------------------------------------------
> Is your legacy SCM system holding you back? Join Perforce May 7 to find
> out:
> • 3 signs your SCM is hindering your productivity
> • Requirements for releasing software faster
> • Expert tips and advice for migrating your SCM now
> http://p.sf.net/sfu/perforce
> _______________________________________________
> Bitcoin-development mailing list
> Bitcoin-development@lists•sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bitcoin-development
>
[-- Attachment #2: Type: text/html, Size: 5690 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 20:25 ` Mike Hearn
2014-05-07 20:31 ` Charlie 'Charles' Shrem
@ 2014-05-07 21:07 ` Wladimir
2014-05-08 0:47 ` Gregory Maxwell
1 sibling, 1 reply; 19+ messages in thread
From: Wladimir @ 2014-05-07 21:07 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
On Wed, May 7, 2014 at 10:25 PM, Mike Hearn <mike@plan99•net> wrote:
> Another is to export stats over regular TCP and make them public so
> literally anyone can listen to the stats feed for any node.
TOR does this as well: http://torstatus.blutmagie.de/
No idea what they use to submit/gather the statistics.
Wladimir
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 21:07 ` Wladimir
@ 2014-05-08 0:47 ` Gregory Maxwell
2014-05-08 7:45 ` Wladimir
0 siblings, 1 reply; 19+ messages in thread
From: Gregory Maxwell @ 2014-05-08 0:47 UTC (permalink / raw)
To: Wladimir; +Cc: bitcoin-development
On Wed, May 7, 2014 at 2:07 PM, Wladimir <laanwj@gmail•com> wrote:
> On Wed, May 7, 2014 at 10:25 PM, Mike Hearn <mike@plan99•net> wrote:
>> Another is to export stats over regular TCP and make them public so
>> literally anyone can listen to the stats feed for any node.
>
> TOR does this as well: http://torstatus.blutmagie.de/
>
> No idea what they use to submit/gather the statistics.
The tor network works based on a centralized (well, in theory,
federated) trusted directory service. (More info in
https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=dir-spec.txt).
Much of data that is not related to relay related is just generated by
probes, e.g. semi-trusted bandwidth authorities that measure node
performance back to the directory authorities.
More info on their monitoring work is available here:
https://metrics.torproject.org/tools.html
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-08 0:47 ` Gregory Maxwell
@ 2014-05-08 7:45 ` Wladimir
2014-05-08 10:08 ` Mike Hearn
0 siblings, 1 reply; 19+ messages in thread
From: Wladimir @ 2014-05-08 7:45 UTC (permalink / raw)
To: Gregory Maxwell; +Cc: bitcoin-development
On Thu, May 8, 2014 at 2:47 AM, Gregory Maxwell <gmaxwell@gmail•com> wrote:
>
> The tor network works based on a centralized (well, in theory,
> federated) trusted directory service. (More info in
> https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=dir-spec.txt).
Thanks!
> Much of data that is not related to relay related is just generated by
> probes, e.g. semi-trusted bandwidth authorities that measure node
> performance back to the directory authorities.
Right, so there is more involved than just adding having nodes provide
measuring points, a third-party is also performing tests on the nodes.
In a way it looks similar to how the Bitcoin DNS seeds work, trying to
find good and stable nodes, although more extensive.
Wladimir
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-08 7:45 ` Wladimir
@ 2014-05-08 10:08 ` Mike Hearn
2014-05-08 10:32 ` Wladimir
0 siblings, 1 reply; 19+ messages in thread
From: Mike Hearn @ 2014-05-08 10:08 UTC (permalink / raw)
To: Wladimir; +Cc: bitcoin-development
[-- Attachment #1: Type: text/plain, Size: 543 bytes --]
>
> In a way it looks similar to how the Bitcoin DNS seeds work, trying to
> find good and stable nodes, although more extensive.
Yeah, it's somewhat similar, except that Tor directory authorities are
authenticated (the public keys are in the source code), whereas DNS seeds
aren't. Also Bitcoin puts way more emphasis on decentralisation than Tor
does. For Tor having a P2P network is just something that's needed in order
to build an anonymity network, but it's not actually the goal. For us,
decentralisation is pretty much the end goal.
[-- Attachment #2: Type: text/html, Size: 812 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-08 10:08 ` Mike Hearn
@ 2014-05-08 10:32 ` Wladimir
0 siblings, 0 replies; 19+ messages in thread
From: Wladimir @ 2014-05-08 10:32 UTC (permalink / raw)
To: Mike Hearn; +Cc: bitcoin-development
On Thu, May 8, 2014 at 12:08 PM, Mike Hearn <mike@plan99•net> wrote:
>> In a way it looks similar to how the Bitcoin DNS seeds work, trying to
>> find good and stable nodes, although more extensive.
>
>
> Yeah, it's somewhat similar, except that Tor directory authorities are
> authenticated (the public keys are in the source code), whereas DNS seeds
> aren't. Also Bitcoin puts way more emphasis on decentralisation than Tor
> does.
Right, having authenticated 'directory authorities' hardcoded in the
source code would be out of the question for bitcoin.
Either the stats would have to be public, or private/authenticated to
parties the owner of the node configures themselves.
Wladimir
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork
2014-05-07 20:28 ` Nelson Castillo
@ 2014-05-08 11:22 ` Wladimir
0 siblings, 0 replies; 19+ messages in thread
From: Wladimir @ 2014-05-08 11:22 UTC (permalink / raw)
To: Nelson Castillo; +Cc: bitcoin-development
On Wed, May 7, 2014 at 10:28 PM, Nelson Castillo
> Is SNMP an option? That way you do not need to implement clients and there
> are many tools written.
It's always good to use an existing standard, but I remember SNMP,
even on a client device/application is pretty complex in itself.
And I suppose the registration for custom OIDs and such is kind of baroque.
I'm not entirely convinced of what the added value in this case would
be. Yes, there are tools for monitoring SNMP, but in my experience
those tools usually support other ways of collecting statistics as
well.
Wladimir
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2014-05-08 11:22 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-05-07 19:12 [Bitcoin-development] Announcing the Statoshi fork Jameson Lopp
2014-05-07 19:46 ` Wladimir
2014-05-07 19:57 ` Jameson Lopp
2014-05-07 20:18 ` Wladimir
2014-05-07 20:25 ` Mike Hearn
2014-05-07 20:31 ` Charlie 'Charles' Shrem
2014-05-07 21:07 ` Wladimir
2014-05-08 0:47 ` Gregory Maxwell
2014-05-08 7:45 ` Wladimir
2014-05-08 10:08 ` Mike Hearn
2014-05-08 10:32 ` Wladimir
2014-05-07 20:28 ` Nelson Castillo
2014-05-08 11:22 ` Wladimir
2014-05-07 20:35 ` Jameson Lopp
2014-05-07 21:04 ` Charlie 'Charles' Shrem
2014-05-07 19:50 ` Mike Hearn
2014-05-07 20:00 ` Jameson Lopp
2014-05-07 20:12 ` Mike Hearn
2014-05-07 19:55 ` Pavol Rusnak
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox