* [Bitcoin-development] Announcing the Statoshi fork
@ 2014-05-07 19:12 Jameson Lopp
2014-05-07 19:46 ` Wladimir
` (2 more replies)
0 siblings, 3 replies; 19+ messages in thread
From: Jameson Lopp @ 2014-05-07 19:12 UTC (permalink / raw)
To: bitcoin-development
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful.
http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/
https://jlopp.github.io/statoshi/
Feedback is appreciated!
- - Jameson
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBAgAGBQJTaoWSAAoJEIch3FSFNiDclLoH/0CXPTum6B2cfoNsacihHuS9
9wt50sOgghttS3J/kloP315ijY7p2HSmhvqL2G/DWYh5Vx0f6gTaUAokQ8H6x4EV
3/pdZG+9a6eegpCtgr+IgphgPSEufzct/Mp7pKTAbH0G61toOM5ZfIgdL2X/2tpx
4TjOmjhZRHuglzsM9934EjezIsR7l2vaRQB0r1LPGSgWmDSKTTb2uK7xvD1zg0tz
QXb0hl7A6rw1xZwmw3i+PujshJCbjVh8QrFT55GYi05yYdBsS6BAG46F5D8Uvn+M
FnCBLdRfWrTzQXIxoLrtBmM1JXOJKdMhmG0p2mwzwXEGR7MR2suS/+Bb7iHJTpA=
=iNNR
-----END PGP SIGNATURE-----
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 19:12 [Bitcoin-development] Announcing the Statoshi fork Jameson Lopp @ 2014-05-07 19:46 ` Wladimir 2014-05-07 19:57 ` Jameson Lopp 2014-05-07 19:50 ` Mike Hearn 2014-05-07 19:55 ` Pavol Rusnak 2 siblings, 1 reply; 19+ messages in thread From: Wladimir @ 2014-05-07 19:46 UTC (permalink / raw) To: Jameson Lopp; +Cc: bitcoin-development On Wed, May 7, 2014 at 9:12 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful. > > http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/ > > https://jlopp.github.io/statoshi/ > > Feedback is appreciated! Ooh nice graphs! We were coincidentally talking about showing stats from a node on a local web site on the #bitcoin-dev IRC a few days ago. At some point, if we're going to offer Bitcoin Core node-only installers, it'd be nice to include something like this so the user can keep an eye on their node(s). Wladimir ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 19:46 ` Wladimir @ 2014-05-07 19:57 ` Jameson Lopp 2014-05-07 20:18 ` Wladimir 0 siblings, 1 reply; 19+ messages in thread From: Jameson Lopp @ 2014-05-07 19:57 UTC (permalink / raw) To: Wladimir; +Cc: bitcoin-development -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The next logical step may be for us to offer a public instance of these graphs; I'd be happy to work with you to set one up. I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer? - - Jameson On 05/07/2014 03:46 PM, Wladimir wrote: > On Wed, May 7, 2014 at 9:12 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> In order to gain more insight into what messages and requests a node is processing, I've created a Bitcoin Core fork that outputs statistics to StatsD. I hope that some of you will find this interesting and potentially useful. >> >> http://coinchomp.com/2014/05/07/announcing-statoshi-realtime-bitcoin-node-statistics/ >> >> https://jlopp.github.io/statoshi/ >> >> Feedback is appreciated! > > Ooh nice graphs! > > We were coincidentally talking about showing stats from a node on a > local web site on the #bitcoin-dev IRC a few days ago. > > At some point, if we're going to offer Bitcoin Core node-only > installers, it'd be nice to include something like this so the user > can keep an eye on their node(s). > > Wladimir > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTapAsAAoJEIch3FSFNiDcyMEIAIds0yo9zeWcqNqGZ+UNltoH hNt8NhYOgL/6WNeLVYdRmCrrdNn/KMSLcAZmOQ0U+W/qL3xh1RB59o3BcBnW05Yr ZxY5ajKKq+oz70ShMNUkVnzFSStMhH9fKnolrF0mgSx4CU9e0YTx/LBc/u9ulypO QNZydiiegwvTFjMxHItgU5xo/wzySazmyxN9x3Gls98vDfSjE3Rt/DTqAwHleD3t SlIu4RU2iPpAW/6MgfWqAw+CrbZ2NNKp7a7+0gsUlbdDP1h6WEvoae5sUzRmvLB3 rHMmRoRvTl4Hl1bG7CKyM4D3piBkpDf/nMqnAAFNYkocS5xVpHM1WrTMDAmkSLk= =/TPp -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 19:57 ` Jameson Lopp @ 2014-05-07 20:18 ` Wladimir 2014-05-07 20:25 ` Mike Hearn ` (2 more replies) 0 siblings, 3 replies; 19+ messages in thread From: Wladimir @ 2014-05-07 20:18 UTC (permalink / raw) To: Jameson Lopp; +Cc: bitcoin-development On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer? Depends on just how much stuff it is. The idea is primarily to have an installer for running a (wallet-less) node as an OS background service. Having some statistics available would be worth some extra download size, otherwise it would be pretty much invisible. We'd already decided that we would need something like Python for the stats service. Implementing things like web services in C++ is just not realistic given the time constraints and the great already-written code that is out there. As an optional tool it should be external, not part of bitcoind itself. I suppose the chart rendering happens client-side? In that case the web service just has to collect and provide the data, and serve static html/js files. Wladimir ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 20:18 ` Wladimir @ 2014-05-07 20:25 ` Mike Hearn 2014-05-07 20:31 ` Charlie 'Charles' Shrem 2014-05-07 21:07 ` Wladimir 2014-05-07 20:28 ` Nelson Castillo 2014-05-07 20:35 ` Jameson Lopp 2 siblings, 2 replies; 19+ messages in thread From: Mike Hearn @ 2014-05-07 20:25 UTC (permalink / raw) To: Wladimir; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 3705 bytes --] I think there a few different possible ways to go here. One is to try and simplify the setup of all the components so it all gets installed together. That might be feasible in some quite restricted setups but the installation instructions for Graphite look kind of terrifying. Another is to export stats over regular TCP and make them public so literally anyone can listen to the stats feed for any node. Then people who dig stats and graphs could work on stats aggregators that give global network visibility independently, effectively crawling the p2p network for data. It'd have the advantage of having zero setup for the node operators and not require much in the way of resources. For what it's worth, although the environment is a bit different inside Google the latter approach is used. Monitoring servers locate servers of interest via a discovery service, connect to them and start streaming stats data into a database service that can then be queried later to get graphs. The stats are also run through various rules to obtain alerts about problematic conditions. For example, if a subset of the network splits it might be hard to notice that if the node operators aren't paying attention and Matt's fork alert/emailing code isn't set up. But if there was a site crawling nodes and aggregating chain heights by version, that could trigger an alert to people who *are* paying attention. I know from practical experience that monitoring and analysis tends to appeal more to certain types of people than others. So I quite like the "let anyone monitor" approach. However, it may not be appropriate in a P2P network, I did not think about it much. Obviously I'm assuming none of the stats expose privacy sensitive data. On Wed, May 7, 2014 at 10:18 PM, Wladimir <laanwj@gmail•com> wrote: > On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> > wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > I agree that it would be awesome to offer these types of stats with the > installer; unfortunately the route I've taken has dependencies on several > other other pieces of software to do all the heavy lifting of stats > aggregation and chart rendering. I'm assuming that you would not want to > build any of that processing into Bitcoin Core itself; would you be opposed > to packaging other software along with the installer? > > Depends on just how much stuff it is. The idea is primarily to have an > installer for running a (wallet-less) node as an OS background > service. > > Having some statistics available would be worth some extra download > size, otherwise it would be pretty much invisible. > > We'd already decided that we would need something like Python for the > stats service. Implementing things like web services in C++ is just > not realistic given the time constraints and the great already-written > code that is out there. As an optional tool it should be external, not > part of bitcoind itself. > > I suppose the chart rendering happens client-side? In that case the > web service just has to collect and provide the data, and serve static > html/js files. > > Wladimir > > > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > [-- Attachment #2: Type: text/html, Size: 4756 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 20:25 ` Mike Hearn @ 2014-05-07 20:31 ` Charlie 'Charles' Shrem 2014-05-07 21:07 ` Wladimir 1 sibling, 0 replies; 19+ messages in thread From: Charlie 'Charles' Shrem @ 2014-05-07 20:31 UTC (permalink / raw) To: Mike Hearn; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 4680 bytes --] I'm going to install this now on my full node, looks really cool! This is my node page: http://199.58.210.124/ Thanks, Charlie CharlieShrem.com | *Please **encrypt messages with my PGP key <http://charlieshrem.com/contact/>* On Wed, May 7, 2014 at 4:25 PM, Mike Hearn <mike@plan99•net> wrote: > I think there a few different possible ways to go here. > > One is to try and simplify the setup of all the components so it all gets > installed together. That might be feasible in some quite restricted setups > but the installation instructions for Graphite look kind of terrifying. > > Another is to export stats over regular TCP and make them public so > literally anyone can listen to the stats feed for any node. Then people who > dig stats and graphs could work on stats aggregators that give global > network visibility independently, effectively crawling the p2p network for > data. It'd have the advantage of having zero setup for the node operators > and not require much in the way of resources. > > For what it's worth, although the environment is a bit different inside > Google the latter approach is used. Monitoring servers locate servers of > interest via a discovery service, connect to them and start streaming stats > data into a database service that can then be queried later to get graphs. > > The stats are also run through various rules to obtain alerts about > problematic conditions. For example, if a subset of the network splits it > might be hard to notice that if the node operators aren't paying attention > and Matt's fork alert/emailing code isn't set up. But if there was a site > crawling nodes and aggregating chain heights by version, that could trigger > an alert to people who *are* paying attention. > > I know from practical experience that monitoring and analysis tends to > appeal more to certain types of people than others. So I quite like the > "let anyone monitor" approach. However, it may not be appropriate in a P2P > network, I did not think about it much. > > Obviously I'm assuming none of the stats expose privacy sensitive data. > > > > On Wed, May 7, 2014 at 10:18 PM, Wladimir <laanwj@gmail•com> wrote: > >> On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> >> wrote: >> > -----BEGIN PGP SIGNED MESSAGE----- >> > Hash: SHA1 >> > >> > I agree that it would be awesome to offer these types of stats with the >> installer; unfortunately the route I've taken has dependencies on several >> other other pieces of software to do all the heavy lifting of stats >> aggregation and chart rendering. I'm assuming that you would not want to >> build any of that processing into Bitcoin Core itself; would you be opposed >> to packaging other software along with the installer? >> >> Depends on just how much stuff it is. The idea is primarily to have an >> installer for running a (wallet-less) node as an OS background >> service. >> >> Having some statistics available would be worth some extra download >> size, otherwise it would be pretty much invisible. >> >> We'd already decided that we would need something like Python for the >> stats service. Implementing things like web services in C++ is just >> not realistic given the time constraints and the great already-written >> code that is out there. As an optional tool it should be external, not >> part of bitcoind itself. >> >> I suppose the chart rendering happens client-side? In that case the >> web service just has to collect and provide the data, and serve static >> html/js files. >> >> Wladimir >> >> >> ------------------------------------------------------------------------------ >> Is your legacy SCM system holding you back? Join Perforce May 7 to find >> out: >> • 3 signs your SCM is hindering your productivity >> • Requirements for releasing software faster >> • Expert tips and advice for migrating your SCM now >> http://p.sf.net/sfu/perforce >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> > > > > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > [-- Attachment #2: Type: text/html, Size: 7904 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 20:25 ` Mike Hearn 2014-05-07 20:31 ` Charlie 'Charles' Shrem @ 2014-05-07 21:07 ` Wladimir 2014-05-08 0:47 ` Gregory Maxwell 1 sibling, 1 reply; 19+ messages in thread From: Wladimir @ 2014-05-07 21:07 UTC (permalink / raw) To: Mike Hearn; +Cc: bitcoin-development On Wed, May 7, 2014 at 10:25 PM, Mike Hearn <mike@plan99•net> wrote: > Another is to export stats over regular TCP and make them public so > literally anyone can listen to the stats feed for any node. TOR does this as well: http://torstatus.blutmagie.de/ No idea what they use to submit/gather the statistics. Wladimir ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 21:07 ` Wladimir @ 2014-05-08 0:47 ` Gregory Maxwell 2014-05-08 7:45 ` Wladimir 0 siblings, 1 reply; 19+ messages in thread From: Gregory Maxwell @ 2014-05-08 0:47 UTC (permalink / raw) To: Wladimir; +Cc: bitcoin-development On Wed, May 7, 2014 at 2:07 PM, Wladimir <laanwj@gmail•com> wrote: > On Wed, May 7, 2014 at 10:25 PM, Mike Hearn <mike@plan99•net> wrote: >> Another is to export stats over regular TCP and make them public so >> literally anyone can listen to the stats feed for any node. > > TOR does this as well: http://torstatus.blutmagie.de/ > > No idea what they use to submit/gather the statistics. The tor network works based on a centralized (well, in theory, federated) trusted directory service. (More info in https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=dir-spec.txt). Much of data that is not related to relay related is just generated by probes, e.g. semi-trusted bandwidth authorities that measure node performance back to the directory authorities. More info on their monitoring work is available here: https://metrics.torproject.org/tools.html ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-08 0:47 ` Gregory Maxwell @ 2014-05-08 7:45 ` Wladimir 2014-05-08 10:08 ` Mike Hearn 0 siblings, 1 reply; 19+ messages in thread From: Wladimir @ 2014-05-08 7:45 UTC (permalink / raw) To: Gregory Maxwell; +Cc: bitcoin-development On Thu, May 8, 2014 at 2:47 AM, Gregory Maxwell <gmaxwell@gmail•com> wrote: > > The tor network works based on a centralized (well, in theory, > federated) trusted directory service. (More info in > https://gitweb.torproject.org/torspec.git?a=blob_plain;hb=HEAD;f=dir-spec.txt). Thanks! > Much of data that is not related to relay related is just generated by > probes, e.g. semi-trusted bandwidth authorities that measure node > performance back to the directory authorities. Right, so there is more involved than just adding having nodes provide measuring points, a third-party is also performing tests on the nodes. In a way it looks similar to how the Bitcoin DNS seeds work, trying to find good and stable nodes, although more extensive. Wladimir ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-08 7:45 ` Wladimir @ 2014-05-08 10:08 ` Mike Hearn 2014-05-08 10:32 ` Wladimir 0 siblings, 1 reply; 19+ messages in thread From: Mike Hearn @ 2014-05-08 10:08 UTC (permalink / raw) To: Wladimir; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 543 bytes --] > > In a way it looks similar to how the Bitcoin DNS seeds work, trying to > find good and stable nodes, although more extensive. Yeah, it's somewhat similar, except that Tor directory authorities are authenticated (the public keys are in the source code), whereas DNS seeds aren't. Also Bitcoin puts way more emphasis on decentralisation than Tor does. For Tor having a P2P network is just something that's needed in order to build an anonymity network, but it's not actually the goal. For us, decentralisation is pretty much the end goal. [-- Attachment #2: Type: text/html, Size: 812 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-08 10:08 ` Mike Hearn @ 2014-05-08 10:32 ` Wladimir 0 siblings, 0 replies; 19+ messages in thread From: Wladimir @ 2014-05-08 10:32 UTC (permalink / raw) To: Mike Hearn; +Cc: bitcoin-development On Thu, May 8, 2014 at 12:08 PM, Mike Hearn <mike@plan99•net> wrote: >> In a way it looks similar to how the Bitcoin DNS seeds work, trying to >> find good and stable nodes, although more extensive. > > > Yeah, it's somewhat similar, except that Tor directory authorities are > authenticated (the public keys are in the source code), whereas DNS seeds > aren't. Also Bitcoin puts way more emphasis on decentralisation than Tor > does. Right, having authenticated 'directory authorities' hardcoded in the source code would be out of the question for bitcoin. Either the stats would have to be public, or private/authenticated to parties the owner of the node configures themselves. Wladimir ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 20:18 ` Wladimir 2014-05-07 20:25 ` Mike Hearn @ 2014-05-07 20:28 ` Nelson Castillo 2014-05-08 11:22 ` Wladimir 2014-05-07 20:35 ` Jameson Lopp 2 siblings, 1 reply; 19+ messages in thread From: Nelson Castillo @ 2014-05-07 20:28 UTC (permalink / raw) To: Wladimir; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 2071 bytes --] On Wed, May 7, 2014 at 3:18 PM, Wladimir <laanwj@gmail•com> wrote: > On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> > wrote: > > -----BEGIN PGP SIGNED MESSAGE----- > > Hash: SHA1 > > > > I agree that it would be awesome to offer these types of stats with the > installer; unfortunately the route I've taken has dependencies on several > other other pieces of software to do all the heavy lifting of stats > aggregation and chart rendering. I'm assuming that you would not want to > build any of that processing into Bitcoin Core itself; would you be opposed > to packaging other software along with the installer? > > Depends on just how much stuff it is. The idea is primarily to have an > installer for running a (wallet-less) node as an OS background > service. > > Having some statistics available would be worth some extra download > size, otherwise it would be pretty much invisible. > > We'd already decided that we would need something like Python for the > stats service. Implementing things like web services in C++ is just > not realistic given the time constraints and the great already-written > code that is out there. As an optional tool it should be external, not > part of bitcoind itself. > > I suppose the chart rendering happens client-side? In that case the > web service just has to collect and provide the data, and serve static > html/js files. > Is SNMP an option? That way you do not need to implement clients and there are many tools written. > Wladimir > > > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > [-- Attachment #2: Type: text/html, Size: 3065 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 20:28 ` Nelson Castillo @ 2014-05-08 11:22 ` Wladimir 0 siblings, 0 replies; 19+ messages in thread From: Wladimir @ 2014-05-08 11:22 UTC (permalink / raw) To: Nelson Castillo; +Cc: bitcoin-development On Wed, May 7, 2014 at 10:28 PM, Nelson Castillo > Is SNMP an option? That way you do not need to implement clients and there > are many tools written. It's always good to use an existing standard, but I remember SNMP, even on a client device/application is pretty complex in itself. And I suppose the registration for custom OIDs and such is kind of baroque. I'm not entirely convinced of what the added value in this case would be. Yes, there are tools for monitoring SNMP, but in my experience those tools usually support other ways of collecting statistics as well. Wladimir ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 20:18 ` Wladimir 2014-05-07 20:25 ` Mike Hearn 2014-05-07 20:28 ` Nelson Castillo @ 2014-05-07 20:35 ` Jameson Lopp 2014-05-07 21:04 ` Charlie 'Charles' Shrem 2 siblings, 1 reply; 19+ messages in thread From: Jameson Lopp @ 2014-05-07 20:35 UTC (permalink / raw) To: Wladimir; +Cc: bitcoin-development -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 The charts are generated on-demand by Graphite, which is a Django app. I will note that one reason I chose StatsD is because it sends the stats via UDP rather than TCP, which is a non-blocking operation. I didn't want the sending of stats to affect the node's performance. - - Jameson On 05/07/2014 04:18 PM, Wladimir wrote: > On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> I agree that it would be awesome to offer these types of stats with the installer; unfortunately the route I've taken has dependencies on several other other pieces of software to do all the heavy lifting of stats aggregation and chart rendering. I'm assuming that you would not want to build any of that processing into Bitcoin Core itself; would you be opposed to packaging other software along with the installer? > > Depends on just how much stuff it is. The idea is primarily to have an > installer for running a (wallet-less) node as an OS background > service. > > Having some statistics available would be worth some extra download > size, otherwise it would be pretty much invisible. > > We'd already decided that we would need something like Python for the > stats service. Implementing things like web services in C++ is just > not realistic given the time constraints and the great already-written > code that is out there. As an optional tool it should be external, not > part of bitcoind itself. > > I suppose the chart rendering happens client-side? In that case the > web service just has to collect and provide the data, and serve static > html/js files. > > Wladimir > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTapj5AAoJEIch3FSFNiDcofAIALHi7XgQi8pf75btujaeBsX3 nniRD0yZIkoAvPlvFLiKQGE8TH+VR8Sb9fQACzmajYx1yjD0gN4xvkJXbI+pkeP5 L8ZryhqxL5qCh/OI4+fkWlsp5Nwx89QvUepdXXdc/AQGQJIEMceiZOLDcjbk29Yb vCsyJL5yhzM9BM0cImuvrOBPtF3/L6DbgHP8OLD2LHRl4paJ1UDtfYCx3HVO9wp8 ZWq1oCaFyoYmUyx8GTUzbLjh9sOgaq43GKYec/kQSLmFxhhMF0dGNDMiwD/xz1i7 LIswjlEKHZYOWWL3SMQg3pLlOTzGH4mHg++BAyrtzZ5CHlc1rjsPSk2d2Df/8Zc= =GFu9 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 20:35 ` Jameson Lopp @ 2014-05-07 21:04 ` Charlie 'Charles' Shrem 0 siblings, 0 replies; 19+ messages in thread From: Charlie 'Charles' Shrem @ 2014-05-07 21:04 UTC (permalink / raw) To: Jameson Lopp; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 3271 bytes --] For those who are interested, here is my 9 step way to install a full node. I tried to make it as universal as possible. http://charlieshrem.com/node/ Thanks, Charlie CharlieShrem.com | *Please **encrypt messages with my PGP key <http://charlieshrem.com/contact/>* On Wed, May 7, 2014 at 4:35 PM, Jameson Lopp <jameson.lopp@gmail•com> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > The charts are generated on-demand by Graphite, which is a Django app. > > I will note that one reason I chose StatsD is because it sends the stats > via UDP rather than TCP, which is a non-blocking operation. I didn't want > the sending of stats to affect the node's performance. > > - - Jameson > > On 05/07/2014 04:18 PM, Wladimir wrote: > > On Wed, May 7, 2014 at 9:57 PM, Jameson Lopp <jameson.lopp@gmail•com> > wrote: > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> I agree that it would be awesome to offer these types of stats with the > installer; unfortunately the route I've taken has dependencies on several > other other pieces of software to do all the heavy lifting of stats > aggregation and chart rendering. I'm assuming that you would not want to > build any of that processing into Bitcoin Core itself; would you be opposed > to packaging other software along with the installer? > > > > Depends on just how much stuff it is. The idea is primarily to have an > > installer for running a (wallet-less) node as an OS background > > service. > > > > Having some statistics available would be worth some extra download > > size, otherwise it would be pretty much invisible. > > > > We'd already decided that we would need something like Python for the > > stats service. Implementing things like web services in C++ is just > > not realistic given the time constraints and the great already-written > > code that is out there. As an optional tool it should be external, not > > part of bitcoind itself. > > > > I suppose the chart rendering happens client-side? In that case the > > web service just has to collect and provide the data, and serve static > > html/js files. > > > > Wladimir > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.14 (GNU/Linux) > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJTapj5AAoJEIch3FSFNiDcofAIALHi7XgQi8pf75btujaeBsX3 > nniRD0yZIkoAvPlvFLiKQGE8TH+VR8Sb9fQACzmajYx1yjD0gN4xvkJXbI+pkeP5 > L8ZryhqxL5qCh/OI4+fkWlsp5Nwx89QvUepdXXdc/AQGQJIEMceiZOLDcjbk29Yb > vCsyJL5yhzM9BM0cImuvrOBPtF3/L6DbgHP8OLD2LHRl4paJ1UDtfYCx3HVO9wp8 > ZWq1oCaFyoYmUyx8GTUzbLjh9sOgaq43GKYec/kQSLmFxhhMF0dGNDMiwD/xz1i7 > LIswjlEKHZYOWWL3SMQg3pLlOTzGH4mHg++BAyrtzZ5CHlc1rjsPSk2d2Df/8Zc= > =GFu9 > -----END PGP SIGNATURE----- > > > ------------------------------------------------------------------------------ > Is your legacy SCM system holding you back? Join Perforce May 7 to find > out: > • 3 signs your SCM is hindering your productivity > • Requirements for releasing software faster > • Expert tips and advice for migrating your SCM now > http://p.sf.net/sfu/perforce > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > [-- Attachment #2: Type: text/html, Size: 5690 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 19:12 [Bitcoin-development] Announcing the Statoshi fork Jameson Lopp 2014-05-07 19:46 ` Wladimir @ 2014-05-07 19:50 ` Mike Hearn 2014-05-07 20:00 ` Jameson Lopp 2014-05-07 19:55 ` Pavol Rusnak 2 siblings, 1 reply; 19+ messages in thread From: Mike Hearn @ 2014-05-07 19:50 UTC (permalink / raw) To: Jameson Lopp; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 695 bytes --] Really nice! We definitely need to put together a team who really cares about the operations side of the network and this is a fantastic start. It'd be nice if you didn't assume knowledge of what statsd is out of the box. Given the name I'd assumed it was a small UNIX daemon but it seems it's actually a Javascript thingy? It looks like putting together a monitored bitcoind setup can be quite a lot of work. I wonder if there are ways to simplify it. For example, would it make sense for someone to run a community statsd and graphite instance, so we can get aggregate statistics across many nodes and the node operators don't have to set everything up themselves? Does that make any sense? [-- Attachment #2: Type: text/html, Size: 804 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 19:50 ` Mike Hearn @ 2014-05-07 20:00 ` Jameson Lopp 2014-05-07 20:12 ` Mike Hearn 0 siblings, 1 reply; 19+ messages in thread From: Jameson Lopp @ 2014-05-07 20:00 UTC (permalink / raw) To: Mike Hearn; +Cc: bitcoin-development -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I completely agree that this setup is far too difficult to reasonably expect anyone to implement it. You're correct that we could run a single StatsD daemon and have quite a few nodes sending statistics to it - this is really what StatsD was designed for - sampling small amounts of stats from high volume systems. There would be an issue of trust, however - StatsD was also only really designed to be run inside of highly secure infrastructure where you trust all of the machines that are talking to it. - - Jameson On 05/07/2014 03:50 PM, Mike Hearn wrote: > Really nice! We definitely need to put together a team who really cares > about the operations side of the network and this is a fantastic start. > > It'd be nice if you didn't assume knowledge of what statsd is out of the > box. Given the name I'd assumed it was a small UNIX daemon but it seems > it's actually a Javascript thingy? > > It looks like putting together a monitored bitcoind setup can be quite a > lot of work. I wonder if there are ways to simplify it. For example, would > it make sense for someone to run a community statsd and graphite instance, > so we can get aggregate statistics across many nodes and the node operators > don't have to set everything up themselves? Does that make any sense? > -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.14 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJTapDsAAoJEIch3FSFNiDcqNMIAKbliLXVJ+CovCB+etqwCYfD oTs0uxEIZvnGSzqn6i3z4tgzF7kbnpVThiHpG2MO2BytasGzfr+FsDLFpvkkvx0Q yffa1J5uB5QhY/POwg6GCN4DEmy8tbd+I5rSTKg1m/JuXZV2B4TFgr6GF+t+gBNT O2TTIngcl77XbGDF2XKOZSAHaIFx2KjdZNg5lfInVxDeA1dDzlU/vmEB37fx80EH IzD7FTllLp5qhlwYxaQTVxXmbjy+pXbtArYsVSRgoscgJI3DDKpxQj3uzjxouEHv 27QkhuS2dzFioQhLPcnz/PtcCkE1l0aDNf6n7OiJ3lGygQ0+AKZHpyuvFujpSY8= =QCYI -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 20:00 ` Jameson Lopp @ 2014-05-07 20:12 ` Mike Hearn 0 siblings, 0 replies; 19+ messages in thread From: Mike Hearn @ 2014-05-07 20:12 UTC (permalink / raw) To: Jameson Lopp; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 422 bytes --] It looks like the packet format statsd expects is rather simple - it should be easy to experiment with. Perhaps a good next step would be to improve your patch so that someone can get a feed of the stats packets via TCP by e.g. ssh tunnelling to their host. Once it's easy to get a feed of simple stats packets, people can easily experiment with different kinds of aggregation and monitoring software from their desktop. [-- Attachment #2: Type: text/html, Size: 615 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [Bitcoin-development] Announcing the Statoshi fork 2014-05-07 19:12 [Bitcoin-development] Announcing the Statoshi fork Jameson Lopp 2014-05-07 19:46 ` Wladimir 2014-05-07 19:50 ` Mike Hearn @ 2014-05-07 19:55 ` Pavol Rusnak 2 siblings, 0 replies; 19+ messages in thread From: Pavol Rusnak @ 2014-05-07 19:55 UTC (permalink / raw) To: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 350 bytes --] On 05/07/2014 09:12 PM, Jameson Lopp wrote: > I've created a Bitcoin Core fork that outputs statistics to StatsD. How complex is the patchset? Would it be possible to merge it into the mainline and enable compilation of this feature conditionally by some build-time option? -- Best Regards / S pozdravom, Pavol Rusnak <stick@gk2•sk> [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 836 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2014-05-08 11:22 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-05-07 19:12 [Bitcoin-development] Announcing the Statoshi fork Jameson Lopp 2014-05-07 19:46 ` Wladimir 2014-05-07 19:57 ` Jameson Lopp 2014-05-07 20:18 ` Wladimir 2014-05-07 20:25 ` Mike Hearn 2014-05-07 20:31 ` Charlie 'Charles' Shrem 2014-05-07 21:07 ` Wladimir 2014-05-08 0:47 ` Gregory Maxwell 2014-05-08 7:45 ` Wladimir 2014-05-08 10:08 ` Mike Hearn 2014-05-08 10:32 ` Wladimir 2014-05-07 20:28 ` Nelson Castillo 2014-05-08 11:22 ` Wladimir 2014-05-07 20:35 ` Jameson Lopp 2014-05-07 21:04 ` Charlie 'Charles' Shrem 2014-05-07 19:50 ` Mike Hearn 2014-05-07 20:00 ` Jameson Lopp 2014-05-07 20:12 ` Mike Hearn 2014-05-07 19:55 ` Pavol Rusnak
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox