* [Bitcoin-development] Bitcoin Testing Project @ 2012-09-25 18:32 steve 2012-09-25 20:41 ` Matt Corallo 2012-09-25 20:49 ` Daniel F 0 siblings, 2 replies; 29+ messages in thread From: steve @ 2012-09-25 18:32 UTC (permalink / raw) To: Bitcoin Development List; +Cc: Bill Hees -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi All, After the failure to get any real testing done for the 0.7 release (all of which is my fault) I have decided to rejig things. I am heavily into test driven development, and I have a strong background in requirements management, and automation. I want to leave bettermeans behind, maybe we might be able to keep the voting aspect of it, and link it into mantis. So, what I have been doing over the past few weeks is developing a rudimentary requirements set, basic requirement tracking, tests to prove/stress the requirements. The next most important thing is to get release/acceptance tests done - these primarily focus on new stuff doesnt break old (ie lose a wallet, etc) and needs no special requirements. To this end I have installed various opensource applications (mantis, salomeTMF, bugzilla, etc) and am currently evaluating the best workflow process. This was a much longer post, but decided against it. :) So, what I want to know is who wants to be a part helping me out with all this? I am finalising the workflow flow diagrams and they should be ready for inspection soon. Anyone interested in helping out/reviewing processes? even if it is just some encouragement, it is all greatly appreciated. Drop me an email if you want access to the current setup and help me review the different software for the bitcoin workflow process. cheers, steve - -- my PGP public key is at pgp.mit.edu id: 0x5016FB50 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYfjMAAoJEFvEB9dQFvtQSmsH/R/FEdOQRB7ncTnHhaP8woLu nIiGX2DgLOWLOF9launSuTCrtVm2G56B9Dgl/BqScFxeuJGbzje7+kp7LgjtA3uy kS9DUZ1zhUfhslGP0UpVJJGX6Yfk8GbQ4nUcuL1VTv6nSZXWP2EvLMDPpRgKwyi5 z1FiyBg2A3Kg3Er+VmHPmpI0zZAGB5ytaenUp4xXGhL7Nk66i5X0twVr51xlEm0L zKCDXHzWTvNNlT7TzMjIxShJ/EcgCI1r6tVD3T+2e9QeVm0QNw3xeNUkMxKn+ul8 d1v1OxJbHD1CsNqW+XgVvFE2SJReizaHNOFwrqcpVCp7bABnWAB5eyTzB9B9IX8= =di5x -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-25 18:32 [Bitcoin-development] Bitcoin Testing Project steve @ 2012-09-25 20:41 ` Matt Corallo 2012-09-26 5:49 ` Wladimir 2012-09-26 12:28 ` steve 2012-09-25 20:49 ` Daniel F 1 sibling, 2 replies; 29+ messages in thread From: Matt Corallo @ 2012-09-25 20:41 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List, Bill Hees Although Jenkins may not be the best system, we already have jenkins and pull-tester (which is a dumb python script I wrote to test all incoming pull requests from github). They both run the same set of scripts, namely those at https://github.com/TheBlueMatt/test-scripts (its pretty basic right now, but since it is on github, I was hoping someone would find the inspiration to add to it). I dont really care if we keep using jenkins, but I figure we might as well keep all the tests in one place? Anyway, I'm all for more testing (I'm always complaining about how we need more tests for stuff...). Matt On Tue, 2012-09-25 at 19:32 +0100, steve wrote: > Hi All, > > After the failure to get any real testing done for the 0.7 release (all > of which is my fault) I have decided to rejig things. > > I am heavily into test driven development, and I have a strong > background in requirements management, and automation. > > I want to leave bettermeans behind, maybe we might be able to keep the > voting aspect of it, and link it into mantis. > > So, what I have been doing over the past few weeks is developing a > rudimentary requirements set, basic requirement tracking, tests to > prove/stress the requirements. > > The next most important thing is to get release/acceptance tests done - > these primarily focus on new stuff doesnt break old (ie lose a wallet, > etc) and needs no special requirements. > > To this end I have installed various opensource applications (mantis, > salomeTMF, bugzilla, etc) and am currently evaluating the best workflow > process. > > This was a much longer post, but decided against it. :) > > So, what I want to know is who wants to be a part helping me out with > all this? I am finalising the workflow flow diagrams and they should be > ready for inspection soon. > > Anyone interested in helping out/reviewing processes? even if it is just > some encouragement, it is all greatly appreciated. > > Drop me an email if you want access to the current setup and help me > review the different software for the bitcoin workflow process. > > cheers, > > steve ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-25 20:41 ` Matt Corallo @ 2012-09-26 5:49 ` Wladimir 2012-09-26 11:41 ` Daniel F 2012-09-26 12:28 ` steve 1 sibling, 1 reply; 29+ messages in thread From: Wladimir @ 2012-09-26 5:49 UTC (permalink / raw) To: Matt Corallo; +Cc: Bitcoin Development List, Bill Hees I think it's a great initiative Steve, On Tue, Sep 25, 2012 at 10:41 PM, Matt Corallo <bitcoin-list@bluematt•me> wrote: > They both run the same set of scripts, namely those at > https://github.com/TheBlueMatt/test-scripts (its pretty basic right now, > but since it is on github, I was hoping someone would find the > inspiration to add to it). The pulltester and Jenkins are really useful, I think it's best to keep these and improve them if needed. > I dont really care if we keep using jenkins, but I figure we might as > well keep all the tests in one place? Which brings me to another issue: we desperately need a page that links all the developer resources for bitcoin dev: - link to my doxygen docs (https://dev.visucore.com/bitcoin/doxygen/) - bluematt's pulltester and jenkins - any testing dashboards Steve is going to add But also: - how to check out the source code, how to find the build instructions for your platform - contribution guidelines - where to ask w/ development problems (this mailing list, #bitcoin-dev) - where to begin, which bugs to solve first ... Maybe on bitcoin.org, but at least easy to find for people that want to contribute, not just some deeply nested wiki page. Ie, something like https://www.mozilla.org/en-US/contribute/ http://www.ogre3d.org/developers ... I'm willing to write this. But I know these kinds of proposals always end in a big discussion about what should be and what should not be on bitcoin.org, however we should be a bit pragmatic here. Wladimir ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 5:49 ` Wladimir @ 2012-09-26 11:41 ` Daniel F 2012-09-26 12:00 ` Luke-Jr 0 siblings, 1 reply; 29+ messages in thread From: Daniel F @ 2012-09-26 11:41 UTC (permalink / raw) To: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 466 bytes --] on 09/26/2012 01:49 AM Wladimir said the following: > I'm willing to write this. But I know these kinds of proposals always > end in a big discussion about what should be and what should not be on > bitcoin.org, however we should be a bit pragmatic here. May I suggest a page bitcoin.org/developers, that links to a wiki page of developer resources? That way there's an easy link from the main site, but the content is readily editable and expandable. [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 224 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 11:41 ` Daniel F @ 2012-09-26 12:00 ` Luke-Jr 0 siblings, 0 replies; 29+ messages in thread From: Luke-Jr @ 2012-09-26 12:00 UTC (permalink / raw) To: bitcoin-development On Wednesday, September 26, 2012 11:41:13 AM Daniel F wrote: > on 09/26/2012 01:49 AM Wladimir said the following: > > I'm willing to write this. But I know these kinds of proposals always > > end in a big discussion about what should be and what should not be on > > bitcoin.org, however we should be a bit pragmatic here. > > May I suggest a page bitcoin.org/developers, that links to a wiki page > of developer resources? > That way there's an easy link from the main site, but the content is > readily editable and expandable. The front page already has wiki links. Adding a direct link to a developer resources page there would probably make sense. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-25 20:41 ` Matt Corallo 2012-09-26 5:49 ` Wladimir @ 2012-09-26 12:28 ` steve 2012-09-26 12:49 ` Wladimir 2012-09-27 0:53 ` [Bitcoin-development] " Matt Corallo 1 sibling, 2 replies; 29+ messages in thread From: steve @ 2012-09-26 12:28 UTC (permalink / raw) To: Matt Corallo; +Cc: Bitcoin Development List, Bill Hees -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Matt, Glad to have another ninja onboard :) On 25/09/2012 21:41, Matt Corallo wrote: > Although Jenkins may not be the best system, we already have > jenkins and pull-tester (which is a dumb python script I wrote to > test all incoming pull requests from github). I have never heard of jenkins before. I need to do some more digging. is this the right thing? https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin Mantis on the other hand, I know exceptionally well. I hate duplication of work/data unless absolutely necessary. I will check jenkins out (just out of interest what is it actually meant to do? the website implies framework, but not what its for) So, currently there are 4 potential places for bugs to be reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 - forum (bitcointalk...) 5? - is there still the ability to add bugs via sourceforge? Adding to this doesnt make sense. Each one of these reporting methods is for a different thing. I am not seeking to replace these (or even unify them) I am looking for software that will take testcases and bug reports against them [and allow for test campaigns]. Mantis is so flexible and industry standard and if the jenkins plugin works... then we can keep things as they are until they fit into better places. The reason I am so behind mantis as the backbone is it works with more or less anything, and can easily modded to work with whatever people are most comfortable with - however it is exceptionally powerful and has had a constant stream of workflow improvements over the past few years. > > They both run the same set of scripts, namely those at > https://github.com/TheBlueMatt/test-scripts (its pretty basic right > now, but since it is on github, I was hoping someone would find > the inspiration to add to it). I will check it out. I wrote a very basic script that wikified the changelog, and linked to the changes and created wiki pages for the testcases. have you seen the stuff I put on bettermeans? bits keep vanishing then re appearing. This is the outline of the testing that I setup for 0.7 https://secure.bettermeans.com/projects/4256/wiki > > I dont really care if we keep using jenkins, but I figure we might > as well keep all the tests in one place? Yes, I would love to unify all build testing and testcases into one place. I am still on the fence as to including unit tests into this. However I do see 3 distinct type of testcases 1 - requirements based testcases (requirements based off the current block chain rules - these are edge cases and known interoperability issues) 2 - Acceptance based testcases - these are testcases that should be run for every build. Check out the General Acceptance Tests in the wiki link for examples and testcases 3 - Testcases for reference implementations of things (like multisig - i see these working like the /test folder when you install a new perl module) These three things alone are a massive task. and they still wont cover everything. I would like to get the workflow so that people can sponsor or donate to a specific campaign (eg a new feature is implemented, people want it tested so can donate just for that campaign [developing testcases, structure, requirements, etc]) Once this is done, I will get to do some exciting stuff (like writing fuzzers, automation, etc) unfortunately I do not know python, only perl. > > Anyway, I'm all for more testing (I'm always complaining about how > we need more tests for stuff...). Nice, I love testing. I think we will get on :) And I would rather go for interoperability between testing rather than rewriting it all. Cheers, steve > > Matt > > On Tue, 2012-09-25 at 19:32 +0100, steve wrote: >> Hi All, >> >> After the failure to get any real testing done for the 0.7 >> release (all of which is my fault) I have decided to rejig >> things. >> >> I am heavily into test driven development, and I have a strong >> background in requirements management, and automation. >> >> I want to leave bettermeans behind, maybe we might be able to >> keep the voting aspect of it, and link it into mantis. >> >> So, what I have been doing over the past few weeks is developing >> a rudimentary requirements set, basic requirement tracking, tests >> to prove/stress the requirements. >> >> The next most important thing is to get release/acceptance tests >> done - these primarily focus on new stuff doesnt break old (ie >> lose a wallet, etc) and needs no special requirements. >> >> To this end I have installed various opensource applications >> (mantis, salomeTMF, bugzilla, etc) and am currently evaluating >> the best workflow process. >> >> This was a much longer post, but decided against it. :) >> >> So, what I want to know is who wants to be a part helping me out >> with all this? I am finalising the workflow flow diagrams and >> they should be ready for inspection soon. >> >> Anyone interested in helping out/reviewing processes? even if it >> is just some encouragement, it is all greatly appreciated. >> >> Drop me an email if you want access to the current setup and help >> me review the different software for the bitcoin workflow >> process. >> >> cheers, >> >> steve > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYvT4AAoJEFvEB9dQFvtQlgkIAJX7JYel5RGmCsbptGdQrCnT BR42tUwTg1t/NRUJ6RA8/Ou8lzallztQquShpLn4mZdQpoalvETdtAwcPnQKnaZb M5inZE/IEq8WJM1y4YkHt3BLou4BJbjwncCNy1/jqcm6f2Oonrg7isVbDwY/7JlP y/epm7XELS7NU4vVubBwQCunwvtsuydXRzuI812LiLXNqpXFMHvG2m8a2RajXE0/ xW4lOMy/hUFzEgYRQWCTAru4Ts2x3Xt26NaEUh/uKvHLwBZJ4xbdu3gpupiPb4sI bCHnVFOC7zoQKOAnfPkCMyvtyoqpzM9HW2+DWI51FoOz851Y2F36N3Fpk/2lii4= =W5xI -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 12:28 ` steve @ 2012-09-26 12:49 ` Wladimir 2012-09-26 13:22 ` steve 2012-09-27 0:53 ` [Bitcoin-development] " Matt Corallo 1 sibling, 1 reply; 29+ messages in thread From: Wladimir @ 2012-09-26 12:49 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List, Bill Hees Steve, > So, currently there are 4 potential places for bugs to be reported > 1 - jenkins (and unit tests) > 2 - git > 3 - mailing list > 4 - forum (bitcointalk...) > 5? - is there still the ability to add bugs via sourceforge? Currently github is the authoritative place to report issues. When someone reports a bug on the mailing list, IRC or forum, they are generally asked to make a github issue (or, someone else makes the issue for them). Failed tests are generally also reported on github, by the pull tester. We currently have 232 issues, mostly classified into categories such as "Bug", "Improvement", "GUI", "Wallet", and so on. Also it's easy to refer to github issues in commits with #123, with automatic linking. I'm not sure it is worth the effort to move to another system (especially if you need a another login etc...). But I'm probably misunderstanding what you're trying to do. Wladimir ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 12:49 ` Wladimir @ 2012-09-26 13:22 ` steve 2012-09-26 16:06 ` Mark Friedenbach 0 siblings, 1 reply; 29+ messages in thread From: steve @ 2012-09-26 13:22 UTC (permalink / raw) To: Wladimir; +Cc: Bitcoin Development List, Bill Hees -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 26/09/2012 13:49, Wladimir wrote: > Steve, Hi Wladimir, > >> So, currently there are 4 potential places for bugs to be >> reported 1 - jenkins (and unit tests) 2 - git 3 - mailing list 4 >> - forum (bitcointalk...) 5? - is there still the ability to add >> bugs via sourceforge? > > Currently github is the authoritative place to report issues. When > someone reports a bug on the mailing list, IRC or forum, they are > generally asked to make a github issue (or, someone else makes the > issue for them). Failed tests are generally also reported on > github, by the pull tester. excellent, that makes things much easier. > > We currently have 232 issues, mostly classified into categories > such as "Bug", "Improvement", "GUI", "Wallet", and so on. > > Also it's easy to refer to github issues in commits with #123, with > automatic linking. > > I'm not sure it is worth the effort to move to another system > (especially if you need a another login etc...). But I'm probably > misunderstanding what you're trying to do. I think you might be misunderstanding a little. I am not trying to replace the current system, I need to make sure that what I do will be compatible with it (seamlessly so for the developer). I do not want this to generate extra work for the development team. However testing is a lot more than just bug reporting, dont get me wrong bug reports are important, but so is running a testcase and that testcase passing, especially if that testcase is linked to the proof of a requirement. I am trying to develop a qa environment that is conducive to testing and will allow the testers to shine in all their glory :) and we need different tools and methodologies. Git is too developer centric to be useful for organising testing. - however there is a large amount of software that is compatible with git, so the core development team only ever need to work with git. The linking between a bug, the requirement, the fix, the retest, and updating of regression testplan is vital. So is the ability to organise testing campaigns and assigning tests, work units and test relevant docs/scripts/ideas, etc. I hope this clears things up a bit? Cheers, steve > > Wladimir > - -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYwESAAoJEFvEB9dQFvtQ/GUH/jv2c5L0OcL/kHkX/z0Yqbl/ 2IntPLdjXNKLuz0A7BMz7XfUyVmWlZrw44qxmi+Vyk5PKNBjYIidm763xHnTeJLN ULQBckYexMvan9hAyYZUOt85IpesdNgqTIsqh8f49y4roHOy8GT4M/2fhzXpnsGg G9d2m8jWGpj/kxl9qE7/WjVQC4APwBi/NiJsCrcHswgweN+zENc/Pot9YBLxAZu/ ACBUX/xFymRdaZN8P2LWBXuKx6E2WEcBdPCCWArX07wPiBlrashx9Gz6tiNzIiNq x2c4ltLzRa45AmiDtQhwqyTprz/DbyeAYO1sIsfpUxDeu9e3xTb/Zd96jfKIWI0= =iHI1 - -----END PGP SIGNATURE----- -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYwGsAAoJEFvEB9dQFvtQxOsIAKgBBOKHNFtoV2cN+GVqzlip yy0qiMvMTZKrraOhEw8QNNuOlB3LRchi+RDR/PvQkVfuwi/jHB2gUBzlapLoECBv EH8pgT/MO281pXzARgRSVkRYqkb3ljhQz3mEQg9RhR9h5t9g2mL3Tvppt7249Bg8 oGXPj6xmMcrbClF5qDbwQUUDGJfOo4eti0jSVD3qp2NE7QpPVQwuN5buchpoKt3P 9aJnjeZdLmuAk2RPoDaLXUFc9unT8AcnW96juD0zoVA9wKvAa6/8IZQf0mzV4iZP yiWGNOQtBZ+jyu2ixiEnvHqqG2ZmjtUVqWtjHkxYgrCyuuK2jOcTMNEWfn7SfKc= =yP7N -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 13:22 ` steve @ 2012-09-26 16:06 ` Mark Friedenbach 2012-09-26 17:10 ` Jeff Garzik 2012-09-26 17:44 ` steve 0 siblings, 2 replies; 29+ messages in thread From: Mark Friedenbach @ 2012-09-26 16:06 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List, Bill Hees [-- Attachment #1: Type: text/plain, Size: 2662 bytes --] Running a concurrent Mantis tracker would be confusing and fragment the development pathway. We have an issue tracker; it's on github. What's being talked about here are two separate things. Jenkins is a continuous integration system. It can be configured to run the suite of unit tests, regression tests, and any kind of automated functional tests for every commit on github and every pull request. Github is our issue tracker. Github, and only github, is where new issues should be reported (unless it's security related, in which case an email should be sent to the core devs directly). Certainly developers should be responsible for making sure that regression tests for bugs they fix make it either into the unit tests or Matt's functional test repository. QA should hold them accountable for that (re-opening tickets for bugs that have been fixed but without regression tests). The other thing we're talking about is coordinated release testing--getting release candidates in the hands of actual users and making sure that issues are reported. This is something that can't be automated as the point really is to pick up on things that the testing suite missed. You sound more qualified than me for coming up with a process, but in the end discovered issues should be reported to github, the final repository of issues that hold up Gavin from doing a release. Just my 0.002BTC Mark On Wed, Sep 26, 2012 at 6:22 AM, steve <steve@mistfpga•net> wrote: > > I think you might be misunderstanding a little. I am not trying to > replace the current system, I need to make sure that what I do will be > compatible with it (seamlessly so for the developer). I do not want this > to generate extra work for the development team. > > However testing is a lot more than just bug reporting, dont get me > wrong bug reports are important, but so is running a testcase and that > testcase passing, especially if that testcase is linked to the proof > of a requirement. I am trying to develop a qa environment that is > conducive to testing and will allow the testers to shine in all their > glory :) and we need different tools and methodologies. > > Git is too developer centric to be useful for organising testing. - > however there is a large amount of software that is compatible with > git, so the core development team only ever need to work with git. > > The linking between a bug, the requirement, the fix, the retest, and > updating of regression testplan is vital. So is the ability to > organise testing campaigns and assigning tests, work units and test > relevant docs/scripts/ideas, etc. > > I hope this clears things up a bit? > > Cheers, > > steve > [-- Attachment #2: Type: text/html, Size: 3149 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 16:06 ` Mark Friedenbach @ 2012-09-26 17:10 ` Jeff Garzik 2012-09-26 17:44 ` steve 1 sibling, 0 replies; 29+ messages in thread From: Jeff Garzik @ 2012-09-26 17:10 UTC (permalink / raw) To: Mark Friedenbach; +Cc: Bitcoin Development List, Bill Hees On Wed, Sep 26, 2012 at 12:06 PM, Mark Friedenbach <mark@monetize•io> wrote: > Certainly developers should be responsible for making sure that regression > tests for bugs they fix make it either into the unit tests or Matt's > functional test repository. QA should hold them accountable for that > (re-opening tickets for bugs that have been fixed but without regression > tests). As a goal or general principle, this is agreeable. But slavish attention to this will only get ignored. There is finite developer resources, and regression tests for certain types of bugs, like prickly P2P network interaction bugs or RPC API bugs, could potentially involve many days or weeks of coding, to sufficiently simulate the environment. The ability to easily, automatically and programmatically reproduce certain classes of bugs is simply out of reach right now, and nobody is going to shut down development to fix that problem. We should move towards this direction, yes, but bitcoin test cases are not always going to be as easy as writing (say) a compiler testcase. We can always use the help of a few good QA coders: simulating a P2P environment and checking the RPC API are two examples of very complicated problems that -can- be automated for testing... with a lot of work. -- Jeff Garzik exMULTI, Inc. jgarzik@exmulti•com ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 16:06 ` Mark Friedenbach 2012-09-26 17:10 ` Jeff Garzik @ 2012-09-26 17:44 ` steve 2012-09-26 18:09 ` Gavin Andresen 1 sibling, 1 reply; 29+ messages in thread From: steve @ 2012-09-26 17:44 UTC (permalink / raw) To: Mark Friedenbach; +Cc: Bitcoin Development List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 26/09/2012 17:06, Mark Friedenbach wrote: > Running a concurrent Mantis tracker would be confusing and fragment > the development pathway. We have an issue tracker; it's on github. I think you misunderstand what I am proposing. QA needs more than just an issue tracker. i have yet to find any opensource software that integrates testcases, nor any method for generating testplans, nor any method for linking testcases and plans to requirements, that work with git. We will need software for this (as well as workflow software) and it is much easier to integrate this into mantis/bugzilla. They are both so much more functional than Git. both mantis and bugzilla have full two way functionality with git. > > What's being talked about here are two separate things. Jenkins is > a continuous integration system. It can be configured to run the > suite of unit tests, regression tests, and any kind of automated > functional tests for every commit on github and every pull > request. well 3 but okay. Jenkins integrates both with mantis (and therefore a testsuite, etc) and with git. I do not see why anything should be any different. again I am not trying to change any current process, just develop some new ones. > > Github is our issue tracker. Github, and only github, is where new > issues should be reported (unless it's security related, in which > case an email should be sent to the core devs directly). You will only ever receive bug reports via git. How they are entered should not be of concern. There will be no space in mantis/zilla for bugs that are not related to testcases. > > Certainly developers should be responsible for making sure that > regression tests for bugs they fix make it either into the unit > tests or Matt's functional test repository. QA should hold them > accountable for that (re-opening tickets for bugs that have been > fixed but without regression tests). I feel very strongly that developers should not do regression testing or any signoff testing on their own code. QA should do the testing. I am 50/50 if they should write the testcases. the QA process should make things easier for the dev team, not generate more work for them. > > The other thing we're talking about is coordinated release > testing--getting release candidates in the hands of actual users > and making sure that issues are reported. This is something that > can't be automated as the point really is to pick up on things that > the testing suite missed. You sound more qualified than me for > coming up with a process, but in the end discovered issues should > be reported to github, the final repository of issues that hold up > Gavin from doing a release. All the core development team will still use git. the extra software is needed by test. (And the third point was coming up with a suite of tests for 3rd party developers to test their interoperability - this will having nothing to do with git, or mantis. But the solution should be compatible with mantis/zilla) > > Just my 0.002BTC Mark > > On Wed, Sep 26, 2012 at 6:22 AM, steve <steve@mistfpga•net> wrote: >> >> I think you might be misunderstanding a little. I am not trying >> to replace the current system, I need to make sure that what I do >> will be compatible with it (seamlessly so for the developer). I >> do not want this to generate extra work for the development >> team. >> >> However testing is a lot more than just bug reporting, dont get >> me wrong bug reports are important, but so is running a testcase >> and that testcase passing, especially if that testcase is linked >> to the proof of a requirement. I am trying to develop a qa >> environment that is conducive to testing and will allow the >> testers to shine in all their glory :) and we need different >> tools and methodologies. >> >> Git is too developer centric to be useful for organising testing. >> - however there is a large amount of software that is compatible >> with git, so the core development team only ever need to work >> with git. >> >> The linking between a bug, the requirement, the fix, the retest, >> and updating of regression testplan is vital. So is the ability >> to organise testing campaigns and assigning tests, work units and >> test relevant docs/scripts/ideas, etc. >> >> I hope this clears things up a bit? >> >> Cheers, >> >> steve >> > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQYz8CAAoJEFvEB9dQFvtQjkMH/Apa95IRh21mfNIuyK8kOSdt 55tLoT9a6DFyF1IPTgjHQlPN/A0JCPy/p2rIEEL7XzWpCMu1zU8BzBNmJsxGAZJG C0ue1eDEywKNFEMPTgQdebC2MbNSfUBA6lGJ5vijQlcXKoIuiV/LS7IMYh57T4u1 6Tc/SGypGe8kBLuFTihmIGH5uFS6arNGlcGgh+HRn+O4jKiAcw06lIoKh7S9Rj5e bmkimvOfproCIZeNQfSJH1BfYZaVVsJ1ouVI7ch6ytFpKsZ622zYF0Iq3042kEEp Fyqh9pDDNTJ/dwbyFpTx0WaxZySdZfZmQOCxFCAeLaCpop/nKeUnW5fy3i0sYno= =rfHO -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 17:44 ` steve @ 2012-09-26 18:09 ` Gavin Andresen 2012-09-29 18:26 ` steve 0 siblings, 1 reply; 29+ messages in thread From: Gavin Andresen @ 2012-09-26 18:09 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List [-- Attachment #1: Type: text/plain, Size: 837 bytes --] There are test cases that can be automated. That's Jenkins, and those will be run automagically. Then there are tests that cannot be automated; things like "Does the GUI look OK on all of the platforms that we support (Windows XP/2000/Vista/7/8, Ubuntu/Debian blah with window managers foo and bar, OSX 10.5/6/7/8)." Thanks to Matt, we're doing great with automated functional test cases (can always do better, of course). We're failing on simple, boring stuff like making sure we actually run on all of the platforms that we say we run on BEFORE final release. That is where I think a QA team can add a lot of value. Steve: I'm worried you're over-designing The Process. A release acceptance test plan could be nothing more than a step-by-step checklist on a wiki page, Google Doc, or Drobox shared folder... -- -- Gavin Andresen [-- Attachment #2: Type: text/html, Size: 1008 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 18:09 ` Gavin Andresen @ 2012-09-29 18:26 ` steve 2012-10-01 13:52 ` Arklan Uth Oslin 0 siblings, 1 reply; 29+ messages in thread From: steve @ 2012-09-29 18:26 UTC (permalink / raw) To: Gavin Andresen; +Cc: Bitcoin Development List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Gavin, Sorry for the delayed response, I wanted to take a couple of days to reflect on your email. On 26/09/2012 19:09, Gavin Andresen wrote: And their are other methods too. The GUI::Test package for perl will allow this to be greatly automated. (I have done this before on the localisation of photoshop.) this why we need detailed testscripts and plans. so we know what has and hasnt been done. The more boring the task the more work that needs to go into testcase development. This is the area I see as my greatest failing last time. I have a large number of virtual machines and should have at least this work. But we need very detailed testcases. with decent testplans just downloading the software, syncing the block chain, syncing an existing wallet, rescanning the blockchain and verifying the balance would cover a large number of tests. The idea behind having lots of very specific testcases is you get to see what tests have not been run. I understand your concern, however I have taken a couple of days to reflect on this and I still strongly feel that in order to make sure that this sticks, and is still useful in 1 years time we need to lay proper foundations. Those foundations are not word documents, spreadsheets, etc. they are selecting the right tools for the job. We can gain so much benefit from using 3rd party software. (bettermeans would rock if it wasnt rotting) I am sure you could do your coding work just using vi, but an sdk makes it much easier and allows you to work in a more productive manner. I have had a couple of off list emails with some testers and they also feel that it is very important to make sure we have a sound foundation (mantis is so much more than just a bug reporting tool, I see the bug reporting functionality as secondary to the main test run functionality - but it doesnt have to be mantis based, we do need workflow and testcase software though - and proper software for this is much better than just a massive google doc.) however I am checking out some other software that has been recommended. It will be very hard to change 'the process' once we have something we are used too (just look at the current resistance) I promise nothing will change for the dev team. But test does need other tools, and processes. If you feel that strongly that I am going about this the wrong way, I am happy to step back and let someone else sort it out (I will still do all the testing I possibly can). I would feel that this would be a real shame and we have the chance to setup requirements to functionality to tests all with traceability. why not do it right from the start? I will open up my vps' somepoint over the next few days and you can see what I mean. I will setup a fake git project, and sort out the interactions. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQZz1pAAoJEFvEB9dQFvtQRLkIAJtPCkW1R9vmMPY9u4o+ET1t w4pV/+W2PXo2p86HnljCIPLV/cua/1I/EJp7XR7s145Nj4KZUbzHGhvUUmwDOHW2 TGvJs+HO1bjsJfh4pWEb6PXcW3TguZxZSt5/rBAAI/5BeomSuRcZOdoV87D1xnK8 TSlgaseWrJcpKLO30/FQA3QnH/bjJ4OBmtHp8WaOtSnfww9Zbb5VYca37O15c2U4 2d0RUunDg1w2kRbkKjztxr3YasSOX+07Uvj4d5Lw7zgA0U93krNWVT1Ypo94dNJ7 6SyKi30UuqDdJ9XxZrMB/LBVNGOLlIBNWL++ocu5GFnOn9pnw57ZMBZM5g6YDpo= =ekQ/ -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-29 18:26 ` steve @ 2012-10-01 13:52 ` Arklan Uth Oslin 2012-10-01 14:28 ` steve 0 siblings, 1 reply; 29+ messages in thread From: Arklan Uth Oslin @ 2012-10-01 13:52 UTC (permalink / raw) To: Bitcoin Development List [-- Attachment #1: Type: text/plain, Size: 4787 bytes --] Hi guys. So, as I mentioned on the bitcointalk.org forums thread about the foundation, I want to get involved in the QA side of bitcoin development. I've done functional testing in the video game industry for years. I've read all the messages in this thread, but I'm left unclear how I can most effectively and quickly being helping out. Could I get a bit of a directional nudge? Arklan ---------- As long as there is light, the darkness holds no fear. And yet, even in the deepest black, there is life. - Arklan Uth Oslin I want to leave this world the same way I came into it: backwards and on fire. - Arklan Uth Oslin On Sat, Sep 29, 2012 at 12:26 PM, steve <steve@mistfpga•net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Gavin, > > Sorry for the delayed response, I wanted to take a couple of days to > reflect on your email. > > On 26/09/2012 19:09, Gavin Andresen wrote: > > > And their are other methods too. > > > > The GUI::Test package for perl will allow this to be greatly > automated. (I have done this before on the localisation of photoshop.) > > > > this why we need detailed testscripts and plans. so we know what has > and hasnt been done. The more boring the task the more work that needs > to go into testcase development. This is the area I see as my > greatest failing last time. I have a large number of virtual machines > and should have at least this work. But we need very detailed > testcases. with decent testplans just downloading the software, > syncing the block chain, syncing an existing wallet, rescanning the > blockchain and verifying the balance would cover a large number of > tests. The idea behind having lots of very specific testcases is you > get to see what tests have not been run. > > > > I understand your concern, however I have taken a couple of days to > reflect on this and I still strongly feel that in order to make sure > that this sticks, and is still useful in 1 years time we need to lay > proper foundations. Those foundations are not word documents, > spreadsheets, etc. they are selecting the right tools for the job. > > We can gain so much benefit from using 3rd party software. > (bettermeans would rock if it wasnt rotting) > > I am sure you could do your coding work just using vi, but an sdk > makes it much easier and allows you to work in a more productive manner. > > I have had a couple of off list emails with some testers and they also > feel that it is very important to make sure we have a sound foundation > (mantis is so much more than just a bug reporting tool, I see the bug > reporting functionality as secondary to the main test run > functionality - but it doesnt have to be mantis based, we do need > workflow and testcase software though - and proper software for this > is much better than just a massive google doc.) however I am checking > out some other software that has been recommended. It will be very > hard to change 'the process' once we have something we are used too > (just look at the current resistance) I promise nothing will change > for the dev team. But test does need other tools, and processes. > > If you feel that strongly that I am going about this the wrong way, I > am happy to step back and let someone else sort it out (I will still > do all the testing I possibly can). I would feel that this would be a > real shame and we have the chance to setup requirements to > functionality to tests all with traceability. why not do it right from > the start? > > I will open up my vps' somepoint over the next few days and you can > see what I mean. I will setup a fake git project, and sort out the > interactions. > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.17 (MingW32) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJQZz1pAAoJEFvEB9dQFvtQRLkIAJtPCkW1R9vmMPY9u4o+ET1t > w4pV/+W2PXo2p86HnljCIPLV/cua/1I/EJp7XR7s145Nj4KZUbzHGhvUUmwDOHW2 > TGvJs+HO1bjsJfh4pWEb6PXcW3TguZxZSt5/rBAAI/5BeomSuRcZOdoV87D1xnK8 > TSlgaseWrJcpKLO30/FQA3QnH/bjJ4OBmtHp8WaOtSnfww9Zbb5VYca37O15c2U4 > 2d0RUunDg1w2kRbkKjztxr3YasSOX+07Uvj4d5Lw7zgA0U93krNWVT1Ypo94dNJ7 > 6SyKi30UuqDdJ9XxZrMB/LBVNGOLlIBNWL++ocu5GFnOn9pnw57ZMBZM5g6YDpo= > =ekQ/ > -----END PGP SIGNATURE----- > > > ------------------------------------------------------------------------------ > How fast is your code? > 3 out of 4 devs don\\\'t know how their code performs in production. > Find out how slow your code is with AppDynamics Lite. > http://ad.doubleclick.net/clk;262219672;13503038;z? > http://info.appdynamics.com/FreeJavaPerformanceDownload.html > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > [-- Attachment #2: Type: text/html, Size: 5896 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-10-01 13:52 ` Arklan Uth Oslin @ 2012-10-01 14:28 ` steve 2012-10-01 16:52 ` Peter Vessenes 0 siblings, 1 reply; 29+ messages in thread From: steve @ 2012-10-01 14:28 UTC (permalink / raw) To: Arklan Uth Oslin; +Cc: Bitcoin Development List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/10/2012 14:52, Arklan Uth Oslin wrote: > Hi guys. > > So, as I mentioned on the bitcointalk.org forums thread about the > foundation, I want to get involved in the QA side of bitcoin > development. I've done functional testing in the video game > industry for years. Nice one, I worked in games for quite a few years. (before getting into finance then pentesting) there are about 6 keen testers now. maybe we should get a bitcoin-test mailing list, where we can discuss stuff without disturbing the dev team. > I've read all the messages in this thread, but I'm left unclear > how I can most effectively and quickly being helping out. Could I > get a bit of a directional nudge? Great question... for me I feel structure is the most important thing to sort out first. However we desperately need detailed testcases for the release of a new version. - Not too much on the change log stuff, more on the noddy stuff (as gavin points out below), downloading and making sure it works on a non dev machine, make sure the wallet isnt overwritten, etc.) doing games qa I imagine this would be an ideal place for you to start. I have a MSDN and TechNet licence so if you need some reference ms virtual machines I can help you out. However we need some testcase software. Please check out what was done on bettermeans for the stuff I was planning out... It details everything from recompense and testcases. bettermeans kinda died a death though... check out: Bitcoin over all- https://secure.bettermeans.com/projects/4180/wiki/Page_index discussion https://secure.bettermeans.com/projects/4180/boards 0.7 https://secure.bettermeans.com/projects/4256/boards and https://secure.bettermeans.com/projects/4256/wiki I still have the testcases, but until we get some proper testcase software I am loathed to publish them in a half arsed format. (they worked well on bettermeans, then just vanished one day...) what testcase software are you familiar with? apart from that, what do you feel you can do for the project? how long have you been involved in bitcoin? It may well be worth reading up all the dev stuff on the wiki so you can get you head around how the bitcoin protocol is different from the daemon and qt client. What do you think you can and will enjoy doing? What is your skill set in regard to networking, crypto and operating systems. (not that you need any, in any we still want and need you. :) there really is room for you to do whatever role you want, and as little or as much as you want - however funding is now a very tricky issue. so much so that I am not sure I want anything to do with it(distribution of coin based on work.). - I just paid for some logo spec work out of my own pocket (for example). I have some testers i know irl who are willing to work for coin. NOTE: This response has nothing to do with the bitcoin foundation. I am not a member of the foundation. I do not speak for them or even probably with them. I am still trying to work out how much qa the foundation should be responsible for, and/how it is supposed to work. I think the games cert process would be ideal for this. This however this a discussion that probably wont have my involvement. (personally I believe that the foundation should publish requirements with example code and testcases for each aspect of the reference client. (on reference platforms - I do not expect many to agree with this though) As a side note, what happens to the donations to the bitcoin testing project? do they get moved over to the foundation? this question is bigger than this email. as far as I know they are all on an address Gavin holds. Actually I would like to be involved in any discussions that would impact QA, does this mean I need to join the foundation or just go lone wolf? tbh I dont really understand foundations. I always thought they were just a tax dodge. Sorry for the long message. :) > > Arklan > > ---------- As long as there is light, the darkness holds no fear. > And yet, even in the deepest black, there is life. - Arklan Uth > Oslin > > I want to leave this world the same way I came into it: backwards > and on fire. - Arklan Uth Oslin > > > > On Sat, Sep 29, 2012 at 12:26 PM, steve <steve@mistfpga•net> > wrote: > > Hi Gavin, > > Sorry for the delayed response, I wanted to take a couple of days > to reflect on your email. > > On 26/09/2012 19:09, Gavin Andresen wrote: > > > And their are other methods too. > > > > The GUI::Test package for perl will allow this to be greatly > automated. (I have done this before on the localisation of > photoshop.) > > > > this why we need detailed testscripts and plans. so we know what > has and hasnt been done. The more boring the task the more work > that needs to go into testcase development. This is the area I see > as my greatest failing last time. I have a large number of virtual > machines and should have at least this work. But we need very > detailed testcases. with decent testplans just downloading the > software, syncing the block chain, syncing an existing wallet, > rescanning the blockchain and verifying the balance would cover a > large number of tests. The idea behind having lots of very > specific testcases is you get to see what tests have not been run. > > > > I understand your concern, however I have taken a couple of days > to reflect on this and I still strongly feel that in order to make > sure that this sticks, and is still useful in 1 years time we need > to lay proper foundations. Those foundations are not word > documents, spreadsheets, etc. they are selecting the right tools > for the job. > > We can gain so much benefit from using 3rd party software. > (bettermeans would rock if it wasnt rotting) > > I am sure you could do your coding work just using vi, but an sdk > makes it much easier and allows you to work in a more productive > manner. > > I have had a couple of off list emails with some testers and they > also feel that it is very important to make sure we have a sound > foundation (mantis is so much more than just a bug reporting tool, > I see the bug reporting functionality as secondary to the main test > run functionality - but it doesnt have to be mantis based, we do > need workflow and testcase software though - and proper software > for this is much better than just a massive google doc.) however I > am checking out some other software that has been recommended. It > will be very hard to change 'the process' once we have something we > are used too (just look at the current resistance) I promise > nothing will change for the dev team. But test does need other > tools, and processes. > > If you feel that strongly that I am going about this the wrong way, > I am happy to step back and let someone else sort it out (I will > still do all the testing I possibly can). I would feel that this > would be a real shame and we have the chance to setup requirements > to functionality to tests all with traceability. why not do it > right from the start? > > I will open up my vps' somepoint over the next few days and you > can see what I mean. I will setup a fake git project, and sort out > the interactions. > > >> >> >> ------------------------------------------------------------------------------ >> >> How fast is your code? >> 3 out of 4 devs don\\\'t know how their code performs in >> production. Find out how slow your code is with AppDynamics >> Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? >> http://info.appdynamics.com/FreeJavaPerformanceDownload.html >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> > > > > ------------------------------------------------------------------------------ > > Got visibility? > Most devs has no idea what their production app looks like. Find > out how fast your code is with AppDynamics Lite. > http://ad.doubleclick.net/clk;262219671;13503038;y? > http://info.appdynamics.com/FreeJavaPerformanceDownload.html > > > > _______________________________________________ Bitcoin-development > mailing list Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQaaieAAoJEFvEB9dQFvtQUi0H/3Eh72DqxwBt6AeNos/hJNqQ ZowMNFRupJQM301EJ7SPQmcnVuc3RF2Jw//ckpAqdpkqhHCgGO9HX/q+Ic2A9erQ CfKbUOwQgqKuLQTZ8eT5UMh969Uw8PMazo711d2I23etU0VhSK/MvjB5IiYIq5fB RJdCEluwvACaiftqfm4RA9HXXJGMnsZtlSWjVQCkg2FyYZKfpctEFdGCg6R8IaP7 ypyglR7TfygcjhUMHeWkQQCP2m2gBOHpx9H99uNBosF+6pLBhCiGUrwVkl8HUiiN 4YfhsvoRQ0BWNgvO9wH1H4EyyZtJsE7S926BNkhd4uewWGV/qe8gML+uJRstPDg= =L66i -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-10-01 14:28 ` steve @ 2012-10-01 16:52 ` Peter Vessenes 2012-10-03 1:15 ` steve 0 siblings, 1 reply; 29+ messages in thread From: Peter Vessenes @ 2012-10-01 16:52 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List [-- Attachment #1: Type: text/plain, Size: 10726 bytes --] I'm a big proponent of a testing project. I think if one could self organize that Gavin and team wanted to bless we could put up some BTC as bounties or funding. We won't have our heads around the foundation budget for a few more weeks, but self-organization is often slower than budgeting. :) This is just my opinion, but I would like very, very much to move the current specification into unit tests so that anyone could validate their alternate bitcoin implementation. This is a lot of work, some of which has been done, much of which hasn't. So, my two cents, plus an offer to bring it up at our next budgeting meeting. Peter On Mon, Oct 1, 2012 at 7:28 AM, steve <steve@mistfpga•net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 01/10/2012 14:52, Arklan Uth Oslin wrote: > > Hi guys. > > > > So, as I mentioned on the bitcointalk.org forums thread about the > > foundation, I want to get involved in the QA side of bitcoin > > development. I've done functional testing in the video game > > industry for years. > > Nice one, I worked in games for quite a few years. (before getting > into finance then pentesting) there are about 6 keen testers now. > maybe we should get a bitcoin-test mailing list, where we can discuss > stuff without disturbing the dev team. > > > I've read all the messages in this thread, but I'm left unclear > > how I can most effectively and quickly being helping out. Could I > > get a bit of a directional nudge? > > Great question... for me I feel structure is the most important thing > to sort out first. However we desperately need detailed testcases for > the release of a new version. - Not too much on the change log stuff, > more on the noddy stuff (as gavin points out below), downloading and > making sure it works on a non dev machine, make sure the wallet isnt > overwritten, etc.) doing games qa I imagine this would be an ideal > place for you to start. I have a MSDN and TechNet licence so if you > need some reference ms virtual machines I can help you out. > > However we need some testcase software. Please check out what was > done on bettermeans for the stuff I was planning out... > > It details everything from recompense and testcases. bettermeans kinda > died a death though... > > check out: > > Bitcoin over all- > https://secure.bettermeans.com/projects/4180/wiki/Page_index > discussion > https://secure.bettermeans.com/projects/4180/boards > > 0.7 > https://secure.bettermeans.com/projects/4256/boards > and > https://secure.bettermeans.com/projects/4256/wiki > > I still have the testcases, but until we get some proper testcase > software I am loathed to publish them in a half arsed format. (they > worked well on bettermeans, then just vanished one day...) what > testcase software are you familiar with? > > apart from that, what do you feel you can do for the project? how long > have you been involved in bitcoin? It may well be worth reading up > all the dev stuff on the wiki so you can get you head around how the > bitcoin protocol is different from the daemon and qt client. What do > you think you can and will enjoy doing? What is your skill set in > regard to networking, crypto and operating systems. (not that you need > any, in any we still want and need you. :) > > there really is room for you to do whatever role you want, and as > little or as much as you want - however funding is now a very tricky > issue. so much so that I am not sure I want anything to do with > it(distribution of coin based on work.). - I just paid for some logo > spec work out of my own pocket (for example). I have some testers i > know irl who are willing to work for coin. > > NOTE: This response has nothing to do with the bitcoin foundation. I > am not a member of the foundation. I do not speak for them or even > probably with them. I am still trying to work out how much qa the > foundation should be responsible for, and/how it is supposed to work. > I think the games cert process would be ideal for this. This however > this a discussion that probably wont have my involvement. (personally > I believe that the foundation should publish requirements with example > code and testcases for each aspect of the reference client. (on > reference platforms - I do not expect many to agree with this though) > > As a side note, what happens to the donations to the bitcoin testing > project? do they get moved over to the foundation? this question is > bigger than this email. as far as I know they are all on an address > Gavin holds. Actually I would like to be involved in any discussions > that would impact QA, does this mean I need to join the foundation or > just go lone wolf? > > tbh I dont really understand foundations. I always thought they were > just a tax dodge. > > Sorry for the long message. :) > > > > > Arklan > > > > ---------- As long as there is light, the darkness holds no fear. > > And yet, even in the deepest black, there is life. - Arklan Uth > > Oslin > > > > I want to leave this world the same way I came into it: backwards > > and on fire. - Arklan Uth Oslin > > > > > > > > On Sat, Sep 29, 2012 at 12:26 PM, steve <steve@mistfpga•net> > > wrote: > > > > Hi Gavin, > > > > Sorry for the delayed response, I wanted to take a couple of days > > to reflect on your email. > > > > On 26/09/2012 19:09, Gavin Andresen wrote: > > > > > > And their are other methods too. > > > > > > > > The GUI::Test package for perl will allow this to be greatly > > automated. (I have done this before on the localisation of > > photoshop.) > > > > > > > > this why we need detailed testscripts and plans. so we know what > > has and hasnt been done. The more boring the task the more work > > that needs to go into testcase development. This is the area I see > > as my greatest failing last time. I have a large number of virtual > > machines and should have at least this work. But we need very > > detailed testcases. with decent testplans just downloading the > > software, syncing the block chain, syncing an existing wallet, > > rescanning the blockchain and verifying the balance would cover a > > large number of tests. The idea behind having lots of very > > specific testcases is you get to see what tests have not been run. > > > > > > > > I understand your concern, however I have taken a couple of days > > to reflect on this and I still strongly feel that in order to make > > sure that this sticks, and is still useful in 1 years time we need > > to lay proper foundations. Those foundations are not word > > documents, spreadsheets, etc. they are selecting the right tools > > for the job. > > > > We can gain so much benefit from using 3rd party software. > > (bettermeans would rock if it wasnt rotting) > > > > I am sure you could do your coding work just using vi, but an sdk > > makes it much easier and allows you to work in a more productive > > manner. > > > > I have had a couple of off list emails with some testers and they > > also feel that it is very important to make sure we have a sound > > foundation (mantis is so much more than just a bug reporting tool, > > I see the bug reporting functionality as secondary to the main test > > run functionality - but it doesnt have to be mantis based, we do > > need workflow and testcase software though - and proper software > > for this is much better than just a massive google doc.) however I > > am checking out some other software that has been recommended. It > > will be very hard to change 'the process' once we have something we > > are used too (just look at the current resistance) I promise > > nothing will change for the dev team. But test does need other > > tools, and processes. > > > > If you feel that strongly that I am going about this the wrong way, > > I am happy to step back and let someone else sort it out (I will > > still do all the testing I possibly can). I would feel that this > > would be a real shame and we have the chance to setup requirements > > to functionality to tests all with traceability. why not do it > > right from the start? > > > > I will open up my vps' somepoint over the next few days and you > > can see what I mean. I will setup a fake git project, and sort out > > the interactions. > > > > > >> > >> > >> > ------------------------------------------------------------------------------ > >> > >> > How fast is your code? > >> 3 out of 4 devs don\\\'t know how their code performs in > >> production. Find out how slow your code is with AppDynamics > >> Lite. http://ad.doubleclick.net/clk;262219672;13503038;z? > >> http://info.appdynamics.com/FreeJavaPerformanceDownload.html > >> _______________________________________________ > >> Bitcoin-development mailing list > >> Bitcoin-development@lists•sourceforge.net > >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development > >> > > > > > > > > > ------------------------------------------------------------------------------ > > > > > Got visibility? > > Most devs has no idea what their production app looks like. Find > > out how fast your code is with AppDynamics Lite. > > http://ad.doubleclick.net/clk;262219671;13503038;y? > > http://info.appdynamics.com/FreeJavaPerformanceDownload.html > > > > > > > > _______________________________________________ Bitcoin-development > > mailing list Bitcoin-development@lists•sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.17 (MingW32) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJQaaieAAoJEFvEB9dQFvtQUi0H/3Eh72DqxwBt6AeNos/hJNqQ > ZowMNFRupJQM301EJ7SPQmcnVuc3RF2Jw//ckpAqdpkqhHCgGO9HX/q+Ic2A9erQ > CfKbUOwQgqKuLQTZ8eT5UMh969Uw8PMazo711d2I23etU0VhSK/MvjB5IiYIq5fB > RJdCEluwvACaiftqfm4RA9HXXJGMnsZtlSWjVQCkg2FyYZKfpctEFdGCg6R8IaP7 > ypyglR7TfygcjhUMHeWkQQCP2m2gBOHpx9H99uNBosF+6pLBhCiGUrwVkl8HUiiN > 4YfhsvoRQ0BWNgvO9wH1H4EyyZtJsE7S926BNkhd4uewWGV/qe8gML+uJRstPDg= > =L66i > -----END PGP SIGNATURE----- > > > ------------------------------------------------------------------------------ > Got visibility? > Most devs has no idea what their production app looks like. > Find out how fast your code is with AppDynamics Lite. > http://ad.doubleclick.net/clk;262219671;13503038;y? > http://info.appdynamics.com/FreeJavaPerformanceDownload.html > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > -- ------------------------------ [image: CoinLab Logo]PETER VESSENES CEO *peter@coinlab•com * / 206.486.6856 / SKYPE: vessenes 811 FIRST AVENUE / SUITE 480 / SEATTLE, WA 98104 [-- Attachment #2: Type: text/html, Size: 14926 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-10-01 16:52 ` Peter Vessenes @ 2012-10-03 1:15 ` steve 2012-10-03 2:02 ` Gregory Maxwell [not found] ` <CAMGNxUu=LTZyAxKt3pAYSVxyhHBU9pyJPCiFs-tA_weYNNXbtw@mail.gmail.com> 0 siblings, 2 replies; 29+ messages in thread From: steve @ 2012-10-03 1:15 UTC (permalink / raw) To: Peter Vessenes; +Cc: Bitcoin Development List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 01/10/2012 17:52, Peter Vessenes wrote: > I'm a big proponent of a testing project. I am very happy to hear this, however, your actual words are slightly evasive. I do not expect you to be up to speed on this. Gavin started a project called 'the bitcoin testing project' This project solicited donations, about 80 coins last time I checked. However these 80 odd coins were donated to 'the bitcoin testing project' This would seem to be an official bitcoin (both protocol and client) testing project. I signed up to work on this, and organise as much as i could of this. for various reasons I did not manage to do the testing I wanted to on 0.7 i over committed myself. Are the donations solicited for the 'bitcoin testing project) funds going to be given to 'the foundation'? Does the foundation support 'the bitcoin testing project'? does the foundation have any involvement with 'the bitcoin testing project'? > > I think if one could self organize that Gavin and team wanted to > bless we could put up some BTC as bounties or funding. We won't > have our heads around the foundation budget for a few more weeks, > but self-organization is often slower than budgeting. :) Im ready to go, more or less. Please check out the links in my previous emails. I have over 400 testcases (8 platforms * 50 release tests) - Also I am not sure what you mean by bless, I take it that is a euphemism for pay? I have tried my hardest to get bettermeans to work, but it doesnt. It does show quite a lot of work that I have done though. If you were to say to me, 'steve, by monday we need end to end, requirements based testing' It would be done. (I have already spent over 4 months on this) Leaderless leadership is something I am having a hard time with, bettermeans is excellent at this. But I have found very little in regards to voting and polling that integrates with the project in an effortless way like bettermeans. I understand that the budget from the foundation is something that needs to be worked on and organised. I offer my services in this area (qa only). I would be happy to submit my cv and refs for this, if required. I am now feeling frustrated and useless. has my last 4 months of work been for nothing? it feels like it. I know I bang on about processes but they are sorted, you can only attract talent like Arklan if he has a process to follow. i feel like a broken record. tl;dr version 1 - Will donations to the 'bitcoin testing project' as started by gavin going to be given to the foundation? 2 - Is the work bill hees and myself going to be binned? 3 - I feel like I have the knowledge and drive to push this, but I cant do it on my own. 4 - Is bill or I entitled to any of the cash raised for 'the bitcoin testing project' 5 - Do I have to join the foundation to have a say in how the project (testing) is done? 6 - sorry for being so mercenary, but am I going to receive any coin for work I have done? 7 - It really probably is the time for a bitcoin-test list to appear. Is there anything I can do to make this happen? > > This is just my opinion, but I would like very, very much to move > the current specification into unit tests so that anyone could > validate their alternate bitcoin implementation. This is a lot of > work, some of which has been done, much of which hasn't. have a look at the stuff in bettermeans. I personally think we can go a step further and publish guidelines (similar to RFC's and all the tests that we would do against a ref client) But I dont want to waste any more time on stuff that is going to be ignored, life is short. > > So, my two cents, plus an offer to bring it up at our next > budgeting meeting. I accept that offer. and I really appreciate it. I have some more questions I would like you to ask in regards to QA. (Gavin and I skyped about this a while ago and we didnt really come to a resolution, weworked out the problems though ;) ) I have an exceptionally detailed qa process (based off the game certification process) - but I have gone on about this at length in previous messages. I thank you for your email and your involvement with this, but do you think we are closer to getting stuff tested? call my bluff... Not one person has asked for login details to my proposals - and i even have a bugzilla version now. I need to sleep. sorry if i rambled. nite nite, steve > > Peter > > > On Mon, Oct 1, 2012 at 7:28 AM, steve <steve@mistfpga•net> wrote: > > On 01/10/2012 14:52, Arklan Uth Oslin wrote: >>>> Hi guys. >>>> >>>> So, as I mentioned on the bitcointalk.org forums thread about >>>> the foundation, I want to get involved in the QA side of >>>> bitcoin development. I've done functional testing in the >>>> video game industry for years. > > Nice one, I worked in games for quite a few years. (before getting > into finance then pentesting) there are about 6 keen testers now. > maybe we should get a bitcoin-test mailing list, where we can > discuss stuff without disturbing the dev team. > >>>> I've read all the messages in this thread, but I'm left >>>> unclear how I can most effectively and quickly being helping >>>> out. Could I get a bit of a directional nudge? > > Great question... for me I feel structure is the most important > thing to sort out first. However we desperately need detailed > testcases for the release of a new version. - Not too much on the > change log stuff, more on the noddy stuff (as gavin points out > below), downloading and making sure it works on a non dev machine, > make sure the wallet isnt overwritten, etc.) doing games qa I > imagine this would be an ideal place for you to start. I have a > MSDN and TechNet licence so if you need some reference ms virtual > machines I can help you out. > > However we need some testcase software. Please check out what was > done on bettermeans for the stuff I was planning out... > > It details everything from recompense and testcases. bettermeans > kinda died a death though... > > check out: > > Bitcoin over all- > https://secure.bettermeans.com/projects/4180/wiki/Page_index > discussion https://secure.bettermeans.com/projects/4180/boards > > 0.7 https://secure.bettermeans.com/projects/4256/boards and > https://secure.bettermeans.com/projects/4256/wiki > > I still have the testcases, but until we get some proper testcase > software I am loathed to publish them in a half arsed format. > (they worked well on bettermeans, then just vanished one day...) > what testcase software are you familiar with? > > apart from that, what do you feel you can do for the project? how > long have you been involved in bitcoin? It may well be worth > reading up all the dev stuff on the wiki so you can get you head > around how the bitcoin protocol is different from the daemon and qt > client. What do you think you can and will enjoy doing? What is > your skill set in regard to networking, crypto and operating > systems. (not that you need any, in any we still want and need you. > :) > > there really is room for you to do whatever role you want, and as > little or as much as you want - however funding is now a very > tricky issue. so much so that I am not sure I want anything to do > with it(distribution of coin based on work.). - I just paid for > some logo spec work out of my own pocket (for example). I have > some testers i know irl who are willing to work for coin. > > NOTE: This response has nothing to do with the bitcoin foundation. > I am not a member of the foundation. I do not speak for them or > even probably with them. I am still trying to work out how much qa > the foundation should be responsible for, and/how it is supposed to > work. I think the games cert process would be ideal for this. This > however this a discussion that probably wont have my involvement. > (personally I believe that the foundation should publish > requirements with example code and testcases for each aspect of the > reference client. (on reference platforms - I do not expect many to > agree with this though) > > As a side note, what happens to the donations to the bitcoin > testing project? do they get moved over to the foundation? this > question is bigger than this email. as far as I know they are all > on an address Gavin holds. Actually I would like to be involved in > any discussions that would impact QA, does this mean I need to join > the foundation or just go lone wolf? > > tbh I dont really understand foundations. I always thought they > were just a tax dodge. > > Sorry for the long message. :) > >>>> >>>> Arklan >>>> >>>> ---------- As long as there is light, the darkness holds no >>>> fear. And yet, even in the deepest black, there is life. - >>>> Arklan Uth Oslin >>>> >>>> I want to leave this world the same way I came into it: >>>> backwards and on fire. - Arklan Uth Oslin >>>> >>>> >>>> >>>> On Sat, Sep 29, 2012 at 12:26 PM, steve <steve@mistfpga•net> >>>> wrote: >>>> >>>> Hi Gavin, >>>> >>>> Sorry for the delayed response, I wanted to take a couple of >>>> days to reflect on your email. >>>> >>>> On 26/09/2012 19:09, Gavin Andresen wrote: >>>> >>>> >>>> And their are other methods too. >>>> >>>> >>>> >>>> The GUI::Test package for perl will allow this to be greatly >>>> automated. (I have done this before on the localisation of >>>> photoshop.) >>>> >>>> >>>> >>>> this why we need detailed testscripts and plans. so we know >>>> what has and hasnt been done. The more boring the task the >>>> more work that needs to go into testcase development. This >>>> is the area I see as my greatest failing last time. I have a >>>> large number of virtual machines and should have at least >>>> this work. But we need very detailed testcases. with decent >>>> testplans just downloading the software, syncing the block >>>> chain, syncing an existing wallet, rescanning the blockchain >>>> and verifying the balance would cover a large number of >>>> tests. The idea behind having lots of very specific >>>> testcases is you get to see what tests have not been run. >>>> >>>> >>>> >>>> I understand your concern, however I have taken a couple of >>>> days to reflect on this and I still strongly feel that in >>>> order to make sure that this sticks, and is still useful in 1 >>>> years time we need to lay proper foundations. Those >>>> foundations are not word documents, spreadsheets, etc. they >>>> are selecting the right tools for the job. >>>> >>>> We can gain so much benefit from using 3rd party software. >>>> (bettermeans would rock if it wasnt rotting) >>>> >>>> I am sure you could do your coding work just using vi, but an >>>> sdk makes it much easier and allows you to work in a more >>>> productive manner. >>>> >>>> I have had a couple of off list emails with some testers and >>>> they also feel that it is very important to make sure we have >>>> a sound foundation (mantis is so much more than just a bug >>>> reporting tool, I see the bug reporting functionality as >>>> secondary to the main test run functionality - but it doesnt >>>> have to be mantis based, we do need workflow and testcase >>>> software though - and proper software for this is much better >>>> than just a massive google doc.) however I am checking out >>>> some other software that has been recommended. It will be >>>> very hard to change 'the process' once we have something we >>>> are used too (just look at the current resistance) I promise >>>> nothing will change for the dev team. But test does need >>>> other tools, and processes. >>>> >>>> If you feel that strongly that I am going about this the >>>> wrong way, I am happy to step back and let someone else sort >>>> it out (I will still do all the testing I possibly can). I >>>> would feel that this would be a real shame and we have the >>>> chance to setup requirements to functionality to tests all >>>> with traceability. why not do it right from the start? >>>> >>>> I will open up my vps' somepoint over the next few days and >>>> you can see what I mean. I will setup a fake git project, and >>>> sort out the interactions. >>>> >>>> >>>>> >>>>> >>>>> > ------------------------------------------------------------------------------ >>>>> >>>>> > > How fast is your code? >>>>> 3 out of 4 devs don\\\'t know how their code performs in >>>>> production. Find out how slow your code is with >>>>> AppDynamics Lite. >>>>> http://ad.doubleclick.net/clk;262219672;13503038;z? >>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html >>>>> >>>>> _______________________________________________ >>>>> Bitcoin-development mailing list >>>>> Bitcoin-development@lists•sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>>>> >>>> >>>> >>>> >>>> > >>>>> - ------------------------------------------------------------------------------ >>>> >>>> > Got visibility? >>>> Most devs has no idea what their production app looks like. >>>> Find out how fast your code is with AppDynamics Lite. >>>> http://ad.doubleclick.net/clk;262219671;13503038;y? >>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html >>>> >>>> >>>> >>>> _______________________________________________ >>>> Bitcoin-development mailing list >>>> Bitcoin-development@lists•sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>>> > >>>> >> >> >> ------------------------------------------------------------------------------ >> >> Got visibility? >> Most devs has no idea what their production app looks like. Find >> out how fast your code is with AppDynamics Lite. >> http://ad.doubleclick.net/clk;262219671;13503038;y? >> http://info.appdynamics.com/FreeJavaPerformanceDownload.html >> _______________________________________________ >> Bitcoin-development mailing list >> Bitcoin-development@lists•sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >> > > > -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQa5GXAAoJEFvEB9dQFvtQUR4H/3o0aOyvF6u4Wi5UUxMtGWXD Bduuag9jNY/GACFshYs365wWqJcK96GXc/svOx0l/TTv10tw9BboaMpKY2x9gWri 7DJ+mym1AorgHBlZUgZJy3kQrb5yUCR7MiYU0O28QAqK5aVoklBa6cGQJqQVRa2j vm9UrKCOeMbHiNMjagLP3s1tcQxYJUwm8hEuALWjyqsDQ+Rk7j2ndBnfonJLTCDc rIy6HhmfAGSCaXFzhH35Ydfl2hV6ASaUHtBSKke3bB/oJ/bA5XspmPPhCkjum4Lo 4b28i4/O8BTisAsr8N3FYQ17dmD2fu8/gmQt1Rl9MBfWBHPJ9YVIIittk+WhcI0= =nER6 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-10-03 1:15 ` steve @ 2012-10-03 2:02 ` Gregory Maxwell 2012-10-03 3:00 ` steve [not found] ` <CAMGNxUu=LTZyAxKt3pAYSVxyhHBU9pyJPCiFs-tA_weYNNXbtw@mail.gmail.com> 1 sibling, 1 reply; 29+ messages in thread From: Gregory Maxwell @ 2012-10-03 2:02 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List On Tue, Oct 2, 2012 at 9:15 PM, steve <steve@mistfpga•net> wrote: > Im ready to go, more or less. Please check out the links in my > previous emails. I have over 400 testcases (8 platforms * 50 release > tests) - Also I am not sure what you mean by bless, I take it that is > a euphemism for pay? Perhaps a bit bluntly here— but since you seem to be rather boldly insisting on getting paid: With all of this testing where can I find the issues you've uncovered? Searching on your name/email in the issue tracker reports nothing, likewise I can't find anything in my email (beyond abstract discussion of testing). ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-10-03 2:02 ` Gregory Maxwell @ 2012-10-03 3:00 ` steve 0 siblings, 0 replies; 29+ messages in thread From: steve @ 2012-10-03 3:00 UTC (permalink / raw) To: Gregory Maxwell; +Cc: Bitcoin Development List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/10/2012 03:02, Gregory Maxwell wrote: > On Tue, Oct 2, 2012 at 9:15 PM, steve <steve@mistfpga•net> wrote: >> Im ready to go, more or less. Please check out the links in my >> previous emails. I have over 400 testcases (8 platforms * 50 >> release tests) - Also I am not sure what you mean by bless, I >> take it that is a euphemism for pay? > > Perhaps a bit bluntly here— but since you seem to be rather boldly > insisting on getting paid: that is not what i was getting at, nor what i meant at all.I am sorry it came across this way. Im a bit drunk. (yeah that old excuse) I was only trying to make an example of the quandary that exists, in what happens to the current donations. There are far more deserving people who have a claim on that coin rather than I. I have stated on more than on occasion (written in public and verbal with gavin) that I would be willing to work for free. But I would love to be able to ditch the day job. (however i see that more as services than mining/testing) rereading what i wrote yeah, fair play i can see how you read my mail like that. it wasnt meant to :/ i feel a bit of a turnip. the tone is not what i meant. it does read pretty mercenary. > > With all of this testing where can I find the issues you've > uncovered? Searching on your name/email in the issue tracker > reports nothing, likewise I can't find anything in my email (beyond > abstract discussion of testing). > This is a different issue. But I see why you have raised it and I would like to address it. I personally believe you should earn bitcoins for writing testcases, executing testcases, passing or failing test cases. in that order (failing a testcase normally generates a new one therefore encouraging indepth and recorded testing.). QA should not be judged based solely on bug reports - this is unfair and will result in race to report bugs. I have worked on a few projects that have tried it. This is one reason we need workflow and testcase software, so we can measure and compensate people for their work. As I have stated in previous emails I should have, but for various excuses did not manage to run a single testcase for 0.7. but I did setup a frame work, and I know some people did do some good work on 0.7 I believe these people should be compensated from the (limited) funds the project received... something needs to happen with those coins. I am sorry you find my emails abstract (and therefore meaningless?) this is a really confusing time. i dont know if i am doing right from wrong. I am trying to lead, without leading... i have always preferred benevolent dictator. I know a number of bitcoin businesses that are really keen to work with the project, but I have no authority, noone seems to. Someone should be able to say, nice one, thanks for the vm's. maybe use $200 worth of coin to get a technet licence, that gives 10 installs of each ms OS. slipstreaming in the service packs means you can have whatever era pc you want (and the vps providers will allow various configuration of vm's...) then even if you do it by hand you can still do 6 or 7 at once... as for bug reports, give me 1 week from now (168hrs), lets see what i can do. (priority, protocol, daemon, qt ref?) but still process is more important... I still think that writing and doing testcases should get more coin than bug reports. but from what I have read, the big bounties will be paid by the foundation. im not sure if that is true or where i read it. (protocol level bugs) it feels like I am wasting peoples time,and I should get back in my box. so I will. contact me off list if you want to have a look at the various workflow stuff i have done. cheers, steve -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQa6pAAAoJEFvEB9dQFvtQ0H8H/jUjvzmsp61w1bmDnHR+KmF4 LNu7WwVLrTvrT8AHSNh3mClWvMM3muJJA7NMb2WthAgVe3jtrGimfreAlstDsObL XNEcGvU6WN1YosH0MkN7hyDl8jnrDFoiH1P5qsMecuZIxwq7Z0vCOHEJ9DPmZilW R+G8OmoGcpaeWs9VqXR6zR7Uyz69KaDAQpMRE1GTu3zQP9HSSolciy3ESeJRR9Sd yO7EcCGdQot80rOG/VIZ0wkOmzGGm1thzYzayD6Zn2eW4Hw+ME1en9ksIbXJFpSv IdgThEm7p5UuBo0jFkbX4Awrf9hfusZSEGWfhZqdASqqkSBnYqLmWF1sLprDRF8= =Iv87 -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
[parent not found: <CAMGNxUu=LTZyAxKt3pAYSVxyhHBU9pyJPCiFs-tA_weYNNXbtw@mail.gmail.com>]
[parent not found: <CAMGNxUuHRBkE_MbmY=A0vQvq=gMfzCFG8Us7SdBn-14KiKMaNg@mail.gmail.com>]
* Re: [Bitcoin-development] Bitcoin Testing Project [not found] ` <CAMGNxUuHRBkE_MbmY=A0vQvq=gMfzCFG8Us7SdBn-14KiKMaNg@mail.gmail.com> @ 2012-10-03 5:04 ` Peter Vessenes 2012-10-03 16:06 ` steve 1 sibling, 0 replies; 29+ messages in thread From: Peter Vessenes @ 2012-10-03 5:04 UTC (permalink / raw) To: steve, Bitcoin Dev [-- Attachment #1: Type: text/plain, Size: 92 bytes --] And, finally, when I say "Ditto to above" I mean "I have no idea", not "nope". Double oops. [-- Attachment #2: Type: text/html, Size: 128 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project [not found] ` <CAMGNxUuHRBkE_MbmY=A0vQvq=gMfzCFG8Us7SdBn-14KiKMaNg@mail.gmail.com> 2012-10-03 5:04 ` Peter Vessenes @ 2012-10-03 16:06 ` steve 2012-10-03 16:11 ` Arklan Uth Oslin 1 sibling, 1 reply; 29+ messages in thread From: steve @ 2012-10-03 16:06 UTC (permalink / raw) To: Peter Vessenes; +Cc: Bitcoin Development List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Peter, Thank you for your in depth, forthright and professional response. You have answered my questions. And you have cleared up a lot of confusion I had in my mind. (and I read ditto as i dont know) I appreciate the extra information. :) This is a top post and there is nothing inline so i snipped it. cheers, steve On 03/10/2012 06:03, Peter Vessenes wrote: > Oops, I accidentally didn't reply-all to Steve. I am ccing my > detailed response to steve to the list since I think people are > wondering about how the Foundation fits in; trash the e-mail if you > don't really care. :) > > On Tue, Oct 2, 2012 at 10:01 PM, Peter Vessenes <peter@coinlab•com> > wrote: -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQbGKXAAoJEFvEB9dQFvtQ5QcIAJR0RlkjUb5Xm09eo7wjV2QV IYXbyOBx6bZWox0wxmbygpPL23grEKxlehavf18Q1S6VjdtFs75K5GV83FZb9KPk YeB0hz5ht48Ig7uQ3zu7MBCaNerTst+6fQ/k5Uu6l2WKCVwk18WykGrnXMnSTbpM qONQCc4HQhOm7sgEsJD9KNp73eGZt/BG0hQAa8zLh2rHA0to8TER9nRUUx+mH0uf Cj58EXKVz5WI5+G31M/4v4UgRRv3Z7efaVbczrvhvv6DZfVkCpuD6t0kt2ZlcFCn oO1sKD05/VtC7DZxQvPEvvkikOp8onmue0xCJBvDpG269Xr55qNqGM039Yjy80c= =1NTS -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-10-03 16:06 ` steve @ 2012-10-03 16:11 ` Arklan Uth Oslin 0 siblings, 0 replies; 29+ messages in thread From: Arklan Uth Oslin @ 2012-10-03 16:11 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List [-- Attachment #1: Type: text/plain, Size: 2387 bytes --] is it just me, or did peter not send his message to anyone but steve - twice? all i see is his "and finally..." then this from steve. Arklan ---------- As long as there is light, the darkness holds no fear. And yet, even in the deepest black, there is life. - Arklan Uth Oslin I want to leave this world the same way I came into it: backwards and on fire. - Arklan Uth Oslin On Wed, Oct 3, 2012 at 10:06 AM, steve <steve@mistfpga•net> wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi Peter, > > Thank you for your in depth, forthright and professional response. You > have answered my questions. And you have cleared up a lot of > confusion I had in my mind. (and I read ditto as i dont know) I > appreciate the extra information. :) > > This is a top post and there is nothing inline so i snipped it. > > cheers, > > steve > > On 03/10/2012 06:03, Peter Vessenes wrote: > > Oops, I accidentally didn't reply-all to Steve. I am ccing my > > detailed response to steve to the list since I think people are > > wondering about how the Foundation fits in; trash the e-mail if you > > don't really care. :) > > > > On Tue, Oct 2, 2012 at 10:01 PM, Peter Vessenes <peter@coinlab•com> > > wrote: > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v2.0.17 (MingW32) > Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ > > iQEcBAEBAgAGBQJQbGKXAAoJEFvEB9dQFvtQ5QcIAJR0RlkjUb5Xm09eo7wjV2QV > IYXbyOBx6bZWox0wxmbygpPL23grEKxlehavf18Q1S6VjdtFs75K5GV83FZb9KPk > YeB0hz5ht48Ig7uQ3zu7MBCaNerTst+6fQ/k5Uu6l2WKCVwk18WykGrnXMnSTbpM > qONQCc4HQhOm7sgEsJD9KNp73eGZt/BG0hQAa8zLh2rHA0to8TER9nRUUx+mH0uf > Cj58EXKVz5WI5+G31M/4v4UgRRv3Z7efaVbczrvhvv6DZfVkCpuD6t0kt2ZlcFCn > oO1sKD05/VtC7DZxQvPEvvkikOp8onmue0xCJBvDpG269Xr55qNqGM039Yjy80c= > =1NTS > -----END PGP SIGNATURE----- > > > ------------------------------------------------------------------------------ > Don't let slow site performance ruin your business. Deploy New Relic APM > Deploy New Relic app performance management and know exactly > what is happening inside your Ruby, Python, PHP, Java, and .NET app > Try New Relic at no cost today and get our sweet Data Nerd shirt too! > http://p.sf.net/sfu/newrelic-dev2dev > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > [-- Attachment #2: Type: text/html, Size: 3329 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* [Bitcoin-development] Fwd: Re: Bitcoin Testing Project [not found] ` <CAMGNxUu=LTZyAxKt3pAYSVxyhHBU9pyJPCiFs-tA_weYNNXbtw@mail.gmail.com> [not found] ` <CAMGNxUuHRBkE_MbmY=A0vQvq=gMfzCFG8Us7SdBn-14KiKMaNg@mail.gmail.com> @ 2012-10-03 16:15 ` steve 2012-10-03 17:09 ` Peter Vessenes 1 sibling, 1 reply; 29+ messages in thread From: steve @ 2012-10-03 16:15 UTC (permalink / raw) To: Bitcoin Development List -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I think he had a typo in the CC. here is a forward of the email. You will have to work out the indentations yourselves :) - -------- Original Message -------- Subject: Re: [Bitcoin-development] Bitcoin Testing Project Date: Tue, 2 Oct 2012 22:01:19 -0700 From: Peter Vessenes <peter@coinlab•com> To: steve <steve@mistfpga•net> On Tue, Oct 2, 2012 at 6:15 PM, steve <steve@mistfpga•net> wrote: > On 01/10/2012 17:52, Peter Vessenes wrote: >> I'm a big proponent of a testing project. > > I am very happy to hear this, however, your actual words are > slightly evasive. I do not expect you to be up to speed on this. > Gavin started a project called 'the bitcoin testing project' This > project solicited donations, about 80 coins last time I checked. > However these 80 odd coins were donated to 'the bitcoin testing > project' This would seem to be an official bitcoin (both protocol > and client) testing project. I signed up to work on this, and > organise as much as i could of this. for various reasons I did not > manage to do the testing I wanted to on 0.7 i over committed > myself. > > Are the donations solicited for the 'bitcoin testing project) > funds going to be given to 'the foundation'? > > > Not as far as I know; sounds like they should go toward testing. > > > Does the foundation support 'the bitcoin testing project'? does > the foundation have any involvement with 'the bitcoin testing > project'? > > I personally support the idea of a testing project. I would like > the Foundation to fund it if it can't crowdsource funding from the > forums; sounds like so far the support hasn't been enough to get > all the work done. The Foundation has no formal role with the > bitcoin testing project that I'm aware of. > > > > >> I think if one could self organize that Gavin and team wanted to >> bless we could put up some BTC as bounties or funding. We won't >> have our heads around the foundation budget for a few more >> weeks, but self-organization is often slower than budgeting. :) > > Im ready to go, more or less. Please check out the links in my > previous emails. I have over 400 testcases (8 platforms * 50 > release tests) - Also I am not sure what you mean by bless, I take > it that is a euphemism for pay? > > Wow, that's awesome! I use bless to mean "Gavin saying that it > sounds good." > > > I have tried my hardest to get bettermeans to work, but it doesnt. > It does show quite a lot of work that I have done though. If you > were to say to me, 'steve, by monday we need end to end, > requirements based testing' It would be done. (I have already spent > over 4 months on this) Leaderless leadership is something I am > having a hard time with, bettermeans is excellent at this. But I > have found very little in regards to voting and polling that > integrates with the project in an effortless way like bettermeans. > > I understand that the budget from the foundation is something that > needs to be worked on and organised. I offer my services in this > area (qa only). I would be happy to submit my cv and refs for > this, if required. > > I am now feeling frustrated and useless. has my last 4 months of > work been for nothing? it feels like it. I know I bang on about > processes but they are sorted, you can only attract talent like > Arklan if he has a process to follow. i feel like a broken record. > > I'm a little late to this conversation, so I don't know what to say > in response. I will answer your questions below, though. > > > tl;dr version 1 - Will donations to the 'bitcoin testing project' > as started by gavin going to be given to the foundation? > > > I don't expect so, although we'd take them if whoever is in charge > of the testing project wants to do so. I'd expect that if the > testing project is good and community approved and supported by the > dev team the funding flow would go the other way, but we'll need to > wait for budgets to get finished. > > > 2 - Is the work bill hees and myself going to be binned? > > > I have no idea whatsoever, I would guess that's up to you and bill > hees and the dev team. > > > 3 - I feel like I have the knowledge and drive to push this, but I > cant do it on my own. > > > Totally understand the feeling! > > > 4 - Is bill or I entitled to any of the cash raised for 'the > bitcoin testing project' > > > I have no idea what the bitcoin testing project finance situation > is. > > > 5 - Do I have to join the foundation to have a say in how the > project (testing) is done? > > > Nope > > > 6 - sorry for being so mercenary, but am I going to receive any > coin for work I have done? > > > Ditto to above > > > 7 - It really probably is the time for a bitcoin-test list to > appear. Is there anything I can do to make this happen? > > I don't have much to do with this. > > >> This is just my opinion, but I would like very, very much to >> move the current specification into unit tests so that anyone >> could validate their alternate bitcoin implementation. This is a >> lot of work, some of which has been done, much of which hasn't. > > have a look at the stuff in bettermeans. I personally think we can > go a step further and publish guidelines (similar to RFC's and all > the tests that we would do against a ref client) > > > Cool, I will check it out. I agree with the RFC oriented approach, > or perhaps a canonical test suite for validation and verification. > > > But I dont want to waste any more time on stuff that is going to > be ignored, life is short. > > >> So, my two cents, plus an offer to bring it up at our next >> budgeting meeting. > > I accept that offer. and I really appreciate it. I have some more > questions I would like you to ask in regards to QA. (Gavin and I > skyped about this a while ago and we didnt really come to a > resolution, weworked out the problems though ;) ) > > I have an exceptionally detailed qa process (based off the game > certification process) - but I have gone on about this at length > in previous messages. > > I thank you for your email and your involvement with this, but do > you think we are closer to getting stuff tested? call my bluff... > Not one person has asked for login details to my proposals - and i > even have a bugzilla version now. > > I need to sleep. sorry if i rambled. > > nite nite, > > steve > > > Thanks for all the details. I'm trying to go into extra detail in > my response to you because people are still figuring out the > Foundation's situation with all things Bitcoin. I can't offer you > any promises for your future, but I'm personally really psyched to > hear about these tests. I'll let other more knowledgeable folks > jump in on the technology side of things. > > > >> Peter > > >> On Mon, Oct 1, 2012 at 7:28 AM, steve <steve@mistfpga•net> >> wrote: > >> On 01/10/2012 14:52, Arklan Uth Oslin wrote: >>>>> Hi guys. >>>>> >>>>> So, as I mentioned on the bitcointalk.org forums thread >>>>> about the foundation, I want to get involved in the QA side >>>>> of bitcoin development. I've done functional testing in >>>>> the video game industry for years. > >> Nice one, I worked in games for quite a few years. (before >> getting into finance then pentesting) there are about 6 keen >> testers now. maybe we should get a bitcoin-test mailing list, >> where we can discuss stuff without disturbing the dev team. > >>>>> I've read all the messages in this thread, but I'm left >>>>> unclear how I can most effectively and quickly being >>>>> helping out. Could I get a bit of a directional nudge? > >> Great question... for me I feel structure is the most important >> thing to sort out first. However we desperately need detailed >> testcases for the release of a new version. - Not too much on >> the change log stuff, more on the noddy stuff (as gavin points >> out below), downloading and making sure it works on a non dev >> machine, make sure the wallet isnt overwritten, etc.) doing games >> qa I imagine this would be an ideal place for you to start. I >> have a MSDN and TechNet licence so if you need some reference ms >> virtual machines I can help you out. > >> However we need some testcase software. Please check out what >> was done on bettermeans for the stuff I was planning out... > >> It details everything from recompense and testcases. bettermeans >> kinda died a death though... > >> check out: > >> Bitcoin over all- >> https://secure.bettermeans.com/projects/4180/wiki/Page_index >> discussion https://secure.bettermeans.com/projects/4180/boards > >> 0.7 https://secure.bettermeans.com/projects/4256/boards and >> https://secure.bettermeans.com/projects/4256/wiki > >> I still have the testcases, but until we get some proper >> testcase software I am loathed to publish them in a half arsed >> format. (they worked well on bettermeans, then just vanished one >> day...) what testcase software are you familiar with? > >> apart from that, what do you feel you can do for the project? >> how long have you been involved in bitcoin? It may well be >> worth reading up all the dev stuff on the wiki so you can get you >> head around how the bitcoin protocol is different from the daemon >> and qt client. What do you think you can and will enjoy doing? >> What is your skill set in regard to networking, crypto and >> operating systems. (not that you need any, in any we still want >> and need you. :) > >> there really is room for you to do whatever role you want, and >> as little or as much as you want - however funding is now a very >> tricky issue. so much so that I am not sure I want anything to >> do with it(distribution of coin based on work.). - I just paid >> for some logo spec work out of my own pocket (for example). I >> have some testers i know irl who are willing to work for coin. > >> NOTE: This response has nothing to do with the bitcoin >> foundation. I am not a member of the foundation. I do not speak >> for them or even probably with them. I am still trying to work >> out how much qa the foundation should be responsible for, and/how >> it is supposed to work. I think the games cert process would be >> ideal for this. This however this a discussion that probably >> wont have my involvement. (personally I believe that the >> foundation should publish requirements with example code and >> testcases for each aspect of the reference client. (on reference >> platforms - I do not expect many to agree with this though) > >> As a side note, what happens to the donations to the bitcoin >> testing project? do they get moved over to the foundation? this >> question is bigger than this email. as far as I know they are >> all on an address Gavin holds. Actually I would like to be >> involved in any discussions that would impact QA, does this mean >> I need to join the foundation or just go lone wolf? > >> tbh I dont really understand foundations. I always thought they >> were just a tax dodge. > >> Sorry for the long message. :) > >>>>> >>>>> Arklan >>>>> >>>>> ---------- As long as there is light, the darkness holds >>>>> no fear. And yet, even in the deepest black, there is life. >>>>> - Arklan Uth Oslin >>>>> >>>>> I want to leave this world the same way I came into it: >>>>> backwards and on fire. - Arklan Uth Oslin >>>>> >>>>> >>>>> >>>>> On Sat, Sep 29, 2012 at 12:26 PM, steve >>>>> <steve@mistfpga•net> wrote: >>>>> >>>>> Hi Gavin, >>>>> >>>>> Sorry for the delayed response, I wanted to take a couple >>>>> of days to reflect on your email. >>>>> >>>>> On 26/09/2012 19:09, Gavin Andresen wrote: >>>>> >>>>> >>>>> And their are other methods too. >>>>> >>>>> >>>>> >>>>> The GUI::Test package for perl will allow this to be >>>>> greatly automated. (I have done this before on the >>>>> localisation of photoshop.) >>>>> >>>>> >>>>> >>>>> this why we need detailed testscripts and plans. so we >>>>> know what has and hasnt been done. The more boring the task >>>>> the more work that needs to go into testcase development. >>>>> This is the area I see as my greatest failing last time. I >>>>> have a large number of virtual machines and should have at >>>>> least this work. But we need very detailed testcases. >>>>> with decent testplans just downloading the software, >>>>> syncing the block chain, syncing an existing wallet, >>>>> rescanning the blockchain and verifying the balance would >>>>> cover a large number of tests. The idea behind having lots >>>>> of very specific testcases is you get to see what tests >>>>> have not been run. >>>>> >>>>> >>>>> >>>>> I understand your concern, however I have taken a couple >>>>> of days to reflect on this and I still strongly feel that >>>>> in order to make sure that this sticks, and is still useful >>>>> in 1 years time we need to lay proper foundations. Those >>>>> foundations are not word documents, spreadsheets, etc. >>>>> they are selecting the right tools for the job. >>>>> >>>>> We can gain so much benefit from using 3rd party software. >>>>> (bettermeans would rock if it wasnt rotting) >>>>> >>>>> I am sure you could do your coding work just using vi, but >>>>> an sdk makes it much easier and allows you to work in a >>>>> more productive manner. >>>>> >>>>> I have had a couple of off list emails with some testers >>>>> and they also feel that it is very important to make sure >>>>> we have a sound foundation (mantis is so much more than >>>>> just a bug reporting tool, I see the bug reporting >>>>> functionality as secondary to the main test run >>>>> functionality - but it doesnt have to be mantis based, we >>>>> do need workflow and testcase software though - and proper >>>>> software for this is much better than just a massive google >>>>> doc.) however I am checking out some other software that >>>>> has been recommended. It will be very hard to change 'the >>>>> process' once we have something we are used too (just look >>>>> at the current resistance) I promise nothing will change >>>>> for the dev team. But test does need other tools, and >>>>> processes. >>>>> >>>>> If you feel that strongly that I am going about this the >>>>> wrong way, I am happy to step back and let someone else >>>>> sort it out (I will still do all the testing I possibly >>>>> can). I would feel that this would be a real shame and we >>>>> have the chance to setup requirements to functionality to >>>>> tests all with traceability. why not do it right from the >>>>> start? >>>>> >>>>> I will open up my vps' somepoint over the next few days >>>>> and you can see what I mean. I will setup a fake git >>>>> project, and sort out the interactions. >>>>> >>>>> >>>>>> >>>>>> >>>>>> > > ------------------------------------------------------------------------------ >>>>>> >>>>>> > > > > How fast is your code? >>>>>> 3 out of 4 devs don\\\'t know how their code performs in >>>>>> production. Find out how slow your code is with >>>>>> AppDynamics Lite. >>>>>> http://ad.doubleclick.net/clk;262219672;13503038;z? >>>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html >>>>>> >>>>>> > >>>>>> _______________________________________________ >>>>>> Bitcoin-development mailing list >>>>>> Bitcoin-development@lists•sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>>>>> >>>>> >>>>> >>>>> >>>>> > >>>>>> >>>>>> > - > > ------------------------------------------------------------------------------ >>>>> >>>>> >> > Got visibility? >>>>> Most devs has no idea what their production app looks >>>>> like. Find out how fast your code is with AppDynamics >>>>> Lite. http://ad.doubleclick.net/clk;262219671;13503038;y? >>>>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Bitcoin-development mailing list >>>>> Bitcoin-development@lists•sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>>>> > >>>>> >>>>> >>> >>> >>> > ------------------------------------------------------------------------------ >>> >>> > > Got visibility? >>> Most devs has no idea what their production app looks like. >>> Find out how fast your code is with AppDynamics Lite. >>> http://ad.doubleclick.net/clk;262219671;13503038;y? >>> http://info.appdynamics.com/FreeJavaPerformanceDownload.html >>> _______________________________________________ >>> Bitcoin-development mailing list >>> Bitcoin-development@lists•sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/bitcoin-development >>> > >>> > > > > - -- - ------------------------------ [image: CoinLab Logo]PETER VESSENES CEO *peter@coinlab•com * / 206.486.6856 / SKYPE: vessenes 811 FIRST AVENUE / SUITE 480 / SEATTLE, WA 98104 -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.17 (MingW32) Comment: Using GnuPG with Mozilla - http://www.enigmail.net/ iQEcBAEBAgAGBQJQbGSIAAoJEFvEB9dQFvtQw1YH+wW3QDMHHV7kf7ZQXvAswv5T S2htfdgp1DBYBatoJB9fUU3hgHTEz1UhENH509SjS1/dxHuIxguJkbgmuSdoeB/4 rDuuYzQVcmym/G1YXvmAkCDk5/fUY4RKlCDGLg9MVMpVFhLIr6Jiwxd0NiIqWm9K aE2bXijiTIRp1sNFxJh5SZTh9/rz1zF8TU3LBWyCegISwyBH2XeIIbVgORoMDBlm n8yKGwPSg5PaH/eWhXR17OUMPbX6xGlMqTmBo7BZDCwPpZU470rkDsPywsn55mAf W6EQhFjxpSCnmZwRhrkdM/Le/0ySSUvM4spPu7MlQ6ekGP2gBzzUPbKnTGHzNgQ= =r57G -----END PGP SIGNATURE----- ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Fwd: Re: Bitcoin Testing Project 2012-10-03 16:15 ` [Bitcoin-development] Fwd: " steve @ 2012-10-03 17:09 ` Peter Vessenes 2012-10-03 17:30 ` Gavin Andresen 0 siblings, 1 reply; 29+ messages in thread From: Peter Vessenes @ 2012-10-03 17:09 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List [-- Attachment #1: Type: text/plain, Size: 104 bytes --] My reply-all forward was blocked (over 40k), sigh. I figured I'd spammed the list enough for one night. [-- Attachment #2: Type: text/html, Size: 142 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Fwd: Re: Bitcoin Testing Project 2012-10-03 17:09 ` Peter Vessenes @ 2012-10-03 17:30 ` Gavin Andresen 0 siblings, 0 replies; 29+ messages in thread From: Gavin Andresen @ 2012-10-03 17:30 UTC (permalink / raw) To: Bitcoin Development List I had hope the Testing Project would be self-organizing, with somebody taking on the QA lead role and figuring out the hard questions like: + How to do fundraising? + If/when bitcoins are available, how to decide who gets rewarded for what? + If somebody wants to help, how do they start? Steve jumped in and started creating a gazillion tests cases, which is great, but creating test cases isn't the hard part. Creating a "community" of testing that gets things done is the hard part that I, frankly, don't have time to do. I hoped that the BetterMeans platform would help, but it sounds like it was more of a hindrance than a help. Ok: live and learn. Failed experiment, lets move on... So, RE: moving on: I'd like to tag a 0.7.1rc1 release in the next few days (I'll start another thread about that). How about a very short-term goal of getting these QA deliverables: 1. A process for QA testers to sanity-test release builds, and sign-off as "Tested/problems found" or "Tested/OK" 2. Some place online I can look to see if all of our supported platforms have been tested before promoting a release candidate to "final release" PS: Thanks to Peter for responding to the "what's the relationship between the Foundation and the Testing Project" (executive summary: no relationship right now). -- -- Gavin Andresen ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-26 12:28 ` steve 2012-09-26 12:49 ` Wladimir @ 2012-09-27 0:53 ` Matt Corallo 2012-09-27 2:29 ` Gregory Maxwell 1 sibling, 1 reply; 29+ messages in thread From: Matt Corallo @ 2012-09-27 0:53 UTC (permalink / raw) To: steve; +Cc: Bitcoin Development List, Bill Hees On Wed, 2012-09-26 at 13:28 +0100, steve wrote: > Hi Matt, > > Glad to have another ninja onboard :) > > On 25/09/2012 21:41, Matt Corallo wrote: > > Although Jenkins may not be the best system, we already have > > jenkins and pull-tester (which is a dumb python script I wrote to > > test all incoming pull requests from github). > > I have never heard of jenkins before. I need to do some more digging. > is this the right thing? > > https://wiki.jenkins-ci.org/display/JENKINS/Mantis+Plugin For a mantis plugin, sure, I guess... > > Mantis on the other hand, I know exceptionally well. I hate > duplication of work/data unless absolutely necessary. I will check > jenkins out (just out of interest what is it actually meant to do? the > website implies framework, but not what its for) Jenkins currently just runs the test script after each new commit to bitcoin (and provides binaries to anyone who wants them), so its pretty basic (though jenkins has way more features than we use). The bitcoin one lives at http://jenkins.bluematt.me/ > > > > > They both run the same set of scripts, namely those at > > https://github.com/TheBlueMatt/test-scripts (its pretty basic right > > now, but since it is on github, I was hoping someone would find > > the inspiration to add to it). > > I will check it out. I wrote a very basic script that wikified the > changelog, We currently keep a changelog at https://en.bitcoin.it/wiki/Changelog (I went back and added tons of logs a while back and it got updated, though 0.7 seems to be missing...) anyway, automating that would be nice... > and linked to the changes and created wiki pages for the > testcases. Having more info on that changelog page would be nice. > have you seen the stuff I put on bettermeans? bits keep > vanishing then re appearing. I have been meaning to catch up with the various attempts at better bitcoin testing that have started up a few times, but I keep never getting around to it... > > This is the outline of the testing that I setup for 0.7 > > https://secure.bettermeans.com/projects/4256/wiki > > > > > I dont really care if we keep using jenkins, but I figure we might > > as well keep all the tests in one place? > > Yes, I would love to unify all build testing and testcases into one > place. I am still on the fence as to including unit tests into this. > However I do see 3 distinct type of testcases Even if unit tests are considered separate, having it all run in one huge test script makes it quite easy to implement new things (like pull-tester) which test some arbitrary bitcoind commit in the same way as every other tester. > 1 - requirements based testcases (requirements based off the current > block chain rules - these are edge cases and known interoperability > issues) The BitcoinjBitcoindComparisonTool.jar file which is run as a part of the test scripts tries to hit as many block acceptance edge cases as possible (I'm sure I missed a ton, but it hits a lot too). I've also been pushing alternate implementation implementors to use it to test their own implementations. > > 2 - Acceptance based testcases - these are testcases that should be > run for every build. Check out the General Acceptance Tests in the > wiki link for examples and testcases > > 3 - Testcases for reference implementations of things (like multisig - > i see these working like the /test folder when you install a new perl > module) > > These three things alone are a massive task. and they still wont cover > everything. I would like to get the workflow so that people can > sponsor or donate to a specific campaign (eg a new feature is > implemented, people want it tested so can donate just for that > campaign [developing testcases, structure, requirements, etc]) > > Once this is done, I will get to do some exciting stuff (like writing > fuzzers, automation, etc) unfortunately I do not know python, only perl. As far as I'm concerned more test cases are more test cases, it may get unwieldy to maintain, but at least we'd have more test cases :) In terms of general testing strategies, I really prefer to script it all, jenkins is quite nice in that it can have slave workers using a different OS which run their own tests and then report back to the main jenkins instance. Getting a real Windows slave to run the installer and test that thoroughly as well as basic Mac things (I know OSX uses a very different build system...) would be nice (though I dont really have time to write all those tests...) re: GUI testing is hard: I've heard Qt's unit test framework is really powerful and can even include things like click scripting and analysis of the current views (though, I agree, its still no doubt hard). Matt ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-27 0:53 ` [Bitcoin-development] " Matt Corallo @ 2012-09-27 2:29 ` Gregory Maxwell 0 siblings, 0 replies; 29+ messages in thread From: Gregory Maxwell @ 2012-09-27 2:29 UTC (permalink / raw) To: Matt Corallo; +Cc: Bitcoin Development List, Bill Hees On Wed, Sep 26, 2012 at 8:53 PM, Matt Corallo <bitcoin-list@bluematt•me> wrote: > Jenkins currently just runs the test script after each new commit to > bitcoin (and provides binaries to anyone who wants them), so its pretty > basic (though jenkins has way more features than we use). The bitcoin > one lives at http://jenkins.bluematt.me/ Jenkins is excellent at cycling through tests, while additional external tools may bring some value they're not required. It's also essential to automate all tests that we really care are run— with our small active development group and volunteer contributors the only tests we can count on being run are the automated ones. Automated tests included with the software— or at least the source— are also the only way to have a good chance of catching gnarly platform interactions. I think more than talking about testing I think we need is actual testing. Code coverage from the current tests (e.g. bitcoin-test and a testnet sync) is very unimpressive, and while coverage isn't some magical silver bullet and does not, by itself, mean the tests are good flaws in uncovered code can't be detected by the tests. We also lack simple testing cycle documentation for people interested in testing manually to walk through, etc. I think all the meta discussion is not very useful until we actually have more substance to put into it. Otherwise I fear we're just building an airport by painting stripes and waiting for the planes to land... If someone wants to help and would like a list of some of the testing I think would be useful, ping me off-list and I can blast some suggestions. But I assume that anyone who actually wants to work on this isn't short of ideas, and at this point "work on what interests you, report what interesting thing you accomplish or discover" is probably a perfectly fine level of coordination. ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-25 18:32 [Bitcoin-development] Bitcoin Testing Project steve 2012-09-25 20:41 ` Matt Corallo @ 2012-09-25 20:49 ` Daniel F 2012-09-25 21:25 ` Gary Rowe 1 sibling, 1 reply; 29+ messages in thread From: Daniel F @ 2012-09-25 20:49 UTC (permalink / raw) To: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 316 bytes --] on 09/25/2012 02:32 PM steve said the following: > Anyone interested in helping out/reviewing processes? even if it is just > some encouragement, it is all greatly appreciated. not enough time in the day for me to seriously help out, but since you asked, here's some encouragement. :) more testing == good. [-- Attachment #2: OpenPGP digital signature --] [-- Type: application/pgp-signature, Size: 224 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
* Re: [Bitcoin-development] Bitcoin Testing Project 2012-09-25 20:49 ` Daniel F @ 2012-09-25 21:25 ` Gary Rowe 0 siblings, 0 replies; 29+ messages in thread From: Gary Rowe @ 2012-09-25 21:25 UTC (permalink / raw) To: Daniel F; +Cc: bitcoin-development [-- Attachment #1: Type: text/plain, Size: 1215 bytes --] This is definitely worth doing and I wish you every encouragement. For my part I'm working on a different area of the Bitcoin ecosystem and that is taking up all my time so I can only cheer you on from the sidelines. On 25 September 2012 21:49, Daniel F <nanotube@gmail•com> wrote: > on 09/25/2012 02:32 PM steve said the following: > > Anyone interested in helping out/reviewing processes? even if it is just > > some encouragement, it is all greatly appreciated. > > not enough time in the day for me to seriously help out, but since you > asked, here's some encouragement. :) more testing == good. > > > > ------------------------------------------------------------------------------ > Live Security Virtual Conference > Exclusive live event will cover all the ways today's security and > threat landscape has changed and how IT managers can respond. Discussions > will include endpoint security, mobile security and the latest in malware > threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ > _______________________________________________ > Bitcoin-development mailing list > Bitcoin-development@lists•sourceforge.net > https://lists.sourceforge.net/lists/listinfo/bitcoin-development > > [-- Attachment #2: Type: text/html, Size: 1827 bytes --] ^ permalink raw reply [flat|nested] 29+ messages in thread
end of thread, other threads:[~2012-10-03 17:30 UTC | newest]
Thread overview: 29+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2012-09-25 18:32 [Bitcoin-development] Bitcoin Testing Project steve
2012-09-25 20:41 ` Matt Corallo
2012-09-26 5:49 ` Wladimir
2012-09-26 11:41 ` Daniel F
2012-09-26 12:00 ` Luke-Jr
2012-09-26 12:28 ` steve
2012-09-26 12:49 ` Wladimir
2012-09-26 13:22 ` steve
2012-09-26 16:06 ` Mark Friedenbach
2012-09-26 17:10 ` Jeff Garzik
2012-09-26 17:44 ` steve
2012-09-26 18:09 ` Gavin Andresen
2012-09-29 18:26 ` steve
2012-10-01 13:52 ` Arklan Uth Oslin
2012-10-01 14:28 ` steve
2012-10-01 16:52 ` Peter Vessenes
2012-10-03 1:15 ` steve
2012-10-03 2:02 ` Gregory Maxwell
2012-10-03 3:00 ` steve
[not found] ` <CAMGNxUu=LTZyAxKt3pAYSVxyhHBU9pyJPCiFs-tA_weYNNXbtw@mail.gmail.com>
[not found] ` <CAMGNxUuHRBkE_MbmY=A0vQvq=gMfzCFG8Us7SdBn-14KiKMaNg@mail.gmail.com>
2012-10-03 5:04 ` Peter Vessenes
2012-10-03 16:06 ` steve
2012-10-03 16:11 ` Arklan Uth Oslin
2012-10-03 16:15 ` [Bitcoin-development] Fwd: " steve
2012-10-03 17:09 ` Peter Vessenes
2012-10-03 17:30 ` Gavin Andresen
2012-09-27 0:53 ` [Bitcoin-development] " Matt Corallo
2012-09-27 2:29 ` Gregory Maxwell
2012-09-25 20:49 ` Daniel F
2012-09-25 21:25 ` Gary Rowe
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox