BitShares Forum

Main => Stakeholder Proposals => Topic started by: wackou on December 21, 2014, 12:58:35 pm

Title: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on December 21, 2014, 12:58:35 pm
delegate name: btstools.digitalgaia (Standby | 7.45% votes)  (http://bitsharesblocks.com/delegates/delegate?name=btstools.digitalgaia)
pay rate: 30%
hangout: https://bitsharestalk.org/index.php?topic=12106.msg163527#msg163527
specs: 2-core xeon, 4GB RAM, 100Mb/s symmetric, location: FR (hosted at https://www.gandi.net)

seed node: seed01.digitalgaia.io:1776  (46.226.109.66:1776)
chain server: seed01.digitalgaia.io:1375  (46.226.109.66:1375)

bts_tools source code: https://github.com/wackou/bts_tools
last published version: 0.2.4  (https://pypi.python.org/pypi/bts_tools)
updates: see last posts in this thread for current updates, official changelog for released versions can be found here: https://github.com/wackou/bts_tools/blob/master/HISTORY.rst



Hi all,

Having learned about Protoshares about a year ago now, I have since then been fascinated by the DAC concept and BitShares in general, which is why I decided to run as delegate since the launch of BitShares (and the dry runs before that), and have been developing some tools in order to maintain my delegate easily. This has been open source since the beginning and the source code can be found here: https://github.com/wackou/bts_tools

What the tools do is that they automate a lot of the tedious tasks needed to properly maintain a delegate, such as building a new version, publishing feeds, not forgetting to update the info of your delegate with the version after an upgrade, monitoring of network connections, etc... and will send you an alert by email or iOS push notifications whenever the client crashes or starts missing blocks.

I have created a 30% paid delegate to fund my development of the tools, and further participate in the BitShares ecosystem (eg: I already provide a seed node too, but would like to provide chain nodes, write howtos about setting up a new node, etc.)


Mission

My mission is to enhance the security, stability and overall reliability of the BitShares platform, by making it easier to run a delegate properly, and enforcing good practices. I also have some ideas about making the network more resilient to defects, lack of connectivity and/or directed attacks which I would like to explore if given the opportunity.

These 2 main avenues of development can currently be seen on my delegate website:


Short-term roadmap

The roadmap I plan to follow for the short term future is to go on developing the bts_tools, fix bugs and implement feature requests that can be seen in the github issues tracker: https://github.com/wackou/bts_tools/issues

Mainly, I would like the tools to be ultra-reliable (what good is a monitoring tool if it crashes?) and start implementing what is needed in order to monitor specialized nodes (eg: seed nodes, chain nodes, backbone nodes) instead of just delegate nodes.

As for network health, I also pledge to set up and maintain 2 seed nodes / chain nodes in each major geographical region (US, Europe, Asia) for a total of 6 nodes for as long as I am voted in.


Long-term roadmap

In the long term, I plan to continue maintaining the delegate tools and make it easy for delegate to perform their duties easily (by writing tutorials, etc.).

Once we know that all delegates are operating in a stable way, I would also like to try to investigate what could be done at higher level in order to ensure that the network functions properly (for instance, I firmly believe that it would be beneficial for the network if it could be ensured that it acts as a small-world network), and would like to build incentives for nodes to coordinate in order to guarantee that, or provide myself nodes that would do that.


About me

My name is Nicolas Wack, and I have been developing in C++ for the last 15 years doing mostly audio analysis (main project of fame: http://essentia.upf.edu/, source at: https://github.com/MTG/essentia) and python for the last 10 years or so, doing a bit of everything (main projects: https://github.com/wackou/guessit (active) and https://github.com/wackou/smewt (retired, for lack of time))
You can see some of my projects on my GitHub account (https://github.com/wackou), as I am a fervent believer in open source and free software as a better way of developing software.
Title: Re: Delegate proposal: btstools.digitalgaia
Post by: wackou on December 21, 2014, 01:05:12 pm
Note: this is a new thread where I will post update of my work, I dropped the old one (https://bitsharestalk.org/index.php?topic=11857.0) because I couldn't remove the poll at the top, and I considered it visual noise as it was closed anyway given that I now have created the delegate.
Title: Re: Delegate proposal: btstools.digitalgaia
Post by: gyhy on December 21, 2014, 01:11:46 pm
 +5%
Title: Re: Delegate proposal: btstools.digitalgaia
Post by: wackou on December 21, 2014, 01:18:32 pm
I just released 0.1.4, which publishes feeds for BTC, gold, silver and all fiat BitAssets. You still only see a few of one in the UI, but they are all published (you can tune which ones are shown by editing the VISIBLE_FEEDS variable https://github.com/wackou/bts_tools/blob/65bfb166d86ad55162a51e1b620e2e2848336efe/bts_tools/feeds.py#L41)

Another main feature of this release is that there is full support for building and monitoring PTS-DPOS clients:

Code: [Select]
$ bts build pts
$ bts run pts
$ bts monitor


You can also build Sparkle, although support is still experimental as there's no official release of the client yet (it does build alpha.7 correctly, though)

Code: [Select]
$ bts build sparkle
$ bts run sparkle
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on December 24, 2014, 02:38:56 pm
updates in master:
 - tools now publish version automatically if client version doesn't correspond to published version
 - smarter caching of some rpc calls resulting in much decreased cpu usage (blockchain_get_delegate_slot_records was particularly taxing)
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on December 25, 2014, 10:39:35 am
as a Christmas gift (and a way to hopefully get voted in :) ), on top of developing the tools, I also hereby pledge to set up and maintain 2 chain nodes in each major geographical region (US, Europe, Asia) for a total of 6 chain nodes for as long as I am voted in.

This will make initial sync with the network much faster!
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on January 06, 2015, 09:06:25 am
I just released 0.1.5 with the following changes:

- smarter caching of some RPC calls (improves CPU usage of the client a lot!)
- automatically publish version of the client if not up-to-date
- added pts command-line tool that defaults to building/running PTS binaries
- new publish_slate command for the command-line tool
- bugfixes / small enhancements
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on January 10, 2015, 09:33:12 pm
Released a separate tool, bts_proxy, that acts as an RPC proxy for the BitShares client and provides fine-grained access control.

Announcement: https://bitsharestalk.org/index.php?topic=13143
Source code and README: https://github.com/wackou/bts_proxy
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: emski on January 10, 2015, 09:35:17 pm
Released a separate tool, bts_proxy, that acts as an RPC proxy for the BitShares client and provides fine-grained access control.

Announcement: https://bitsharestalk.org/index.php?topic=13143
Source code and README: https://github.com/wackou/bts_proxy

Good Work!
This might pave the way for development of more advanced (or closed source) delegate tools without the need to trust the tool creator explicitly.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: ripplexiaoshan on January 10, 2015, 10:52:59 pm
May I ask, what's the difference between your tools and Alt's tools which are being used by most of the delegates?
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on January 10, 2015, 11:46:27 pm
May I ask, what's the difference between your tools and Alt's tools which are being used by most of the delegates?

not completely familiar with Alt's tools, but as I understand it they are more a collection of scripts that help do the same thing, while I went with an integrated approach where you manage everything from a single config file (https://github.com/wackou/bts_tools/blob/master/bts_tools/config.yaml), and which also gives you a web interface to monitor your delegate, see screenshots on this page: https://github.com/wackou/bts_tools

I also have a few cmd line utilities, for instance to build the latest version, I just need to run
Code: [Select]
$ bts build
$ bts run # this runs the latest client

There is also support for DevShares, PTS and Sparkle:
Code: [Select]
$ dvs build # just added to github, no released version yet
$ pts build
$ bts build sparkle
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on January 26, 2015, 10:59:51 am
Released 0.1.6, which contains (amongst other fixes):

- full support for DevShares
- fixed issue with new naming of tags (bts/X.X.X and dvs/X.X.X)
- include slate for btstools.digitalgaia as an example slate
- send notifications grouped by clients (for multiple delegates in same wallet)
- fixed tools for new API in 0.6.0 (blockchain_get_delegate_slot_records)

This version should support both 0.5.3 and 0.6.0, but there might be minor quirks, please let me know if this is the case.

The other main task that I have been working on lately is to write documentation and a tutorial for setting up a delegate. This is still work in progress, but is already in a state which I consider useful. The annoucement for the tutorial is here:

https://bitsharestalk.org/index.php?topic=13627

In the coming weeks, I plan to work mostly on expanding the documentation a lot, and obviously fix the bugs in the tools that are going to pop up.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on February 02, 2015, 03:19:03 pm
Last week's time was spent mainly on writing more documentation, both reference documentation for the tools and making the tutorial clearer. This can be found here:

http://bts-tools.readthedocs.org
(announcement for the tutorial is at: https://bitsharestalk.org/index.php?topic=13627)

I also fixed running the GUI client in linux (bts run_gui). This week I will continue to polish the documentation, as it starts looking nice but has still a few rough edges.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on February 06, 2015, 02:42:20 pm
Released 0.1.7, which fixes the 0.1.6 release that was kinda messed up due to being released too hastily. It should now all be working!
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on February 11, 2015, 06:54:17 pm
Released 0.1.8 that fixes a few quirks and annoyances that happen when setting up a new delegate (thanks Thom for the beta-testing!) and refined documentation as well as tutorial to make things simpler and avoid confusion for people not familiar with python.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on February 19, 2015, 11:54:22 am
Released 0.1.9, which fixes the following:

- allow to pass additional args to "bts run", eg: "bts run --rebuild-index"
- fixed feeds due to bter being down
- completed (for now) documentation and tutorial
- tools display their version in footer of web pages, or using "bts version"

The next thing that I'd like to work, apart from ongoing development and maintenance of the bts_tools, would be to setup some nodes such as what is described here: http://digitalgaia.io/backbone.html
This would allow all delegates (not just mine) to be able to thwart a DDoS attack such as what is described here: https://bitsharestalk.org/index.php?topic=14150.msg184358#msg184358

However, in order to do this, I will have to get a few more VPS instances, which I will be able to afford only if my delegate gets voted in first. Remember, it's only 30%  ;)
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: onceuponatime on February 19, 2015, 06:46:27 pm
Released 0.1.9, which fixes the following:

- allow to pass additional args to "bts run", eg: "bts run --rebuild-index"
- fixed feeds due to bter being down
- completed (for now) documentation and tutorial
- tools display their version in footer of web pages, or using "bts version"

The next thing that I'd like to work, apart from ongoing development and maintenance of the bts_tools, would be to setup some nodes such as what is described here: http://digitalgaia.io/backbone.html
This would allow all delegates (not just mine) to be able to thwart a DDoS attack such as what is described here: https://bitsharestalk.org/index.php?topic=14150.msg184358#msg184358

However, in order to do this, I will have to get a few more VPS instances, which I will be able to afford only if my delegate gets voted in first. Remember, it's only 30%  ;)

I think your work is very valuable and I have already voted for you.

Can you please tell us how much a "few more VPS instances" would cost?
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on February 19, 2015, 08:16:46 pm
Released 0.1.9, which fixes the following:

- allow to pass additional args to "bts run", eg: "bts run --rebuild-index"
- fixed feeds due to bter being down
- completed (for now) documentation and tutorial
- tools display their version in footer of web pages, or using "bts version"

The next thing that I'd like to work, apart from ongoing development and maintenance of the bts_tools, would be to setup some nodes such as what is described here: http://digitalgaia.io/backbone.html
This would allow all delegates (not just mine) to be able to thwart a DDoS attack such as what is described here: https://bitsharestalk.org/index.php?topic=14150.msg184358#msg184358

However, in order to do this, I will have to get a few more VPS instances, which I will be able to afford only if my delegate gets voted in first. Remember, it's only 30%  ;)

I think your work is very valuable and I have already voted for you.

Can you please tell us how much a "few more VPS instances" would cost?

Thanks! I'd like to start with 6 nodes, and distribute them 2 in Europe, 2 in the US and 2 in Asia. The ones I have currently are ~35$/month, they are a bit on the expensive side, but quality is top notch. They also need 4G of RAM, which is expensive on VPSes and is the limiting resource of the BitShares client as of now, so I could probably get some cheaper, but not by much. In total, that would amount to ~200$/month.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: onceuponatime on February 19, 2015, 09:21:45 pm
Released 0.1.9, which fixes the following:

- allow to pass additional args to "bts run", eg: "bts run --rebuild-index"
- fixed feeds due to bter being down
- completed (for now) documentation and tutorial
- tools display their version in footer of web pages, or using "bts version"

The next thing that I'd like to work, apart from ongoing development and maintenance of the bts_tools, would be to setup some nodes such as what is described here: http://digitalgaia.io/backbone.html
This would allow all delegates (not just mine) to be able to thwart a DDoS attack such as what is described here: https://bitsharestalk.org/index.php?topic=14150.msg184358#msg184358

However, in order to do this, I will have to get a few more VPS instances, which I will be able to afford only if my delegate gets voted in first. Remember, it's only 30%  ;)

I think your work is very valuable and I have already voted for you.

Can you please tell us how much a "few more VPS instances" would cost?

Thanks! I'd like to start with 6 nodes, and distribute them 2 in Europe, 2 in the US and 2 in Asia. The ones I have currently are ~35$/month, they are a bit on the expensive side, but quality is top notch. They also need 4G of RAM, which is expensive on VPSes and is the limiting resource of the BitShares client as of now, so I could probably get some cheaper, but not by much. In total, that would amount to ~200$/month.

I have sent you bitUSD to get started. If you want to, you can pay me back a little each month after you are elected.


Everyone:  Please vote for Delegate: btstools.digitalgaia
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on February 20, 2015, 03:41:23 pm
I have sent you bitUSD to get started. If you want to, you can pay me back a little each month after you are elected.

Thank you for the BitUSD (my first BitUSDs, nice!), but I think I will send them back to you, for 2 reasons:

- in effect, this is a loan that I contract in order to be able to start working now for free. When I would repay it later, I would also be paying the rent for the VPS myself, so I'd be paying more than what I would have paid had I started only after being elected.

- I don't have huge amounts of free time right now to work on BitShares stuff, and setting up what I envision would take a non-negligible amount of additional time, which I would have a hard time justifying to my family. If that work would be paid (by having my delegate active), then it would be easier for me to find some work time.

As of right now, I still have enough ideas / bug fixes to keep me occupied on the bts_tools project for quite some time, so my plan would be to keep at this rhythm while my delegate is not active (just bts_tools), and step up my game (bts_tools + backbone/network infrastructure) once it gets voted in. Thank you though for the support  :)
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: onceuponatime on February 20, 2015, 04:28:23 pm
I have sent you bitUSD to get started. If you want to, you can pay me back a little each month after you are elected.

Thank you for the BitUSD (my first BitUSDs, nice!), but I think I will send them back to you, for 2 reasons:

- in effect, this is a loan that I contract in order to be able to start working now for free. When I would repay it later, I would also be paying the rent for the VPS myself, so I'd be paying more than what I would have paid had I started only after being elected.

- I don't have huge amounts of free time right now to work on BitShares stuff, and setting up what I envision would take a non-negligible amount of additional time, which I would have a hard time justifying to my family. If that work would be paid (by having my delegate active), then it would be easier for me to find some work time.

As of right now, I still have enough ideas / bug fixes to keep me occupied on the bts_tools project for quite some time, so my plan would be to keep at this rhythm while my delegate is not active (just bts_tools), and step up my game (bts_tools + backbone/network infrastructure) once it gets voted in. Thank you though for the support  :)

Here's a counter-offer:

If you want to start the the nodes now, you don't pay me back (at the rate of a small amount each month) when you are elected until the market cap rises enough that your delegate income would justify it.

If you want to discuss any other options, we can do it in PMs.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: Shentist on February 27, 2015, 08:53:36 pm
you should post on the general thread and maybe ask the chinese community for help to get the ball rolling!
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: onceuponatime on February 27, 2015, 09:07:24 pm
you should post on the general thread and maybe ask the chinese community for help to get the ball rolling!

He has a thread in the General, but not sure about the Chinese area:

https://bitsharestalk.org/index.php?topic=14501.0
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on March 19, 2015, 11:14:02 am
btstools.digitalgaia is now an active delegate :D thanks to everyone that voted!

I will now proceed to buy and setup the VPS instances as soon as I can, setup the seed nodes and backbone nodes, and then make the backbone available to the public and ensure it is as stable as possible.

Watch this thread for updates!
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: xeroc on March 19, 2015, 11:19:35 am
F.. i .. n .. a .. l .. l .. y!! Congratulations!
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: cube on March 20, 2015, 04:01:04 am
Congraz!  It has been a long wait.  You deserve this celebration. :)
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on March 20, 2015, 09:46:49 am
Thanks! But I'm out again... :'( :'( Whoever voted me in, also voted me out (btc38? maybe I got voted out as collateral damage from the yunbi delegates coming up in the top 101 and getting kicked out too...)
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: xeroc on March 20, 2015, 10:01:53 am
+- 150M BTS .. that hurts .. and sucks ..
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on March 26, 2015, 04:32:32 pm
ohh, forgot to mention: I released 0.1.10 about 3 days ago, which (apart from the usual bugfixes and optimizations) contains the following:

- feed checking is a bit more robust in case one of the feed providers fail
- started modularization of the monitoring thread, in preparation of a bigger 0.2 release which will refactor monitoring into plugins, to make it easy for 3rd parties to write their own and/or contribute them to the mainline
- added payroll distribution "plugin", contributed by user Thom, that distributes a delegate's pay amongst the configured accounts (still experimental)

0.2 will include some changes and refactoring to the config.yaml file, too, to make the general architecture more modular and the tools easier to extend. I hope to get it out as soon as possible :)

And by the way, given that btstools.digitalgaia is in standby (has only been active for 2 days :'( ), this means that I won't start working on the backbone proposal. If that's something that you want, please remember to vote for btstools.digitalgaia!
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: BunkerChainLabs-DataSecurityNode on April 13, 2015, 03:34:46 pm
Not sure how I missed this.. but just voted for you.

 +5% to everyone else to case your vote here if you want a technically stronger delegate ecosystem.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on April 14, 2015, 11:43:51 am
Released 0.2, which includes the following changes:

Quote
* now requires python3.4
* API CHANGE: format of the config.yaml file has changed, and you will need to update it.
  Run "bts list" and it should tell you what to fix in your config file. For more details,
  see: http://bts-tools.readthedocs.org/en/latest/config_format.html#nodes-list
* added support for building DVS and BTS client >= 0.9.0
* added support for building PLAY client (pls)
* internal refactoring and modularization of the monitoring plugins

The important thing here is that you need to update your config.yaml file. The tools should tell you what to update, but if you're not too sure, just delete your config.yaml file and restart the tools so that they generate a new, valid one for you.

In particular, make sure that you specify the "missed" and "network_connections" plugins in your monitoring list, as they were done automatically before, but you now have to specify them explicitly. See http://bts-tools.readthedocs.org/en/latest/config_format.html#nodes-list for more info.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on April 22, 2015, 09:06:03 am
Released 0.2.1, which fixes the feeds publishing and display for bts >= 0.9.0 (works on RC1 too)
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: Shentist on April 25, 2015, 04:56:09 am
I am a none coder, but the tools you provide, will make it easier to run a delegate.

Do you have a demo side etc. where some interested voters or delegates can have a look, without the need to install all the stuff?
I think it would make it easier to get elected for you!

Hope you will get elected soon, keep up, i appreciate your work  +5%
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on April 27, 2015, 01:22:01 pm
Do you have a demo side etc. where some interested voters or delegates can have a look, without the need to install all the stuff?

I was thinking of making the install of my seed nodes + bts_tools public, so people can have a look at it, but haven't gotten around to do it yet. It would indeed be a good showcase of what the tools can do.
In the meantime, there are some screenshots here (not as interactive, but hey :) ) : http://bts-tools.readthedocs.org/en/latest/monitor.html#screenshots
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on May 04, 2015, 01:58:45 pm
Released 0.2.2, which fixes slate publishing for bts >= 0.9.0
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on June 03, 2015, 02:13:52 am
I just released 0.2.3 and 0.2.4 with the following changes:

Quote
0.2.4 (2015-06-03)
------------------

* added feed for NASDAQC, NIKKEI, HANGSENG market-pegged assets
* list of visible feeds can be configured in config.yaml file


0.2.3 (2015-06-02)
------------------

* added feed for SHENZHEN market-pegged asset
* fixed payroll plugin (contributed by @ThomasFreedman)

for the discussion on new market pegged assets, see https://bitsharestalk.org/index.php/topic,16616.0/all.html

And my delegate btstools.digitalgaia is now in position 102... One more to go!
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: Yao on July 07, 2015, 10:09:52 am
 +5%
Fantastic work. Voted!
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on August 19, 2015, 10:27:03 am
I have now joined forces with Thom of delegate.verbaltech as can be seen here: https://bitsharestalk.org/index.php/topic,13837.msg230185.html#msg230185

I will still post relevant updates to this thread, but news/updates about the backbone will be published in the other thread mainly.
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: 38PTSWarrior on August 19, 2015, 10:46:37 am
Vote for this delegate please
Title: Re: Delegate btstools.digitalgaia: proposal and updates
Post by: wackou on October 13, 2015, 01:24:05 pm
delegates wackou-delegate and btstools.digitalgaia have stopped producing blocks a 9am EST today and have now been shutdown for good, may they rest in peace :) Like the phoenix, a new witness will be reborn from their ashes (to be announced very soon)

What an adventure the past year has been! I have learnt a lot about crypto-currencies and am very proud to be part of the BitShares community, which I hold in the highest respect because of the way that technical excellence is valued here first and foremost (instead of politics), for the openness of the core devs and for all the different opinions of everyone here (I could try to list all of you but Ander did a superb job here: https://bitsharestalk.org/index.php?topic=18455.0, so I'll leave it there).

Eagerly waiting for the impending release of BitShares 2.0. The future has never looked more interesting!