BitShares Forum
Main => Stakeholder Proposals => Topic started by: wackou on January 23, 2015, 02:40:48 pm
-
Hi all,
(tl;dr: tutorial is at: http://bts-tools.readthedocs.org/en/latest/howto.html)
I completed the first version of the tutorial I had planned to write for some time on running a delegate.
The tutorial starts from scratch, and guides you in installing Debian on a VPS, installing the BTS Tools I developed, and then using them to build, run and monitor the BitShares client. Hopefully it is clear enough, and I hope this will motivate more people to look into the BTS Tools.
Tutorial can be found at: http://bts-tools.readthedocs.org/en/latest/howto.html
Note also that this is a first version, it is not 100% complete (but mostly), and any comments/suggestions for improvements are welcome.
I have also started to write more doc in general for the tools (not only the tutorial) but this is still very much a work in progress. Doc is at: http://bts-tools.readthedocs.org/
Please remember to also vote for my delegate: btstools.digitalgaia, it still needs ~5% to get voted in. For a 30%-only delegate this gets you:
- continued development and funding of the BTS Tools (https://bitsharestalk.org/index.php?topic=12534)
- more documentation like this one
- 6 seed nodes/chain servers (2 US, 2 Europe, 2 Asia) for the BitShares and the DevShares network
Hope you like it!
-
Your tools are incredible .. I wonder why you are still not voted in .. at least to get a pay for your work ..
having 6 seed nodes is worth being voted in anyway
-
+5% +5% +5%
-
Really nice!
If you could expand the moniting app tutorials a bit that would be great!
Things I would like to know:
How to install it?
What is a WSGI server?
Can I use it with a remote server using ssh key?
-
Thanks guys for the kind words!
If you could expand the moniting app tutorials a bit that would be great!
will do, that's definitely the part that I think is useful so I will try to make it clearer
How to install it?
you get all in the same package, so
pip install bts_tools
is all you ever have to do. It is really recommended to take 5 mins to understand and use virtualenvs, though, it will save you a lot of headaches in the future. See http://bts-tools.readthedocs.org/en/latest/howto_install.html
What is a WSGI server?
the monitoring tools are run by default as a python script, and serving the http requests is done by an embedded http server written in python. This is nice for development, but is usually not recommended for production.
In production, the usual way of doing this is to have a "real" webserver, such as Apache or nginx do the http part, and ask the python code to give it just the contents of the page, but not serve it. This has many advantages, such as a better security model, more features (eg: SSL if supported by your webserver, etc.)
WSGI is the standard allowing a webserver to execute directly python code (think: mod_php, but for python)
Can I use it with a remote server using ssh key?
if you mean having the client run on a machine, and the monitoring tools run on a different one but monitoring the client through ssh, then yes, it is possible. I would have to review this part, though, as I did it with convenience in mind first, without taking into account the security too much. There is nothing in the doc yet, will get on it too.
-
This is fantastic work!
-
This is fantastic work!
+5% I concur. Voted!
-
Your tools are incredible .. I wonder why you are still not voted in .. at least to get a pay for your work ..
having 6 seed nodes is worth being voted in anyway
+5% +5% +5%
@all: Come on pls show some love for wackou as delegate!
Dan, Nikolai etc... give him your vote....NOW!!! :)
-
i'll include this tut into delegate page of ravamped .org site - if i'll get your GO - and link to your delegate proposal here!
-
i'll include this tut into delegate page of ravamped .org site - if i'll get your GO - and link to your delegate proposal here!
Sure, go ahead, I'd love it! I see that the countdown is extended and is now a week, that'll give me a bit more time to fine tune it.
-
I only read a little so far but this this seems like a really useful guide.
The easier we can make it to run delegates obviously the better, great work.
-
Fantastic work.
Riverhead is doing similar things, according to this https://bitsharestalk.org/index.php?topic=10861.0
So are you going to compete with him? :P
-
So are you going to compete with him? :P
I consider that I have a profile very similar to Riverhead, that's true, but I wouldn't call it competition :)
As far as I can tell, Riverhead is more active on the forums, and I spend most of my time developing the bts_tools instead, but we have the same mission: ensuring that the network is as robust as possible. In any case, I think that in the first stages of development of a project, it is better to let creative initiatives grow organically and freely, and then take the better ideas between them and merge them into something useful and stable.
You could compare this to writing code as fast as possible to make it work first, and only then refactor it to make it clean. I like this way better because when you refactor and do a proper design, you already have the experience of making it work, which avoids overlooking some aspects that you could have missed had you started thinking about the problem beforehand.
I do foresee a future where Riverhead and myself, but also Xeroc, Emski, etc get together to shape some common tools / docs / etc. which represent the consensus about what is the absolute best way to manage the network infrastructure (ie: delegate nodes, seed nodes, backup solutions, DDoS prevention, etc). For instance, I can see a not so distant future where we take Xeroc's, Alt's and my python code to create a standard python BitShares wrapper API too (like this: https://bitsharestalk.org/index.php?topic=13685, but more complete)
However, before reaching a consensus, I believe we first need to come up with ideas and code as freely as possible. Friendly competition, if you will ;)
-
btw, I plan to integrate some of the comments I already received on the tutorial, hopefully over this weekend. This will fix some missing parts in the docs and rough edges about the installation. Stay tuned!
-
I'm new to the delegate scene and I've been working with wackou's bts_tools. I love the simplicity of their use, and his tutorial is a very good start. I had some difficulties but I don't blame the tools for them, I think I had some issues on my test rig. I'll be focused on using the tools over the weekend so I will offer my perspective as I go.
Most excellent work tho! This guy needs more votes!
I can't quite see the full picture about the virtualenv however. I understand the sandbox concept, but unless I'm mistaken doesn't the client need to run within the virtualenv context (i.e. directory created by the virtualenv cmd), or is it only a sandbox for python includes/libs/imports?
When you create the virtualenv & source it, does that extend the sandbox outside of the virtualenv folder?
Also, been trying to find info on "seed nodes"; what are they? What is their function? There's no info in the wiki.bitshares.org about them.
-
I can't quite see the full picture about the virtualenv however. I understand the sandbox concept, but unless I'm mistaken doesn't the client need to run within the virtualenv context (i.e. directory created by the virtualenv cmd), or is it only a sandbox for python includes/libs/imports?
it is only for the python libs/imports, nothing else is changed (basically what it does it to set the $PYTHONPATH environment variable so that libs in the virtualenv are imported first). This has absolutely no effect whatsoever on running the BitShares client, or anything that doesn't use python.
When you create the virtualenv & source it, does that extend the sandbox outside of the virtualenv folder?
Yes, the sandbox is either active or inactive, regardless of which directory you are in. That's also one reason I like virtualenvwrapper a lot, is that it "hides" the location of the virtualenv directory, and the only thing you ever have to think of is to activate it ("workon myvirtualenv") or deactivate it ("deactivate"), and you can call this from any folder you are in. Makes it so much simpler to interiorize
Also, been trying to find info on "seed nodes"; what are they? What is their function? There's no info in the wiki.bitshares.org about them.
seed nodes are just normal nodes, but they have an IP address that is publicly known, and are used by people connecting to the network when they launch the client: they need an entry point to the network, and that is the function provided by seed nodes. This places a few desired properties on them: high uptime, lots of connections, etc. To sum up, they are like full nodes in bitcoin + public IP.
-
I decided to follow your tutorial as closely as possible, and so I set out to install debian jessie on a KVM VM on my local system. It was my first encounter with KVM and so far I really like it. My host is a Dell quad core i3 system running Linux Mint 17 with 80GB disk & 4GB RAM.
It's been a few years since I've installed a Debian distro. It only required 1 DVD before, but it's up to 3 now! It took almost 14 hours to download the ISOs with BitTorrent.
Had a minor hiccup in that I installed directly from the ISO image but when the installer asked for the second disk there was no way to provide it. So I selected the network as the update source and completed the installation.
I began to follow your tut to upgrade Wheezy to Jessie and found out your repository link doesn't work. The link you provided to mirrors.gandi.net resolves to 215.70.184.233 but apt can't connect to it(I can ping that address just fine).
I presume you're using a gandi.net VPS and the mirror is available internally only. So I had to find others. It took a little playing around but these seem to be working:
deb http://http.debian.net/debian jessie main contrib non-free
deb http://security.debian.org/ jessie/updates main contrib non-free
I'm curious why you decided to go with Jessie rather than Wheezy. It's not generally good practice to run a production server on an unreleased beta OS version. I do however actually like the debian "rolling release" approach. It's slower but it avoids many of the typical upgrade issues of other distros like Ubuntu. I haven't tried it yet but Mint also offers a version based on debian. The reason I haven't is b/c it's always behind the ubuntu release schedule and I'm not sure if Mint will continue the debian offering. I have no info they won't, but it is quite a high overhead to support two releases as they now do.
I'll keep you posted as I follow along and let you know if I encounter any other issues.
-
It's been a few years since I've installed a Debian distro. It only required 1 DVD before, but it's up to 3 now! It took almost 14 hours to download the ISOs with BitTorrent.
Actually, the recommended way to install Debian is not to download the DVDs (which contain all the packages in their repos), but the single CD (if you have a slow connection and want to install it multiple times) or the Netinst (highly recommended) which contain only the base system, and then from there you complete the installation by downloading updated packages directly.
See here to get the netinst ISO (and others): https://www.debian.org/devel/debian-installer/
I began to follow your tut to upgrade Wheezy to Jessie and found out your repository link doesn't work. The link you provided to mirrors.gandi.net resolves to 215.70.184.233 but apt can't connect to it(I can ping that address just fine).
Oops sorry, yeah that was a copy-paste oversight (because I'm on gandi, those mirrors are blazing fast, but not available outside), you should definitely use the official debian repos. Actually, if you install directly from the Jessie installer, you don't have to do anything with our apt sources, they already point to the correct ones. I did the Wheezy -> Jessie upgrade way as Gandi doesn't offer Jessie images yet, but installing from the Jessie installer is the better solution.
I'm curious why you decided to go with Jessie rather than Wheezy.
I did it mainly for 2 reasons:
- Jessie is already frozen (ie: they only accept bugfixes until the final release, no new packages or updates), so it's very similar to what will be the stable release (from experience, only minor packages are still waiting for bug fixes, and I never had a problem with a RC in Debian in 10 years)
- In Wheezy, gcc, cmake and boost are all outdated versions that cannot be used to compile the BitShares client. So you would have to update those anyway by taking them from testing, which ends up making your distro look more like Jessie than Wheezy anyway.
So Jessie it is :)
Thanks for all the proofreading ;)
-
Actually, the recommended way to install Debian is not to download the DVDs (which contain all the packages in their repos), but the single CD (if you have a slow connection and want to install it multiple times) or the Netinst (highly recommended) which contain only the base system, and then from there you complete the installation by downloading updated packages directly.
See here to get the netinst ISO (and others): https://www.debian.org/devel/debian-installer/
I tried the netinst first but ran into trouble. It may have been related to using RMPrepUSB to boot the iso from a USB hard drive. I couldn't find a single install CD for Jessie on the debian site or that's what I would have gone with for sure.
Anyway, it took much longer but I got it done. I'm rather surprised by the distro, as it looks extremely similar to the OpenSuSE 12.2 distro I run on another system. I like it very much!!! Debian is long overdue for a face lift.
I'm basing that first impression mostly from the GUI presentation, which isn't what I intend to use on the target VPS (it will be headless to conserve disk & memory resources). I may reconsider using Debian Jessie as my favoured linux distro rather than Mint, tho to be fair I will check out the Debian flavour of Mint first and assess it's long term viability and support.
So now that I have a VM with Jessie I will go thru the rest of your tutorial. I will also use your bts tools to build a client for DevShares. I noticed the config.yaml lists the same github repository for DevShares as the main BTS repo. I may have done that myself by mistake but you should check it out on your end and fix it if necessary.
Thanks for all your efforts to make this easier for people. Looking forward to enhancements that provide for more robust remote management and monitoring using ssh and secure email notification.
Take care!
-
I'm having trouble due to the mixture of python 2.7 & python 3. Your instructions need to take this into account. For example, the mkvirtualenv cmd is not in the path, and pip will reference the default version for python 2.7 not python 3.
I had to add python3-pip to the long list of packages you cite as dependencies. Also, to use the virtualenvwrapper script it must be sourced, such as in the ~/.bashrc profile. The location of the wrapper script can be found with something like:
pushd /; find . | grep virtualenvwrapper.sh; popd
Then add that path to the end of the ~/.bashrc file or your favorite shell:
source <path to virtualenvwrapper.sh>
Even still that may not produce the correct results, but I finally found a definitive solution for using the virtualenvwrapper so that it utilizes the correct python interpreter. You must use the -p parameter to mkvirtualenv to be certain the target virtual environment is using python3, otherwise it's going to be the system default which is probably python2.x.
mkvirtualenv -p`which python3` bts_tools
I noticed you changed the docs & removed the references to virtualenv. I've wasted alot of time screwing around with this on fresh installs of Debian Jessie and looked at plenty of websites about python & virtualenv/wrapper. It's quite confusing and rather difficult to get it setup right which may be why you removed it from the docs. But your reason for using virtualenvs is valid and justified; it's a good practice.
I cannot tell you if the -p solution will work on a fresh Debian Jessie as I have tried so many variations of apt-get install / pip3 install (and their remove / uninstall variants) that I'm no longer sure of the context. Hopefully you can distill this info and add to your docs to help others avoid the pitfalls I ran into.
You might consider adding a version arg to your bts tool which will make it easy for users to report the version people may have issues with.
-
Thanks Thom for your comments. I have already integrated some of those comments, still need to make the python3 issue clearer.
Updated doc can be found at: http://bts-tools.readthedocs.org/
There are 2 main parts written now: reference doc and the tutorial. Reference doc should be more complete, while the tutorial is more step-by-step. For instance, ref doc doesn't mention (yet) the use of virtualenv, as it is not absolutely required for the tools, but the tutorial still shows how to do it. I will keep updating both until there is very little room for confusion left.
-
I'm having trouble due to the mixture of python 2.7 & python 3. Your instructions need to take this into account. For example, the mkvirtualenv cmd is not in the path, and pip will reference the default version for python 2.7 not python 3.
I had to add python3-pip to the long list of packages you cite as dependencies. Also, to use the virtualenvwrapper script it must be sourced, such as in the ~/.bashrc profile. The location of the wrapper script can be found with something like:
pushd /; find . | grep virtualenvwrapper.sh; popd
Then add that path to the end of the ~/.bashrc file or your favorite shell:
source <path to virtualenvwrapper.sh>
Even still that may not produce the correct results, but I finally found a definitive solution for using the virtualenvwrapper so that it utilizes the correct python interpreter. You must use the -p parameter to mkvirtualenv to be certain the target virtual environment is using python3, otherwise it's going to be the system default which is probably python2.x.
All this should be unnecessary, once you install the virtualenvwrapper package, (apt-get install virtualenvwrapper), next time you open a shell (but you need to open a new one, you probably stayed in the same one and that's why you had to do everything manually) you will have virtualenvwrapper functions in your shell, no need to activate anything manually (this is done through the /etc/bash_completion.d/virtualenvwrapper file)
You might consider adding a version arg to your bts tool which will make it easy for users to report the version people may have issues with.
definitely needed, will do (https://github.com/wackou/bts_tools/issues/19)
-
Looking forward to enhancements that provide for more robust remote management and monitoring using ssh and secure email notification.
Not sure what you mean by secure email notification... The tools can already send an email to you using a provider you configured, using TLS. Did you have something else in mind?
-
Thanks Thom for your comments. I have already integrated some of those comments, still need to make the python3 issue clearer.
Updated doc can be found at: http://bts-tools.readthedocs.org/
There are 2 main parts written now: reference doc and the tutorial. Reference doc should be more complete, while the tutorial is more step-by-step. For instance, ref doc doesn't mention (yet) the use of virtualenv, as it is not absolutely required for the tools, but the tutorial still shows how to do it. I will keep updating both until there is very little room for confusion left.
Which is the ref doc? Is it this: http://bts-tools.readthedocs.org/en/latest/cmdline.html (http://bts-tools.readthedocs.org/en/latest/cmdline.html)?
I expected the tutorial (the step by step guide) to be something most people could follow and result in a working client / gui / delegate. I haven't found that to be the case so far. The devil is in the details my friend! I wholeheartedly applaud your efforts and I'm devoting the time to help you refine your tutorial so it may benefit others.
Linux can be tough to document such a step by step guide tho. There are many variations. That's why I wanted to go thru your tutorial in a very meticulous and detailed manor, including the base OS you're using. Here's another example of the devilish details I'm referring to - on this page http://bts-tools.readthedocs.org/en/latest/install.html (http://bts-tools.readthedocs.org/en/latest/install.html) you state the first place to start is with:
$ pip3 install bts_tools
but if the python3-pip package isn't already installed that step will fail, which it in fact does on Debian Jessie. Logging out then back in after installing virtualenvwrapper is another example, which apparently is overlooked by many, based on the number of websites I looked at that discussed the installation of it that fail to mention it, just as you did. Sounds more like Windows than linux!
I'm not sure how much more time I can devote to figuring out all of the nuances to using the bts_tools, I'm falling behind on other things I need to get done. Perhaps by the end of the day today this learning experience will yield satisfaction.
The last thing I did before going to bed last night was kick off a build which failed. I think I have the tools & virtualenv installed and working correctly now, but obviously I need to track down the failure. I decided to take a full backup image of the disk and figure out the problem today. I should have made the image just after installing the fresh copy of Jessie before installing any other packages, but I didn't.
What I meant by secure email is a way to mask the origin of it, so the delegate node can remain hidden, such as through a VPN. That may not be important to many now but it will provide better security if the node's location can't be easily tracked by the origins of the notification emails. It's a minor concern right now for sure.
-
The last thing I did before going to bed last night was kick off a build which failed. I think I have the tools & virtualenv installed and working correctly now, but obviously I need to track down the failure.
what is the error you get? Probably it should be something easy to fix now that you've made it this far...
(will reply to your other comments in a while, no time right now)
-
All this should be unnecessary, once you install the virtualenvwrapper package, (apt-get install virtualenvwrapper), next time you open a shell (but you need to open a new one, you probably stayed in the same one and that's why you had to do everything manually) you will have virtualenvwrapper functions in your shell, no need to activate anything manually (this is done through the /etc/bash_completion.d/virtualenvwrapper file)
Logging in from a cold boot I received errors that /usr/share/virtualenvwrapper/virtualenvwrapper_lazy.sh did not exist which it didn't. I removed the sourcing of the script from my .bashrc file and then looked at the /etc/bash_completion.d/ folder to see if virtualenvwrapper was present and it was. It was hard coded to use the lazy version (USE_FULL=no).
I removed the package with apt-get remove and installed it with pip3 install last night so neither script was present in /usr/share/virtualenvwrapper, so why is the script still present in /etc/bash_completion.d/? Poor clean up on the part of the apt-get removal process or is it related to using pip3 to install the package?.
I corrected this by reinstalling with apt-get install virtualenvwrapper. No errors occured. Then I logged out of that shell & started a new one, but still mkvirtualenv was not to be found in my path. So logged out of the entire session & back in, still not in my path. Checked both /etc/bash_completion.d/ and /usr/share/virtualenvwrapper/ and they looked OK. The contents of that file are:
# bash completion snippet for virtualenvwrapper
USE_FULL=no
if [ "$USE_FULL" = "yes" ]; then
. /usr/share/virtualenvwrapper/virtualenvwrapper.sh
else
. /usr/share/virtualenvwrapper/virtualenvwrapper_lazy.sh
fi
Inspecting the environment with set showed 2 virtualenvwrapper settings:
VIRTUALENVWRAPPER_SCRIPT=/usr/share/virtualenvwrapper/virtualenvwrapper.sh
_VIRTUALENVWRAPPER_API=' mkvirtualenv rmvirtualenv lsvirtualenv showvirtualenv workon add2virtualenv cdsitepackages cdvirtualenv lssitepackages toggleglobalsitepackages cpvirtualenv setvirtualenvproject mkproject cdproject mktmpenv'
root@Jessie:/etc/bash_completion.d#
However the mkvirtualenv command still fails with command not found.
I'm not the only one that has problems with the wrapper script. Look at the comments here for example: http://stackoverflow.com/questions/25774829/mkvirtualenv-command-not-found (http://stackoverflow.com/questions/25774829/mkvirtualenv-command-not-found)
In my case the .bashrc file is simple, small and clean so that is unlikely the issue.
The workaround I'm using was to add this to the end of my .bashrc:
source /etc/bash_completion.d/virtualenvwrapper
Probably not the correct way to resolve whatever the actual problem is but it works and I have no time to dig into it anymore.
Could it be that /etc/bash_completion.d scripts are probably executed on termination of the shell, not the start of it? If so that would explain it. From several quick google searches it seems others use that to run scripts upon shell startup, so scratch that idea.
However it is still necessary to use the -p parameter to get the correct python version:
mkvirtualenv -p`which python3` bts_tools
It looks like I've overcome the last major issue and the build is working well, at least it has gotten to 70% complete now so I think I'm good. Onto the next series of steps...
-
In my case the .bashrc file is simple, small and clean so that is unlikely the issue.
maybe that was the issue, debian provides you with a default .bashrc that contains this towards the end.
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
if [ -f /usr/share/bash-completion/bash_completion ]; then
. /usr/share/bash-completion/bash_completion
elif [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi
fi
If you overwrote it with yours and it doesn't have this, then yes, you need to call the virtualenvwrapper yourself there
Probably not the correct way to resolve whatever the actual problem is but it works and I have no time to dig into it anymore.
If it works, it works :)
However it is still necessary to use the -p parameter to get the correct python version:
mkvirtualenv -p`which python3` bts_tools
This is the correct way to do it, it has been like that in the tutorial from the beginning if I recall correctly (not in the install section of the reference doc, though)
Sorry for all the mess with python and the virtualenv, I realize now it's maybe not that easy to setup correctly if you don't know how to, but be glad because a working virtualenvwrapper will save you many headaches in the future, I can guarantee that. Don't hesitate to pm me if you have any more problems.
-
I didn't alter anything that was in the default .bashrc file, I simply appended one line to it. Something is not working in that bit of shell code you posted, I saw that code too but it doesn't look like it handles the scripts in the /etc/bash_completions.d/ folder, at least not directly. There's another piece of the puzzle that must loop thru the /etc folder and combine those scripts into the one sourced by the code you highlighted.
Also, there's a big difference between the root user's .bashrc & a normal user, the root user's is far smaller.
I was very careful in reading your tut and I never saw the -p switch used. I'm also not sure what you are calling the reference doc.
-
Also, there's a big difference between the root user's .bashrc & a normal user, the root user's is far smaller.
ah, maybe that's the issue, I believe this is only active for a normal user. Actually, that's the whole point of the virtualenvs, is that you don't need to install anything as root, so you don't mess up your system. Everything is contained in your project, which you should run as a normal user anyway. The only thing really needed to install as root with apt-get are: python3, virtualenvwrapper, and maybe python3-pip. All the rest, creation of the virtualenv and pip installing stuff inside should be run as a normal user. I will also add this to the doc.
I'm also not sure what you are calling the reference doc.
well, just the main doc about the tools (http://bts-tools.readthedocs.org/) which is not the tutorial (http://bts-tools.readthedocs.org/en/latest/howto.html)
Hope you got your setup working in the end, and thanks for bearing with me while I refine the documentation.
-
Documentation and tutorial are now mostly complete, there are no missing sections and lots of quirks have been worked out (Thanks Thom for going through it and reporting the issues).
Writing documentation is of course a never-ending task, but I feel it is now good enough that people with no a priori knowledge of the BitShares ecosystem should be able to dive into it without too many issues.