Author Topic: Tutorial about running a delegate  (Read 7670 times)

0 Members and 1 Guest are viewing this topic.

Offline wackou

Documentation and tutorial are now mostly complete, there are no missing sections and lots of quirks have been worked out (Thanks Thom for going through it and reporting the issues).

Writing documentation is of course a never-ending task, but I feel it is now good enough that people with no a priori knowledge of the BitShares ecosystem should be able to dive into it without too many issues.
Please vote for witness wackou! More info at http://digitalgaia.io

Offline wackou

Also, there's a big difference between the root user's .bashrc & a normal user, the root user's is far smaller.

ah, maybe that's the issue, I believe this is only active for a normal user. Actually, that's the whole point of the virtualenvs, is that you don't need to install anything as root, so you don't mess up your system. Everything is contained in your project, which you should run as a normal user anyway. The only thing really needed to install as root with apt-get are: python3, virtualenvwrapper, and maybe python3-pip. All the rest, creation of the virtualenv and pip installing stuff inside should be run as a normal user. I will also add this to the doc.

I'm also not sure what you are calling the reference doc.

well, just the main doc about the tools (http://bts-tools.readthedocs.org/) which is not the tutorial (http://bts-tools.readthedocs.org/en/latest/howto.html)

Hope you got your setup working in the end, and thanks for bearing with me while I refine the documentation.
Please vote for witness wackou! More info at http://digitalgaia.io

Offline Thom

I didn't alter anything that was in the default .bashrc file, I simply appended one line to it. Something is not working in that bit of shell code you posted, I saw that code too but it doesn't look like it handles the scripts in the /etc/bash_completions.d/ folder, at least not directly. There's another piece of the puzzle that must loop thru the /etc folder and combine those scripts into the one sourced by the code you highlighted.

Also, there's a big difference between the root user's .bashrc & a normal user, the root user's is far smaller.

I was very careful in reading your tut and I never saw the -p switch used. I'm also not sure what you are calling the reference doc.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline wackou

In my case the .bashrc file is simple, small and clean so that is unlikely the issue.

maybe that was the issue, debian provides you with a default .bashrc that contains this towards the end.

Code: [Select]
# enable programmable completion features (you don't need to enable
# this, if it's already enabled in /etc/bash.bashrc and /etc/profile
# sources /etc/bash.bashrc).
if ! shopt -oq posix; then
  if [ -f /usr/share/bash-completion/bash_completion ]; then
    . /usr/share/bash-completion/bash_completion
  elif [ -f /etc/bash_completion ]; then
    . /etc/bash_completion
  fi
fi

If you overwrote it with yours and it doesn't have this, then yes, you need to call the virtualenvwrapper yourself there

Probably not the correct way to resolve whatever the actual problem is but it works and I have no time to dig into it anymore.

If it works, it works :)

However it is still necessary to use the -p parameter to get the correct python version:

Code: [Select]
mkvirtualenv -p`which python3` bts_tools

This is the correct way to do it, it has been like that in the tutorial from the beginning if I recall correctly (not in the install section of the reference doc, though)

Sorry for all the mess with python and the virtualenv, I realize now it's maybe not that easy to setup correctly if you don't know how to, but be glad because a working virtualenvwrapper will save you many headaches in the future, I can guarantee that. Don't hesitate to pm me if you have any more problems.
Please vote for witness wackou! More info at http://digitalgaia.io

Offline Thom

All this should be unnecessary, once you install the virtualenvwrapper package, (apt-get install virtualenvwrapper), next time you open a shell (but you need to open a new one, you probably stayed in the same one and that's why you had to do everything manually) you will have virtualenvwrapper functions in your shell, no need to activate anything manually (this is done through the /etc/bash_completion.d/virtualenvwrapper file)

Logging in from a cold boot I received errors that /usr/share/virtualenvwrapper/virtualenvwrapper_lazy.sh did not exist which it didn't. I removed the sourcing of the script from my .bashrc file and then looked at the /etc/bash_completion.d/ folder to see if virtualenvwrapper was present and it was. It was hard coded to use the lazy version (USE_FULL=no).

I removed the package with apt-get remove and installed it with pip3 install last night so neither script was present in /usr/share/virtualenvwrapper, so why is the script still present in /etc/bash_completion.d/? Poor clean up on the part of the apt-get removal process or is it related to using pip3 to install the package?.

I corrected this by reinstalling with apt-get install virtualenvwrapper. No errors occured. Then I logged out of that shell & started a new one, but still mkvirtualenv was not to be found in my path. So logged out of the entire session & back in, still not in my path. Checked both /etc/bash_completion.d/ and /usr/share/virtualenvwrapper/ and they looked OK. The contents of that file are:

Code: [Select]
# bash completion snippet for virtualenvwrapper
USE_FULL=no

if [ "$USE_FULL" = "yes" ]; then
    . /usr/share/virtualenvwrapper/virtualenvwrapper.sh
else
    . /usr/share/virtualenvwrapper/virtualenvwrapper_lazy.sh
fi

Inspecting the environment with set showed 2 virtualenvwrapper settings:

Code: [Select]
VIRTUALENVWRAPPER_SCRIPT=/usr/share/virtualenvwrapper/virtualenvwrapper.sh

_VIRTUALENVWRAPPER_API=' mkvirtualenv rmvirtualenv lsvirtualenv showvirtualenv workon add2virtualenv cdsitepackages cdvirtualenv lssitepackages toggleglobalsitepackages cpvirtualenv setvirtualenvproject mkproject cdproject mktmpenv'
root@Jessie:/etc/bash_completion.d#

However the mkvirtualenv command still fails with command not found.

I'm not the only one that has problems with the wrapper script. Look at the comments here for example: http://stackoverflow.com/questions/25774829/mkvirtualenv-command-not-found

In my case the .bashrc file is simple, small and clean so that is unlikely the issue.

The workaround I'm using was to add this to the end of my .bashrc:

Code: [Select]
source /etc/bash_completion.d/virtualenvwrapper
Probably not the correct way to resolve whatever the actual problem is but it works and I have no time to dig into it anymore.

Could it be that /etc/bash_completion.d scripts are probably executed on termination of the shell, not the start of it? If so that would explain it. From several quick google searches it seems others use that to run scripts upon shell startup, so scratch that idea.

However it is still necessary to use the -p parameter to get the correct python version:

Code: [Select]
mkvirtualenv -p`which python3` bts_tools
It looks like I've overcome the last major issue and the build is working well, at least it has gotten to 70% complete now so I think I'm good. Onto the next series of steps...
« Last Edit: February 02, 2015, 07:33:11 pm by Thom »
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline wackou

The last thing I did before going to bed last night was kick off a build which failed. I think I have the tools & virtualenv installed and working correctly now, but obviously I need to track down the failure.

what is the error you get? Probably it should be something easy to fix now that you've made it this far...

(will reply to your other comments in a while, no time right now)
Please vote for witness wackou! More info at http://digitalgaia.io

Offline Thom

Thanks Thom for your comments. I have already integrated some of those comments, still need to make the python3 issue clearer.

Updated doc can be found at: http://bts-tools.readthedocs.org/

There are 2 main parts written now: reference doc and the tutorial. Reference doc should be more complete, while the tutorial is more step-by-step. For instance, ref doc doesn't mention (yet) the use of virtualenv, as it is not absolutely required for the tools, but the tutorial still shows how to do it. I will keep updating both until there is very little room for confusion left.

Which is the ref doc? Is it this: http://bts-tools.readthedocs.org/en/latest/cmdline.html?

I expected the tutorial (the step by step guide) to be something most people could follow and result in a working client / gui / delegate. I haven't found that to be the case so far. The devil is in the details my friend! I wholeheartedly applaud your efforts and I'm devoting the time to help you refine your tutorial so it may benefit others.

Linux can be tough to document such a step by step guide tho. There are many variations. That's why I wanted to go thru your tutorial in a very meticulous and detailed manor, including the base OS you're using. Here's another example of the devilish details I'm referring to - on this page http://bts-tools.readthedocs.org/en/latest/install.html you state the first place to start is with:

Code: [Select]
$ pip3 install bts_tools
but if the python3-pip package isn't already installed that step will fail, which it in fact does on Debian Jessie. Logging out then back in after installing virtualenvwrapper is another example, which apparently is overlooked by many, based on the number of websites I looked at that discussed the installation of it that fail to mention it, just as you did. Sounds more like Windows than linux!

I'm not sure how much more time I can devote to figuring out all of the nuances to using the bts_tools, I'm falling behind on other things I need to get done. Perhaps by the end of the day today this learning experience will yield satisfaction.

The last thing I did before going to bed last night was kick off a build which failed. I think I have the tools & virtualenv installed and working correctly now, but obviously I need to track down the failure. I decided to take a full backup image of the disk and figure out the problem today. I should have made the image just after installing the fresh copy of Jessie before installing any other packages, but I didn't.

What I meant by secure email is a way to mask the origin of it, so the delegate node can remain hidden, such as through a VPN. That may not be important to many now but it will provide better security if the node's location can't be easily tracked by the origins of the notification emails. It's a minor concern right now for sure.
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline wackou

Looking forward to enhancements that provide for more robust remote management and monitoring using ssh and secure email notification.

Not sure what you mean by secure email notification... The tools can already send an email to you using a provider you configured, using TLS. Did you have something else in mind?
Please vote for witness wackou! More info at http://digitalgaia.io

Offline wackou

I'm having trouble due to the mixture of python 2.7 & python 3. Your instructions need to take this into account. For example, the mkvirtualenv cmd is not in the path, and pip will reference the default version for python 2.7 not python 3.

I had to add python3-pip to the long list of packages you cite as dependencies. Also, to use the virtualenvwrapper script it must be sourced, such as in the ~/.bashrc profile. The location of the wrapper script can be found with something like:

Code: [Select]
pushd /; find . | grep virtualenvwrapper.sh; popd

Then add that path to the end of the ~/.bashrc file or your favorite shell:

Code: [Select]
source <path to virtualenvwrapper.sh>

Even still that may not produce the correct results, but I finally found a definitive solution for using the virtualenvwrapper so that it utilizes the correct python interpreter. You must use the -p parameter to mkvirtualenv to be certain the target virtual environment is using python3, otherwise it's going to be the system default which is probably python2.x.

All this should be unnecessary, once you install the virtualenvwrapper package, (apt-get install virtualenvwrapper), next time you open a shell (but you need to open a new one, you probably stayed in the same one and that's why you had to do everything manually) you will have virtualenvwrapper functions in your shell, no need to activate anything manually (this is done through the /etc/bash_completion.d/virtualenvwrapper file)

You might consider adding a version arg to your bts tool which will make it easy for users to report the version people may have issues with.
definitely needed, will do (https://github.com/wackou/bts_tools/issues/19)
Please vote for witness wackou! More info at http://digitalgaia.io

Offline wackou

Thanks Thom for your comments. I have already integrated some of those comments, still need to make the python3 issue clearer.

Updated doc can be found at: http://bts-tools.readthedocs.org/

There are 2 main parts written now: reference doc and the tutorial. Reference doc should be more complete, while the tutorial is more step-by-step. For instance, ref doc doesn't mention (yet) the use of virtualenv, as it is not absolutely required for the tools, but the tutorial still shows how to do it. I will keep updating both until there is very little room for confusion left.
Please vote for witness wackou! More info at http://digitalgaia.io

Offline Thom

I'm having trouble due to the mixture of python 2.7 & python 3. Your instructions need to take this into account. For example, the mkvirtualenv cmd is not in the path, and pip will reference the default version for python 2.7 not python 3.

I had to add python3-pip to the long list of packages you cite as dependencies. Also, to use the virtualenvwrapper script it must be sourced, such as in the ~/.bashrc profile. The location of the wrapper script can be found with something like:

Code: [Select]
pushd /; find . | grep virtualenvwrapper.sh; popd

Then add that path to the end of the ~/.bashrc file or your favorite shell:

Code: [Select]
source <path to virtualenvwrapper.sh>

Even still that may not produce the correct results, but I finally found a definitive solution for using the virtualenvwrapper so that it utilizes the correct python interpreter. You must use the -p parameter to mkvirtualenv to be certain the target virtual environment is using python3, otherwise it's going to be the system default which is probably python2.x.

Code: [Select]
mkvirtualenv -p`which python3` bts_tools
I noticed you changed the docs & removed the references to virtualenv. I've wasted alot of time screwing around with this on fresh installs of Debian Jessie and looked at plenty of websites about python & virtualenv/wrapper. It's quite confusing and rather difficult to get it setup right which may be why you removed it from the docs. But your reason for using virtualenvs is valid and justified; it's a good practice.

I cannot tell you if the -p solution will work on a fresh Debian Jessie as I have tried so many variations of apt-get install / pip3 install (and their remove / uninstall variants) that I'm no longer sure of the context. Hopefully you can distill this info and add to your docs to help others avoid the pitfalls I ran into.

You might consider adding a version arg to your bts tool which will make it easy for users to report the version people may have issues with.


« Last Edit: February 02, 2015, 08:59:03 am by Thom »
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline Thom

Actually, the recommended way to install Debian is not to download the DVDs (which contain all the packages in their repos), but the single CD (if you have a slow connection and want to install it multiple times) or the Netinst (highly recommended) which contain only the base system, and then from there you complete the installation by downloading updated packages directly.

See here to get the netinst ISO (and others): https://www.debian.org/devel/debian-installer/

I tried the netinst first but ran into trouble. It may have been related to using RMPrepUSB to boot the iso from a USB hard drive. I couldn't find a single install CD for Jessie on the debian site or that's what I would have gone with for sure.

Anyway, it took much longer but I got it done. I'm rather surprised by the distro, as it looks extremely similar to the OpenSuSE 12.2 distro I run on another system. I like it very much!!! Debian is long overdue for a face lift.

I'm basing that first impression mostly from the GUI presentation, which isn't what I intend to use on the target VPS (it will be headless to conserve disk & memory resources). I may reconsider using Debian Jessie as my favoured linux distro rather than Mint, tho to be fair I will check out the Debian flavour of Mint first and assess it's long term viability and support.

So now that I have a VM with Jessie I will go thru the rest of your tutorial. I will also use your bts tools to build a client for DevShares. I noticed the config.yaml lists the same github repository for DevShares as the main BTS repo. I may have done that myself by mistake but you should check it out on your end and fix it if necessary.

Thanks for all your efforts to make this easier for people. Looking forward to enhancements that provide for more robust remote management and monitoring using ssh and secure email notification.

Take care!
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline wackou

It's been a few years since I've installed a Debian distro. It only required 1 DVD before, but it's up to 3 now! It took almost 14 hours to download the ISOs with BitTorrent.

Actually, the recommended way to install Debian is not to download the DVDs (which contain all the packages in their repos), but the single CD (if you have a slow connection and want to install it multiple times) or the Netinst (highly recommended) which contain only the base system, and then from there you complete the installation by downloading updated packages directly.

See here to get the netinst ISO (and others): https://www.debian.org/devel/debian-installer/

I began to follow your tut to upgrade Wheezy to Jessie and found out your repository link doesn't work. The link you provided to mirrors.gandi.net resolves to 215.70.184.233 but apt can't connect to it(I can ping that address just fine).

Oops sorry, yeah that was a copy-paste oversight (because I'm on gandi, those mirrors are blazing fast, but not available outside), you should definitely use the official debian repos. Actually, if you install directly from the Jessie installer, you don't have to do anything with our apt sources, they already point to the correct ones. I did the Wheezy -> Jessie upgrade way as Gandi doesn't offer Jessie images yet, but installing from the Jessie installer is the better solution.

I'm curious why you decided to go with Jessie rather than Wheezy.

I did it mainly for 2 reasons:
 - Jessie is already frozen (ie: they only accept bugfixes until the final release, no new packages or updates), so it's very similar to what will be the stable release (from experience, only minor packages are still waiting for bug fixes, and I never had a problem with a RC in Debian in 10 years)
 - In Wheezy, gcc, cmake and boost are all outdated versions that cannot be used to compile the BitShares client. So you would have to update those anyway by taking them from testing, which ends up making your distro look more like Jessie than Wheezy anyway.
So Jessie it is :)

Thanks for all the proofreading ;)
Please vote for witness wackou! More info at http://digitalgaia.io

Offline Thom

I decided to follow your tutorial as closely as possible, and so I set out to install debian jessie on a KVM VM on my local system. It was my first encounter with KVM and so far I really like it. My host is a Dell quad core i3 system running Linux Mint 17 with 80GB disk & 4GB RAM.

It's been a few years since I've installed a Debian distro. It only required 1 DVD before, but it's up to 3 now! It took almost 14 hours to download the ISOs with BitTorrent.

Had a minor hiccup in that I installed directly from the ISO image but when the installer asked for the second disk there was no way to provide it. So I selected the network as the update source and completed the installation.

I began to follow your tut to upgrade Wheezy to Jessie and found out your repository link doesn't work. The link you provided to mirrors.gandi.net resolves to 215.70.184.233 but apt can't connect to it(I can ping that address just fine).

I presume you're using a gandi.net VPS and the mirror is available internally only. So I had to find others. It took a little playing around but these seem to be working:
Quote
deb http://http.debian.net/debian jessie main contrib non-free
deb http://security.debian.org/ jessie/updates main contrib non-free

I'm curious why you decided to go with Jessie rather than Wheezy. It's not generally good practice to run a production server on an unreleased beta OS version. I do however actually like the debian "rolling release" approach. It's slower but it avoids many of the typical upgrade issues of other distros like Ubuntu. I haven't tried it yet but Mint also offers a version based on debian. The reason I haven't is b/c it's always behind the ubuntu release schedule and I'm not sure if Mint will continue the debian offering. I have no info they won't, but it is quite a high overhead to support two releases as they now do.

I'll keep you posted as I follow along and let you know if I encounter any other issues.
« Last Edit: February 01, 2015, 01:50:30 am by Thom »
Injustice anywhere is a threat to justice everywhere - MLK |  Verbaltech2 Witness Reports: https://bitsharestalk.org/index.php/topic,23902.0.html

Offline wackou

I can't quite see the full picture about the virtualenv however. I understand the sandbox concept, but unless I'm mistaken doesn't the client need to run within the virtualenv context (i.e. directory created by the virtualenv cmd), or is it only a sandbox for python includes/libs/imports?

it is only for the python libs/imports, nothing else is changed (basically what it does it to set the $PYTHONPATH environment variable so that libs in the virtualenv are imported first). This has absolutely no effect whatsoever on running the BitShares client, or anything that doesn't use python.


When you create the virtualenv & source it, does that extend the sandbox outside of the virtualenv folder?

Yes, the sandbox is either active or inactive, regardless of which directory you are in. That's also one reason I like virtualenvwrapper a lot, is that it "hides" the location of the virtualenv directory, and the only thing you ever have to think of is to activate it ("workon myvirtualenv") or deactivate it ("deactivate"), and you can call this from any folder you are in. Makes it so much simpler to interiorize

Also, been trying to find info on "seed nodes"; what are they? What is their function? There's no info in the wiki.bitshares.org about them.

seed nodes are just normal nodes, but they have an IP address that is publicly known, and are used by people connecting to the network when they launch the client: they need an entry point to the network, and that is the function provided by seed nodes. This places a few desired properties on them: high uptime, lots of connections, etc. To sum up, they are like full nodes in bitcoin + public IP.
Please vote for witness wackou! More info at http://digitalgaia.io