Author Topic: [Worker] Deploy and maintain independent BitShares infrastructure  (Read 4012 times)

0 Members and 1 Guest are viewing this topic.

Offline JohnR

Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #45 on: March 01, 2018, 03:51:25 am »
Asking the tough questions.  I am interested to see the response here.  Bitshares has a very bright future if dialogue like this becomes the norm.

We have summarized our development efforts now in this final report.

Main post is also updated.

Hello to both Blockchain Projects BV and younger accidental brother Blockchain BV, as well as ChainSquad GMBH. Since you've posted your final report, and until now as we all know, there was nobody around to do review on your work as you're doing on others, i was happy to step up and p by doing pro-bono review of your delivery since it would be too centralized that Blockchain Projects BV getting reviewed by itself in another form known as Bitshares Foundation, as it was the case in the past.

[NOTICE]: Since CEO of the Blockchain BV and Blockchain Projects BV is at the same time being hired/funded by the blockchain to represents Bitshares interests as spokeperson, and Dr. Stefan Schießl is software engineer not a Project Manager/CTO, to completely remove any possible conflict of interests, i would request politely that @xeroc (Dr. Ing. Fabian Schuh), as the CTO of the company hired by the blockchain, be collaborative and provide explanations on certain details of this review in transparent way to stake-holders who upvoted his offer/worker proposal as presented first time in early December 2017.

Review of the ORIGINAL PROPOSAL and ROADMAP

http://www.bitshares.foundation/worker/budget/2017-12-infrastructure ( offtopic - you're missing ssl on bitshares.foundation domain )

Highlights and requirements (blue) for clarifications:

1) Distributed Network of BitShares
"The core component of this proposal is the distribution of the BitShares network by means of deploying multiple nodes and offer public API endpoints to improve latency, robustness and availability for the BitShares ecosystem. Since this proposal is funded by the BitShares ecosystem, we limit the use of the APIs to non-commercial activity."

Q) Please explain in which way exactly and/or how you limited the use/usage of the API to non-commercial one ?

2) Public nodes with load balancer in the U.S. on separate servers for redundancy
"a. Deploy two BitShares nodes on a cloud hosting service located in the U.S.
b. Deploy a loadbalancer controlling the traffic to the two new nodes on a cloud
hosting service located in the U.S."

Q) Why balancer and nodes are keep rebooting ? Uptime is not even a day, and downtime is not even an hour. - https://us.nodes.bitshares.ws/stats
Q) us.nodes.bitshares.ws resolves to 96.126.96.40 - Let's encrypt SSL deployed on it doesn't have a valid CA Root certificate, by presenting itself to potential visitor/user as not being trusted by all browsers. At the same time by checking bitshares.eu it does return properly installed SSL. Why ?
Q) On the invoices presented there is no transparent way to know which node is this, please clarify in the future proposals and reports to the community. Since I've been keen enough to do whois against IP, i've discovered that provider is Linode. Can you please clarify which node from the Linode invoices are these 3 ? Load balancer + 2.
Q) Could you please disclose location of the 2 deployed nodes ? In the current infrastructure problem, redundancy was not important as solution on the single location how much decreasing latency and new availability on more locations having balancer in front.


3) Support SSL encryption on the load balancers
"a. Free letsencrypt certificate is used
b. Upgrade to a premium SSL certificate, if necessary"

Comment: Wildcard SSL by Comodo/Positive is roughly around 250$, even though you stated price 800$ (no reason for it). We have inbound price for the same Wildcard a bit over 100$. It would cover all servers/subdomains under that domain for 1 year giving it full encryption and insurance policy. This was offered and discussed publicly in DEX channel earlier 2017. It covers all sub-domains and the main domain without expiry and stronger encryption chain as live certificate. Free SSL is not same as paid SSL, and im sure you are well aware since you are having highest degree of education and positioned as CTO of the Blockchain company managing DAC business after all. We are not running darkweb or home based servers that we can't afford cost of real SSL for all.



REVIEW OF FINAL REPORT

1) Test deployment and loadbalanced nodes
" Internal testing is completed and connection reliability is monitored. There is a
connection issue with the web wallet (“Node out of sync”) that has to be investigated,
but the nodes seem to be unaffected by this issue (no significant load when such
connection issues appear). There might be a connection management problem
inside the backend or the web wallet, but this lies outside the scope of this worker. "

Q) Who is responsible as authority/person to overview and review work being done by Worker Bill Butler ? - Offtopic, but related to the integrity of authority being responsible.

2) Public nodes with load balancer in Asia on separate servers for redundancy
"Two machines are deployed, which is a change to the original plan . One runs the
loadbalancer and one runs a normal node."

Q) Is it truth that adding load balancer in front of a single node will not do nothing except increasing latency by adding additional hop in front to the node ?
Q) Since worker proposal is being approved as written/offered by stake-holders and based on initial goal, please clarify who approved change to reduce original proposal from 3 servers to 2 for Asia and to waste resources ?
Q) Why balancer and nodes are keep rebooting ? Uptime is not even a day, and downtime is not even an hour. - https://sg.nodes.bitshares.ws/stats
Q) sg.nodes.bitshares.ws resolves to 172.104.167.98 and Let's Encrypt SSL deployed is missing CA Root certificate on the domain. Why ?


3) "If traffic increases significantly on this loadbalancer, additional nodes can be
deployed. This was a cost-benefit decision due to the price of asian dedicated hosts"

Q) Please clear on Linode's invoice which nodes are Asian ones/specified by ip.
Q) You are using excuse under the "price of asian dedicated hosts" while you are running cloud instance on Linode. Can you please be more clear on the actual reason of decreasing/leaving original roadmap/project ?



- Please be kind and clarify which node is running which IP and belongs to which invoice.
- Please be kind and explain what is legal relation with company ChainSquad gmbH presenting email printed reviews with its own stamp on it ? Isn't responsible party for the worker proposal Blockchain Projects BV ?
- Please be kind how from Infrastructure Urgent proposal additional milestone is related to infrastructure ( faucet ) and why it was primarily deployed as priority while even worker proposal approved infrastructure was started in February.
- Please clarify is Hetzner (invoice R0007759821) additional milestone for EU and who approved it ?

Finance/Payment review

http://www.bitshares.foundation/download/invoices/201712-infrastructure/20180216-Factuur%202018-4.pdf

You've asked for 3300 USD regarding one-time setup. Initial setup of infrastructure started in January and it was not fully operational by February since unexpected downtime happened at your initial deployment.

If your ONE-TIME setup had a period of 1 month, and you haven't delivered any eta on days/hours by setting fixed price, as taken from your own report i would be suggesting that you refund 1 of the 2 month (QTY) payments you've charged for the management of the infrastructure since you've been managing it only February (period after initial cost/setup), before this escalate more.

I also will hit reminder that having centralized autonomy, power, invoicing to yourself, approving changes and reviewing it yourself, changing proposals as you go without consent of each stake-holder, or moving funds in resources never approved was not something that i was expecting from someone who should set example on the blockchain.

Overall is that you are charging more than you are doing, lack of interest on long-term solutions (SSL example above), lack of professionalism (You spent 3 weeks to deploy 5 nodes and 3 are having broken SSL, but you've done Testing ), you are doing what was never agreed to be done, there is no serious reporting and your own KPI failed by the nodes status in first month as well as current up-time overall.

Many thanks for clarifications and answers, community is waiting patiently.

Offline blockchainprojectsbv

  • Newbie
  • *
  • Posts: 9
    • View Profile
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #46 on: March 01, 2018, 10:32:27 am »
Hello to you too,

and thank you for your feedback. We have now realized that we can improve our transparency with respect to the deployed machines and costs. We appreciate that you take the time to ask the questions, your are doing the whole community a great favor in reviewing delivery. We will answer your questions below.

Quote
Hello to both Blockchain Projects BV and younger accidental brother Blockchain BV, as well as ChainSquad GMBH. Since you've posted your final report, and until now as we all know, there was nobody around to do review on your work as you're doing on others, i was happy to step up and p by doing pro-bono review of your delivery since it would be too centralized that Blockchain Projects BV getting reviewed by itself in another form known as Bitshares Foundation, as it was the case in the past.
Blockchain BV is a holding that owns intellectual property and everything related and is the owner of Blockchain Projects BV, which is for our operating business with staff. The BitShares Blockchain Foundation (BBF) is seperate from both and a non-profit legal entity without any staff. When needed, the BBF hires Blockchain Projects BV or outside contractors. Public appearances under the flag of the BBF should always have one thing that stands out: Only to create understanding, knowledge and interest in BitShares and its core token BTS for wider use.
ChainSquad GmbH is another independent entity that offered to run the faucet software at a cheap rate and additional offered to provide statistics that we would have needed to develop from scratch. It was the most economical thing to simply outsource the faucet to an existing business. If competition in that regard grows, we will re-evaluate that decision.

Quote
to completely remove any possible conflict of interests
This is something that we are fully aware and we aim for complete transparency here. That is why your efforts are much appreciated and we will answer all of your questions.

We do realize that the overlap of personal is not ideal, in particular when it comes to reviewing delivery and we would have preferred it to be different but need to live with what we have right now. We try hard to keep those entities separate as much as possible and be transparent to the maximum possible. The fact that we basically review our own work is far from ideal - we know that - and thus we publish reports in great detail and allow everyone else to review the delivery independently.

Quote
Q) Please explain in which way exactly and/or how you limited the use/usage of the API to non-commercial one ?
This is only ensured as per disclaimer, nothing is planned to ensure this. The reason behind is as follows: The BTS holders effectively pay for the infrastructure. Commercial users should strengthen the infrastructure even more by providing their additional nodes.

Quote
Q) Why balancer and nodes are keep rebooting ? Uptime is not even a day, and downtime is not even an hour. - https://us.nodes.bitshares.ws/stats
They are not rebooting, we have identified an issue with the health-check script that results in repeated  appearance of downtime due to looking into the maintenance interval. We are looking into resolving the problem with the health-checker while not breaking the intentions of the check.

Quote
Q) us.nodes.bitshares.ws resolves to 96.126.96.40 - Let's encrypt SSL deployed on it doesn't have a valid CA Root certificate, by presenting itself to potential visitor/user as not being trusted by all browsers. At the same time by checking bitshares.eu it does return properly installed SSL. Why ?
BitShares.eu is an independent entity and has its own setup and deployment, which differs from the approach of this infrastructure worker. The infrastructure worker uses haproxy. During development, we have identified a few issues with haproxy that made us force SNI. IP addresses never validate with any SSL certificate as it has no FQDN.

Quote
Q) On the invoices presented there is no transparent way to know which node is this, please clarify in the future proposals and reports to the community. Since I've been keen enough to do whois against IP, i've discovered that provider is Linode. Can you please clarify which node from the Linode invoices are these 3 ? Load balancer + 2.
In Singapore, we have 2 linode machines (running 1 loadbalancer and 1 node)
In the US, we have 2 linode machines (running 1 loadbalancer and 1 node) and one AWS instance which runs another node
In Europe, we have 3 Hetzner machines (running 1 loadbalancer and 3 nodes)
The IPs of the backends are not public since we need people to go through the load balancer to be able to update backend nodes without giving notice. We also keep the option to switch the loadbalancer without needing to redeploy the backend nodes as well, e.g. if haproxy proves insufficient, or in case of denial of service attacks

Quote
Q) Could you please disclose location of the 2 deployed nodes ? In the current infrastructure problem, redundancy was not important as solution on the single location how much decreasing latency and new availability on more locations having balancer in front.
There are 3 nodes in Central Europe
There are 2 nodes in the US
There is 1 node in Singapore
All of those have their own loadbalancer located near them. The latency of the UI is an ongoing topic, but the deployment is always the same to allow for dynamic addition or removal of nodes. So far, we cannot confirm that the latency is caused by the node deployment. Our latency tests (outside of the UI) consistently show below 100ms of delay.

Quote
Free SSL is not same as paid SSL, and im sure you are well aware since you are having highest degree of education and positioned as CTO of the Blockchain company managing DAC business after all. We are not running darkweb or home based servers that we can't afford cost of real SSL for all.
We agree. A premium certificate would be nicer but an initial delivery was easier to achieve with letsencrypt then with paid certificates. We don’t see that letsencrypt certificates are by any means less secure or have a bad reputation. Furthermore, the end user doesn’t actually see those certificates since the websocket connection is opened by the browser in the background and merely needs to provide a valid certificate. The browser bar does not change if we pay money for a premium cert. That said, we reduced the costs for the worker by not going for the paid certificate just yet. Again, the worker only pays for costs that have actually been billed for.

Quote
Q) Who is responsible as authority/person to overview and review work being done by Worker Bill Butler ? - Offtopic, but related to the integrity of authority being responsible.
The infrastructure worker is provided by Blockchain Projects BV, is escrowed by the BBF and ultimately has to keep the BitShares voters happy, as to keep the worker active. The latter holds true for every worker on BitShares including the UI worker. For the reviewing process, the BBF works closely with Sigve Kvalsvik who is in charge of doing the code review - as per the worker proposal. The BBF, being the escrow for the worker funds verifies delivery.

Quote
Q) Is it truth that adding load balancer in front of a single node will not do nothing except increasing latency by adding additional hop in front to the node ?
In short: yes. The latency to the loadbalancer cannot be improved, the increased latency from the loadbalancer to the backend nodes can be seen in the stats page. This setup allows new backend nodes to be dynamically added to increase capacity the loadbalancer. To improve this, we could either change the specifications of websockets to allow redirects (probably won’t happen), or add another layer to the whole thing to identify the location of the best possible loadbalancer near you. This raises the question why multiple loadbalancers at all? First, for redundancy, second the whole need for multiple loadbalancers is an issue with websockets. If we were HTTP-only, we could let the loadbalancer redirect requests geo-locationally, but that is not possible for websockets.

Quote
Q) Since worker proposal is being approved as written/offered by stake-holders and based on initial goal, please clarify who approved change to reduce original proposal from 3 servers to 2 for Asia and to waste resources ?
The reduction of the asian nodes from 3 to 2 was a temporary decision that was caused by the linode nodes in Singapore to be too expensive in combination with AWS limitations that prevented us from deploying AWS nodes in Asia. This was solely our decision, and this limitation has been removed since. This question probably arouse of our report being called final, our efforts will of course continue. We will update said report.
Currently we prefer AWS over linode since we need over 14GB of RAM and over 20GB of disk. To run a single instance on linode, we need to pay 160$ a month. On AWS we are more flexible with respect to disk and RAM, i.e. cheaper. Furthermore dedicated hosts are prefered over virtual ones. Ideally, we would like to have Hetzner performance and price (roughly 60$ for a dedicated host that allows deployment of two nodes) in Asia, but unfortunately, we haven’t found such a service yet.

Quote
Q) Why balancer and nodes are keep rebooting ? Uptime is not even a day, and downtime is not even an hour. - https://sg.nodes.bitshares.ws/stats
The answer to this can be found above. This is a limitation of the health-checking script and we are looking into it.


Quote
Q) sg.nodes.bitshares.ws resolves to 172.104.167.98 and Let's Encrypt SSL deployed is missing CA Root certificate on the domain. Why ?
Same answer as above.

Quote
Q) Please clear on Linode's invoice which nodes are Asian ones/specified by ip.
We cannot publish backend IP addresses. The linodes in Singapore are linode5406954 and linode5837679.

Quote
Q) You are using excuse under the "price of asian dedicated hosts" while you are running cloud instance on Linode. Can you please be more clear on the actual reason of decreasing/leaving original roadmap/project ?
Please elaborate why you believe we need an excuse? As you can see from the worker proposal, we invoice to the worker the costs that we have for the infrastructure. Keeping costs low is the benefit of the shareholders. Also, the worker proposal, at no point, mentions to obtain dedicated servers but explicitly mentions that we are going to pick the most economical thing available that offers what is needed. Interestingly, Hetzner offers dedicated servers in Germany for much cheaper than we can get VPS or dedicated machines anywhere else.
As for the reduced number of nodes in Asia, again, this has been temporarily and will be changed in the next few weeks.

Quote
- Please be kind and clarify which node is running which IP and belongs to which invoice.
We cannot publish backend IP addresses to prevent denial of service attacks against the backend nodes.

Quote
- Please be kind and explain what is legal relation with company ChainSquad gmbH presenting email printed reviews with its own stamp on it ? Isn't responsible party for the worker proposal Blockchain Projects BV ?
ChainSquad runs bitshares.eu and the faucet service. They have been hired by us to take that job and get paid as per the worker proposal. This was an economic decision that allows us to reuse existing services instead of redeveloping thh wheel.

Quote
- Please be kind how from Infrastructure Urgent proposal additional milestone is related to infrastructure ( faucet ) and why it was primarily deployed as priority while even worker proposal approved infrastructure was started in February.
The faucet was always part of the worker, simply forgot as a milestone.
https://github.com/bitshares-foundation/bitshares.foundation/blob/ac0c21a5ea619a10e408df950814a222635fd1fe/_budgets/2017-12-infrastructure.md

Quote
- Please clarify is Hetzner (invoice R0007759821) additional milestone for EU and who approved it ?
The worker allows for us to decide to deploy additional nodes, which covers the EU milestone.

Quote
If your ONE-TIME setup had a period of 1 month, and you haven't delivered any eta on days/hours by setting fixed price, as taken from your own report i would be suggesting that you refund 1 of the 2 month (QTY) payments you've charged for the management of the infrastructure since you've been managing it only February (period after initial cost/setup), before this escalate more.
The setup costs mostly contain the work that needed to be done for the orchestration tooling and is thus independent of the delivery or any time frame (that’s why it is a setup fee). The fact that there was a downtime for the asian and the US nodes was caused by an update of the backend software that needed to be installed quickly to close a DOS attack vector.

The management fee covers the management of deployed nodes. Given that we have an orchestration environment, the management does not actually depend on the amount of deployed nodes and we expect it to grow only slightly with the growth of new nodes.

The nodes themselves are billed separately from the management as is the setup fee that was for initial development of said tools.

Quote
I also will hit reminder that having centralized autonomy, power, invoicing to yourself, approving changes and reviewing it yourself, changing proposals as you go without consent of each stake-holder, or moving funds in resources never approved was not something that i was expecting from someone who should set example on the blockchain.
We are fully aware of this problematic setup, but suspending all our BitShares related efforts due to running the BBF was not an option. We aim for transparency.
Furthermore, the infrastructure worker was setup as a “budget worker” specifically to allow some flexibility with respect to the deliverables. It allows us to “overdeliver” without asking for shareholder approval every time we see need to adapt for new situations. This also allowed us to deploy an elasticsearch node for the UI development team even though it was not part of the original worker proposal. Of course, always keeping in mind best cost-efficiency.
Additionally, we are not aware of any changes made unilaterally that caused harm to the BTS holders. Please let us know if you feel differently.

Quote
Overall is that you are charging more than you are doing, lack of interest on long-term solutions (SSL example above), lack of professionalism (You spent 3 weeks to deploy 5 nodes and 3 are having broken SSL, but you've done Testing ), you are doing what was never agreed to be done, there is no serious reporting and your own KPI failed by the nodes status in first month as well as current up-time overall.
All our efforts aim for long-term and dynamically adaptable solutions, to be able to adjust the infrastructure to demands of the network. Please do elaborate where you see “lack of professionalism” (note that we do not work full time for the infrastructure worker). As for the reporting, which information do you see lacking? We are happy to fill those in.
The nodes failed due to an unexpected issue we ran into when upgrading the backend software for the latest release. Your issues with SSL have been addressed above. So far, we don’t see that we have overcharged the shareholders even though we may have had a bumpy start.
« Last Edit: March 01, 2018, 11:07:13 am by blockchainprojectsbv »

Offline Digital Lucifer

Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #47 on: March 01, 2018, 11:58:37 am »
Hello to you too,

and thank you for your feedback. We have now realized that we can improve our transparency with respect to the deployed machines and costs. We appreciate that you take the time to ask the questions, your are doing the whole community a great favor in reviewing delivery. We will answer your questions below.

Quote
Hello to both Blockchain Projects BV and younger accidental brother Blockchain BV, as well as ChainSquad GMBH. Since you've posted your final report, and until now as we all know, there was nobody around to do review on your work as you're doing on others, i was happy to step up and p by doing pro-bono review of your delivery since it would be too centralized that Blockchain Projects BV getting reviewed by itself in another form known as Bitshares Foundation, as it was the case in the past.
Blockchain BV is a holding that owns intellectual property and everything related and is the owner of Blockchain Projects BV, which is for our operating business with staff. The BitShares Blockchain Foundation (BBF) is seperate from both and a non-profit legal entity without any staff. When needed, the BBF hires Blockchain Projects BV or outside contractors. Public appearances under the flag of the BBF should always have one thing that stands out: Only to create understanding, knowledge and interest in BitShares and its core token BTS for wider use.
ChainSquad GmbH is another independent entity that offered to run the faucet software at a cheap rate and additional offered to provide statistics that we would have needed to develop from scratch. It was the most economical thing to simply outsource the faucet to an existing business. If competition in that regard grows, we will re-evaluate that decision.

Quote
to completely remove any possible conflict of interests
This is something that we are fully aware and we aim for complete transparency here. That is why your efforts are much appreciated and we will answer all of your questions.

We do realize that the overlap of personal is not ideal, in particular when it comes to reviewing delivery and we would have preferred it to be different but need to live with what we have right now. We try hard to keep those entities separate as much as possible and be transparent to the maximum possible. The fact that we basically review our own work is far from ideal - we know that - and thus we publish reports in great detail and allow everyone else to review the delivery independently.

Quote
Q) Please explain in which way exactly and/or how you limited the use/usage of the API to non-commercial one ?
This is only ensured as per disclaimer, nothing is planned to ensure this. The reason behind is as follows: The BTS holders effectively pay for the infrastructure. Commercial users should strengthen the infrastructure even more by providing their additional nodes.

Quote
Q) Why balancer and nodes are keep rebooting ? Uptime is not even a day, and downtime is not even an hour. - https://us.nodes.bitshares.ws/stats
They are not rebooting, we have identified an issue with the health-check script that results in repeated  appearance of downtime due to looking into the maintenance interval. We are looking into resolving the problem with the health-checker while not breaking the intentions of the check.

Quote
Q) us.nodes.bitshares.ws resolves to 96.126.96.40 - Let's encrypt SSL deployed on it doesn't have a valid CA Root certificate, by presenting itself to potential visitor/user as not being trusted by all browsers. At the same time by checking bitshares.eu it does return properly installed SSL. Why ?
BitShares.eu is an independent entity and has its own setup and deployment, which differs from the approach of this infrastructure worker. The infrastructure worker uses haproxy. During development, we have identified a few issues with haproxy that made us force SNI. IP addresses never validate with any SSL certificate as it has no FQDN.

Quote
Q) On the invoices presented there is no transparent way to know which node is this, please clarify in the future proposals and reports to the community. Since I've been keen enough to do whois against IP, i've discovered that provider is Linode. Can you please clarify which node from the Linode invoices are these 3 ? Load balancer + 2.
In Singapore, we have 2 linode machines (running 1 loadbalancer and 1 node)
In the US, we have 2 linode machines (running 1 loadbalancer and 1 node) and one AWS instance which runs another node
In Europe, we have 3 Hetzner machines (running 1 loadbalancer and 4 nodes)
The IPs of the backends are not public since we need people to go through the load balancer to be able to update backend nodes without giving notice. We also keep the option to switch the loadbalancer without needing to redeploy the backend nodes as well, e.g. if haproxy proves insufficient, or in case of denial of service attacks

Quote
Q) Could you please disclose location of the 2 deployed nodes ? In the current infrastructure problem, redundancy was not important as solution on the single location how much decreasing latency and new availability on more locations having balancer in front.
There are 3 nodes in Central Europe
There are 2 nodes in the US
There is 1 node in Singapore
All of those have their own loadbalancer located near them. The latency of the UI is an ongoing topic, but the deployment is always the same to allow for dynamic addition or removal of nodes. So far, we cannot confirm that the latency is caused by the node deployment. Our latency tests (outside of the UI) consistently show below 100ms of delay.

Quote
Free SSL is not same as paid SSL, and im sure you are well aware since you are having highest degree of education and positioned as CTO of the Blockchain company managing DAC business after all. We are not running darkweb or home based servers that we can't afford cost of real SSL for all.
We agree. A premium certificate would be nicer but an initial delivery was easier to achieve with letsencrypt then with paid certificates. We don’t see that letsencrypt certificates are by any means less secure or have a bad reputation. Furthermore, the end user doesn’t actually see those certificates since the websocket connection is opened by the browser in the background and merely needs to provide a valid certificate. The browser bar does not change if we pay money for a premium cert. That said, we reduced the costs for the worker by not going for the paid certificate just yet. Again, the worker only pays for costs that have actually been billed for.

Quote
Q) Who is responsible as authority/person to overview and review work being done by Worker Bill Butler ? - Offtopic, but related to the integrity of authority being responsible.
The infrastructure worker is provided by Blockchain Projects BV, is escrowed by the BBF and ultimately has to keep the BitShares voters happy, as to keep the worker active. The latter holds true for every worker on BitShares including the UI worker. For the reviewing process, the BBF works closely with Sigve Kvalsvik who is in charge of doing the code review - as per the worker proposal. The BBF, being the escrow for the worker funds verifies delivery.

Quote
Q) Is it truth that adding load balancer in front of a single node will not do nothing except increasing latency by adding additional hop in front to the node ?
In short: yes. The latency to the loadbalancer cannot be improved, the increased latency from the loadbalancer to the backend nodes can be seen in the stats page. This setup allows new backend nodes to be dynamically added to increase capacity the loadbalancer. To improve this, we could either change the specifications of websockets to allow redirects (probably won’t happen), or add another layer to the whole thing to identify the location of the best possible loadbalancer near you. This raises the question why multiple loadbalancers at all? First, for redundancy, second the whole need for multiple loadbalancers is an issue with websockets. If we were HTTP-only, we could let the loadbalancer redirect requests geo-locationally, but that is not possible for websockets.

Quote
Q) Since worker proposal is being approved as written/offered by stake-holders and based on initial goal, please clarify who approved change to reduce original proposal from 3 servers to 2 for Asia and to waste resources ?
The reduction of the asian nodes from 3 to 2 was a temporary decision that was caused by the linode nodes in Singapore to be too expensive in combination with AWS limitations that prevented us from deploying AWS nodes in Asia. This was solely our decision, and this limitation has been removed since. This question probably arouse of our report being called final, our efforts will of course continue. We will update said report.
Currently we prefer AWS over linode since we need over 14GB of RAM and over 20GB of disk. To run a single instance on linode, we need to pay 160$ a month. On AWS we are more flexible with respect to disk and RAM, i.e. cheaper. Furthermore dedicated hosts are prefered over virtual ones. Ideally, we would like to have Hetzner performance and price (roughly 60$ for a dedicated host that allows deployment of two nodes) in Asia, but unfortunately, we haven’t found such a service yet.

Quote
Q) Why balancer and nodes are keep rebooting ? Uptime is not even a day, and downtime is not even an hour. - https://sg.nodes.bitshares.ws/stats
The answer to this can be found above. This is a limitation of the health-checking script and we are looking into it.


Quote
Q) sg.nodes.bitshares.ws resolves to 172.104.167.98 and Let's Encrypt SSL deployed is missing CA Root certificate on the domain. Why ?
Same answer as above.

Quote
Q) Please clear on Linode's invoice which nodes are Asian ones/specified by ip.
We cannot publish backend IP addresses. The linodes in Singapore are linode5406954 and linode5837679.

Quote
Q) You are using excuse under the "price of asian dedicated hosts" while you are running cloud instance on Linode. Can you please be more clear on the actual reason of decreasing/leaving original roadmap/project ?
Please elaborate why you believe we need an excuse? As you can see from the worker proposal, we invoice to the worker the costs that we have for the infrastructure. Keeping costs low is the benefit of the shareholders. Also, the worker proposal, at no point, mentions to obtain dedicated servers but explicitly mentions that we are going to pick the most economical thing available that offers what is needed. Interestingly, Hetzner offers dedicated servers in Germany for much cheaper than we can get VPS or dedicated machines anywhere else.
As for the reduced number of nodes in Asia, again, this has been temporarily and will be changed in the next few weeks.

Quote
- Please be kind and clarify which node is running which IP and belongs to which invoice.
We cannot publish backend IP addresses to prevent denial of service attacks against the backend nodes.

Quote
- Please be kind and explain what is legal relation with company ChainSquad gmbH presenting email printed reviews with its own stamp on it ? Isn't responsible party for the worker proposal Blockchain Projects BV ?
ChainSquad runs bitshares.eu and the faucet service. They have been hired by us to take that job and get paid as per the worker proposal. This was an economic decision that allows us to reuse existing services instead of redeveloping thh wheel.

Quote
- Please be kind how from Infrastructure Urgent proposal additional milestone is related to infrastructure ( faucet ) and why it was primarily deployed as priority while even worker proposal approved infrastructure was started in February.
The faucet was always part of the worker, simply forgot as a milestone.
https://github.com/bitshares-foundation/bitshares.foundation/blob/ac0c21a5ea619a10e408df950814a222635fd1fe/_budgets/2017-12-infrastructure.md

Quote
- Please clarify is Hetzner (invoice R0007759821) additional milestone for EU and who approved it ?
The worker allows for us to decide to deploy additional nodes, which covers the EU milestone.

Quote
If your ONE-TIME setup had a period of 1 month, and you haven't delivered any eta on days/hours by setting fixed price, as taken from your own report i would be suggesting that you refund 1 of the 2 month (QTY) payments you've charged for the management of the infrastructure since you've been managing it only February (period after initial cost/setup), before this escalate more.
The setup costs mostly contain the work that needed to be done for the orchestration tooling and is thus independent of the delivery or any time frame (that’s why it is a setup fee). The fact that there was a downtime for the asian and the US nodes was caused by an update of the backend software that needed to be installed quickly to close a DOS attack vector.

The management fee covers the management of deployed nodes. Given that we have an orchestration environment, the management does not actually depend on the amount of deployed nodes and we expect it to grow only slightly with the growth of new nodes.

The nodes themselves are billed separately from the management as is the setup fee that was for initial development of said tools.

Quote
I also will hit reminder that having centralized autonomy, power, invoicing to yourself, approving changes and reviewing it yourself, changing proposals as you go without consent of each stake-holder, or moving funds in resources never approved was not something that i was expecting from someone who should set example on the blockchain.
We are fully aware of this problematic setup, but suspending all our BitShares related efforts due to running the BBF was not an option. We aim for transparency.
Furthermore, the infrastructure worker was setup as a “budget worker” specifically to allow some flexibility with respect to the deliverables. It allows us to “overdeliver” without asking for shareholder approval every time we see need to adapt for new situations. This also allowed us to deploy an elasticsearch node for the UI development team even though it was not part of the original worker proposal. Of course, always keeping in mind best cost-efficiency.
Additionally, we are not aware of any changes made unilaterally that caused harm to the BTS holders. Please let us know if you feel differently.

Quote
Overall is that you are charging more than you are doing, lack of interest on long-term solutions (SSL example above), lack of professionalism (You spent 3 weeks to deploy 5 nodes and 3 are having broken SSL, but you've done Testing ), you are doing what was never agreed to be done, there is no serious reporting and your own KPI failed by the nodes status in first month as well as current up-time overall.
All our efforts aim for long-term and dynamically adaptable solutions, to be able to adjust the infrastructure to demands of the network. Please do elaborate where you see “lack of professionalism” (note that we do not work full time for the infrastructure worker). As for the reporting, which information do you see lacking? We are happy to fill those in.
The nodes failed due to an unexpected issue we ran into when upgrading the backend software for the latest release. Your issues with SSL have been addressed above. So far, we don’t see that we have overcharged the shareholders even though we may have had a bumpy start.

Thank you for a very professional and carefully crafted reply. I do publicly apologize to Fabian Schuh and/or anyone from the Team, and i do hope in the future we will continue with transparency and reporting such as this, since you do see a lot of info was missing between test and production period on bugs/fixes and management that is going on.

Regarding SSL, I don't have issue with it, just pointed out that certificate you deployed is not done properly without any mention in any of your reports until i've asked. You said the problem was HAProxy so, here is a brief info how to deploy with specific notice on adding CA Root:

Quote
You need to add manually CA Root to the file certificate.

Another reason why Paid SSL should be much more appropriate and easier approach for Balancer and Orchestration tools deployment/development with less stress.


Many thanks, and talk to you soon. Will be in a need for counter review on your side very soon.

Cheers,

DL.
Keep up the good work and more detailed communication for better transparency, it will save both sides some time.

 
« Last Edit: March 01, 2018, 12:02:32 pm by Digital Lucifer »
Milos (Mike) Preocanin - General Manager @ Syntek Solutions
TOANDI Co., LTD. (BOI Approved) - TAX ID: 0205549016913 - 95/5 Moo 4 Siam Country Club Rd.
Nong Prue, Bang Lamung, Chonburi 20250, Thailand.

Offline Customminer

  • Sr. Member
  • ****
  • Posts: 452
  • Bitshares FTW!
    • View Profile
    • Gridcoin.US
  • GitHub: grctest
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #48 on: March 07, 2018, 09:00:10 pm »
Attempting to perform direct JSON requests to these api nodes fails: https://eu.nodes.bitshares.works/

Code: [Select]
11 eof_exception: End Of File
stringstream
    {}
    th_a  sstream.cpp:109 peek

    {"str":""}
    th_a  json.cpp:478 from_string

For the mean time, I'll be removing these nodes from the HUG REST API (only when performing these json requests). Cheers
Hertz, Beyond Bitshares, Gridcoin!

Offline sschiessl

Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #49 on: March 07, 2018, 09:25:26 pm »
Attempting to perform direct JSON requests to these api nodes fails: https://eu.nodes.bitshares.works/

Code: [Select]
11 eof_exception: End Of File
stringstream
    {}
    th_a  sstream.cpp:109 peek

    {"str":""}
    th_a  json.cpp:478 from_string

For the mean time, I'll be removing these nodes from the HUG REST API (only when performing these json requests). Cheers

That is how actually the expected response, nothing broken there. Do you have an example of a node that behaves differently?

Offline Customminer

  • Sr. Member
  • ****
  • Posts: 452
  • Bitshares FTW!
    • View Profile
    • Gridcoin.US
  • GitHub: grctest
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #50 on: March 07, 2018, 11:30:21 pm »
Attempting to perform direct JSON requests to these api nodes fails: https://eu.nodes.bitshares.works/

Code: [Select]
11 eof_exception: End Of File
stringstream
    {}
    th_a  sstream.cpp:109 peek

    {"str":""}
    th_a  json.cpp:478 from_string

For the mean time, I'll be removing these nodes from the HUG REST API (only when performing these json requests). Cheers

That is how actually the expected response, nothing broken there. Do you have an example of a node that behaves differently?

Depends on the node, but here's the output from this script: https://github.com/BTS-CM/Bitshares-HUG-REST-API/blob/master/work_in_progress/testing_requests.py

Other online nodes work properly.

Code: [Select]
./\/\/\.
Not online: https://bitshares.crypto.fans/ws
.\/\/\/.
---
Online: https://bit.btsabc.org/ws
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===
---
Online: https://api.bts.blckchnd.com
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===
---
Online: https://openledger.hk/ws
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===
---
Online: https://bitshares-api.wancloud.io/ws
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===
...
Error: https://dex.rnglab.org
HTTPSConnectionPool(host='dex.rnglab.org', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
...
./\/\/\.
Not online: https://dexnode.net/ws
.\/\/\/.
---
Online: https://kc-us-dex.xeldal.com/ws
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===
---
Online: https://la.dexnode.net/ws
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===
...
Error: https://eu.nodes.bitshares.works
HTTPSConnectionPool(host='eu.nodes.bitshares.works', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
...
...
Error: https://us.nodes.bitshares.works
HTTPSConnectionPool(host='us.nodes.bitshares.works', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
...
...
Error: https://sg.nodes.bitshares.works
HTTPSConnectionPool(host='sg.nodes.bitshares.works', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))
...

---

Update: Added 'verify=False' to the request and got the expected output with warnings:

Code: [Select]
/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
---
Online: https://eu.nodes.bitshares.works
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===
/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
---
Online: https://us.nodes.bitshares.works
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===
/lib/python3.6/site-packages/urllib3/connectionpool.py:858: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
---
Online: https://sg.nodes.bitshares.works
<Response [200]>
{"id":1,"jsonrpc":"2.0","result":100}
===

---

So if it's the SSL cert, please update this to A+, B grade seems insufficient: https://www.ssllabs.com/ssltest/analyze.html?d=eu.nodes.bitshares.works

https://certbot.eff.org/ certbot/letsencrypt can provide grade 'A' SSL certs for free.
« Last Edit: March 07, 2018, 11:40:54 pm by Customminer »
Hertz, Beyond Bitshares, Gridcoin!

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12535
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #51 on: March 12, 2018, 09:39:43 am »
Interesting, on those nodes, we do use letsencrypt/certbot .. I didn't notice they offer domains other than "grade-A"
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 3077
    • View Profile
    • Steemit Blog
  • BitShares: abit
  • GitHub: abitmore
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #52 on: March 12, 2018, 06:39:03 pm »
@xeroc:
Code: [Select]
$ openssl s_client -connect eu.nodes.bitshares.works:443
CONNECTED(00000003)
140507123943072:error:14077458:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 unrecognized name:s23_clnt.c:770:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 305 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
---

Code: [Select]
~$ openssl s_client -connect eu.openledger.info:443
CONNECTED(00000003)
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=api.openledger.info
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
...
BTS account: abit
BTS committee member: abit
BTS witness: in.abit

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12535
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #53 on: March 12, 2018, 06:51:20 pm »
Thanks @abit, I've seen that problem too, will look into it. For some reason, HAproxy deals with the certs differently than I thought ..
fine tuning :D

edit: As i expected, it's caused by SNI (which helps the load balancer to distinguish between multiple domains)
if you use

Code: [Select]
openssl s_client -connect eu.nodes.bitshares.works:443 -servername eu.nodes.bitshares.works

instead, it works all fine.
« Last Edit: March 12, 2018, 06:53:54 pm by xeroc »
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12535
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #54 on: March 19, 2018, 09:06:23 am »
We've identified the issue with the certificate and now have a A+ rating:
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 3077
    • View Profile
    • Steemit Blog
  • BitShares: abit
  • GitHub: abitmore
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #55 on: March 19, 2018, 02:35:17 pm »
Still same error for me.
BTS account: abit
BTS committee member: abit
BTS witness: in.abit

Offline xeroc

  • Board Moderator
  • Hero Member
  • *****
  • Posts: 12535
  • ChainSquad GmbH
    • View Profile
    • ChainSquad GmbH
  • BitShares: xeroc
  • GitHub: xeroc
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #56 on: March 20, 2018, 02:39:30 pm »
Still same error for me.
The issue is actuall on openssl binary since the default does not use SNI which is required on the haproxy.
To make it work you need to specify the SNI domain:

openssl s_client -connect eu.nodes.bitshares.works:443 -servername eu.nodes.bitshares.works
Give BitShares a try! Use the http://testnet.bitshares.eu provided by http://bitshares.eu powered by ChainSquad GmbH

Offline abit

  • Committee member
  • Hero Member
  • *
  • Posts: 3077
    • View Profile
    • Steemit Blog
  • BitShares: abit
  • GitHub: abitmore
Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #57 on: May 18, 2018, 08:08:40 am »
Below is from Telegram channel, by clockworkgr

If get this error when connecting with cli_wallet:

0 exception: unspecified
TLS handshake failed
    {"message":"TLS handshake failed"}
    asio  websocket.cpp:487 operator()

It's perhaps due to cli_wallet sends a very small list of supported ciphers....if haproxy has those set it will fail with handshake failure due to not being able to negotiate a matching cipher

E.G. for nginx, this won't work:

ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-

----------- update ----------
by @clockworkgr:

I debugged the issue on my node running haproxy yesterday
hopefully it's the same reason
using ssldump during handshaking I noted 2 things
a) cli_wallet submits a Client Hello with a max supported SSL version of 3.1
that's TLS1.1
(my node had TLS 1.2 forced...so handshaking failed)
after I set it to allow TLS 1.1
b) I Noted that the ciphers supported by cli_wallet were not the ones in my allowed ciphers list in haproxy config
also failing handshake
cli_wallet's supported cipher list according to ssldump is the following:
        TLS_DHE_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_DSS_WITH_AES_256_CBC_SHA
        TLS_DH_RSA_WITH_AES_256_CBC_SHA
        TLS_DH_DSS_WITH_AES_256_CBC_SHA
        TLS_RSA_WITH_AES_256_CBC_SHA
        TLS_DHE_RSA_WITH_AES_128_CBC_SHA
        TLS_DHE_DSS_WITH_AES_128_CBC_SHA
        TLS_DH_RSA_WITH_AES_128_CBC_SHA
        TLS_DH_DSS_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_AES_128_CBC_SHA
        TLS_RSA_WITH_RC4_128_SHA
        TLS_RSA_WITH_RC4_128_MD5
        TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA
        TLS_DH_DSS_WITH_3DES_EDE_CBC_SHA
        TLS_RSA_WITH_3DES_EDE_CBC_SHA
you'll have to find what literals those correspond to in your nginx config and make sure at least one of them exists
« Last Edit: May 18, 2018, 02:01:19 pm by abit »
BTS account: abit
BTS committee member: abit
BTS witness: in.abit

Offline sschiessl

Re: [Worker] Deploy and maintain independent BitShares infrastructure
« Reply #58 on: May 18, 2018, 08:54:42 pm »
Also from telegram

Alex M - clockwork:
RESULT!

A+ Rating on ssllabs + haproxy + load balancer + working with cli_wallet

   ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
   ssl-default-bind-options no-sslv3    no-tls-tickets
   ssl-default-server-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
  ssl-default-server-options no-sslv3   no-tls-tickets

i made a mistake btw ..cli-wallet reporting max version SSL 3.1 is TLS1.0 not TLS1.1

it appears to not work only under IE8 on windows XP , Java6 and android 2.3.7 (lack of SNI)

the above config i mean

@xeroc & @sschiessl seeing as infrastructure nodes suffer from the same issue (cli-wallet dies with handshake failed) I suggest you use the above config in your haproxy setup