Hello to you too,
and thank you for your feedback. We have now realized that we can improve our transparency with respect to the deployed machines and costs. We appreciate that you take the time to ask the questions, your are doing the whole community a great favor in reviewing delivery. We will answer your questions below.
Hello to both Blockchain Projects BV and younger accidental brother Blockchain BV, as well as ChainSquad GMBH. Since you've posted your final report, and until now as we all know, there was nobody around to do review on your work as you're doing on others, i was happy to step up and p by doing pro-bono review of your delivery since it would be too centralized that Blockchain Projects BV getting reviewed by itself in another form known as Bitshares Foundation, as it was the case in the past.
Blockchain BV is a holding that owns intellectual property and everything related and is the owner of Blockchain Projects BV, which is for our operating business with staff. The BitShares Blockchain Foundation (BBF) is seperate from both and a non-profit legal entity without any staff. When needed, the BBF hires Blockchain Projects BV or outside contractors. Public appearances under the flag of the BBF should always have one thing that stands out: Only to create understanding, knowledge and interest in BitShares and its core token BTS for wider use.
ChainSquad GmbH is another independent entity that offered to run the faucet software at a cheap rate and additional offered to provide statistics that we would have needed to develop from scratch. It was the most economical thing to simply outsource the faucet to an existing business. If competition in that regard grows, we will re-evaluate that decision.
to completely remove any possible conflict of interests
This is something that we are fully aware and we aim for complete transparency here. That is why your efforts are much appreciated and we will answer all of your questions.
We do realize that the overlap of personal is not ideal, in particular when it comes to reviewing delivery and we would have preferred it to be different but need to live with what we have right now. We try hard to keep those entities separate as much as possible and be transparent to the maximum possible. The fact that we basically review our own work is far from ideal - we know that - and thus we publish reports in great detail and allow everyone else to review the delivery independently.
Q) Please explain in which way exactly and/or how you limited the use/usage of the API to non-commercial one ?
This is only ensured as per disclaimer, nothing is planned to ensure this. The reason behind is as follows: The BTS holders effectively pay for the infrastructure. Commercial users should strengthen the infrastructure even more by providing their additional nodes.
Q) Why balancer and nodes are keep rebooting ? Uptime is not even a day, and downtime is not even an hour. - https://us.nodes.bitshares.ws/stats
They are not rebooting, we have identified an issue with the health-check script that results in repeated appearance of downtime due to looking into the maintenance interval. We are looking into resolving the problem with the health-checker while not breaking the intentions of the check.
Q) us.nodes.bitshares.ws resolves to 96.126.96.40 - Let's encrypt SSL deployed on it doesn't have a valid CA Root certificate, by presenting itself to potential visitor/user as not being trusted by all browsers. At the same time by checking bitshares.eu it does return properly installed SSL. Why ?
BitShares.eu is an independent entity and has its own setup and deployment, which differs from the approach of this infrastructure worker. The infrastructure worker uses haproxy. During development, we have identified a few issues with haproxy that made us force SNI. IP addresses never validate with any SSL certificate as it has no FQDN.
Q) On the invoices presented there is no transparent way to know which node is this, please clarify in the future proposals and reports to the community. Since I've been keen enough to do whois against IP, i've discovered that provider is Linode. Can you please clarify which node from the Linode invoices are these 3 ? Load balancer + 2.
In Singapore, we have 2 linode machines (running 1 loadbalancer and 1 node)
In the US, we have 2 linode machines (running 1 loadbalancer and 1 node) and one AWS instance which runs another node
In Europe, we have 3 Hetzner machines (running 1 loadbalancer and 3 nodes)
The IPs of the backends are not public since we need people to go through the load balancer to be able to update backend nodes without giving notice. We also keep the option to switch the loadbalancer without needing to redeploy the backend nodes as well, e.g. if haproxy proves insufficient, or in case of denial of service attacks
Q) Could you please disclose location of the 2 deployed nodes ? In the current infrastructure problem, redundancy was not important as solution on the single location how much decreasing latency and new availability on more locations having balancer in front.
There are 3 nodes in Central Europe
There are 2 nodes in the US
There is 1 node in Singapore
All of those have their own loadbalancer located near them. The latency of the UI is an ongoing topic, but the deployment is always the same to allow for dynamic addition or removal of nodes. So far, we cannot confirm that the latency is caused by the node deployment. Our latency tests (outside of the UI) consistently show below 100ms of delay.
Free SSL is not same as paid SSL, and im sure you are well aware since you are having highest degree of education and positioned as CTO of the Blockchain company managing DAC business after all. We are not running darkweb or home based servers that we can't afford cost of real SSL for all.
We agree. A premium certificate would be nicer but an initial delivery was easier to achieve with letsencrypt then with paid certificates. We don’t see that letsencrypt certificates are by any means less secure or have a bad reputation. Furthermore, the end user doesn’t actually see those certificates since the websocket connection is opened by the browser in the background and merely needs to provide a valid certificate. The browser bar does not change if we pay money for a premium cert. That said, we reduced the costs for the worker by not going for the paid certificate just yet. Again, the worker only pays for costs that have actually been billed for.
Q) Who is responsible as authority/person to overview and review work being done by Worker Bill Butler ? - Offtopic, but related to the integrity of authority being responsible.
The infrastructure worker is provided by Blockchain Projects BV, is escrowed by the BBF and ultimately has to keep the BitShares voters happy, as to keep the worker active. The latter holds true for every worker on BitShares including the UI worker. For the reviewing process, the BBF works closely with Sigve Kvalsvik who is in charge of doing the code review - as per the worker proposal. The BBF, being the escrow for the worker funds verifies delivery.
Q) Is it truth that adding load balancer in front of a single node will not do nothing except increasing latency by adding additional hop in front to the node ?
In short: yes. The latency to the loadbalancer cannot be improved, the increased latency from the loadbalancer to the backend nodes can be seen in the stats page. This setup allows new backend nodes to be dynamically added to increase capacity the loadbalancer. To improve this, we could either change the specifications of websockets to allow redirects (probably won’t happen), or add another layer to the whole thing to identify the location of the best possible loadbalancer near you. This raises the question why multiple loadbalancers at all? First, for redundancy, second the whole need for multiple loadbalancers is an issue with websockets. If we were HTTP-only, we could let the loadbalancer redirect requests geo-locationally, but that is not possible for websockets.
Q) Since worker proposal is being approved as written/offered by stake-holders and based on initial goal, please clarify who approved change to reduce original proposal from 3 servers to 2 for Asia and to waste resources ?
The reduction of the asian nodes from 3 to 2 was a temporary decision that was caused by the linode nodes in Singapore to be too expensive in combination with AWS limitations that prevented us from deploying AWS nodes in Asia. This was solely our decision, and this limitation has been removed since. This question probably arouse of our report being called final, our efforts will of course continue. We will update said report.
Currently we prefer AWS over linode since we need over 14GB of RAM and over 20GB of disk. To run a single instance on linode, we need to pay 160$ a month. On AWS we are more flexible with respect to disk and RAM, i.e. cheaper. Furthermore dedicated hosts are prefered over virtual ones. Ideally, we would like to have Hetzner performance and price (roughly 60$ for a dedicated host that allows deployment of two nodes) in Asia, but unfortunately, we haven’t found such a service yet.
Q) Why balancer and nodes are keep rebooting ? Uptime is not even a day, and downtime is not even an hour. - https://sg.nodes.bitshares.ws/stats
The answer to this can be found above. This is a limitation of the health-checking script and we are looking into it.
Q) sg.nodes.bitshares.ws resolves to 172.104.167.98 and Let's Encrypt SSL deployed is missing CA Root certificate on the domain. Why ?
Same answer as above.
Q) Please clear on Linode's invoice which nodes are Asian ones/specified by ip.
We cannot publish backend IP addresses. The linodes in Singapore are linode5406954 and linode5837679.
Q) You are using excuse under the "price of asian dedicated hosts" while you are running cloud instance on Linode. Can you please be more clear on the actual reason of decreasing/leaving original roadmap/project ?
Please elaborate why you believe we need an excuse? As you can see from the worker proposal, we invoice to the worker the costs that we have for the infrastructure. Keeping costs low is the benefit of the shareholders. Also, the worker proposal, at no point, mentions to obtain dedicated servers but explicitly mentions that we are going to pick the most economical thing available that offers what is needed. Interestingly, Hetzner offers dedicated servers in Germany for much cheaper than we can get VPS or dedicated machines anywhere else.
As for the reduced number of nodes in Asia, again, this has been temporarily and will be changed in the next few weeks.
- Please be kind and clarify which node is running which IP and belongs to which invoice.
We cannot publish backend IP addresses to prevent denial of service attacks against the backend nodes.
- Please be kind and explain what is legal relation with company ChainSquad gmbH presenting email printed reviews with its own stamp on it ? Isn't responsible party for the worker proposal Blockchain Projects BV ?
ChainSquad runs bitshares.eu and the faucet service. They have been hired by us to take that job and get paid as per the worker proposal. This was an economic decision that allows us to reuse existing services instead of redeveloping thh wheel.
- Please be kind how from Infrastructure Urgent proposal additional milestone is related to infrastructure ( faucet ) and why it was primarily deployed as priority while even worker proposal approved infrastructure was started in February.
The faucet was always part of the worker, simply forgot as a milestone.
https://github.com/bitshares-foundation/bitshares.foundation/blob/ac0c21a5ea619a10e408df950814a222635fd1fe/_budgets/2017-12-infrastructure.md- Please clarify is Hetzner (invoice R0007759821) additional milestone for EU and who approved it ?
The worker allows for us to decide to deploy additional nodes, which covers the EU milestone.
If your ONE-TIME setup had a period of 1 month, and you haven't delivered any eta on days/hours by setting fixed price, as taken from your own report i would be suggesting that you refund 1 of the 2 month (QTY) payments you've charged for the management of the infrastructure since you've been managing it only February (period after initial cost/setup), before this escalate more.
The setup costs mostly contain the work that needed to be done for the orchestration tooling and is thus independent of the delivery or any time frame (that’s why it is a setup fee). The fact that there was a downtime for the asian and the US nodes was caused by an update of the backend software that needed to be installed quickly to close a DOS attack vector.
The management fee covers the management of deployed nodes. Given that we have an orchestration environment, the management does not actually depend on the amount of deployed nodes and we expect it to grow only slightly with the growth of new nodes.
The nodes themselves are billed separately from the management as is the setup fee that was for initial development of said tools.
I also will hit reminder that having centralized autonomy, power, invoicing to yourself, approving changes and reviewing it yourself, changing proposals as you go without consent of each stake-holder, or moving funds in resources never approved was not something that i was expecting from someone who should set example on the blockchain.
We are fully aware of this problematic setup, but suspending all our BitShares related efforts due to running the BBF was not an option. We aim for transparency.
Furthermore, the infrastructure worker was setup as a “budget worker” specifically to allow some flexibility with respect to the deliverables. It allows us to “overdeliver” without asking for shareholder approval every time we see need to adapt for new situations. This also allowed us to deploy an elasticsearch node for the UI development team even though it was not part of the original worker proposal. Of course, always keeping in mind best cost-efficiency.
Additionally, we are not aware of any changes made unilaterally that caused harm to the BTS holders. Please let us know if you feel differently.
Overall is that you are charging more than you are doing, lack of interest on long-term solutions (SSL example above), lack of professionalism (You spent 3 weeks to deploy 5 nodes and 3 are having broken SSL, but you've done Testing ), you are doing what was never agreed to be done, there is no serious reporting and your own KPI failed by the nodes status in first month as well as current up-time overall.
All our efforts aim for long-term and dynamically adaptable solutions, to be able to adjust the infrastructure to demands of the network. Please do elaborate where you see “lack of professionalism” (note that we do not work full time for the infrastructure worker). As for the reporting, which information do you see lacking? We are happy to fill those in.
The nodes failed due to an unexpected issue we ran into when upgrading the backend software for the latest release. Your issues with SSL have been addressed above. So far, we don’t see that we have overcharged the shareholders even though we may have had a bumpy start.
Topic: [Worker] Deploy and maintain independent BitShares infrastructure (Read 25260 times)
