Bitshares BOINC project brainstorming thread! All ideas welcome!

[R]

I have created a thread on Gridcoin's forum to drive up interest in this project: https://cryptocointalk.com/topic/53393-brainstorming-bitshares-boinc-project/

[R]

Stress-test whitepaper released! : http://docs.bitshares.eu/_downloads/bitshares-stresstest.pdf

Read the above PDF report for the recent stress test within the Bitshares testnet!

Relevant quotes from above whitepaper:

Transaction Production
In  order  to stress-test  a  network, we need to produce actual stress which is not easy to achieve at the scale that we needed for our tests. Furthermore, we wanted to simulate a more or less realistic scenario in which multiple in-dependent parties distributed on the globe produce transactions, each transaction trying to make it into the next block.

For this reasons, we have asked the BitSharesTalk community to participate in the stress-testing not only by means of providng validation  nodes  across  the  planet,  but  also  to  produce  the  re-quired stress.

Sounds like we could help out here!

However, the Peer-2-Peer networking code has been optimized for a block interval of about 10 s and thus leaves a lot of room for  optimizations  when  talking  about  high-throughput  at  low latencies.

Ok so the lack of computer resources available to create transactions wasn't the sole bottleneck for maximizing TPS, this is potentially an area of witness experimentation or machine learning (to find optimal parameter optimization for ranges of TPS load).

Conclusion section
The Peer-2-Peer code seems to be working nicely but results show that it could be one of the bottlenecks currently preventing us from going further. The computational resources of our validators have more than enough back-off for higher throughputs but the networking code was not able to provide sufficient data (e.g. transactions) to raise it during our stress-test.

Ok, so perhaps the lack of computing power isn't required until the network code is optimized?

[R]

BOINC vs Commercial Cloud Computing
Research papers

• The cost of virtualization implementation in volunteer computing http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=6914205&queryText=boinc&pageNumber=2&newsearch=true
• Increasing GP Computing Power for Free via Desktop GRID Computing and Virtualization http://ieeexplore.ieee.org/xpl/articleDetails.jsp?arnumber=4912963&queryText=boinc&pageNumber=2&newsearch=true
• [PDF] Cost-Benefit Analysis of Cloud Computing versus Desktop Grids http://mescal.imag.fr/membres/derrick.kondo/pubs/kondo_hcw09.pdf

Research TL;DR: (Note - 7 year old statistics!)

• In the best-case scenario, hosts register at a rate of 124 cloud nodes per day.
• The ratio of volunteer nodes needed to achieve the compute power of a small EC2 instance is about 2.83 active volunteer hosts to 1.
• Effectively, with a commercial BOINC project, there exists the opportunity to create a cloud computing service without actually owning/providing the computing capabilities yourself. Imagine creating Azure at a fraction of the cost..

[Fox]

Overall, I like your thoughts on how to use BOINC to generate more transactions against the TESTNET. Let's get the technical discussion going.

[Thom]

Witness experiments

We currently have ~25 witnesses, down from 100 in the switch from BTS 1.0 to BTS 2.0, we could experiment with different quantities of witnesses in an automated manner to evaluate optimal quantities of witnesses.

Interesting!

I like this idea. Such an experiment might lead to new insights on what the optimal number of witnesses are (irrespective of economic considerations) or what effects larger numbers of witnesses might have on the operation of the BitShares network.

[R]

Content mirrored: https://steemit.com/bitshares/@cm-steem/bitshares-boinc-project-brainstorming-thread-all-ideas-welcome

Hey,

I'm Customminer from the Gridcoin https://gridcoin.us community, I've been a holder of BTS since BTSX days & have a very positive outlook on the future of Bitshares!

I have been paying attention to the recent Bitshares testnet stress-testing https://bitsharestalk.org/index.php/topic,23829.0.html which yielded a rate of approx 3k TX/s, it was mentioned in the 13th Bitshares hangout https://soundcloud.com/beyond-bitcoin-hangouts/bitshares-hangout-13-w-christoph-hering-fuzzy that a potential reason that we were limited to 3k TX/s was the lack of computing power at the disposal of the stress-test organizers - I believe that the BOINC & Gridcoin community can help!

BOINC https://boinc.berkeley.edu/ is an open-source kit for volunteer distributed computing, it can distribute any kind of computing task and has between 250k & 560k active users (differing estimates depending on what site you go to.. it's difficult to guesstimate) & 4 million registered users in total. To put the scale of available computing power in perspective, team Gridcoin has 100k hosts at its disposal and currently only rewards 0.5% of the active BOINC community. The majority of the BOINC community contributes their resources entirely for free, those that crunch under team Gridcoin are rewarded GRC by the network on behalf of whitelisted projects (project owners pay nothing).

I spoke during episode 200 of the BeyondBitcoin hangout https://soundcloud.com/beyond-bitcoin-hangouts/beyondbitcoin-spotlight-gridcoin-03-24-17 and the 14th Bitshares hangout Chris4210[/member]/bitshares-hangout-14-03-31-17]https://steemit.com/bitshares/@Chris4210/bitshares-hangout-14-03-31-17 on this topic.

I try to encourage the development of new BOINC projects each month, join in the discussion: https://steemit.com/@cm-steem/brainstorming-boinc-projects-006

The tools:
https://github.com/marius311/boinc-server-docker
https://github.com/marius311/boinc2docker
https://www.docker.com/
https://github.com/xeroc/python-bitshares
https://github.com/xeroc/bitshares-docker
https://github.com/grctest/project-rain-site (Outdated, needs uplifted to latest BOINC version)

The ideas:

Bitshares Testnet Load Generator!

It was mentioned in the last Bitshares hangout that the Bitshares testnet stress test may have been limited due to a lack of resources available for creating a larger transaction volume - what about distributing testnet load generating software out to a few thousand computers around the world via a BOINC project?

Such a project wouldn't be eligible for Gridcoin whitelist status (no GRC rewards) due to the bursty nature of stress testing (GRC needs continuous work availability).

You could easily distribute an UIA to your volunteers proportionally to their completed work by adding a Bitshares field to the BOINC signup page and user profile.

I'd imagine that for this idea, you would distribute a docker container which contained python dependencies, the testnet stress-test python script, and the bitshares client.

Once running on an user's machine, it would need to create a testnet user account, the project admin would send enough testnet BTS to each volunteer bitshares account, each account would register as lifetime (for less fees, but not neccessary),  then the main stress-test script would begin bombarding testnet with transactions.

The 3000+ TX/s rate achieved in the recent Bitshares testnet stress-test was great, but let's try to push it to 1 million + TX/s!

Relevant links:
Bitshares testnet stress-test thread : https://bitsharestalk.org/index.php?topic=23829.15

---

Bitshares Full/API node distribution

Alongside the testnet load generator, you could distribute full node servers out to volunteers to supply additional network resources to the Bitshares network.

This would require the development of a proof-of-bandwidth credit mechanism (how to reward credit to users based on their full node availability/capacity/consumed-resources).

Since full-nodes are always required, this would make a Bitshares BOINC project eligible for Gridcoin whitelisting.

This would require setting minimum computer spec requirements, so not everyone would be able to participate unlike the load generator.

---

Machine Learning + BTS

You could distribute TensorFlow https://www.tensorflow.org/ docker containers, and perform machine learning against data gathered from the BTS DEX.

---

Witness experiments

We currently have ~25 witnesses, down from 100 in the switch from BTS 1.0 to BTS 2.0, we could experiment with different quantities of witnesses in an automated manner to evaluate optimal quantities of witnesses.

---

Volunteer incentives!

The BOINC web server is programmed in PHP, you can easily add a Bitshares (and bitshares testnet) account field to the account signup page (and profile page), then expose said account name within the user statistics xml files (which is generated every 24hrs) then you can proportionally distribute an UIA on the BTS network to each BOINC volunteer proportionally to their contributed computation (RAC). Since these xml files are public, anyone could distribute their own UIA against your volunteers (new sharedrop vector!).

If there is a continuous availability of work units for volunteers, then the project would be eligible for Gridcoin whitelist status (the Gridcoin network would distribute rewards to your volunteers).

---

Do you have an idea for Bitshares which requires vast amount of distributed computing power? Please post your ideas in this thread! 

Thoughts?

Regards,
CM.