Storj and Bitshares Collaboration

[super3]

Which is why we spent a great deal of time developing Metadisk. A decentralized application that we run most of the core code on. Solves two problems:

1) Allows us to have reward full nodes because they have an application built on top of them
2) The nodes can cache the DHT

Because the nodes also may have the file, you can submit multiple queries to multiple nodes. Obviously they would want to return a result as quickly as possible, as being the fastest would probably mean they also get to get paid for transferring the file to you.

[bytemaster]

That's only a extra millisecond or two of overhead

Assuming you use a DHT for your lookup, then you have 20 ms RTT per query (log N) so for large data sets it could end up taking the better part of a second to look up a chunk of data.

Of course you could have 'directory nodes' that accelerate this process, but then you have a load balancing challenge.

You are working a tough problem, I really hope you solve it because there is a need for it.

[super3]

Well profit model is also involved. If you look at Dropbox they just take Amazon cloud and mark it up 100x for their premium service. Its $1 GB per year. Take your free hard drive and add those numbers up(divide by 3 to account for redundancy). Have I convinced you?

Sure lookup times will increase in complexity. That's only a extra millisecond or two of overhead. We are taking about transfer at P2Ps speeds though so you gain 10x-100x transfer rates over centralized systems. Not to mention decentralized system can automatically scale to any size.

[luckybit]

The decentralized storage space is fundamentally limited by economics like everything else.

1) Lets clearly state the problem to be solved, because without understanding the problem all we have are solutions in search of a problem.   I would describe the problem is that governments have declared certain information (bits) contraband and thus attacks those who would dare to provide the service of hosting it.   The solution is to "decentralize" and host it everywhere and nowhere all at the same time.

2) The other claim maid by these decentralized storage providers is that they are "faster", "cheaper", and have "higher availability" than centralized counterparts.  All of these claims are lies unless you are talking about contraband bits.    When I download a movie from iTunes it is faster than any torent and availability is higher too.   

3) People have correctly identified why Tor and file sharing services are slower:  no economic incentive to build infrastructure.  So projects like Storj and Maidsafe are trying to build out an economic model that will motivate users to add resources.

I am certain that you will get a vast increase in the performance of contraband networks, at a cost.  You will pay more per byte on one of these networks and your latency will make browsing a challenge because you are doing a decentralized Log N lookup of your content.  When you do find it you will be streaming it from someones home PC at a slow/cheap rate or you will be paying a vastly higher price per byte than your ISP is currently charging you. 

4) Decentralization always has costs.  Hosting files from your home computer consumes disk space, bandwidth, CPU, etc.   These costs have to be paid by the users.

5) Usually these systems boil down to micro-payment systems and this is the biggest challenge of them all.  Cryptography is not cheap, and all transactions have a mental-cost (making a decision) even if the user asks the computer to do it for them.   Negotiating payments of a fraction of a cent with 1000 different peers is economically challenged to say the least.

That said what these systems ultimately boil down to is this:
1) Provide a standardized API for finding content by hash across many different service providers
2) Provide a standardized payment system to automate paying any provider you may come across.
3) Lowering the barrier to entry by making the use of this system transparent to the users.

The above steps are possible but assumes competitors (data providers) will work together to allow their customers to shop around for the best deal.  In the process of shopping for the best deal economies of scale will once again result in complete centralization of hosting.  Sure anyone can run a server and use the API, but only one company will have the economies of scale necessary to provide "instant lookup", "largest selection", "fastest downloads". 

Of course the thing that prevents such a universal service from existing today is that hosting providers are forced to remove contraband bits.  Lastly, there is the issue that those looking for contraband are looking for "free" and those looking to pay can probably already get what they want.

So we can probably boil the whole thing down to an attempt to build a censor resistant data-store, the black-market of cloud hosting.  I think this is possible, but it will be expensive to use and slow compared to the alternatives.   Remember, black-market contraband almost always carries a risk premium.

Perhaps a hybrid approach:
1) Use 1 API for legit & contraband bits
2) Legit bits can be found quickly and easily and relatively cheaply
3) Contraband bits can be found slower and relatively more expensively.

The space is challenging and so far I haven't seen anyone focus on the economics of the problem.
 

Some of what you say is true. It does have initially higher costs because the infrastructure isn't set up for decentralization. I expect these initially higher costs to go down though as infrastructure is set up to bring costs down.

So something like a decentralized ISP is possible and could be operated at a lower cost but it would require some new hardware or perhaps the purchase and repurposing of the current infrastructure.

Centralization itself has hidden costs. If you use centralized services you're at the mercy of corporations who may or may not take information security seriously. Ebay for example just leaked or was hacked. Too many humans in the chain between the user and their data increases the risks on the users losing control of their data.

In the long term as data becomes even more precious than it is today the only way to make it harder for hackers to steal it is to never store it all in one place and while you can use the centralized corporate cloud then you have to trust the government, the corporations, the employees who work for each of these organizations, while with Storj you have to trust the code and your computer.

http://arstechnica.com/security/2014/07/weaponized-exploit-can-steal-user-cookies-on-ebay-tumblr-other-sites/

http://www.cnet.com/news/ebay-hacked-requests-all-users-change-passwords/

The benefit of something like Storj if it were to reach critical mass is that people could securely store their data in the decentralized cloud. No more local storage would be required. Thin clients in the form of a smart phone would be all most people would need and in the developing world the smart phone is all most people are going to have. So for the developing world which already has a wireless Internet the proposition will be quite a bit different but even for us in the developed world if our phone gets stolen we could just not care if our files are backed up on Storj.

Laptops could do something similar. I understand Google is trying to do that now but I don't see why we can't have a StorjOS for laptops.

[luckybit]

I'm interested interested in using Bitshares because we can use the platform to create pegged digital commodities. While Storjcoin is a deflationary cryptocurrency, increasing in worth as the network expands it might be useful to have something like a BitGB.

Therefore I can purchase a BitGB, and use it in the application knowing that it will always be worth 1 GB of space.

[CLains]

I'm also interested in Storj using BitShares, lol. I'd love to see bytemaster pitch why DPOS is the superior alternative in this case. But even further, I would encourage you to visit them at location super3, cost of travel is a drop in the ocean compared to the magnitude and potential of your project, and it will give a whole new dimension to the potential collaboration!

Take a leap. Be bold, brave, ambitious!

[super3]

I'm interested interested in using Bitshares because we can use the platform to create pegged digital commodities. While Storjcoin is a deflationary cryptocurrency, increasing in worth as the network expands it might be useful to have something like a BitGB.

Therefore I can purchase a BitGB, and use it in the application knowing that it will always be worth 1 GB of space.

[super3]

Great post Bytemaster. Let me try to break it down a bit.

Our solution is to encrypt all bits in the network. Therefore all bits can be "fungible" and treated the same. The problem when occurs when the user shares the hash+decryption key to the bits (all data is encrypted client side). At that point, the user has voided the contract to keep the data secure. Any contraband bits can be public greylisted, and nodes and applications can refuse to server them.

This doesn't not invite censorship because Bob can always share any file with Alice as long as he adds some salt to his data, so it does not appear as a publically listed file. Public files will be removed only if a particular network achieves consensus on the greylist. So some nodes will serve all content, some will choose to implement perhaps a copyright greylist. The users will have the the power to choose which nodes they want via their wallet and usage.

That said what these systems ultimately boil down to is this:
1) Provide a standardized API for finding content by hash across many different service providers
2) Provide a standardized payment system to automate paying any provider you may come across.
3) Lowering the barrier to entry by making the use of this system transparent to the users.

1) Our distributed application Metadisk implements a standardized API. This application can span multiple blockchains and networks, and make it very easy to integrate though a nice web API.
2) Cryptocurrency obviously.
3) We are designing out applications in a way that an average user can use them.

1) Use 1 API for legit & contraband bits
2) Legit bits can be found quickly and easily and relatively cheaply
3) Contraband bits can be found slower and relatively more expensively.

Done through Metadisk. The more contraband a bit is, the more it will be pushed toward the edge of the network.

[sang]

Bytemaster, I suggest checking out this paper on TorCoin - they seem to have figured out a way to do Proof of Bandwidth: http://www.nrl.navy.mil/itd/chacs/sites/www.nrl.navy.mil.itd.chacs/files/pdfs/14-1231-1559.pdf

(I have not read the entire thing yet)

[bytemaster]

The decentralized storage space is fundamentally limited by economics like everything else.

1) Lets clearly state the problem to be solved, because without understanding the problem all we have are solutions in search of a problem.   I would describe the problem is that governments have declared certain information (bits) contraband and thus attacks those who would dare to provide the service of hosting it.   The solution is to "decentralize" and host it everywhere and nowhere all at the same time.

2) The other claim maid by these decentralized storage providers is that they are "faster", "cheaper", and have "higher availability" than centralized counterparts.  All of these claims are lies unless you are talking about contraband bits.    When I download a movie from iTunes it is faster than any torent and availability is higher too.   

3) People have correctly identified why Tor and file sharing services are slower:  no economic incentive to build infrastructure.  So projects like Storj and Maidsafe are trying to build out an economic model that will motivate users to add resources.

I am certain that you will get a vast increase in the performance of contraband networks, at a cost.  You will pay more per byte on one of these networks and your latency will make browsing a challenge because you are doing a decentralized Log N lookup of your content.  When you do find it you will be streaming it from someones home PC at a slow/cheap rate or you will be paying a vastly higher price per byte than your ISP is currently charging you. 

4) Decentralization always has costs.  Hosting files from your home computer consumes disk space, bandwidth, CPU, etc.   These costs have to be paid by the users.

5) Usually these systems boil down to micro-payment systems and this is the biggest challenge of them all.  Cryptography is not cheap, and all transactions have a mental-cost (making a decision) even if the user asks the computer to do it for them.   Negotiating payments of a fraction of a cent with 1000 different peers is economically challenged to say the least.

That said what these systems ultimately boil down to is this:
1) Provide a standardized API for finding content by hash across many different service providers
2) Provide a standardized payment system to automate paying any provider you may come across.
3) Lowering the barrier to entry by making the use of this system transparent to the users.

The above steps are possible but assumes competitors (data providers) will work together to allow their customers to shop around for the best deal.  In the process of shopping for the best deal economies of scale will once again result in complete centralization of hosting.  Sure anyone can run a server and use the API, but only one company will have the economies of scale necessary to provide "instant lookup", "largest selection", "fastest downloads". 

Of course the thing that prevents such a universal service from existing today is that hosting providers are forced to remove contraband bits.  Lastly, there is the issue that those looking for contraband are looking for "free" and those looking to pay can probably already get what they want.

So we can probably boil the whole thing down to an attempt to build a censor resistant data-store, the black-market of cloud hosting.  I think this is possible, but it will be expensive to use and slow compared to the alternatives.   Remember, black-market contraband almost always carries a risk premium.

Perhaps a hybrid approach:
1) Use 1 API for legit & contraband bits
2) Legit bits can be found quickly and easily and relatively cheaply
3) Contraband bits can be found slower and relatively more expensively.

The space is challenging and so far I haven't seen anyone focus on the economics of the problem.
 

[Simeon II]

1.There is giant problem in this space.

2. It seems nobody has found a solution.

3. Storj is offering new product in the space, even he/they not claiming a decision for the fundamental problem.

What gains do you foresee in collaborating with just another player who does not know how to solve the problem?



I know you all want to play nice, but other than that what’s your point?

How do we find solutions to problems? Thinking and communicating ideas across a broad spectrum of issues...that is what the hangouts are for.  Updates, questions and brainstorming sessions essentially.

You can daydream, say good words –collaboration, communications, hangouts, brainstorming. Those are empty words for me.
The truth is the problem is big and complex… and the solution will probably be found by single someone driving on the road by himself, or maybe sleeping in a tent in the middle of nowhere.

Let me give you an example:
If it is 1940 and you organize 100 hangouts/brainstorming sessions with random people, chances are you not gonna discover the computer. Even better if you organize 100 hangout with random physicians chances are you not gonna discover the computer. It took one single genius, who had put 1000 h thinking about that and related and unrelated problems to make the breakthrough. Driving by himself, with no particular aim or destination, in the middle of nowhere.

So much for  collaboration, communication and brainstorming sessions.


Do not get me wrong – hangouts are great when the fundamental breakthrough is already there. Example:
When the idea of BTS X is already born, you can hang out and polish it. Hell, you can even find the best voting algo in one of those brainstorming sessions.

But in the field we are talking about the breakthrough is not there yet. Maidsafe have not found it, Storj ans Sia have not found it, even our in house genius have put finding the solution on backburner. So, you can brainstorm and dream how great the undiscovered ‘computer’ will be, but chances are you not gona discovery it… collaboratively brainstorming

[cass]

1.There is giant problem in this space.

2. It seems nobody has found a solution.

3. Storj is offering new product in the space, even he/they not claiming a decision for the fundamental problem.

What gains do you foresee in collaborating with just another player who does not know how to solve the problem?



I know you all want to play nice, but other than that what’s your point?

How do we find solutions to problems? Thinking and communicating ideas across a broad spectrum of issues...that is what the hangouts are for.  Updates, questions and brainstorming sessions essentially.

agreed

[fuzzy]

1.There is giant problem in this space.

2. It seems nobody has found a solution.

3. Storj is offering new product in the space, even he/they not claiming a decision for the fundamental problem.

What gains do you foresee in collaborating with just another player who does not know how to solve the problem?



I know you all want to play nice, but other than that what’s your point?

How do we find solutions to problems? Thinking and communicating ideas across a broad spectrum of issues...that is what the hangouts are for.  Updates, questions and brainstorming sessions essentially. 

[donkeypong]

DonJoe may be vindictive (I'm not saying I agree at all; I have no history there, and Super3 certainly seems to have history). But DonJoe was using real quotes from people we trust, including Stan and BM. I wouldn't sweep all that aside just for the sake of reconciliation; it needs to be addressed. The Storj idea sounds promising. If there were a collaboration in the works, I would enthusiastically look forward to supporting and possibly investing in it. But as an investor, performing my own due diligence, I sure as heck would want to investigate any smoke under the hood. You can blast DonJoe for shoving it down your throat, but it is not appropriate to blast him for repeating what some trusted people wrote. I would want explanations and assurances from those trusted people before I invested. It sounds like at the very least, the communication was poor, and perhaps beyond that it was just a misunderstanding.

[Simeon II]

1.There is giant problem in this space.

2. It seems nobody has found a solution.

3. Storj is offering new product in the space, even he/they not claiming a decision for the fundamental problem.

What gains do you foresee in collaborating with just another player who does not know how to solve the problem?



I know you all want to play nice, but other than that what’s your point?