Over the next two weeks, I'd like to work on documenting the blockchain format.  In particular, going from a block as a sequence of bytes; to fields with particular data types (uint32, hash); to semantic information about entities like accounts, balances, delegates, orders, and votes.  To keep the scope manageable, I'd like to limit my contribution to focusing on parsing the serialized blockchain.

I'll include information on the following:

- Byte-by-byte documentation of the serialized block format
- Constraints between various fields (e.g. the value in each block's parent block hash field must be equal to the value obtained by computing the hash of the serialization of the parent block).
- Reasons for design decisions
- Genesis-block parameters that can easily be tweaked
- Highlight major BitShares Toolkit features that either mimic features of other cryptocoins, or are different from other cryptocoins
- Describe in detail the implementation of high-level semantic nouns and verbs (e.g.:  What is a short position and how do you cover it?  How do we make TITAN transactions invisible?)

I'll post here with the Github URL when I start.

Also, since the bounty was posted before snapshot was taken, I'd like to ask if the bounty will include the BTSX spawned by the 500 PTS?

Hopefully this post satisfies the "scope and requirements" laid out in the opening post, and we can change from pending to active.

RFC 3986 specifies reserved delimiters:

      gen-delims  = ":" / "/" / "?" / "#" / "[" / "]" / "@"

      sub-delims  = "!" / "$" / "&" / "'" / "(" / ")"
                  / "*" / "+" / "," / ";" / "="

I guess the idea is to include a reserved delimiter from this list, to have our domain names that deliberately violate RFC 3986 (and hence, will hopefully never exist in legacy DNS).

We need to select a character not used by popular applications.  Off the top of my head:

: is used by git, scp, and in proto:// syntax
/ is used in HTTP to indicate paths
? is used in HTTP to indicate query string
# is used to indicate an anchor in web browsers
@ is used by ssh and email

So here are the remaining possible delimiters:

[]!$&'()*+,;=

We probably want to stay away from things that might make quoting problematic, and grouping symbols probably won't look right.  Which leaves:

!$&*+

Maybe + or * is a better candidate than ! since + and * already have short words ("plus" or "star"), and visually delimit words.

A move to decentralized allocation of .com would be most successful if managed via an "airdrop" to current .com holders, if there was wide industry support.   

"Airdrop to current .com holders" is a technical problem.  I think I have a solution.

For this, I'll refer to "domains," "legacy domains" or "legacy DNS" as the mainstream technology most internet users currently use,
and "cryptomains" as the domains of some DNS replacement cryptocoin technology that attempts to airdrop to legacy
DNS domain owners.

So the goal is to give all (or almost all) legacy domain holders an airdrop of the corresponding cryptomain.  You need two things:
(1) An "inherited set" consisting of some big set of legacy domains, so they can't be registered unless you prove you own the legacy domain.
(2) Some way for current domain holders to prove their ownership and ("claiming your inheritance.")

To seed the inherited set before the genesis block:

- Obtain a list of domain names.  Common crawl [1] seems to be not under NDA, although it's hosted by a proprietary vendor (AWS).
Will probably need prefunding (in USD) to pay for AWS nodes to walk the data and extract domain names.
- Common crawl data is ~100TB, but most of that is content.  Verisign official .com database appears to be about ~2 GB [2].  So the per-node storage requirements for an airdrop to all existing .com domains would be reasonable.
- The base list is called the "crawl set."  This covers domains everybody knows about.  So google, yahoo, facebook, reddit, etc. will surely be included.
  No humans make decisions about what does or doesn't make the cut, and nobody at those companies has to be convinced to do anything special.  Most organizations
  that people actually care about will be included.

After the genesis block, the inherited set can be grown as follows:

- Any registered TITAN account can submit, for free, up to ~8 domains every ~24 hours, which will considered by the network for inclusion.  This makes up the "ping set."
- ~20 randomly selected delegates will ping each domain in the ping-set over ~60-day period by doing a DNS lookup (in legacy DNS) and seeing if it resolves.  We check the signature on the ping,
  and check that the delegate was the one randomly selected by the random process.  But the DNS lookup itself is not audited, because we don't want to DDOS anybody!  We just take the delegate's
  word for whether the domain worked or not.
- If >50% of the pings agree that the domain appears to exist, the domain is moved from the ping set to the launch set.
- Pings are initiated by ordinary users to make sure domains aren't inadvertently excluded.
- We may have a browser extension or DNS proxy, to aid in gathering domain names.
- ~8 months after launch, new pings can no longer be submitted.
- ~10 months after launch, the last pings are resolved and the inherited set is now fixed for eternity.

If you register a cryptomain, the process goes like follows:

- If the cryptomain is not in the inherited set, great!  You got your shiny new cryptomain.
- If you register a cryptomain in the inherited set, the registration will be ineffective until you prove ownership of the legacy domain.
- If your previously registered cryptomain gets at least 5 pings, with >50% success rate, at any point in time, the cryptomain will be suspended until either unsuccessful ping(s) bring the rate
  below 50%, or you prove ownership of the legacy domain.

You can prove ownership of the legacy domain by entering (the hash of) your account's public key in a TXT record in legacy DNS, and using that public key to pay a fee to the network.  ~200
randomly selected delegates check that the TXT record exists and contains the correct key over a ~60 day period.  If at least 20 such checks have been performed with greater than 90% success,
then you have successfully proven ownership of the legacy domain.

So basically:

- Holders of domains that were crawlable at the Common Crawl used to initialize the genesis block will have their domain reserved in the inherited set.
- Holders of domains that got / became popular enough that at least one user submitted them during ~8 months post-genesis will have their domain reserved in the inherited set.
- If you register a new cryptomain during the first ~8 months, you need to control the corresponding legacy domain to ensure the cryptomain won't be suspended.

The owners of ".com" would not want to give up their control and influence for nothing so they would have to be allocated a large stake in the new system.  But getting a government to agree to this is like getting the voting dac adopted by governments.

The beauty of an airdrop is that neither the ICANN administrators, nor the domain holders, need to consent ahead of time.  They're simply given a stake in the system that they can claim at anytime.

[1] http://commoncrawl.org/

[2] http://www.leandomainsearch.com/blog/16-how-to-get-access-to-the-official-verisign--com-zone-file

What's up with invictusinnovations/bitshares then?  There are people committing to that one too.  It seems to have a bunch of junk related to email, chat and Keyhotee ID (about which I filed an issue [1] but it was ignored).

[1] https://github.com/InvictusInnovations/BitShares/issues/66

Lets address the issues here:

1) front running ... this particular issue does not apply given my matching algorithm.  The person placing the order will get the same price regardless of what any other party does.   In fact, one could argue that the blockchain earns revenue by intentional front-running and that the shareholders are the ones to benefit from it.   

Suppose every buyer wants to maximize the probability that their order is filled, subject to the constraint that the price is less than HPBWP (Highest Price Buyer Is Willing To Pay).  Obviously, a buyer can achieve these goals by setting a bid price equal to HPBWP.

However, the person who assembles the block might have a better option which achieves the same goals.  Since they know all other transactions in the block, they can compute exactly where in the order their transaction will be.  They can then compute the best Ask price at that point in the sequence, and set their order to that price (Best Ask At Time Of Execution or BAATOE).

There is value in the spread HPBWP - BAATOE.  One goal of BitShares X is to capture this value and return it to shareholders.  If the person who assembles the block has the capability to input their own transaction as the last transaction in the block, they can effectively circumvent the capture mechanism and extract the value for themselves.

My solution is a protocol design which requires all transactions to be signed by several randomly chosen notaries to be included in a block.  Thus a single notary or a small group of colluding notaries cannot perform the above attack.

Therefore, I am expecting the same kind of behavior from users of BTS X except instead of having to point their hashing power at a pool, they merely change a setting in their wallet.  This setting can even be setup with a chain of command so that users do not have to think much about it.  Instead of only miners voting, everyone can vote.

I wonder if "chain of command" means the concept I've been calling "transitive proxies."  By "transitive proxy" I mean a relationship where you can assign someone to vote on your behalf, and they can assign someone to vote on their behalf, et cetera.  It seems like a good idea because it lets small shareholders delegate their votes to someone they personally know and trust, who has a better grasp of the issues and responsibilities of voting, and perhaps spends more time following the BitShares world.  That person in turn can delegate their votes to someone they know and trust.  So everyone deals with people they know who can explain things, those to whom responsibility is dedicated often have a finite number of persons to please or persuade, and small shareholders can pass off the responsibility to stay informed and be available.

There are two potential problems with transitive proxies:  Cycles and long chains.

For example, say Eve has N small addresses with tiny balances.  If she combines them into a single large chain, she might be able to cause every node to do O(N^2) computation using O(N) transactions.  E.g. maybe she repeatedly creates a new address and tells it to delegate to the first address in her chain.  Then each added address requires walking all the way to the end of the chain to discover where the votes end up.

If try to fix the above problem by adding a field to the data structure to keep track of the ultimate destination where everyone's votes end up, then a chain built in the other direction becomes just as problematic.  So if Eve creates a new chain by always telling the last node to delegate to a different address, then changing the last node's delegation requires the ultimate destination of all prior nodes in the chain to be changed.

Okay, maybe we should just forbid long chains?  In that case, Eve can actually block a level 1 node from becoming a level 2 node by attaching a chain just under the length limit to it.  You might suppose the proper thing to do, then, is to allow the level-1 node to transition to level 2, and just cut off the now-too-long chain where it bumps against the length limit.  But this doesn't work either; if Eve is a level 1 node with followers, she can change from level 1 to level L-k by delegating to a chain of her own addresses, cutting off links between nodes further down the chain.  However, this may not be a problem.  It reduces Eve's vote total, reducing her influence on the network; the mischief she causes by causing the connections to be broken is less than she's costing herself to perform the attack.  In addition, now there's a public record that Eve's the sort of person who jumps levels, and should be obvious if the chain Eve's jumped to is a bunch of sock puppet accounts with tiny balances delegating to each other.

This post discusses an algorithm for choosing a proposer set randomly.  I suggest ordering the notaries by sorting with H(H(shiftreg) + H(notary_pubkey)) as
the sort key.  The value shiftreg contains the low bit from the last 128 hashes.  If we used the hash of the previous block instead, then a number of colluding
notaries might fall early in the order by chance.  Then they would be able to front-run the next block and manipulate the next block's hash at will.  They
could then "mine" a block hash that results in a favorable ordering for the *next* block hash, and maintain control for multiple blocks.  This attack would
leave irrefutable statistical evidence after a time.  But shiftreg is a trivial modification that makes it much harder for this attack to maintain control for several blocks:
If the colluders by luck obtain a favorable ordering, they can still front-run the next block and manipulate its hash, but they can only choose from two
possible orderings for the next block; unless the colluding group is very large, there will be a very high probability that neither choice allows them to
maintain control.  I call this attack the "notary ordering selection attack."

The proposer set is just the first TXSET_QUORUM notaries in the ordering; any that fail are replaced by the next ones in the ordering.