BitShares X Status Update Thread 2.0

[bytemaster]

We just started the latest dry run with BitAssets.

We have a Market GUI that is somewhat functional, but has much left to do.

[bytemaster]

BTS X with bitassets is now implemented and functional, but we are working on a few last minute market security features.  The basic code for DPOS + TITAN + Asset exchange is now stable enough for a MVP release of the toolkit to developers such as DAC Sun Limited and others. 

We will continue with dryruns with the latest market functions on the XT chain.     

[toast]

3) We are now preparing Dry Run 9 which will have Market Issued Assets (BitUSD!) available from the command line interface.

[bytemaster]

It has been a while since I have posted an update here because we have been doing 8 dry runs of BitShares X that have given everyone an opportunity to experience first hand where we are.   I would like to summarize the past month's progress now:

1) We have a working / tested Delegated Proof of Stake Chain operating at 10 transactions per second and 10 second blocks.
2) We have a usable Mac/Windows GUI wallet that supports User Issued Assets
3) We are now preparing Dry Run 9 which will have Market Issued Assets (BitUSD!) available from the command line interface.

A team is preparing BTSX for launch and will do so as soon as they feel comfortable.  They are preparing information, tutorials, etc to make sure the launch is as smooth as possible.

[bytemaster]

The latest test nets have proven very successful and we are preparing for a dry-run of the XT chain launch with some real delegates operating the chain.

Today I also figured out how to apply TITAN to multi-sig accounts and cross-chain-transfers. 

We are preparing a lot of documentation on the system as a whole and expect things to be operational soon.

[bytemaster]

We started a new test net and have started testing delegate registration and voting.   Sending to named accounts seems to be working. 

The web interface is coming along nicely and with any luck we should have a usable test release of the web interface this week. 

Check out https://bitsharestalk.org/index.php?topic=4836.0 for the status updates on testing.

[bytemaster]

We have successfully migrated the wallet to be based upon TITAN (Transfer Invisibly to Any Name) without having to hard-fork the blockchain.  To enhance maintainability and facilitate development of 3rd party APIs we have also switched to using code generation for a large part of the RPC API.   We can now use this JSON description of our API to auto-generate JavaScript wrappers for the API and help keep everything in sync as things change in the future.

The TITAN updates significantly enhance usability of the wallet as well as the privacy of the users.   The 97 character long addresses that you are swapping around today with every individual are now gone and instead you can publish a single ~40 character address once  (for everyone).  Then you can register this address (account) with the blockchain.

Users no longer deal with 100's of addresses or manually creating a new address for everyone they deal with.  You wallet is no longer a hodge-podge of 'pseudo-accounts' and instead you have real accounts each with their own balance and all transfers are 'from' one account 'to' another account.   Whether that other account is yours or someone elses. 

As a result the web interface will become very familiar to anyone who had used online banking + billpay.

[bytemaster]

There has been significant progress and developments over the past several days.   One of the most significant is the introduction of a random number generation scheme and random delegate selection today.   This was done for two reasons: it resolves a security vulnerability discovered with sequential DPOS and it facilitates one of the more challenging parts of the Lotto DAC.

We have implemented UPNP networking.

The web wallet is successfully transferring shares and is looking better by the day.

We have introduced a proposal system whereby delegates may introduce and vote on alerts to be displayed to the user.  This replaces the developer alert message that is built into bitcoin and facilitates reaching consensus in the event that a hard-fork is needed in the future.   

[bytemaster]

Today I worked through much of the blockchain fork handling code and have unit tests passing for forks of blocks that contain no data.  I also revamped / fixed bugs with many of the bugs that would have popped up during fork reorganization.   At this point I believe that forking is mostly handled in all cases and we just need a few more unit tests to prove it. 

In the process of debugging the fork handling code I produced some chain visualization tools:

[bytemaster]

Progress continues to advance as the team is getting better coordinated through the use of a development skype chat and improved process of using issues on github.  We are actively working on enhancing process to facilitate the rapid delegation of tasks to new users.

Today I implemented the fork handling algorithm at the blockchain level, but testing remains.  This evening I will be focused on producing the tests that can confirm that fork handling is working as planned.

[bytemaster]

I have updated the testing instructions here: http://bitshares.org/documentation/group__dpos__manual__testing.html

Primary changes are in the API usage and wallet management.    The primary goal is to get people away from sending to an 'address' and instead sending to an 'account' where the account is a hierarchal key.  Every time you send to an account it will use a different public key (address) and eventually this will allow automation of privacy enhancing transactions.   

We have also extended the API / implemented more of it.   See info here:

Code: [Select]

blockchain_get_block <block_hash>                                       Retrieves the block header for the given block hash
blockchain_get_block_by_number <block_number>                           Retrieves the block header for the given block number
blockchain_get_blockcount                                               Returns the number of blocks in the longest block chain.
blockchain_get_blockhash <block_num>                                    Returns hash of block in best-block-chain at index provided..
blockchain_get_delegates [first] [count]                                Returns the list of delegates sorted by vote
blockchain_get_name_record <name>                                       Retrieves the name record
blockchain_get_names [first] [count]                                    Returns the list of reserved names sorted alphabetically
blockchain_get_transaction <transaction_id>                             Get detailed information about an in-wallet transaction
get_info                                                                Provides common data, such as balance, block count, connections, and lock time
help [command]                                                          Lists commands or detailed help for specified command.
network_add_node <node> <command>                                       Attempts add or remove <node> from the peer list or try a connection to <node> once
network_get_connection_count                                            Returns the number of connections to other nodes.
network_get_peer_info                                                   Returns data about each connected node.
stop                                                                    Stop BitShares server
validate_address <address>                                              Return information about given BitShares address.
wallet_close                                                            Closes the curent wallet if one is open.
wallet_create <wallet_name> <password>                                  Opens the wallet of the given name
wallet_create_receive_account <account_name>                            Add new account for receiving payments
wallet_create_sending_account <account_name> <account_key>              Add new account for sending payments
wallet_get_account <account_name>                                       Lists all foreign addresses and their labels associated with this wallet
wallet_get_balance [account_name] [minconf] [asset]                     Returns the wallet's current balance
wallet_get_name                                                         Returns the wallet name passed to wallet_open
wallet_get_transaction_history [count]                                  Retrieves all transactions into or out of this wallet.
wallet_import_bitcoin <filename> <password>                             Import a BTC/PTS wallet
wallet_import_private_key <key> [account_name] [wallet_rescan_blockchain]   Import a BTC/PTS private key in wallet import format (WIF)
wallet_list_receive_accounts [start] [count]                            Lists all foreign addresses and their labels associated with this wallet
wallet_list_reserved_names [account_name]                               Lists all reserved names controlled by this wallet, filtered by account.
wallet_list_sending_accounts [start] [count]                            Lists all foreign addresses and their labels associated with this wallet
wallet_lock                                                             Lock the private keys in wallet, disables spending commands until unlocked
wallet_open <wallet_name> <password>                                    Opens the wallet of the given name
wallet_open_file <wallet_file> <password>                               Opens the wallet at the given path.
wallet_register_delegate <name> <data>                                  Registeres a delegate to be voted upon by shareholders.
wallet_rename_account <current_account_name> <new_account_name>         Lists all reserved names controlled by this wallet, filtered by account.
wallet_rescan_blockchain [starting_block]                               Rescan the block chain from the given block
wallet_rescan_blockchain_state                                          Rescans the genesis block and state (not the transactions)
wallet_reserve_name <name> <data>                                       Retrieves the name record
wallet_transfer <amount> <sending_account_name> [invoice_memo] [from_account] [asset_id]   Sends given amount to the given address, assumes shares in DAC
wallet_unlock <spending_pass> <timeout>                                 Unlock the private keys in the wallet to enable spending operations
If you are curious about the new transaction structure you can view an example json dump here:

Code: [Select]

{
           "expiration": null,
           "delegate_id": null,
           "operations": [{
               "type": "withdraw_op_type",
               "data": {
                 "balance_id": "XTSKh6JP6QzMmeUmmK1T16gxgadrJ547ocGT",
                 "amount": 154321,
                 "claim_input_data": ""
               }
             },{
               "type": "deposit_op_type",
               "data": {
                 "amount": 54321,
                 "condition": {
                   "asset_id": 0,
                   "delegate_id": 8,
                   "condition": "withdraw_signature_type",
                   "data": {
                     "owner": "XTSAg5YxD9reTXr5iQWBZEJrLzzMxMxvnFhy"
                   }
                 }
               }
             }
           ],
           "signatures": [
             "2028da68efe2696ab7895e86c1b35affbb7cd1b71d8f33124fe81ad785ace3ed55226e0e622098b9c45f8660801b4b929556102c3da549d887c5da99262b5c6275"
           ]
}

As you can see this transaction is entirely human readable outside the context of the blockchain,  it is transferring 154321 from XTSKh6JP6QzMmeUmmK1T16gxgadrJ547ocGT and sending 54321  to XTSAg5YxD9reTXr5iQWBZEJrLzzMxMxvnFhy while paying a fee of 100000. 

[bytemaster]

Today we reached a major milestone in our testing:

   1) Fully P2P Network Layer
   2) Fully DPOS block production
   3) Updated Blockchain Transaction / Database structure
   4) Updated manual testing instructions: http://bitshares.org/documentation/group__dpos__manual__testing.html
   5) Updated wallet backed by level_db

We will be making some renaming updates to follow a consistent convention as well as updating the wallet account tracking, most of this is done in a branch but waiting on a few last tweaks before merging into master.

We have implemented most of the logic for handling forks at the blockchain level, but have the following tasks remaining:

   1) design unit test for blockchain forks and fix any bugs discovered
   2) update P2P network code to handle synchronizing in the presence of forks.
   3) generate unit test for P2P network code fork synchronization

Once these changes are in place we will have a system that in theory could launch and be fully decentralized.  This system will be far more robust and contain far more backend functionality than the MVP we had originally planned.   So while we have slipped our MVP date, we have made great strides in system architecture and functionality.

[dannotestein]

Below is a graph of a transaction propagating on the bitshares peer-to-peer network.



The test network consisted of 50 nodes that each attempt to maintain a minimum of 3 connections and a maximum of 5 (we're using reduced connectivity to get longer paths between nodes in the network). This graph shows how a transaction generated by node 0 propagates to the other nodes in the network. Dashed lines indicate unused peer connections for each node, solid directed lines indicate the actual paths taken by the packet (including the propagation time across that link). This is all taking place on a single computer, so the slowdowns on later links is due to cpu bottlenecking for block validation rather than network propagation delays.

[bytemaster]

I have started a test network for XT...  https://bitsharestalk.org/index.php?topic=4480.msg56265#msg56265

I will be taking bug reports and issuing rapid updates in the week ahead.    If you are a developer give it a shot. 

[bytemaster]

Eric has been doing some great testing on the P2P code and validating the network connectivity graph.   As part of this testing we produced this graph:



The test network consisted of 50 nodes that each attempt to maintain 8 connections with a maximum of 12.  There are various simulated pauses used during the connecting phase.

[bytemaster]

Some major progress has been made in the past few days as we are rapidly moving toward feature completeness.

1) I have been able to use a web interface to transfer funds between wallets
2) Nikita is making great strides to making the initial web interface presentable
3) Today I implemented support for multi-sig, cross-chain-trading, and firing delegates who have provably signed invalid statements
4) I also updated the documentation on how blockchain validation works while I was reviewing the code, so you can now read the algorithm here:

   Block Validation
   http://bitshares.org/documentation/group__block__verification__algorithm.html
   
   Transaction Validation
   http://bitshares.org/documentation/group__transaction__verification__algorithm.html

We have been working very hard to finalize the RPC API for release so that others can build on top of it.   

[bytemaster]

Today I made more progress on the web interface which now supports the concept of logging in to open and decrypt your wallet.   It will also request your balance and display your user name.    In effect this is kind of like supporting multiple wallets under different users.   

Once logged in you can 'transfer' funds if you are connected to the network. 

It is not pretty, but I suspect you web wizards can fix that in no time flat.   So here is what you need to do to help me with the interface:

Edit the HTDOCS in the repository here to produce a beautiful interface that leverages the RPC calls which I will be posting shortly. 

https://github.com/BitShares/bitshares_toolkit/tree/master/programs/bts_xt/htdocs

The existing code shows everything you need to create a wallet and login.  It is VERY bare bones and needs to add a lot of extra features such as:  validating the password check (reentry) matches... verify password quality... all of which can be done in javascript in the browser prior to making the RPC call.

For those who can read the self documenting code you can view the available RPC calls here:

https://github.com/BitShares/bitshares_toolkit/blob/master/libraries/rpc/rpc_server.cpp  starting line 632

We will be adding many new calls tomorrow in an attempt to flush out every feature the GUI will require. 

To test the web interface you can use this command:

./bts_xt_client --data-dir clientc --trustee-address LEzgqiySszvp8VcZovD3tPh2jwHKWPmQD --server

For more information about doing the manual testing see this:

http://bitshares.org/documentation/group__manual__testing.html

[bytemaster]

Today Eric put a lot of effort into automated testing of multiple clients.  To write these tests he has improved the JSON-RPC interface to the clients significantly.   Additionally he produced the following document: http://bitshares.org/documentation/group__manual__testing.html  on how to perform manual testing.   I will be testing his manual testing document and improve upon it.

The manual testing is currently using the concept of a trustee to sign off on blocks.  Eventually the trustee will be replaced by the proper delegate once we are sure all of the delegate election code is solid.   So the current blockchain will elect delegates but not give them any signing authority. 

Today I updated the security of the BTS addresses to use 160 bits instead of 128 bits and removed the checksum from the binary representation.  This means that we increased security without increasing blockchain bloat.  We still use checksums for base58 representation so user generated addresses are now 7 bytes longer than before.   

Eric is setting up the unit tests so that we can have the same set of tests work with both client/server or P2P modes of operation and eventually also delegate modes.  We should have P2P fully integrated and tested this week under the trustee model.   At this point we will launch the chain with an intention of doing a hard-fork to activate DPOS once enough delegates have been elected and the system is sound.   

When we launch this chain to make BTS XT liquid, understand that if a bug is found we may reset the chain back to the snapshot.... so do not sell your positions in XT until after the code has been throughly reviewed.  During this testing phase we will be working with the exchanges to integrate our daemon and list BTS XT. 

Dan, Eric, and I are all working in the same room now in an effort to accelerate development and improve efficiency. 

[bytemaster]

Today I fixed the vote accounting to be based upon percentages.

I updated the xt_server to allocate initial votes among 100 delegates evenly and initialize the initial delegate IDs.

I reverted from 128 bit asset types to 64 bit asset types... this should reduce transaction size and make it easier for javascript developers. 

I also updated the code for calculating bips from shares and have established a minimum fee per byte of '1 satoshi' the lowest minimum fee possible. 

Tomorrow I will check the wallet code for signing up to be a delegate and configuring the trusted delegate list. 

Once all the accounting is verified I will move switch the trustee out for the current delegate. 

[bytemaster]

Two steps forward, one step backward...

1) Initialize genesis block with delegates
2) Blockchain detects initial delegate balances
3) Wallet now scans and detects delegates for which it has the keys
4) Wallet prints out delegates

Simulation and unit tests showed everything to be working but detected a small problem in the accounting:

1) Due to transaction fees, the number of future votes is always less than the number of past votes and thus delegates end up with perpetual votes that cannot be removed.  This is a problem.

After talking with Stan he had the great idea to do accounting for votes in BIPS instead of shares.  Initial white board calculations show that this will solve the problem.  Tomorrow I will update the accounting for this.

[bytemaster]

Today I spent time on the DPOS delegate vote tracking.  Votes now transfer between delegates and the wallet randomly selects one of their delegates each time a transaction is made.  This is a temporary fix to keep delegates load balanced on votes.

[bytemaster]

We are making daily progress on this code.  P2P code is being integrated by Dan N and Eric and I am working on DPOS.   

[bytemaster]

Today I met with Dan N. and Eric F. and they demonstrated the P2P code working nicely.  They will be integrating that work with the blockchain work I have done.   We are retargeting the Keyhotee team to focus on the GUI for BitShares XT wallet.   

We then had a solid discussion on Delegated Proof of Stake and I turned that into a white paper.

http://107.170.30.182/security/delegated-proof-of-stake.php

[bytemaster]

I have just posted a draft presentation of the talk for NYC and will be returning to the code tomorrow.  The recent developments regarding the design of the proof of work system to move beyond a single notary/trustee to a board of directors elected by the shareholders has been very fruitful and given me confidence that once again BTS X will be as fast as ripple while being more secure and truly decentralized.   It has been a long road and I am very grateful for everyone on this forum who has helped me think through and consider these difficult design decisions. 

For BTS XT I will keep a single trustee model with a plan to upgrade it later so that I do not hinder making XT users liquid.   I still plan on launching a test-test for BTS XT prior to NYC and then launching the final BTS XT network when I get back.  I didn't want any major emergencies while I was on travel and unable to respond in a timely manner.

[bytemaster]

Today I had some major revelations regarding the security model of the blockchain and thus managed to bypass the whole mining thing entirely.   I have posted my thoughts on this here: https://bitsharestalk.org/index.php?topic=3865.0

Essentially the role of producing a block can be left to trustee, the security is provided by TaPOS, and the trustee can be voted out at any time by the shareholders if they produce a single alternative block or are found to be rejecting valid transactions as part of a denial of service attack.    The trustee is an unpaid position and could operate behind a TOR node.   A DAC could set up a chain of command for trustees to handle rapid fail-over in the event that one trustee violates the trust.

As a result of this change we can now have validation times as fast as Ripple, no need to worry about chain forks, and yet the network can easily handle reallocation of the trustee position should that be necessary.

As a trustee you cannot double spend (you will be caught and fired).
As a trustee you cannot perform Denial of Service without being caught and fired.

This will be particularly helpful for BitShares X where attacks on block generation could really mess with the system and where chain forks are much more painful than they are with Bitcoin.

With these changes in place I can bypass a whole lot of complex code that is most likely to be the source of security concerns.  Initial testing shows that aside from some user interface and RPC updates BitShares XT is very close to being able to launch making the Feb 28th snapshot liquid. 

[bytemaster]

By popular demand I am creating this thread to facilitate tracking the status of BitShares X development.

At this time BitShares X has already had its snapshot taken (Feb 28th, 2014) and I am in the process of making that snapshot liquid with a chain that allows trading of shares and testing of the TaPOS system.

A quick summary of the reasons for delay after the snapshot:
1) Multiple market manipulation attacks were discovered that require protective measures
       -  the algorithm for determining when trading can begin based upon market depth must be implemented
       -  maximum price movement per block must be implemented
2) The ability to pay fees in any BitAsset seems to be a critical feature that was missing.
3) The ability to cover short positions using the collateral in the short rather than external funds should be in the base version.

While we are working on those features we also decided that it would be better to test the system in phases and increase functionality over time.  We decided to refactor the code to make it extensible for new DACs which are being developed in parallel (Thanks Toast & HackFisher).   This refactoring helps accelerate overall productivity at the expense of a couple of weeks delay in launch.    It is already paying dividends though so we think it was a good choice.

I will continue to post updates on the progress of BitShares X here.  This thread is locked to 3rd party comments, if you have any feedback to discuss this thread, please use the old BitShares X status update thread for general discussion.

Old Thread: https://bitsharestalk.org/index.php?topic=1890.0