BitShares Forum

Other => Graveyard => BitShares AGS => Topic started by: 5chdn on January 01, 2014, 05:18:55 pm

Title: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 01, 2014, 05:18:55 pm
I've written a ruby parser which displays AGS donations in realtime by parsing the BTC and PTS blockchains.

It's open source and available on github. https://github.com/donSchoe/ags-parser

See realtime parsing on my demo site  (v0.4):
http://q39.protoshar.es/ags/4/pts.csv.txt
http://q39.protoshar.es/ags/4/btc.csv.txt
http://q39.protoshar.es/ags/3/balances.json

Using git to backup the chain every hour: http://git.jdqirc.net/donschoe/ags-datafeed
http://git.jdqirc.net/donschoe/ags-datafeed/raw/master/4/pts.csv.txt
http://git.jdqirc.net/donschoe/ags-datafeed/raw/master/4/btc.csv.txt
http://git.jdqirc.net/donschoe/ags-datafeed/raw/master/3/balances.json

You can use it to create some nice angelshares sites and push the community forward. I'm no frontend guy, but would be nice to see this in action. The text version is enough for my personal needs :D
It's the first time I tried to programmatically parse a blockchain, damn nice experience! But dont expect magic and be aware of bugs!

Report them to me or on github or use this testing thread: https://bitsharestalk.org/index.php?topic=1989.0

If you like it, feel free to send some Coins my way:
BTC: 1Bzc7PatbRzXz6EAmvSuBuoWED96qy3zgc
PTS: PcDLYukq5RtKyRCeC1Gv5VhAJh88ykzfka

Cheers!



Urgent: Upgrade to v0.4 immidiatly: v0.3 is bugged, v0.1 API is inaccurate and will be disabled.



Overview: Sites using my API so far, donate some some coins too, if you like them:

Yo @earthbound, a tip would be nice.

http://agsexplorer.com

More features will be added.

BTC: 1Bb6V1UStz45QMAamjaX8rDjsnQnBpHvE8
PTS: PgULYtxcDJM7SKABfECipFiSjrmtjiPQd9
If you found my PHP front end and source code useful (http://joelooney.org/ags/ (http://joelooney.org/ags/)), you can tip me here (no pressure  :P)...

BTC:  16TsCgrKwQ3efzw6LkNuBHmvP6aJy21HGq
PTS:  PYDhxRmdd8YQdYLD757s7robuvafFdMARi

Thanks!

Note: this code will show the previous period's data until BTC and PTS donations are made in the current period.

Update: removed links to older versions.
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: maqifrnswa on January 01, 2014, 07:30:34 pm
Thanks for posting!
I see in the code it should reset for a new day but
http://q39.qhor.net/ags/pts.txt
didn't output what the code should. Don't know if it's a bug or you're running different code, just wanted to point it out
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: 5chdn on January 01, 2014, 07:36:12 pm
Thanks for posting!
I see in the code it should reset for a new day but
http://q39.protoshar.es/ags/4/pts.txt
didn't output what the code should. Don't know if it's a bug or you're running different code, just wanted to point it out

Thanks for your feedback!

The code will reset every day. Note that all payments from 2013 are counted to Jan/01/2014. Thats why you dont see any reset yet.
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: Evan on January 01, 2014, 07:47:28 pm
Is the BTC parser updating properly?  I sent BTC to the AGS address yesterday, but the last entry on the BTC parser is dated 2013-12-29.

Edit:  I see it's updating now.
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: seraphim on January 01, 2014, 08:20:28 pm
Wow, great!
We definitely needed that!
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: 5chdn on January 01, 2014, 08:30:16 pm
Is the BTC parser updating properly?  I sent BTC to the AGS address yesterday, but the last entry on the BTC parser is dated 2013-12-29.

Edit:  I see it's updating now.

Yes the BTC chain is still being parsed. The bitcoin network is huge and thats why it will take some hours to catch up!

The bot on IRC now tells us the value of AGS:

21:34:30     donschoe | .ags pts
21:34:30   +sternburg | Angleshares value: 2.247 AGS/PTS = 8.97 USD. Total donations today: 2225.249 PTS.
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: toast on January 01, 2014, 08:43:17 pm
Is the BTC parser updating properly?  I sent BTC to the AGS address yesterday, but the last entry on the BTC parser is dated 2013-12-29.

Edit:  I see it's updating now.

Yes the BTC chain is still being parsed. The bitcoin network is huge and thats why it will take some hours to catch up!

The bot on IRC now tells us the value of AGS:

21:34:30     donschoe | .ags pts
21:34:30   +sternburg | Angleshares value: 2.247 AGS/PTS = 8.97 USD. Total donations today: 2225.249 PTS.

what channel is this bot in?
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: 5chdn on January 01, 2014, 08:56:54 pm
The bot on IRC now tells us the value of AGS:

21:34:30     donschoe | .ags pts
21:34:30   +sternburg | Angleshares value: 2.247 AGS/PTS = 8.97 USD. Total donations today: 2225.249 PTS.

what channel is this bot in?

on freenode:
Code: [Select]
    bot.rooms.new("#memorycoin").join
    bot.rooms.new("#extasie").join
    bot.rooms.new("##mmc-spam").join
    bot.rooms.new("#protoshares").join
    bot.rooms.new("#bitsharestalk").join
    bot.rooms.new("##pts-spam").join
    bot.rooms.new("##primecoin").join
    bot.rooms.new("#beeeeer.org").join
    bot.rooms.new("##xpm-spam").join
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: bytemaster on January 01, 2014, 09:34:30 pm
The bot on IRC now tells us the value of AGS:

21:34:30     donschoe | .ags pts
21:34:30   +sternburg | Angleshares value: 2.247 AGS/PTS = 8.97 USD. Total donations today: 2225.249 PTS.

what channel is this bot in?

on freenode:
Code: [Select]
    bot.rooms.new("#memorycoin").join
    bot.rooms.new("#extasie").join
    bot.rooms.new("##mmc-spam").join
    bot.rooms.new("#protoshares").join
    bot.rooms.new("#bitsharestalk").join
    bot.rooms.new("##pts-spam").join
    bot.rooms.new("##primecoin").join
    bot.rooms.new("#beeeeer.org").join
    bot.rooms.new("##xpm-spam").join

Awesome work guys!
Title: Re: [ANN] Open Source AGS-Blockchain-Parser
Post by: bytemaster on January 01, 2014, 09:34:57 pm
Who can I send a 1 PTS tip to for this bot?
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: cass on January 02, 2014, 11:14:41 am
amazing work ... thx donschoe pls provide tip jar
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 02, 2014, 11:29:32 am
Who can I send a 1 PTS tip to for this bot?
amazing work ... thx donschoe pls provide tip jar
As mentioned in the first post, the donation addresses are: 1Bzc7PatbRzXz6EAmvSuBuoWED96qy3zgc or PcDLYukq5RtKyRCeC1Gv5VhAJh88ykzfka  8)

Thanks! If you have any feature requests let me know!
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: aasl on January 02, 2014, 01:58:22 pm
nice work!
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: cass on January 02, 2014, 10:39:08 pm
https://coinplorer.com/PTS/Transactions/bb2c83d49d66f253a949b902bf3ea885ff25c325fb309cf4e175a9a466fc880a

Donations sent - Danke Donschoe für deinen Einsatz ;) thx for your efforts to the community

Cass
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 03, 2014, 08:47:46 am
Thanks for all the donations! :)

And now vote me for MMC executive!
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: mint chocolate chip on January 03, 2014, 10:46:30 am
Really nice, easy to understand the daily breakdown looking at this, I'll surely be revisiting it over the next couple hundred days
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 03, 2014, 11:02:00 am
You can use command line args to start parsing at a certain day. To get today's stats only, just run it with:

Code: [Select]
$ ruby pts_chain.rb 38069
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: stuartcharles on January 03, 2014, 11:30:35 am
great work
Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: wasthatawolf on January 04, 2014, 06:08:42 pm
With the help of Bootstrap, I hacked together a page using PHP to beautify donshoe's parsed txt files.  Still a work in progress.

http://joelooney.org/ags/



Title: Re: [ANN][AGS] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 04, 2014, 07:27:50 pm
With the help of Bootstrap, I hacked together a page using PHP to beautify donshoe's parsed txt files.  Still a work in progress.

http://joelooney.org/ags/

Nice, pls check if this helps you:

UPDATE: CLEAN CSV FILES HERE:
http://q39.protoshar.es/ags/4/pts.csv.txt
http://q39.protoshar.es/ags/4/btc.csv.txt

These files are improved.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: wasthatawolf on January 04, 2014, 08:14:31 pm
I ended up using the daily summary data break as a checkpoint for collecting data for each day.  I'm not sure how much faster the page would load at this point if I used the cleaner txt file.

The several second load time is likely due to parsing the entire txt file for both btc and pts each time.

What I think I'll do to speed it up is store daily summary information (top deposit, number of deposits, sum of deposits) in a MySQL database.  Then, all I would need is a csv file with the current period data for each blockchain.

Great work by the way.  Thanks for parsing the two blockchains and posting the live data.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: boombastic on January 04, 2014, 09:50:17 pm
I've put up a frond-end based on donschoe's daily parsed data input.  You can check it out at:

http://agsexplorer.com/

I was trying to build on live blockchains, but since I am new working with blockchain, still have some issues to solve.  donschoe's data is good enough for now.  Thanks.

There's one question, the time in the txt file looks not consistent with the same transaction found on blockchained.info. Isn't the time should be equal to the block in which the transaction is included?
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 04, 2014, 10:07:32 pm
I've put up a frond-end based on donschoe's daily parsed data input.  You can check it out at:

http://agsexplorer.com/

I was trying to build on live blockchains, but since I am new working with blockchain, still have some issues to solve.  donschoe's data is good enough for now.  Thanks.

There's one question, the time in the txt file looks not consistent with the same transaction found on blockchained.info. Isn't the time should be equal to the block in which the transaction is included?

This looks damn awesome!

I'm using the timestamp of the block where the transaction is picked up in. Maybe the time of the transaction can be issued earlier. I'm not sure - does that matter?

Please rethink using the v0.2 csv files. They are basicly the same but I will shut down v0.1 API soon as they are not accurate enough!

I ended up using the daily summary data break as a checkpoint for collecting data for each day.  I'm not sure how much faster the page would load at this point if I used the cleaner txt file.

The several second load time is likely due to parsing the entire txt file for both btc and pts each time.

What I think I'll do to speed it up is store daily summary information (top deposit, number of deposits, sum of deposits) in a MySQL database.  Then, all I would need is a csv file with the current period data for each blockchain.

Great work by the way.  Thanks for parsing the two blockchains and posting the live data.

You also may want to upgrade to the v0.2 API which has more accurate results. For daily statistics you can use the last line of each day.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: boombastic on January 04, 2014, 10:10:59 pm
Thanks, will check out v2 tomorrow. Gotta go to bed now, it's already morning here in Shanghai.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: wasthatawolf on January 04, 2014, 10:50:32 pm
Do you plan to shut down the v0.1 API once the bitcoin blockchain is finished parsing in v0.2?
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 04, 2014, 11:48:32 pm
No, i will keep running the v0.1 untill my own services are migrated. but just be safe and upgrade soon.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: boombastic on January 05, 2014, 06:23:26 pm
hi donschoe,

I've upgraded to v2 api and found some problems.

The subtotal seems to be incorrectly calculated.  It seems it only happen when several donation transactions happen within the same blocks.

Here are some snippets of 01-15's data

Quote
"278645";"2014-01-05 01:09:04 UTC";"1M16SzSwY9RXpxadEyQ6vdjnNUntk4iogu";"0.12345";"0.22445";"22276.67631989"
"278645";"2014-01-05 01:09:04 UTC";"1765s6HMMane9jMf74Nsop2eBCXbtQUWCv";"0.01313365";"0.72445";"6901.78756298"
"278645";"2014-01-05 01:09:04 UTC";"16PtgDmLb8gixJZKAzgXa8PWeyos9n4vur";"0.48686635";"0.72445";"6901.78756298"

You will see the second line's total is incorrect, the same pattern repeats and therefore the last line total is untrustful.  Please verify.

Another question, if one donation transaction contains multiple inputs, which input address will you credit as donation credit? inputs[0]?
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 05, 2014, 08:28:47 pm
@boombastic, i can answer all your questions at once:

if you see the same sum value, its a single transaction. its not desired but no big issue. will fix that in future releases. edit: fixed (https://github.com/donSchoe/ags-parser/commit/40b91b6b0bbc697af5aaf35fe8e5f1888c6a6506).

multiple input addresses are all displayed by the script. there is a discussion goin on here you might wish to join https://bitsharestalk.org/index.php?topic=2036.0
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: boombastic on January 05, 2014, 08:53:16 pm
Thanks @donschoe, I will follow that post.


Sent from my iPhone using Tapatalk (http://tapatalk.com/m?id=1)
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: wasthatawolf on January 05, 2014, 11:05:30 pm
Updated with v0.2 data - Angelshares Summary Data

http://joelooney.org/ags/ (http://joelooney.org/ags/)

Source: http://pastebin.com/w7quHJeM (http://pastebin.com/w7quHJeM)

Decided to keep it simple.  PHP only, this should be a good solution for anyone that wants to run the code and view Angelshares summary data locally with no javascript.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: earthbound on January 05, 2014, 11:46:31 pm
Y'all who've made pretty and more detailed front ends for this, I'd like to tip you (in addition to donschoe).

Where should I send the tips?

donschoe, if they provide addresses, will you please update your OP with info to tip to them as well?

One other thing: it'd be nice if these listed the time zone and time at which a donation period resets (for example, in donschoe's version, this could be given in parenthesis after the time countdown). I forget when that daily reset is . . . is it midnight Greenwich Mean Time?

Thanks!
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: wasthatawolf on January 06, 2014, 12:03:45 am
If you found my PHP front end and source code useful (http://joelooney.org/ags/ (http://joelooney.org/ags/)), you can tip me here (no pressure  :P)...

BTC:  16TsCgrKwQ3efzw6LkNuBHmvP6aJy21HGq
PTS:  PYDhxRmdd8YQdYLD757s7robuvafFdMARi

Thanks!

Note: this code will show the previous period's data until BTC and PTS donations are made in the current period.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: boombastic on January 06, 2014, 05:31:03 am
Yo @earthbound, a tip would be nice.

http://agsexplorer.com

More features will be added.

BTC: 1Bb6V1UStz45QMAamjaX8rDjsnQnBpHvE8
PTS: PgULYtxcDJM7SKABfECipFiSjrmtjiPQd9
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 06, 2014, 10:23:07 am
Updated OP, added new feature.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: earthbound on January 06, 2014, 03:51:15 pm
Sorry to say they may be paltry tips, but I'll get to them . . .
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: bytemaster on January 06, 2014, 10:20:38 pm
Can this script be modified to generate a list of all unspent outputs necessary for building a genesis block?  I believe there is another bounty where such a modified script would be helpful to win.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 06, 2014, 10:44:04 pm
Can this script be modified to generate a list of all unspent outputs necessary for building a genesis block?  I believe there is another bounty where such a modified script would be helpful to win.

Not sure what you mean by "unspent outputs"? It lists all holding addresses and their connected AGS balance, I thought that was the info needed to build a genesis block, but I never did that before, just let me know what else you need, the script is easily modified.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 06, 2014, 11:58:17 pm
will release api version 0.3 tomorrow and shut down 0.1 as soon as possible. upgrade your applications  8)
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: bytemaster on January 07, 2014, 12:39:39 am
Can this script be modified to generate a list of all unspent outputs necessary for building a genesis block?  I believe there is another bounty where such a modified script would be helpful to win.

Not sure what you mean by "unspent outputs"? It lists all holding addresses and their connected AGS balance, I thought that was the info needed to build a genesis block, but I never did that before, just let me know what else you need, the script is easily modified.

This was directed more toward not honoring AGS but honoring PTS in a new blockchain.  If there was a website that provided the current unspent balance of every output in the blockchain then someone wanting to build a new genesis block would have a very easy task.  Namely, import the table as a single send to many transaction. 
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 07, 2014, 09:55:01 am
Can this script be modified to generate a list of all unspent outputs necessary for building a genesis block?  I believe there is another bounty where such a modified script would be helpful to win.

Not sure what you mean by "unspent outputs"? It lists all holding addresses and their connected AGS balance, I thought that was the info needed to build a genesis block, but I never did that before, just let me know what else you need, the script is easily modified.

This was directed more toward not honoring AGS but honoring PTS in a new blockchain.  If there was a website that provided the current unspent balance of every output in the blockchain then someone wanting to build a new genesis block would have a very easy task.  Namely, import the table as a single send to many transaction.

Ah no that's not so easy. Got to think about that. How did FreeTrade do it?
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: wasthatawolf on January 07, 2014, 02:17:24 pm
Seems to be an issue with the BTC API v0.2, a lot of blank space and the earliest donation shown is at 2014-01-07 12:29:52 UTC
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 07, 2014, 02:47:56 pm
Seems to be an issue with the BTC API v0.2, a lot of blank space and the earliest donation shown is at 2014-01-07 12:29:52 UTC

Not sure what happened. Better upgrade to v0.3 immidiatly:
<removed>

I will keep v0.1 running until v0.2 is fixed.



A bug was introduced in v0.3, upgrade to v0.4 immidiatly.

http://q39.protoshar.es/ags/4/pts.csv.txt
http://q39.protoshar.es/ags/4/btc.csv.txt
http://q39.protoshar.es/ags/3/balances.json

I promise not to change the API unless there are bugs found. This v0.4 will be final. I made some in-depth checks on that one, this should be really, really accurate now!
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 09, 2014, 11:20:09 am
I'm using git to backup the blockchains. http://git.jdqirc.net/donschoe/ags-datafeed

Updating the chain every hour, you can use the raw feed:
http://git.jdqirc.net/donschoe/ags-datafeed/raw/master/4/pts.csv.txt
http://git.jdqirc.net/donschoe/ags-datafeed/raw/master/4/btc.csv.txt
http://git.jdqirc.net/donschoe/ags-datafeed/raw/master/3/balances.json

I hope in future I dont have to reparse the blockchain so often!
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: earthbound on January 09, 2014, 05:19:53 pm
Okay, no, really, seriously, I'll get to those paltry donations :) When I do, I'll edit this post to say: [Date blah blah blah--sent.]

It's just that I've been forced to re-download the Bitcoin and Protoshares blockchains, so that I can't send any transactions from my personal wallets, yet. Ah, snap! Maybe I can make use of the git repository donschoe posted . . .

Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: earthbound on January 10, 2014, 06:55:20 pm
Three mugs O' beer finally sent :)
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: wasthatawolf on January 10, 2014, 07:19:28 pm
Thanks!
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on January 10, 2014, 09:25:35 pm
Three mugs O' beer finally sent :)

Thanks earthbound :)
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: Evan on February 05, 2014, 09:25:59 pm
The AGS parser demo site does not appear to be updating at this moment.

The timestamp of the last BTC donation - 2014-02-05 05:53:09 UTC
The timestamp of the last PTS donation - 2014-02-05 07:27:13 UTC

I have personally made both BTC and PTS donations since the timestamp of the most recent parsed transaction, so I would expect the timestamps to be more recent.
Title: Re: [ANN][AGS-PRASER] Open Source Angelshares-Blockchain-Parser
Post by: 5chdn on February 10, 2014, 11:52:05 am
The AGS parser demo site does not appear to be updating at this moment.

The timestamp of the last BTC donation - 2014-02-05 05:53:09 UTC
The timestamp of the last PTS donation - 2014-02-05 07:27:13 UTC

I have personally made both BTC and PTS donations since the timestamp of the most recent parsed transaction, so I would expect the timestamps to be more recent.

The machine running these example APIs died on feb/05. I will setup a better machine soon and publish it here. Until then you have to live with the other sites available. Sorry for this :)