BitShares Forum

Other => Graveyard => BitShares PTS => Topic started by: bobb on November 10, 2013, 11:48:53 pm

Title: Client keeps crashing
Post by: bobb on November 10, 2013, 11:48:53 pm
hey,

after updating the client (git pull from github) and recompiling the client, it quickly crashes.
I already tried to re-download the whole blockchain and started from scratch (just kept the wallet.dat), but it still doesn't work (regardless if I mine or not)

(running on linux 64bit, ubuntu 12.04, reproducible on various machines)

any ideas? thanks in advance.

Title: Re: client keeps crashing
Post by: victor on November 11, 2013, 12:02:40 am
I have the same issue on Ubuntu 13.10 64bit.

The old client worked fine, but the new one with new diff adjustment just crashes after a few minutes, with or without miner enabled. It is not even possible to do normal transactions.

I think the diff adjustment should be harmless, however, commit 776832f419649211cb7b173ffb437e7d2af68b8b could cause this problem since there are some big changes.
Title: Re: client keeps crashing
Post by: FreeTrade on November 11, 2013, 03:31:07 am
I think the diff adjustment should be harmless, however, commit 776832f419649211cb7b173ffb437e7d2af68b8b could cause this problem since there are some big changes.

Hi Victor,

Can you try commenting out this line -
emit miningChanged(dHashesPerSec, nThreads);

in src/qt/clientmodel.cpp (line 87)

and let us know if that helps?
Title: Re: client keeps crashing
Post by: victor on November 11, 2013, 04:18:01 am
I think the diff adjustment should be harmless, however, commit 776832f419649211cb7b173ffb437e7d2af68b8b could cause this problem since there are some big changes.

Hi Victor,

Can you try commenting out this line -
emit miningChanged(dHashesPerSec, nThreads);

in src/qt/clientmodel.cpp (line 87)

and let us know if that helps?

I am not sure if it is relevant, since I only build the CLI (bitcoind) without QT...
Title: Re: client keeps crashing
Post by: FreeTrade on November 11, 2013, 05:05:57 am
I am not sure if it is relevant, since I only build the CLI (bitcoind) without QT...

Ok - probably not relevant then. What do you see at the bottom of the log, immediately after the crash?
Title: Re: Client keeps crashing
Post by: bobb on November 11, 2013, 07:03:35 am
This is exactly what happens right before the crash on all systems after the update:

Error: System error: CDB() : can't open database file wallet.dat, error -30973
*** System error: CDB() : can't open database file wallet.dat, error -30973
Error: System error: CDB() : can't open database file wallet.dat, error -30973
ERROR: AcceptBlock() : AddToBlockIndex failed
ERROR: ProcessBlock() : AcceptBlock FAILED
dumpaddr thread stop
msghand thread interrupt
addcon thread interrupt
Flush(false)
wallet.dat refcount=0
wallet.dat checkpoint
wallet.dat detach
wallet.dat closed
DBFlush(false) ended               0ms
StopNode()
net thread interrupt
Flushed 3978 addresses to peers.dat  44ms
terminate called after throwing an instance of 'std::runtime_error'
  what():  CDB() : can't open database file wallet.dat, error -30973
Title: Re: Client keeps crashing
Post by: victor on November 11, 2013, 07:25:29 am
Crash log 1:

Code: [Select]
getblocks 12094 to 000013f13f50db53a53d758bc5806aff4208856cfb8f0b1c13ddeb985a821558 limit 500
  getblocks stopping at 12103 000013f13f50db53a53d758bc5806aff4208856cfb8f0b1c13ddeb985a821558
getblocks 12094 to 00001d75ecbde6b0d68856923465284a33532130a0b0c3d7f633b2cfd8391543 limit 500
  getblocks stopping at 12107 00001d75ecbde6b0d68856923465284a33532130a0b0c3d7f633b2cfd8391543
getblocks 12094 to 00001026c23e93e093b00fb5b2fdb3505ac6964a9e17834161493a9d53df2234 limit 500
  getblocks stopping at 12140 00001026c23e93e093b00fb5b2fdb3505ac6964a9e17834161493a9d53df2234
getblocks 12094 to 0000096bd5afd9bbdea554e820271bb596863c7d9123ccd89726c41d194601cb limit 500
  getblocks stopping at 12144 0000096bd5afd9bbdea554e820271bb596863c7d9123ccd89726c41d194601cb
getblocks 12094 to 00000debd784366414f0e74b710a5970634809a2d982206306960ca7829a3e2d limit 500
  getblocks stopping at 12146 00000debd784366414f0e74b710a5970634809a2d982206306960ca7829a3e2d
getblocks 12094 to 00001b2926f366440b32d5eae8c22f0b8ae215964f2c03b2e9b339c3dc2bb3f5 limit 500
  getblocks stopping at 12154 00001b2926f366440b32d5eae8c22f0b8ae215964f2c03b2e9b339c3dc2bb3f5
received getdata (59 invsz)
getblocks 12094 to 000015998ef76ec5b326e869972093bd325d39e5be40de22c147b3653834e3f5 limit 500
  getblocks stopping at 12097 000015998ef76ec5b326e869972093bd325d39e5be40de22c147b3653834e3f5
getblocks 12094 to 000015998ef76ec5b326e869972093bd325d39e5be40de22c147b3653834e3f5 limit 500
  getblocks stopping at 12097 000015998ef76ec5b326e869972093bd325d39e5be40de22c147b3653834e3f5
getblocks 12094 to 00001b454a10ada02429aa55aa552cd1bae1fb2b195acf0bca771fbb8a2f3e72 limit 500
  getblocks stopping at 12100 00001b454a10ada02429aa55aa552cd1bae1fb2b195acf0bca771fbb8a2f3e72
getblocks 12094 to 00001026c23e93e093b00fb5b2fdb3505ac6964a9e17834161493a9d53df2234 limit 500
  getblocks stopping at 12140 00001026c23e93e093b00fb5b2fdb3505ac6964a9e17834161493a9d53df2234
getblocks 12094 to 00000debd784366414f0e74b710a5970634809a2d982206306960ca7829a3e2d limit 500
  getblocks stopping at 12146 00000debd784366414f0e74b710a5970634809a2d982206306960ca7829a3e2d
received getdata (37 invsz)
getblocks 12094 to 000004c63a1ebb204c84fabf27cac560f0c1b164abf04c43bf9708fd3a745b53 limit 500
  getblocks stopping at 12101 000004c63a1ebb204c84fabf27cac560f0c1b164abf04c43bf9708fd3a745b53
getblocks 12094 to 0000028729900e6ad655cad22e7be7c4a940f5915a218a11cc7b696efb71e00c limit 500
  getblocks stopping at 12123 0000028729900e6ad655cad22e7be7c4a940f5915a218a11cc7b696efb71e00c
getblocks 12094 to 000004c7ab85ac74edca488399183448258990e81c747bb677b9abbf84520fe0 limit 500
  getblocks stopping at 12129 000004c7ab85ac74edca488399183448258990e81c747bb677b9abbf84520fe0
getblocks 12094 to 0000096bd5afd9bbdea554e820271bb596863c7d9123ccd89726c41d194601cb limit 500
  getblocks stopping at 12144 0000096bd5afd9bbdea554e820271bb596863c7d9123ccd89726c41d194601cb
received getdata (32 invsz)
socket recv error 104
disconnecting node 218.75.114.210:1072

Crash log 2:

Code: [Select]
connection timeout
trying connection 146.185.171.65:3888 lastseen=18.8hrs
connect() failed after select(): Connection refused
trying connection 54.221.104.112:3888 lastseen=132.2hrs
connection timeout
trying connection 71.82.34.194:3888 lastseen=13.8hrs
getblocks 12089 to 0000000000000000000000000000000000000000000000000000000000000000 limit 500
getblocks 12089 to 000010bb273ac0c42e2c9c30733b1a0ae9f78211bc06ce8b82bdeb2ca176956a limit 500
  getblocks stopping at 12106 000010bb273ac0c42e2c9c30733b1a0ae9f78211bc06ce8b82bdeb2ca176956a
received getdata (50 invsz)
Title: Re: Client keeps crashing
Post by: dzhan on November 11, 2013, 08:30:11 am
I have exactly the same problem.
Title: Re: Client keeps crashing
Post by: bobb on November 11, 2013, 08:37:00 am
I think  we are dealing with two related problems here.

When starting from scratch (without any former wallet.dat files) I experience the same behaviour victor already described.
It crashes and the last lines in the log file look like these:

Code: [Select]
getblocks stopping at 12105 00001fa9bc685efadcbdc3c65603e817d8e02c846b24f47660cd56368d7e71d1
getblocks 12089 to 00001d392ce5eaec0320429b14a66c25cbd574778f9d95fa8e0cba0cc8f502c1 limit 500
getblocks stopping at 12230 00001d392ce5eaec0320429b14a66c25cbd574778f9d95fa8e0cba0cc8f502c1

The even more concerning part is this that the client is not able to access the wallet files after these crashes.

Code: [Select]
Error: System error: CDB() : can't open database file wallet.dat, error -30973

Please note that the wallet issues are the same on a range of different machines.

Is there an official (64bit) linux build, that I can use to test?

It is kind of frustrating that there is no way to build a working linux client at the moment.
In combination with wallet error, this is kind of making me nervous right now.
I have a reasonable amount of coins in these wallets.

If I can help to track down/solve the problem, let me know.
Thanks in advance.
Title: Re: Client keeps crashing
Post by: bytemaster on November 11, 2013, 08:55:29 am
This is the bug that forced me to take down my pool.   It kept crashing on me.

I have been spending the past several hours trying to track down this issue.

Have not seen the database corruption though.
Title: Re: Client keeps crashing
Post by: bytemaster on November 11, 2013, 08:56:14 am
Caught this in GDB:
 0x00007ffff67581f7 in send () from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00000000004cc37c in SocketSendData (pnode=pnode@entry=0x7fffdc0048c0) at net.cpp:716
#2  0x00000000004d20bc in ThreadSocketHandler () at net.cpp:1031
#3  0x00000000004d306b in TraceThread<void (*)()> (name=0x5efc61 "net", func=0x4d1360 <ThreadSocketHandler()>) at util.h:591
#4  0x00007ffff7549242 in thread_proxy () from /usr/local/lib/libboost_thread.so.1.54.0
#5  0x00007ffff6750f8e in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6  0x00007ffff5c5ce1d in clone () from /lib/x86_64-linux-gnu/libc.so.6
Title: Re: Client keeps crashing
Post by: FreeTrade on November 11, 2013, 09:40:33 am
So I think we have two colliding networks - some bug is causing the command line version to be affected, but not the QT client.

As a stop-gap measure we could compile a list of misbehaving IPs and block them at a higher level.
Title: Re: Client keeps crashing
Post by: victor on November 11, 2013, 09:48:04 am
So I think we have two colliding networks - some bug is causing the command line version to be affected, but not the QT client.

As a stop-gap measure we could compile a list of misbehaving IPs and block them at a higher level.

I think we should make the new client more robust instead. Some users have not upgraded to the latest client, and when a new client is communicating with the old ones is just crashes. It is hard to block all these IPs; and what if they upgrade to the latest client?
Title: Re: Client keeps crashing
Post by: FreeTrade on November 11, 2013, 09:50:47 am
Are you guys seeing this in your log files much?

ERROR: AcceptBlock() : incorrect proof of work
ERROR: ProcessBlock() : AcceptBlock FAILED
Misbehaving: 60.247.75.39:3888 (0 -> 100) DISCONNECTING
disconnecting node 60.247.75.39:3888

I've got about 10 of these since the fork - it might be that these are not being handled correctly
Title: Re: Client keeps crashing
Post by: Vizierx on November 11, 2013, 09:52:31 am
So I think we have two colliding networks - some bug is causing the command line version to be affected, but not the QT client.

As a stop-gap measure we could compile a list of misbehaving IPs and block them at a higher level.

I think we should make the new client more robust instead. Some users have not upgraded to the latest client, and when a new client is communicating with the old ones is just crashes. It is hard to block all these IPs; and what if they upgrade to the latest client?
right you are - or do something like what to stop crush
Title: Re: Client keeps crashing
Post by: ryen123 on November 11, 2013, 11:47:08 am
Same here. I ran it at DigitalOcean and it keeps stopping after 15/20 minutes.
Title: Re: Client keeps crashing
Post by: JeffreyLee on November 11, 2013, 12:33:19 pm
I have exactly the same problem.Ubuntu 12.04.3 desktop and Ubuntu 12.04.3 Server
Title: Re: Client keeps crashing
Post by: craggietx on November 11, 2013, 03:25:31 pm
ditto here.   finally just shut down my servers last night as the client would crash within 10 secs of starting.   

Sounds like you guys are about to get this fixed.   Thanks for all the hard work!
Title: Re: Client keeps crashing
Post by: gigawatt on November 11, 2013, 04:50:55 pm
Running a few tests to see if I can replicate the issue.  I've got a watchdog wrapper for my protoshare client, so if it crashes I'll capture the debug.log file for analysis.

I'll let you know if I spot anything.
Title: Re: Client keeps crashing
Post by: bytemaster on November 11, 2013, 06:13:36 pm
Running a few tests to see if I can replicate the issue.  I've got a watchdog wrapper for my protoshare client, so if it crashes I'll capture the debug.log file for analysis.

I'll let you know if I spot anything.

We are aware of this issue and have a 1 BTC bounty for any patches that can help resolve it quickly. 
Title: Re: Client keeps crashing
Post by: FreeTrade on November 11, 2013, 06:32:18 pm
Testing the protosharesd daemon on windows . . . not seeing any of the problems as reported for linux. Not yet anyway. Think you might be looking for something linux specific - maybe something specific to a version of the boost libs.


Title: Re: Client keeps crashing
Post by: bytemaster on November 11, 2013, 06:33:06 pm
Testing the protosharesd daemon on windows . . . not seeing any of the problems as reported for linux. Not yet anyway. Think you might be looking for something linux specific - maybe something specific to a version of the boost libs.

It seems to be when an old node connects, can we add a checkpoint at 12097?
Title: Re: Client keeps crashing
Post by: victor on November 11, 2013, 08:49:44 pm
Running a few tests to see if I can replicate the issue.  I've got a watchdog wrapper for my protoshare client, so if it crashes I'll capture the debug.log file for analysis.

I'll let you know if I spot anything.

We are aware of this issue and have a 1 BTC bounty for any patches that can help resolve it quickly.

My quick patch below should fix this issue. Please test it.

http://pastebin.com/FQv3snaP

The cause of this problem is that the old client disconnects the new client due to the wrong proof of work; however there is a race condition that sometimes the new client still wants to communicate with the old client, but since the old client has closed this connection, a "broken pipe" signal will be passed to the new client and it is terminated.

Interestingly, on Windows platform a flag MSG_NOSIGNAL is passed to send() to avoid this problem. This should also apply to Linux since MSG_NOSIGNAL is supported after kernel 2.2+

Code: [Select]
--- ProtoShares/src/net.cpp     2013-11-11 11:46:13.019922787 -0800
+++ ProtoShares-new/src/net.cpp 2013-11-11 12:34:22.155890315 -0800
@@ -710,7 +710,7 @@
     while (it != pnode->vSendMsg.end()) {
         const CSerializeData &data = *it;
         assert(data.size() > pnode->nSendOffset);
-#ifdef WIN32
+#if 1
         int nBytes = send(pnode->hSocket, &data[pnode->nSendOffset], data.size() - pnode->nSendOffset, MSG_NOSIGNAL | MSG_DONTWAIT);
 #else
         int nBytes = send(pnode->hSocket, &data[pnode->nSendOffset], data.size() - pnode->nSendOffset, MSG_DONTWAIT);
Title: Re: Client keeps crashing
Post by: bytemaster on November 11, 2013, 08:51:05 pm
GREAT WORK! 
Title: Re: Client keeps crashing
Post by: dzhan on November 11, 2013, 11:05:12 pm
can you implement it into the existing code?
Title: Re: Client keeps crashing
Post by: JeffreyLee on November 12, 2013, 01:20:58 am
I had modified net.cpp ,It run good,not crashed any longer an hour.
-#ifdef WIN32
+#if 1
         int nBytes = send(pnode->hSocket, &data[pnode->nSendOffset], data.size() - pnode->nSendOffset, MSG_NOSIGNAL | MSG_DONTWAIT);
 #else
         int nBytes = send(pnode->hSocket, &data[pnode->nSendOffset], data.size() - pnode->nSendOffset, MSG_DONTWAIT);