BitShares Forum
Main => General Discussion => Topic started by: abit on May 04, 2016, 10:08:42 am
-
Someone is attacking BitShares? Or bug?
1535394ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 83.221.132.47:1776
1541939ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 188.40.91.213:1776
1541939ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 114.92.236.175:62015
1556942ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 71.197.2.119:1776
1561942ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 45.32.247.234:1776
1567965ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 82.211.31.175:1776
1571306ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 159.203.15.153:1776
1572544ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 188.166.188.206:1776
1573421ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 107.170.245.89:1777
1573421ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 54.85.252.77:39705
1573421ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 104.236.144.84:1777
1573421ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 40.127.190.171:1777
1573421ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 185.25.22.21:1776
1573422ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 23.95.43.126:50696
1573422ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 109.73.172.144:50696
1573422ms th_a application.cpp:177 reset_p2p_node ] Adding seed node 128.199.143.47:2015
~$ telnet 83.221.132.47 1776
Trying 83.221.132.47...
^C
~$ telnet 188.40.91.213 1776
Trying 188.40.91.213...
telnet: Unable to connect to remote host: Connection refused
~$ telnet 71.197.2.119 1776
Trying 71.197.2.119...
telnet: Unable to connect to remote host: Connection refused
~$ telnet 45.32.247.234 1776
Trying 45.32.247.234...
telnet: Unable to connect to remote host: Connection refused
~$ telnet 82.211.31.175 1776
Trying 82.211.31.175...
telnet: Unable to connect to remote host: Connection refused
~$ telnet 159.203.15.153 1776
Trying 159.203.15.153...
telnet: Unable to connect to remote host: Connection refused
~$ telnet 188.166.188.206 1776
Trying 188.166.188.206...
telnet: Unable to connect to remote host: Connection refused
~$ telnet 107.170.245.89 1777
Trying 107.170.245.89...
Connected to 107.170.245.89.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
~$ telnet 54.85.252.77 39705
Trying 54.85.252.77...
^C
~$ telnet 104.236.144.84 1777
Trying 104.236.144.84...
^C
~$ telnet 40.127.190.171 1777
Trying 40.127.190.171...
^C
~$ telnet 185.25.22.21 1776
Trying 185.25.22.21...
^C
~$ telnet 23.95.43.126 50696
Trying 23.95.43.126...
^C
~$ telnet 109.73.172.144 50696
Trying 109.73.172.144...
^C
~$ telnet 128.199.143.47 2015
Trying 128.199.143.47...
telnet: Unable to connect to remote host: Connection refused
After found it's not responding, I just restarted my seed node. A working seed node should respond like this:
~$ telnet 114.92.236.175 62015
Trying 114.92.236.175...
Connected to 114.92.236.175.
Escape character is '^]'.
]▒▒a▒X
▒o▒▒▒g▒▒s▒a▒Z▒f▒^]
-
Thanks for the alert. Restarted bitcube seed. It is working now.
-
And I thought it's just been my connection .. will check more closely when I run into this again
-
FYI steempty@steem made a simple bash script to monitor seed node:
[ -n "`echo EOF | nc steem.clawmap.com 2001 -w 10 -q 2`" ] && echo Ok || echo Failed
-
Are we talking BitShares or STEEM? The OP says BitShares, but your last post looks like STEEM. Don't you guys have a separate channel for STEEM witnesses, and why post here on BitShares General Discussion? For those witnesses like myself who aren't engaged as a STEEM witness (I find ramping up on STEEM project too frustrating, knowledge limited and scattered, making it hard to follow), please be clear so as not to give us false alerts.
According to my tools bitshares seeds (at least the 4 I run) have not shown any issues in the last 24 hours.
Thanks.
-
You misunderstood it. abit just illustrated a method to monitor the seed node of bitshares, and that method is got from steem.
Are we talking BitShares or STEEM? The OP says BitShares, but your last post looks like STEEM. Don't you guys have a separate channel for STEEM witnesses, and why post here on BitShares General Discussion? For those witnesses like myself who aren't engaged as a STEEM witness (I find ramping up on STEEM project too frustrating, knowledge limited and scattered, making it hard to follow), please be clear so as not to give us false alerts.
According to my tools bitshares seeds (at least the 4 I run) have not shown any issues in the last 24 hours.
Thanks.
-
Are we talking BitShares or STEEM? The OP says BitShares, but your last post looks like STEEM. Don't you guys have a separate channel for STEEM witnesses, and why post here on BitShares General Discussion? For those witnesses like myself who aren't engaged as a STEEM witness (I find ramping up on STEEM project too frustrating, knowledge limited and scattered, making it hard to follow), please be clear so as not to give us false alerts.
According to my tools bitshares seeds (at least the 4 I run) have not shown any issues in the last 24 hours.
Thanks.
@Thom Please try the script by yourself.
Your BitShares seed nodes are either refusing or not responding to new connections, although they are in sync (which means established connections and perhaps outgoing connections are working fine so far).
$ [ -n "`echo EOF | nc seed04.bitsharesnodes.com 1776 -w 10 -q 2`" ] && echo Ok || echo Failed
Failed
Thanks @ripplexiaoshan for the explanation.
-
Yeah, I get the same results.
I don't understand the logic of that script, or perhaps I don't understand the protocol. Looks like you send EOF to a node with netcat and wait 2 seconds for a response then terminate the connection. The script will wait up to 10 seconds to establish the connection. If the witness doesn't return a string the script result is failed otherwise it's ok.
Is EOF End of File or End of Frame? Either way the witness is supposed to respond to basically null data with an ack of some sort?
As I said I get the same result as you, yet as you say the nodes are all talking. Sorry if this sounds negative, but this is pretty much what frustrates me about how the devs in this community communicate. It's just too terse and abbreviated and I struggle to follow as a result. I think it's safe to say if transactions weren't being processed the indication would be more obvious.
You clearly have deeper knowledge of BitShares internals than I do, but I'm afraid you'll have to provide a few more details before I'll understand what the issue is. As for monitoring my nodes I am using wackou's bts_tools which is far more comprehensive at monitoring the nodes than I've seen from other sources. Not saying it's the best at it, there may be other tools which are better but I haven't seen them described or documented as well as bts_tools. The tools could use some refinement, but no code is perfect.
If you want me to do other tests or want me to check anything just let me know and I'll do my best to help you out.
-
Yeah, I get the same results.
I don't understand the logic of that script, or perhaps I don't understand the protocol. Looks like you send EOF to a node with netcat and wait 2 seconds for a response then terminate the connection. The script will wait up to 10 seconds to establish the connection. If the witness doesn't return a string the script result is failed otherwise it's ok.
Is EOF End of File or End of Frame? Either way the witness is supposed to respond to basically null data with an ack of some sort?
As I said I get the same result as you, yet as you say the nodes are all talking. Sorry if this sounds negative, but this is pretty much what frustrates me about how the devs in this community communicate. It's just too terse and abbreviated and I struggle to follow as a result. I think it's safe to say if transactions weren't being processed the indication would be more obvious.
You clearly have deeper knowledge of BitShares internals than I do, but I'm afraid you'll have to provide a few more details before I'll understand what the issue is. As for monitoring my nodes I am using wackou's bts_tools which is far more comprehensive at monitoring the nodes than I've seen from other sources. Not saying it's the best at it, there may be other tools which are better but I haven't seen them described or documented as well as bts_tools. The tools could use some refinement, but no code is perfect.
If you want me to do other tests or want me to check anything just let me know and I'll do my best to help you out.
Sorry I have no time to explain. Perhaps I'll get @wackou to talk with you, seems you trust him more. Best if @bytemaster has written a white paper for the protocol, but right now all info is in the code.
-
Most nodes are still down. Or down again.
83.221.132.47:1776
Failed
188.40.91.213:1776
Failed
114.92.236.175:62015
Failed
71.197.2.119:1776
Ok
45.32.247.234:1776
Failed
82.211.31.175:1776
Failed
159.203.15.153:1776
Failed
188.166.188.206:1776
Failed
107.170.245.89:1777
Ok
54.85.252.77:39705
Failed
104.236.144.84:1777
Failed
40.127.190.171:1777
Failed
185.25.22.21:1776
Failed
23.95.43.126:50696
Failed
109.73.172.144:50696
Failed
128.199.143.47:2015
Failed
My node is stuck again. The last "accepted inbound connection" entry in p2p.log:
2016-05-05T06:45:19 p2p:accept_loop accept_loop ] accepted inbound connection from 211.149.148.148:58502
node.cpp:4178
2016-05-05T06:45:19 p2p:accept_connection_task scope_logger ] entering peer_connection::accept_connection()
peer_connection.cpp:190
2016-05-05T06:45:19 p2p:accept_connection_task accept_connection ] error accepting connection 11 eof_exception: End Of File
End of file
{"message":"End of file"}
asio asio.cpp:36 operator() peer_connection.cpp:217
2016-05-05T06:45:19 p2p:accept_connection_task ~scope_logger ] leaving peer_connection::accept_connection()
peer_connection.cpp:191
Hmm.. wait, why canceling accept_or_connect_task? and accept_or_connect_task completed?
p2p.log.20160505T060000:2016-05-05T06:46:00 p2p:accept_loop destroy ] calling close_connection()
peer_connection.cpp:127
p2p.log.20160505T060000:2016-05-05T06:46:00 p2p:accept_loop destroy ] close_connection completed normally
peer_connection.cpp:129
p2p.log.20160505T060000:2016-05-05T06:46:00 p2p:accept_loop destroy ] canceling _send_queued_messages task
peer_connection.cpp:142
p2p.log.20160505T060000:2016-05-05T06:46:00 p2p:accept_loop destroy ] cancel_and_wait completed normally
peer_connection.cpp:144
p2p.log.20160505T060000:2016-05-05T06:46:00 p2p:accept_loop destroy ] canceling accept_or_connect_task
peer_connection.cpp:157
p2p.log.20160505T060000:2016-05-05T06:46:00 p2p:accept_loop destroy ] accept_or_connect_task completed normally peer_connection.cpp:159
-
[...] I think it's safe to say if transactions weren't being processed the indication would be more obvious.
Please note: without any seed nodes accepting connections, the BitShares network is essentially invisible to new nodes attempting to connect, without manual and informed intervention. Obviously no transactions happening then.
Currently, it is conceivable to systematically DDOS individual witness nodes until none (or is it only <11) remain to advance the chain state due to their inability to resync the blockchain following disconnect, given zero seed or known peer nodes which are actively synced and accepting connections.
@bytemaster, et al. devs: This is a series issue that needs to be resolved by analyzing the failed seed nodes (likely peer nodes are similarly impacted) identifying the root cause and issuing a patch or guidance on existing node re-configuration.
Respectfully as always,
Fox
-
Yeah, I get the same results.
I don't understand the logic of that script, or perhaps I don't understand the protocol. Looks like you send EOF to a node with netcat and wait 2 seconds for a response then terminate the connection. The script will wait up to 10 seconds to establish the connection. If the witness doesn't return a string the script result is failed otherwise it's ok.
Is EOF End of File or End of Frame? Either way the witness is supposed to respond to basically null data with an ack of some sort?
As I said I get the same result as you, yet as you say the nodes are all talking. Sorry if this sounds negative, but this is pretty much what frustrates me about how the devs in this community communicate. It's just too terse and abbreviated and I struggle to follow as a result. I think it's safe to say if transactions weren't being processed the indication would be more obvious.
You clearly have deeper knowledge of BitShares internals than I do, but I'm afraid you'll have to provide a few more details before I'll understand what the issue is. As for monitoring my nodes I am using wackou's bts_tools which is far more comprehensive at monitoring the nodes than I've seen from other sources. Not saying it's the best at it, there may be other tools which are better but I haven't seen them described or documented as well as bts_tools. The tools could use some refinement, but no code is perfect.
If you want me to do other tests or want me to check anything just let me know and I'll do my best to help you out.
Sorry I have no time to explain. Perhaps I'll get @wackou to talk with you, seems you trust him more. Best if @bytemaster has written a white paper for the protocol, but right now all info is in the code.
I'm empathetic to your lack of time, and this isn't an indictment on you personally abit. However if the only source of info about the problem is in the code after 6 months of it being released, that is IMO, a bigger more fundamental problem that speaks to lots of issues, such as lack of adoption and inability to entice developers to become interested in BitShares, which as I mentioned previously keeps me from wanting to get involved with STEEM as a witness.
Also, it's not a matter of trust, it's a matter of who can / is willing / has the time to provide information. Any "distrust" you may perceive probably has its roots in the questions I raise due to the lack of information I need to understand the issue / situation. Like you I don't have the time to decipher the C++ code to understand what it intends to accomplish.
I will offer my assistance to run whatever scripts or tests you may need me to run to gather data or provide logs etc. if that is useful to you. I suspect it won't be but short of other suggestions that's about as much as I can offer.
Good luck in tracking down the issue (sounds like a flaw in the code design to me) and keep us posted on your progress.
-
If it is a successful attack, should I assume the same attack can accomplished to steem as well?
-
If it is a successful attack, should I assume the same attack can accomplished to steem as well?
It's caused by a bug, which is exploitable. I believe steem has same issue.
Already notified BM.
@Thom : if a test returns "failed" on your node, please restart the node. Fox has explained already.
I got some time now. Let me write more. A seed node should always accept new connections when load is low. When a new node connected to it, it should respond with a "hello" message. The script above checks if a target is accepting connections and responding with any message. Wackou's old script perhaps checks the former only.
-
Ok, got it.
I'm noticing the traffic is dropping off now. Probably won't be long before nodes loose sync and we're all notified by the telegram bot of missing blocks.
The situation if getting urgent very quickly!
-
Fox has explained already.
Where? I don't see it in this thread or telegram. Is this what you say is explanation? https://bitsharestalk.org/index.php/topic,22385.msg291616.html#msg291616
Anyway, restarting my nodes...
-
Ok, got it.
I'm noticing the traffic is dropping off now. Probably won't be long before nodes loose sync and we're all notified by the telegram bot of missing blocks.
The situation if getting urgent very quickly!
There's no obvious sign of an attack, so I don't think you need to worry about the network missing any blocks right now. Simply restarting the problematic seed nodes is a fine solution in the short term. The concern expressed is more about the possibility that someone knowledgeable enough could exploit the vulnerability.
-
How is it that the seeds can be restarted and regain 20 - 30 or more connections and yet fail the basic protocol test?
I restarted 2 of my nodes and waited for them to regain sync. That process looked quite normal. As soon as they were synced I reran this test:
[ -n "`echo EOF | nc seed05.bitsharesnodes.com 1776 -w 10 -q 2`" ] && echo Ok || echo Failed
and they still show failed.
I'm guessing part of the answer to that lies in the protocol. If I use that script on the P2P port it responds with Ok. Hence the issue has to do with the port for "new" connections on 1776, right?
As I said tho, restarting the nodes doesn't seem to affect that, the script still fails on port 1776 after the restart.
-
How is it that the seeds can be restarted and regain 20 - 30 or more connections and yet fail the basic protocol test?
I restarted 2 of my nodes and waited for them to regain sync. That process looked quite normal. As soon as they were synced I reran this test:
[ -n "`echo EOF | nc seed05.bitsharesnodes.com 1776 -w 10 -q 2`" ] && echo Ok || echo Failed
and they still show failed.
I'm guessing part of the answer to that lies in the protocol. If I use that script on the P2P port it responds with Ok. Hence the issue has to do with the port for "new" connections on 1776, right?
As I said tho, restarting the nodes doesn't seem to affect that, the script still fails on port 1776 after the restart.
Tried "telnet seed05.bitsharesnodes.com 1776" and I got a response "Trying 82.211.31.175... telnet: Unable to connect to remote host: Connection refused", which means that server is not listening on port 1776. Is the IP address correct? And witness_node running?
//Update:
I tried https://seed04.bitsharesnodes.com/info and found the p2p port is 1779. Is it supposed? If yes please submit a pull request to Github, like this: https://github.com/bitshares/bitshares-2/pull/14/files
All your other seed nodes are listening on port 1779 as well.
-
//Update:
I tried https://seed04.bitsharesnodes.com/info and found the p2p port is 1779. Is it supposed? If yes please submit a pull request to Github, like this: https://github.com/bitshares/bitshares-2/pull/14/files
All your other seed nodes are listening on port 1779 as well.
yes, that's the issue. You were still able to get connections because other active nodes know the "correct" port of 1779.
-
Here is the first few lines in my config.ini file. I see no reason why it should use port 1779 instead of port 1776:
# Endpoint for P2P node to listen on
p2p-endpoint = 0.0.0.0:1776
# P2P nodes to connect to on startup (may specify multiple times)
#seed-node = 45.55.6.216:1776
#seed-node = 114.92.254.159:62015
#seed-node = seed01.bitsharesnodes.com:1776
#seed-node = seed02.bitsharesnodes.com:1776
#seed-node = seed03.bitsharesnodes.com:1776
#seed-node = seed04.bitsharesnodes.com:1776
#seed-node = seed05.bitsharesnodes.com:1776
#seed-node = seed06.bitsharesnodes.com:1776
#seed-node = seed07.bitsharesnodes.com:1776
#seed-node = 188.165.233.53:1777
#seed-node = 104.236.51.238:2005
#seed-node = 114.92.254.159:62015
As you can see the p2p-endpoint is defined to use 1776 and all of the seed definitions are built into the code, none are configured in the runtime config file. When I use netstat there are no listeners on 1776 but plenty on 1779, so the only conclusion I would be inclined to make is the definitions in the code must specify 1779 to override the p2p-endpoint defined in the config file.
-
Maybe not, are these are 4 connections on 1776 or 1779:
# netstat | grep 1776
tcp 0 0 seed05.bitsharesnodes:1779 81-89-101-133.blue:1776 ESTABLISHED
tcp 0 48 seed05.bitsharesnodes:1779 c-71-197-2-119.hsd:1776 ESTABLISHED
tcp 0 0 seed05.bitsharesnodes:1779 120.55.181.181:1776 ESTABLISHED
tcp 0 0 seed05.bitsharesnodes:1779 host-95-71-22-178.:1776 ESTABLISHED
-
Here is the first few lines in my config.ini file. I see no reason why it should use port 1779 instead of port 1776:
# Endpoint for P2P node to listen on
p2p-endpoint = 0.0.0.0:1776
# P2P nodes to connect to on startup (may specify multiple times)
#seed-node = 45.55.6.216:1776
#seed-node = 114.92.254.159:62015
#seed-node = seed01.bitsharesnodes.com:1776
#seed-node = seed02.bitsharesnodes.com:1776
#seed-node = seed03.bitsharesnodes.com:1776
#seed-node = seed04.bitsharesnodes.com:1776
#seed-node = seed05.bitsharesnodes.com:1776
#seed-node = seed06.bitsharesnodes.com:1776
#seed-node = seed07.bitsharesnodes.com:1776
#seed-node = 188.165.233.53:1777
#seed-node = 104.236.51.238:2005
#seed-node = 114.92.254.159:62015
As you can see the p2p-endpoint is defined to use 1776 and all of the seed definitions are built into the code, none are configured in the runtime config file. When I use netstat there are no listeners on 1776 but plenty on 1779, so the only conclusion I would be inclined to make is the definitions in the code must specify 1779 to override the p2p-endpoint defined in the config file.
You're starting with bts_tools right? @wackou any idea?
-
Maybe not, are these are 4 connections on 1776 or 1779:
# netstat | grep 1776
tcp 0 0 seed05.bitsharesnodes:1779 81-89-101-133.blue:1776 ESTABLISHED
tcp 0 48 seed05.bitsharesnodes:1779 c-71-197-2-119.hsd:1776 ESTABLISHED
tcp 0 0 seed05.bitsharesnodes:1779 120.55.181.181:1776 ESTABLISHED
tcp 0 0 seed05.bitsharesnodes:1779 host-95-71-22-178.:1776 ESTABLISHED
Your end is 1779, the other end is 1776.
-
Here is the first few lines in my config.ini file. I see no reason why it should use port 1779 instead of port 1776:
# Endpoint for P2P node to listen on
p2p-endpoint = 0.0.0.0:1776
# P2P nodes to connect to on startup (may specify multiple times)
#seed-node = 45.55.6.216:1776
#seed-node = 114.92.254.159:62015
#seed-node = seed01.bitsharesnodes.com:1776
#seed-node = seed02.bitsharesnodes.com:1776
#seed-node = seed03.bitsharesnodes.com:1776
#seed-node = seed04.bitsharesnodes.com:1776
#seed-node = seed05.bitsharesnodes.com:1776
#seed-node = seed06.bitsharesnodes.com:1776
#seed-node = seed07.bitsharesnodes.com:1776
#seed-node = 188.165.233.53:1777
#seed-node = 104.236.51.238:2005
#seed-node = 114.92.254.159:62015
As you can see the p2p-endpoint is defined to use 1776 and all of the seed definitions are built into the code, none are configured in the runtime config file. When I use netstat there are no listeners on 1776 but plenty on 1779, so the only conclusion I would be inclined to make is the definitions in the code must specify 1779 to override the p2p-endpoint defined in the config file.
You're starting with bts_tools right? @wackou any idea?
yes, you should look into the "~/.bts_tools/config.yaml" file that defines the clients to run, not in the config.ini file. There you can add the "p2p_port" field set to 1776, eg:
nodes:
bts:
p2p_port: 1776
...
and restart your seed node for the change to take effect.
-
A view of the status of all seed nodes can be seen here:
http://seed.bitsharesnodes.com/network/bts/seednodes
As you can see, the situation is pretty dire, so let's make this a wake up call:
SEED NODE OPERATORS, PLEASE FIX YOUR SEED NODES!!
-
A view of the status of all seed nodes can be seen here:
http://seed.bitsharesnodes.com/network/bts/seednodes
As you can see, the situation is pretty dire, so let's make this a wake up call:
SEED NODE OPERATORS, PLEASE FIX YOUR SEED NODES!!
This is very useful. +5% +5%
By the way my node got stuck again. Same situation: "accept_loop" is destroyed. However the latest incoming connection was successfully established, so it should not be the cause.
Will add more logs and check next time.
-
+5% for bts_tools!!
-
A view of the status of all seed nodes can be seen here:
http://seed.bitsharesnodes.com/network/bts/seednodes
As you can see, the situation is pretty dire, so let's make this a wake up call:
SEED NODE OPERATORS, PLEASE FIX YOUR SEED NODES!!
This is very useful. +5% +5%
By the way my node got stuck again. Same situation: "accept_loop" is destroyed. However the latest incoming connection was successfully established, so it should not be the cause.
Will add more logs and check next time.
I fixed mine today. Have you discovered why the seeds get "stuck"? You mention accept_loop so I presume you're looking into a bug in the code. Any more news or info to report?
-
Bump!
We're still low on reliable seed nodes, see http://seeds.quisquis.de/bitshares.html .
If you want to run a seed node you should apply this fix to avoid crashes:
https://github.com/abitmore/bitshares-2/commit/aa844f56faac4dda0d542520a672c4704fb0e617
And please use port 1776 for the p2p socket. That will make your node reachable under our new DNS seed seeds.bitshares.eu:1776 .
-
Please add my primary seed node to your list.
212.47.249.84:50696
-
Please add my primary seed node to your list.
212.47.249.84:50696
Hm... it should get added automatically, eventually. Provided that it is well-connected itself, and accepts connections.
-
Please add my primary seed node to your list.
212.47.249.84:50696
Hm... it should get added automatically, eventually. Provided that it is well-connected itself, and accepts connections.
Strange - my node is up and has been for some time according to http://status.bitsharesnodes.com/
-
Please add my primary seed node to your list.
212.47.249.84:50696
Hm... it should get added automatically, eventually. Provided that it is well-connected itself, and accepts connections.
Strange - my node is up and has been for some time according to http://status.bitsharesnodes.com/
Why is it strange? I ran all of my nodes on port 1779 for months and they always had between 40 and 60 connections each, despite them not being advertised. Now that they're running on port 1776 I get closer to 100 connections on each node.
Apparently the nodes have a way of finding each other. Also, for whatever reason, all 4 of my nodes have not experienced the "stuck" state for a full week now.
-
Please add my primary seed node to your list.
212.47.249.84:50696
Hm... it should get added automatically, eventually. Provided that it is well-connected itself, and accepts connections.
Strange - my node is up and has been for some time according to http://status.bitsharesnodes.com/
Why is it strange? I ran all of my nodes on port 1779 for months and they always had between 40 and 60 connections each, despite them not being advertised. Now that they're running on port 1776 I get closer to 100 connections on each node.
Apparently the nodes have a way of finding each other. Also, for whatever reason, all 4 of my nodes have not experienced the "stuck" state for a full week now.
iHashFury's node should get added with time as pc mentioned, as I guess his crawler hasn't been running for too long on the bitshares network and hasn't had the time to map the entire network yet. I believe there are more nodes in the bitshares network than the ~10 nodes on pc's page. Give it some time :)
As for you, Thom, what happened is that your nodes are declared to be on port 1776 in the official list of the bitshares client (see https://github.com/BitShares/bitshares-2/blob/bitshares/libraries/app/application.cpp#L154).
When they were running on port 1779 they were not acting as seed nodes and so were "just" full nodes. Now that you are on port 1776 you get connected to as seed nodes by people starting the client, hence the higher number of connections I guess.
-
there are quite a few seed nodes down or stuck again...
@lafona @puppies @betax @liondani @cube @iHashFury @Harvey
http://status.bitsharesnodes.com/
-
there are quite a few seed nodes down or stuck again...
@lafona @puppies @betax @liondani @cube @iHashFury @Harvey
http://status.bitsharesnodes.com/
Thanks for the heads-up.
-
we should get these out of the default seed node list for the next update ..
-
we should get these out of the default seed node list for the next update ..
Happy to remove 104.168.154.160:50696 from the default list as it not playing nice, my other seed node is solid.
-
there are quite a few seed nodes down or stuck again...
@lafona @puppies @betax @liondani @cube @iHashFury @Harvey
http://status.bitsharesnodes.com/
@abit was working on a patch but it has a compile problem. Abit, have you fixed it?
-
mine is online now.
-
there are quite a few seed nodes down or stuck again...
@lafona @puppies @betax @liondani @cube @iHashFury @Harvey
http://status.bitsharesnodes.com/
@abit was working on a patch but it has a compile problem. Abit, have you fixed it?
I thought I had replied earlier..
git remote add abit https://github.com/abitmore/bitshares-2.git
git fetch abit
git checkout 2.0.160328
git submodule update --init --recursive
git cherry-pick aa844f56faac4dda0d542520a672c4704fb0e617
cmake .
make
-
there are quite a few seed nodes down or stuck again...
@lafona @puppies @betax @liondani @cube @iHashFury @Harvey
http://status.bitsharesnodes.com/
@abit was working on a patch but it has a compile problem. Abit, have you fixed it?
I thought I had replied earlier..
git remote add abit https://github.com/abitmore/bitshares-2.git
git fetch abit
git checkout 2.0.160328
git submodule update --init --recursive
git cherry-pick aa844f56faac4dda0d542520a672c4704fb0e617
cmake .
make
I cannot find the commit aa844f56faac4dda0d542520a672c4704fb0e617. Can you show me the direct link to aa844f56faac4dda0d542520a672c4704fb0e617?
-
there are quite a few seed nodes down or stuck again...
@lafona @puppies @betax @liondani @cube @iHashFury @Harvey
http://status.bitsharesnodes.com/
@abit was working on a patch but it has a compile problem. Abit, have you fixed it?
I thought I had replied earlier..
git remote add abit https://github.com/abitmore/bitshares-2.git
git fetch abit
git checkout 2.0.160328
git submodule update --init --recursive
git cherry-pick aa844f56faac4dda0d542520a672c4704fb0e617
cmake .
make
I cannot find the commit aa844f56faac4dda0d542520a672c4704fb0e617. Can you show me the direct link to aa844f56faac4dda0d542520a672c4704fb0e617?
https://github.com/abitmore/bitshares-2/commit/aa844f56faac4dda0d542520a672c4704fb0e617
-
there are quite a few seed nodes down or stuck again...
@lafona @puppies @betax @liondani @cube @iHashFury @Harvey
http://status.bitsharesnodes.com/
@abit was working on a patch but it has a compile problem. Abit, have you fixed it?
I thought I had replied earlier..
git remote add abit https://github.com/abitmore/bitshares-2.git
git fetch abit
git checkout 2.0.160328
git submodule update --init --recursive
git cherry-pick aa844f56faac4dda0d542520a672c4704fb0e617
cmake .
make
I cannot find the commit aa844f56faac4dda0d542520a672c4704fb0e617. Can you show me the direct link to aa844f56faac4dda0d542520a672c4704fb0e617?
https://github.com/abitmore/bitshares-2/commit/aa844f56faac4dda0d542520a672c4704fb0e617
I applied this one-liner change and it causes a compile error. I mentioned the error in the telegram channel.
-
thanks guys on the quick reaction time!
@dannotestein are you the one in charge of the following seed nodes?
seed.blocktrades.us:1776
bitshares.openledger.info:1776
faucet.bitshares.org:1776
if not, do you happen to know who is?
-
the latter two should be removed.
faucet.bitshares.org does not exist anylonger
and openledger does not provide a public seed node.
-
we should get these out of the default seed node list for the next update ..
Happy to remove 104.168.154.160:50696 from the default list as it not playing nice, my other seed node is solid.
I confirm that my seed-node 185.25.22.21:1776 should be removed from the list also...
-
the latter two should be removed.
faucet.bitshares.org does not exist anylonger
and openledger does not provide a public seed node.
Shouldn't we build more nodes using the fund accumulated by "refund" worker?
-
Shouldn't we build more nodes using the fund accumulated by "refund" worker?
Either this, or ask witnesses to provide a seed node on top of the witness node, and vote out those who don't.
If the first case, then I would also fund Thom, who's been providing 4 seed nodes for a long time now without any sort of compensation.
-
Thx for the mention Nico.
BTW, have any of you witnesses noticed any changes to your nodes this week?
My witness node has missed a block every day (today it has already missed 2) since June 4th.
I have looked but see nothing in the disk / log space, memory and cpu use that correlates to the missed blocks.
I didn't spend any time confirming this, but I noticed a very odd (coincidence?) of me checking the Telegram bot on my WiFi connected cell phone to a single missed block. More than once I would pick up the phone, scroll over to the Telegram app, bring up the bot and boom! a missed block would occur. I dismissed it as coincidence, but was I too hasty? Seems like a stretch they are related tho...
-
My witness node has missed a block every day (today it has already missed 2) since June 4th.
I've had several misses on the 4th, 5th and 6th but none since then.
-
https://github.com/abitmore/bitshares-2/commit/aa844f56faac4dda0d542520a672c4704fb0e617
I applied this one-liner change and it causes a compile error. I mentioned the error in the telegram channel.
This is the error messages you posted in Telegram:
In file included from /root/bitshares-2/libraries/fc/include/fc/log/logger.hpp:5:0,
from /root/bitshares-2/libraries/fc/include/fc/exception/exception.hpp:6,
from /root/bitshares-2/libraries/fc/include/fc/thread/future.hpp:5,
from /root/bitshares-2/libraries/fc/include/fc/thread/task.hpp:2,
from /root/bitshares-2/libraries/fc/include/fc/thread/thread.hpp:5,
from /root/bitshares-2/libraries/net/node.cpp:61:
/root/bitshares-2/libraries/net/node.cpp: In member function âvoid graphene::net::detail::node_impl::accept_loop()â:
/root/bitshares-2/libraries/fc/include/fc/log/log_message.hpp:161:19: error: expected primary-expression before â(â token
fc::log_message( FC_LOG_CONTEXT(LOG_LEVEL), FORMAT, fc::mutable_variant_object()__VA_ARGS__ )
But I have no idea what caused it..
OS version? boost version? GCC version?
Try a clean clone?
You can also try to replace that line with:
} catch (...) {}
to catch exceptions without logging.
-
remove mine.. I had to put it down due to server / hosting moves.