!!!CAUTION!!! All public seed nodes down??

[abit]

Maybe not, are these are 4 connections on 1776 or 1779:

Code: [Select]

# netstat | grep 1776
tcp        0      0 seed05.bitsharesnodes:1779 81-89-101-133.blue:1776 ESTABLISHED
tcp        0     48 seed05.bitsharesnodes:1779 c-71-197-2-119.hsd:1776 ESTABLISHED
tcp        0      0 seed05.bitsharesnodes:1779 120.55.181.181:1776     ESTABLISHED
tcp        0      0 seed05.bitsharesnodes:1779 host-95-71-22-178.:1776 ESTABLISHED


Your end is 1779, the other end is 1776.

[abit]

Here is the first few lines in my config.ini file. I see no reason why it should use port 1779 instead of port 1776:

# Endpoint for P2P node to listen on
p2p-endpoint = 0.0.0.0:1776

# P2P nodes to connect to on startup (may specify multiple times)
#seed-node = 45.55.6.216:1776
#seed-node = 114.92.254.159:62015
#seed-node = seed01.bitsharesnodes.com:1776
#seed-node = seed02.bitsharesnodes.com:1776
#seed-node = seed03.bitsharesnodes.com:1776
#seed-node = seed04.bitsharesnodes.com:1776
#seed-node = seed05.bitsharesnodes.com:1776
#seed-node = seed06.bitsharesnodes.com:1776
#seed-node = seed07.bitsharesnodes.com:1776
#seed-node = 188.165.233.53:1777
#seed-node = 104.236.51.238:2005
#seed-node = 114.92.254.159:62015

As you can see the p2p-endpoint is defined to use 1776 and all of the seed definitions are built into the code, none are configured in the runtime config file. When I use netstat there are no listeners on 1776 but plenty on 1779, so the only conclusion I would be inclined to make is the definitions in the code must specify 1779 to override the p2p-endpoint defined in the config file.

You're starting with bts_tools right? @wackou any idea?

[Thom]

Maybe not, are these are 4 connections on 1776 or 1779:

Code: [Select]

# netstat | grep 1776
tcp        0      0 seed05.bitsharesnodes:1779 81-89-101-133.blue:1776 ESTABLISHED
tcp        0     48 seed05.bitsharesnodes:1779 c-71-197-2-119.hsd:1776 ESTABLISHED
tcp        0      0 seed05.bitsharesnodes:1779 120.55.181.181:1776     ESTABLISHED
tcp        0      0 seed05.bitsharesnodes:1779 host-95-71-22-178.:1776 ESTABLISHED


[Thom]

Here is the first few lines in my config.ini file. I see no reason why it should use port 1779 instead of port 1776:

# Endpoint for P2P node to listen on
p2p-endpoint = 0.0.0.0:1776

# P2P nodes to connect to on startup (may specify multiple times)
#seed-node = 45.55.6.216:1776
#seed-node = 114.92.254.159:62015
#seed-node = seed01.bitsharesnodes.com:1776
#seed-node = seed02.bitsharesnodes.com:1776
#seed-node = seed03.bitsharesnodes.com:1776
#seed-node = seed04.bitsharesnodes.com:1776
#seed-node = seed05.bitsharesnodes.com:1776
#seed-node = seed06.bitsharesnodes.com:1776
#seed-node = seed07.bitsharesnodes.com:1776
#seed-node = 188.165.233.53:1777
#seed-node = 104.236.51.238:2005
#seed-node = 114.92.254.159:62015

As you can see the p2p-endpoint is defined to use 1776 and all of the seed definitions are built into the code, none are configured in the runtime config file. When I use netstat there are no listeners on 1776 but plenty on 1779, so the only conclusion I would be inclined to make is the definitions in the code must specify 1779 to override the p2p-endpoint defined in the config file.

[dannotestein]

//Update:
I tried https://seed04.bitsharesnodes.com/info and found the p2p port is 1779. Is it supposed? If yes please submit a pull request to Github, like this: https://github.com/bitshares/bitshares-2/pull/14/files
All your other seed nodes are listening on port 1779 as well.

yes, that's the issue. You were still able to get connections because other active nodes know the "correct" port of 1779.

[abit]

How is it that the seeds can be restarted and regain 20 - 30 or more connections and yet fail the basic protocol test?

I restarted 2 of my nodes and waited for them to regain sync. That process looked quite normal. As soon as they were synced I reran this test:

[ -n "`echo EOF | nc seed05.bitsharesnodes.com 1776 -w 10 -q 2`" ] && echo Ok || echo Failed

and they still show failed.

I'm guessing part of the answer to that lies in the protocol. If I use that script on the P2P port it responds with Ok. Hence the issue has to do with the port for "new" connections on 1776, right?

As I said tho, restarting the nodes doesn't seem to affect that, the script still fails on port 1776 after the restart.

Tried "telnet seed05.bitsharesnodes.com 1776" and I got a response "Trying 82.211.31.175... telnet: Unable to connect to remote host: Connection refused", which means that server is not listening on port 1776. Is the IP address correct? And witness_node running?

//Update:
I tried https://seed04.bitsharesnodes.com/info and found the p2p port is 1779. Is it supposed? If yes please submit a pull request to Github, like this: https://github.com/bitshares/bitshares-2/pull/14/files
All your other seed nodes are listening on port 1779 as well.

[Thom]

How is it that the seeds can be restarted and regain 20 - 30 or more connections and yet fail the basic protocol test?

I restarted 2 of my nodes and waited for them to regain sync. That process looked quite normal. As soon as they were synced I reran this test:

[ -n "`echo EOF | nc seed05.bitsharesnodes.com 1776 -w 10 -q 2`" ] && echo Ok || echo Failed

and they still show failed.

I'm guessing part of the answer to that lies in the protocol. If I use that script on the P2P port it responds with Ok. Hence the issue has to do with the port for "new" connections on 1776, right?

As I said tho, restarting the nodes doesn't seem to affect that, the script still fails on port 1776 after the restart.

[dannotestein]

Ok, got it.

I'm noticing the traffic is dropping off now. Probably won't be long before nodes loose sync and we're all notified by the telegram bot of missing blocks.

The situation if getting urgent very quickly!

There's no obvious sign of an attack, so I don't think you need to worry about the network missing any blocks right now. Simply restarting the problematic seed nodes is a fine solution in the short term. The concern expressed is more about the possibility that someone knowledgeable enough could exploit the vulnerability.

[Thom]

Fox has explained already.

Where? I don't see it in this thread or telegram. Is this what you say is explanation? https://bitsharestalk.org/index.php/topic,22385.msg291616.html#msg291616


Anyway, restarting my nodes...

[Thom]

Ok, got it.

I'm noticing the traffic is dropping off now. Probably won't be long before nodes loose sync and we're all notified by the telegram bot of missing blocks.

The situation if getting urgent very quickly!

[abit]

If it is a successful attack, should I assume the same attack can accomplished to steem as well?

It's caused by a bug, which is exploitable. I believe steem has same issue.
Already notified BM.

@Thom : if a test returns "failed" on your node, please restart the node. Fox has explained already.

I got some time now. Let me write more. A seed node should always accept new connections when load is low. When a new node connected to it, it should respond with a "hello" message. The script above checks if a target is accepting connections and responding with any message. Wackou's old script perhaps checks the former only.

[liondani]

If it is a successful attack, should I assume the same attack can accomplished to steem as well?

[Thom]

Yeah, I get the same results.

I don't understand the logic of that script, or perhaps I don't understand the protocol. Looks like you send EOF to a node with netcat and wait 2 seconds for a response then terminate the connection. The script will wait up to 10 seconds to establish the connection. If the witness doesn't return a string the script result is failed otherwise it's ok.

Is EOF End of File or End of Frame? Either way the witness is supposed to respond to basically null data with an ack of some sort?

As I said I get the same result as you, yet as you say the nodes are all talking. Sorry if this sounds negative, but this is pretty much what frustrates me about how the devs in this community communicate. It's just too terse and abbreviated and I struggle to follow as a result. I think it's safe to say if transactions weren't being processed the indication would be more obvious.

You clearly have deeper knowledge of BitShares internals than I do, but I'm afraid you'll have to provide a few more details before I'll understand what the issue is. As for monitoring my nodes I am using wackou's bts_tools which is far more comprehensive at monitoring the nodes than I've seen from other sources. Not saying it's the best at it, there may be other tools which are better but I haven't seen them described or documented as well as bts_tools. The tools could use some refinement, but no code is perfect.

If you want me to do other tests or want me to check anything just let me know and I'll do my best to help you out.

Sorry I have no time to explain. Perhaps I'll get @wackou to talk with you, seems you trust him more. Best if @bytemaster has written a white paper for the protocol, but right now all info is in the code.

I'm empathetic to your lack of time, and this isn't an indictment on you personally abit. However if the only source of info about the problem is in the code after 6 months of it being released, that is IMO, a bigger more fundamental problem that speaks to lots of issues, such as lack of adoption and inability to entice developers to become interested in BitShares, which as I mentioned previously keeps me from wanting to get involved with STEEM as a witness.

Also, it's not a matter of trust, it's a matter of who can / is willing / has the time to provide information. Any "distrust" you may perceive probably has its roots in the questions I raise due to the lack of information I need to understand the issue / situation. Like you I don't have the time to decipher the C++ code to understand what it intends to accomplish.

I will offer my assistance to run whatever scripts or tests you may need me to run to gather data or provide logs etc. if that is useful to you. I suspect it won't be but short of other suggestions that's about as much as I can offer.

Good luck in tracking down the issue (sounds like a flaw in the code design to me) and keep us posted on your progress.

[Fox]

[...] I think it's safe to say if transactions weren't being processed the indication would be more obvious.

Please note: without any seed nodes accepting connections, the BitShares network is essentially invisible to new nodes attempting to connect, without manual and informed intervention. Obviously no transactions happening then.

Currently, it is conceivable to systematically DDOS individual witness nodes until none  (or is it only <11) remain to advance the chain state due to their inability to resync the blockchain following disconnect, given zero seed or known peer nodes which are actively synced and accepting connections.

@bytemaster, et al. devs: This is a series issue that needs to be resolved by analyzing the failed seed nodes (likely peer nodes are similarly impacted) identifying the root cause and issuing a patch or guidance on existing node re-configuration.   

Respectfully as always,
Fox

[abit]

Most nodes are still down. Or down again.

Code: [Select]

83.221.132.47:1776
Failed
188.40.91.213:1776
Failed
114.92.236.175:62015
Failed
71.197.2.119:1776
Ok
45.32.247.234:1776
Failed
82.211.31.175:1776
Failed
159.203.15.153:1776
Failed
188.166.188.206:1776
Failed
107.170.245.89:1777
Ok
54.85.252.77:39705
Failed
104.236.144.84:1777
Failed
40.127.190.171:1777
Failed
185.25.22.21:1776
Failed
23.95.43.126:50696
Failed
109.73.172.144:50696
Failed
128.199.143.47:2015
Failed

My node is stuck again. The last "accepted inbound connection" entry in p2p.log:

Code: [Select]

2016-05-05T06:45:19       p2p:accept_loop          accept_loop ] accepted inbound connection from 211.149.148.148:58502
        node.cpp:4178
2016-05-05T06:45:19 p2p:accept_connection_task         scope_logger ] entering peer_connection::accept_connection()
        peer_connection.cpp:190
2016-05-05T06:45:19 p2p:accept_connection_task    accept_connection ] error accepting connection 11 eof_exception: End Of File
End of file
    {"message":"End of file"}
    asio  asio.cpp:36 operator()                        peer_connection.cpp:217
2016-05-05T06:45:19 p2p:accept_connection_task        ~scope_logger ] leaving peer_connection::accept_connection()
        peer_connection.cpp:191

Hmm.. wait, why canceling accept_or_connect_task? and accept_or_connect_task completed?

Code: [Select]

p2p.log.20160505T060000:2016-05-05T06:46:00       p2p:accept_loop              destroy ] calling close_connection()
                peer_connection.cpp:127
p2p.log.20160505T060000:2016-05-05T06:46:00       p2p:accept_loop              destroy ] close_connection completed normally
                     peer_connection.cpp:129
p2p.log.20160505T060000:2016-05-05T06:46:00       p2p:accept_loop              destroy ] canceling _send_queued_messages task
                    peer_connection.cpp:142
p2p.log.20160505T060000:2016-05-05T06:46:00       p2p:accept_loop              destroy ] cancel_and_wait completed normally
                      peer_connection.cpp:144
p2p.log.20160505T060000:2016-05-05T06:46:00       p2p:accept_loop              destroy ] canceling accept_or_connect_task
                        peer_connection.cpp:157
p2p.log.20160505T060000:2016-05-05T06:46:00       p2p:accept_loop              destroy ] accept_or_connect_task completed normally                       peer_connection.cpp:159