AWS connections

Posted on
Sun Feb 18, 2018 8:32 am
snowjay offline
Posts: 274
Joined: Aug 09, 2006

AWS connections

Any idea what would cause the plugin to spawn (and seemingly orphan) all these AWS connections?

Code: Select all
Active Internet connections (including servers)
Proto Recv-Q Send-Q  Local Address          Foreign Address        (state)
tcp4       0      0  macmini.49531          ec2-54-174-104-1.9553  CLOSE_WAIT
tcp4      37      0  macmini.49530          ec2-35-168-98-11.https CLOSE_WAIT
tcp4       0      0  macmini.49529          ec2-54-174-104-1.9553  CLOSE_WAIT
tcp4      37      0  macmini.49528          ec2-35-168-98-11.https CLOSE_WAIT
tcp4       0      0  macmini.49519          ec2-54-174-104-1.9553  CLOSE_WAIT
tcp4      37      0  macmini.49518          ec2-35-168-98-11.https CLOSE_WAIT
tcp4       0      0  macmini.49517          ec2-54-174-104-1.9553  CLOSE_WAIT
tcp4      37      0  macmini.49516          ec2-35-168-98-11.https CLOSE_WAIT
tcp4       0      0  macmini.49514          ec2-54-174-104-1.9553  CLOSE_WAIT
tcp4       0      0  macmini.49509          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49508          ec2-52-87-80-95..https LAST_ACK
tcp4       0      0  macmini.49507          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49506          ec2-52-87-80-95..https LAST_ACK
tcp4       0      0  macmini.49501          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49500          ec2-52-87-80-95..https LAST_ACK
tcp4       0      0  macmini.49499          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49498          ec2-52-87-80-95..https LAST_ACK
tcp4       0      0  macmini.49496          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49495          ec2-52-87-80-95..https LAST_ACK
tcp4       0      0  macmini.49494          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49493          ec2-52-87-80-95..https LAST_ACK
tcp4       0      0  macmini.49490          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49489          ec2-52-87-80-95..https LAST_ACK
tcp4       0      0  macmini.49488          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49487          ec2-52-87-80-95..https LAST_ACK
tcp4       0      0  macmini.49479          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49478          ec2-35-168-98-11.https LAST_ACK
tcp4       0      0  macmini.49477          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49476          ec2-35-168-98-11.https LAST_ACK
tcp4       0      0  macmini.49474          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49473          ec2-52-70-30-52..https LAST_ACK
tcp4       0      0  macmini.49472          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49471          ec2-52-70-30-52..https LAST_ACK
tcp4       0      0  macmini.49469          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49468          ec2-52-70-30-52..https LAST_ACK
tcp4       0      0  macmini.49467          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49466          ec2-52-70-30-52..https LAST_ACK
tcp4       0      0  macmini.49464          ec2-54-174-104-1.9553  LAST_ACK
tcp4       0      0  macmini.49463          ec2-52-70-30-52..https LAST_ACK

Jason

Posted on
Sun Feb 18, 2018 8:52 am
Colorado4Wheeler offline
User avatar
Posts: 2754
Joined: Jul 20, 2009
Location: Colorado

Re: AWS connections

I'm assuming they are API calls, the plugin makes a metric ton of those things (so much so I had to change my plugins to not be overwhelmed by all the state changes). With all of the calls it makes I'm not surprised that there are remnants. This may also be why there is such a massive memory leak in the plugin (I restarted mine yesterday as it sat at 900MB!). Since Chameleon is MIA (he was dealing with health issues the last I heard, hope he is OK) I've considered forking this project to try to plug the leak but I'm neck deep in alligators right now with my new plugin. It's on the roadmap ;).

My Modest Contributions to Indigo:

HomeKit Bridge | Device Extensions | Security Manager | LCD Creator | Room-O-Matic | Smart Dimmer | Scene Toggle | Powermiser | Homebridge Buddy

Check Them Out Here

Posted on
Sun Feb 18, 2018 10:08 am
snowjay offline
Posts: 274
Joined: Aug 09, 2006

Re: AWS connections

Thanks for the update, I had no idea he was MIA as I haven't been active here in a while myself. (My Indigo/Insteon just works!)

I only noticed the issue after I started sending my dd-wrt logs to Splunk and was seeing all sorts of outgoing traffic to Amazon getting dropped. Further investigation showed the Indigo server constantly sending FIN ACKs which were getting dropped which then send me down the rabbit whole to finding out what was causing it.

I may experiment with changing the update time (right now at 60 secs) to see if I can cut down on the amount of orphaned connections, otherwise I'll probably disable it all together as the only thing I was using it for was home/away status.

If you ever tackle forking the project over keep me in the loop, I can provide logs for you.

Jason

Jason

Posted on
Sun Feb 18, 2018 12:23 pm
lanbrown offline
Posts: 772
Joined: Sep 26, 2017

Re: AWS connections

On my system, it has 16GB of RAM to it but uses nowhere near that much. Currently 4.75GB used and 2.65GB of caches files. The largest thing running on my Indigo system is Indigo server itself. The plugins are all below it; so nothing is over 60MB per Activity Monitor.

If you have large number of close_wait connections and you see in Splunk that your firewall is dropping traffic to AWS/Nest, I would start looking at your firewall. I have no close_wait sessions on my system at all and I'm running the Nest plugin with a 60 second value. If Nest (AWS) is sending you a FIN, is your system sending an ACK back and then sending it's own FIN? If the session is not be closed properly, then you will see a large number of close_wait sessions. I have none on my system and in my firewall I have very few open sessions. It has four to Apple, three to my internal network (Indigo sits in a DMZ) and then one for DomoPad.

Example (I hid my IP address) but this is the closure of a connection to AWS for Nest purposes as evidenced with TCP/9553 being used to AWS. The "F" is a FIN and the ack is an ACK. First packets shows the FIN from AWS, my system sends a few ACK's for other data in that stream as well as the FIN sent by AWS/Nest. My Indigo system then sends a FIN of its own and then it is ACK'd by AWS/Nest. The session is closed and my system would not be in a close_wait state.

90: 12:06:31.704217 802.1Q vlan#260 P0 54.204.224.218.9553 > 1.2.3.4.59599: F 4124060302:4124060302(0) ack 4043580820 win 235 <nop,nop,timestamp 215359941 2388593644>
91: 12:06:31.704309 802.1Q vlan#260 P0 1.2.3.4.59599 > 54.204.224.218.9553: . ack 4124056265 win 4010 <nop,nop,timestamp 2388593770 215359941>
92: 12:06:31.704370 802.1Q vlan#260 P0 1.2.3.4.59599 > 54.204.224.218.9553: . ack 4124057633 win 4096 <nop,nop,timestamp 2388593770 215359941>
93: 12:06:31.704385 802.1Q vlan#260 P0 1.2.3.4.59599 > 54.204.224.218.9553: . ack 4124060302 win 4012 <nop,nop,timestamp 2388593770 215359941>
94: 12:06:31.704416 802.1Q vlan#260 P0 1.2.3.4.59599 > 54.204.224.218.9553: . ack 4124060303 win 4096 <nop,nop,timestamp 2388593770 215359941>
95: 12:06:31.704950 802.1Q vlan#260 P0 1.2.3.4.59599 > 54.204.224.218.9553: F 4043580820:4043580820(0) ack 4124060303 win 4096 <nop,nop,timestamp 2388593770 215359941>
96: 12:06:31.751532 802.1Q vlan#260 P0 54.204.224.218.9553 > 1.2.3.4.59599: . ack 4043580821 win 235 <nop,nop,timestamp 215359953 2388593770>

Posted on
Sun Feb 18, 2018 4:39 pm
snowjay offline
Posts: 274
Joined: Aug 09, 2006

Re: AWS connections

These are my drops:

Code: Select all
Feb 18 17:09:59 192.168.0.1 Feb 18 17:11:18 kernel: DROP IN=br0 OUT=vlan2
SRC=1.2.3.4 DST=52.216.229.123 LEN=40 TOS=0x00 PREC=0x00 TTL=63 ID=51922 DF PROTO=TCP SPT=38568 DPT=80
SEQ=237400789 ACK=538878403 WINDOW=1403 RES=0x00 ACK FIN URGP=0


Update:
After turning off the plugin, I started seeing the same pattern of traffic from my Fire TVs to AWS. Interesting that it only seems to be AWS traffic that triggers it, but some research reveals it's a known issue with iptables, Seems that after the first FIN it tears down the connection and drops everything else thinking it's invalid. So now I need to figure out how to allow invalid traffic leaving my internal network so it's not clogging up my logs.

Jason

Posted on
Sun Feb 18, 2018 6:02 pm
lanbrown offline
Posts: 772
Joined: Sep 26, 2017

Re: AWS connections

snowjay wrote:
So now I need to figure out how to allow invalid traffic leaving my internal network so it's not clogging up my logs.


That is wrong and you should not do that. What you need to figure out is how to get whatever firewall you are using to work with TCP RFC's. Your firewall should not just tear a session down as soon as it sees the first FIN and if that is what it wants to do, then it should be sending a RST to the inside so it knows to kill the session. Ultimately, your firewall does not conform to TCP/IP standards that have long been established. Allowing it is send invalid traffic is then a security issue and should not be the answer to the problem. Sending a FIN ACK is valid traffic, not invalid traffic.

Posted on
Sun Feb 18, 2018 6:27 pm
snowjay offline
Posts: 274
Joined: Aug 09, 2006

Re: AWS connections

I realize I shouldn't do it, but I'm less concerned about letting FIN/ACKs out of my internal network right now, than having my logs be useless. Everywhere I research this, it seems to be a bug in iptables, and I doubt I'm going to change that overnight. ;) I've seen threads dating back to 2000 describing the issue.

Jason

Posted on
Sun Feb 18, 2018 6:37 pm
lanbrown offline
Posts: 772
Joined: Sep 26, 2017

Re: AWS connections

You need to realize the downside though, you are asking your firewall not to be stateful by allowing invalid traffic. That also means that connections from the outside that are not established would be allowed in, the only issue would be that there would be no NAT for that session. If for some reason there was an entry, then that traffic that would have been blocked would then be allowed. Your firewall is now worse than what the majority of ISP's provide you; you now would have a router that only performs NAT and does not care about state at all.

It is your network. I know I wouldn't use a security device that is now just a device because of bug that is nearly two decades old. Makes you wonder what other bugs there are?

Posted on
Sun Feb 18, 2018 6:51 pm
snowjay offline
Posts: 274
Joined: Aug 09, 2006

Re: AWS connections

If I make the change, my intent isn't to allow all invalid traffic, it would be to only allow FIN/ACKs sourcing from my internal network.

Iptables isn't some obscure firewall and maybe I'm just not finding the right solution to the issue.

Jason

Posted on
Sun Feb 18, 2018 7:28 pm
lanbrown offline
Posts: 772
Joined: Sep 26, 2017

Re: AWS connections

snowjay wrote:
Iptables isn't some obscure firewall and maybe I'm just not finding the right solution to the issue.


While it isn't obscure, it doesn't seem to be well maintained in that the symptom you see has been around for quite a long time. Either it is an iptables issue or how iptables was implemented on your firewall. Maybe a default rule that is set by firmware you are running? How old is the iptables version your firewall is running?

Just these week the *NIX admins ran into an iptables issue in that it decided to start blocking traffic from certain IP's when it should not have. One day everything worked, the next day certain IP's were blocked for no reason.

Page 1 of 1

Who is online

Users browsing this forum: No registered users and 0 guests