Weird email behavior

Okay sports fans...
I use a dedicated gmail account for all my home automation stuff. Gmail account is used to send email to my iCloud account on all my devices/computers. Has been working like a champ for over a year...

...until this week. Now I'm not getting emails from Indigo at all. I logged in to the gmail account and sent a test message from the browser...worked just fine. Looked at the sent items box in gmail...shows all the emails Indigo tried to send. Checked Indigo logs...Indigo has no errors.

In summary, direct emails from the gmail account transit to iCloud, but emails sent by Indigo from the gmail account appear to send from gmail but never arrive at iCloud.

Any ideas?
Phil

I'm zeroing in on the problem...I think.

I run Indigo on a Mac Mini Server that provides a bunch of other common services for the house. I've altered the server name in the below example so as not to invite adventuresome computer enthusiasts(!)...let's call it FRED. So it appears that emails showing gmail.com in the return path, but not gmail.com in the message ID are being filtered by iCloud, probably for security reasons. My unRAID server is able to send emails no problem, but it has a check box in the setup to change the message ID to reflect the email server. I think because Indigo does not change the message ID, the emails are being filtered.

Below is the "original text" from the email as shown on the gmail web account (with ip addresses and email usernames xxx'd over):

Return-Path: <XXXXXXXXXXXXX@gmail.com>
Received: from localhost.localdomain (c-XX-XX-XXX-XX.hsd1.md.comcast.net. [XX.XX.XXX.XX])
by mx.google.com with ESMTPS id ey7sm6732169ved.0.2012.11.20.05.07.24
(version=TLSv1/SSLv3 cipher=OTHER);
Tue, 20 Nov 2012 05:07:24 -0800 (PST)
Date: Tue, 20 Nov 2012 08:07:24 -0500
Subject: The Basement Door is OPEN!!!!
Message-ID: <99a74d120329406c5b03638f90726564@FRED.com>

So, I need to either ask Matt and Jay to please, purty pleeeeze, introduce some kind of emulation thingy - or if anyone out there knows how to skin this cat with settings on the Mac Mini, please hold forth. I think this is a recent introduction on the part of Apple's iCloud service - because it worked no problem for the longest time.

Thanks,
Phil

[jay (support)]

I gotta think this is a bug in iCloud mail - not only because it's ridiculous, but mainly because iCloud doesn't give you any notification that an email has been rejected. No mail system should ever reject an incoming mail message with absolutely no indication to the user - put it in the junk folder, or send a proxy mail message saying an incoming message from blah has been rejected.

I don't believe clients can control the Message-ID field - according to the SMTP rfc the creation of that field is server dependent. Using it in this manner is a really poor decision by Apple which makes me think it's just a bug in the iCloud mail servers.

[jay (support)]

Well, after a little more testing, Message-ID is not the problem. I created a hotmail account, switched Indigo to use those settings, and sent a message to my iCloud account which had a message id of: <BLU0-SMTP136B23BDBE618137998BE21CA550@phx.gbl> and it was delivered correctly.

Clearly that ID doesn't match hotmail.com, so something else is hanging the message up. It's unclear whether this is an iCloud or Gmail problem, but given the relationship between Apple and Google...

Guess I'll switch my SMTP settings to the hotmail account until this problem gets solved.

[EDIT] Apparently on my production Indigo server I already had emails going out from smtp.me.com and was only scanning incoming mails from gmail - maybe I ran across this already. As I said in a previous post, failing memory is a sure sign of age.

Jay,
I initially tried (ages ago) to set up outgoing on me.com and it wouldn't work - that's why I set up a gmail account. Sounds like Apple shifts security settings both directions without notice. Agree with you, this must be an Apple problem. Weird that only Indigo is having the issue for me. I'll work at it a bit more to see if I can zero in on the cause.
Thanks,
Phil

For comparison, here's the message header for the emails auto-generated by my unRAID server at loss/gain of comms with the UPS (again with PID XXXX'd out):

Return-Path: <XXXXXXXXXXXX@gmail.com>
Received: from XXXXXXXXXXXX@gmail.com (c-XX-XX-XXX-XX.hsd1.md.comcast.net. [XX.XX.XX.XXX])
by mx.google.com with ESMTPS id a10sm6712055vez.10.2012.11.20.05.02.43
(version=TLSv1/SSLv3 cipher=OTHER);
Tue, 20 Nov 2012 05:02:45 -0800 (PST)
Message-ID: <50ab7f75.aa543a0a.1049.0f7d@mx.google.com>
Received: by XXXXXXXXXXXX@gmail.com (sSMTP sendmail emulation); Tue, 20 Nov 2012 08:02:43 -0500
From: Console and webGui login account <XXXXXXXXXXXX@gmail.com>
Date: Tue, 20 Nov 2012 08:02:43 -0500
To: root
Subject: Library Communications with UPS restored
User-Agent: Heirloom mailx 12.3 7/15/07
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

AND HERE'S ONE THAT DIDN'T GO (ORIGINATED FROM INDIGO)

Return-Path: <XXXXXXXXXXXX@gmail.com>
Received: from localhost.localdomain (c-XX-XX-XXX-XX.hsd1.md.comcast.net. [XX.XX.XX.XXX])
by mx.google.com with ESMTPS id dh10sm6754053veb.8.2012.11.20.05.37.41
(version=TLSv1/SSLv3 cipher=OTHER);
Tue, 20 Nov 2012 05:37:41 -0800 (PST)
Date: Tue, 20 Nov 2012 08:37:41 -0500
Subject: Motion sensor dusk
Message-ID: <b1e64b072281e11f9df1b5a13975d495@FRED.com>
MIME-Version: 1.0 (Generated by Pantomime 1.2.0)
From: XXXXXXXXXXXX@gmail.com
To: XXXXXXXXXXXX@me.com
X-Mailer: Indigo Server 5.1.6
Content-Type: text/plain; charset="utf-8"

[jay (support)]

It's definitely not the Message-ID unless Apple is specifically checking only emails from Google given my results with Hotmail.

[jay (support)]

OK - so, the library we're using is generating the message id. However, Hotmail is changing it. Which, if you follow the RFC exactly, is probably the correct behavior, since the message id is supposed to be unique to the instance on any given server. Gmail isn't changing it (probably unless it happens to hit a conflict) but rather just passing through the one from the source.

In any event, Apple is totally screwing up by assigning any kind of meaning to the message id. Other than guaranteeing uniqueness, it should have no relevance to anything else. Idiots.

We'll add a feature request to try to work around the issue in some future version. For now, I suggest switching to iCloud to send mail (I checked and I've been doing it that way for well over a year).

Jay,
Thanks for chasing it. I'll shift over to me.com or hotmail.com - it's not a showstopper, but it is quite annoying. When I first started learning about computers in elementary school in the early 80s, "GIGO" was sacrosanct. It was always correct to assume the computer was just doing what you told it to do - you were just telling it wrong. With the advancements in computing and the way operating systems are structured these days it is becoming less certain that a computer or process will perform the same way every time. Frustrating - means you spend all your time chasing fireflies.
Thanks,
Phil

[jay (support)]

More info: it doesn't appear to be the message id at all. We've tested various message id constructs (including one that's pretty much identical to what you get when sending via the gmail ui and one that's virtually identical to the one you post above) and nothing is getting through. So, something else is causing it. Doubt we'll have much more time to dive into it though - it's a problem Google/Apple will have to work out.

Yeah, Apple and Google "working things out"....call me when the little devils in hell are strapping on their ice skates and sipping iced tea...

All, i have had a ticket open with Apple for a while regarding this. Mainly because my emails were going into Junk. But junk on the cloud server, not junk on in my mail app. you can them if you log into icloud.com. It appears that they never make it to the mail app.

they have been unhelpful in resolving. Even if i said "Not Junk" to the emils it would still almost always go into the junk folder. Their explanation was that there was still a high enough spam score to ignore my "not junk" declaration. This is backwards if you ask me but that is not why I am writing.

So i took the same message that went into the junk folder and signed into my dedicated indigo gmail account (which is what indigo is using) and sent the same exact same subject and body. went through no problem. So i think that it is in the header someplace.

I have not gotten around to try scripting a few emails through the mail app on my indigo server to see if that helps or not.

So for now I am sending to a different email address.

I am wondering if they are zeroing in on the sender app or something like that...

Jay I can send you both emails if you like. Just let me know.

[jay (support)]

We think we know what the problem is and are continuing to investigate. I also found that you can (at least sometimes) get the emails by logging into iCloud. And, let me tell you, this is a MUCH bigger problem for Apple. I have quite a bit of non-junk that's getting caught. Interestingly, it doesn't appear to be all of it though.

Anyway, we'll continue to attempt to figure out how to work around the problem.

Jay,
Damnedest thing. I just upgraded to Beta 6 this morning, and voila! - emails from gmail again. I can't say for sure there's a direct cause and effect. Something must have changed in Indigo 6, or else a "fresh" install of Indigo made things start working again.
We'll see how long it lasts.
Best,
Phil

[matt (support)]

Yes, Jay hacked around the problem by having our email client spoof the HELO command with a bogus LAN IP address. iCloud is definitely not following RFC 1123. Note we have found iCloud flagging other valid emails (not from Indigo) as junk, too. Hopefully, Apple will fix the problems on their end eventually.