The error the sender receives is Remote host said: 557 The server encountered an error while processing this message. Please retry later.

 

That "557" is fatal - sending servers do not retry. And we end up looking like chumps.

 

Can anyone shed light on why an internal processing error would generate a permanent failure code instead of a temporary "4xx" error so emails could be re-queued by the sending system?

 

This issue is pretty random. I'll see it in a logfile along with hundreds of successful deliveries. Here's the SpamFilter excerpt of the event that triggers that 557 error...

 

03/11/10 05:16:54:265 -- (644) Connection from: x.x.x.x  -  Originating country : United States
03/11/10 05:16:55:171 -- (644) Received MAIL FROM: <sender@senderdomain.com>
03/11/10 05:16:55:249 -- (644) Received RCPT TO: recipient@example.com
03/11/10 05:16:55:265 -- (644) Resolving x.x.x.x - mail.senderdomain.com
03/11/10 05:17:02:515 -- (644) - SPF analysis for senderdomain.com done: - none
03/11/10 05:17:02:515 -- (644) Mail from: sender@senderdomain.com
03/11/10 05:17:23:515 -- (644) - MAPS search done...
03/11/10 05:17:23:515 -- (644) RCPT TO: recipient@example.com accepted
03/11/10 05:17:23:874 -- (644) Checking SFDC
03/11/10 05:17:24:218 -- (644) SFDC - Added x.x.x.x - Response: Error=0
03/11/10 05:17:24:234 -- (644) Exception occurred during RECEIVEMESSAGE: ( 1 2 2b 3 3a 4 5 6 7 8 9 11 12 13 15 16 17 18 18b 19 20b 20c 29 30 31 32 111) Access violation at address 00413B4E in module 'SpamFilterSvc.exe'. Read of address 00000005 --
03/11/10 05:17:24:312 -- (644) Disconnect

 

If we restart the service, things are smooth again for a few days. But this random rejection business is difficult to explain to customers. Any help greatly appreciated.

When I do manual DNS lookups, generally speaking they do not time out.

I wonder.... could it be anything to do with the black lists I'm using? For example, if they're no longer in service, may this effect things or report timeouts?

I'm seeing this a lot in my logs:

The behavior you describe is very odd. An incoming email is not complete until the <CR>.<CR> end-of-transmission sequence is received. SpamFilter cannot process an incoming email unless that sequence is received. If the remote server is dropping the connection (or timing out) before sending that sequence, it *should* be impossible to process the email (the should is because in IT you've never ever seen it all....)

What could happen in theory is that the remote server does send the <CR>.<CR> sequence (which will cause SpamFilter to accept the email), but then for some reason does not receive the "250 OK" acceptance SMTP code from SpamFilter, which tells the remote server that the email was received correctly. Without seeing this 250 return code, it would be likely that the remote server will retry. It's however rather strange that there could be a timeout that prevents SpamFilter from successfully sending the 250 return code while at the same time allowing the incoming email to make it thru.

If you can zip us to support @ logsat.com any logs you have (even the remote server's logs - we'll try to use those as well), we'll try to see what is happening. If you have the ability to also configure a full 2-way network capture (with Wireshark or similar) of the SMTP traffic to/from the remote server in question, this would greatly help in identifying the problem.

On a number of occasions we have been informed by a few different clients that emails from particular (but varied) senders are being received multiple times - like 100's!

Looking at the logs our end, we don't see a problem - to us we just see the sending server make a connection, send an email which we forward onto the destination server. I've now been able to get hold of some sort of log file from one of the senders which are experiencing this 'duplication' of emails. From what I can gather, there system connects and sends SF the email, SF does whatever checks it does, but before SF accepts/confirms delivery, the senders end hits a time out, drops the connection and re-queues the email for another delivery attempt later. Despite the senders end timing out before our server has confirmed receipt or acceptance, our server still forwards the email - presumably because our server knows its received the email in its entirety having receiving the <CR>.<CR> instruction? So assuming the email passed spam checks, the email is sent on. 

The above course of events cause a loop, whereby our server receives and delivers the email, the senders continually times out before our server has confirmed acceptance, causing the sending server to re-queue, and re-deliver the email - and then the loop begins again.

If this is what's happening, I think either one of the following is wrong:

1) The senders end's time out is too short?

2) SF should not deliver an email if the connection is dropped/times out - just like it wouldn't if the connection was dropped part way through the DATA stage.

If number 2 is changed, so that SF does not deliver emails from which the connection is dropped even after a <CR>.<CR> is received, this will not prevent the senders end from timing out, so instead of receiving duplicates, our server would not forward any, meaning the email would never be received, and eventually a bounce back would be generated. What are your thoughts on this? And what do people thing the fix might be? 

The version of SpamFilter you're running is very old :-) That was a known issue and was solved a long time ago starting with SpamFilter v3.5.3.657. The partial release notes for that build are as follows: