Fixed in the next release. Added a date parameter to uncached attachment URLs, which corresponds to when it was edited. If the date parameter is present, status 304 not modified can be sent immediately.
If the date parameter is not set (as appropriate for URLs stored in the post cache, etc), the attachment request will be processed at least far enough to fetch the attachment data from the database. Status 304 not modified will then be sent if the HTTP_NOT_MODIFIED_SINCE header is newer than the last update time stored in the attachment data. Otherwise, the attachment will be sent as if it's a cache-less request.