Frequently Asked Questions about Windows Live Sender Reputation Data

Return Path's Sender Score Certified program added Windows Live Sender Reputation Data as a new reputation data source and associated complaint metric on April 1, 2007. This complaint measurement is an addition to the existing Quantitative Requirements. Within the Sender Score Certified program, this quantitative standard is referred to as "Windows Live Sender Reputation Data".

The following provides an overview of how the new reputation data is used in the Sender Score Certified Program. Also included is a summary of the Windows Live program that generates the data used for this performance measurement.

While the raw data being evaluated is provided to Return Path by Microsoft Windows Live team, the analysis, modeling, acceptable thresholds, and enforcement of the data standard is solely determined and administered by Return Path. Questions regarding the data, its' use or enforcement should be directed to the Sender Score Certified Compliance Team at compliance@senderscorecertified.com.

 

The Windows Live Sender Reputation Data and Sender Score Certified

How is this data incorporated into the compliance model for Sender Score Certified?
Sender Score Certified uses data from Return Path's proprietary email reputation system, Reputation Monitor, in conjunction with data from MSN® Hotmail, SpamCop, SenderBase, and public blacklists to evaluate and ensure compliance with the program's Quantitative Requirements.

What Sender Score Certified compliance actions will be taken if the acceptable thresholds are exceeded?
Where a sender's performance for this metric does not meet acceptable thresholds as determined by Return Path, suspensions can occur. The acceptable thresholds are described in more detail below. If either an individual IP address or an IP Group has exceeded acceptable thresholds, the associated IP addresses will be suspended. Suspensions result in the non-compliant IP addresses being removed from the Sender Score Certified list, and occur within 24 hours.

In normal operation, the Sender Score Certified Compliance Team reviews the Windows Live Sender Reputation Data each day. Unlike the existing data sources, there are two levels of compliance for this data source. The first is at the IP level (similar to existing sources), and the second is at the IP Group level.

IP Address Compliance
Compliance actions for individual IP addresses must meet the following condition:

Where any individual IP address has generated a minimum of 5 messages, compliance action will be taken under the following circumstances:

IP Group Compliance
Compliance actions for an IP Group must meet the following condition:

Where the Group has generated a minimum of 30 messages for the Windows Live Sender Reputation Data source, compliance action will be taken under the following circumstances:

 

When a Group or IP address is suspended due to violation of this data source, what actions are needed before the Group or IP address is reinstated in the Sender Score Certified program?
As with violations of the existing data sources, once a Group or IP address becomes compliant with the thresholds for this data source, any suspensions will be removed automatically by the Compliance Team.

How is this data reported to senders in the Sender Score Certified program?
The Windows Live Sender Reputation Data will be reported via the Daily Performance Report along with the existing data sources used to monitor sender compliance with standards

For each certified IP address, the Daily Performance Report will show:

Compliance with the IP Group standard can be tracked by referring to the IP Group Total row at the bottom of the report as seen in the screenshot below.

Is information provided for suspended IP addresses?
Yes. When an IP address is suspended for any reason, including exceeding the thresholds for the Windows Live Sender Reputation Data, the Daily Performance Report will continue to report on this data source.

If a sender has been suspended because of non-compliance with the Windows Live Sender Reputation Data, does Return Path have recommendations about how to fix this problem?

Windows Live Sender Reputation Data is reflective of recipients voting that the mail they received is "junk" or "not junk". Understanding why your recipients consider your mail to be "junk" or complain about your mail can drive changes to your email program that reduce complaints overall. Return Path has the following recommendations for reducing complaints and complaint rates:

  1. Sign-Up for Feedback Loops

    Return Path also recommends that senders sign up for and analyze complaint data from ISP feedback loops available from AOL, USA.net, and other ISPs. Data sources such as Microsoft's Smart Network Data Service (SNDS) and Return Path's Sender Score Reputation Monitor can also be invaluable in diagnosing the root cause of user complaints.

  2. Reduce Volume of Mail Sent from Certified IP Addresses
    A short-term solution to reducing Windows Live Sender Reputation Data rates may be to reduce the volume of mail being sent from certified IP addresses, remove the problematic IP addresses from the program, or segment the types of email being sent through separate IP addresses

  3. Make It Easier to Unsubscribe than to Complain
    Always make it easier for recipients to unsubscribe than it is for them to "report spam" and block communication from you permanently. One benefit of being a Sender Score Certified sender is that email sent over certified IP addresses will have an unsubscribe prompt where you use a list unsubscribe header. You can find more information about implementing list unsubscribe at http://www.list-unsubscribe.com.

    Placing unsubscribe instructions in an area where users are most likely to see it, allowing users to unsubscribe by offering a "one-click" mechanism, and providing multiple methods to unsubscribe (like a link to a simple web form or replying with "unsubscribe" in the subject line) are ways to promote unsubscribing rather than clicking on "report spam".

  4. Objectively Analyze Your Email Program for Reasons to Understand Why Your Subscribers Complain
    The long-term solution is to address problems with your email program. To gain an understanding as to why recipients think mail is junk, examine the user experience, paying special attention to:

    • Consent and Disclosure statements at Point of Collection of user email address
    • Recipient expectation on content and delivery (do they expect to see 3rd-party offers? Are they clearly labelled as coming from you?)
    • Clear, conspicuous, and functional unsubscribe process including your mail being RFC compliant.
    • Content relevancy
    • Content frequency
    • Age of user records and email addresses on your mailing list
    • Permission Practices / Privacy Policy
    • User preference centers
    • Data sources and data partner vetting and auditing

  5. Segment Mail Streams
    Another approach is to segment mail streams and deliver each mail stream over a separate IP address. For example, send transactional messages over one IP, third party marketing messages over another IP, and new data from partners over another IP. Monitor the complaints generated one each IP to gain additional knowledge about which program is driving high complaint rates.

  6. Resources
    Additional information about resolving complaint problems is available on the Return Path Sender Score Certified site at http://www.senderscorecertified.com/about/suspension.php.

    Additional information about Best Practices for Senders is available at:

 

About Windows Live Sender Reputation Data

What is the Windows Live Sender Reputation Data?
Both Windows Live™ Hotmail and MSN® Hotmail services utilize Microsoft's patented SmartScreen anti-spam filtering technology. This technology uses a machine-learning approach to help protect users' inboxes from junk e-mail. SmartScreen technology learns from known spam and Phishing threats as well as from Windows Live Hotmail or MSN® Hotmail customers who have chosen to participate in the Feedback Loop Program (FBL).

The Windows Live Sender Reputation Data is a collection of non-biased responses from FBL participants over time. Along with other sources of reputation data such as the Junk e-Mail Reporting Program (JMRP), the Windows Live Sender Reputation Data helps to train and improve the way SmartScreen technology properly classifies messages based on email content and sender reputation.

How are participants selected for the program?
To ensure representative sampling across the e-mail user base:

What is the current distribution of participants in this program?

How do participants provide feedback about the e-mail they have received?
Random samplings of email are extracted prior to any anti-spam technology which may subsequently filter out a particular piece of mail. This allows for the ability to learn from e-mail that otherwise may not have been delivered to the recipient.

Program participants are always presented e-mail for classification that was originally addressed to them. They are simply asked to classify the message as "not junk" or "junk". This is an important part of the program – users are only asked to comment on e-mail that was addressed to them, and they are never asked to comment on e-mail that was not addressed to them. Users must choose to classify the message either as "junk mail" or "not junk mail", and then are prompted to confirm their selection.

Is this data representative of all e-mail sent via Microsoft e-mail systems?
The program captures data for all senders delivering into MSN® Hotmail and Windows Live Hotmail. Return Path receives Windows Live Sender Reputation Data on the Sender Score Certified membership for use as a performance measurement for the program. Sender Score Certified members will see their measured data only.

How is participant feedback solicited on an on-going basis?
Once a user signs up to classify randomly selected messages, they periodically receive email that has a subject of "Junk Email Classification". Users only classify e-mail originally addressed to them. E-mail is presented for classification within 12 to 24 hours of the messages original receipt for delivery to them.

Examples of the invitation and classification email messages:

 

Example 1: Sample Invitation to MSN Hotmail Classic Member

 

Example 2: Sample Classification Email for MSN Hotmail Classic Member

 

Example 3: Sample Classification Confirmation for MSN Hotmail Classic Member

 

Example 4: Sample Invitation to Windows Live Hotmail Member

 

Example 5: Sample Classification Email for Windows Live Hotmail Member

Does Microsoft determine the use of this data for Sender Score Certified?

While the raw data is provided by Microsoft Windows Live team to Return Path, the analysis of the data, determination of acceptable thresholds for compliance with the data, and enforcement of the data standard is strictly and solely determined by and administered by Return Path.

Questions regarding the use and enforcement of the data should be directed to the Sender Score Certified Compliance Team at compliance@senderscorecertified.com.

Isn't the presentation in the message requesting classification biased?
No. Members are sent e-mail classification request for email that was addressed only to them. Messages being classified may have been previously delivered to the inbox or intended to be delivered but either routed to the junk folder or deleted by the junk mail filter. Participants are then asked in a non-biased and objective manner how they would classify the message.

Isn't the program skewed toward participants who would want to only vote mail as Junk?
No. Participants are selected at random from over 200 countries without any bias or consideration for their previous voting record. Reports from one user alone cannot create junk status and requires additional feedback. Combining feedback alone with other sources of reputation data allows for junk or not-junk status to be applied.

Is the sampling really random? Is there ever any intentional selection for a particular IP address or sending domain?
E-mail is selected for classification at random without any consideration of any particular sender, IP Address, or sending domain.