A proposal to easily set up user accounts in PIMs

Exec Summary

• Entering account settings is one of the most cumbersome and difficult aspect of using an Email client. This is a huge barrier against adopting a new PIM, such as Chandler.
• Ideally, users should be able to enter just their email address as a means of specifying their email account settings.
• We can, largely, accomplish this by creating a publicly accessible Accounts Settings Database (AccountsDB), containing email settings of potentially all email providers. Setting up an email account then simply entails looking up an email provider's settings and substituting these generic settings with the user's email userid.
• Just as with FreeDB, the public CD info database, we populate this database through user contributions, building in an auto-user-submission process into Chandler as a form of commons based peer production.

The Problem and Background

• Our interviews with LPFI brought up an obvious (in hindsight) but nonetheless illustrative observation: Filling out account settings is one of the most difficult and obtuse areas of a PIM application for most users. LPFI users were all pretty experienced Outlook users, but all of them (without exception) relies on Chris (our sysadmin) to decipher their account settings. No one set up their accounts themselves
• It was also a key reason why they relied on Outlook Web Access for checking email from home. It's not the only reason, but getting Chris to set up their home machine (often a desktop) was unfeasible and the envisioned pain of setting up their own accounts, made them feel that using OWA was an easier alternative, even though many of them complained about OWA's clunky web interface.
• In most web email implementations, these account settings are not needed. This may be another reason for why web email was so quickly and easily adopted.
• Account settings are obtuse for many reasons:
  • Few users understand what POP and IMAP mean
  • Users don't understand why an email account requires both an account for receiving email and a different account for sending email (SMTP)
  • Server names don't necessarily correspond to email addresses. Users have a hard time figuring out what server names are for
  • Port numbers are an even bigger mystery. There is no user mental model for port numbers
  • The purpose of SSL and its variants are seldom explained to the user. It also needs to be communicated that SSL requires server support. Even if the user knows what SSL is, she may not know if her mail server supports SSL. At OSAF we use self-signed certificates which is an even harder abstraction to grok.
  • Multiple accounts and the combinatorial explosion of SMTP, IMAP and POP server combinations, further create unnecessary complexity for the user

Why this is a difficult problem and current solutions

• Mimi observed that the user knows two things about her email account well:
  • email address (they need to give this out anyway)
  • password
  • [OK, maybe also, "display name" or "Full name"]
• Thus, the cleanest, most ideal interface is simply to ask for the user's email address and password. if only the application can fill out everything else based on this information.
• This is in practice difficult because there is no easily computable 1-to-1 mapping between an email address to all the other account settings. And these settings, in practice, vary quite a bit. There are no simple heuristics to figure out common or default setting values.
• Most PIMs today either do not deal with this problem or use step-by-step guided Wizards to help the user.
• In the Wizards, some default simplifications are applied. For example, many PIM wizards help users fill out the minimum information required, ignoring "advanced" settings like SSL. But when SSL is required, as is increasingly so, this drives system administrators crazy because the wizard interface no longer works for their end users.

A proposal

• Today's solutions focus on how single individuals can fill out these settings in isolation. But, we observe that most of these settings are almost identical for all users within the same email provider (e.g. ISP, Yahoo! mail, office email server). Within the same provider, in most cases, the variability in settings can all be deduced from the user's email address.
• Thus, if one person fills out the settings for her email account, those settings can be applied to all other users (within the same provider), with a few simple substitutions based on the particular user's email address.
• So, if we had a publicly accessible database of default email settings, indexed by email provider, we could then fill out a user's email account settings simply by asking for the user's email address. Here're a few example entries of such a database:
  • Index: @osafoundation.org
    • IMAP: mail.osafoundation.org: 993, TLS 1.0, username=emailID
    • SMTP: mail.osafoundation.org:587, TLS 1.0, username=emailID
  • Index: @yahoo.com
    • POP3: pop.mail.yahoo.com:110, username=emailID
    • SMTP: smtp.mail.yahoo.com:25, username=emailID
  • Index: @earthlink.net
    • POP3: pop.earthlink.net:110, username=emailaddress
    • SMTP: smtpauth.earthlink.net:25, username=emailaddress
  • AccountsXml
• How do we populate this AccountsDB?
  • We can seed the database with the most popular settings from the largest email providers (e.g. earthlink.net, yahoo, comcast, etc.)
  • FreeDB^? created a massive database of CD information, including detailed info of each song or track on the CD, all through individual user contributions.
  • Analogous to FreeDB^?, we can accept user contributions of email settings for all email providers
  • We can create an automatic opt-in user submission of email settings into Chandler, so that the first few users that encounters a email provider that is not in the database are prompted to submit this information back to the email AccountsDB. Subsequent users then benefit from this submitted info.
  • AccountsDB would never store the username and password. It would never need the user's password during the submission.
  • We can open up the API for auto-submission, so that other email clients (such as Thunderbird or even Outlook) can also utilize this publicly available database for easy account set up and also for auto-submission to AccountsDB
• Once we retrieve the relevant user settings from the AccountsDB, we can apply these settings to the user's email address and password through Morgen's routines to test if they work. If not, user can be asked to manually to fill in the right settings.
• Obviously, the actual workflow and UI for downloading and submitting these need to be further thought out, but we can learn a lot from how MP3 players (e.g. iTunes) interact with CDDBs.

Security issues

• GrantBaillie points out that the system needs to guard against fake servers and password phishing. A user could submit a malicious email setting that directs users to an incorrect fake (IMAP/POP/SMTP) server specially set up to harvest a user's username and password combination. The malicious user can then apply this password combination to access the user's reall email account.
• We should require user submission of email settings to be confirmed only through email verification of the same email address whose settings are being submitted (e.g. if I'm submitting settings for chao@yahoo.com, I need to prove that I have access to mail received from that account). Each submission is also logged. This should allow a malicious submission to be traced to a specific email address within the organization.
• We could also restrict server domain name settings so that they correspond to the email address' top and second level domain (e.g. server name settings for mimi@yahoo.com has to end with "yahoo.com"). GrantBaillie points that we may need to restrict this lower domain name levels based on specific Top Level Domains, e.g. sf.ca.us would need to be restricted to third level domains
• In Chandler, this automatic accounts set up should be made optional. For example, some universities may want to disable this option for some campus deployment.

Other thoughts

• We should be able to extend this DB from email settings to other account settings (e.g. CalDAV or WebDAV settings). The key is to have a common user handle and index.
• The publicly available AccountsDB can be placed under a suitable CreativeCommons^? license to encourage widespread adoption and to prevent any party from taking proprietary or unfair advantage of this system.
• This proposal is clearly not exhaustive and just a start of a discussion. There may be instances where an email address maps to multiple account settings for different users. This may be common in large enterprises. More advanced UI for both settings retrieval and submission will be needed to deal with these cases. Fortunately, enterprises are not the target users for Kibble

Links and external resources

Server settings of some major ISPs:

• SBCGlobal: http://help.sbcglobal.net/article.php?item=287
• Comcast: http://www.comcast.com/Support/Corp1/FAQ/Faq2_196_0.html
• Earthlink: http://support.earthlink.net/mu/1/psc/img/walkthroughs/other/1694.psc.html
• Yahoo: http://help.yahoo.com/help/us/sbc/mail/pop/pop-11.html
• Hurricane Electric: http://www.he.net/faq/email.html

Should also look into:

• ACAP

Comments

BrianKirsch 20050302

Chao, Overall I think this a great idea. The concept of an AccountsDB if done right could end up as a standard among email clients.

Thoughts: 1. Many service providers alias the Server name for IMAP and POP accounts. This will make automatic lookup difficult. For example an Earthlink account with username demo1 might have a server name of demo1.imap.earthlink.com. We would need more advanced heuristics to properly deduce server settings in these cases.
ChaoLam: Yes, this will be a problem under the current proposal. If this is a common problem, the backend can try to be smart and match the username as a variable and confirm with the submitter that the server setting indeed varies with the username. A similar matching problem occurs in the backend to figure out if the username for the account password is just the username of the email address (i.e. everything before the @ sign) or the entire email address.

2. When accounts are configured Chandler should not only test if the account settings are correct (like webDAV is currently doing) but test if the server has SSL enabled. This way we can move the burden of SSL configuration from the user to the application. For example the IMAP SSL port is 993. When testing the account settings Chandler first tries 993 if that fails, Chandler tries the default IMAP port of 143. IMAP allows a STARTTLS command. So on port 143 we see if STARTTLS is support. If it is not then the IMAP account in not secure.

The UI should distinguish between a secure and non-secure connection similar to how a web browser does for http vs. https. However, we should do a better job of making the distinction between a secure and non-secure connection to the user. Web browsers display a lock icon on the status bar when a connection is secure this is not sufficient in my opinion.

3. There should still be an easy to use Account Setup Wizard for cases where an account can not be determined from info stored in the AccountsDB. Also what happens if the database returns the wrong account info? If 90% of the account is correct but the server name is wrong for example it is going to be very hard for the average user to identify the issue and correct it.

ChaoLam: You're right. We need data to figure out how accurate this approach is empirically. A few ways to mitigate this issue are:

1. User only submits info if settings work for the user herself.
2. We could build it into the API to report failures from submission. So, Chandler could report back that the submission look up returned the wrong account info. After N such reports, the account submission for a particular domain can be disabled.
3. Only show the submitted info from AccountsDB after the client checks that the submitted info works (using Morgen's routines)

4. Responding to Grant's concern about password sniffing. Is it really necessary to store specific user account information in the AccountsDB? Just storing the email address domain and general account info and not the personal account info (username, email address, password) seems sufficient. I would be very concerned the security if we were to store passwords.
ChaoLam: No username or password should be stored in AccountsDB (although the username is sent to AccountsDB for matching as described above) The password sniffing that Grant raised happens in the much more rare scenario that someone maliciously submits a fake servername that points to that person's own server to harvest the username and password of future unsuspecting users who rely on this fake submission. I think requiring submitters to verify their email address should avoid this kind of attack.

MorgenSagen - 01 Mar 2005:

Agreed, this is a neat idea. You definitely wouldn't want to (or need to) send passwords to the AccountsDB; probably just send everything after the @ sign, right?
ChaoLam: Definitely, no storing of username and passwords. It's useful to transmit the email address and username to figure out server username and to address issues Brian raised above. But this could all be done at the client end too.

There are cases where this won't work, though. For example, my brother has an account at Hurricane Electric, and his account is something@he.net. However, his particular login has been assigned to a specific machine such as foo.he.net as the IMAP server (whereas other users might be assigned to bar.he.net), and the AccountsDB couldn't know about this mapping. In this case, a client trying to use AccountsDB could be given the wrong server. I doubt this happens frequently, but it's possible.
ChaoLam: That's a good point. We could build it into the API to report failures from submission. So, Chandler could report back that the submission look up returned the wrong account info. After N such reports, the account submission for a particular domain can be disabled. Alternatively, we could return a set of possible submissions - one of which may work for any particular user. Chandler can automatically try all the submissions, until one works. This may be too confusing tho'.

I see Brian has pointed out another failure case above (#1). I wonder how sensitive sys admins would be to having a central clearinghouse for this type of info. I guess it's not really a security risk -- I mean how hard is it to guess that OSAF's imap server is 'mail.osafoundation.org'? As long as no list of usernames or passwords are in the AccountsDB, I guess it's not a problem.

AccountsDB would basically be a separate project, I think, with a server-type person doing the implementation, deployment, management tools (plus some sysadmin work to keep the service up and running), and Chandler could be the first client of it.
ChaoLam: I'm asking Ted to see if this could be a community project, since it has few dependencies on core Chandler. It's perfect example of commons-based peer-production that Ted and Mitch has been talking about.

MimiYin: Questions for engineering

• Do users need to be able adjust the port setting? Can we just have a checkbox called: Turn on extra security or something to that effect?
• How likely is it that someone enters an smtp server without an incoming mail server? Can we conflate the two into a single account with a single set of username/passwords?
• Do people ever have different username / passwords for the the same incoming/outgoing mail server pair?
• Do people ever have usernames that are different from the part of their email address before the @ sign?
• Can we get feedback for the "Test this account" feature in the WebDAV account setup screen?

HeikkiToivonen:

• Proposed AccountsXml as an XML format.
• I think there should be multiple ways to get the acounts information:
  • Download an accounts db over the web from some URL, and/or have a local copy of it
    • Works great for ISPs, companies etc. that can provide a small file good for most of their users.
    • Also reasonable default for some high volume email addresses like hotmail etc.
  • Query some accounts db(s) over the web
    • Need some beefy servers
    • Info easier to keep up to date
  • Hybrid approach should also be possible, and caching query results and query servers returning "no changes" would reduce traffic

• The AccountsDB should be public, but controlled by trusted moderators. Everyone should be able to suggest changes.
• The client should not send email or let the server know the user's email address for privacy reasons
• The client could let the AccountsDB know which of the servers it tried, which worked and which did not, to improve quality of db.
• The client should not blindly trust what AccountsDB returns, but use certain rules:
  • mail server can be queried from DNS (see also LisaDusseault20050309)
  • mail server can be found in the same domain as what is after @-sign
  • Maybe present the user with prefilled dialog and ask the user to confirm - really only helps advanced users

• See alternative ways to make this work: LisaDusseault20050309

0.6 account setup proposal

• Here is a lightweight proposal for .6 account setup which does not take the Accounts DB proposed above into consideration.

• A couple of key assumptions
• Every outgoing mail account has an incoming mail account with the same username password
• Usernames are always the same as the part of the email address before the @ sign
• Users don't need to customize the port settings

• Workflow
• User creates a new account
• User then sets account type: IMAP, POP, Sharing
• Each account deals with settings for both incoming and outgoing mail
• ZeroPointSixAccounts^?.gif:
  

• ZeroPointSixAccountsSharing^?.gif: