Working from China is a constant reminder that the location of internet services is super important. North America and Europe are just a few internet hops from our servers in Hetzner’s carbon neutral, green energy data center in Germany. From China, however, we’re often routed out through Beijing to San Jose, then to New York, London, Amsterdam, and finally Frankfurt. On a good day we can do that with only 2-3% packet loss.
Back in the good old days Gmail’s imap service worked perfectly even though Google was blocked, but eventually email was blocked too. At that point we added a managed mail server at Hetzner in Germany. Despite having our own non-blocked mail server, Hetzner’s data centers are unreachable during peak periods of the day because of peering agreements, congestion, geolocation, and a million other factors not worth arguing about.
The image above is a WinMTR report, a combination of ping and traceroute that helps locate network problems. 219.* is where we exit on China Unicom’s cable in Beijing and connect at San Jose, California. Nearly 50% packet loss, average ping time of 482ms. This is better than normal because San Jose connected directly to Frankfurt when this test was taken, instead of the more typical New York-> London-> Amsterdam route.
So how bad is it exactly? 5 tries to send a simple text email without attachments. “Failed to save draft, try again?” prompts every few seconds. Ability to see mail subject headers, but not download the message text. Want to send or receive an attachment? Better schedule that for late at night or first thing in the morning.
International bandwidth from China
There are three cable landings in China: Beijing, Shanghai, and Guangzhou. Each is dominated by one of the three state-owned ISPs. China Unicom (our provider) is big in the north with international traffic routing through Beijing and around 1TB/s bandwidth. China Telecom is big in the south (e.g. Shenzhen area) with around 2.5TB/s of bandwidth through Guangzhou. There are also a lot of bit players like Great Wall and TopWay that have city and regional backbones that eventually dump onto the major state-owned ISP infrastructure.
Part of our problem is using a northern ISP (Unicom) in the south. All of our traffic is routed to Beijing before leaving China. While we can see Hong Kong from the office and can walk there in about 15 minutes, connections to Hong Kong websites are routed up to Beijing and back for a 3000km+ journey. This isn’t our choice, Unicom has a monopoly in our Huaqiangbei office and nothing else is available.
This was intended to be an epic post using MTR to analyze the optimal routes and geographical locations to stash internet services with the best chance to be reachable from China. That idea bombed because nearly everything changed dramatically day to day. A test from yesterday is different than a test today, which will probably be different than a test tomorrow. Follow below for tests of several major email providers and their accessibility from China.
Rackspace Business Email is a disaster from China
Over the past few months we tested a lot of email services. Rackspace is a well respected company with business email hosting for $2 per box per month. Signup from China triggered a review, so we had to call support to complete the order. The Rackspace rep volunteered that they have constant complaints from users in China, not something you want to hear. The MTR report shows why, routing to their imap server is a disaster (30% loss, 361ms average ping).
Microsoft Office365 email works great from China, but is itself a disaster
Probably the most distressing part of daily life in China is using Microsoft’s Bing search engine. It’s a terrible search engine, but it’s always super fast within China. You might even find what you’re looking for if you skip directly to the third page of results, link number 5…
Microsoft’s Office365 imap mail service also works very well inside China, and at $4 per box per month it isn’t very expensive. The MTR report suggests Microsoft is running a server in Hong Kong that connects directly to China Unicom (219.*). The service is fast (21ms ping) and very accessible (very few lost packets).
(Un)fortunately Microsoft’s automatic email migration tool sprayed crap all over the place. Their suggestion was to hire an authorized partner for support. After a while all the subscription plans, conditions, and lack of support started to seem quite sleazy. We canceled when mail migration failed and they demanded 1 year commitments for each test account to help debug it.
ABCHK.net is our email hero
ABCHK.net is a hosting and email provider located in Hong Kong that specializes in email service for Mainland China. We were super skeptical that we could get a stable connection to Hong Kong, but they provided a test account that blew everyone away. 10 meg attachments? Uploaded and sent in seconds. It works perfectly at all times of the day, and the MTR report shows a direct, clean route to the imap server. No packets to destination lost, 31ms average ping.
For less than $7/month we get five mail boxes and 100GB of storage. Real people answered emails and handled the mail migration from our old server in Germany. After a month we are still extremely thrilled to be able to use email “normally” from inside China. Attachments upload and download super fast, and the server is always 100% reachable.
This applies to Unicom only!
Tests were done on commercial and residential China Unicom 100Mbps fiber connections. China Unicom is not the optimal ISP for Shenzhen though. It makes more sense to be on China Telecom with 2.5x more international bandwidth exiting just an hour north in Guangzhou. In the office Unicom has a monopoly, but we had a Telecom connection installed at home and will run the tests again on Telecom in a few days. Anecdotal evidence from other Telecom users doesn’t seem particularly promising though.
Admittedly this is all niche info, but it was hard won and seemed worth sharing here. At the very least someone in a similar situation might find this on Google. Or on Bing, page 3, result 5.