Monday, December 29, 2014

The 5000mph car

If your car dealer offers, for the additional sum of $500, to upgrade your car to a top speed of 5,000mph, would you take it?

When it comes to broadband Internet access, people have been downing the Kool-Aid with exceptional enthusiasm.

In some cities with weak or non-existent consumer laws, ISPs are offering 100Mbps, 200Mbps, 500Mbps and 1Gbps service. Each bigger number option (note I do not say faster option) costs a few dollars more than the smaller one. In some places, the ISPs have stopped offering 100Mbps altogether and start at 200Mbps. The reason couldn't be more obvious. The cost of providing 200Mbps is the same as 100Mbps. The small print in the T&Cs' essentially allows the ISPs to not do anything other than provide the specified speed between the customer and their first node. So, a higher priced product means more profits.

Buying 200Mbps or 500Mbps is exactly the same as buying a 2,000mph or 5,000mph car respectively. People can easily relate to the uselessness of buying a 5,000mph car, but alas most cannot understand the same for a 500Mbps connection. Come on, most people do not have a home LAN capable of 500Mbps. Even most computers don't have an effective IO bus speed anywhere near that.

Taking a spin at is the same as taking your 5,000mph car to a specially engineered race track to check its maximum speed. The moment you leave the race track, your car is as good as one that has a top speed of 80mph. Similarly, departing and going to your typical web site, you will likely experience a typical 2Mbps.

Why do you even need 100Mbps? A web page will render on your computer fully in 1 second if you have 1Mbps. A full HD video can stream without a single jitter at far less than 10Mbps. But today we can't even get interruption free YouTube at a lowly 640x480, and who can afford more bandwidth than Google? The bottleneck may not be at the web server. It can be anywhere along a hundred points from your device to the server. And the cause need not be network speed or bandwidth. It can be something as mundane as a slow disk.

Recently, I managed to get fiber at 100Mbps. It was only possible through a patient battle. The telco won't sell any fiber service below 200Mbps. I only got my wish by refusing to let go of my 6Mbps ADSL line until they relented. I wanted fiber not for its x00Mbps, but for its symmetric bandwidth as I often connect to my servers and cameras at home from outside. As it turned out, the 100Mbps service was slightly cheaper than the ADSL.

Yeah, showed great numbers in excess of 90Mbps both ways. But when I uploaded a file to, which has 300Gbps connectivity, the throughput was 0.4Mbps - something that even my ADSL's miserly 600kbps upstream could handle just fine.

Meaningless numbersMeaningless numbers

January 2015 Update: As if to prove my claim that all these are meaningless numbers, two months after I signed up the 100Mbps service, the telco sent me this:

It took me a while to realize the false generosity. In two years when the minimum service period is up, my plan will continue as a 200Mbps one which costs more!

Friday, December 5, 2014

Serializing that convoluted cookielib.CookieJar

The Python cookielib.CookieJar object is a very convenient feature to manage cookies automatically as you traverse a series of Http web requests back and forth. However, the data structure of the class is a convoluted collection of Python dict.

cookielib.CookieJar has a _cookies property which is a dictionary of a dictionary of a dictionary of cookielib.Cookie.

To understand the data structure in the CookieJar object cj, try:

for domain in cj._cookies.keys():
   for path in cj._cookies[domain]:
     for name in cj._cookies[domain][path]:
       cookie = cj._cookies[domain][path][name]
       print domain, path,, '=', cookie.value

However, the class-defined __iter__ method makes the above effort unnecessary if you just want to find the value of a cookie. The __iter__ method returns a cookielib.Cookie object for each iteration. You can simply go:

for cookie in cj:
    print cookie.domain, cookie.path,, cookie.value # etc

If you want your CookieJar to persist in a file that can later be read back to create a Cookiejar object, the following two methods should work. They require the cPickle and base64 modules.

import cPicker, base64

def writeCookieJarFile(cj, cookieJarFile):
    f = open(cookieJarFile,'w')
    for domain in cj._cookies.keys():
        serialized = cPickle.dumps(cj._cookies[domain])

def readCookieJarFile(cookieJarFile):
    cj = cookielib.CookieJar()
        with open(cookieJarFile,'r') as f:
    except Exception as exception:
        print "readCookieJarFile: %s" % exception
    lines = text.split('\n')
    for line in lines:
        if line=='': continue
        cookieObject = cPickle.loads(base64.b64decode(line))
        firstCookie = cookieObject[cookieObject.keys()[0]].keys()[0]
        domain = cookieObject[cookieObject.keys()[0]][firstCookie].domain
        cj._cookies[domain] = cookieObject
    return cj

Note that cookieObject in the read method above is not a cookielib.Cookie object. It is a dictionary (keyed by domain) of a dictionary (keyed by path) of a dictionary of Cookie (keyed by name).

Tuesday, July 29, 2014

Eurail Pass - No more valuable

Thirty-five years ago, I traveled Europe on a rail pass as a student. Recently, we planned for a trip as a family of three. We almost bought the Eurail Pass - €576+€490x2 - but at the last minute we decided to drive.

The main reason we found the Eurail Pass not suitable was that despite the high cost, nothing can be a be planned with any degree of certainty. In France, the number of seats available for Eurail Pass holders are limited and nobody could say how limited. With such unknowns, it was difficult to book hotels. The other main reason was that lots of time would be spent waiting at train stations to catch connections.

As we flew into Amsterdam, we booked a Hertz car from Schipol Airport. The total rental bill for sixteen days worked out to only €483.96.

Hertz Rental Record (ignore pen marks)

Notice that there is a Location Service Charge of 18% or €61.01. I didn't see it till I got home. From my experience with Hertz in the US, this is usually because the car is rented out of an airport location. Try to rent from town and see if you can get rid of it.

For that €483.96, we had a huge car, a Kia SportAge. It was a stick shift, like most European rental cars, but it came with a built-in GPS Navigator (I didn't opt for it but the car came with it) which I found it priceless. Without it, I think we would easily spend much much more time undoing the mistakes from driving on the wrong lanes and taking the wrong exits. Like all Hertz cars I had rented, this one was also under 10,000km.

A car has some advantages in addition to costing only 1/3 to 1/4 of the Eurail Pass equivalent for a family of three or four:

  • Flexibility - if you meet delays, you can speed up by spending less time at your destination, or droping some destinations
  • Much cheaper hotels as you can stay away from big cities. In Paris, we stayed in Gennevillers next to a tram station for only €39 a night.
  • Mobility - you are not locked down to visit only places near train stations.

The time spent driving is not any longer than taking trains as there is no time spent waiting to change trains.

The cost of tolls and gas is not significant.

Driving needs lots of preparatory work, but I don't think any more than that needed to use a Eurail Pass effectively. You have to find out the driving requirements and practices in each European country. For example, in Germany you need a green Umweltzone sticker, which I got it without fuss at the first TUV station after crossing the border. In Austria, you need a reflective vest, which I found it in the safety package in the trunk. In Austria, you need to buy a vignette sticker to use motorways, which I bought it easily at a gas station in Germany near the border. In France, you need an unused breath analyzer. I didn't bother to check whether the safety package contained one, and I heard that it can be bought for €2 at Hertz.

The tough part with driving may be parking in cities. But most cities now have Park and Ride schemes. In Amsterdam, we could park the whole day for €1 with heavily discounted tram/train/bus tickets into town. In Berlin, we could park for free at a U- or S-Bahn station on the city fringes.

The big risk with driving is that you may get into accidents. As you are likely to be in a country other than the one in which you rented the car, this becomes an international incident and I don't know how it will be handled. I guess your holiday will be completely ruined if you are involved in an accident.

For the autobahn experience, see here.

Monday, July 28, 2014

Speed Unlimited

After five minutes on the famed autobahns, you get a full realization of the often quoted "you are more likely to die from an auto accident than an airplane accident" or "commercial flying is safer than driving".

First of all, the lanes are a bit undersized. For a saloon car, there is about a meter left on each side for most parts of the network.

Next, the traffic is heavy! This is probably due to the complete lack of tolls. If you are on the outer lane to pass another vehicle, you have less than 10 seconds to complete your job, because that is the time when another vehicle will come charging down behind you.

Tailgating is dangerous, but at the same time you hate to be traveling too "slowly". So passing often is a necessary evil. And it is a sheer act of terror. Most of the time, it is necessary to pass at about 130kph. At that speed, and with a high sided truck about a meter next to the right, the fluid mechanics sent my rental car slightly airborne each time.

Speed unlimited
Two diagonal lines across 130 means the speed limit is no more 130kph!

Driving on the autobahn is a super high workload activity. You have to stare with 100% concentration at the car in front, and at five times a second, you have to compare each 2D image of the back of the car in front with the previous image to compute and determine whether you are closing in. If the car is about 200m ahead, and you are driving at 120kph, you have about one second of reaction if the car in front suddenly slows down.

It is not infrequent that traffic comes to a sudden and complete standstill. I was driving eastward from Hanover on the E30 on July 16, 2014. Suddenly, about 500m in front, all the lanes were filled with stationary vehicles. I slowed down, switched on my hazard lights switched them off immediately after two blinks. The driver behind me imitated me, and that was when he made a near fatal mistake. He did not switch off his hazard lights. Apparently, he then signaled right, but because both of his turn lights were blinking, the car in the right lane was not aware he wanted to ease right. There was a loud bang, both cars nearly disintegrated and vehicle parts were flying everywhere.

If I was any closer the flying debris would make me part of the accident, and my holiday would be disastrously ruined. It would be an international accident, with my car rented out of Amsterdam. It would take months or years to determine culpability. Even if Hertz were to act immediately, it could take easily a week for them to send a replacement car from Holland.

The traffic jam in front of me, unrelated to the accident behind me, cost me almost two hours of delay. It was apparently caused by a major accident involving trucks.

I drove for a week in Germany, and every other day, there was a jam of about an hour duration on the autobahns.

When I entered France from Saarbruken, the situation changed suddenly. The French motorway was nearly deserted (the toll from the border to Paris cost €30), and it was a pleasure to be able to drive at your own speed, without having someone in front holding you up, and without the fear of me holding somebody else up.

The above is totally my personal experience. The statistics on the Wikipedia link suggests that the German autobahns are better than most other places.

Tuesday, April 29, 2014

Smart Phones are so passé

Smart phones and tablets are so last decade.

New features have already plateaued, and everyone who is anyone or no one, 5 years old or 100 years old, has, not one, but a couple.

Samsung and Apple are resisting the race to the bottom, and still winning. I don't miss a thing with a sub-$200 full-price Asus 5.5" full HD LTE phone.

It is no more about which phone is more capable, because they are all similar. It's about how you use them, just like this.