gevent urllib is slow

Go To StackoverFlow.com

6

I've created a set of demos of a TCP server however my gevent examples are noticely slower. I'm really not testing performance but at the moment something is making the gevent version 5 times slower.

I'm sure must be how I compiled gevent but can't work out the problem. I'm using OSX leopard using fink compiled python 2.6 and 2.7. I've tried both the stable gevent and gevent 1.0b1 and it acts the same. The echo takes 5 seconds to respond, where the other examples take <1sec. If I remove the urllib call then the problem goes away.

I put all the code in https://github.com/djay/geventechodemo

To run the examples I'm using zc.buildout so to build

$ python2.7 bootstrap.py
$ bin/buildout

To run the gevent example:

$ bin/py geventecho3.py &
[1] 80790
waiting for connection...
$ telnet localhost 8080
Trying 127.0.0.1...
...connected from: ('127.0.0.1', 56588)
Connected to localhost.
Escape character is '^]'.
hello
echo: avast

This will take 3-4 seconds to respond on my system.

However the threaded example

$ bin/py threadecho2.py

or the twisted example

$ bin/py twistedecho2.py

Is less than 1s. Any idea what I'm doing wrong?

2012-04-04 05:24
by djay
It seems to depend on my local network. At my home gevent urlib call is slow compared to normal urlib, but at the office they are both the same speed. Very strange - djay 2012-04-05 01:28


1

The answer was change the default DNS resolver being used as outlined in this conversation.

https://groups.google.com/forum/#!topic/gevent/5uNfkdgzWVc

setting GEVENT_RESOLVER=thread made it work as the expected speed

2012-08-19 23:35
by djay


0

Just tried that on Windows XP. Doesn't respond immediately, but is much quicker than 3 seconds. Will mock up a client to measure the exact timing.

PS Building libevent on Windows is not flawless! Had to play with includes and fix one bug in the actual code. I'll stick to Linux for libevent/Gevent in the future ;)

2012-04-04 12:02
by Bugagotti


0

urllib does not support http 1.1 connection reuse. Each time you get the page it creates a new TCP connection and a new TCP handshake occurs. urllib is and always will be slow with or without gevent.

2012-08-18 16:10
by gwik
if you look at the code provided you'll notice I was comparing urllib in both cases so this was not the cause - djay 2012-08-19 23:16
Ads