Aaron McNeal | 7 Jan 2011 21:45
Favicon

Dynamic VHost Crash

Hi everyone, 

I have been running some tests recently on a dynamic virtual hosting setup running through nginx + uwsgi. I
am currently experiencing some issues with app switching, and am seeing lots of crashes. 

The general work flow of the problem is like this. 

* Services are started.
* First request comes through for app1
* App 1 initializes and is served.
* Request for App 2 comes through, and app2 is usually initialized OK.
* More traffic comes in and things break, resulting in errors like this:
uwsgi: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
interpreter for app 0 initialized.
DAMN ! process 5256 died :( trying respawn ...

At this point, neither app is serving correctly, and a "Page is unavailable" message is returned to the
visitor until things respawn and reinitialize, but then the problem will happen again.

Here is a full debug log from the following scenario:

A uwsgi server is launched with the following parameters: --socket /tmp/uwsgi.sock --vhost -M -C -d
/var/log/uwsgi.log -L --no-site -p 6

An apache Benchmark is run with a concurrency of 12, and a total request count of 1000, into app1
The entire test runs, with no failures reported. Yay!
I then manually surf to app2 i my browser, and the errors begin. 
At that point, I need to refresh the page three times before I get app2 to appear.

Does anyone see something obvious that I might be doing wrong with the configuration of uwsgi server?

Thanks very much for any advice or thoughts on this.

Cheers,
Aaron

DEBUG LOG:
chmod() socket to 666 for lazy and brave users
initializing hooks...done.
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI master process (pid: 5253)
spawned uWSGI worker 1 (pid: 5255)
spawned uWSGI worker 2 (pid: 5256)
spawned uWSGI worker 3 (pid: 5257)
spawned uWSGI worker 4 (pid: 5258)
spawned uWSGI worker 5 (pid: 5259)
spawned uWSGI worker 6 (pid: 5260)
interpreter for app 0 initialized.
interpreter for app 0 initialized.
interpreter for app 0 initialized.
interpreter for app 0 initialized.
interpreter for app 0 initialized.
interpreter for app 0 initialized.
application 0 (app1.site.com|) ready
setting default application to 0
application 0 (app1) ready
setting default application to 0
application 0 (app1.site.com|) ready
setting default application to 0
application 0 (app1.site.com|) ready
setting default application to 0
application 0 (app1.site.com|) ready
setting default application to 0
application 0 (app1.site.com|) ready
setting default application to 0
writev(): Broken pipe [wsgi_headers.c line 168]
write(): Broken pipe [pyutils.c line 101]
interpreter for app 1 initialized.
application 1 (app2.site.com|) ready
uwsgi: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
DAMN ! process 5255 died :( trying respawn ...
Respawned uWSGI worker (new pid: 5268)
interpreter for app 1 initialized.
application 1 (app2.site.com|) ready
uwsgi: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
interpreter for app 1 initialized.
DAMN ! process 5257 died :( trying respawn ...
Respawned uWSGI worker (new pid: 5269)
application 1 (app2.site.com|) ready
interpreter for app 1 initialized.
application 1 (app2.site.com|) ready
uwsgi: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
interpreter for app 1 initialized.
DAMN ! process 5258 died :( trying respawn ...
Respawned uWSGI worker (new pid: 5270)
application 1 (app2.site.com|) ready
interpreter for app 1 initialized.
application 1 (app2.site.com|) ready
uwsgi: Modules/gcmodule.c:277: visit_decref: Assertion `gc->gc.gc_refs != 0' failed.
interpreter for app 0 initialized.
DAMN ! process 5256 died :( trying respawn ...
Respawned uWSGI worker (new pid: 5271)
application 0 (app2.site.com|) ready
setting default application to 0
interpreter for app 0 initialized.
application 0 (app2.site.com|) ready
setting default application to 0
interpreter for app 0 initialized.
interpreter for app 0 initialized.
application 0 (app2.site.com|) ready
setting default application to 0
application 0 (app2.site.com|) ready
setting default application to 0

Gmane