Hello, I am the developer of this patch, there are a few things to note:
this patch requires a few extensions. i didn't apply more rigorous extension checking before but i'll add that tonight, for now make sure you defiantly have the following (windows users can use a program called glcaps or something like that, i forget. Linux has glxinfo of course).
GL_EXT_packed_depth_stencil (or GL_NV_packed_depth_stencil)
GL_ARB_depth_texture
GL_EXT_framebuffer_object
GL_ARB_vertex_program
GL_ARB_vertex_shader
GL_ARB_fragment_program
GL_ARB_fragment_shader
GL_ARB_shading_language_100
GL_ARB_texture_non_power_of_two
people having problems when stencil buffers are enabled i would guess do not have packed depth stencil support, this chart suggests that some ati cards for whatever reason do not have this at all:
http://www.labri.fr/perso/granier/OpenGL/ext/videocards.html if someone with an ati card could clarify this for me that would be great, if it turns out that ATI really don't have any form of depth-stencil framebuffer support i will just auto-disable stencil so that it at least works.
for people who seem to be having problems with the lighting looking flat shaded, i would first make sure that its not a problem that already exists that is just 'highlighted' by the colour modification the rotoscope shader does, just turn off the bloom/rotoscope effects and take a very close look at the model in question (also if you could post a screenshot with that setup that would be great, so i can take a close look for myself)
for the person apparently having problems with sdl make sure that you are using the latest svn version, with that version there is only sdl support iirc.
i would also like to clarify some of the cvar names (i really need to make these more obvious)
"r_framebuffer" - toggles the framebuffer rendering path
"r_framebuffer_bloom" - toggles the bloom code
"r_framebuffer_blur_size" - the size (in pixels) of the blurred image, higher values will be a higher quality with less popping but obviously, slower (you may need to increase the blur amount too)
"r_framebuffer_blur_ammount" - the size in percentage of the blur, the 'best' value for this is 100/blur_samples.
"r_framebuffer_blur_samples" - only three values will work for this 5, 7 and 9. higher values = higher quality.
"r_framebuffer_bloom_sharpness" - the sharpness between the bright colours and the dark ones when blooming
"r_framebuffer_bloom_brightness" - pretty obvious, 0.85 is a sensible value. for testing i use 10.0 (to make it really show)
"r_framebuffer_rotoscope" - enables rotoscoping
"r_framebuffer_rotoscope_zedge" - uses a different method for edge detection which requires the zbuffer be bound to a texture.
finally how the effect 'should' look, this is my box with regular quake 3 demo four.
http://s239.photobucket.com/albums/ff277/gordallott/