I've tested this out myself and it's not very slow. Probably the same speed as the normal lighting calc. Another thing is, when doing multiple lights, you can't use this skip the same way. This is my setup :
//if ( incoming <= 0 ) {
//*(int *)&colors[i*4] = ambientLightInt; // can't use this here now
//continue;
//}
if( incoming < 0) // needed because incoming can be -1 while bounce is 1 (opposite sides)
incoming = 0;
if( bounce < 0)
bounce = 0;
if( (incoming <= 0) && (bounce <= 0) ){
*(int *)&colors[i*4] = ambientLightInt; // now we can use this little skip!
continue;
}