[PATCH] security: Don't stop restoring labels too early