Maxim Dounin [Mon, 5 Mar 2012 12:36:51 +0000 (12:36 +0000)]
Merge of r4471:
Variables: honor no_cacheable for not_found variables.
Variables with the "not_found" flag set follow the same rules as ones with
the "valid" flag set. Make sure ngx_http_get_flushed_variable() will flush
non-cacheable variables with the "not_found" flag set.
This fixes at least one known problem with $args not available in a subrequest
(with args) when there were no args in the main request and $args variable was
queried in the main request (reported by Laurence Rowe aka elro on irc).
Also this eliminates unneeded call to ngx_http_get_indexed_variable() in
cacheable case (as it will return cached value anyway).
Maxim Dounin [Mon, 5 Mar 2012 12:33:06 +0000 (12:33 +0000)]
Merge of r4470:
Fix for proxy_store leaving temporary files for subrequests.
Temporary files might not be removed if the "proxy_store" or "fastcgi_store"
directives were used for subrequests (e.g. ssi includes) and client closed
connection prematurely.
Non-active subrequests are finalized out of the control of the upstream
module when client closes a connection. As a result, the code to remove
unfinished temporary files in ngx_http_upstream_process_request() wasn't
executed.
Fix is to move relevant code into ngx_http_upstream_finalize_request() which
is called in all cases, either directly or via the cleanup handler.
Maxim Dounin [Mon, 5 Mar 2012 12:15:02 +0000 (12:15 +0000)]
Merge of r4461: upstream: fixed "too big header" check.
If header filter postponed processing of a header by returning NGX_AGAIN
and not moved u->buffer->pos, previous check incorrectly assumed there
is additional space and did another recv() with zero-size buffer. This
resulted in "upstream prematurely closed connection" error instead
of correct "upstream sent too big header" one.
Maxim Dounin [Sun, 5 Feb 2012 20:06:50 +0000 (20:06 +0000)]
Merge of r4422:
Fixed error handling in ngx_event_connect_peer().
Previously if ngx_add_event() failed a connection was freed two times (once
in the ngx_event_connect_peer(), and again by a caller) as pc->connection was
left set. Fix is to always use ngx_close_connection() to close connection
properly and set pc->connection to NULL on errors.
Maxim Dounin [Sun, 5 Feb 2012 20:05:11 +0000 (20:05 +0000)]
Merge of r4416:
Fixed AIO error handling on FreeBSD.
The aio_return() must be called regardless of the error returned by
aio_error(). Not calling it resulted in various problems up to segmentation
faults (as AIO events are level-triggered and were reported again and again).
Additionally, in "aio sendfile" case r->blocked was incremented in case of
error returned from ngx_file_aio_read(), thus causing request hangs.
Maxim Dounin [Sun, 5 Feb 2012 19:27:18 +0000 (19:27 +0000)]
Merge of r4405:
Fixed division by zero exception in ngx_hash_init().
The ngx_hash_init() function did not expect call with zero elements count,
which caused FPE error on configs with an empty "types" block in http context
and "types_hash_max_size" > 10000.
Maxim Dounin [Sun, 5 Feb 2012 19:25:24 +0000 (19:25 +0000)]
Merge of r4404:
Fixed sched_setaffinity(2) to correctly pass size.
Second argument (cpusetsize) is size in bytes, not in bits. Previously
used constant 32 resulted in reading of uninitialized memory and caused
EINVAL to be returned on some Linux kernels.
Maxim Dounin [Sun, 5 Feb 2012 19:15:09 +0000 (19:15 +0000)]
Merge of r4401, r4415:
SSL changes:
*) Added support for TLSv1.1, TLSv1.2 in ssl_protocols directive.
Support for TLSv1.1 and TLSv1.2 protocols was introduced in
OpenSSL 1.0.1 (-beta1 was recently released). This change makes it
possible to disable these protocols and/or enable them without other
protocols.
*) Removed ENGINE_load_builtin_engines() call.
It's already called by OPENSSL_config(). Calling it again causes
some openssl engines (notably GOST) to corrupt memory, as they don't
expect to be created more than once.
Maxim Dounin [Sun, 5 Feb 2012 16:12:55 +0000 (16:12 +0000)]
Merge of r4384, r4385:
Fixes for limit_rate:
*) Fixed throughput problems with large limit_rate.
Previous attempt to fix this was in r1658 (0.6.18), though that one
wasn't enough (it was a noop).
*) Fixed interaction of limit_rate and sendfile_max_chunk.
It's possible that configured limit_rate will permit more bytes per
single operation than sendfile_max_chunk. To protect disk from
takeover by a single client it is necessary to apply sendfile_max_chunk
as a limit regardless of configured limit_rate.
See here for report (in Russian):
http://mailman.nginx.org/pipermail/nginx-ru/2010-March/032806.html
Maxim Dounin [Sun, 5 Feb 2012 15:47:58 +0000 (15:47 +0000)]
Merge of r4381, r4400:
Fixed limit_conn_log_level/limit_req_log_level inheritance. The
directives did not work if there were no limit_conn/limit_req
specified on the same level.
The code for limit_conn is different in 1.0.x, conflict resolved
manually.
Maxim Dounin [Sun, 5 Feb 2012 13:53:50 +0000 (13:53 +0000)]
Merge of r4372, r4373, r4374:
SCGI fixes:
*) Fixed incorrect use of r->http_version in scgi module.
The r->http_version is a version of client's request, and modules
must not set it unless they are really willing to downgrade protocol
version used for a response (i.e. to HTTP/0.9 if no response headers
are available). In neither case r->http_version may be upgraded.
The former code downgraded response from HTTP/1.1 to HTTP/1.0 for no
reason, causing various problems (see ticket #66). It was also
possible that HTTP/0.9 requests were upgraded to HTTP/1.0.
*) Removed duplicate function declaration.
*) Removed error if there is no Status header.
The SCGI specification doesn't specify format of the response, and
assuming CGI specs should be used there is no reason to complain.
RFC 3875 explicitly states that "A Status header field is optional,
and status 200 'OK' is assumed if it is omitted".
Maxim Dounin [Sun, 5 Feb 2012 13:34:08 +0000 (13:34 +0000)]
Merge of r4339, r4340, r4341:
Cache fixes:
*) Obsolete code removed.
The ngx_http_cache() and ngx_http_no_cache_set_slot() functions
were replaced in 0.8.46 and no longer used since then.
*) Handling of cache files with long headers.
There are two possible situations which can lead to this: response
was cached with bigger proxy_buffer_size value (and nginx was
restared since then, i.e. shared memory zone content was lost), or
due to the race in the cache update code (see [1]) we've end up
with fcn->body_start from a different response stored in shared
memory zone.
*) Only complain on long locked entries.
There have been multiple reports of cases where a real locked entry
was removed, resulting in a segmentation fault later in a worker
which locked the entry. It looks like default inactive timeout isn't
enough in real life.
For now just ignore such locked entries, and move them to the top of
the inactive queue to allow processing of other entries.
Maxim Dounin [Sun, 5 Feb 2012 12:42:36 +0000 (12:42 +0000)]
Merge r4336:
Microoptimization of sendfile(2) usage under FreeBSD.
FreeBSD kernel checks headers/trailers pointer against NULL, not
corresponding count. Passing NULL if there are no headers/trailers
helps to avoid unneeded work in kernel, as well as unexpected 0 bytes
GIO in traces.
Maxim Dounin [Wed, 14 Dec 2011 18:06:21 +0000 (18:06 +0000)]
Merge of r4315:
Allowed add_header for proxied 206 replies.
It was working for nginx's own 206 replies as they are seen as 200 in the
headers filter module (range filter goes later in the headers filter chain),
but not for proxied replies.
Maxim Dounin [Wed, 14 Dec 2011 15:30:02 +0000 (15:30 +0000)]
Merge of r4298:
Fixed flv header to match specification.
Used "\x5" in 5th byte to claim presence of both audio and video. Used
previous tag size 0 in the beginning of the flv body (bytes 10 .. 13) as
required by specification (see http://www.adobe.com/devnet/f4v.html).
Maxim Dounin [Wed, 14 Dec 2011 15:23:23 +0000 (15:23 +0000)]
Merge of r4284:
Introduction of simple ngx_write_stderr() instead of ngx_log_stderr()
for output of ./configure options, etc., since ngx_log_stderr() output
length is limited by 2048 characters defined as NGX_MAX_ERROR_STR.
Maxim Dounin [Wed, 14 Dec 2011 15:13:25 +0000 (15:13 +0000)]
Merge of r4275, r4276, r4278, r4279:
Fixes for proxy_set_header, fastcgi/scgi/uwsgi_param inheritance:
*) Fixed proxy_set_header inheritance with proxy_cache (ticket #45).
Headers cleared with cache enabled (If-Modified-Since etc.) might be
cleared in unrelated servers/locations without proxy_cache enabled
if proxy_cache was used in some server/location.
In both cases If-Modified-Since header wasn't sent to backend in
location /2.
Fix is to not modify conf->headers_source, but instead merge user-supplied
headers from conf->headers_source and default headers (either cache or not)
into separate headers_merged array.
*) Fixed proxy_set_header inheritance with proxy_set_body.
*) Separate functions to merge fastcgi/scgi/uwsgi params.
No functional changes.
*) Fixed fastcgi/scgi/uwsgi_param inheritance. The following problems were
fixed:
1. Directive fastcgi_cache affected headers sent to backends in unrelated
servers / locations (see ticket #45).
2. If-Unmodified-Since, If-Match and If-Range headers were sent to
backends if fastcgi_cache was used.
3. Cache-related headers were sent to backends if there were no
fastcgi_param directives and fastcgi_cache was used at server level.
Maxim Dounin [Wed, 14 Dec 2011 13:34:16 +0000 (13:34 +0000)]
Merge of r4231, r4300, r4303, r4304:
Configure/build changes and fixes:
*) Revamped "configure --help" text.
*) FreeBSD 10-current has recently gotten POSIX_FADV_* macros.
A fix for the broken build applied.
*) AIX 7 defines sys_nerr in errno.h, therefore <errno.h> included
in the sys_nerr test.
When sys_nerr and _sys_nerr are missed on a particular platform
and our euristic for a maximum errno detection applied always
print the maximum errno number we reached instead of printing void.
This makes possible to build nginx on AIX 7.1.
*) Made it possible to build/install from the SVN checkout.
Maxim Dounin [Tue, 13 Dec 2011 18:59:18 +0000 (18:59 +0000)]
Merge of r4221, r4222:
Fixed unix ngx_write_chain_to_file() to return total bytes written.
Previously result of last iteration's writev() was returned. This was
unnoticed as return value was only used if chain contained only one or
two buffers.
Maxim Dounin [Tue, 13 Dec 2011 18:34:34 +0000 (18:34 +0000)]
Merge of r4153:
Better handling of late upstream creation.
Configuration with duplicate upstream blocks defined after first use, i.e.
like
server {
...
location / {
proxy_pass http://backend;
}
}
upstream backend { ... }
upstream backend { ... }
now correctly results in "duplicate upstream" error.
Additionally, upstream blocks defined after first use now handle various
server directive parameters ("weight", "max_fails", etc.). Previously
configuration like
server {
...
location / {
proxy_pass http://backend;
}
}
upstream backend {
server 127.0.0.1 max_fails=5;
}
incorrectly resulted in "invalid parameter "max_fails=5"" error.
Maxim Dounin [Tue, 13 Dec 2011 18:30:15 +0000 (18:30 +0000)]
Merge of r4144:
Upstream: clearing of u->peer.connection on close.
This fixes crashes observed with some 3rd party balancer modules. Standard
balancer modules (round-robin and ip hash) explicitly set pc->connection
(aka u->peer.connection) to NULL and aren't affected.
Igor Sysoev [Tue, 1 Nov 2011 14:13:25 +0000 (14:13 +0000)]
Merging r4195, r4196:
AIO related fixes:
*) Fix for connection drops with AIO.
Connections serving content with AIO to fast clients were dropped with
"client timed out" messages after send_timeout from response start.
*) Fix for socket leak with "aio sendfile" and "limit_rate".
Second aio post happened when timer set by limit_rate expired while we have
aio request in flight, resulting in "second aio post" alert and socket leak.
The patch adds actual protection from aio calls with r->aio already set to
aio sendfile code in ngx_http_copy_filter(). This should fix other cases
as well, e.g. when sending buffered to disk upstream replies while still
talking to upstream.
The ngx_http_writer() is also fixed to handle the above case (though it's
mostly optimization now).
Igor Sysoev [Tue, 1 Nov 2011 14:09:15 +0000 (14:09 +0000)]
Merging r4193, r4194:
Autoindex fixes:
*) Autoindex: escape '?' in file names.
For files with '?' in their names autoindex generated links with '?' not
escaped. This resulted in effectively truncated links as '?' indicates
query string start.
This is an updated version of the patch originally posted at [1]. It
introduces generic NGX_ESCAPE_URI_COMPONENT which escapes everything but
unreserved characters as per RFC 3986. This approach also renders unneeded
special colon processing (as colon is percent-encoded now), it's dropped
accordingly.
Igor Sysoev [Tue, 1 Nov 2011 13:49:31 +0000 (13:49 +0000)]
Merging r4151, r4152, r4177:
HTTP cache related fixes:
*) Cache: fix for sending of empty responses.
Revert wrong fix for empty responses introduced in 0.8.31 and apply new
one, rewritten to match things done by static module as close as possible.
*) Cache: fix for sending of stale responses.
For normal cached responses ngx_http_cache_send() sends last buffer and then
request finalized via ngx_http_finalize_request() call, i.e. everything is
ok.
But for stale responses (i.e. when upstream died, but we have something in
cache) the same ngx_http_cache_send() sends last buffer, but then in
ngx_http_upstream_finalize_request() another last buffer is send. This
causes duplicate final chunk to appear if chunked encoding is used (and
resulting problems with keepalive connections and so on).
Fix this by not sending in ngx_http_upstream_finalize_request()
another last buffer if we know response was from cache.
*) Fixed cache bypass caching of non-cacheable replies (ticket #21).
If cache was bypassed with proxy_cache_bypass, cache-controlling headers
(Cache-Control, Expires) wasn't considered and response was cached even
if it was actually non-cacheable.
Igor Sysoev [Tue, 1 Nov 2011 13:45:33 +0000 (13:45 +0000)]
Merging r4147, r4148, r4149, r4150, r4207:
Fixes of combination of error_page and return directives:
*) Fix for incorrect 201 replies from dav module.
Replies with 201 code contain body, and we should clearly indicate it's
empty if it's empty. Before 0.8.32 chunked was explicitly disabled for
201 replies and as a result empty body was indicated by connection close
(not perfect, but worked). Since 0.8.32 chunked is enabled, and this
causes incorrect responses from dav module when HTTP/1.1 is used: with
"Transfer-Encoding: chunked" but no chunks at all.
Fix is to actually return empty body in special response handler instead
of abusing r->header_only flag.
See here for initial report:
http://mailman.nginx.org/pipermail/nginx-ru/2010-October/037535.html
*) Fix for double content when return is used in error_page handler.
Test case:
location / {
error_page 405 /nope;
return 405;
}
location /nope {
return 200;
}
This is expected to return 405 with empty body, but in 0.8.42+ will return
builtin 405 error page as well (though not counted in Content-Length, thus
breaking protocol).
Fix is to use status provided by rewrite script execution in case
it's less than NGX_HTTP_BAD_REQUEST even if r->error_status set. This
check is in line with one in ngx_http_script_return_code().
Note that this patch also changes behaviour for "return 302 ..." and
"rewrite ... redirect" used as error handler. E.g.
will actually return redirect to "http://example.com/" instead of builtin
405 error page with meaningless Location header. This looks like correct
change and it's in line with what happens on e.g. directory redirects
in error handlers.
*) Fix for "return 202" not discarding body.
Big POST (not fully preread) to a
location / {
return 202;
}
resulted in incorrect behaviour due to "return" code path not calling
ngx_http_discard_request_body(). The same applies to all "return" used
with 2xx/3xx codes except 201 and 204, and to all "return ... text" uses.
Fix is to add ngx_http_discard_request_body() call to
ngx_http_send_response() function where it looks appropriate.
Discard body call from emtpy gif module removed as it's now redundant.
Reported by Pyry Hakulinen, see
http://mailman.nginx.org/pipermail/nginx/2011-August/028503.html
*) Incorrect special case for "return 204" removed.
The special case in question leads to replies without body in
configuration like
Igor Sysoev [Tue, 1 Nov 2011 13:00:30 +0000 (13:00 +0000)]
Merging r4034, r4186, r4187, r4229, r4235, r4237:
SSL related fixes:
*) Better handling of various per-server ssl options with SNI.
SSL_set_SSL_CTX() doesn't touch values cached within ssl connection
structure, it only changes certificates (at least as of now, OpenSSL
1.0.0d and earlier).
As a result settings like ssl_verify_client, ssl_verify_depth,
ssl_prefer_server_ciphers are only configurable on per-socket basis while
with SNI it should be possible to specify them different for two servers
listening on the same socket.
Workaround is to explicitly re-apply settings we care about from context
to ssl connection in servername callback.
Note that SSL_clear_options() is only available in OpenSSL 0.9.8m+. I.e.
with older versions it is not possible to clear ssl_prefer_server_ciphers
option if it's set in default server for a socket.
*) Disabling SSL compression. This saves about 300K per SSL connection.
The SSL_OP_NO_COMPRESSION option is available since OpenSSL 1.0.0.
*) Releasing memory of idle SSL connection. This saves about 34K per SSL
connection. The SSL_MODE_RELEASE_BUFFERS option is available since
OpenSSL 1.0.0d.
*) Decrease of log level of some SSL handshake errors.
*) Fixed segfault on configuration testing with ssl (ticket #37).
The following config caused segmentation fault due to conf->file not
being properly set if "ssl on" was inherited from the http level:
http {
ssl on;
server {
}
}
*) Silently ignoring a stale global SSL error left after disabled renegotiation.
Igor Sysoev [Tue, 1 Nov 2011 11:26:56 +0000 (11:26 +0000)]
Merging r4020:
SIGWINCH/NOACCEPT signal is disabled now in non-daemon mode.
Non-daemon mode is currently used by supervisord, daemontools and so on
or during debugging. The NOACCEPT signal is only used for online upgrade
which is not supported when nginx is run under supervisord, etc.,
so this change should not break existant setups.
Igor Sysoev [Tue, 1 Nov 2011 11:25:15 +0000 (11:25 +0000)]
Merging r4017:
Move SO_ACCEPTFILTER and TCP_DEFER_ACCEPT checks into configure.
NetBSD 5.0+ has SO_ACCEPTFILTER support merged from FreeBSD, and having
accept filter check in FreeBSD-specific ngx_freebsd_config.h prevents it
from being used on NetBSD. Therefore move the check into configure (and
do the same for Linux-specific TCP_DEFER_ACCEPT, just to be in line).
Igor Sysoev [Tue, 1 Nov 2011 11:23:26 +0000 (11:23 +0000)]
Merging r4013, r4200:
error_log related fixes:
*) Complain on invalid log levels.
Previously only first log level was required to be correct, while
error_log directive in fact accepts list of levels (e.g. one may
specify "error_log ... debug_core debug_http;"). This resulted
in (avoidable) wierd behaviour on missing semicolon after error_log
directive, e.g.
error_log /path/to/log info
index index.php;
silently skipped index directive and it's arguments (trying to
interpret them as log levels without checking to be correct).
*) Fixed configuration summary and manpage contents for the special
--error-log-path=stderr case.
*) Fix of names of the referer hash size directives introduced in r3940.
*) Cosmetics: replaced NGX_CONF_TAKE1 to NGX_CONF_FLAG for "sendfile"
and "chunked_transfer_encoding" directives, to be in line with all
directives taking a boolean argument. Both flags will ensure that
a directive takes one argument.
*) Improved ngx_parse_time() code readability.
*) Preallocating exact number of default MIME types entries.
*) Stylistic change in checking the boolean expression.
*) Replaced magic constants representing default values of some directives
with appropriate #define's.
Igor Sysoev [Tue, 1 Nov 2011 11:06:31 +0000 (11:06 +0000)]
Merging r3992, r4192:
access_log related fixes:
*) Removal of error message about %name log_format parameters,
they have been deleted long ago in 0.5.0-RELEASE.
*) Improved access log escaping to better protect other software.
Some character sets (notably ISO-8859-1) have C1 control characters in
upper half, make sure to escape them.
Igor Sysoev [Fri, 30 Sep 2011 14:36:19 +0000 (14:36 +0000)]
Merging r3986, r4006, r4007, r4073:
Request body related fixes:
*) Always set timer in discard body handler, this fixes the cases
when request for static file is redirected by error_page to an SSI page.
*) Correctly set body if it's preread and there are extra data.
Previously all available data was used as body, resulting in garbage after
real body e.g. in case of pipelined requests. Make sure to use only as many
bytes as request's Content-Length specifies.
*) Fix body with request_body_in_single_buf.
If there were preread data and request body was big enough first part
of the request body was duplicated.
See report here:
http://mailman.nginx.org/pipermail/nginx/2011-July/027756.html
*) Bugfix: read event was not blocked after reading body.
Read event should be blocked after reading body, else undefined behaviour
might occur on additional client activity. This fixes segmentation faults
observed with proxy_ignore_client_abort set.
Igor Sysoev [Fri, 30 Sep 2011 14:30:01 +0000 (14:30 +0000)]
Merging r4011, r4012, r4136:
Proxy related fixes:
*) Fixing cpu hog with all upstream servers marked "down".
The following configuration causes nginx to hog cpu due to infinite loop
in ngx_http_upstream_get_peer():
upstream backend {
server 127.0.0.1:8080 down;
server 127.0.0.1:8080 down;
}
server {
...
location / {
proxy_pass http://backend;
}
}
Make sure we don't loop infinitely in ngx_http_upstream_get_peer() but stop
after resetting peer weights once.
Return 0 if we are stuck. This is guaranteed to work as peer 0 always exists,
and eventually ngx_http_upstream_get_round_robin_peer() will do the right
thing falling back to backup servers or returning NGX_BUSY.
*) Upstream: properly allocate memory for tried flags.
Previous allocation only took into account number of non-backup servers, and
this caused memory corruption with many backup servers.
See report here:
http://mailman.nginx.org/pipermail/nginx/2011-May/026531.html
*) Fix of cpu hog in event pipe.
If client closed connection in ngx_event_pipe_write_to_downstream(), buffers
in the "out" chain were lost. This caused cpu hog if all available buffers
were in the "out" chain. Fix is to call ngx_chain_update_chains() before
checking return code of output filter to avoid loosing buffers in the "out"
chain.
Note that this situation (all available buffers in the "out" chain) isn't
normal, it should be prevented by busy buffers limit. Though right now it
may happen with complex protocols like fastcgi. This should be addressed
separately.
Igor Sysoev [Fri, 30 Sep 2011 14:20:52 +0000 (14:20 +0000)]
Merging r4114, r4129:
Buffers reuse in chunked filter.
There were 2 buffers allocated on each buffer chain sent through chunked
filter (one buffer for chunk size, another one for trailing CRLF, about
120 bytes in total on 32-bit platforms). This resulted in large memory
consumption with long-lived requests sending many buffer chains. Usual
example of problematic scenario is streaming though proxy with
proxy_buffering set to off.
Introduced buffers reuse reduces memory consumption in the above problematic
scenario.
See here for initial report:
http://mailman.nginx.org/pipermail/nginx/2010-April/019814.html