Forum

9.13 reverse proxy: 100 continue and libevhtp

Klemens Schölhorn
28 May 2015, 19:20
First off, this is maybe not a bug in hiawatha but in libevhtp, but I wanted to hear your opinion on it.

I noticed a problem when using the reverse proxy to send large requests to a libevhtp (https://github.com/ellzey/libevhtp) server.
I originally saw this issue with seafile (see https://github.com/haiwen/seafile/issues/1229), but it can be reproduced using just curl and libevhtp itself (the included test_basic to be specific).
Basically, curl does not receive a final http response and so waits until it hits its timeout:

$ curl -v --data-binary @200KiB.txt http://domain.test/simple/
* Trying 127.0.0.1...
* Connected to domain.test (127.0.0.1) port 80 (#0)
> POST /simple/ HTTP/1.1
> Host: domain.test
> User-Agent: curl/7.42.1
> Accept: */*
> Content-Length: 204800
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
* Done waiting for 100-continue
< HTTP/1.1 100 Continue
* Empty reply from server
* Connection #0 to host domain.test left intact
curl: (52) Empty reply from server


This does not occur when contacting libevhtp directly or when leaving out the "Expect: 100-continue":
$ curl -v --data-binary @200.txt http://localhost:8081/simple/
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 8081 (#0)
> POST /simple/ HTTP/1.1
> Host: localhost:8081
> User-Agent: curl/7.42.1
> Accept: */*
> Content-Length: 204800
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 200 OK
< Content-Length: 6
< Content-Type: text/plain
<
* Connection #0 to host localhost left intact
simple

# curl -v -H "Expect:" --data-binary @200.txt http://domain.test/simple/
* Trying 127.0.0.1...
* Connected to domain.test (127.0.0.1) port 80 (#0)
> POST /simple/ HTTP/1.1
> Host: domain.test
> User-Agent: curl/7.42.1
> Accept: */*
> Content-Length: 204800
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Content-Length: 6
< Content-Type: text/plain
<
* Connection #0 to host domain.test left intact
simple


The hiawatha log looks like this:
::|Thu 28 May 2015 19:08:11 +0200|100|205228||POST /simple/ HTTP/1.1|Host: domain.test|User-Agent: curl/7.42.1|Accept: */*|Content-Length: 204800|Content-Type: application/x-www-form-urlencoded|Expect: 100-continue
::|Thu 28 May 2015 19:08:23 +0200|200|205251||POST /simple/ HTTP/1.1|Host: domain.test|User-Agent: curl/7.42.1|Accept: */*|Content-Length: 204800|Content-Type: application/x-www-form-urlencoded

The configuration is relatively standard:
VirtualHost {
Hostname = domain.test
WebsiteRoot = /srv/http/domain.test/www
WebDAVapp = yes
ReverseProxy .* http://127.0.0.1:8081 60
}


So this is either a bug in hiawatha or libevhtp gets confused whit hiawatha's "Expect" handling. (see https://www.hiawatha-webserver.org/forum/topic/1613/)
I suppose it is the latter, because I captured the conversation between hiawatha and libevhtp in the seafile issue (linked above) and libevhtp just did not send a response (except for the "HTTP/1.1 100 Continue").

I could also provide tcpdumps for this example, if it helps.

Thanks!
Hugo Leisink
10 June 2015, 15:47
Sorry for the too-long delay. I must have missed this post. I'll take a look at the 100 Continue issue soon. It's probably a Hiawatha issue, because it doesn't support that HTTP code.
Klemens Schölhorn
10 June 2015, 21:43
No problem, don't put yourself under pressure.
Hugo Leisink
11 June 2015, 19:38
Place the patch below in a file named 100-expect.patch, move that file to your Hiawatha source directory and apply the patch via the following command:
patch -p1 < 100-expect.patch

And of course let me know if it works.

diff -u old/src/http.c new/src/http.c
--- old/src/http.c 2015-05-04 17:51:49.000000000 +0200
+++ new/src/http.c 2015-06-11 19:28:02.169859000 +0200
@@ -21,6 +21,7 @@
#include <sys/time.h>
#include <sys/stat.h>
#include "global.h"
+#include "alternative.h"
#include "session.h"
#include "libstr.h"
#include "liblist.h"
@@ -29,6 +30,7 @@
#include "log.h"
#include "monitor.h"
#include "memdbg.h"
+#include "send.h"

#define REQUEST_BUFFER_CHUNK 4 * KILOBYTE
#define NO_REQUEST_LIMIT_TIME 300
@@ -186,6 +188,12 @@
session->bytes_in_buffer = header_length;
}

+ /* Handle 100-continue
+ */
+ if (strnstr(session->request, "Expect: 100-continue\r\n", header_length) != NULL) {
+ send_buffer(session, "HTTP/1.1 100 Continue\r\n\r\n", 25);
+ send_buffer(session, NULL, 0);
+ }
}
}

diff -u old/src/rproxy.c new/src/rproxy.c
--- old/src/rproxy.c 2015-03-13 21:52:33.000000000 +0100
+++ new/src/rproxy.c 2015-06-11 19:27:00.717859000 +0200
@@ -452,6 +452,10 @@

for (http_header = options->http_headers; http_header != NULL; http_header = http_header->next) {
if (rproxy->hostname != NULL) {
+ if (strcasecmp(http_header->data, "Expect: 100-Continue") == 0) {
+ continue;
+ }
+
if (strncasecmp(http_header->data, "Host:", 5) == 0) {
continue;
}
Klemens Schölhorn
12 June 2015, 17:41
Thanks for the patch. Unfortunately it does not solve the issue.
Curl now receives two "HTTP/1.1 100 Continue" (one from libevhtp and one from hiawatha?), but still no final response with the actual content.
$ curl -v --data-binary @200KiB.txt http://domain.test/simple/
* Trying 127.0.0.1...
* Connected to domain.test (127.0.0.1) port 80 (#0)
> POST /simple/ HTTP/1.1
> Host: domain.test
> User-Agent: curl/7.42.1
> Accept: */*
> Content-Length: 204800
> Content-Type: application/x-www-form-urlencoded
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
< HTTP/1.1 100 Continue
* Empty reply from server
* Connection #0 to host domain.test left intact
curl: (52) Empty reply from server

After the second "< HTTP/1.1 100 Continue" line there is a delay of 30 seconds.

PS: The new forum design is missing the BB-Code reference.
Hugo Leisink
14 June 2015, 16:15
Can you make your final website available from my webserver (IP: 141.138.201.249). I want to see for myself what goes wrong by configuring my own webserver as a reverse proxy for your website.
Klemens Schölhorn
14 June 2015, 17:01
I was just using the test_basic.c example of libevhtp version 1.2.10 (https://github.com/ellzey/libevhtp/releases/tag/1.2.10). It can be build relatively simple with cmake (only dependency is libevent2):
cd build
cmake ..
make
make examples


If you don't want to build it yourself, I can also start an instance on my machine and send you the link via email.
Klemens Schölhorn
14 June 2015, 18:38
Ok, I have no idea what is going on here, but it seems to work now. I was just about to set up two instances of the libevhtp example on my vps for further testing, when I noticed that it was actually working now (using hiawatha 9.13 with the above patch).
I will investigate further.
Hugo Leisink
15 June 2015, 18:50
I'll take a look at it when I have the time. Please note that the patch (or what I'm willing to implement) is not a full implementation of the 100 response code. It's simply a hack that simulates the communication. Hiawatha will always answer with a "HTTP/1.1 100 Continue" when it sees the "Expect: 100-continue" header in a request. No check is done.
This topic has been closed.