Forum

Monitor not fast enough to get all statistics

Aquanet
30 May 2013, 12:49
Hello,

We are still having issues with monitor. We get around 1.5 Mln requests via hiawatha per day now and 1 thread of monitor is just not able to catch up with such speed. So we need to launch several threads (up to 10) but they overload the server.

Please let us know if there is any other solution to this.

Regards
Andrew
Hugo Leisink
31 May 2013, 09:08
Try setting MonitorStatsInterval to 1. There is a mistake in the manualpage for this option. The parameter is in minutes, not in seconds.
Aquanet
3 June 2013, 16:30
it seems we are also getting error 404 on our hiawatha nodes in monitor-access.log:



xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:06 +0000|404|889||GET /1369838017.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:07 +0000|404|889||GET /1369838028.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:07 +0000|404|889||GET /1369838035.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:07 +0000|404|889||GET /1369838037.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:07 +0000|404|889||GET /1369838039.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:08 +0000|404|889||GET /1369838045.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:08 +0000|404|889||GET /1369838050.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
xxx.xx.xx.xxx|Mon 03 Jun 2013 14:28:08 +0000|200|5022||GET /1369838054.txt.gz HTTP/1.1|Host: monitor|Connection: close|User-Agent: Mozilla/5.0 (compatible; Banshee PHP framework HTTP library)|Accept-Encoding: gzip
Hugo Leisink
3 June 2013, 17:52
That's because you run multiple log fetch threads at the same time. Don't do that. They all request an index. But if one downloads a .gz file, another thread might also try to download it. After downloading a monitor log file (the .txt.gz files), Hiawatha deletes it automaticly.
Aquanet
4 June 2013, 14:09
we now run 1 thread only, it seems it processes around 200,000 requests per 2-3 hours.
Aquanet
4 June 2013, 14:11
we are worried that we wont be able to process more than 5-10 mln. requests this way
Hugo Leisink
4 June 2013, 20:04
It could very well be possible that my monitoring solution isn't capable of handling that amount of requests. My problem is that I don't have a server/website with that amount of traffic to do the required testing.

Do you use 'MonitorRequests = yes' for one or more virtual hosts?
Aquanet
4 June 2013, 22:45
we have MonitorRequests set to yes for Default virtual host, and generally we wanted to use it for all other virtual hosts, but clearly it will be difficult to meet with 1 monitoring thread.

if you have in mind some high performance design to handle 10-100 million requests per day, we are ready to hire developers for implementing it and even making the code public after that...contributing to this community
Hugo Leisink
5 June 2013, 22:24
I will give this some thought. The downloading of the statistical information from the webservers is not the issue, right? The load is on the database server while inserting the statistical information, correct?
Aquanet
6 June 2013, 01:39
yes, that's correct. i understand that inserting hundreds of thousands of entries into db leads to high load, even on SSD disks that we have.
Hugo Leisink
6 June 2013, 17:00
I've made some changes to fetch_webserver_logs, which can now insert multiple records via one query instead of a separate query per record. Perhaps that will lower the load on the database.

I will do some last checks and bug testing before sending it to you. Please send me your e-mailaddress via the contact form.
Aquanet
16 June 2013, 20:26
hello hugo, thank you for the script, we are getting the following error:

[\u@\h \W]#\[\033[0m\]./fetch_webserver_logs
PHP Notice: Undefined offset: 1 in /var/www/monitor/database/fetch_webserver_logs on line 202
PHP Fatal error: Call to private method database_connection::type_to_format() from context 'MySQLi_connection_di' in /var/www/monitor/database/fetch_webserver_logs on line 166
[\u@\h \W]#\[\033[0m\]
Hugo Leisink
16 June 2013, 21:12
I think I told to make the function type_to_format() in libraries/database/database_connection.php 'protected'. It's 'private' now. Did I forget to mention that in the e-mail?
Aquanet
18 June 2013, 06:51
Thank you, Hugo, my bad.

http://i42.tinypic.com/n65and.png

We are now seeing this going at super high speed in ssh window.

Is that normal? For 1 day we see no update in any of the statistics in web pages.
Hugo Leisink
18 June 2013, 10:43
Those are SQL queries to insert log information into the database. I have no idea why they end up on your screen. I don't print them in my script. What did you do to see those SQL queries?

And to answer your question: no, that's not normal.
This topic has been closed.