Server problems today

[Mr. Bill]

We're having issues this morning with the server for thesurvivalpodcast.com and survivalpodcast.net suddenly slowing to a crawl for 5 to 30 minutes at a time.  Problem not identified yet -- just wanted to let you know we're working on it.

[Stwood]

Maybe the BS from the last coupla days is oozing around in the server yet. 

[Smurf Hunter]

Assuming you haven't changed anything, one of the biggest clues on a linux machine is file handles.

https://en.wikipedia.org/wiki/Lsof

If you have the parent process ID of your HTTP server, you can get a listing of all the open files associated.  Remember in unix a network socket is also a file.
Often the count of open files can grow over time.  You can get a count doing something like this:

Code: [Select]

lsof 2> /dev/null | awk {'print $2'} 2> /dev/null | grep $PID | wc -l
Compare the results immediately after a restart of your HTTP to several days later.

Hope the unsolicited advice is not taken badly.

[Mr. Bill]

Thanks for the suggestion -- I'll look at that (but I think Archer did already).  At the moment we can't find anything wrong on the server, so we're suspecting it's an issue with either the hardware or something that the server provider needs to fix.  Awaiting more data.

[surfivor]

What is running, how much disk:


$ top

top - 15:57:06 up  4:58,  1 user,  load average: 0.54, 0.46, 0.25
Tasks: 254 total,   3 running, 251 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.4 us,  0.0 sy,  0.0 ni, 97.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  4025404 total,   299848 free,  3247088 used,   478468 buff/cache
KiB Swap:  1287164 total,   952336 free,   334828 used.   335708 avail Mem

   PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND     
  4223 lbob   20   0  503136  28580  13680 S  2.2  0.7   0:07.00 gnome-term+
     1 root      20   0  119888   4056   2944 S  0.0  0.1   0:02.32 systemd     
     2 root      20   0       0      0      0 S  0.0  0.0   0:00.01 kthreadd   
     4 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 kworker/0:+
     6 root      20   0       0      0      0 S  0.0  0.0   0:01.16 ksoftirqd/0
     7 root      20   0       0      0      0 S  0.0  0.0   0:13.05 rcu_sched   
     8 root      20   0       0      0      0 S  0.0  0.0   0:00.00 rcu_bh     
     9 root      rt   0       0      0      0 S  0.0  0.0   0:00.00 migration/0
    10 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 lru-add-dr+
    11 root      rt   0       0      0      0 S  0.0  0.0   0:00.04 watchdog/0 
    12 root      20   0       0      0      0 S  0.0  0.0   0:00.00 cpuhp/0     
    13 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kdevtmpfs   
    14 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 netns       
    15 root      20   0       0      0      0 S  0.0  0.0   0:00.02 khungtaskd 
    16 root      20   0       0      0      0 S  0.0  0.0   0:00.00 oom_reaper 
    17 root       0 -20       0      0      0 S  0.0  0.0   0:00.00 writeback   
    18 root      20   0       0      0      0 S  0.0  0.0   0:00.00 kcompactd0

============

$ df

df: /mnt/hgfs: Protocol error
Filesystem     1K-blocks     Used Available Use% Mounted on
udev             1989176        0   1989176   0% /dev
tmpfs             402544    11408    391136   3% /run
/dev/sda1       45044148 16888592  25844388  40% /
tmpfs            2012700     8104   2004596   1% /dev/shm
tmpfs               5120        4      5116   1% /run/lock
tmpfs            2012700        0   2012700   0% /sys/fs/cgroup
cgmfs                100        0       100   0% /run/cgmanager/fs
tmpfs             402544       32    402512   1% /run/user/1000
/dev/sr0         1855008  1855008         0 100% /media/lbob/Linux Mint 18.3 Cinnamon 64-bit

[outoforder2day]

If it's a virt and you haven't changed anything, it's probably a noisy neighbor. Not much you can do about that except open a ticket with your host provider to help troubleshoot.
In AWS, you can restart the system and it will move to another hypervisor.

[Mr. Bill]

If it's a virt and you haven't changed anything, it's probably a noisy neighbor. Not much you can do about that except open a ticket with your host provider to help troubleshoot. ...

It's a dedicated server.  Archer opened a ticket but they didn't provide any useful information.

$ top

Yup, that's the first thing we tried, and that's what's weird: nothing was using a lot of CPU.  Disk has plenty of free space, swap file not being used.

We're keeping an eye on it -- maybe we can catch it in the act and find some useful evidence.  Anyway, for whatever reason it's working plenty fast at the moment.

[archer]

looks like it was something upstream, possibly in the isp. but they wont admit to anything..

[outoforder2day]

Dedicated server, not a vps? Nice.
I do a lot of debugging for web and db systems in nix and windows. If you want a second set of eyes, feel free to reach out.

[Mr. Bill]

[nerdstuff]

There are probably a gazillion tools for pinging a site and monitoring the response time, but I've been using this bash script:

prettyping

It runs in a terminal and uses colored blocks to output a little bar chart of response time.  With some minor fiddling it will work with httping instead of ping.



Oughta run on any Linux or Mac OS X.

[/nerdstuff]

[Smurf Hunter]

curl is your friend.

I'd setup a crontab that does a curl for some URL and log the output.

Here's an example of how curl can break down the time spent:

https://stackoverflow.com/questions/18215389/how-do-i-measure-request-and-response-times-at-once-using-curl?utm_medium=organic&utm_source=google_rich_qa&utm_campaign=google_rich_qa

[surfivor]

What about nmap ? There is probably a way with nmap.
You can put this is a script or ruby program with a loop and a sleep 10 seconds each time

https://stackoverflow.com/questions/22173720/nmap-repeatedly-scan-a-port-for-a-syn-ack-reply-and-output-to-file



nmap -n -sn -PS80 thesurvivalpodcast.com

Starting Nmap 7.01 ( https://nmap.org ) at 2018-04-11 14:34 EDT
Nmap scan report for thesurvivalpodcast.com (206.190.141.186)
Host is up (0.082s latency).
Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

=================

nmap -n -Pn -p 80 thesurvivalpodcast.com

Starting Nmap 7.01 ( https://nmap.org ) at 2018-04-11 14:37 EDT
Nmap scan report for thesurvivalpodcast.com (206.190.141.186)
Host is up (0.080s latency).
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

[surfivor]

# Ruby script in myping.rb, then run with "ruby ./myping.rb"
# this will run the nmap command every 15 seconds until you hit control-c
#
#######

cmd = 'nmap -n -Pn -p 80 thesurvivalpodcast.com'

while true
  system(cmd)
  sleep 15
end

[Smurf Hunter]

What about nmap ? There is probably a way with nmap.
You can put this is a script or ruby program with a loop and a sleep 10 seconds each time

https://stackoverflow.com/questions/22173720/nmap-repeatedly-scan-a-port-for-a-syn-ack-reply-and-output-to-file



nmap -n -sn -PS80 thesurvivalpodcast.com

Starting Nmap 7.01 ( https://nmap.org ) at 2018-04-11 14:34 EDT
Nmap scan report for thesurvivalpodcast.com (206.190.141.186)
Host is up (0.082s latency).
Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

=================

nmap -n -Pn -p 80 thesurvivalpodcast.com

Starting Nmap 7.01 ( https://nmap.org ) at 2018-04-11 14:37 EDT
Nmap scan report for thesurvivalpodcast.com (206.190.141.186)
Host is up (0.080s latency).
PORT   STATE SERVICE
80/tcp open  http

Nmap done: 1 IP address (1 host up) scanned in 0.14 seconds

That's not good enough.  nmap would only validate that TCP was listening on port 80.

You could delete all the TSP content and that would still pass.  Curl could additionally download index.html and then you could parse that looking for signs of life.

e.g.

Code: [Select]

curl http://thesurvivalpodcast.com/forum/index.php
grep "The Survival Podcast Forum - Index" --output index.php
RESULT=$?
if [ $RESULT -eq 0 ]; then
  echo success
else
  echo failed
fi

So

[surfivor]

# I guess the similar ruby code is something like this.
# I never use PHP anymore and not much shell scripts
# https://jhawthorn.github.io/curl-to-ruby/

require 'net/http'
require 'uri'

uri = URI.parse("http://thesurvivalpodcast.com/forum/index.php")
response = Net::HTTP.get_response(uri)

# response.body should have html
# response.code
# response.body

[surfivor]

# simple version
# put this in myping.rb

require 'net/http'
require 'uri'

uri = URI.parse("http://thesurvivalpodcast.com/forum/index.php")
response = Net::HTTP.get_response(uri)

puts response.code
puts response.body[0..300]


if response.body =~ /The Survival Podcast Forum - Index/
  puts 'SUCCESS!'
else
 puts 'FAILED!'
end

[surfivor]

# continuous version of script, pings every 15 seconds
# - I tested it
# ruby ./myping.rb
#
# we could make it keep track of how many failures there are or exit after 20 failures etc

require 'net/http'
require 'uri'

def ping_tsp
  uri = URI.parse("http://thesurvivalpodcast.com/forum/index.php")
  response = Net::HTTP.get_response(uri)

  if response.body =~ /The Survival Podcast Forum - Index/
    puts 'SUCCESS!'
  else
   puts ' ***************** FAILED! **********************'
  end

end


while true
  ping_tsp
  sleep 15
end

[Smurf Hunter]

ruby?  Come on man.  We only needed a bash shell script and you HAD to go there... 

[Mr. Bill]

If you can't do it in Applesoft BASIC, it doesn't need to be done.

[outoforder2day]

All the cool kids are using Prometheus or the TICK stack. 

We're POCing both of the above actually. It's going to upgrade our current Zabbix deployment that I set up a few years back. Then again, nothing beats a good old (and simple) nagios check script.

I think the problem here, though, is that it's more than just website availability monitoring. You really want to find the underlying performance issue on the server.  That's where tools like the tick stack come in. Telegraf, the T in TICK, has a ridiculously robust plugin ecosystem. It can enumerate performance data for apache/nginx/mysql/postegresql/etc right out of the box with no real config. You can then shoot it to whatever system you want to grab and analyze the metrics.

[surfivor]

 I know, I get accused of advocating solutions that I like, but Ruby is an excellent tool and I think is installed on most linux systems. It is one of the better object oriented scripting languages. Some people like Python too

When you may want to add complex features, logging,and constant monitoring in your script then ruby will give you the lift and I can help with that. It is a popular language and good to know about

[Smurf Hunter]

I realize TSP forum is only a quasi-commercial venture.  I mean, if you were making $10,000 daily in website transactions, paying for NewRelic, adobe analytics, etc. are well worth it.
We recently had a problem with our shopping cart checkout, where the credit card fields were inactive (user could not input character) on a specific iOS version only on 2nd gen iPad.
Most of the above solutions wouldn't have caught that, and truthfully it was < 1% of our business, so we didn't noticed a decline in sales for that device/segment.

You can take things as far as you want, but there's a solution for literally anything you might think of.

[surfivor]

 I would think there would be some freebie solutions, some python utilities that you could use or develop something yourself.

 One thing is that it is an interesting problem so you can learn something from the problem or in monitoring it, then if you go to a job interview you can always claim some experience in that area even if you exaggerate a little as long as you have done some home work and know some important details about the topic.

 That was why I was saying Ruby, some people always want the simplest and cheapest solution that takes no time but if you want to learn something then you can work with it and fiddle around. The first part of the problem can be identifying exactly what you are trying to figure out. If you have some fancy tool that does everything then you don't always have to explain the problem since it can solve mutliple problems. For me in that case I am not really learning anything other than how to use fancy tools and that is not the area of expertise I try to cultivate because I am more of a programmer/developer. Security is an interesting area I find, but it is not what I know the most about but I am always looking to see what I can learn related to programming, linux tools etc

[shawnyadav]

Bill, just my two cents. The Survival Podcast should have it's own dedicated app so listeners can get interrupted access to the podcasts. The forum has been loading slow lately as well. A mobile app may fix this concern do give it a thought. I remember there used to be an android app but I can no longer can find it.

[Mr. Bill]

Bill, just my two cents. The Survival Podcast should have it's own dedicated app so listeners can get interrupted access to the podcasts. The forum has been loading slow lately as well. A mobile app may fix this concern do give it a thought. I remember there used to be an android app but I can no longer can find it.

I'm not involved in the actual podcast, and (horrors!) I actually don't own a smartphone, so I can't be too helpful with a podcast app.  I believe there used to be a TSP app, but I think most folks are using one or another general-purpose podcast app nowadays.  (Somebody help me out on this?)

I got sidetracked fixing some totally unrelated server problems, and haven't gotten back to the intermittent-slowness issue yet.  I'll see what I can learn this coming week.

[outoforder2day]

I would think there would be some freebie solutions, some python utilities that you could use or develop something yourself.

 One thing is that it is an interesting problem so you can learn something from the problem or in monitoring it, then if you go to a job interview you can always claim some experience in that area even if you exaggerate a little as long as you have done some home work and know some important details about the topic.

 That was why I was saying Ruby, some people always want the simplest and cheapest solution that takes no time but if you want to learn something then you can work with it and fiddle around. The first part of the problem can be identifying exactly what you are trying to figure out. If you have some fancy tool that does everything then you don't always have to explain the problem since it can solve mutliple problems. For me in that case I am not really learning anything other than how to use fancy tools and that is not the area of expertise I try to cultivate because I am more of a programmer/developer. Security is an interesting area I find, but it is not what I know the most about but I am always looking to see what I can learn related to programming, linux tools etc

Don't reinvent the wheel. Use the free tools that are out there to get to where you want. It's the same principle that you're advocating, but just not with ruby. Don't write your own monitoring system. Use an existing one and extend it if you need to. You'll get a better performing more feature filled solution that way, usually with much less effort long-term.

[shawnyadav]

I'm not involved in the actual podcast, and (horrors!) I actually don't own a smartphone, so I can't be too helpful with a podcast app.  I believe there used to be a TSP app, but I think most folks are using one or another general-purpose podcast app nowadays.  (Somebody help me out on this?)

I got sidetracked fixing some totally unrelated server problems, and haven't gotten back to the intermittent-slowness issue yet.  I'll see what I can learn this coming week.

Forums these days mostly using tapatalk to make their app, but a dedicated app can allow you to personalize it according to yourself which is a better option. Myself being a developer I can surely get an app done as a part of contribution towards TSP community. What say?

Shawn Yadav
https://showbox.onl/ https://mobdro.onl/ https://dltutuapp.com/

[surfivor]

 I have a possible opportunity to go to defcon in Vegas, anyone know much about that ?

It's hard to figure out what is going to be there, it seems like a big smorgsborg but it's a hacking conference

https://www.defcon.org/

[archer]

I have a possible opportunity to go to defcon in Vegas, anyone know much about that ?

It's hard to figure out what is going to be there, it seems like a big smorgsborg but it's a hacking conference

https://www.defcon.org/

if you go, leave your electronic devices at home.