Tcpdump usage examples

See the list of interfaces on which tcpdump can listen:

tcpdump -D

Listen on interface eth0:

tcpdump -i eth0

Listen on any available interface (cannot be done in promiscuous mode. Requires Linux kernel 2.2 or greater):

tcpdump -i any

Be verbose while capturing packets:

tcpdump -v

Be more verbose while capturing packets:

tcpdump -vv

Be very verbose while capturing packets:

Making simple Splunk Nginx dashboard

As a DevOps guy I often do incident analysis, post deployment monitoring and usual logs checks. If you also is using Splunk as me when let me show for you few effective Splunk commands for Nginx logs monitoring.

Extract fileds

To make commands works Nginx log fields have to be extracted into variables.
Where are 2 ways to extract fields:

1. By default Splunk recognise “access_combined” log format which is default format for Nginx. If it is your case congratulations nothing to do for you!
2. For custom format of logs you will need to create regular expression. Splunk has built in user interface to extract fields or you can provide regular expression manually.

Simple shell script to backup MySQL databases

#!/bin/bash
# Simple script to backup MySQL databases

# Parent backup directory
backup_parent_dir="/var/backups/mysql"

# MySQL settings
mysql_user="root"
mysql_password=""

# Read MySQL password from stdin if empty
if [ -z "${mysql_password}" ]; then
  echo -n "Enter MySQL ${mysql_user} password: "
  read -s mysql_password
  echo
fi

How to Redirect Nginx traffic based on the Client’s IP Address

Sometimes developers need to redirect part of their traffic to another web area or specific URL. Depending on the IP address the visitors have, it’s ideal to geo-locate traffic or simply protect areas from unwanted specific IP addresses. So, how do you redirect traffic from IP 1.2.3.4 to http://www.thisurl.com/page2.html and the rest of the traffic to http://www.thisurl.com/? We’ll go over this today.

Redirect using Nginx HttpAccessModule

As Nginx supports conditional configurations, first let’s look at an example of how to redirect traffic using HttpAccessModule running a regex against $remote_addr variable.

server {
if ($remote_addr = 1.2.3.4) {
rewrite ^ http://www.yourwebsite.com/otherpage.htm;
}
}

In this example, 1.2.3.4 is the IP address you want to redirect.

SYN Flooding using SCAPY and Prevention using iptables

DoS (Denial of Service) attacks against Web services make them unavailable for legitimate users, affecting the website owner’s potential business. These involve intentional consumption of network, CPU and memory resources. In this article, I will demonstrate how to do a SYN flood using the SCAPY framework, along with other preventive measures.

Over time, DoS attacks have become more complicated, concealing malicious client requests as legitimate ones. Also, a distributed approach, the DDoS (Distributed Denial of Service) is now being adopted, which involves generating multiple requests to create a flood scenario. One type of DDoS flood attack is the TCP SYN queue flood.
A SYN queue flood attack takes advantage of the TCP protocol’s “three-way handshake”. A client sends a TCP SYN (S flag) packet to begin a connection to the server. The target server replies with a TCP SYN-ACK (SA flag) packet, but the client does not respond to the SYN-ACK, leaving the TCP connection “half-open”. In normal operation, the client should send an ACK (a flag) packet followed by the data to be transferred, or an RST reply to reset the connection. On the target server, the connection is kept open, in a “SYN_RECV” state, as the ACK packet may have been lost due to network problems.

How to Fix Nf_conntrack Table Full Dropping Packet

Issue

Packet drops on this system for connections using ip_conntrack or nf_conntrack. Following messages seen in /var/log/kern on the centos nodes when one of the instances drops packets:

$ tail -f /var/log/kern
Jul  4 03:47:16 centos kernel: : nf_conntrack: table full, dropping packet
Jul  4 03:47:16 centos kernel: : nf_conntrack: table full, dropping packet

This can happen when you are being attacked, or is also very likely to happen on a busy server even if there is no malicious activity.

NOTE: By default, CentOS will set this maximum to 65,536 connections. This is enough for lightly loaded servers, but can easily be exhausted on heavy traffic servers.

How to Fix

Nginx: 24: Too Many Open Files Error And Solution

How do I fix this problem under CentOS / RHEL / Fedora Linux or UNIX like operating systems?

Linux / UNIX sets soft and hard limit for the number of file handles and open files. You can use ulimit command to view those limitations:
su - nginx
To see the hard and soft values, issue the command as follows:
ulimit -Hn
ulimit -Sn

Increase Open FD Limit at Linux OS Level

Your operating system set limits on how many files can be opened by nginx server. You can easily fix this problem by setting or increasing system open file limits under Linux. Edit file /etc/sysctl.conf, enter:
# vi /etc/sysctl.conf
Append / modify the following line:
fs.file-max = 70000
Save and close the file. Edit /etc/security/limits.conf, enter:
# vi /etc/security/limits.conf
Set soft and hard limit for all users or nginx user as follows:

nginx       soft    nofile   10000
nginx       hard    nofile  30000

Save and close the file. Finally, reload the changes with sysctl command:
# sysctl -p