Network Troubleshooting
Nick Urbanik <nicku@vtc.edu.hk>
Copyright Conditions: GNU FDL (see http://www.gnu.org/licenses/fdl.html)
Identifying and Solving Problems on the Network

Reference: Network Troubleshooting Tools, Joseph D. Sloan, O'Reilly, August 2001
Focus: Basics and Standard Tools
•	Solving network problems depends a lot on your understanding
•	Simple tools can tell you what you need to know
•	Example: ping is incredibly useful!
Troubleshooting
•	Avoid it by:
•	redundancy
•	documentation
•	training
•	Try quick fixes first
•	simple problems often have big effects:
•	is the power on?
•	is the network cable plugged into the right socket?  Is LED flashing?
•	has anything changed recently?
•	Change only one thing at a time
•	test thoroughly after the change
•	Be familiar with the system
•	maintain documentation
•	Be familiar with your tools
•	before trouble strikes
Troubleshooting: Learn as you go
•	Study and be familiar with the normal behaviour of your network
•	Monitoring tools can tell you when things are wrong
•	if you know what things look like when they are right
•	Using tools such as Ethereal can help you understand
•	your network, and
•	TCP/IP — better
Documentation
•	Maintain an inventory of equipment and software
•	a list mapping MAC addresses to machines can be very helpful
•	Maintain a change log for each major system, recording:
•	each significant change
•	each problem with the system
•	each entry dated, with name of person who made the entry
•	Two categories of documentation:
•	Configuration information
•	describes the system
•	use system tools to obtain a snapshot, e.g., sysreport in Red Hat Linux
•	Procedural information
•	How to do things
•	use tools that automatically document what you are doing, e.g., script
Documentation Tools
•	Use script:
	$ script ~/logs/logfile-$(date +%F-%R).log
•	starts a new shell
•	all you type, all output goes into the file
•	Add comments with # I tried this...
•	Use tee:
	$ arp –a | tee outfile
•	Use sudo: all commands are recorded in /var/log/secure
•	Use plod from
http://bullwinkle.deer-run.com/~hal/plod/
•	lets you record a worksheet easily
•	Perl, so fine on any platform
Purchasing Equipment
•	Better to:
•	spend enough for the short term (one or two years) or
•	“invest for the long term?”
•	Moore’s Law: exponential growth in “bang for the buck”
•	Maintenance costs more for older equipment
•	Count all the costs
•	Conclusion: often (but not always), getting cheaper equipment to cover needs for the next two years will save money
•	Buying excess capacity can waste a lot of money 

Host Network Configuration tools
•	ps — information about processes
•	top — dynamic information about processes
•	netstat — show connections and services, routing
•	lsof — list open files
•	ifconfig — shows and changes network interfaces
•	route — shows, changes routing table
•	ip — show, change, set network configuration
•	arp — shows MAC addresses
•	nmap — portscanner: shows open ports, identifies OS
Using ps to See If Server Running
•	Is the network service running on the server?
•	Is the web server running?
	$ ps aux | grep httpd
•	Is the DHCP server running?
	$ ps aux | grep dhcpd
•	Is the directory server running?
	$ ps aux | grep slapd
•	Windows: use the task manager
Using top to see Resource Hogs
•	The program top shows:
•	load average (the average number of processes that are ready to run, but for which no CPU is available)
•	a load average of 4 or more is “quite high”
•	processes that use the most resources
Using netstat –tua to See Network Connections
•	netstat –tua shows all network connections, including those listening
•	netstat –tu shows only connections that are established
•	netstat –i is like ifconfig, shows info and stats about each interface
•	netstat –nr shows the routing table, like route –n
•	Windows provides netstat also.
lsof: List Open Files
•	An amazingly useful tool
•	Available for almost any Unix system
•	lsof –i shows output to Internet and X.25 files, but won’t show connections that have terminated
•	lsof –i@ictlab.tyict.vtc.edu.hk will show only connections to that machine
•	Can monitor progress of an FTP transfer, many, many other applications
•	See manpage, FAQ and quick start guide.
•	Apparently, no equivalent tool available on Windows.
ifconfig
•	ifconfig eth0 — show stats on network interface eth0
•	sudo ifconfig lo 127.0.0.1 — configure the loopback interface, start it up
•	sudo ifconfig eth0 172.19.233.5 netmask 255.255.255.0 — configure eth0 with IP address 172.19.233.5/24
•	ifconfig — show all configured network interfaces
•	ifconfig –a — show all interfaces, including those not configured yet.
route
•	route –n — print routing table
•	route add 127.0.0.1 — add a route to localhost;
•	should have been done automatically when created device with ifconfig
•	route add –net 172.19.233.0 — add a route to the eth0 configured on previous slide
•	should have been done automatically by ifconfig
•	route add 172.19.64.0 gw 172.19.233.254 — add a static route to network 172.19.64.0 through router 172.19.233.254
•	route add default gw 172.19.233.253 — add a default route to 172.19.233.253 through eth0
Connectivity Testing: Cabling
•	Label cables clearly at each end
•	Cable testers 
•	ensure wired correctly, check:
•	attenuation
•	length — is it too long?
•	100BaseT: less than 100m
•	Is the activity light on the interface blinking?
Software tools: ping
•	Most useful check of connectivity
•	Universal
•	If ping hostname, includes a rough check of DNS
•	Sends an ICMP (Internet Control Message Protocol) ECHO_REQUEST
•	Waits for an ICMP ECHO_REPLY
•	Most pings can display round trip time
•	Most pings can allow setting size of packet
•	Can use to make a crude measurement of throughput
ping: Roughly Estimating Throughput
•	Example:
•	ping with packet size = 100 bytes, round-trip time = 30ms
•	ping with packet size = 1100 bytes, round-trip time = 60ms
•	So takes 30ms extra (15ms one way) to send additional 1000 bytes, or 8000 bits
•	Throughput is roughly 8000 bits per 15ms, or about 540,000 bits per second
•	A very crude measurement: no account for other traffic, treats all links on path, there and back, as one.
ping: Roughly Estimating Throughput
•	This can be expressed as a simple formula:
What ping Result is Good, Bad?
•	A steady stream of consistent replies indicates probably okay
•	Usually first reply takes longer due to ARP lookups at each router
•	After that, ARP results are cached
•	ICMP error messages can help understand results:
•	Destination Network Unreachable indicates the host doing ping cannot reach the network
•	Destination Host Unreachable may come from routers further away
How to Use ping?
•	Ensure local host networking is enabled first: ping localhost, local IP address
•	ping a known host on local network
•	ping local and remote interfaces on router
•	ping by IP as well as by hostname if hostname ping fails
•	confirm DNS with dig (or nslookup) — see later
•	Ping from more than one host

fping: flood ping
•	Designed to test a large number of hosts
•	more efficient than ping
•	Used extensively by monitoring software such as mon: http://www.kernel.org/software/mon/, nagios: http://www.nagios.org/
•	take care not to flood too much!
•	RPMs are available; I have built one and put on ictlab under ~ftp/pub/redhat/contrib
arping: uses ARP requests
•	Limited to local network
•	Can work with MAC or IP addresses
•	use to probe for ARP entries in router (very useful!)
•	packet filtering
•	can block ICMP pings, but
•	won't block ARP requests
Path Discovery: traceroute
•	Sends UDP packets
•	(Microsoft tracert sends ICMP packets)
•	increments Time to Live (TTL) in IP packet header
•	Sends three packets at each TTL
•	records round trip time for each
•	increases TTL until enough to reach destination
traceroute: How it Works
•	As IP packets pass through each router, TTL in IP header is decremented
•	Packet is discarded when TTL decrements to 0
•	ROUTER sends ICMP TIME_EXCEEDED message back to traceroute host
•	When UPD packet reaches destination, gets ICMP PORT_UNREACHABLE, since uses an unused high UDP port
traceroute Limitations
•	Each router has a number of IP addresses
•	but traceroute only shows the one it used
•	get different addresses when run traceroute from other end
•	sometimes route is asymmetric
•	router may be configured to not send ICMP TIME_EXCEEDED messages
•	get stars: * instead of round-trip time in traceroute output

Performance Measurements: delay
•	Three sources of delay:
•	transmission delay — time to put signal onto cable or media
•	depends on transmission rate and size of frame
•	propagation delay — time for signal to travel across the media
•	determined by type of media and distance
•	queuing delay — time spent waiting for retransmission in a router
Performance Measurements  2
•	bandwidth — the transmission rate through the link
•	relates to transmission time
•	throughput — amount of data that can be sent over link in given time
•	relates to all causes of delay
•	is not the same as bandwidth
•	Other measurements needed
•	I.e., for quality of service for multimedia
Throughput: Measuring with ping  						1
•	Measure throughput between two remote hosts: may use tools like ping
•	ping two locations with two packet sizes (4 pings altogether, minimum)
•	Example:

Throughput: Measuring with ping  						2
•	Time difference / 2 (round trip time (RTT) -> one way)
•	Divide by size difference in bits: 8000
•	Multiply by 1000 (ms -> seconds)
•	Convert bps to Mbps
Throughput: Measuring with ping  						3
Throughput: Measuring with ping  						4
Throughput: Measuring with ping  						5
•	Completing calculation for throughput between 205.153.61.1 and 205.153.61.2:
Path Performance: Other tools
•	Could use a tool like pathchar, bing, clink, pchar, or tmetric that performs this calculation for you
•	Use http://www.google.com to locate these tools
•	pathchar is only available in binary form
•	Others in source form, need compile with commands something like this:
$ cd bing-1.1.3
$ make
$ sudo make install
Path measurement with pathchar
$ sudo ./pathchar sina.com.hk
pathchar to sina.com.hk (202.85.139.140)
 can't find path mtu - using 1500 bytes.
 doing 32 probes at each of 45 sizes (64 to 1500 by 32)
 0 localhost (127.0.0.1)
 |   106 Mb/s,   293 us (698 us),  +q 1.18 ms (15.7 KB)
 1 172.19.35.246 (172.19.35.246)
 |    28 Mb/s,   488 us (2.10 ms)
 2 192.168.83.2 (192.168.83.2)
 3  * 1   448 798       2
 |    20 Mb/s,   273 us (3.25 ms)
 4 cw7204.vtc.edu.hk (202.40.210.220)
 |   6.8 Mb/s,   521 us (6.04 ms)
 5 210.176.123.37 (210.176.123.37)
 |    52 Mb/s,   20 us (6.31 ms)
 6 210.87.254.61 (210.87.254.61)
 |   136 Mb/s,   116 us (6.63 ms)
 7 g5-0-0.wttbr01.imsbiz.com (210.87.254.129)
 |    33 Mb/s,   0.94 ms (8.88 ms),  +q 1.48 ms (6.10 KB) *6
 8 iadvantage3-RGE.hkix.net (202.40.161.172)
 |   164 Mb/s,   45 us (9.04 ms),  +q 1.74 ms (35.6 KB) *6
 9 v005-m02.hk01.iadvantage.net (202.85.129.53)
 |   ?? b/s,   -66 us (8.88 ms)
10 202.85.129.136 (202.85.129.136)
 |   ?? b/s,   459 us (9.79 ms)
11 202.85.139.11 (202.85.139.11)
11 hops, rtt 6.18 ms (9.79 ms), bottleneck 6.8 Mb/s, pipe 9361 bytes
Path Performance: measuring
•	May use ftp to transfer a large file, measure time
•	tests whole path
•	problem: affected by disk I/O, xinetd
•	Use ttcp, not affected by disk I/O
•	Consists of a client and server
•	Need have installed at both ends
•	Part of Red Hat Linux, Cisco IOS
Example of use of ttcp
•	First, start receiver on ictlab:
$ ttcp -r -s
ttcp-r: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp
ttcp-r: socket
ttcp-r: accept from 172.19.32.30
ttcp-r: 16777216 bytes in 1.45 real seconds = 11285.88 KB/sec +++
ttcp-r: 9704 I/O calls, msec/call = 0.15, calls/sec = 6684.46
ttcp-r: 0.0user 0.2sys 0:01real 14% 0i+0d 0maxrss 0+2pf 0+0csw
•	Second, start transmitter on nickpc:
$ ttcp -t -s ictlab
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp  -> ictlab
ttcp-t: socket
ttcp-t: connect
ttcp-t: 16777216 bytes in 1.45 real seconds = 11335.64 KB/sec +++
ttcp-t: 2048 I/O calls, msec/call = 0.72, calls/sec = 1416.95
ttcp-t: 0.0user 0.0sys 0:01real 4% 0i+0d 0maxrss 0+2pf 0+0csw

The ip program, iproute
•	The ip program in the iproute package provides complete control over TCP/IP networking in a Linux system
•	Provides more networking control facilities than other TCP/IP implementations
•	Supports tunneling in many forms
•	iproute documentation is in two manuals, one for ip routing, the other for tunnelling
iproute and iptables
•	Between these software packages, you can:
•	throttle bandwidth for certain computers
•	throttle bandwidth to certain computers
•	fairly share bandwidth
•	protect your network from DoS attacks
•	protect Internet from your customers
•	multiplex many servers into one, for load balancing or for high availability
•	restrict access to your computers
•	limit access of your users to other hosts
•	do routing based on user id, MAC address, source IP, port, type of service, time of day or content
•	See the Linux Advanced Routing and Traffic Control HOWTO at http://tldp.org for details
Traffic Measurements: netstat -i
•	The netstat program can show statistics about network interfaces
•	Linux netstat shows lost packets in three categories:
•	errors,
•	drops (queue full: shouldn’t happen!)
•	overruns (last data overwritten by new data before old data was read: shouldn’t happen!)
•	drops and overruns indicate faulty flow control — bad!
•	These values are cumulative (since interface was up)
•	Could put a load on interface to see current condition, with ping –l, to send large number of packets to destination
•	See the difference in values
Measuring Traffic: netstat -i
•	Here we run netstat –i on ictlab:
$ netstat -i
Kernel Interface table
Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0    1500   0  407027830  0         0          0         1603191764  0         0          3       BMRU
lo        16436  0  2858402     0         0          0         2858402       0         0          0       LRU
•	Notice that of the 1.6 billion bytes transmitted, there were 3 overuns.
•	Next, blast the path you want to test with packets using ping –l or the spray program, and measure again.
Traffic measurements: ifconfig, ip
•	ifconfig and ip give more information than netstat –i:
$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:00:E2:35:AF:EE
          inet addr:172.19.64.52  Bcast:172.19.127.255  Mask:255.255.192.0
          IPX/Ethernet 802.2 addr:33001601:0000E235AFEE
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:407579600 errors:0 dropped:0 overruns:0 frame:0
          TX packets:1605655688 errors:0 dropped:0 overruns:3 carrier:0
          collisions:0 txqueuelen:100
          RX bytes:3055300191 (2913.7 Mb)  TX bytes:2048217058 (1953.3 Mb)
          Interrupt:18 Base address:0xd000
$ ip -s link list eth0
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:e2:35:af:ee brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    3058362227 407610495 0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    2140511920 1605768150 0       0       0       0
Getting more info using ip
•	The –s (-statistics) option to ip provides statistics.  Adding a second gives you even more:
$ ip -s -s link list eth0
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:00:e2:35:af:ee brd ff:ff:ff:ff:ff:ff
    RX: bytes  packets  errors  dropped overrun mcast
    3070792102 407726727 0       0       0       0
    RX errors: length  crc     frame   fifo    missed
               0        0       0       0       0
    TX: bytes  packets  errors  dropped carrier collsns
    2445799644 1606151878 0       0       0       0
    TX errors: aborted fifo    window  heartbeat
               0        3       0       0
Quick Guide to using ip: set up interface
•	Here we set up a network interface and give it the IP address 192.168.0.1/24:
$ ip link set dev eth1 up
$ ip addr add 192.168.0.1/24 brd + dev eth1
•	Two important points:
•	If you do not specify the netmask, a netmask of /32 is assumed
•	brd + means obtain broadcast address by setting the host bits
Quick Guide to using ip: set up routes
$ ip route add default dev eth1 via 192.168.0.254
$ ip route add 192.168.1.0/24 via 192.168.0.10
•	The last adds a static route to another network
•	the first adds the default route.
•	You can omit the device if the network can be reached through a particular interface without any ambiguity
•	I.e., ip is smart enough to figure out which network device to use, though specifying it doesn’t hurt.
Packet  Capture
tcpdump, Ethereal, ntop
What is Packet Capture?
•	Real time collection of data as it travels over networks
•	Tools called:
•	packet sniffers
•	packet analysers
•	protocol analysers, and sometimes even
•	traffic monitors
When Packet Capture?
•	Most powerful technique
•	When need to see what client and server are actually saying to each other
•	When need to analyse type of traffic on network
•	Requires understanding of network protocols to use effectively
Warning: Don’t Get Sacked!
•	Be sure that your boss agrees with you capturing packets on your company’s network
•	People have been sacked for doing this without permission!
•	Do not invade the privacy of others
•	Capturing passwords with insecure protocols such as telnet, ftp, http (that is not encrypted with TLS) is very easy
•	DON’T DO IT!
tcpdump
•	Available everywhere
•	Windows: http://windump.polito.it/
•	Syntax also used by other programs (such as Ethereal)
•	Often it is the only tool available, so good to know
•	Works by putting network interface into promiscuous mode
•	normal Ethernet interface will ignore packets not addressed to it
•	in promiscuous mode, will examine all packets that arrive, even those not addressed to it
How to use tcpdump
•	Can just type its name (as root):
$ sudo tcpdump
•	...but get a huge amount of data!
•	Can restrict the data collected using a filter
•	A filter may select addresses, protocols, port numbers,...
tcpdump: some options
•	-c n — capture a count of n packets then stop
•	-w file — write raw data to file.
•	Very useful — can filter and analyse this later with tcpdump, ethereal or other tools
•	but you cannot see what you are capturing till later!
•	-i interface — collect from interface instead of lowest numbered network interface
•	-s bytes — collect no more than bytes of data from each packet instead of default 68 bytes
•	-e  — show link level info, e.g., Ethernet addresses
•	-x — gives a hexadecimal dump of packets
•	excluding link level data
•	-X — display ASCII as well as hexadecimal if have –x option too
•	Many more options: man tcpdump
tcpdump Filters: host and port
•	Show all network traffic to and from 192.168.0.1:
tcpdump host 192.168.0.1
•	Show packets to 192.168.0.1:
tcpdump dst 192.168.0.1
•	Show packets to port 68 on 192.168.0.1:
tcpdump dst 192.168.0.1 and port 68

tcpdump filters: networks
•	Capture traffic to or from 205.153.60/24:
tcpdump net 172.19.64/18
•	can specify network as source or destination:
tcpdump src net 205.153.60/24
tcpdump dst net 172.19.64/18
tcpdump filters: protocol
•	tcpdump ip
•	tcpdump tcp
•	tcpdump ip proto ospf
•	This will catch DNS name lookups, but not zone transfers (which use tcp):
•	tcpdump udp port 53
tcpdump filters: combining
•	This will not work as you might expect:
•	tcpdump host ictlab and udp or arp
•	Instead, need group with parentheses, and quote:
•	tcpdump “host ictlab and (udp or arp)”
•	many more ways of filtering: man tcpdump
IP Header
TCP Header
UDP Header
Writing data to a file
sudo tcpdump -c 1000 -w ~/tmp/tcpdump.pcap
tcpdump: listening on eth0
1014 packets received by filter
0 packets dropped by kernel

Reading a dumped file
$ tcpdump -nr ~/tmp/tcpdump.pcap arp
22:32:41.751452 arp who-has 172.19.127.254 tell 172.19.127.29
22:32:41.863173 arp who-has 172.19.64.52 tell 172.19.64.63
22:32:41.863198 arp reply 172.19.64.52 is-at 0:0:e2:35:af:ee
22:32:42.082584 arp who-has 172.19.65.16 tell 172.19.125.229
22:32:43.113655 arp who-has 172.19.123.211 tell 172.19.65.2
22:32:44.635149 arp who-has 172.19.65.16 tell 172.19.127.106
22:32:44.874117 arp who-has 172.19.65.6 tell 172.19.126.174
22:32:45.147178 arp who-has 172.19.65.16 tell 172.19.126.240
22:32:45.209507 arp who-has 172.19.127.254 tell 172.19.125.127
22:32:45.212484 arp who-has 172.19.127.175 tell 172.19.125.127
22:32:45.239445 arp who-has 172.19.127.254 tell 172.19.125.212
22:32:45.455863 arp who-has 172.19.65.16 tell 172.19.126.194
22:32:45.540507 arp who-has 172.19.126.50 (44:30:54:59:43:4d) tell 172.19.65.10
22:32:45.562004 arp who-has 172.19.126.50 tell 172.19.65.2
HTTP
tcpdump -nr ~/tmp/tcpdump.pcap port http
22:43:32.633636 192.168.25.9.14075 > 172.19.64.52.http: S 1015952778:1015952778(0) win 6144 <mss 1460> (DF)
22:43:32.633693 172.19.64.52.http > 192.168.25.9.14075: S 1929920485:1929920485(0) ack 1015952779 win 5840 <mss 1460> (DF)
22:43:32.635828 192.168.25.9.14075 > 172.19.64.52.http: P 1:590(589) ack 1 win 6144 (DF)
22:43:32.635906 172.19.64.52.http > 192.168.25.9.14075: . ack 590 win 6479 (DF)
22:43:32.636758 172.19.64.52.http > 192.168.25.9.14075: P 1:217(216) ack 590 win 6479 (DF)
22:43:32.636982 172.19.64.52.http > 192.168.25.9.14075: F 217:217(0) ack 590 win 6479 (DF)
22:43:32.639080 192.168.25.9.14075 > 172.19.64.52.http: R 590:590(0) ack 217 win 0 (DF)
tcpdump: When reading TCP
•	format:
•	src > dst: flags data-seqno ack window urgent options
•	Flags are some combination of S (SYN), F (FIN), P (PUSH) or R (RST)  or a  single '.' (no flags).
•	The first  time  tcpdump  sees a tcp 'conversation', it prints the sequence number from the packet. 
•	On subsequent packets of the conversation, the difference  between  the current packet's sequence number and this initial sequence number is printed. 
Window
•	win nnn specifies data window the sending host will accept in future packets
•	I.e., the maximum number of bytes
•	TCP flow-control:
•	host reduces this number if congested or overloaded
•	will sometimes set to 0 to temporarily halt incoming traffic in this connection

Ethereal
King of the Packet Analysers!
Available for Linux, Unix, Windows
Ethereal
•	Ethereal can read data captured by tcpdump, e.g.,
$ ethereal –r tcpdump.pcap
•	or File -> Open
•	Can capture data itself
•	Uses same filter language as tcpdump

You can expand any protocol:
•	If we click on the + next to Bootstrap Protocol, we can see the details of the DHCP Request: 

Display Filters
•	Note the box at the bottom of Ethereal for display filters
•	Select only some of the packets captured for display
•	see man ethereal and search for DISPLAY FILTER SYNTAX
•	Different syntax than the syntax for capture filters
•	Example:
ip.src==172.19.64.52 and ip.dest==172.19.64.57
Tools -> Follow TCP Stream
•	Can view the contents of an entire TCP stream conversation, in ASCII or in hexadecimal.
•	Be careful not to invade your customers’ privacy.
•	Can use to check if a communications stream is really encrypted
Ntop: monitoring data at a point
•	The Ntop program
•	listens on a network interface
•	puts an Ethernet interface into promiscuous mode and
•	displays statistics through a web interface
•	Shows:
•	percentages of protocols,
•	which machines generate most traffic
•	which traffic is purely local, which traffic comes from outside, which traffic goes from inside to outside of network
Ntop RPM
•	I have made an RPM package of ntop
•	it’s the best one available, or at least it was when I made it :-)
•	Can get from /home/nfs/redhat/contrib/ntop-2.1.51-20021031nu2.i386.rpm
•	source rpm is there too
•	Or search for it on http://rpmfind.net/
•	Note that you will be prompted for a password when you install it.
Switched Networks
•	Problem: a switched network is really a point-to-point network
•	You cannot normally capture the unicast traffic from other hosts on a single switch port
•	Solution: many switches support port monitoring, where one port can monitor all traffic on a specified VLAN
•	Example: Cisco 3500XL switches provide the port monitor command:
•	port monitor vlan VLAN1
How monitor one machine?
•	You are asked to check out a server on a switched network: what to do?
•	Use a small hub, and use a notebook running the capture software

Are switched networks secure?
•	Is all unicast traffic on one port of a switch private?
•	No, there are tools (dsniff) freely available to temporarily make a switch behave like a hub, or that provide other ways to compromise switch security.
Port Scanning
Identify services offered by a remote computer
What is a port scanner?
•	Sends packets to various ports on a network device
•	Best one available everywhere is nmap
•	can identify the OS of the target machine
•	Do not port scan arbitrary machines in your company's network without permission!
•	May be interpreted as a cracking attempt
How does nmap identify OS?
•	RFCs leave interpretation of some things up to the implementer
•	RFCs do not specify how should work if get contradictory flags, strange sequences of inconsistent packets
•	Most TCP/IP implementations are not complete
•	Every implementation of TCP/IP is different; the “grey areas” are different from one OS to another.
•	nmap sends “strange” packets to the machine, detects how reacts, matches this against a file of OS fingerprints

Running nmap: Use xnmap
•	$ sudo –v
•	$ sudo xnmap &
•	Enter the IP address of machine(s) to identify
•	select other choices from buttons
•	press Start
•	xnmap is simply a way to easily generate command line options to nmap using a graphical interface
Uses of nmap
•	Identify the type of a computer that is causing trouble on the network
•	Check what network services a computer is really offering
•	compare with netstat -tua output
•	A cracked computer may be hiding some services with trojaned utilities
•	nmap can help you discover such services
Troubleshooting Protocols
DNS
email
using telnet
DNS troubleshooting
•	Suspect DNS when get long timeouts before see any response
•	ping name, IP address, see if only IP address works
•	tools on Linux, Unix:
•	dig, nslookup, host
•	tools on Windows:
•	nslookup
DNS: dig
•	The people who write the most common name server (Bind) promote dig, deprecate nslookup
•	dig output is in form of DNS resource records
•	can copy and paste straight into DNS database files
dig: Checking forward DNS lookup
$ dig sysadmin.no-ip.com

; <<>> DiG 9.2.1 <<>> sysadmin.no-ip.com
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23568
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3

;; QUESTION SECTION:
;sysadmin.no-ip.com.            IN      A

;; ANSWER SECTION:
sysadmin.no-ip.com.     60      IN      A       202.69.77.139

dig: checking forward DNS  2
;; AUTHORITY SECTION:
no-ip.com.              60      IN      NS      nf1.no-ip.com.
no-ip.com.              60      IN      NS      nf2.no-ip.com.
no-ip.com.              60      IN      NS      nf3.no-ip.com.

;; ADDITIONAL SECTION:
nf1.no-ip.com.          60      IN      A       66.185.166.131
nf2.no-ip.com.          60      IN      A       66.185.162.100
nf3.no-ip.com.          60      IN      A       216.66.37.10

;; Query time: 254 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Mon Feb 24 10:55:26 2003
;; MSG SIZE  rcvd: 154

dig: reverse lookup		1
$ dig -x 202.69.77.139

; <<>> DiG 9.2.1 <<>> -x 202.69.77.139
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22117
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0

;; QUESTION SECTION:
;139.77.69.202.in-addr.arpa.    IN      PTR

;; ANSWER SECTION:
139.77.69.202.in-addr.arpa. 3600 IN     PTR     077-139.onebb.com.
dig: reverse lookup		2
;; AUTHORITY SECTION:
77.69.202.in-addr.arpa. 3600    IN      NS      ns2.onebb.com.
77.69.202.in-addr.arpa. 3600    IN      NS      ns1.onebb.com.

;; Query time: 310 msec
;; SERVER: 172.19.64.52#53(172.19.64.52)
;; WHEN: Mon Feb 24 11:07:04 2003
;; MSG SIZE  rcvd: 111

dig syntax
dig [options] [@server] name type
•	main option is –x
•	server is the name server to query
•	by default, use first server in /etc/resolv.conf
•	name is what you want to look up
•	type can be: any, a, mx, axfr, soa, etc.
•	default is to get A record(s)
dig: axfr (Zone Transfer)
•	dig can request a complete zone transfer:
dig @ictlab tyict.vtc.edu.hk axfr
•	result can be copied and pasted as a master file in a DNS server
nslookup: an interactive program
$ nslookup
Note:  nslookup is deprecated and may be removed from future releases.  Consider using the `dig' or `host' programs instead.  Run nslookup with the `-sil[ent]' option to prevent this message from appearing.
> sysadmin.no-ip.com
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   sysadmin.no-ip.com
Address: 202.69.77.139
nslookup: reverse lookups
> 202.69.77.139
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
139.77.69.202.in-addr.arpa      name = 077-139.onebb.com.

Authoritative answers can be found from:
77.69.202.in-addr.arpa  nameserver = ns1.onebb.com.
77.69.202.in-addr.arpa  nameserver = ns2.onebb.com.
ns1.onebb.com   internet address = 202.180.160.1
ns2.onebb.com   internet address = 202.180.161.1
>
Email: testing with telnet
•	Email protocols SMTP, POP3 are text
•	telnet a good tool to test them
•	syntax:
	telnet server portnumber
•	SMTP: port 25
•	POP3: port 110

Test the VTC mail server:
$ telnet smtp.vtc.edu.hk 25
Trying 192.168.79.191...
Connected to smtp.vtc.edu.hk (192.168.79.191).
Escape character is '^]'.
220 pandora.vtc.edu.hk ESMTP Mirapoint 3.2.2-GA; Tue, 25 Feb 2003 11:15:30 +0800 (HKT)
helo nickpc.tyict.vtc.edu.hk
250 pandora.vtc.edu.hk Hello [172.19.32.30], pleased to meet you
mail from:<nicku@vtc.edu.hk>
250 <nicku@vtc.edu.hk>... Sender ok
rcpt to:<nicku@vtc.edu.hk>
250 <nicku@vtc.edu.hk>... Recipient ok
data
354 Enter mail, end with "." on a line by itself
My message body.
.
250 AFF21826 Message accepted for delivery
quit
221 pandora.vtc.edu.hk closing connection
Connection closed by foreign host.
SMTP commands for sending mail
•	helo — identify your computer
•	mail from — specify sender
•	rcpt to — specify receiver
•	data — indicates start of message body
•	quit — terminate session
•	Use names, not IP addresses, to specify destination
Testing the VTC pop3 server 1
$ telnet pop.vtc.edu.hk 110
Trying 192.168.79.12...
Connected to pop.vtc.edu.hk (192.168.79.12).
Escape character is '^]'.
+OK carme.vtc.edu.hk POP3 service (iPlanet Messaging Server 5.2 Patch 1 (built Aug 19 2002))
user nicku
+OK Name is a valid mailbox
pass password
+OK Maildrop ready
stat
+OK 1 673

Testing the pop3 server		2
retr 1
+OK 673 octets
Return-path: <nicku@vtc.edu.hk>
Received: from pandora.vtc.edu.hk (pandora.vtc.edu.hk [192.168.79.191])
 by carme.vtc.edu.hk (iPlanet Messaging Server 5.2 Patch 1 (built Aug 19 2002))
 with ESMTP id <0HAU00I35H3HGL@carme.vtc.edu.hk> for nicku@ims-ms-daemon
 (ORCPT nicku@vtc.edu.hk); Tue, 25 Feb 2003 11:16:29 +0800 (CST)
Received: from nickpc.tyict.vtc.edu.hk ([172.19.32.30])
        by pandora.vtc.edu.hk (Mirapoint Messaging Server MOS 3.2.2-GA)
        with SMTP id AFF21826; Tue, 25 Feb 2003 11:16:01 +0800 (HKT)
Date: Tue, 25 Feb 2003 11:15:30 +0800 (HKT)
From: Nick Urbanik <nicku@vtc.edu.hk>
Message-id: <200302250316.AFF21826@pandora.vtc.edu.hk>

My message body.
.
dele 1
+OK message deleted
quit
+OK
Connection closed by foreign host.

pop3 commands: retrieving mail
•	See RFC 1939 for easy-to-read details
•	First, must authenticate:
•	user username
•	pass password
•	stat — shows number of messages and total size in bytes
•	list — list all the message numbers and size in bytes of each message
•	retr messagenum — retrieve the message with number messagenum
•	dele messagenum — delete the message with message number messagenum
•	quit
telnet: Testing Other Applications
•	Many network protocols are text.  telnet can be helpful in checking:
•	IMAP servers:
•	telnet hostname 143
•	Web servers:
•	telnet hostname 80
•	Ftp servers:
•	telnet hostname 21
•	Even ssh (can check version, if responding):
•	telnet hostname 22
Conclusion
•	Check the simple things first
•	Document what you do
•	Become familiar with common tools
•	Use the tools to become familiar with your network before troubles strike
•	Know what is “normal”
•	Get permission from the boss before using packet sniffing and port scanners