Network Troubleshooting Troubleshooting Tools Nick Urbanik Copyright Conditions: GNU FDL (see http://www.gnu.org/licenses/fdl.html) References . . . . . . . . . . . . . . . . . . . Introduction Focus: Basics and Standard Tools . Troubleshooting. . . . . . . . . . . . . . Troubleshooting: Learn as you go . Dcoumentation Documentation . . . . . . . . . . . . . . Documentation Tools . . . . . . . . . . General Troubleshooting Problem Solving . . . . . . . . . . . . . Identify the Problem . . . . . . . . . . Gather the Facts . . . . . . . . . . . . . Consider Possibilities . . . . . . . . . . Action Plan . . . . . . . . . . . . . . . . Implement Action Plan . . . . . . . . Observe Results. . . . . . . . . . . . . . document . . . . . . . . . . . . . . . . . . modify action plan. . . . . . . . . . . . TCP/IP OSI—TCP/IP . . . . . . . . . . . . . . . IP Header—Layer 3 . . . . . . . . . . . IP Header . . . . . . . . . . . . . . . . . . IP Header (continued) . . . . . . . . . TCP Header—Layer 4 . . . . . . . . . TCP Header . . . . . . . . . . . . . . . . TCP Header (continued) . . . . . . . UDP Header—Layer 4 . . . . . . . . . Troubleshooting TCP/IP Troubleshooting TCP/IP . . . . . . . Troubleshooting TCP/IP . . . . . . . Host Configuration Host Configuration . . . . . . . . . . . Host Configuration . . . . . . . . . . . Boot Linux . . . . . . . . . . . . . . . . . Boot Windows. . . . . . . . . . . . . . . Determine Addresses . . . . . . . . . . MAC ↔ IP mapping — 1. . . . . . . . . . . . . . . . . . . . . . . . . . slide 2 . . . . . . . . . . . . . . . . . . . slide 3 . . . . . . . . . . . . . . . . . . . slide 4 . . . . . . . . . . . . . . . . . . . slide 5 . . . . . . . . . . . . . . . . . . . slide 6 . . . . . . . . . . . . . . . . . . . slide 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide 9 slide 10 slide 11 slide 12 slide 13 slide 14 slide 15 slide 16 slide 17 slide slide slide slide slide slide slide slide 19 20 21 22 23 24 25 26 . . . . . . . . . . . . . . . . . . slide 28 . . . . . . . . . . . . . . . . . . slide 29 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide slide slide slide slide slide 30 31 32 33 34 35 MAC ↔ IP mapping — 2. . . . . . . Routing Table . . . . . . . . . . . . . . . Access Controls . . . . . . . . . . . . . . DNS resolver . . . . . . . . . . . . . . . . Checking services provided . . . . . . Server Running? . . . . . . . . . . . . . top. . . . . . . . . . . . . . . . . . . . . . . netstat -tua . . . . . . . . . . . . . . . lsof: List Open Files . . . . . . . . . ifconfig . . . . . . . . . . . . . . . . . . route . . . . . . . . . . . . . . . . . . . . . Cables Connectivity Testing: Cabling . . . Ping Software tools: ping . . . . . . . . . . Good ping, Bad ping?. . . . . . . . . How to Use ping? . . . . . . . . . . . . fping: flood ping . . . . . . . . . . . . hping2 . . . . . . . . . . . . . . . . . . . . arping: uses ARP requests . . . . . traceroute Path Discovery: traceroute . . . . traceroute: How it Works . . . . . traceroute Limitations . . . . . . . . Measurements Performance Measurements: delay. bandwidth and throughput. . . . . . Quality of a Link . . . . . . . . . . . . . Throughput: ping . . . . . . . . . . . . Througput: ping One . . . . . . . . . Throughput ping 2 remote . . . . . . Throughput ping 2 remote . . . . . . Throughput ping 2 remote . . . . . . Throughput ping 2 remote . . . . . . Throughput ping 2 remote . . . . . . Limitations of ping . . . . . . . . . . . Path Performance: Other tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide slide slide slide slide slide slide slide slide slide slide 36 37 38 39 40 41 42 43 44 45 46 . . . . . . . . . . . . . . . . . . slide 47 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide slide slide slide slide slide 49 50 51 52 53 54 . . . . . . . . . . . . . . . . . . slide 55 . . . . . . . . . . . . . . . . . . slide 56 . . . . . . . . . . . . . . . . . . slide 57 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide slide slide slide slide slide slide slide slide slide slide slide 58 59 60 61 62 63 64 65 66 67 68 69 pathchar . . . . . . . . . . . . . . . . . . . Measuring Throughput. . . . . . . . . . Measuring Throughput with ttcp . . iproute The ip program, iproute . . . . . . . . iproute and iptables . . . . . . . . . . Traffic Measurements: netstat -i . Measuring Traffic: netstat -i . . . . Traffic measurements: ifconfig, ip Getting more info using ip . . . . . . . Guide to ip 1 . . . . . . . . . . . . . . . . Guide to ip 2 . . . . . . . . . . . . . . . . Packet Capture What is Packet Capture? . . . . . . . . When Packet Capture? . . . . . . . . . Warning: Don’t Get Sacked! . . . . . . tcpdump . . . . . . . . . . . . . . . . . . . . How to use tcpdump. . . . . . . . . . . . tcpdump: some options. . . . . . . . . . tcpdump Filters: host and port . . . . tcpdump filters: networks . . . . . . . . tcpdump filters: protocol . . . . . . . . tcpdump filters: combining . . . . . . . Writing data to a file . . . . . . . . . . . Reading a Dumped File . . . . . . . . . HTTP . . . . . . . . . . . . . . . . . . . . . tcpdump: When reading TCP . . . . . Window . . . . . . . . . . . . . . . . . . . . Ethereal Ethereal . . . . . . . . . . . . . . . . . . . . Ethereal Screenshot . . . . . . . . . . . . Ethereal Screenshot . . . . . . . . . . . . You can expand any protocol:. . . . . Ethereal 2 . . . . . . . . . . . . . . . . . . . Display Filters. . . . . . . . . . . . . . . . Tools → Follow TCP Stream . . . . . Ntop. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide 70 . . . . . . . . . . . . . . . . . slide 71 . . . . . . . . . . . . . . . . . slide 72 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide 74 75 76 77 78 79 80 81 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 . slide 99 slide 100 slide 101 slide 102 slide 103 slide 104 slide 105 slide 106 Ntop. . . . . . . . . . . . . . . . . . . . . . . Switched Networks Port Monitoring . . . . . . . . . . . . . . How monitor one machine?. . . . . . . Are switched networks secure? . . . . Port Scanning What is a port scanner? . . . . . . . . . How does nmap identify OS? . . . . . . Running nmap: Use xnmap . . . . . . . Uses of nmap . . . . . . . . . . . . . . . . . DNS troubleshooting DNS troubleshooting . . . . . . . . . . . DNS: dig . . . . . . . . . . . . . . . . . . . dig: Checking forward DNS lookup. dig: reverse lookup 1 . . . . . . . . . . . dig syntax . . . . . . . . . . . . . . . . . . dig: axfr (Zone Transfer) . . . . . . . . nslookup: an interactive program . . nslookup: reverse lookups . . . . . . . telnet Email: testing with telnet . . . . . . Test the VTC mail server: . . . . . . . SMTP commands for sending mail . Testing the VTC pop3 server 1 . . . . Testing the pop3 server 2 . . . . . . . . pop3 commands: retrieving mail . . . telnet: Testing Other Applications Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide 107 . . . . . . . . . . . . . . . . slide 109 . . . . . . . . . . . . . . . . slide 110 . . . . . . . . . . . . . . . . slide 111 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide slide 113 114 115 116 118 119 120 121 122 123 124 125 127 128 129 130 131 132 133 134 References Joseph D. Sloan, Network Troubleshooting Tools, O’Reilly, August 2001, ISBN: 059600186-X, TK5105.5 .S557 2001 Cisco’s Troubleshooting Overview at:         http://www.cisco.com/univercd/cc/td/doc/cisintwk/itg v1/tr1901.htm Craig Hunt, TCP/IP Network Administration, Third Edition O’Reilly, April 2002, ISBN: 059600297-1. Chapter 13 covers troubleshooting TCP/IP. Martin A. Brown, Guide to IP Layer Network Administration with Linux, http://linux-ip.net/ Noah Davids, Don’t forget to check your ARP cache, http://members.cox.net/∼ndav1/self published/The ARP cache.doc SNM — ver. 1.4 Network Troubleshooting — slide 2 Focus: Basics and Standard Tools Solving network problems depends a lot on your understanding Simple tools can tell you what you need to know Example: ping is incredibly useful! Network Troubleshooting — slide 3 SNM — ver. 1.4         Troubleshooting Avoid it by: ◦ redundancy ◦ documentation ◦ training Try quick fixes first ◦ simple problems often have big effects: ◦ is the power on? ◦ is the network cable plugged into the right socket? Is LED flashing? ◦ has anything changed recently? Change only one thing at a time ◦ test thoroughly after the change Be familiar with the system ◦ maintain documentation Be familiar with your tools ◦ before trouble strikes SNM — ver. 1.4 Network Troubleshooting — slide 4 Troubleshooting: Learn as you go Study and be familiar with the normal behaviour of your network Monitoring tools can tell you when things are wrong ◦ if you know what things look like when they are right Using tools such as Ethereal can help you understand ◦ your network, and ◦ tcp/ip— better SNM — ver. 1.4 Network Troubleshooting — slide 5     Documentation Maintain an inventory of equipment and software ◦ a list mapping mac addresses to machines can be very helpful Maintain a change log for each major system, recording: ◦ each significant change ◦ each problem with the system ◦ each entry dated, with name of person who made the entry Two categories of documentation: ◦ Configuration information – describes the system – use system tools to obtain a snapshot, e.g., sysreport in Red Hat Linux ◦ Procedural information – How to do things – use tools that automatically document what you are doing, e.g., script SNM — ver. 1.4 Network Troubleshooting — slide 6                   Documentation Tools Use script: $ ∼ General Troubleshooting   ¨¤ ! # $" ¨ © ¦    ¨§ ¨§ ¡ © ¡ ¢ £ ¤¥ ¦ ©      ©& ¦  ¨¤   ◦ starts a new shell ◦ all you type, all output goes into the file ◦ Add comments with # I tried this... Use tee: $    £¥ Use sudo: all commands are recorded in /var/log/secure Use plod from http://bullwinkle.deer-run.com/∼hal/plod/ ◦ lets you record a worksheet easily ◦ Perl, so fine on any platform       ¦ %  SNM — ver. 1.4 Network Troubleshooting — slide 7 Problem Solving Solve Problem Define problem Gather facts Consider possibilities based on facts Create action plan Implement action plan Observe results NO Symptoms stop? YES Document process Problem solved SNM — ver. 1.4 Network Troubleshooting — slide 9 Identify the Problem Problem is reported by a person or by software Often involves communicating with others ◦ Somewhat like gathering requirements in software design ◦ An iterative process Possible questions to ask: ◦ What does not work? ◦ What does work? ◦ Are the things that do and do not work related? ◦ Has the thing that does not work ever worked? ◦ When the problem was first noticed? ◦ What has changed since the last time it did work? ◦ Did anything unusual happen since the last time it worked? ◦ When exactly does the problem occur? ◦ Can the problem be reproduced and if so, how can it be reproduced? SNM — ver. 1.4 Network Troubleshooting — slide 10 Gather the Facts You probably need to find out more about the problem from other sources, including ◦ Asking other people: affected users, network administrators, managers, and other key people ◦ Network management http://nagios.org/ systems, such as Nagios         ◦ Tools such as Ethereal, tcpdump, ntop (http://ntop.org/) — see slides starting at 83 ◦ Server log files ◦ Documentation about your servers and network created by local staff ◦ Documentation about software and hardware that are provided by the vendors SNM — ver. 1.4 Network Troubleshooting — slide 11 Consider Possibilities based on Facts Using the information you have gathered, try to eliminate some potential problems from your list. Network Troubleshooting — slide 12 SNM — ver. 1.4   ' Create an Action Plan Start with the most likely ◦ . . . and those that are easiest to test Plan needs to be methodical Plan to change only one thing at a time ◦ You can then understand the cause of the problem ◦ Aim to understand the problem so you can learn from it, solve (or prevent) similar problems in the future Aim higher than just removing the symptoms! Network Troubleshooting — slide 13 SNM — ver. 1.4 Otherwise, Modify Action Plan Go back to the steps in slide 13: ◦ Modify your action plan, selecting the next most likely action from your list ◦ You may have discovered more information in your investigation, so this can help you focus on likely causes. If you have exhausted all the items on your list, and cannot think of what else to do: ◦ Get help from the vendor ◦ Get help from mailing lists ◦ Discuss the problem with your network of colleagues (e.g., the people who are now studying with you, but who move on to work in a similar field!) ◦ You could even track me down and ask me! Quite a few of my ex-students do. Network Troubleshooting — slide 14 SNM — ver. 1.4 Network Troubleshooting — slide 17       Implement Action Plan Perform each step carefully Test to see if symptoms go away SNM — ver. 1.4 Observe Results Gather results as you change each variable Use same techniques you used in slide 11: ◦ Check with the key people ◦ Check with your tools SNM — ver. 1.4 Network Troubleshooting — slide 15       TCP/IP OSI—TCP/IP Layer 7 6 5 4 3 2 1 OSI Application Presentation Session Transport Network Data link Physical TCP/IP HTTP, FTP, SMTP Telnet, SSH, SMTP SNMP, NFS, SMB RPC, DNS, OSPF, BGP         ' Application If Solved: Document Solution Record the problem and its resolution in the documentation you maintain for your company. Ensure others in your team can benefit from the insight you have gained Network Troubleshooting — slide 16 TCP, UDP IP ARP ICMP Transport Internet Network Access SNM — ver. 1.4     SNM — ver. 1.4 Network Troubleshooting — slide 19 IP Header—Layer 3 12 16 20 24 28 31 Bits 0 4 8 IP Header (continued) Flags — a flag is a 3-bit field of which the 2 low-order bits control fragmentation. One bit specifies whether the packet can be fragmented; the second bit specifies whether the packet is the last fragment in a series of fragmented packets. Time-to-Live — this field maintains a counter that gradually decrements down to zero, at which point the datagram is discarded. This prevents packets from looping endlessly. Protocol — protocol indicates which upper-layer protocol receives incoming packets after IP processing is complete. Header Checksum — this field helps ensure IP header integrity. Source Address — this field specifies the sending node. Destination Address — this field specifies the receiving node. Options — this field allows IP to support various options, such as security. Data — this field contains upper-layer information. Network Troubleshooting — slide 22 1 2 Words 3 4 5 Version IHL Type of Service DF MF Total Length Fragmentation Offset Header Checksum Identification Time to Live Protocol Source Address Destination Address Options (0 to 40 bytes) Padding 5−16 Your data starts here SNM — ver. 1.4 Network Troubleshooting — slide 20 IP Header Version — this is a 4-bit IP header length field that indicates the version of IP currently used. The current version of IP is 4 (IPv4) but IPv6 is already being implemented experimentally and will be supported on future versions of the IOS. IP Header Length (IHL) — this indicates the datagram header length in 32-bit words. Type of Service (ToS) — ToS specifies how a particular upper-layer protocol would like the current datagram to be handled. Datagrams can be assigned various levels of importance with this field. Total length — this specifies the length of the entire IP packet, including data and header, in bytes. Identification — this field contains an integer that identifies the current datagram. This field is used to help piece together datagram fragments. Network Troubleshooting — slide 21   SNM — ver. 1.4 SNM — ver. 1.4                         TCP Header—Layer 4 12 16 20 24 28 31 Bits 0 4 8 TCP Header (continued) Window — this field specifies the size of the sender’s receive window (that is, buffer space available for incoming data). Checksum — this field indicates whether the header was damaged in transit. Urgent pointer — this field points to the first urgent data byte in the packet. Options — this field specifies various TCP options. Data — this field contains upper-layer information. Network Troubleshooting — slide 25 1 2 Words 3 header 4 length 5 Source Port Sequence Number Destination Port Acknowledgement Number URG ACK PSH RST Reserved Checksum SYN FIN Window Urgent Pointer Padding Options (0 to 40 bytes) 5−15 Your data starts here SNM — ver. 1.4 Network Troubleshooting — slide 23 SNM — ver. 1.4 TCP Header Source port and destination port — these fields identify the points at which upper-layer source and destination processes receive TCP services. Sequence number — this field usually specifies the number assigned to the first byte of data in the current message. Under certain circumstances, it can also be used to identify an initial sequence number to be used in the upcoming transmission. Acknowledgment number — this field contains the sequence number of the next byte of data the sender of the packet expects to receive. Data offset — this field indicates the number of 32-bit words in the TCP header. Reserved — this field is reserved for future use. Flags — this field carries a variety of control information. Network Troubleshooting — slide 24 UDP Header—Layer 4 16 31 Destination Port Checksum Your data starts here Bits 0                         Source Port Length Source and Destination Port fields serve the same functions as they do in the TCP header. Length field specifies the length of the UDP header and data Checksum field allows packet integrity checking. It is optional. Network Troubleshooting — slide 26 SNM — ver. 1.4 SNM — ver. 1.4     Troubleshooting TCP/IP Troubleshooting TCP/IP Local host YES configuration okay? NO can YES you ping the local router? NO can you reach all routers in path to remote Host Network Configuration tools — information about processes — dynamic information about processes — show connections and services, routing YES host? remote host configuration okay? NO ¨ ¡ Fix any problems with address, subnet mask, default gateway, DNS entry Fix any router configuration problems, and any LAN or switch problems Fix any router configuration problems, LAN or WAN problems Fix remote host configuration £ ©& — show, change, set network configuration — shows MAC addresses — portscanner: shows open ports, identifies os Network Troubleshooting — slide 30 SNM — ver. 1.4 Network Troubleshooting — slide 28 Troubleshooting TCP/IP Step 1 First, determine whether your local host is properly configured (for instance, correct subnet mask and default gateway configuration). Step 2 Next, use the ping or traceroute commands to determine whether the routers through which you must communicate can respond. Start with the most local router and progressively ping outwards through the Internet or use traceroute. Step 3 If you cannot get through a particular node, examine the node configuration and use the various show commands to determine the state of the router (these include show ip route, show ip arp, show running-configuration, and so on.) Step 4 If you can get to all the routers in the path, check the host configuration at the remote host (or get someone’s help to do so), and check its configuration. SNM — ver. 1.4 Network Troubleshooting — slide 29 () SNM — ver. 1.4 Checking (and Setting) Host Configuration Solving Boot problems: 32, 33 ¥  £¥ ¤¥  ¦ ( ¢ © ¤ NO © ¤ ¦ ( ¡ ¦ ©¥ ¦ ¥ ¡ — list open files — shows and changes network interfaces ¦ — shows, changes routing table '   Determine ip address, netmask, broadcast address: 34 Deterine correct mac ↔ ip address mapping: 35, 36 '   '   Examine routing table: 37 Examine access controls: 38 Examine web proxy settings: check web browser Examine dns resolver settings: 39 Determine services provided: 40, 41       ' '   '   Determine cpu, memory load conditions (is the server overloaded?) 42   SNM — ver. 1.4 ' ' ' Network Troubleshooting — slide 31 ' ' Boot problems: Linux Use grub to interactively boot the computer (see my extensive grub handout: http://ictlab.tyict.vtc.edu.hk/ossi/lab/grub/grub.pdf) Verify that /etc/fstab mounts the correct filesystems Use a rescue disk such as Knoppix or the Red Hat installation cdrom. This gives you full access to the system and repairing boot problems is pretty straightforward. Network Troubleshooting — slide 32 Determine Addresses Linux: On Linux, these commands all show the ip address, mac address, netmask and broadcast address for all (or the specified) interface. The commands ip and ifconfig are in the directory /sbin; netstat is in /bin. $ $ $ $ $       £  £  ©1 0 ¤ ¦ ¦ ( ¡  ¦ (¤ £& ( ( ¤  ©1 0  ¨§ ¨ ( ¢ ©1 0 £ 3 (¦ ¡ ¤¥   ¢ ©1 0 ¤ £ (¦ ¡ ¤¥   ¢ Network Troubleshooting — slide 34  ¤¥ ¤   ¤ ( ( ¢ © ¤ SNM — ver. 1.4 Windows: C:\> Cisco IOS: these are both privileged commands, as shown by the prompt: Use the installation Windows cd to enter the (extremely limited) system repair mode. I believe this is called the recovery console. Use the Linux floppy bootdisk ∼pnordahl/ntpasswd/ http://home.eunet.no/ to the Administrator password Use the bootable http://www.nu2.nu/pebuilder/; Windows at replace cdrom:   SNM — ver. 1.4     ◦ Gives full access to the NTFS file system. ◦ Not as good with Windows as Knoppix is with Linux, but better than another reinstall. ◦ takes some time to build. ◦ Henry Leung (in A204d) has built some. SNM — ver. 1.4 Network Troubleshooting — slide 33    £ Boot problems: Windows Router# Router# Router# ¡ ( ¢ ¤¥ © ¤ ¦ ¢ © 02 ¤ ¦ © ¤  ¡ 02 ¤¥ ¤ MAC ↔ IP mapping — 1 Linux: $ $ MAC ↔ IP mapping — 2 Windows: C:\>  £¥    IP address £¥  £¥ 4   £¥   0 ©1 0 ¤¥ ( ¤ ¡ You may wish to clear the arp cache on a Windows machine with: The lifetime of the arp cache entries is settable in /proc/sys/ net/ipv4/neigh/ interface /gc stale time and is normally 60 seconds. Cisco IOS: Router# C:\> or clear the entire arp cache with: ©1 0 C:\> ¡ since the Windows arp cache lives 10 minutes by default, a rather Note that this document: http://members.cox.net/∼ndav1/self published/The ARP cache.doc(too?) long time. has a good discussion on troubleshooting arp. It can be changed by two registry entries under The online book at http://linux-ip.net/ has an excellent chapter on HKEY LOCAL MACHINE\SYSTEM\CurrentControlSet\Services\ arp. Tcpip\Parameters. arp is probably the most dangerously exposed protocol in a lan, and SNM — ver. 1.4 Network Troubleshooting — slide 36 can easily be subverted by tools such as dsniff and Ettercap. Hardcode important arp entries to avoid attack. SNM — ver. 1.4 Network Troubleshooting — slide 35  ¤¥ £¥    Routing Table Linux: The commands ip and route are in netstat is in /bin. $ $ $ /sbin, the command DNS resolver settings Linux: The configuration for the resolver is /etc/resolv.conf. This determines what name servers the system will ask. It also tells what domain will be appended to a hostname. The /etc/hosts file is usually the first way hostname ↔ ip address mappings are made, but this can be changed in /etc/ nsswitch.conf. To ask the operating system for what it can see there, do: $ £ £© ¦ ( ¡  ( £ ¦ ¦  &  ¦ ( ©& Windows: C:\> C:\>  ¦ ¤¥ ¦ ¦ ( ¡ ¦ (£ £¤ ¦ (¦ £© & ¥ Cisco IOS: Router# SNM — ver. 1.4 Linux provides three tools for troubleshooting dns and dns servers: dig, host and nslookup. Windows: See the output of ©1 0  ¡ £ ©&  ¦ ¤¥ Network Troubleshooting — slide 37 C:\> Access Controls Access controls can block access mysteriously unless you think to check for it. for the names of the dns server the resolver will use. Recent versions of Windows provide the program nslookup to help with dns troubleshooting. See Slides starting at 118 for details of dig and nslookup. SNM — ver. 1.4   '  ¨§ ¨ ( ¢ ¤¥ © ¤ ©0 ¡ ¦ ¡ (¦    ¦ Network Troubleshooting — slide 39 Linux: There are two main things to check. The iptables command is in the /sbin directory. $ 3¨ ¦ ¤¥    ¡ 5 ( Note that Linux and many other posix systems implement the tcpwrappers access control in /etc/hosts.allow and /etc/hosts. deny. See man hosts.allow and man hosts.deny. Cisco IOS: Router# SNM — ver. 1.4 ©1 0 ¨ ¡ ¡¦  ¤¥ ¢ ¢  ¡ Network Troubleshooting — slide 38 ¡ ¤ Checking services provided Linux: There are four main ways to check: Verify the processes with ps (see 41) Verify the services that are configured to start when the system boots: $ Using   ' to see Resource Hogs @A The program top shows: ◦ load average (the average number of processes that are ready to run, but for which no cpu is available) – a load average of 4 or more is “quite high” ◦ processes that use the most resources     06 ( ¨ ¢ ¢ © ¡ ¦ ¤ ¤  % £ ¥   © ( Check that the service is listening on the network interface: $ SNM — ver. 1.4 7 Network Troubleshooting — slide 42   ¦ ( ¡   ¦ ¦ ¡   ¦ ' ¦& ¦ will show all network connections to this machine. The lsof program can be helpful in diagnosing problems with network services. See 44. Windows: Check network connections with C:\>   SNM — ver. 1.4 ( ¦ Network Troubleshooting — slide 40 Using   to See If Server Running BC @8   78 Is the network service running on the server? Is the web server running? $ : See Network Connections @ @D           SNM — ver. 1.4 netstat -tua shows all network connections, including those listening sudo netstat -tuap shows all network connections, including those listening, and the processes responsible netstat -tul shows all network listeners netstat -t shows only tcp connections that are established netstat -i is like ifconfig, shows info and stats about each interface netstat -nr shows the routing table, like route -n Windows provides netstat also. Network Troubleshooting — slide 43   % ¦0 ¥ ¡ & 9  £ ¥ Is the DHCP server running? $   % Is the directory server running? $   % ¡¨ ¥ ¡ & 9  £ ¥ Windows: use the task manager Network Troubleshooting — slide 41   ¥  ¢¥ 0  ¥ ¡ & 9  £ ¥  ¦¥  SNM — ver. 1.4   E D @F : List Open Files 8 HG A An amazingly useful tool Available for almost any Unix system shows output to Internet and X.25 files, but won’t show connections that have terminated                 will show only connections    ¨  © ¡ ¤ P¤ ¤  ¨ © ¨ #3 ¦Q to that machine Can monitor progress of an FTP transfer, many, many other applications See manpage, faq and quick start guide. Apparently, no equivalent tool available on Windows. Network Troubleshooting — slide 44 # ¢¦ R # ¢¦  # & ¤  ¢¦ ¡  06     SNM — ver. 1.4   SNM — ver. 1.4 C@ cA route -n — print routing table route add 127.0.0.1 — add a route to localhost; should have been done automatically when created device with ifconfig route add -net 172.19.233.0 — add a route to the eth0 configured on previous slide should have been done automatically by ifconfig route add 172.19.64.0 gw 172.19.233.254 — add a static route to network 172.19.64.0 through router 172.19.233.254 route add default gw 172.19.233.253 — add a default route to 172.19.233.253 through eth0 Network Troubleshooting — slide 46 I Connectivity Testing: Cabling Label cables clearly at each end Cable testers ◦ ensure wired correctly, check: ◦ attenuation ◦ length — is it too long? – 100BaseT: less than 100m Is the activity light on the interface blinking? Network Troubleshooting — slide 47 SNM — ver. 1.4 S I UVS T HI A B — show stats on network interface eth0 — configure the loopback inter- ¤ (   ¤ ¦ ¢ © 02 ¤ face, start it up ¤ — configure eth0 with IP address 172.19.233.5/24 — show all configured network interfaces — show all interfaces, including those not configured Network Troubleshooting — slide 45 # XY # W` X # aa b ( )¦  ¡6 X # bb X # bb X # bb 2 © ( 02   ¦ ¡& ¢ © W ¤ WX #Y #2 #2 W © ( ¡& ¢ ©   © ¨ ¤ ¤ ¤   ( ¢ © ¤ ( ¢ © ¤   yet. SNM — ver. 1.4         F Ping Software tools:   Universal If hostname, includes a rough check of dns How to Use   ? Ensure local host networking is enabled first: ping localhost, local ip address ping a known host on local network ping local and remote interfaces on router ping by IP as well as by hostname if hostname ping fails ◦ confirm DNS with (or ) — see slide 118 7 BU S Most useful check of connectivity   ¥   ( ¤       7 BU S ¤ (¡ ¨ Sends an icmp (Internet Control Message Protocol) ECHO REQUEST Waits for an icmp ECHO REPLY Most Most s can display round trip time s can allow setting size of packet Network Troubleshooting — slide 49   Ping from more than one host Network Troubleshooting — slide 51 SNM — ver. 1.4   ¥   ¥   Can use to make a crude measurement of throughput—see 61 ( ¤ ( ¤ SNM — ver. 1.4 What     Result is Good, Bad? S '   : flood ping 7 BU A steady stream of consistent replies indicates probably okay Usually first reply takes longer due to arp lookups at each router ◦ After that, arp results are cached icmp error messages can help understand results: ◦ Destination Network Unreachable indicates the host doing ping cannot reach the network ◦ Destination Host Unreachable may come from routers further away 7 I             SNM — ver. 1.4 Network Troubleshooting — slide 50 SNM — ver. 1.4   BU S Designed to test a large number of hosts more efficient than ping Used extensively by monitoring software such as mon: http://www.kernel.org/software/mon/, nagios: http://www.nagios.org/ take care not to flood too much! rpms are available; I built one (a long time ago) and put it on ictlab under ∼ftp/pub/redhat/contrib Network Troubleshooting — slide 52 ©6 © & ¥ ' : ping anything with anything : How it Works d S e @ able to send custom TCP/IP packets and display target replies like ping program does with icmp replies. Can install with $ As ip packets pass through each router, ttl in ip header is decremented Packet is discarded when TTL decrements to 0 ROUTER sends icmp TIME EXCEEDED message back to traceroute host When udp packet reaches destination, gets PORT UNREACHABLE, since uses an unused high udp port icmp       ¤  ¦¨ ¨ Q ( X Q & ) (¡ 0¥ ¤ on Fedora Core 1. See http://www.hping.org/. Network Troubleshooting — slide 53 SNM — ver. 1.4 SNM — ver. 1.4         C@ cD T C cA F 7 BU Network Troubleshooting — slide 56   D c         7 : uses ARP requests BU S Limited to local network Can work with mac or ip addresses use to probe for arp entries in router (very useful!) packet filtering ◦ can block icmp pings, but ◦ won’t block arp requests SNM — ver. 1.4 Network Troubleshooting — slide 54 Limitations @ Path Discovery:   @ C@ cD T C cA F Each router has a number of ip addresses but traceroute only shows the one it used get different addresses when run traceroute from other end sometimes route is asymmetric router may be configured to not send icmp TIME EXCEEDED messages ◦ get stars: * instead of round-trip time in output Sends udp packets ◦ (Microsoft tracert sends icmp packets) increments Time to Live (ttl) in ip packet header Sends three packets at each ttl records round trip time for each increases ttl until enough to reach destination Network Troubleshooting — slide 55                 C@ cD T C cA F ¦ £ ¢  £©& SNM — ver. 1.4   SNM — ver. 1.4 Network Troubleshooting — slide 57  ¦ Performance Measurements: delay Three sources of delay: transmission delay — time to put signal onto cable or media ◦ depends on transmission rate and size of frame propagation delay — time for signal to travel across the media ◦ determined by type of media and distance queuing delay — time spent waiting for retransmission in a router Network Troubleshooting — slide 58 SNM — ver. 1.4 Quality of a Link Other measurements needed ◦ i.e., for quality of service for multimedia SNM — ver. 1.4 Network Troubleshooting — slide 60   Is Bandwidth == Throughput? bandwidth — the difference between the upper frequency and the lower frequency that a channel can carry ◦ measured in Hertz throughput — amount of data that can be sent over link in given time ◦ is not the same as bandwidth, which really has no direct meaning with digital information bandwidth is related to throughput by the Shannon-Hartley Theorem; throughput ∝ bandwidth if signal to noise ratio is fixed: Cmax = B log2 1 + where Cmax B S N SNM — ver. 1.4           7             BU S S N bits/sec = maximum channel capacity, = bandwidth in Hz, = signal power in W, = noise power in W. Network Troubleshooting — slide 59 Roughly Estimating Throughput Example: measuring throughput between this machine and one remote machine. ping with packet size = 100 bytes, round-trip time = 30ms ping with packet size = 1100 bytes, round-trip time = 60ms So takes 30ms extra (15ms one way) to send additional 1000 bytes, or 8000 bits Throughput is roughly 8000 bits per 15ms, or about 530,000 bits per second A very crude measurement: ◦ no account for other traffic, treats all links on path, there and back, as one. ◦ Routers sometimes send packets onwards with much higher priority than with which they answer pings. See slide 68.     SNM — ver. 1.4 Network Troubleshooting — slide 61 ' Throughput:   One Remote Host Throughput:   Two Remote Hosts S 7 BU This can be expressed as a simple formula: T P = 16 × Pl Ps tl ts Here we have: T P = 16 × 1100 − 100 (60 − 30) × 10−3 1000 = 16 × 30 × 10−3 16 = × 106 30 ≈ 530,000 bps Network Troubleshooting — slide 62 Pl − P s bits per second, where tl − t s = size of large packet = size of small packet = round-trip time for large packet = round-trip time for small packet Measure throughput between two remote hosts: may use tools like ping ping two locations with two packet sizes (4 pings altogether, minimum) ◦ Many ping programs calculate average ping time: better to make a number of pings, use the average ping time. ◦ First ping time may be longer due to the time to get an answer to the arp request ◦ May be better to ping once, then start pinging again, and use the average ping time. Example: RTT 100 bytes 1.380 ms 4.985 ms 8.621 ms RTT 1100 bytes 5.805 ms 12.823 ms 26.713 ms Network Troubleshooting — slide 63 Address 205.153.61.1 205.153.60.2 165.166.36.17 SNM — ver. 1.4 SNM — ver. 1.4     7 UB S Throughput: Address 205.153.61.1 205.153.60.2 165.166.36.17 Two Remote Hosts — 2 RTT 1100 bytes 5.805 ms 12.823 ms 26.713 ms Throughput: T P = 16 × where: Pl Ps tnl tns tf l tf s Two Remote Hosts — 3 bits per second S 7 BU RTT 100 bytes 1.380 ms 4.985 ms 8.621 ms Pl − P s tf l − tf s − tnl + tns Time difference / 2 (round trip time (RTT) → one way) Divide by size difference in bits: 8000 = 8 × (1100 − 100) Multiply by 1000 (ms → seconds) Mbs = bps/106 Far link Time difference Est. Throughput 4.69 Mbps 1.56 Mbps   = l arge packet size, bytes = small packet size, bytes = ping time for l arger packet to the near link, seconds = ping time for smaller packet to the near link, seconds = ping time for l arger packet to the f ar link, seconds = ping time for smaller packet to the f ar link, seconds Network Troubleshooting — slide 65 Near link       SNM — ver. 1.4 205.153.61.1 205.153.60.2 3.413 ms 205.153.60.2 165.166.36.17 10.254 ms Throughput: 7 7 UB S UB S Two Remote Hosts — 4 3.413 ms = (12.823 − 4.985) − (5.805 − 1.380) ms 10.254 ms = (26.713 − 8.621) − (12.823 − 4.985) ms SNM — ver. 1.4 Network Troubleshooting — slide 64 Pl = l arge packet size, bytes = 1100 bytes Ps = small packet size, bytes = 100 bytes tnl = ping time for l arger packet to the near link, seconds = 5.805 × 10−3 seconds tns = ping time for smaller packet to the near link, seconds = 1.380 × 10−3 seconds tf l = ping time for l arger packet to the f ar link, seconds = 12.823 × 10−3 seconds tf s = ping time for smaller packet to the f ar link, seconds = 4.985 × 10−3 seconds T P = 16 × Pl − P s bits per second tf l − tf s − tnl + tns 1100 − 100 = 16 × (12.823 − 4.985 − 5.805 + 1.380) × 10−3 1000 = 16 × 3.413 × 10−3 ≈ 4,687,958 bits per second ≈ 4.69 Megabits per second SNM — ver. 1.4 Network Troubleshooting — slide 66 Throughput: Two Remote Hosts — 5 Limitations of measuring with   S 7 BU Pl = l arge packet size, bytes = 1100 bytes Ps = small packet size, bytes = 100 bytes tnl = ping time for l arger packet to the near link, seconds = 12.823 × 10−3 seconds tns = ping time for smaller packet to the near link, seconds = 4.985 × 10−3 seconds tf l = ping time for l arger packet to the f ar link, seconds = 26.713 × 10−3 seconds tf s = ping time for smaller packet to the f ar link, seconds = 8.621 × 10−3 seconds T P = 16 × Pl − P s bits per second tf l − tf s − tnl + tns 1100 − 100 = 16 × (26.713 − 8.621 − 12.823 + 4.985) × 10−3 1000 = 16 × 10.254 × 10−3 ≈ 1,560,366 ≈ 1.56 Megabits per second Most modern routers give high priority to routing ◦ Expecially switching routers, such as the Cisco 6509 in our Campus ◦ Many give much lower priority to answering pings ◦ The difference can be so great that the ping reply sometimes comes sooner from a more distant router, which according to our formula, indicates a negative throughput! ◦ Do not blindly apply this formula! bits per second Measurements may not match the kind of traffic created by the application you support The big advantages of these icmp measurements are: ◦ you do not need access to the machines, and ◦ you do not need to install any special software on them. SNM — ver. 1.4 Network Troubleshooting — slide 67 SNM — ver. 1.4     Network Troubleshooting — slide 68 Path Performance: Other tools Could use a tool like pathchar, bing, clink, pchar, or tmetric that performs this calculation for you Use http://www.google.com/ to locate these tools pathchar is only available in binary form Others in source form, need compile with commands something like this: $ $ $          3 ¤ #W )   6 © 6 ¤ #W ¢  ( SNM — ver. 1.4  ¦¨ a ¡& )   (¡ ¨ Network Troubleshooting — slide 69 7 BU S Path measurement with fg p ih s qr vs tu u w fp ys x p vi€ u Measuring Throughput with Use @ Td Dd 7D c @ @T Network Troubleshooting — slide 72 $ pathchar to sina.com.hk (202.85.139.140) can’t find path mtu - using 1500 bytes. doing 32 probes at each of 45 sizes (64 to 1500 by 32) 0 localhost (127.0.0.1) | 106 Mb/s, 293 us (698 us), +q 1.18 ms (15.7 KB) 1 172.19.35.246 (172.19.35.246) | 28 Mb/s, 488 us (2.10 ms) 2 192.168.83.2 (192.168.83.2) 3 *1 448 798 2 | 20 Mb/s, 273 us (3.25 ms) 4 cw7204.vtc.edu.hk (202.40.210.220) | 6.8 Mb/s, 521 us (6.04 ms) 5 210.176.123.37 (210.176.123.37) | 52 Mb/s, 20 us (6.31 ms) 6 210.87.254.61 (210.87.254.61) | 136 Mb/s, 116 us (6.63 ms) 7 g5-0-0.wttbr01.imsbiz.com (210.87.254.129) | 33 Mb/s, 0.94 ms (8.88 ms), +q 1.48 ms (6.10 KB) *6 8 iadvantage3-RGE.hkix.net (202.40.161.172) | 164 Mb/s, 45 us (9.04 ms), +q 1.74 ms (35.6 KB) *6 9 v005-m02.hk01.iadvantage.net (202.85.129.53) | ?? b/s, -66 us (8.88 ms) 10 202.85.129.136 (202.85.129.136) | ?? b/s, 459 us (9.79 ms) 11 202.85.139.11 (202.85.139.11) 11 hops, rtt 6.18 ms (9.79 ms), bottleneck 6.8 Mb/s, pipe 9361 bytes , not affected by disk i/o   Consists of a client and server Need have installed at both ends Part of Red Hat Linux, Cisco ios Example: first, start receiver on ictlab: $ ttcp-r: ttcp-r: ttcp-r: ttcp-r: ttcp-r: ttcp-r:         ƒ„ ‚‚ SNM — ver. 1.4 Network Troubleshooting — slide 70 Second, start transmitter on nickpc: $ ttcp-t: ttcp-t: ttcp-t: ttcp-t: ttcp-t: ttcp-t:   …† buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp socket accept from 172.19.32.30 16777216 bytes in 1.45 real seconds = 11285.88 KB/sec +++ 9704 I/O calls, msec/call = 0.15, calls/sec = 6684.46 0.0user 0.2sys 0:01real 14% 0i+0d 0maxrss 0+2pf 0+0csw …‡ ¢¥ ¦ ¦ ƒ„ ‚‚ …‚ …‡ buflen=8192, nbuf=2048, align=16384/0, port=5001 tcp -> ictlab socket connect 16777216 bytes in 1.45 real seconds = 11335.64 KB/sec +++ 2048 I/O calls, msec/call = 0.72, calls/sec = 1416.95 0.0user 0.0sys 0:01real 4% 0i+0d 0maxrss 0+2pf 0+0csw SNM — ver. 1.4 Measuring Throughput May use to transfer a large file, measure time   ◦ tests whole path ◦ problem: affected by disk i/o, xinetd May use a web browser and measure download time ◦ Problem: may be affected by caching in the web browser ◦ May be affected by caching in web proxies Better: measure using traffic similar to that created by the application. Network Troubleshooting — slide 71 SNM — ver. 1.4     ¦¥  ‚ ‰ ‘ ˆƒ 7 iproute The   Traffic Measurements:   BC The netstat program can show statistics about network interfaces Linux netstat shows lost packets in three categories: ◦ errors, ◦ drops (queue full: shouldn’t happen!) ◦ overruns (last data overwritten by new data before old data was read: shouldn’t happen!) ◦ drops and overruns indicate faulty flow control — bad! These values are cumulative (since interface was up) Could put a load on interface to see current condition, with ping -l, to send large number of packets to destination See the difference in values Network Troubleshooting — slide 76 program, iproute 7 S The ip program in the iproute package provides complete control over tcp/ip networking in a Linux system Provides more networking control facilities than other tcp/ip implementations Supports tunneling in many forms iproute documentation is in two manuals, one for ip routing, the other for tunnelling Network Troubleshooting — slide 74       SNM — ver. 1.4 iproute and   SNM — ver. 1.4 S Between these software packages, you can: ◦ throttle bandwidth for certain computers ◦ throttle bandwidth to certain computers ◦ fairly share bandwidth ◦ protect your network from DoS attacks ◦ protect Internet from your customers ◦ multiplex many servers into one, for load balancing or for high availability ◦ restrict access to your computers ◦ limit access of your users to other hosts ◦ do routing based on user id, mac address, source ip, port, type of service, time of day or content See the Linux Advanced Routing and Traffic Control HOWTO at http://tldp.org for details Network Troubleshooting — slide 75 CG 7 @D 8 ’ Measuring Traffic:           BC Here we run netstat -i on ictlab: $ Kernel Interface table ¦ ¦ ( ¡ Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg 3 BMRU eth0 1500 0 407027830 0 0 0 1603191764 0 0 lo 16436 0 2858402 0 0 0 2858402 0 0 0 LRU Notice that of the 1.6 billion bytes transmitted, there were 3 overuns. Next, blast the path you want to test with packets using ping -l or the spray program, and measure again. Network Troubleshooting — slide 77   SNM — ver. 1.4 SNM — ver. 1.4      ¦ ¤ E @ @8 @D S E @ @8 @D S Traffic measurements:   $ eth0 , Quick Guide to using   : set up interface S I U VS S TI A B 7 ifconfig and ip give more information than netstat -i: ƒ” “ˆ •“ Here we set up a network interface and give it the IP address 192.168.0.1/24: $ $ —‚ Link encap:Ethernet HWaddr 00:00:E2:35:AF:EE inet addr:172.19.64.52 Bcast:172.19.127.255 Mask:255.255.192.0 IPX/Ethernet 802.2 addr:33001601:0000E235AFEE UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:407579600 errors:0 dropped:0 overruns:0 frame:0 TX packets:1605655688 errors:0 dropped:0 overruns:3 carrier:0 collisions:0 txqueuelen:100 RX bytes:3055300191 (2913.7 Mb) TX bytes:2048217058 (1953.3 Mb) Interrupt:18 Base address:0xd000 ˜™ ˆ– ¨ (6 ¤  R ¦ ¡   £ #X #f #2 X§    W` We W g 3  ¦ & ¥ 0W ¤¥ 7 S  R   £  Two important points: ◦ If you do not specify the netmask, a netmask of /32 is assumed ◦ means obtain broadcast address by setting the host Network Troubleshooting — slide 80 $ 2: eth0: mtu 1500 qdisc pfifo_fast qlen 100 link/ether 00:00:e2:35:af:ee brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped overrun mcast 3058362227 407610495 0 0 0 0 TX: bytes packets errors dropped carrier collsns 2140511920 1605768150 0 0 0 0 „ˆ …‡ ‰ • dˆ ‰ ‡ ‚ˆ —‚ ˜™   3 bits SNM — ver. 1.4 SNM — ver. 1.4 Network Troubleshooting — slide 78 Quick Guide to using $ $ £   : set up routes ¤ X # W` #X We # f We #2 #f W2 # 2 X b g Network Troubleshooting — slide 81     ¨  7 S ¦ ¦ £©&   & X§ R  ¦ #X #f #W R¤   W` We 2 g £©&   ¦ The last adds a static route to another network the first adds the default route. You can omit the device if the network can be reached through a particular interface without any ambiguity ◦ I.e., ip is smart enough to figure out which network device to use, though specifying it doesn’t hurt. Getting more info using   The -s (-statistics) option to ip provides statistics. Adding a second gives you even more: $ 2: eth0: mtu 1500 qdisc pfifo_fast qlen 100 link/ether 00:00:e2:35:af:ee brd ff:ff:ff:ff:ff:ff RX: bytes packets errors dropped overrun mcast 3070792102 407726727 0 0 0 0 RX errors: length crc frame fifo missed 0 0 0 0 0 TX: bytes packets errors dropped carrier collsns 2445799644 1606151878 0 0 0 0 TX errors: aborted fifo window heartbeat 0 3 0 0 7 S SNM — ver. 1.4 „ˆ …‡ …‡ ‰ • dˆ ‰ ‡ ‚ˆ —‚ ˜™ SNM — ver. 1.4 Network Troubleshooting — slide 79 Packet Capture        W` ¤¥ R 0W ¤¥  ¦ 0W ¤¥ , Ethereal and Ntop l mk h Warning: Don’t Get Sacked! Be sure that your boss agrees with you capturing packets on your company’s network People have been sacked for doing this without permission! Some have suffered long lawsuits and criminal records: ◦ See http://www.stonehenge.com/merlyn/, http://www.lightlink.com/spacenka/fors/ for mous example Do not invade the privacy of others ◦ Capturing passwords with insecure protocols such as telnet, ftp, http (that is not encrypted with tls) is very easy – DON’T DO IT! SNM — ver. 1.4 Network Troubleshooting — slide 85 i j n j What is Packet Capture? Real time collection of data as it travels over networks Tools called: ◦ packet sniffers ◦ packet analysers ◦ protocol analysers, and sometimes even ◦ traffic monitors SNM — ver. 1.4 Network Troubleshooting — slide 83       and a fa-     Fo @T 7 Available everywhere Windows: http://windump.polito.it/ Syntax also used by other programs (such as Ethereal) Often it is the only tool available, so good to know Works by putting network interface into promiscuous mode ◦ normal Ethernet interface will ignore packets not addressed to it ◦ in promiscuous mode, will examine all packets that arrive, even those not addressed to it SNM — ver. 1.4 Network Troubleshooting — slide 86   Network Troubleshooting — slide 84 When Packet Capture? Most powerful technique When need to see what client and server are actually saying to each other When need to analyse type of traffic on network Requires understanding of network protocols to use effectively   SNM — ver. 1.4                 p 7 How to use   Filters: host and port Fo Fo @T 7   $ @T 7 p 7 Can just type its name (as root): $ Show all network traffic to and from 192.168.0.1: © ¦ & p & 7 ©0 ¡ ¦ #X W` #f #f #f We ¢¥ ¦ & )¥ ¡ ¦ #X W` We ¢¥ ¦ & )¥ ¡ ¦ #X W` We #2 W #2 ¥  ( W . . . but get a huge amount of data! Can restrict the data collected using a filter A filter may select addresses, protocols, port numbers, . . . Network Troubleshooting — slide 87 Show packets to 192.168.0.1: $         SNM — ver. 1.4 Show packets to port 68 on 192.168.0.1: $   #2 ¡& ¢¥ )¥ W ¦ SNM — ver. 1.4 Network Troubleshooting — slide 89 : some options Fo filters: networks Fo @T 7 p 7 @T 7 n capture a count of n packets then stop file write raw data to file . Very useful — can filter and analyse this later with tcpdump, ethereal or other tools but you cannot see what you are capturing till later! Capture traffic to or from 205.153.60/24: $  ¢  1   p 7   can specify network as source or destination: $ $     interface collect from interface instead of lowest numbered network interface bytes collect no more than bytes of data from each packet instead of default 68 bytes show link level info, e.g., Ethernet addresses gives a hexadecimal dump of packets  SNM — ver. 1.4 filters: protocol Fo  ¡ @T 7 tcpdump ip tcpdump tcp tcpdump ip proto ospf This will catch DNS name lookups, but not zone transfers (which use tcp): tcpdump udp port 53 Network Troubleshooting — slide 91    9 excluding link level data display ascii as well as hexadecimal if have -x option too Network Troubleshooting — slide 88  q Many more options: man tcpdump SNM — ver. 1.4 SNM — ver. 1.4           p 7 W2 # XY # b # W` Wb e #a e g§ 2 W X§ f g Network Troubleshooting — slide 90 & ¦ ¢¥ )¥ ¡ £ & ¡ ¦ ¦ ¢ ¢¥ )¥ ¤ ( ¦ ( ¦ X W # XY # W` e g§ W f & ¦ ¢¥ )¥ ( ¦ £¦ ¢¥ )¥ © e f filters: combining HTTP $ ∼ 22:43:32.633636 192.168.25.9.14075 > 172.19.64.52.http: S 1015952778:1015952778(0) win 6144 (DF) 22:43:32.633693 172.19.64.52.http > 192.168.25.9.14075: S 1929920485:1929920485(0) ack 1015952779 win 5840 (DF) 22:43:32.635828 192.168.25.9.14075 > 172.19.64.52.http: P 1:590(589) ack 1 win 6144 (DF) 22:43:32.635906 172.19.64.52.http > 192.168.25.9.14075: . ack 590 win 6479 (DF) 22:43:32.636758 172.19.64.52.http > 192.168.25.9.14075: P 1:217(216) ack 590 win 6479 (DF) 22:43:32.636982 172.19.64.52.http > 192.168.25.9.14075: F 217:217(0) ack 590 win 6479 (DF) 22:43:32.639080 192.168.25.9.14075 > 172.19.64.52.http: R 590:590(0) ack 217 win 0 (DF) Fo @T 7 p 7 u„ ‚ v ƒ„ ‚ v This will not work as you might expect: tcpdump host ictlab and udp or arp Instead, need group with parentheses, and quote: …•        r ¤ & ©0 ¡ ¦ ¨ 3 ¥ ¦ ¢¦ ¢¥ )¥   ( & © £   many more ways of filtering: man tcpdump Network Troubleshooting — slide 92 SNM — ver. 1.4    £¥ r " SNM — ver. 1.4 „ ‚‚ ˜ ƒ„ ‚ †‚ u„ st † u„w st „ ƒ„ „” Network Troubleshooting — slide 95 Writing data to a file $ ∼ tcpdump: listening on eth0 1014 packets received by filter 0 packets dropped by kernel § © ¦ & W2 2 2 § & )¥ Network Troubleshooting — slide 93 )¥ ¦ ¡& ¢¥ )¥ ¢ 1 ¢¥ ¦   SNM — ver. 1.4 # ¥ ¢ ¥ : When reading TCP Fo @T 7 Reading a Dumped File $ ∼ 22:32:41.751452 arp 22:32:41.863173 arp 22:32:41.863198 arp 22:32:42.082584 arp 22:32:43.113655 arp 22:32:44.635149 arp 22:32:44.874117 arp 22:32:45.147178 arp 22:32:45.209507 arp 22:32:45.212484 arp 22:32:45.239445 arp 22:32:45.455863 arp 22:32:45.540507 arp tell 172.19.65.10 22:32:45.562004 arp format: src ¿ dst: flags data-seqno ack window urgent options Flags are some combination of S (SYN), F (FIN), P (PUSH) or R (RST) or a single ‘.’ (no flags). The first time tcpdump sees a tcp ’conversation’, it prints the sequence number from the packet. On subsequent packets of the conversation, the difference between the current packet’s sequence number and this initial sequence number is printed. Network Troubleshooting — slide 96 …• u„ ‚ v ƒ„ ‚ v ƒ„ ‚ u„ st † u„w st „ ƒ„ who-has 172.19.127.254 tell 172.19.127.29 who-has 172.19.64.52 tell 172.19.64.63 reply 172.19.64.52 is-at 0:0:e2:35:af:ee who-has 172.19.65.16 tell 172.19.125.229 who-has 172.19.123.211 tell 172.19.65.2 who-has 172.19.65.16 tell 172.19.127.106 who-has 172.19.65.6 tell 172.19.126.174 who-has 172.19.65.16 tell 172.19.126.240 who-has 172.19.127.254 tell 172.19.125.127 who-has 172.19.127.175 tell 172.19.125.127 who-has 172.19.127.254 tell 172.19.125.212 who-has 172.19.65.16 tell 172.19.126.194 who-has 172.19.126.50 (44:30:54:59:43:4d) who-has 172.19.126.50 tell 172.19.65.2 „ † SNM — ver. 1.4 Network Troubleshooting — slide 94 SNM — ver. 1.4           p 7 Window win nnn specifies data window the sending host will accept in future packets ◦ I.e., the maximum number of bytes tcp flow-control: ◦ host reduces this number if congested or overloaded ◦ will sometimes set to 0 to temporarily halt incoming traffic in this connection SNM — ver. 1.4 Network Troubleshooting — slide 97     Ethereal King of the Packet Analysers! Available for Linux, Unix, Windows Ethereal Ethereal can read data captured by tcpdump, e.g., $ SNM — ver. 1.4 Network Troubleshooting — slide 100   0  ¨ #¥ & ) ¥ ¢ ¦ £ £  or File → Open Can capture data itself Uses same filter language as tcpdump Network Troubleshooting — slide 99 SNM — ver. 1.4       ¢¥ ¦ SNM — ver. 1.4 Network Troubleshooting — slide 101 SNM — ver. 1.4 Network Troubleshooting — slide 103 Display Filters Note the box at the bottom of Ethereal for display filters Select only some of the packets captured for display see man ethereal and search for DISPLAY FILTER SYNTAX Different syntax than the syntax for capture filters Example: ip.src==172.19.64.52 and ip.dest==172.19.64.57 Network Troubleshooting — slide 104 You can expand any protocol: If we click on the + next to Bootstrap Protocol, we can see the details of the DHCP Request: Network Troubleshooting — slide 102   SNM — ver. 1.4 SNM — ver. 1.4             Tools → Follow TCP Stream Can view the contents of an entire tcp stream conversation, in ascii or in hexadecimal. Be careful not to invade your customers’ privacy. Can use to check if a communications stream is really encrypted Network Troubleshooting — slide 105 Ntop: Installing Installation is pretty easy. On my Fedora Core 1 machine:   SNM — ver. 1.4 $ $ $ -rwx------ 1 root root 13203 Apr 27 03:47 /etc/ntop.conf.sample $ $ # temporarily comment out the line --daemon $ $   ~} ~} ~ ‰|  €~ „  Ž xyz Šƒ € ‚ ƒ„ Ž  ‹Œ † yŠ „z Œ Ž „ ‚—– Ž ‚ }Š | ‘ Ž ’‘ Ž ‚ ƒ„ Ž ’“ † Ž xyz …”•} y €†‡… ˆ †€ ˆ ‰xŠ xyz {| x {| €€ ‚ ƒ„ y €† … †€ ˆ …”•} ˆ     Then open the web browser on http://localhost:3000/ Network Troubleshooting — slide 107 SNM — ver. 1.4 Ntop: monitoring data at a point The Ntop program: ◦ listens on a network interface ◦ puts an Ethernet interface into promiscuous mode and ◦ displays statistics through a web interface Shows: ◦ percentages of protocols, ◦ which machines generate most traffic ◦ which traffic is purely local, which traffic comes from outside, which traffic goes from inside to outside of network SNM — ver. 1.4 Network Troubleshooting — slide 106     Switched Networks   ‰ | Ž x‰  }Š x { Ž ‚„ ƒ ‚ } Ž y ‚„ ƒ ‰ƒ y ˜x  š Ž ƒ Šƒ ‚„ ƒ Ž † yŠ „ ‚– €› ‰|  ‰|  „„ ‰|  ‰|  „„ z Šy ˜Š € ‰˜ Ž Ž Šƒ Šƒ ‚„ ƒ Ž ‚„ ƒ Ž † yŠ † yŠ „„ ‚ –™ † ‚ – ‰z˜ y~  Ž Šƒ ‚„ ƒ Ž † yŠ „ ‚– ‰ † xyz Using Ethereal, , Ntop in a switched network l mk h i j n j How monitor one machine? You are asked to check out a server on a switched network: The switch does not support port monitoring ( 109), or you do not have administrative access to the switch: what to do? Use a small hub, and use a notebook running the capture software Ethernet Switch     mini−hub Device under test e.g., a server notebook running capture software SNM — ver. 1.4 Network Troubleshooting — slide 110 Port Monitoring: Switched Networks Problem: a switched network is really a point-to-point network You cannot normally capture the unicast traffic from other hosts on a single switch port How do you use Ethereal, tcpdump or Ntop to monitor traffic between a number of hosts?   Are switched networks secure? Is all unicast traffic on one port of a switch private? No, there are tools (dsniff and Ettercap) freely available to automate arp spoofing and man-in-the-middle attacks, that provide various ways to compromise switch security. Network Troubleshooting — slide 111     Solution: many switches support port monitoring, where one port can monitor all traffic on a specified VLAN Example: Cisco 3500XL switches provide the command: SNM — ver. 1.4   (¤ ¦ R¨ œ  ¥ £¦ © ) © © £ SNM — ver. 1.4  ( 5 žW Network Troubleshooting — slide 109 (¤ ¦ £¦ ¥ © ) © © £     ' Port Scanning Running $ $ : Use B pD 7 Ÿ B © ¡& ©  R ¡& What is a port scanner? Sends packets to various ports on a network device Best one available everywhere is nmap can identify the OS of the target machine Do not port scan arbitrary machines in your company’s network without permission! May be interpreted as a cracking attempt Network Troubleshooting — slide 113 Enter the ip address of machine(s) to identify select other choices from buttons press Start xnmap is simply a way to easily generate command line options to nmap using a graphical interface Network Troubleshooting — slide 115       SNM — ver. 1.4 SNM — ver. 1.4     Uses of           9 B Identify the type of a computer that is causing trouble on the network Check what network services a computer is really offering ◦ compare with netstat -tua output   How does   identify OS? ◦ A cracked computer may be hiding some services with trojaned utilities ◦ nmap can help you discover such services SNM — ver. 1.4 Network Troubleshooting — slide 116 B rfcs leave interpretation of some things up to the implementer rfcs do not specify how should work if get contradictory flags, strange sequences of inconsistent packets Most tcp/ip implementations are not complete Every implementation of tcp/ip is different; the “grey areas” are different from one os to another. sends “strange” packets to the machine, detects how reacts, matches this against a file of os fingerprints   ()       ¥ Dp 7 SNM — ver. 1.4 Network Troubleshooting — slide 114 DNS troubleshooting pD () ¥ 7   pD 7 Troubleshooting DNS Servers : Checking forward DNS lookup # (¤  (© ¡Q ¡ ) # ¤¥ ¢ ©) DNS troubleshooting Suspect dns when get long timeouts before see any response ping name, ip address, see if only ip address works tools on Linux, Unix: ◦ dig, nslookup, host tools on Windows: ◦ nslookup SNM — ver. 1.4 Network Troubleshooting — slide 118 ; <<>> DiG 9.2.1 <<>> sysadmin.no-ip.com ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 23568 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3 ;; QUESTION SECTION: ;sysadmin.no-ip.com. IN A ;; ANSWER SECTION: 60 IN A 202.69.77.139 sysadmin.no-ip.com. ;; AUTHORITY SECTION: no-ip.com. 60 IN NS nf1.no-ip.com. no-ip.com. 60 IN NS nf2.no-ip.com. no-ip.com. 60 IN NS nf3.no-ip.com. ;; ADDITIONAL SECTION: nf1.no-ip.com. 60 IN A 66.185.166.131 nf2.no-ip.com. 60 IN A 66.185.162.100 nf3.no-ip.com. 60 IN A 216.66.37.10 ;; Query time: 254 msec ;; SERVER: 127.0.0.1#53(127.0.0.1) ;; WHEN: Mon Feb 24 10:55:26 2003 ;; MSG SIZE rcvd: 154         SNM — ver. 1.4 ¤ o US $ Network Troubleshooting — slide 120 : reverse lookup 1 # YY W a` 9X  2X # #` e DNS:   The people who write the most common name server (Bind) promote dig, deprecate nslookup dig output is in form of dns resource records ◦ can copy and paste straight into dns database files Network Troubleshooting — slide 119 ; <<>> DiG 9.2.1 <<>> -x 202.69.77.139 ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 22117 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 2, ADDITIONAL: 0 ;; QUESTION SECTION: ;;139.77.69.202.in-addr.arpa. IN PTR ;; ANSWER SECTION: 139.77.69.202.in-addr.arpa. 3600 IN PTR 077-139.onebb.com. ;; AUTHORITY SECTION: 77.69.202.in-addr.arpa. 3600 IN NS ns2.onebb.com. 77.69.202.in-addr.arpa. 3600 IN NS ns1.onebb.com. ;; Query time: 310 msec ;; SERVER: 172.19.64.52#53(172.19.64.52) ;; WHEN: Mon Feb 24 11:07:04 2003 ;; MSG SIZE rcvd: 111 SNM — ver. 1.4   o US SNM — ver. 1.4 ¤ o US $ Network Troubleshooting — slide 121 syntax options server name type : an interactive program o US AG 8B P¡ ¢ (¡ ¨ Network Troubleshooting — slide 122 ¡ ¢ main option is -x server is the name server to query ◦ by default, use first server in /etc/resolv.conf name is what you want to look up type can be: any, a, mx, axfr, soa, etc. default is to get A record(s) $ Note: nslookup is deprecated and may be removed from future releases. Consider using the ‘dig’ or ‘host’ programs instead. Run nslookup with the ‘-sil[ent]’ option to prevent this message from appearing. > sysadmin.no-ip.com Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: Name: sysadmin.no-ip.com Address: 202.69.77.139 SNM — ver. 1.4 Network Troubleshooting — slide 124 ¤ SNM — ver. 1.4           ©6 © & ¥ F§ F§ A AG 8B A ¦ d ˜ † “ : axfr (Zone Transfer) dig can request a complete zone transfer: $ o US ; <<>> DiG 9.2.2-P3 <<>> @ns tyict.vtc.edu.hk axfr ;; global options: printcmd tyict.vtc.edu.hk. 86400 IN SOA ns.tyict.vtc.edu.hk. nicku.vtc.edu.hk. 2004031000 3600 1800 604800 600 tyict.vtc.edu.hk. 86400 IN NS ns.tyict.vtc.edu.hk. tyict.vtc.edu.hk. 86400 IN NS ns1.tyict.vtc.edu.hk. tyict.vtc.edu.hk. 86400 IN NS dns1.vtc.edu.hk. tyict.vtc.edu.hk. 86400 IN NS dns2.vtc.edu.hk. 000081667.tyict.vtc.edu.hk. 86400 IN A 172.19.64.92 ... > 202.69.77.139 Server: 127.0.0.1 Address: 127.0.0.1#53 Non-authoritative answer: 139.77.69.202.in-addr.arpa name = 077-139.onebb.com. Authoritative answers can be found from: 77.69.202.in-addr.arpa nameserver = ns1.onebb.com. 77.69.202.in-addr.arpa nameserver = ns2.onebb.com. ns1.onebb.com internet address = 202.180.160.1 ns2.onebb.com internet address = 202.180.161.1 >   •‡ £ ¤‚ w ˆƒ ‚ w ¥ƒ ‚ w — st sˆ– SNM — ver. 1.4 7 7 : reverse lookups Network Troubleshooting — slide 125 result can be copied and pasted as a master file in a DNS server Network Troubleshooting — slide 123 SNM — ver. 1.4   Telnet: Troubleshooting Email and Other Protocols Email: testing with   C@ G @ BC SMTP commands for sending mail identify your computer specify sender ¨¤ )  £ ¢¥ ¦ ¦  ¨& ¦¤   © Email protocols smtp, pop3 are text telnet a good tool to test them syntax: $ server portnumber  0¨ ©     SNM — ver. 1.4 ¦ indicates start of message body terminate session Use names, not IP addresses, to specify destination Network Troubleshooting — slide 129 ¦ ¨ SMTP: port 25 POP3: port 110 Network Troubleshooting — slide 127 SNM — ver. 1.4 Test the VTC mail server: $ ¦ ¨     ( R # ¢¦  # & ¦ 06 ( Trying 192.168.79.191... Connected to smtp.vtc.edu.hk (192.168.79.191). Escape character is ’^]’. 220 pandora.vtc.edu.hk ESMTP Mirapoint 3.2.2-GA; Tue, 25 Feb 2003 11:15:30 +0800 (HKT) helo nickpc.tyict.vtc.edu.hk 250 pandora.vtc.edu.hk Hello [172.19.32.30], pleased to meet you mail from: 250 ... Sender ok rcpt to: 250 ... Recipient ok data 354 Enter mail, end with ‘‘.’’ on a line by itself My message body. . 250 AFF21826 Message accepted for delivery quit 221 pandora.vtc.edu.hk closing connection Connection closed by foreign host. # ¦¥ ¡) ¦ X b Testing the VTC pop3 server 1 $ Trying 192.168.79.12... Connected to pop.vtc.edu.hk (192.168.79.12). Escape character is ’^]’. +OK carme.vtc.edu.hk POP3 service (iPlanet Messaging Server 5.2 Patch 1 (built Aug 19 2002)) user nicku +OK Name is a valid mailbox pass password +OK Maildrop ready stat +OK 1 673 •— ‰ „” ‚ £© )  specify receiver —‚ SNM — ver. 1.4 Network Troubleshooting — slide 128 SNM — ver. 1.4 w ¥ƒ ‚ w — st „w ˜d© © ™ Network Troubleshooting — slide 130 Testing the pop3 server 2 retr 1 +OK 673 octets Return-path: Received: from pandora.vtc.edu.hk (pandora.vtc.edu.hk [192.168.79.191]) by carme.vtc.edu.hk (iPlanet Messaging Server 5.2 Patch 1 (built Aug 19 2002)) with ESMTP id <0HAU00I35H3HGL@carme.vtc.edu.hk> for nicku@ims-ms-daemon (ORCPT nicku@vtc.edu.hk); Tue, 25 Feb 2003 11:16:29 +0800 (CST) Received: from nickpc.tyict.vtc.edu.hk ([172.19.32.30]) by pandora.vtc.edu.hk (Mirapoint Messaging Server MOS 3.2.2-GA) with SMTP id AFF21826; Tue, 25 Feb 2003 11:16:01 +0800 (HKT) Date: Tue, 25 Feb 2003 11:15:30 +0800 (HKT) From: Nick Urbanik Message-id: <200302250316.AFF21826@pandora.vtc.edu.hk> My message body. . dele 1 +OK message deleted quit +OK Connection closed by foreign host. : Testing Other Applications C@   BC G Many network protocols are text. telnet can be helpful in checking: ◦ IMAP servers: $ hostname @ ¦ ¨ ◦ Web servers: $ hostname ¦ ¨ ( ¦ ◦ Ftp servers: $ hostname ¦ ¨ ( ¦ ◦ Even ssh (can check version, if responding): $ hostname SNM — ver. 1.4 Network Troubleshooting — slide 131 ¦ ¨ ( ¦ SNM — ver. 1.4 ( ¦ X Network Troubleshooting — slide 133 Network Troubleshooting — slide 134 Conclusion pop3 commands: retrieving mail See RFC 1939 for easy-to-read details First, must authenticate: username password shows number of messages and total size in bytes Know what is “normal” Get permission from the boss before using packet sniffing and port scanners list all the message numbers and size in bytes of each message messagenum retrieve the message with number messagenum messagenum delete messagenum the message with message number SNM — ver. 1.4 Check the simple things first Be methodical Document what you do Become familiar with common tools Use the tools to become familiar with your network before troubles strike     & ¡  £ ¥  ¦ ¡ ¦ ¡ ¡ ¨ ¡ ¦ ¤ ¨& SNM — ver. 1.4 ¦¤  ¨  ¦ £ £ Network Troubleshooting — slide 132               X X W f2 W g a