An Intrusion Detection System (IDS) monitors a network and its servers for suspicious activity. An IDS can be hardware or software. Some are passive and only report suspicious activity, while others are reactive and will take defensive actions in response to alerts. A reactive IDS is sometimes called an "Intrusion Detection and Prevention System" (IDPS). Most of the time, the terms can be use interchangeably.
A Network IDS (NIDS) is located at strategic points in a network so that it can monitor traffic between all devices on the subnet. It can perform rapid, real-time analysis of data packets. In some cases it can also perform slower, deeper analysis of stored data. NIDS will watch for known attacks or intrusion indicators and can recognize potential threats by comparing current activity against an established baseline of activity.
A Host IDS (HIDS) monitors all traffic to and from a single server. In addition to traffic monitoring, HIDS can also monitor server files. Production server system files and configurations change infrequently. HIDS includes a File Integrity Checker which takes a snapshot of critical system files and then alerts an admin if the snapshot changes.
The line between IDS and firewall is blurry. Both are similar tools which analyze packets of data, searching for patterns. And, in fact, some hardware and software will perform both roles. In general, the distinction is that a firewall analyzes data moving in/out of a network and either allows or denies traffic, but send no alerts, while an IDS analyzes data moving within a network and sends alerts when problems are detected.
One frequent criticism of IDS is that false positives can be too common and send too many alerts. If alerts are not taken seriously each time, this can be a security vulnerability. Another concern is that as a network and its traffic grows, the IDS will need to be bigger and faster to keep up. Encryption presents challenges for IDS. Encryption is more secure for data transport, but it becomes difficult for the IDS to analyze legitimate traffic. However, encrypted traffic cannot simply be ignored, because attackers use encryption too.
IDS uses three techniques to identify intrusions. Signature-based analysis watches traffic for known patterns ("signatures") used by malware and attacker trying to exploit a vulnerability. It is similar to the signature matching done by anti-virus programs. Anomaly-based analysis watches for out-of-the-ordinary traffic. This could be abnormal usage of bandwidth, protocols, or ports. The third techique is very different from the first two. IDS often uses honeypots to identify intruders.
A honeypot is a valuable-looking decoy resource which is being monitored by the IDS. The decoy resource could be a server (real or virtual), a program or service, a directory or file, or a database. The goal is to attract and identify attackers inside the network. It is a form of "baiting" the attackers into revealing themselves. It gets its name because it is similar to how Winnie the Pooh is uncontrollably attracted to pots of honey.
A honeypot can identify attacker traffic, block attackers, analyze their attack techniques, and gather forensic data for legal prosecution. Honeypots have a low number of false positives because the honeypot has no legitimate purpose. Any access to the honeypot is either a curious user or an attacker. Honeypots can be very useful tools against new attacks and zero-day exploits which slip past other defenses. It can also be a cat-and-mouse game though; attackers can use honeypot detection tools.