I think the two products are complementary. IDS/IPS looks for all sorts of signatures either in incoming or outgoing packets whereas app-attack-detector (fail2ban underneath the skin) monitors your logs for things you may not want so their trigger mechanisms are very different. IDS/IPS is heavy on resources, f2b is light. IDS/IPS can look for specific vulnerabilities and exploits whereas f2b can pick up on not so malicious traffic, for example on traffic probing your web site for non-existant files (which show up in the httpd error log). It is much easier to create and edit your own filters in f2b. IPS/IDS is almost impossible without packet capture and analysis knowledge.