Opsec WG P. Savola Internet-Draft CSC/FUNET Intended status: Informational January 23, 2008 Expires: July 26, 2008 Experiences from Using Unicast RPF draft-savola-bcp84-urpf-experiences-03.txt Status of this Memo By submitting this Internet-Draft, each author represents that any applicable patent or other IPR claims of which he or she is aware have been or will be disclosed, and any of which he or she becomes aware will be disclosed, in accordance with Section 6 of BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF), its areas, and its working groups. Note that other groups may also distribute working documents as Internet- Drafts. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." The list of current Internet-Drafts can be accessed at http://www.ietf.org/ietf/1id-abstracts.txt. The list of Internet-Draft Shadow Directories can be accessed at http://www.ietf.org/shadow.html. This Internet-Draft will expire on July 26, 2008. Copyright Notice Copyright (C) The IETF Trust (2008). Abstract RFC 3704 (BCP 84) published in March 2004 provided an ingress filtering technique update to RFC 2827 (BCP 38). This memo tries to document operational experiences learned practising ingress filtering techniques, in particular ingress filtering for multihomed networks. Savola Expires July 26, 2008 [Page 1] Internet-Draft Unicast RPF Experiences January 2008 Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 3 2. Common uRPF Drops . . . . . . . . . . . . . . . . . . . . . . 3 2.1. Unused Address Space Ping-Pong . . . . . . . . . . . . . . 4 2.2. Private Address Leak . . . . . . . . . . . . . . . . . . . 4 2.3. Wrong IP Address . . . . . . . . . . . . . . . . . . . . . 5 3. Multihoming uRPF Drops . . . . . . . . . . . . . . . . . . . . 5 3.1. Incorrect Source Address Selection . . . . . . . . . . . . 5 3.2. Point-to-Point Interface Routes . . . . . . . . . . . . . 6 3.3. Multiple Routers on a LAN use LAN for Transit . . . . . . 7 4. Special uRPF Failures Cases . . . . . . . . . . . . . . . . . 8 4.1. PMTUD and Private/Non-routed Addresses . . . . . . . . . . 8 5. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9 6. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . 9 7. Security Considerations . . . . . . . . . . . . . . . . . . . 9 8. References . . . . . . . . . . . . . . . . . . . . . . . . . . 9 8.1. Normative References . . . . . . . . . . . . . . . . . . . 9 8.2. Informative References . . . . . . . . . . . . . . . . . . 10 Author's Address . . . . . . . . . . . . . . . . . . . . . . . . . 10 Intellectual Property and Copyright Statements . . . . . . . . . . 11 Savola Expires July 26, 2008 [Page 2] Internet-Draft Unicast RPF Experiences January 2008 1. Introduction RFC 3704 [RFC3704] (BCP 84) published in March 2004 provided an ingress filtering technique update to RFC 2827 [RFC2827] (BCP 38). This memo tries to document operational experiences learned practising ingress filtering techniques, in particular ingress filtering for multihomed networks. Specifically, this version describes the lessons learned in author's network where strict unicast RPF (uRPF) ingress filtering, using "feasible paths" variant [RFC3704] has been used for all the customer interfaces (whether single- or multihomed) since 2003. In feasible paths strict uRPF, only an accepted equal length prefix (even if not preferred) is considered feasible. Although in some cases a more specific or even a less specific might be acceptable, such condition would not necessarily be correct in general. We use the typical "customer" and "ISP" terms to refer to the subject of strict uRPF filtering and the party doing filtering. The same considerations also apply for other business relationships (e.g., "internal customers" inside an ISP). According to a study, there is substantial ingress filtering deployment, even 75% of addresses were not spoofable [SPOOFER]. We note explicitly that Loose mode RPF is NOT a sufficient solution in any way to ingress filtering as it creates a false sense of protection. Even its use as a "contract validation" [RFC3704] is tenuous at best. NOTE IN DRAFT: comments should be directed to the author or the OPSEC mailing list (opsec@ops.ietf.org). However, it is not clear what should be the next steps wrt. these experiences. Update to the ingress filtering RFCs? Publish separately? Keep as a standing document for now? Integrate with OPSEC document work? In any case, feedback on other experiences is encouraged. In the second section, we'll first look at the most common types of uRPF drops and their causes. In the third section, we'll look at a few special cases observed on multihoming or multi-connecting scenarios. More special filtering failures are discussed in the fourth section. 2. Common uRPF Drops Most uRPF packet drops are in fact due to anomalies which have nothing to do with spoofing source addresses but are detected (and Savola Expires July 26, 2008 [Page 3] Internet-Draft Unicast RPF Experiences January 2008 prevented) by the uRPF methodology. In this section, we'll describe the most common causes for uRPF drops which apply to both single- and multi-homed networks, and respective ways to eliminate or mitigate dropping. 2.1. Unused Address Space Ping-Pong By far, the most common cause for uRPF drops seems to be the case where a prefix P is routed to the customer (e.g., using a static route), but the customer doesn't use all of P, and an attacker A is port-scanning the unused address space. In this case, typically packets destined to the unused part of "P" lack a more specific route, and are forwarded back to the ISP through a default route. The ISP's router sees these as sourced from attacker A (an IP address in the Internet), destined to the customer's prefix P. This fails uRPF check and is dropped. Note: if uRPF is not employed, the scan may may cause ping-pong effect up to the remaining hop count/TTL of the packet, consuming even 250 times the bandwidth and packet processing. This has been briefly described in [I-D.ietf-ipngwg-p2p-pingpong]. Hence employing uRPF significantly mitigates the impact of this kind of packet looping. The ping-pong effect has also been used in Internet Exchanges to game peer selection or traffic balance data. Therefore, the customer should install static discard aggregate routes (or equivalent) for all of its address space upon assignment, so that if no better route exists, such probe packets are discarded. An alternative is applying a similar filtering in egress interface towards the ISP. There isn't much an ISP can do to prevent this unless it wants to create customer-specific uRPF access-lists. 2.2. Private Address Leak Very often, packes from all kinds of private addresses also leak to the ISP and are obviously dropped by uRPF. This is probably a result of misconfigured NATs or inadequate firewall rules. Even (constant) rates of hundreds of packets per second have been observed, which makes one wonder which kinds of users' communications must be failing or otherwise working in a non-optimal fashion due to this kind of misconfiguration... This is actually one of the most convincing reasons from the users' perspective why (they or the ISP) using uRPF could give benefits: it allows them to notice and fix network misconfiguration and Savola Expires July 26, 2008 [Page 4] Internet-Draft Unicast RPF Experiences January 2008 malfunction "at the source" and as a result, communication should work more reliably and new issues would be easier to notice. The obvious fix is to ensure that the customer is filtering out (and logging) these packets, and uses that to figure out what is causing such address leaks and fixes the misconfiguration or other problem(s). 2.3. Wrong IP Address It's also not atypical to see other kinds of wrong source addresses. These can be classified in three main categories: a) nomadic laptops trying their old IP from a previous network attachment point, b) spoofed/misconfigured/typoed public, routable IP address, or c) an unroutable ("bogon") IP address. (It should be noted that Loose uRPF would only spot the last category.) Many spoofed attacks are usually a result of a worm or a botnet (DoS) attack. A recent case was using recursive DNS servers for reflection [I-D.ietf-dnsop-reflectors-are-evil], but a lot of different usages have been observed. The same considerations as for leaking private addresses apply here, except that these wouldn't typically get this far if the customer had been using unicast RPF at its LAN interfaces -- uRPF can and should be applied recursively [RFC3704]. 3. Multihoming uRPF Drops We'll describe a few multihomed/multi-connected network scenarios which cause uRPF drops, and how to eliminate these drops. Bearing these in mind, uRPF can be employed with multihomed networks as well. We note that a customer can multihome and even perform traffic engineering with feasible paths uRPF provided that the consistency requirement is fulfilled. In other words, AS-path prepending, setting communities to lower local-preference, etc. are all valid mechanisms to ensure the prefix is advertised to every provider, but actually may not ever end up being used. 3.1. Incorrect Source Address Selection Hosts attaching to multiple LANs with different IP addresses need to be careful with their source address selection. The same applies to networks with multiple prefixes as explored in [I-D.huitema-shim6-ingress-filtering]. Savola Expires July 26, 2008 [Page 5] Internet-Draft Unicast RPF Experiences January 2008 For example, assume the host has a default route through interface 1 with address A1 from prefix P1, and only a more specific route through interface 2 with address A2 from prefix P2. When a host in P1 sends a packet to A2, the response may go out through interface 1; similarly, when a host in P2 sends a packet to A1, the response may go out through interface 2. This problem can be fixed by the customers by setting up source-based address selection on the multi-interfaced hosts or source-based routing with multi-prefixed networks. Alternatively, making an exclusion in the uRPF filter list allows sourcing from the other prefix. The exclusion is typically not a good solution when the ISP doesn't control both the prefixes, because an ISP originating these excluded packets would be indistinguishable from IP address spoofing. 3.2. Point-to-Point Interface Routes Feasible path strict uRPF works well, but assumes that the routes in all the directions are consistent (i.e., exist). This principle is often violated with the interface routes between the ISP and the customer (ie., point-to-point links). In some cases, the point-to-point link may be unnumbered but this has other issues (e.g., eBGP is more complicated). If the links have addresses, the address blocks usually need to be separate. The addresses might be more specifics of the customer's aggregate(s) or from the ISP's address space. In either case, the similar source address selection issue as described in the previous section applies for communication (e.g., pinging the CPE's p2p address) to the customer's point-to-point addresses. Internet / | / | / | +------+-+ +-+------+ |Router 1|---|Router 2| +-+------+ +--+-----+ 192.0.2.253/30| |192.0.2.249/30 link 1 | | link 2 192.0.2.254/30| |192.0.2.250/30 .---------------------. / multi-conn. customer \ \ 192.0.2.0/24 / '---------------------' Figure 1 Savola Expires July 26, 2008 [Page 6] Internet-Draft Unicast RPF Experiences January 2008 This issue can demonstrated by examining Figure 1. Assuming that the customer has configured link 1 to be primary for egress traffic, when someone in the Internet pings 192.0.2.250 (or traceroutes to a customer's address), router 1 receives a response packet with 192.0.250 source address from link 1. Router 1 thinks the "correct" direction the packet should have come from is the link towards Router 2 (due to router 2 advertising the more specific connected route in IGP/iBGP), not from the customer, even though the packet's source address is in the customer's superblock. Even though the CPE router would not have any services, these kind of packets are emitted at least as a result of traceroute or brute-force address space scanning. The easiest fix is to add dummy static routes with a higher preference/distance on all the border routers, so that every router facing the customer knows all the point-to-point address blocks used on other routers; using a higher preference implies that the route is actually never used, but is still valid from uRPF perspective. In the example above, this would mean adding a 192.0.2.252/30 route pointing to 192.0.2.250 on router 2 and a 192.0.2.248/30 route pointing to 192.0.2.254 on router 1. Another possibility, if the addresses come from the customer's aggregate, is to not propagate the point-to-point addresses in iBGP or IGP at all so that there are no more specifics to mess up the uRPF feasible path consistency, but this may have manageability concerns if the aggregate goes down (i.e., can't ping the point-to-point address except on the router connecting the customer). As already mentioned, using unnumbered interfaces is also possible in some cases but may have manageability or configuration concerns. 3.3. Multiple Routers on a LAN use LAN for Transit When multiple routers attach to the same network subnet (typically when e.g., VRRP/HSRP is used), packets destined to router 2 (R2)'s interface addresses towards the LAN transiting router 1 use R1's LAN interface to reach R2. (In most cases, the primary path between routers should go via dedicated link(s), not via a LAN. IGP is enabled on the direct R1-R2 link but not on the LAN.) These packets fail uRPF check at R2 (and vice versa at R1). Savola Expires July 26, 2008 [Page 7] Internet-Draft Unicast RPF Experiences January 2008 Internet / | / | / | +------+-+ +-+------+ Customer A -- |Router 1|---|Router 2| +-+------+ +--+-----+ 192.0.2.254/24| |192.0.2.253/24 -------+-------------+------- VRRP-enabled stub LAN Figure 2 This scenario is demonstrated by examining Figure 2. When customer A pings router 2's loopback address (outside 192.0.2.0/24), packets get forwarded through R1-R2 link. When customer A pings 192.0.2.253, router 1 forwards the packets to the LAN (due to connected route always being better than IGP/iBGP learned route), and router 2 receives them through the LAN. However, from router 2's perpective, it seems that someone in the LAN has spoofed Customer A's source addresses and uRPF drops the packets. There are two obvious fixes: 1. Force forwarding to the interface addresses from R1 to R2 (and vice versa) go through a dedicated link (practically this requires a static route as advertising an interface address instead of interface prefix in IGP or iBGP is difficult), or 2. Make an exception to uRPF configuration to allow such "transit LAN" usage. Both options have their tradeoffs: the latter allows an attacker in the LAN to spoof an address to the LAN router's interface address(es) (for example, circumventing remote login access lists), while the former introduces operational complexity. 4. Special uRPF Failures Cases 4.1. PMTUD and Private/Non-routed Addresses A disturbing issue is that some large operators seem to think it's perfectly legitimate to send private-source addressed ICMP messages (e.g., from PMTUD) across AS boundaries [PRIVIP]. While the reasoning is different, the result is similar for non-routed, but uniquely assigned address space. This might prevent applying strict packet-based source filtering from the direction of that network. Savola Expires July 26, 2008 [Page 8] Internet-Draft Unicast RPF Experiences January 2008 Private IP addresses for infrastructure are a bad idea. But even worse is doing that and deploying links in such infrastructure which have lower MTU than the egress link, i.e., are guaranteed to send ICMP fragmentation needed messages under certain circumstances. Deploying such networks that require PMTUD to work while happily originating RFC1918 traffic (and translating it at the edge) seems like very bad design from network hygiene perspective. 5. IANA Considerations This memo makes no request to IANA. 6. Acknowledgements Danny McPherson, Matsuzaki Yoshinobu, Barry Greene, Fred Baker, and Christian Vogt provided comments on earlier revisions of this document. 7. Security Considerations This document describes uRPF experiences. The most important security impact comes from applying particular fixes to uRPF issues noted, i.e., what kind of spoofing window or other unintended usage that would allow. As already stated, in invalid source address selection scenario, making an exception to allow prefixes which you don't control is typically a big mistake, as then you become indistinguishable from someone spoofing that address. Also as already stated, in the case of transit LAN, making an exception might allow one to spoof an address destined to the LAN router's interface address(es) which usually has a security impact. 8. References 8.1. Normative References [RFC2827] Ferguson, P. and D. Senie, "Network Ingress Filtering: Defeating Denial of Service Attacks which employ IP Source Address Spoofing", BCP 38, RFC 2827, May 2000. [RFC3704] Baker, F. and P. Savola, "Ingress Filtering for Multihomed Networks", BCP 84, RFC 3704, March 2004. Savola Expires July 26, 2008 [Page 9] Internet-Draft Unicast RPF Experiences January 2008 8.2. Informative References [I-D.huitema-shim6-ingress-filtering] Huitema, C., "Ingress filtering compatibility for IPv6 multihomed sites", draft-huitema-shim6-ingress-filtering-00 (work in progress), September 2005. [I-D.ietf-dnsop-reflectors-are-evil] Damas, J. and F. Neves, "Preventing Use of Recursive Nameservers in Reflector Attacks", draft-ietf-dnsop-reflectors-are-evil-05 (work in progress), December 2007. [I-D.ietf-ipngwg-p2p-pingpong] Hagino, J., JINMEI, T., and B. Zill, "Avoiding ping-pong packets on point-to-point links", draft-ietf-ipngwg-p2p-pingpong-00 (work in progress), July 2001. [PRIVIP] NANOG mailing-list thread, "private IP addresses from ISP", May 2006, . [SPOOFER] MIT ANA, "Spoofer Project", . Author's Address Pekka Savola CSC/FUNET Espoo Finland Email: psavola@funet.fi Savola Expires July 26, 2008 [Page 10] Internet-Draft Unicast RPF Experiences January 2008 Full Copyright Statement Copyright (C) The IETF Trust (2008). This document is subject to the rights, licenses and restrictions contained in BCP 78, and except as set forth therein, the authors retain all their rights. This document and the information contained herein are provided on an "AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Intellectual Property The IETF takes no position regarding the validity or scope of any Intellectual Property Rights or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; nor does it represent that it has made any independent effort to identify any such rights. Information on the procedures with respect to rights in RFC documents can be found in BCP 78 and BCP 79. Copies of IPR disclosures made to the IETF Secretariat and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the IETF on-line IPR repository at http://www.ietf.org/ipr. The IETF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights that may cover technology that may be required to implement this standard. Please address the information to the IETF at ietf-ipr@ietf.org. Acknowledgment Funding for the RFC Editor function is provided by the IETF Administrative Support Activity (IASA). Savola Expires July 26, 2008 [Page 11]