Mystery "ghost" MAC addresses?

Discussion in 'Cisco' started by John Oliver, Sep 12, 2008.

  1. John Oliver

    John Oliver Guest

    While troubleshooting some firewall issues, I started to see some
    mystery broadcast traffic:

    15:22:20.192899 02:02:0a:00:0a:c8 (oui Unknown) > Broadcast, ethertype
    Unknown (0x886f), length 1510:
    0x0000: bf01 dec0 0402 0000 0200 0000 0a00 0ac8
    .................
    0x0010: 0a00 0a15 0100 0100 0100 0200 0f01 8542
    ................B
    0x0020: a737 0000 0000 0000 0000 0000 00f0 ff6f
    ..7.............o
    0x0030: 0000 0000 0000 0000 0000 0000 0000 0000
    .................
    0x0040: 0000 0000 0000 0000 0000 0000 0000 0000
    .................
    0x0050: 0000


    I log in to the 2960 that's at the "heart" of this particular network
    and use sh mac-address-table to find that MAC. It says it's on Gi0/6 I
    follow that cable to a Dell 2748, so I log in to it... and it tells me
    that MAC is on port 47, which leads back to the 2960! To make matters
    worse, 02:02:0a is showing up as unassigned.

    I've had issues like this with Cisco switches before, where mysterious
    traffic seems to originate in the middle of wires. And I know others
    have seen the same issue, because once in a while I get an email from
    someone who saw an old post I made about it, asking if I ever found an
    answer because they're tearing their hair out!

    --
    * John Oliver http://www.john-oliver.net/ *
     
    John Oliver, Sep 12, 2008
    #1
    1. Advertising

  2. John Oliver

    Al Guest

    On Sep 13, 4:44 am, "Thrill5" <> wrote:
    > First thoughts....
    >
    > The fact that the source MAC is "weird", the traffic is a broadcast, and is
    > a full length packet (1510 bytes) leads me to believe that you have a device
    > on your network that has a virus.
    >
    > I did work on a problem more than 12 years ago where we had bridges
    > (CrossComm ILAN's) that replicated a randomly generated packet, but this was
    > due to a bug in the bridge software.  I have been working with Cisco
    > switches and routers for more than 10 years and I have never seen or heard
    > of a problem like this with Cisco equipment.  I would suspect the "other"
    > vendors equipment before Cisco.
    >
    > "John Oliver" <> wrote in message
    >
    > news:...
    >
    > > While troubleshooting some firewall issues, I started to see some
    > > mystery broadcast traffic:

    >
    > > 15:22:20.192899 02:02:0a:00:0a:c8 (oui Unknown) > Broadcast, ethertype
    > > Unknown (0x886f), length 1510:
    > >        0x0000:  bf01 dec0 0402 0000 0200 0000 0a00 0ac8
    > > ................
    > >        0x0010:  0a00 0a15 0100 0100 0100 0200 0f01 8542
    > > ...............B
    > >        0x0020:  a737 0000 0000 0000 0000 0000 00f0 ff6f
    > > .7.............o
    > >        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000
    > > ................
    > >        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000
    > > ................
    > >        0x0050:  0000

    >
    > > I log in to the 2960 that's at the "heart" of this particular network
    > > and use sh mac-address-table to find that MAC.  It says it's on Gi0/6  I
    > > follow that cable to a Dell 2748, so I log in to it... and it tells me
    > > that MAC is on port 47, which leads back to the 2960!  To make matters
    > > worse, 02:02:0a is showing up as unassigned.

    >
    > > I've had issues like this with Cisco switches before, where mysterious
    > > traffic seems to originate in the middle of wires.  And I know others
    > > have seen the same issue, because once in a while I get an email from
    > > someone who saw an old post I made about it, asking if I ever found an
    > > answer because they're tearing their hair out!

    >
    > > --
    > > * John Oliver                              http://www.john-oliver.net/*


    One time I saw some similar traffic it turned out to be to do with
    network load balancing enabled on two NICs on two different servers.
    The core switch started reporting a flapping mac-address & sure
    enough, 1 second it would be on one uplink - to server A - and the
    next it would be on another - to server B. As neither server had dual
    NICs we just disabled the feature as it was unnecessary.
    Unfortunately, I do not remember the MAC address & whether it was b/
    cast or other. All I remember is that it wasn't a registered OUI & I
    couldn't find anything about it online...
     
    Al, Sep 13, 2008
    #2
    1. Advertising

  3. Well, let try this as a wild guess: if the switch itself emitted a malformed
    packet due to a bug, this could be the problem. The packet is pretty
    clearly malformed, looking only at the ethertype. And it is a maximum sized
    packet, which is suspicious for a broadcast that is not part of a stream.
    Some switch seems to have a pointer to memory which is hoarked.

    Let's presume it came from the Dell for the moment. The Cisco switch will
    flood it (since it is a broadcast) and mark the interface where it heard the
    broadcast (Gi0/6) as the source. The packet will flood and the Dell will
    ultimately hear the packet back on that same port (depending on the state of
    the spanning tree or nature of the bug that caused this packet), and it will
    mark the interface where it heard it from. Since it has no way to know that
    it generated this malformed packet itself, it adds it to the MAC address
    forwaring table, presuming it must be a distant host.

    This is how you get a "synthesized" packet in the middle of the wire - One
    of the two switches emitted a malformed broadcast packet and did not
    recognize the packet as its own when it came back.

    The partial hex dump of the packet has no obvious clues (to me) about what
    it is. Sometimes, these are clearly legitimate packets shifted by a number
    of bytes. A more motivated person might play around with bitshifting the
    hex dump or looking at a more complete hex dump to see if any recognizable
    patterns emerge. But as near as I can tell, this is probably some kind of
    bug in one of the switches, and it seems harmless as long as these emissions
    are rare.

    Are the MAC address entries timestamped? If they are timestamped with high
    precision and the switches are synchronized, the one with the oldest MAC
    address entry is probably the innocent one (since that switch heard the
    impossible packet first).

    Does the packet appear regularly? If you can somehow diagnose which side
    "shoots first" you will be closer to understanding where it comes from.
     
    Phillip Remaker, Sep 14, 2008
    #3
  4. John Oliver

    Stephen Guest

    On Sat, 13 Sep 2008 00:26:53 +0200 (CEST), John Oliver
    <> wrote:

    >While troubleshooting some firewall issues, I started to see some
    >mystery broadcast traffic:
    >
    >15:22:20.192899 02:02:0a:00:0a:c8 (oui Unknown) > Broadcast, ethertype
    >Unknown (0x886f), length 1510:
    > 0x0000: bf01 dec0 0402 0000 0200 0000 0a00 0ac8
    >................
    > 0x0010: 0a00 0a15 0100 0100 0100 0200 0f01 8542
    >...............B
    > 0x0020: a737 0000 0000 0000 0000 0000 00f0 ff6f
    >.7.............o
    > 0x0030: 0000 0000 0000 0000 0000 0000 0000 0000
    >................
    > 0x0040: 0000 0000 0000 0000 0000 0000 0000 0000
    >................
    > 0x0050: 0000
    >
    >
    >I log in to the 2960 that's at the "heart" of this particular network
    >and use sh mac-address-table to find that MAC. It says it's on Gi0/6 I
    >follow that cable to a Dell 2748, so I log in to it... and it tells me
    >that MAC is on port 47, which leads back to the 2960! To make matters
    >worse, 02:02:0a is showing up as unassigned.


    that Ethertype 886f "belongs" to Microsoft.
    AFAIR it is something to do with server load balancing.
    http://standards.ieee.org/regauth/ethertype/type-pub.html

    i think when this is used the server uses a different source address -
    but it makes it up from the NIC address.

    Look to see if 1 of the source adr MACs on the server is the same
    apart from a couple of the high order bits.

    >
    >I've had issues like this with Cisco switches before, where mysterious
    >traffic seems to originate in the middle of wires. And I know others
    >have seen the same issue, because once in a while I get an email from
    >someone who saw an old post I made about it, asking if I ever found an
    >answer because they're tearing their hair out!

    --
    Regards

    - replace xyz with ntl
     
    Stephen, Sep 14, 2008
    #4
  5. > that Ethertype 886f "belongs" to Microsoft.
    > AFAIR it is something to do with server load balancing.
    > http://standards.ieee.org/regauth/ethertype/type-pub.html



    Well, I guess you can disregard my last post, then! :)

    With your clue, I Googled some more: Google "0x886f load balancing"

    Protocol 0x886f is described at
    http://www.iss.net/security_center/advice/Exploits/Ports/Ethertype/0x886f/default.htm

    Which says:

    Microsoft's Windows NT Load Balancing Service (WLBS), a clustering service.
    Packets sent using this Ethertype have the following properties:

    a.. The destination MAC address is either the broadcast (FFFFFFFFFFFF) or
    a multicast address that is set in the registry.
    b.. The source MAC address is 00 BF followed by the IP address of the
    machine sending the packet. That machine also has a corresponding MAC
    address of 02 BF ... that you can use to ping the machine with.
    c.. Status messages appear about once per second per machine in the
    cluster.

    so from the original post it looks like it belongs to a server with an IP
    address 0A:00:0A:C8 or 10.0.10.200
     
    Phillip Remaker, Sep 14, 2008
    #5
  6. John Oliver

    Sam Wilson Guest

    In article <>,
    John Oliver <> wrote:

    > While troubleshooting some firewall issues, I started to see some
    > mystery broadcast traffic:
    >
    > 15:22:20.192899 02:02:0a:00:0a:c8 (oui Unknown) > Broadcast, ethertype
    > Unknown (0x886f), length 1510:
    > [snip]
    >
    > ... To make matters
    > worse, 02:02:0a is showing up as unassigned.


    One snippet that hasn't come up in the thread so far is that the second
    least significant bit (0x02) of the first octet of a MAC address is the
    "locally administered bit". That means that any MAC address with that
    bit set is (supposed) not to be globally assigned but generated locally.
    Except for a few historical oddities any address with that bit set is
    not a universally registered OUI.

    Sam
     
    Sam Wilson, Sep 16, 2008
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. kl
    Replies:
    2
    Views:
    4,432
    aladin
    Jul 19, 2004
  2. Arunkumar
    Replies:
    2
    Views:
    1,442
    Dave Keith
    Jan 14, 2004
  3. Han Valk
    Replies:
    3
    Views:
    688
    M.C. van den Bovenkamp
    Jan 27, 2004
  4. Replies:
    0
    Views:
    940
  5. van
    Replies:
    2
    Views:
    4,686
    Barry Watzman
    Oct 14, 2003
Loading...

Share This Page