Very Strange Network Problem HELP!!!

Discussion in 'Cisco' started by Craig N., Jul 16, 2004.

  1. Craig N.

    Craig N. Guest

    I have a very strange problem here, been working on it for 4 months now,
    hopefully someone can give me some advice.

    Anyways, I have a client with 200 users running Citrix. The network
    configuration is:

    all main servers are HP DL380's, Dual xeon 3.0's, 1 gig of ram
    2 domain controllers - win'2000
    4 Citrix servers
    1 SQL server
    1 Print Server
    1 EMC Clarian SAN
    1 Lotus
    1 HP-UX running BPCS
    Switches:
    4 Intel Express Hubs 12 Port
    2 3COM hubs
    2 HP 2524 Switches
    2 Cisco Catalyst 2900 Switches
    1 Cisco Catalyst 3500 Switch
    4 Old SMC Hubs for the phone system

    Anyways, the client was running old servers, so they upgraded to HP DL380
    servers, and Citrix ran great, then a few days later it went to crap. The
    logins were slow, trying to open office documents through office was slow
    (but opening through explorer was not), and when typing, it would lag real
    bad. (This is Citrix Specific)

    I came in to try to fix it, and we tried everything until we finally rebuilt
    the domain controllers and Citrix farm, since the problem really seemed like
    a server issue. Well, after rebuilding from scratch, we stil had the same
    problem, so we tested everything, and it looked good. I then got an idea, it
    felt like it might be network traffic.

    I took a catalyst 2900 switch that had never been connected to the network,
    and ran the servers to it, and 6 thin clients. We turned it on, and the
    thing cruised. It rocked, logins were instant, it was the first time it
    worked in 4 months. So, I figured I would track down the problem, I started
    plugging in switches one at a time until the problem surfaced. I got all the
    Intel hubs connected, nothing, then I connected the 3coms, still nothing,
    and then the HP switches, so far so good.

    By now, its 1:00 AM, and I decide to go home, I quickly connect in the rest
    of the switches at once, and BOOM, the problem shows up, so I figure I have
    it down to only a few switches, lets go home and try tomorrow. I move the
    network cables over to the old servers, then I Connect this switch into the
    network, and leave. (It was easier, than moving cables everywhere, since the
    switch room is seperate from the server room, and I was tired)

    Next Day, I'm excited, so I decide to show it off. WELL, I move my network
    cables to the old servers, and unplug the switch from the rest of the
    network. The problem is back!!! I reboot the switch, I reboot the servers,
    nothing. I even swap network cables. This is the exact same configuration as
    last night when it was running great. I dont get it. I guess maybe something
    from one cisco switch has infected this one, who knows.

    I do know that the HP switches didnt cause a problem, so I empty them out,
    and plug in only my servers and thin clients, nothing else, completely
    seperate from the network. STILL THE SAME PROBLEM!!

    This is weird, and I need to fix it. Its like whenever you add a switch, it
    acquires this problem, but a brand new switch runs great until it is
    connected to the network for any length of time. It ran great on the other
    switch for a good 5 hours, so I know its not the servers, gotta be the
    switch. Does anyone have any ideas?

    I am resetting my catslyst to factory default then going to try it. Someone
    please help, I need this up by monday.
    Craig N., Jul 16, 2004
    #1
    1. Advertising

  2. Craig N.

    PES Guest

    I would make absolutely sure that there were no smb based worms like blaster
    on the network. I would then make sure that there are no duplex mismatches.
    If a pc/server card is set to auto duplex, make sure the corresponding
    switch port is auto. If duplex is forced full on either end (but not both)
    you will have a duplex mismatch. At that point, I would use a packet
    sniffer to troubleshoot the problem. If you've not done quite a bit of
    this, it would be worth your money to hire someone who looks at packets on a
    daily (or at least weekly basis).

    "Craig N." <> wrote in message
    news:aRJJc.262249$...
    > I have a very strange problem here, been working on it for 4 months now,
    > hopefully someone can give me some advice.
    >
    > Anyways, I have a client with 200 users running Citrix. The network
    > configuration is:
    >
    > all main servers are HP DL380's, Dual xeon 3.0's, 1 gig of ram
    > 2 domain controllers - win'2000
    > 4 Citrix servers
    > 1 SQL server
    > 1 Print Server
    > 1 EMC Clarian SAN
    > 1 Lotus
    > 1 HP-UX running BPCS
    > Switches:
    > 4 Intel Express Hubs 12 Port
    > 2 3COM hubs
    > 2 HP 2524 Switches
    > 2 Cisco Catalyst 2900 Switches
    > 1 Cisco Catalyst 3500 Switch
    > 4 Old SMC Hubs for the phone system
    >
    > Anyways, the client was running old servers, so they upgraded to HP DL380
    > servers, and Citrix ran great, then a few days later it went to crap. The
    > logins were slow, trying to open office documents through office was slow
    > (but opening through explorer was not), and when typing, it would lag real
    > bad. (This is Citrix Specific)
    >
    > I came in to try to fix it, and we tried everything until we finally

    rebuilt
    > the domain controllers and Citrix farm, since the problem really seemed

    like
    > a server issue. Well, after rebuilding from scratch, we stil had the same
    > problem, so we tested everything, and it looked good. I then got an idea,

    it
    > felt like it might be network traffic.
    >
    > I took a catalyst 2900 switch that had never been connected to the

    network,
    > and ran the servers to it, and 6 thin clients. We turned it on, and the
    > thing cruised. It rocked, logins were instant, it was the first time it
    > worked in 4 months. So, I figured I would track down the problem, I

    started
    > plugging in switches one at a time until the problem surfaced. I got all

    the
    > Intel hubs connected, nothing, then I connected the 3coms, still nothing,
    > and then the HP switches, so far so good.
    >
    > By now, its 1:00 AM, and I decide to go home, I quickly connect in the

    rest
    > of the switches at once, and BOOM, the problem shows up, so I figure I

    have
    > it down to only a few switches, lets go home and try tomorrow. I move the
    > network cables over to the old servers, then I Connect this switch into

    the
    > network, and leave. (It was easier, than moving cables everywhere, since

    the
    > switch room is seperate from the server room, and I was tired)
    >
    > Next Day, I'm excited, so I decide to show it off. WELL, I move my network
    > cables to the old servers, and unplug the switch from the rest of the
    > network. The problem is back!!! I reboot the switch, I reboot the servers,
    > nothing. I even swap network cables. This is the exact same configuration

    as
    > last night when it was running great. I dont get it. I guess maybe

    something
    > from one cisco switch has infected this one, who knows.
    >
    > I do know that the HP switches didnt cause a problem, so I empty them out,
    > and plug in only my servers and thin clients, nothing else, completely
    > seperate from the network. STILL THE SAME PROBLEM!!
    >
    > This is weird, and I need to fix it. Its like whenever you add a switch,

    it
    > acquires this problem, but a brand new switch runs great until it is
    > connected to the network for any length of time. It ran great on the other
    > switch for a good 5 hours, so I know its not the servers, gotta be the
    > switch. Does anyone have any ideas?
    >
    > I am resetting my catslyst to factory default then going to try it.

    Someone
    > please help, I need this up by monday.
    >
    >
    PES, Jul 16, 2004
    #2
    1. Advertising

  3. Craig N.

    Craig N. Guest

    Thanks, I ran a sniffer, and I had 0 netowkr connectivity during the system
    hang. Virus scans show no virus activity, but I will keep it in mind.


    "PES" <NO*SPAMpestewartREMOVE**SUCKS> wrote in message
    news:40f7c755$...
    >I would make absolutely sure that there were no smb based worms like
    >blaster
    > on the network. I would then make sure that there are no duplex
    > mismatches.
    > If a pc/server card is set to auto duplex, make sure the corresponding
    > switch port is auto. If duplex is forced full on either end (but not
    > both)
    > you will have a duplex mismatch. At that point, I would use a packet
    > sniffer to troubleshoot the problem. If you've not done quite a bit of
    > this, it would be worth your money to hire someone who looks at packets on
    > a
    > daily (or at least weekly basis).
    >
    > "Craig N." <> wrote in message
    > news:aRJJc.262249$...
    >> I have a very strange problem here, been working on it for 4 months now,
    >> hopefully someone can give me some advice.
    >>
    >> Anyways, I have a client with 200 users running Citrix. The network
    >> configuration is:
    >>
    >> all main servers are HP DL380's, Dual xeon 3.0's, 1 gig of ram
    >> 2 domain controllers - win'2000
    >> 4 Citrix servers
    >> 1 SQL server
    >> 1 Print Server
    >> 1 EMC Clarian SAN
    >> 1 Lotus
    >> 1 HP-UX running BPCS
    >> Switches:
    >> 4 Intel Express Hubs 12 Port
    >> 2 3COM hubs
    >> 2 HP 2524 Switches
    >> 2 Cisco Catalyst 2900 Switches
    >> 1 Cisco Catalyst 3500 Switch
    >> 4 Old SMC Hubs for the phone system
    >>
    >> Anyways, the client was running old servers, so they upgraded to HP DL380
    >> servers, and Citrix ran great, then a few days later it went to crap. The
    >> logins were slow, trying to open office documents through office was slow
    >> (but opening through explorer was not), and when typing, it would lag
    >> real
    >> bad. (This is Citrix Specific)
    >>
    >> I came in to try to fix it, and we tried everything until we finally

    > rebuilt
    >> the domain controllers and Citrix farm, since the problem really seemed

    > like
    >> a server issue. Well, after rebuilding from scratch, we stil had the same
    >> problem, so we tested everything, and it looked good. I then got an idea,

    > it
    >> felt like it might be network traffic.
    >>
    >> I took a catalyst 2900 switch that had never been connected to the

    > network,
    >> and ran the servers to it, and 6 thin clients. We turned it on, and the
    >> thing cruised. It rocked, logins were instant, it was the first time it
    >> worked in 4 months. So, I figured I would track down the problem, I

    > started
    >> plugging in switches one at a time until the problem surfaced. I got all

    > the
    >> Intel hubs connected, nothing, then I connected the 3coms, still nothing,
    >> and then the HP switches, so far so good.
    >>
    >> By now, its 1:00 AM, and I decide to go home, I quickly connect in the

    > rest
    >> of the switches at once, and BOOM, the problem shows up, so I figure I

    > have
    >> it down to only a few switches, lets go home and try tomorrow. I move the
    >> network cables over to the old servers, then I Connect this switch into

    > the
    >> network, and leave. (It was easier, than moving cables everywhere, since

    > the
    >> switch room is seperate from the server room, and I was tired)
    >>
    >> Next Day, I'm excited, so I decide to show it off. WELL, I move my
    >> network
    >> cables to the old servers, and unplug the switch from the rest of the
    >> network. The problem is back!!! I reboot the switch, I reboot the
    >> servers,
    >> nothing. I even swap network cables. This is the exact same configuration

    > as
    >> last night when it was running great. I dont get it. I guess maybe

    > something
    >> from one cisco switch has infected this one, who knows.
    >>
    >> I do know that the HP switches didnt cause a problem, so I empty them
    >> out,
    >> and plug in only my servers and thin clients, nothing else, completely
    >> seperate from the network. STILL THE SAME PROBLEM!!
    >>
    >> This is weird, and I need to fix it. Its like whenever you add a switch,

    > it
    >> acquires this problem, but a brand new switch runs great until it is
    >> connected to the network for any length of time. It ran great on the
    >> other
    >> switch for a good 5 hours, so I know its not the servers, gotta be the
    >> switch. Does anyone have any ideas?
    >>
    >> I am resetting my catslyst to factory default then going to try it.

    > Someone
    >> please help, I need this up by monday.
    >>
    >>

    >
    >
    Craig N., Jul 16, 2004
    #3
  4. Craig N.

    Craig N. Guest

    Sniffer Output

    Here are a few things I caught using ethereal, if anyone can tell me what it
    means. Everything looks like normal traffic, except from one PC, and the
    citrix boxes.I can export the file to text and e-mail it if anyone wants
    some more detail, just email me at .

    This is only half of it, but you get the idea, out of hundreds of pc's amd
    about 15 random servers, these are the only ones doing this particulaar
    thing.

    Source Destination
    Info

    Colleen-pc.company.int 192,168.102.6 TCP
    3480 > 5321 [PSH, ACK] Seq=1 Ack=0 Win=54512 Len=120.
    Colleen-pc.company.int 192,168.102.6 TCP
    3480 > 5321 [ACK] Seq=1 Ack=0 Win=54512 Len=120
    Colleen-pc.company.int 192,168.102.6 TCP
    3480 > 5321 [SYN] Seq=0 Ack=0 Win=54512 Len=120 MSS=1460
    Colleen-pc.company.int 192.168.102.14 TCP
    3479 > 1352 [ACK] Seq=1 Ack=0 Win=64512 Len=0
    Colleen-pc.company.int 192.168.102.14 TCP
    3479 > 1352 [SYN] Seq=0 Ack=0 Win=64512 Len=0 MSS=1460
    ---------------
    Then on Citrix, I have a bunch of these, on all the servers:

    Cxp03.company.int 192.168.102.150 TCP
    1494 > 1041 [ACK] Seq=0 Ack=0 Win=63412 Len=0
    -----------------------------------
    Along with a LOT of these:

    Cxp03.company.int 192.168.102.150 TCP
    [TCP Previous segment lost] 1494 > 1041 [PSH, ACK] Seq=121622 Ack=4049
    Win=63783 Len=1459

    Cxp03.company.int 192.168.102.150 TCP
    [TCP Previous segment lost] 1494 > 1041 [PSH, ACK] Seq=2045131 Ack=22451
    Win=63783 Len=1459
    Craig N., Jul 16, 2004
    #4
  5. Craig N.

    PES Guest

    Re: Sniffer Output

    "Craig N." <> wrote in message
    news:GxYJc.102237$...
    > Here are a few things I caught using ethereal, if anyone can tell me what

    it
    > means. Everything looks like normal traffic, except from one PC, and the
    > citrix boxes.I can export the file to text and e-mail it if anyone wants
    > some more detail, just email me at .
    >
    > This is only half of it, but you get the idea, out of hundreds of pc's amd
    > about 15 random servers, these are the only ones doing this particulaar
    > thing.
    >
    > Source Destination
    > Info



    May be Firehotcker (a trojan). I'm not familiar with it.

    > Colleen-pc.company.int 192,168.102.6

    TCP
    > 3480 > 5321 [PSH, ACK] Seq=1 Ack=0 Win=54512 Len=120.
    > Colleen-pc.company.int 192,168.102.6

    TCP
    > 3480 > 5321 [ACK] Seq=1 Ack=0 Win=54512 Len=120
    > Colleen-pc.company.int 192,168.102.6

    TCP
    > 3480 > 5321 [SYN] Seq=0 Ack=0 Win=54512 Len=120 MSS=1460


    May be Lotus Notes

    > Colleen-pc.company.int 192.168.102.14

    TCP
    > 3479 > 1352 [ACK] Seq=1 Ack=0 Win=64512 Len=0
    > Colleen-pc.company.int 192.168.102.14

    TCP
    > 3479 > 1352 [SYN] Seq=0 Ack=0 Win=64512 Len=0 MSS=1460
    > ---------------
    > Then on Citrix, I have a bunch of these, on all the servers:


    Normal for an established citrix session

    > Cxp03.company.int 192.168.102.150

    TCP
    > 1494 > 1041 [ACK] Seq=0 Ack=0 Win=63412 Len=0
    > -----------------------------------
    > Along with a LOT of these:



    Why art the tcp segments lost? Could you not have the same sniffing
    capacity as the server has connectivity? Or could the network be dropping
    packets? Also, I'm not sure, but a length of 1459 seems large for citrix.
    Is there a router between cxp03.company.int and 192.168.102.150? If so, is
    ip unreachables enabled? How about duplex settings on the the switchport
    that cxp03 and 192.168.102.150 is on. Input or output errors on the switch
    ports?

    > Cxp03.company.int 192.168.102.150

    TCP
    > [TCP Previous segment lost] 1494 > 1041 [PSH, ACK] Seq=121622 Ack=4049
    > Win=63783 Len=1459
    >
    > Cxp03.company.int 192.168.102.150

    TCP
    > [TCP Previous segment lost] 1494 > 1041 [PSH, ACK] Seq=2045131 Ack=22451
    > Win=63783 Len=1459
    >
    >
    PES, Jul 17, 2004
    #5
  6. Craig N.

    Craig N. Guest

    Problem seems to be fixed. This is very very strange. It is either the HP
    Lights out board, or, it is the Kernel that HP installs if you use the HP
    install disks. We disabled the board, then reinstalled the server with an
    actual Win2k disk, and I cant recreate the problem. Of course, I had it
    working once before and well that went to crap.

    Anyways, I still dont get why it was working great the other day on that one
    single switch, and bad when I plugged in the others. BUT, it works now
    plugged into all switches.

    Thanks for your help.


    "Craig N." <> wrote in message
    news:EUWJc.265975$...
    > Thanks, I ran a sniffer, and I had 0 netowkr connectivity during the
    > system hang. Virus scans show no virus activity, but I will keep it in
    > mind.
    >
    >
    > "PES" <NO*SPAMpestewartREMOVE**SUCKS> wrote in
    > message news:40f7c755$...
    >>I would make absolutely sure that there were no smb based worms like
    >>blaster
    >> on the network. I would then make sure that there are no duplex
    >> mismatches.
    >> If a pc/server card is set to auto duplex, make sure the corresponding
    >> switch port is auto. If duplex is forced full on either end (but not
    >> both)
    >> you will have a duplex mismatch. At that point, I would use a packet
    >> sniffer to troubleshoot the problem. If you've not done quite a bit of
    >> this, it would be worth your money to hire someone who looks at packets
    >> on a
    >> daily (or at least weekly basis).
    >>
    >> "Craig N." <> wrote in message
    >> news:aRJJc.262249$...
    >>> I have a very strange problem here, been working on it for 4 months now,
    >>> hopefully someone can give me some advice.
    >>>
    >>> Anyways, I have a client with 200 users running Citrix. The network
    >>> configuration is:
    >>>
    >>> all main servers are HP DL380's, Dual xeon 3.0's, 1 gig of ram
    >>> 2 domain controllers - win'2000
    >>> 4 Citrix servers
    >>> 1 SQL server
    >>> 1 Print Server
    >>> 1 EMC Clarian SAN
    >>> 1 Lotus
    >>> 1 HP-UX running BPCS
    >>> Switches:
    >>> 4 Intel Express Hubs 12 Port
    >>> 2 3COM hubs
    >>> 2 HP 2524 Switches
    >>> 2 Cisco Catalyst 2900 Switches
    >>> 1 Cisco Catalyst 3500 Switch
    >>> 4 Old SMC Hubs for the phone system
    >>>
    >>> Anyways, the client was running old servers, so they upgraded to HP
    >>> DL380
    >>> servers, and Citrix ran great, then a few days later it went to crap.
    >>> The
    >>> logins were slow, trying to open office documents through office was
    >>> slow
    >>> (but opening through explorer was not), and when typing, it would lag
    >>> real
    >>> bad. (This is Citrix Specific)
    >>>
    >>> I came in to try to fix it, and we tried everything until we finally

    >> rebuilt
    >>> the domain controllers and Citrix farm, since the problem really seemed

    >> like
    >>> a server issue. Well, after rebuilding from scratch, we stil had the
    >>> same
    >>> problem, so we tested everything, and it looked good. I then got an
    >>> idea,

    >> it
    >>> felt like it might be network traffic.
    >>>
    >>> I took a catalyst 2900 switch that had never been connected to the

    >> network,
    >>> and ran the servers to it, and 6 thin clients. We turned it on, and the
    >>> thing cruised. It rocked, logins were instant, it was the first time it
    >>> worked in 4 months. So, I figured I would track down the problem, I

    >> started
    >>> plugging in switches one at a time until the problem surfaced. I got all

    >> the
    >>> Intel hubs connected, nothing, then I connected the 3coms, still
    >>> nothing,
    >>> and then the HP switches, so far so good.
    >>>
    >>> By now, its 1:00 AM, and I decide to go home, I quickly connect in the

    >> rest
    >>> of the switches at once, and BOOM, the problem shows up, so I figure I

    >> have
    >>> it down to only a few switches, lets go home and try tomorrow. I move
    >>> the
    >>> network cables over to the old servers, then I Connect this switch into

    >> the
    >>> network, and leave. (It was easier, than moving cables everywhere, since

    >> the
    >>> switch room is seperate from the server room, and I was tired)
    >>>
    >>> Next Day, I'm excited, so I decide to show it off. WELL, I move my
    >>> network
    >>> cables to the old servers, and unplug the switch from the rest of the
    >>> network. The problem is back!!! I reboot the switch, I reboot the
    >>> servers,
    >>> nothing. I even swap network cables. This is the exact same
    >>> configuration

    >> as
    >>> last night when it was running great. I dont get it. I guess maybe

    >> something
    >>> from one cisco switch has infected this one, who knows.
    >>>
    >>> I do know that the HP switches didnt cause a problem, so I empty them
    >>> out,
    >>> and plug in only my servers and thin clients, nothing else, completely
    >>> seperate from the network. STILL THE SAME PROBLEM!!
    >>>
    >>> This is weird, and I need to fix it. Its like whenever you add a switch,

    >> it
    >>> acquires this problem, but a brand new switch runs great until it is
    >>> connected to the network for any length of time. It ran great on the
    >>> other
    >>> switch for a good 5 hours, so I know its not the servers, gotta be the
    >>> switch. Does anyone have any ideas?
    >>>
    >>> I am resetting my catslyst to factory default then going to try it.

    >> Someone
    >>> please help, I need this up by monday.
    >>>
    >>>

    >>
    >>

    >
    >
    Craig N., Jul 17, 2004
    #6
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. GET OFF

    very strange problem

    GET OFF, Feb 19, 2004, in forum: Computer Support
    Replies:
    7
    Views:
    452
    Harrison
    Feb 19, 2004
  2. Thomas Reed

    Quick Book file access very very very slow

    Thomas Reed, Apr 9, 2004, in forum: Computer Support
    Replies:
    7
    Views:
    732
    Palindrome
    Apr 9, 2004
  3. Ram

    System on/off problem - Very Strange!!

    Ram, Nov 3, 2004, in forum: Computer Support
    Replies:
    3
    Views:
    398
    dadiOH
    Nov 3, 2004
  4. elaich

    Very strange W2K problem

    elaich, Jan 17, 2006, in forum: Computer Support
    Replies:
    1
    Views:
    310
  5. Gordon

    Very strange network problem

    Gordon, Sep 29, 2009, in forum: Wireless Networking
    Replies:
    6
    Views:
    516
    Pavel A.
    Oct 4, 2009
Loading...

Share This Page