IP Cam-Specific Radio Link Problem?

Discussion in 'Wireless Internet' started by (PeteCresswell), Jan 3, 2015.

  1. I posted this on comp.os.ms-windows.networking.tcp-ip
    and somebody suggested taking it here... so here goes...
    -----------------------------------------------------------

    Just finished 3 weeks of banging my head against the wall.

    Windows 7 PC running an IP cam server at location A, radio link to
    location B where 4 IP cams are running.

    Network Diagram: http://tinyurl.com/mkfmvcx

    One day, 3 of the 4 cams went down as far as the server was concerned.

    The 4th cam, no problem.... ever. Yes, the 3 problem cams are the same
    make/model (HikVision DS-2CD2032-I) and the "Good" cam is a different
    make/model (TrendNet TV-IP672P). Both claim to adhere to the same POE
    standard ("Active 802.3af POE"... whatever that implies) and one of the
    3 problem cams is located within 18" of the "Good" cam on with the Cat5e
    cable the same length.

    Here's a screen snap of all 4 cams under Ping -t The "Good" cam being
    in the center: http://tinyurl.com/ouqrd95

    Go to the remote location, pull the Cat5e cable to the radio link,
    connect a laptop, start the 4 Ping -t commands, and all 4 cams are
    running a-ok.... the Replys just keep rolling off the screen.


    To cut to the chase, after a loooong time of assuming that no radio link
    problem could be camera-specific, I re-booted both radios and the
    problem went away. And, just to put a cap on it, sometime later I was
    admiring my working cams when all 3 problem cams went offline
    simultaneously.... again, re-booted the radio links and they immediately
    came back.

    Soooo.... my assumption is that it's the radio link.


    The Question:

    Can anybody postulate how 3 cams can have a problem and one cam can be
    running a-ok the whole time?

    Or is there a flaw in my logic/assumption?
     
    (PeteCresswell), Jan 3, 2015
    #1
    1. Advertisements

  2. (PeteCresswell)

    John Navas Guest

    Troubleshoot the link. Is there transparent bridging at both ends?
    Use your laptop to run Wireshark. <https://www.wireshark.org/>
    Verify that pings to the problematic IP addresses are making it over
    the link.
     
    John Navas, Jan 3, 2015
    #2
    1. Advertisements

  3. I'll trade you. I burned some quality time last night trying to
    figure out why a previously working laptop decided to play dead after
    I tore it apart to replace some broken plastic parts. It took an hour
    to notice I had forgotten to reinstall the memory chips.
    Nice Visio drawing. Do you have a version with all the device details
    included viewable with IE (active-x required)?
    I need a little more detail.
    1. Which camera (by IP or location) is the one that's working?
    2. Are all the devices showing IP addresses setup for a static IP
    address or are they dynamic? If static, what IP are you using for the
    default route?
    3. What other devices are on the 10.0.0.xxx LAN that might have a
    built in DHCP server?
    4. Are the Nanostation M5 radios setup as a bridge or router? If
    router, do you have the DHCP servers at both ends turned off?
    5. What is the maker and model of the "Extreme West" WAP? If it's a
    router being used as an access point, is the DHCP server turned off?
    6. What's the maker and model of the PoE switch? Is it a managed
    switch with an IP address?
    7. What's the maker and model of the Comcast modem/router/wifi box?
    Some of these are little better than garbage.
    Sure. I see it all the time. It's easy to use a wireless router as a
    bridge or access point. However, things go directly to hell if you
    have more than one DHCP server running on the system. Make sure every
    box has the DHCP server turned off, except the Comcast cable
    modem/router/wireless box. Rebooting the Nanostations to fix the
    problem suggests that one of them is a possible source of the extra
    DHCP server.
    Dunno. I always ignore the customers diagnostics because they act as
    a diversion. I look at what you have, what you've done, and draw my
    own conclusions.

    What I see is a mess. Never mind the obvious environmental issues.
    You have everyone at Toledo Ave and the windsurfing shack all on the
    same network. You're not going to get anywhere with this unless you
    can isolate each section and test them individually. Tear it apart,
    and test each segment individually, starting at the Comcast router.
    I'm not a big fan of running camera servers on the same Class C IP
    block as the "productivity" machines, but this is not the time to make
    the topology more complicated. Still, does the PoE switch support a
    VLAN?

    However, it's always fun to take a wild guess. I'll guess(tm) that if
    you disconnect the shop PC, shop owners laptops, Extreme West WAP, and
    any other devices that might have snuck onto the shop LAN, things will
    magically work as expected.
     
    Jeff Liebermann, Jan 3, 2015
    #3
  4. Something is missing, methinks. It seems odd that the owner of the
    surf shop would use your wi-fi bridge to connect his store and
    customers laptops too the internet. More likely, he had a previous
    internet connection, which is now replaced by the wireless link and
    your cable modem connection. Any chance that the previous router is
    still present and plugged in the network somewhere?

    The reason I suspect a 2nd router is that I made the same mistake once
    (and only once). I bridged two locations that previously had
    impendent connections to the internet. Since the "remote office" then
    didn't require its own internet connection, I just turned off the
    power to the router, which was not needed. I left it and the DSL
    modem connected because I wanted a fall back just in case something
    went wrong with the wireless link. After a month, the plan was to
    remove the router permanently.

    That worked fine until someone noticed that the router was powered
    off, and turned it back on. Now, I had two DHCP servers in the
    network. It might have worked because both DHCP servers were
    providing IP addresses in the same IP block. However, both routers
    used the same IP address, and therefore, the same default IP address
    for the client computers default route, but different MAC addresses.
    Eventually, ARP requests ended up going to the wrong router and the
    route to the internet was lost for some (not all) machines. I tried
    to fix it by remote control, so I couldn't tell that the 2nd router
    was running. However, I did notice that:
    arp -a
    was returning the right IP address, but sometimes the wrong associated
    MAC address. Even so, it then took a few hours and 2 cups of coffee
    to figure out the cause.

    Anyway, check for a decommissioned router at the surf shop.
     
    Jeff Liebermann, Jan 3, 2015
    #4
  5. Per John Navas:
    Installed Wireshark on the server PC and on the remote PC in the shack
    with the cams.

    Now I am climbing the learning curve vis-a-vis WireShark filtering.

    Am seeing a lot of 224.0.0.251, which seems to be something called
    "Multicast"... but not yet able to see the activity from Ping -T
    10.0.0.145....

    It's going to be awhile.... but that's ok because everything is working
    right now and I just need to be ready if/when it stops working again...
     
    (PeteCresswell), Jan 3, 2015
    #5
  6. Per Jeff Liebermann:
    No, but that sounds like a feature of Visio that I should look in to -
    because the drawing is already a bit cluttered. Assuming that those
    details are behind hyperlinks..

    The cam that is working is 10.0.0.133.

    All cams have static IP addresses - set both within the cam
    and with entries in the router's Static IP table.

    I do not know what "Default route" is.
    Will look it up and get back to you.

    The camera server PC and the troubleshooting PC
    at the shack.

    The M5's are in bridge mode. On the "NETWORK" tab of each
    M5 there is a "Management Network Settings | Management
    IP Address:" which is set to "Static" on both (as opposed
    to "DHCP") and below that the IP Address, NetMask, and so-forth
    are all filled in.

    The WAP is a D-Link DAP-1522. There is a rocker
    switch on it with choices "AP", "AUTO", and "BRIDGE".
    Hopefully it is set to "AP", but I'm not down there
    so could not swear to it. It does not show up
    when I run NetScan... FWIW.

    El-cheapo, consumer-grade TrendNet TPE-S44. No IP addr, not
    managed.
    Arris TG862G

    (lotta stuff snipped)
    It's easier than that: unplug the M5 at the shop, plug in a laptop, and
    all is well - as evidenced by a Ping -t command for each cam.

    Since it is off-season, the actual network is probably a lot simpler
    than the diagram implies: just the Comcast box, the cam server, the shop
    PC, switch, WAP, and the cams.

    What I have so far is that one of the prime suspects is some device that
    is spontaneously going in-and-out of DHCP mode and that the M5's are
    high on that list.

    Next time things go South, I will try to resist the temptation to just
    remotely reset the M5's and, instead, drive down there and start
    unplugging things to see what changes/when.
     
    (PeteCresswell), Jan 3, 2015
    #6
  7. Per (PeteCresswell):
    Oops... typo.... sb ...143
     
    (PeteCresswell), Jan 3, 2015
    #7
  8. I must be slipping. The IP address is all over the screen and I
    didn't notice:
    <https://plus.google.com/photos/108149798664924808733/albums/5781878959793669377/6093934978803039202>
    Looks like you have Teamviewer running. Good program and very handy
    for remote admin.

    Ok, TrendNet TV-IP672P parking lot camera. As you mentioned, it's the
    only one of that type, while the others are HikVision DS-2CD2032-I.
    I'm familiar with various Trendnet cameras. Fairly straight forward
    with few surprises or bugs. Make sure you get the updated firmware.
    Hmmm... do you have a TV-IP672P or TV-IP672P1 ?? They're probably the
    same as the firmware versions for each are identical.
    <http://www.trendnet.com/support/supportdetail.asp?prod=185_TV-IP672P>

    Checking the emulator:
    <http://www.trendnet.com/emulators/TV-IP672P_V1.0R/mainFrame.xml?nav=Setup>
    it support UPnP, which has been somewhat of a PITA on IP camera
    system. If you have port forwarding properly setup, you don't need
    it.
    <http://foscam.us/forum/please-always-disable-upnp-in-your-camera-here-is-why-t7282.html>
    Not much else that I see that might cause a problem. (Yes, I know
    this camera is working).


    The HikVision DS-2CD2032-I camera:
    <http://www.hikvision.com/en/us/Products_show.asp?id=9090>
    has some detail on the network setup in the manual at:
    <http://www.hikvision.com/UploadFile/image/2013101808045467171.pdf>
    Sorry, but I couldn't find anything that might cause a problem.


    I still suspect that you have more than one DHCP server active.
    Find a copy of nmap and run:
    nmap --script broadcast-dhcp-discover -p67 [IP_address_range]
    See below for my results. The DHCPOFFER indicates that my DHCP server
    at 192.168.1.1 is active and working. I'll see if I can find another
    router later. If there are two, I think you'll see two "pre-scan"
    responses:

    nmap --script broadcast-dhcp-discover -p67 192.168.1.0/24

    Starting Nmap 6.46 ( http://nmap.org ) at 2015-01-03 19:14 Pacific
    Standard Time
    Pre-scan script results:
    | broadcast-dhcp-discover:
    | IP Offered: 192.168.1.165
    | DHCP Message Type: DHCPOFFER
    | Server Identifier: 192.168.1.1
    | IP Address Lease Time: 0 days, 0:02:00
    | Renewal Time Value: 0 days, 0:01:00
    | Rebinding Time Value: 0 days, 0:01:45
    | Subnet Mask: 255.255.255.0
    | Broadcast Address: 192.168.1.255
    | Domain Name Server: 192.168.1.1
    |_ Router: 192.168.1.1
    etc...

    You might also want to use namp to see what happens with the cameras
    crap out. Just scan for IP addresses that can be seen. If everything
    on the other side of the wireless WAN bridge goes away, including the
    bridge IP address, then something is goofy in the bridge.
    nmap -sP 192.168.1.0/24

    Some good examples of using nmap:
    <http://www.cyberciti.biz/networking/nmap-command-examples-tutorials/>
     
    Jeff Liebermann, Jan 4, 2015
    #8
  9. Sorta. The standard and 3rd party shapes include all the information
    in a database that is part of the Visio file. Fun Internet Exploder
    and:
    Ctrl + Right-Click
    on one of the computah icons at:
    <http://802.11junk.com/jeffl/K6BJ-Network/K6BJ-Network.htm>
    for a lousy example. It should produce a table in the left window
    pane with a bunch of info that I probably forgot to fill in. I should
    have something with everything filled out, but can't find it.
    Looks like you have almost everything setup with static IP's. That's
    good but adds to the mystery in that there's very little left that
    could mess up the connection with a bogus DHCP server. I'm beginning
    to suspect that blaming DHCP was a bad idea. Argh.
    The default route is where the client computers send their packets
    when the destination IP address is not in the route table
    (specifically not in the local network). It basically points to the
    gateway to the internet.
    Oh-oh. Neither of those two should have a DHCP server running. Notice
    that I said server, not client. They both probably have DHCP clients
    running.

    In a Windoze "cmd" window, run:

    ipconfig /all | find /I "dhcp"
    Dhcp Enabled. . . . . . . . . . . : Yes
    DHCP Server . . . . . . . . . . . : 192.168.1.1
    You might want to login to the network with Teamviwer and take a look
    at the DAP-1522 settings to make sure it's really an access point and
    really does have the DHCP disabled. I'm assuming it does, but with
    D-Link, I get too many surprises.
    Retch, but I guess it will work:
    Retch. It has a flaky wi-fi section, but the rest seems to work as
    expected.
    Unless something is seriously broken, I don't think there's anything
    that goes in and out of DHCP client mode versus static IP that would
    bring down 3 cameras at once. The closest might be something strange
    inside the PoE ethernet switch. Can you run one of the HikVision
    cameras from local power instead of PoE and see if it acts any
    differently?
    The Ubiquiti M5's will do SNMP (simple network management protocol)
    monitoring. Kinda messy, but it tells me quite a bit about what's
    moving on the network.
    <http://wiki.ubnt.com/SNMP_and_MRTG_Monitoring>
    Or, just use AirOS to see what's happening. You might also want to
    check the radios for transparent bridge mode setup:
    <https://community.ubnt.com/t5/airMA...o-Point-Link-Layer-2-Transparent/ta-p/419941>
    Note that WDS (wireless distribution service) must be enabled on both
    ends.

    Good luck.
     
    Jeff Liebermann, Jan 4, 2015
    #9
  10. (PeteCresswell)

    miso Guest

    miso, Jan 4, 2015
    #10
  11. (PeteCresswell), Jan 4, 2015
    #11
  12. Here's the original Ubiquiti shape archive, which is what I'm using:
    <http://dl.ubnt.com/UBNT-visio-shapes.zip>

    This looks newer:
    <http://community.ubnt.com/t5/Ubiqui...-and-stencils-for-diagrams-Visio/ta-p/455771>

    More (with probably some duplicates):
    <http://dl.ubnt.com/media/community/Wiki_Icons.zip>
    <http://gregsowell.com/?p=1797>
    <http://wiki.ispsupplies.com/index.php/Ubiquiti_Visio_Stencils>
    <http://community.spiceworks.com/topic/644325-ubiquiti-unifi-visio-2013-stencil>
    <https://www.graffletopia.com/stencils/1356>

    The modem and router are just JPG pictures that I borrowed from
    somewhere on the internet. Essentially, it's just clip art. There's
    no device data behind them, although I could easily add that. When
    I'm in a hurry (or don't care much about the results), I tend to do it
    this way.
     
    Jeff Liebermann, Jan 4, 2015
    #12
  13. Per Jeff Liebermann:
    No chance - I personally dropped the old DSL modem in the trash last
    time I went down there.

    But we keep coming back to that extra DHCP server....

    But a PC having a DHCP server is *not* the same as
    Network Connections | Local Area Connection | Properties | Internet
    Protocol (TCP/IP) | Properties | Obtain an IP Address Automatically &
    Obtain DNS server address automatically, right?

    That being the case, would somebody have to had install or activate
    something on the specified PC to make it a DHCP server? Or is it still
    just a matter of fat-fingering some property?

    Also...probably moot... but I just noted this in the server PC's Windows
    Event Log: "TCP/IP has chosen to restrict the scale factor due to a
    network condition. This could be related to a problem in a network
    device and will cause degraded throughput." This is roughly (very
    roughly.... I have no clue when the problem resurfaced...) coincident
    with a series of 3 onsets of the original problem today.

    Now we are at a place where re-booting the radios makes the problem go
    away.... but only for awhile (20 minutes? 10 minutes?.... gotta put up
    a ping log now to get it more exact).

    But I note that today it's really wild down there: wind averaging 25,
    gusting into the low forties.... and the cam views (when they're up)
    show it with the 2x4 that 2 of the cams are mounted on visibly rocking
    back-and-forth. Given that the shop radio is mounted on something even
    whippier (a windsurfer mast) I'm starting to wonder if the radio's
    swaying around in the wind could have something to do with this.


    Bottom line though: can I have a DHCP server active just by
    fat-fingering a setting, or would something have to have been
    installed/activated?
     
    (PeteCresswell), Jan 5, 2015
    #13
  14. (PeteCresswell)

    Char Jackson Guest

    Right, they are not the same. The latter case above is an example of a DHCP
    *client*.
    If it's a Windows server OS, I suppose it might have a DHCP server
    available, and possibly even running by default, but the desktop OSs aren't
    likely to have DHCP server capabilities, AFAIK. Do you know what OS is
    running there?

    Routers, OTOH, have a DHCP server on their LAN side that typically runs by
    default. They also typically have a DHCP client available on their WAN side,
    so it's possible to have both client and server available at the same time,
    although not on the same interface.
     
    Char Jackson, Jan 5, 2015
    #14
  15. Per (PeteCresswell):
    I think we've struck gold this time.

    Posted the above theory in the Ubiquiti forum and two of the resident
    experts jumped right on me: "You are using shielded cable, right?".....

    OOPS!.... RTFM... it's right there in the manual: "ESD attacks are the
    leading cause for device failures. The diagram below illustrates the
    areas vulnerable....."

    Today's series of failures correlated nicely with peak wind speeds
    between 10:00 and 16:00: http://tinyurl.com/nb8a846

    And it's a great time of year to be testing because it gets pretty wild
    down there on a fairly regular basis.

    Three more correlations like this and zero failures otherwise and I'm
    going to call it "Case Closed".

    Now I just have to modify my Ping -T bat file so it knows to start a new
    text file every time the date rolls over.... then I'll just let it run
    against one of the problem cameras 24-7....
     
    (PeteCresswell), Jan 5, 2015
    #15
  16. I've installed 7 pairs of various Ubiquiti wireless bridges and never
    used shielded cable. Three links are on RF polluted mountain tops and
    they don't use shielded cable. I also know of other such links that
    I'm fairly sure (but not certain) that they don't use shielded cable.
    RF can certainly cause problems, but not disconnect 3 out of 4
    cameras, leaving everything else working. That would require some
    rather unique IP selective RF interference.
    Define "failure". Was the Trendnet camera still working?
    Wind driven RF interference fixed by adding shielded CAT5? Huh?

    For what its worth, most of the problems I've had with links are due
    to lousy fade margin (weak signals) caused by antenna misalignment,
    Fresnel zone cancellation, reflections off moving reflectors (cars),
    and plain old interference. I also had one rediculous problem dealing
    with mismatched versions (xm versus xw):
    I don't gamble, but I'll be willing to speculate that this is going to
    be an ongoing problem until you identify the cause and culprit. Of
    course, after you find it, everything will be obvious.
    Try FreePing. Set it up to ping literally everything, especially both
    Ubiquit radios. When something goes down, you'll know what parts of
    the network are failing. It won't tell you the cause, but it will
    tell you where to look.

    Incidentally, I took a closer look at your ping results at:
    <https://plus.google.com/photos/108149798664924808733/albums/5781878959793669377/6093934978803039202>
    That's definately NOT what interference looks like. With
    interference, you get random increases in latency, with ocassional
    timeout errors. The increases in latency are caused by retransmission
    delays.

    Also, get some data on the signal levels, SNR, and BER between the
    Ubiquiti radios. The built in graphs aren't good enought for long
    term monitoring. Offhand, they look ok:
    <https://plus.google.com/photos/108149798664924808733/albums/5781878959793669377/6099034450766445106>
    <https://plus.google.com/photos/108149798664924808733/albums/5781878959793669377/6099034451179001362>

    Hmmm... you're on version 5.5.8. Current version is 5.5.10.
    <http://www.ubnt.com/download/#NanoStation:M5>

    I suggest setting up SNMP and running MRTG to make the graphs. If
    that's too much, setup SNMP and use the demo version of PRTG to draw
    the graphs. PRTG is easier to setup:
    <http://wiki.ubnt.com/SNMP_and_MRTG_Monitoring>
    <http://www.paessler.com/prtg>
    <http://oss.oetiker.ch/mrtg/>
     
    Jeff Liebermann, Jan 6, 2015
    #16
  17. Jeff Liebermann, Jan 6, 2015
    #17
  18. (PeteCresswell)

    Char Jackson Guest

    That caught my eye, as well.

    Just when I thought that there was nothing new in the world.
     
    Char Jackson, Jan 6, 2015
    #18
  19. Per (PeteCresswell):
    Well, I think I have disposed of that little theory by having several
    failures when the wind was not blowing at any unusual strength.

    But I seem to be on the track of *something* with the cam server
    application. I now have a series of things I can do that will provoke
    the exact symptoms I have been having.

    1) Close the Blue Iris application

    2) Stop the Blue Iris service.

    3) (observe that cams are still 100% pingable)

    4) Re-start the Blue Iris application (which
    automagically restarts the service)

    5) (observe that the three problem cams are
    no longer continuously pingable)

    6) Re-boot both radio links.

    7) (observe that all cams are back to normal)

    Whhhhaaaaaaat !!!?????

    It would appear that somehow the BI application/service may somehow be
    able to interfere with a camera's continuous pingability AND somehow
    rebooting the radios remedies that situation.

    Jeff: does this seem possible?

    Next step is to wait for the homeowner of the server site to come back,
    provoke the problem, and then have them power off the server PC and see
    if the pingability returns without rebooting the radios.

    Maybe that will tell me whether it's the BI app/service that's stepping
    on things or some state that it has put the PC into... ??


    And if I wind up going down to install the smart switch before they come
    back, I'll try to install that Belkin remote switch I was talking about
    before - so I can toggle the PC's power remotely.
     
    (PeteCresswell), Jan 10, 2015
    #19
  20. Yeah, it's possible. One way is if the Blue Iris server is running a
    DHCP server and the cameras are accepting IP addresses assigned via
    DHCP. However, we ruled that out when you said that the cameras were
    all on static IP's. Still, check for a 2nd DHCP server. Just
    temorarily turn off the DHCP server in the router. Reboot a few
    devices and see if they get an IP address. If the get
    169.254.xxx.xxx, then there's no hidden DHCP server. However, if
    they get a valid IP address, then there's a 2nd DHCP server.

    Another way to screw things up is to run something like:
    arp -s 192.168.1.123 00-00-00-00-00-00
    where the IP address is some existing device IP, and the MAC address
    is that of one of the 3 cameras that is failing. In effect, it
    convinced your client computah that the camera has changed IP address,
    without actually changing the IP address on the camera. For the grand
    prize, use the router IP address, and the MAC address of some other
    device on the LAN, and watch the whole system come to a grinding halt
    as all the packets destined for the internet to a camera instead of
    the router. Lots of other variations on the same theme such as
    duplicated MAC addresses on counterfeit cameras and devices, or
    Comcast routers that deliver both routeable and private IP addresses
    to the client machines.

    Wireshark and other network sniffers will usually report if there are
    two IP addresses sharing the same MAC address, or two MAC addresses
    with the same IP address.

    At the risk of being rude, have you tried ANY of the suggestions that
    I've made in previous postings? I don't think so and you seem intent
    on discovering the problem yourself. Fine. I'll offer a simple
    strategy.

    Tear the system apart by removing everything that doesn't need to be
    running, and put it back together section by section until it stops
    working. Since you now have a way to reproduce the problem, that
    should be easy. Also, since you're using static IP's, make a
    spreadsheet listing each devices IP address, Netmask, Default route
    (gateway), and DNS servers. However, don't take them from what you
    assume them to be. Login to the management interface, and get the
    numbers from the devices directly. If ambitious, use some
    autodiscovery software such as Visio or TheDude:
    <https://www.mikrotik.com/thedude.php>
    When I have a topology problem, I do it to see if I missed something.
    That usually finds my sloppy typing errors.
     
    Jeff Liebermann, Jan 11, 2015
    #20
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.