NFS settings for video

Discussion in 'Linux Networking' started by S.K.R. de Jong, Aug 15, 2012.

  1. I have a box A with an external (USB) disk directly attached to it
    containing video material. The disk has two jfs partitions that are NFS-
    exported. Another box B, in the same network, mounts those partions locally
    using the following entries in B's /etc/fstab file:

    A:/Videos/One /One nfs rw,user,rsize=8192,wsize=8192,hard,intr,nfsvers=3,actimeo=0,addr=192.168.0.2,addr=192.168.0.2 0 0
    A:/Videos/Two /Two nfs rw,user,rsize=8192,wsize=8192,hard,intr,actimeo=0,addr=192.168.0.2,addr=192.168.0.2 0 0

    where 192.168.0.2 is A's IP address.

    With this setup playing videos in /One or /Two from B gives me a choppy
    performance, that is more evident for high resolution videos. In this case,
    the video plays fine for a few seconds, freezes for one second, and then the
    same, over and over again. For lower resolution videos the effect is less
    noticeable (the freeze happens less often and is more brief) but it is still
    there.

    Both A and B can play all videos without any problems, when the videos
    are stored in a disk local to A and B, respectively. Both machines have 100Mb
    NICs, and the network is indeed a 100Mb one. I have other disks local to A which
    show the same behavior when NFS-mounted on B. All this makes me think that this
    is an NFS issue.

    Any suggestions on how to fine tune NFS for this purpose? I tried to
    change rsize and wsize above to be 32768, but it did not seem to make any
    difference. Any ideas?
     
    S.K.R. de Jong, Aug 15, 2012
    #1
    1. Advertisements

  2. I have similar experiences. Can you run top and have a look at at the
    memory usage during playback? Not the server processes' usage, but the
    kernel mem at the top few lines. It is the 'cached' line that I am
    interested in. Does it get very large and then jump back down to sane
    values? The lags ocurring when it jumps back down?

    If yes, it means your problem may be the kernel memory management
    itself. On reading from the usb disk, the kernel will cache _any_ block
    it has read, swapping out data at random(to local disk). This generates
    data transfers usb->mem and mem->swap, both about the bitrate of video
    playback. This may be a tad too much for you bus. also, there will be
    delays, when running applications have to be swapped back in, which will
    happen more frequently. If this is the case running an occasional 'echo
    3 > /proc/sys/vm/drop_caches' as root may help temporarily. But you'd
    have to do it regularly to keep the cache from filling up. You could
    also tinker with the swappiness values or disable swap area altogether.

    If the above does not apply, look for radio equipment generating
    interference(phones,WiFi...) and turn it off. USB seems rather
    susceptible to it.
     
    Johann Klammer, Aug 15, 2012
    #2
    1. Advertisements

  3. S.K.R. de Jong

    Rick Jones Guest

    I am assuming your 100Mb network is a wired network. Is it also
    running full-duplex?

    You might take a snapshot of netstat statistics over a 60 second
    interval while the video is playing across the network and then run it
    through the likes of beforeafter (ftp://ftp.netperf.org/netperf/ -
    someone did a better version of that sort of tool but I cannot recall
    who)

    netstat -s > before
    sleep 60
    netstat -s > after
    beforeafter before after > delta

    and then examine "delta" for things like retransmissions. If you see
    any you might also include ethtool -S <interface> statistics and check
    those.

    Have you run something like netperf to verify that the network itself
    is working OK? I would suggest grabbing netperf 2.6.0 bits from
    netperf.org (or top-of-trunk via subversion to
    http://www.netperf.org/svn/netperf2/trunk and then do a
    ../configure;make on both sides. Start netserver on at least the
    system acting as the NFS server and then:

    netperf -H <nfsserver> -t TCP_RR -l 60 -f m -- -b <readahead> -r 256,8448

    Where readahead should be however many read-ahead requests the NFS
    client code is putting out there (not sure where to find that
    information, though tuning read-ahead might be a good thing...). The
    values for the -r option are the request and response size
    respectively and I've ass-u-me-d that an NFS message header is 256
    bytes. So, that will simulate at the TCP level a stream of NFS Reads
    from the system running netperf to the NFS server. An example using a
    burst size of 3, so there were 4 transactions outstanding at any one
    time, over a 1 GbE network:

    [email protected]:~/netperf2_trunk$ src/netperf -H raj-8510w.americas.hpqcorp.net -t TCP_RR -l 20 -f m -- -b 3 -D -r 256,8448
    MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to raj-8510w.local () port 0 AF_INET : nodelay : histogram : demo : first burst 3
    Local /Remote
    Socket Size Request Resp. Elapsed
    Send Recv Size Size Time Throughput
    bytes Bytes bytes bytes secs. 10^6bits/sec

    16384 87380 256 8448 20.00 931.26
    16384 87380

    (it might be a good idea to add a -s 1M -S 1M to the end there because
    netperf TCP_RR has no select and if the request or response size times
    the number of outstanding is greater than the socket buffer size, the
    test runs the risk of hanging)

    That though is only an average throughput over the length of the test.
    You can also add a --enable-histogram to the ./configure and add a -v
    2 and you will see the likes of:

    [email protected]:~/netperf2_trunk$ src/netperf -H raj-8510w.americas.hpqcorp.net -t TCP_RR -l 20 -f m -v 2 -- -b 3 -D -r 256,8448
    MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to raj-8510w.local () port 0 AF_INET : nodelay : histogram : demo : first burst 3
    Local /Remote
    Socket Size Request Resp. Elapsed
    Send Recv Size Size Time Throughput
    bytes Bytes bytes bytes secs. 10^6bits/sec

    16384 87380 256 8448 20.00 930.27
    16384 87380
    Alignment Offset RoundTrip Trans Throughput
    Local Remote Local Remote Latency Rate 10^6bits/s
    Send Recv Send Recv usec/Tran per sec Outbound Inbound
    8 0 0 0 299.409 13359.667 27.361 902.900

    Histogram of request/response times
    UNIT_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
    TEN_USEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
    HUNDRED_USEC : 0: 12: 110006: 150468: 5546: 670: 125: 71: 51: 33
    UNIT_MSEC : 0: 111: 24: 16: 22: 18: 8: 2: 2: 12
    TEN_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
    HUNDRED_MSEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
    UNIT_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
    TEN_SEC : 0: 0: 0: 0: 0: 0: 0: 0: 0: 0
    HIST_TOTAL: 267197

    And you will have a histogram of the individual transaction times. If
    your video player doesn't buffer enough to deal with the higher
    transaction latencies...

    Also, someone mentioned caching and filling the memory with it and
    such. If that is indeed an issue, it might be a good idea to get the
    video player to make a (posix_)fadvise() call to tell the kernel when
    it is no longer interested in a given range of the video file. In
    that way, the kernel will be less likely to let the video file fill
    all the available memory and start tossing-out otherwise more
    interesting pages. Perhaps one of these days Linux will be willing to
    have a configurable upper bound on the size of the file cache...

    rick jones

    And if you want to get a whole lot of output, you can make sure that
    --enable-demo is present in the ./configure (should be default in
    2.5.0 and later though) you can look at the interim results as the
    test runs by adding a -D option with the desired interval in seconds:

    [email protected]:~/netperf2_trunk$ src/netperf -H raj-8510w.americas.hpqcorp.net -t TCP_RR -l 20 -f m -v 2 -D 1 -- -b 3 -D -r 256,8448
    MIGRATED TCP REQUEST/RESPONSE TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to raj-8510w.local () port 0 AF_INET : nodelay : histogram : demo : first burst 3
    Interim result: 935.88 10^6bits/s over 1.000 seconds ending at 1345142033.022
    Interim result: 940.71 10^6bits/s over 1.000 seconds ending at 1345142034.022
    Interim result: 918.71 10^6bits/s over 1.024 seconds ending at 1345142035.046
    Interim result: 922.61 10^6bits/s over 1.000 seconds ending at 1345142036.046
    ....
     
    Rick Jones, Aug 16, 2012
    #3
  4. S.K.R. de Jong

    Jorgen Grahn Guest

    .
    Might have been my 'cdiff'. I must admit though that I haven't used it
    much myself, so noone knows if it's really better:

    http://snipabacken.se/~grahn/comp/#cdiff

    Also, someone at work showed me the the 'watch' utility, which is part
    of the 'procps' package on Linux. Very useful, and overlapping in
    functionality -- but not good if you want exact delta figures.

    /Jorgen
     
    Jorgen Grahn, Aug 16, 2012
    #4
    1. Advertisements

Ask a Question

Want to reply to this thread or ask your own question?

You'll need to choose a username for the site, which only take a couple of moments (here). After that, you can post your question and our members will help you out.