Linux sripting help neede

Discussion in 'NZ Computing' started by Rob S, Jul 29, 2007.

  1. Rob S

    Rob S Guest

    I have a directory with >500 files. I want to divide the 500 odd files
    into 4 groups, each group containing a random assortment of filenames.
    I've tried using awk with rand() and srand()until my head was ready to
    explode. I'm obviously getting the syntax wrong somewhere, but seeing as
    how I've forgotten nearly everything I ever knew about awk, it's not
    surprising. Any ideas? Any language will do as long as it does the job.

    Cheers
    --

    Rob
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    http://www.aspir8or.com
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


    Linux means productivity and fun. NT means 'Not Today'.
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Rob S, Jul 29, 2007
    #1
    1. Advertising

  2. Rob S

    Adam Guest

    Rob S wrote:

    > I have a directory with >500 files. I want to divide the 500 odd files
    > into 4 groups, each group containing a random assortment of filenames.


    OK a sort by file size or name or date is not what you wish to use, but a
    true random of some sort.

    > I've tried using awk with rand() and srand()until my head was ready to
    > explode. I'm obviously getting the syntax wrong somewhere, but seeing as
    > how I've forgotten nearly everything I ever knew about awk, it's not
    > surprising. Any ideas? Any language will do as long as it does the job.


    Common Lisp or Python both come to mind. To allow you to tinker
    non-destructively at a REPL.

    Perhaps you could form an array of the filenames, then a same-sized array of
    unique positions for the filenames, and read the names from the first array
    by sequencing through the second. And arrays are available in Bash,
    although more simply juggled in Lisp or Python.

    Alternatively, create two files - one of filenames and one of unique (non
    repeating) random numbers, and then stream the filenames into array
    positions from the second - then stream the filename array out in its
    shuffled order when finshed.
    Adam, Jul 29, 2007
    #2
    1. Advertising

  3. Rob S

    Shane Guest

    Rob S wrote:

    > I have a directory with >500 files. I want to divide the 500 odd files
    > into 4 groups, each group containing a random assortment of filenames.
    > I've tried using awk with rand() and srand()until my head was ready to
    > explode. I'm obviously getting the syntax wrong somewhere, but seeing as
    > how I've forgotten nearly everything I ever knew about awk, it's not
    > surprising. Any ideas? Any language will do as long as it does the job.
    >
    > Cheers


    bash has RANDOM which you access thus.
    $ echo $RANDOM
    29264

    from the manpage
    RANDOM
    Each time this parameter is referenced, a random integer between
    0 and 32767 is generated. The sequence of random numbers may be
    initialized by assigning a value to RANDOM. If RANDOM is unset,
    it loses its special properties, even if it is subsequently
    reset.

    For your usage (You want a rand(4)?)
    $echo $(($RANDOM%4))
    1

    (ie the random number modulus 4)

    Your next job is to use that number to assign which directory to move your
    file to.

    For completeness I should mention that $echo $(($RANDOM%4)) will return 0,
    1, 2, or 3

    HTH


    --
    Q: What's big, grey, and proves the uncountability of the reals?
    A: Cantor's diagonal elephant.
    Shane, Jul 29, 2007
    #3
  4. Rob S

    NR Guest

    On Jul 30, 9:56 am, Shane <-a-geek.net> wrote:
    > Rob S wrote:
    > > I have a directory with >500 files. I want to divide the 500 odd files
    > > into 4 groups, each group containing a random assortment of filenames.
    > > I've tried using awk with rand() and srand()until my head was ready to
    > > explode. I'm obviously getting the syntax wrong somewhere, but seeing as
    > > how I've forgotten nearly everything I ever knew about awk, it's not
    > > surprising. Any ideas? Any language will do as long as it does the job.

    >
    > > Cheers

    >
    > bash has RANDOM which you access thus.
    > $ echo $RANDOM
    > 29264
    >
    > from the manpage
    > RANDOM
    > Each time this parameter is referenced, a random integer between
    > 0 and 32767 is generated. The sequence of random numbers may be
    > initialized by assigning a value to RANDOM. If RANDOM is unset,
    > it loses its special properties, even if it is subsequently
    > reset.
    >
    > For your usage (You want a rand(4)?)
    > $echo $(($RANDOM%4))
    > 1
    >
    > (ie the random number modulus 4)
    >
    > Your next job is to use that number to assign which directory to move your
    > file to.
    >
    > For completeness I should mention that $echo $(($RANDOM%4)) will return 0,
    > 1, 2, or 3
    >
    > HTH
    >
    > --
    > Q: What's big, grey, and proves the uncountability of the reals?
    > A: Cantor's diagonal elephant.


    something like

    #!/bin/bash

    #run from inside the directory you want to randomly sort.

    mkdir -p {0,1,2,3}
    for file in $(find . -maxdepth 1 -type f) ; do
    mv $file $(($RANDOM%4))
    done ;
    NR, Jul 30, 2007
    #4
  5. Rob S

    Rob S Guest

    NR wrote:
    > On Jul 30, 9:56 am, Shane <-a-geek.net> wrote:
    >> Rob S wrote:
    >>> I have a directory with >500 files. I want to divide the 500 odd files
    >>> into 4 groups, each group containing a random assortment of filenames.
    >>> I've tried using awk with rand() and srand()until my head was ready to
    >>> explode. I'm obviously getting the syntax wrong somewhere, but seeing as
    >>> how I've forgotten nearly everything I ever knew about awk, it's not
    >>> surprising. Any ideas? Any language will do as long as it does the job.
    >>> Cheers

    >> bash has RANDOM which you access thus.
    >> $ echo $RANDOM
    >> 29264
    >>
    >> from the manpage
    >> RANDOM
    >> Each time this parameter is referenced, a random integer between
    >> 0 and 32767 is generated. The sequence of random numbers may be
    >> initialized by assigning a value to RANDOM. If RANDOM is unset,
    >> it loses its special properties, even if it is subsequently
    >> reset.
    >>
    >> For your usage (You want a rand(4)?)
    >> $echo $(($RANDOM%4))
    >> 1
    >>
    >> (ie the random number modulus 4)
    >>
    >> Your next job is to use that number to assign which directory to move your
    >> file to.
    >>
    >> For completeness I should mention that $echo $(($RANDOM%4)) will return 0,
    >> 1, 2, or 3
    >>
    >> HTH
    >>
    >> --
    >> Q: What's big, grey, and proves the uncountability of the reals?
    >> A: Cantor's diagonal elephant.

    >
    > something like
    >
    > #!/bin/bash
    >
    > #run from inside the directory you want to randomly sort.
    >
    > mkdir -p {0,1,2,3}
    > for file in $(find . -maxdepth 1 -type f) ; do
    > mv $file $(($RANDOM%4))
    > done ;
    >

    After loads of mucking around and badgering a couple of coders, this
    ruby script does the job perfectly, with an equal number of filenames in
    random order in each of 4 files named wp_list_0, wp_list_1, wp_list_2,
    and wp_list_3.

    #!/usr/bin/ruby
    list = Dir[ "*" ].sort_by{ rand }
    chunk = ( list.size / 4.0 ).ceil
    4.times{ |i|
    File.open( "wp_list_#{i}", "w" ){ |f|
    f.puts( list[ i * chunk, chunk ].sort )
    }
    }

    but thanks for the suggestions. I'll tackle them as well, if only to
    keep the creaky old gears turning. :)

    --

    Rob
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    http://www.aspir8or.com
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


    The Windows98 Eveready Bunny: It's still loading, and loading...
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Rob S, Jul 30, 2007
    #5
  6. Rob S

    Shane Guest

    Rob S wrote:

    > NR wrote:
    >> On Jul 30, 9:56 am, Shane <-a-geek.net> wrote:
    >>> Rob S wrote:
    >>>> I have a directory with >500 files. I want to divide the 500 odd files
    >>>> into 4 groups, each group containing a random assortment of filenames.
    >>>> I've tried using awk with rand() and srand()until my head was ready to
    >>>> explode. I'm obviously getting the syntax wrong somewhere, but seeing
    >>>> as how I've forgotten nearly everything I ever knew about awk, it's not
    >>>> surprising. Any ideas? Any language will do as long as it does the job.
    >>>> Cheers
    >>> bash has RANDOM which you access thus.
    >>> $ echo $RANDOM
    >>> 29264
    >>>
    >>> from the manpage
    >>> RANDOM
    >>> Each time this parameter is referenced, a random integer between
    >>> 0 and 32767 is generated. The sequence of random numbers may be
    >>> initialized by assigning a value to RANDOM. If RANDOM is unset,
    >>> it loses its special properties, even if it is subsequently
    >>> reset.
    >>>
    >>> For your usage (You want a rand(4)?)
    >>> $echo $(($RANDOM%4))
    >>> 1
    >>>
    >>> (ie the random number modulus 4)
    >>>
    >>> Your next job is to use that number to assign which directory to move
    >>> your file to.
    >>>
    >>> For completeness I should mention that $echo $(($RANDOM%4)) will return
    >>> 0, 1, 2, or 3
    >>>
    >>> HTH
    >>>
    >>> --
    >>> Q: What's big, grey, and proves the uncountability of the reals?
    >>> A: Cantor's diagonal elephant.

    >>
    >> something like
    >>
    >> #!/bin/bash
    >>
    >> #run from inside the directory you want to randomly sort.
    >>
    >> mkdir -p {0,1,2,3}
    >> for file in $(find . -maxdepth 1 -type f) ; do
    >> mv $file $(($RANDOM%4))
    >> done ;
    >>

    > After loads of mucking around and badgering a couple of coders, this
    > ruby script does the job perfectly, with an equal number of filenames in
    > random order in each of 4 files named wp_list_0, wp_list_1, wp_list_2,
    > and wp_list_3.
    >
    > #!/usr/bin/ruby
    > list = Dir[ "*" ].sort_by{ rand }
    > chunk = ( list.size / 4.0 ).ceil
    > 4.times{ |i|
    > File.open( "wp_list_#{i}", "w" ){ |f|
    > f.puts( list[ i * chunk, chunk ].sort )
    > }
    > }
    >
    > but thanks for the suggestions. I'll tackle them as well, if only to
    > keep the creaky old gears turning. :)
    >


    Hmm yeah I had wondered if I had misread your intentions. Still, good to
    see youve got a solution that works.

    --
    Q: Why do mathematicians often confuse Christmas and Halloween?
    A: Because Oct 31 = Dec 25.
    Shane, Jul 30, 2007
    #6
  7. Rob S

    Jasen Betts Guest

    On 2007-07-29, Rob S <Here@home> wrote:
    > I have a directory with >500 files. I want to divide the 500 odd files
    > into 4 groups, each group containing a random assortment of filenames.
    > I've tried using awk with rand() and srand()until my head was ready to
    > explode. I'm obviously getting the syntax wrong somewhere, but seeing as
    > how I've forgotten nearly everything I ever knew about awk, it's not
    > surprising. Any ideas? Any language will do as long as it does the job.


    here's how to do it in bash using the rolldice tool.

    mkdir 1 2 3 4
    for x in path/to/files/*
    do
    mv "$x" `rolldice d4`
    done

    rolldice is in the bsdgames package.

    Bye.
    Jasen
    Jasen Betts, Jul 31, 2007
    #7
  8. Re: Linux scripting help needed

    In message <>, Rob S wrote:

    > I have a directory with >500 files. I want to divide the 500 odd files
    > into 4 groups, each group containing a random assortment of filenames.
    > I've tried using awk with rand() and srand()until my head was ready to
    > explode. I'm obviously getting the syntax wrong somewhere, but seeing as
    > how I've forgotten nearly everything I ever knew about awk, it's not
    > surprising. Any ideas? Any language will do as long as it does the job.


    Since nobody has volunteered a solution using Python, how about this. :)

    import os
    import random

    target_dir = # name of directory containing files to divide up #
    list_of_filenames = os.listdir(target_dir)
    nr_groups = 4
    grouping = range(0, nr_groups) * (len(list_of_filenames) / 4) + \
    range(0, nr_groups)[:len(list_of_filenames) % 4]
    random.shuffle(grouping)
    lists = [[] for i in range(0, nr_groups)]
    for i in range(0, len(files)) :
    lists[grouping].append(list_of_filenames)

    After the above sequence, the variable "lists" will hold four lists, each
    including a random subset of the original names from "list_of_filenames".
    Actually moving each list of files into a separate subdirectory is left as
    an exercise for the reader. :)
    Lawrence D'Oliveiro, Aug 1, 2007
    #8
  9. Rob S

    Rob S Guest

    Re: Linux scripting help needed

    Lawrence D'Oliveiro wrote:
    > In message <>, Rob S wrote:
    >
    >> I have a directory with >500 files. I want to divide the 500 odd files
    >> into 4 groups, each group containing a random assortment of filenames.
    >> I've tried using awk with rand() and srand()until my head was ready to
    >> explode. I'm obviously getting the syntax wrong somewhere, but seeing as
    >> how I've forgotten nearly everything I ever knew about awk, it's not
    >> surprising. Any ideas? Any language will do as long as it does the job.

    >
    > Since nobody has volunteered a solution using Python, how about this. :)
    >
    > import os
    > import random
    >
    > target_dir = # name of directory containing files to divide up #
    > list_of_filenames = os.listdir(target_dir)
    > nr_groups = 4
    > grouping = range(0, nr_groups) * (len(list_of_filenames) / 4) + \
    > range(0, nr_groups)[:len(list_of_filenames) % 4]
    > random.shuffle(grouping)
    > lists = [[] for i in range(0, nr_groups)]
    > for i in range(0, len(files)) :
    > lists[grouping].append(list_of_filenames)
    >
    > After the above sequence, the variable "lists" will hold four lists, each
    > including a random subset of the original names from "list_of_filenames".
    > Actually moving each list of files into a separate subdirectory is left as
    > an exercise for the reader. :)


    After enveloping target_dir in quotes, I got an undefined variable on
    line 11 " files". I spent ages mucking around trying to sort it out, but
    usually ended up with no output after changing 'files' to any of your
    declared variables, not even any errors,

    python listby4.py >err.log 2>&1 invariably came up empty.

    I'm only starting to get my head around ruby, I think learning python at
    the same time would turn me into more of a jibbering idiot than I am
    already. :)
    --

    Rob
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    http://www.aspir8or.com
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


    "If you want to make an apple pie from scratch, you must first create
    the universe." - Carl Sagan
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Rob S, Aug 1, 2007
    #9
  10. Re: Linux scripting help needed

    In message <46b071ca$>, Rob S wrote:

    > After enveloping target_dir in quotes, I got an undefined variable on
    > line 11 " files". I spent ages mucking around trying to sort it out, but
    > usually ended up with no output after changing 'files' to any of your
    > declared variables, not even any errors,


    Oops, sorry, "files" should of course be "list_of_filenames".

    > python listby4.py >err.log 2>&1 invariably came up empty.


    Note you'll need to pull the lists out of "lists" at the end.

    > I'm only starting to get my head around ruby, I think learning python at
    > the same time would turn me into more of a jibbering idiot than I am
    > already. :)


    Given that I currently have to maintain code I've written for clients in
    Perl, C, C++, Python and Tcl, I figure it just adds to the fun. :)

    By the way, note one important difference between my solution and all the
    other ones I've seen so far in this thread: mine guarantees that, as far as
    possible, the different partitions of the original list of files are equal
    in length. The others don't.
    Lawrence D'Oliveiro, Aug 2, 2007
    #10
  11. Rob S

    Rob S Guest

    Re: Linux scripting help needed

    Lawrence D'Oliveiro wrote:
    > In message <46b071ca$>, Rob S wrote:
    >
    >> After enveloping target_dir in quotes, I got an undefined variable on
    >> line 11 " files". I spent ages mucking around trying to sort it out, but
    >> usually ended up with no output after changing 'files' to any of your
    >> declared variables, not even any errors,

    >
    > Oops, sorry, "files" should of course be "list_of_filenames".
    >
    >> python listby4.py >err.log 2>&1 invariably came up empty.

    >
    > Note you'll need to pull the lists out of "lists" at the end.
    >

    Got it working now thanks.
    >> I'm only starting to get my head around ruby, I think learning python at
    >> the same time would turn me into more of a jibbering idiot than I am
    >> already. :)

    >
    > Given that I currently have to maintain code I've written for clients in
    > Perl, C, C++, Python and Tcl, I figure it just adds to the fun. :)
    >

    I did a course in C about 20 years ago, hated it, learnt Perl instead,
    but haven't written anything in it for about 12 years, but it's slowly
    coming back to me. Once I've got Ruby down, then Ruby on Rails, I
    probably will have a go at Python. I'm not really interested in coding
    business apps or suchlike, more web apps, widgets etc. I blame it on my
    addictive personality, instant gratification and all that. :)

    By the way, note one important difference between my solution and all the
    > other ones I've seen so far in this thread: mine guarantees that, as far as
    > possible, the different partitions of the original list of files are equal
    > in length. The others don't.


    My Ruby solution also creates equal numbers. Any idea why Ruby starts
    counting at 0?

    --

    Rob
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    http://www.aspir8or.com
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -


    Barium: What you do with dead chemists.
    - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
    Rob S, Aug 2, 2007
    #11
  12. Re: Linux scripting help needed

    In message <46b12c31$>, Rob S wrote:

    > Any idea why Ruby starts counting at 0?


    You mean, when indexing arrays/lists? They all do that these days. :)
    Lawrence D'Oliveiro, Aug 2, 2007
    #12
    1. Advertising

Want to reply to this thread or ask your own question?

It takes just 2 minutes to sign up (and it's free!). Just click the sign up button to choose a username and then you can ask your own questions on the forum.
Similar Threads
  1. Nick de Graeve

    Importing TB mail from linux in linux

    Nick de Graeve, Oct 4, 2004, in forum: Firefox
    Replies:
    0
    Views:
    638
    Nick de Graeve
    Oct 4, 2004
  2. Adrian de los Santos

    Linux-Cisco-ADSL-Cisco-Linux Connection hangs....

    Adrian de los Santos, Jul 22, 2003, in forum: Cisco
    Replies:
    6
    Views:
    893
    David Van Cleef
    Jul 22, 2003
  3. me_here

    neede in australia

    me_here, Feb 1, 2004, in forum: Digital Photography
    Replies:
    1
    Views:
    642
    Proctor
    Feb 1, 2004
  4. Have a nice cup of pee

    Linux... yeah linux.. Linux

    Have a nice cup of pee, Apr 12, 2006, in forum: NZ Computing
    Replies:
    19
    Views:
    642
    Bette Noir
    Apr 17, 2006
  5. phil johnson

    Serious assistance neede re: Mail problem

    phil johnson, Apr 15, 2010, in forum: The Lounge
    Replies:
    1
    Views:
    1,310
    Neggsy
    Apr 16, 2010
Loading...

Share This Page