Question 37456 in Ubuntu Question posted in
The official Ubuntu documentation can be found here.

Bash looping a file and placing lines in another file based on finding string in line – Ubuntu

I am trying to process a large file 4 Gig and want each line to go to a specific file but cant seem to get it. Can someone help me find out what works best for this in Ubuntu 20. I have been at this for hours but cannot seem to get it. Am I even close?

test.log

go to file 1
should go to file 2
should be seen in file 3
another for file 3 I belong in file 3
file 2 is my place

processLog.sh

  while read -r LINE
  do
     grep -h "file 1" > file1.txt
     grep -h "file 2" > file2.txt
     grep -h "file 3" > file3.txt
  done < test.log

Expected Results

file1.txt
    go to file 1
file2.txt
    should go to file 2
    file 2 is my place
file3.txt
    should be seen in file 3
    another for file 3 I belong in file 3

Final Used

@William Pursell thank you for the answer it rips thru the 4Gig file like a Lightsaber thru butter!

In the end, I used the following; to help others it’s here in its entirety. Called on the bash CLI which is why the trailing backslashes are there. Notice the first line is a negation (!) so it gets all that does NOT have "INSERT INTO ". The rest of the lines look for a specific string and place it into a specific file. Happy Awking!

awk '
!/INSERT INTO / {print > "DATE_mysql_om.sql"; next } 
/INSERT INTO `objecttype`/ {print > "insert_om_objecttype.sql"; next } 
/INSERT INTO `objecttag`/ {print > "insert_om_objecttag.sql"; next } 
/INSERT INTO `object_objecttag`/ {print > "insert_om_object_objecttag.sql"; next } 
/INSERT INTO `indexlogger`/ {print > "insert_om_indexlogger.sql";}' 
*_mysqldump.sql
Tags:

2

Answers


  1. 0

    grep is the wrong tool for something like this. If you want lines that match the string "file 1" to be written to file1.txt, etc, you could do something like:

    awk '/file 1/ { print > "file1.txt"; next } /file 2/ {print > "file2.txt"; next }` input-file
    Login or Signup to reply.
  2. 0

    This awk should work for you in a single action block using match function:

    awk 'match($0, /file [0-9]+/) {
   print > (substr($0, RSTART, 4) substr($0, RSTART+5, RLENGTH-5) ".txt")
}' file
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search