Using Python Under Linux To Count Lines In All The Files In A Folder Via Terminal

I am using Ubuntu with python 2.7, I need to take all the files in a folder, and count the lines in every file seperatly and dump it to a file. I found how to do it via terminal

Solution 1:

You can use stuff in the standard library without having to shell out:

import os

from multiprocessing import Pool

folder = '.'

fnames = (name for name in os.listdir(folder)
          if os.path.isfile(os.path.join(folder, name)))


deffile_wc(fname):
    withopen(fname) as f:
        count = sum(1for line in f)
    return count


pool = Pool()

print(pool.map(file_wc, list(fnames)))

If you want to record the file names

deffile_wc(fname):
    withopen(fname) as f:
        count = sum(1for line in f)
    return (fname, count)

print(dict(pool.map(file_wc, list(fnames))))

Solution 2:

Count files, dirs and path's in folder

import ospath, dirs, files = os.walk("/home/my_folder").next()
file_count = len(files)

Count lines in file, I tried to find a way to count the lines without open the file but I can't

withopen(<pathtofile>) as f:
    printlen(f.readlines())

Now you have a list of the files (variable files in the firts example) you just need to join this 2 pieces of code to get the number of lines for every file in the variable files

Solution 3:

Actually you don't need to use external processes to do this task in python. Python can do it for you. Here is python3 snippet:

import osfor x inos.listdir(): 
    ifos.path.isfile(x):
        with open(x, 'rb') as f:
            print('{} lines: {}'.format(x, sum(1for line in x)))

Here are some additional information about listening files in dir, getting number of lines in file and counting lines for huge files

Solution 4:

You can use multiprocessing together with system calls. You don't have to use the queue here and just print the results directly.

import multiprocessing as mp
from subprocess import Popen, PIPE


output = mp.Queue()


def count_lines(path, output):
    popen = Popen(["wc", "-l", path], stdout=PIPE, stderr=PIPE)
    res, err = popen.communicate()
    output.put(res.strip())


popen = Popen(["ls", "."], stdout=PIPE, stderr=PIPE)
res, err = popen.communicate()


processes = [mp.Process(target=count_lines, args=(path.strip(), output)) forpathin res.split('\n') ifpath]

# Run processes
for proc in processes:
    proc.start()

for proc in processes:
    proc.join()


results = [output.get() for proc in processes]
non_empty = [result for result in results if result]
print(non_empty)

Reference:

https://sebastianraschka.com/Articles/2014_multiprocessing.html