I want to read multi big files that exist on centos server with python.I wrote a simple code for that and it’s worked but entire file came to a paramiko object (paramiko.sftp_file.SFTPFile) after that I can process line. it has not good performance and I want process file and write to csv piece by piece because process entire file can affect performance. Is there a way to solve the problem?
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, port, username, password)
sftp_client = ssh.open_sftp()
remote_file = sftp_client.open(r'/root/bigfile.csv')
try:
for line in remote_file:
#Proccess
finally:
remote_file.close()
2
Answers
Reading in chunks will help you here:
Update:
Yeah, I’m aware that my answer written based on a local file. Just giving example for reading file in chunks.
To answer the question, check out this one:
Here could solve your problem.
This will avoid reading the whole thing into memory at once.