Flat File Length Management Script
Posted: Wed Jul 15, 2015 8:40 pm
I have a Python script that writes select device state information to a flat file (CSV file) for use in constructing Gnuplot charts. Due mainly to laziness, I always just added another line to the end of the file and got on with my life. After a while, the file started to get too large--and for no good reason. For example, I use 3 days' worth of data--288 observations--but my CSV file could quickly grow to many thousands of lines. At any event, I finally decided to whip out a simple script to keep the file to a more rational size. I run it through a schedule that fires at a time where there is no chance that the CSV file will be in use by another process. I only run it once per day. There are five configurable variables--but only two need explaining:
keep_first_lines: When TRUE, the script will retain the first N lines of the data. When FALSE, the script will retain the last N lines of the data.
keep_header_lines: This value controls how many header lines to retain. A setting of 0 retains no header lines. A setting of 1 retains one line...
Feel free to post improvements or criticisms, and use at your own risk.
keep_first_lines: When TRUE, the script will retain the first N lines of the data. When FALSE, the script will retain the last N lines of the data.
keep_header_lines: This value controls how many header lines to retain. A setting of 0 retains no header lines. A setting of 1 retains one line...
Feel free to post improvements or criticisms, and use at your own risk.
- Code: Select all
import os
import shutil
import sys
################################################################
# User configurable variables.
source = '/Users/.../Dropbox/Public/charts.csv'
backup = '/Users/.../Dropbox/Public/charts backup.csv'
target_lines = 10
keep_first_lines = False
keep_header_lines = 0
################################################################
try:
# Make a backup of the original file in case something goes wrong.
shutil.copyfile(source, backup)
# Open the original file in read-only mode and count the number of lines.
with open(source, 'r') as oldfile:
lines = oldfile.readlines()
old_num_lines = sum(1 for line in lines)
# Reopen the orignal file in write mode and write-back only the lines we want. If we
# have fewer lines to start with, move on.
# Keep the last lines of the file.
if old_num_lines > target_lines and keep_first_lines == False:
with open(source, 'w') as newfile:
if keep_header_lines > 0:
newfile.writelines(lines[ 0 : keep_header_lines ])
newfile.writelines(lines[ (old_num_lines - target_lines) : old_num_lines ])
# Keep the first lines of the file.
elif old_num_lines > target_lines and keep_first_lines == True:
with open(source, 'w') as newfile:
newfile.writelines(lines[ 0 : (target_lines + keep_header_lines) ])
else:
pass
# Remove the backup file if we have been successful. NOTE: The deleted file *will not*
# be sent to the Trash folder.
os.remove(backup)
indigo.server.log(u"Flat file truncated.")
except:
indigo.server.log(u"Something has gone wrong.")