Automating Daily Filesystem and System Maintenance with Python

Learn how to automate daily filesystem and system maintenance tasks using Python's built-in standard libraries.

In this tutorial, you'll learn how to leverage Python's built-in standard libraries—specifically pathlib, shutil, os, and glob—to automate daily filesystem and system maintenance tasks. We'll cover practical examples for finding files, moving and archiving logs, and writing safe cleanup scripts. By the end, you'll have a robust set of tools to help you manage your filesystem and system more efficiently.

Why Automate with Python?

Python is a versatile language that can handle a wide range of tasks, from simple text processing to complex data analysis. When it comes to system administration and filesystem management, Python's standard libraries provide a powerful and flexible toolkit. These libraries are built into Python, so you don't need to install any additional dependencies.

Setting Up Your Environment

Before we dive into the code, make sure you have Python installed on your system. You can check the version by running:

python3 --version

We'll use Python 3 for this tutorial, as it is the most widely used version.

Finding Files with `glob` and `pathlib`

The glob module in Python provides a way to match filenames against a pattern. The pathlib module allows us to work with filesystem paths in a more object-oriented way. Together, they make it easy to find and manipulate files.

Example: Recursively Find All Log Files

Let's start by writing a script that finds all log files in a directory and its subdirectories. We'll use glob to match the pattern and pathlib to handle the paths.

import glob
from pathlib import Path
 
# Define the directory to search
log_dir = Path('/var/log')
 
# Find all log files recursively
log_files = glob.glob(str(log_dir / '**' / '*.log'), recursive=True)
 
for log_file in log_files:
    print(log_file)

Example: Find Files Modified in the Last 24 Hours

You can also use glob to find files modified within a specific time frame. Here's how to find files modified in the last 24 hours.

import time
from pathlib import Path
 
# Define the directory to search
log_dir = Path('/var/log')
 
# Get the current time
now = time.time()
 
# Find all log files modified in the last 24 hours
log_files = glob.glob(str(log_dir / '**' / '*.log'), recursive=True)
 
for log_file in log_files:
    file_path = Path(log_file)
    file_time = file_path.stat().st_mtime
    if now - file_time < 24 * 60 * 60:
        print(file_path)

Moving and Archiving Logs with `shutil`

The shutil module provides a high-level interface for common file operations, including moving and copying files. We can use it to move old log files to an archive directory.

Example: Move and Archive Old Log Files

Let's write a script that moves old log files to an archive directory.

import glob
from pathlib import Path
import shutil
 
# Define the source and archive directories
log_dir = Path('/var/log')
archive_dir = Path('/var/log/archive')
 
# Create the archive directory if it doesn't exist
archive_dir.mkdir(parents=True, exist_ok=True)
 
# Find all log files modified in the last 24 hours
log_files = glob.glob(str(log_dir / '**' / '*.log'), recursive=True)
 
for log_file in log_files:
    file_path = Path(log_file)
    file_time = file_path.stat().st_mtime
    if now - file_time < 24 * 60 * 60:
        # Define the new path for the archived file
        archive_path = archive_dir / file_path.name
        # Move the file to the archive directory
        shutil.move(file_path, archive_path)
        print(f"Moved {file_path} to {archive_path}")

Safe Cleanup Scripts with `os` and `shutil`

When performing cleanup tasks, it's important to ensure that files are deleted safely. The os and shutil modules provide functions to delete files and directories, with options to avoid errors if the files don't exist.

Example: Safe File Deletion

Let's write a script that safely deletes old log files that are more than 30 days old.

import glob
import os
import time
from pathlib import Path
import shutil
 
# Define the source directory
log_dir = Path('/var/log')
 
# Get the current time
now = time.time()
 
# Find all log files modified more than 30 days ago
log_files = glob.glob(str(log_dir / '**' / '*.log'), recursive=True)
 
for log_file in log_files:
    file_path = Path(log_file)
    file_time = file_path.stat().st_mtime
    if now - file_time > 30 * 24 * 60 * 60:
        try:
            # Remove the file
            os.remove(file_path)
            print(f"Deleted {file_path}")
        except OSError as e:
            print(f"Error: {file_path} : {e.strerror}")

Example: Safe Directory Deletion

Now, let's extend the script to delete old directories that are empty or contain only log files.

import glob
import os
import time
from pathlib import Path
import shutil
 
# Define the source directory
log_dir = Path('/var/log')
 
# Get the current time
now = time.time()
 
# Find all log files modified more than 30 days ago
log_files = glob.glob(str(log_dir / '**' / '*.log'), recursive=True)
 
# Find all directories modified more than 30 days ago
dir_files = glob.glob(str(log_dir / '**' / '*'), recursive=True)
dirs_to_delete = [Path(d) for d in dir_files if os.path.isdir(d)]
 
for log_file in log_files:
    file_path = Path(log_file)
    file_time = file_path.stat().st_mtime
    if now - file_time > 30 * 24 * 60 * 60:
        try:
            # Remove the file
            os.remove(file_path)
            print(f"Deleted {file_path}")
        except OSError as e:
            print(f"Error: {file_path} : {e.strerror}")
 
for dir_to_delete in dirs_to_delete:
    if not os.listdir(dir_to_delete):
        try:
            # Remove the empty directory
            shutil.rmtree(dir_to_delete)
            print(f"Deleted {dir_to_delete}")
        except OSError as e:
            print(f"Error: {dir_to_delete} : {e.strerror}")

Conclusion

In this tutorial, we've covered how to automate daily filesystem and system maintenance tasks using Python's built-in standard libraries. We've seen how to find files, move and archive logs, and write safe cleanup scripts. These techniques can be adapted to fit various use cases, from simple file management to more complex system administration tasks.

By automating these tasks, you can save time and reduce the risk of human error. Remember to test your scripts in a safe environment before deploying them to a production system. Happy scripting!