Building Your Own Utilities

Database Regeneration - Practical Scripting

Course 1 · Chapter 6 · Building Your Own Utilities

🛠️ Building Your Own Utilities

You've learned the core patterns: paths, configuration, databases, and automation. Now it's time to take these patterns and apply them to build reusable utilities for any project. This final chapter teaches best practices for structure, documentation, testing, and sharing your utilities.

📋 The Utility Template

Every utility you build should follow this structure:

my-utility/
├── .env.example              ← Configuration template
├── requirements.txt          ← Python dependencies
├── README.md                 ← Usage instructions
├── utils.py                  ← Core functionality
├── main.py                   ← Entry point
└── tests/
    └── test_utils.py        ← Unit tests

✅ Best Practices

1. Separate Concerns

# ❌ BAD: Everything in one file
main.py:
  - Load config
  - Connect to database
  - Process files
  - Insert into database
  - Display results

# ✅ GOOD: Separate modules
config.py        → Configuration loading
db_utils.py      → Database operations
file_utils.py    → File operations
main.py          → Orchestration

2. Configuration Management

# ✅ GOOD: All config in one place
config.py:
  - Load from .env
  - Set defaults
  - Validate
  - Return structured config

# ✅ GOOD: .env.example for documentation
.env.example:
  DB_HOST=localhost
  DB_USER=your_username
  DB_PASS=your_password

3. Error Handling

# ✅ GOOD: Graceful failure
try:
    # Main operation
except SpecificError as e:
    print(f"❌ {e}")
    return False
finally:
    # Cleanup

4. Type Hints

def process_file(filepath: str, config: dict) -> bool:
    """Process a file and return success status."""
    pass

def find_files(directory: str) -> list[str]:
    """Find all files in directory."""
    pass

5. Documentation

# README.md should explain:

1. What it does (1 paragraph)
2. Installation (pip install -r requirements.txt)
3. Setup (.env configuration)
4. Usage (how to run it)
5. Examples (real-world usage)
6. Troubleshooting (common issues)

🔄 Adapting Patterns to New Projects

From TypeScript Course to Any Course

Component	TypeScript Example	Generic Pattern
Find Files	glob("ts1-*.html")	glob("{prefix}-*.{ext}")
Parse Metadata	Extract from <h2>	Extract key data from file
Database Entity	TypeScript Fundamentals	Any course name
Insert Operation	insert_chapter_with_content()	insert_<item>()

Real-World Examples

Example 1: Import Blog Posts

Find: blog-*.md files
Parse: Extract title, date, content
Database: Insert into posts table

Example 2: Sync User Data

Find: users.csv file
Parse: Read CSV rows
Database: Update users table

Example 3: Process Images

Find: *.jpg files in directory
Parse: Extract metadata (size, dimensions)
Database: Insert into images table

🧪 Testing Your Utility

Unit Tests

import pytest
from utils import parse_metadata, find_files

def test_parse_metadata():
    """Test metadata parsing."""
    result = parse_metadata("chapter-1.html")
    assert result['number'] == 1
    assert result['title'] is not None

def test_find_files(tmp_path):
    """Test file discovery."""
    # Create test files
    (tmp_path / "chapter-1.html").touch()
    (tmp_path / "chapter-2.html").touch()

    # Test finding them
    files = find_files(tmp_path)
    assert len(files) == 2

Integration Tests

def test_full_workflow(tmp_path, test_db):
    """Test complete workflow."""
    # Create test files
    # Run main workflow
    # Verify database
    pass

🎯 Common Utility Patterns

Pattern 1: File Discovery + Database Insert

Find files → Parse → Insert into database

Pattern 2: Data Transformation

Read source → Transform → Write result

Pattern 3: API Sync

Fetch from API → Parse → Store in database

Pattern 4: CSV/Excel Import

Read file → Parse rows → Insert records

Pattern 5: Batch Processing

Find items → Process each → Report results

📝 Documentation Checklist

✅ README.md with overview and usage

✅ .env.example showing all config options

✅ requirements.txt with all dependencies

✅ Type hints on all functions

✅ Docstrings on modules and functions

✅ Error messages that explain what went wrong

✅ Examples of how to use the utility

✅ Troubleshooting section for common issues

💻 Final Challenges

Challenge 1: Create a Blog Post Importer

Build a utility that:

Finds all blog-*.md files
Parses title and date from front matter
Inserts into database
Reports count

Goal: Apply patterns to a new domain (blog posts vs. courses).

→ Solution

Challenge 2: Data Transformation Utility

Build a utility that:

Reads a CSV file
Transforms data (cleanup, validation)
Writes to database
Generates report

Goal: Practice the data transformation pattern.

→ Solution

Challenge 3: Reusable Utility Package

Build a complete utility package with:

Main script with proper structure
Configuration management
Error handling and logging
Unit tests
Complete documentation

Goal: Build a production-ready utility you could share.

→ Solution

🎓 Course Conclusion

🏆 You've Mastered

✅ File system navigation with Path and glob
✅ Configuration management with .env
✅ Database connections and operations
✅ Building reusable utility modules
✅ Complete automation workflows
✅ Best practices for production code

What You Can Build Now

With these patterns, you can build utilities for:

📚 Course content management systems
📝 Blog post importers
📊 Data migration tools
🔄 API sync utilities
📁 Batch file processors
🗄️ Database administration tools
🚀 CI/CD automation scripts

Next Steps

Pick a project: What do you want to automate?
Follow the template: Use the structure from this course
Test thoroughly: Write unit and integration tests
Document well: Help future users understand it
Share it: Put it on GitHub so others can use it

💡 Remember

The patterns you've learned work for any project. The details change (different file types, different databases, different APIs), but the structure stays the same. Master the patterns, adapt them to your needs, and you can build anything!