Software Development/Application Configuration

When you're coding for yourself, it's easy to keep all the variables inside the code files, since you can just open the files and change the values whenever you want. But when you have outside users, you don't necessarily want them changing the code, and if the code is compiled into binary executables, they can't change it anyway. Plus, separating code and data makes your program easier to think about. So you'll want to put the user-settable data in external files.

There are at least two questions to answer: what format to store the settings in and where to store them.

Format

Here are some discussions that cover several formats:

• A Configuration Comparison in Python - PyCon 2009 - PyVideo (video).
• Configuration files in Python - Martin Thoma - Examples of most of the formats I'm listing here.
• Python configuration file: Any file format recommendation? INI format still appropriate? Seems quite old school - Stack Overflow
• What's the best practice using a settings file in Python? - Stack Overflow
• Configuration file with list of key-value pairs in python - Stack Overflow
• What is the difference between YAML and JSON? When to prefer one over the other - Stack Overflow
• Why bother with python and config files? - RobG3d

I've sorted these formats roughly in order of increasing complexity.

CSV

• csv - The Python Standard Library

Pros:

• It's simple.
• Some version of it is supported in many languages.
• It's fairly easy to import and export.

Cons:

• It's a little hard to read, especially when the lines get long.
• It's limited to simple data.
• It doesn't enforce data types.
• It doesn't have a standard format.

INI

• configparser - The Python Standard Library
• ConfigObj - PyPI - This one has a few more features than configparser.

Pros:

• It's easy to read.
• It's easy to avoid syntax errors (given a particular format).
• Some version of it is supported in many languages.
• It's easy to import and export.

Cons:

• It's limited to simple data.
• It doesn't enforce data types (unless you use ConfigObj).
• It doesn't have a standard format.

JSON

• json - The Python Standard Library

Pros:

• It enables multiple data types.
• It enables complex data structures.
• It's supported in many languages.
• It's easy to import and export.

Cons:

• It's easy to commit syntax errors, especially if you're not familiar with the format.
• It can be hard to read, especially if it's not pretty printed or if it uses complex data structures.

YAML

• PyYAML

Pros:

• It enables multiple data types.
• It enables complex data structures.
• It's supported in many languages.
• It's easy to import and export.
• It's easy to read if you stick to simple data structures.

Cons:

• It can be hard to write if you use its more complex features.

I'm trying out YAML as my default choice.

XML

There are a few major Python tools for working with XML:

• lxml.etree Tutorial - lxml
• Beautiful Soup - Crummy: The Site - It's mainly meant for HTML, but it parses XML too.
• xml.etree.ElementTree - The Python Standard Library

Here's some discussion of the use of XML for configuration:

• Hierarchical Configurations - Apache Commons Configuration User Guide

Pros:

• It enables multiple data types.
• It enables complex data structures.
• It's supported in many languages.
• It's easy to import.

Cons:

• It doesn't have a standard format for representing configuration.
• It can be hard to read.
• It can be hard to export if the starting format isn't compatible.
• Enforcing data types requires an additional schema.

Python

You can use regular Python code to define settings, usually by putting them in their own module. I'd personally only use Python for configuration in special cases. Here are some recommendations and warnings:

• Loading settings.py config file into a dict - Stack Overflow
• load python code at runtime - Stack Overflow
• Most Pythonic way to provide global configuration variables in config.py? - Stack Overflow
• pyconfig - PyPI
• Python is Not a Configuration File Format - Lateral Opinion

Pros:

• It enables multiple data types.
• It enables complex data structures.
• It enables using conditions to set values.
• If your code is already in Python, it doesn't require learning an additional language.
• Importing is very easy.

Cons:

• It mixes code and data, which makes the program harder to think about.
• It can create a security risk, since users can add arbitrary code to the configuration file for the program to run.
• Exporting can be hard or impossible.
• It can be hard to read, depending on the code.
• It doesn't have a standard format for representing configuration.
• It can be easy to commit syntax errors.

Database

The formats I've already covered are plain text formats that a human can read and edit. Another option is putting the settings in a binary database and accessing them through a separate UI. The advantages are that you can enforce data types and the user isn't editing the settings file directly, which leaves it less open to syntax errors. For settings to be set by non-technical end users, this is probably the best option.

• sqlite3 - The Python Standard Library

Pros:

• It's hard for the user to commit syntax errors.
• It enables multiple data types.
• You can control access to the settings.
• It's supported in many languages.
• It can be easy to import and export.

Cons:

• Complex data structures are hard to represent, or they require an additional format.
• You have to code a UI for viewing and setting values.

Location

Where should you put your config file? Depending on the circumstances, you might need more than one file, and they might need to go in different locations. Generally speaking, you might separate global and user settings into different files. Global settings might go in the app's root directory or a config subdirectory, and user settings somewhere in a user directory.

• appdirs - PyPI - A Python module for finding various directories across platforms, such as the user data, a possible location for a config file.