Handlers¶

Handlers are what provide functionality to serialize to and deserialize from various file formats. All of the supported file formats should be attribute-value pair data exchange formats that can be serialized to and from dictionaries.

Whenever a new config instance is created some generic serialization and deserialization methods are magically created based on what handlers are currently available. The exposed methods are the following:

config_instance.dump_x(file_object) - dumps an instance to a file object

config_instance.dumps_x() - dumps the content to a string

ConfigClass.load_x(file_object) - loads an instance from a file object

ConfigClass.loads_x(string) - loads an instance from a string

where x is the generic name of the handler (e.g. json, ini, pickle, toml, etc…). I will commonly refer to these methods as the config serialization methods.

Important

Since handlers such as JSONHandler have support for multiple serializer libraries, a keyword prefer is supplied to all the config serialization methods. You can use this keyword to indicate what library you wish to use for that specific dump or load.

# if you have python-rapidjson installed but would prefer to use the builtin json
config_instance.dumps_json(prefer="json")

The value of the prefer keyword is the module name not the package name. For example pyyaml’s package name is pyyaml but it’s importable module name is yaml. So you would put prefer="yaml" rather than prefer="pyyaml".

For the following documentation I will be using the following config instance to showcase the handler’s serialization.

from typing import List, Dict
from enum import Enum
import file_config


class Status(Enum):
   STOPPED = 0
   STARTED = 1


@file_config.config
class ProjectConfig(object):

   @file_config.config
   class Dependency(object):
      name = file_config.var(str, min=1)
      version = file_config.var(file_config.Regex(r"^v\d+$"))

   name = file_config.var(str, min=1)
   type_ = file_config.var(str, name="type", required=False)
   keywords = file_config.var(List[str], min=0, max=10)
   status = file_config.var(Status)
   dependencies = file_config.var(Dict[str, Dependency])


config_instance = ProjectConfig(
   name="My Project",
   type_="personal-project",
   keywords=["example", "test"],
   status=Status.STOPPED,
   dependencies={
      "a-dependency": ProjectConfig.Dependency(name="A Dependency", version="v12")
   },
)

JSON¶

JSON is probably the most popular, supported, and straightforward data exchange formats available right now.

Available formatting options for JSON serializers are…

indent=2 - the number of spaces to use for indentation

sort_keys=True - sorts keys alphabetically in the resulting json

`json`¶

Uses the builtin json module for serialization.

config_instance.dumps_json(prefer="json", indent=2)

{
   "name": "My Project",
   "type": "personal-project",
   "keywords": [
      "example",
      "test"
   ],
   "status": 0,
   "dependencies": {
      "a-dependency": {
         "name": "A Dependency",
         "version": "v12"
      }
   }
}

python-rapidjson ¶

Compatability with python-rapidjson requires the installation of it as an extra…

pipenv install file-config[python-rapidjson]

Serialization is the same as the default json handler…

config_instance.dumps_json(prefer="rapidjson", indent=2)

{
   "name": "My Project",
   "type": "personal-project",
   "keywords": [
      "example",
      "test"
   ],
   "status": 0,
   "dependencies": {
      "a-dependency": {
         "name": "A Dependency",
         "version": "v12"
      }
   }
}

ujson ¶

In my opinion people shouldn’t be using ujson since they don’t follow the JSON spec and are thus incompatible with the default json module for many edge cases. But I know that lots of packages still depend on it (for unknown reasons).

Support for ujson requires it to be installed as an extra…

pipenv install file-config[ujson]

Usage is similar, however notice the resulting json lacks whitespace between key value pairs (a quirk of ujson).

config_instance.dumps_json(prefer="ujson", indent=2)

{
   "name":"My Project",
   "type":"personal-project",
   "keywords":[
      "example",
      "test"
   ],
   "status":0,
   "dependencies":{
      "a-dependency":{
         "name":"A Dependency",
         "version":"v12"
      }
   }
}

INI¶

INI is another popular configuration file format. Although it lacks features available in other file formats people still tend to use it over better solutions (TOML following a very similar specification).

Important

INI does not support arrays of dictionaries. Doing so would break the specification and be difficult to read. If your resulting dictionary from file_config.to_dict(config_instance) has an array containing dictionaries the config serialization methods for ini serialization will raise a ValueError.

If you need to structure your data this way, please switch to using TOML as that is one of their key features.

Available formatting options for INI serializers are…

root="root" - the name of the root section of the resulting ini

delimiter=":" - the delimiter to use to indicate nested dictionaries

empty_sections=True - allows empty ini sections to exist

`configparser`¶

Uses the builtin configparser module for serialization. A custom INIParser is used since the default configparser module is pretty lackluster.

config_instance.dumps_ini(prefer="configparser", root="root")

[root]
name = "My Project"
type = personal-project
keywords = example
   test
status = 0

[root:dependencies:a-dependency]
name = "A Dependency"
version = v12

Pickle¶

You really shouldn’t ever be serializing and storing things to pickle syntax since it has pretty serious security flaws with how it is loaded back in. Anyway, here is how you can do it in file_config.

There are no format options available for the pickle handler.

`pickle`¶

Uses the builtin pickle module for serialization.

config_instance.dumps_pickle(prefer="pickle")

\x80\x04\x95\xc6\x00\x00\x00\x00\x00\x00\x00\x8c\x0bcollections\x94\x8c\x0bOrderedDict\x94\x93\x94)R\x94(\x8c\x04name\x94\x8c\nMy Project\x94\x8c\x04type\x94\x8c\x10personal-project\x94\x8c\x08keywords\x94]\x94(\x8c\x07example\x94\x8c\x04test\x94e\x8c\x06status\x94K\x00\x8c\x0cdependencies\x94}\x94\x8c\x0ca-dependency\x94h\x02)R\x94(h\x04\x8c\x0cA Dependency\x94\x8c\x07version\x94\x8c\x03v12\x94usu.

TOML¶

There are a thousand libraries for parsing toml and they are all terrible. Maybe toml is just poorly designed but every single toml parsing library I’ve used (in Python) either doesn’t fully parse toml correctly or has some odd quirks that make it hard to work with.

Nevertheless, here are the three best libraries that currently exist for parsing toml.

The format options available to toml are the following:

inline_tables=["dependences.*"] - a glob pattern for inlining tables

tomlkit ¶

Using tomlkit requires it to be installed as an extra…

pipenv install file-config[tomlkit]

Usage is just as you might expect…

config_instance.dumps_toml(prefer="tomlkit", inline_tables=["dependencies.*"])

name = "My Project"
type = "personal-project"
keywords = ["example", "test"]
status = 0

[dependencies]
a-dependency = {name = "A Dependency",version = "v12"}

toml ¶

Using toml requires it to be installed as an extra…

pipenv install file-config[toml]

Usage is just the same as tomlkit…

config_instance.dumps_toml(prefer="toml", inline_tables=["dependencies.*"])

name = "My Project"
type = "personal-project"
keywords = [ "example", "test",]
status = 0

[dependencies]
a-dependency = { name = "A Dependency", version = "v12" }

pytoml ¶

Using pytoml requires it to be installed as an extra…

pipenv install file-config[pytoml]

Pytoml does not support serializing inline tables in any easy manner. So the inline_tables keyword won’t be applied when dumping with pytoml.

config_instance.dumps_toml(prefer="pytoml")

name = "My Project"
type = "personal-project"
keywords = ["example", "test"]
status = 0

[dependencies]

[dependencies.a-dependency]
name = "A Dependency"
version = "v12"

YAML¶

Yaml is a data exchange langauge used by many projects (Docker, TravisCI, etc…) for simple configuration files.

There are no format options for yaml serialization.

pyyaml (yaml)¶

Using pyyaml requires it to be installed as an extra…

pipenv install file-config[pyyaml]

Usage is straightforward…

config_instance.dumps_yaml(prefer="yaml")

name: My Project
type: personal-project
keywords: [example, test]
status: 0
dependencies:
a-dependency:
   name: A Dependency
   version: v12

Message Pack¶

MessagePack is a byte sized json format which retains the same structure as json. It’s valuable for quick and fast json streams over network protocols.

There are no format options for msgpack.

msgpack ¶

Using msgpack requires it to be installed as an extra…

pipenv install file-config[msgpack]

Usage is just as you would expect…

config_instance.dumps_msgpack(prefer="msgpack")

\x85\xa4name\xaaMy Project\xa4type\xb0personal-project\xa8keywords\x92\xa7example\xa4test\xa6status\x00\xacdependencies\x81\xaca-dependency\x82\xa4name\xacA Dependency\xa7version\xa3v12

XML¶

XML is an older data exchange format that follows a nested tag-attribute type structure. A custom XMLParser is used since the many dictionary to xml helper packages out there are not reflective.

The format options available to xml are the following:

root="root" - the name of the root element in the resulting xml

pretty=True - indicates that the resulting xml should be pretty formatted

xml_declaration=True - indicates that the xml header should be added

encoding="utf-8" - the encoding to use for the resulting xml document

lxml ¶

Using XML requires lxml to be installed as an extra…

pipenv install file-config[lxml]