Handlers¶
Handlers are what provide functionality to serialize to and deserialize from various file formats. All of the supported file formats should be attribute-value pair data exchange formats that can be serialized to and from dictionaries.
Whenever a new config instance is created some generic serialization and deserialization methods are magically created based on what handlers are currently available. The exposed methods are the following:
config_instance.dump_x(file_object)
- dumps an instance to a file object
config_instance.dumps_x()
- dumps the content to a string
ConfigClass.load_x(file_object)
- loads an instance from a file object
ConfigClass.loads_x(string)
- loads an instance from a string
where x
is the generic name of the handler (e.g. json
, ini
, pickle
,
toml
, etc…). I will commonly refer to these methods as the
config serialization methods.
Important
Since handlers such as JSONHandler
have support for multiple serializer libraries, a keyword prefer
is supplied
to all the config serialization methods. You can use this keyword to indicate what
library you wish to use for that specific dump or load.
# if you have python-rapidjson installed but would prefer to use the builtin json
config_instance.dumps_json(prefer="json")
The value of the prefer
keyword is the module name not the package name.
For example pyyaml’s package name is pyyaml
but it’s importable module name is yaml
. So you would put prefer="yaml"
rather than prefer="pyyaml"
.
For the following documentation I will be using the following config instance to showcase the handler’s serialization.
from typing import List, Dict
from enum import Enum
import file_config
class Status(Enum):
STOPPED = 0
STARTED = 1
@file_config.config
class ProjectConfig(object):
@file_config.config
class Dependency(object):
name = file_config.var(str, min=1)
version = file_config.var(file_config.Regex(r"^v\d+$"))
name = file_config.var(str, min=1)
type_ = file_config.var(str, name="type", required=False)
keywords = file_config.var(List[str], min=0, max=10)
status = file_config.var(Status)
dependencies = file_config.var(Dict[str, Dependency])
config_instance = ProjectConfig(
name="My Project",
type_="personal-project",
keywords=["example", "test"],
status=Status.STOPPED,
dependencies={
"a-dependency": ProjectConfig.Dependency(name="A Dependency", version="v12")
},
)
JSON¶
JSON is probably the most popular, supported, and straightforward data exchange formats available right now.
Available formatting options for JSON serializers are…
indent=2
- the number of spaces to use for indentation
sort_keys=True
- sorts keys alphabetically in the resulting json
json
¶
Uses the builtin json
module for serialization.
config_instance.dumps_json(prefer="json", indent=2)
{
"name": "My Project",
"type": "personal-project",
"keywords": [
"example",
"test"
],
"status": 0,
"dependencies": {
"a-dependency": {
"name": "A Dependency",
"version": "v12"
}
}
}
python-rapidjson¶
Compatability with python-rapidjson
requires the installation of it as an extra…
pipenv install file-config[python-rapidjson]
Serialization is the same as the default json handler…
config_instance.dumps_json(prefer="rapidjson", indent=2)
{
"name": "My Project",
"type": "personal-project",
"keywords": [
"example",
"test"
],
"status": 0,
"dependencies": {
"a-dependency": {
"name": "A Dependency",
"version": "v12"
}
}
}
ujson¶
In my opinion people shouldn’t be using ujson
since they don’t follow the JSON spec
and are thus incompatible with the default json
module for many edge cases.
But I know that lots of packages still depend on it (for unknown reasons).
Support for ujson
requires it to be installed as an extra…
pipenv install file-config[ujson]
Usage is similar, however notice the resulting json lacks whitespace between key value
pairs (a quirk of ujson
).
config_instance.dumps_json(prefer="ujson", indent=2)
{
"name":"My Project",
"type":"personal-project",
"keywords":[
"example",
"test"
],
"status":0,
"dependencies":{
"a-dependency":{
"name":"A Dependency",
"version":"v12"
}
}
}
INI¶
INI is another popular configuration file format. Although it lacks features available in other file formats people still tend to use it over better solutions (TOML following a very similar specification).
Important
INI does not support arrays of dictionaries. Doing so would break
the specification and be difficult to read. If your resulting dictionary from
file_config.to_dict(config_instance)
has an array containing dictionaries the
config serialization methods for ini serialization will raise a ValueError
.
If you need to structure your data this way, please switch to using TOML as that is one of their key features.
Available formatting options for INI serializers are…
root="root"
- the name of the root section of the resulting ini
delimiter=":"
- the delimiter to use to indicate nested dictionaries
empty_sections=True
- allows empty ini sections to exist
configparser
¶
Uses the builtin configparser
module for serialization. A custom
INIParser
is used since the default configparser
module is pretty lackluster.
config_instance.dumps_ini(prefer="configparser", root="root")
[root]
name = "My Project"
type = personal-project
keywords = example
test
status = 0
[root:dependencies:a-dependency]
name = "A Dependency"
version = v12
Pickle¶
You really shouldn’t ever be serializing and storing things to pickle
syntax
since it has pretty serious security flaws with how it is loaded back in. Anyway, here
is how you can do it in file_config
.
There are no format options available for the pickle handler.
pickle
¶
Uses the builtin pickle
module for serialization.
config_instance.dumps_pickle(prefer="pickle")
\x80\x04\x95\xc6\x00\x00\x00\x00\x00\x00\x00\x8c\x0bcollections\x94\x8c\x0bOrderedDict\x94\x93\x94)R\x94(\x8c\x04name\x94\x8c\nMy Project\x94\x8c\x04type\x94\x8c\x10personal-project\x94\x8c\x08keywords\x94]\x94(\x8c\x07example\x94\x8c\x04test\x94e\x8c\x06status\x94K\x00\x8c\x0cdependencies\x94}\x94\x8c\x0ca-dependency\x94h\x02)R\x94(h\x04\x8c\x0cA Dependency\x94\x8c\x07version\x94\x8c\x03v12\x94usu.
TOML¶
There are a thousand libraries for parsing toml and they are all terrible. Maybe toml is just poorly designed but every single toml parsing library I’ve used (in Python) either doesn’t fully parse toml correctly or has some odd quirks that make it hard to work with.
Nevertheless, here are the three best libraries that currently exist for parsing toml.
The format options available to toml are the following:
inline_tables=["dependences.*"]
- a glob pattern for inlining tables
tomlkit¶
Using tomlkit requires it to be installed as an extra…
pipenv install file-config[tomlkit]
Usage is just as you might expect…
config_instance.dumps_toml(prefer="tomlkit", inline_tables=["dependencies.*"])
name = "My Project"
type = "personal-project"
keywords = ["example", "test"]
status = 0
[dependencies]
a-dependency = {name = "A Dependency",version = "v12"}
toml¶
Using toml requires it to be installed as an extra…
pipenv install file-config[toml]
Usage is just the same as tomlkit…
config_instance.dumps_toml(prefer="toml", inline_tables=["dependencies.*"])
name = "My Project"
type = "personal-project"
keywords = [ "example", "test",]
status = 0
[dependencies]
a-dependency = { name = "A Dependency", version = "v12" }
pytoml¶
Using pytoml requires it to be installed as an extra…
pipenv install file-config[pytoml]
Pytoml does not support serializing inline tables in any easy manner. So the
inline_tables
keyword won’t be applied when dumping with pytoml.
config_instance.dumps_toml(prefer="pytoml")
name = "My Project"
type = "personal-project"
keywords = ["example", "test"]
status = 0
[dependencies]
[dependencies.a-dependency]
name = "A Dependency"
version = "v12"
YAML¶
Yaml is a data exchange langauge used by many projects (Docker, TravisCI, etc…) for simple configuration files.
There are no format options for yaml serialization.
pyyaml (yaml)¶
Using pyyaml requires it to be installed as an extra…
pipenv install file-config[pyyaml]
Usage is straightforward…
config_instance.dumps_yaml(prefer="yaml")
name: My Project
type: personal-project
keywords: [example, test]
status: 0
dependencies:
a-dependency:
name: A Dependency
version: v12
Message Pack¶
MessagePack is a byte sized json format which retains the same structure as json. It’s valuable for quick and fast json streams over network protocols.
There are no format options for msgpack.
msgpack¶
Using msgpack requires it to be installed as an extra…
pipenv install file-config[msgpack]
Usage is just as you would expect…
config_instance.dumps_msgpack(prefer="msgpack")
\x85\xa4name\xaaMy Project\xa4type\xb0personal-project\xa8keywords\x92\xa7example\xa4test\xa6status\x00\xacdependencies\x81\xaca-dependency\x82\xa4name\xacA Dependency\xa7version\xa3v12
XML¶
XML is an older data exchange format that follows a
nested tag-attribute type structure. A custom XMLParser
is
used since the many dictionary to xml helper packages out there are not reflective.
The format options available to xml are the following:
root="root"
- the name of the root element in the resulting xml
pretty=True
- indicates that the resulting xml should be pretty formatted
xml_declaration=True
- indicates that the xml header should be added
encoding="utf-8"
- the encoding to use for the resulting xml document
lxml¶
Using XML requires lxml
to be installed as an extra…
pipenv install file-config[lxml]
Usage is straightforward…
config_instance.dumps_xml(prefer="lxml", pretty=True, xml_declaration=True)
<?xml version='1.0' encoding='UTF-8'?>
<ProjectConfig>
<name type="str">My Project</name>
<type type="str">personal-project</type>
<keywords>
<keywords type="str">example</keywords>
<keywords type="str">test</keywords>
</keywords>
<status type="int">0</status>
<dependencies>
<a-dependency>
<name type="str">A Dependency</name>
<version type="str">v12</version>
</a-dependency>
</dependencies>
</ProjectConfig>