Implementing Your Own Backend¶

Support for this is still incredibly provisional. In theory, all you need to do is implement the PicPocket protocol and everything should work. Unfortunately, getting initialize or load requires modifying the APIS dict in the base module, so it can’t yet be done in a way that allows use of the cli <api/cli>.

See the definition of testenv:py{311}-tests-postgres in tox.ini for information on how to run all tests against your backend.

Adding Support for a New DBAPI 2.0-Compatible Database¶

If you want to support a database that has an asynchronous DBAPI-compatible package, it’s fairly straight-forward. PicPocket’s PostgreSQL and SQLite backends use the same code for almost all operations, and should require very little changes to the non-shared code as features are added to PicPocket.

You will need to subclass picpocket.database.dbapi.DbApi and picpocket.database.logic.SQL.

In practice, this might be slightly more difficult if your database’s functionality differs greatly from both PostgreSQL and SQLite.

DbApi¶

class picpocket.database.dbapi.DbApi(*args, **kwargs)¶

A DBAPI 2.0 Implementation of PicPocket

This class implements all required to support the PicPocket protocol other than the configuration and mounts properties and the parse_connection_info, initialize, get_api_version, get_version, and matching_version methods

To add support for a new database you need to implement those methods yourself as well as some new methods related to getting connection and cursor objects, and a class for generating sql statements (which is largely modelled on psycopg’s sql module).

It will be assumed that an implementation is using the tables listed as class attributes (presently, LOCATIONS_TABLE, TASKS_TABLE, TASK_INVOCATIONS_TABLE, IMAGES_TABLE), that each of those tables will be named that attribute name in lowercase and without the _table. For all tables, it will be assumed that all columns listed exist. If your table uses different types, you will need to override that class attribute

Todo

DbApi additionaly assumed a tags table (with columns id, name, escaped_name, depth, description) and an image_tags table (with columns image, tag).

Todo

DbApi only really does type formatting on columns it knows differ between the postgres and sqlite. Once PicPocket adds proper JSON support for filtering on exif data, the Types system will probably be addapted so that it accounts for input and output formats (e.g., postgres’s JSON types expects text input but provides a deserialized output vs. sqlite’s where input and output will both be strings but json operations are supported).

abstract property sql: SQL¶

The sql object to use when dynamically generating sql.

See picpocket.types

abstract async connect()¶

Establish a connection with the underlying database.

Returns:: A dbapi-compatible Connection object

abstract cursor(connection, commit: bool = False) → AsyncContextManager¶

Yield a cursor that rolls back on error and optionally commits

Parameters:

connection – A connection object returned by connect
commit – Whether to commit on (successful) exit

Returns:

A dbapi-compatible Cursor object

async ensure_foreign_keys(cursor)¶

Ensure a cursor will support cascading changes.

This will be a no-op for most DBs (or an exception for DBs that cannot support it I guess?) and only exists because an SQLite cursor needs to run a pragma to enable this and we don’t want to do that on every single cursor creation.

async import_data(path: Path, locations: list[str] | dict[str, Optional[pathlib.Path]] | None = None)¶

Import a PicPocket backup

Load data (locations, tasks, images, tags) from a PicPocket backup created using export_data(). This is the recommended way to migrate between backends.

This method is safe to rerun as PicPocket will skip duplicates instead of overwriting.

Note

This file will not contain the images themselves, just the metadata you’ve created for the image (tags, captions, alt text, etc.).

Parameters:

path – The JSON file to load data from.
locations – Only import data related to this set of locations. This can be supplied either as a list of locations or as a dict of location/path pairs. If a path is provided, it will be mounted prior to loading images and then unmounted afterward. When importing a location without a set path, this must be provided as a dict. For locations with paths definied within the backup, you can pass None as the path to use the default location.

async export_data(path: Path, locations: list[str | int] | None = None)¶

Create a PicPocket backup

Export data (locations, tasks, iamges, tags) from PicPocket. This is the recommended way to migrate between backends.

Warning

This file will not contain the images themselves, just the metadata you’ve created for the image (tags, captions, alt text, etc.).

Parameters:

path – Where to save teh JSON file to store data to.
locations – Only export images and task related to this location.

async add_location(name: str, path: Path | None = None, *, description: str | None = None, source: bool = False, destination: bool = False, removable: bool = True) → int¶

Add a new location to PicPocket.

Locations represent sources to fetch images from (e.g., cameras) and storage locations to store data in. A location must be specified either as a source to import images from, a destination to import images to, or both.

Parameters:

name – The (unique) name of the location being added. E.g. photos, “EOS 6D”, phone
path – The path where the location is/gets mounted. If this path is not consistent (or not unique), you may want to leave this blank and instead mount/unmount the drive as necessary.
description – A text description of what the location is/how it is used. E.g., “main photo storage”, “my digital camera”
source – Whether the location is somewhere images may be copied from (to a new location)
destination – Whether the location is somewhere images may be compied to.
removable – Whether the location is a permanent or removable storage device.

Returns:

The integer id representing the location

Edit a PicPocket location.

Edit information about about a PicPocket location.

Parameters:

name/id – the name or id of the location.
new_name – What to rename the location to.
path – Where the location gets mounted.
description – A description of the location.
source – Whether the location is a place images are imported from
destination – Whether the location is a place images are imported to.
removable – Whether the location is a permanent or removable storage device.

async remove_location(name_or_id: int | str, /, *, force: bool = False)¶

Remove a location from PicPocket.

The location will not be removed if it has images associated with it unless explicitly asked. Even in that case, it will not touch the image files on disk.

Parameters:

name/id – the name or of the location.
force – Remove the location even if it currently has images associated with it and remove those images from the database.

Returns:

Whether the location was deleted.

async get_location(name_or_id: str | int, /, *, expect: bool = False) → Location | None¶

Get information about a location

Parameters:

name/id – The location to find
expect – Error if the location doesn’t exist

Returns:

The matching location if it exists.

async list_locations() → list[picpocket.database.types.Location]¶: Fetch all known locations

async mount(name_or_id: str | int, /, path: Path)¶

Supply a path where a location is currently available

Mount a location to a specific path. For removable locations without unique or consistent mount points, this is how you alert PicPocket to its present location. This will also work for locations that have set attach points but are temporarily available at a different location. Mounting an already-mounted location will replace the previous mount point.

Parameters:

name/id – The location to mount
path – Where to mount the location

async unmount(name_or_id: str | int)¶

Remove a temporary mount point

Unmount a location that was previously mounted. Unmounting a drive that isn’t mounted won’t cause an error.

Parameters:: name/id – The location to mount

async import_location(name_or_id: str | int, /, *, file_formats: set[str] | None = None, batch_size: int = 1000, creator: str | None = None, tags: list[str] | None = None) → list[int]¶

Import all images from a location

Import all images that are stored within a location. If an image already exists within PicPocket, its user-supplied won’t be updated but properties of the file itself (such as dimensions or exif data) will be updated in PicPocket if its changed on disk.

Parameters:

name/id – The location to import
file_formats – Only import images that have these suffixes. Suffixes can be supplied with or without leading dot. If not supplied, the PicPocket default file formats will be imported.
batch_size – How often to commit while importing images.
creator – Mark all images as having been created by this person.
tags – Apply all these tags to all imported images.

Returns:

The ids of all images that were added (or edited)

Add a repeatable task to PicPocket.

Todo

Perform operations on images, clean up path stuff, scheduled and automatic tasks

Warning

This method will be changing substantially as support for other types of task are added.

Add a task to PicPocket. Currently, all supported tasks are imports from one known location to another. Support for things like converting images on import are forthcoming. Images will be copied from the source location without modifying the source image. Tasks will refuse to overwrite existing files on import.

see picpocket.tasks

Parameters:

name – What to call the task (must be unique)
source – The name/id of the location to import images from
destination – The name/id of the location to copy images to
creator – Who to list as the creator of all imported images
tags – Apply these tags to all imported images
source_path –
Where, (relative to the source location’s root) to start looking for images. If not supplied, root will be used. The following dyamic directory names are allowed:
- {year}: A year
- {month}: A month
- {day}: A day
- {date:FORMAT}: A date in the supplied format
- {regex:PATTERN}: A name that matches a regex
- {str:RAW}: A folder name that contains {
destination_format –
Where (relative to the destination root) images should be copied to. This is supplied as a format string with the following allowed formatters:
- directory: The source filepath (relative to the source) root.
- file: The file’s name
- name: The file’s name (without extension)
- extension: The file’s extension (without the leading dot).
- uuid: A UUID4 hex
- date: A date object representing the source image’s last modified time. You can supply the format to save the date using strftime formatting (e.g. date:%Y-%m-%d-%H-%M-%S)
- hash: A hash of the image contents
- index: A 1-indexed number representing the order images are encountered by the running task. PicPocket makes no promises about the order images are added
file_formats – Only import images that have these suffixes. Suffixes can be supplied with or without leading dot. If not supplied, the PicPocket default file formats will be imported.
force – Overwrite any previously-existing task with this name

async run_task(name: str, *, since: datetime | None = None, full: bool = False, tags: list[str] | None = None) → list[int]¶

Run a task

Todo

Allow offsets for last ran.

Running a task will never overwrite an image file, but it will re-add images that were deleted if the source image still matches all criteria. When running a task, the time the task previously ran will be supplied and any specified date-based path segments from before that last ran date will be skipped, as will any files with last-modified dates that are earlier.

Parameters:

name – The name of the task to run
since – Ignore files with last-modified dates older than this date and in folders representing earlier dates. Ignored if full is supplied.
full – Don’t filter directories based on the last-run date of the task.
tags – Any tags to include to this specific run

Returns:

The ids of all imported images.

async remove_task(name: str)¶

Remove a task

Running this on a non-existent task will not error.

Parameters:: remove (name The name of the task to) –

async get_task(name: str) → Task | None¶: Get the definition of an existing task

async list_tasks() → list[picpocket.database.types.Task]¶: Get all tasks in PicPocket

Copy an image into PicPocket

Create a copy of a supplied image and then add that image to PicPocket.

Parameters:

source – The image to create a copy of
name/id – The location to create a copy of the image in
destination – The path (relative to the location root) to copy the image to
creator – Who made the image
title – The title of the image
caption – A caption of the image
alt – Descriptive text of the image to be used as alt text
rating – A numeric rating of the image
tags – Any number of tags to apply to the image

Returns:

The id of the copied image

async edit_image(id: int, *, creator: str | ~picpocket.internal_use.NotSupplied | None = <picpocket.internal_use.NotSupplied object>, title: str | ~picpocket.internal_use.NotSupplied | None = <picpocket.internal_use.NotSupplied object>, caption: str | ~picpocket.internal_use.NotSupplied | None = <picpocket.internal_use.NotSupplied object>, alt: str | ~picpocket.internal_use.NotSupplied | None = <picpocket.internal_use.NotSupplied object>, rating: int | ~picpocket.internal_use.NotSupplied | None = <picpocket.internal_use.NotSupplied object>)¶

Edit information about an image.

Edit user-supplied metadata about an image. For all properties, supplying None will erase the existing value of that property

Parameters:

id – The image to edit
creator – Who made the image
title – The title of the image
caption – A caption of the image
alt – Descriptive text of the image to be used as alt text
rating – A numeric rating of the image

async tag_image(id: int, tag: str)¶

Apply a tag to an image

Tags in PicPocket are nestable, and are supplied as lists of strings (e.g., [“bird”, “goose”, “Canada Goose”]). When adding a tag, it’s recommended that you don’t add its parents.

Parameters:

id – The image to tag
tag – The tag to apply

async untag_image(id: int, tag: str)¶

Remove a tag from an image

Todo

allow cascading?

Removing a non-existent tag will not error.

Parameters:

id – The image to untag
tag – The tag to remove

async move_image(id: int, path: Path, location: int | None = None)¶

Move an image.

PicPocket will attempt to move both the on-disk location as well as its internal record. If the current file location is empty and a file exists at the destination, PicPocket will assume the file was already moved on-disk and will update its records. In all other cases, if a file exists at the destination (either on-disk or in PicPocket), PicPocket will error.

Parameters:

id – the image to move
path – The new location (relative to the root of its destination location).
location – The id of the location to move the image to. If unsupplied, the file will be moved within its source location.

async remove_image(id: int, *, delete: bool = False)¶

Remove an image from PicPocket

PicPocket will only delete the file on-disk if specifically requested.

Parameters:

id – The image to remove from PicPocket
delete – Delete the image on-disk as well. If True, the image will only be removed from PicPocket if the delete is successful

async find_image(path: Path, tags: bool = False) → Image | None¶

Attempt to find an image given its path

Parameters:

path – The full path to the image
tags – Whether to fetch the image’s tags

Returns:

The image, if it exists in PicPocket

async get_image(id: int, tags: bool = False) → Image | None¶

Get an image in PicPocket

Parameters:

id – The image to fetch
tags – Whether to fetch the image’s tags

Returns:

The image matching the id (if it exists)

Count the number of images in PicPocket

Get the number of images in PicPocket that match a criteria.

You can filter the result on any of the following properties:

name: The name of the image (without extension)
extension: The file suffix (without leading dot)
creator: Who created the image
location: Only include images at this location
path: The text path, relative to its location’s root
title: The title of the image
caption: The image caption
alt: The image alt text
rating: The rating of the image.

Parameters:

filter – Only include images that match this filter
tagged – Only include images that are(n’t) tagged
any_tags – Only include images that are tagged with at least one of the supplied tags
all_tags – Only include images that are tagged with all of the supplied tags
no_tags – Don’t include images that have any of the supplied tags
reachable – Only include images that are currently reachable

Returns:

The number of images that match the criteria

Get the ids of images in PicPocket

Get the ids of images in PicPocket that match a criteria.

You can filter and order the result on any of the following properties:

name: The name of the image (without extension)
extension: The file suffix (without leading dot)
creator: Who created the image
location: Only include images at this location
path: The text path, relative to its location’s root
title: The title of the image
caption: The image caption
alt: The image alt text
rating: The rating of the image.

When ordering images, properties will be sorted in ascending order by default (e.g., for ratings: Lower rated images will be supplied first). Putting a minus sign (-) in front of that property will sort in descending order instead. You can also use the exclamation mark (!) to specify whether images lacking a property (e.g., unrated images) should be sorted at the beginning or the end (e.g., !rating will order unrated images and then worst-to-best; rating! will order unrated images after the best images). When supplying unspecified properties in descending order, the exclamation mark should be supplied after the minus sign (e.g., -!rating will return untagged images and then best-to-worst). As well as above properties, ‘random’ can be used to order images randomly. Random ordering can be done after properties (e.g., -!rating,random will return images unrated and then best to worst, but with unrated images and images with a given rating will be returned in a random order).

Note

Orderings are not preserved across invocations, so running repeatedly with increasing offsets is not guaranteed to include all images or supply each images exactly once. That said, if you don’t add/edit/remove images between invocations (and don’t use random ordering), it should be fine.

Parameters:

filter – Only include images that match this filter
tagged – Only include images that are(n’t) tagged
any_tags – Only include images that are tagged with at least one of the supplied tags
all_tags – Only include images that are tagged with all of the supplied tags
no_tags – Don’t include images that have any of the supplied tags
reachable – Only include images that are currently reachable
order – how to sort the returned ids as a list of properties
limit – Only return this many image ids
offset – If limit is supplied, start returing images from this offset.
reachable – Only include images that are(n’t) currently reachable

Returns:

The ids of all images that match this criteria

Search images in PicPocket

Get images in PicPocket that match a criteria.

You can filter and order the result on any of the following properties:

name: The name of the image (without extension)

extension: The file suffix (without leading dot)

creator: Who created the image

location: Only include images at this location

path: The text path, relative to its location’s root

title: The title of the image

caption: The image caption

alt: The image alt text

rating: The rating of the image.

When ordering images, properties will be sorted in ascending order by default (e.g., for ratings: Lower rated images will be supplied first). Putting a minus sign (-) in front of that property will sort in descending order instead. You can also use the exclamation mark (!) to specify whether images lacking a property (e.g., unrated images) should be sorted at the beginning or the end (e.g., !rating will order unrated images and then worst-to-best; rating! will order unrated images after the best images). When supplying unspecified properties in descending order, the exclamation mark should be supplied after the minus sign (e.g., -!rating will return untagged images and then best-to-worst). As well as above properties, ‘random’ can be used to order images randomly. Random ordering can be done after properties (e.g., -!rating,random will return images unrated and then best to worst, but with unrated images and images with a given rating will be returned in a random order).

Note

Orderings are not preserved across invocations, so running repeatedly with increasing offsets is not guaranteed to include all images or supply each images exactly once. That said, if you don’t add/edit/remove images between invocations (and don’t use random ordering), it should be fine.

Parameters:

filter – Only include images that match this filter
tagged – Only include images that are(n’t) tagged
any_tags – Only include images that are tagged with at least one of the supplied tags
all_tags – Only include images that are tagged with all of the supplied tags
no_tags – Don’t include images that have any of the supplied tags
reachable – Only include images that are currently reachable
order – how to sort the returned ids as a list of properties
limit – Only return this many image ids
offset – If limit is supplied, start returing images from this offset.
reachable – Only include images that are(n’t) currently reachable

Returns:

The images that match this criteria

async verify_image_files(*, location: int | None = None, path: Path | None = None, reparse_exif: bool = False) → list[picpocket.database.types.Image]¶

Check that images in PicPocket exist on-disk

Check that images exist on-disk and that stored file information is accurate. If file metadata (e.g., dimensiosns, exif data) has changed, it will be updated as necessary. PicPocket will not remove missing images from its database.

This method will skip any removable locations that aren’t currently attached.

Parameters:

location – Only check images in this location
path – Only scan images within this directory
reparse_exif – Reread EXIF data even if the image hasn’t been touched

Returns:

All images that exist in PicPocket that can’t be found on-disk.

async add_tag(tag: str, description: str | ~picpocket.internal_use.NotSupplied | None = <picpocket.internal_use.NotSupplied object>)¶

Add a tag to PicPocket

Tags in PicPocket are nestable, and are supplied as lists of strings (e.g., [“bird”, “goose”, “Canada Goose”]). You don’t need to explicitly add tags (tagging an image will create that tag). You only need to add a tag if you want to have a description associated with a tag.

Adding an existing tag will not error, and if supplied, the description will be updated.

Parameters:

tag – The tag to add
description – A description of the tag

async move_tag(current: str, new: str, /, cascade: bool = True) → int¶

Move a tag to a new name

If a tag already exists with the same name, the two tags will be merged (keeping that tag’s description).

Note

Cascade is always based on the supplied tag, not where the change was made in the tag. Even if you edit a parent section of the tag’s name, it will only cascade to this tag’s children.

Parameters:

current – The name of the tag
new – The new name of the tag
cascade – Whether to move child tags as well.

Returns:

The number of moved tags

async remove_tag(tag: str, cascade: bool = False)¶

Remove a tag from PicPocket

Parameters:

tag – The tag to remove
cascade – If True, also remove any tags that are descendents of this tag

async get_tag(tag: str, children: bool = False) → Tag¶

Get the description of a tag

Parameters:

name – The tag to get the description of
children – Fetch children as well

Returns:

The tag

async all_tag_names() → set[str]¶

Get the names of all used tags

Returns:: The list of tags

async all_tags() → dict[str, Any]¶

Return the tags PicPocket is aware of

Returns:: A recursive dict representing the known tag structure within PicPocket. The outer level will be the names of root level tags and for each tag, their value will be a dict containing their description and the dict of their child tags.

async get_tag_set(*image_ids: int, minimum: int = 1) → dict[str, int]¶

Get the set of tags applied to any of a set of images

Parameters:

image_ids – The images to find tags for
minimum – Only return tags that have been applied to this many images.

Returns:

A dict with the tags as keys and their counts

async create_session(data: dict[str, Any], expires: datetime | None = None) → int¶

Store temporary data about a user query

Parameters:

data – The JSON-serializable data
expires – The expiration date for the data (if not supplied, the default will be used).

Returns:

the session id

async get_session(session_id: int) → dict[str, Any] | None¶

Fetch session data

Parameters:: session_id – The id of the session
Returns:: the session data, if it exists

async prune_sessions()¶: remove any expired sessions

SQL¶

class picpocket.database.logic.Types(value, names=None, *, module=None, qualname=None, type=None, start=1, boundary=None)¶

Types of PicPocket property

BOOLEAN = 'boolean'¶

DATETIME = 'datetime'¶

ID = 'id'¶

JSON = 'json'¶

NUMBER = 'number'¶

TEXT = 'text'¶

class picpocket.database.logic.SQL¶

A class for configuring how to manage SQL for a given database.

This includes some checks for what features the database supports as well as methods for crafting safe query strings the database will accept. The latter are modelled on psycopg’s sql <https://www.psycopg.org/psycopg3/docs/api/sql.html> module with the default implementations providing the support SQLite would require.

abstract property types: set[picpocket.database.logic.Types]¶: The supported data types

property arrays: bool¶: Whether the database supports arrays

abstract property param: str¶

The symbol the database uses for positional paramaters.

We can’t just use ‘paramstyle’ because APIs are inconsistent on whether they use that to refer to their positional or named param style.

Warning

This method should not be used in conjunction with any of the below formatting methods!

Returns:: a symbol to use when generating SQL that uses positional placeholders.

format(statement: str, *parts)¶

add sections (e.g., identifiers, placeholders) to a statement

The base statement should be using {} for positional formatting and each part being formatted in should represent already-formatted sql (using either format or any of the other sql methods).

Parameters:

statment – The statement to format
parts – the sections to add to the statement

Returns:

The formatted statement.

join(joiner: str, parts: Iterable)¶

join together several sections of a statement

Parts should already be formatted by another sql format method before being passed in to join.

Parameters:

joiner – The string to use to join the sections
parts – The sections to join together

Returns:

The formatted section

identifier(identifier: str)¶

Format an identifier (i.e., table, column names)

Warning

The default implementation is written with the assumption that identifiers will only be passed in from a reliable source (i.e., PicPocket’s own API) and not from user input. This method is not thorough enough to be trusted with user-supplied identifiers.

Parameters:: identifier – the table/column name
Returns:: A formatted version of the identifier

literal(literal: Any)¶

Format a literal (e.g., a number or string)

Parameters:: literal – the literal to format
Returns:: A formatted version of that literal

abstract placeholder(placeholder: str)¶

Format a named placeholder

Parameters:: placeholder – the placeholder to format
Returns:: A formatted version of that placeholder

property escape: str | None¶: The symbol used to escape search terms for LIKE queries

Implementing Your Own Backend¶

Adding Support for a New DBAPI 2.0-Compatible Database¶

DbApi¶

SQL¶

PicPocket

Navigation

Related Topics