API docs¶

Here you will find API docs. If you’re python user you will probably understand it. In other case you should visit:

And you probably want to use CodernityDB-HTTP instead this embedded version.

Database¶

Note

Please refer to Database operations flow for general description

Standard¶

class CodernityDB.database.Database(path)¶

Bases: object

A default single thread database object.

add_index(new_index, create=True, ind_kwargs=None)¶

Parameters:	new_index (string) – New index to add, can be Index object, index valid string or path to file with index code create (bool) – Create the index after add or not
Returns:	new index name

all(index_name, limit=-1, offset=0, with_doc=False, with_storage=True)¶

Alows to get all records for given index

Parameters:	index_name – Index to perform the operation limit – defines limit for query offset – defines offset (how many records from start it will ignore) with_doc – if `True` data from id index will be included in output with_storage – if `True` data from index storage will be included, otherwise just metadata

close()¶: Closes the database

compact()¶: Compact all indexes. Runs _compact_indexes() behind.

compact_index(index)¶

Compacts index Used for better utilization of index metadata. The deleted documents will be not more in structure.

Parameters:	index (CodernityDB.index.Index` instance, or string) – the index to destroy

count(target_funct, *args, **kwargs)¶

Counter. Allows to execute for example

db.count(db.all, 'id')

And it will return then how much records are in your id index.

Warning

It sets kwargs['with_storage'] = False and kwargs['with_doc'] = False

create(path=None, **kwargs)¶

Create database

Parameters:	path – path where to create the database
Returns:	database path

create_new_rev(old_rev=None)¶: Creates new revision number based on previous one. Increments it + random bytes. On overflow starts from 0 again.

custom_header = ''¶

delete(data)¶

Delete data from database.

data has to contain _id and _rev fields.

Parameters:	data – data to delete

destroy()¶

Allows to destroy database.

not reversable operation!

destroy_index(index)¶

Destroys index

Parameters:	index (CodernityDB.index.Index` instance, or string) – the index to destroy

edit_index(index, reindex=False, ind_kwargs=None)¶

Allows to edit existing index. Previous working version will be saved with _last suffix (see revert_index()

Parameters:	reindex (bool) – should be the index reindexed after change
Returns:	index name

exists(path=None)¶

Checks if database in given path exists

Parameters:	path – path to look for database

flush()¶: Flushes all indexes. Runs flush_indexes() behind.

flush_indexes()¶: Flushes all indexes

fsync()¶: It forces the kernel buffer to be written to disk. Use when you’re sure that you need to.

get(index_name, key, with_doc=False, with_storage=True)¶

Get single data from Database by key.

Parameters:	index_name – index to get data from key – key to get with_doc – if `True` data from id index will be included in output with_storage – if `True` data from index storage will be included, otherwise just metadata.

get_db_details()¶

Get’s database details, size, indexes, environment etc.

Returns:	database details

get_index_code(index_name, code_switch='All')¶

It will return full index code from index file.

Parameters:	index_name – the name of index to look for code

get_index_details(name)¶

Will return index properties.

Returns:	index details

get_many(index_name, key=None, limit=-1, offset=0, with_doc=False, with_storage=True, start=None, end=None, **kwargs)¶

Allows to get multiple data for given key for Hash based indexes. Also allows get range queries for Tree based indexes with start and end arguments.

Parameters:

Parameters:	index_name – Index to perform the operation key – key to look for (has to be `None` to use range queries) limit – defines limit for query offset – defines offset (how many records from start it will ignore) with_doc – if `True` data from id index will be included in output with_storage – if `True` data from index storage will be included, otherwise just metadata. start – `start` parameter for range queries end – `end` parameter for range queries
Returns:	iterator over records

index_name – Index to perform the operation
key – key to look for (has to be None to use range queries)
limit – defines limit for query
offset – defines offset (how many records from start it will ignore)
with_doc – if True data from id index will be included in output
with_storage – if True data from index storage will be included, otherwise just metadata.
start – start parameter for range queries
end – end parameter for range queries

Returns:

iterator over records

initialize(path=None, makedir=True)¶

Initialize new database

Parameters:	path – Path to a database (allows delayed path configuration), if not provided self.path will be used makedir – Make the `_indexes` directory or not
Returns:	the database path

insert(data)¶

It’s using reference on the given data dict object, to avoid it copy it before inserting!

If data will not have _id field, it will be generated (random 32 chars string)

Parameters:	data – data to insert

open(path=None)¶

Will open already existing database

Parameters:	path – path with database to open

reindex()¶: Reindex all indexes. Runs _reindex_indexes() behind.

reindex_index(index)¶

Performs reindex on index. Optimizes metadata and storage informations for given index.

You can’t reindex id index.

Parameters:	index (CodernityDB.index.Index` instance, or string) – the index to reindex

revert_index(index_name, reindex=False, ind_kwargs=None)¶

Tries to revert index code from copy. It calls edit_index() with previous working.

Parameters:	index_name (string) – index name to restore

run(index_name, target_funct, *args, **kwargs)¶

Allows to execute given function on Database side (important for server mode)

If target_funct==sum then given index must have run_sum method.

Parameters:	index_name – index name to perform action. target_funct – target function name (without run prefix) args – `args` for function kwargs – `kwargs` for function

set_indexes(indexes=[])¶

Set indexes using indexes param

Parameters:	indexes (iterable of `CodernityDB.index.Index` objects.) – indexes to set in db

update(data)¶

It’s using reference on the given data dict object, to avoid it copy it before updating!

data must contain _id and _rev fields.

Parameters:	data – data to update

exception CodernityDB.database.DatabaseConflict¶: Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.DatabaseException¶: Bases: exceptions.Exception

exception CodernityDB.database.DatabaseIsNotOpened¶: Bases: CodernityDB.database.PreconditionsException

exception CodernityDB.database.DatabasePathException¶: Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.PreconditionsException¶: Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.RecordDeleted¶: Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.RecordNotFound¶: Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.RevConflict¶: Bases: CodernityDB.database.DatabaseException

CodernityDB.database.header_for_indexes(index_name, index_class, db_custom='', ind_custom='', classes_code='')¶

Thread Safe Database¶

class CodernityDB.database_thread_safe.ThreadSafeDatabase(path, *args, **kwargs)¶

Bases: CodernityDB.database_safe_shared.SafeDatabase

Thread safe version of CodernityDB that uses several lock objects, on different methods / different indexes etc. It’s completely different implementation of locking than SuperThreadSafe one.

Super Thread Safe Database¶

class CodernityDB.database_super_thread_safe.SuperLock¶

Bases: type

static wrapper(f)¶

class CodernityDB.database_super_thread_safe.SuperThreadSafeDatabase(*args, **kwargs)¶

Bases: CodernityDB.database.Database

Thread safe version that always allows single thread to use db. It adds the same lock for all methods, so only one operation can be performed in given time. Completely different implementation than ThreadSafe version (without super word)

add_index(*args, **kwargs)¶

all(*args, **kwargs)¶

Alows to get all records for given index

Parameters:	index_name – Index to perform the operation limit – defines limit for query offset – defines offset (how many records from start it will ignore) with_doc – if `True` data from id index will be included in output with_storage – if `True` data from index storage will be included, otherwise just metadata

close(*args, **kwargs)¶: Closes the database

compact(*args, **kwargs)¶: Compact all indexes. Runs _compact_indexes() behind.

compact_index(*args, **kwargs)¶

Compacts index Used for better utilization of index metadata. The deleted documents will be not more in structure.

Parameters:	index (CodernityDB.index.Index` instance, or string) – the index to destroy

count(*args, **kwargs)¶

Counter. Allows to execute for example

db.count(db.all, 'id')

And it will return then how much records are in your id index.

Warning

It sets kwargs['with_storage'] = False and kwargs['with_doc'] = False

create(*args, **kwargs)¶

create_new_rev(*args, **kwargs)¶: Creates new revision number based on previous one. Increments it + random bytes. On overflow starts from 0 again.

delete(*args, **kwargs)¶

Delete data from database.

data has to contain _id and _rev fields.

Parameters:	data – data to delete

destroy(*args, **kwargs)¶

Allows to destroy database.

not reversable operation!

destroy_index(*args, **kwargs)¶

Destroys index

Parameters:	index (CodernityDB.index.Index` instance, or string) – the index to destroy

edit_index(*args, **kwargs)¶

exists(*args, **kwargs)¶

Checks if database in given path exists

Parameters:	path – path to look for database

fsync(*args, **kwargs)¶: It forces the kernel buffer to be written to disk. Use when you’re sure that you need to.

get(*args, **kwargs)¶

Get single data from Database by key.

Parameters:	index_name – index to get data from key – key to get with_doc – if `True` data from id index will be included in output with_storage – if `True` data from index storage will be included, otherwise just metadata.

get_db_details(*args, **kwargs)¶

Get’s database details, size, indexes, environment etc.

Returns:	database details

get_index_code(*args, **kwargs)¶

It will return full index code from index file.

Parameters:	index_name – the name of index to look for code

get_index_details(*args, **kwargs)¶

Will return index properties.

Returns:	index details

get_many(*args, **kwargs)¶

Allows to get multiple data for given key for Hash based indexes. Also allows get range queries for Tree based indexes with start and end arguments.

Parameters:

Parameters:	index_name – Index to perform the operation key – key to look for (has to be `None` to use range queries) limit – defines limit for query offset – defines offset (how many records from start it will ignore) with_doc – if `True` data from id index will be included in output with_storage – if `True` data from index storage will be included, otherwise just metadata. start – `start` parameter for range queries end – `end` parameter for range queries
Returns:	iterator over records

index_name – Index to perform the operation
key – key to look for (has to be None to use range queries)
limit – defines limit for query
offset – defines offset (how many records from start it will ignore)
with_doc – if True data from id index will be included in output
with_storage – if True data from index storage will be included, otherwise just metadata.
start – start parameter for range queries
end – end parameter for range queries

Returns:

iterator over records

initialize(*args, **kwargs)¶

Initialize new database

Parameters:	path – Path to a database (allows delayed path configuration), if not provided self.path will be used makedir – Make the `_indexes` directory or not
Returns:	the database path

insert(*args, **kwargs)¶

It’s using reference on the given data dict object, to avoid it copy it before inserting!

If data will not have _id field, it will be generated (random 32 chars string)

Parameters:	data – data to insert

open(*args, **kwargs)¶

reindex(*args, **kwargs)¶: Reindex all indexes. Runs _reindex_indexes() behind.

reindex_index(*args, **kwargs)¶

Performs reindex on index. Optimizes metadata and storage informations for given index.

You can’t reindex id index.

Parameters:	index (CodernityDB.index.Index` instance, or string) – the index to reindex

revert_index(*args, **kwargs)¶

Tries to revert index code from copy. It calls edit_index() with previous working.

Parameters:	index_name (string) – index name to restore

run(*args, **kwargs)¶

Allows to execute given function on Database side (important for server mode)

If target_funct==sum then given index must have run_sum method.

Parameters:	index_name – index name to perform action. target_funct – target function name (without run prefix) args – `args` for function kwargs – `kwargs` for function

set_indexes(*args, **kwargs)¶

Set indexes using indexes param

Parameters:	indexes (iterable of `CodernityDB.index.Index` objects.) – indexes to set in db

super_lock = <_RLock owner=None count=0>¶

update(*args, **kwargs)¶

It’s using reference on the given data dict object, to avoid it copy it before updating!

data must contain _id and _rev fields.

Parameters:	data – data to update

Gevent Database¶

class CodernityDB.database_gevent.GeventDatabase(path, *args, **kwargs)¶: Bases: CodernityDB.database_safe_shared.SafeDatabase

Indexes¶

Note

If you’re not interested in CodernityDB development / extending you don’t need to read this section, please then refer to Database indexes, otherwise please remember that index methods are called from Database object.

General Index¶

exception CodernityDB.index.DocIdNotFound¶: Bases: CodernityDB.index.ElemNotFound

exception CodernityDB.index.ElemNotFound¶: Bases: CodernityDB.index.IndexException

class CodernityDB.index.Index(db_path, name)¶

Bases: object

all(start_pos)¶

close_index()¶

compact(*args, **kwargs)¶

create_index()¶

custom_header = ''¶

delete(key, start, size)¶

destroy(*args, **kwargs)¶

flush()¶

fsync()¶

get(key)¶

get_many(key, start_from=None, limit=0)¶

insert(doc_id, key, start, size)¶

insert_with_storage(doc_id, key, value)¶

make_key(data)¶

make_key_value(data)¶

open_index()¶

update(doc_id, key, start, size)¶

update_with_storage(doc_id, key, value)¶

exception CodernityDB.index.IndexConflict¶: Bases: CodernityDB.index.IndexException

exception CodernityDB.index.IndexException¶: Bases: exceptions.Exception

exception CodernityDB.index.IndexNotFoundException¶: Bases: CodernityDB.index.IndexException

exception CodernityDB.index.IndexPreconditionsException¶: Bases: CodernityDB.index.IndexException

exception CodernityDB.index.ReindexException¶: Bases: CodernityDB.index.IndexException

exception CodernityDB.index.TryReindexException¶: Bases: CodernityDB.index.ReindexException

Hash Index specific¶

Note

Please refer to Hash Index for description

class CodernityDB.hash_index.DummyHashIndex(db_path, name, entry_line_format='<32s4sIIcI', *args, **kwargs)¶

Bases: CodernityDB.hash_index.IU_HashIndex

all(*args, **kwargs)¶

delete(*args, **kwargs)¶

destroy()¶

get(*args, **kwargs)¶

get_many(*args, **kwargs)¶

insert(*args, **kwargs)¶

make_key_value(data)¶

update(*args, **kwargs)¶

class CodernityDB.hash_index.HashIndex(db_path, name, entry_line_format='<32s{key}IIcI', hash_lim=1048575, storage_class=None, key_format='c')¶

Bases: CodernityDB.hash_index.IU_HashIndex

That class is designed to be used in custom indexes.

class CodernityDB.hash_index.IU_HashIndex(db_path, name, entry_line_format='<32s{key}IIcI', hash_lim=1048575, storage_class=None, key_format='c')¶

Bases: CodernityDB.index.Index

That class is for Internal Use only, if you want to use HashIndex just subclass the HashIndex instead this one.

That design is because main index logic should be always in database not in custom user indexes.

all(limit=-1, offset=0)¶

close_index()¶

compact(hash_lim=None)¶

create_index()¶

delete(doc_id, key, start=0, size=0)¶

destroy()¶

get(key)¶

get_many(key, limit=1, offset=0)¶

insert(doc_id, key, start, size, status='o')¶

make_key(key)¶

make_key_value(data)¶

open_index()¶

update(doc_id, key, u_start=0, u_size=0, u_status='o')¶

class CodernityDB.hash_index.IU_MultiHashIndex(*args, **kwargs)¶

Bases: CodernityDB.hash_index.IU_HashIndex

Class that allows to index more than one key per database record.

It operates very well on GET/INSERT. It’s not optimized for UPDATE operations (will always readd everything)

delete(doc_id, key, start=0, size=0)¶

get(key)¶

insert(doc_id, key, start, size, status='o')¶

make_key_value(data)¶

update(doc_id, key, u_start, u_size, u_status='o')¶

class CodernityDB.hash_index.IU_UniqueHashIndex(db_path, name, entry_line_format='<32s8sIIcI', *args, **kwargs)¶

Bases: CodernityDB.hash_index.IU_HashIndex

Index for unique keys! Designed to be a id index.

That class is for Internal Use only, if you want to use UniqueHashIndex just subclass the UniqueHashIndex instead this one.

That design is because main index logic should be always in database not in custom user indexes.

all(limit=-1, offset=0)¶

delete(key, start=0, size=0)¶

destroy()¶

get_many(*args, **kwargs)¶

insert(key, rev, start, size, status='o')¶

insert_with_storage(_id, _rev, value)¶

make_key_value(data)¶

update(key, rev, u_start=0, u_size=0, u_status='o')¶

update_with_storage(_id, _rev, value)¶

class CodernityDB.hash_index.MultiHashIndex(*args, **kwargs)¶

Bases: CodernityDB.hash_index.IU_MultiHashIndex

That class is designed to be used in custom indexes.

class CodernityDB.hash_index.UniqueHashIndex(db_path, name, entry_line_format='<32s8sIIcI', *args, **kwargs)¶

Bases: CodernityDB.hash_index.IU_UniqueHashIndex

That class is designed to be used in custom indexes. It’s designed to be id index.

B+Tree Index specific¶

Note

Please refer to B Plus Tree Index for description

class CodernityDB.tree_index.IU_MultiTreeBasedIndex(*args, **kwargs)¶

Bases: CodernityDB.tree_index.IU_TreeBasedIndex

Class that allows to index more than one key per database record.

It operates very well on GET/INSERT. It’s not optimized for UPDATE operations (will always readd everything)

delete(doc_id, key, start=0, size=0)¶

get(key)¶

insert(doc_id, key, start, size, status='o')¶

make_key_value(data)¶

update(doc_id, key, u_start, u_size, u_status='o')¶

class CodernityDB.tree_index.IU_TreeBasedIndex(db_path, name, key_format='32s', pointer_format='I', meta_format='32sIIc', node_capacity=10, storage_class=None)¶

Bases: CodernityDB.index.Index

all(limit=-1, offset=0)¶: Traverses linked list of all tree leaves and returns generator containing all elements stored in index.

close_index()¶

compact(node_capacity=0)¶

create_index()¶

custom_header = 'from CodernityDB.tree_index import TreeBasedIndex'¶

delete(doc_id, key, start=0, size=0)¶

destroy()¶

get(key)¶

get_between(start, end, limit=1, offset=0, inclusive_start=True, inclusive_end=True)¶

get_many(key, limit=1, offset=0)¶

insert(doc_id, key, start, size, status='o')¶

insert_first_record_into_leaf(leaf_start, key, doc_id, start, size, status)¶

make_key(key)¶

make_key_value(data)¶

open_index()¶

update(doc_id, key, u_start=0, u_size=0, u_status='o')¶

class CodernityDB.tree_index.MultiTreeBasedIndex(*args, **kwargs)¶

Bases: CodernityDB.tree_index.IU_MultiTreeBasedIndex

It allows to index more than one key for record. (ie. prefix/infix/suffix search mechanizms) That class is designed to be used in custom indexes.

exception CodernityDB.tree_index.NodeCapacityException¶: Bases: CodernityDB.index.IndexException

class CodernityDB.tree_index.TreeBasedIndex(db_path, name, key_format='32s', pointer_format='I', meta_format='32sIIc', node_capacity=10, storage_class=None)¶: Bases: CodernityDB.tree_index.IU_TreeBasedIndex

Storage¶

class CodernityDB.storage.DummyStorage¶

Bases: object

Storage mostly used to fake real storage

close(*args, **kwargs)¶

create(*args, **kwargs)¶

data_from(*args, **kwargs)¶

data_to(*args, **kwargs)¶

flush(*args, **kwargs)¶

fsync(*args, **kwargs)¶

get(*args, **kwargs)¶

insert(*args, **kwargs)¶

open(*args, **kwargs)¶

save(*args, **kwargs)¶

update(*args, **kwargs)¶

class CodernityDB.storage.IU_Storage(db_path, name='main')¶

Bases: object

close()¶

create()¶

data_from(data)¶

data_to(data)¶

destroy()¶

flush()¶

fsync()¶

get(start, size, status='c')¶

insert(data)¶

open()¶

save(data)¶

update(data)¶

class CodernityDB.storage.Storage(db_path, name='main')¶: Bases: CodernityDB.storage.IU_Storage

exception CodernityDB.storage.StorageException¶: Bases: exceptions.Exception

Patches¶

CodernityDB.patch.patch_cache_lfu(lock_obj)¶: Patnches cache mechanizm to be thread safe (gevent ones also)

Note

It’s internal CodernityDB mechanizm, it will be called when needed

CodernityDB.patch.patch_cache_rr(lock_obj)¶: Patches cache mechanizm to be thread safe (gevent ones also)

Note

It’s internal CodernityDB mechanizm, it will be called when needed

CodernityDB.patch.patch_flush_fsync(db_obj)¶

Will always execute index.fsync after index.flush.

Note

It’s for advanced users, use when you understand difference between flush and fsync, and when you definitely need that.

It’s important to call it AFTER database has all indexes etc (after db.create or db.open)

Example usage:

...
db = Database('/tmp/patch_demo')
db.create()
patch_flush_fsync(db)
...

Labs

CodernityDB Download

API docs¶

Database¶

Standard¶

Thread Safe Database¶

Super Thread Safe Database¶

Gevent Database¶

Indexes¶

General Index¶

Hash Index specific¶

B+Tree Index specific¶

Storage¶

Patches¶