Labs

CodernityDB Download

API docs

Here you will find API docs. If you’re python user you will probably understand it. In other case you should visit:

And you probably want to use CodernityDB-HTTP instead this embedded version.

Database

Note

Please refer to Database operations flow for general description

Standard

class CodernityDB.database.Database(path)

Bases: object

A default single thread database object.

add_index(new_index, create=True, ind_kwargs=None)
Parameters:
  • new_index (string) – New index to add, can be Index object, index valid string or path to file with index code
  • create (bool) – Create the index after add or not
Returns:

new index name

all(index_name, limit=-1, offset=0, with_doc=False, with_storage=True)

Alows to get all records for given index

Parameters:
  • index_name – Index to perform the operation
  • limit – defines limit for query
  • offset – defines offset (how many records from start it will ignore)
  • with_doc – if True data from id index will be included in output
  • with_storage – if True data from index storage will be included, otherwise just metadata
close()

Closes the database

compact()

Compact all indexes. Runs _compact_indexes() behind.

compact_index(index)

Compacts index Used for better utilization of index metadata. The deleted documents will be not more in structure.

Parameters:index (CodernityDB.index.Index` instance, or string) – the index to destroy
count(target_funct, *args, **kwargs)

Counter. Allows to execute for example

db.count(db.all, 'id')

And it will return then how much records are in your id index.

Warning

It sets kwargs['with_storage'] = False and kwargs['with_doc'] = False

create(path=None, **kwargs)

Create database

Parameters:path – path where to create the database
Returns:database path
create_new_rev(old_rev=None)

Creates new revision number based on previous one. Increments it + random bytes. On overflow starts from 0 again.

custom_header = ''
delete(data)

Delete data from database.

data has to contain _id and _rev fields.

Parameters:data – data to delete
destroy()

Allows to destroy database.

not reversable operation!

destroy_index(index)

Destroys index

Parameters:index (CodernityDB.index.Index` instance, or string) – the index to destroy
edit_index(index, reindex=False, ind_kwargs=None)

Allows to edit existing index. Previous working version will be saved with _last suffix (see revert_index()

Parameters:reindex (bool) – should be the index reindexed after change
Returns:index name
exists(path=None)

Checks if database in given path exists

Parameters:path – path to look for database
flush()

Flushes all indexes. Runs flush_indexes() behind.

flush_indexes()

Flushes all indexes

fsync()

It forces the kernel buffer to be written to disk. Use when you’re sure that you need to.

get(index_name, key, with_doc=False, with_storage=True)

Get single data from Database by key.

Parameters:
  • index_name – index to get data from
  • key – key to get
  • with_doc – if True data from id index will be included in output
  • with_storage – if True data from index storage will be included, otherwise just metadata.
get_db_details()

Get’s database details, size, indexes, environment etc.

Returns:database details
get_index_code(index_name, code_switch='All')

It will return full index code from index file.

Parameters:index_name – the name of index to look for code
get_index_details(name)

Will return index properties.

Returns:index details
get_many(index_name, key=None, limit=-1, offset=0, with_doc=False, with_storage=True, start=None, end=None, **kwargs)

Allows to get multiple data for given key for Hash based indexes. Also allows get range queries for Tree based indexes with start and end arguments.

Parameters:
  • index_name – Index to perform the operation
  • key – key to look for (has to be None to use range queries)
  • limit – defines limit for query
  • offset – defines offset (how many records from start it will ignore)
  • with_doc – if True data from id index will be included in output
  • with_storage – if True data from index storage will be included, otherwise just metadata.
  • startstart parameter for range queries
  • endend parameter for range queries
Returns:

iterator over records

initialize(path=None, makedir=True)

Initialize new database

Parameters:
  • path – Path to a database (allows delayed path configuration), if not provided self.path will be used
  • makedir – Make the _indexes directory or not
Returns:

the database path

insert(data)

It’s using reference on the given data dict object, to avoid it copy it before inserting!

If data will not have _id field, it will be generated (random 32 chars string)

Parameters:data – data to insert
open(path=None)

Will open already existing database

Parameters:path – path with database to open
reindex()

Reindex all indexes. Runs _reindex_indexes() behind.

reindex_index(index)

Performs reindex on index. Optimizes metadata and storage informations for given index.

You can’t reindex id index.

Parameters:index (CodernityDB.index.Index` instance, or string) – the index to reindex
revert_index(index_name, reindex=False, ind_kwargs=None)

Tries to revert index code from copy. It calls edit_index() with previous working.

Parameters:index_name (string) – index name to restore
run(index_name, target_funct, *args, **kwargs)

Allows to execute given function on Database side (important for server mode)

If target_funct==sum then given index must have run_sum method.

Parameters:
  • index_name – index name to perform action.
  • target_funct – target function name (without run prefix)
  • *args*args for function
  • **kwargs**kwargs for function
set_indexes(indexes=[])

Set indexes using indexes param

Parameters:indexes (iterable of CodernityDB.index.Index objects.) – indexes to set in db
update(data)

It’s using reference on the given data dict object, to avoid it copy it before updating!

data must contain _id and _rev fields.

Parameters:data – data to update
exception CodernityDB.database.DatabaseConflict

Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.DatabaseException

Bases: exceptions.Exception

exception CodernityDB.database.DatabaseIsNotOpened

Bases: CodernityDB.database.PreconditionsException

exception CodernityDB.database.DatabasePathException

Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.PreconditionsException

Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.RecordDeleted

Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.RecordNotFound

Bases: CodernityDB.database.DatabaseException

exception CodernityDB.database.RevConflict

Bases: CodernityDB.database.DatabaseException

CodernityDB.database.header_for_indexes(index_name, index_class, db_custom='', ind_custom='', classes_code='')

Thread Safe Database

class CodernityDB.database_thread_safe.ThreadSafeDatabase(path, *args, **kwargs)

Bases: CodernityDB.database_safe_shared.SafeDatabase

Thread safe version of CodernityDB that uses several lock objects, on different methods / different indexes etc. It’s completely different implementation of locking than SuperThreadSafe one.

Super Thread Safe Database

class CodernityDB.database_super_thread_safe.SuperLock

Bases: type

static wrapper(f)
class CodernityDB.database_super_thread_safe.SuperThreadSafeDatabase(*args, **kwargs)

Bases: CodernityDB.database.Database

Thread safe version that always allows single thread to use db. It adds the same lock for all methods, so only one operation can be performed in given time. Completely different implementation than ThreadSafe version (without super word)

add_index(*args, **kwargs)
all(*args, **kwargs)

Alows to get all records for given index

Parameters:
  • index_name – Index to perform the operation
  • limit – defines limit for query
  • offset – defines offset (how many records from start it will ignore)
  • with_doc – if True data from id index will be included in output
  • with_storage – if True data from index storage will be included, otherwise just metadata
close(*args, **kwargs)

Closes the database

compact(*args, **kwargs)

Compact all indexes. Runs _compact_indexes() behind.

compact_index(*args, **kwargs)

Compacts index Used for better utilization of index metadata. The deleted documents will be not more in structure.

Parameters:index (CodernityDB.index.Index` instance, or string) – the index to destroy
count(*args, **kwargs)

Counter. Allows to execute for example

db.count(db.all, 'id')

And it will return then how much records are in your id index.

Warning

It sets kwargs['with_storage'] = False and kwargs['with_doc'] = False

create(*args, **kwargs)
create_new_rev(*args, **kwargs)

Creates new revision number based on previous one. Increments it + random bytes. On overflow starts from 0 again.

delete(*args, **kwargs)

Delete data from database.

data has to contain _id and _rev fields.

Parameters:data – data to delete
destroy(*args, **kwargs)

Allows to destroy database.

not reversable operation!

destroy_index(*args, **kwargs)

Destroys index

Parameters:index (CodernityDB.index.Index` instance, or string) – the index to destroy
edit_index(*args, **kwargs)
exists(*args, **kwargs)

Checks if database in given path exists

Parameters:path – path to look for database
fsync(*args, **kwargs)

It forces the kernel buffer to be written to disk. Use when you’re sure that you need to.

get(*args, **kwargs)

Get single data from Database by key.

Parameters:
  • index_name – index to get data from
  • key – key to get
  • with_doc – if True data from id index will be included in output
  • with_storage – if True data from index storage will be included, otherwise just metadata.
get_db_details(*args, **kwargs)

Get’s database details, size, indexes, environment etc.

Returns:database details
get_index_code(*args, **kwargs)

It will return full index code from index file.

Parameters:index_name – the name of index to look for code
get_index_details(*args, **kwargs)

Will return index properties.

Returns:index details
get_many(*args, **kwargs)

Allows to get multiple data for given key for Hash based indexes. Also allows get range queries for Tree based indexes with start and end arguments.

Parameters:
  • index_name – Index to perform the operation
  • key – key to look for (has to be None to use range queries)
  • limit – defines limit for query
  • offset – defines offset (how many records from start it will ignore)
  • with_doc – if True data from id index will be included in output
  • with_storage – if True data from index storage will be included, otherwise just metadata.
  • startstart parameter for range queries
  • endend parameter for range queries
Returns:

iterator over records

initialize(*args, **kwargs)

Initialize new database

Parameters:
  • path – Path to a database (allows delayed path configuration), if not provided self.path will be used
  • makedir – Make the _indexes directory or not
Returns:

the database path

insert(*args, **kwargs)

It’s using reference on the given data dict object, to avoid it copy it before inserting!

If data will not have _id field, it will be generated (random 32 chars string)

Parameters:data – data to insert
open(*args, **kwargs)
reindex(*args, **kwargs)

Reindex all indexes. Runs _reindex_indexes() behind.

reindex_index(*args, **kwargs)

Performs reindex on index. Optimizes metadata and storage informations for given index.

You can’t reindex id index.

Parameters:index (CodernityDB.index.Index` instance, or string) – the index to reindex
revert_index(*args, **kwargs)

Tries to revert index code from copy. It calls edit_index() with previous working.

Parameters:index_name (string) – index name to restore
run(*args, **kwargs)

Allows to execute given function on Database side (important for server mode)

If target_funct==sum then given index must have run_sum method.

Parameters:
  • index_name – index name to perform action.
  • target_funct – target function name (without run prefix)
  • *args*args for function
  • **kwargs**kwargs for function
set_indexes(*args, **kwargs)

Set indexes using indexes param

Parameters:indexes (iterable of CodernityDB.index.Index objects.) – indexes to set in db
super_lock = <_RLock owner=None count=0>
update(*args, **kwargs)

It’s using reference on the given data dict object, to avoid it copy it before updating!

data must contain _id and _rev fields.

Parameters:data – data to update

Gevent Database

class CodernityDB.database_gevent.GeventDatabase(path, *args, **kwargs)

Bases: CodernityDB.database_safe_shared.SafeDatabase

Indexes

Note

If you’re not interested in CodernityDB development / extending you don’t need to read this section, please then refer to Database indexes, otherwise please remember that index methods are called from Database object.

General Index

exception CodernityDB.index.DocIdNotFound

Bases: CodernityDB.index.ElemNotFound

exception CodernityDB.index.ElemNotFound

Bases: CodernityDB.index.IndexException

class CodernityDB.index.Index(db_path, name)

Bases: object

all(start_pos)
close_index()
compact(*args, **kwargs)
create_index()
custom_header = ''
delete(key, start, size)
destroy(*args, **kwargs)
flush()
fsync()
get(key)
get_many(key, start_from=None, limit=0)
insert(doc_id, key, start, size)
insert_with_storage(doc_id, key, value)
make_key(data)
make_key_value(data)
open_index()
update(doc_id, key, start, size)
update_with_storage(doc_id, key, value)
exception CodernityDB.index.IndexConflict

Bases: CodernityDB.index.IndexException

exception CodernityDB.index.IndexException

Bases: exceptions.Exception

exception CodernityDB.index.IndexNotFoundException

Bases: CodernityDB.index.IndexException

exception CodernityDB.index.IndexPreconditionsException

Bases: CodernityDB.index.IndexException

exception CodernityDB.index.ReindexException

Bases: CodernityDB.index.IndexException

exception CodernityDB.index.TryReindexException

Bases: CodernityDB.index.ReindexException

Hash Index specific

Note

Please refer to Hash Index for description

class CodernityDB.hash_index.DummyHashIndex(db_path, name, entry_line_format='<32s4sIIcI', *args, **kwargs)

Bases: CodernityDB.hash_index.IU_HashIndex

all(*args, **kwargs)
delete(*args, **kwargs)
destroy()
get(*args, **kwargs)
get_many(*args, **kwargs)
insert(*args, **kwargs)
make_key_value(data)
update(*args, **kwargs)
class CodernityDB.hash_index.HashIndex(db_path, name, entry_line_format='<32s{key}IIcI', hash_lim=1048575, storage_class=None, key_format='c')

Bases: CodernityDB.hash_index.IU_HashIndex

That class is designed to be used in custom indexes.

class CodernityDB.hash_index.IU_HashIndex(db_path, name, entry_line_format='<32s{key}IIcI', hash_lim=1048575, storage_class=None, key_format='c')

Bases: CodernityDB.index.Index

That class is for Internal Use only, if you want to use HashIndex just subclass the HashIndex instead this one.

That design is because main index logic should be always in database not in custom user indexes.

all(limit=-1, offset=0)
close_index()
compact(hash_lim=None)
create_index()
delete(doc_id, key, start=0, size=0)
destroy()
get(key)
get_many(key, limit=1, offset=0)
insert(doc_id, key, start, size, status='o')
make_key(key)
make_key_value(data)
open_index()
update(doc_id, key, u_start=0, u_size=0, u_status='o')
class CodernityDB.hash_index.IU_MultiHashIndex(*args, **kwargs)

Bases: CodernityDB.hash_index.IU_HashIndex

Class that allows to index more than one key per database record.

It operates very well on GET/INSERT. It’s not optimized for UPDATE operations (will always readd everything)

delete(doc_id, key, start=0, size=0)
get(key)
insert(doc_id, key, start, size, status='o')
make_key_value(data)
update(doc_id, key, u_start, u_size, u_status='o')
class CodernityDB.hash_index.IU_UniqueHashIndex(db_path, name, entry_line_format='<32s8sIIcI', *args, **kwargs)

Bases: CodernityDB.hash_index.IU_HashIndex

Index for unique keys! Designed to be a id index.

That class is for Internal Use only, if you want to use UniqueHashIndex just subclass the UniqueHashIndex instead this one.

That design is because main index logic should be always in database not in custom user indexes.

all(limit=-1, offset=0)
delete(key, start=0, size=0)
destroy()
get_many(*args, **kwargs)
insert(key, rev, start, size, status='o')
insert_with_storage(_id, _rev, value)
make_key_value(data)
update(key, rev, u_start=0, u_size=0, u_status='o')
update_with_storage(_id, _rev, value)
class CodernityDB.hash_index.MultiHashIndex(*args, **kwargs)

Bases: CodernityDB.hash_index.IU_MultiHashIndex

That class is designed to be used in custom indexes.

class CodernityDB.hash_index.UniqueHashIndex(db_path, name, entry_line_format='<32s8sIIcI', *args, **kwargs)

Bases: CodernityDB.hash_index.IU_UniqueHashIndex

That class is designed to be used in custom indexes. It’s designed to be id index.

B+Tree Index specific

Note

Please refer to B Plus Tree Index for description

class CodernityDB.tree_index.IU_MultiTreeBasedIndex(*args, **kwargs)

Bases: CodernityDB.tree_index.IU_TreeBasedIndex

Class that allows to index more than one key per database record.

It operates very well on GET/INSERT. It’s not optimized for UPDATE operations (will always readd everything)

delete(doc_id, key, start=0, size=0)
get(key)
insert(doc_id, key, start, size, status='o')
make_key_value(data)
update(doc_id, key, u_start, u_size, u_status='o')
class CodernityDB.tree_index.IU_TreeBasedIndex(db_path, name, key_format='32s', pointer_format='I', meta_format='32sIIc', node_capacity=10, storage_class=None)

Bases: CodernityDB.index.Index

all(limit=-1, offset=0)

Traverses linked list of all tree leaves and returns generator containing all elements stored in index.

close_index()
compact(node_capacity=0)
create_index()
custom_header = 'from CodernityDB.tree_index import TreeBasedIndex'
delete(doc_id, key, start=0, size=0)
destroy()
get(key)
get_between(start, end, limit=1, offset=0, inclusive_start=True, inclusive_end=True)
get_many(key, limit=1, offset=0)
insert(doc_id, key, start, size, status='o')
insert_first_record_into_leaf(leaf_start, key, doc_id, start, size, status)
make_key(key)
make_key_value(data)
open_index()
update(doc_id, key, u_start=0, u_size=0, u_status='o')
class CodernityDB.tree_index.MultiTreeBasedIndex(*args, **kwargs)

Bases: CodernityDB.tree_index.IU_MultiTreeBasedIndex

It allows to index more than one key for record. (ie. prefix/infix/suffix search mechanizms) That class is designed to be used in custom indexes.

exception CodernityDB.tree_index.NodeCapacityException

Bases: CodernityDB.index.IndexException

class CodernityDB.tree_index.TreeBasedIndex(db_path, name, key_format='32s', pointer_format='I', meta_format='32sIIc', node_capacity=10, storage_class=None)

Bases: CodernityDB.tree_index.IU_TreeBasedIndex

Storage

class CodernityDB.storage.DummyStorage

Bases: object

Storage mostly used to fake real storage

close(*args, **kwargs)
create(*args, **kwargs)
data_from(*args, **kwargs)
data_to(*args, **kwargs)
flush(*args, **kwargs)
fsync(*args, **kwargs)
get(*args, **kwargs)
insert(*args, **kwargs)
open(*args, **kwargs)
save(*args, **kwargs)
update(*args, **kwargs)
class CodernityDB.storage.IU_Storage(db_path, name='main')

Bases: object

close()
create()
data_from(data)
data_to(data)
destroy()
flush()
fsync()
get(start, size, status='c')
insert(data)
open()
save(data)
update(data)
class CodernityDB.storage.Storage(db_path, name='main')

Bases: CodernityDB.storage.IU_Storage

exception CodernityDB.storage.StorageException

Bases: exceptions.Exception

Patches

CodernityDB.patch.patch_cache_lfu(lock_obj)

Patnches cache mechanizm to be thread safe (gevent ones also)

Note

It’s internal CodernityDB mechanizm, it will be called when needed

CodernityDB.patch.patch_cache_rr(lock_obj)

Patches cache mechanizm to be thread safe (gevent ones also)

Note

It’s internal CodernityDB mechanizm, it will be called when needed

CodernityDB.patch.patch_flush_fsync(db_obj)

Will always execute index.fsync after index.flush.

Note

It’s for advanced users, use when you understand difference between flush and fsync, and when you definitely need that.

It’s important to call it AFTER database has all indexes etc (after db.create or db.open)

Example usage:

...
db = Database('/tmp/patch_demo')
db.create()
patch_flush_fsync(db)
...