Tryag File Manager

//proc/self/root/usr/share/doc/mx-2.0.6/BeeBase/Doc/mxBeeBase.html

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<HTML>
<HEAD>
 <TITLE>mxBeeBase - On-disk B+Tree Based Database Kit</TITLE>
 <META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
 <STYLE TYPE="text/css">
 p { text-align: justify; }
 ul.indent { }
 body { }
 </STYLE>
</HEAD>

<HR NOSHADE WIDTH="100%">
 <H2>mxBeeBase - On-disk B+Tree Based Database Kit</H2>

<HR SIZE=1 NOSHADE WIDTH="100%">
 <TABLE WIDTH="100%">
 <TR>
	<TD>
	 
	 Interface (
	 <A HREF="#BeeDict">BeeDict</A> :
	 <A HREF="#BeeStorage">BeeStorage</A> :
	 <A HREF="#BeeIndex">BeeIndex</A> )
	 <A HREF="#Examples">Examples</A> :
	 <A HREF="#Structure">Structure</A> :
	 <A HREF="#Support">Support</A> :
 <A HREF="http://www.egenix.com/files/python/eGenix-mx-Extensions.html#Download-mxBASE">Download</A> :
	 <A HREF="#Copyright">Copyright &amp; License</A> :
	 <A HREF="#History">History</A> :
	 <A HREF="" TARGET="_top">Home</A>
	
	</TD>
	<TD ALIGN=RIGHT VALIGN=TOP>
	 
	 Version 2.0.2
	 
	</TD>
 </TABLE>
 <HR SIZE=1 NOSHADE WIDTH="100%">

<H3>Introduction</H3>

<HR>

XXX This documentation is unfinished... the software
	 itself has been in use for quite a while, but the
	 documentation still needs to be completed. For now, all I
	 can offer you is to read the code which is well documented.

<HR>

mxBeeBase is a high performance construction kit for disk
	 based indexed databases. It offers components which you can
	 plug together to easily build your own custom mid-sized
	 databases (the current size limit is
	 <CODE>sizeof(long)</CODE> which gives you an address range
	 of around 2GB on 32-bit platforms).

The two basic building blocks in mxBeeBase are
	 storage and index. Storage is implemented as
	 variable record length data storage with integrated data
	 protection features, automatic data recovery and locking for
	 multi process access. Indexes use a high performance
	 optimized B+Tree implementation built on top of <A
	 HREF="http://epaperpress.com/">Thomas
	 Niemann</A>'s Cookbook B+Tree implementation.

Note: Due to the file locking mechanism currently
	 used in mxBeeBase, the package will only work on systems
	 which support symbolic links, e.g. Windows does not support
	 symbolic links and therefore parts of mxBeeBase don't work
	 on that platform.

</UL>

<H3>BeeDict Module</H3>

The BeeDict module provides two high level on-disk
	 dictionary implementations. The first (BeeDict) can
	 work with arbitrary hashable key objects, while the second
	 (BeeStringDict) uses limited sized strings as basis
	 providing slightly better performance. Both variants need
	 pickleable Python objects as keys and values.

Data transfer to and from the dictionaries is done in the
	 same way as for in-memory dictionaries, e.g. <CODE>d['key']
	 = 1; print d['key']; del d['key]</CODE>, so usage should be
	 transparent to Python programs using either in-memory or
	 on-disk dictionaries. Not all dictionary methods are
	 understood though.

<H4>BeeDict Objects</H4>

BeeDict objects are on-disk dictionaries which use a
	 hash-to-address index. Both Keys and values must be
	 pickleable and can have arbitrary size (keys shouldn't be
	 too long though); keys have to be hashable.

Hash collisions are treated by sequential reads of all
	 records with the same hash value and testing for equality of
	 keys. This can be expensive !

BeeDicts use a BeeStorage.BeeKeyValueStorage instance as
	 storage object and a BeeIndex.BeeIntegerIndex instance as
	 index object.

BeeDict objects are constructed using:

<DL>

<DT><CODE>
		BeeDict(name,min_recordsize=0,readonly=0,recover=0,
		autocommit=0,validate=0)</CODE></DT>

<DD>
	 Create an instance using <CODE>name</CODE> as basename for
	 the data and index files. Two files will be created:
	 name.dat and name.idx.

<CODE>min_recordsize</CODE> is passed to the BeeStorage
	 as indicator of the minimum size for data
	 records. <CODE>readonly</CODE> can be set to true to
	 open the files in read-only mode, preventing any disk
	 modifications.

To open the dictionary in recovery mode, pass a keyword
	 <CODE>recover=1</CODE>. Then run <CODE>.recover()</CODE>
	 and reopen using the normal settings. The
	 <CODE>AutoRecover()</CODE> wrapper can take care of this
	 action for you automatically.

If <CODE>autocommit</CODE> is true the cache control
	 will do an automatic <CODE>.commit()</CODE> whenever the
	 transaction log overflows.
 
	 
	 If <CODE>validate</CODE> is true, the dictionary will
	 run a validation check after having successfully opened
	 storage and index. <CODE>RecreateIndexError</CODE> or
	 <CODE>RecoverError</CODE> exceptions could be raised in
	 case inconsistencies are found.

</DD>

</DL>

</UL>

<H5>BeeDict Instance Methods</H5>

<UL CLASS="indent">
 
	
	 BeeDict objects have the following methods:

<DL>

<DT><CODE>
		flush()</CODE></DT>

<DD>
	 Flush buffers to disk.
	 </DD>

<DT><CODE>
		close()</CODE></DT>

<DD>
	 Flush buffers and close.
	 
	 This does not issue a <CODE>.commit()</CODE>, so the
	 current transaction is rolled back.

</DD>

<DT><CODE>
		commit()</CODE></DT>

<DD>
	 Commits all changes and starts a new transaction.

</DD>

<DT><CODE>
		rollback()</CODE></DT>

<DD>
	 Rolls back the current transaction. All changes are undone.

</DD>

<DT><CODE>
		changed()</CODE></DT>

<DD>
	 Return true in case the current transaction includes changes
 to the database, false otherwise.

</DD>

<DD>
	 Check if the dictionary has an item indexed by key.
	 
	 Successfully found items are put in the cache for fast
	 subsequent access.

</DD>

<DT><CODE>
		get(key,default=None)</CODE></DT>

<DD>
	 Get item indexed by key from the dictionary or default if
	 no such item exists.
	 
	 This first tries to read the item from cache and reverts
	 to the disk storage if it is not found.

</DD>

<DT><CODE>
		cursor(key,default=None)</CODE></DT>

<DD>
	 Return a BeeDictCursor instance for the dictionary.
	 
	 In case the key is not found, default is returned
	 instead.
	 
	 Note that cursors operate with the data on disk meaning
	 that any uncommitted changes will not be seen by the
	 cursor.

</DD>

<DT><CODE>
		garbage()</CODE></DT>

<DD>
	 Determine the amount of garbage in bytes that has accumulated
 in the storage file.
	 
	 This amount would be freed if .collect() were run.

</DD>

<DT><CODE>
		collect()</CODE></DT>

<DD>
	 Run the storage garbage collector.
	 
	 Storage collection can only be done for writeable
	 dictionaries and then only if the current transaction
	 does not contain any pending changes.
	 
	 This can take a while depending on the size of the
	 dictionary.

</DD>

<DT><CODE>
		recover()</CODE></DT>

<DD>
	 Recover all valid records and recreate the index.

</DD>

<DD>
	 Return a list of keys.
	 
	 The method does not load any data into the cache, but
	 does take notice of uncommitted changes.

</DD>

<DT><CODE>
		values()</CODE></DT>

<DD>
	 Return a list of values.
	 
	 The method does not load any data into the cache, but
	 does take notice of uncommitted changes.

</DD>

<DT><CODE>
		items()</CODE></DT>

<DD>
	 Return a list of items.
	 
	 The method does not load any data into the cache, but
	 does take notice of uncommitted changes.

</DD>

</DL>

</UL>

<H5>BeeDict Instance Attributes</H5>

<UL CLASS="indent">
 
	
	 BeeDict objects have the following read-only attributes:

<DL>

<DD>
	 Basenname of the dictionary. The implementation uses two
	 files for the on-disk representation: name.dat and name.idx.
	 </DD>

<DT><CODE>
		closed</CODE></DT>

<DD>
	 Closed flag.
	 </DD>

<DT><CODE>
		readonly</CODE></DT>

<DT><CODE>
		autocommit</CODE></DT>

<DD>
	 Autocommit flag.
	 </DD>

<DT><CODE>
		FirstKey</CODE></DT>

<DD>
	 Special key object useable to represent the first key in
	 the (sorted) B+Tree index.
	 
	 </DD>

<DT><CODE>
		LastKey</CODE></DT>

<DD>
	 Special key object useable to represent the last key in
	 the (sorted) B+Tree index.
	 
	 </DD>

</DL>

</UL>

<H4>BeeDictCursor Objects</H4>

BeeDickCursor objects are intended to iterate over the
	 database one item at a time without the need to read all
	 keys. You can then read/write to the current cursor position
	 and thus modify the dictionary in place.

Note that modifying the targetted dictionary while using a
	 cursor can cause the cursor to skip new entries or fail due
	 to deleted items. Especially deleting the key to which the
	 cursor currently points can cause errors to be raised. In
	 all other cases, the cursor will be repositioned.

BeeDictCursor objects are constructed using the BeeDict
	 <CODE>.cursor()</CODE> method.

</UL>

<H5>BeeDictCursor Instance Methods</H5>

<UL CLASS="indent">
 
	
	 BeeDictCursor objects have the following methods:

<DL>

<DT><CODE>
		position(key,value=None)</CODE></DT>

<DD>
	 Position the index cursor to index[key]. If value is given,
 index[key] == value is assured.
	 
	 key may also be FirstKey or LastKey (in which case value
	 has to be None).

</DD>

<DD>
	 Moves to the next entry in the dictionary.
	 
	 Returns true on success, false if the end-of-data has
	 been reached.

</DD>

<DD>
	 Moves to the previous entry in the dictionary.
	 
	 Returns true on success, false if the end-of-data has
	 been reached.

</DD>

<DD>
	 Reads the object from the dict to which the cursor
 currently points.

</DD>

<DD>
	 Reads the key object from the dict to which the cursor
	 currently points.

</DD>

<DT><CODE>
		write()</CODE></DT>

<DD>
	 Writes the object to the dict under the key to which
 the cursor currently points.
	 
	 The new data is not written to disk until the
	 dictionary's current transaction is committed.

</DD>

</DL>

</UL>

<H5>BeeDictCursor Instance Attributes</H5>

<UL CLASS="indent">
 
	
	 BeeDictCursor don't have any useful attributes. Use the instance
	 methods to query the key and value objects.

</UL>

<H4>BeeStringDict Objects</H4>

BeeStringDict objects are on-disk dictionaries which use a
	 limited size string to address index. Values must be
	 pickleable and can have arbitrary size.

Since hash collisions cannot occur this dictionary type may
	 have some performance advantages over the standard BeeDict
	 dictionary.

BeeStringDict objects are constructed using:

<DL>

<DT><CODE>
		BeeStringDict(name,keysize=10,min_recordsize=0,readonly=0,
		recover=0,autocommit=0,validate=0)</CODE></DT>

<DD>
	 Create an instance using <CODE>name</CODE> as basename for
	 the data and index files. Two files will be created:
	 name.dat and name.idx.

<CODE>keysize</CODE> gives the maximal size of the
	 strings used as index keys. <CODE>min_recordsize</CODE>
	 is passed to the BeeStorage as indicator of the minimum
	 size for data records. <CODE>readonly</CODE> can be set
	 to true to open the files in read-only mode, preventing
	 any disk modifications.

Note that the keysize is currently not stored in the
	 dictionary itself -- you'll have to store this
	 information in some other form. This may change in
	 future versions.

</DD>

</DL>

</UL>

<H5>BeeStringDict Instance Methods</H5>

<UL CLASS="indent">
 
	
	 BeeStringDict objects have the following methods:

<DL>

<DT><CODE>
		commit()</CODE></DT>

<DD>
	 Commits all changes and starts a new transaction.

</DD>

<DT><CODE>
		rollback()</CODE></DT>

<DD>
	 Rolls back the current transaction. All changes are undone.

</DD>

<DT><CODE>
		changed()</CODE></DT>

<DD>
	 Return true in case the current transaction includes changes
 to the database, false otherwise.

</DD>

<DD>
	 Check if the dictionary has an item indexed by key.
	 
	 Successfully found items are put in the cache for fast
	 subsequent access.

</DD>

<DT><CODE>
		get(key,default=None)</CODE></DT>

</DD>

<DT><CODE>
		cursor(key,default=None)</CODE></DT>

<DD>
	 Return a BeeStringDictCursor instance for the dictionary.
	 
	 In case the key is not found, default is returned
	 instead.
	 
	 Note that cursors operate with the data on disk meaning
	 that any uncommitted changes will not be seen by the
	 cursor.

</DD>

<DT><CODE>
		garbage()</CODE></DT>

<DD>
	 Determine the amount of garbage in bytes that has accumulated
 in the storage file.
	 
	 This amount would be freed if .collect() were run.

</DD>

<DT><CODE>
		collect()</CODE></DT>

</DD>

<DT><CODE>
		recover()</CODE></DT>

<DD>
	 Recover all valid records and recreate the index.

</DD>

<DD>
	 Return a list of keys.
	 
	 The method will raise an error if there are uncommitted
	 changes pending. Output is sorted ascending according to
	 keys.

</DD>

<DT><CODE>
		values()</CODE></DT>

<DD>
	 Return a list of values.
	 
	 The method will raise an error if there are uncommitted
	 changes pending. Output is sorted ascending according to
	 keys.

</DD>

<DT><CODE>
		items()</CODE></DT>

<DD>
	 Return a list of items.
	 
	 The method will raise an error if there are uncommitted
	 changes pending. Output is sorted ascending according to
	 keys.

</DD>

</DL>

</UL>

<H5>BeeStringDict Instance Attributes</H5>

<UL CLASS="indent">
 
	
	 BeeDict objects have the following read-only attributes:

<DL>

<DD>
	 Basenname of the dictionary. The implementation uses two
	 files for the on-disk representation: name.dat and name.idx.
	 </DD>

<DT><CODE>
		closed</CODE></DT>

<DD>
	 Closed flag.
	 </DD>

<DT><CODE>
		readonly</CODE></DT>

<DT><CODE>
		autocommit</CODE></DT>

<DD>
	 Autocommit flag.
	 </DD>

<DT><CODE>
		FirstKey</CODE></DT>

<DD>
	 Special key object useable to represent the first key in
	 the (sorted) B+Tree index.
	 
	 </DD>

<DT><CODE>
		LastKey</CODE></DT>

<DD>
	 Special key object useable to represent the last key in
	 the (sorted) B+Tree index.
	 
	 </DD>

</DL>

</UL>

<H4>BeeStringDictCursor Objects</H4>

BeeStringDictCursor objects are constructed using the BeeStringDict
	 <CODE>.cursor()</CODE> method.

</UL>

<H5>BeeStringDictCursor Instance Methods</H5>

<UL CLASS="indent">
 
	
	 BeeStringDictCursor objects have the following methods:

<DL>

<DT><CODE>
		position(key,value=None)</CODE></DT>

</DD>

<DD>
	 Moves to the next entry in the dictionary.
	 
	 Returns true on success, false if the end-of-data has
	 been reached.

</DD>

<DD>
	 Moves to the previous entry in the dictionary.
	 
	 Returns true on success, false if the end-of-data has
	 been reached.

</DD>

<DD>
	 Reads the object from the dict to which the cursor
 currently points.

</DD>

<DD>
	 Reads the key object from the dict to which the cursor
	 currently points.

</DD>

<DT><CODE>
		write()</CODE></DT>

</DD>

</DL>

</UL>

<H5>BeeStringDictCursor Instance Attributes</H5>

<UL CLASS="indent">
 
	
	 BeeStringDictCursor objects have the following read-only
	 attributes:

<DL>

<DD>
	 Key string which dereferences to the current cursor position.
	 </DD>

</DL>

</UL>

<H4>Functions</H4>

These functions are available:

<DL>

<DT><CODE>
		AutoRecover(Class,*args,**kws)</CODE></DT>

<DD>
	 Wrapper that can be used around the dictionary
	 constructors to provide automatic recovery whenever needed
	 (if possible).
	 
	 Example: <CODE>diskdict = AutoRecover(BeeDict.BeeDict,
	 'test')</CODE>

</DD>

</DL>

</UL>
 
 <H4>Constants</H4>

These constants are available:

<DL>

<DT><CODE>
		Error</CODE></DT>

<DD>
	 Baseclass for errors related to this module. It is a
	 subclass of exceptions.StandardError. </DD>

<DT><CODE>
		RecreateIndexError</CODE></DT>

<DD>
	 This error is raised in case the index for a dictionary
	 was not found and/or needs to be recreated by running
	 recovery. It is a subclass of Error. </DD>

<DT><CODE>
		RecoverError</CODE></DT>

<DD>
	 This error is raised in case the storage for a dictionary
	 was found to be in an inconsistent state. It is a subclass
	 of Error. </DD>

<DT><CODE>
		FirstKey</CODE></DT>

<DD>
	 Special index key object representing the key of the first
	 entry in the index (B+Tree's sort their data).

</DD>

<DT><CODE>
		LastKey</CODE></DT>

<DD>
	 Special index key object representing the key of the last
	 entry in the index (B+Tree's sort their data).

</DD>

</DL>

</UL>

</UL>
 
 <A NAME="BeeStorage"></A>

<H3>BeeStorage Module</H3>

XXX Not yet documented. Please refer to the source code.

</UL>
 
 <A NAME="BeeIndex"></A>

<H3>BeeIndex Module</H3>

XXX Not yet documented. Please refer to the source code.

</UL>
 
 <A NAME="Examples"></A>

<H3>Examples of Use</H3>

Here is a very simple one:

<PRE>from mx import BeeBase

#...XXX not written yet...

</PRE>

More examples will eventually appear in the Examples
	 subdirectory of the package.

</UL>

<H3>Package Structure</H3>
 
 <UL CLASS="indent">

<PRE>
[BeeBase]
 Doc/
 [mxBeeBase]
 test.py
 BeeBase.py
 BeeDict.py
 BeeIndex.py
 BeeStorage.py
 showBeeDict.py
 	</PRE>

Entries enclosed in brackets are packages (i.e. they are
	 directories that include a <TT>__init__.py</TT> file). Ones
	 without brackets are just simple subdirectories that are not
	 accessible via <CODE>import</CODE>.

The package imports all symbols from the BeeBase sub module
	 which in turn imports the extension module, so you only need
	 to '<CODE>import BeeBase</CODE>' to start working.

</UL>

<H3>Support</H3>

eGenix.com is providing commercial support for this
	 package. If you are interested in receiving information
	 about this service please see the <A
	 HREF="http://www.egenix.com/files/python/eGenix-mx-Extensions.html#Support">eGenix.com
	 Support Conditions</A>.

</UL>

<H3>Copyright &amp; License</H3>

&copy; 1999-2000, Copyright by Marc-Andr&eacute; Lemburg;
	 All Rights Reserved. mailto: <A
	 HREF="mailto:mal@lemburg.com">mal@lemburg.com</A>
	
	 &copy; 2000-2001, Copyright by eGenix.com Software GmbH,
	 Langenfeld, Germany; All Rights Reserved. mailto: <A
	 HREF="mailto:info@egenix.com">info@egenix.com</A>

This software is covered by the <A
	 HREF="mxLicense.html#Public">eGenix.com Public
	 License Agreement</A>. The text of the license is also
	 included as file "LICENSE" in the package's main directory.

By downloading, copying, installing or otherwise using
	 the software, you agree to be bound by the terms and
	 conditions of the eGenix.com Public License
	 Agreement.

Parts of this package are based on an ANSI C implementation
 	 of a B+Tree implementation written by Thomas Nieman,
 	 Portland, Oregon. The files in question are btr.c and btr.h
 	 which were heavily modified for the purpose of inclusion in
 	 this package by the above author.

The original files were extracted from btr.c -- an ANSI C
 	 implementation included in the source code distribution of

<PRE>
 SORTING AND SEARCHING ALGORITHMS: A COOKBOOK

by THOMAS NIEMANN Portland, Oregon 
	home: <A HREF="http://epaperpress.com/">http://epaperpress.com/</A>
	</PRE>
 
	
	 From the cookbook:
	<BLOCKQUOTE>
	 Permission to reproduce this document, in whole or in part,
 	 is given provided the original web site listed below is
 	 referenced, and no additional restrictions apply. Source
 	 code, when part of a software project, may be used freely
 	 without reference to the author.
	</BLOCKQUOTE>

</UL>

<H3>History & Future</H3>

Things that still need to be done:

<UL>

<LI> Provide more examples.

<LI> Test the package some more.

<LI> Write more documentation.

</UL>

There were no significant changes between 2.0.0 and 2.0.3.

Version 2.0.0 was the intial release.

</UL>

<HR WIDTH="100%">
 <CENTER>
 
 &copy; 1998-2000, Copyright by Marc-Andr&eacute; Lemburg;
 All Rights Reserved. mailto: <A
 HREF="mailto:mal@lemburg.com">mal@lemburg.com</A>
 
 &copy; 2000-2001, Copyright by eGenix.com Software GmbH; 
 All Rights Reserved. mailto: <A
 HREF="mailto:info@egenix.com">info@egenix.com</A>
 </CENTER>
 </CENTER>

</BODY>
</HTML>