Miscellanea menu
Unified storage for javascript objects including classes

Purpose. Introduction to object storage.

Applications need to save and retrieve data. They do this by having some label (we'll call a key) and a convention for representing useful things in a form the program can use and the storage media can handle. In the world of Javascript applications there are two sorts of storage which we might use:
local Per user, machine based. Typically insecure, relatively inaccessible to other applications and unlikely to be backed-up. However, for suitable applications it's cheap, simple and administration-free. An application running on www.faraway.doIcare.com can save user preferences without the server caring about data protection, logins or any sort of overhead.
server Centrally stored, managed, secured and curated data. However there are technical and organisational issues which increase the overheads, and so the cost, and development timescale. Typically a blogging system might utilise data files and a database. This architecture is ideal for managing your orders and corporate data.
The issue as far as Javascript classes is concerned the same in both cases. We need a way to convert functional classes, complete with methods and inheritance, to and from a format which can be processed by whatever storage method we use. OK, that's been taken care of by CJC so what's left to do?

Well, in an object oriented system the things that tend to get stored are complex and quite possibly fluid in their definition. This is not how traditional relational databases work. The alternative is to have a sort of dictionary where you ask the store for whatever it holds for a given 'key'. "Document reference:123" and that sort of thing. Browser local storage already works to this 'key' system and with CJC we can do things like put('configuration',myConfigObject) and of course myJobs = get('toDoList'). What happens if we want to look at files in a file system. You might make an AJAX request for http://someServer/data/email/inbox.idx then, having perused the index try doing something with http://someServer://data/email/bodies/123.eml. We know how to use file systems (and although we need the assistance of a server and a bit of protocol) we can use that. In this case the keys were inbox.idx and bodies/123.eml on top of a root of http://someServer/data/email/. (Having a root is a common way to restrict the realm the storage system can poke around in.)

What if we wanted to know how many email bodies there were? This is slightly more tricky as we've got to do more than fetch a known key. Now we're asking how many 'sub-keys' are there to "bodies/"? We might ask the file system to sort them in date order as that's what they're good at. Or the storage mechanism might be a non-sql database. Or pigeonholes, monkeys and typewriters.

So we need to decide what API will cover all these possibilities so that we might switch storage platforms or transfer objects between them. We want to work with a standardised API and leave the grunt and grime to a lower-level that we don't need to think about. (Also to be honest, I don't want to have to learn the syntax, quirks and re-test of multiple systems.) Remembering that in an object-oriented world we contain a lot more information in single objects, we ought to be looking at what we want any storage system to do for us to support our application programming. Cstore is the first iteration. We specify a template of methods which every storage system should be able to support.

Wht not use indexedDb? Have you tried to use it? It's a Moebius knot.

Hierarchy of keys, put, get and typical management operations

In an OO system we don't worry so much about defining tables in a database, but we still need a way to access data in the store which may be organised but not explicitly in our OO knowledge. It's a bit like when we go shopping we have a shopping list (keys if you like) but also we browse the displays, perhaps on the lookout for special offers. We definitely need to be able to explore the data store.

Keys

With localStorage and file systems we already use strings as keys. Let's reuse the files paradigm of path/path/path/etcetera. Immediately we have a mapping between keys and the files if we choose to use it. (Why not? Because we've got to be careful not to allow illegal characters. So there's an immediate restriction on key-string values we need to note.)

If we use our pseudo-file-system then we should be able to ask it "List me the items in /foo/bar". ie. What are the sub-keys.

That was easy. Reuse the proven hierarchical path system for structuring keys.

Operations

Firstly we need to establish some sort of connection between the underlying store and our Cstore abstraction layer. In OO thats a constructor. We almost certainly want to establish some root so that this application is limited to a particular realm.

Then put an object into the store identified by a key and similarly get. (We will be using CJC for formatting, but in principle it could be any converter to-from internal representation.) Getting is not always going to find what we're looking for, so what then? We're expecting the store to give us a certain sort of object back and even objects with 'no data' may have defaults and methods that can be used. This is why we have a .GetOrCreate() method which delivers an object regardless of whether it's found in the store or not. Now we need a delete and we've got our essentials for storing and retrieving objects.

To explore the key-space we need some method which will list any sub-keys.

Overall management requires bulk import and export methods for data transfer (especially between stores) and replication. Also some sort of nuke-the-lot method.

That was simple. There doesn't seem to be anything complicated about implementing a standard storage layer for objects. (It's just eight methods, although of course there might be tricky aspects in actual implementation.)

Cstore, ClocalStore and CfileStore

Cstore is an abstract class used as a template for actual implementations.
ClocalStore is a pure in-browser implementation which doesn't require any server.
CfileStore talks to a very simple server which takes POST requests with some action to accept or serve files. The main use is to store Javascript objects, but a trivial tweak means we can read and write any text.

The method framework is consistent in the use of keys and what the methods do, but there's no escaping the fact that CfileStore has to operate asynchronously which means we need to add ThenDo follow up functions.

Cstore - abstract class as a foundation for keys and catalogue of methods

Constructor

This is where we establish the 'application root' we'll be confining ourselves to. All data will be within this realm. We might want to check we're able to access the store. For example 'myApp'.

Writing data

We'll call this .Put(). For a local store all we need to do is Put(Key,String) and we're done but for a remote store we may want to wait until either this particular .Put is complete or a shoal of .Puts is complete.

Reading data

We'll call this .Get(). One particularly useful thing we can do when reading objects is to construct a default object if it isn't found in the store. In this way we always have a functional object returned even if the data elements are as yet unpopulated. ClocalStore implements this functionality with .GetOrCreate() and that's that, but in the more complex asynchronous world of CfileStore we pre-register what happens when we get a key using .PrimeGet() which sets up all the default constructoring and Thendo. Then .Get() is an event trigger rather than a function returning anything. Sometimes we might want to wait until a shoal of .Gets have all been completed.

Poking around

.Keys() works can be used to return the whole 'tree' of keys or a 'sub-branch'. Note that this is not the same as an actual file system tree. Yes, other server-side processes might read and write to files in your keystore, but you can't use it to browse arbitary directories. But look here.

Delete

.Delete() can delete a single key or a branch. In CfileStore we have a Then-do which will be given a flag to say if all deletes have been completed as we may be making a shoal of deletions and only want to act when they've all been done. The issue is that we don't know which server call will be the last to complete.

Individual key status

Does a key exist? In a file based store we can examine the timestamp

ClocalStore - Dead simple local functional object storage

Characteristics
instant availabilityNo need to mess around with configuring a server
per user storageIdeal for configuration and private workspaces.
simple callsCalls return immediately without the necessity of dealing with promises etc.
hierarchical keys...Just like a familiar file system
...on a partitionTo clarify management

Example

var ls=new ClocalStore('myApp'); var cfg = ls.GetOrCreate('settings',CmyAppConfig); if(!cfg.settingsSet){DoSettingsWizardScreen();} ...application at work... cfg.AddToRecentList(someThing); cfg.Save(); // because we used GetOrCreate // cfg knows how to save itself ...application at work... ls.Put(['logs',thisProjName],errorsTextArray); ...later (or another page)... var logList = ls.Keys('logs'); // list any logs var logsInFull = logList.map(K=>ls.Get(K)); // read actual data
Using ClocalStore
General
Requirements CJC.js and Cstore.js All object to store operations use CJC
<script src="/jsLib/CJC.js" type="text/javascript"></script>
Keys Slashed string or array of components Strings are 0-9A-Aa-z_ and are relative to the application root specified in the constructor
ls.Put('users/fred/emails',myEmails); var todo = ls.Get(['users','fred','todo']);
Security No serious way to lock-out others. You could reasonably save password hashes but not encrypted passwords you expect to decrypt.
A P I
Constructor ClocalStore(AppRoot) The application root means that all actions remain within a single 'partition'. Obviously you're responsible for managing this. All keys are relative to this. You can't do sneaky things like having a key of '../../foo/bar
var ls=new ClocalStore('myApp');
Write an object Put(Key,Object) The object is encoded into a string using CJC then stored in local storage under the key appRoot/slashed keys
ls.Put('configuration',config);
Simple read Get(Key) Reads an item from local storage. This returns either an object (if an object was stored) or undefined
You would use this when you're not expecting an object.
var nameOfThing = ls.Get('name'); // string or undefined
Read an object GetOrCreate(Key,Constructor
ConArg1 ... ConArg4)
If the key isn't found then create the required object. This always returns an instance of the class specified. A typical use-case is look-up some data but if not found return an empty data structure (with all it's methods of course). Up to four optional arguments can be supplied to the constructor if it's needed.
var cfg = ls.GetOrCreate('configuration',Csettings)
Save myself magic
If you declare this.store and this.storeKey in your object constructor then GetOrLoad() will automatically add this filestore to .store and the key used as .storeKey. This means that your object can save itself with very simple internal code as in the example
if(this.changed){ this.store.Put(this.storeKey); this.changed = false; }
List keys Keys(KeyPart) Return an array of strings. If an argument is given you'll get just sub-keys.
var allKeys = ls.Keys('');
Delete Delete(Keys,AndSubs) Delete the specified key. Or if this is a partial 'path' then you can opt to wipe the sub-keys as well.
// remove user and worksheets ls.Delete(['users',userName],true);
Other methods
Does key exist Exists(Keys) Returns boolean
if(ls.Exists(['users',newUserName]){ // duplicate!
Bulk load Import(KeysAndVals,
WipeFirst)
Give this an array of [keyStrings,valueStrings] doublets. See Export().
ls.Import(kvDoublets,true);
Bulk export Export() Returns an array of [keyStrings,valueStrings] doublets. You may find this is handy if you want to transfer all or some of your store to either another store (say on a server) or some backup media. Note that this exports the actual data in the store and theres no object (CJC) conversion.
var kvdoublets = ls.Export();
Delete everything WipeAreYouReallySure(Really) Really needs to be "YES" for this to work. The funny name is to give you pause for thought.
WipeAreYouReallySure("YES");
Properties
Any changes changed Boolean set true if there are any Puts or Deletes. This is read/write.
if(ls.changed){ ExportToBackup(ls.Export()); ls.changed = false; }
Key partition keyRoot Having a key root implements a partition so you have everything in the one known place. (If you find you're changing this then you probably need to re-think your architecture.
var kroot = ls.keyRoot;

Filestore introduction

Client

The asynchronous nature of accessing a server adds the complication of having to process a response only when it has returned. There are layers to this, but the basic principle is We have to tell CfileStore what the ThenDo function will be at the time of making the call.

There are other wrinkles to make multiple actions 'safe' and efficient.

As file systems are good at last-modified time stamps we'll add that functionality to the client-side API.

Server

We need to handle requests on the server. We can do this with say for example PHP running on Apache, or write a all-in-one server in say node.js. This application will exercise caution and limit what's possible on the file system by allocating a 'partition' which is coded into the server.

Obviously we need a set of commands and responses which we'll call a protocol for the client-server interaction.

Because implementing a server is a separate thing there's a separate page for it here. (It's easy if you use PHP, but it ought to be explained properly to those interested.)

CfileStore - Dead simple functional object storage on a server

Characteristics
asynchronousYou have to get used to event-driven programming.
accessable by other apps and usersIdeal for shared data.
few callsThere are only a few calls to get used to.
hierarchical keys...Just like a familiar file system
...on a partitionFor safe and clear management
r/w access to raw text filesWithin the partition, eg errlog.txt

Example

var fs=new CfileStore('./fss.php','myApp'); fs.PrimeGet([myUserName,'messages.txt'],HandleNewMessages); fs.PrimeGet([myUserName,'myPrefs'],ApplyPrefs,Cpreferences); fs.Get([myUserName,'myPrefs']); function ApplyPrefs(Prefs){ // do something with preferences object }
Using CfileStore
General
Requirements jquery, CJC.js, Cstore.js and CfileStore.js All object to store operations use CJC
<script src="/jsLib/CJC.js" type="text/javascript"></script>
Keys Slashed string or array of components Strings are 0-9A-Aa-z_ and are relative to the application root specified in the constructor
fs.Get('users/fred/emails'); fs.Put(['users','fred','todo'],myTodoList);
Files as well
as objects
Here's a clever thing. The default formatting for the filestore is CJC. That is fs.Put('config',cfgObj) and the equivilent .Get() will automaticlly save and return full objects. This is the least effort for most value. But an extra feature is being to specify an actual file to be read or written as text. To do this add an extension to the key exactly as a filename. for example fs.Put('logs.txt',logs.array.join('\n')). Or fs.Get('sharePrices.xml') will pass a text file to the then-do. The server will vet extensions.
Security All security is implemented by the server.
ThenDo The general pattern is call a method to start a client-server action, then call a second function when this returns with a response from the server. For example fs.Exists('errors',function(Ky,Ex){console.log('Key:'+Ky+' Exists:'+Ex)});

There are wrinkles and such as when deleting a shoal of files how do you know when they's all completed? You can't rely on calling order! A really useful method is .WhenAllDo(ThenDo) which will execute the given function when all activity is finished.

A P I
Constructor CfileStore( ServerURL, AppName) The server will restrict access to files in some place. In addition AppName is used to distinguish this application from others that may be using the same file store. Note that AppName is easily hackable but the server's 'partition' is fixed.
var fs = new CfileStore('./fss.php','myAppName');
Write Put(Keys, Data, Delay, ThenDo)
Gotcha!
fs.Put('foo',DontBotherFun); fs.Put('bar',AllPutsDoneFun);
The 'bar' write may return before the 'foo write!
In crude terms this writes an object to a file. Actually writes are put into a queue for a few milliseconds, say 1/3rd of a second before being sent to the server. This is to trap an application making a lot of changes to the same self-saving object in quick succession. You can optionally change this delay. The ThenDo function will be passed two arguments. Firstly the key and secondly a boolean to say if write queue is empty. The latter can be used when writing a shoal of files then acting when they've all been completed.
{ some code block // save working data fs.Put('config',cfgObj,0,LogOut); fs.Put('recentEdits',editHistory,0,LogOut); fs.Put('currentWorksheet',worksheet,0,LogOut); } stand-alone code block // only closes down when all tidy-up Puts done // called three times and any order function Logout(Ky,AllDone){ if(!AllDone){return;} some logout actions }
Read PrimeGet(...)
Get(Key)
There are essentially three phases to reading a file or object.
  1. A one-off priming where we set up all the necessary parameters
  2. A Get which triggers an immediate server request
  3. A then-do function to handle the data/object when it's received from the server.
In practice you might code the then-do in-line with the prime step a bit like you might wire up a circuit. The Get step just sends a pulse through the circuit.

Priming

PrimeGet(Keys,Target,ThenDo,Constructor,ConstructorArg1..4)
The Target is a reference to an object. This can be done as follows if you like:

ThenDo is a function which takes CfileStore_ReadCatalogue_Item object. The two properties you'll probably want are .key and .target. Target is the reference to the returned object which is also referenced by the variable you used as Target. See below for more details.

The remaining arguments are for an optional constructor which is used when the filesystem can't find the requested key. This ensures that Target will always be an object of the specified class no matter what.

Trigger the read

Get(Keys) kicks-off the fetching process as defined above. The Keys argument must match something previously primed. .Get returns true if reading is actually happening. (False could indicate already reading or key not primed.)

Handling the returned value (ThenDo function)

A typical pattern is to use the ThenDo in the priming. An alternative, discussed below is the .WhenAllDoneDo method.

The safest way to access your returned object (because of scoping issues) is to use the reference provided via the .target property of the CfileStore_ReadCatalogue_Item passed to the handler function. This is the full ticket, not a copy or clone but the real-thing. See example. If you want to use the same handler for a more than one file then a useful property of the CRI is .key .

If you're always going to use the returned data inside the handler then you can give an anonymous null object to .PrimeGet.

Reading raw text

As discussed above, if you provide an extension such as .text or .html or whatever the data on the file is returned as-is. However .PrimeGet requires an object reference as its target. So you need to do the same empty object set-up and then read the .text property of that.
Simple prime with in-line then-do var myObj = {}; empty object. Will be constructed from file fs.PrimeGet('someKey',myObj,function(RCI){RCI.target.DoSomething();},CsomeClass);

Using constructor to guarantee an object is returned var myDoc = {}; Empty object. Will become a Cdocument instance fs.PrimeGet(['documents',currentDoc],myDoc,HandleDocLoaded,Cdocument,userName,currentDocName); fs.Get(['documents',currentDoc]); // Fetch or create document Handler function function HandleDocLoaded(RCI){ console.log('Handling:+RCI.key); RCI.target.Render(); // guarenteed to be a Cdocument }

Illustrating target referencing var myDoc = {}; Empty object. Will become a Cdocument instance . . . PrimeGet and Get . . . Handler function function HandleDocLoaded(RCI){ // .target will always be in scope RCI.target.someProperty = 'foo'; // myDoc may or may not be in scope var shouldBeFoo = myDoc.someProperty; // foo }

Fetching raw text var myHtml = {}; Empty object. Will have a .text property fs.PrimeGet('boilerplate/foo.htm',myHtml,HandleBoilerplateLoad); fs.Get('boilerplate/foo.htm'); // Fetch raw html Handler function function HandleBoilerplateLoad(RCI){ $('#doc')append(RCI.target.text); // shove into document }

Fetching raw text (compact) Does same as previous example fs.PrimeGet('boilerplate/foo.htm',{},function(RCI){ $('#doc')append(RCI.target.text); // shove into document });
Waiting for multiple operations to finish WhenAllDoneDo( ThenDo ) Sometimes it's necessary to wait for everything to finish before carrying on. There is a way for put-handlers to test if they are the last successful put to return. This has been discussed above. An alternative, which lends itself to in-line, on-the spot coding, is WhenAllDoneDo(ThenDoFunction) This only triggers when all reads, writes, deletes and other actions have been completed. You can see from the example how the code appears to flow normally, with what-happens-next right below the actions.

Remember though that if there were other activities going on with the file store this wouldn't activate until those were complete as well. Don't confuse yourself by setting this a second time before the first has fired.

The fileserver has two boolean properties you might want to query if WhenAllDoneDo is too embracing. .anyLeftToGet and isWriteBufferEmpty

Waiting for end of multiple deletes obsoleteItems.forEach(I=>fs.Delete(I)); fs.WhenAllDoneDo(function(){ console.log('All deletes done'); now get on with something });
List keys Keys( KeyStem, ThenDo, DontTrim) Returns an array of strings. If a KeyStem is not "" only sub-keys (and perhaps the exact key) will be returned. ThenDo will be passed a reference to the filestore. Use the .keys property (Naming comvention. Methods are capitalised, properties aren't.) If the optional DontTrim flag is true then keys that would normally rendered as having no extension will have the default .cjc added.
Getting sub-keys fs.Keys('foo',function(FS){ console.log(FS.keys.join('\n')); // list of keys in the foo tree });
Deleting Delete(Keys, AndChildren, ThenDo) This is a sort of unlink or del which also can be used to remdir. Keys is the only required argument, with the other two in any order or not at all. And See .WhenAllDoneDo above
Testing Exists( Keys, ThenDo, Timestamp) Does the item Keys exist and if so what about the timestamp?
Keys is the usual slashed-string or array. ThenDo is function (Keys,Result) where result will be one of: "missing", "older", "newer" or a number which is the timestamp of the file in client-reference time. Timestamp is optional, and when provided gives a datum for the "older" and "newer" responses.

Note: All timestamps (milliseconds) are notionally in client-time having been translated from server-time. However this tranlation is not precise as it depends (amongst other things) on connection latency and how awake the server file system happens to be when writing. So don't expect any precision better than a couple of seconds. Two enquiries to the same file will almost certainly give different numbers.

fs.Exists('rawFile.txt',function(R){ var stamp = Number.parseInt(R,10); var d = new Date(stamp); console.log('rawfile.txt last modified:'+d); },0); // zero or missing prompts numeric result
Utilities and settings
Catch server errors SetFailFunction( IfReadFailsFun) A single 'onReadFail' function for all difficulties reading. It is given a CfileStore_ReadCatalogue_Item in the context of this CfileStore. Typically look at item.errorStr. The item.key is going to be useful too.
Complete wipe WipeAreYouReallySure( Really, ThenDo) Really must be "YES". (This is to make programmers think before taking this step.) Remember this will only clear all 'application root' files.
Bulk load Import( KeysAndValues, WipeFirst, OlderThan, ThenDo) KeysAndValues is an array of [KeyString,ValueString] doublets. This is the sort of thing produced by .Export(). WipeFirst is an optional boolean. Optional OlderThan IS NOT YET IMPLEMENTED.
Bulk export Export( ThenDo) Creates an array of [keyStrings,valueStrings] doublets which is passed to ThenDo. You may find this is handy if you want to transfer all or some of your store to either another store (say on a server) or some backup media. Note that this exports the actual data in the store and theres no object (CJC) conversion.
fs.Export(function(KV){ console.log('Number of items exported:'+KV.length); do something, eg Import to another store }
Properties .server Server URL .appName .writeIntervalMs Write buffer delay .isWriteBufferEmpty boolean