| local | Per user, machine based. Typically insecure, relatively inaccessible to other applications and unlikely to be backed-up. However, for suitable applications it's cheap, simple and administration-free. An application running on www.faraway.doIcare.com can save user preferences without the server caring about data protection, logins or any sort of overhead. |
| server | Centrally stored, managed, secured and curated data. However there are technical and organisational issues which increase the overheads, and so the cost, and development timescale. Typically a blogging system might utilise data files and a database. This architecture is ideal for managing your orders and corporate data. |
Well, in an object oriented system the things that tend to get stored are complex and quite possibly fluid in their definition. This is not how traditional relational databases work. The alternative is to have a sort of dictionary where you ask the store for whatever it holds for a given 'key'. "Document reference:123" and that sort of thing. Browser local storage already works to this 'key' system and with CJC we can do things like put('configuration',myConfigObject) and of course myJobs = get('toDoList'). What happens if we want to look at files in a file system. You might make an AJAX request for http://someServer/data/email/inbox.idx then, having perused the index try doing something with http://someServer://data/email/bodies/123.eml. We know how to use file systems (and although we need the assistance of a server and a bit of protocol) we can use that. In this case the keys were inbox.idx and bodies/123.eml on top of a root of http://someServer/data/email/. (Having a root is a common way to restrict the realm the storage system can poke around in.)
What if we wanted to know how many email bodies there were? This is slightly more tricky as we've got to do more than fetch a known key. Now we're asking how many 'sub-keys' are there to "bodies/"? We might ask the file system to sort them in date order as that's what they're good at. Or the storage mechanism might be a non-sql database. Or pigeonholes, monkeys and typewriters.
So we need to decide what API will cover all these possibilities so that we might switch storage platforms or transfer objects between them. We want to work with a standardised API and leave the grunt and grime to a lower-level that we don't need to think about. (Also to be honest, I don't want to have to learn the syntax, quirks and re-test of multiple systems.) Remembering that in an object-oriented world we contain a lot more information in single objects, we ought to be looking at what we want any storage system to do for us to support our application programming. Cstore is the first iteration. We specify a template of methods which every storage system should be able to support.
If we use our pseudo-file-system then we should be able to ask it "List me the items in /foo/bar". ie. What are the sub-keys.
Then put an object into the store identified by a key and similarly get. (We will be using CJC for formatting, but in principle it could be any converter to-from internal representation.) Getting is not always going to find what we're looking for, so what then? We're expecting the store to give us a certain sort of object back and even objects with 'no data' may have defaults and methods that can be used. This is why we have a .GetOrCreate() method which delivers an object regardless of whether it's found in the store or not. Now we need a delete and we've got our essentials for storing and retrieving objects.
To explore the key-space we need some method which will list any sub-keys.
Overall management requires bulk import and export methods for data transfer (especially between stores) and replication. Also some sort of nuke-the-lot method.
| Examplevar ls=new ClocalStore('myApp');
var cfg = ls.GetOrCreate('settings',CmyAppConfig);
if(!cfg.settingsSet){DoSettingsWizardScreen();}
...application at work...
cfg.AddToRecentList(someThing);
cfg.Save(); // because we used GetOrCreate
// cfg knows how to save itself
...application at work...
ls.Put(['logs',thisProjName],errorsTextArray);
...later (or another page)...
var logList = ls.Keys('logs'); // list any logs
var logsInFull = logList.map(K=>ls.Get(K)); // read actual data
| ||||||||||||||||||
| Using ClocalStore | |||
General | |||
| Requirements | CJC.js and Cstore.js | All object to store operations use CJC | <script src="/jsLib/CJC.js" type="text/javascript"></script> |
| Keys | Slashed string or array of components | Strings are 0-9A-Aa-z_ and are relative to the application root specified in the constructor | ls.Put('users/fred/emails',myEmails);
var todo = ls.Get(['users','fred','todo']); |
| Security | No serious way to lock-out others. You could reasonably save password hashes but not encrypted passwords you expect to decrypt. | ||
A P I | |||
| Constructor | ClocalStore(AppRoot) | The application root means that all actions remain within a single 'partition'. Obviously you're responsible for managing this. All keys are relative to this. You can't do sneaky things like having a key of '../../foo/bar | var ls=new ClocalStore('myApp'); |
| Write an object | Put(Key,Object) | The object is encoded into a string using CJC then stored in local storage under the key appRoot/slashed keys | ls.Put('configuration',config); |
| Simple read | Get(Key) | Reads an item from local storage. This returns either an object (if an object was stored) or undefined You would use this when you're not expecting an object. |
var nameOfThing = ls.Get('name');
// string or undefined |
| Read an object | GetOrCreate(Key,Constructor ConArg1 ... ConArg4) |
If the key isn't found then create the required object. This always returns an instance of the class specified. A typical use-case is look-up some data but if not found return an empty data structure (with all it's methods of course). Up to four optional arguments can be supplied to the constructor if it's needed. | var cfg = ls.GetOrCreate('configuration',Csettings) |
| Save myself magic |
If you declare this.store and this.storeKey in your object constructor then GetOrLoad() will automatically add this filestore to .store and the key used as .storeKey. This means that your object can save itself with very simple internal code as in the example
|
if(this.changed){
this.store.Put(this.storeKey);
this.changed = false;
} |
|
| List keys | Keys(KeyPart) | Return an array of strings. If an argument is given you'll get just sub-keys. | var allKeys = ls.Keys(''); |
| Delete | Delete(Keys,AndSubs) | Delete the specified key. Or if this is a partial 'path' then you can opt to wipe the sub-keys as well. | // remove user and worksheets
ls.Delete(['users',userName],true); |
Other methods | |||
| Does key exist | Exists(Keys) | Returns boolean | if(ls.Exists(['users',newUserName]){ // duplicate! |
| Bulk load | Import(KeysAndVals, WipeFirst) |
Give this an array of [keyStrings,valueStrings] doublets. See Export(). | ls.Import(kvDoublets,true); |
| Bulk export | Export() | Returns an array of [keyStrings,valueStrings] doublets. You may find this is handy if you want to transfer all or some of your store to either another store (say on a server) or some backup media. Note that this exports the actual data in the store and theres no object (CJC) conversion. | var kvdoublets = ls.Export(); |
| Delete everything | WipeAreYouReallySure(Really) | Really needs to be "YES" for this to work. The funny name is to give you pause for thought. | WipeAreYouReallySure("YES"); |
Properties | |||
| Any changes | changed | Boolean set true if there are any Puts or Deletes. This is read/write. | if(ls.changed){
ExportToBackup(ls.Export());
ls.changed = false;
} |
| Key partition | keyRoot | Having a key root implements a partition so you have everything in the one known place. (If you find you're changing this then you probably need to re-think your architecture. | var kroot = ls.keyRoot; |
There are other wrinkles to make multiple actions 'safe' and efficient.
As file systems are good at last-modified time stamps we'll add that functionality to the client-side API.
Obviously we need a set of commands and responses which we'll call a protocol for the client-server interaction.
| Examplevar fs=new CfileStore('./fss.php','myApp');
fs.PrimeGet([myUserName,'messages.txt'],HandleNewMessages);
fs.PrimeGet([myUserName,'myPrefs'],ApplyPrefs,Cpreferences);
fs.Get([myUserName,'myPrefs']);
function ApplyPrefs(Prefs){
// do something with preferences object
}
| ||||||||||||||||||
| Using CfileStore | |||
General | |||
| Requirements | jquery, CJC.js, Cstore.js and CfileStore.js | All object to store operations use CJC | <script src="/jsLib/CJC.js" type="text/javascript"></script> |
| Keys | Slashed string or array of components | Strings are 0-9A-Aa-z_ and are relative to the application root specified in the constructor | fs.Get('users/fred/emails');
fs.Put(['users','fred','todo'],myTodoList); |
| Files as well as objects |
Here's a clever thing. The default formatting for the filestore is CJC. That is fs.Put('config',cfgObj) and the equivilent .Get() will automaticlly save and return full objects. This is the least effort for most value. But an extra feature is being to specify an actual file to be read or written as text. To do this add an extension to the key exactly as a filename. for example fs.Put('logs.txt',logs.array.join('\n')). Or fs.Get('sharePrices.xml') will pass a text file to the then-do. The server will vet extensions. | ||
| Security | All security is implemented by the server. | ||
| ThenDo | The general pattern is call a method to start a client-server action, then call a second function when this returns with a response from the server. For example fs.Exists('errors',function(Ky,Ex){console.log('Key:'+Ky+' Exists:'+Ex)});
There are wrinkles and such as when deleting a shoal of files how do you know when they're all completed? You can't rely on calling order! A really useful method is .WhenAllDo(ThenDo) which will execute the given function when all activity is finished. |
||
A P I | |||
| Constructor | CfileStore( ServerURL, AppName) | The server will restrict access to files in some place. In addition AppName is used to distinguish this application from others that may be using the same file store. Note that AppName is easily hackable but the server's 'partition' is fixed. | var fs = new CfileStore('./fss.php','myAppName'); |
| Write | Put(Keys, Data, Delay, ThenDo) | Gotcha!
In crude terms this writes an object to a file. Actually writes are put into a queue for a few milliseconds, say 1/3rd of a second before being sent to the server. This is to trap an application making a lot of changes to the same self-saving object in quick succession. You can optionally change this delay. The ThenDo function will be passed two arguments. Firstly the key and secondly a boolean to say if write queue is empty. The latter can be used when writing a shoal of files then acting when they've all been completed.
|
{ some code block
// save working data
fs.Put('config',cfgObj,0,LogOut);
fs.Put('recentEdits',editHistory,0,LogOut);
fs.Put('currentWorksheet',worksheet,0,LogOut);
}
stand-alone code block
// only closes down when all tidy-up Puts done
// called three times and any order
function Logout(Ky,AllDone){
if(!AllDone){return;}
some logout actions
}
|
| Read | PrimeGet(...) Get(Key) |
There are essentially three phases to reading a file or object.
PrimingPrimeGet(Keys,Target,ThenDo,Constructor,ConstructorArg1..4)The Target is a reference to an object. ThenDo is a function which takes a CfileStore_ReadCatalogue_Item object. The two properties you'll probably want are .key and .target. Target is the reference to the returned object which is also referenced by the variable you used as Target. See below for more details. The remaining arguments are for an optional constructor which is used when the filesystem can't find the requested key. This ensures that Target will always be an object of the specified class no matter what.
Trigger the readGet(Keys) kicks-off the fetching process as defined above. The Keys argument must match something previously primed. .Get returns true if reading is actually happening. (False could indicate already reading or key not primed.)Handling the returned value (ThenDo function)A typical pattern is to use the ThenDo in the priming. An alternative, discussed below is the .WhenAllDoneDo method.The safest way to access your returned object (because of scoping issues) is to use the reference provided via the .target property of the CfileStore_ReadCatalogue_Item passed to the handler function. This is the full ticket, not a copy or clone but the real-thing. See example. If you want to use the same handler for a more than one file then a useful property of the CRI is .key . If you're always going to use the returned data inside the handler then you can give an anonymous null object to .PrimeGet. Reading raw textAs discussed above, if you provide an extension such as .text or .html or whatever the data on the file is returned as-is. However .PrimeGet requires an object reference as its target. So you need to do the same empty object set-up and then read the .text property of that. |
Simple prime with in-line then-do
var myObj = {}; empty object. Will be constructed from file
fs.PrimeGet('someKey',myObj,function(RCI){RCI.target.DoSomething();},CsomeClass);
Using constructor to guarantee an object is returned
var myDoc = {}; Empty object. Will become a Cdocument instance
fs.PrimeGet(['documents',currentDoc],myDoc,HandleDocLoaded,Cdocument,userName,currentDocName);
fs.Get(['documents',currentDoc]); // Fetch or create document
Handler function
function HandleDocLoaded(RCI){
console.log('Handling:+RCI.key);
RCI.target.Render(); // guarenteed to be a Cdocument
}
Illustrating target referencing
var myDoc = {}; Empty object. Will become a Cdocument instance
. . . PrimeGet and Get . . .
Handler function
function HandleDocLoaded(RCI){
// .target will always be in scope
RCI.target.someProperty = 'foo';
// myDoc may or may not be in scope
var shouldBeFoo = myDoc.someProperty; // foo
}
Fetching raw text
var myHtml = {}; Empty object. Will have a .text property
fs.PrimeGet('boilerplate/foo.htm',myHtml,HandleBoilerplateLoad);
fs.Get('boilerplate/foo.htm'); // Fetch raw html
Handler function
function HandleBoilerplateLoad(RCI){
$('#doc')append(RCI.target.text); // shove into document
}
Fetching raw text (compact)
Does same as previous example
fs.PrimeGet('boilerplate/foo.htm',{},function(RCI){
$('#doc')append(RCI.target.text); // shove into document
});
|
| Waiting for multiple operations to finish | WhenAllDoneDo( ThenDo ) | Sometimes it's necessary to wait for everything to finish before carrying on. There is a way for put-handlers to test if they are the last successful put to return. This has been discussed above. An alternative, which lends itself to in-line, on-the spot coding, is WhenAllDoneDo(ThenDoFunction) This only triggers when all reads, writes, deletes and other actions have been completed. You can see from the example how the code appears to flow normally, with what-happens-next right below the actions.
Remember though that if there were other activities going on with the file store this wouldn't activate until those were complete as well. Don't confuse yourself by setting this a second time before the first has fired. The fileserver has two boolean properties you might want to query if WhenAllDoneDo is too embracing. .anyLeftToGet and .isWriteBufferEmpty |
Waiting for end of multiple deletes
obsoleteItems.forEach(I=>fs.Delete(I));
fs.WhenAllDoneDo(function(){
console.log('All deletes done');
now get on with something
}); |
| List keys | Keys( KeyStem, ThenDo, DontTrim) | Returns an array of strings. If a KeyStem is not "" only sub-keys (and perhaps the exact key) will be returned. ThenDo will be passed a reference to the filestore. Use the .keys property (Naming convention. Methods are capitalised, properties aren't.) If the optional DontTrim flag is true then keys that would normally rendered as having no extension will have the default .cjc added. | Getting sub-keys
fs.Keys('foo',function(FS){
console.log(FS.keys.join('\n')); // list of keys in the foo tree
});
|
| Deleting | Delete(Keys, AndChildren, ThenDo) | This is a sort of unlink or del which also can be used to remdir. Keys is the only required argument, with the other two in any order or not at all. AndChildren is a boolean. | See .WhenAllDoneDo above |
| Testing | Exists( Keys, ThenDo, Timestamp) | Does the item Keys exist and if so what about the timestamp? Keys is the usual slashed-string or array. ThenDo is function (Keys,Result) where result will be one of: "missing", "older", "newer" or a number which is the timestamp of the file in client-reference time. Timestamp is optional, and when provided gives a datum for the "older" and "newer" responses. Note: All timestamps (milliseconds) are notionally in client-time having been translated from server-time. However this tranlation is not precise as it depends (amongst other things) on connection latency and how awake the server file system happens to be when writing. So don't expect any precision better than a couple of seconds. Two enquiries to the same file will almost certainly give different numbers. |
fs.Exists('rawFile.txt',function(R){
var stamp = Number.parseInt(R,10);
var d = new Date(stamp);
console.log('rawfile.txt last modified:'+d);
},0); // zero or missing prompts numeric result |
Utilities and settings | |||
| Catch server errors | SetFailFunction( IfReadFailsFun) | A single 'onReadFail' function for all difficulties reading. It is given a CfileStore_ReadCatalogue_Item in the context of this CfileStore. Typically look at item.errorStr. The item.key is going to be useful too. | |
| Complete wipe | WipeAreYouReallySure( Really, ThenDo) | Really must be "YES". (This is to make programmers think before taking this step.) Remember this will only clear all 'application root' files. | |
| Bulk load | Import( KeysAndValues, WipeFirst, OlderThan, ThenDo) | KeysAndValues is an array of [KeyString,ValueString] doublets. This is the sort of thing produced by .Export(). WipeFirst is an optional boolean. Optional OlderThan IS NOT YET IMPLEMENTED. | |
| Bulk export | Export( ThenDo) | Creates an array of [keyStrings,valueStrings] doublets which is passed to ThenDo. You may find this is handy if you want to transfer all or some of your store to either another store (say on a server) or some backup media. Note that this exports the actual data in the store and theres no object (CJC) conversion. | fs.Export(function(KV){
console.log('Number of items exported:'+KV.length);
do something, eg Import to another store
} |
| Properties | .server Server URL .appName .writeIntervalMs Write buffer delay .isWriteBufferEmpty boolean | ||