Cirrus is a DB Engine that brings together Datalog and essentially logs/streams for SkyDB. It allows you to construct datalog queries to extract data from the facts that are included in the Database. That was alot of coded terminology. Essentially Cirrus provides a way for you to query your data as well as other peoples data in a typed and well structured format. Additionally it utilizes datalog in order to allow you to extract information from the data sets.
The thought here is that SkyDB could benefit from building a database solution that works in line with a streaming architecture. Instead of building mutable data built of less composable items, build immutable atomic data, that can be used to materialize views that represent the current state. Doing this allows for a host of portable applications that can improve the developer experience.
Simply setting your dataset to be a log stream isn't enough, people need ways to extract value from the data streams. Since business data often mirrors a graph, a future infrastructure leans into this notion to achieve better user query reliability and general data processing performance embracing this see YedaLog and Datomic
The goal here is to build a framework, that can allow developers that use skydb to extract value, and to build an extensible knowledge graph across applications, without needing to maintain additional servers. This is a proof of concept implementation and is Experimental
- node v15.x.x or greater.
- npm v6.10.3 or greater.
Install via npm package
mkdir -p example
cd example
npm init
npm install -g @createdreamtech/cirrus
touch example.js
copy pasta below int example.js
compile for webShort Example
const cirrus = require('@createdreamtech/cirrus')
const datalog = require("@createdreamtech/datalog")
const {Cirrus, SkyDBStorage} = cirrus;
//import {Cirrus, SkyDBStorage} from "@createdreamtech/cirrus";
//import datalog from "@createdreamtech/datalog";
(async function(){
const PeopleSchema = {
id: datalog.NumberType,
name: datalog.StringType,
}
const ParentSchema = {
parentId: datalog.NumberType,
childId: datalog.NumberType
}
const appKey = "my-ancestry-clone-dot-com"
const secret = `cirrus-seed-${Math.random()}`
const storage = new SkyDBStorage(secret, appKey)
const pubKey=storage.origin()
console.log(`my pubKey: ${storage.origin()}`)
const db = new Cirrus(storage,storage.origin())
await db.init()
const People = await db.newTable("People",PeopleSchema)
const ParentOf = await db.newTable("ParentOf", ParentSchema)
People.asserts({id: 1, name: "Jane"})
People.asserts({id: 2, name: "Joan"})
People.asserts({id: 10, name: "Jackie"})
People.asserts({id: 12, name: "Johnne"})
People.asserts({id: 22, name: "Grandpa Jones"})
People.asserts({id: 20, name: "Grandma Jones"})
ParentOf.asserts({parentId:10,childId:1})
ParentOf.asserts({parentId:12,childId:2})
ParentOf.asserts({parentId:12,childId:1})
ParentOf.asserts({parentId:22,childId:12})
ParentOf.asserts({parentId:20,childId:10})
//saves all data even data from remote dbs
// await db.save();
//save only actions related to your database (asserts/retractions)
await db.saveOnly((action)=>{action.__origin === pubKey})
//Parents of Jane
const qq = datalog.query(({id, parentId})=> {
People({id,name:"Jane"})
ParentOf({parentId,childId:id})
})
console.log(qq.view().readAllData())
const qqq = datalog.query(({id, name, parentId})=> {
People({id,name:"Jane"})
ParentOf({parentId,childId:id})
People({id:parentId,name})
})
console.log(qqq.view().readAllData())
//Grandparents
//Good explantion of how this works out https://x775.net/2019/03/18/Introduction-to-Datalog.html
const q = datalog.query(({name,id, parentId, childId})=> {
People({id,name})
ParentOf({parentId:id,childId})
ParentOf({parentId:childId,childId:parentId})
})
console.log(q.view().readAllData())
})();outputs
[ { id: 1, parentId: 10 }, { id: 1, parentId: 12 } ]
[
{ id: 1, parentId: 10, name: 'Jackie' },
{ id: 1, parentId: 12, name: 'Johnne' }
]
[
{ id: 20, name: 'Grandma Jones', childId: 10, parentId: 1 },
{ id: 22, name: 'Grandpa Jones', childId: 12, parentId: 1 },
{ id: 22, name: 'Grandpa Jones', childId: 12, parentId: 2 }
]Live React Example Example based off the example app that allows you to retrieve someone else's notes created from the example. Github link
src
├── actions
│ └── index.ts
├── cirrus
│ └── index.ts
├── index.ts
├── integration
│ └── index.test.ts
├── serialize
│ ├── actions.ts
│ ├── index.ts
│ └── schema.ts
└── storage
├── index.ts
├── memory.ts
├── skydb.ts
└── storage.ts
These are equivalent to datalog facts. Kind of like database inserts
These allow you retract a fact, essentially undoing the effect of an fact.
An application key is just a developer generated public key that they would like to identify their app data by
Pubkey generally refers to a user's SkyDB public key, never an application key
+-----------------------+
| |
| Datalog Query Engine |
| |
+-----------------------+ -
+
+------------------+
| MemoryStorage |
+------------------+
+----------------------+
| |
| SkyDB Storage |
| |
+----------------------+
Here the datalog query engine follows closely with datafrog model from the rust borrow checker to deep dive see here. Long and short it works, by linking together facts called asserts in our lib, and does a few things to speed that up.
Here Memory storage is used as an alternative store and a cache for SkyDB. Before things are written to skydb they are first written to the cache. this allows us to have smart filters on what we store to the DB.
The SkyDBStorage is where we construct the local Storage and have the ability to play actions from remote stores. This allows us to integrate changes periodically from other SkyDBs. The cadence is actually set by the application, with their ability to call refresh, which in turn allows the DB. to cycle through it's index of foreign SkyDB application entries.
Contains the code to define actions for datalog and the underlying db. These are the fundamental operations that compose views of the database.
- assert: Asserts a fact, adding data to a table
- retract: Retracts a fact, canceling data from the table
- newTable: Creates a new set of Facts unified by a schema
{
export interface Action {
type: ActionType // "assert", "retract", "newTable"
datum: any, // Any object data you'd like to store
schema: any, // Table Schema object
tableName: string, // a string name/label for the table in use
__origin: string. // origin of the action an identifier tied to public key
}Contains the code that encapsulates the database, it is the interface that wraps the datalog query engine to make it useful for SkyDB,
const db = new Cirrus(new SkyDBStorage(...))
db.newTable...Contains the serialization code for Actions and schemas, which are written to the DB as well, which in theory would allow for dynamic creation of typed data.
Contains the code that defines what makes a data store it contains, the specific logic for writing to SkyDB. It also provides the ability for users to create their own DB extensions, which allows for application specific customizations to the Storage layer.
See the open issues for a list of proposed features (and known issues).
How to contribute, build and release are outlined in CONTRIBUTING.md, BUILDING.md and RELEASING.md respectively. Commits in this repository follow the CONVENTIONAL_COMMITS.md specification.
Apache License 2.0
Experimental software and currently/ temporarily restricted to web use.
Datalog Primer super Quick StrangeLoop about Streaming Architectures Datalog definition Datascript a datalog impl with ClojureScript Datafrog in Rust used in borrow checker (Domain Modeling Entity Attribute Value)[https://www.youtube.com/watch?v=oo-7mN9WXTw]
Shoutout to and Special thanks to Datalog in TS
Because of this excellent work I was able to hook in and make a few minor tweaks to be able to have a type safe query engine for Cirrus