Comments on: Multiuser Synchronized Distributed Database
As the GUI moves ahead, there's some talk around about how to do a lean synchronized distributed database for REBOL apps.
You already have one, you just don't know it yet. R3 Chat system (DevBase-2) is a third generation implementation of such a synchronized message-based database. It's a full redesign of what I implemented for AltME, which as a full redesign of what we implemented for REBOL IOS. I also learned some things from DevBase-1. So, that's about a ten year concept development span (but goes back even further if you consider my message-based systems developments began in the 1980s, with RPC prototypes, and implementations like the Amiga MT Exec, which even predates IBM MQ, 1992.)
The newest message-sync implementation solves many problems that earlier systems did not. For example, it solves the message deletion problem (which is more than just technical in nature), groups, threads, message movement between groups, file tagging, file sync, file renaming, file deletion, and much more. It's an excellent design, and one I've simplified to a tiny kernel of less than 10 KB (server and client.)
In addition, the system is "virtualized" -- in the VM sense of the word that messages can be fetched-on-demand -- reducing download overhead and client startup overhead if so desired. The approach also focuses on minimizing server overhead and loading, maximizing throughput, and efficient distributed backup. The same technique provides "user views" of shared data, where user-private data is only shared with those who are allowed to see it. User-selective snapshots.
It can be used for a lot more than just messages too. It's easily extended to contact database, calendars, file sharing, revision control, sales records, client info, and much more.
Anyway, I just thought I'd put this out there to see if there's any interest. I've not promoted it much, because I figured everyone these days was an SQL dead-head. The approach I use is much leaner, faster, and maintainable. But, it's not SQL, nor do I want it to be.
23 Comments Comments:
Kaj
14-Nov-2010 9:59:29 |
You're just in time for the NoSQL movement, which is finally catching on.
http://en.wikipedia.org/wiki/No_sql | Brian Hawley
14-Nov-2010 10:07:06 |
I am not the one to normally make this argument, but in this case I'll make an exception: Without the source, we don't really "have" it. DevBase 2 (I thought it was 3) is a great platform. If you ignore the UI, it is one of the best collaborative development platforms I've ever used.
But we can't really get the potential benefits of the database platform unless we can also adapt it to other uses. For that we would need either the source, documentation of the algorithms involved (it is only 10k of R3 code, we can replicate that from docs), or a really good reverse engineering of the system by someone who was willing to release their findings. Preferably one of the first two.
We know we can get the client code because we can run it, but without the server code it is incomplete. And we can't just use your server - it's already running a service with a specific use, not a general-purpose data service. | Carl Sassenrath
14-Nov-2010 10:18:42 |
Kaj, interesting, thanks for noting it.
Brian, once we have github/rebol established, I'll move some other sources to it, including this S-DB and WIP3 too, etc.
| ingo
14-Nov-2010 13:14:39 |
Yes, there is interest.
(At least I am interested.) | Me
14-Nov-2010 13:51:31 |
fileshareing? So you could use this as an update-tool to solve [0497] ? :) | Luis.
15-Nov-2010 2:52:30 |
(at)Me: That's what I thought as soon as I read through the initial post.
Brings to mind 'Can't see the wood for the trees'... :)
Cheers,
Luis.
| Kaj
15-Nov-2010 4:06:08 |
That's not the point - it's a different forest. The issue in article 0497 is that Cheyenne is stored in Subversion, so there's no way around Subversion if you want regular updates. | DocKimbel
15-Nov-2010 7:05:11 |
Cheyenne SVN revisions tarballs are now available in TAR and ZIP formats at :
http://cheyenne-server.org/download.shtml
The update process is now fully automated and files will be available one hour (or less) after last commit on SVN repository.
I've tried also to upload them on Google Code directly, but the Google API for uploading files seems broken since at least a month.
| Maarten Koopmans
15-Nov-2010 7:36:47 |
+1 for release and putting it in the NoSQL camp. How well does it scale, and can we also use it as middleware other than user based chat? | sqlab
15-Nov-2010 9:34:21 |
I am interested in knowing more about it. But foremost I wonder about the reliability. | Gregg Irwin
15-Nov-2010 10:05:33 |
Of course we want this, but we need more info. It sounds like that's coming, which is great. I'm not sure if it should be called a distributed database though, since it imposes a data model (unless it's truly a generic db engine we can extend without hacking code).
Like Maarten, I want to know how it scales, and how to use it in a distributed fashion (i.e. virtual synchrony). Like sqlab, reliability is key for me. | Luis.
16-Nov-2010 2:14:12 |
(at)Kaj: Yes, but I don't think you're looking at the same one I am.. :)
This would be purely to keep the update process simple by keeping a database of different versions (not 'diff' in the usual sense of SVN). If, as Article 0497 points out, that a minimal update system is required, I can't think of anything smaller than a database with pointers to the different versions.
Cheers,
Luis.
| David den Haring
16-Nov-2010 6:37:15 |
Atronix Engineering is interested in evaluating it for use in ASCADA. | onetom
16-Nov-2010 11:33:48 |
(at)carl: of course we are interested in this messaging system just as the WIP wiki! theoretical description would be awesome!
we would even help writing documentation too as we are discovering these gems.
(at)brian: well, i would say the chat source is open: http://www.rebol.com/r3/chat.r (taken from "source chat")
http://picolisp.com has an awesome distributed graph database with an sql-like structure definition and a prolog implementation as a query language.
persistent objects are 1st class in the language. they appear as normal symbols actually. "like words in rebol".
and it's also quite small, has a web server, web app framework, etc...
i wish we could have a fusion of the 2 languages ;)
| Brian Hawley
16-Nov-2010 12:21:18 |
Onetom, that is just the client. We need the server too - it isn't fully distributed, at least in the chat implementation. Carl, I look forward to it! | Greg
17-Nov-2010 1:51:57 |
Sounds very interesting.
Maybe some code samples would help understanding how it works. | Multicast tunnel
19-Nov-2010 19:13:27 |
and yet 26 years plus, it seems there's still No interest in including a simple "Multicast tunnel" (to simply get over the worlds ISP blocking this long standing protocol over the web)to put all these message threads and binary data into them....
or seeing as its now 2010/11, extending that to even simpler fast end to end encrypted DHT/torrent rootless services Inside these tunnels so again these world ISP's cant easily Deed packet intercept your darknet commercially and private traffic and make it slow or readable for their commercial profit.
the concept of OpenDHT (distributed hash table) multicast and torrent are well understood subjects and there exists ready made example muticast tuneled DHT in bamboo java code for a long time now
http://bamboo-dht.org/tutorial.html
the point is , a simple REBOL 'Multicast tunnel' framework that all these Multiuser Synchronized Distributed Database AND App's would use and become very popular are only held back by people not including them and writing the core backbone scripts to show them off in a simple way.
and that a shame, think about the long term bandwidth and speed potential saved , try and include this long standing 'Multicast tunnel' framework and make people happy, even thoughs that dont know they need it yet/ | jblake
25-Nov-2010 17:03:21 |
Isn't that an SOA?
Break apps down into little "Services" and re-use them?
| ScottT
8-Jan-2011 12:39:15 |
Carl:
I read all I could find on the associative data model when you first talked about it in relation to Chat. I was never really indoctrinated into the sql camp.
I would love to read a further explication of the generalized model of the system. Diagram.
I had hoped that this associative model would learn to grow into the source management, wiki, docs, blogs, script libraries, etc.
Seems since certain base data structures are the same, they should have a pretty sneaky interoperability betwixt all the above.
It's been some months since I obsessed over it until I thought I had it in my mind, I'd love another dose, maybe some fresh links. | buddyontobuddy
4-Jul-2012 5:24:40 |
I enjoyed reading your article and found it to be informative and to the point. Thank you for not rambling on and on just to fill the page.
article
| tempraments
9-Jul-2012 0:15:26 |
I am exploring this topic for a report I have to write. You have good content and I would like to know if you have any other posts about this?
bmo on line banking
| bhenmtvnet
9-Jul-2012 1:31:10 |
This was an excellent article. It has some valuable content on this topic. Thank you for compiling it into an easy to read and well written post.
ipad 4 release
| jojosipi
15-Jul-2012 21:16:08 |
This post has helped me to have another perspective. I am researching this topic for a paper I am writing. Your article provided me great insight of my topic.
Yoga Accessories
|
Post a Comment:
You can post a comment here. Keep it on-topic.
|
