Things have changed a bit since 1999

I am Chris’s pager. You know me as 555.818.1234 . March 9th. 1999 was the most traumatic day of my being. I had faithfully woken him up at 5am, had the screen window cleaned, and been put in my place of honor at his left hip. We left home to go to the airport for a day trip to Houston. Trips to Houston are always exciting because I get to show my wares many times in a day.


Flying however is not my favorite activity because he always stuffs me, along with my great rival the cell phone, into a dark, black bag before going through security. He explained this once, but my battery was low, so I switched off.

On this morning, he stuffed me into the bag, and unceremoniously dumped me into the overhead bin of a Southwest Airlines plane to Houston. During the flight, I was paged, and managed through vibrating hard, to escape into the bin. The bin was clean, and uncluttered – and best of all, the cell phone was still in the bag. The cell phone was less trouble than normal, it had been switched off for the flight.

The flight landed, and a somewhat groggy Chris pulled down the bag, and didn’t notice that I had escaped. I vibrated at 60-second intervals to try to alert him, but to no avail. He ignored me.

So there I was. Abandoned in a strange city. In a strange compartment thinking I would never find him again. Every time I was called, I hoped it was he, but no he seemed to have forgotten all about me.

I can only imagine how frantic he was without me. I kept getting called – almost 20 times in the day, and knew I was important. The cell phone had pride of place that day, though. I hoped that Chris wasn’t going to think that I was dispensable, and rely totally on the cell phone in the future.

Meanwhile, the plane was cleaned and took off on an odyssey across country. We even left Texas and went to Rhode Island. At the Providence airport, a cleaning crew member found me, and handed me to lost and found. I heard those clever people at South West airlines wondering how to reunite me with Chris.

One of them turned me over and deciphered the number tattooed on my back. She called it, and was given Chris’s name. They looked in some thing they called the reservation system, and found his reservation with our phone number in it. Luckily it was our home and not that cell phone’s number. Several people tried calling but there was no one home to answer. My worst fears were realized. I had been tossed aside and replaced. And then late that night, the phone rang. I heard the familiar voice. Chris called! He was worried about me after all. He wanted me home. He authorized the use of his credit card to have me shipped. He left a long message warning that if the Southwest people were to try to call him at work, I would vibrate.

I passed a more peaceful night than I expected. Still I wakened with a ringing in myself at 5:00am. It was no help though. I was still up North where it was cold out. I spent a fretful day, and eventually found myself on another of those beautiful Southwest Airlines planes going home to Dallas. Every time the plane stopped, I hoped that I had arrived, but no. In fact it was worse, I was going to Houston. I didn’t know the plan had changed and that Chris was going back to Houston that night, and would rescue me.

I sat quietly in my box, not knowing when he would pick me up. And then, I heard his voice. He was coming to get me. I had not been abandoned after all. We had a happy reunion. He illuminated my display to check the pages. I felt needed again. He put me back in my holster, we were a team once more.

As we left Houston the following day, the cell phone’s battery ran out. I was again the only way to reach him.

Grumpy Old Man Plays the Role of Query Optimizer

Another in the continuing saga of MongoDB. As you maye gleaned from other posts, I am yet to become an all in fan of the product. I have some appreciation of its capabilities, but still am finding it syntactically and semantically hard going.
The class from the 10gen team is very well done. a few glitches, but nothing major. I have been on expensive classes where the material has been of lesser quality.
So kudos to 10gen - especially Andrew.
The strange bits. And this week there are 2. First off there is thing called the "aggregation pipeline". Cool concept, you specify as steps in a pipeline what to do with the data, so you can do things like group, sort, and generally report usefully on the data contents. Cleverly (of course), the results from each stage of an execution pipeline are mongo (JSON) documents. So you can operate on them just like any others. Nice.
But, in order to do quite complicated queries, you have to specify every step of the pipeline yourself. And you have to figure out the order you perform them in(to get the right results and to optimize the performance). If I sort before I filter, then I am probably doing too much work, for example. In SQL databases - at least the better versions, the query optimizer is supposed to figure this out for you. So, niow I am having to be my own query optimizer. Not happy about that. Yeah there are reasons, sharding might be tricky to optimize (I don't know).
The second - and this is quite uncomfortable feling, but I dare say I will get used to it is referencing. This is a bit tricky so I will illustrate (I hope correctly!)
if I want to group by the value of category, where category is a key in a document,
I would have to right something like {"group":{"_id":"$category"....}}

I read this as make the _id field the value obtained from the category key. Each one in quotes (programming in strings again, uh), and then the $ to dereference the name of the category key so I can use its value. That's an awful lot of symbology to remember.

Grumpiness quotient has gone up this week!

Grumpy Old Man and MongoDB - Indexes and things

This week (week 4 in the excellent 10gen class on MongoDB) has us looking at things like indexes, profiling, etc.
I am getting used to the syntax (but still dislike the "programming in quotes" model and the use of cryptic special values for specifying sort sequence, etc.).
Lovely looking feature for geospatial indexes, but quite tricky to use. At the base level, the distance measures on the spherical model are expressed in radians. So we have to do that conversion somewhere. PITA so far. I can see why, but that isn't exactly habdy. Would like (and will build) some other mechanisms to sort that out.
If for no other reason, the radians based model doesn't distinguish well between directionality. maybe I want coffee shops within 10 miles North of me (because I am heading that direction, none south of me and maybe within 1 mile each side of the route). I am sure I could code that!
And then for some reason, the MongoDB shell treats using the geospatial spherical model differently from other models. It is invoked through the db.runcommand(...) syntax and not the usual db.dbname.find(...) syntax.
Also since you don't specify which index to use in the db.runcommand(..) syntax, and if you happen to have 2 2d indexes defined it fails. Promising feauture, but could use work.
Utilities are handy - Mongotop and Mongostat are helpful indeed
In many ways MongoDB reminds me of the 1970s system ADABAS, but with updated syntax.
 

Grumpy old man and MongoDB - Transactions

I am beyond scared by the possibility of using MongoDB for any kind of meaningful transactional system.

We always have a balance between "getting stuff through the system" and "getting sufficient accuracy". Sufficient here is really key. ACID properties are vital to ensure that we don't see incomplete transactions WHEN THE KINDS OF TRANSACTIONS WE ARE PROCESSING MUST NOT BE SEEN UNTIL COMPLETE. (caps deliberate).

The "classic" example is the movement of money from one account to another. While the money is being moved, decisions based on the value of either account will be flawed. The "from account" will have a smaller balance than we think, and the "to account" a larger one. So we should probably wait until the transfer transaction has completed before allowing any process to make decisons based on the balance in either account.

In the MongoDB world the update to each account is itself atomic, but there appears to be no overarching transaction context. So it is possible (not very probable) for the document that represents the "from" account to show that the amount has been debited, but that the "to account" has yet to be credited. Assuming that the system does debits before credits. It, of course, doesn't have to, although I think it would be foolish not to.

The designers of the major database management systems (relational or not) have thought carefully through those kinds of implications. They have made sure that records are somehow locked to prevent this kind of behavior. They ensure that updates on both the "from" and "to" sides of the transaction are both handled - or neither is.

Do I really trust a developer with the kind of skills I have to get this right in every case if I get no help from the underlying data management system? I don't think so. I would much rather see the transactional systems using transactional databases. And use these powerful engines (like MongoDB) for situations where I don't have to rely on transactional behaviors.

Now the actual number of cases where transactional behavior of this nature is actually required may be smaller than we think. Often times we see a small transactional component (moving the money) not tied to the delivery of the goods. See this excellent post from Gregor Hohpe.

Grumpy old man and MongoDB - Database Design

It is week three in the MongoDB class put on by 10gen. The instructors have done a great job. The material flows well and is presented nicely. So kudos to the guys.

One of the privileges of being old and grumpy is that you learn that there are no mysteries in system design. However, there are new paradigms sometimes. We have that in the MongoDB world and there are many cases where it can make a big difference. Essentially I am now beginning to think of MongoDB as "relational database with embedded arrays". I don't know for sure (I haven't done the math and nor am I likely to), that MongoDB will support the Relational Calculus. It should (probably, but again, I have not done the math!) support SQL pretty well. Especially a very vanilla form that doesn't use constraints, etc. I am not sure of the value of the DDL aspects of SQL, although I guess one could do that. Much more important would be the layering oof SQL for data manipulation.
Even expressing a join would be fine - and if the data were embedded more power to it. SQL as data access layer vs SQL all the way through the storage subsystem.
There are some semantics changes of course - because of the lack of a real "key" in an embedded document, some of the join-like processing will potentially be a bit odd. Essentially we have to treat the values in an embedded document as we would in a materialized view.
SQ Update and Delete operations are less likely to behave as they do in an RDB. The implications of deletion on embedded documents are subtle. However I can see some great opportunities for some stereotypes here.

This post by Bill Kent is one of the all time great articles on thinking about choices in representation of a simple 'fact', The paper was written in 1988.

As a long time teacher of data modeling (my classes pre-date relational databases!), I have come to a couple of realizations:

• The approach that I take to logical (E/R, not expressed as tables) modeling won't change with MongoDB
• There should be some pretty simple guidelines for converting an E/R model to a MongoDB implementation
• The best looking uses for MongoDB are where something else has already done the validation and linking - insertion into MongoDB becomes an organizational exercise.
• MongoDB gives some flexibility in order of insertion even when things are linked. So some of the convoluted exercises we have done when creating systems of references in conventional relational databases may go away.
• The modeling tools (like Embarcadero and ER/WIN) are less help than they used to be - except maybe as pure diagramming tools. This one I am less sure of, since all I have ever seen from these tools is modeling as a relational exercise. If there are other ways possible, I haven't really seen them.

I am looking forward to week 4.

syntactic sucralose

In programming languages there is a concept "syntactic sugar". As wikipedia describes it In computer science, syntactic sugar is syntax within a programming language that is designed to make things easier to read or to express.
In some languages (especially the MongoDB shell), there is the reverse concept. There are language features that are present to "make it work" but have no bearing on anything in the program's context. I call these syntactic sucralose. They are the only things available to get the desired result, but leave a bitter taste in your mouth afterwards.
The case that riled me up today was the $unset "operator" in Mongodb's shell interface. To unset (eliminate a name value pair in a MongoDB document), you write something of the following form for the second argument of the .update method.
{$unset :{foo : 1}}. The 1 in this case is a mandatory positional parameter that has no relation to the current value of foo. In fact you could put anything that is a legitimate value (string, date, objectID, integer, boolean...) in place of the 1.  In fact whatever is placed there is evaluated.  So for example, the code fragment

x=0
,{$unset :{foo,x++}} does result in both foo becoming unset and x being incremented.

Even if the value is an unbound variable name, it still is acceptable.

Lots of scope for mischief. This should come with a government health warning

MongoDB and the Relational Car

Sometimes you want your data all nicely normalized, and sometimes you don't. This has come into sharp relief as I go through MongoDB training this week. By way of background, I have a fair amount of experience with many types of databases, data modeling and data thinking in general, so it is interesting and fun to learn about new ways of thinking.
But forst a story. In the dark ages (maybe 1974 or 1975), I wondered about mailing lists. So much so that I devised a way of tracking some of the uses of data among companies - especially early markets in buying and selling of information. I would sign up for a magazine using some unique variant of my names. Keep track of which variant I used for which magazine, and then see what solicitations I would get through the mail using that name variant. Most instructive. American Express sent mail to the largest number of variants.
I do the same things to this day - making up email names addresses just for long enough to validate that I want the service I have signed up for, and then wait and see what else arrives. Of course everything that arrives is by definition spam. But I digress.
I also thought about the "relational car" i.e. what would the world be like if I normalized my vehicles. Kept the wheels with the wheels, the engines with the engines,... You get the idea. I think it is likely that I would be late for work every day. First join all the piece parts together to make a suitable version (assuming that the children hadn't emptied the fuel tank the night before, thus putting the equivalent of a lock on the tank). Then drive off. After coming home, put the updated parts back. Updated???? Yes, because the tires are now more worn...
Clearly from the primary use of having a vehicle as transport, the relational car is far from ideal. I am much better off with the assembled car.
That's kind of how I think in Mongo. Often times the data are much more useful when put together by primary usage than when all normalized and accessed with joins. But, of course, not all the time.
That led me onto thinking about the quality of "relationships" (among the data entities). Many others, in more learned writings than mine, have fussed about different qualities in relationships. Composition is different from association, etc. So a Purchase Order might be composed of Many (at least 1) Purchase Order Line Items, so it seems reasonable to think of these line items inherently bound up with the POs. So the document oriented approach looks pretty good. But, things are less rosy, when I think about the association between the Product and the PO. There it is probably unreasonable to bury the product data inside a PO document. And there are several different kinds of associations we might want to consider.
So, when learning the document oriented DBMS (MongoDB), I am finding myself revisiting types of relationships and whether the distinctions are important. For me I have come down on the side of, I probably care in my master systems, those systems of record that actually run the business. But in those which are simply systems of reference, maybe it makes a whole lot of sense not to worry about the normalization, required schemas and other aspects that make the document oriented databases so interesting.
In the relational model the foreign key is the only relationship condtruct available. Even the cretion of "link" relations relies on the Forign Key. That doesn't seem to me to be a powerful enough construct to express the nuances of the kind of relationship and thus its associated semantics.
Oh, and circling around to something interesting about composition types of relationships, we do have some interesting delete anomalies. If we say that an A is composed of 1 or more Bs, what do we do when attempting to remove the last remaining B. That should somehow kill off the A, of course - or we expressed the rule incorrectly.