How to work around the lack of transactions in MongoDB?

I know there are similar topics on the site but they are either telling me to switch back to regular RDBMS systems if i need transactions or use atomic operations or two-phase commit. The second solution seems the best choice. The third is a way i don’t wish to follow because it seems that many things could go wrong and i can’t test it in every aspect. I’m having hard times to redefine my project to meet the second method. I don’t know if this cames from the viewpoint i have because i’ve only worked with SQL databases so far or it really can’t be done.

We like to pilot test MongoDB at our company. We have chosen a relatively simple project which is an SMS gateway. It allows our softwares to send SMS messages to the cellular network and the gateway does the dirty work: actually communicating with the providers which can have different communication protocols. The gateway also manages the billing of the messages. Every customer who applies for the service have to buy some credits. The system automatically decreases the user’s balance when a message is sent and denies the access if the balance is insufficient. Also because we are customers of third party SMS providers we may also have our own balance by them. We have to keep track of those as well.

I started thinking about how can I store the required data with MongoDB if i cut down some complexity (external billing, queued SMS sending). If i came from SQL world i would create a separate table for the users, another one for SMS messages, and one for storing the transactions regarding the users balance. Let’s say I create separate collections for all of those in MongoDB.

Imagine an SMS sending task with the following steps in the simplified system:

  1. check if the user has sufficient balance; deny access if there’s not enough credits available

  2. send and store the message in the SMS collection with the details and cost (in the live system message would have a status attribute and a task would pick up it for delivery and set the price of the SMS according to it’s current state)

  3. reduce the users’s balance with the cost of the sent message

  4. log the transaction in the transaction collection

Now what’s the problem with that? MongoDB can do atomic updates only on one document. In the previous flow it could happen that some kind of error creeps in and the message gets stored in the database but the user’s balance is not reduced and/or the transaction is not logged.

I came up with two ideas:

  • Create a single collection for the users, and store the balance as a field, user related transactions and messages as sub documents in the user’s document. Because we can update documents in atomic manner it actually solves the transaction problem. Disadvantages: if the user sends many SMS messages, the size of the document could become large and the 4MB document limit could be reached. Maybe i can create history documents in such scenarios, but i don’t think this would be a good idea. Also i don’t know how fast the system would be if i push more and more data to the same big document.

  • Create one collection for users, and one for transactions. There can be two kind of transactions: credit purchase with positive balance change and message send with negative balance change. Transaction may have a subdocument for example by message send the details of the SMS can be embedded in the transaction. Disadvantages: i don’t store the current user balance so i have to calculate it every time a user tries to send a message to tell if the message could go or not. I’m affraid this calculation can became slow as the number of stored transactions grows.

I’m a little bit confused which way to choose. Do i have other solutions? I don’t find any best practices online how can I work around this kinds of problems. I guess many of the programmers who are trying to become familiar in NoSQL world facing similar problems in the beginnings.

An Accounting RESTful API or, circumventing MongoDB’s lack of DECIMAL data type and support for TRANSACTIONS

I am evaluating MongoDB as a storage solution for our financial/accounting RESTful API. I know about MongoDB’s lack of transactions and decimal data type, however: I suppose I could overcome the lack

MongoDB transactions?

Playing around with MongoDB and NoRM in .NET. Thing that confused me – there are no transactions (can’t just tell MongoConnection.Begin/EndTransaction or something like that). I want to use Unit of wo

Work around for lack of sqlProjection support within criteria builder?

http://jira.grails.org/browse/GRAILS-2803 concerns lack of support for sqlProjection. It was closed as a duplicate and supposedly fixed in Grails v1.2. But the duplicate issue concerned sqlRestricti

Work around the lack of templates in java

In C++ I build a little wrapper around cout like this: template <class T> void output( T a ){ if(is_this_enabled==true) std::cout<<a<<endl; return; } When certain kinds of output fl

Transactions In MongoDB

I am using a NoSQL database MongoDB with Java and Spring Data. I am aware that MongoDB only supports transactions for a single document. I am using Spring Transactions to carry out MongoDB transcation

SQL Server: how transactions work

In SQL Server, how many transactions will this produce? DECLARE @deleted BIGINT SET @deleted = 100000 WHILE @deleted = 100000 BEGIN DELETE TOP(100000) FROM MYTABLE WITH (ROWLOCK) where Col1 = 7048 and

Transactions around finder methods

Is it wise to start transactions around finder methods: @Transactional public E getParticularEvent(final String id) { return (E)em.find(carClass, id); } Should this be done?

MySQL Transactions and how does Rollback work?

I just came across Transactions and I wonder what are the pros and cons of using it, how does the rollback work, are the old values kept in memory, if bigger, what happens? For 1. I understand you h

How to design around lack of multiple inheritance?

Using interfaces won’t work because I want a single implementation. Using this solution would end in a lot of redundant code because I plan on having quite a few sub classes (composition vs inheritanc

How to solve lack of space issue in mongodb

I am getting the following error in mongod.log file: journal dir=/var/lib/mongodb/journal Mon Jan 20 15:10:19.742 [initandlisten] recover : no journal files present, no recovery needed Mon Jan 20 15:1

Answers

The project is simple, but you have to support transactions for payment, which makes the whole thing difficult. So, for example, a complex portal system with hundreds of collections (forum, chat, ads, etc…) is in some respect simpler, because if you lose a forum or chat entry, nobody really cares. If you, on the otherhand, lose a payment transaction that’s a serious issue.

So, if you really want a pilot project for MongoDB, choose one which is simple in that respect.

Bring it to the point: if transactional integrity is a must then don’t use MongoDB but use only components in the system supporting transactions. It is extremely hard to build something on top of component in order to provide ACID-similar functionality for non-ACID compliant components. Depending on the individual usecases it may make sense to separate actions into transactional and non-transactional actions in some way…

Transactions are absent in MongoDB for valid reasons. This is one of those things that make MongoDB faster.

In your case, if transaction is a must, mongo seems not a good fit.

May be RDMBS + MongoDB, but that will add complexities and will make it harder to manage and support application.

Now what’s the problem with that? MongoDB can do atomic updates only on one document. In the previous flow it could happen that some kind of error creeps in and the message gets stored in the database but the user’s balance is not gets reduced and/or the transaction is not gets logged.

This is not really a problem. The error you mentioned is either a logical (bug) or IO error (network, disk failure). Such kind of error can leave both transactionless and transactional stores in non-consistent state. For example, if it has already sent SMS but while storing message error occurred – it can’t rollback SMS sending, which means it won’t be logged, user balance won’t be reduced etc.

The real problem here is the user can take advantage of race condition and send more messages than his balance allows. This also applies to RDBMS, unless you do SMS sending inside transaction with balance field locking (which would be a great bottleneck). As a possible solution for MongoDB would be using findAndModify first to reduce the balance and check it, if it’s negative disallow sending and refund the amount (atomic increment). If positive, continue sending and in case it fails refund the amount. The balance history collection can be also maintained to help fix/verify balance field.

Check this out, by Tokutek. They develop a plugin for Mongo that promises not only transactions but also a boosting in performance.

Living Without Transactions

Transactions support ACID properties but although there are no transactions in MongoDB, we do have atomic operations. Well, atomic operations means that when you work on a single document that that work will be completed before anyone else sees the document. They’ll see all the changes we made or none of them. And using atomic operations, you can often accomplish the same thing we would have accomplished using transactions in a relational database. And the reason is that, in a relational database, we need to make changes across multiple tables. Usually tables that need to be joined and so we want to do that all at once. And to do it, since there are multiple tables, we’ll have to begin a transaction and do all those updates and then end the transaction. But with MongoDB, we’re going to embed the data, since we’re going to pre-join it in documents and they’re these rich documents that have hierarchy. We can often accomplish the same thing. For instance, in the blog example, if we wanted to make sure that we updated a blog post atomically, we can do that because we can update the entire blog post at once. Where as if it were a bunch of relational tables, we’d probably have to open a transaction so that we can update the post collection and comments collection.

So what are our approaches that we can take in MongoDB to overcome a lack of transactions?

  • restructure – restructure the code, so that we’re working within a single document and taking advantage of the atomic operations that we offer within that document. And if we do that, then usually we’re all set.
  • implement in software – we can implement locking in software, by creating a critical section. We can build a test, test and set using find and modify. We can build semaphores, if needed. And in a way, that is the way the larger world works anyway. If we think about it, if one bank need to transfer money to another bank, they’re not living in the same relational system. And they each have their own relational databases often. And they’ve to be able to coordinate that operation even though we cannot begin transaction and end transaction across those database systems, only within one system within one bank. So there’s certainly ways in software to get around the problem.
  • tolerate – the final approach, which often works in modern web apps and other applications that take in a tremendous amount of data is to just tolerate a bit of inconsistency. An example would, if we’re talking about a friend feed in Facebook, it doesn’t matter if everybody sees your wall update simultaneously. If okey, if one person’s a few beats behind for a few seconds and they catch up. It often isn’t critical in a lot of system designs that everything be kept perfectly consistent and that everyone have a perfectly consistent and the same view of the database. So we could simply tolerate a little bit of inconsistency that’s somewhat temporary.

Update, findAndModify, $addToSet (within an update) & $push (within an update) operations operate atomically within a single document.