Comments on: Storing Post Revisions / Post Locking

By: Dog Cow

Dog Cow — Wed, 18 Nov 2009 19:46:54 +0000

Fair enough.

By: Dave Rathbun

Dave Rathbun — Wed, 18 Nov 2009 02:52:32 +0000

Because it’s convenient.

This is all a big experiment, and I may come around to your way of thinking eventually, but right now it makes it very simple to deal with.

By: Dog Cow

Dog Cow — Tue, 17 Nov 2009 16:24:19 +0000

Why have that data stored there if it is so infrequently accessed?

Since you could be storing elsewhere, that is….

By: Dave Rathbun

Dave Rathbun — Tue, 17 Nov 2009 01:10:10 +0000

I read it again and I’m not sure what the “second question” is, can you humor me and clarify what it is?

By: Dog Cow

Dog Cow — Wed, 04 Nov 2009 17:16:33 +0000

Ok, so what about my second question?

By: Dave Rathbun

Dave Rathbun — Wed, 04 Nov 2009 02:55:51 +0000

Bigger table != performance problem With a unique key of post_id + post_version only one row (or one block, depending on how the database works) is retrieved. It doesn’t matter if it’s 100 rows or 100M rows, a unique key read should be nearly instant. Yes, a unique key index on a larger table is going to be larger than a unique key index on a smaller table, but it’s not a linear performance hit. When I query a single post using the unique key on my post table with over 580,000 posts the query time is not measurable. According to MySQL, everything comes back in 0.00 seconds. The query plan (from the explain command) shows that the primary key index will be used, and because of the where clause exactly one row will be returned. That’s about as fast as it gets.

By: Dog Cow

Dog Cow — Tue, 03 Nov 2009 16:44:31 +0000

There is certainly a performance hit every time the table grows. Why have that data stored there if it is so infrequently accessed?

I must admit, though, that I do like how every post action becomes an INSERT. Nice job on that one.

By: Dave Rathbun

Dave Rathbun — Tue, 03 Nov 2009 14:48:51 +0000

I had not read the topic, thanks for the link. As mentioned in the topic, performance is one concern. The way I have implemented it has no negative impact on performance and in fact may have a slight benefit. Inserts can be more efficient than updates since no locking is done. Since every post “edit” is in fact converted to an insert there might be a benefit there. Note I have not done any benchmarking to determine if this is the case.

However, I don’t like the idea of storing the data in a separate table. It’s different versions of the post text only; the post itself is not changed. Why not store it – in versioned form – in the posts text table? Since there is no performance impact of storing versions and there are certainly advantages (less code to change, easier to retrieve and display the archived versions) I think it’s a good choice.

Time will tell; I plan to roll out the collection portion of the code this week. Giving the moderators the ability to review / revert code is coming later.

By: Dog Cow

Dog Cow — Mon, 02 Nov 2009 19:29:45 +0000

Secondly, I don’t agree with how you have designed the versioning system. I think that upon every edit, you should be inserting a new row of data into a post_version table. This table would be denormalized and the data stored compressed as mentioned earlier. Some columns would be post_id, post_version, user_id, topic_id, post_date, post_text, etc…

Then you could make the primary key on post_id, post_version and query this table separately. Therefore, the posts table and posts_text table would store _only_ the most current post revision, and nothing else. You also would not need the additional field.

By: Dog Cow

Dog Cow — Mon, 02 Nov 2009 19:26:19 +0000

I posted a topic to phpBB.com some months ago on this very subject, which you may have read: http://www.phpbb.com/community/viewtopic.php?f=64&t=1573125

One of the problems is that current RDBM systems (that I know of) don’t have a third dimension of time. They are just row/column-oriented. Thus, you’d have to use a JOIN to another table. Having to deal with a JOIN just introduces another facet of complexity due to the inherent problem that a JOINing two or more large tables presents. Quite frankly, when one is dealing with millions of rows and gigabytes of data, the only solution then becomes to remove all joins and query tables as key-value pairs, otherwise the memory and CPU involved becomes prohibitively expensive.

Now, one of the bonuses we have on our side is that of frequency of reference. It just like having a collection of volumes in a library. One uses compact shelving to save space because these historical volumes are infrequently accessed. Likewise, we can store all versioned posts as compressed binary strings in a separate table, or even on a separate database server.