Comments on: Experimenting With FULLTEXT Indexing

By: Dave Rathbun

Dave Rathbun — Thu, 17 Sep 2009 04:35:38 +0000

I’m not saying what I want to very well it seems.

I know that when I search by text fields it doesn’t pay attention to dates. What I had hoped, or rather what I had expected, was that the most “relevant” posts would be the most recent because the discussions are better. Instead when I search for words X, Y, and Z often times I come up with a topic with a single post from four years ago. It’s not very helpful, and I know there are other topics (posts) that in my mind would match better.

It’s an issue of the relevance engine, not really the dates. I wasn’t clear about that before.

By: Dog Cow

Dog Cow — Wed, 16 Sep 2009 21:34:56 +0000

MySQL FULLTEXT doesn’t know anything about the dates of posts. Therefore, you need to use an AND clause with post_time column

By: Dave Rathbun

Dave Rathbun — Wed, 16 Sep 2009 06:16:21 +0000

Use an AND clause … to do what? Limit the search to more recent posts?

What I had expected (hoped) was the relevance engine would do that job for me. I know my board; I know the content very well. Okay, so I don’t have all 500,000 posts memorized, but I do know what I expect to see. So far, I’m not getting it.

More experimentation is required.

By: Dog Cow

Dog Cow — Mon, 14 Sep 2009 23:54:25 +0000

“They keep pulling up posts from three or four years back.”
You can use an AND clause, you know.

By: Dave Rathbun

Dave Rathbun — Mon, 14 Sep 2009 23:10:54 +0000

I have been testing with IN BOOLEAN MODE and specifying various different search terms that I am very familiar with. They keep pulling up posts from three or four years back. If I go this route, I believe I am going to have to come up with some sort of weighting so that more recent posts carry more weight somehow.

By: Dog Cow

Dog Cow — Mon, 14 Sep 2009 18:16:47 +0000

Are you using the right search type? There’s boolean, and then there’s some other types as well. You need > 5.0 for the best searching features.

By: Dave Rathbun

Dave Rathbun — Mon, 14 Sep 2009 15:44:47 +0000

So far I have been underwhelmed by the accuracy of the search results. When ordered by “relevance” I get lots of unrelevant posts. Or I get posts that I know are not even close to the best answer to the question.

I have been experimenting with running search queries directly on the database. Eventually I will alter search.php to provide the interface required to let the users do the same sort of search.

By: Dog Cow

Dog Cow — Sun, 13 Sep 2009 19:06:59 +0000

Good to see you’re making the right decision. Yeah, FULLTEXT seems like it’s got some draw-backs (one of them being index size roughly 3/4 of data size), but really, when you weigh the pros versus cons, it’s truly the way to go when you want a lot of flexibility in the search. MySQL has come a long way since a lot of the discussions on it at phpbb.com back in 2003-04.