Comments on: How does search work? Part IV: Evolution of a Regular Expression

By: Dave Rathbun

Dave Rathbun — Tue, 06 Nov 2007 04:49:14 +0000

Hi, Scott, thanks for your comments. In the phpBB search process special characters like the ‘ in you’re are removed prior to the regex, leaving the word youre instead. So they have that part covered. I’ve been using the code that I posted earlier on an english language board without issues, but it does cause problems on non-english boards. I might still have some data that I was loaned for testing that I can use… when I get time I will try your suggestion. Thanks!

PS – Hope you don’t mind but I edited your comment to include the “pre” tag so that your extra space will show up as you wanted. I moved one of your inline comments to a separate line to keep the screen from scrolling horizontally.

By: Scott

Scott — Mon, 05 Nov 2007 17:51:53 +0000

Well actually the following method will work to remove any one or two character words from a string without using the \b tags which break words like "don't" and "you're":

// replaces each space with two spaces, yes there are two spaces in the second argument..
$string = preg_replace( '/[ ]/', '  ', $string );
$string = preg_replace( '/\s\S{1,2}\s/', ' ', $string );  // kill one and two character words
$string = preg_replace( '/\s+/', ' ', $string ) // replaces multiple spaces with a single space

Not the prettiest solution ever, but it is working for me.

By: Scott

Scott — Mon, 05 Nov 2007 17:41:43 +0000

I just built a very similar regex for a similar purpose this morning (to remove one and two letter terms from a search).

It is worth mentioning that the \b boundary notation will break words at any character other than a-z0-9, so the regex shown above will turn the word: “you’re” into “you” because it will see “you’re” as two words: “you” and “re” and the “re” is only 2 characters so it is dropped.

I have not yet found a nice way to get around this.

By: dave.rathbun

dave.rathbun — Mon, 12 Feb 2007 15:26:31 +0000

Update posted here. The news is not good; foreign language posts are not being properly processed by this expression.

By: dave.rathbun

dave.rathbun — Sat, 10 Feb 2007 01:25:44 +0000

I have tested the regex presented in this post quite thoroughly on an english language board with no issues. I have had less success with some foreign language boards. I will post some further results when I have something more concrete to share.