What i was asking for is how you maitain insertion to that table? I don’t believe that you update that table by hand. As it looks like very automated process which originaly saves data to .txt file

Indeed it is a manual process for now. I only update the contents once or twice a year and it was not worth the time to create an admin panel page to manage the table contents.

another thing (which is up in mind just now) is question how you reindexing your site so fast. Even with all tweaks (and many yours hint) my reindexing takes about 26hours (around 500.000 post now) and my server isn’t just old slow piece of furniture

I suspect part of that was related to your follow up comment where you suggested that your stopwords list was not being used. That could easily double the time taken to reindex the site because every single word (even common words like “the” and “and” and so on) would be included in the index.

after i’ve commented part with this if statement and left only part in else, search works smooth and fast!
dunno why this statement did not work because my $lang['ENCODING'] = “utf-8″ so strstr may find occurence and go to false in that statement

weird
so i’ll look today closely for all this-like statements!!!

unfortunately this is not really answer to my question. My question is about whole process how is stopwords made or maitained.
I’m pretty sure that i’ve never touched content of that file and allready have more than 1000 rows in that file. So it must be generated some way (i did not check how and where yet)
So as i posted before that information is somewhere from http://www.phpbb.com/community/viewtopic.php?f=18&t=135383 this discussion

As we can see that function maitain process which retrieve stop word from DB (which is much more faster than from file) and then do normal stuff
What i was asking for is how you maitain insertion to that table? I don’t believe that you update that table by hand. As it looks like very automated process which originaly saves data to .txt file

another thing (which is up in mind just now) is question how you reindexing your site so fast. Even with all tweaks (and many yours hint) my reindexing takes about 26hours (around 500.000 post now) and my server isn’t just old slow piece of furniture
Maybe is there way to reindex searchindex from shell or something similar which will not be involving apache… dunno

sorry for my english, it’s long time when i was using english on daily basis

i’m pretty interested in synonyms/stopwords. i saw few not complete attepts on How to optimize search on big forums (phpbb) but i think that it is not complete


if (trim($entry) !== '') //subject lines are blank sometimes so skip this if needed
{
//get array of words
$word_array = split_words($entry);

if ( !($result = $db->sql_query($sql)) )
{
message_die(GENERAL_ERROR, 'Could not select stopwords', '', __LINE__, __FILE__, $sql);
}
while ( $row = $db->sql_fetchrow($result) )
{
$stopword = trim($row['word_text']);
if ( $mode == 'post' || ( $stopword != 'not' && $stopword != 'and' && $stopword != 'or' ) )
{
$entry = str_replace(' ' . strtolower($stopword) . ' ', ' ', $entry);
}
}
$db->sql_freeresult($result)
}
}


this is pretty good start, but this “mod” won’t solve INSERT into stopwords … which i can’t really figure out how this is realized

What a post. I’ll touch on most (but not all) of the points you brought up…

If you decided to share some of your data with me for testing, all I would really need is the content from the topics, posts, and posts_text tables. I would not need any user data. In fact, what I would do (and what I did last time I got some external data for testing) is assign every single post and topic to the same user. That helps some with privacy concerns, as it addresses the user identities. It does not address the privacy concerns about what the text that was actually posted. I can’t really address that, other than to take every possible precaution that the data is used only for my testing, and deleted immediately afterwards.

You could even extract topic and post data from one visible forum and dump that. I’ve done that before using a Select… Into command. If you’re not familiar with that syntax it’s a great way to get a reduced set of data. Suppose that you wanted to get a list of topics from forum 42 into a new table:

create table forum42_topics as
select * from phpbb_topics
where forum_id = 42

After that command completes you will have a table with the identical structure as that used for phpbb_topics (minus any indexes) and content based on topics from the selected forum. If you want to create an empty table with the same structure simply supply a where clause that will never be true, as in:

create table forum42_topics as
select * from phpbb_topics
where forum_id = -99

The tricky part is capturing the phpbb_posts_text table since the forum_id does not appear, but you can do that with a simple join:

create table forum42_posts_text as
select pt.post_id as post_id
, bbcode_uid
, post_subject
, post_text
from phpbb_posts_text pt
, phppbb_posts p
where pt.post_id = p.post_id
and p.forum_id = 42

You can’t get away with “select *” in this case as you’re using a join of two tables, and you don’t want everything from the phpbb_posts table to come along too. You can replicate this type of query for many purposes.

It doesn’t matter at the moment, as it’s going to be a few months before I get back to testing anyway. And if your board is english then I don’t really need to get your data, as I have plenty of english posts to use.

Now to your rant… first, I’m glad you felt better afterwards.

I will admit that I see more than one side of the story. I can agree that phpBB2 is a solid product, and I believe it has life moving forward even after the release of phpBB3. I will also agree that from what you can see from the front area51 does not appear to have dramatically different features. Yes, attachments are built in. Yes, subforums are now a core feature instead of a cumbersome add-on. But if you only look from the front you’re not seeing the bulk of the upgrades. Like an iceberg, you have to look under the surface to see the bigger part of the picture.

SQL handling is different. Language handling is different. Moderations tasks are different. The admin control panel is in a different galaxy, it’s so different from what we have today. Security has been rebuilt from the ground up. The fact that phpBB3 looks and acts so much like phpBB2 is – in my opinion – simply because it’s a forum. There are only so many ways that you can process forums, topics, and posts.

I really don’t want to comment on why phpBB3 has taken so long or the personnel turnover or other things that you touched on. I have an opinion, but I prefer to keep it to myself.

Regarding the MOD process… I have submitted a few MODs, and most of them have been approved. I have chronicled the story of my Page Permissions MOD in quite a few blog posts. It went through numerous submissions (and rejections). The first time it was rejected I got what I considered to be some very inappropriate comments from the MOD validator. Frankly, it pissed me off. I had a nasty response all written out and ready to send when discretion took over and I cancelled the message.

I came back a few days later and composed my reply instead. There were, in fact, legitimate concerns raised by the MOD validator and I had no problems with that. In fact, I learned from the experience, which is something that I value. What I sent to the MOD Validator was a brief comment explaining that while I valued the content of their review, the mechanism and style used to deliver their message was way out of line. Technical content? Good. Editorial comments? Very bad.

It was partially because of that feedback that it took so long to get Page Permissions approved. I also lost interest in the process. I did eventually come back and finalize the submission and ultimately got the MOD released.

I personally would never want to join the MOD Validator Team. I think it’s a no-win situation to be in. They are always behind, partially because of the enthusiasm of the phpBB2 MOD community. And that, to some degree, is because of the longevity of the phpBB2 codebase. The longer something stays the same, the more people become familiar with it, and the more potential MOD writers there are. And the further behind they get.

I’m sorry you had a bad experience. From my experience I would suggest that if enough time has passed you might try another submission to see if things have improved. If you feel that they have not, then a properly structured personal message (meaning constructive criticism) sent to wGEric (the MOD Team leader) might be appropriate.

If you want to take this offline, feel free to send me a PM at phpbb.com (username “drathbun”).

You mentioned looking at phpbbhacks for rebuild MODs. They may very well have some, but you won’t find any of my MODs at that site. Let me put it another way: if you find one of my MODs at that site then they were not submitted with my approval. At this time I only release MODs at this site (phpbbdoctor.com) or at phpbb.com.

I would be willing to install the rebuild mod and the efficient clean words mod. If you are asking for a complete mysql dump of the board, that can be tricky. While I only have one confidential forum (the rest are public view), the users table is very much enhanced and contains privacy data far beyond what a normal phpbb forum contains, and I really can’t release that info. perhaps a selective dump for that table would be the answer. I’m not sure how a stock english board would help you though.

On a separate rant regarding Olympus: I’ve actually released 2 successful MOD’s for PHPBB 2. I found working with the PHPBB community so distasteful both during the approval process. I also thought they handled the PHPBB 2.2 update process (originally scheduled for 2003?) and the justified criticism for their programmatic decisions, blown deadlines, unrealistic goals and milestones extremely poorly and with hypersensitivity. Now, almost 5 years later, I’m looking in Area 51 (which I had been on and off for years) and I just don’t see a huge functionality improvement. I read about how people can’t stand PHPBB 2 after running PHPBB 3 beta, but I’m completely missing what they are raving about. From outward appearances, it’s a minor version (i.e. V2.2) update of PHPBB 2.0. So why would anybody in their right mind rush to PHPBB 3 when PHPBB 2 is only now becoming usable from a security standpoint and while the PHPBB community has fragmented (read: all the original guys are long gone) and proven they can’t manage a project well. There will be years of bug fallout from the release of PHPBB 3. I think PHPBB 2 had plenty of momentum in 2002 and was well on their way to give all forum projects a run for their money, but that swell is gone.

And to here the community, you’d think they were building a database engine or a new operating system. It’s a freaking message board. I personally, with no help, could start from scratch a build a forum with more functionality in 4 months, so 5 years for an upgrade is absurd.

To top it off, I had a lot of great ideas for new MOD’s, for functionality that doesn’t exist (similar to my Topic Cement MOD), but after such a negative experience there’s no way I was going to take the time to develop and contribute any more than I did. I was asked once or twice to port my mods to PHPBB 3, but I’m not inclined to do that.

I have no idea why I ranted, but it felt good. I guess I was supported your decision to wait a long time to migrate to PHPBB 3. Maybe it’ll give you an idea for a blog topic.

John

I have not had time to complete my research into why the regex fails on non-english language boards. When I was asking about your language, I wasn’t trying to pry, but I was wondering if you would be willing to test some code on your system.

The search synonyms is wrong, as I posted recently. To fix that would require fixing the synonyms data file, or implementing the search syonyms table. I will be releasing the search stopwords/synonyms as a MOD, but not before I add one more feature to it. I want to add “search addwords”, which are words that would normally be eliminated because of their length or because they contain symbols and whatnot. This is a real complaint at my biggest board where it is impossible to search for version numbers like 6.5.1 because the period is dropped and the remaining characters are all too short to be indexed.

Once phpBB.com comes back up (which I would imagine would have to be soon at this point) and you can find a rebuild MOD, would you be willing to try the Efficient Cleanwords MOD and give me your feedback? Alternatively, if you have a smaller board in a language other than german/swiss german that you could allow me to use for testing, I would appreciate it. The data would be treated as strictly confidential, and installed only on a server at my house which is not exposed to the Internet. I completely understand if the answer to both questions is “No”, so don’t worry about hurting my feelings.

I did not see your rebuild mod on phpbbhacks, and phpbb.com is down indefinitely, so I don’t know where I would get a copy. I guess what would be nice is a combined mod or set of mods that would fix the regex, update the cleanwords, and provide an admin utility to rebuild the search database. With those tools (and perhaps some tweaking to stopwords), wouldn’t that “repair” a stock 2.0.22 board? Or is there something else that needs tweaking?