Thanks for taking the time to have a look. I will do some benchmarking and post the results.

I haven’t had a chance to look over this in detail yet, but there is one flaw in the code that may or may not have an impact on performance. It’s a common error on MySQL databases. When you create a query with an aggregate function (like sum() as you have) then you need to have every non-aggregate column appear in the GROUP BY. I will have to post more about that in a blog post. MySQL allows you to be “sloppy” and get away with not doing that, but it can have an impact on your data.

Also, you are doing sum(x) + sum(y) where both x and y come from the same table. I would think that sum(x+y) could be slightly more efficient, although it won’t change your index usage.

Final question: there is no “where clause” other than the expected joins. I believe you said that this was to get the top five cars with the most dollars spent on mods, right? What I would do is get those first in a separate query, retrieve the five vehicle IDs from that query, then go back and do the second query to get just those five.

Here’s why I think two queries is more efficient than one in this case. First, when you have a group by (even an incomplete one ) the entire query has to run to completion before the first row can come back. That means that all of your joins are being done for every car in your database, even though you only want five.

I would do this:

SELECT	m.vehicle_id,
	SUM(m.install_price + m.price) AS POI,
FROM (
	phpbb_garage_modifications m
)
GROUP BY m.vehicle_id
ORDER BY POI DESC
LIMIT 5

That query should run much faster than the larger query that you have. It will return a dataset of five car id values. Next, replace your bigger query with this:

SELECT v.id,
	CONCAT_WS(' ', v.made_year, mk.make, md.model) AS vehicle,
	v.user_id,
	u.username,
	v.currency,
	u.user_colour,
	u.user_id
FROM (
	phpbb_garage_vehicles v,
	phpbb_garage_makes mk,
	phpbb_garage_models md,
	phpbb_users u
)
WHERE	v.id IN ( ... insert ids from prior query here ... )
	AND v.make_id = mk.id
	AND mk.pending = 0
	AND v.model_id = md.id
	AND md.pending = 0
	AND v.user_id = u.user_id

You will notice that I skipped the “modifications” table in the second query. You have the total modification dollars (currency type) already from the prior query, so you don’t need to have the overhead of recalculating it in this query.

The net result – without any benchmarking on my part, I should add – should be an improvement, I would think. The first query still runs a GROUP BY and a SUM but only on one table. No joins are required. From that query you will get a dataset (array) of vehicle ID values and their total modification dollars. The second query gets the user and vehicle specifics for those top 5 vehicles.

This query is to get vehicles with the most money spent on them.
Vehicles are stored in phpbb_garage_vehicles. Each vehicle can have lots of modifications, each modification is stored in phpbb_garage_modifications. Each modification has two “costs” associated with them, a purchase and install price. We need to get the sum of these and return the vehicles with the highest sum. Each vehicle is own by one user stored in the regular table and each vehicle consists of a make and a model stored in table phpbb_garage_makes & phpbb_garage_models

Hope that makes sense. Query is as below.

SELECT v.id,
	CONCAT_WS(' ', v.made_year, mk.make, md.model) AS vehicle,
	v.user_id,
	(SUM(m.install_price) + SUM(m.price)) AS POI,
	u.username,
	v.currency,
	u.user_colour,
	u.user_id
FROM (
	phpbb_garage_vehicles v,
	phpbb_garage_makes mk,
	phpbb_garage_models md,
	phpbb_users u,
	phpbb_garage_modifications m
)
WHERE m.vehicle_id = v.id
	AND v.make_id = mk.id
	AND mk.pending = 0
	AND v.model_id = md.id
	AND md.pending = 0
	AND v.user_id = u.user_id
GROUP BY v.id
ORDER BY POI DESC
LIMIT 5

So our explain plan gives us the follow. We can see we have “using temporary”, which all my reading I believe to be a indicator of poor index/query structure.

+-----+---------------+----------+------------+-----------------------------+---------+---------+----------------------+-------+----------------------------------+
|  Id |   Select Type |   Table  |    Type    |       Possible Keys         |   Key   |  KeyLen	|         Ref          |  Rows |      Extra                       |
+-----+---------------+----------+------------+-----------------------------+---------+---------+----------------------+-------+----------------------------------+
|  1  |     SIMPLE    |     m    |     ALL    |	vehicle_id_2,vehicle_id     |         |         |                      |   3   |  Using temporary; Using filesort |
|  1  |     SIMPLE    |     v    |    eq_ref  |	PRIMARY,user_id             | PRIMARY |    4    | phpbb3.m.vehicle_id  |   1   |                                  |
|  1  |     SIMPLE    |     u    |    eq_ref  |	PRIMARY                     | PRIMARY |    3    | phpbb3.v.user_id     |   1   |  Using where                     |
|  1  |     SIMPLE    |     mk   |    eq_ref  |	PRIMARY                     | PRIMARY |    4    | phpbb3.v.make_id     |   1   |  Using where                     |
|  1  |     SIMPLE    |     md   |    eq_ref  |	PRIMARY                     | PRIMARY |    4    | phpbb3.v.model_id    |   1   |  Using where                     |
+-----+---------------+----------+------------+-----------------------------+---------+---------+----------------------+-------+----------------------------------+

So we have 5 table involved and we have the following structure, I will not bother listing phpbb_users as you know that one!!

Table description for phpbb_garage_vehicles

+------------------+----------------------------+---------+-------+---------------+----------------+
+  Field           |              Type          |   Null  |  Key  | Default       |    Extra       |
+------------------+----------------------------+---------+-------+---------------+----------------+
| id               | int(10) unsigned           |   NO    |  PRI  | 	NULL      | auto_increment |
| user_id          | int(10)                    |   NO    |  MUL  | 	0         |                |
| made_year        | varchar(4)                 |   NO    |       |     2003      |                |
| engine_type      | varchar(32)                |   NO    |       |     NULL      |                |
| colour           | varchar(128)               |   YES   |       |     NULL      |                |
| mileage          | int(10) unsigned           |   NO    |       |     0         |                |
| mileage_unit     | varchar(32)                |   NO    |       |     Miles     |                |
| price            | int(10) unsigned           |   YES   |       |     NULL      |                |
| currency         | varchar(32)                |   NO    |       |     USD       |                |
| comments         | varchar(255)               |   YES   |       |     NULL      |                |
| views            | int(10) unsigned           |   NO    |  MUL 	|     0         |                |
| date_created     | int(10)                    |   YES   |  MUL  | 	NULL      |                |
| date_updated     | int(10)                    |   YES   |  MUL 	|     NULL      |                |
| make_id          | int(10) unsigned           |   NO    |       |     0         |                |
| model_id         | int(10) unsigned           |   NO    |       |     0         |                |
| main_vehicle     | tinyint(1)                 |   NO    |       |     0         |                |
| weighted_rating  | double                     |   NO    |       |     0         |                |
| pending          | tinyint(1)                 |   NO    |       |     0         |                |
+------------------+----------------------------+---------+-------+---------------+----------------+

Indexes for phpbb_garage_vehicles

+---------------+----------+-------------------+--------------------------------+
|    Keyname    |   Type   |    Cardinality    |  Field                         |
+---------------+----------+-------------------+--------------------------------+
| PRIMARY       | PRIMARY  |  3                | id                             |
| date_created  | INDEX    |  None             | date_created                   |
| date_updated  | INDEX    |  None             | date_updated                   |
| user_id       | INDEX    |  None             | user_id                        |
| views         | INDEX    |  None             | views                          |
+---------------+----------+-------------------+--------------------------------+

Table description for phpbb_garage_makes

+------------------+----------------------------+-------------+-------+---------------+----------------+
+  Field           |              Type          |    Null     |  Key  | Default       |    Extra       |
+------------------+----------------------------+-------------+-------+---------------+----------------+
| id               | int(10) unsigned           | NO          |  PRI  | NULL          | auto_increment |
| make             | varchar(255)               | NO          |  MUL  |               |                |
| pending          | tinyint(1)                 | NO          |       | 1             |                |
+------------------+----------------------------+-------------+-------+---------------+----------------+

Indexes for phpbb_garage_makes

+---------------+----------+-------------------+--------------------------------+
|    Keyname    |   Type   |    Cardinality    |  Field                         |
+---------------+----------+-------------------+--------------------------------+
| PRIMARY       | PRIMARY  |  91               |  id                            |
| make          | INDEX    |  None             |  make                          |
+---------------+----------+-------------------+--------------------------------+

Table description for phpbb_garage_models

+------------------+----------------------------+-------------+-------+---------------+----------------+
+  Field           |              Type          |    Null     |  Key  | Default       |    Extra       |
+------------------+----------------------------+-------------+-------+---------------+----------------+
| id               | int(10) unsigned           |  NO         |  PRI  | NULL          | auto_increment |
| make_id          | int(10) unsigned           |  NO         |  MUL  | 0             |                |
| model            | varchar(255)               |  NO         |       |               |                |
| pending          | tinyint(1)                 |  NO         |       | 1             |                |
+------------------+----------------------------+-------------+-------+---------------+----------------+

Indexes for phpbb_garage_models

+---------------+----------+-------------------+--------------------------------+
|    Keyname    |   Type   |    Cardinality    |  Field                         |
+---------------+----------+-------------------+--------------------------------+
|    PRIMARY    |  PRIMARY |        811        | id                             |
|    make_id    |  INDEX   |        None       | make_id                        |
+---------------+----------+-------------------+--------------------------------+

Table description for phpbb_garage_modifications

+------------------+----------------------------+-------------+-------+---------------+----------------+
+  Field           |              Type          |    Null     |  Key  | Default       |    Extra       |
+------------------+----------------------------+-------------+-------+---------------+----------------+
| id               | int(10) unsigned           |     NO      |  PRI  | NULL          | auto_increment |
| vehicle_id       | int(10) unsigned           |     NO      |  MUL  | 0             |                |
| user_id          | int(10)                    |     NO      |  MUL  | 0             |                |
| category_id      | int(10) unsigned           |     NO      |  MUL  | 0             |                |
| manufacturer_id  | int(10) unsigned           |     NO      |       | 0             |                |
| product_id       | int(10) unsigned           |     NO      |       | 0             |                |
| price            | int(10) unsigned           |     NO      |       | 0             |                |
| install_price    | int(10) unsigned           |     NO      |       | 0             |                |
| product_rating   | tinyint(2)                 |     YES     |       | NULL          |                |
| purchase_rating  | tinyint(2)                 |     YES     |       | NULL          |                |
| install_rating   | tinyint(2)                 |     YES     |       | NULL          |                |
| shop_id          | int(10)                    |     YES     |       | NULL          |                |
| installer_id     | int(10)                    |     YES     |       | NULL          |                |
| comments         | text                       |     YES     |       | NULL          |                |
| install_comments | text                       |     YES     |       | NULL          |                |
| date_created     | int(10)                    |     YES     |  MUL  | NULL          |                |
| date_updated     | int(10)                    |     YES     |  MUL  | NULL          |                |
+------------------+----------------------------+-------------+-------+---------------+----------------+

Indexes for phpbb_garage_modifications

+---------------+----------+-------------------+--------------------------------+
|    Keyname    |   Type   |    Cardinality    |  Field                         |
+---------------+----------+-------------------+--------------------------------+
| PRIMARY       | PRIMARY  | 3                 | id                             |
| user_id       | INDEX    | None              | user_id                        |
| vehicle_id_2  | INDEX    | None              | vehicle_id, category_id        |
| category_id   | INDEX    | None              | category_id                    |
| vehicle_id    | INDEX    | None              | vehicle_id                     |
| date_created  | INDEX    | None              | date_created                   |
| date_updated  | INDEX    | None              | date_updated                   |
+---------------+----------+-------------------+--------------------------------+

If you care to post a query and the explain plan, along with the index structure for the tables involved, I will be happy to use it as a case study for a future post.

Search synonyms are, in fact, a topic of an upcoming post. They don’t work the way I expected them to. They work, of course, but they’re just different. In my revised code they have also been moved into the database, and they have been given the same treatment. That is to say that after moving them to the database I can then check for synonyms only of the words that are actually in use rather than applying the entire synonym file… even to a post with only one word.

I like the fact that search synonyms can be used to “correct” for words that are commonly spelled wrong.

I had not known who the main developer was for the search process. And thanks for confirming that some of this research might even be appropriate for phpBB3 as well.

I should make one thing very clear: I like the phpBB2 search process. I think that if we could figure out a way to effectively add a “search exact phrase” option to the process it would be awesome. The point of this series of posts has been to justify to detractors why I like the search process and along the way, as I have discovered things, to document some tweaks that can make it even better.

As far as future posts, I am going to talk about the search synonym process, the “common words” problem, an enhancement to search.php that I’m writing that will help the user understand how to get better search results… and a few more that at the moment I don’t remember. For example we have the concept of search stopwords which I have covered now in great detail, the concept of search synonyms which is coming… I have an idea for a MOD called Search Addwords aka the Jargon MOD. There are certain “words” that get dropped because they are too short (or too long) or structured in such a way that they get ignored. For certain boards it might be vital to search for certain two-letter words, but not all two-letter words.

I plan to expand on the process of putting search words into the database by creating a MOD that allows board owners to add words that they want to ensure get included in the search index. Those words would likely be very unique to each board. For example, on your board maybe you want people to be able to search for “Mustang V8 GT” or something like that. Well, GT and V8 would be ignored because they’re too short. With a Seach Addwords table you could ensure that those words do, in fact, get included in the search index. It seems to be a logical extension of the existing search process; I’m surprised I have not seen it suggested somewhere else before.

Which doesn’t mean that it hasn’t been… it just means I haven’t seen it.

Thanks again for your comment; I wish I knew what you had done to trigger the spam filter. If at some point a comment of yours gets removed, it was an accident and nothing personal. Some times I open the spam list and click “Delete All” without reading through the list; it depends on how many are waiting.

As always a great post with some very interesting results. Be interesting to hear someone like naderman comment on this type of research. As he seems to be the dev handling searching in phpBB3, and we still have the same concept now called ignore words.

You plan on covering search synonyms at all? I know are used on the query side of the search engine + phpBB ships with some funny ones. My fav being “gynecology gynaecology”, useful for 0.0001% of boards!!