More Real World PHP Routing Benchmarks
Sharing some updates on the real-world example benchmark of Symfony's Routing component and FastRoute
June 7, 2021
One week after publishing the results from my real-world PHP routing benchmark, there are few interesting things to share.
Reception
I created the benchmark in order to use it as a measuring stick when comparing the performance of the two most popular routing libraries in PHP, Symfony Routing and FastRoute. My intention behind sharing publicly the results was to invite the scrutiny that is associated with every open-source project, and have both my code and my findings challenged.
What happened is there was a lot of positive feedback, mainly on Twitter. I was happy at the attention my project was getting, and I (somewhat foolishly) assumed that my code is OK. Well, it was not 😃
It turns out that most people just read the blog post, but didn't pay a lot of attention to the code.
First Bug
In my original findings I wondered why both benchLast (benchmark for matching the last route in the route collection) and benchLongest (benchmark for matching the longest route in the route collection) have so similar results. In theory they shouldn't as with the different strategies there are remedies for combating the penalty of matching the last route, and I assumed that it should perform better then benchLongest.
After so many likes and retweets I thought that someone would've picked up if something
was wrong, but still I never stop to doubt myself, and I started digging. As it happens
in a lot of cases, indeed I was visited by the Fuck-up Fairy and there was a
bug
in quick-benchmark.php where both benchmarks were calling benchLongest.
I've run the tests again after the fix, and I've updated the original post with the new results, and a fresh list of takeaways. It turns out that indeed benchLast is faster than benchLongest 😎
Second Bug
Several days later, Nikita Popov and Saif Eddin Gmati
pointed out that I am not using the cachedDispatcher() in FastRoute:
As @azjezz pointed out to me, it looks like these benchmarks were run with FastRoute caching disabled :) Probably your conclusion is still right, but it makes those benchmark results really misleading.
Nikita Popov June 4, 2021
I did added that to the benchmark project by introducing four new fast_cached-* bench cases, and that drastically changed the results:
I haven't committed my changes yet, but unless I've fucked up something else, the results are drastically different
Kaloyan K. Tsvetkov June 4, 2021
The screenshot above is from running quick-benchmark.php locally on my beaten-up
Mac Mini running PHP7.1.3, and it shows that cached FastRoute on average performs
at least 1.3x to 1.5x faster than Symfony Routing.
Here are the new updated results, and you can also see the reports from Github Actions:
PHP 7.1
+-------------------------+--------------+--------+------------------+-----------------+
| Case | Scenario | Routes | Time | Per Second |
+-------------------------+--------------+--------+------------------+-----------------+
| fast_cached_mark | benchAll | 364 | 0.245389 seconds | 1483.3591673824 |
| fast_cached_char_count | benchAll | 364 | 0.249807 seconds | 1457.125594837 |
| fast_cached_group_pos | benchAll | 364 | 0.254083 seconds | 1432.6031935787 |
| fast_cached_group_count | benchAll | 364 | 0.260062 seconds | 1399.665061699 |
| symfony_compiled | benchAll | 364 | 0.413057 seconds | 881.23411950566 |
| fast_cached_mark | benchLast | 300 | 0.340637 seconds | 880.70299950656 |
| fast_cached_group_count | benchLast | 300 | 0.342824 seconds | 875.08463684742 |
| fast_cached_char_count | benchLast | 300 | 0.351849 seconds | 852.63887580713 |
| fast_cached_group_pos | benchLast | 300 | 0.365263 seconds | 821.32603666784 |
| symfony_compiled | benchLast | 300 | 0.493613 seconds | 607.76356609928 |
| fast_cached_mark | benchLongest | 300 | 1.061979 seconds | 282.49144695766 |
| fast_cached_group_pos | benchLongest | 300 | 1.076209 seconds | 278.75624619514 |
| fast_cached_group_count | benchLongest | 300 | 1.087883 seconds | 275.76495010203 |
| fast_cached_char_count | benchLongest | 300 | 1.112363 seconds | 269.69610910862 |
| symfony_compiled | benchLongest | 300 | 1.215672 seconds | 246.77708791194 |
| fast_char_count | benchAll | 364 | 2.768154 seconds | 131.49556745285 |
| fast_group_pos | benchAll | 364 | 2.787809 seconds | 130.56849077365 |
| fast_mark | benchAll | 364 | 2.854463 seconds | 127.51960252867 |
| fast_group_count | benchAll | 364 | 2.888068 seconds | 126.03581527453 |
| fast_mark | benchLast | 300 | 2.428645 seconds | 123.52566284945 |
| fast_char_count | benchLast | 300 | 2.504715 seconds | 119.77409626942 |
| fast_group_count | benchLast | 300 | 2.540682 seconds | 118.07853915284 |
| fast_group_pos | benchLast | 300 | 2.569429 seconds | 116.75746216514 |
| symfony | benchAll | 364 | 3.659201 seconds | 99.475270496406 |
| fast_char_count | benchLongest | 300 | 3.158607 seconds | 94.978577399213 |
| fast_group_count | benchLongest | 300 | 3.186798 seconds | 94.138376825474 |
| fast_mark | benchLongest | 300 | 3.187436 seconds | 94.119533765834 |
| fast_group_pos | benchLongest | 300 | 3.340925 seconds | 89.795491354189 |
| symfony | benchLongest | 300 | 3.612994 seconds | 83.033629085927 |
| symfony | benchLast | 300 | 4.843723 seconds | 61.935831411289 |
+-------------------------+--------------+--------+------------------+-----------------+PHP 7.2
+-------------------------+--------------+--------+------------------+-----------------+
| Case | Scenario | Routes | Time | Per Second |
+-------------------------+--------------+--------+------------------+-----------------+
| fast_cached_mark | benchAll | 364 | 0.213650 seconds | 1703.7211333838 |
| fast_cached_group_count | benchAll | 364 | 0.230613 seconds | 1578.4019576929 |
| fast_cached_group_pos | benchAll | 364 | 0.231448 seconds | 1572.7079173805 |
| fast_cached_char_count | benchAll | 364 | 0.233461 seconds | 1559.1475705827 |
| fast_cached_mark | benchLast | 300 | 0.306500 seconds | 978.7921948108 |
| symfony_compiled | benchAll | 364 | 0.373521 seconds | 974.50990099327 |
| fast_cached_char_count | benchLast | 300 | 0.319633 seconds | 938.57640914251 |
| fast_cached_group_count | benchLast | 300 | 0.319820 seconds | 938.02785402355 |
| fast_cached_group_pos | benchLast | 300 | 0.325751 seconds | 920.94930974062 |
| symfony_compiled | benchLast | 300 | 0.452147 seconds | 663.50101927714 |
| fast_cached_group_pos | benchLongest | 300 | 0.826421 seconds | 363.01109466367 |
| fast_cached_char_count | benchLongest | 300 | 0.839799 seconds | 357.22836770222 |
| fast_cached_mark | benchLongest | 300 | 0.843073 seconds | 355.84111779518 |
| fast_cached_group_count | benchLongest | 300 | 0.852549 seconds | 351.88590125425 |
| symfony_compiled | benchLongest | 300 | 0.957260 seconds | 313.39451497233 |
| fast_mark | benchAll | 364 | 2.072627 seconds | 175.62252548766 |
| fast_char_count | benchAll | 364 | 2.103555 seconds | 173.04040361108 |
| fast_group_pos | benchAll | 364 | 2.121528 seconds | 171.57443809145 |
| fast_group_count | benchAll | 364 | 2.126081 seconds | 171.20702444539 |
| fast_mark | benchLast | 300 | 1.848153 seconds | 162.32421311617 |
| fast_char_count | benchLast | 300 | 1.856099 seconds | 161.62931919579 |
| fast_group_pos | benchLast | 300 | 1.876896 seconds | 159.83837956819 |
| fast_group_count | benchLast | 300 | 1.970900 seconds | 152.21473811394 |
| symfony | benchAll | 364 | 2.571703 seconds | 141.54045240237 |
| fast_mark | benchLongest | 300 | 2.346373 seconds | 127.85690494098 |
| fast_char_count | benchLongest | 300 | 2.369537 seconds | 126.60700610622 |
| fast_group_pos | benchLongest | 300 | 2.376117 seconds | 126.2564095669 |
| fast_group_count | benchLongest | 300 | 2.457300 seconds | 122.08520623162 |
| symfony | benchLongest | 300 | 2.673281 seconds | 112.22165014515 |
| symfony | benchLast | 300 | 3.383559 seconds | 88.664037190817 |
+-------------------------+--------------+--------+------------------+-----------------+PHP 7.3
+-------------------------+--------------+--------+------------------+-----------------+
| Case | Scenario | Routes | Time | Per Second |
+-------------------------+--------------+--------+------------------+-----------------+
| fast_cached_mark | benchAll | 364 | 0.212982 seconds | 1709.0651027469 |
| fast_cached_group_count | benchAll | 364 | 0.218750 seconds | 1664 |
| fast_cached_group_pos | benchAll | 364 | 0.221451 seconds | 1643.7041491851 |
| fast_cached_char_count | benchAll | 364 | 0.226611 seconds | 1606.2775517504 |
| symfony_compiled | benchAll | 364 | 0.345792 seconds | 1052.6564898549 |
| fast_cached_mark | benchLast | 300 | 0.290941 seconds | 1031.1368972257 |
| fast_cached_group_pos | benchLast | 300 | 0.298009 seconds | 1006.6812807015 |
| fast_cached_group_count | benchLast | 300 | 0.300559 seconds | 998.139986483 |
| fast_cached_char_count | benchLast | 300 | 0.322342 seconds | 930.68806901771 |
| symfony_compiled | benchLast | 300 | 0.439923 seconds | 681.93744644805 |
| fast_cached_group_count | benchLongest | 300 | 0.854665 seconds | 351.01480616081 |
| fast_cached_mark | benchLongest | 300 | 0.872547 seconds | 343.8209615578 |
| fast_cached_char_count | benchLongest | 300 | 0.878194 seconds | 341.61022443322 |
| fast_cached_group_pos | benchLongest | 300 | 0.900564 seconds | 333.1245917984 |
| symfony_compiled | benchLongest | 300 | 1.016180 seconds | 295.22327604169 |
| fast_char_count | benchAll | 364 | 2.216095 seconds | 164.2528884421 |
| fast_group_pos | benchAll | 364 | 2.221537 seconds | 163.85053254518 |
| fast_mark | benchAll | 364 | 2.247656 seconds | 161.94648232648 |
| fast_group_count | benchAll | 364 | 2.302350 seconds | 158.09933025128 |
| fast_mark | benchLast | 300 | 1.959023 seconds | 153.13755896805 |
| fast_group_pos | benchLast | 300 | 1.980528 seconds | 151.47474935707 |
| fast_char_count | benchLast | 300 | 2.041746 seconds | 146.93305607013 |
| fast_group_count | benchLast | 300 | 2.094186 seconds | 143.25376278568 |
| symfony | benchAll | 364 | 2.811152 seconds | 129.48428345902 |
| fast_mark | benchLongest | 300 | 2.530609 seconds | 118.5485441453 |
| fast_group_count | benchLongest | 300 | 2.543774 seconds | 117.93499929986 |
| fast_char_count | benchLongest | 300 | 2.576504 seconds | 116.436846565 |
| fast_group_pos | benchLongest | 300 | 2.733477 seconds | 109.75032433397 |
| symfony | benchLongest | 300 | 3.127849 seconds | 95.912555310366 |
| symfony | benchLast | 300 | 3.844360 seconds | 78.036398718119 |
+-------------------------+--------------+--------+------------------+-----------------+PHP 7.4
+-------------------------+--------------+--------+------------------+-----------------+
| Case | Scenario | Routes | Time | Per Second |
+-------------------------+--------------+--------+------------------+-----------------+
| fast_cached_mark | benchAll | 364 | 0.184830 seconds | 1969.3752544393 |
| fast_cached_char_count | benchAll | 364 | 0.190946 seconds | 1906.2994762045 |
| fast_cached_group_pos | benchAll | 364 | 0.198573 seconds | 1833.0779803957 |
| fast_cached_group_count | benchAll | 364 | 0.200405 seconds | 1816.3230136136 |
| fast_cached_char_count | benchLast | 300 | 0.265815 seconds | 1128.6043973096 |
| symfony_compiled | benchAll | 364 | 0.323630 seconds | 1124.7408882992 |
| fast_cached_mark | benchLast | 300 | 0.267429 seconds | 1121.7935985194 |
| fast_cached_group_count | benchLast | 300 | 0.272139 seconds | 1102.3775356263 |
| fast_cached_group_pos | benchLast | 300 | 0.274821 seconds | 1091.6204122212 |
| symfony_compiled | benchLast | 300 | 0.367107 seconds | 817.20009430048 |
| fast_cached_mark | benchLongest | 300 | 0.723163 seconds | 414.84415953495 |
| fast_cached_char_count | benchLongest | 300 | 0.729227 seconds | 411.39449421304 |
| fast_cached_group_pos | benchLongest | 300 | 0.739985 seconds | 405.41362918457 |
| fast_cached_group_count | benchLongest | 300 | 0.775798 seconds | 386.69855832926 |
| symfony_compiled | benchLongest | 300 | 0.854253 seconds | 351.18399485124 |
| fast_mark | benchAll | 364 | 1.739359 seconds | 209.27250815583 |
| fast_group_count | benchAll | 364 | 1.753623 seconds | 207.57026920172 |
| fast_group_pos | benchAll | 364 | 1.808081 seconds | 201.31842422393 |
| fast_mark | benchLast | 300 | 1.535626 seconds | 195.36004613032 |
| fast_char_count | benchAll | 364 | 1.875456 seconds | 194.08612178155 |
| fast_char_count | benchLast | 300 | 1.546621 seconds | 193.97123384788 |
| fast_group_pos | benchLast | 300 | 1.564864 seconds | 191.70993107962 |
| fast_group_count | benchLast | 300 | 1.571994 seconds | 190.84041501369 |
| symfony | benchAll | 364 | 2.233018 seconds | 163.00807872597 |
| fast_mark | benchLongest | 300 | 2.023856 seconds | 148.2318955379 |
| fast_group_pos | benchLongest | 300 | 2.036537 seconds | 147.30889250637 |
| fast_group_count | benchLongest | 300 | 2.042478 seconds | 146.88040095373 |
| fast_char_count | benchLongest | 300 | 2.085564 seconds | 143.84598826018 |
| symfony | benchLongest | 300 | 2.308640 seconds | 129.9466350681 |
| symfony | benchLast | 300 | 2.949923 seconds | 101.6975684065 |
+-------------------------+--------------+--------+------------------+-----------------+An Interesting Fork
After the second bug, Saif Eddin Gmati and I started an interesting discussion. Saif shared a fork of my project on which he worked to test out some of his ideas about how to run the tests. He experimented with JIT and PHP8, with APCU, and he was able to verify the cached results for FastRoute before me.
In addition to that, Saif added two interesting additions to his benchmark fork.
-
First, he wanted to track the performance of PHP-based port hack-router by Facebook. You can find this project at github.com/azjezz/hack-routing, and it is included in his benchmark repo. He is actively working on this and I guess soon he will have some interesting observations to share.
-
Second, Saif also demonstrated benchmark cases that have the dispatcher object preserved, and re-used. These so called "instance" benchmarks show how the routers will perform in web applications that use Swoole or ReactPHP or similar techniques, which are gaining larger audience and bigger popularity lately. His benchmarks showed that once loaded and preserved, symfony_compiled does show the best results. You can see some of the earlier runs of these benchmarks here:
❯ php -dopcache.jit=1235 -dopcache.enable_cli=yes -dopcache.enable=yes -dapc.enable=1 -dapc.enable_cli=1 scripts/quick-benchmark.php
18/18 [============================] 100%
+---------------------------+--------------+--------+------------------+-----------------+
| Case | Scenario | Routes | Time | Per Second |
+---------------------------+--------------+--------+------------------+-----------------+
| symfony_instance | benchAll | 364 | 0.001703 seconds | 213737.45709085 |
| symfony_instance | benchLast | 300 | 0.001650 seconds | 181807.71564803 |
| fast_mark_instance | benchLast | 300 | 0.002801 seconds | 107097.72746617 |
| fast_char_count_instance | benchAll | 364 | 0.003597 seconds | 101194.84695433 |
| fast_group_pos_instance | benchAll | 364 | 0.004016 seconds | 90639.198290193 |
| fast_mark_instance | benchAll | 364 | 0.004131 seconds | 88112.578980781 |
...Finally, I would say I am really happy with the feedback and ideas I got from this. There is a lot more work to be done, but this was a nice first step.