What have I done this week?
Did a few simple adjustments to backpack.tf’s MongoDB modeling. Cut updates/second to half while doing so. We need to consider downscaling the hardware at some point, because I think it’s too much.
I’ve been helping an old friend with some Elasticsearch queries, since he was using MongoDB and getting poor results. The problem was an interesting one: given a player’s inventory of trading cards, find the deck that has gotten the highest score in recent games. He wrote an aggregation pipeline to do so in MongoDB, but it was slow, returning results in 10 seconds, and wondered if it could be done better with Elasticsearch. Since I’m kind of experienced with it, I gave it a look.
The terms set query came in handy for matching against a set, although it’s a very new feature and I guess it’s not fully baked yet: to use it, I had to use a script with a constant value. It was either that, or bake a
minimum_should_match_field property into each source document. I don’t know why there is not an option to use a simple value input for these queries, but there should probably be one. I’m not sure what the overhead is of using a script for this sort of thing, but it seems negligible.
But whatever. ES was handling a query in 100 friggin’ milliseconds, as opposed to MongoDB’s ten thousand. It seemed pretty scalable, too, with 50 concurrent requests for different sets of various sizes averaging at about 250ms/response. Considering the size of the deck permutation collection (about 6.4m or so documents), and the size of the inputs, this seemed pretty decent. I’m still wondering if it could be improved, since a) is a script with a constant value adding performance overhead to the query? and b) I’m sure there’s something I could do to optimise the query further, since it’s just a simple bool query, filter and sort.
I’ve updated my php-steam-inventory module to add helper methods for Steam inventory documents. This change uses a nested model approach to add methods for assets and descriptions.
What am I going to do next?
Working on AJAX inventories for backpack.tf, maybe will play with PHP-PM again now it’s stable. Have I said that before? I’ve been laying it off so much, but probably because I’m nervous about it falling over. I’ll probably end up running it in production for a few weeks, but on a different port, so it’s not completely replacing php-fpm and we can test for side effects and stability with real users.