search – Manos Tsagkias

The end of 2019 will mark the fifth anniversary of 904Labs. Wouter and I founded 904Labs in late 2014 and embarked in an adventure of becoming entrepreneurs while holding tight to our scientist trait. We took courses in entrepreneurship, joined our university’s startup incubator, poured all the money we had saved by that moment in the company, and we set sail for bringing state-of-the-art search to the masses.

A lot has happened in these five years. We met with incredible people in all ranks, involved in from local startups to worldwide multinationals. Each one helped us in their own way, from validating our ideas, shaping our product-A.I. for Search-, to running pilots. We received support from our university, University of Amsterdam, through grants and loans, which helped us go through the development phase of our product and bring it to the market. Within a year since we started, we grew from just two founders to a team of six, and our code base went from a bunch of scripts to a full blown, distributed, horizontally scalable, multi-tenant architecture. In our second year we ran our first pilot, with massive success: A.I. for Search managed to increase search driven revenue by 38%! The same year we had our first big customer and the year after a second, bigger one. By 2019, A.I. for Search has served searches from more than 15 countries and an equal number of languages. It has battled and won in more than 10 A/B tests, in all of which it has proven to substantially increase search driven revenue. Tech-wise, 904Labs has been success story, really.

Business-wise, the story is slightly different. Wouter and I, our business developers and salespeople, found it hard to sell A.I. for Search. Customers were excited with the tech and the potential, but the deals were not closing. We’ve been banging our heads against the wall for a long time but didn’t manage to find a solution to why the deals were not closing, at least not early enough. Three years in and without a good customer base, we’ve hit a financial storm. We had to let all of our people go, and our offices as well. Despite the financial stress, we decided to keep the company running while hoping for better days; we minimized all possible expenses to squeeze out all the time we could buy until new deals come in–which would allow us to reboot the company. Two years since, we didn’t manage to close new deals, but the epiphany we were looking for since our early days stroke us: deals weren’t closing because we’ve been targeting a very narrow and difficult customer segment, which was insufficient to sustain the company growth we needed and wished for. The realization hit us hard. We spent a lot of time thinking of alternatives but couldn’t find a satisfying alternative. In the beginning of 2019, one of our two customers left, and their departure marked the beginning of winding down 904Labs.

We decided that despite the difficult customer segment we chose to target, 904Labs’ tech was still magical and had to remain with the world no matter what was the future of 904Labs as company. We’ve reached out to anyone and any entity that we believed our tech will be useful: our professional network, potential customers, competitors. We offered a one time license to our source code. After taking hold of the code one would be able to do whatever they want with it. In the one time license fee, we included our latest version of our code base, our infrastructure, documentation, our processes, and, of course, onsite training and support. We’ve been actively talking with a few companies now, and they will bring in A.I. for Search to either complement and strengthen their offering, or to use it internally in different projects. The next few quarters may bring the company of 904Labs to an end, but its tech will survive, and it will keep helping people find what they’re looking for; by learning from one click at a time, as it used to.

As to what Wouter and I are up to next, it has been a tough decision but we’re on good and exciting new paths. Wouter has joined Zeta-Alpha-Vector, a Dutch deep learning research company based in Amsterdam, and I’m joining Apple in California to work with the Siri Understanding team. While at 904Labs, we both learned so much: building and running a business, developing a product, managing engineering and sales teams. It’s been an invaluable experience. 904Labs gifted us the knowledge of what it takes to build a product in the real-world, which is very different from what we had in mind when we left Academia. By now we know that a product needs to be fun from a tech/scientific perspective but it is equally important to satisfy customers needs and solve specific problems–at the same time it is bound by limited resources, either be it time, people, or budget. Finding the right balance on all three dimensions is key to product success. The lesson may have come late for 904Labs, but we are excited to be able to apply this hard earned knowledge as well as earlier and newly acquired skills to new environments and challenges!

Before waving goodbye, we would like to take a moment and extend a big thank you to everyone who has been involved with 904Labs, directly or indirectly: Our customers, engineers, sales people, university, mentors, shareholders, network, and last, but not least, our families and friends who’ve been very supportive throughout this entire journey. It’s been a fun ride but not always easy. We wouldn’t have made it so far without their support. Thank you!

I was invited to give a talk at the SIGIR Workshop On eCommerce 2019 but unfortunately, I am not attending SIGIR this year. Instead I wrote down some thoughts on interesting
problems, ideas and challenges in the e-commerce domain. Here they are:

The vocabulary gap is still an open problem. People refer to products in different ways that products are described in the catalogue. Neural networks, learning to rank, query intent engines are all important. Our experience at 904Labs shown that a query intent engine that boosts specific product categories given a query, boosts our learning to rank system by more than 16% in additional revenue. These results suggest that effective initial ranking is very important for effective learning to rank. To this end, we foresee an increasing interest in methods that can re-rank the entire collection and not only the top-N documents. This is particularly important for larger shops with large inventories (> hundred of thousands of items) where a query can return hundreds of items, and only the top few are re-ranked.
E-commerce search is as much as about exploration as it is about finding the best match. From our experience at 904Labs, we see a large fraction of queries to revolve around categories, or combinations of categories, e.g., “red shoes”, “kitchen tables”, “dvd players”, “ebooks for 12 years old”. This type of queries go beyond our typical search and require understanding the query and generating a list of recommended relevant items. This proposition is supported by the surprising effectiveness of sorting by popularity; the most popular items for a query are potential good candidates for this “recommendation list” that the user is looking for. Back to query understanding of this type of exploratory queries, one would think that natural language processing can help here but in the e-commerce setting, queries are very short and any language analysis falls short. An open question here is how can we go from these broad queries to a good set of recommendations? A natural way is devise a hybrid system of search and recommendations: We fire the query to a search system and then we take the first few items as seed to a recommender system for getting similar items. Or a system that transforms a query to an image (think AttnGAN or similar) and each product to an image and then rank documents by their image similarity to the query’s–the image representation may constraint the latent space, abstract the language of the query and that of the document and be able to capture semantics that are otherwise difficult to encode in textual form. The image representation of queries and documents also offers explainability when it comes to explaining the rankings of the system; which is becoming increasingly quite important in machine learning-based systems.
Evaluation metrics in e-commerce. There several directions here that we need more work. First, the e-commerce setting is a conjunction of exploratory search, typical search, and recommendations. Using the standard IR precision and recall measures for e-commerce may not tell us the entire story for how happy makes its users. We need to discover the aspects of a system that makes users happy for designing one or more metrics for evaluating search and recommendation systems. These metrics should also correlate well with revenue but also with customer loyalty (measured in returning customers and in shortening the time between returns). Such a metric (or a multiple of metrics) will then allow us to run offline experiments and make predictions on revenue, which is the main KPI that systems are evaluated in production for most e-commerce business.

Extra tip: we found that boolean scoring may be on par or outperform tf.idf or BM25 scoring in the e-commerce domain, it’s worth checking its effectiveness on your own data 😉

Tag: search

904Labs: Goodbye and thanks for all the clicks!

Thoughts on the future of Search in E-commerce