Learning to Rank with XGBoost
Have you ever come across a situation where you used a search engine and not found what you’re looking for within the first few items returned? Perhaps, you go back and amend the query but the results get even worse. Most of the contemporary search engines such as Google, Bing, Netflix, Amazon, etc. are often very good at finding all the results for a given query. Think how good are they, when it comes to identifying which ones are the best and returning those first.
When we say search engines, we are not only talking about Google and Bing but there are a huge amount of search engines available almost everywhere on the internet. One thing we can all agree about is that, despite the application, a good search functionality would always lead to satisfied customers, better business revenues and so on. Therefore, having the ability to rank the document quickly and accurately is very important. That’s where the learning to rank algorithms come in handy. In this article, we will talk about how we can use the XGBoost package for this task.
first, let’s formally look at what is learning to rank (LTR).
Learning to Rank
Learning to rank or machine-learned ranking (MLR) methods are commonly employed in the development of ranking models for information retrieval systems, where the training data consists of a collection of items in a certain order. This order is mostly determined by a numerical or ordinal score. Once it is trained, the Learning to Rank model should thereafter be able to generate a permutation of an unknown itemset.
There are three primary kinds of learning to rank algorithms, according to Tie-Yan Liu’s book, Learning to Rank for Information Retrieval: Pointwise, Pairwise, and Listwise approaches. According to the number of documents the algorithm considers when computing the loss function, we can identify three main types of approaches in Learning to Rank. Let’s have look at each of these methods.
Pointwise approach
In the pointwise method, only one point document is considered at a time. It’s as simple as using a regression/classification model to predict a document’s rank value/class, where sorting the expected regression/classification result yields the rank. The relevance score/class of the texts is represented by the projected regression scores/classification class (i.e. very relevant to irrelevant at all). The score of the document generated by this method is unrelated to the scores of the other documents in the list.
This method assigns separate ratings to each document based on its relevance to the search query. This approach’s specific objective function is provided below.
This can be either regression or classification loss.
Pairwise approach
In the pairwise method, the documents loss function is assessed as a pair. When given a pair of documents, this method attempts to determine the best ordering and compare it to the actual order. The objective is to keep the ranking inversion to a minimum. Intuitively, this method follows the natural way of determining the proper order, and it performs better in practice when dealing with rank problems.
The model tries to discover the proper order of document pairings in this way, and it minimizes the objective function as shown below.
Listwise approach
The listwise approach treats all of the documents as a single list. It then selects portions of it to create the right ordering for it. As a result, when compared to the other two techniques, the listwise approach might have a more complicated calculation.
The model examines the documents as a full list in this manner. The goal of this exercise is to discover a ranking function that minimizes the predicted loss. The listwise approach’s loss function is specified across all the documents connected with the query, as illustrated below.
Now, we have an idea of what Learning to Rank is. Let’s see how we can implement Learning to Rank with XGBoost.
Learning to Rank with XGBoost
XGBoost is a version of the gradient boosting decision tree method that has been enhanced in terms of speed. Chen and Guestrin were the first to introduce it in 2016. You can read their paper from here. It was part of the Distributed (Deep) Machine Learning Community (DMLC) group’s research output. It is currently accessible as an open-source package, with contributions from numerous developers and community support, and can be viewed on Github. This technique has several advantages over the previous gradient boosting methods, including quicker execution speed, scalability in all circumstances, and enhanced model performance. It was demonstrated when the winners of multiple Kaggle tournaments employed XGBoost. It boosts the popularity of this product in a variety of domains. Gradient boosting is a method for making a final forecast by including a new model that can predict the residual errors of the prior model. When adding a new model, the method is known as gradient boosting since it employs the gradient descent technique as its objective function. As it develops, the XGBoost program may now be used to learn ranking problems using the three techniques outlined above.
