{"id":9090,"date":"2025-10-25T15:58:18","date_gmt":"2025-10-25T13:58:18","guid":{"rendered":"https:\/\/www.codecon.sk\/?p=9090"},"modified":"2026-03-11T16:03:37","modified_gmt":"2026-03-11T15:03:37","slug":"search-is-not-magic-with-postgresql","status":"publish","type":"post","link":"https:\/\/www.codecon.sk\/en\/search-is-not-magic-with-postgresql\/","title":{"rendered":"Search is not magic with PostgreSQL"},"content":{"rendered":"
\n
\n
\n

Here I want to show you several approaches making search better for your data. I will show you powerful tools and some tips that I know thanks to days and weeks in it. In order not to be pure SQL I will also show implementation of approaches in the Elixir language with the great Ecto library. It\u2019s important to say that this \u201cproblem\u201d can not be solved only in the database. It has to be solved on both places where you have logic and data.<\/p>\n

A long time ago, when someone said \u201csearch\u201d to me, the first thing I thought of was:<\/p>\n

select<\/span> *<\/span> from<\/span> products where<\/span> name ilike '%banana%`;<\/span><\/span><\/pre>\n
It\u2019s easy with Ecto:<\/p>\n
text = \"banana\"\r\nfrom(p in Products, where: ilike(p.name, \"%#{text}%\")) \r\n|> Repo.all()<\/span><\/pre>\n
Is it good enough? Is it safe? Well, it worked. We can still say that it works in some cases. But people are more lazy at these days, they don\u2019t pay attention to what they write. So if customers write\u00a0banna<\/strong>na<\/code>\u00a0it will not work. Because all your products with the name\u00a0banana<\/code>\u00a0will just not match. You can say:<\/p>\n
\n
Yeah, I don\u2019t care, it\u2019s just a search\u2026<\/p>\n<\/blockquote>\n
Well, if you think it doesn\u2019t matter you can Google it or ask someone. For example some random statistics from the internet:<\/p>\n
\n
43% of retail customers\u00a0<\/strong>head directly to the search bar on a website.<\/em><\/p>\n
On-site search is used more frequently than the navigation menu (49%), filter feature (37%), and homepage recommendations (30%), with\u00a078%.<\/strong><\/em><\/p>\n
As many as\u00a068% of customers would not return<\/strong>\u00a0to a site with a\u00a0poor search<\/strong>\u00a0experience.<\/em><\/p>\n<\/blockquote>\n
Source:\u00a0<\/em>https:\/\/www.mailmodo.com\/guides\/ecommerce-site-search-statistics\/<\/em><\/a><\/p>\n
And I know a lot of sites where you can see that more than 80% of customers are using search instead of categories.<\/p>\n
Let\u2019s do a very simple calculation with fake data: You have 1000 visitors (per day\/month, doesn\u2019t matter). Based on the text below 790 visitors will use search. Let\u2019s say that 30% will get wrong results in search. That\u2019s 237 visitors who will get wrong results. What does it mean? Let\u2019s say you have a 20% conversion rate for converting regular visitors to paying customers (who have bought something). That\u2019s 47.4 customers you lost.
\nLet\u2019s say your average shopping cart is 80 (EUR, Dollars, \u2026). Basically this means that you have lost 3 792 per day\/month\/\u2026 . Take your statistics and work this out for yourself, or ask your boss to do it. Have I hot your attention?<\/em><\/p>\n<\/div>\n<\/div>\n<\/div>\n
<\/div>\n
\n
\n
\n
Are there other options?<\/h2>\n
You probably already know the answer, so take a look at what we have on our plate for today with Elixir\/Ecto and Postgresql:<\/p>\n
    \n
  1. Similarity<\/li>\n
  2. TS Vectors<\/li>\n
  3. Semantic Search<\/li>\n
  4. Hybrid Search<\/li>\n
  5. 3rd party engines\/providers<\/li>\n<\/ol>\n
    Similarity<\/h2>\n
    It\u2019s a function or logic in Postgresql thanks to the\u00a0pg_trgm<\/em>\u00a0extension, definition from\u00a0docs<\/a>\u00a0is:<\/p>\n
    \n
    The\u00a0pg_trgm<\/code>\u00a0module provides functions and operators for determining the similarity of alphanumeric text based on trigram matching, as well as index operator classes that support fast searching for similar strings.<\/p>\n<\/blockquote>\n
    Ok, but what about trigrams? What are they? It\u2019s important to understand what you\u2019re doing, not just to use a function without knowing what it does. Even more so when it\u2019s not that complicated to understand it.<\/p>\n
    \n
    A trigram is a group of three consecutive characters taken from a string. We can measure the similarity of two strings by counting the number of trigrams they share. This simple idea turns out to be very effective for measuring the similarity of words in many natural languages.<\/p>\n<\/blockquote>\n
    That\u2019s much better but let\u2019s show you some examples to understand it better.<\/p>\n
    CREATE<\/span> EXTENSION IF NOT<\/span> EXISTS<\/span> pg_trgm;<\/span><\/pre>\n
    select<\/span> show_trgm('banana'<\/span>);\r\n=<\/span>><\/span> {  b, ba,ana,ban,na ,nan}\r\n\r\nselect<\/span> show_trgm('effective for measuring the similarity'<\/span>);\r\n=<\/span>><\/span> {  e,  f,  m,  s,  t, ef, fo, me, si, th,ari,asu,cti,eas,ect,eff,fec,ffe,for<\/span>,he ,ila,imi,ing,ity,ive,lar,mea,mil,ng ,or<\/span> ,rin,rit,sim,sur,the,tiv,ty ,uri,ve }<\/span><\/pre>\n
    Basically super simple explanation: it splits text into multiple small texts and compares them with each other and counts how many of these small parts are the same.<\/p>\n
    First, this function just shows how text is split into multiple strings that are compared and ranked for the actual search. Another function is used for the search:<\/p>\n
    select<\/span> name, similarity(name, 'banana'<\/span>)\r\nfrom<\/span> products\r\norder<\/span> by<\/span> 2<\/span> desc<\/span>;\r\n=<\/span>><\/span>\r\n+<\/span>-------------+----------+<\/span>\r\n|<\/span>name         |<\/span>similarity|<\/span>\r\n+<\/span>-------------+----------+<\/span>\r\n|<\/span>Banana Bread |<\/span>0.54545456<\/span>|<\/span>\r\n|<\/span>Banan (lime) |<\/span>0.5<\/span>       |<\/span>\r\n|<\/span>Banana Milk. |<\/span>0.46153846<\/span>|<\/span>\r\n+<\/span>-------------+----------+<\/span><\/span><\/pre>\n
    As you can see, it\u2019s quite simple. Thanks to the ranking you can sort by it, filter because it will show you ranking for everything in the table. So you will also have products with a ranking of 0. It depends on your dataset which sensitivity you want to achieve. Ideally you should try to limit it like this:<\/p>\n
    select<\/span> name, similarity(name, 'banana'<\/span>)\r\nfrom<\/span> products\r\nwhere<\/span> similarity<\/span>(name, 'banana'<\/span><\/span>) > 0.241\r\norder by<\/span> 2 desc<\/span>;<\/span><\/pre>\n
    This allows you to see how many results you will get and which results are at the end of the results. If it makes sense to show them last, try to reduce the number and see results even later. This will help you find the ideal number for you. Don\u2019t be afraid to use a longer number with more decimals.<\/p>\n
    What about Elixir and Ecto? There is nothing special to use if you allow extension in your database.<\/p>\n
    text<\/span> = \"banana\"<\/span>\r\nfrom<\/span>(p in<\/span> Products,\r\n  where<\/span>: fragment(\"similarity(?, ?)\"<\/span>, p.name, ^text<\/span>) > 0.241<\/span>,\r\n  order_by: [desc: fragment(\"similarity(?, ?)\"<\/span>, p.name, ^acc_query)]\r\n)\r\n|> Repo.all()<\/span><\/pre>\n
    Don\u2019t afraid to use similarity on multiple attributes together and combine them to get the best results. Thanks to\u00a0indexes<\/a>\u00a0you can get really fast responses. If you have several attributes or even a table merged with\u00a0UNION<\/code>\u00a0or something like that, you can add constants as weights to lower some results. Because maybe it\u2019s more important to search in the product name than in ingredients, brands, shops, etc.<\/p>\n
    Are we done? Is this the best what we can do for the search? Again, it depends on your dataset and your usage. It may be sufficient, but may also have a lot of false positive search results. Just because some words contain the same trigrams, \u201csmall parts\u201d. It can be enough thanks to speed and a lot of results. But human languages are complex, there are different words with same meaning, changing letters in words like 1 banana and 2 bananas, etc. This can play an important role in search.<\/p>\n
    TS Vectors<\/h2>\n
    CREATE<\/span> EXTENSION IF NOT<\/span> EXISTS<\/span> vector;<\/span><\/pre>\n
    To move forward and a little speed up reading a little, we can simplify it. Search with TS Vectors parses text into tokens based on the language configuration. Easier! Text is parsed into small words based on language, so for example\u00a0cakes<\/code>\u00a0is parsed to\u00a0cake<\/code>\u00a0with English language. Have a look:<\/p>\n
    select<\/span> to_tsvector('english'<\/span>, 'banana cakes for birthday parties'<\/span>);\r\n=<\/span>><\/span> 'banana'<\/span>:1<\/span> 'birthday'<\/span>:4<\/span> 'cake'<\/span>:2<\/span> 'parti'<\/span>:5<\/span><\/span><\/pre>\n
    See how sensitive it is when you change the language:<\/p>\n
    select<\/span> to_tsvector<\/span>('norwegian'<\/span>, 'banana cakes for birthday parties'<\/span><\/span>)<\/span>;\r\n=> 'banan'<\/span>:1<\/span> 'birthday'<\/span>:4<\/span> 'cak'<\/span>:2<\/span> 'parti'<\/span>:5<\/span><\/span><\/pre>\n
    cakes<\/code>\u00a0is not cake but\u00a0cak<\/code>\u00a0because the system used a different dictionary to define the tokens and created a list of lexemes which are similar but different. It\u2019s a simple example but imagine some complications in your language which the change the way the word looks. You know better than me. But look at some examples in other languages:<\/p>\n
    SELECT<\/span> to_tsvector('french'<\/span>, '\u00e9conomie fran\u00e7aise est en croissance rapide.'<\/span>);\r\n=<\/span>><\/span> 'croissanc'<\/span>:5<\/span> 'fran\u00e7ais'<\/span>:2<\/span> 'rapid'<\/span>:6<\/span> '\u00e9conom'<\/span>:1<\/span>\r\n\r\nSELECT<\/span> to_tsvector('german'<\/span>, 'Die wirtschaftliche Lage in Deutschland ist stabil und wachsend.'<\/span>);\r\n=<\/span>><\/span> 'deutschland'<\/span>:5<\/span> 'lag'<\/span>:3<\/span> 'stabil'<\/span>:7<\/span> 'wachsend'<\/span>:9<\/span> 'wirtschaft'<\/span>:2<\/span>\r\n\r\nSELECT<\/span> to_tsvector('english'<\/span>, 'The quick brown fox jumps over the lazy dog.'<\/span>);\r\n=<\/span>><\/span> 'brown'<\/span>:3<\/span> 'dog'<\/span>:9<\/span> 'fox'<\/span>:4<\/span> 'jump'<\/span>:5<\/span> 'lazi'<\/span>:8<\/span> 'quick'<\/span>:2<\/span><\/span><\/pre>\n
    It\u2019s important to say how the results are cleaned up for unimportant words. See the last example with English. All\u00a0the<\/code>\u00a0are ignored. See also examples in other languages. They\u2019re called\u00a0stop words<\/code>\u00a0and are basically discarded to make the search more clearer.<\/p>\n
    Ok first part is done, now the query itself, that\u2019s a little bit complicated. Simple example:<\/p>\n
    select<\/span>  to_tsquery('english'<\/span>,'cooking'<\/span>);\r\n=<\/span>><\/span> 'cook'<\/span><\/span><\/pre>\n
    Nice, but not usable, let\u2019s see in action:<\/p>\n
    select<\/span> name\r\nfrom<\/span> products\r\nwhere<\/span> to_tsvector('english'<\/span>, name) @@ to_tsquery('english'<\/span>, 'cooking'<\/span>);\r\n=<\/span>><\/span>\r\n+<\/span>-------------------------------+<\/span>\r\n|<\/span>name                           |<\/span>\r\n+<\/span>-------------------------------+<\/span>\r\n|<\/span>Beef -<\/span> Cooked, Corned          |<\/span>\r\n|<\/span>Ham -<\/span> Cooked Italian           |<\/span>\r\n|<\/span>Ham -<\/span> Cooked                   |<\/span>\r\n|<\/span>Oil -<\/span> Cooking Spray            |<\/span>\r\n|<\/span>Pork -<\/span> Bacon Cooked Slcd       |<\/span>\r\n|<\/span>Lobster -<\/span> Cooked               |<\/span>\r\n|<\/span>Shrimp -<\/span> 16<\/span> -<\/span> 20<\/span> Cooked, Peeled|<\/span>\r\n+<\/span>-------------------------------+<\/span><\/span><\/pre>\n
    Well, it works but it would be nice to add sorting so that we are sure that the best results are at the top.<\/p>\n
    select<\/span> name, ts_rank_cd(to_tsvector('english'<\/span>, name), to_tsquery('english'<\/span>, 'cooking'<\/span>))\r\nfrom<\/span> products\r\nwhere<\/span> to_tsvector('english'<\/span>, name) @@ to_tsquery('english'<\/span>, 'cooking'<\/span>)\r\norder<\/span> by<\/span> ts_rank_cd(to_tsvector('english'<\/span>, name), to_tsquery('english'<\/span>, 'cooking'<\/span>));\r\n=<\/span>><\/span> \r\n+<\/span>-------------------------------+----------+<\/span>\r\n|<\/span>name                           |<\/span>ts_rank_cd|<\/span>\r\n+<\/span>-------------------------------+----------+<\/span>\r\n|<\/span>Beef -<\/span> Cooked, Corned          |<\/span>0.1<\/span>       |<\/span>\r\n|<\/span>Ham -<\/span> Cooked Italian           |<\/span>0.1<\/span>       |<\/span>\r\n|<\/span>Ham -<\/span> Cooked                   |<\/span>0.1<\/span>       |<\/span>\r\n|<\/span>Oil -<\/span> Cooking Spray            |<\/span>0.1<\/span>       |<\/span>\r\n|<\/span>Pork -<\/span> Bacon Cooked Slcd       |<\/span>0.1<\/span>       |<\/span>\r\n|<\/span>Lobster -<\/span> Cooked               |<\/span>0.1<\/span>       |<\/span>\r\n|<\/span>Shrimp -<\/span> 16<\/span> -<\/span> 20<\/span> Cooked, Peeled|<\/span>0.1<\/span>       |<\/span>\r\n+<\/span>-------------------------------+----------+<\/span><\/span><\/pre>\n
    \n
    This function (ts_rank_cd<\/em><\/code>) computes the cover density ranking for the given document vector and query, as described in Clarke, Cormack, and Tudhope\u2019s \u201cRelevance Ranking for One to Three Term Queries\u201d in the journal \u201cInformation Processing and Management\u201d, 1999. Cover density is similar to\u00a0ts_rank<\/code>\u00a0ranking except that the proximity of matching lexemes to each other is taken into consideration.<\/p>\n<\/blockquote>\n
    I\u2019ve added thets_rank_cd<\/code>\u00a0column just to show you what\u2019s happening. This function uses weights to give a rank to the results. The defaults are<\/p>\n
    {0.1<\/span>, 0.2<\/span>, 0.4<\/span>, 1.0<\/span>}<\/span><\/pre>\n
    Weights are used for labelled words\/vectors. Labeled words, vectors? What? Let\u2019s explain how it works by default.<\/p>\n
    Weights are defined by label, labels are\u00a0{D, C, B, A}<\/code>\u00a0so weight for label\u00a0D<\/code>\u00a0is\u00a00.1<\/code>\u00a0, etc. If weight is not defined for word\/column, label\u00a0D<\/code>\u00a0is used. To make it even simpler, our search term\u00a0cooking<\/code>\u00a0is (after being converted to ts vectors) is only present once in the result column, so the is\u00a01 * 0.1<\/code>. So in this case it would be wise to add another sorting based on your business logic, for example sorting by how many times the product has been bought.<\/p>\n
    However, search only by product name is most often used in conjunction with searching by description.<\/p>\n
    SELECT<\/span> name,\r\n       ts_rank_cd(\r\n               to_tsvector('english'<\/span>, name) ||<\/span>\r\n               to_tsvector('english'<\/span>, description),\r\n               to_tsquery('english'<\/span>, 'cooking'<\/span>)\r\n       ) AS<\/span> rank\r\nFROM<\/span> products\r\nWHERE<\/span> (to_tsvector('english'<\/span>, name) ||<\/span>\r\n       to_tsvector('english'<\/span>, description)) @@ to_tsquery('english'<\/span>, 'cooking'<\/span>)\r\nORDER<\/span> BY<\/span> rank DESC<\/span>;<\/span><\/pre>\n
    This takes vectors for\u00a0name<\/code>\u00a0and\u00a0description<\/code>\u00a0and concat vectors. Or you can use regular\u00a0or<\/code>\u00a0expression:<\/p>\n
    SELECT<\/span> name,\r\n       ts_rank_cd(\r\n               to_tsvector('english'<\/span>, name) ||<\/span>\r\n               to_tsvector('english'<\/span>, description),\r\n               to_tsquery('english'<\/span>, 'cooking'<\/span>)\r\n       ) AS<\/span> rank\r\nFROM<\/span> products\r\nWHERE<\/span> to_tsvector('english'<\/span>, name) @@ to_tsquery('english'<\/span>, 'cooking'<\/span>)\r\n   OR<\/span> to_tsvector('english'<\/span>, description) @@ to_tsquery('english'<\/span>, 'cooking'<\/span>)\r\nORDER<\/span> BY<\/span> rank DESC<\/span>;\r\n=<\/span>><\/span>\r\n+<\/span>---------------------------------+----+<\/span>\r\n|<\/span>name                             |<\/span>rank|<\/span>\r\n+<\/span>---------------------------------+----+<\/span>\r\n|<\/span>Handcrafted Italian Pasta        |<\/span>0.7<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Penne Primavera, Single  |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Cappellini, Dry          |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Tortellini, Fresh        |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Fusili, Dry              |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Penne, Rigate, Dry       |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Fusili Tri -<\/span> Coloured    |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Rotini, Colour, Dry      |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Shells, Medium, Dry      |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Cannelloni, Sheets, Fresh|<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Orzo, Dry                |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Lasagna, Dry             |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Fettuccine, Egg, Fresh   |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Lasagna Noodle, Frozen   |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Linguini, Dry            |<\/span>0.1<\/span> |<\/span>\r\n+<\/span>---------------------------------+----+<\/span><\/span><\/pre>\n
    But it\u2019s important to note that if you use two columns to search, you can have two ranks. There are many ways to combine two ranks. By default you can concat both vectors and do a rank based on that, like in the previous example, so one rank. These ranks are just numbers so you can do whatever your maths skills will allow you.<\/p>\n
    So now we have a search in two columns but it\u2019s more important to show products that match the name better than description. So now we can use this weight.<\/p>\n
    SELECT<\/span> name,\r\n       ts_rank_cd(\r\n           setweight(to_tsvector('english'<\/span>, name), 'D'<\/span>) ||<\/span>\r\n           setweight(to_tsvector('english'<\/span>, description), 'D'<\/span>),\r\n           to_tsquery('english'<\/span>, 'pasta'<\/span>)\r\n       ) AS<\/span> rank\r\nFROM<\/span> products\r\nWHERE<\/span> setweight(to_tsvector('english'<\/span>, name), 'D'<\/span>) @@ to_tsquery('english'<\/span>, 'pasta'<\/span>)\r\n   OR<\/span> setweight(to_tsvector('english'<\/span>, description), 'D'<\/span>) @@ to_tsquery('english'<\/span>, 'pasta'<\/span>)\r\nORDER<\/span> BY<\/span> rank DESC<\/span>\r\nLIMIT 20<\/span>;\r\n=<\/span>><\/span>\r\n+<\/span>---------------------------------+----+<\/span>\r\n|<\/span>name                             |<\/span>rank|<\/span>\r\n+<\/span>---------------------------------+----+<\/span>\r\n|<\/span>Handcrafted Italian Pasta        |<\/span>0.7<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Penne Primavera, Single  |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Cappellini, Dry          |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Tortellini, Fresh        |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Fusili, Dry              |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Penne, Rigate, Dry       |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Fusili Tri -<\/span> Coloured    |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Rotini, Colour, Dry      |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Shells, Medium, Dry      |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Cannelloni, Sheets, Fresh|<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Orzo, Dry                |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Lasagna, Dry             |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Fettuccine, Egg, Fresh   |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Lasagna Noodle, Frozen   |<\/span>0.1<\/span> |<\/span>\r\n|<\/span>Pasta -<\/span> Linguini, Dry            |<\/span>0.1<\/span> |<\/span>\r\n+<\/span>---------------------------------+----+<\/span><\/span><\/pre>\n
    You can see the exact same results. But we want the match with the product name to be at the top and the description to be a little lower.<\/p>\n
    SELECT<\/span> name,\r\n       ts_rank_cd(\r\n               '{0.1, 0.2, 0.9, 1.0}'<\/span>,\r\n               setweight(to_tsvector('english'<\/span>, name), 'A'<\/span>) ||<\/span>\r\n               setweight(to_tsvector('english'<\/span>, description), 'B'<\/span>),\r\n               to_tsquery('english'<\/span>, 'pasta'<\/span>)\r\n       ) AS<\/span> rank\r\nFROM<\/span> products\r\nWHERE<\/span> setweight(to_tsvector('english'<\/span>, name), 'A'<\/span>) @@ to_tsquery('english'<\/span>, 'pasta'<\/span>)\r\n   OR<\/span> setweight(to_tsvector('english'<\/span>, description), 'B'<\/span>) @@ to_tsquery('english'<\/span>, 'pasta'<\/span>)\r\nORDER<\/span> BY<\/span> rank DESC<\/span>\r\nLIMIT 20<\/span>;\r\n=<\/span>><\/span>\r\n+<\/span>---------------------------------+---------+<\/span>\r\n|<\/span>name                             |<\/span>rank     |<\/span>\r\n+<\/span>---------------------------------+---------+<\/span>\r\n|<\/span>Handcrafted Italian Pasta        |<\/span>6.3999996<\/span>|<\/span>\r\n|<\/span>Pasta -<\/span> Penne Primavera, Single  |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Cappellini, Dry          |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Tortellini, Fresh        |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Fusili, Dry              |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Penne, Rigate, Dry       |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Fusili Tri -<\/span> Coloured    |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Rotini, Colour, Dry      |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Shells, Medium, Dry      |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Cannelloni, Sheets, Fresh|<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Orzo, Dry                |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Lasagna, Dry             |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Fettuccine, Egg, Fresh   |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Lasagna Noodle, Frozen   |<\/span>1<\/span>        |<\/span>\r\n|<\/span>Pasta -<\/span> Linguini, Dry            |<\/span>1<\/span>        |<\/span>\r\n+<\/span>---------------------------------+---------+<\/span><\/span><\/pre>\n
    So this query will set weight to name with label\u00a0A<\/code>\u00a0and label\u00a0B<\/code>\u00a0for description. It also sets actual numbers for these labels. The results are the same, but the ranking is different. You can see howthe ranking has changed. But the current dataset doesn\u2019t show big differences.<\/p>\n
    It\u2019s important to have correct data for the search to work correctly.<\/p>\n
    Let\u2019s change the dataset. I found on good old internet (without AI) this dataset with products (bottles, nothing better, sorry) from\u00a0data.world<\/a>.<\/p>\n
    SELECT<\/span> name,\r\n       category,\r\n       ts_rank_cd(\r\n               '{0.1, 0.2, 0.9, 1.0}'<\/span>,\r\n               setweight(to_tsvector('english'<\/span>, name), 'A'<\/span>) ||<\/span>\r\n               setweight(to_tsvector('english'<\/span>, category), 'B'<\/span>),\r\n               to_tsquery('english'<\/span>, 'decanters'<\/span>)\r\n       ) AS<\/span> rank\r\nFROM<\/span> bottle_products\r\nWHERE<\/span> setweight(to_tsvector('english'<\/span>, name), 'A'<\/span>) @@ to_tsquery('english'<\/span>, 'decanters'<\/span>)\r\n   OR<\/span> setweight(to_tsvector('english'<\/span>, category), 'B'<\/span>) @@ to_tsquery('english'<\/span>, 'decanters'<\/span>)\r\nORDER<\/span> BY<\/span> rank DESC<\/span>\r\nLIMIT 20<\/span>;\r\n=<\/span>><\/span> \r\n+<\/span>-------------------------------------------+------------------------------+----+<\/span>\r\n|<\/span>name                                       |<\/span>category                      |<\/span>rank|<\/span>\r\n+<\/span>-------------------------------------------+------------------------------+----+<\/span>\r\n|<\/span>Giori Casino Decanter                      |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Evan Williams Master Select<\/span> Decanter       |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Giori Two Dolphins Decanter                |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Courvoisier Erte Vigne Cognac Decanter 1993<\/span>|<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Giori Octopus Decanter                     |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Giori Butterfly Decanter                   |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Courvoisier Erte Decanter 1994<\/span>             |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Giori Unicorn Decanter                     |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Drambuie Liqueur W\/<\/span>empty<\/span> Decanter          |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Jack Daniels 1954<\/span> Gold Medal Decanter      |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Jack Daniels 1914<\/span> Gold Medal Decanter      |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Jack Daniels 1913<\/span> Gold Medal Decanter      |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Giori Golf Decanter                        |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Jim Beam Black Holiday Decanter            |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>1.9<\/span> |<\/span>\r\n|<\/span>Jack Daniels 1904<\/span> Gold Medal Decanter      |<\/span>TENNESSEE WHISKIES            |<\/span>1<\/span>   |<\/span>\r\n|<\/span>Jack Daniels 1905<\/span> Gold Medal Decanter      |<\/span>TENNESSEE WHISKIES            |<\/span>1<\/span>   |<\/span>\r\n|<\/span>Jack Daniels 150<\/span>th Anniversary Decanter    |<\/span>TENNESSEE WHISKIES            |<\/span>1<\/span>   |<\/span>\r\n|<\/span>Cutty Sark W\/<\/span>decanter                      |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>0.9<\/span> |<\/span>\r\n|<\/span>Cutty Sark Scotch W\/<\/span>ceramic Tumblers       |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>0.9<\/span> |<\/span>\r\n|<\/span>Glenmorangie 10<\/span>yo W\/<\/span>glass                  |<\/span>DECANTERS &<\/span> SPECIALTY PACKAGES|<\/span>0.9<\/span> |<\/span>\r\n+<\/span>-------------------------------------------+------------------------------+----+<\/span><\/span><\/pre>\n
    Nothing big but\u2026 So we search for word\u00a0dencaters<\/code>\u00a0. You can see a better ranking for products containing word\u00a0decanter<\/code>\u00a0in the name and also in the category. So thanks to this we will see all products in this order:<\/p>\n
      \n
    1. with name with search term and products from relevant category,<\/li>\n
    2. with name with search term from other categories,<\/li>\n
    3. other products from the relevant category.<\/li>\n<\/ol>\n
      Let\u2019s see the Elixir code for this:<\/p>\n
      text =<\/span> \"decanters\"\r\nweights =<\/span> \"{0.1, 0.2, 0.9, 1.0}\"\r\nlanguage<\/span> =<\/span> \"english\"\r\nlimit =<\/span> 20<\/span>\r\noffset<\/span> =<\/span> 0<\/span>\r\n\r\nfrom<\/span>(bp in<\/span> BottleProducts,\r\n  where<\/span>:\r\n    fragment(\r\n      \"setweight(to_tsvector(?::text::regconfig', ?), 'A') @@ to_tsquery(?::text::regconfig, ?)\",\r\n      ^<\/span>language<\/span>,\r\n      bp.name,\r\n      ^<\/span>language<\/span>,\r\n      ^<\/span>text\r\n    ) or<\/span>\r\n      fragment(\r\n        \"setweight(to_tsvector(?::text::regconfig', ?), 'B') @@ to_tsquery(?::text::regconfig, ?)\",\r\n        ^<\/span>language<\/span>,\r\n        bp.category,\r\n        ^<\/span>language<\/span>,\r\n        ^<\/span>text\r\n      ),\r\n  limit: ^<\/span>limit,\r\n  offset<\/span>: ^<\/span>offset<\/span>,\r\n  order_by: [\r\n    desc<\/span>:\r\n      fragment(\r\n        \"ts_rank_cd(?, setweight(to_tsvector(?::text::regconfig, ?), 'A') || setweight(to_tsvector(?::text::regconfig, ?), 'B'))\",\r\n        ^<\/span>weights,\r\n        ^<\/span>language<\/span>,\r\n        bp.name,\r\n        ^<\/span>language<\/span>,\r\n        bp.category\r\n      )\r\n  ]\r\n)\r\n|<\/span>><\/span> Repo.all()<\/span><\/pre>\n
      It\u2019s a pretty long query and basically all the logic is in fragment where we have raw SQL. There is no other way to do it. You can write some macros which will show it much nicer in case you will use it on multiple places. Maybe there are some libraries for Ecto but basically it will just be a wrapper around fragment as in our case.<\/p>\n
      This type of query does a lot of computation and it\u2019s possible to optimise it with a better query or simply by creating indexes. For example:<\/p>\n
      CREATE<\/span> INDEX idx_bottle_products_name_tsvector ON<\/span> bottle_products\r\nUSING<\/span> gin(to_tsvector('english'<\/span>, name));<\/span><\/pre>\n
      If you search with only one column:<\/p>\n
      SELECT<\/span> name,\r\n       category,\r\n       ts_rank_cd(\r\n               to_tsvector('english'<\/span>, name),\r\n               to_tsquery('english'<\/span>, 'decanters'<\/span>)\r\n       ) AS<\/span> rank\r\nFROM<\/span> bottle_products\r\nWHERE<\/span> to_tsvector('english'<\/span>, name) @@ to_tsquery('english'<\/span>, 'decanters'<\/span>)\r\nORDER<\/span> BY<\/span> rank DESC<\/span>\r\nLIMIT 20<\/span>;<\/span><\/pre>\n
      I got results in 50 to 80ms without index and 20 to 30ms with index. It\u2019s important to say that you create an index for specific language. So if you have records that are in multiple languages, you need to create an index for each language. You can have multiple indexes for the same column. You just add a condition at the end. For example:<\/p>\n
      CREATE<\/span> INDEX idx_bottle_products_name_english_tsvector ON<\/span> bottle_products\r\nUSING<\/span> gin(to_tsvector('english'<\/span>, name)) where<\/span> language_id =<\/span> 1<\/span>;\r\n\r\nCREATE<\/span> INDEX idx_bottle_products_name_german_tsvector ON<\/span> bottle_products\r\nUSING<\/span> gin(to_tsvector('german'<\/span>, name)) where<\/span> language_id =<\/span> 2<\/span>;<\/span><\/pre>\n
      When you do search with these ts vectors with multiple columns it can be a little bit difficult to make a proper and fast query. Personally, I think that similarity shows better results than ts vectors, so I haven\u2019t gone deeper. But again, it\u2019s very dependent on your data. It may be much better in your cases.<\/p>\n
      In real life customers, visitors, users don\u2019t search with just one word. Usually two or sometimes even more.<\/p>\n
      SELECT<\/span> name,\r\n       category,\r\n       ts_rank_cd(\r\n               to_tsvector('english'<\/span>, name),\r\n               to_tsquery('english'<\/span>, 'decanters bottle'<\/span>)\r\n       ) AS<\/span> rank\r\nFROM<\/span> bottle_products\r\nWHERE<\/span> to_tsvector('english'<\/span>, name) @@ to_tsquery('english'<\/span>, 'decanters bottle'<\/span>)\r\nORDER<\/span> BY<\/span> rank DESC<\/span>\r\nLIMIT 20<\/span>;\r\n=<\/span>><\/span> \r\nERROR: syntax error in<\/span> tsquery: \"decanters bottle\"<\/span><\/pre>\n
      That\u2019s because\u00a0to_tsquery<\/code>\u00a0allows you to use a single word or query. This will be a bit wild. You have several options what to do. You can see a lots of examples in the\u00a0documentation<\/a>:<\/p>\n
      \n
      tsquery<\/code>\u00a0operators\u00a0&<\/code>\u00a0(AND),\u00a0|<\/code>\u00a0(OR),\u00a0!<\/code>\u00a0(NOT), and\u00a0<-><\/code>\u00a0(FOLLOWED BY)<\/p>\n<\/blockquote>\n
      Let\u2019s look at two basic examples:<\/p>\n
      to_tsquery('english'<\/span>, 'decanters | bottle'<\/span>)\r\n=> decanters OR<\/span> bottle\r\nto_tsquery('english'<\/span>, 'decanters & bottle'<\/span>)\r\n=> decanters AND<\/span> bottle\r\n<\/span><\/pre>\n
      To move this problem to real code, let\u2019s look at the updated function for making query:<\/p>\n
      text = \"decanters bottle\"<\/span> |> String.split(\" \") <\/span>|> Enum.join(\" & \"<\/span>)\r\nweights = \"{0.1, 0.2, 0.9, 1.0}\"<\/span>\r\nlanguage = \"english\"<\/span>\r\nlimit = 20<\/span>\r\noffset = 0<\/span>\r\n\r\nfrom(bp in<\/span> BottleProducts,\r\n  where:<\/span>\r\n    fragment(\r\n      \"setweight(to_tsvector(?::text::regconfig', ?), 'A') @@ to_tsquery(?::text::regconfig, ?)\"<\/span>,\r\n      ^language,\r\n      bp.name,\r\n      ^language,\r\n      ^text\r\n    ) or<\/span>\r\n      fragment(\r\n        \"setweight(to_tsvector(?::text::regconfig', ?), 'B') @@ to_tsquery(?::text::regconfig, ?)\"<\/span>,\r\n        ^language,\r\n        bp.category,\r\n        ^language,\r\n        ^text\r\n      ),\r\n  limit:<\/span> ^limit,\r\n  offset:<\/span> ^offset,\r\n  order_by:<\/span> [\r\n    desc:<\/span>\r\n      fragment(\r\n        \"ts_rank_cd(?, setweight(to_tsvector(?::text::regconfig, ?), 'A') || setweight(to_tsvector(?::text::regconfig, ?), 'B'))\"<\/span>,\r\n        ^weights,\r\n        ^language,\r\n        bp.name,\r\n        ^language,\r\n        bp.category\r\n      )\r\n  ]\r\n)\r\n|> Repo.all()<\/span><\/span><\/pre>\n
      That\u2019s a very small change:<\/p>\n
      text = \"decanters bottle\"<\/span> |> String.split(\" \") <\/span>|> Enum.join(\" & \"<\/span>)<\/span><\/pre>\n
      But it\u2019s not good because:<\/p>\n
        \n
      1. input can be different and it can lead to wrong query<\/li>\n
      2. it assume we want to use only & operator<\/li>\n<\/ol>\n
        Let\u2019s see the first problem:<\/p>\n
         \"decanters   bottle\"<\/span> |> String<\/span>.split(\" \"<\/span>) |> Enum<\/span>.join<\/span>(\" & \"<\/span>)\r\n\"decanters &  &  & bottle\"<\/span><\/span><\/pre>\n
        How can we fix it? It can be complex and handle more use cases but this is very simple solution:<\/p>\n
         \"decanters   |bottle & whisky  \"<\/span>\r\n    |> String<\/span>.trim()\r\n    |> String<\/span>.downcase()\r\n    |> String<\/span>.replace(\"\\&\"<\/span>, \"\"<\/span>)\r\n    |> String<\/span>.replace(\"\\|\"<\/span>, \"\"<\/span>)\r\n    |> String<\/span>.replace(~r\/\\s+\/, \" \"<\/span>)\r\n    |> String<\/span>.split(\" \"<\/span>)\r\n    |> Enum<\/span>.join<\/span>(\" & \"<\/span>)\r\n\r\n\"decanters & bottle & whisky\"<\/span><\/span><\/pre>\n
        The point is to normalise the input, remove the chars which the query uses and do the actual query with word parsing. It\u2019s very simple, probably can be done with one RegEx or another way. It\u2019s up to you to make it work with your logic and skills.<\/p>\n
        It\u2019s important to mention why I made\u00a0String.downcase<\/code>. It\u2019s not necessary but it\u2019s nice to use some kind of common format and in the case of using caching search results, it can save a lot of hints when visitors enter something with capital letters.<\/p>\n
        The second problem is more complex, because if the search term consists of several words, we assume that it\u2019s an \u201cand\u201d operator. So each search result must contain all the words. It\u2019s very complex and really depends on the use of this search. I\u2019m not going to write code on how to do it, because basically it can be done in a whole new article. I think there are a lot of articles about parsing queries and making logic. Very simple code which I suggest not to use in production, can be something like this:<\/p>\n
        \"decanters  or whisky\"<\/span>\r\n    |> String<\/span>.trim()\r\n    |> String<\/span>.downcase()\r\n    |> String<\/span>.replace(\"\\&\"<\/span>, \"\"<\/span>)\r\n    |> String<\/span>.replace(\"\\|\"<\/span>, \"\"<\/span>)\r\n    |> String<\/span>.replace(~r\/\\s+\/, \" \"<\/span>)\r\n    |> String<\/span>.split(\" \"<\/span>)\r\n    |> Enum<\/span>.join<\/span>(\" & \"<\/span>)\r\n    |> String<\/span>.replace(\"& or &\"<\/span>, \"|\"<\/span>)\r\n\r\n\"decanters | whisky\"<\/span><\/span><\/pre>\n
        It will only work with one operator, when you enter multiple operators it may cause create logical problems.<\/p>\n
        \"decanters  or whisky or glass and water\"<\/span>\r\n    |> String<\/span>.trim()\r\n    |> String<\/span>.downcase()\r\n    |> String<\/span>.replace(\"\\&\"<\/span>, \"\"<\/span>)\r\n    |> String<\/span>.replace(\"\\|\"<\/span>, \"\"<\/span>)\r\n    |> String<\/span>.replace(~r\/\\s+\/, \" \"<\/span>)\r\n    |> String<\/span>.split(\" \"<\/span>)\r\n    |> Enum<\/span>.join<\/span>(\" & \"<\/span>)\r\n    |> String<\/span>.replace(\"& or &\"<\/span>, \"|\"<\/span>)\r\n    |> String<\/span>.replace(\"& and &\"<\/span>, \"&\"<\/span>)\r\n\r\n\"decanters | whisky | glass & water\"<\/span><\/span><\/pre>\n
        You may have the problem that you will have products with similar names but really different meanings. For example chocolate and plate in chocolate colour. Similarity search will find similar parts and assume it\u2019s similar, ts vectors as well. Maybe it will be lower in results but still. It will show plates when customer want to buy chocolate for girl.<\/p>\n
        Semantic Search<\/h2>\n
        Big player in the game. Semantic search. Sounds modern, sounds like something with AI. Well, we can easily simplify it or define it as \u201cusing the meaning of words to search\u201d. People, especially developers, realised a very long time ago that search uses human languages which are much more complex and not as precise as programming languages. You can read more about the history of \u201cusing meaningful search\u201d in this nice article from Algolia (a really good 3rd party search provider):<\/p>\n
        \n
        \"\"<\/a><\/p>\n<\/div>\n
        You can read in the article AI search in 2007, vector search in 2013. So nothing to do with current (2024) hype about AI. These technologies are so old that they can already walk to school alone without their parents. But they still need to be guided and shown the right way to live.<\/p>\n
        It\u2019s important to mention that Semantic Search it\u2019s a slightly broader concept, not anexact way how to do it. Main goal is to use tools that take meanings of what search term want to achieve and execute it with understanding of content and context with relevant data.<\/p>\n
        For this purpose I will show the vector search. PostgreSQL is wondeful for using real data with vectors thanks to the\u00a0pgvector<\/code>\u00a0extension.<\/p>\n
        \n
        \n
        \n
        \n
        \"\"<\/a><\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
        The definition what is a vector, or vector search is quite complex. I\u2019m not the right guy to explain it properly. I will give you just very simple definition which I used to explain my non-technical colleagues.<\/p>\n
        First of all, the following visualisation of the vectors is really good.<\/p>\n
        \n
        \n
        \"Words<\/a><\/div>\n<\/div>
        Words plotted in 3-dimensional space. Embeddings can have hundreds or thousands of dimensions \u2014 too many for humans to visualise.<\/figcaption><\/figure>\n
        I borrowed this image from already mentioned Algolia article, and they borrowed it from this article:<\/p>\n
        \n
        \n
        \n
        \"\"<\/a><\/h3>\n<\/div>\n<\/div>\n<\/div>\n
        You can see words and connections between them in this picture. It\u2019s in 3-dimensional space. Each dimension has its own meaning. It can be distance from North Pole, or 0 can be black 1 can be white and all other colours can be in between. It really depends on the author of this \u201cspace\u201d. These spaces are called embedding models. I explain it to non-technical people as \u201cGPS\u201d coordinates in fake word. Where vectors are several numbers similar to latitude and longitude in our world. And model is map. So basically vectors are coordinates on map. Yes, it\u2019s not correct and my university teachers would probably try to physically attack me for that definition, but it\u2019s as it is, I\u2019m a practical guy.<\/p>\n
        Nice example is model from OpenAI. They have model\u00a0text-embedding-3-large<\/em><\/strong>\u00a0with size of 3072 dimensions. This means that if you put some text into this model, it will return you an array\/list of 3072 numbers. These numbers are basically coordinates in their world named \u201ctext-embedding-3-large\u201d. They have other models too. You can read more about how this model is created and some comparisons in their\u00a0article<\/a>.<\/p>\n
        So let\u2019s move on to the practical and technical. Using vectors for search consists of two parts:<\/p>\n
          \n
        1. Obtaining and storing Vectors<\/li>\n
        2. Calculations with Vectors<\/li>\n<\/ol>\n
          Obtaining and storing Vectors<\/h3>\n
          First you need to know your data. Getting the right vectors is very important. It\u2019s different for getting vectors for articles and for products or small text or big text. The problem with big or huge text like book or article is that it can contain a lot of meanings. One part of article can be about bad experience with A and second part can be about good experience with B. So embedding model will have hard work to define which vectors to use, where is the position on every dimension. Or product description can also contain tutorial how to cook with it. Thanks to this, the model can make a decision and understand that the meaning of this content is about this meal not about the product.<\/p>\n
          Context<\/strong><\/p>\n
          For example for articles, books, documentation or basically any large text. It\u2019s common practice to split text into several blocks and vectorize only those blocks. You will get better vectors thanks to smaller possible meanings and you can also address results to more details. For example instead of showing the result that your topic is mentioned in book A, you can show the result that your topic is mentioned in book A in chapter 23, paragraph 5.<\/p>\n
          If you have a product with description, you need to make sure that you don\u2019t have any directly related text in it. How to cook with this product, history of the company that made this product, or whatever is not directly related to product itself. It\u2019s important because we\u2019re not in the state of models\/AI (2024) where it can automatically define which context is main and important and which is irrelevant and should be ignored. It\u2019s possible but you have to take the right steps first. For example you can take only a few of sentences from the description or use summarised description if you have one. Or you can use Other AI models to generate the right description from your big description. It\u2019s important to focus on the right context.<\/p>\n
          Sometimes it\u2019s not about about removing text, It\u2019s about adding detail. Imagine you are an online shop for coffee. You have names of all products, brands, etc\u2026 You have a good search for products. But you also want to add a search for brands. So you want to use brand name and description, maybe location. Great. It will work, but it can work better. Imagine that some brands don\u2019t just sell coffee, but also equipment to make coffee, some other ingredients or specialise in some special categories of coffee. So , for example, adding the top 5 categories of products that this brand sells in your store will help create a better context for the model to decide the right vectors.<\/p>\n
          Important note! Do not forget to remove html or other tags from your text. They\u2019re not needed to get proper vectors and can waste some potential.<\/p>\n
          Even more context<\/strong><\/p>\n
          So we have proper data, big text is broken down to small, product contains only directly relevant data, brands contains top categories. Great, are we done with proper context? NO!<\/p>\n
          Imagine that you just met some random friend and you tell that person:<\/p>\n