Source code for advertools.word_frequency

.. _word_frequency:

Text Analysis

Absolute and Weighted Word Count

When analyzing a corpus of documents (I'll simply call it a text list), one of
the main tasks to accomplish to start text mining is to first count the words.
While there are many text mining techniques and approaches, the
:func:`word_frequency` function works mainly by counting words in a text list.
A "word" is defined as a sequence of characters split by whitespace(s), and
stripped of non-word characters (commas, dots, quotation marks, etc.).
A "word" is actually a phrase consisting of one word, but you have the option
of getting phrases that have two words, or more. This can be done simply
by providing a value for the :attr:`phrase_len` parameter.

Absolute vs Weighted Frequency

In social media reports, analytics, keyword reports, url and page reports, we
get more information than simply the text. We get numbers describing those
posts or page titles, or product names, or whatever the text list might
contain. Numbers can be pageviews, shares, likes, retweets, sales, bounces,
sales, etc. Since we have numbers to quantify those phrases, we can improve our
counting by taking into consideration the number list that comes with the text

For example, if you have an e-commerce site that has two products, let's say
you have bags and shoes, then your products are split 50:50 between bags and
But what if you learn that shoes generate 80% of your sales?
Although shoes form half your products, they generate 80% of your revenue. So
the *weighted count* of your products is 80:20.

Let's say two people post two different posts on a social media platform. One
of them says, "It's raining", and the other says, "It's snowing". As in the
above example, the content is split 50:50 between "raining" and "snowing", but
we get a much more informative picture if we get the number of followers of
each of those accounts (or the number of shares, likes, etc.). If one of them
has a thousand followers, and other has a million (which is typical on social
media, as well as in pageviews report, e-commerce and most other datasets),
then you get a completely different picture about your dataset.

These two simple examples contain two posts, and a word each. The
:func:`word_frequency` function can provide insight on hidden trends especially
in large datasets, and when the sentences or phrases are also longer then a
word or two each.

Let's take a look at how to use the :func:`word_frequency` function, and what
the available parameters and options are.

.. glossary::

        The list of phrases or documents that you want to analyze. Here are
        some possible ideas that you might use this for:

            * keywords, whether in a PPC or SEO report
            * page titles in an analytics report
            * social media posts (tweets, Facebook posts, YouTube video titles
              or descriptions etc.)
            * e-commerce reports (where the text would be the product names)

        Ideally, if you have more than one column describing :attr:`text_list`
        you should experiment with different options. Try weighting the words
        by pageviews, then try by bounce rate and see if you get different
        interesting findings.
        With e-commerce reports, you can see which word appears the most, and
        which word is associated with more revenue.

        You should also experiment with different lengths of phrases. In many
        cases, one-word phrases might not be as meaningful as two-words or

        The default is to simply split words by whitespace, and provide phrases
        of length :attr:`phrase_len`.
        But you may want to count the occurrences of certain patterns of text.
        Check out the :ref:`regex <regex>` module for the available regular
        expressions that might be interesting. Some of the pre-defined ones are
        hashtags, mentions, questions, emoji, currencies, and more.

        A list of words to remove and ignore from the count. Known as
        stop-words these are the most frequently used words in a language,
        the most used, but don't add much meaning to the content (a, and, of,
        the, if, etc.). By default a set of English stopwords is provided
        (which you can check and possibly may want to modify), or run
        ``adv.stopwords.keys()`` to get a list of all the available stopwords
        in the available languages.
        In some cases (like page titles for example), you might get "words"
        that need to be removed as well, like the pipe "|" character for

        The returned DataFrame contains the default columns
        ``[word, abs_freq, wtd_freq, rel_value]``. You can get extra
        columns for percentages and cumulative percentages that add perspective
        to the other columns. Set this parameter to :attr:`True` if you want

Below are all the columns of the returned DataFrame:

=========================  ================================================
:attr:`word`               Words in the document list each on its own row.
                           The length of these words is determined by
                           :attr:`phrase_len`, essentially phrases if
                           containing more than one word each.
:attr:`abs_freq`           The number of occurrences of each word in all
                           the documents.
:attr:`wtd_freq`           Every occurrence of :attr:`word` multiplied by
                           its respective value in :attr:`num_list`.
:attr:`rel_value`          :attr:`wtd_freq` divided by :attr:`abs_freq`,
                           showing the value per occurrence of :attr:`word`
:attr:`abs_perc`           Absolute frequency percentage.
:attr:`abs_perc_cum`       Cumulative absolute percentage.
:attr:`wtd_freq_perc`      Weighted frequency percentage.
:attr:`wtd_freq_perc_cum`  Cumulative weighted frequency percentage.
=========================  ================================================

.. thebe-button::
    Run this code

.. code-block::
    :class: thebe, thebe-init

    import advertools as adv
    import pandas as pd
    tweets = pd.read_csv('data/tweets.csv')

====  ================================================================================================================================================  =================
  ..  tweet_text                                                                                                                                          followers_count
====  ================================================================================================================================================  =================
   0  @AERIALMAGZC @penguinnyyyyy you won't be afraid if I give you a real reason :D                                                                                  157
   1  Vibing in the office to #Metallica when the boss is on a coffee break                                                                                          4687
   2  I feel like Ann says she likes coffee and then gets drinks that are 99% sugar and 1% coffee                                             104
   3  A venti iced coffee with four pumps of white mocha, sweet cream and caramel drizzle might just be my new favorite drink. Shout out to TikTok lol                126
   4  I was never a coffee person until I had kids. β˜•οΈ this cup is a life saver.                                                             1595
   5  Who's excited about our next Coffee Chat? We know we are!πŸ₯³                                                                                                    5004

      We're also adding Representative John Bradford to this lineup to discuss redistricting in the area. You won't want to miss it!

      Join the meeting:
   6  he paid for my coffee= husbandπŸ’—                                                                                                                                165
   7  It's nipply outside, and now I side too :)                                                                                                                        0
      That sounds like blowjob in front of a fire and visit with coffee after :)
      I'm still out of coffee
      I could have green tea instead
      I want to spend the morning pampering you ...
   8  Good morning πŸ˜ƒπŸŒžβ˜€οΈ I hope everyone has a great Tuesday morning. Enjoy your day and coffee β˜•οΈ β™₯οΈβ€οΈπŸ’•πŸ₯°πŸ˜˜                                                           189
   9  @MarvinMilton2 I nearly choked on my coffee πŸ€ͺ                                                                                                                 1160
====  ================================================================================================================================================  =================

.. thebe-button::
    Run this code

.. code-block::
    :class: thebe, thebe-init

    word_freq = adv.word_frequency(text_list=tweets['tweet_text'],

    # try sorting by 'abs_freq', 'wtd_freq', and 'rel_value':
    word_freq.sort_values(by='abs_freq', ascending=False).head(25)

"""  # noqa: E501

import re
from collections import defaultdict

import pandas as pd

import advertools as adv
from advertools.word_tokenize import word_tokenize

[docs] def word_frequency( text_list, num_list=None, phrase_len=1, regex=None, rm_words=adv.stopwords["english"], extra_info=False, ): """Count the absolute as well as the weighted frequency of words in :attr:`text_list` (based on :attr:`num_list`). Parameters ---------- text_list : list Typically short phrases, but could be any list of full blown documents. Usually, you would use this to analyze tweets, book titles, URLs, etc. num_list : list A list of numbers with the same length as :attr:`text_list`, describing a certain attribute of these 'documents'; views, retweets, sales, etc. regex : str The regex used to split words. Doesn't need changing in most cases. phrase_len : int The length in words of each token the text is split into (ngrams), defaults to 1. rm_words : set Words to remove from the list a.k.a 'stop-words'. The default uses. To get all available languages run ``adv.stopwords.keys()``. extra_info : bool Whether or not to give additional metrics about the frequencies. Returns ------- abs_wtd_df : pandas.DataFrame Absolute and weighted counts DataFrame. Examples -------- >>> import advertools as adv >>> text_list = ['apple orange', 'apple orange banana', ... 'apple kiwi', 'kiwi mango'] >>> num_list = [100, 100, 100, 400] >>> adv.word_frequency(text_list, num_list) word abs_freq wtd_freq rel_value 0 kiwi 2 500 250.0 1 mango 1 400 400.0 2 apple 3 300 100.0 3 orange 2 200 100.0 4 banana 1 100 100.0 Although "kiwi" occurred twice :attr:`abs_freq`, and "apple" occurred three times, the phrases in which "kiwi" appear have a total score of 500, so it beats "apple" on :attr:`wtd_freq` even though "apple" wins on :attr:`abs_freq`. You can sort by any of the columns of course. :attr:`rel_value` shows the value per occurrence of each word, as you can see, it is simply obtained by dividing :attr:`wtd_freq` by :attr:`abs_freq`. >>> adv.word_frequency(text_list) # num_list values default to 1 each word abs_freq wtd_freq rel_value 0 apple 3 3 1.0 1 orange 2 2 1.0 2 kiwi 2 2 1.0 3 banana 1 1 1.0 4 mango 1 1 1.0 >>> text_list2 = ['my favorite color is blue', ... 'my favorite color is green', 'the best color is green', ... 'i love the color black'] Setting :attr:`phrase_len` to 2, "words" become two-word phrases instead. Note that we are setting :attr:`rm_words` to the empty list so we can keep the stopwords and see if that makes sense: >>> word_frequency(text_list2, phrase_len=2, rm_words=[]) word abs_freq wtd_freq rel_value 0 color is 3 3 1.0 1 my favorite 2 2 1.0 2 favorite color 2 2 1.0 3 is green 2 2 1.0 4 is blue 1 1 1.0 5 the best 1 1 1.0 6 best color 1 1 1.0 7 i love 1 1 1.0 8 love the 1 1 1.0 9 the color 1 1 1.0 10 color black 1 1 1.0 The same result as above showing all possible columns by setting :attr:`extra_info` to :attr:`True`: >>> adv.word_frequency(text_list, num_list, extra_info=True) word abs_freq abs_perc abs_perc_cum wtd_freq wtd_freq_perc wtd_freq_perc_cum rel_value 0 kiwi 2 0.222222 0.222222 500 0.333333 0.333333 250.0 1 mango 1 0.111111 0.333333 400 0.266667 0.600000 400.0 2 apple 3 0.333333 0.666667 300 0.200000 0.800000 100.0 3 orange 2 0.222222 0.888889 200 0.133333 0.933333 100.0 4 banana 1 0.111111 1.000000 100 0.066667 1.000000 100.0 """ # noqa: E501 if num_list is None: num_list = [1 for _ in range(len(text_list))] if isinstance(regex, str): regex = re.compile(regex) text_list = [" ".join(regex.findall(text)) for text in text_list] word_freq = defaultdict(lambda: [0, 0]) for text, num in zip(word_tokenize(text_list, phrase_len=phrase_len), num_list): for word in text: if word.lower() in rm_words: continue word_freq[word.lower()][0] += 1 word_freq[word.lower()][1] += num columns = ["abs_freq", "wtd_freq"] abs_wtd_df = ( pd.DataFrame.from_dict(word_freq, orient="index", columns=columns) .sort_values("wtd_freq", ascending=False) .assign(rel_value=lambda df: df["wtd_freq"] / df["abs_freq"]) .round() ) if extra_info: abs_wtd_df.insert( 1, "abs_perc", value=abs_wtd_df["abs_freq"] / abs_wtd_df["abs_freq"].sum() ) abs_wtd_df.insert(2, "abs_perc_cum", abs_wtd_df["abs_perc"].cumsum()) abs_wtd_df.insert( 4, "wtd_freq_perc", abs_wtd_df["wtd_freq"] / abs_wtd_df["wtd_freq"].sum() ) abs_wtd_df.insert(5, "wtd_freq_perc_cum", abs_wtd_df["wtd_freq_perc"].cumsum()) abs_wtd_df = abs_wtd_df.reset_index().rename(columns={"index": "word"}) if set(num_list) == {1}: abs_wtd_df = abs_wtd_df.drop(["wtd_freq", "rel_value"], axis=1) return abs_wtd_df