Comprehensive Guide to Text Analytics for Market Research

This comprehensive guide to text analytics for market research covers everything from thematic coding to sentiment analysis and - most importantly - how to implement it effectively. It is useful for anyone looking to extract insights from text and should show you that getting started with text analytics doesn't have to be difficult.

If you are still curious or want more advanced examples, check out Displayr's resources to keep learning. 

Introduction to text analytics

Text analytics is the process of analyzing large amounts of text and presenting insights. Advancements in technology mean it is now not just about detecting keywords and patterns, but striving to understand your text and the meaning behind it.

Businesses collect large volumes of text daily, whether through social media comments, customer feedback, employee engagement surveys, emails, or text messages. As more text comes through the door, these businesses need a way to analyze the information quickly and gather insights.

The days of humans taking the time to code text manually have long passed. Text analytics tools have made it possible to automate this process and even provide recommendations on how to action customer feedback.

How does it work?

Many different technological components allow text analytics to be effective. One of the most important is natural language processing (NLP). NLP is a field within artificial intelligence that utilizes machine learning techniques to enable computers to comprehend human language. In text analytics, NLP serves as an essential resource for refining unstructured text data and preparing it for analysis. This capability empowers market researchers to examine larger volumes of text data with enhanced precision.

Machine learning (ML) is another branch of AI that helps power text analytics using algorithmic models. By training on labeled datasets, these models learn to recognize patterns and make predictions, such as identifying sentiments or extracting key entities. This is how text analytics can be scaled to efficiently handle large volumes of unstructured text, ultimately improving the accuracy and depth of insights derived from the data.

Examples of text analytics techniques in market research

Although text analytics is singularly focused on deriving insights from text, there are a number of different ways to do so. Below are some common examples of market research text analytics techniques.

• Information extraction involves collecting entities, attributes, and their relationships for subsequent analysis.
• Information retrieval refers to the use of relevant keywords and patterns to observe and analyze user behavior.
• Text categorization entails assigning various words and phrases to specific predefined categories.
• Clustering involves grouping entire documents that share similar content.
• Visualization uses graphical representations, such as word clouds, to show the density of a category.
• Summarization focuses on condensing a document by highlighting its essential details.

Sentiment analysis

Sentiment analysis is all about evaluating the emotional tone behind customer feedback. Think online surveys, social media interactions, and even customers' exchanges with chatbots. Put simply, sentiment analysis classifies the sentiment of specific text as positive, negative, or neutral.

It is a way for businesses to gain an overview of their customer's experience with the brand. Advancements in artificial intelligence mean sentiment analysis can now be used to process large volumes of data in just minutes. Key applications of sentiment analysis include;

• Brand monitoring
• Competitive analysis
• Employee engagement assessment
• Product evaluation

Despite its benefits, sentiment analysis does have some drawbacks. The complexities of human language can lead to misunderstandings, particularly in recognizing sarcasm or ambiguous terms, even as AI's accuracy improves. Additionally, sentiment analysis is relatively one-dimensional. It primarily categorizes text as positive, negative, or neutral - without exploring the underlying reasons for these sentiments.

Thematic coding

Thematic coding (a.k.a. thematic analysis) involves examining text to uncover recurring themes. Unlike sentiment analysis, thematic coding can categorize different pieces of text under the same theme, even when their sentiments differ.

As such, thematic coding and sentiment analysis should be viewed as complementary approaches - not an either-or. By integrating both techniques, market researchers can identify key themes while also gauging the emotions associated with them. Together, these methods can provide valuable insights into KPIs like customer retention and Net Promoter Score (NPS).

A word on context

Regardless of the study, context is always going to be an important factor in determining the success of a text analytics project. Context is what makes research insightful and shows us the text's true meaning.

While advances in NLP and ML mean AI-powered text analytics is much better at picking up context than previous iterations, it is still not perfect. More often than not, when there is an error in text analysis, it is because context has not been taken into account at some stage. When supervising these automated systems (more on that later), it is essential to be on the lookout for contextual errors that might have skewed the results.

How text analytics helps market research

For market researchers, text analysis offers countless opportunities to find new insights and deliver value for clients. Some common market research examples of text analysis include;

• Extracting key themes from focus group sessions using categorization
• Analyzing competitors' brand messaging and popularity online
• Whitespace analysis using reviews of existing products
• Identifying regional differences in customer's perception of a brand/product
• Identifying upcoming popular trends

Many of these use cases rely on publicly accessible data, such as social media posts and online reviews. This means text analysis is a great starting point for any market researcher.   

A step-by-step guide for market researchers

Note: This is just one potential use case for text analysis. There are a variety of different ways to analyze text. Selecting the right method is going to depend on your project, team, and the skills and tools you have available to you.

Step 1: Define your goals

Before you start thinking about how you are going to perform your text analysis, you should understand why you are doing it. Sit down with every stakeholder and define the objectives.

If the goal is for a business to improve its customer service, then it could be worth conducting a sentiment analysis based on Google Reviews. If the goal is to see how frequently a brand name appears online, Named Entity Recognition could work. By defining clear goals, you can start to think about what techniques you will use to perform your text analysis.

Step 2: Gather data

Once you have a clear idea of the objective, it should be reasonably easy to identify the data you need. On the other hand, gathering and collating this data can prove more challenging. You may find all the relevant data is mixed in with a trove of irrelevant data, or it could be dispersed into different languages.

These issues can be smoothed out in the data cleaning process (keep reading!), but it is worth being prepared for some ups and downs. It is also essential to collect diverse data that accurately reflects the entire subset of people (i.e., customers, employees, etc) you are trying to understand.

Step 3: Clean and preprocess data

Text data must be thoroughly cleaned and preprocessed before it can be successfully analyzed. This means removing irrelevant content, standardizing the format (e.g., making it all lowercase), tokenizing the text, removing stop words, and performing stemming and lemmatization to reduce words to their base form.

Be wary of this step, as this is often where errors occur. Although these techniques are in place to improve the overall quality of your analysis, there are some cases where they could adjust the meaning of specific words (particularly when stemming and lemmatizing) and, as a result, impact the analysis.

Step 4: Categorize your text

Text categorization is a fantastic starting point in any text analysis project. By organizing your text into predefined categories, you can start to see some themes emerge as you begin your research. Displayr AI offers automated text categorization by quickly sorting open-ended and textual responses into clear and meaningful groups, then comes up with category names based on the ideas and intentions in the text. This makes analyzing and understanding the story in your text data even easier. Using Displayr, you can easily add categorized text to your reports and visualizations.

Step 5: Perform sentiment analysis

Once the text has been categorized, it can be further analyzed through sentiment analysis. As mentioned earlier, sentiment analysis involves classifying text as positive, negative, or neutral and is a great tool for providing a snapshot of your customer's (or anyone else's) mood.

By performing sentiment analysis on categorized text, you can identify patterns. For example, suppose a business categorizes online reviews by different products/services. They can see how these different items impact overall customer sentiment, using sentiment analysis, and identify product/service-specific changes to enhance the overall experience.

By integrating these two approaches, market researchers can both uncover themes within the text and gain insights into the underlying emotions. Merging these methods is a valuable strategy when it comes to understanding metrics like customer churn and Net Promoter Score (NPS).

Step 6: Visualize data

Creating a visual representation of your findings is an effective way of telling the story behind your data. One particularly effective visualization for text analysis is a word cloud. A word cloud is, as it sounds, a visual collection of the different words and phrases that have been captured in the survey, with the size of the words corresponding with the frequency in which that word was mentioned. Word clouds are effective and easy to make, providing an engaging visual data summary. You can find a how-to guide on building word clouds in Displayr.

Step 7: Interpret results

Once the data has been analyzed and visualized, it's time to interpret the results and identify actionable insights. What these look like is going to vary from study to study. For example, the sentiment analysis and categorization might have shown that customers are significantly less likely to churn when they have a positive experience with one specific product.

It is also worth considering how your findings match your initial research goals. If the study aimed to improve the customer experience, be sure to seek out insights related to this.  

Step 8: Report your findings

Another important step in the process is to compile your findings into a detailed report. The report should cover the methodology, findings and offer some recommendations based on the data. Generative AI has emerged as a way for researchers to fast-track the report writing process, especially when working towards a tight deadline. Leaning on AI to write the entire report obviously opens up the risk of accuracy errors and hallucinations. However, it can be a valuable tool when writing something like an executive summary.

Step 9: Implement

The final—and arguably most important—step for any text analytics project is implementation. This means sharing the results with relevant teams, developing strategies based on the insights, and then monitoring the impact of the changes moving forward.

Best practices for text analytics in market research

There are a number of text analysis best practises to keep in mind when following this guide.

• Address biases - text is often heavily biased, especially when looking at social media. Bias in text can lead to skewed outcomes during analysis. These biases must be acknowledged and addressed, and researchers should always strive to analyze a balanced data set.
• Protect data privacy - standard privacy best practices for research apply when performing text analysis. Data should be anonymized where possible to prevent reidentification and the data should be audited during preprocessing to ensure it is compliant with industry standards (GDPR, HIPAA, etc).
• Consider context -  As mentioned earlier, understanding the context in which text exists is essential for any text analysis. If you are working with data from the medical industry, for example, you must consider the industry jargon when trying to extract insights from the text.  
• Choose a tool that fits your needs - With so many text analytics tools available, it is worth taking some time to research what is going to be the best fit for you. Market researchers should look at industry-specific tools like Forsta, Blix, Canvs AI and Displayr.

Top text analytics tools for market research

As mentioned above, text analysis tools are commonplace in the market research industry. Each tool is slightly different, so it is worth understanding what's out there so you can make an informed decision before investing in your solution. A few to consider are;

1. Forsta - offers 12-emotion segmentation as part of its granular sentiment analysis offering, AI-powered categorization, and self-service controls.
2. Blix - has automated topic discovery, AI-powered coding and automated reports and summaries.
3. Canvs AI - provides emotion measurement, topical & thematic analysis and a generative AI co-pilot to support users.
4. Displayr - delivers AI-powered text analytics using advanced AI and NLP to undertsand the meaning behind your text. It goes beyond identifying keywords and categorizes verbatims into reliable themes, while offering users the ability to customize the solution where needed.

You can find a full list of the best text analysis tools here.

Emerging Trends in Text Analytics for Market Research

The text analytics market research space is evolving rapidly, driven by advancements in artificial intelligence (AI), natural language processing (NLP), and sentiment analysis. These technologies are reshaping how market researchers analyze and extract insights from vast amounts of unstructured text data, providing businesses with new opportunities to better understand consumer behavior, market trends, and competitive landscapes.

1. AI and Machine Learning in Text Analytics
AI and machine learning are revolutionizing text analytics for market research by enabling more accurate and scalable analysis of large datasets. Machine learning algorithms can quickly identify patterns, categorize content, and even predict future trends based on historical text data. This automation significantly reduces the time and effort required for manual text coding, while improving the accuracy and depth of insights, which is critical in the fast-paced world of market research.

2. Natural Language Processing (NLP) for Deeper Understanding
NLP has become a cornerstone of text analytics in market research, allowing businesses to go beyond basic keyword detection to understand the deeper meaning behind the text. With semantic analysis, NLP technologies interpret context, tone, and intent—making it easier to extract actionable insights from customer feedback, social media interactions, surveys, and more. By analyzing text at this deeper level, market researchers can uncover hidden patterns and insights that would have been missed with traditional methods.

3. Sentiment Analysis for Real-Time Insights
Sentiment analysis is an increasingly vital tool in text analytics market research, allowing businesses to assess consumer emotions and opinions in real time. Whether it's positive, negative, or neutral sentiment, sentiment analysis helps businesses track customer reactions across various platforms like social media, online reviews, or focus groups. The ability to track sentiment trends over time provides businesses with a competitive edge, enabling them to respond quickly to customer concerns or capitalize on emerging opportunities.

As the text analytics market research landscape continues to evolve, the integration of AI, NLP, and sentiment analysis will only deepen, providing market researchers with more powerful tools to make data-driven decisions and stay ahead of the competition.

Get Started with text analytics today

Displayr's #1 text analytics software utilizes the latest large language models to provide an out-of-the-box solution that can understand the why behind your text data in seconds. Not only can you use Displayr to code text with a high level of accuracy, but you can also use prompts to ask highly detailed questions of your data—meaning sentiment analysis and intention detection are only a click away.

Want to see it yourself? Start a free trial.