Ten years ago, I explored the process of insight generation. I presented its key stages and emphasized the characteristics distinguishing true data-driven insights from mere correlations. While the foundations of insight generation remain relevant, since the publication of my original article, the methods, tools, and implications have expanded dramatically. Generative AI is reshaping how insights are derived, validated, and applied across industries.
The Core Tenets of Insight Generation
In an era where data-driven automated decision-making has become a competitive necessity, insight generation is a cornerstone of strategic advantage. In my original piece, I defined insight as a novel, interesting, plausible, and understandable relation, or set of associated relations, that is selected from a more extensive set of patterns derived from a data set. I argued that insights must be actionable, measurable, stable, reproducible, robust, and enduring. These qualities set insights apart from mere correlations. I also presented a framework for generating insights.
Insight generation requires pattern recognition, information synthesis, and the creation of meaningful causal connections that lead to actionable knowledge. In the original framework, I showed that human expertise was crucial for selecting the data that would be provided to the insight generation system, providing seed domain knowledge, assessing the generated insights-action plan pairs, and evaluating the decisions that result from the application of these plans.
While the overall structure of the proposed framework does not change, generative AI accelerates the insight generation process and can potentially improve the quality and quantity of the generated insights. However, the human role in contextualizing, evaluating, and acting upon the generated insights remains essential.
Generative AI’s Role in Insight Generation
Generative AI introduces three enhancements to my insight generation framework:
- AI-Enhanced Data Exploration: Generative AI significantly improves the framework’s Knowledge Extractors’ effectiveness to generate models, establish causal relations among identified entities and determine which of the identified patterns are causally related, identify interesting outliers, and develop benchmarks. Large Language Models (LLMs) can process structured and unstructured real and synthetic data to quickly create various such knowledge structures that become insight candidates. Synthetic data is useful for addressing the shortcomings of real data, improving the quality of the data, which improves the precision of the generated insight candidates, addressing ethical considerations such as privacy, and generating controlled scenarios to test insights and improve their robustness. However, even with the ability to generate appropriate synthetic data, which generative AI systems can do well, the importance of clean proprietary data should never be underestimated. Both high-value proprietary data and appropriately generated synthetic data positively impact the quality of the generated tokens and, therefore, the extracted knowledge.
- Expanded Access to Analytical Capabilities: The framework’s Insight Generator capabilities can significantly improve by combining its planning component and the domain ontologies and domain-specific insight/action plans it has access to with the reasoning abilities of frontier models and domain-specific LLMs. This combination enables the Insight Generator to filter out irrelevant or weak insight candidates, tease out causal relations among entities, generate an appropriate action plan for each, and even simulate the stability, reproducibility, and measurability of each plan before ultimately associating it with an insight.
- Augmented Decision-Making: The Decisioning System that is used during the Insight Evaluation and Selection step of the overall process can employ generative AI agents that act as “collaborative thinkers” to analyze alternative viewpoints and refine the generated insights, consider counterfactuals, simulate various scenarios by accessing appropriate digital twins, and propose potential actions in addition to those created by the Insight Generator.
Challenges to a Generated Insight’s Characteristics
Even though generative AI enhances the process first presented ten years ago, it introduces new challenges to a candidate insight’s defining characteristics because of how frontier and large language models reason and may hallucinate as they respond to prompts. In particular, generative AI can impact an insight’s characteristics in the following ways:
- Actionability: It may not be possible to perform the actions associated with an insight under real-world constraints.
- Measurability: Generated insights may not remain stable, and it may not be possible to measure their effectiveness consistently.
- Reproducibility: The output of any generative AI system is probabilistic, meaning that different runs on the same dataset may result in different outputs, resulting in hallucinations and a lack of precision. Hallucinated insight candidates and/or action plan candidates will impact the user’s trust in the system.
- Robustness and Endurance: Candidate insights may not endure across different contexts and over time because they are generated in a way that makes them susceptible to change.
Where Do We Go From Here?
Looking ahead, the convergence of several key trends will shape the future of insight generation:
- The continued evolution of generative AI: As frontier models and domain-specific LLMs improve, their ability to produce increasingly nuanced, context-aware, domain-specific, and creative insight candidates will grow. Future research should explore how to best leverage these capabilities while ensuring the actionability, measurability, reproducibility, robustness, and endurance of the generated insights.
- The rise of multimodal AI: The integration of multiple data modalities (text, images, audio, etc.) will enable a more holistic understanding of complex phenomena, leading to richer and more comprehensive insight candidates.
- The development of more sophisticated human-AI collaboration: The focus will shift from simply using AI to augment human capabilities to creating truly collaborative systems where humans and AI work together synergistically to generate insight/action plan pairs, with each contributing their unique strengths.
- Increased emphasis on ethical considerations: As AI plays a larger role in generating insights that inform decision-making, it will become increasingly important to address the ethical implications of these insights, including issues of bias, fairness, and transparency.
- The development of new business models: The expanded ability to generate insights of greater variety in different domains will drive new business models to monetize the insights that are generated rather than just the tools that lead to their generation.
The need to generate insights and effectively apply them is more important for organizations than mere pattern generation, which almost all are now able to do. The challenge lies in establishing the right insight generation process that ensures that those insights are reliable, ethical, and ultimately drive better outcomes.