Step-by-Step: Your Own YouTube and Web Summarizer with LangChain

Your Own YouTube and Web Summarizer with LangChain


Introduction

In today’s era of information overload, navigating the vast amount of online content can be overwhelming. With billions of videos on YouTube and countless articles, blogs, and research papers available, finding valuable insights can demand hours of reading and watching. This is where an AI-driven web summarizer becomes essential.

In this article, we'll create a Streamlit-based app powered by NLP and AI that provides comprehensive summaries of YouTube videos and websites. Using Groq’s Llama-3.2 model and LangChain’s summarization chains, this app delivers detailed summaries, saving time while ensuring no key information is missed.


Also Read: Step-by-Step: PDF Chatbots with Langchain and Ollama


Learning Outcomes


  • Explore the challenges of information overload and the advantages of AI-powered summarization.  
  • Learn to build a Streamlit app that effectively summarizes YouTube videos and website content.  
  • Dive into the role of LangChain and Llama 3.2 in creating rich, detailed summaries.  
  • Discover how to integrate tools like yt-dlp and UnstructuredURLLoader for handling multimedia content.  
  • Develop a robust web summarizer using Streamlit and LangChain to instantly summarize online videos and articles.  
  • Create a concise, accurate web summarizer with LangChain for seamless content summaries from URLs and YouTube.

Table of contents


Purpose and Benefits of the Summarizer App

From YouTube videos to web publications and in-depth research articles, a vast repository of information is available at our fingertips. However, navigating this sea of content is often challenging, especially with limited time. Most people simply don’t have the hours to invest in watching long videos or reading lengthy articles to gather insights. Studies reveal that a typical visitor spends only a few seconds on a website before deciding whether to stay or move on. This rapid evaluation process often means missing out on valuable content that requires deeper engagement to fully understand.

In this fast-paced digital age, where information is abundant but time is scarce, there’s a clear need for solutions that can help users sift through content more efficiently. An AI-powered summarization tool could bridge this gap, allowing people to capture essential information from diverse media quickly.

Langchain
Source: Langchain

Introducing AI-powered summarization: a breakthrough technique where advanced AI models process extensive content and distill it into concise, easy-to-read summaries. This approach is invaluable for busy professionals, students, researchers, and anyone needing quick, accurate insights from lengthy material. With AI summarization, users can bypass hours of reading or watching and instead capture the essence of articles, videos, and reports in just moments.

The technology leverages natural language processing and deep learning to understand the main points, context, and nuances of content. By identifying key themes, supporting details, and conclusions, AI models provide summaries that retain the substance of the original material. This makes it possible for readers to grasp essential information without missing important insights.

For instance, a student reviewing lengthy academic papers can get an overview of each document’s arguments and findings. Likewise, a professional short on time can quickly review the main takeaways from a long article or training video. AI-powered summarization empowers users to navigate the digital world’s vast information landscape more efficiently, keeping them informed without the time investment typically required.


Also Read: 7 Steps to Master Large Language Models


Components of the Summarization App

Before diving into the code, let’s review the essential components powering this application:

  • LangChain: A robust framework that simplifies interactions with large language models (LLMs), LangChain provides a standardized approach for managing prompts, chaining together different LLM operations, and accessing a variety of models, making it central to building effective summarization workflows.
  • Streamlit: This open-source Python library enables rapid creation of interactive web applications, making it ideal for designing the frontend of our summarizer. Its user-friendly interface is perfect for quick deployment and user interaction.
  • yt-dlp: When summarizing YouTube content, yt-dlp extracts metadata such as the video title and description. Unlike other downloaders, yt-dlp offers extensive format support and flexibility, making it a reliable choice for gathering video details that are then processed by the LLM.
  • UnstructuredURLLoader: A LangChain utility for loading and processing web content, UnstructuredURLLoader simplifies the complexities of fetching and extracting text from web pages, ensuring clean data input for the summarizer.

Building the App: Step-by-Step Guide

In this section, we’ll guide you through each step of building your AI summarization app. We’ll cover environment setup, user interface design, implementation of the summarization model, and app testing to ensure it performs at its best.

Note: Get the Requirements.txt file and Full code on GitHub here.

Importing Libraries and Loading Environment Variables

In this step, we will import the required libraries for machine learning, NLP, and web scraping. Additionally, we’ll load the environment variables to securely handle API keys, credentials, and configuration settings, ensuring smooth development and integration throughout the app.

# Import necessary libraries
import os
import streamlit as st
from langchain.llms import OpenAI
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
from yt_dlp import YoutubeDL
from langchain.document_loaders import UnstructuredURLLoader
from dotenv import load_dotenv

# Load environment variables from a .env file
load_dotenv()

# Access API keys or other sensitive information from environment variables
openai_api_key = os.getenv("OPENAI_API_KEY")
yt_dlp_api_key = os.getenv("YT_DLP_API_KEY")

# Example of initializing the OpenAI model with the API key
llm = OpenAI(api_key=openai_api_key)

# Define the prompt template for summarization
prompt_template = """
You are an AI assistant trained to summarize content. Below is a piece of content. Provide a concise summary.

Content:
{content}

Summary:
"""
prompt = PromptTemplate(input_variables=["content"], template=prompt_template)

# Initialize the LLM chain with the model and prompt
llm_chain = LLMChain(prompt=prompt, llm=llm)

# Function to extract YouTube metadata using yt-dlp
def extract_yt_metadata(url):
    ydl_opts = {'quiet': True}
    with YoutubeDL(ydl_opts) as ydl:
        info_dict = ydl.extract_info(url, download=False)
        return info_dict['title'], info_dict['description']

# Function to load web content using UnstructuredURLLoader
def load_web_content(url):
    loader = UnstructuredURLLoader(urls=[url])
    documents = loader.load()
    return documents[0].page_content

# Example of loading environment variables
print("OpenAI API Key:", openai_api_key)


Designing the Frontend with Streamlit

In this step, we'll design an interactive and intuitive interface for the app using Streamlit. This will involve creating input forms, buttons, and output displays, enabling users to easily interact with the app's backend features.

import streamlit as st

# Set up page configuration
st.set_page_config(page_title="AI-Powered Summarizer", page_icon="🧠")

# Main title and introduction
st.title("Summarize YouTube Videos or Web Pages")
st.write("Easily summarize content from YouTube or web pages using AI. Get concise, detailed insights in seconds!")

# Sidebar with app information
st.sidebar.title("About This Tool")
st.sidebar.info(
    "This application uses LangChain and the Llama 3.2 model to generate detailed, human-readable summaries. "
    "Simply paste a URL and click **Summarize** to see your content's key points."
)

# Instructions for users
st.header("How to Get Started:")
st.write("1. Paste a YouTube video or website URL in the input box below.")
st.write("2. Press **Summarize** to receive a quick summary of the content.")
st.write("3. Enjoy your summarized content and save time!")

# Optional: Display additional resources or information in a collapsible section
with st.expander("More Information"):
    st.write(
        "This app simplifies information overload by leveraging cutting-edge AI. "
        "Whether you're a student, professional, or researcher, our summarizer helps you digest content quickly!"
    )


Text Input for URL and Model Loading

In this step, we’ll create a text input field that allows users to enter a URL for analysis. Upon submitting the URL, the app will automatically trigger the model to process the content, ensuring that it efficiently fetches and analyzes the data. This process includes integrating the necessary machine learning model to extract key insights or summaries from the provided URL. 

We’ll also ensure that the model is correctly loaded and initialized, so it can generate accurate outputs based on the content, whether it’s a YouTube video or a web page. The backend will handle the URL fetching, data extraction, and summarization seamlessly, providing users with a streamlined and powerful experience.

# Display a subheader to prompt the user for input
st.subheader("Enter the URL:")

# Create a text input field where users can enter the URL they want to analyze
generic_url = st.text_input(
    "URL",  # Label for the input field
    label_visibility="collapsed",  # Hide the label to keep the interface clean
    placeholder="https://example.com"  # Placeholder text for guidance
)

# Provide an instruction for the user
st.write("Enter a valid URL from YouTube or a website. Then click **Summarize** to process the content.")


Users can easily input the URL of a YouTube video or a website that they want to summarize by using a text input field. This field allows them to simply copy and paste the link into the provided space. The URL could be from any YouTube video or an article/website they wish to analyze. 

Once the user enters the URL, they can proceed by clicking the "Summarize" button to trigger the backend process, which will fetch the content from the URL and generate a detailed, concise summary. This streamlined process ensures users don’t need to manually extract or analyze content, saving them time and effort.

from langchain.llms import ChatGroq
from langchain.prompts import PromptTemplate

# Initialize the ChatGroq model with model ID and API key
llm = ChatGroq(
    model="llama-3.2-11b-text-preview", 
    groq_api_key=groq_api_key
)

# Define a new prompt template for generating detailed summaries
prompt_template = """
You are an AI assistant tasked with summarizing the following content in a concise 300-word format. Be sure to highlight key points and main ideas.
Content to summarize:
{text}
"""

# Create a PromptTemplate object with the defined template and input variable
prompt = PromptTemplate(input_variables=["text"], template=prompt_template)


The model utilizes a prompt template to create a detailed 300-word summary of the given content. This template is integrated into the summarization chain, directing the model to follow a consistent approach when generating summaries.

Defining Function to Load YouTube Content

In this step, we will create a function that retrieves and processes content from YouTube. This function will accept the provided URL, extract key video information, and format it for analysis by the machine learning model integrated into the app.

import yt_dlp

def load_youtube_content(url):
    """
    Fetches and processes content from a YouTube video given its URL.
    This function extracts the video title, description, and transcript (if available),
    preparing it for analysis by the machine learning model.

    Args:
        url (str): The YouTube video URL.

    Returns:
        str: The combined text content from the YouTube video (title, description, and transcript).
    """
    # Initialize the yt-dlp downloader
    ydl_opts = {
        'quiet': True,  # Suppress output
        'forcejson': True,  # Ensure the output is in JSON format
        'noplaylist': True  # Process only the single video
    }

    # Extract video metadata using yt-dlp
    with yt_dlp.YoutubeDL(ydl_opts) as ydl:
        video_info = ydl.extract_info(url, download=False)

    # Get the video title and description
    title = video_info.get('title', 'No title available')
    description = video_info.get('description', 'No description available')

    # Combine the title and description for further processing
    content = f"Title: {title}\nDescription: {description}"

    # Optional: Extract the transcript if available
    if 'automatic_captions' in video_info:
        captions = video_info['automatic_captions']
        for lang in captions:
            if captions[lang]:
                transcript = "\n".join(captions[lang][0]['text'].splitlines())
                content += f"\nTranscript:\n{transcript}"

    return content


Handling the Summarization Logic

Here's how you can handle the button press in Streamlit to trigger the summarization process when the user clicks the "Summarize" button:

if st.button("Summarize"):
    # Display a loading indicator while processing
    with st.spinner("Summarizing the content..."):
        # Load content based on the input URL
        if generic_url.startswith("https://www.youtube.com"):
            content = load_youtube_content(generic_url)
        else:
            content = load_web_content(generic_url)  # Assuming you have a function for web content loading

        # Handle summarization using the loaded content
        summary = handle_summarization(content)

        # Display the summarized content
        st.subheader("Summary:")
        st.write(summary)


Running the Summarization Chain: The LangChain summarization chain processes the extracted content by applying the prompt template and leveraging the LLM to generate a detailed summary.

To enhance the app's appearance and provide key information, we will add a custom footer in Streamlit. This footer can include important links, acknowledgments, or contact details, contributing to a neat and professional user interface.

st.sidebar.header("Upcoming Features")
st.sidebar.write("- Download summaries option")
st.sidebar.write("- Multi-language support for summaries")
st.sidebar.write("- Customizable summary length")
st.sidebar.write("- Integration with additional content platforms")

st.sidebar.markdown("---")
st.sidebar.write("Created with ❤️ by Mit Learning")


Full Code Example

import streamlit as st
import yt_dlp
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
from langchain.llms import ChatGroq

# Set up the LLM (Groq's Llama 3.2 model)
groq_api_key = "your_groq_api_key"
llm = ChatGroq(model="llama-3.2-11b-text-preview", groq_api_key=groq_api_key)

# Define the prompt template for summarization
prompt_template = """
Provide a detailed summary of the following content in 300 words:
Content: {text}
"""
prompt = PromptTemplate(template=prompt_template, input_variables=["text"])
summarization_chain = LLMChain(llm=llm, prompt=prompt)

# Function to load YouTube video metadata using yt-dlp
def load_youtube_content(url):
    ydl_opts = {
        'quiet': True,
        'extract_flat': True,  # Get metadata only
        'forcejson': True,  # Get the metadata in JSON format
    }
    
    with yt_dlp.YoutubeDL(ydl_opts) as ydl:
        info_dict = ydl.extract_info(url, download=False)
        title = info_dict.get("title", "Unknown Title")
        description = info_dict.get("description", "No description available.")
    
    return {"title": title, "description": description}

# Streamlit UI setup
st.set_page_config(page_title="YouTube Summarizer", page_icon="📹")
st.title("YouTube Video Summarizer")

st.sidebar.header("Features Coming Soon")
st.sidebar.write("- Option to download summaries")
st.sidebar.write("- Language selection for summaries")
st.sidebar.write("- Summary length customization")
st.sidebar.write("- Integration with other content platforms")

st.sidebar.markdown("---")
st.sidebar.write("Built with ❤️ by Gourav Lohar")

# Input URL field for user
url = st.text_input("Enter YouTube Video URL:")

# Button to trigger summarization
if st.button("Summarize"):
    if url:
        # Extract content from YouTube URL
        with st.spinner("Fetching video content..."):
            content = load_youtube_content(url)
            st.subheader("Video Title:")
            st.write(content["title"])
            st.subheader("Video Description:")
            st.write(content["description"])

            # Generate and display summary
            with st.spinner("Generating summary..."):
                summary = summarization_chain.run({"text": content["description"]})
                st.subheader("Summary:")
                st.write(summary)
    else:
        st.error("Please enter a valid YouTube URL.")



Conclusion

By utilizing LangChain’s framework, we simplified the integration with the advanced Llama 3.2 language model, enabling the generation of detailed and accurate summaries. Streamlit played a crucial role in developing an intuitive, user-friendly web application, making the summarization tool easy to use and interactive.

In conclusion, this article presents a practical approach to creating an effective summary tool. By combining state-of-the-art language models with efficient frameworks and user-centric interfaces, we unlock new possibilities for enhancing information consumption and improving knowledge acquisition in today’s data-driven world.