The ‘Not-So-Ultimate’ Guide to Pagination in Modern Web Apps!

Table of Contents

Pagination is a fundamental technique in web development that allows applications to break large datasets into manageable chunks, improving both performance and user experience. In today’s data-driven applications, efficient pagination is not just a nice-to-have—it’s essential for creating responsive, scalable systems. This comprehensive guide explores various pagination strategies with a focus on the MERN stack (MongoDB, Express.js, React.js, Node.js), covering both SQL and NoSQL implementations. So how to handle pagination for large data retrieval ? Let’s understand What ? Why ? & How ? about pagination.

Understanding Pagination Fundamentals

Pagination is the process of dividing content into discrete pages, allowing users to navigate through large sets of data without overwhelming the system or the user interface. Without pagination, applications would need to load entire datasets at once, leading to several problems: performance degradation due to large data transfers, increased server load and database query time, poor user experience with excessive scrolling, unnecessary memory consumption, and slower page load times affecting bounce rates and SEO.

Effective pagination solves these problems by loading only what’s immediately necessary, providing mechanisms to access additional data when needed. Modern pagination goes beyond traditional numbered pages to include seamless experiences like infinite scrolling and dynamic content loading.

Types of Pagination Strategies

In modern web development, several pagination approaches have emerged, each with distinct advantages and use cases. Understanding these approaches is crucial for implementing the right solution for your specific application needs.

Offset-based Pagination

Offset-based pagination, also known as skip-limit pagination, is perhaps the most straightforward approach. It works by using two parameters: limit (the maximum number of items to return) and offset (the number of items to skip before returning results).

For example, to fetch the second page of results with 10 items per page, you would use an offset of 10 and a limit of 10, effectively saying “skip the first 10 items and give me the next 10.” This approach is commonly used in traditional web applications with numbered pagination controls.

Cursor-based Pagination

Cursor-based pagination uses a unique identifier (the “cursor”) to mark the position in the dataset. Instead of using numerical positions that can shift when data changes, cursor pagination tracks specific items. This is especially valuable in dynamic datasets where items might be added or removed frequently.

Cursor-based pagination is particularly well-suited for feeds or posts pages in social media-like applications, where the latest content appears at the top and new content is frequently added. The cursor acts as a bookmark, allowing the system to efficiently retrieve the next set of results without being affected by data changes.

Other Pagination Types

Beyond the two main approaches, several other pagination strategies exist:

Keyset (Seek) Pagination: Uses a key to determine the starting point of a request, similar to cursor-based pagination but typically leveraging existing indexed columns rather than a dedicated cursor value.

Token-based Pagination: Uses a token in a given request to signify the starting point for fetching data, often implemented in APIs where the service provider generates opaque tokens that clients use to request subsequent pages.

Time-based Pagination: Utilizes time-related parameters like “start_time” and “end_time” to define a time range for data retrieval, particularly useful for chronologically ordered data.

Page-based Pagination: Uses a “page” parameter to designate the desired page number, a common approach seen in many web applications where users can navigate to specific numbered pages.

Offset-based Pagination: Implementation and Best Practices

Offset-based pagination is widely used due to its simplicity and straightforward implementation. Let’s explore how it works in the MERN stack.

How Offset Pagination Works

Offset pagination works by skipping a predetermined set of data before returning the requested items. The formula is essentially:

SKIP = (page_number - 1) * page_size
LIMIT = page_size

So for page 3 with 10 items per page, you would skip the first 20 items and return the next 10.

Implementation in MERN Stack

Backend (Node.js/Express with MongoDB)

// Route for getting paginated to-do items
app.get('/todos', async (req, res) => {
  try {
    const page = parseInt(req.query.page) || 0;
    const pageSize = parseInt(req.query.limit) || 10;
    
    const skip = page * pageSize;
    
    const todos = await Todo.find({})
      .skip(skip)
      .limit(pageSize)
      .sort({ createdAt: -1 });
      
    const total = await Todo.countDocuments({});
    
    res.status(200).json({
      data: todos,
      page,
      totalPages: Math.ceil(total / pageSize),
      totalItems: total
    });
  } catch (error) {
    res.status(500).json({ message: error.message });
  }
});

This implementation demonstrates how MongoDB’s .skip() and .limit() methods enable offset pagination. The server calculates how many documents to skip based on the requested page number and page size.

Frontend (React)

import React, { useState, useEffect } from 'react';
import axios from 'axios';
 
const PaginatedList = () => {
  const [todos, setTodos] = useState([]);
  const [loading, setLoading] = useState(false);
  const [page, setPage] = useState(0);
  const [totalPages, setTotalPages] = useState(0);
  
  const fetchTodos = async (pageNumber) => {
    setLoading(true);
    try {
      const response = await axios.get(`/todos?page=${pageNumber}&limit=10`);
      setTodos(response.data.data);
      setTotalPages(response.data.totalPages);
    } catch (error) {
      console.error('Error fetching todos:', error);
    } finally {
      setLoading(false);
    }
  };
  
  useEffect(() => {
    fetchTodos(page);
  }, [page]);
  
  const handleNextPage = () => {
    if (page < totalPages - 1) {
      setPage(page + 1);
    }
  };
  
  const handlePrevPage = () => {
    if (page > 0) {
      setPage(page - 1);
    }
  };
  
  return (
    <div>
      {loading ? (
        <p>Loading...</p>
      ) : (
        <>
          <ul className="todos-list">
            {todos.map(todo => (
              <li key={todo._id}>{todo.title}</li>
            ))}
          </ul>
          
          <div className="pagination-controls">
            <button 
              onClick={handlePrevPage} 
              disabled={page === 0}
            >
              Previous
            </button>
            <span>Page {page + 1} of {totalPages}</span>
            <button 
              onClick={handleNextPage} 
              disabled={page === totalPages - 1}
            >
              Next
            </button>
          </div>
        </>
      )}
    </div>
  );
};
 
export default PaginatedList;

The React component manages pagination state, fetches data for the current page, and provides controls for navigating between pages. This pattern is common in applications with traditional pagination interfaces.

SQL Implementation Example

For SQL databases, offset pagination is typically implemented using the OFFSET and LIMIT clauses:

SELECT * FROM products
ORDER BY created_at DESC
LIMIT 10 OFFSET 20;  -- Returns the 3rd page (items 21-30) with 10 items per page

This SQL query demonstrates how offset pagination works in relational databases, skipping the first 20 records and returning the next 10.

Advantages and Disadvantages

Offset-based pagination offers several advantages: it’s simple to implement and understand, works well for static data, supports jumping directly to any page, and makes it easy to calculate total pages and show page numbers.

However, it also has significant disadvantages: performance degrades with large offsets as the database must scan and discard the skipped rows, it can lead to inconsistent results if items are added or removed during pagination, there’s potential for duplicates or missed items in dynamic datasets, and it’s less efficient for large datasets.

Cursor-based Pagination: Implementation and Best Practices

Cursor-based pagination is increasingly preferred for modern applications, especially those with real-time or frequently updated data.

How Cursor-based Pagination Works

Cursor-based pagination uses a unique identifier (cursor) that points to a specific item in the dataset. When requesting the next page, the cursor is used to find items that come after (or before) the cursor. This approach is stateless and maintains consistency even when items are added or removed.

Cursors are often based on unique ID fields or timestamps of the items being paginated, providing a stable reference point regardless of dataset changes.

Implementation in MERN Stack

Backend (Node.js/Express with MongoDB)

// Route for cursor-based pagination of posts
app.get('/posts', async (req, res) => {
  try {
    const limit = parseInt(req.query.limit) || 10;
    const cursor = req.query.cursor; // Typically a date or unique ID
    
    // Build the query
    let query = {};
    if (cursor) {
      // Assuming posts are sorted by createdAt in descending order
      query = { createdAt: { $lt: new Date(cursor) } };
    }
    
    // Fetch one more item than needed to determine if there are more pages
    const posts = await Post.find(query)
      .sort({ createdAt: -1 })
      .limit(limit + 1);
      
    const hasNextPage = posts.length > limit;
    if (hasNextPage) {
      posts.pop(); // Remove the extra item
    }
    
    // The cursor for the next page is the createdAt of the last item
    const nextCursor = posts.length > 0 
      ? posts[posts.length - 1].createdAt.toISOString() 
      : null;
    
    res.status(200).json({
      data: posts,
      nextCursor,
      hasNextPage
    });
  } catch (error) {
    res.status(500).json({ message: error.message });
  }
});

This implementation shows how to implement cursor-based pagination in a Node.js/Express backend with MongoDB. Instead of using skip(), it filters documents based on a cursor value, typically a timestamp or unique ID.

Frontend (React with Infinite Scrolling)

import React, { useState, useEffect } from 'react';
import axios from 'axios';
import { useInView } from 'react-intersection-observer';
 
const InfiniteScrollFeed = () => {
  const [posts, setPosts] = useState([]);
  const [loading, setLoading] = useState(false);
  const [cursor, setCursor] = useState(null);
  const [hasNextPage, setHasNextPage] = useState(true);
  const { ref, inView } = useInView();
  
  const fetchPosts = async () => {
    if (loading || !hasNextPage) return;
    
    setLoading(true);
    try {
      const url = cursor 
        ? `/posts?cursor=${cursor}&limit=10` 
        : '/posts?limit=10';
        
      const response = await axios.get(url);
      
      setPosts(prevPosts => [...prevPosts, ...response.data.data]);
      setCursor(response.data.nextCursor);
      setHasNextPage(response.data.hasNextPage);
    } catch (error) {
      console.error('Error fetching posts:', error);
    } finally {
      setLoading(false);
    }
  };
  
  useEffect(() => {
    if (inView) {
      fetchPosts();
    }
  }, [inView]);
  
  // Initial load
  useEffect(() => {
    fetchPosts();
  }, []);
  
  return (
    <div className="feed">
      {posts.map(post => (
        <div key={post._id} className="post">
          <h3>{post.title}</h3>
          <p>{post.content}</p>
          <span>{new Date(post.createdAt).toLocaleDateString()}</span>
        </div>
      ))}
      
      {hasNextPage && (
        <div ref={ref} className="loading-indicator">
          {loading ? 'Loading more posts...' : 'Scroll for more'}
        </div>
      )}
      
      {!hasNextPage && posts.length > 0 && (
        <div className="end-message">No more posts to load</div>
      )}
    </div>
  );
};
 
export default InfiniteScrollFeed;

This React component implements infinite scrolling with cursor-based pagination, a pattern commonly seen in social media feeds and content platforms like Facebook, Instagram, and LinkedIn. It uses the Intersection Observer API to detect when the user has scrolled to the end of the current content, triggering a request for the next page.

SQL Implementation Example

For SQL databases, cursor-based pagination requires using a WHERE clause that filters based on the cursor value:

-- Assuming 'created_at' is the cursor field with descending order
SELECT * FROM posts
WHERE created_at < '2023-09-18T10:00:00Z'  -- The cursor value
ORDER BY created_at DESC
LIMIT 10;

This SQL query demonstrates cursor-based pagination in a relational database, filtering records based on a timestamp cursor.

Advantages and Disadvantages

Cursor-based pagination offers significant advantages: consistent results even when data changes during pagination, better performance for large datasets as it doesn’t require counting or skipping records, no duplication or missed items in dynamic feeds, more efficient for databases with proper indexing, and ideal for infinite scrolling and real-time data.

Its disadvantages include: more complex implementation than offset pagination, difficulty in supporting direct page jumps, requirement for a stable unique field as the cursor, and potentially more challenging implementation in some frontend frameworks.

Choosing the Right Pagination Strategy

Selecting the appropriate pagination strategy is crucial for creating efficient, user-friendly applications. Here’s a framework for making this decision based on your specific requirements:

When to Use Offset-based Pagination

Offset-based pagination is generally suitable for:

Smaller, relatively static datasets (thousands of records, not millions)
Applications where users need to jump to specific page numbers
Admin interfaces and dashboards where completeness is more important than performance
When the total count of items is required for UI elements
Simple CRUD applications with predictable data volumes

When to Use Cursor-based Pagination

Cursor-based pagination is preferred for:

Large datasets (thousands to millions of records)
Real-time data or feeds where items are frequently added/removed
Social media feeds, activity streams, or comment threads
Implementing infinite scrolling user interfaces
Mobile applications where performance is critical
When consistency during pagination is essential

The table below summarizes the key differences between these approaches, based on information from search result:

Feature	Offset Pagination	Cursor-based Pagination
Implementation	Uses numeric offset to skip records	Uses a unique identifier as cursor
Complexity	Simple to implement	More complex implementation
Performance with large datasets	Degrades with larger offsets	Maintains consistent performance
Data consistency	Can miss or duplicate items when data changes	Consistent even with changing data
Use cases	Small to medium datasets, admin interfaces	Large or dynamic datasets, feeds, infinite scrolling

Industry Standards and Practices

Major tech companies have largely adopted cursor-based pagination for their APIs and user interfaces:

Facebook/Meta uses cursor-based pagination for news feeds
Twitter implements cursor-based pagination for timeline feeds
Instagram uses cursor-based pagination with infinite scrolling
LinkedIn uses cursor-based pagination for post feeds (mentioned in search result)
Google emphasizes proper pagination techniques for SEO purposes (mentioned in search result)

This industry trend reflects the growing importance of handling large, dynamic datasets efficiently in modern web applications.

Implementation Best Practices

Frontend Implementation (React)

When implementing pagination in React applications, consider these best practices:

State Management for Pagination

// For offset pagination
const [page, setPage] = useState(1);
const [totalPages, setTotalPages] = useState(0);
const [data, setData] = useState([]);
 
// For cursor pagination
const [items, setItems] = useState([]);
const [cursor, setCursor] = useState(null);
const [hasNextPage, setHasNextPage] = useState(true);

Proper state management is crucial for tracking pagination position and controlling UI elements.

Loading States and Error Handling

const [isLoading, setIsLoading] = useState(false);
const [error, setError] = useState(null);
 
// In your JSX
{isLoading && <div className="loader">Loading...</div>}
{error && (
  <div className="error-message">
    Error loading data: {error.message}
    <button onClick={retry}>Retry</button>
  </div>
)}

Always provide clear feedback to users when data is loading or when errors occur.

Backend Implementation (Node.js/Express)

Standardized API Responses

// Consistent response format for paginated data
res.status(200).json({
  data: items,          // The actual data items
  pagination: {
    nextCursor,         // For cursor-based pagination
    hasNextPage,        // Boolean flag for more data
    page,               // Current page (offset pagination)
    totalPages,         // Total available pages
    totalItems          // Total count of items
  }
});

Consistent API response formats make client-side implementation more straightforward and maintainable.

Input Validation and Security

// Sanitize and validate pagination parameters
const limit = Math.min(
  parseInt(req.query.limit) || 10,
  100  // Maximum allowed limit
);
 
const page = Math.max(
  parseInt(req.query.page) || 1,
  1    // Minimum page number
);

Always validate and sanitize pagination parameters to prevent potential security issues or performance problems.

Database Optimizations

Efficient database queries are essential for performant pagination:

Proper Indexing

For MongoDB:

// Create indexes on fields used for pagination
db.posts.createIndex({ createdAt: -1 });
 
// Compound indexes for more complex pagination
db.posts.createIndex({ userId: 1, createdAt: -1 });

For SQL:

-- Index on timestamp field for cursor pagination
CREATE INDEX idx_posts_created_at ON posts(created_at);
 
-- Compound index for filtered pagination
CREATE INDEX idx_posts_category_created_at ON posts(category_id, created_at);

Proper indexing is critical for pagination performance, especially for cursor-based pagination where filtering on cursor fields is frequent.

Efficient Queries

For MongoDB:

// Use projection to limit fields returned
db.posts.find({ createdAt: { $lt: cursor } })
  .sort({ createdAt: -1 })
  .limit(10)
  .projection({ title: 1, content: 1, author: 1 });

For SQL:

-- Select only necessary fields
SELECT id, title, content, author_id, created_at FROM posts
WHERE created_at < ? 
ORDER BY created_at DESC
LIMIT 10;

Always request only the fields you need to minimize data transfer and processing time.

SEO Considerations

From search result, Google provides specific pagination best practices:

Link pages sequentially - Include links from each page to the next using <a href> tags
Unique URLs for each page - Give each page a unique URL (e.g., using ?page=n parameters)
Proper canonical URLs - Each page should have its own canonical URL
Avoid URL fragments - Don’t use URL fragment identifiers (#) for page numbers
Consider using preload/prefetch - Optimize performance for users moving to the next page

These practices ensure search engines can properly crawl and index your paginated content.

Advanced Pagination Scenarios

Real-time Data with Pagination

For applications with real-time updates:

Combining WebSockets with cursor pagination
- Use cursor pagination for initial data load
- Stream real-time updates via WebSockets
- Merge new items into the existing dataset based on cursor position
Handling conflicts
- Develop strategies for when new items appear during pagination
- Consider notification systems for new content

This approach is commonly used in chat applications, social media feeds, and collaborative tools.

Handling Deletions and Insertions

Cursor stability
- Use immutable properties (like UUIDs or timestamps) as cursors
- Consider composite cursors (combination of timestamp and ID)
Duplication detection
- Maintain a client-side cache of item IDs to detect and remove duplicates
- Implement deduplication logic in the frontend

These strategies ensure consistent pagination even when the underlying dataset changes frequently.

Pagination with Complex Filtering and Sorting

Consistent sorting
- Always include a unique field in the sort criteria to ensure consistency
- For example: ORDER BY category, created_at DESC, id DESC
Cursor transformation
- When allowing users to change sort order, transform or reset cursors
- Store sort parameters along with cursor information

These approaches help maintain pagination consistency when users can change how data is filtered or sorted.

Best Practices Summary

To conclude, here are the key pagination best practices to remember:

Choose the right pagination strategy based on your dataset size, dynamism, and user needs
Implement proper indexing for fields used in pagination queries
Validate and sanitize all pagination parameters on the server
Provide consistent API responses with comprehensive pagination metadata
Handle loading states and errors gracefully in the UI
Consider SEO implications and follow Google’s pagination guidelines
Optimize database queries by selecting only necessary fields
Implement proper cursor encoding/decoding for cursor-based pagination
Add comprehensive error handling for pagination edge cases
Test pagination with large datasets to ensure performance at scale

By following these guidelines, you can create efficient, user-friendly pagination systems that enhance the overall performance and usability of your web applications.