Search engines are built to answer questions with pages. Ask for data, and you often get a blog post about data, a listicle, or a tool that wants your email address. The raw files sit a few clicks deeper.
Here are five large, free datasets worth knowing, what each one contains, and the catch. All figures come from the project’s pages as of November 2026.
1. OpenAlex: a map of scholarly research
What it holds: An open catalog of the global research system, covering works, authors, sources, institutions, topics, publishers, and funders.
How to get it: A free public snapshot, updated quarterly, plus a website and an API.
The catch: The JSON Lines snapshot runs about 330 GB compressed and roughly 1.6 TB unpacked. Unless you plan to analyze the whole catalog, use the website or the API instead.
In our directory: OpenAlex in the Further Index directory
2. OpenStreetMap planet file: every road and trail
What it holds: All the nodes, ways, and relations that make up the OpenStreetMap map.
How to get it: A new planet file every week, with daily, hourly, and minutely update files. As of November 1, 2026, the compressed PBF version is 89.1 GB.
The catch: The data uses the Open Database License, which asks for credit and has share-alike rules for databases you derive from it. Most projects start with a smaller regional extract instead of the full planet.
In our directory: OpenStreetMap in the Further Index directory
3. Common Crawl: a copy of a huge slice of the web
What it holds: Raw web page data, metadata, and text extracts, collected regularly since 2008. Common Crawl’s own overview says the corpus holds petabytes of data.
How to get it: Free access to the copy hosted on Amazon. You can run jobs on AWS or download all or part of it, and a URL index lets you look up captured pages.
The catch: This is raw web content, so the pages inside belong to their original publishers, and the project’s Terms of Use apply. It suits people comfortable with cloud tools, not casual browsing.
In our directory: Common Crawl in the Further Index directory
4. Smithsonian Open Access: millions of CC0 objects
What it holds: 2-D and 3-D images with their metadata, dedicated to the public domain under CC0. The program launched in February 2020 with 2.8 million items, a subset of the Smithsonian’s 155 million objects.
How to get it: Weekly refreshes through the API (which needs a free key), a GitHub repository of JSON files, or a public Amazon S3 bucket that needs no account.
The catch: Only the open-access subset is CC0, and third-party rights can still apply to some items. Check each item’s terms before commercial use.
In our directory: Smithsonian Open Access in the Further Index directory
5. Our World in Data: tidy, citable numbers
What it holds: Charts, articles, and data on major global topics.
How to get it: An API provides direct CSV downloads and JSON metadata, for a whole dataset or only the slice shown in a chart.
The catch: Our World in Data’s own work is licensed CC BY, so you must credit it when you reuse it. Data it borrows from other providers follows those providers’ licenses.
In our directory: Our World in Data in the Further Index directory
How to pick one
Start from your question.
- Papers and citations: OpenAlex
- Places and routes: OpenStreetMap
- The web itself: Common Crawl
- Museum objects and images: Smithsonian Open Access
- Global statistics: Our World in Data
If you don’t know which dataset exists for your topic, Google Dataset Search finds datasets across many hosts and sends you to the organizations that publish them. We list it in the Further Index directory.
Before you download anything
Check four things first: the license, the file size, the date of the last update, and the credit line the source requires.
Size matters most. A 1.6 TB unpacked snapshot will not fit on a standard laptop drive, so if you plan to process snapshots locally, plan on external storage. Look for at least 2 TB of capacity and a USB-C connection that supports fast transfers. You can browse external SSDs on Amazon.
As an Amazon Associate, Further Index earns from qualifying purchases.