🥞 The Stack v3 — full corpus

stack-v3-full is the complete corpus behind The Stack v3: 110+ TB of source code from 224M GitHub repositories (770 languages), before near-dedup selection and quality filtering. It retains all near-duplicates (with cluster IDs), pre-filtering signals, and a stub entry for every excluded file — so you can build your own training mix from scratch.

If you just want to train a model, use stack-v3-train instead — the deduplicated, quality-filtered, PII-redacted training dataset with contents inline, grouped by repository. It is also the main entry point for all Stack v3 documentation: dataset creation, licensing, opt-out, and usage considerations are covered in detail there.

Unlike stack-v3-train, the file contents in this bucket are not PII-redacted. They are the raw file contents as crawled from GitHub and may contain emails, credentials, API keys, and other personal data that was public on GitHub. You are responsible for performing PII redaction on any data you derive from stack-v3-full.

• Layout
• How to access it
• Data fields
• Stub entries
• Near-duplicates: rolling your own dedup and filters
• Dataset statistics
• Licensing, opt-out, and limitations

Layout

The corpus is split into two parts so that identical file blobs shared across many repositories are stored only once:

• metadata/ — one row per repository, with a files[] array describing every file (path, language, licenses, dedup cluster, …). Each file references its content by content_id.
• contents/ — one row per unique file blob, keyed by content_id, Hive-partitioned by language (contents/language=Python/, contents/language=JavaScript/, …). This is where the actual content text lives. language is encoded in the directory name, not stored as a column inside the Parquet files — engines with Hive-partitioning support (Spark, DuckDB, PyArrow datasets) reconstruct it as a virtual column.

To attach content to files, join the two parts on content_id.

How to access it

stack-v3-full is hosted as a Hugging Face Storage Bucket (not a regular dataset repo). See the bucket access documentation for all access methods, including the S3-compatible API. At 110+ TB, downloading the whole bucket is rarely what you want — start with a language partition or a metadata sample.

With DuckDB (register the HF filesystem, then query directly):

import duckdb
from huggingface_hub import HfFileSystem

duckdb.register_filesystem(HfFileSystem())

duckdb.sql("""
    WITH files AS (
        SELECT repo_path, UNNEST(files) AS f
        FROM 'hf://buckets/HuggingFaceCode/stack-v3-full/metadata/*.parquet'
    )
    SELECT
        files.repo_path,
        files.f.file_path,
        files.f.language,
        files.f.license_type,
        c.content
    FROM files
    JOIN 'hf://buckets/HuggingFaceCode/stack-v3-full/contents/language=Python/*.parquet' AS c
      ON files.f.content_id = c.content_id
    LIMIT 10
""").show()

With pandas / PyArrow (reads via hf://buckets/ fsspec paths):

import pandas as pd
from huggingface_hub import HfFileSystem

fs = HfFileSystem()

# Read the first metadata partition
meta_files = fs.glob("hf://buckets/HuggingFaceCode/stack-v3-full/metadata/*.parquet")
meta = pd.read_parquet(f"hf://{meta_files[0]}")

# Read all contents for a single language
contents = pd.read_parquet("hf://buckets/HuggingFaceCode/stack-v3-full/contents/language=OCaml/")

With PySpark (for large-scale joins — use the S3-compatible API, or use pyspark_huggingface):

from pyspark.sql import SparkSession, functions as F

spark = (
    SparkSession.builder
    .config("spark.jars.packages", "org.apache.hadoop:hadoop-aws:3.4.1")
    # Point s3a at the HF Storage Bucket gateway
    .config("spark.hadoop.fs.s3a.endpoint", "https://s3.hf.co/HuggingFaceCode")
    .config("spark.hadoop.fs.s3a.access.key", "HFAK...")
    .config("spark.hadoop.fs.s3a.secret.key", "...")
    .config("spark.hadoop.fs.s3a.path.style.access", "true")
    .config("spark.hadoop.fs.s3a.endpoint.region", "us-east-1")
    # The HF gateway returns 302 redirects to CDN on GetObject.  It only
    # proxies bytes directly for user-agents it knows can't follow redirects
    # (aws-cli, botocore, aws-sdk-rust).  The AWS Java SDK used by Hadoop
    # s3a doesn't follow the redirect, so we spoof the UA.
    .config("spark.hadoop.fs.s3a.user.agent.prefix", "botocore/1.35.0")
    .config("spark.hadoop.fs.s3a.change.detection.mode", "none")
    .config("spark.hadoop.fs.s3a.aws.credentials.provider",
            "org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider")
    .getOrCreate()
)

meta = spark.read.parquet("s3a://stack-v3-full/metadata/")
contents = spark.read.parquet("s3a://stack-v3-full/contents/")

files = meta.select("repo_path", F.explode("files").alias("f"))
joined = (
    files.join(contents, files["f.content_id"] == contents["content_id"])
         .select("repo_path", "f.file_path", "f.language", "f.license_type", "content")
)

Data fields

metadata (one row per repository)

• repo_path (string): owner/name slug of the repository on GitHub.
• repo_id (int64): Numeric GitHub repository ID.
• commit_id (string): Commit (HEAD of the default branch) the snapshot was taken from.
• github_metadata (struct): Repository-level metadata scraped from the GitHub repository page: branch, commit_count, repo_created_at, is_fork, is_org_owned, forked_from, stars, forks, issues, pull_requests (see the stack-v3-train card for per-field descriptions).
• num_files (int64): Number of files in this repository row.
• files (list[struct]): Every file in the repository, including stubs:
  • content_id (string): SHA-1 of the UTF-8 file content. Join key into contents. null when the content was never captured (see stubs).
  • file_path (string): Path within the repository.
  • file_timestamp (int64): File modification time (Unix seconds). Only as accurate as the GitHub metadata.
  • size_bytes (int64): Length of the file content in UTF-8 bytes.
  • language (string): Language detected by go-enry. null when detection failed (see stubs).
  • is_generated (bool): Whether the file is auto-generated (per go-enry). May be unreliable.
  • is_vendor (bool): Whether the file is vendored (per go-enry).
  • detected_licenses (list[string]): SPDX license identifiers detected by ScanCode.
  • license_type (string): permissive or no_license.
  • dedup_cluster (int64): Near-duplicate cluster ID — files sharing an ID are near-duplicates of each other. null for stubs and for very short files (fewer than 5 tokens) that were not indexed for deduplication.

contents (one row per unique file blob)

• content_id (string): SHA-1 of the UTF-8 content. Join key into metadata.
• content (string): The UTF-8 file content (not PII-redacted — see the warning above).
• size_bytes (int64): Length in UTF-8 bytes.
• dedup_cluster (int64): Near-duplicate cluster ID.
• repo_ids (list[int64]): All repository IDs that contain this blob.

The partition directory (language=…) carries the detected language; there is no partition for undetected languages — those files exist only as stubs in metadata.

Jupyter notebooks in contents are the raw .ipynb JSON as crawled (outputs included, subject to the crawl-time size and binary limits below) — unlike stack-v3-train, where cell outputs, inline images, and volatile metadata are stripped. Use stack-v3-full if you want to apply your own notebook processing.

Stub entries

metadata lists the complete file listing of every repository — 43.9B file entries, of which 28.3 billion are stubs: files present in the repository but whose content is not included in contents. A stub is any entry with content_id IS NULL OR language IS NULL:

• content_id = null (13.5B entries): the content was never captured during the crawl — binary files, files larger than 5 MB, symlinks and other non-regular files. language is also null for these.
• content_id set, language = null (14.8B entries): the file was decoded as text, but go-enry could not detect a language (data files, PDFs-as-text, domain-specific formats, …). The content is not included in contents; other fields such as size_bytes and license_type may still be populated.

Stubs carry file_path, so together with the repository's repo_path and commit_id you can locate any excluded file on GitHub and re-crawl it yourself — e.g. if you need PDFs, binary assets, or a language that language detection missed.

Near-duplicates: rolling your own dedup and filters

stack-v3-full deliberately retains all near-duplicates. MinHash signatures (256 permutations, 5-grams, min. 5 tokens) were indexed with LSH; candidate pairs with Jaccard similarity ≥ 0.7 (verified, language-agnostic) were grouped into clusters via connected components. Every file's cluster ID is exposed as dedup_cluster in both parts.

To deduplicate, keep one file per dedup_cluster. For stack-v3-train, the representative was chosen by (in order): highest stars, highest forks, permissive license, earliest repository creation — but with the cluster IDs and the pre-filtering signals (is_generated, is_vendor, licenses, github_metadata) you can apply any selection or quality filtering you like.

Dataset statistics

Per-language and per-license breakdowns for the full corpus are published alongside the train dataset: stats/full/ (totals, by language, by license, stub counts, repository-size distributions).

Licensing, opt-out, and limitations

The dataset is released under the Open Data Commons Attribution License (ODC-By) v1.0 license.

Both released datasets contain only files classified as permissive or no_license; files with non-permissive detected licenses are excluded. Any use of the code must abide by the terms of the original licenses, including attribution; repo_path, commit_id, and detected_licenses provide provenance for each file.

Developers can check inclusion via the "Am I in The Stack?" Space and request removal following the opt-out instructions.

For the full documentation — dataset creation pipeline, license detection methodology, known limitations, social impact, and citation — see the stack-v3-train dataset card.