# Massive 5-year US stock data kit

These scripts download the raw market data behind our 5-year tests of your balance method and our desk's lanes,
straight from Massive (the new name of Polygon.io), on your own Massive subscription. Nothing here contains data or keys:
you run it, and it fetches everything into a folder on your machine.

## What you get (about 7 GB, as traded, not adjusted for later splits)

| Folder | What | Size (our run) |
|---|---|---|
| `grouped/` | the whole US stock market's daily bars, one file per trading day | 0.4 GB |
| `reference/` | every ticker (active and delisted) with its type, every split | small |
| `selection/` | each day's movers (rules below) + the day after each mover + SPY/QQQ/IWM | small |
| `minute/` | 1-minute candles 04:00–20:00 New York time for every selected stock-day **and its previous day** | 5.2 GB |
| `daily/rth/` | per-stock regular-session daily bars for the selected stocks | 0.1 GB |
| `ref/` | per stock: company details (SIC code, market cap, shares), current free float, every news item with Massive's per-ticker sentiment, short interest, daily short volume, quarterly financials, ticker changes; plus every IPO | 0.7 GB |
| `seconds/` | 1-second candles for the 3,291 trade windows of our tests (`seconds_windows.csv`) | 0.3 GB |
| `minute3/` | 3-minute candles built from the 1-minute files (`make_3min.py`) | ~2 GB |

**The movers ("selection")**: common stocks and ADRs with $250k+ traded and a high of $0.50+ that either had a
regular-hours high 10%+ over the prior close, a 20%+ range, or a close 10%+ away from the prior close; plus every stock
that moved only outside regular hours (high since 16:00 the day before 10%+ over the prior close, or a 20%+ range from
04:00 to 20:00) found by the `pmcand` phase; plus the day after each mover.

## What you need

* A Massive **Stocks Starter** plan (or higher). It gives 5 years of history, minute and second candles and the
  reference endpoints used here. The history window is rolling 5 years, so your first day will be a few days later
  than ours (2021-10-04) and you also get the newest days.
* Python 3.10 or newer (no extra packages), about 10 GB of free disk, a few hours.

## Steps

```bash
export MASSIVE_API_KEY=your_key_here          # your own key; never put it in a file you share
export MASSIVE_ROOT=./massive_data            # where the data goes (default ./massive_data)

python massive_download.py daily              # ~1,300 calls, about 15 min
python massive_download.py reference          # every ticker and its type
python massive_download.py splits
python massive_download.py select             # each day's movers (computed locally, no calls)
python massive_download.py minute             # ~164,000 calls, about 4 hours (8 workers)
python massive_download.py pmcand             # ~116,000 calls, about 2 hours: movers outside regular hours
python massive_download.py rthdaily           # computed locally
python massive_reference.py                   # ~7,600 stocks x 8 kinds, 1-2 hours
python massive_seconds.py --windows seconds_windows.csv   # ~3,300 calls, about 15 min
python make_3min.py                           # computed locally
python massive_download.py status             # what is on disk
```

Every phase can be stopped and started again: files already on disk are skipped.

## Formats

* 1-minute and 3-minute files: a JSON list of `{"timestamp": "...Z" (UTC, the candle's start), "open", "high", "low", "close", "volume"}`.
* 1-second files: `[[epoch milliseconds, open, high, low, close, volume], ...]`; only seconds with trades are present.
* `seconds_windows.csv`: `day,symbol,start,end` in New York time; the start is the minute we bought in our tests.

## Things to know

* **Late prints ("ghost prints")**, the FINRA late reports you warned us about, are in the raw candles, mostly around
  08:00. Our tests cleaned them: your own rule (08:00–09:30, a candle whose bigger wick is over 70% of its range is
  clamped to ±15% of the median close of the minutes around it), then the same ±15% clamp on every candle at any
  hour, then a wick sticking out more than 5% beyond its body and 2.5% beyond the minutes around it is cut to the body.
* Prices are as traded on the day. On the day a split takes effect, compare with the prior close multiplied by the
  split ratio (`reference/splits.json.gz`).
* Massive's 1-minute candles include odd-lot and other prints that Interactive Brokers' candles leave out, so a
  single minute's low can sit a few percent under what an IBKR chart shows.