Data: files, connectors, and datasets

Your data already lives somewhere — on a disk, behind an HTTP endpoint, in a cloud drive, in an object-storage bucket, on a machine you reach over SSH. Scellis draws one sharp line through all of it: bytes coming in stream onto your device under credentials only you hold; bytes going out require a separate, stronger, destination-named consent. In between, data becomes what everything else in Scellis already is — versioned, content-addressed, and citable by hash. This page walks that path: files, connectors, credentials, the direction rule, and how raw bytes become a dataset you can reproduce by hash.

Local files stream straight from disk#

The most common source is the least dramatic. A local file streams straight from disk through a file handle: a multi-gigabyte dataset is read in place, never uploaded by default, and never squeezed through the browser's local database. That database holds your working set — the local-first catalog of what you have — not your raw data. Opening a large file therefore costs what reading it costs, nothing more: no import step, no silent duplication of gigabytes into browser storage, no copy you later have to reconcile with the original.

Connectors are content, not engine features#

Everything else arrives through a connector — and a connector is content, not an engine feature. It bundles three things: a declared effect, so a workflow that reaches the network says so before it runs; typed IO blocks, so incoming data enters the graph with a real shape; and an auth binding, so the credential rules below apply uniformly. Underneath, the record has one shape:

# A connector is content — three parts, checked like any block.
name: user/ada/s3_source           # namespaced — never shadows a builtin
version: 1
effect:  { id: net.ingress, direction: ingress }    # declared before it runs
auth:    { kind: api_key, store: device_only }       # never synced, never provenance-stamped
io:
  in:  [{ name: bucket, dtype: string }, { name: key, dtype: string }]
  out: [{ name: object, dtype: bytes }]               # enters the graph with a real shape
transport: client_signed           # SigV4 signed on device — the secret never leaves

The built-in catalog covers the protocols most data lives behind — local files and uploads, HTTP in and out, FTP/FTPS/SFTP over a relay, S3 and GCS with client-side signing, Google Drive and Dropbox over OAuth, SMTP email, a Telegram bot, outbound and inbound webhooks, Google Sheets append, and Colab export:

Because a connector is ordinary content, that set is a floor, not a ceiling. OneDrive, Slack, Discord, a Gmail-API client — none is built in, and none needs to be: each is an afternoon of authoring the same way you author any other block, catalogued, shared, and consented like one. A missing integration is never a maintainer queue — it is the flywheel that runs the Catalog.

Credentials never leave your device#

Credentials are radically client-side. A token, key, or password is encrypted at rest on your device — AES-256-GCM under a device-bound key — and that is where its life ends. It is never synced. It never joins the sync queue that holds your other changes for the server. It is never stamped into provenance. A run records what arrived, never how you got in. You can export a passphrase-wrapped copy for yourself; the platform keeps none. The full trust model is the subject of security and privacy.

OAuth, acquired as content#

Even acquiring an OAuth token is content. An OAuthProviderSpec declares a provider's endpoints, scopes, and PKCE flow, so adding one is a spec you author — not engine code — and a user-authored provider brings its own client id. Scope-minimalism is author-checked: the built-in Google connector uses the per-file picker scope only, never blanket drive access. The one relay-transiting secret (SSH/FTP auth, which a browser genuinely cannot avoid) is consented, connection-scoped, and held in memory only.

Direction is the privacy line#

Scellis does not treat “network access” as one permission. Ingress — pulling data onto your device — is a consented network effect: the connector declares it, you approve it, and afterwards the bytes live locally, back inside the ordinary local-first story. Egress — data leaving your device — is a separate, stronger consent that names the destination, and it can never ride on an ingress approval. Consenting to “fetch from this source” never quietly authorizes sending anything anywhere.

The line holds when work spans devices. In a distributed job, connectors stay device-local: ingress runs on the convener's device before the work is partitioned, and egress runs after the results merge. Peers receive work to compute — never your credentials.

Sending out, and letting the world in#

Two moments earn their own discipline: sending data out, and letting something reach in. Every act-block — an email, a webhook, a message — runs a pure dry-run first. It produces the exact outbound envelope with zero egress, so you see precisely what would be sent before anything is:

# Every act-block runs a pure dry-run first — the exact envelope, zero egress.
dry_run  email_send
  to       ada@example.org         # consent copy names this recipient
  subject  "run 812 finished"
  body     2.4 KB
  egress   none                    # nothing left the device
  ledger   send-key 5f1a…          # idempotent — a re-run will not double-send

The real send is idempotent, guarded by a send-ledger so a re-run does not double-send; the consent copy names the recipient; budgets are loud. Inbound is deliberately modest. A webhook-in is a platform-relayed inbox — a rotatable capability URL, size-capped and TTL'd — not a socket the browser listens on. And triggers fire only while a tab is open: with no tab, events queue, loudly bounded, until you return. Arming a trigger is itself the consent — a standing approval over the workflow's full effect closure, minted by a deliberate gesture and auto-disarmed the moment an edit widens that closure. This is emphatically not unattended automation: a closed tab runs nothing. (Send-dedup is per device, so a re-run on another machine may resend — provenance records the keys — a bound the platform states plainly rather than hides.)

Formats and datasets are content too#

A foreign file format is not something the engine happens to understand; it is a decode block — and, on the way out, an encode block. The built-in set covers CSV, Parquet, and Arrow; ONNX, safetensors, and PyTorch state-dicts; XLSX and DOCX export; and BPE/WordPiece tokenizers whose vocabulary is itself a content-addressed artifact. Drag a foreign file onto the canvas and it routes through connector and codec into dataset creation in one gesture; a codec the built-in set lacks is authorable, like everything else on the platform. The whole surface is cataloged content — you can see it on the data page.

Decoded data becomes a dataset: a first-class, versioned catalog entity. Every version is content-addressed — same data, same hash — and a training run references the dataset version it consumed by that hash. “What data trained this model?” is a query, not archaeology. Splits go just as deep: a train/val/test split is itself a versioned, content-addressed artifact — a deterministic, seeded partition — so the exact held-out rows behind a result are reproducible, not approximately re-drawn.

Every dataset carries a dataset card — provenance, license, consent basis, known limitations — and a profiling view: columns, dtypes, missingness, class balance. When data needs labels before supervised training can start, a labeling and annotation surface is part of the platform — not a tool you leave it for.

Bigger than memory is the normal case#

Multi-gigabyte inputs are not an edge case to survive; they are the case the engine plans for. Every block declares its streaming class — single-pass, bounded multi-pass, or full-materialization — and the engine plans chunked execution against explicit budgets for RAM, VRAM, and storage. Feasibility is checked before execution, not discovered during it: a run that cannot fit is refused up front, with named options — reduce the batch, enable streaming, change precision — never as an out-of-memory surprise mid-run. A refusal you can act on beats a crash whose cause you have to dig out afterwards, and that is the trade Scellis makes everywhere.