Skip to main content

Co-op file columns

The co-op file is a flat Parquet table, one per provider: coop_up2data (every LinkedIn profile fetched through Up2Data), coop_harvestapi (through HarvestAPI) and coop_fetchin (through Fetchin), in GET /files/. One row per profile, its latest answer. Each column is a field of that provider’s answer, named as the provider names it (LinkedIn Profile API): an object becomes one column per subfield, <field>_<subfield> (location_city). A list stays one column, a Parquet list, and an object inside it a struct whose fields are listed below as <column>[].<field>. No column holds JSON text: a field keyed by what the provider names (Fetchin’s text per language, keyed by locale) is a Parquet map. A field missing from an answer is null. An email or a phone number written in a profile reads [redacted]. When a provider adds or changes a field, we add it here and to the files. Until then the daily build logs it and leaves it out.

coop_up2data

coop_harvestapi

coop_fetchin

The member’s birthday (Fetchin’s birthDate) is left out of the file, like an email or a phone number.