Partner Data API

Problem

A partner needed programmatic access to Archaeology Data Service holdings for a virtual research environment they were building, as part of HSDS projects deliverables — which ADS operates within, so this is partner work rather than an outside request. Researchers using that VRE need to find a collection, see what is in it, and load the actual files into a working environment where they can be manipulated — none of which is possible against a website designed for human browsing.

The data also had to stay under our control. Handing over a database dump would have been simpler and completely wrong: no access control, no revocation, no record of what left.

Approach

Built with one other senior developer, who guided me and shared the work. I own it as the technical contact for the API and for questions about ADS data.

The service is API Platform on Symfony, containerised with Docker in both development and production. Around ten endpoints model the archive the way it actually works — three resources rather than one flat export:

• Collections — searchable, so the partner can find material by subject
• Objects — paginated within a collection, since collections get large
• Representations — the file versions of an object, both the original preservation copy and the dissemination copy

Almost everything is a GET. A single PUT endpoint exists so the partner can request several specific objects in one call rather than issuing a request per item. Nothing writes back into the archive.

Access is bearer token. Users are provisioned through a Doctrine console command into a small SQLite store — there are four of them, which is the right number, and a full user database would have been architecture for its own sake. Tokens are issued once through a secrets link, and the whole service is restricted to a static IP allowlist. Two independent controls, so a leaked token alone is not enough.

Bulk file delivery is deliberately not the API's job. Serving tens of terabytes through a metadata service would have been the wrong shape, so files are mirrored to an AWS S3 bucket on the partner's side and pulled from there.

The mirror

The second half of the project, and the part that runs unattended.

A Python service using the AWS SDK pushes archive files into the partner's S3 bucket, scheduled with cron to run at weekends when there is capacity for it. Over 26.7 TB has been transferred.

Two things keep it from becoming a one-off migration that quietly rots:

• New archives enrol themselves. Rather than maintaining a list, the job reads a database field marking an archive as designated for transfer, and picks up anything new on its next run. Adding an archive to the mirror is a data change, not a code change.
• A daily reconciliation job checks the mirror against the archive and reports any archive whose files are missing on the far side, alongside log checks on the transfer itself.

That second job is the one that matters. A copy you never verify is not a mirror, it is an assumption — and at 26.7 TB, nobody is going to notice a gap by looking.

Outcome

Live and in use. The partner's VRE presents researchers with a desktop environment holding the files they selected, ready for scientific work.

Weekly meetings with the partner team have run since June 2025. Beyond the implementation, a lot of my contribution has been advising on how the front end should present archaeological data types, and shaping endpoints so that presentation is straightforward on their side — decisions that are much cheaper to get right in the API than to work around in a client.