Quay Missing Image Layers

Ran into a scenario where Quay had run out of storage but oc-mirror continued to push images to the registry without error.

This meant that the Quay database had the images but the actual data was not on the filesystem / s3 bucket.

The below postgresql query helped locate each of the repositories that linked to that sha256 layer. The query is run against the quay database inside postgres.

psql


SELECT
    r.name AS repository,
    u.username AS namespace,
    t.name AS tag
FROM
    manifestblob mb
JOIN
    manifest m ON mb.manifest_id = m.id
JOIN
    repository r ON m.repository_id = r.id
JOIN
    "user" u ON r.namespace_user_id = u.id
JOIN
    tag t ON t.manifest_id = m.id
JOIN
    imagestorage s ON mb.blob_id = s.id
WHERE
    s.content_checksum = 'sha256:<missing sha256 here>';

The output will allow you to see everything that references that sha256 layer. These can then be deleted from Quay and re-pushed / mirrored without the Layer already exists message.