Data Copy API
  • 21 Sep 2020
  • 2 Minutes To Read
  • Contributors
  • Print
  • Share
  • Dark
    Light

Data Copy API

  • Print
  • Share
  • Dark
    Light

Igneous provides a RESTful API for copying data between supported sources and destinations. Customers may implement applications that leverage the Copy API to ingest data from remote devices, to tier data between primary and secondary storage tiers, or for any backup and archive use cases - both on-prem and in the cloud. The API is able to move data in the Igneous compact (compressed and "blobbed") format as well as in native format.

Supported source and destination combinations

Sources and Destinations

Note that while "nfs://HOST" is a native filer destination and the URL host field needs to be the NFS host, "nfs-compact://BUCKET" is a compact cloud destination and the URL host field needs to be the bucket name.

Data movement request syntax

Source and destination can be formatted as parameters within URL strings or as JSON properties. for example, the can be specifed in the URL strings:

{
  "source": "nfs://HOST/EXPORT",
  "destination": "s3-compact://BUCKET/PATH?credential=USERNAME&tier=S3_TIER&region=S3_REGION&index=IGNEOUS_BUCKET"
}

Or, they can be in JSON format:

{
  "source": {
    "url": "nfs://HOST/EXPORT"
  },
  "destination": {
    "url": "s3-compact://BUCKET/PATH",
    "credential": "USERNAME",
    "tier": "S3_TIER",
    "region": "S3_REGION",
    "index": "IGNEOUS_BUCKET"
  }
}

Combinations of the two are supported. For example, the following is identical to the previous example:

{
  "source": "nfs://HOST/EXPORT",
  "destination": {
    "url": "s3-compact://BUCKET/PATH?tier=S3_TIER&region=S3_REGION&index=IGNEOUS_BUCKET",
    "credential": "USERNAME"
  }
}

Note that if a parameter is given in both a URL string and a JSON parameter, the JSON parameter will take precedence. For example:

{
  "url": "s3://BUCKET/PATH?tier=S3_TIER&index=IGNEOUS_BUCKET&region=S3_REGION",
  "region": "OTHER_REGION"
}

is effectively the same as:

{
  "url": "s3://BUCKET/PATH?tier=S3_TIER&index=IGNEOUS_BUCKET&region=OTHER_REGION"
}

Stored and transient credentials

The API allows use of both stored and transient credentials. Stored credentials are credentials that are stored within Igneous system and can be used for multiple tasks requested via the API or the Igneous dashboard. They are specified with the credential=username query parameter where username is stored in the Igneous system.

Transient credentials are used once for the request and then immediatly discarded. Transient credentials are specified with a set of transientCredentials parameters which are dependent on the type of destination.

AWS S3 Destinations

Transient credentials are specified with the accessKey and secretKey parameters, either as query parameters or JSON properties.

{
    "source": "nfs://nfshost/nfsexport",
    "destination": "s3://bucket1?region=us-west-2&tier=STANDARD&index=igbucket&transientCredentials[accessKey]=AKIAQ4T2XKPLM26PC2U4&transientCredentials[secretKey]=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"
}

Note the special syntax with square brackets to specify transient credentials parameters as URL query parameters: transientCredentials[accessKey] and transientCredentials[secretKey]

The JSON equivelent is:

{
    "source": "nfs://nfshost/nfsexport",
    "destination": {
        "url": "s3://bucket1?region=us-west-2&tier=STANDARD&index=igbucket",
        "transientCredentials": {
          "accessKey": "AKIAQ4T2XKPLM26PC2U4",
          "secretKey": "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"
        }
    }
}

Azure Destinations

Transient credentials are specified with the accountName and accountKey parameters, either as query parameters or JSON properties.

{
  "source": "nfs://nfshost/nfsexport",
  "destination": "azs://bucket1?tier=Hot&index=igbucket&transientCredentials[accountName]=user0&transientCredentials[accountKey]=password0"
}

The equivelent JSON request:

{
  "source": "nfs://nfshost/nfsexport",
  "destination": {
    "url": "azs://bucket1?tier=Hot&index=igbucket",
    "transientCredentials": {
      "accountName": "user0",
      "accountKey": "password0"
    }
  }
}

Google Cloud destinations

Transient credentials are specified with the projectId and jwt parameters as either query parameters or JSON properties:

{
  "source": "nfs://nfshost/nfsexport",
  "destination": "gcs://bucket1?tier=standard&index=igbucket&transientCredentials[projectId]=project1&transientCredentials[jwt]=%7B%22Type%22%3A%22service_account%22%2C%22Project_id%22%3A%22project1%22%2C%22Private_key_id%22%3A%22324a56f75017adfd7bab089e157f29626d32e735%22%2C%22Private_key%22%3A%22-----BEGIN+PRIVATE+KEY-----%5CnMIIEvgIBADANBgkqhkiG9e0BAQEFAASCBKgwggSkAgEAAoIBAQCYqzsYhSwwpG92%5CnSin8%2BRSaAMgX7WoyqP%2FznL1x7jMcPJzSAS%2FWNxu4VNbs%2FfPW9UVI9FFP38vi3MV7%5Cn656w94LuCBLIC6xA4JkTDkb6Zh0OtD%2BcpHUp71SsjBOH7ud%2FaRXHZ3V6kNvWJjO%2F%5CnPmSEyj37DHQ63vJ1J2MB79QRZg%2FPMQwBR91nwjlqO0rzEuq9zRdBnDmzjmSrH3%2Fh%5CnttOiM%2FzSTPj9cd5ut01fIcuiuhVzHuKaH6kT%2FGhgTz2so5pVb6uB6pQJFxuWLcJJ%5CnqorEYhIfMmv3cbpjRhbeflP2AGg4%2BBf1pFMbzROhSupiA1g%2FsQH9VF87LjSEmY%2BR%5Cn%2FmgUkgX3AgMBAAECggEAEW6DueIC0K8F1xDSyIK8OnVaY3kFRjGLwAomi2qyRUga%5Cnc9QU3TW%2FoYV2YOsCO4oX1iOnYSJhik7A8cxQTv6D4Z%2FkXRkQGvxKHiXkLtUX%2FkTQ%5CnepdAlYAxP95ioIPDxQV%2FqhwDCvKdV09XWz%2FJhGv%2FNboPT2WDc6MCqAb1RONNbM%2B0%5Cns1Ed0Xj0M0Hqf%2BXO7rRiMRBfmykzzTMS1ZRiCMhQikoVZu9i58uykc1CYJxHmwJs%5CnCoWgaRgi2Rva8jFW4Zzcnw%2Bzt8zx6gsLKOSAfkEmfLfQgGDWr0p7UNDaIl6DvSzF%5Cndd7EA4AsPlE%2Baj5dJfngQbQNyy6DC29sN8M%2FeHTemQKBgQDI3no7xiTj37h29uH9%5Cnt%2BqGaCJH2jyMJhZ9AsM2ZJO2KCw4SICCZhBMMSsIc5QeBRP9Y0kapvVYHk1IW74e%5CnI%2FGjoued2Fj5Tz9DkK1p0m4S4RB2wnRFLa7c2eP%2BrsFBwWv%2FOWIoluyjyK1GNfTD%5Cn%2FR3dMQMgKkeEt5wiweUMC3h6dQKBgQDCkhj1h8sNEz95o%2Fwk8LKbEDHFZARkxUDX%5CnmyNixMgEne1yboKFj0BkyNeom1W3r2JnskhGjvIyZFTQrKyoiEKHr4QL%2F%2B9xB%2Bqo%5CnDTF%2BCpAcMIplaX38aAMJ34M1UVnUv3DpVNTMQkvD4rD6xwFtwt1fljgnHD%2BpVjGb%5CnxzA0U6r5OwKBgQCn2mnQe2KmXXQ7TVFKAJ9lU62Z13%2BTKdzO%2BDkCwgdBR86z9boV%5CnvY5O7SNlu5eBgoLS0l3rfKs2yJOSLL%2FxVd39%2FQqJt5maNCPMpSCUdnjvI83rP9GD%5CnYYpnGVlRoNzzwtspet1Qx9XVg8NwEXSJNoH3LZbMESiUeg5qaRaq%2BwbhZQKBgQCB%5Cntbfh51ccLlrAkuvDDJtk%2BFTVzKmJLF7f3VKnw5u8SDRFdxqYQzmgdqwegP%2BR5t%2Fp%5CnVVcNgro%2FkddhDdxEtiH0JrQW89YCOk7bHYCDEDOmN03r8XnFzOlJqbHk14jVZZK2%5CnWpFKYl%2F0BKoLZKO1MpU15gUlPZXLI%2BUKfq3KFIUEhQKBgC9u%2B9xEnNEFJqO2ugtz%5CnyeWszgXBbwsHkULITpjPMqZScrlEZb6xI6cVvpyDwlnMuuhn%2Ff73gMlMJoaO1EVX%5CnWJxXzeNwNgZPxsu84zpL3%2BtIXMcp7kkjQw5xrCqB6NfUGa2qinZMQu0ipBbYmueG%5Cn5fGfiJzpA7QvJX4xlZX2sq%2Fn%5Cn-----END+PRIVATE+KEY-----%5Cn%22%2C%22Client_email%22%3A%22igneous-tier1%40project1.iam.gserviceaccount.com%22%2C%22Client_id%22%3A%22109940094586184102946%22%2C%22Auth_uri%22%3A%22https%3A%2F%2Faccounts.google.com%2Fo%2Foauth2%2Fauth%22%2C%22Token_uri%22%3A%22https%3A%2F%2Foauth2.googleapis.com%2Ftoken%22%2C%22Auth_provider_x509_cert_url%22%3A%22https%3A%2F%2Fwww.googleapis.com%2Foauth2%2Fv1%2Fcerts%22%2C%22Client_x509_cert_url%22%3A%22https%3A%2F%2Fwww.googleapis.com%2Frobot%2Fv1%2Fmetadata%2Fx509%2Figneous-tiering%2540tiering-170717.iam.gserviceaccount.com%22%7D"
}

Note: the Google Cloud JWT key needs to be URL encoded if passed as a URL query parameter. We recommend passing it as a JSON property, which has a syntax less prone to entry errors.

The JSON equivelent looks like this:

{
  "source": "nfs://nfshost/nfsexport",
  "destination": {
    "url": "gcs://bucket1?tier=standard&index=igbucket",
    "transientCredentials": {
      "projectId": "project1",
      "jwt": "{\"Type\":\"service_account\",\"Project_id\":\"project1\",\"Private_key_id\":\"324a56f75017adfd7bab089e157f29626d32e735\",\"Private_key\":\"-----BEGIN+PRIVATE+KEY-----\nMIIEvgIBADANBgkqhkiG9e0BAQEFAASCBKgwggSkAgEAAoIBAQCYqzsYhSwwpG92\nSin8+RSaAMgX7WoyqP/znL1x7jMcPJzSAS/WNxu4VNbs/fPW9UVI9FFP38vi3MV7\n656w94LuCBLIC6xA4JkTDkb6Zh0OtD+cpHUp71SsjBOH7ud/aRXHZ3V6kNvWJjO/\nPmSEyj37DHQ63vJ1J2MB79QRZg/PMQwBR91nwjlqO0rzEuq9zRdBnDmzjmSrH3/h\nttOiM/zSTPj9cd5ut01fIcuiuhVzHuKaH6kT/GhgTz2so5pVb6uB6pQJFxuWLcJJ\nqorEYhIfMmv3cbpjRhbeflP2AGg4+Bf1pFMbzROhSupiA1g/sQH9VF87LjSEmY+R\n/mgUkgX3AgMBAAECggEAEW6DueIC0K8F1xDSyIK8OnVaY3kFRjGLwAomi2qyRUga\nc9QU3TW/oYV2YOsCO4oX1iOnYSJhik7A8cxQTv6D4Z/kXRkQGvxKHiXkLtUX/kTQ\nepdAlYAxP95ioIPDxQV/qhwDCvKdV09XWz/JhGv/NboPT2WDc6MCqAb1RONNbM+0\ns1Ed0Xj0M0Hqf+XO7rRiMRBfmykzzTMS1ZRiCMhQikoVZu9i58uykc1CYJxHmwJs\nCoWgaRgi2Rva8jFW4Zzcnw+zt8zx6gsLKOSAfkEmfLfQgGDWr0p7UNDaIl6DvSzF\ndd7EA4AsPlE+aj5dJfngQbQNyy6DC29sN8M/eHTemQKBgQDI3no7xiTj37h29uH9\nt+qGaCJH2jyMJhZ9AsM2ZJO2KCw4SICCZhBMMSsIc5QeBRP9Y0kapvVYHk1IW74e\nI/Gjoued2Fj5Tz9DkK1p0m4S4RB2wnRFLa7c2eP+rsFBwWv/OWIoluyjyK1GNfTD\n/R3dMQMgKkeEt5wiweUMC3h6dQKBgQDCkhj1h8sNEz95o/wk8LKbEDHFZARkxUDX\nmyNixMgEne1yboKFj0BkyNeom1W3r2JnskhGjvIyZFTQrKyoiEKHr4QL/+9xB+qo\nDTF+CpAcMIplaX38aAMJ34M1UVnUv3DpVNTMQkvD4rD6xwFtwt1fljgnHD+pVjGb\nxzA0U6r5OwKBgQCn2mnQe2KmXXQ7TVFKAJ9lU62Z13+TKdzO+DkCwgdBR86z9boV\nvY5O7SNlu5eBgoLS0l3rfKs2yJOSLL/xVd39/QqJt5maNCPMpSCUdnjvI83rP9GD\nYYpnGVlRoNzzwtspet1Qx9XVg8NwEXSJNoH3LZbMESiUeg5qaRaq+wbhZQKBgQCB\ntbfh51ccLlrAkuvDDJtk+FTVzKmJLF7f3VKnw5u8SDRFdxqYQzmgdqwegP+R5t/p\nVVcNgro/kddhDdxEtiH0JrQW89YCOk7bHYCDEDOmN03r8XnFzOlJqbHk14jVZZK2\nWpFKYl/0BKoLZKO1MpU15gUlPZXLI+UKfq3KFIUEhQKBgC9u+9xEnNEFJqO2ugtz\nyeWszgXBbwsHkULITpjPMqZScrlEZb6xI6cVvpyDwlnMuuhn/f73gMlMJoaO1EVX\nWJxXzeNwNgZPxsu84zpL3+tIXMcp7kkjQw5xrCqB6NfUGa2qinZMQu0ipBbYmueG\n5fGfiJzpA7QvJX4xlZX2sq/n\n-----END+PRIVATE+KEY-----\n\",\"Client_email\":\"igneous-tier1@project1.iam.gserviceaccount.com\",\"Client_id\":\"109940094586184102946\",\"Auth_uri\":\"https://accounts.google.com/o/oauth2/auth\",\"Token_uri\":\"https://oauth2.googleapis.com/token\",\"Auth_provider_x509_cert_url\":\"https://www.googleapis.com/oauth2/v1/certs\",\"Client_x509_cert_url\":\"https://www.googleapis.com/robot/v1/metadata/x509/igneous-tiering%40tiering-170717.iam.gserviceaccount.com\"}"
    }
  }
}

Independent prefix paths on source and destination

Prefix paths are specified independently on the source and the destination locations with the prefix parameter:

{
  "source": "nfs://host1/export1?prefix=export1/path1",
  "destination": "nfs://host2/export2?prefix=export2/path2"
}
Was This Article Helpful?