Data sources

Lochness supports pulling data from a number of data sources. This page will show you how to configure these data sources both in the keyring and in the PHOENIX metadata files.

Beiwe

For each Beiwe instance you will be pulling data from, add a new connection details section to the root of your keyring. This name of this section can be whatever you like as long as it is valid JSON. Under your new section, you must add the URL, ACCESS_KEY, and SECRET_KEY fields

{
  "beiwe.example": {
    "URL": "https://beiwe.example.org",
    "ACCESS_KEY": "...",
    "SECRET_KEY": "..."
  }
}

metadata file entry

A valid metadata file entry for Beiwe should look as follows

Active,...,Beiwe,...
1,...,beiwe.example:STUDY:SUBJECT,...

The beiwe.example should be a valid keyring section name, STUDY should be the first few characters from the 24-character Study ID, and SUBJECT should be a valid Beiwe subject.

Attention

The STUDY component of the Beiwe metadata file entry should be the Study ID, not the Study Name. You do not need to enter all 24 characters of the Study ID either. You only need enough characters (e.g., the first 5 or so) to make it uniquely identifiable.

backfilling

There is always the chance that you’re deploying Lochness well after you’ve started capturing data with Beiwe. Lochness will not attempt to download all of your data in one enormous payload. That could be on the order of many gigabytes and you will likely run into issues. Instead, Lochness will request your data in day or week-sized chunks from some starting point up to the current date. This process is called backfilling.

While Lochness will politely download your data in these digestible chunks, one detail it cannot easily predict is what date to start the backfill. The user is expected to set this using the backfill_start field within the Lochness configuration file. Please refer to the configuration file backfill_start documentation for more details.

REDCap

To have Lochness download data from REDCap, you need a few things.

redcap keyring section

First you need to create a section at the root of your keyring for your REDCap connection details. You can name this section whatever you like as long as it is valid JSON. Within this section you’ll need to add a URL field and a subsection named API_TOKEN where you will store all of your REDCap Project API tokens

{
  "redcap.demo": {
    "URL": "https://redcap.demo.org",
      "API_TOKEN": {
        "Project 1": "...",
        "Project 2": "..."
      }
  }
}

Note

To generate a REDCap Project API Token, use the API section under your REDCap Project Settings page. You must generate an API Token for each project.

lochness keyring subsection

Within the keyring file, add a REDCAP subsection to the primary lochness section

{
  "lochness": {
    "REDCAP": {
    }
}

Within this new REDCAP section, add a subsection for each PHOENIX study name. In the following example, we’ll assume the study name is StudyA

{
  "lochness": {
    "REDCAP": {
      "StudyA": {
      }
    }
  }
}

Within the new StudyA section, create yet another section with the name of your REDCap keyring section (e.g., redcap.example) followed by a list of REDCap projects you want Lochness to search and pull data from

{
  "lochness": {
    "REDCAP": {
      "StudyA": {
        "redcap.example": [
          "Project 1",
          "Project 2"
        ]
      }
    }
  }
}

metadata file entry

A valid metadata file entry should look as follows

Active,...,REDCap,...
1,...,redcap.example:SUBJECT,...

Where redcap.example would be a valid keyring section and SUBJECT would be a valid REDCap subject.

de-identification

For each PHOENIX study, you may add an entry to the Lochness configuration file indicating that data from REDCap should be de-identified before being saved to the filesystem. Please refer to the redcap configuration file documentation for more details.

XNAT

For each XNAT instance you wish to pull data from, add a new connection details section to the root of your keyring. The name of this section can be whatever you like as long as it’s valid JSON. Within your new section, add the URL, USERNAME, and PASSWORD fields

{
  "xnat.example": {
    "URL": "https://xnat.example.org",
    "USERNAME": "...",
    "PASSWORD": "..."
  }
}

metadata file entry

A valid metadata file entry should look as follows

Active,...,XNAT,...
1,...,xnat.example:PROJECT:SUBJECT,...

Where xnat.example would be a valid keyring section, PROJECT would be a valid XNAT project, and SUBJECT would be a valid XNAT Subject.

Attention

The SUBJECT component of this metadata entry should be a valid XNAT Subject, not just a MR Session. All MR Sessions for that XNAT Subject will be downloaded.

Dropbox

To have Lochness download data automatically from Dropbox, you need a few things.

create access token

First, you need to create an Access Token using the Dropbox App Console. The token should be a 64-character alphanumeric string.

create keyring section

Next, you need to create a new keyring section for your Dropbox instance. This section must be named dropbox.xxxxx where xxxxx can be any string that is both valid JSON and valid as a Python module name. Behind the scenes, Lochness will use this string to import a module. Within your new section, you must add your Dropbox Acsess Token to an API_TOKEN field

{
  "dropbox.xxxxx": {
    "API_TOKEN": "..."
  }
}

metadata file entry

A valid metadata file entry should look as follows

Active,...,Dropbox,...
1,...,dropbox.xxxxx:SUBJECT,...

Where dropbox.example would be a valid keyring section and SUBJECT would be a valid Subject folder name in Dropbox. This folder name does not necessarily have to match the PHOENIX subject.

delete on success

You can configure Lochness to delete files from Dropbox on successful download. For details, please refer to the dropbox delete_on_success configuration file documentation

dropbox base

You can configure Lochness to begin searching your Dropbox account starting from a specific subdirectory. For details, please refer to the dropbox base configuration file documentation.