Prepare pipeline and data

[relevant workers: general, get_data, observation_config, prepare_data]

Directories and input file names

A run of MeerKATHI must always start by setting up a number of directory and file names. This is done through the general and get_data workers.

In the general worker users give:

  • if necessary, the data_path directory where to find the files that should be converted to .MS format (general: data_path);
  • the msdir directory where to find/write .MS files (general: msdir);
  • the output directory where to write all output data products (general: output);
  • the input directory where to find various input files, such as AOflagger strategies etc. (general: input);
  • the prefix for the output data products (general: prefix).

If they do not exist yet, the above directories can be created by setting general: init_pipeline to true. This also copies files from the meerkathi/data/meerkat_files directory to the input directory set above.

In the get_data worker users give the name of the .MS files to be processed (get_data: dataid). Furthermore, the following optional steps are available:

  • convert from HDF5/MVF format to .MS (get_data: mvftoms); this step includes the following additional conversion options:
    • create a .MS.TAR file,
    • convert only visibilities in a selected channel range,
    • discard cross-polarisation products;
  • untar an existing .MS.TAR file (get_data: untar);
  • virtually concatenate all .MS input files (get_data: combine); optionally users can delete an existing concatenate file and tar/untar the concatenated file.

Metadata

Finally, before starting the actual data processing users need to provide some info about the content of the input .MS files through the observation_config worker (e.g., target and calibrators name, channelisation, etc.). In principle, this can be done by editing the relevant parameters of this worker:

In fact, MeerKATHI can automatically extract most of these info from the .MS files themselves. To do so users should enable the observation_config: obsinfo parameter. This writes a .JSON and a .TXT file to disc. Once the .JSON file is on disc, parameters set to auto (or in some cases to 0) in the observation_config worker will be automatically read from that file.

Note that the reference antenna information is often missing from the metadata. In those cases, users should carefully select a good reference antenna for calibration and set it with the observation_config: reference_antenna parameter.

[missing from this page: vampirisms, primary beam]