Introduction

What is MeerKATHI?

MeerKATHI is a pipeline to reduce radio interferometry continuum and spectral line data in full polarisation. It works on data from any radio interferometer as long as they are in “measurement set” format.

In the simplest terms, MeerKATHI is a collection of Python/Stimela scripts. Stimela is a platform-independent radio interferometry scripting framework based on Python and Docker/Singularity. Stimela allows users to execute tasks from many different data reduction packages in Python without having to install those packages individually (e.g., CASA, MeqTrees, AOflagger, SoFiA, etc.). Using Stimela, the different software packages are available through a unified scheme. MeerKATHI consists of a sequence of Stimela scripts, which it links and runs sequentially.

Within MeerKATHI – and throughout this documentation – the individual Stimela scripts are called “workers”. Each MeerKATHI worker corresponds to a specific section of the data reduction process (e.g., flagging, cross-calibration, spectral line imaging, etc.). Each worker executes several tasks from the interferometry packages included in Stimela (e.g., the cross-calibration worker can calibrate delays, bandpass, gains and flux scale).

In practice, users tell MeerKATHI what to do – and how to do it – via a YAML configuration file. The configuration file has one section for each run of a worker. By editing the configuration file users control the workers’ options, deciding which tasks to run and with what settings. A detailed explanation of the configuration file syntax is given in the Configuration file section of this manual.

Normally, users will not have to touch anything but the configuration file. They can check what has happened through a variety of data products, including images, diagnostic plots and log files. A list of all MeerKATHI data products is available at the Data products section of this manual.

In the rest of this Introduction we give a brief description of each worker. A more comprehensive description is available in the Data reduction section of this manual, which follows the flow of a typical data reduction process. The full list of parameters available for the individual workers through the configuration file can be found at the Workers parameters section of this manual or following the links below.

Brief description of workers

The following workers are available in MeerKATHI. Typically, they are executed in the same order in which they are given below. Only the first three workers (general, get_data and observation_config) should always be executed. All other workers are optional.

general

This worker sets up the name of various input/output directories and the prefix used for the output data products (e.g., diagnostic plots, images, etc.).

get_data

This worker sets up the name of the files to be processed and whether any conversion to .MS format is necessary. It can also virtually concatenate several .MS files together.

observation_config

This worker collects basic information on the content of the .MS files to be processed (e.g., target and calibrators’ name, channelisation, etc.). The worker can also extract this information automatically from the .MS metadata. Finally, it can create a primary beam image cube on a user-defined pixel- and frequency grid.

prepare_data

This worker prepares the data for calibration and imaging. For example, it can recalculate UVW coordinates, add a BITFLAG column to the input .MS files, or add spectral weights based on Tsys measurements.

flagging

This worker flags the data and returns statistics on the flags. As all other workers, it can be run multiple times within a single MeerKATHI run as explained at Configuration file (though this feature is not necessarily useful for many other workers). It can flag data based on, e.g., channel-, antenna- and time selection, or using automated algorithms that run on autocorrelations (to catch antennas with clear problems) or crosscorrelations.

cross_cal

This worker cross-calibrates the data. Users can calibrate delays, bandpass, gains and flux scale. The calibration can be applied to the calibrators’ visibilities (for later inspection) and to the target. Numerous parameters are available for users to decide how to calibrate. Flagging based on closure errors is available in this worker.

polcal

TBD

inspect_data

This worker produces diagnostic plots based on the calibrated calibrators’ visibilities.

split_target

This worker creates new .MS files which contain the targets’ calibrated visibilities only. Time and frequency averaging is available, as well as phase rotation to a new phase centre. Crosscalibration can be applied on the fly while splitting.

masking

This worker creates an a-priori clean mask based on NVSS or SUMSS catalogues, to be used during the continuum imaging/self-calibration loop. It can also merge the resulting mask with a mask based on an existing image.

self_cal

This worker performs continuum imaging and standard (i.e., direction-independent) self-calibration. Automated convergence of the calibration procedure is optionally available. This worker can also interpolate and transfer sky model and calibration tables to another .MS (e.g., from a coarse- to a fine-channel .MS file).

image_line

This worker creates spectral-line cubes and images. It can subtract the continuum via both model and UVLIN-like subtraction, Doppler correct, flag solar RFI, perform automated iterative cleaning with 3D clean masks, and, finally, run a spectral-line source finder.