Test hipercam installation
Login to Engaging with the -X option and begin an interactive session on a compute node:
ssh -X <username>@orcd-login001.mit.edu
salloc --partition=sched_mit_kburdge_r8 --time=1:00:00 --mem=8G --cpus-per-task=4 --x11Let’s load the hipercam module:
module use /orcd/data/kburdge/001/modulefiles
module load hipercamTest it:
python -c "import hipercam; print('hipercam loaded')"The module simply loads a Conda environment (/orcd/data/kburdge/001/conda_envs/hipercam_env) and sets environment variables (e.g. PGPLOT_DIR) needed for hipercam to run. If you want to install extra packages, one can clone the Conda environment.
Reduce data
Now, let’s reduce some data! For the following tutorial I’ll assume this is ULTRACAM data. Set your data source to the local ULTRACAM option (“ul”) by:
export HIPERCAM_DEFAULT_SOURCE="ul"(See Hidden parameters documentation.)
Navigate to your night’s directory YYYY-MM-DD/, which will hold .dat and .xml files (see File Organization). Point a browser at 20XX-XX-XX/20XX-XX-XX_log.html to locate biases, flats, and science images. For example, on your local macOS:
$ scp USERNAME@orcd-login001.mit.edu:<path-to-engaging-dir>/20XX-XX-XX_log.html ./
$ open 20XX-XX-XX_log.htmlNote the readout speed (gain=ccd corresponds to slow readout speed; gain=fbb is fast), binning, and filters for your target. Note the bias frame corresponding to the same readout speed and binning. For example:
| file | object | num | bin | gain |
| run005 | Bias | 51 | 1×1 | cdd |
| run010 | Flat | 259 | 1×1 | cdd |
| run017 | OBJNAME | 756 | 1×1 | cdd |
Note: If your science frames are 2×2 binned but your flats were taken in 1×1 binning (as is standard), the HiPERCAM pipeline can safely rebin the flats in software. However, you must use 2×2 bias frames when debiasing the science images, and 1×1 bias frames when debiasing the flats, since the bias level depends on the binning.
Let’s produce an averaged bias frame by using the makebias command, which will average the (51, in my case) bias frames. Prior to running the command, it is good practice to inspect the raw frames using rtplot:
$ rtplot
run - run name: run005
first - first frame to plot: 1
trim - do you want to trim edges of windows?: True
ccd - CCD(s) to plot [0 for all]: 0
nx - number of panels in X: 3
bias - bias frame ['none' to ignore]: none
flat - flat frame ['none' to ignore]: none
defect - defect file ['none' to ignore]: none
setup - display current hdriver window settings: False
msub - subtract median from each window?: True
iset - set intensity a(utomatically), d(irectly) or with p(ercentiles)?: a
xlo - left-hand X value: -56.0
xhi - right-hand X value: 1137.0
ylo - lower Y value: -8.0
yhi - upper Y value: 1041.0If setaper fails to launch with an error like “qt.qpa.xcb: could not connect to display localhost:10.0”, it’s likely due to a stale X11 display socket. You can fix this by resetting the DISPLAY variable:
unset DISPLAY
export DISPLAY=localhost:10.0 # Replace with your original working value
Or, log out and log back in.
Now let’s run makebias, After pressing enter, it will prompt you for the input file name (run005, in my case), some input parameters, and your desired output name, which I recommend making something memorable, for example:
$ makebias
run - run name: run005
first - first frame to grab: 1
last - last frame to grab: 0
sigma - number of RMS deviations to clip: 3.0
plot - plot mean levels versus frame number? False
output - output name: biasThis will produce a file named bias.hcm. You can inspect the bias using hplot:
$ hplot
input - frame to plot: bias
ccd - CCD(s) to plot [0 for all]: 0
nx - number of panels in X: 3
msub - subtract median from each window?: True
iset - set intensity a(utomatically), d(irectly) or with p(ercentiles)?: a
xlo - left-hand X value [-56.0]:
xhi - right-hand X value [1137.0]:
ylo - lower Y value [-8.0]:
yhi - upper Y value [1041.0]: We will also want a dark frame for our reduction. Dark current is important for ULTRACAM since the CCDs run relatively warm. Since it isn’t typical to take a dark frame every night, I have already downloaded an ULTRACAM dark frame from May 27, 2021, located at /orcd/data/kburdge/001/calibs/. Create a soft link in your data directory with:
$ ln -s /orcd/data/kburdge/001/calibs/ultracam_dark_r006_2021_05_27.hcm dark.hcmNow let’s also produce a combined flat field using the makeflat function, which will use the bias.hcm and dark.hcm located in the current directory:
$ makeflat
run - run name: run010
first - first frame to average: 1
last - last frame to average (0 for all): 0
bias - bias frame: bias
dark - dark frame: dark
ngroup - number of frames per median average group: 5
ccd - CCD(s) to process [0 for all]: 0
lower - lower limits on mean count level for included flats, 1 per CCD: (5000, 5000, 5000)
upper - lower limits on mean count level for included flats, 1 per CCD: (50000, 50000, 50000)
output - output average [run010]: flatYou may want to experiment with larger ngroup values, which control how many frames are combined in each group for median stacking. Increasing ngroup can improve the removal of stars from your flats, especially if the telescope was spiralled during observations.
Next, if your filters include z-band, then perform defringing.
Now, let’s perform aperture photometry, starting with defining apertures. If your science exposures are short (e.g. 10 s), you might have trouble seeing your target in a single exposure. I suggest examining a median-combined exposure using averun prior to aperture selection. I also like to create a soft link to rename the science .xml/.dat files to something more memorable.
$ ln -s run017.xml sci.xml
$ ln -s run017.dat sci.dat
$ averun
run - run name: sci
first - first frame to average: 1
last - last frame to average: 0
trim - do you want to trim edges of windows?: True
bias - bias frame ['none' to ignore]: bias
dark - dark frame ['none' to ignore]: dark
flat - flat field frame ['none' to ignore]: flat
fmap - fringe map ['none' to ignore]: none
output - output average: sci_medianThis will produce a median-combined image sci_median.hcm. You can inspect this .hcm file to check which filters were used and which CCDs they correspond to using the hinfo command:
$ hinfo
input - frame to plot: sci_medianIn the output, look for the FILTERS keyword near the bottom:
FILTERS = 'Super u'' Super g'' Super r''These indicate the filters used during the observation. Throughput curves can be found here. Further up in the output, each CCD’s metadata contains a line like:
('CCDNAME', 'Red CCD', '')These identify the physical detectors. The mapping is always:
| CCD | CCDNAME |
| 1 | Red CCD |
| 2 | Green CCD |
| 3 | Blue CCD |
Pull up your target on Aladin, then run setaper (see all actions here):
$ setaper
mccd - frame to plot: sci_median
aper - name of aperture file: sci
ccd - CCD(s) to plot [0 for all]: 0
rtarg - target aperture radius [unbinned pixels]: 10.0
rsky1 - inner sky aperture radius [unbinned pixels]: 15.0
rsky2 - outer sky aperture radius [unbinned pixels]: 25.0
xlo - left-hand X value: -56.0
xhi - right-hand X value: 1137.0
ylo - lower Y value: -8.0
yhi - upper Y value: 1041.0
nx - number of panels in X: 3
msub - subtract median from each window?: True
iset - set intensity a(utomatically), d(irectly) or with p(ercentiles)?: p
plo - lower intensity limit percentile: 5.0
phi - upper intensity limit percentile: 95.0
a(dd), b(reak), c(entre), d(elete), e(xtra) h(elp), l(ink), m(ask), p(rofit), q(uit), r(eference), i(njected), C(opy):This will pull up a matplotlib window, so as usual you can use upperleft zoom and pan options. these comparison stars.) Note that all setaper commands must be pressed while in your plotting window, in particular with your cursor within the axes. You can see all the available setaper commands here. Also, note that the aperture radii can be changed afterwards, during the reduction step.
Consulting Aladin (the FOV ~ 6˚), add an aperture over your science target by moving your cursor over the star and pressing ‘a’. If nearby stars are within your aperture, initiate the mask function by pressing ‘m’. Then move your cursor over the center of the nearby star and press ‘m’, then move to the edge of the nearby star and again press ‘m’. Copy your target aperture to the other CCDs with ‘shift-C’. For each CCD, correct the centering of the aperture by hovering over your target star and pressing ‘c’.
If your target is significantly fainter in one of the bands (e.g. u-band), then consider linking your target to a bright, well-isolated reference star in the field. First add (‘a’) an aperture onto your bright star, copy (‘C’) to other CCDs, and center (‘c’) on each CCD. Then, for each CCD, hover over your target star and press ‘l’, then hover over the reference star and press ‘l’.
Add (‘a’), optionally apply masks (‘m’), copy (‘C’), and center (‘c’) apertures around potential comparison stars. These comparison stars are used to perform differential photometry — a technique where the brightness of your target is measured relative to one or more nearby stars assumed to be constant in brightness. The helps correct for changes in sky transparency, atmospheric extinction, and other time-dependent systematics. In the reduction step, you can test different comparison stars to see which ones produce the most stable and noise-free light curve.
If you’re aiming for absolute flux calibration, the Gaia XP spectra allows you to convert instrumental magnitudes into physical units more accurately. You can search your target on the Gaia Archive and use the ESASky preview window to display nearby Gaia stars; then, enable the filter “Has Sampled BP/RP Spectrum” and set it to true to identify stars with available Gaia XP spectra.
Press ‘q’ when finished. This produces a file sci.ape.
Generate a reduction file with genred.
$ genred
apfile - aperture input file: sci
rfile - reduce output file: sci
bias - bias frame ['none' to ignore]: bias
dark - dark field frame ['none' to ignore]: dark
flat - flat field frame ['none' to ignore]: flat
fmap - fringe frame ['none' to ignore]: none
seeing - approximate seeing [arcsec, 0 to ignore]: 1.0
sworst - worst expected seeing [arcsec]: 3.0
binfac - binning factor: 1
template - template reduce file ['none' to ignore]: none
inst - the instrument-telescope [hipercam-gtc]: ultracam-ntt
Reduce file written to sci.redThis produces a file sci.red. Open in a text editor and go to section [extraction] (around line 221) to define how the apertures will be resized and how the flux will be extracted from the aperture. There is one line per CCD with format:
<CCD label> = <resize> <extract method> [scale min max] [scale min max] [scale min max]where:
- <resize> : either ‘variable’ or ‘fixed’ apertures (usually choose ‘variable’ to fit profile and scale aperture size relative to FWHM)
- <extract method> : either ‘optimal’ or ‘normal’ extraction (usually choose ‘normal’ for straight sum of sky subtracted flux over the aperture)
- [scale min max] correspond to the target aperture radius, the inner sky radius, and the outer sky radius:
- scale : a scale factor relative to FWHM for the aperture radius
- minimum : the minimum aperture radius in unbinned pixels
- maximum : the maximum aperture radius in unbinned pixels
You’ll want to experiment with changing the [extraction] parameters, then running and inspecting the reduction.
Run and inspect reduction by running reduce, which produces a file sci.log. This runs much faster with lplot=False and implot=False.
$ reduce
run - run name: sci
first - first frame to plot: 1
trim - do you want to trim edges of windows?: True
rfile - reduce file: sci
log - name of log file to store results: sci
tkeep - number of minute of data to keep in internal buffers (0 for all): 0
lplot - do you want to plot light curves?: False
implot - do you want to plot images?: FalseYou can inspect the reduction using Python (see documentation of hlog object here):
import matplotlib.pyplot as plt
import hipercam as hcam
# load the reduction log. Creates a hipercam.hlog.Hlog object
hlg = hcam.hlog.Hlog.rascii('sci.log')
# Extract a hipercam.hlog.Tseries from aperture '1' from CCD '2', which has attributes targ.x, targ.y, targ.ye
targ = hlg.tseries('2','1')
# now the comparison (aperture '2')
comp = hlg.tseries('2','2')
# take their ratio (propagates uncertainties correctly)
lc = targ / comp
# Make an interactive plot
lc.mplot(plt)
plt.savefig('lightcurve.png')CCD ‘1’, ‘2’, ‘3’ usually corresponds to r-band, g-band, u-band. Now, edit reduction file and rerun reduction! For example:
- Try decreasing the target radius in u band (usually CCD 3)
- Try using ‘optimal’ extraction method (uses Tim Naylor’s profile weighting instead straight sub of sky subtracted flux over the aperture)
- Try fixing the sky radii by setting the minima and maxima equal to each other