Handling ASCII data tables - Solutions

First familiarize yourself with the available supported formats:

  • Open up the Supported formats page and click through the documentation for each format.
  • Which formats document support for table metadata like column units, comments or additional descriptions?

Answer:

The 'cds', 'daophot', and 'ipac' formats have support for units and/or metadata.

Read the following files in the data/ascii/ sub-directory and show the contents for each one. Try to use the minimum effort and number of non-default options possible.

  • 'basic.dat'
  • 'ipac.dat' -- from NASA/IPAC Infrared Science Archive
  • 'daophot.dat.gz' -- Hint: unzipping is automatic
  • 'cds/lhs2065.dat' -- Hint: read the docs! Note that Vizier catalog results are in CDS format.
  • 'fixed_width.dat' -- Hint: check out the Fixed width gallery.
In [1]:
from astropy.table import Table
In [2]:
Table.read('data/ascii/basic.dat', format='ascii')
Out[2]:
obsidredshiftXYobjectradius
8770.2243783892Source 8212.5
31020.3241674085Q1250+568-A9.0
In [3]:
Table.read('data/ascii/ipac.dat', format='ascii.ipac')
Out[3]:
designationradecsigrasigdecsigradecw1mprow1sigmprow1snrw1rchi2w2mprow2sigmprow2snrw2rchi2w3mprow3sigmprow3snrw3rchi2w4mprow4sigmprow4snrw4rchi2nbnaw1satw2satw3satw4satpmrasigpmrapmdecsigpmdeccc_flagsext_flgvar_flgph_qualmoon_levw1nmw1mw2nmw2mw3nmw3mw4nmw4mdistangle
degdegarcsecarcsecarcsecmagmagmagmagmagmagmagmagmaspyrmaspyrmaspyrmaspyrarcsecdeg
J095542.55+690421.2148.927326369.07256210.04010.039-0.007911.0060.02347.3124.211.310.02249.342.310.0690.06915.71.5858.3990.3133.50.9217100.00.00.00.00193740-4241dhhd3n20nAAAB000038383838171701756.72761362.457064
J095540.45+690439.1148.918575969.07753610.05190.04940.01111.2330.02937.383.9911.510.02642.533.4910.530.09711.11.3468.2120.311.90.8811200.00.00.00.001-5865457059hhd03441nAAAU000038383838161811858.9197641.495112
J095522.94+690330.8148.845609569.05856490.04120.04050.004411.3530.02249.7155.811.6930.02248.636.429.8160.0618.12.8358.0030.1746.21.095200.00.00.00.03725026354hhhd3n41nAAAB000038383838171731759.899981246.2178
J095533.22+690354.7148.888450869.06521570.02550.0259-0.00485.7420.03828.648.736.4720.01956.972.045.0920.01475.919.272.9570.02151.44.403100.1040.0310.00.034024-32324000053420AAAA000038383838181818180.386686126.771975
J095543.03+690346.4148.929302669.06290780.03850.0378-0.007310.7550.02347.7148.111.050.02151.455.89.5920.04922.02.8357.7640.157.21.258100.00.00.00.064039-27541hhh03n30nAAAB000038383838171751753.54544699.158058
In [4]:
Table.read('data/ascii/daophot.dat.gz', format='ascii.daophot')
Out[4]:
IDXCENTERYCENTERMAGMERRMSKYNITERSHARPNESSCHIPIERPERROR
pixelspixelsmagnitudesmagnitudescountsperrors
14138.538256.40515.4610.00334.859554-0.0320.8020No_error
1818.114280.17022.3290.20630.127844-2.5441.1040No_error
In [5]:
Table.read('data/ascii/cds/lhs2065.dat', readme='data/ascii/cds/ReadMe', format='ascii.cds')
Out[5]:
LambdaFnu
0.1 nmmJy
6476.090.383329
6476.280.515559
6476.470.288042
6476.660.373343
In [6]:
Table.read('data/ascii/fixed_width.dat', header_start=1, position_line=2, data_end=-1,
           format='ascii.fixed_width_two_line')
Out[6]:
obsidredshiftXYobjectradius
8770.2243783892Source 8212.5
31020.3241674085Q1250+568-A9.0

Metadata

For the table from 'ipac.dat', there is table metadata that is in an ordered dictionary in the table meta attribute. If you read the table into a variable named dat then you would be looking at dat.meta.

  • Print the keys of dat.meta. What are they?
  • Print each of the keywords and their values from dat.meta, one per line in the format keyword: value.
  • For each column in the table print the column name and unit. HINT remember colnames.
In [7]:
dat = Table.read('data/ascii/ipac.dat', format='ascii.ipac')
dat.meta.keys()
Out[7]:
['comments', 'keywords']
In [8]:
for name, keyword in dat.meta['keywords'].items():
    print('{0}: {1}'.format(name, keyword['value']))
fixlen: T
primary: 0
RowsRetrieved: 5
QueryTime: 00:00:00.00837
ORIGIN: IPAC Infrared Science Archive (IRSA), Caltech/JPL
DATETIME: 2014-06-22 18:51:50
DataTag: ADS/IRSA.Gator#2014/0622/185150_31848
DATABASE: AllWISE Source Catalog (wise_allwise_p3as_psd)
EQUINOX: J2000
SKYAREA: within 1 arcmin of  ra=148.88821 dec=+69.06528 Eq J2000 
StatusFile: /workspace/TMP_Tfkh5F_30754/Gator/irsa/31848/log.31848.html
SQL: 'WHERE (no constraints)SELECT (47 column names follow in next row.)

In [9]:
for colname in dat.colnames:
    print('{0}: {1}'.format(colname, dat[colname].unit))
designation: 
ra: deg
dec: deg
sigra: arcsec
sigdec: arcsec
sigradec: arcsec
w1mpro: mag
w1sigmpro: mag
w1snr: 
w1rchi2: 
w2mpro: mag
w2sigmpro: mag
w2snr: 
w2rchi2: 
w3mpro: mag
w3sigmpro: mag
w3snr: 
w3rchi2: 
w4mpro: mag
w4sigmpro: mag
w4snr: 
w4rchi2: 
nb: 
na: 
w1sat: 
w2sat: 
w3sat: 
w4sat: 
pmra: maspyr
sigpmra: maspyr
pmdec: maspyr
sigpmdec: maspyr
cc_flags: 
ext_flg: 
var_flg: 
ph_qual: 
moon_lev: 
w1nm: 
w1m: 
w2nm: 
w2m: 
w3nm: 
w3m: 
w4nm: 
w4m: 
dist: arcsec
angle: deg

Harder reading (if you have time)

Read two slightly challenging tables:

  • hard1.dat -- This is part of a real data file produced by a Sybase SQL query.
  • hard2.dat -- Your neighbor asked you to help computerize his garden and handed you this data file to read. You'll need to study up on fill_values.
In [10]:
t = Table.read('data/ascii/hard1.dat', format='ascii', delimiter='|', data_start=2)
t.remove_column('col0')
t
Out[10]:
objIDosrcidxsrcidSpecObjIDradecobsidccdidzmodelMag_imodelMagErr_imodelMag_rmodelMagErr_rexpothetarad_ecf_39detlim90fBlim90
277955213S000.7044P00.7513XS04861B6_005109431360.7044530.751336486160.0865515.462060.0038416.063650.0038885104.6212610.1055333.02238215.1177120.311318
889974380S002.9051P14.7003XS03957B7_004211898322.90519514.700391395770.1318216.466050.00480716.992690.0049171479.2070350.118553.01634217.364280.880407
In [11]:
Table.read('data/ascii/hard2.dat', format='ascii.no_header', delimiter=':', data_start=5,
           names=['year', 'peas', 'beans', 'carrots'], fill_values=[('not sure', 0)])
Out[11]:
yearpeasbeanscarrots
1992234
1993101214
1994--05
1995765

Writing

Start from the table from 'ipac.dat' and make a new table that has just first 2 lines of the first 3 columns. This is done with:

>>> dat3 = dat['designation', 'ra', 'dec'][:2]
In [12]:
dat3 = dat['designation', 'ra', 'dec'][:2]

Find the output format output that will create each of the following three outputs from the table t3. You can use the pattern:

dat3.write(sys.stdout, format='...', ...)  # assuming import sys beforehand
Format 1
designation,ra,dec,sigra,sigdec
J095542.55+690421.2,148.9273263,69.0725621,0.0401,0.039
J095540.45+690439.1,148.9185759,69.0775361,0.0519,0.0494
In [13]:
import sys
dat3.write(sys.stdout, format='ascii', delimiter=',')
designation,ra,dec
J095542.55+690421.2,148.9273263,69.0725621
J095540.45+690439.1,148.9185759,69.0775361

Format 2
        designation          ra        dec
------------------- ----------- ----------
J095542.55+690421.2 148.9273263 69.0725621
J095540.45+690439.1 148.9185759 69.0775361
In [14]:
dat3.write(sys.stdout, format='ascii.fixed_width_two_line')
        designation          ra        dec
------------------- ----------- ----------
J095542.55+690421.2 148.9273263 69.0725621
J095540.45+690439.1 148.9185759 69.0775361

Format 3
\begin{table}
\begin{tabular}{ccc}
designation & ra & dec \\
J095542.55+690421.2 & 148.9273263 & 69.0725621 \\
J095540.45+690439.1 & 148.9185759 & 69.0775361 \\
\end{tabular}
\end{table}
In [15]:
dat3.write(sys.stdout, format='ascii.latex')
\begin{table}
\begin{tabular}{ccc}
designation & ra & dec \\
J095542.55+690421.2 & 148.9273263 & 69.0725621 \\
J095540.45+690439.1 & 148.9185759 & 69.0775361 \\
\end{tabular}
\end{table}

IPAC
  • Now write the t3 table back out in IPAC format. Does it look like the output has the same units and metadata as the original?
In [16]:
dat3.write(sys.stdout, format='ipac')
WARNING: Comment string > 78 characters was automatically wrapped. [astropy.io.ascii.ipac]
WARNING:astropy:Comment string > 78 characters was automatically wrapped.

\ designation
\ ___ unique WISE source designation
\ ra (deg)
\ ___ right ascension (J2000)
\ dec (deg)
\ ___ declination (J2000)
\ sigra (arcsec)
\ ___ uncertainty in RA
\ sigdec (arcsec)
\ ___ uncertainty in DEC
\ sigradec (arcsec)
\ ___ cross-term of RA and Dec uncertainties
\ w1mpro (mag)
\ ___ instrumental profile-fit photometry magnitude, band 1
\ w1sigmpro (mag)
\ ___ instrumental profile-fit photometry flux uncertainty in mag units, band 1
\ w1snr
\ ___ instrumental profile-fit photometry S/N ratio, band 1
\ w1rchi2
\ ___ instrumental profile-fit photometry reduced chi^2, band 1
\ w2mpro (mag)
\ ___ instrumental profile-fit photometry magnitude, band 2
\ w2sigmpro (mag)
\ ___ instrumental profile-fit photometry flux uncertainty in mag units, band 2
\ w2snr
\ ___ instrumental profile-fit photometry S/N ratio, band 2
\ w2rchi2
\ ___ instrumental profile-fit photometry reduced chi^2, band 2
\ w3mpro (mag)
\ ___ instrumental profile-fit photometry magnitude, band 3
\ w3sigmpro (mag)
\ ___ instrumental profile-fit photometry flux uncertainty in mag units, band 3
\ w3snr
\ ___ instrumental profile-fit photometry S/N ratio, band 3
\ w3rchi2
\ ___ instrumental profile-fit photometry reduced chi^2, band 3
\ w4mpro (mag)
\ ___ instrumental profile-fit photometry magnitude, band 4
\ w4sigmpro (mag)
\ ___ instrumental profile-fit photometry flux uncertainty in mag units, band 4
\ w4snr
\ ___ instrumental profile-fit photometry S/N ratio, band 4
\ w4rchi2
\ ___ instrumental profile-fit photometry reduced chi^2, band 4
\ nb
\ ___ number of blend components used in each fit
\ na
\ ___ active deblend flag (=1 if actively deblended)
\ w1sat
\ ___ fraction of pixels affected by saturation, band 1
\ w2sat
\ ___ fraction of pixels affected by saturation, band 2
\ w3sat
\ ___ fraction of pixels affected by saturation, band 3
\ w4sat
\ ___ fraction of pixels affected by saturation, band 4
\ pmra (maspyr)
\ ___ Apparent motion in RA
\ sigpmra (maspyr)
\ ___ Uncertainty in the RA motion estimate
\ pmdec (maspyr)
\ ___ Apparent motion in Dec
\ sigpmdec (maspyr)
\ ___ Uncertainty in the Dec  motion estimate
\ cc_flags
\ ___ prioritized artifacts affecting the source in each band
\ ext_flg
\ ___ probability that source morphology is not consistent with single PSF
\ var_flg
\ ___ probability that flux varied in any band greater than amount expected from
\ unc.s
\ ph_qual
\ ___ photometric quality of each band (A=highest, U=upper limit)
\ moon_lev
\ ___ level of moon contamination in coadd (ceiling(#frmmoon/#frames*10)), 1 per
\ band
\ w1nm
\ ___ number of profile-fit flux measurements for source with SNR >= 3, band 1
\ w1m
\ ___ number of profile-fit flux measurements for source, band 1
\ w2nm
\ ___ number of profile-fit flux measurements for source with SNR >= 3, band 2
\ w2m
\ ___ number of profile-fit flux measurements for source, band 2
\ w3nm
\ ___ number of profile-fit flux measurements for source with SNR >= 3, band 3
\ w3m
\ ___ number of profile-fit flux measurements for source, band 3
\ w4nm
\ ___ number of profile-fit flux measurements for source with SNR >= 3, band 4
\ w4m
\ ___ number of profile-fit flux measurements for source, band 4
\ angle (deg)
\ ___ Position Angle in degree.
\ dist (arcsec)
\ ___ Distance between the target position and each source in arcsec.
\fixlen='T'
\primary=0
\RowsRetrieved=5
\QueryTime='00:00:00.00837'
\ORIGIN='IPAC Infrared Science Archive (IRSA), Caltech/JPL'
\DATETIME='2014-06-22 18:51:50'
\DataTag='ADS/IRSA.Gator#2014/0622/185150_31848'
\DATABASE='AllWISE Source Catalog (wise_allwise_p3as_psd)'
\EQUINOX='J2000'
\SKYAREA='within 1 arcmin of  ra=148.88821 dec=+69.06528 Eq J2000 '
\StatusFile='/workspace/TMP_Tfkh5F_30754/Gator/irsa/31848/log.31848.html'
\SQL="'WHERE (no constraints)SELECT (47 column names follow in next row.)"
|        designation|         ra|       dec|
|               char|     double|    double|
|                   |        deg|       deg|
|               null|       null|      null|
 J095542.55+690421.2 148.9273263 69.0725621 
 J095540.45+690439.1 148.9185759 69.0775361