Gaia ARI. ARI s Gaia Workshop. Grégory Mantelet. 20 th June 2018 CDS / ARI

Transcription

1 Gaia ARI ARI s Gaia Workshop Grégory Mantelet CDS / ARI 20 th June 2018 Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

2 1 1. Available services 2 2. ARI Feedback Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

3 1. Available services Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

4 First glance - Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

5 1.a. Single Source Search Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

6 1.a. Single Source Search - What is it? Web-service to research a single Gaia source by: position or object name (resolved by Sesame/Simbad), or source ID Also possible to search by list whose each line can contain any of the above information. Some interesting features: AladinLite preview to display other candidates IDs in other Gaia Data Releases Result displayable online or downloadable (VOTable, CSV, Json) SAMP-capable (i.e. directly send the result to a VO tool like TOPCAT or Aladin) Time-Series (Light Curves) displayed automatically when available Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

7 1.a. Single Source Search Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

8 1.b. Cone Search Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

9 1.b. Cone Search 1/2 - What is it? A VO web-service to search all sources in a specified spatial cone (i.e. central position + radius). Interesting features: VO standard => well-defined access and behaviour => several clients can access it (e.g. TOPCAT/STILTS, Python) Simple access protocol => generally fast and simple access to data Verbosity level in order to reduce or extend the number of information to retrieve for each resulting source Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

10 1.b. Cone Search 2/2 The following features are available only on the web page of the ARI s Gaia Cone Search Service: Estimation of the cone search result (+ warning when result will be truncated) SAMP-capable Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

11 1.c TAP Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

12 1.c TAP 1/2 - What is it? A VO protocol giving a very flexible access to astronomical table data. (TAP stands for Table Access Protocol) Queries are expressed in ADQL (for Astronomical Data Query Language). Based on SQL, it provides additional functions for astronomical data. Interesting features: User table upload Select astronomical objects by position (e.g. CONTAINS(POINT(...), CIRCLE(...))=1) Custom functions: gaia_healpix_index(hpx_level, source_id) ivo_apply_pm(ra, dec, pmra, pmdec, EPOCH_DIST) full list available at gaia.ari.uni-heidelberg.de/ tap/ capabilities Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

13 1.c TAP 2/2 Query examples visible on the web page but also in TOPCAT STILTS script for each query example The following features are only available on the web page of the ARI s Gaia TAP Service: auto-completion while writing the ADQL query (triggered by Ctrl+Space) list of tables and columns SAMP-capable Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

14 1.d. Data discovery Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

15 1.d. Data discovery - full description of columns search columns basic statistics histograms skymaps (Healpix map) Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

16 1.e. Light curves Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

17 1.e. Light curves Light curves (time series) are not available through a TAP table for Gaia DR2. You have two ways to get them: with the Single Source Search service.... but only if your closest result has time series through Datalink available in a TAP query result if gaiadr2.gaia_source.source_id is selected with a URL like: In both case, TOPCAT is able to use the result. Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

18 2. ARI Feedback Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

19 2.a. Some usage statistics Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

20 2.a. Some usage statistics 1/6 Size in database? DR1: TB DR2: TB (~3x more) With indexes? DR1: TB DR2: TB (~3-4x more) Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

21 2.a. Some usage statistics 2/6 Figure 1: Daily activity on all services since Gaia-DR1 Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

22 2.a. Some usage statistics 3/6 Before DR2 Service Mean Median SS-TGAS 11 4 SS-DR1 5 2 SS-DR2 - - CS-TGAS CS-DR CS-DR2 - - TAP-sync TAP-async After DR2 Service Mean Median SS-TGAS 9 4 SS-DR1 2 2 SS-DR CS-TGAS CS-DR CS-DR TAP-sync TAP-async (nb queries per day) Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

23 2.a. Some usage statistics 4/6 Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

24 2.a. Some usage statistics 5/6 Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

25 2.a. Some usage statistics 6/6 What are the most used tables? (last week) table COUNT gaiadr2_complements.geometric_distance gaiadr2.gaia_source 7350 tap_schema.columns 270 tap_upload.user_table 156 gaiadr2.apassdr9_best_neighbour 121 ivoa.obscore 86 gaiadr1.gaia_source 80 tap_schema.tables 69 gaiadr2.gsc23_best_neighbour 49 gaiadr2.ppmxl_best_neighbour 48 Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

26 2.b. Known issues Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

27 2.b. Known issues encountered issues since the DR2 release: negative declination while using Single Source Search service with a CSV file => fixed unable to access the website or any other resource of the ARI s Gaia Services => blocked HTTP requests => fixed my query is queued for a long time. Why? Is there a problem? complex and/or slow queries not reported minor issues: missing units for the hidden information in Single Source Search need examples for Gaia-DR2 help sections not updated for DR2 (e.g. nothing about Datalink and Time-Series) Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

28 2.c. Query optimisation Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29

29 2.c. Query optimisation Less output columns (more columns you have bigger will be your output table file for the same number of rows) Avoid ORDER BY and GROUP BY as much as possible Put constraint on indexed terms instead of on an (math) expression Reduce as much as possible the input sample to crossmatch with (especially with table uploads) Grégory Mantelet (CDS / ARI) Gaia ARI 20 th June / 29