TO: All GCN Notice recipients RE: New Sites Configuration method and new Filter Functions DT: 23-May-11 INTRO: A new format for the "sites.cfg" file has been adopted. And new filtering functions have been added. MOTIVATION FOR THE NEW FILE FORMAT: There are several motivations for converting from the old fixed-format (hex numbers) to the new free-format (english style) method: 1) The old config file was getting unwieldy -- too many columns; and the new features/functions and new Notice types required more columns that would have made it even more unwieldy. 2) This new "english style" format is easier to read. 3) Makes it easier for me to maintain/manage/daily_operations. 4) Allows Notice type-by-type filtering parameters: Error, TimeDelay, Intensity, etc; which makes the Intensity filter work again because it is now able to handle the variations in units and dynamic_ranges from notice type to notice type. 5) Allows for Notice type-by-type attachment specification. 6) Allows many other filtering functions (Cel/Gal/Ecl coordinates, Sun, Moon, etc). CONFIGURATION FILE FORMAT CHANGE: The switch is from the fixed-column hex-number format with the entire configuration for a given site on a single line to an english-style free-form "token value" format. Examples of the old and new formats: Old: SBmail2 -77.45 38.85 -1 360.1 999.9 2C00 0002 3Aef e308 0000 0000 0 0 0 0 0 0 0 0 0 0 ALL NONE LMAIL scott@milkyway.gsfc.nasa.gov D C ; 25jun05 08jan11 New: BEGIN_SITE # Site_administrative items: SITE_NAME SBmail2 DATES 25jun05 08jan11 SITE_LON_LAT -77.45 38.85 DIST_METHOD LMAIL DIST_ADDRESS scott@milkyway.gsfc.nasa.gov POC_NAME Scott Barthelmy POC_ADDRESS scott@milkyway.gsfc.nasa.gov DAILY_REPORT_ENABLE 0 DAILY_REPORT_ADDRESS scott@milkyway.gsfc.nasa.gov # Site-level filter items: TRIGGER_ID 0 INTENSITY -1.0 ERROR 360.1000 DELAY 999.9 SIGNIFICANCE 0.00 CONF_LEVEL 0.00 OBSERVABILITY ALL ATTACHMENT NONE SIMBAD_NED 0 # Type-level filter items: INTEGRAL_WEAK KONUS_LC SWIFT_BAT_GRB_ALERT SWIFT_BAT_GRB_POS trig_id=0 SWIFT_BAT_GRB_NACK_POS SWIFT_BAT_TRANS SWIFT_BAT_GRB_LC trig_id=0 SWIFT_FOM_2OBS trig_id=0 SWIFT_SC_2SLEW trig_id=0 SWIFT_XRT_POS SWIFT_XRT_NACK_POS SWIFT_XRT_IMAGE SWIFT_XRT_PROC_IMAGE SWIFT_UVOT_POS inten=99 SWIFT_UVOT_NACK_POS SWIFT_UVOT_IMAGE SWIFT_UVOT_PROC_IMAGE SWIFT_UVOT_SRCLIST SWIFT_UVOT_PROC_SRCLIST SWIFT_TOO_FOM SWIFT_TOO_SLEW END_SITE NEW FORMAT CONTENT: In the example above, the configuration information for a site is divided up into 3 parts: (a) the Site_adminstrative items, (b) the Site-level (global) filtering speciications, and (c) the type-by-type-level filtering specifications. For parts "a" and "b", the "token value" format provides self-documatation, but for more detail, please refer to: http://gcn.gsfc.nasa.gov/tech_describe.html#tc27 Here is a template example of the part "c" Type-specific filtering format and syntax: SWIFT_BAT_GRB_POS [inten=X] [error=Y] [delay=Z] [signif=S] [conf=P] [obs=all/vis/night/custom/etc] [trig_id=1/0] [sim_ned=1/0] [attach=sss] The square brackets ([]) indicate that these are optional arguments; the square brackets are not part of the actual invocation (eg "SWIFT_BAT_GRB_POS inten=500"). NEW FILTERING DESCRIPTION: In the past, to receive a given notice it had to pass all 6 filter rules of your configuration: 1) you had to have that notice type enabled AND 2) the position uncertainty had to be less than your error limit AND 3) it it had to be less than the time delay from the original trigger_time AND 4) the intensity measure of the event had to be greater than your intensity threshold AND 5) that particular trigger event had to be identified as the event type in question if you had Trigger_ID enabled AND 6) the RA,Dec location on the sky had to pass your observability specification. (These filter criteria stay exactly the same from the old way to the new way.) The 10 additional filter rules add to the filtering process making a 16-fold logical-AND requirement. In the config-file format conversion from old to new, the "wide open" default values for each of the new filter rules has been specified for each site, so that they do not restrict what you receive (e.g. the Sun_Distance is set to 0 degrees; you get everything that is farther than 0 deg from the Sun). It will be up to you to request changes to these new filter-rule values to provide actual filtering operation for your configuration entry. (And of course, the values for the original 6 filtering were copied from old to new.) The old system of filtering had 6 dimensions: Notice Type, Error Size, Time Delay, Intensity (applicable to only the 4 (now obsolete) CGRO_BATSE-derived types), Trigger_ID (applicable to Swift-BAT_Pos and the obsolete BATSE types), Observability (All, Visible, Night, ). The new system of filtering has 16 dimensions (the same 6 old dimensions plus 10 new dimensions): Notice Type (defaults is disabled), Error Size (default is 360 deg), Time Delay (default is infinity hours), Intensity (and now available for all types that have an "intensity" field; it is units specific), Significance (default is 0.0), Confidence (applicable to a small subset of types) (default is 0%), Trigger_ID (applicable to a small subset of types) (default is 0), Sun_distance (deg) (default is 0), Sun_hour_angle_dist (hours) (default is 0), Moon_distance (deg) (default is 0), Moon_Illumination (%) (default is 100), Celestial coords (RA,Dec ranges) (default is 0-360,-90-+90), Galactic coords (Lon,Lat ranges) (default is 0-360,-90-+90), Ecliptic coords (Lon,Lat ranges) (default is 0-360,-90-+90), Time-of-day window (default is 0.0-24.0), Observability (All, Visible, Night, ) (default is All). The meaning of each filtering parameters are: Notice_type: The notice type has to be enabled to be received. Error_size: The location error has to be less than the specified value. Time_delay: The time delay has to be less than the specified value. Intensity: The intensity has to be greater than the specified value. Significance: The significance [sigma] has to be greater than the specified value. Confidence: The confidence level [%] has to be greater than the specified value. Trigger_ID: The trigger identity has to be what the TypeName indicates (ie no false positives). Observability: The observability has to satisfy your All, Visible, Night, filter. Sun_distance: The distance (in deg) to the Sun has to be greater than the specified value. Sun_hour_angle_dist: The delta_RA hour_angle (lead or lag) has to be greater than the specified value. Moon_distance: The distance (in deg) to the Moon has to be greater than the specified value. Moon_illumination: The Moon illumination has to be less than the specified value. Celestial_coords: The location has to be within the specified RA,Dec ranges. Galactic_coords: The location has to be within the specified Gal_Lon,Lat ranges. Ecliptic_coords: The location has to be within the specified Ecl_Lon,Lat ranges. Time-of-day_window: The notice time has to be within the specified time window (UT). And all of these can be reversed (negated). The "!" character (as a suffix to the Token) is used to "reverse" the meaning of the filter range. Example: The normal way to exclude notices during bright Moon time: MOON_ILLUM 50 This will block all notices when the Moon is brighter than 50% illumination. But if you want notices when the Moon IS bright (eg. you have bright-time on a telescope to make near-IR observations): MOON_ILLUM! 50 This will pass notices when the Moon is brighter than 50% illumination (and block everything when the Moon is fainter than 50% illumination). Other Examples: If you wanted events ONLY on the Galactic Plane, then GAL_LAT_LIM -20.0 +20.0 or if you want everything off the Plane (i.e. nothing IN the Plane) GAL_LAT_LIM! -20.0 +20.0 If you only want items in the Galactic Bulge: GAL_LON_LIM 340.0 +20.0 GAL_LAT_LIM -20.0 +20.0 If you have a northern hemisphere telescope, you can restrict the Declination to what you can observe: CEL_DEC_LIM -30 +90 If you only want items in the LMC: CEL_RA_LIM 75.4 86.5 CEL_DEC_LIM -74 -64 The table below lists all the filter functions (old and new ways), and it lists which functions are Global (i.e. applies to all notice types) and which are Type-specific (i.e. can be specified for each of the notice types). Type-specific override any Global specifications. (The original 6 filter functions were only Global in the old way.) FILTER PARAM | OLD WAY(1) | NEW WAY(1) | | Global | Type-Specific(4) =============================================================================== Notice_type | X | X(3) | X Error_size | X | X[!] | X Time_delay | X | X[!] | X Intensity | X(2) | X[!] | X Significance | | X[!] | X Confidence | | X[!] | X Trigger_ID | X | X(3) | X Observability | X | X(3) | X | | | Sun_distance (deg) | | X[!] | Sun_hour_angle_dist (hr) | | X[!] | Moon_distance (deg) | | X[!] | Moon_illumination (%) | | X[!] | | | | Celestial coords (RA,Dec ranges) | | X[!] | Galactic coords (Lon,Lat ranges) | | X[!] | Ecliptic coords (Lon,Lat ranges) | | X[!] | Time-of-day Window | | X[!] | Table Notes: (1) An "X" indicates that the given function is implemented in the Old or New ways. Note that the first 8 filter rules are applicable at the global level (i.e. to all Notice Types) and at the Type-by-Type level. The next 8 filter rules (new way) are only applicable at the global level. If you want different, say, Gal Lon,Lat values for different Notice Types, then you will have to create two entries with the desired Types in each entry with the different Gal Lon,Lat ranges. (2) Intensity filtering was applicable to only the BATSE-based notice types (Old way). Now it is applicable to all notice types that contain an intensity field; and since it can be specified on a Type-by-Type basis, the different units and dynamic ranges of intensities of each Type can be handled appropriately. (3) There is no negation operation for the global Source_Type, Trigger_ID, and Observability. If you don't want, say, SWIFT_BAT_POS, then don't enable it. There is almost no practical need for the negation of Trigger_ID. And it makes no practical sense to implement a "not visible" observability filter. Negation capability on a filter function is indicated by the "[!]". (4) There is no negation operation for the Type-specific filters. The Type-specific specifications override the Global specifications. Example: SIMBAD_NED 1 ... AGILE_GRB_WAKEUP_POS sim_ned=0 means you will get SIMBAD_NED notices for all the Position-based notice types (that you have enabled), except for AGILE_GRB_WAKEUP_POS for which you have specifically disabled its SIMBAD_NED notice. Not all filtering functions are applicable to all notice types. The table in http://gcn.gsfc.nasa.gov/gcn/filtering.html shows which filters are active on which notice types. For notice types for which the filter function is not defined, there is no filtering done (no site-global, nor type-specific for that specific filter parameter); .e. that particular notice always "passes" the test no matter what is specified in the site's configuration. There are some generalities and some uniques: a) All intensity-containing notice types have intensity filtering. This is new; the old way only had intensity filtering on BATSE_POS. This was because there was only a global intensity threshold specification, and now there are different instruments (each with their own threshold value and units) a single number is not appropriate. b) Significance filtering is somewhat different than intensity filtering. It works in signal-to-noise sigma space (not counts or other raw units space). For the notice types that provide only significance (not intensity), GCN copies the significance value into the intensity slot in the notice packet. c) Currently, there are only 2 notice types that have a Confidence Level in their data packet: FERMI_GBM_FLT_POS and FERMI_GBM_TRANS. d) Currently, there are only 4 notice types that have a Trigger_ID in their data packet: SWIFT_BAT_GRB_POS, SWIFT_BAT_LIGHTCURVE, SWIFT_FOM_2OBS, and SWIFT_SC_2SLEW. RELATED FUNCTIONALITY: 1) The attachment_format specification is now also available at the notice type-by-type level. Example: if you want JPEG format for the XRT_BAT_LIGHTCURVE and FITS for the UVOT_IMAGE, you can now do that: SWIFT_BAT_LIGHTCURVE attach=jpeg SWIFT_UVOT_IMAGE attach=fits 2) The SIMBAD_NED search notices are now also specifiable at the type-by-type level. TESTING: I have been running the new format & filtering method in parallel with the old format and filtering method for about 12 months with continuous crosschecking on a site-by-site notice-by-notice basis and there have been no differences for the last 5 months in the filtering/distribution decisions. ACTION TO YOU: 1a) If you are happy with your current configuration (i.e. you do not want any of the new filtering functions) then there is nothing to do. (Your current configuration has been converted to the new format and is appended below for your convenience.) 1b) If you do want any of these new filter functions or features, go to the Config_builder web page http://gcn.gsfc.nasa.gov/gcn/config_builder.html , go to selection #2 (i.e. "Modify" a config), type in your "sitename", click "continue", and change your configuration (click/unclick Notice radio buttons, fill in filtering values, etc). (I suggest you wait on making any Type-specific filtering changes for Intensity, Significance, Confidence, or Trigger_ID. I have yet to do the analysis (on past notices) to show the distributions of those quantities vs the true-/false-positive_percentage. Without that information, these Type-specific filters are of little value, or could even be detrimental.) 2) Fill in the: POC_NAME your_name_here POC_ADDRESS your@address.goes.here All sites are strongly encouraged to fill in the POC_NAME and POC_ADDRESS. (For some sites I have filled these in based on recent email communications, but I may be wrong, so please check/correct.) 3) Of course, you can always take this opportunity to request other changes in your configuration. Your current configuration is listed below. A reminder that MAXI_UNKNOWN/_KNOWN, INTEGRAL_WEAK, & BAT_MONITOR have been aded to the list of available notice types in the last year. Sincerely, Scott Scott Barthelmy NASA-GSFC, Code 661, Greenbelt, MD 20771 PHONE: 301-286-3106 (office) PHONE: 301-346-3733 (cell) FAX: 301-286-0677 (1st choice, -1682 2nd choice) EMAIL: scott@lheamail.gsfc.nasa.gov PAGER: 3013463733@cingularme.com WEB: http://gcn.gsfc.nasa.gov/gcn