Duplicate picks in events from FDSN- (fixed from Post May2023 onward)

[calum-chamberlain]

Hi all,
I wondered if there was a good reason for duplicate picks appearing in events downloaded from the FDSN service? For instance event 2022p788472 has duplicate picks for multiple stations including duplicate P-picks for BFZ. There are two P-picks sharing the same pick id (smi:nz.org.geonet/20221019.173832.28-AIC-NZ.BFZ.10.HHZ) that, to my eyes, have all the same information.

If there isn't a good reason, would it be possible to fix this?

Thanks,
Calum.

[salichon]

Hello @calum-chamberlain
that s indeed a feature that i noticed a while ago - This is very occasional I reckon and a direct output from seiscomp
I had a few cases and i had no chance to get some clarification around the processing ~eg. been able to reproduce that feature and address it to Gempa. (Glitch over the streams from the station - or from the picking in seiscomp)
I ll resurrect this topic in the forum.

With regards to fixing this the answer is yes and easy, ..although the "process" to perform this
in an operational manner is much less so, for now.

To me this is an example of event level catalogue data curation (FYI @elidana @JonoHanson ) that should be added into the to do list.
We can expect an implementation of an environment that allows this to be performed immediatly in the incoming year.

NB:Please keep on postingevents there to keep on building the case.

thanks a lot
Jerome

[calum-chamberlain]

This isn't rare at all in my experience. Consider the example below, which assumes that duplicate pick resource ids represent duplicated picks without checking other information explicitly:

from obspy.clients.fdsn import Client
from obspy import UTCDateTime
from collections import Counter

client = Client("GEONET")
cat = client.get_events(starttime=UTCDateTime(2022, 1, 1), endtime=UTCDateTime(2022, 2, 1))

duplicate_pick_count = 0
events_with_duplicates = 0
for ev in cat:
    pid_counts = Counter(p.resource_id for p in ev.picks)
    duplicate_picks = [pid for pid, count in pid_counts.items() if count > 1]
    duplicate_pick_count += len(duplicate_picks)
    if len(duplicate_picks):
        events_with_duplicates += 1

print(f"From {len(cat)} events, {events_with_duplicates} had duplicate picks.")
print(f"Total duplicate picks: {duplicate_pick_count}")

For this month there were 2,101 events returned, of which 1,877 had duplicate picks - there were 15,395 duplicate pick ids. Note that this is not all picks, it isn't obvious from my end which picks are duplicated. I haven't checked extensively whether it is only auto-picks that are duplicated, but so far I haven't seen any manual picks duplicated.

[salichon]

@calum-chamberlain thanks a lot for making the case ...
I went Through this observation and then amend my comments above:
1- This is not the same odd random feature i thought about then where 2 same station picks are set in the seiscompml file solutions within a really short delay (without exact same times)

2- Indeed 2,101 events returned, of which 1,877 had duplicate picks looks definitely more than i was expecting .. see above!!

so...
3 - With regards to #105 (comment)
Event file verification gives:
- for seiscompml (Base/reference event format ): only one BFZ pick
- for fdsnxml (inflight converted event format for webservices): Duplicated BFZ picks (*swear_word here)

e.g

 `
> **Seiscompml event file seiscompml07.s3-website-ap-southeast-2.amazonaws.com/2022p788472.xml**
>  **grep -A3  -E "publicID=\"20221019.173832.28-AIC-NZ.BFZ.10.HHZ" dup2022p788472.xml** 
>     
> <pick publicID="20221019.173832.28-AIC-NZ.BFZ.10.HHZ">
>       <time>
>         <value>2022-10-19T17:38:32.28Z</value>
>       </time>
> 
> 
> **FDSN XML  event File : https://service.geonet.org.nz/fdsnws/event/1/query?eventid=2022p788472** 
>  **grep -A3  -E "publicID=\"smi:nz.org.geonet/20221019.173832.28-AIC-NZ.BFZ.10.HHZ" dup2022p788472_fdsn.xml** 
> 
> 
>       <pick publicID="smi:nz.org.geonet/20221019.173832.28-AIC-NZ.BFZ.10.HHZ">
>         <time>
>           <value>2022-10-19T17:38:32.28Z</value>
>         </time>
> --
>       <pick publicID="smi:nz.org.geonet/20221019.173832.28-AIC-NZ.BFZ.10.HHZ">
>         <time>
>           <value>2022-10-19T17:38:32.28Z</value>
>         </time>
> --

"Arrival" for BFZ is unique in both xml files
NB for clarity ( andother users) Picks =/= Arrivals. Arrivals are used in the event localizations. Picks may be or not associated to that solution

First conclusion/assumption:
- seiscompml to fdsnxml event files conversion is looking buggy

Actions:

• Replicate with the GeoNet conversion process

• Confirm the pick hickhock, confirm a Bug and level of it.

• Remediate

• Since this is not related to the archived event files (seiscompml ) most likely - this could be fixed "easily" --> bug reports and fix. @calum-chamberlain

FYI @elidana - this is looking like an fdsn event xml service question to address (to me)

dup2022p788472.xml.txt
dup2022p788472_fdsn.xml.txt

[salichon]

This may relate to thhis : https://github.com/GeoNet/fdsn/tree/main/cmd/fdsn-quake-consumer

[elidana]

@calum-chamberlain thanks a lot from my end as well for raising this.
And good catch @salichon about the issue being only on the FDSN service side of things! Definitively agree that this is reassuring.
@calum-chamberlain , have you noticed this issue only recently or is it something you have encountered since a while?
We have upgraded the event service about a month ago (main change from the user perspective was the addition of the event_type), so trying to pinpoint the timeframe might help with troubleshooting.

[calum-chamberlain]

@elidana I am pretty sure this is a recent thing, probably in the one-month timeframe, but I'm no certain. I haven't had to cope with it until this year at least.

[elidana]

thanks @calum-chamberlain , that's very useful to know!

[junghao]

@salichon @elidana
The QuakeMLs are generated by tool xsltproc (http://xmlsoft.org/xslt/xsltproc.html), not much about our own code.

You can try install xsltproc in your computer and run the command:

xsltproc sc3ml_0.11__quakeml_1.2.xsl 2022p788472.xml 

where the sc3ml_0.11__quakeml_1.2.xsl is provided by SeisComP3 here https://github.com/SeisComP3/seiscomp3/blob/master/src/trunk/libs/xml/0.11/sc3ml_0.11__quakeml_1.2.xsl

Have tested the command above and got the same result (having "duplicated pick").
Not sure if it's due to a bug (of xsltproc or the xsl file) , or it's the source data (sc3ml) makes it.

I guess we can file a bug to SeisCompP3 but seems need some elaboration about the event so probably not what I can do?

[salichon]

super @junghao thanks for that
note1 : This repository has been archived by the owner on Oct 14, 2022. It is now read-only.
we now shall point to https://github.com/SeisComP/common/tree/master/libs/xml/0.11 as point of reference and currently maintained repository.
and more generally to https://github.com/SeisComP/common/tree/master/libs/xml (0.12 is on the horizon .. :)

[junghao]

Thanks @salichon .

Based on this PR SeisComP/common@132fc95#diff-e60a95631541fd9f0ff3a585e41fc7d493aec710d6f75c3e0edc3d7891d5ff71R262 , not sure if the comment (we exclude picks already referenced in amplitudes) is our case? (If yes then seems it didn't fix)

[salichon]

@junghao I confirm the conversion prioducing the same output thanks
the only diffrence so far (with xsltproc) is the "agency name" from the repo it is "org.gfz-potsdam.de/geofon/"
so we would have to use option ""-stringparam ID_PREFIX smi:nz.org.geonet" instead i suppose.

the event 2022p pick id output for 2022p788472
gives 40 picks ID (composed of duplicated ones such BFZ) as opposed to the original 29 picks

Duplication as shown in that screenshot

This shows a "sort of" random duplication

PickID files qml1.txt for quakeml and scml1.txt for original are attached

qml1.txt
scml1.txt
early comments: What the damned!