Bug 1990742 - Ingest perfherder_data from JSON artifacts instead of parsing logs #8997

junngo · 2025-09-26T14:18:01Z

Currently, Treeherder ingests performance data (PERFHERDER_DATA:) by parsing raw logs.
This patch supports reading data from the perfherder-data.json artifact instead.
For now, both the existing log parsing and the new JSON ingestion run in parallel to maintain compatibility.

bugzilla :https://bugzilla.mozilla.org/show_bug.cgi?id=1990742

gmierz · 2025-09-29T12:34:19Z

treeherder/log_parser/tasks.py

    return artifact_list
+
+
+def post_perfherder_artifacts(job_log):


@junngo I think it would be better for us to put this into a separate area. This folder seems to be specifically for parsing logs, but we're parsing JSONs instead. What do you think about having this task defined here in the perf directory? https://github.com/mozilla/treeherder/blob/505ad6b4047f77fc3ecdea63e57881116340d0fb/treeherder/perf/tasks.py

@gmierz Splitting the code is a great idea. Creating a separate file under the code directory [0] looks good to me. It feels more cohesive to put it there, since the log parsing [1] also lives in that folder.
Please consider my opinion and feel free to tell me about the directory location.

[0]
https://github.com/mozilla/treeherder/tree/505ad6b4047f77fc3ecdea63e57881116340d0fb/treeherder/log_parser
[1]

treeherder/treeherder/log_parser/artifactbuildercollection.py

Line 85 in 505ad6b

with make_request(self.url, stream=True) as response:

I added the new file based on your feedback. It seems more suitable since the JSON artifact isn’t part of the log parsing process :)

gmierz · 2025-09-29T12:43:47Z

treeherder/etl/perf.py

+                existing_replicates = set(
+                    PerformanceDatumReplicate.objects.filter(
+                        performance_datum=subtest_datum
+                    ).values_list("value", flat=True)


I'm guessing this is happening because of duplicate ingestion tasks (log, and json). I think we should find a way to default to using the JSON if they exist, and ignore the data we find in the logs. Maybe we could have a list of tests that we start with for testing this out? I'm thinking we could start with these tasks since the data they produce is not useful so any failures won't be problematic: https://treeherder.mozilla.org/jobs?repo=autoland&searchStr=regress&revision=6bd2ea6b9711dc7739d8ee7754b9330b11d0719d&selectedTaskRun=K87CGE6IT1GHl6wD4Skbyw.0

Exactly, log parsing and the JSON file feature are both active right now, so I handled the duplication.
I’ll revert that, add an allowlist, and only call _load_perf_datum for whitelisted tests when needed.

treeherder/etl/perf.py

junngo · 2025-10-01T03:39:42Z

ID	Framework	Enabled	Suites
1	talos	true
2	build_metrics	true	compiler warnings, compiler_metrics, decision ...
4	awsy	true
5	awfy	false
6	platform_microbench	true
10	raptor	true
11	js-bench	true
12	devtools	true
13	browsertime	true	constant-regression ...
14	vcs	false
15	mozperftest	true
16	fxrecord	true
17	telemetry	true

I have a list of frameworks generated locally by django code.
It would be good to gradually reflect the less important framework-suite mappings one by one.

[0]
compiler warnings: https://firefoxci.taskcluster-artifacts.net/NE-naCeqSyenKogxu0nD4Q/0/public/build/perfherder-data-building.json
compiler_metrics: https://firefoxci.taskcluster-artifacts.net/P1T_HaXURD-r59ymlz5GWA/0/public/build/perfherder-data-compiler-metrics.json
decision: https://firefoxci.taskcluster-artifacts.net/OKsoq3lARpCjUhwVjqDddA/0/public/perfherder-data-decision.json

junngo

note:

# treeherder/etl/jobs.py
parse_logs.apply_async(queue=queue, args=[job.id, [job_log.id], priority])

~~I considered splitting the queues, but decided to keep using the existing ones to avoid code duplication and increased complexity.~~

https://github.com/mozilla/treeherder/pull/8997/files#diff-937b3e21ad52eec5277a7f52f51572348a072addafb88a049f9fe302ae437e76R369

junngo · 2025-10-07T12:43:52Z

Hi there :) I updated the code.
There is log parsing feature. I didn’t modify the existing log parsing feature. Instead, I created the new queue and task for handling the perfherder-data.json artifacts. I separated the processing of logs and perfherder-data.json artifacts so that they can run on different queues.

treeherder/perf/tasks.py

gmierz

Great start @junngo! It looks like we're getting close :)

gmierz · 2025-10-07T14:18:37Z

treeherder/etl/jobs.py


+        job_log_name = job_log.name.replace("-", "_")
+        if job_log_name.startswith("perfherder_data"):
+            _schedule_perfherder_ingest(job, job_log, result, repository)


Instead of calling the schedule function here, we should call it in the _load_job method similar to where we call the _schedule_log_parsing function.

treeherder/etl/perf.py

gmierz · 2025-10-07T14:21:17Z

treeherder/perf/tasks.py

+        )
+
+    first_exception = None
+    for job_log in job_logs:


It looks like this is parsing the logs, but this new task should only be responsible for handling the JSON artifacts.

Thanks for the review :)
(I thought the purpose of JobLog table was as follows.) The job_logs variable is built from the JobLog table, but that table isn’t just for raw log parsing. It’s a generic job reference table that also tracks artifacts like live_backing_log and perfherder-data-artifact.json and so on.
We store references to whatever needs further processing there, and then different Celery queues pick them up and handle them.
I agree the wording around job_logs could be confusing, so I’ll rename things to make it clear!
If you have any other feedback or ideas, I’d be happy to hear them.

treeherder/etl/perf.py

gmierz · 2025-10-09T12:54:13Z

treeherder/perf/tasks.py

+
+    first_exception = None
+    for job_artifact in job_artifacts:
+        job_log_name = job_artifact.name.replace("-", "_")


nit: change this to job_artifact_name

gmierz · 2025-10-09T12:54:54Z

treeherder/perf/tasks.py

+
+        if job_artifact.status not in (JobLog.PENDING, JobLog.FAILED):
+            logger.info(
+                "Skipping ingest_perfherder_data for job %s since log already processed.  Log Status: %s",


nit: "since artifact already processed."

gmierz

Looking a lot better now :) a few questions/minor things below. I think the major thing is where we're checking the should_ingest stuff.

gmierz · 2025-10-14T20:06:50Z

treeherder/etl/jobs.py

    )

-    log_refs = job_datum.get("log_references", [])
+    log_refs = [


Do you think we could split this out of log_refs and make some additional code to handle the JogLog creation for those artifacts below the log ones? e.g.

if perf_refs: for artifact in perf_refs: ... _schedule...

Maybe some of the code could be generalized here too.

Okay, I split log_refs and perfherder_data_references and updated it :)

treeherder/etl/perf.py

gmierz · 2025-10-14T20:16:25Z

treeherder/perf/ingest_data.py

+
+    try:
+        serialized_artifacts = serialize_artifact_json_blobs(artifact_list)
+        store_job_artifacts(serialized_artifacts)


Could we call store_performance_artifact here directly instead of going through the store_job_artifacts method?

Yes, I called the store_performance_artifact method directly instead of store_job_artifacts method.

gmierz

r+ changes look great to me now, great work @junngo!

…arsing logs

junngo · 2025-10-16T13:00:37Z

docker/entrypoint_prod.sh

+elif [ "$1" == "worker_perf_ingest" ]; then
+    export REMAP_SIGTERM=SIGQUIT
+    exec newrelic-admin run-program celery -A treeherder worker --without-gossip --without-mingle --without-heartbeat -Q perf_ingest --concurrency=7


@gmierz
In production, we might need to run a worker_perf_ingest process for the perf_ingest queue.
But I’m not fully sure whether this entrypoint is actually required.

junngo marked this pull request as draft September 26, 2025 14:18

gmierz self-requested a review September 29, 2025 12:27

gmierz reviewed Sep 29, 2025

View reviewed changes

junngo force-pushed the ingest-perfherder-data branch from 34855c7 to 26bc32d Compare September 30, 2025 14:44

junngo marked this pull request as ready for review September 30, 2025 14:44

junngo requested review from beatrice-acasandrei and esanuandra as code owners September 30, 2025 14:44

junngo commented Sep 30, 2025

View reviewed changes

treeherder/etl/perf.py Show resolved Hide resolved

junngo commented Oct 3, 2025

View reviewed changes

junngo force-pushed the ingest-perfherder-data branch from 26bc32d to 7ec7ee8 Compare October 7, 2025 12:29

junngo commented Oct 7, 2025

View reviewed changes

treeherder/perf/tasks.py Show resolved Hide resolved

junngo force-pushed the ingest-perfherder-data branch from 7ec7ee8 to b29d246 Compare October 7, 2025 13:58

gmierz requested changes Oct 7, 2025

View reviewed changes

junngo commented Oct 7, 2025

View reviewed changes

treeherder/etl/perf.py Show resolved Hide resolved

junngo force-pushed the ingest-perfherder-data branch 2 times, most recently from 69ce6a2 to c319a67 Compare October 9, 2025 11:22

gmierz reviewed Oct 9, 2025

View reviewed changes

junngo force-pushed the ingest-perfherder-data branch 2 times, most recently from 3f6fcf6 to 845ee84 Compare October 14, 2025 14:48

gmierz requested changes Oct 14, 2025

View reviewed changes

junngo force-pushed the ingest-perfherder-data branch from 845ee84 to cb5b351 Compare October 15, 2025 15:47

gmierz approved these changes Oct 15, 2025

View reviewed changes

Bug 1990742 - Ingest perfherder_data from JSON artifacts instead of p…

5b4abc0

…arsing logs

junngo force-pushed the ingest-perfherder-data branch from cb5b351 to 5b4abc0 Compare October 16, 2025 12:54

junngo commented Oct 16, 2025

View reviewed changes

gmierz merged commit af7211c into mozilla:master Oct 20, 2025
6 checks passed

Bug 1990742 - Ingest perfherder_data from JSON artifacts instead of parsing logs #8997

Bug 1990742 - Ingest perfherder_data from JSON artifacts instead of parsing logs #8997

Uh oh!

Conversation

junngo commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

junngo commented Oct 1, 2025

Uh oh!

junngo left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

junngo commented Oct 7, 2025

Uh oh!

Uh oh!

gmierz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

junngo Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmierz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gmierz left a comment

Choose a reason for hiding this comment

Uh oh!

junngo Oct 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

junngo commented Sep 26, 2025 •

edited

Loading

junngo left a comment •

edited

Loading

junngo Oct 8, 2025 •

edited

Loading

junngo Oct 16, 2025 •

edited

Loading