Skip to content

Conversation

MarcelGeo
Copy link
Contributor

@MarcelGeo MarcelGeo commented Aug 15, 2025

Added ➕

  • integration to v2 push chunks mechanism - spliting some factory methods for UploadJob and methods in UploadQueueItem to v2 versions (we can discuss that, if it's ok)
  • ignore 409 error in push check_only to prevent stopping upload
  • added retry mechanism for chunks - 2 attempts after 5 seconds
  • added retry mechanism for sync command - compatible with v1 and v2push version using checks for status codes, server codes and response texts. This retry is doing n attempts every 5 seconds. Retry is doing if
    • 409 AntherUploadRunning, ProjectVersionExists
    • 429 - reate limit
    • for v1 if 400 and specific text in detail
  • introducing uploaded chunks cache which is used by v2 endpoints and cleaned by successful update
  • introducing LocalChanges class to handle dictionaries with changes (which nobody knows what's there) - it's used just in push methods to finalize push (time for improvement)
  • introducing server_features which will handle in future all feature flags from server
  • getting project info directly from v2 POST versions

Updated tests:

  • added remove only files test
  • added test for low level sync methods
  • added test for retry mechanism

  • not sure what to do with progress bar in sync command, as it's not possible to update maximum size (length) of that on the fly.
  • some tests will fail, becasue we are there using v2 api which is not in app.dev yet - chunks cache test is skipped

Chunks cache

Logging following to logs

2025-08-13 14:09:10,282 will upload 1 items with total size 92025-08-13 14:09:10,282 Starting upload chunks for project 459a8014-9c05-439d-9b6b-2687cfdfb4fa2025-08-13 14:09:10,283 Uploading /home/marcelkocisek/Documents/mm-workspaces/localhost/hejj7/hej3.txt part=02025-08-13 14:09:10,283 Chunk 76ade537-12f2-4e0e-b25b-c22929f10c5a already uploaded, skipping upload

Retry of sync_project

2025-08-15 13:36:32,558 Attempting sync process due to conflicts between server and local directory or another user is syncing.

Workflow
image

Introduce mapping based on chunk_id (generated by client) and server_chunk_id generated by server.
- +v1 is raising error just from push start (push_project_async), because there are not chunks caches
prevent conflcted copy fix
Copy link
Contributor

@varmar05 varmar05 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good 🚀

mc.post(
f"/v2/projects/{project_id}/versions",
{**data, "check_only": True},
{"Content-Type": "application/json"},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to set these headers here, as they are most of the time json?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this is how the post is created

if headers.get("Content-Type", None) == "application/json":
            data = json.dumps(data, cls=DateTimeEncoder).encode("utf-8")

But I can move it to some const

from .utils import is_versioned_file
from .common import MAX_UPLOAD_MEDIA_SIZE, MAX_UPLOAD_VERSIONED_SIZE

MAX_UPLOAD_CHANGES = 100
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe move next to others?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants