List view
We currently rely on a patchwork of python dependency handling in development and actual deployments with some installation going through `pip` with: `pip install -r requirements.txt` `pip install -r recommended.txt` `pip install -r local-requirements.txt` and some handled with explicit versioned package installation as in `pip install 'paramiko==VERSION'` `pip install 'openstack==VERSION'` ... in the `Dockerfile`s in docker-migrid. We also have overlap in `requirements.txt` and `recommended.txt`, which only make things more complex. We decided to restructure the existing requirements files into a single main one with all *mandatory* requirements needed for _all_ migrid installs, plus individual `requirements-FEATURE.txt` extensions that _only_ include requirements to enable a particular FEATURE like sftp, jupyter, cloud, etc. That will allow us to generally use `pip install -r requirements.txt` followed by `pip install -r requirements-FEATURE.txt` for each enabled FEATURE and hopefully reduce the requirements files overlap to a minimum. We will want to then gradually move to use that structure both in the development/testing use e.g. in containerized unit tests and linting as well as in the docker-migrid `Dockerfile`s. Please note that we will stick to installing various non-python library dependencies (openssl, cracklib, etc.) as well as the python dependencies we _can_ avoid maintaining ourselves through the Linux distro channels. As long as our `requirements*.txt` files don't directly require versions conflicting with those OS package versions, that will work fine and leave the OS-installed version alone. In other words we _just_ have to avoid `requirements*.txt` specifying that package X *must* be a version bigger than what the OS provides for those packages we intend the OS package to provide.
No due date•0/1 issues closedOur existing sharelinks already come with a number of convenient features like easy data exchange with read-only, read-write or write-only support and efficient access through SFTP/FTPS/WebDAVS for the read+write flavor. This allows the use of sharelinks as a non-personal data store e.g. from instruments or lab computers. Yet, we could extend these capabilities to cover additional use cases and improve the ease of use. This milestone gathers various ideas and design plans with a certain amount of overlap in that field. * Efficient download of read-only shared folders * Efficient upload to write-only shared folders * Limited access to sharelinks only with SFTP/SSHFS using key auth (no https, no password) * Simple web access to sharelink folders with huge amount of files (without timeout) * Single file shares with easy download using the proper name and type * Sharelinks with custom passwords * Time-limited sharing * Longer sharelink IDs for stronger protection against brute-force guessing * Optional use of read-only backend storage (bind) mount for SFTP access to RO sharelinks? * Optional tweak or filter of `listdir` for write-only sharelinks to return empty or allow listing? * Optional tweak to allow any `chattr`/`chmod` without actual effect for read-only sharelinks?
No due date•6/9 issues closedWe currently have a number of clean up tasks implemented as cron jobs. This milestone is about implementing a dedicated janitor service (say, `grid_janitor.py`) , which apart from handling our current cron clean up tasks should also take over the vgrid cache updates and check that pending requests are valid and mark or simply reject them if not. The current client-driven concurrent vgrid cache updates with http timeout after 300s are fragile and inefficient. They should really be replaced by a single handler without such time limits. The janitor would be an obvious place for taking care of the updates in a simple and efficient way. It can be adjusted to use memory caching of owners and members in `mig_system_run` for better performance when netfs is a bottleneck. Site operators report that there's quite a bit of overhead in the existing account handling process and in several cases simple errors or mismatches render the process moot midway through. E.g. if the user already has an account with a slightly different ID or if trying to renew an expired account and not providing the existing password to authenticate it. The janitor could therefore check pending requests and detect password mismatch, ID clash and invalid peers for starters. Having password checks in a detached service like that rather than during form submission also allows delaying the check/reject long enough for password guessing that way to remain infeasible. On-going work to simplify account renewal and password change from the Accounts page will reduce some of the hurdles in account handling but this service would still help reduce the remaining nuisances. We expect the work to contain at least three iterations: 1. Basic janitor service focusing on integration in the migrid stack and some low hanging fruits regarding account request pruning (PR #322). 2. Additions to replace the need for the separate clean up cron jobs. Maybe some polish in follow-up to first iteration (PR #322 + XXX). 3. Additions to replace the currently client-driven vgrid/user/people cache upgrades with a janitor-only version to eliminate races and overhead (PR 341)
No due date•14/21 issues closedWe have decided to begin moving scripts from `./`, `./mig/` and `./mig/server/` into the new FHS-compliant and easier to find `./bin/` at the root. Similarly the service launchers in `./mig/server/` and any other system or service executables will migrate into `./sbin/` . In relation to the move we have also decided to introduce a `./mig/lib/` for hosting supporting code modules or collections of such in sub-directories. We will use the opportunity to clean up and lint the modules migrating there and add github actions to monitor for regressions once there.
No due date•3/4 issues closedWith Python officially unsupported upstream and vendors only providing important security fixes for it we need to migrate completely to Python3 ASAP. The core code base is mostly ported in terms of the backends and facilities related to general data handling, but especially the original grid compute parts are less tested on Python3. We're testing and weeding out any remaining issues on development systems and a few smaller production sites. Any issues related to the original complete grid compute stack should be gradually tested and handled here.
Due by December 31, 2025•10/11 issues closedWe have gradually phased in OpenID Connect authentication support in the migrid stack and will continue in that direction. There are some rough corners and missing pieces in relation to completing the move away from OpenID 2.0 completely. This milestone tries to sum up and work as an umbrella for the associated tasks. * The built-in _external_ user account auth handling implemented in grid_openid is OpenID 2.0 - we need a corresponding grid_openidc service * Jupyter sessions fail ("Protected Location") on OpenID Connect (e.g. `Start DAG`) unless also logged in on OpenID 2.0. * OpenID Connect login currently does _not_ auto-renew _local_ user accounts like OpenID 2.0 logins does. * Explicit sign up should no longer be necessary for _local_ users with OpenID Connect * Log out may or may not be completely supported - in particular with WAYF OpenID Connect * Further testing of _local_ user OpenID Connect sign up and handling of affiliation+role values is needed
Overdue by 1 month(s)•Due by September 1, 2025•11/11 issues closedWe are in the process of updating the user interface and polish various details both in relation to user feedback and to address our own findings. This milestone is an umbrella to group all such user interface updates and improvements.
No due date•10/14 issues closedHaving the full X509 distinguished name as primary user ID and a slightly modified version for the on-disk home directory made sense back when migrid started and relied on user certificates, but with the introduction of other authentication methods etc. it has become ever more clear that it introduces artificial limitations and complicates quite a few things in the stack. We're looking into reworking the handling of IDs and home directories to sever that rigid binding. Changing it now of course is a bit tricky because quite a few places in the code and Apache setup more or less clearly make assumptions about it being so. Still it makes a lot of sense to proceed to allow things like easy adjustment of user affiliation fields without e.g. a costly full home directory renaming.
No due date•3/4 issues closedWith Python officially unsupported upstream and vendors only providing important security fixes for it we need to migrate completely to Python3 ASAP. The core code base is mostly ported in terms of the backends and facilities related to hosting general research data. We have basically ported the components related to hosting sensitive data as well, but it needs more testing to discover and fix any remaining issues. We're testing and weeding out any remaining issues on development systems and will continue with production sites. Any issues related to the basic research data and the original complete grid compute stack should be gradually tested and handled in the separate milestones for those parts specifically.
No due date•46/48 issues closedWith Python officially unsupported upstream and vendors only providing important security fixes for it we need to migrate completely to Python3 ASAP. The core code base is mostly ported in terms of the backends and facilities related to hosting general research data. We're testing and weeding out any remaining issues on development systems and a few smaller production sites. Any issues related to the original complete grid compute stack should be gradually tested and handled in the separate milestone for that part specifically.
No due date•92/102 issues closed