Poor quality, weak, old, outdated model selection algorithm on Artbot, models_to_load, models_to_skip

[Luke2642]

There seems to be a strange self-sustaining negative feedback loop going on. I think it's an Artbot UI problem as well as a horde model algorithm selection problem.

Bad, old, outdated, poor quality models have got a strong foothold on the horde. Why are the illustrious, cyberrealistic and other popular models not present in the models folder when I have top 10 and a civitai api key in the config?

1. The most popular models on civitai are badly represented.
2. Alphabetical ordering disproportionately affects model selected - why is AlbedoBase so popular, but not at all popular on civitai? And is it downloading v0.1... or the latest?
3. Ancient weak models like stable diffusion 1.5 and ICBINP 1.5 get downloaded and seem to account for a large number of jobs when their output quality is awful.

Manually hacking this seems a poor solution that will need regular checking. I want to offer the best and not waste energy on poor models.

With top 15 set, I now have:

models_to_skip:

• "pix2pix" # Not currently supported
• "SDXL_beta::stability.ai#6901" # Do not remove this, as this model would never work
• "A to Zovya RPG" # This model is known to cause problems with reGen
• "stable_diffusion"
• "BB95 Furry Mix"
• "BB95 Furry Mix v14"
• "ICBINP - I Can't Believe It's Not Photography"
• "Anything Diffusion"
• "Deliberate"
• "ALL SD21" # Stable diffusion 2.1 has bigger memory requirements than 1.5, so if your card cannot lift, it, disable it
• "AlbedoBase XL (SDXL)"
• "Anything v3"

Does anyone else have any thoughts on this?

[tazlin]

This is really a existential question about much larger issues, though perhaps indirectly and somewhat unintendedly. I think your concerns could be rephrased as follows:

How does a crowd sourced platform which is volunteer run (and whose generative image model offerings are determined at the whims of independent third parties) fairly and accurately decide what models to offer without heavy-handed and unilateral top-down methods?

I'd like to highlight the following as food for thought:

• Each individual worker operator has their own motivations and reasons for running the worker software and that may influence their decision to offer certain models (anime vs realistic vs furry, for example)
• Workers often have limited resources.
  • This includes disk space and bandwidth. SDXL models are 8gb a piece and are hosted by (slow) third parties using bandwidth we don't control and the AI-Horde team cannot afford to host it ourselves to improve this.
  • Having to frequently download, prune, or otherwise worry/think about downloading models is a burden on many operators, especially in view of the fact they are donating their time and electricity.
• The inclusion or exclusion of a given model to the officially allowed model list is fraught with subjectivity. Said differently, what exactly should the benchmark for quality be?
  • Is CivitAI popularity somehow more relevant than on-horde popularity?
  • There are a wide variety of use cases for the horde, some of which are niche. Should we interfere with current usage patterns just to create wider appeal?
  • Older models typically require more effort into getting results that the user intended. However, a large subset of workers can only offer older models, as newer (and heavier) models are beyond their system capabilities. Many anonymous users only get results by virtue of using an older model, otherwise, their requests would timeout while competing for the higher-capacity workers.
• Models are popular on the horde because end users make requests for those models.
• Prior community consensus by community stakeholders in the project determined that the top 10 models require protection and enjoy the privilege of not being modified in place nor removed without good reason.

As for your opening thesis here:

There seems to be a strange self-sustaining negative feedback loop going on.

I completely agree. However, I do not think there are simple solutions, or at least, engineering one may be possible but requires community censuses on what that would look like as I would be extremely remiss to unilaterally change this dynamic. Which, in fairness, this dynamic is largely an organic one based on the actual behavior of worker operators and end users, but I will agree that specific implementations (such as the TOP 10 behavior) influences this.

Workers are considered stakeholders in the project as you are donating compute. You can exercise your rights associated with that by coming to the discord, logging into the bot there (with /login), and starting a discussion in the #stakeholders channel, and I would also recommend joining the loomio so you can see the prior discussion there on a very similar past issue.

[Luke2642]

Thanks for the thorough reply! I do understand it's a community dynamic, but the "default behaviour" of the humans and the algoirthm is to blame here, because as a new worker, seeing ancient .ckpt files for outdated checkpoints appearing is a bad look, then watching the logs and seeing them getting used and knowing the outputs are gonna be awful is just sad.

I understand some of the horde are running hardware more suitable for SD1.5, but my cycles shouldn't be wasted on it? The default behaviour should factor that in surely?

Similarly, downloading and pruning absolutely should be a regular, automatic scheduled job for every worker on by default. A download cap per X would be a neat solution in the background thread.

I would say simply, yes, civitai popularity is key, but basic curation is needed too! You can see from the community that SDXL fractured into Pony, Illustrious, NAI, Noob, BigAsp, Animagine, before we even consider flux.

[tazlin]

I understand some of the horde are running hardware more suitable for SD1.5, but my cycles shouldn't be wasted on it? The default behaviour should factor that in surely?

So if the root of your problem is that you don't want to offer SD15 models, you can add ALL SD15 to your models_to_skip and you'll never offer a model of that kind unless you explicitly add it. I would like to gently reiterate my point that every worker operator has their own motivations and preconceptions about what they should and should not offer. Ultimately, this is the kind of decision making I am talking about. I recognize that you have this position about SD15, but not all workers do. And more to the point, the end users who make the decision about what model to use might not appreciate being told that the model choice they are making (which they may be very happy with) is somehow "wrong".

Please also keep in mind that SD15 generations reward a roughly proportional number of kudos less than other models so they are cheaper for end users. Further, with decent hardware, it is possible to do many hundreds to thousands of requests per hour doing only SD15, where only a fraction of that may be possible with other model baselines at higher resolutions. Delivering more images to more users, even if they are not of the absolute highest quality (and again, who gets to decide quality anyway?), and maintaining that kind of traffic to those models is an indication to me that there is merit in offering them.

Similarly, downloading and pruning absolutely should be a regular, automatic scheduled job for every worker on by default. A download cap per X would be a neat solution in the background thread.

This is somewhat easier said than done, though the only real blocker to it amounts to a willingness by someone to invest the engineering effort into it. See also #416 and #4 (which is I will admit is one of the earliest languishing issues).

I'd like to reiterate the point that this is an offered-for-free and operated by a not-for-profit project, for and by volunteers, and there is not a specific drive towards mass appeal or marketability - though I obviously do strive to make the project perform and be the best it can, I do think there is a great deal of subjectivity on what is and isn't "good". There are around 50 active image workers, many of which have months or even years of worker uptime. There are over 15k monthly unique active users and around 100k image requests per day. The dynamics you are identify are properties of their behavior. I am very cautious about making top-down changes that have major practical impacts as a result.

[tazlin]

Also, while its linked by proxy, I want to highlight that I have thought about and been aware of the problem you originally mention, seen here: #353, so I am not dismissing your concerns out of hand, I am only trying to highlight that any unilateral resolution is sure to upset someone and so any proposed solution will need community consensus.

[Luke2642]

Sorry, I didn't mean to be combative, it's a great project! As a worker I think you just want the feeling that you're helping users make something that is aesthetic and intentional. I'm sure it's a widely held opinion that the original SD1.5 model that is today still offered by 19 workers delivers neither, for 95% of generations. ICBINP is little better without skilled prompting, I would bet money most generations are bad.

I've read those other threads you linked, and the issue is deep!

So, yes, if the default template file had a comment suggesting putting ALL SD15 in models_to_skip, it would have addressed 50% of my initial confusion. The unusual popularity of a relatively unpopular model like "AlbedoBaseXL" could be addressed by nudging behaviour with UI tweaks in other projects, and tweaking download priorities in workers.

It seems to me, that the link is broken between workers and users in their UIs. Selecting a model is only part of it.

Purely because of economics, Civitai has a tighter feedback loop, as their generation interface and buzz pricing makes their economics work. There's something to learn from it!

For example, their defaults of 832x1216 which is 4% fewer pixels than 1024x1024 and in shape people actually want, so win win. Then, the 3 buzz pricebreak at 19 steps with Euler A, and a slight discount when batching, so 4 images for 10 buzz, supports this idea of economics reflecting compute.

For horde UIs, I think you could consider having a "simple mode", focused on enabling novice users to get aesthetic results even with poor prompting skills. Simple mode positive prompts should be limited to 250 characters. They shouldn't even be choosing a model, they should be choosing a 'style' - anime, photoreal, etc. Each month or two choose three hyper models by community curation concensus to cover 90% of novice user simple mode generation needs, ensure all the horde has those models, and push heavily in the upstream to be set as default in UIs.

It can use all the same tricks as civitai, portrait and landscape resolutions, but also decent negative prompts, steps and cfg scale set to guarantee a baseline quality. The aim is to use the resources more efficently, so all users can experience faster, more reliable and higher quality generations. Something like 832x1216, cfg 4, 8 steps. On a good hyper sdxl model this will be faster, higher resolution, higher quality and better prompt following than 25 steps on an old sd1.5 model.

This kind of smarter allocation of resources would be more efficient for workers, improve the average generation quality, and faster queues for "advanced mode" to serve users that want to select all the settings and choose any model.

[Unit1208]

This is essentially what AIScribbles does, providing a simplified generation interface for those who don't want to have to worry about prompting specifics or model details. Because of how diverse and decentralized the Horde's UIs are, there is no one person who can decide that all horde UIs must have a simple and an advanced mode. That would be up to the UI's creator to decide what features to show and which to hide from the user to simplify their experience.

This brings up a deeper facet of the horde: There is no single "AI Horde experience" that every user is exposed to. Some may only use simplified UIs like AIScribbles, not knowing that they might be able to have more control over their prompting. Others may use Artbot or HordeNG, not knowing that they are getting bad results due to their lack of knowledge of prompting. While CivitAI can present the same front-facing user interface to everyone who uses their service, the Horde doesn't have a single frontend for everyone.

With that being said, many UI creators are in the DB0 discord server and can work together to providing better defaults for people to use the horde more efficiently.

[Momi-V]

@tazlin
As a worker there might be some value in having a "BEST" models option to "keep up with the times".

Users will often choose whatever model gives them results back fastest, and workers might not want to check and update what models they offer every week or two (and might indeed just host whatever model has the most users).

Having a ranking system, or, since creating a definitive rank for models might be difficult "quality tiers" to prefer generally, on average "better" models of some sizes to "worse" ones might have some value.

So if someone sets SD2.1 best 10 and there are say 100 different SD2.1 models, with quality scores ranging from 1-5 they get a TOP N subset weighted towards quality 1 models with older, less performant models being de-prioritized.

That way model maintainers might be able to steer the horde slightly instead of having to rely on (probably uninformed) users to request different models by actively selecting a "lower ranked" model or having Node operators peer through their config every weekend to manually adjust what they offer.

Top N can still be used, but without a significant amount of people offering ALL, negative self reinforcing effects through sampling biases can indeed take effect.

[Luke2642]

I stopped running a default worker after modifying the code to observe the quality of generations, and seeing how atrociously poor it was. Something like 1 in 300 images had some artistic merit after leaving it running overnight.

The problem is threefold: Models, Skill and Subject.

As an experiment, I further modified the code to force the changes I suggested above upon the users lucky enough to use my worker for a short duration, and attempted to fix all three.

To fix the model, regardless of what the request was, it would use the same good ~8 step SDXL model, a good sampler, and apply some excessive parsing regex to crop down the prompts, remove an awful lot of stuff and replace the negative.

The artistic subject is the most challenging and controversial to improve, but I attempted two ways:

1. I used a model that follows prompts much better than most of the options on the horde.
2. I augmented the prompts to make the compositions more varied, and more colours, lighting, shadows, etc. But, it was very much one style fits all. More intelligent parsing code, could apply a variety of artistic styles.

The skill gap was the easiest to fix, with good settings, generations were much better, a consistently high technical quality, good texture, detail, correct anatomy, good hands, and returned in a couple of seconds instead of 30. It could serve a lot more requests!

If I get around to parsing the prompt properly and modifying the code further to keep two models in memory, a photorealistic model and anime model, it would be worth putting in a fork for anyone that wants to "pollute" the horde with unexpectedly better results for 99% of requests, based on what is undoubtedly an extremely controversial philosophy of AI art outcomes

Although heavy-handed and unilateral, it's not a top-down solution though eh? ;-)

[tazlin]

@Luke2642

While I disagree with forcibly changing user requests, I can see an optional "improve_request" sort-of flag being added that does roughly what it is you are saying. If you are willing to PR a change to the worker to this effect (for it to support such a flag using your methods) we could go through the process of seeing what that could look like as an official feature.

[arcaria]

@Luke2642
What's your worker's name?