Support dtype conversion from processor output

[donin1129]

Hello team,

First off, I want to express my sincere appreciation for the excellent work on the library! Our team has genuinely enjoyed using it, and we find it incredibly valuable.

We've recently encountered a specific scenario where we believe an enhancement to the TransformerMultiModal class could significantly benefit users, particularly concerning dtype handling.

The Challenge:
Currently, within the _prepare_model_inputs method (located here: https://github.com/dottxt-ai/outlines/blob/main/outlines/models/transformers.py#L658), we've identified a need for a mechanism to convert the data type (dtype) of inputs. While we understand that general dtype handling is typically managed by the processor, certain hardware-specific data types, such as bfloat16 (which is crucial for our team's current setup), ideally shouldn't be part of the processor's call function due to performance or architectural considerations.

We'd love to hear your thoughts and support on this. Would you be open to integrating a feature that allows for explicit dtype conversion at this stage, or perhaps you have an alternative approach in mind that would address this need effectively?

Thank you.

[RobinPicard]

Hi @donin1129! Thanks for creating a discussion, I'm really glad to hear that outlines has been useful to you and enjoyable to work with.

I have never encountered this issue, but that sounds like something important and we're definitely willing to look into it. Do you think you could open a draft PR with a super basic implementation (no tests or docs needed) of what you'd like the library to do? From there we could discuss how we can make it fit best in our current architecture and what interface we should propose.

[donin1129]

Hi @RobinPicard , thnak you very much for your quick feedback. Here is my submitted PR. #1760.
And a possible use case could be:

outline_model = TransformersMultiModal(model, processor)
outline_model.device_dtype = torch.bfloat16
output = outline_model(input_dict, type_hint)

[RobinPicard]

Thanks! An alternative would be to pass it through the inference kwargs, but I feel it makes more sense as a model attribute that can be set at initialization. What do you think @rlouf?

[rlouf]

I think it would make sense in this example to have TransformersMultiModal(model, processor, device_dtype=torch.bfloat16)

[donin1129]

Yes. thanks for the response. I was a bit hasty and forgot to add the other recommended use case from your front page. I think the proposed change will cover both use case:

outline_model = TransformersMultiModal(model, processor, device_dtype=torch.bfloat16)
output = outline_model(input_dict, type_hint)

model = outlines.from_transformers(
AutoModelForCausalLM.from_pretrained(MODEL_NAME, device_map="auto"),
AutoTokenizer.from_pretrained(MODEL_NAME)
)
model .device_dtype = torch.bfloat16
output = outline_model(input_dict, type_hint)

[RobinPicard]

I've opened a PR with tests and documentation updated: #1762