Add florence 2 #560

MadeWithStone · 2024-08-01T02:54:13Z

Description

This PR adds support for florence2 in workflows. It allows for a wide variety of computer vision tasks with prompts and finetuned loras. The node returns both raw outputs from the model, and parsed supervision detections.

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
This change requires a documentation update

How has this change been tested, please provide a testcase or example of how you tested the change?

hosting the inference server on a gpu virtual machine (gcp ce) and connecting to the server using roboflow.com as the client

…into add-florence-2

CLAassistant · 2024-08-01T02:54:18Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ MadeWithStone
❌ roboflowmaxwell
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

inference/models/florence2/florence2.py

inference/core/workflows/core_steps/models/foundation/florence_2.py

…into add-florence-2

inference/models/florence2/florence2.py

PawelPeczek-Roboflow · 2024-08-13T14:29:06Z

inference/core/workflows/core_steps/common/serializers.py

@@ -41,8 +41,10 @@ def serialise_sv_detections(detections: sv.Detections) -> dict:
        detection_dict[X_KEY] = x1 + detection_dict[WIDTH_KEY] / 2


The nature of this change makes me worrying about integrity of the whole thing after the change.
If you don't have confidence, then output is not really sv.Detections, the same with class name being empty string. Those problems must be addressed at the block level

ok, just realised that there is this from_lmm(...) constructor for sv.Detections, but still - objects produced by the block will be incompatible with other blocks due to optional property

The issue is that different vision tasks have different information. I dont really see how this is different than a an sv detection for bounding boxes having a box vs. a detection for classification or detection for segmentation. We already support optional properties in supervision and multimodal models are bound to have these incomplete detections so i dont really see a way around it. I think all blocks should assume incompleteness and we may consider a separate method for validating supervision blocks for necessary info on the receiving end rather than generation end of a supervision detection.

PawelPeczek-Roboflow · 2024-08-13T14:30:44Z

inference/core/workflows/core_steps/models/foundation/florence_2.py

+]
+
+
+class BlockManifest(WorkflowBlockManifest):


must be adjusted to be aligned with main - we got rid of asyncio and introduced versioning patterns. Please apply (take a look at other blocks, in doubts I can help)

Q: What happens when LMM output does not match expectations - any error handling would be possible / error would be raised / empty output will be yielded?

not sure what you mean by doesnt match expectation. Microsoft has already handled output validation and error handling so we can assume their preprocessor returns valid output data or empty data.

PawelPeczek-Roboflow · 2024-08-13T14:33:20Z

inference/core/workflows/core_steps/models/foundation/florence_2.py

+            return await self.run_locally(
+                images=images, vision_task=vision_task, prompt=prompt, model_id=model_id
+            )
+        elif self._step_execution_mode is StepExecutionMode.REMOTE:


I am not up to date with new models that are to be supported on the platform, in priv please provide me the info about models to be hosted on the platform and the way on how we want to host them - in particular - are we going to let this model to be run only locally / do we let people train the model etc.

Like paligemma, florence 2 requires intensive compute to generate results in a reasonable amount of time so its only supported on local/dedicated deployments. we dont have a dedicated endpoint at this time. we currently support training loras outside of roboflow and uploading to the platform (and loading through this block)

PawelPeczek-Roboflow · 2024-08-13T14:41:47Z

inference/core/workflows/core_steps/models/foundation/florence_2.py

+        return [
+            OutputDefinition(name="parent_id", kind=[BATCH_OF_PARENT_ID_KIND]),
+            OutputDefinition(name="root_parent_id", kind=[BATCH_OF_PARENT_ID_KIND]),
+            OutputDefinition(name="image", kind=[BATCH_OF_IMAGE_METADATA_KIND]),


this output is no longer used given sv.Detections are there under predictions

raw output is the raw text from the model, structured output is the formatted dict generated by the microsoft provided output text processor, predictions is sv detections processed by our from_lmm method in supervision

PawelPeczek-Roboflow · 2024-08-13T14:42:39Z

inference/core/workflows/entities/types.py

@@ -313,6 +313,24 @@ def __hash__(self) -> int:
    docs=DETECTION_KIND_DOCS,
 )

+DETECTION_KIND_DOCS = """


could you explain why this additional kind is needed?

each sv detection wont necessarily fall exactly into one of the current types i.e. missing class, class_id, etc so I created the batch of Detection kind as a catch all

PawelPeczek-Roboflow

For the PR to be approved:

integration tests to be created - as must have I would like to see how output is practically used by other block
there is high chance that UQL operations on detections are incompatible with sv.Detections produced as output of this step
deployment on hosted platform to be clarified

PawelPeczek-Roboflow · 2024-08-30T08:43:09Z

Hi there,
we need to push that forward, I need orientation to make some decisions:

@MadeWithStone - are you willing to continue the task
@probicheaux - what is the status of Florence 2 model on the platform, in particular:
- what kind of pre-trained models we support
- where Florence 2 could run - my brief analysis indicated that only on GPU server builds - if so, the block itself is not practically useful, even potentially harmful once random users got hyped, enter Workflows and see something is broken, without proper info on how to run
- do we have plan on making good docs about this VLM?
- do we have plan to onboard the model on the inference platform? what about CPU builds?

PawelPeczek-Roboflow · 2024-09-19T20:14:41Z

shall we close in favour of #661

roboflowmaxwell added 16 commits July 16, 2024 22:09

add florence 2 preliminary workflow block

e20660f

add florence 2 preliminary workflow block

0ce69b7

Merge branch 'add-florence-2' of https://github.com/roboflow/inference …

4a7d7eb

…into add-florence-2

add florence 2 preliminary workflow block

a3317a4

trying version

cc546db

add florence 2 preliminary workflow block

93c7889

add florence 2 preliminary workflow block

13af7dc

trying version

ca5b33b

Merge branch 'add-florence-2' of https://github.com/roboflow/inference …

c48f90b

…into add-florence-2

working through inference api

c5f0820

trying to get florence 2 pretrained weights to work

d36c265

Merge branch 'add-florence-2' of https://github.com/roboflow/inference …

1fd3a47

…into add-florence-2

model works with new generation params

451062a

get florence2 working

20e45f4

remove unnecessary prints and imports

0ed6aca

remove remote support

966e640

MadeWithStone requested review from PawelPeczek-Roboflow, grzegorz-roboflow, yeldarby, probicheaux and hansent as code owners August 1, 2024 02:54

run formatter

8117c3e

probicheaux reviewed Aug 1, 2024

View reviewed changes

inference/models/florence2/florence2.py Outdated Show resolved Hide resolved

MadeWithStone added 2 commits August 1, 2024 15:16

Merge branch 'main' into add-florence-2

f52f6e0

Merge branch 'main' into add-florence-2

399625e

yeldarby reviewed Aug 5, 2024

View reviewed changes

inference/core/workflows/core_steps/models/foundation/florence_2.py Outdated Show resolved Hide resolved

inference/core/workflows/core_steps/models/foundation/florence_2.py Outdated Show resolved Hide resolved

roboflowmaxwell added 3 commits August 6, 2024 22:05

fix task options

9590187

Merge branch 'add-florence-2' of https://github.com/roboflow/inference …

091fb2b

…into add-florence-2

add batch detections and remove wildcard

54136a6

roboflowmaxwell and others added 3 commits August 7, 2024 00:01

fix output structuring

8366954

finish formatting and fix style

88d4dd9

Merge branch 'main' into add-florence-2

1b4596d

MadeWithStone requested review from yeldarby and probicheaux August 7, 2024 06:02

roboflowmaxwell added 2 commits August 7, 2024 21:35

update lora florence to support parsed output

01cc989

Merge branch 'add-florence-2' of https://github.com/roboflow/inference …

6baaeb6

…into add-florence-2

probicheaux requested changes Aug 8, 2024

View reviewed changes

inference/models/florence2/florence2.py Outdated Show resolved Hide resolved

inference/models/florence2/florence2.py Outdated Show resolved Hide resolved

roboflowmaxwell added 2 commits August 8, 2024 01:40

update florence2 and transformers using better practices

f2c4cd7

fix duplication mistake

72b8f9c

PawelPeczek-Roboflow reviewed Aug 13, 2024

View reviewed changes

PawelPeczek-Roboflow requested changes Aug 13, 2024

View reviewed changes

MadeWithStone and others added 3 commits August 19, 2024 11:44

Merge branch 'main' into add-florence-2

e90cff5

update for v16 changes

cef187b

florence 2 working on latest inference

b952d5f

PawelPeczek-Roboflow added release 0.18.0 release 0.19.0 and removed release 0.18.0 labels Sep 3, 2024

PawelPeczek-Roboflow added release 0.20.0 and removed release 0.19.0 labels Sep 16, 2024

PawelPeczek-Roboflow closed this Sep 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add florence 2 #560

Add florence 2 #560

MadeWithStone commented Aug 1, 2024

CLAassistant commented Aug 1, 2024 •

edited

Loading

PawelPeczek-Roboflow Aug 13, 2024

PawelPeczek-Roboflow Aug 13, 2024

MadeWithStone Aug 19, 2024

PawelPeczek-Roboflow Aug 13, 2024

PawelPeczek-Roboflow Aug 13, 2024

MadeWithStone Aug 19, 2024

PawelPeczek-Roboflow Aug 13, 2024

MadeWithStone Aug 19, 2024

PawelPeczek-Roboflow Aug 13, 2024

MadeWithStone Aug 19, 2024

PawelPeczek-Roboflow Aug 13, 2024

MadeWithStone Aug 19, 2024

PawelPeczek-Roboflow left a comment

PawelPeczek-Roboflow commented Aug 30, 2024

PawelPeczek-Roboflow commented Sep 19, 2024

		@@ -41,8 +41,10 @@ def serialise_sv_detections(detections: sv.Detections) -> dict:
		detection_dict[X_KEY] = x1 + detection_dict[WIDTH_KEY] / 2

Add florence 2 #560

Add florence 2 #560

Conversation

MadeWithStone commented Aug 1, 2024

Description

Type of change

How has this change been tested, please provide a testcase or example of how you tested the change?

CLAassistant commented Aug 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

PawelPeczek-Roboflow left a comment

Choose a reason for hiding this comment

PawelPeczek-Roboflow commented Aug 30, 2024

PawelPeczek-Roboflow commented Sep 19, 2024

CLAassistant commented Aug 1, 2024 •

edited

Loading